felhom-controller/CLAUDE.md

# CLAUDE.md — Project Instructions for Claude Code (`felhom-controller`)

> Read automatically at session start. Stable orientation only — **current state lives in
> `CONTEXT.md` and the top of `CHANGELOG.md`**, never here. Cross-repo orientation: workspace-root
> `/mnt/5_hdd/felhom.eu/git/CLAUDE.md`.

!!! IMPORTANT !!!
- Always update CHANGELOG.md whenever you modified the code, and pushed to git!!
- IF controller feature changed (new/modify/remove) always update the relevant part of controller/README.md with the architectural change!!

## Project overview

Felhom is a managed home-server business for Hungarian customers. This repo contains the
**felhom-controller** — the Go application that manages Docker Compose stacks inside each customer
LXC guest via a Hungarian-language web dashboard.

Read in this order:
- **`REUSE.md`** — before writing new code (canonical helpers, patterns, traps, seams).
- `CONTEXT.md` — current project state, decisions, roadmap (update after each session).
- `controller/README.md` — full feature/architecture reference (update when features change).
- `TASK.md` — the current task to implement (if it exists).

## System context — the three-component model

The project runs **on Proxmox**, with a locked three-component model:
- **Hub** (`felhom.eu/hub/`) — operator backend on k3s.
- **Host agent** (`felhom-agent/`) — one per Proxmox host; operator-tier; owns ALL Proxmox interaction.
- **In-guest controller** (THIS repo) — one per customer LXC; **Docker-only; holds NO Proxmox
  credentials**. De-privileged: disk/host/Proxmox concerns are delegated to the host agent via the
  pinned local-API client (`internal/agentapi`); the controller keeps the app domain — stack/deploy
  management, the Hungarian web UI, app-data backup, metrics/telemetry, integrations, git-sync,
  notifications. Whole-guest backup (PBS vzdump) is the agent's.

> **Authoritative maps:** `felhom.eu/documentation/architecture/01/02/03-*.md` (topology/trust,
> controller module map, host agent) + the code-verified feature docs in
> `felhom.eu/documentation/controller/`. Match the current code, not summaries, if they drift.

**Don't confuse the two ex-"controllers":** `felhom-agent` (host, operator-tier, was
`proxmox-controller`) vs this `felhom-controller` (in-guest, was `deploy-felhom-compose`).

## Layout (verified against the tree)

```
controller/cmd/controller/   entry point + startup wiring (scheduler block, init-only setters)
controller/internal/
  agentapi/       pinned-TLS client to the host agent's per-guest local API (THE disk seam)
  api/            REST /api/* router (writeJSON envelope, limitBody, config writes)
  appbackup/      felhom-data paths/namespaces, DB dumps, userdata skeleton (shared primitives)
  appexport/      .fab export/import bundles (password crypto, strict segment validation)
  assets/         app logo/screenshot sync from the hub
  backup/         app-data backup manager, recovery units, tier-2 copies, offbox restic
  bootstrap/      bootstrap.json ingest → controller.yaml (Day-0 + refresh)
  channelhealth/  agent-channel health checker (debounce + born-down alerting)
  cloudflare/     geo-enforcement remnant (agent-delegated)
  config/         controller.yaml load/validate (LoadPermissive = setup-mode only)
  crypto/         AES-256-GCM app.yaml secret encryption (ENC: prefix)
  infra/          traefik/cloudflared/filebrowser base-stack templates
  integrations/   app-to-app integrations (e.g. OnlyOffice)
  mailrelay/      app-email SMTP shim → hub relay
  metrics/        telemetry collection
  monitor/        health checks, protected containers
  notify/         hub event push (typed Notify* wrappers)
  quiesce/        quiesce loop for whole-guest backup (marker + recover)
  recovery/       recovery-unit restore
  report/         hub report builder/pusher + pull-based config refresh
  scheduler/      background jobs (Every/Daily, Budapest DST-safe)
  selftest/       startup self-checks
  selfupdate/     controller image self-update via the agent swap
  settings/       settings.json persistence (registry, flags, corruption recovery)
  setup/          first-boot setup wizard (own CSRF)
  stacks/         compose ops: deploy/delete/migrate/state (THE app domain core)
  sync/           git-sync of the app catalog
  system/         mounts/probes (linux + permissive _other stubs)
  util/           small shared helpers
  web/            dashboard UI: server, auth/CSRF, handlers, funcmap, templates (Hungarian)
```

Per-package helpers/seams/traps: **`REUSE.md`** (maintained same-commit as helper changes).

## Conventions & cardinal rules

- **Trunk-based — no branches.** All shippable work commits directly to `main`; `main` equals what is
  deployed. Report-only artifacts → `felhom.eu/documentation/` (`audits/`, `backlog/`). Risky fixes
  are implemented during the supervised session itself, on `main`; if a fix can't be verified/shipped,
  revert + report — never park on a branch.
- Code quality: double-check for bugs/edge cases; add debug logging; **ask rather than guess**.
- All UI text is Hungarian (Budapest timezone). Design tokens/gates: use the `felhom-ui-design` skill.
- **Run `python3 controller/scripts/controller_gates.py` (from `controller/`) after ANY change in
  this repo.** It is the ONE entry point and runs all seven local gates — `template_id_gate`,
  `emoji_gate`, `native_confirm_gate`, `offbox_rename_gate`, `app_row_dedup_gate`, `mojibake_gate`,
  `docker_run_volume_path_gate` — plus `reuse_refs_check` on the repo root, streaming each gate's
  own output and exiting non-zero if any fails. `--fast` selects the gates that touch no network and
  no container runtime; today that is all of them. A missing gate script is a FAILURE, never a skip.
  **Why a runner and not seven invocations** (2026-08-02, R-29): a census of all thirteen gates
  across the four repos found that every check a `CLAUDE.md` names was passing, and two of the four
  nobody is told to run were failing. This file used to name two of the seven; the other five were
  reachable only through a line in `REUSE.md`, and `docker_run_volume_path_gate.py` was RED.
  **The shared `reuse_refs_check.py` lives in `felhom.eu/scripts/` and is never copied here** —
  a copy would recreate the drift it detects; an absent sibling clone FAILS the gate.
  **The pre-push hook** (`.githooks/pre-push`) runs it with `--fast` and refuses a failing push. It
  is per-clone — switch it on once with `git config core.hooksPath .githooks`, and a manual run
  WARNS when this clone is unarmed. `git push --no-verify` bypasses it deliberately; **say so in the
  session report when you use it.** Both facts are why CI is still owed (`OPEN-ITEMS.md` R-168).
- Testing doctrine (non-hollow tests, red-proofs, seams): use the `felhom-testing` skill.
- **Logging**: new leveled lines use `internal/logx` (DEBUG always reaches the debug ring; stdout
  respects `logging.level`); English, keys-never-values, durations on outcomes — full rules in
  `felhom.eu/documentation/runbooks/logging-conventions.md`.
- Update `REUSE.md` if you added/changed/deprecated a shared helper or pattern (same commit).
- **Coupled features** (controller behavior that depends on a specific agent version): add a
  `featureProbes` table row in `internal/agentapi/features.go` + a `Supports` gate call at the
  feature's entry point; declare `MinAgent: X.Y.Z` in the CHANGELOG entry header. Rules:
  `felhom.eu/documentation/runbooks/publish-train-rules.md`.

> **In every repository where you make a change, update both files in that repo:**
> - **`CHANGELOG.md`** — cumulative log, newest on top.
> - **`REPORT.md`** — **overwrite** with the most recent implementation/validation summary only.
>
> **Never write secrets** into any committed file — reference them as "stored out-of-band".

## Live validation

Exercise the SERVER-SIDE PIPELINE a real user triggers, end-to-end (connect → enroll → deploy). The
forbidden shortcut is BYPASSING that pipeline (the F9 episode: raw agent guest-attach + hand-set
state). **`claude-in-chrome` is NOT available in the DooPlex environment** — the standard method is
endpoint-level: invoke the exact endpoint the UI invokes (no server logic is skipped, only
rendering) and say which method was used. Strict end-to-end UI coverage is a manual click-through.

Two traps in that method, both from the 2026-07-20 remediation:
- **Grep the fetched page with ASCII-only substrings.** Accented Hungarian patterns get mangled
  through the `ssh → pct exec → bash -c` chain and return a false `0` — which reads exactly like the
  banner/string being gone. Use `kezel`, `Utols`, `Biztons`; never let an accented pattern gate a
  conclusion (it nearly produced a wrong "banner cleared" claim).
- **Credentials with `!` or `'` break in heredoc-built helper scripts** (history expansion eats
  `!!`). Use the proven inline `-d "password=$PW"` form for authed curl, and delete any
  credential-bearing helper from `/tmp` (host AND guest) when done.

## Environment & access

Claude Code runs **on DooPlex (192.168.0.180, Debian 13, user `kisfenyo`)**; repos in
`/mnt/5_hdd/felhom.eu/git/`, build dirs in `/mnt/5_hdd/felhom.eu/build/`. All repos hosted at
`gitea.dooplex.hu/admin/`. Builds are local commands; felhom-pve is one SSH hop.

| Host | Access | Role |
|------|--------|------|
| **DooPlex (this host)** | local — `/mnt/5_hdd/felhom.eu/{git,build}/` | build + push images, `sudo kubectl` |
| Demo Proxmox host `demo-felhom` | `ssh felhom-pve` (root@192.168.0.162) | `pct` into guests; live validation |
| Demo guest 9201 | `ssh felhom-pve "pct exec 9201 -- ..."` | the live demo controller (golden/bootstrap-managed) |
| Demo host `demo-hp` (HP t740) | `ssh demo-hp` (TS `100.76.96.79` / LAN `192.168.0.87`); **no baked key** — break-glass root via hub `host_recovery/demo-hp-bb76ea` + `sshpass` (recipe in `felhom.eu/documentation/operations/nodes.md`) | 2nd demo node (guest 9201 `demo-hp`, agent 0.93.0). **Designated drill+build VM host** per the 2026-07-25 ruling — but **no drill VM is provisioned there yet** (forward-looking); the drill `drill.qcow2` still lives on DooPlex (off). See nodes.md. |
| felhotest (legacy) | `ssh -p 33022 kisfenyo@router.abonet.hu` | OLD /opt/docker compose mechanism |

> **Legacy: Windows workstation.** Until 2026-07-19 CC ran on Windows 11 with repos in `E:\git\`,
> and every remote command needed `SSH=/c/Windows/System32/OpenSSH/ssh.exe` (Git Bash's ssh lacks
> the Windows agent and fails silently — see `docs/vscode-ssh-fix.md`), plus `MSYS_NO_PATHCONV=1`
> for `pct exec`. Retained in case that environment is revived.

> **felhom-pve is back on the home LAN (as of 2026-07-25).** The host holds `192.168.0.162` again and
> the **agent is UP** — `localapi` binds `192.168.0.162:8443`, the service is `active`, capabilities
> self-check clean, and all agent-backed features (storage, PBS backup, quiesce, restore-test, DR) are
> reachable. `ssh felhom-pve` remains a **Tailscale** alias (`100.70.170.35`, location-independent, the
> N100 is travel-portable) with `Host felhom-pve-lan` as the direct-LAN fallback; both work. The earlier
> "remote site — agent DOWN (binds a stale `.162`)" block was the 2026-07-20→24 vacation window and is
> now retired; the historical record stays in
> `felhom.eu/documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md`.

External access via Cloudflare Tunnel → Traefik; Pi-hole forwards `*.demo-felhom.eu` → .162 locally.

## Build & deploy — MANDATORY after code changes

**Full runbook: use the `felhom-build-deploy` skill.** Summary (guest 9201 is bootstrap-managed —
**no compose file**; `felhom-controller-bootstrap.service` runs the tag in `/etc/felhom-controller-image`):

> **Clean-tree gate before any build:** `git status --porcelain` must be empty and
> `git rev-parse HEAD` must equal `git rev-parse origin/main` in the repo being built. An unpushed
> change does not exist — never build a dirty or unpushed tree. The `git pull` in the build step
> stays (it is a no-op when you work in this tree, and load-bearing if anything was pushed from
> elsewhere).

| Step | Command |
|------|---------|
| 1. Commit + push | `git add <explicit paths> && git commit -m "..." && git push` |
| 2. Build + push image | `cd /mnt/5_hdd/felhom.eu/build/felhom-controller && git -C /mnt/5_hdd/felhom.eu/git/felhom-controller pull && ./build.sh <VER> --push` (build.sh does NOT pull — the explicit pull is load-bearing) |
| 3. Deploy (9201) | `ssh felhom-pve "pct exec 9201 -- bash -c 'docker pull gitea.dooplex.hu/admin/felhom-controller:<VER> && echo gitea.dooplex.hu/admin/felhom-controller:<VER> > /etc/felhom-controller-image && systemctl restart felhom-controller-bootstrap.service'"` |
| 4. Verify | `ssh felhom-pve "pct exec 9201 -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'"` + container logs |

Hub build/deploy lives in `felhom.eu` (GitOps) — see that repo's CLAUDE.md / the skill. Catalog
changes (`app-catalog-felhom.eu`): commit+push; controller sync picks them up ≤15 min or via the
"Sablonok frissítése" button.

## Session-critical invariants (the rest live in REUSE.md)

- `docker compose restart` does NOT pick up new images/env — always `up -d` (`RedeployFromEnv`).
- Docker's `.State` says "running" even for unhealthy containers — `.Status` parse is the truth.
- In-memory `Deployed` flag is set BEFORE `compose up -d` (slow-pull race); reverted on failure.
- `compose up -d` exits 0 on crash-loops — post-start status check is the detection.
- Env var KEYS are logged, never values. Protected stacks (traefik, cloudflared, felhom-controller)
  can't be stopped from the UI.
- Verify a container image HAS the healthcheck tool before using it (BusyBox wget / python3 / curl —
  catalog REUSE.md maps the families).

## Working with CHANGELOG.md

**DO NOT read the full file** — it is large and will waste context.
- Session start: use `CONTEXT.md` + `controller/README.md` for current state.
- Adding an entry: Read only the top ~30 lines for format, then Edit-insert after line 1.
- History: Grep for topics instead of reading.

## End-of-session checklist

1. **Commit and push** all code changes
2. **Build, push, and deploy** the new controller image (if controller code changed)
3. **Update CHANGELOG.md** with what was done
4. **Update CONTEXT.md** with decisions made, state and what's next
5. **Update controller/README.md** if architecture or features changed
6. **Verify** the deployment is working (check `docker ps` and logs)
7. **Update REUSE.md** if you added/changed/deprecated a shared helper or pattern (same commit)