felhom-controller/CONTEXT.md

# CONTEXT.md — Project Memory

> This file serves as persistent project memory across Claude Code sessions.
> It replaces the auto-generated "Memory" from the claude.ai Project.
> **Update this file at the end of each working session** with current state,
> recent decisions, and anything the next session needs to know.
>
> Ask Claude Code: "Please update CONTEXT.md with what we did today"

Last updated: 2026-07-11 (v0.116.1 — observability pass; pairs with agent v0.83.0 + hub v0.46.0)

> **2026-07-11 — v0.116.0/0.116.1: OBSERVABILITY PASS (LIVE on 9201; agent v0.83.0 + hub v0.46.0).
> MinAgent: 0.81.0 unchanged.** The debug ring (`LogBuffer`) now ALWAYS exists — logger =
> `MultiWriter(LevelFilterWriter(stdout, logging.level), ring)`, so DEBUG detail is remotely
> readable on an `info` box while docker logs keep the configured level. New `internal/logx`
> leveled helpers = the sweep standard (netstorage_job phases/verdicts/durations, netprobe,
> validation refusals, orphan WARN, `SupportsWithSource` gate line, agentapi per-call DEBUG,
> migrate phases, tier2/offbox unswallowed persists). Report ACK gains `controller_log_requested` →
> next report ships `controller_log_tail` (selftail.go, consume-once, 128 KB; the customer-visible
> `operator log pull served` INFO rides in the tail; app-tail wire byte-compatible). Debug page:
> `Vezérlő | Ügynök` tabs — the agent tab proxies agent `GET /debug/logs` (`Client.DebugLogs`;
> typed-404 → the "after the agent's next update" notice). **v0.116.1 (found by live validation):
> `/debug` + `/api/debug/*` + the nav item were STILL gated on logging.level=debug — ungated (auth
> unchanged), the incident's actual blind spot.** Live-proven at info: a real refused NAS add is
> fully reconstructable on both tabs (capability gate w/ source=version, phase lines, 502+duration,
> category). Conventions: `felhom.eu/documentation/runbooks/logging-conventions.md`. OPEN: operator
> clicks the hub's two "Request logs" buttons (hub UI password-gated) to close the live bundle
> round-trip; legacy `isDebug()` emission sites left for incremental migration. NOT published —
> Peti stays 0.113/0.81.

> **2026-07-11 — v0.115.0: version-aware Supports + DSM-validated NAS guidance (LIVE on 9201, pairs
> with agent v0.82.0 + hub v0.45.0). MinAgent: 0.81.0.** Capability detection now compares the agent
> version from agent v0.82.0's `X-Felhom-Agent-Version` header (`Client.noteAgentVersion` captures it
> on every response, strict semver; `features.go featureMinAgent` table + version-first `Supports`)
> instead of route-probing — the probe stays as the fallback for header-less (≤0.81) agents, so
> nothing changed for Peti's box. THE one comparator moved to `internal/util/version.go` (selfupdate
> aliases it). Part A DSM spike (real DSM 7.2 via virtual-dsm) validated the consumer recipes E2E; the
> NAS-page NFS guidance gained the verified Synology steps (File Services → NFS → **NFSv4.1**; "Map
> all users to admin"; `/volume1/<share>`); caveat narrowed to QNAP-only. Live-checked on demo: a real
> add shows `capability gate: netstorage_verify=yes` via the version compare, zero probes. **Q1c
> (Part E, supervised) FAILED**: a NAS automount trigger does NOT survive a guest reboot (guest sees
> an empty dir; agent has no network-mount reassert) — fix is felhom-agent's, spec'd at
> felhom.eu/documentation/backlog/FOLLOWUP-nas-automount-guest-reboot-reassert.md (controller
> health-cross-check follow-on noted there). NOT published (agent 0.82 demo-only; Peti 0.81).

> **2026-07-11 — v0.114.0: agent-capability gate (option-1) + publish-train rules (option-2).**
> Answer to the 0.81/0.113 train's 9-minute controller-before-agent skew (Peti's box): the box now
> protects itself. `internal/agentapi/features.go` — `Supports(Feature)` route-probes the agent
> (`GET /netstorage/verify-status` = the v0.81.0 coupling signal; typed `StatusError` 404 ⇒ No, 2xx
> ⇒ Yes, transport/5xx ⇒ Unknown NEVER refused; `SupportCache` TTL 5m both polarities, Unknown
> uncached). `handleNetStorageAdd` refuses on No BEFORE the single-flight claim (412 +
> `agent_outdated` + honest Hungarian message); settings page swaps the add form for a banner
> (list/remove untouched in every state). remove/list NOT gated. NO agent/hub changes; NOT
> published, floor untouched, Peti stays 0.113.0 — the gate is inert protection until the next
> train. Rules codified: `felhom.eu/documentation/runbooks/publish-train-rules.md` (manifest before
> floor; floor field LAST — hub_settings DB row overrides env + acts immediately; MinAgent fleet
> gate — CHANGELOG header convention starts with this release; the gate as box-level backstop).
> Tests T1–T6 + wire-level 404-typing; red-proofs RP1–RP5 in REPORT.md. Roadmap: agent
> version-in-envelope upgrade of `Supports`; hub floor-UI separation = its own task. The
> `agent_outdated` branch is test-proven only (demo agent is current — downgrade not justified).

> **2026-07-11 — v0.113.0: NAS verify-before-commit + page redesign (LIVE on 9201, pairs with agent
> v0.81.0 + host-install v1.13.0).** `POST /api/storage/netstorage/add` no longer registers blind
> (the bogus-share-at-Készenlét bug is dead): detached single-flight orchestration
> (`internal/web/netstorage_job.go`, migrate shape; poll `GET .../add/status`) = agent add (units +
> agent-side detached verify with journal classification + auto-rollback) → controller **uid-1000
> re-exec write probe** (`--netprobe`, SysProcAttr.Credential — catches the squash trap) → register
> LAST. Any failure = full rollback; verify-lost after agent restart ⇒ controller rollback; unregistered
> agent shares surface as remove-only "Árva megosztás" rows. §3.2 Hungarian error map server-side
> (`netAddMessage`; `nfs_export` MERGES not-found/not-permitted — NFSv4 identical strings).
> storage_network.html rebuilt on the storage_attach pattern (form-row/form-input killed), SMB listed
> first, NFS two-recipe guidance with live computed uid+100000. Agent v0.81.0: NFS `retry=0`
> (dead-NAS access 91 s→3.8 s), `ClassifyNetVerifyFailure` (Q4-verbatim), unprivileged journal read
> (systemd-journal group — host-install v1.13.0 adds it; NO new sudoers). Live-validated A–E on 9201
> vs an isolated sim NAS (all transcripts + red-proofs in REPORT.md); Route A proven in production
> (alien-uid squash → server-side 1060:1060). Authoritative doc:
> felhom.eu/documentation/controller/network-storage-nas.md. NOT published (0.81.0 not in Gitea /
> Day-0 manifest; Peti untouched — his rollout incl. the usermod one-liner comes with the floor bump).
> Gotcha for future sessions: the controller container is bridge-only — in-guest API tests need the
> CONTAINER IP + `Host: felhom.demo-felhom.eu` (127.0.0.1:8080 is stale advice).

> **2026-07-10 — v0.112.0: self-update without credentials (LIVE on 9201, pairs with hub v0.43.1).**
> Root cause on Peti's box: the updater refused without Git Sync creds, but the public package is
> anonymously pullable. `queryRegistry` now does the Docker v2 anonymous token dance when both creds are
> empty (realm/service parsed FROM the WWW-Authenticate header — never hardcoded); `pullImage` skips
> `docker login` credential-less; creds path byte-unchanged (private catalogs); half-configured pair =
> loud misconfig; denial = "registry denied anonymous access — a private registry requires Git Sync
> credentials". Settings panel gains the mode line "Registry: nyilvános (hitelesítés nélkül) /
> hitelesített" — credential-less is a supported mode, not an error. Red-proof green (old guard restored
> → anonymous tests fail with the old message). LIVE-PROVEN on the credential-less demo (git creds are
> quoted-empty): /api/selfupdate/check → ok, latest=0.112.0, no error; settings shows "nyilvános".
> PENDING OPERATOR: floor-bump Peti to 0.112.0, then delete his temp Git Sync creds → clean "nyilvános"
> check. runCommand/runCommandStdin are now package VARS (test seam).

> **2026-07-10 — v0.111.0: remote app-log diagnostics (LIVE on 9201, pairs with hub v0.43.0).** The
> telemetry scraper now attaches `LogIssue.Context` (±5 raw lines around the FIRST occurrence of each
> error-severity issue; ≤11 lines, ≤400 chars/line, 16KB/report budget dropping lowest-count first; warns
> carry none) and `metrics.RedactLine` sanitizes EVERY off-box context/tail line (password/token/api-key/
> authorization/bearer → `[REDACTED]`, 64-hex → `[REDACTED-HEX64]`). On-demand log tails ride the ACK pull
> pattern: hub ACK `log_tail_requests` → next report `log_tails` (200 lines via stacks.GetLogs /
> FetchContainerLogTail, ordered, ≤64KB/app newest-kept, redacted, consume-once drain). Hub v0.43.0 stores
> context (first-capture-wins + `context_customer` provenance), renders click-to-expand copyable issues,
> fixes the period filter on Known Issues, replaces issue deletion with DISMISSAL (`dismissed_at`,
> resurface only on `last_seen > dismissed_at`), adds `?customer=` filtered drill-down, and keeps the last
> 2 tails per app with an ordered viewer + .log download. All red-proofs green (capture, redaction,
> consume-once ×2, dismissal guard, range filter, context clobber). Live-proven on demo: synthetic error →
> hub row with ordered 11-line context and `password=[REDACTED]`. OPERATOR: one click ("Request log tail"
> on demo felhom-controller) completes the live tail round-trip — hub UI is password-gated, CC cannot.

> **2026-07-09 — v0.106.0: offsite provisioning SLICE 2 — the apply-bridge (pairs with hub v0.38.0).** On
> startup the controller reconciles the hub-served `offsite:` descriptor into a key-only offbox target:
> `internal/offsiteapply.Bridge.Reconcile` — verify-pin the box host key against `host_fingerprint` (NO blind
> TOFU) → generate keypair → consume the one-time password (`POST /api/v1/offsite/consume-password/{id}`,
> single-use, never logged) → `sshpass ssh-copy-id -s -f` install + verify → `Manager.ApplyOffsiteTarget`
> (fork-4 enable → `EscrowState="pending"`) → persist a descriptor-hash marker LAST. **Idempotent** (no
> re-consume of a spent password) + **fail-safe** (any step fails → nothing persisted, retry next restart;
> consumed-but-failed install = loud "reset on the hub"). Seams faked in tests; both red-proofs (no-TOFU,
> marker-after-success) green. `Dockerfile` + `sshpass`. **NOT yet live-applied** — supervised end-to-end
> (hub provision → controller apply) is the next runbook, gated on the hub's new scoped `HETZNER_TOKEN`.
> NEXT slices: SLICE 3 (escrow auto-confirm), SLICE 4 (soft-quota).

> **2026-07-09 — v0.105.0: fork-4 offsite password custody (pairs with agent v0.77.0).** The restic-offsite
> repo password now rides the **customer-R escrow** (age-under-R in the agent `IdentityBundle`; custody spike
> `febdc56`). Enable → controller pushes the password (`StageEscrowSecret` → agent `POST /escrow/stage-secret`)
> → `EscrowState="pending"`. **Atomicity gate:** no offsite RUN until `EscrowState="escrowed"` (operator
> `POST /backup/offbox/confirm-escrow` after the escrow ceremony) — so no un-recoverable offsite ciphertext
> exists. **DR:** `POST /backup/offbox/inject-password` pre-places the recovered password (honored by
> `WriteOffboxSecrets`). DR recipe gains non-secret `offsite_restic` coords (`DRResticCoord`); the SFTP key is
> regenerated at DR (not escrowed). Atomicity + inject companion red-proofs green. **NOT yet live-validated**
> — the supervised escrow ceremony (enable→stage→escrow-create→confirm→gated run) is operator-run; NEXT =
> hub-verified auto-confirm + customer-self-serve enable (provisioning task). Deployed to 9201; see REPORT.

> **2026-07-09 — v0.104.0: off-box discovery over inference + no-silent-success.** The Storage-Box spike
> found offbox reporting `ok`/0 snapshots while backing up nothing; DIAG pinned it: offbox resolved each
> toggled app's recovery unit via `AppNamespaceRoot`→`GetAppDrivePath`, which reads the app's *live*
> `app.yaml` `HDD_PATH` and **silently falls back to `systemDataPath`** when the app isn't deployed → it
> looked on the wrong drive. **Decision: DISCOVER, don't infer** — scan the durable storage registry
> (schedulable, non-decommissioned paths ∪ systemDataPath) for `backups/primary/<app>`, deployment-state
> independent; newest-by-manifest wins on drive churn. **Silent-success closed:** 0-of-N toggled → hard
> error + operator alert; partial → `ok` + customer `LastWarning`. Write paths + `AppNamespaceRoot`
> untouched. Unit suite + both companion red-proofs green. **NEXT:** supervised box re-provision + a real
> offbox→Storage-Box endpoint round-trip (this task did NOT re-point at the live box — spike creds were
> torn down). Deployed to 9201; see REPORT.md.

> **2026-07-07 — v0.103.0: F-C2-1 (LIVE on 9201).** The config loader ran `os.ExpandEnv` over the
> whole YAML before parse, silently corrupting a bcrypt `web.password_hash` (`$2a$10$…` → `"a0"`) — a
> silent auth-integrity bug. Removed both `ExpandEnv` calls (parse raw bytes); typed
> `FELHOM_WEB_PASSWORD_HASH` override unchanged. Live-proven: a bcrypt hash in controller.yaml now
> loads intact and login succeeds (pre-fix it corrupted → login fail). Behavior change: literal
> `${VAR}` in a value is now preserved verbatim (no repo config depends on the old expansion).

Last updated: 2026-07-06 (v0.102.0 — async restore family; F4 re-adjudicated + fixed)

> **2026-07-06 — v0.102.0: async restore family (F4 UX fix, LIVE on 9201).** All three restore surfaces
> (`/backup/restore`, `/backup/tier2/restore`, `/backup/offbox/restore`) blocked the HTTP request until
> completion → through cloudflared's 100s cap a customer got an error page while the restore succeeded
> (offbox worse: bounded on `r.Context()`, canceling the SFTP restore mid-flight). Now async (offboxRun
> shape): fast-path IsRunning refuse → background goroutine (offbox ctx off r.Context()→Background+30m) →
> instant redirect. New `GET /api/backup/restore-status` + mutex op-status (`opstatus.go`) + 3s-polling
> `backups.html` banner. Live-proven: restore POST 0.018s internal / **0.235s external (F4 tunnel)**, canary
> bit-identical, status transitions. Restore single-flight unchanged. OPEN: op-status is in-memory (no
> persistence, by design).

> **2026-07-06 — v0.101.0: no-mercy campaign findings.** F3: git subprocess deadline in
> `internal/sync/sync.go` (`gitCmdTimeout=120s`, `exec.CommandContext`) — a hung remote no longer
> wedges `syncing=true` until restart. F2 evidence gap: `agentapi.EjectDisk`/`Decommission` now use
> `postWithStatus` + `refusalError` so the agent's `"…refused (role: X)"` reaches the operator
> instead of a bare `HTTP 403`. Companion: catalog `d86e256` (F1 vaultwarden `_ENABLE_SMTP` boot-gate
> — fresh email-off deploys crash-looped; live-validated Scenarios A/B on 9201). F2 diagnosed to a
> verdict (REAL finding — `roleForMountPath` over-refuses an enrolled user-data drive that isn't a
> PVE storage; fail-safe direction; agent fix DEFERRED). Full triage:
> `felhom.eu/documentation/audits/CAMPAIGN-nomercy-2026-07-06.md` addendum. OPEN follow-ups: agent
> `roleForMountPath` fallback; the targeted P1–P3 campaign re-run for clean backup/restore coverage.

> **2026-07-05 — v0.100.0 (TASK C2): drill finding F2 CLOSED — one-click class-C file restore.**
> `POST /backup/tier2/restore` + "Fájlok visszaállítása" on the Tier-2 row: in-place, ADDITIVE-ONLY
> (`rsync -a --ignore-existing` from the recorded Tier-2 copy — never overwrites, never deletes).
> Serves "I deleted my files"; corruption/point-in-time stays offbox/operator. **The C-series
> (drill findings F1/F2/F3/O4) is now fully closed.** Reindex caveat (e.g. Nextcloud occ files:scan)
> documented in backup-architecture.md.

> **2026-07-05 — v0.99.0 restore-path fixes (TASK C1): drill findings F1/F3/O4 RESOLVED.**
> F1: `GET /api/backup/snapshots` implemented (`backup.ListRestorePoints`) — the restore panel
> populates and the restore button enables. F3: `runVolumeDumps` wired into the nightly/manual
> backup run (volume gate BEFORE DumpAppVolumesSafe; before unit capture). O4: unrecoverable
> resettable secrets get a generated replacement (`GenerateSecretForField` + `SetSecretGenerator`
> seam); data-key gate untouched. **F2 (one-click in-place class-C restore) remains OPEN → TASK C2.**
> O4 residual: restored volume tar with an OLD credential hash may still need a manual in-DB reset.

> **2026-07-04 — app-data restore drill** → see `felhom.eu/documentation/audits/DRILL-appdata-restore-2026-07-04.md`.
> Keep-side restore proven live on 9201 (class-A DB replay + fail-closed data-key gate + non-destruction). **F1 (HIGH): UI restore is dead — `/api/backup/snapshots` has no handler, so the restore button never enables.** F2: no one-click in-place class-C (HDD bind-mount) restore. F3: named-volume data never backed up (`DumpAppVolumes*` has no caller). *(F1/F3/O4 resolved in v0.99.0, see above.)*

> **2026-07-03 — CLAUDE.md slimmed to stable orientation** (full package map, verified 9201 deploy
> summary, no version-pinned state). Deep runbooks/design/testing doctrine now in the personal
> skills `felhom-build-deploy` / `felhom-ui-design` / `felhom-testing` (source: `felhom.eu/skills/`).

> **2026-07-03 — `REUSE.md` exists at the repo root** (canonical helpers / patterns / traps / seams, code-verified). Check it before writing new code; update it in the same commit that adds/changes a shared helper (maintenance rule now in CLAUDE.md).

> **2026-07-02 — v0.98.0 (deployed on 9201): Tárhely IA follow-up.** User feedback on D1: NAS-add and
> local-drive enrollment buttons sat side by side — confusing. `/storage` split into two subpages
> under the Tárhely nav item (nested sub-links, `.nav-links-nested`): **/storage = Meghajtók**
> (drives + agent view + wizards + manual add) and **/storage/network = Hálózati tárhely (NAS)**
> (NAS-megosztások list + add form; own openDialog copy). `networkStoragePageData` (page key
> `storage-network`) split out of `storagePageData`. No API/storage-semantics change.

> **2026-07-02 — v0.97.0 (deployed on 9201): TASK-D1 — settings split + Tárhely page (IA only).**
> The 1451-line `settings.html` monolith is split into four pages: `/storage` (new main-nav
> **Tárhely**: Adattárolók + NAS + unified agent drive view + wizard entry), `/settings` (Rendszer:
> konfig, verzió/frissítés, vezérlő+kiszolgáló újraindítás), `/settings/notifications` (Értesítések
> + Alkalmazás-email), `/settings/security` (Jelszó, Földrajzi korlátozás, Vészhelyzeti információk).
> Sidebar gains the Tárhely item + a "Beállítások" group with three sub-links. **No storage/agent/API
> behavior change** — pages moved, logic didn't; every `/api/storage/*` endpoint + POST action
> unchanged. **Decisions:** (a) storage flashes + wizards moved to `/storage`; old
> `/settings/storage/{init,attach}` **301** to `/storage/{init,attach}`. (b) The two duplicated drive
> views MERGED: registry cards render server-side, then JS enriches each connected user-data card in
> place from the agent `/api/disks` (role tag, durable-id, agent actions) joined on mount path, plus
> two groups — **Rendszermeghajtók** (system/backup, read-only, lock tag, NO actions) and **Nem
> regisztrált** (register only); agent-down → one warn note, registry cards still render. (c) Every
> native `confirm()`/`prompt()` on the four pages now routes through a light `.confirm-overlay`
> (`openDialog`; texts verbatim; type-to-confirm kept only where it existed) — the D0 tab-freeze is
> gone. **Contract for future UI:** new templates must pass `scripts/template_id_gate.py` (every JS
> element-ID resolves in its own template) and `scripts/emoji_gate.py` (0 emoji — the D0 grep gate
> false-negatived multibyte emoji on Windows; 8 survivors removed). `.badge-lock`/`.lock-ico` CSS
> deleted; other `.badge*` still used on secondary pages (D2 to finish). **Next: D2** (badge→tag on
> backups/secondary pages; storage.html is still ~700 lines — NAS-add partial candidate). NOT
> live-validated: agent-down degradation (don't stop the live agent — static/unit only); destructive
> storage ops via the moved overlay paths (endpoints unchanged; supervised session).

> **2026-07-02 — v0.96.0 (deployed on 9201): TASK-D0 — design system v2 re-skin (appearance only).**
> The whole customer UI (dashboard AND setup wizard) now renders in the approved v2 language: navy
> token palette (`--bg-0/1/2, --line, --text-1/2/3, --blue, --warn, --crit`), single 2px radius, no
> shadows; **exception-based status color** — nominal is blue/neutral, amber/red only on deviation;
> zone bars → the hairline `.meter` (neutral 70/85 ticks, warn/crit flag „Fogyóban a hely" /
> „Kritikusan kevés hely"); pills → `.tag` / `.metarow`. **Decisions:** (a) `stateColor` semantics
> changed — **stopped/exited = neutral, NOT red** (operator-approved; failures surface via unhealthy
> + alerts); restarting = warn; outputs are now `run/progress/warn/neutral/off` and
> `nominal/warn/crit` — any new template MUST use these suffixes (truth tables guarded by unit tests,
> `stateLabel` copy frozen byte-identical). (b) Fonts (PJS + JBM variable woff2, latin+latin-ext) and
> a 30-icon Lucide sprite are **vendored in the binary** (`/static/fonts/`, `templates/icons.html`)
> — no CDN; `fonts.googleapis.com`, `box-shadow`, 999px and the emoji set are banned strings (grep
> gate in REPORT §5, 143→0). (c) Setup wizard `handleCSS` bug fixed: it read a dataDir-derived path
> that never exists in the container → always served minimalCSS in production; now serves embedded
> `web.StyleCSS()`. (d) Found+fixed pre-existing v0.93.0 bug: `/backups` 500'd once an off-box backup
> had run (`OffboxTarget.LastRun` is an RFC3339 string fed to `timeAgo`; new `timeAgoStr`).
> Canonical reference: `felhom.eu/documentation/design/design-system.md`. **Next: D1** (settings IA
> split + Tárhely main-nav promotion; also migrate the remaining native `confirm()` dialogs to the
> type-to-confirm overlay and finish badge→tag markup on secondary pages). NOT live-validated: setup
> wizard visuals (unit-only), warn/crit meters on real hardware (demo healthy).

> **2026-07-01 — v0.95.0 (deployed on 9201): Impl-2b — raw-drive enrollment wizards.** Both enrollment
> wizards (`storage_init.html` / `storage_attach.html`) now fetch candidates from the agent's raw-device
> scan `GET /api/disks/candidates` (proxy `agentDiskCandidatesHandler` → agent Impl-2a) instead of the
> `Observe()`-based `/api/disks`, so a brand-new (non-PVE-storage) drive is discoverable + enrollable.
> `agentapi.ListCandidates` + `resolveEnrollUUID` (a raw candidate isn't in `/disks` → resolve its
> fs-UUID from the scan). Live end-to-end on felhom-pve: raw `/dev/sdd` (SD card) → wizard → format (Impl-1
> guard) / attach → mount → bind → intent → guest-bind → **"Aktív"** in the UI (3rd drive, no false
> detach). Needed a chain of AGENT fixes (v0.56–0.58: durableIDForMount / ReassertGuestBinds / HostReader
> wiring / /disks GuestPath+BoundUnderParent for registry rows) — the whole enroll+tracking machinery had
> assumed a PVE-storage drive. **Follow-ups:** `runStorageInit` should poll the agent's detached-format
> status (`/disks/format/status`) so a SLOW-device format completes in one flow (pre-existing); Impl-3
> shared-box operator format gate.

> **2026-06-30 — v0.94.0 (deployed on 9201): pull-based config-refresh.** The hub report ACK now also
> carries a per-customer **`config_version`** (hub v0.26.0; a stored counter bumped on every config save).
> `OnPushResponse` → `ConfigRefresher.Reconcile` (`internal/report/config_refresh.go`): on a change vs.
> `settings.applied_config_version`, `bootstrap.RefreshConfig` re-pulls `controller.yaml` (re-merging
> `local_api` from bootstrap.json; overwrites controller.yaml, never settings.json) → record → graceful
> self-restart (`api.GracefulSelfRestart`). First-run records baseline (no restart); unchanged = no-op
> (no storm); failed pull keeps config + retries. This is the box-pulls-config replacement for the hub's
> retired inbound "Push Config" — the hub never connects into the box. **Companion hub v0.26.0** also
> retired Trigger Update / Pull Config / Show Diff and dropped geo-disable's inbound notify (kept the
> hub→Cloudflare WAF removal), and fixed the stale `docker-setup.sh` setup command → host-install.
> Live-validated on 9201: first-run baseline (applied=1, no restart) → DB bump 1→2 → re-pull+self-restart
> (session_secret rotated, local_api preserved, RestartCount 0→1) → 4 cycles no-loop → apps stayed up.

> **2026-06-29 — Self-health arc complete (agent v0.48.0 + hub v0.22.1).** The hub now proactively
> watches every agent's served local-API **leaf fingerprint** fleet-wide: the agent reports
> `leaf_fingerprint` on its host report (v0.48.0), and the hub `HostLeafChecker` (`monitor/host_leaf.go`)
> raises `host_leaf_changed` when it changes (trust-on-first-report baseline; empty fp = unknown, never
> alerts; reads from report_json, no migration; hub-generated so no allowlist change). Independent of the
> controller channel-check. Live: a leaf regen raised `host_leaf_changed` (`82078fab…→60b5974d…`) AND the
> controller's `agent_channel_pin_mismatch` — complementary. NOTE: v0.22.0 shipped the checker but the
> main.go wiring never applied (concurrent file-touch); the LIVE TEST caught it (no event on regen) →
> fixed in v0.22.1. The original silent-multi-day-outage incident class is now caught from THREE angles
> (agent capabilities v0.44.0 / controller channel-check v0.90.0+F2 / hub leaf-fp v0.22.x) AND prevented
> (agent v0.46.0 loud-regenerate + host-install --preserve-state-from). Backlog: the served-fp-vs-pinned-fp
> authoritative cross-check; the test-run coverage gaps (capability→hub live, host-reboot doubling).

> **2026-06-29 — v0.91.0: F2 (channel-health born-down alerting).** The v0.90.0 channel-health check
> alerted only on a live up→down transition; a channel broken at **startup/reseed** (e.g. the
> controller boots right after a leaf regen → first observation is `pin_mismatch`) was dashboard-only,
> no operator email — forever. Fixed with an `alerted` flag in `channelhealth/checker.go`: a confirmed
> down coming from up/unseeded OR a reason-change re-arms then alerts once; steady-down no dup; recovery
> re-arms; debounce intact (transient born-down still N≥2; healthy first-obs silent). Live-validated:
> controller restarted into a regenerated-leaf channel → `[channel] DOWN (unseeded->down:pin_mismatch)`
> + `Event pushed: agent_channel_pin_mismatch` (~60s) — the capstone's missing half. Companions: hub
> v0.21.0 (same fix for HostCapabilityChecker/HostStalenessChecker — seed only healthy, born-degraded/
> stale emits on first Check, cooldown dedups), agent v0.46.0 (EnsureLeaf loud-WARNs a REGENERATED leaf
> + host-install `--preserve-state-from`/populated-host guard = the prevention for the original incident
> class). Self-health story now complete: agent watches its capabilities, controller watches its link,
> detection works at startup, and a reinstall can preserve the leaf. Backlog: F1 swap-rollback
> RestartCount hardening; full live reinstall (supervised); hub-side leaf-fp fleet comparison.

> **2026-06-29 — v0.90.0: controller→agent channel health-check (self-health slice).** A ~60s
> scheduler job (`internal/channelhealth`) probes the local-API channel via the PRODUCTION memoized
> client (`Server.ProbeAgentChannel` → `s.agentClient()` + GET /storage — NOT a fresh client, per the
> spike: self-heals, mirrors the disk UI, no transport leak). Classifies failures (spike Q1 map:
> pin_mismatch/unauthorized/unreachable/timeout/misconfigured/construction_error/unknown), **debounces**
> transient reasons (refused/timeout need N≥2 consecutive so the ~1s agent-restart blip doesn't page;
> pin/401/DNS/construction alert first-obs), seeds the first observation silently, and on a transition
> emits an **English operator-only** event (`Notifier.NotifyAgentChannelDown/Recovered`) + a Hungarian
> dashboard banner (`AlertManager.SetAgentChannelAlert`). Closes the R1-incident gap (channel was only
> checked once at startup). **Required a hub change** (felhom-hub v0.20.0): the `agent_channel_*` event
> types were rejected by the hub's `allowedEventTypes` allowlist (HTTP 400) — controller-pushed events
> are gated, unlike the hub-generated host_* ones. Live-validated: transient restart → no alert;
> sustained stop → debounce(1/2) then DOWN transition + dashboard banner; start → recovered. No agent
> change. Self-health story now: agent watches its own capabilities (v0.44.0) + controller watches its
> link to the agent (this). Backlog: hub-side leaf-fp comparison.

> **2026-06-29 — OPS (no code change): controller↔agent TLS leaf-pin mismatch RESOLVED via R1.**
> The 2026-06-28 root→non-root agent migration regenerated the agent's local-API leaf
> (`60b5974d…`→`ced34036…`) and truncated its token store, so the in-guest controller's pinned
> `agentapi` client failed every call (`TLS pin mismatch`) — disk UI dead AND the quiesce
> app-consistent-backup loop dead. Fix: restored the pre-migration `local-api.{crt,key}` **+**
> `local-tokens.log` from `/root/agent-backup-bundle-test/aside-var-lib/` on felhom-pve (R1 RUNBOOK,
> supervised). Agent now serves `60b5974d…` again (== controller pin + bootstrap.json seed) and the
> restored token store's 9201 entry matches the controller's current `local_api.token` — **zero
> in-guest change.** Gates passed: leaf loaded (not regenerated), wire fp `60b5974d…`, authenticated
> `GET /disks` 200/5-disks (bogus token 401), and the next quiesce cycle ran a real backup job.
> Rollback point kept at `/root/agent-rollback-20260629-150137/`. **OPEN (carried forward):** the
> migration/Day-0 path must carry these three files into the new `felhom-agent`-owned data dir (or
> treat their loss as a re-bootstrap trigger), else any re-migration reintroduces this. Full record:
> `felhom.eu/documentation/audits/SPIKE-agent-pin-mismatch-2026-06-29.md` (§9 Outcome).

> **2026-06-27 — v0.86.0 (deployed on 9201): Phase 2 managed updates — controller-version FLOOR.**
> On top of the Phase 1 opt-in "update to latest" button, the controller now honors an operator-enforced
> **minimum version (FLOOR)** delivered on the hub **report ACK** (`min_controller_version` +
> `latest_version`; `internal/report/pusher.go` `PushResponse`). `OnPushResponse` →
> `updater.SetFloor()` + `updater.MaybeAutoUpdate()` (rides the report cycle — no new timer). Below the
> floor → **auto-update to the floor** (reuses Phase 1 `performUpdate`: in-guest pull → agent swap →
> rollback; `initiatedBy="auto-floor"`). At/above floor → nothing (does NOT chase latest — that's the
> button). Guards: dev/no-agent/backup → skip; floor must be pullable (floor ≤ latest; floor>latest →
> warn+noop); one attempt per below-floor condition (in-mem + persisted state) → no flapping. Floor source
> + operator UI are hub-side (felhom-hub v0.15.0: per-customer override + global default + report ACK).
> **No agent change** (reuses Phase 1 `POST /controller/swap`). Day-0 now ships current (golden rebuilt at
> 0.85.1) AND stays current (floor) — the fleet-currency story is closed. Live: dogfood 0.85.1→0.86.0 via
> the button, then floor 0.87.0 → auto 0.86.0→0.87.0 (no click).


> **2026-06-26 — v0.84.0 (deployed on 9201): show an app's auto-generated first-login on its page.**
> General, catalog-driven mechanism: `.felhom.yml initial_credentials: {file, format json|regex|plain,
> username_key/password_key | username_pattern/password_pattern, note}`. The controller reads the file
> **live** from the container (`internal/stacks/initialcreds.go` `ReadInitialCredentials` + pure
> `parseInitialCreds`, unit-tested), never persists it, and renders a "Kezdeti belépési adatok" card on
> `/apps/{slug}` (masked password + reveal/copy). First consumer: crafty-controller (Crafty writes a
> random admin password to `/crafty/app/config/default-creds.txt`). Live-verified on 9201: card shows
> username `admin` + the real extracted password. Reuse for any future self-seeding app. Security: same
> exposure class as the existing post-deploy reveal / default_creds card — relies on the prod dashboard
> being auth-gated (demo public-unauth is the separate tracked issue). Backlog idea unchanged: a
> `backend_scheme` hint for TLS backends (v0.83.0 line).

> **2026-06-26 — v0.83.0 (deployed on 9201): scoped Traefik backend transport for self-signed HTTPS apps.**
> Crafty is the **first/only catalog app with an HTTPS backend** (self-signed TLS on `:8443`, no plain-HTTP
> port). The crafty healthcheck fix un-withheld its route, exposing a pre-existing 502 (Traefik proxied
> HTTP to the HTTPS backend). Since traefik v3 forbids `insecureSkipVerify` via Docker labels, the
> controller now renders a file-provider dynamic file `dynamic/serverstransports.yml` defining a **named**
> `insecure-skip-verify` transport (`infra.RenderServersTransports` + `stacks.ensureServersTransports`,
> written from `EnsureBaseStack` outside the traefik-running guard, write-if-changed, hot-loaded). Apps opt
> in per-service via catalog labels `scheme=https` + `serverstransport=insecure-skip-verify@file`.
> **No global `insecureSkipVerify`** — verification stays ON for every other backend (scoped Option B).
> Pattern to reuse for any future self-signed HTTPS backend. Live-verified: `minecraft.demo-felhom.eu`
> 502→302; filebrowser (HTTP) unaffected. Backlog: a `.felhom.yml backend_scheme` hint so the catalog
> convention sets these labels instead of hand-adding.

> **2026-06-22 — v0.76.0 (deployed on 9201): three campaign-#3 hardening fixes.**
> - **S1**: corrupt `settings.json` no longer crash-loops — `save()` writes a last-known-good `.bak`
>   (after the primary rename); `Load()` recovers from `.bak`, else preserves the corrupt file as
>   `*.corrupt-<ts>` + safe defaults. New `Settings.LoadWarning` → dashboard banner. (`main.go` Fatalf
>   now only on the IO-unreadable path.)
> - **F2**: `web.validStackName` gates `backupRestoreHandler` + `apiExportStart` (reject `/ \ .. NUL`)
>   before any restore/export — closes the traversal defense-in-depth gap (no escape had occurred, but
>   it relied on downstream map-lookups).
> - **S3**: `quiesce.readMarker` logs `[WARN]` + quarantines a corrupt marker to `*.corrupt-<ts>`
>   instead of silently dropping it.
> All three live-validated on 9201 (truncate settings → recover from .bak no crash-loop; traversal
> restore → rejected, /etc intact; corrupt quiesce marker → quarantined). Tests + red-proofs.
> Still-open campaign-#3 deferrals (NOT done): time-chaos on a dedicated VM, host reboot (supervised),
> `.fab` import compose-fuzzing. See `felhom.eu/documentation/tests/test-campaign-3-2026-06-22-findings.md`.

Last updated: 2026-06-22 (v0.75.0 — gate userdata MkdirAll on a live mountpoint)

> **2026-06-22 — v0.75.0 (deployed on 9201): userdata MkdirAll gated on a live mountpoint.**
> Both `MkdirAll`-into-`<drive>/userdata` sites — the deploy belt (`stacks.ensureUserdataMounts`) and
> the FileBrowser sync (`web.syncFileBrowserMounts`) — now skip when an **external** drive root (under
> `StableParentDir=/mnt/felhom-drives`, `!= sysDataPath`) is **not a live mountpoint** (`system.IsMountPoint`).
> Closes the campaign-#2 hazard where a drive-absent window produced `mkdir …/userdata: permission denied`
> + transient `Created` flapping AND could write app data onto the guest **rootfs** (shadowed when the
> drive returns). The app is held by `planDriveGates` instead. System/local path is never gated. New
> `Manager.isMountPoint` seam + pure `web.skipFileBrowserPath` helper for tests. Live-proven via a C6
> re-run: 0 permission-denied, 0 shadow dirs on the rootfs, apps recover on reconnect.
>
> **Boot-ordering design note (NOT implemented — decide separately):** the *boot-time* `mkdir …
> permission denied` is a different cause — **docker's** boot-restore auto-starts drive-backed
> containers (`restart: unless-stopped`) before the agent mounts the drives, so docker (not the belt)
> tries to create the bind source; `planDriveGates` recovers them after mount convergence (why a reboot
> ends healthy). Options: (A) accept + suppress (apps self-heal; lowest risk; maybe downgrade the
> boot-window log level) — recommended now; (B) drive-backed app containers don't docker-auto-start at
> boot (restart policy `no`/`on-failure`) so the controller's gate is the sole starter after mounts
> converge — cleaner but bigger (must cover crash-restart too). See
> `felhom.eu/documentation/tests/test-campaign-2-finding1-recovery-diagnosis.md` (boot-ordering observation).

Last updated: 2026-06-22 (v0.74.0 — controller→agent connection-leak fix)

> **2026-06-22 — v0.74.0 (deployed on demo guest 9201): fixed the controller→agent connection leak.**
> `Server.agentClient()` built a new `agentapi.Client` (new bare `http.Transport`, `IdleConnTimeout:0`)
> per call → one leaked idle ESTABLISHED socket per agent call to `192.168.0.162:8443`, exhausting the
> ephemeral source-port range after ~5 days of controller uptime → EADDRNOTAVAIL, which had taken down
> storage UI + host-metrics + whole-guest backup (found in the 2026-06-22 unattended campaign). Fix:
> memoize ONE shared client via `sync.Once` + harden the Transport (`MaxIdleConnsPerHost:2`,
> `IdleConnTimeout:90s`). Live-proven: idle sockets to `:8443` stay flat at 2 across a 120-call burst
> (pre-fix grew ~1/call → 132). Agent/firewall untouched. Diagnosis + campaign findings live in
> `felhom.eu/documentation/tests/unattended-test-campaign-2026-06-22-*.md`. **Separate open item:** the
> defense-in-depth host firewall rule scoping `:8443` to the guest bridge subnet is still absent
> (pve-firewall disabled) — to be closed independently.

Last updated: 2026-06-13 (v0.60.0 backlog-Medium cleanup)

> **Live version: controller v0.60.0** (deployed on demo guest 9201), agent **v0.30.0** (AGENT-001 deployed), hub **v0.11.0**.
> **NOTE:** the long-form sections far below this banner are stale (last full pass ~v0.16.1). Current
> state is the dated entries immediately below + `CHANGELOG.md` + the now-authoritative **central docs at
> `felhom.eu/documentation/controller/`** (code-verified) + auto-memory `MEMORY.md`.
>
> **2026-06-13 — v0.60.0 backlog-Medium cleanup (BUGHUNT reconcile):**
> - **M25** (Server.integrationMgr data race — constructor goroutine vs post-construction `SetIntegrationManager`):
>   **FIXED** via `atomic.Pointer`; `-race`-verified. **M4/M5/M6** verified already FIXED (no action).
> - **M18** (dump re-validation every 5 min — perf) and **M19** (`deriveStackName` misattribution — low
>   incidence) verified LIVE but cross-package-entangled → branches `fix/m18-dump-validation-cache` /
>   `fix/m19-stackname-crossref` (notes + fix plan, PENDING REVIEW, not deployed).
>
> **2026-06-13 — v0.59.0 audit fixes + documentation centralization:**
> - Fixed the validated 2026-06-13 audit findings (records: `felhom.eu/documentation/audits/`):
>   **CTRL-001** (`.fab` import path traversal — manifest segment validator, fail the parse);
>   **CTRL-T2-1** (ghost-deployed on crash — `app.yaml` now persists `deployed:false` until `compose up -d`
>   succeeds, flipped true only on success; in-memory flag still true during pull for UX);
>   **H10** (plaintext secret on encrypt failure — `SaveAppConfig` now fail-closed, returns an error);
>   **M2** (misleading lock on init-only `SetStackProvider` removed). All with regression tests.
> - **AGENT-001** (wrong-disk wipe TOCTOU) fixed on agent branch `fix/agent-001-wipe-durable-reresolve`
>   — PENDING REVIEW, NOT deployed (supervised merge+golden-rebake reserved; see `AGENT-001-FIX-NOTES.md`).
> - Built + deployed v0.59.0 to demo guest 9201 (bootstrap mechanism); verified healthy, dashboard 200.
> - Documentation centralized under `felhom.eu/documentation/` (controller subtree + audits + top index),
>   code-verified; `controller/README.md` banner points there; this CONTEXT banner refreshed.
>
> **2026-06-13 — v0.58.0 OS/Docker-data split prevention layer (Phase 2; Phase 1 = agent v0.29.0):**
> - OS rootfs + Docker data split onto separate local-lvm volumes (golden bakes 32G rootfs + 256G
>   /var/lib/docker, backup=1, **overlay2** so images live on the data vol). Infra protected by
>   PREVENTION not placement: `system.GetDockerVolumeHeadroom()` (statfs "/" = data vol) reserves
>   max(5GB,10%); `deployStack` refuses HTTP 507 below buffer; deploy.html banner+disable; monitor
>   (warn80/crit90) watches the same vol; log rotation baked into the golden daemon.json.
> - Live-validated by DESTROYING + RE-PROVISIONING 9201 from the split golden: split layout, overlay2,
>   images on data vol, lean rootfs, OS isolation (data vol 100% → rootfs healthy), deploy gate 507,
>   ActualBudget volume on data vol, hub config pull, external access via CF. RomM/USB re-enroll = the
>   documented final restore step (RomM data safe on host USB). See memory [[os-data-split]].
>
> **2026-06-13 — v0.57.0 UI fixes (Part A of the UI-fixes/storage-spike spec):**
> - A1: fixed the RIGHT storage list — `#host-storage-bars` (the JS-filled, agent-PVE-storage list:
>   `local`/`local-lvm`/`felhom-pbs`/`felhom-usb`), which reordered on every poll. Now
>   `enrichHostStorageTargets` sorts `/api/host-metrics` server-side + adds friendly Hungarian
>   labels/purpose. Display-only — PVE storage ids never renamed. (v0.56.0's 4C had sorted the OTHER,
>   server-rendered user-data list.)
> - A2: per-app Tier-2 config panel at `GET/POST /stacks/{name}/backup`; the dead-end "Beállítás" button
>   (was → deploy page) is repointed there. Pin a target drive / toggle Tier 2 off; prefs
>   (`UserDisabled`/`PreferredTarget`) persist on `CrossDriveBackup` and survive the runner's status
>   writes (`withTier2Prefs`). Always visible incl. single-SSD + non-HDD (PBS-context) apps.
> - Part B (storage OS/data split spike) = build-nothing; findings → `felhom-agent/REPORT-storage-split-spike.md`.
> - Live-validated on guest 9201; build/deploy = golden bootstrap (`/etc/felhom-controller-image` + restart `felhom-controller-bootstrap.service`).
>
> **2026-06-13 — v0.56.0 Phase 4: FileBrowser scoping + UI polish (SLICE COMPLETE):**
> - 4A: FileBrowser bind scoped to `<drive>/appdata` (recovery units + Tier 2 copies under `backups/`
>   NOT mounted → customer can't browse/delete the restore source). 4B: deploy storage step states
>   files-on-drive / DB-on-fast-SSD. 4C: `buildStorageBars` stable sort + purpose description on the
>   monitoring list (user-data drives only; agent local/local-lvm/pbs live on the storage page, not here).
> - Live-validated (9201): FileBrowser mount `/mnt/felhom-usb/appdata -> /srv/felhom-usb` (backups hidden);
>   deploy + monitoring text rendered. **All 5 phases (1, 2, 2b, 3, 4) shipped + live-validated, v0.52→v0.56.**
>
> **2026-06-13 — v0.55.0 Phase 3: auto off-drive Tier 2 (rootfs-headroom guard):**
> - `internal/backup/tier2.go`: rsync `-a --delete` of each HDD app's recovery unit + appdata → a
>   DIFFERENT physical disk (`<target>/backups/secondary/<app>/`). Auto target: prefer another registered
>   drive (off-disk via `system.SamePhysicalDevice`), else internal SSD for SMALL units only.
> - **Rootfs-headroom guard** (`tier2FitsHeadroom`, unit-tested): SSD = ~8G guest rootfs, so REFUSE
>   unless the unit fits leaving reserve = max(2G, 20%) free; honest "needs 2nd HDD" status when nothing
>   fits — never fills the rootfs. Status via surviving `settings.CrossDriveBackup`; "2. mentés" UI card
>   now populated (`buildAppBackupRows`). Daily `tier2-backup` 03:30 + `POST /api/backup/tier2`.
> - **Live-validated (9201):** happy path (RomM → SSD, off felhom-usb, 77KB, "[SSD: DB/config only]");
>   refuse path (1G userdata dummy → REFUSED with honest msg, rootfs not filled); UI card shows
>   "Sikeres → belső SSD (csak DB/konfiguráció)". Demo cleaned.
> - Next: Phase 4 (FileBrowser scoping + deploy-UI DB-on-SSD note + monitoring sort).
>
> **2026-06-13 — v0.53.0/v0.53.1 Phase 2: per-app recovery unit (capture side, SECRET-FREE):**
> - Each app's `backups/primary/<app>/` becomes a self-contained recovery unit: `compose/`
>   (docker-compose.yml + .felhom.yml + **secret-stripped** app.yaml) + db-dumps/ + volume-dumps/ +
>   `manifest.json` (image pins, secret env-var NAMES, data_key names, checksums, secret_source note).
> - **Secret-free by design.** Decided after reading the ACTUAL hub code: hub is zero-knowledge (no app
>   secrets); app.yaml + key live on the guest rootfs → in the PBS whole-guest snapshot. So the unit
>   stores no secret/data-key/image; restore recovers secrets from the guest's app.yaml (live/PBS),
>   regenerates nothing. `data_key` (DeployField.DataKey; AdventureLog SECRET_KEY marked) = fail-closed
>   restore annotation only.
> - Capture needs no decryption (non-secret env is plaintext; excludes secret-named + encrypted keys).
>   Wired into RunDBDumps AND the periodic RefreshCache (idempotent checksum-skip → no USB thrash).
> - **Deploy mechanism resolved:** controller in guest 9201 is golden/bootstrap-managed —
>   `felhom-controller-bootstrap.service` docker-runs the tag from `/etc/felhom-controller-image`
>   (gitea anon-pull). Deploy = build+push → anon-pull → update tag file → restart the service.
> - **Live-validated (9201):** RomM unit captured (images=3, secrets=3, data_keys=0), secret-leak grep
>   = NO_LEAK.
> - **v0.54.0 Phase 2b (restore-from-unit + fail-closed gate):** `RestoreFromRecoveryUnit` recreates an
>   app from its unit + secrets recovered from the GUEST's live app.yaml (`RecoverStackSecrets`,
>   `stacks.RedeployFromEnv`), regenerating nothing. `reconcileRestoreSecrets` (pure, unit-tested) is the
>   fail-closed gate: missing/empty data-key → REFUSE (needs PBS whole-guest restore); missing resettable
>   secret → warn+proceed. Wired into `/backup/restore`. Gate + orchestration + data_key parsing
>   unit/integration-tested; deployed v0.54.0 healthy.
> - **LIVE-validated (9201, AdventureLog):** unit manifest `data_key_env_vars:[SECRET_KEY]`
>   (catalog→manifest live); with SECRET_KEY made unrecoverable, `POST /backup/restore` REFUSED with the
>   exact fail-closed message BEFORE any compose-up. Demo has NO dashboard password → API open (auth+CSRF
>   skipped), driven via public URL. NOTE: full deploy-with-data→restore e2e blocked because AdventureLog
>   images don't fit the 8G guest rootfs ("no space left") — that's the Phase 3 rootfs-headroom concern
>   seen live. Demo left clean (AdventureLog reverted to not-deployed).
> - Next: Phase 3 (Tier 2 auto off-drive, rootfs-headroom guard), Phase 4 (FileBrowser + UI).
>
> **2026-06-13 — v0.52.0 Phase 1 GATE: deploy-side double-nest fix (catalog) + path-agreement test:**
> - The `felhom-data` double-nest lived in the **app-catalog compose templates**
>   (`${HDD_PATH}/felhom-data/appdata/<app>`), not in `deploy.go`. On a Model-A in-guest drive the mount
>   already IS the `felhom-data` namespace, so it double-nested on disk while the v0.51.0 backup helpers
>   resolved single-nested → divergence. Fixed all four HDD templates (romm, nextcloud, immich,
>   paperless-ngx) → `${HDD_PATH}/appdata/<app>`.
> - New `internal/stacks/hddpath_agreement_test.go` locks deploy-resolver (`ParseComposeHDDMounts`) ==
>   backup helper (`AppDataDir(NamespaceRoot(.,true))`). No controller runtime change → no image rebuild
>   (deployed stays 0.51.0, functionally current; golden not rebaked for a no-op).
> - **Live (guest 9201):** git-sync auto-delivered the fix to all four stack files; RomM migrated
>   (stop→move→verify→redeploy) from `/mnt/felhom-usb/felhom-data/appdata/romm` →
>   `/mnt/felhom-usb/appdata/romm`, healthy + HTTP 200, no data loss, old namespace empty. **GATE PASSED.**
> - Next: Phase 2 (per-app recovery unit), Phase 3 (auto-enabled off-drive Tier 2 w/ rootfs-headroom
>   guard), Phase 4 (FileBrowser scoping + deploy-UI DB-on-SSD note + monitoring sort).
>
> **2026-06-12 — storage UX polish (v0.45.0), pairs with felhom-agent v0.24.0:**
> - **Agent eject role-gate (Part A, felhom-agent v0.24.0):** `POST /disks/eject` now refuses to
>   unmount system/backup storage *at the agent* (fail-safe to protected on ambiguity) — the UI hiding
>   the button was never the control. Validated live on guest 9201 (eject `/var/lib/vz` → 403, no unmount).
> - **Controller (Part B, v0.45.0):** B1 deterministic `/api/disks` order (user-data→system→backup,
>   alpha within); B2 init wizard excludes mounted drives; B3 **Regisztrálás** primary action for a
>   mounted-but-unregistered user-data drive (`POST /api/storage/register`); B4 per-card purpose
>   descriptions + app-backing tags + tiering note (`local` & `local-lvm` both kept); B5 eject already
>   names affected apps. All validated live on guest 9201.
> - **Golden REBAKED** to controller 0.45.0 (`/root/build-golden.sh`; gitea allows anon pull, no creds);
>   archive `local:backup/vzdump-lxc-9100-2026_06_12-09_53_03.tar.zst`, build guest 9100 purged.

---

## About Viktor (project owner)

- Works at Deutsche Telekom (Budapest), building Felhom.eu as a side business
- Felhom.eu: managed home-server service for Hungarian households
- Technical but prefers pragmatic solutions over over-engineering
- Runs all infrastructure on Gitea (gitea.dooplex.hu), k3s cluster for management
- Customer deployments use Docker Compose (not Kubernetes) for simplicity

### felhom-controller (this repo)
- **Version:** v0.16.1
- **Phase 1:** ✅ COMPLETE — Stack Manager + Deploy Flow
- **Phase 2:** ✅ COMPLETE — Monitoring & Health (scheduler, CPU/temp, healthchecks.io pings)
- **Phase 3:** ✅ COMPLETE — Backups (DB dumps, restic integration, manual trigger, **dedicated backup page**)
- **Phase 4:** ✅ COMPLETE — Monitoring Page with Metrics Store (SQLite, Chart.js, system + container metrics)
- **Phase 5:** ✅ COMPLETE — Authentication, Persistence & Settings Page (settings.json, password change, session management)
- **Phase 6:** ✅ COMPLETE — Monitoring Warnings, Dashboard Alerts & Notification System
- **Phase 7:** ✅ COMPLETE — Storage Overview, Per-App Backup Toggles & Limited Restore
- **Phase A:** ✅ COMPLETE — Storage Paths Foundation (registry, auto-discovery, per-app HDD_PATH, deploy dropdown, health monitoring)
- **Phase B:** ✅ COMPLETE — Storage Management UI Polish & Health Severity Fix (flash messages, label editing, app details, FS info, deploy free space, backup context)
- **Phase C:** ✅ COMPLETE — Storage Init Wizard, Data Migration & Startup Fix (disk scan/format/mount wizard, rsync-based migration, startup pings)
- **v0.11.1 bugfix:** ✅ COMPLETE — Storage Scan: system disk detection via host fstab + blkid UUID resolution; FSType enrichment via `blkid -o export`
- **v0.11.2 bugfix:** ✅ COMPLETE — /host-dev mount for block device access; `HostDevicePath()` helper; all format/scan/safety ops use /host-dev
- **v0.11.3 bugfix:** ✅ COMPLETE — Added `fdisk` package to Dockerfile (provides `sfdisk`; not in `util-linux` on Debian bookworm)
- **v0.11.4 bugfix:** ✅ COMPLETE — FormatAndMount: fixed sfdisk (wipefs+force+`,,`), mount (explicit device path), mount propagation (rshared), ASCII label, smart partition skip, findmnt verification
- **v0.11.6:** ✅ COMPLETE — FileBrowser auto-mount sync (`syncFileBrowserMounts()`) + 3 UI fixes (badge color, progress bar, button text)
- **v0.11.7:** ✅ COMPLETE — Stale data cleanup + FileBrowser sync after migration + deploy page title fix
- **v0.11.8:** ✅ COMPLETE — Per-App Cross-Drive Backup (3-2-1 rule): rsync/restic to secondary drive, deploy page UI, backup page summary, scheduler jobs, API endpoints
- **v0.11.9:** ✅ COMPLETE — UI Polish Fixes: spacing, tooltip on "Módszer", status dot instead of disabled checkbox, progressive disclosure, emoji cleanup
- **First app deployed:** Paperless-ngx on demo-felhom.eu (2026-02-13)
- **Running on:** demo-felhom (N100 mini PC) at 192.168.0.162:8080, felhotest (Proxmox VM) at router.abonet.hu:33022
- **All Phase 1-5 features working:** deploy, start/stop/restart/update, logs, health-aware states, auth, monitoring, backups, backup detail page, system monitoring page, settings page

## Architecture decisions

| Decision | Rationale |
|----------|-----------|
| Go stdlib for web (no Gin/Echo) | Minimal dependencies, single binary, easy to embed templates |
| Templates as go:embed HTML/CSS files | Zero runtime file dependencies (compiled into binary), but each template is a separate editable file |
| Docker Compose for customers (not k8s) | Simpler troubleshooting, customers don't need k8s knowledge |
| k3s for management infra only | Viktor's own services (gitea, monitoring, website) run on k3s |
| Cloudflare Tunnel for remote access | No port forwarding needed, works behind any NAT |
| app.yaml per stack | Separates deploy config from compose files, survives git pulls |
| Password fields require explicit input | Prevents accidental empty-password deployments |
| Health-aware state from Docker Status field | Docker's State says "running" even for unhealthy containers |
| Memory limits via deploy.resources.limits | Prevents runaway containers; ~50% headroom over expected usage |
| System info from /proc/meminfo + statfs | No external dependencies, cheap to read on each page load |
| mem_request vs mem_limit (K8s-inspired) | Requests = expected usage (hard block), limits = peak (overcommit OK) |
| 384MB reserved for system | Prevents deploying apps that would starve the OS/controller |
| Logo SVG embedded as Go constant | Same approach as CSS/HTML — zero external file deps |
| Git sync via os/exec git CLI | No Go git library needed, git is in the container image |
| SHA-256 for content comparison | Only copy changed files, avoid unnecessary disk writes |
| 30s debounce on manual sync | Prevents spamming the git server |
| Orphan = deployed but not in catalog | Safe lifecycle: remove from catalog → mark orphaned → user deletes via UI |
| FileBrowser as infra (not catalog) | Needed even after apps deleted (user browses HDD data); deployed by setup script |
| Protected HDD paths | Safety net: never delete top-level HDD dirs (media, storage, Dokumentumok, appdata) |
| Central scheduler (not ad-hoc goroutines) | Single place to register/monitor all periodic tasks, graceful shutdown, skip-if-running |
| CPU sampling via background goroutine | /proc/stat delta needs two readings — collector runs every 5s, GetInfo() reads cached value |
| Temperature from /host/sys (Docker mount) | Container can't read host /sys directly — mount /sys:/host/sys:ro, try /host/sys first |
| Restic password auto-generated | No manual setup needed — generated on first backup run, stored in named volume |
| DB discovery via docker inspect | No config needed — discovers postgres/mariadb containers by image name + env vars |
| Backup orchestrator with running flag | Prevents concurrent backups, supports both scheduled and manual trigger |
| modernc.org/sqlite (pure Go) | No CGO/gcc needed in Docker build stage — keeps `CGO_ENABLED=0` static binary |
| AlertManager state-based refresh | Alerts regenerated every 5min from health report — no persistent storage needed, always reflects current state |
| Notification relay via hub | Controller → hub → Resend → email. Hub acts as central relay: knows customer email, handles Resend API. Controller only needs hub URL + API key |
| In-memory notification cooldowns | Per-event-type cooldown map (default 6h). Lost on restart = acceptable (better to re-notify than miss). No persistence needed |
| Health status change detection | Only notify on degradation (ok→warn, ok→fail, warn→fail). Avoids spam on flapping. First run records baseline, doesn't notify |
| Resend HTTP API (no SMTP) | Direct POST to api.resend.com — same pattern as website contact-mailer. Simpler than SMTP setup, good deliverability |
| Preferences sync on save + startup | Controller pushes prefs to hub (not pull). Startup sync handles hub DB rebuild. Local save always succeeds even if sync fails |
| Chart.js embedded locally | Customer hardware may not have internet — CDN not reliable for offline environments |
| StackDataProvider interface | backup package needs stack data but can't import stacks (circular). Interface in backup, thin adapter in main.go |
| Password sync to hub via report | Restic password in Docker named volume on SSD. Hub sync provides redundancy for disaster recovery |
| App backup via HDD mounts only | Docker volumes at /var/lib/docker/volumes/ not mounted in controller. HDD data is the important user data; DB in volumes covered by nightly dump |
| Restore uses running mutex | Prevents concurrent backup+restore on same restic repo. Reuses existing `m.running` flag |
| Storage paths registry in settings.json | Multi-storage support: each app's HDD_PATH from app.yaml is authoritative. Auto-discovery on startup avoids manual config. Registry enables UI management + health monitoring per path |
| /mnt:/mnt:rw mount in controller | Replaces per-path HDD_PATH mount. Enables multi-storage + restore writes. All customer HDD mounts are under /mnt/ by convention |
| Per-app HDD_PATH resolution (app.yaml > global) | App's own env HDD_PATH is Priority 1, registered storage paths as fallback. Eliminates dependency on global controller.yaml hdd_path |
| Mount-point detection via syscall.Stat_t.Dev | Compares device ID of path vs parent dir — reliable check that path is on separate filesystem. Prevents data writes to SSD |
| Health severity: mount-point = warning | Non-mount-point is informational, not a service failure. FAIL reserved for genuinely broken things. Avoids false alarms on demo/test environments |
| FS info via findmnt + sysfs | `findmnt -n -o SOURCE,FSTYPE --target <path>` for filesystem type/device. `/sys/block/<dev>/device/model` for disk model. Best-effort, returns nil on failure |
| Query param flash messages | Stateless, no session store needed. Consistent with backup page pattern. `?storage_msg=success&storage_detail=...` |
| StorageLabels map on stacks page | Separate map passed to template (not modifying Stack struct). Built from deployed apps' HDD_PATH → registered path label lookup |
| Metrics downsampling via SQL | Bucket-based AVG in GROUP BY keeps Chart.js responsive with up to 30 days of data |
| 60s metrics collection interval | Good balance of resolution vs. storage — ~44K rows/month for system metrics |
| /etc/os-release mounted read-only | Container can't read host OS info directly — mount to /host/etc/os-release:ro |

## Key file locations on demo-felhom

```
/opt/docker/felhom-controller/         # Controller compose + config
  ├── controller.yaml                  # Customer config (domain, auth, paths)
  ├── docker-compose.yml               # Controller's own compose
  └── data/                             # Controller persistent data (named volume)

/opt/docker/stacks/                    # All app stacks
  ├── traefik/                         # Reverse proxy (protected)
  ├── cloudflared/                     # Tunnel (protected)
  ├── paperless-ngx/                   # First deployed app ✅
  │   ├── docker-compose.yml
  │   ├── .felhom.yml                  # App metadata
  │   └── app.yaml                     # Deploy config (env vars, locked fields)
  └── whoami/                          # Test stack (not deployed)

/mnt/hdd_placeholder/storage/          # HDD storage for apps
  └── paperless/
      ├── consume/                     # Drop files here for OCR
      ├── media/                       # Processed documents
      └── export/                      # Backup exports
```

## Related repositories and their state

| Repository | Status | Notes |
|------------|--------|-------|
| felhom-controller | Active | This repo. Controller code + deploy scripts |
| app-catalog-felhom.eu | Active | 10 app templates, all with .felhom.yml metadata + memory limits |
| felhom.eu | Active | Website + hub/ subfolder (felhom-hub service) + k8s manifests |
| homelab-manifests | Stable | k3s cluster running (dooplex.hu services) |
| misc-scripts | Utility | collect-repo.sh, backup helpers |

## Gotchas & lessons learned

- `docker compose restart` ≠ `docker compose up -d` — restart doesn't pick up new images
- Go maps have random iteration order — always sort slices before displaying
- Docker `.State`="running" doesn't mean healthy — check `.Status` for "(health: starting)" / "(unhealthy)"
- Paperless-ngx needs `PAPERLESS_OCR_LANGUAGES` (plural) to install language packs, `PAPERLESS_OCR_LANGUAGE` (singular) to select
- In-memory Deployed flag must be set BEFORE `docker compose up -d` (not after) — compose can take 30-60s for image pulls, during which the UI would show a stale "Telepítés" button
- Cloudflare Tunnel handles *.demo-felhom.eu → Traefik handles Host()-based routing to containers
- BIOS "AC Power Recovery" must be enabled on N100 for auto-restart after power outage
- `docker compose up -d` returns exit 0 even when containers immediately crash-loop — need post-start status check to detect this
- When logging env vars for debugging, only log keys (not values) to avoid leaking secrets in log files
- Mealie image (`ghcr.io/mealie-recipes/mealie`) doesn't include wget/curl — use Python TCP socket check for healthcheck
- Mealie DB migrations on first start take ~40s (alembic) — use `start_period: 60s` to avoid premature unhealthy status
- Alpine-based images (filebrowser, vaultwarden) have wget via BusyBox — healthchecks with `wget --spider` work fine
- Deploy `sed` command to update image version must target only the `image:` line — naive `sed 's|name:OLD|name:NEW|'` also matches the service name line (e.g., `felhom-controller:` → `felhom-controller:0.2.12`), breaking YAML. Use `sudo sed -i 's|image:.*felhom-controller:[^ ]*|image: ...felhom-controller:NEW|'` or similar scoped pattern
- Hungarian quotation marks `„"` in YAML: `„` (U+201E) is safe inside YAML double-quoted strings, but the closing `"` must NOT be ASCII `"` (0x22) — it terminates the YAML string. Use `\"` escape or Unicode `"` (U+201D). This caused a silent parse failure for the entire `.felhom.yml` file
- Never silently swallow parse errors — always log them. Silent failures make debugging impossible (took a dedicated debug session to find a simple quoting issue)