felhom-controller/CONTEXT.md

# CONTEXT.md — Project Memory

> This file serves as persistent project memory across Claude Code sessions.
> It replaces the auto-generated "Memory" from the claude.ai Project.
> **Update this file at the end of each working session** with current state,
> recent decisions, and anything the next session needs to know.
>
> Ask Claude Code: "Please update CONTEXT.md with what we did today"

Last updated: 2026-07-16 (v0.138.0 — escrow "awaiting hub confirmation" waiting state)

> **2026-07-16 — v0.138.0: escrow "awaiting hub confirmation" waiting state.** Fixes the customer-zero
> (N100) UX gap: after a completed escrow ceremony the Távoli mentés page kept showing the yellow
> "Helyreállítási kód szükséges" card for ~15 min until the next hub-report ACK flipped
> pending→escrowed. **Phase-0 diagnosis (read-only) = verdict A (report-cycle lag), already resolved on
> the box:** demo logs show the ceremony claimed `16:13:39`, the next `hub-report` ACK at `16:27:58`
> auto-confirmed via hash-match (`d517ce7f…`); `settings.json` = `escrow_state:"escrowed"`. **Hub
> Hypothesis B verified FALSE → no hub change:** `SaveHostEscrow`'s `ON CONFLICT … stale_at = NULL`
> already clears stale on upload (store.go:2084); withhold only fires while `stale_at != ""`
> (store.go:2150). **Part 1 (code):** new persisted `OffboxTarget.CeremonyCompletedAt` (stamped on the
> recovery-code claim, zeroed on the flip + manual confirm); `offboxCeremonyWaitState` +
> `escrowCeremonyGraceWindow`=35m; `backups_remote.html` gains an info "megerősítésre vár, legfeljebb 15
> perc" card → warn "a megerősítés nem érkezett meg" past the window; `backups_escrow.html` final step
> gains a "Mi történik ezután?" note. Test `web/escrow_wait_state_test.go` + red-proof. No
> scheduler/agent/hub/endpoint changes. Deploy 0.136.0→0.138.0 to 9201. First "Távoli mentés most" =
> Viktor's click (NOT done). Note: demo host key changed (box reprovisioned for N100) → known_hosts
> refreshed.


> **2026-07-15 — v0.137.0: cleanup bundle (email-wipe guard + carried hygiene).** Closes the arc's
> carried micro-queue. **Part 1 (code):** `settingsNotificationsHandler` now REFUSES a save with a
> blank email box while events are enabled (it would push empty to the hub → wipe the customer's
> provisioning-seeded alert address — the 2026-07-15 demo incident). Returns before
> SetNotificationPrefs + sync, Hungarian error, repaints submitted checkboxes; empty+zero-events
> clear-all still allowed. No HTML `required` (it would block the legit clear-all). Tests + red-proof
> in `web/notifications_guard_test.go`. Deployed 0.137.0 to 9201 (healthy). **Part 2 (hygiene):**
> removed the confirmed-older `felhom-flash/backups/primary/immich` recovery unit (CreatedAt 06-23 <
> live usb 07-15, 44M); STOPPED audiobookshelf/komga/romm on flash (CreatedAt TIED with usb →
> tie-break is drive-order-dependent, not confirmable — manual disposition pending). **Part 3:**
> campaign6 is a bare empty leftover dir (not a live mount); safe `rmdir` refused (Permission
> denied — autofs-ghost/immutable); no mount disturbed → left for Viktor's reboot window. **Part 4:**
> tagged the campaign6 6D-audit finding track-only (felhom.eu `dee72cd`).

> **2026-07-15 — v0.136.0: `.fab` exclusion scoping (Task 4).** Architecture §2 `.fab` row + SQ5
> verdict + R1-C. SQ6 over-capture FIXED for classified apps: the userdata root tar is exclude-scoped
> (keeps only ancestors/descendants of a SELECTED bind relpath — the tier2Reconcile keep-rule); no
> selected userdata bind ⇒ no root tar (radarr state-only). New `appbackup.ComputeFabBuckets` (shared
> `resolveGuardCollapse` pipeline; class buckets; guards over ALL classes; no cross-bucket containment).
> New `appexport/fabplan.go`: `computeFabPlan` (SkipMounts/SkipUserdataTar/UserdataExcludeRels) +
> `tarDirectoryExcluding` + `fabEstimateSplit`. `ExportRequest` += DeselectOptional/OptInExcluded (both
> start handlers — two-call-site); mandatory is a SERVER-SIDE floor. Manifest v1 + import UNTOUCHED;
> legacy apps byte-identical to v0.130.0. Estimate gains an additive class split; export UI shows
> locked-mandatory / optional-checkboxes / excluded-opt-in + the two-number warning. All 6 §10
> red-proofs verified. CAMPAIGN-6D Accept #1 (≥1 GiB .fab full circle) now runs against this shape.

> **2026-07-15 — v0.135.0: tier-2 engine rework (Task 3b).** Architecture §2/§8. New
> `tier2_capture.go`: classified apps get the `TierSecondary` per-bind legs (paperless copy shrinks —
> export drops); legacy apps keep the byte-identical resolver set. v2 relpath-mirroring layout
> (`backups/secondary/<stack>/{.felhom-tier2-layout marker LAST, recovery-unit/, hdd/<rel>/,
> userdata/<rel>/}`) — N>1 native (flat-appdata refusal + `errTier2MultiDir`/`tier2AppDataName`
> deleted). Migration=delete-and-rebuild + reconcile (prunes dest dirs a bind no longer covers); all
> `os.RemoveAll` via `tier2SafeRemove` (refuses outside backups/secondary/). SSD=state-only tier
> (unit+mandatory). `selectTier2Target` never picks NETWORK storage (pinned+auto, F-6C-1). Restore
> reads v2 behind a marker gate (pre-v2 refused); two-subtree missing-only merge. **Part 0:**
> offbox_enlarge_blocked is now a persisted one-time Load seed (`OffboxEnlargeNoticeSeeded`), NOT a
> getter append — opt-out STICKS (fixes the 3a-fix un-disableable checkbox). **Part 0.5:** offsite
> restore scratch prefers a local (non-network) path. Full v2 test suite + all 10 §10 red-proofs
> verified. Every destructive write bounded to backups/secondary/.

> **2026-07-15 — v0.134.1 (+ hub v0.55.0): Task 3a-fix.** Placement hardening in
> `offbox_restore.go`: F-3a-1a live target uses raw `GetStackHDDPath` (not `AppNamespaceRoot` — its
> systemDataPath fallback would merge userdata onto the SSD; empty ⇒ undeployed ⇒ refuse); F-3a-1b
> placement headroom gate; F-3a-4 stat PRE-PASS over all placements before any copy (no partial
> writes); F-3a-3 `mapOffsiteRestorePaths` refuses the namespace root itself; F-3a-2 scratch removed
> on success (place button then gone), kept on failure. Enlarge-blocked notification delivery chain:
> `DefaultEnabledEvents` + `GetNotificationPrefs` append-if-absent migration + settings checkbox +
> handler slice (controller), and hub v0.55.0 allowlists `offbox_enlarge_blocked` — NO
> customerMessages entry (raw dynamic message must survive). +8 controller tests, +2 hub; all §10
> red-proofs verified. Hub LIVE (ArgoCD synced, :0.55.0). Migration trade-off noted (getter re-enables
> on opt-out — future persisted marker). 3a deferred list shrinks after §13 live legs.

> **2026-07-14 — v0.134.0: offsite tier policy engine (Task 3a — FIRST behavior change).**
> Implements architecture §2/§6/§7/§9. Each toggled app's offsite push = ONE multi-path restic
> snapshot (recovery unit + TierOffsite mandatory userdata via `ComputeCaptureSet`); legacy/undeployed
> stay unit-only. New `offbox_capture.go` (`offboxCaptureSet` + loud gaps: restic 0.14.0 silently
> skips missing paths, SP-3.4) + `offbox_restore.go` (ID-first snapshot introspection, scratch off the
> rootfs + headroom gate F-A1, `RestoreOffboxScratch(full)` unit-only default via `--include`,
> `PlaceOffsiteRestore` missing-only merge, pure `mapOffsiteRestorePaths`). Quota → `stats --mode
> raw-data` (SP-1; **displayed size drops once after deploy**). Pre-push enlargement gate blocks the
> userdata enlargement over-quota (unit-only push continues; `OffboxTarget.EnlargedBlocked`;
> edge-triggered notify). `forget --group-by host,tags` on both sites (SP-2). UI: /backups/restore
> three actions (unit / full two-step / place-to-live); /backups/remote per-app blocked note. New
> route `POST /backup/offbox/place`. **HUB FLAG:** `offbox_enlarge_blocked` event needs hub
> allowlist+customerMessages for push delivery (in-dashboard LastWarning works now). +13 tests, all 10
> §10 red-proofs verified. NOT-live-yet (6D): PlaceOffsiteRestore, large full restore, live
> enlarge-block, notification delivery, SQ3 immich full-circle. Tier-2 (3b) + .fab (Task 4) untouched.

> **2026-07-14 — v0.133.0: capture-set computation (Task 3-core, INERT).** Task 3-core of the
> backup-classification-redesign arc (architecture `felhom.eu/documentation/architecture/07-backup-architecture.md`
> §3; spike verdicts `SPIKE-restic-snapshot-shape-2026-07-14.md`). New
> `appbackup/captureset.go`: pure `ComputeCaptureSet(binds, hasClassification, tier, hddPath)` →
> `CaptureSet{HasClassification, Paths []CapturePath, Skipped []SkippedPath}`. Pipeline: legacy
> short-circuit → tier filter (`TierOffsite`=mandatory only, `TierSecondary`=mandatory+optional,
> excluded dropped) → structural guards (traversal / bare HDD drive-root / reserved `backups/` →
> `Skipped` with English reasons; bare userdata allowed) → equal-Abs collapse (mandatory>optional) →
> containment dedup (keep ancestor) → sort by Abs. Slash algebra only (no `filepath`). Pure
> `CrossAppOverlaps` advisory (WARN wiring deferred to 3a/3b). **Deliberately INERT — no engine
> consumes it yet; 3a (offsite policy) and 3b (tier-2 rework) are the consumers.** ARCHITECTURE
> IMPACT from the spike (SP-3.4, already in §2.5): restic 0.14.0 does NOT error on a missing source
> path (exit 0, silent partial snapshot) → the stat-filter in 3a/3b is load-bearing. Wiring test
> through a real Manager (F-S3 no-seam); all 6 §10 red-proofs verified. felhom.eu §3 docs aligned
> (`8d85da7`).

> **2026-07-14 — v0.132.0: backup classification (Task 2, INERT).** Task 2 of the
> backup-classification-redesign arc (spike: `felhom.eu/documentation/audits/SPIKE-backup-classification-2026-07-14.md`).
> Ships the referential-coupling classification as DATA + PARSER + PURE CLASSIFIER, **deliberately
> inert** — no backup tier changes behavior. New `appbackup/classify.go`: `BackupSpec`/`BindSpec`
> (the `.felhom.yml` `backup:` block), `ComposeBind` (`${VAR}`-relative + `:ro`), `ClassifyBinds`
> (SQ5 two-level default: explicit beats `:ro`; unlisted writable→mandatory, unlisted `:ro`→excluded;
> **no block → legacy/false**), `ValidateBackupSpec` (whole-block-reject on any defect). New
> `stacks/classify_binds.go` `ParseComposeClassifiableBinds` (relative-space, keeps `:ro` — NOT
> `ParseComposeHDDMounts`/`ExportDataMounts`, the classifier-input traps). `LoadMetadata` is the
> SINGLE validation choke point (bad catalog block → nil + one `[ERROR]` within one sync cycle →
> legacy). Wired seam `Manager.ClassifiedBinds` + `StackDataProvider.GetStackClassifiedBinds`
> (nil-stubbed in every fake) so **Task 3 (tier policy engine)** consumes a tested seam, not a fresh
> one. Inertness proven: full pre-existing suite green with ZERO test-logic edits. The 13 catalog
> `backup:` blocks ship in the same `app-catalog-felhom.eu` change (controller deployed FIRST so the
> parser validates on first sync). audiobookshelf PENDING-VETO: media/audiobooks ruled **optional**
> (consistency with komga/romm) pending a Viktor veto to excluded. +14 tests, RP-1..RP-4 confirmed.
> **Next: Task 3** consumes `ClassifiedBinds` to scope offsite/tier-2/`.fab` capture by class.

> **2026-07-14 — v0.131.0: F-S2 + F-S3 (compose-derived appdata dir resolution).** Task 1 of the
> backup-classification-redesign arc (spike: `felhom.eu/documentation/audits/SPIKE-backup-classification-2026-07-14.md`).
> The controller assumed `appdata/<stackName>`; paperless-ngx writes `appdata/paperless` (stack
> `paperless-ngx`). ONE canonical resolver `appbackup.AppDataDirNames(hddPath, stackName, mounts)`
> derives the real dir name(s) from compose `${HDD_PATH}` binds (deduped/sorted; fallback `[stackName]`);
> all consumers use it. **F-S2** (spike-proven): `RunTier2`/`Tier2Info`/`RestoreTier2Files` now hit the
> resolved dir (paperless documents got NO tier-2 copy before — the appdata leg stat-skipped a dir that
> never existed). **F-S3 (NEW, found this session):** `migrate.go` keyed all six per-app appdata legs by
> stack name; **scope="app"** has no merge walk, so migrating paperless-ngx copied nothing, verified
> vacuously, flipped HDD_PATH → **empty media dir** (scope="all" was saved by the merge walk — data safe,
> accounting off). All six legs now loop resolved names. **Multi-dir (N>1) refusal** is defensive (no
> catalog app hits it today: immich/nextcloud/romm match, paperless mismatches, each app = exactly ONE
> dir): tier-2 backup/info/restore refuse loudly (Hungarian); **migrate supports N naturally**. This
> limitation is **deferred to Task 3 (tier-policy engine)**, which owns the destination layout. Storage
> page sums resolved dirs. Truth repairs: the v0.130.0 CHANGELOG/CONTEXT "tier-2 copies the namespace
> wholesale" claim is FALSE — corrected in the v0.131.0 CHANGELOG entry + `main.go` export-adapter
> comment; tier-2 copies the recovery unit + resolved `appdata/<name>` ONLY (NOT userdata — F-S1,
> unaddressed here). New seam `tier2Mirror`; `migSeams.resolveNames`. +9 tests, RP-1..RP-5 all
> confirmed. Controller-only, no agent/hub coupling. **NOT live-validated here:** scope="app" migration
> of a real app between drives (F-S3 live proof — supervised leg, Viktor's session).

> **2026-07-14 — v0.130.0: CRITICAL C6B-F1 (hollow .fab export) + C6B-F2 (share-removal guard).**
> CAMPAIGN-6B proved `.fab` export shipped **config-only, data-free bundles** for 12/13 `needs_hdd`
> catalog apps (sonarr 4.17 GB → 2308 B, success, past the v0.125.0 guard). Three compounding fixes
> (all red-proofed run→fail→revert): (1) `stacks.ExportDataMounts` — export mount discovery unions
> `${HDD_PATH}` binds + the `${USERDATA_PATH}` **ROOT** (single `userdata` entry; root-not-per-bind
> is LOAD-BEARING: the manifest keys tars by basename and the untouched import maps basename →
> `<HDD_PATH>/<subdir>` — per-bind subpaths would restore to wrong places; this deviates from the
> task's literal per-bind+namespaced-names instruction, which could not round-trip without import
> changes the task forbade); (2) export + estimate are ADDITIVE for `needs_hdd` apps (HDD data AND
> named volumes — sonarr_config was silently dropped); (3) anti-hollow guard: `needs_hdd` manifest
> with zero data fails loudly. Plus §8: basename collision between mounts = loud Hungarian failure
> (was silent overwrite). **C6B-F2:** `netstorage/remove` refuses 409 while a DEPLOYED app's
> HDD_PATH is on the share (the orphaned-autofs trigger); resolves via the netAgent seam.
> **Residual flagged for a felhom-agent task:** RemoveNetworkMount's tolerate-and-continue stop
> (felhom-agent netmount.go:434-443) still deletes unit files under a busy mount if some non-product
> path calls it. Scheduled/tier-2 backup path was NOT affected and is untouched (`stackAdapter`
> deliberately unchanged). CAMPAIGN-6C's first acceptance test = the full-circle byte-compare this
> unblocks.

> **2026-07-13 night — v0.128.1 + demo storage hygiene (ruling F5).** `classTag` suppresses the
> rotational class hint for `type==='usb'` (card already carries the USB tag; hub `ClassHint`
> UNCHANGED; pinned by `TestStorageTemplate_USBClassBadgeSuppressed` + red-proof). Host op on
> demo-felhom: the two pre-intermediary legacy `dir:` storages (`felhom-usb`, `felhom-flash`,
> content=Backup, is_mountpoint) RETIRED via `pvesm remove` after G1/G2/G3 gates all PASSED
> (agent-owned UUID .mount units; both `enrolled` in drive-intents.json; zero /etc/pve refs, empty
> dump/, no customer app on either drive). Post-removal: mounts+binds intact (marker round-trip
> through the guest), `GET /disks` shows both registry-sourced (role+durable-id intact, class
> absent), `pvesm status` clean. **The demo node now matches the fresh-install storage shape** —
> drives are registry+units-sourced only, no legacy PVE dir: storages. v0.128.1 LIVE on demo 9201
> (drill guest skipped — optional, no behavioral dependency; it runs 0.128.0).

> **2026-07-13 night — v0.128.0: CHUNKED BROWSER .FAB UPLOAD on /import (ruling F3: chunked).**
> Step-0 probe on the real tunnel PROVED the Cloudflare edge cap (120 MiB POST → edge 413 with
> `Server: cloudflare` on Content-Length alone; 80 MiB → origin 302 /login; local DNS overrides
> the hostname to the LAN guest, probe needed `--resolve` onto CF's public IP). Design: JS
> `File.slice` 64 MiB strictly-sequential chunks → `POST /api/export/upload/{init,chunk,finalize,
> abort}` inside `ServeExportAPI` (inherits RequireAuth+CsrfProtect; single-flight; offset must
> equal received else 409+echo; 96 MiB request cap; free-space gate size+1 GiB; finalize =
> exact-size + fsync + atomic rename, collision → lowest-free `"name (N).fab"`). Lands in the
> DEFAULT drive's exports dir — scan/validate/import pipeline untouched. No client hash
> (deliberate: .fab self-validates). In-memory state: startup GC of `*.part-*`, 15-min idle
> abort. §7 A–F tested + 3 red-proofs. `appexport.DiskFree` exported (REUSE.md row).
> **NOT live-validated: the end-to-end multi-GB browser upload through the real tunnel needs a
> dashboard login → Viktor's 5-minute leg (export an app to .fab, download, re-upload, import —
> full circle).** Possible follow-up if it itches: per-drive target picker (v1 = default drive only).

> **2026-07-13 eve — v0.127.0: CUSTOMER-FACING ESCROW CEREMONY WIZARD (/backup/escrow) +
> Scenario-F stale-blob re-check. MinAgent 0.88.0 (wizard only). LIVE on demo 9201 + drill guest
> (both healthy).** The friend-alpha missing piece: preflight → warnings → password re-auth
> (login rate limiter) → **re-stage-first** (abort on failure — the UI can never mint a hash-less
> blob) → agent job (poll 2 s) → ONE-SHOT R reveal (no-store; R only in the claim XHR + page JS;
> 10-min TTL → void) → typed-back (two random words) → finish. Ruling F1: R over the CF tunnel
> once = accepted (threat model in felhom.eu RUNBOOK-escrow-ceremony.md). Scenario F: an ESCROWED
> box re-checks the ACK hash — mismatch/hash-less ⇒ stale flag (card warning + CTA) + one WARN
> per hash; never flips, never blocks; **fired LIVE on both boxes' hash-less blobs at first ACK**
> (drill = the spike's superseded blob, since REPAIRED via a real ceremony —
> `restic_pw_sha256` now covers; demo = its legacy blob, warning stays until a wizard run).
> Manual-confirm BUTTON removed (endpoint stays, deprecated). **OPEN: one supervised full-browser
> wizard pass with Viktor's login (re-auth needs the customer-owned password — CC validated
> everything beneath it endpoint-exact); demo wizard run to clear its stale warning.**

> **2026-07-13 — v0.126.0: UI UNIFORMITY BUNDLE (shared app-list rows + infra identity +
> restore-form polish + mojibake gate + honest stale line). Presentation-layer only — NO
> backup/toggle/engine behavior change. MinAgent 0.81 + floor unchanged.**
> (A) `templates/app_row.html` `app_list_row`/`app_list_row_end` is THE canonical list row
> (icon+name left, caller action right, compact 44px) — dashboard installed-apps, Távoli mentés
> toggles, Visszaállítás restore-to-verify + .fab lists render through it; the backups-apps
> expander header is ALIGNED (own markup, allowlisted); gate `scripts/app_row_dedup_gate.py`
> (red-proven). funcmap: `dict`/`appHref`/`infraMeta`. (B) `inframeta.go`: cloudflared →
> „Cloudflare Tunnel", traefik → „Traefik", filebrowser → „FileBrowser" + Hungarian descriptions
> + generic `/static/infra-logo.svg` fallback; filebrowser = the ONLY Linked infra
> (files.<domain>); render test counts exactly one customer link (red-proven). (C) .fab password
> field standard („Opcionális jelszó" + helper; import-page input got `.form-input`).
> (D) `scripts/mojibake_gate.py` — templates+Go strict UTF-8, zero Ã/Â/Ă-signature chars,
> allowlist ZERO (red-proven); source had NO mojibake — the live „Tárhely" is the felhom-usb
> drive-label DATA, repaired via the label-edit UI (live step). (E) `offboxWarningDisplay`
> display pick — stale „nincs mentésre jelölt alkalmazás" run-warning → „A kijelölés módosult…"
> note once ≥1 app toggled (neutral color); 0 toggled unchanged (red-proven).
> Housekeeping: one-shot `backups_split_move_check.py` RETIRED (served its purpose).
> **Live QA fixes:** v0.126.1 — `.form-input`/`.form-row` had NO CSS rule at all (root cause of
> the unstyled .fab password field; styled as the `.form-control` twin). v0.126.2 — CF edge
> caches /static/style.css 4h → stylesheet link now `?v={{.Version}}` (auto-bust per release).
> **0.126.2 LIVE drill+demo (both healthy).** §13 visual QA ran on the DRILL box via a
> reversible SSH gate-lift (hash restored byte-identical, gate verified back ON) — the demo is
> customer-claimed and CC does not enter credentials. OPEN human step: demo login → felhom-usb
> label repair via the label-edit UI (data = 'Tárhely (felhom-usb)' in storage_paths, documented
> read-only; the Part D gate closed the code side).

> **2026-07-13 — v0.125.0: .FAB VOLUME PATH-STRAND DATA LOSS FIXED (IA finding 1, HIGH).
> MinAgent 0.81 unchanged; floor may advance to 0.125.0 next train (must NOT halt above 0.124.0
> without this).** Both volume legs stream via docker cp (helper container + `dockerExec` seam —
> zero shared paths, correct bare-metal AND containerized; §3 live probe first). Export FAILS
> LOUD on any missing/empty claimed tar (`assertBundleDataComplete`); import VALIDATES BEFORE it
> destroys (`validateBundleData` in step 0 — hollow bundle → refusal, app untouched). Class
> extinguished by `scripts/docker_run_volume_path_gate.py` (every `"-v"` allowlisted with WHY;
> Tier-1/2 mounts documented host-visible). Live: the exact failed ActualBudget leg round-trips
> byte-identically (`ec8ea6cb…` before==after); engine-invalid volume → loud export failure.
> **ASYMMETRY (needs a customer-docs line):** .fab bundles exported by containerized ≤0.124.0
> controllers are hollow — re-export; the import guard refuses them loudly.

> **2026-07-13 — v0.124.0: BACKUPS IA RESTRUCTURE. MinAgent 0.81.0 + floor unchanged.
> Operator decisions (2026-07-13, treat as settled):** (1) single active offsite destination per
> box STANDS — the dual-destination `managed_by` model is the separate queued Task B;
> (2) the Felhom-offsite status card NEVER changes anything — display + opt-in pointers only;
> (3) .fab export/download is PORTABILITY, not a backup tier — no scheduling, no status surface,
> point-in-time framing. Mechanics: four sub-pages (`/backups{,/remote,/apps,/restore}`, sections
> moved VERBATIM — `scripts/backups_split_move_check.py` gates vs df7ad37), status card (3 states,
> display-only), .fab download exit (existing exporter + staging dir + guarded stream + 1h TTL;
> traversal guard red-proven). Live-validated on drill+demo incl. a supervised import round-trip.
> **NEW FINDINGS:** **(HIGH)** containerized .fab export strands Docker-VOLUME tars on the guest
> host (`docker run -v <container-tmp>` → host path) — bundle ships empty volumes, import brings
> the app up EMPTY; fix = host-visible staging + fail-loud post-export assertion. **(MEDIUM,
> agent)** legacy-boot PVE (LVM root, no ESP mount) → SystemDisks empty → sysKnown=false → drive
> wizard offers ZERO candidates ever. Follow-ups: .fab browser-upload; mega-zip parked.

> **2026-07-13 — v0.123.0: POLISH BATCH (take-two F-15/F-11 + rename + zero-toggle). MinAgent
> 0.81.0 unchanged; floor 0.122 unchanged. Requires hub v0.52.0 for F-15 (old hub = clean no-op).**
> (1) F-15: the reset-request RESPONSE carries the rotated code hash, applied via the ACK's
> generation-guarded ClaimSync — emailed codes work immediately (live: 1 s, first-try accept).
> (2) F-11: zero native `confirm()` — `felhomConfirm`/`data-confirm` inline Igen/Mégse (layout.html);
> gate `scripts/native_confirm_gate.py`. (3) Tier-3 customer branding is **"Távoli mentés"**
> (NAS-mentés gone; manual form generalized to any SFTP target; gate
> `scripts/offbox_rename_gate.py`; "Hálózati tárhely" feature untouched). (4) Zero-toggle honesty:
> hint + "Sikeres — nincs mentésre jelölt alkalmazás" run copy. Also: `atomicPromoteTar` O_RDWR
> fsync (Windows dev-box green gate was permanently red). Deployed drill qm300 + demo 9201.
> NOT started (separate queued task, operator fork pending): offbox `managed_by` coexistence.

> **2026-07-12 — v0.122.0: CUSTOMER-CLAIM PASSWORD GATE (closes DRILL-day0-vm F-4/F-5). MinAgent
> 0.81.0 unchanged. Requires hub v0.50.0.** The dashboard password is CUSTOMER-OWNED via a one-time
> claim code the hub emails to the registered address — the "no password → open dashboard" default
> is GONE. Unclaimed box (code hash delivered, no password) → serves ONLY `/claim`; every other route
> → claim page (302) or 401 (API). A set password disables the gate (auth wins). Reset rides the same
> code engine (login "Elfelejtett jelszó"). Legacy-open (no password + no hash) → red transition
> banner until the hub delivers a hash. `internal/web/claim.go` (gate + pages + HMAC pre-auth CSRF +
> 5-try/15-min lockout → `claim_lockout` event), `report/claim_sync.go` (ACK cache, idempotent by
> generation), settings `Claimed`/`ClaimCode*`/`ClaimConsumedGeneration`, `config.web.claim_code_*`
> (hub-baked), `--print-reset-code` root hatch. Gate-coverage signature test + 4 red-proofs.
> **LIVE-PROVEN on drill guest 9201 (0.122.0): gate ON via the real edge (/ → 302 claim page, /api →
> 401), code emailed to demo-vm-felhom's registered address.** Floor raise 0.120→0.122 = operator's
> LAST step (supervised). Details: felhom.eu/documentation/audits/DRILL-day0-vm-2026-07-12.md (F-4/F-5
> RESOLVED).

> **2026-07-12 — v0.121.0: BACKUPS PAGE TRUTH PASS. MinAgent 0.81.0 unchanged. Controller-only, no
> agent-API change.** Pure UI/data-plumbing on `/backups`; no backup-engine behavior change. Fixes the
> self-contradicting live page: (1) **removed the dead "Részletek" card** (operator decision — redundant;
> per-app rows + Adatbázisok section already carry the truth) — kills the last uses of the never-set
> template fields `Tier2DriveGroups`/`ResticPassword`, the `restic-pw` element, and the `toggleTier`/
> `toggleResticPw`/`copyResticPw` JS. (2) **per-app "3. mentés" row now shows real off-box state** via a
> new pure `tier3State` (configured→toggle→escrow precedence): unconfigured/off/escrow_pending/active —
> the "Hamarosan — B2/S3/SFTP" placeholder is gone. (3) **SQLite-honest DB messaging** via pure
> `dbSectionState(discovered,dumps)` → dumps/pending/embedded (embedded-only box shows "–" + "beágyazott
> DB-k a kötetmentésben", not a bare "0"). (4) **dead/raw fields fixed** — `Tier1LastRun`/`Tier1LastStatus`
> now populated from `ListRestorePoints`; Tier-1/Tier-2 labels via `timeAgoStr` (relative), confirm()
> dialog keeps raw. (5) **terminology split** — off-box section = "Távoli mentés (3. mentés)" (+
> `#offbox-section` anchor); whole-guest PBS card = "Távoli rendszermentés" (was both "Távoli mentés").
> (6) deploy page gains a "Mentési beállítások →" link. DECISIONS: Részletek removed as redundant
> (operator-approved); "Távoli mentés (3. mentés)" (app off-box) vs "Távoli rendszermentés" (PBS whole-CT)
> are two distinct customer-facing names. Pure helpers in `internal/web/backup_page_state.go`. +9 web
> tests, 4 red-proofs. Observations: orphaned style.css classes from the Részletek removal left in place
> (details-tier*, repo-encryption*, restic-pw-field, drive-detail-*, tier-empty-state, repo-info-row*,
> repo-tier-title) — noted, not cleaned. Backlog: felhom.eu backup-architecture.md offbox refresh (separate task).

> **2026-07-12 — v0.120.0: fix-3 + fix-6 → CAMPAIGN-3 CLOSED (LIVE on 9201 + hub 0.48.0).
> MinAgent 0.81.0 unchanged.** **fix-3:** a `deadapp-check` job (30s, 90s boot grace) flags a DEPLOYED
> app in stopped/exited state (`stacks.IsDownState`) → self-clearing WARN dashboard banner + one
> `app_start_failed` hub event per running→down transition (`Notifier.NotifyAppStartFailures`, in-memory
> tracker, hub owns cooldown). **fix-6:** ring cap 1000→5000 (display cap raised too); periodic
> scheduler/refresh success lines → `[TRACE]` (ring drops at write-time, failures never TRACE); atomic
> JSON-lines spill to `<DataDir>/debug-ring.log` (SSD, survives recreate) every 30s + shutdown, loaded
> on boot. **hub v0.48.0** accepts `app_start_failed` (allowlist + customerMessages). LIVE: docker stop
> seerr → banner + ONE hub event across 3 cycles (anti-spam) → docker start → banner self-cleared; ring
> 0 spam lines + restart PRESERVED the pre-restart window (oldest unchanged, 63KB spill on SSD volume).
> **CAMPAIGN-3 CLOSED** (F12/F11/F10/F9/F2/F1→agent 0.85; F7/F6/F5→0.118; F8/F4→0.119; fix-3/6→0.120).
> Follow-ups: agent-ring persistence; F13 (active-nfs-mp8 rc255); publish train (agent 0.85 + ctrl
> 0.118/0.119/0.120 + hub 0.48) to Peti. Seams: deadapp scanDeployedAppRunStates, notify.pushFn.

> **2026-07-12 — v0.119.0: STORAGE-HEALTH COHERENCE (LIVE on 9201). MinAgent 0.81.0 unchanged.** Fixes
> CAMPAIGN-3 F8+F4. **F8 (MED):** the share row's health came only from the agent's server-level TCP
> dial (blind to a single unexported share) → it showed benign "Készenlét" while the stacks cards
> showed the stub — a contradictory UI. `networkStorageItems`→`fuseNetHealth` now reuses the SAME
> `system.ClassifyPathFS` the stacks stub badge reads (§3 fork = option B, controller-only): a new
> `stub` health (badge "Hibás — az alkalmazások nem a NAS-t látják") overrides idle/ok when the
> namespace sees local disk; `unreachable` still wins over stub; autofs/network/unknown leave agent
> health intact (never force-mount). Row + stacks badge now share ONE classifier → can't contradict.
> **F4 (LOW):** `handleNetStorageAdd` range-checks container uid/gid 1..65533 (`validMappedID`) →
> friendly 400, nothing installed (was raw agent_error on 101000). LIVE: F8 row=stub matching stacks
> badge through an exportfs cut, cleared to ok on re-export; F4 uid 101000→400, uid 1000 passes.
> Seam: `s.classifyFSPath`. Task D (fix-3 alerting + ring revision) still queued.

> **2026-07-12 — v0.118.0: BACKUP INTEGRITY (LIVE on 9201). MinAgent 0.81.0 unchanged.** Fixes
> CAMPAIGN-3 backup findings (`documentation/audits/CAMPAIGN-3-2026-07-11.md`). **F7 (HIGH) atomic
> volume dumps:** `DumpAppVolumes` writes `<vol>.tar.tmp` → fsync → `os.Rename` over the `.tar` only on
> success (`atomicPromoteTar`), mirroring dbdump.go DumpOne; a mid-write NFS cut can no longer
> truncate the last good tar to 0 bytes. **F6 (LOW) no single-copy:** `RunAllTier2` no longer skips
> volume-only apps (they now get a cross-drive tier-2 copy); sys_drive restore-point label is clear
> ("Belső SSD (rendszer)"); single-drive box shows an honest `SingleCopyWarning` banner. **F5 (LOW)
> stale-primary sweep:** `pruneStalePrimaryDirs` removes an orphaned `backups/primary/<app>` dir on an
> OLD drive after an HDD_PATH move (guarded: deployed + different-current-drive only, never a restore
> point). **Part 4 locality fork → operator chose (A) keep locality, doc-only** (NAS tier-1 stays on
> the NAS; tier-2 is the off-NAS leg). LIVE: F7 money-shot (all NAS tars byte-identical through a
> mid-write cut, no 0-byte, success:false); F6 (actualbudget/seerr on felhom-usb/secondary); F5
> (seeded stale dir swept, current kept); restore round-trip byte-identical. Seams: `tarVolume`,
> `perAppTier2`. Task C (F8/F4) + Task D (ring/alerting) still queued; Peti reaches 0.118 + agent 0.85
> at his next train (agentless-on-proxmox2 gap noted).

> **2026-07-11 — v0.116.0/0.116.1: OBSERVABILITY PASS (LIVE on 9201; agent v0.83.0 + hub v0.46.0).
> MinAgent: 0.81.0 unchanged.** The debug ring (`LogBuffer`) now ALWAYS exists — logger =
> `MultiWriter(LevelFilterWriter(stdout, logging.level), ring)`, so DEBUG detail is remotely
> readable on an `info` box while docker logs keep the configured level. New `internal/logx`
> leveled helpers = the sweep standard (netstorage_job phases/verdicts/durations, netprobe,
> validation refusals, orphan WARN, `SupportsWithSource` gate line, agentapi per-call DEBUG,
> migrate phases, tier2/offbox unswallowed persists). Report ACK gains `controller_log_requested` →
> next report ships `controller_log_tail` (selftail.go, consume-once, 128 KB; the customer-visible
> `operator log pull served` INFO rides in the tail; app-tail wire byte-compatible). Debug page:
> `Vezérlő | Ügynök` tabs — the agent tab proxies agent `GET /debug/logs` (`Client.DebugLogs`;
> typed-404 → the "after the agent's next update" notice). **v0.116.1 (found by live validation):
> `/debug` + `/api/debug/*` + the nav item were STILL gated on logging.level=debug — ungated (auth
> unchanged), the incident's actual blind spot.** Live-proven at info: a real refused NAS add is
> fully reconstructable on both tabs (capability gate w/ source=version, phase lines, 502+duration,
> category). Conventions: `felhom.eu/documentation/runbooks/logging-conventions.md`. OPEN: operator
> clicks the hub's two "Request logs" buttons (hub UI password-gated) to close the live bundle
> round-trip; legacy `isDebug()` emission sites left for incremental migration. NOT published —
> Peti stays 0.113/0.81.

> **2026-07-11 — v0.115.0: version-aware Supports + DSM-validated NAS guidance (LIVE on 9201, pairs
> with agent v0.82.0 + hub v0.45.0). MinAgent: 0.81.0.** Capability detection now compares the agent
> version from agent v0.82.0's `X-Felhom-Agent-Version` header (`Client.noteAgentVersion` captures it
> on every response, strict semver; `features.go featureMinAgent` table + version-first `Supports`)
> instead of route-probing — the probe stays as the fallback for header-less (≤0.81) agents, so
> nothing changed for Peti's box. THE one comparator moved to `internal/util/version.go` (selfupdate
> aliases it). Part A DSM spike (real DSM 7.2 via virtual-dsm) validated the consumer recipes E2E; the
> NAS-page NFS guidance gained the verified Synology steps (File Services → NFS → **NFSv4.1**; "Map
> all users to admin"; `/volume1/<share>`); caveat narrowed to QNAP-only. Live-checked on demo: a real
> add shows `capability gate: netstorage_verify=yes` via the version compare, zero probes. **Q1c
> (Part E, supervised) FAILED**: a NAS automount trigger does NOT survive a guest reboot (guest sees
> an empty dir; agent has no network-mount reassert) — fix is felhom-agent's, spec'd at
> felhom.eu/documentation/backlog/FOLLOWUP-nas-automount-guest-reboot-reassert.md (controller
> health-cross-check follow-on noted there). NOT published (agent 0.82 demo-only; Peti 0.81).

> **2026-07-11 — v0.114.0: agent-capability gate (option-1) + publish-train rules (option-2).**
> Answer to the 0.81/0.113 train's 9-minute controller-before-agent skew (Peti's box): the box now
> protects itself. `internal/agentapi/features.go` — `Supports(Feature)` route-probes the agent
> (`GET /netstorage/verify-status` = the v0.81.0 coupling signal; typed `StatusError` 404 ⇒ No, 2xx
> ⇒ Yes, transport/5xx ⇒ Unknown NEVER refused; `SupportCache` TTL 5m both polarities, Unknown
> uncached). `handleNetStorageAdd` refuses on No BEFORE the single-flight claim (412 +
> `agent_outdated` + honest Hungarian message); settings page swaps the add form for a banner
> (list/remove untouched in every state). remove/list NOT gated. NO agent/hub changes; NOT
> published, floor untouched, Peti stays 0.113.0 — the gate is inert protection until the next
> train. Rules codified: `felhom.eu/documentation/runbooks/publish-train-rules.md` (manifest before
> floor; floor field LAST — hub_settings DB row overrides env + acts immediately; MinAgent fleet
> gate — CHANGELOG header convention starts with this release; the gate as box-level backstop).
> Tests T1–T6 + wire-level 404-typing; red-proofs RP1–RP5 in REPORT.md. Roadmap: agent
> version-in-envelope upgrade of `Supports`; hub floor-UI separation = its own task. The
> `agent_outdated` branch is test-proven only (demo agent is current — downgrade not justified).

> **2026-07-11 — v0.113.0: NAS verify-before-commit + page redesign (LIVE on 9201, pairs with agent
> v0.81.0 + host-install v1.13.0).** `POST /api/storage/netstorage/add` no longer registers blind
> (the bogus-share-at-Készenlét bug is dead): detached single-flight orchestration
> (`internal/web/netstorage_job.go`, migrate shape; poll `GET .../add/status`) = agent add (units +
> agent-side detached verify with journal classification + auto-rollback) → controller **uid-1000
> re-exec write probe** (`--netprobe`, SysProcAttr.Credential — catches the squash trap) → register
> LAST. Any failure = full rollback; verify-lost after agent restart ⇒ controller rollback; unregistered
> agent shares surface as remove-only "Árva megosztás" rows. §3.2 Hungarian error map server-side
> (`netAddMessage`; `nfs_export` MERGES not-found/not-permitted — NFSv4 identical strings).
> storage_network.html rebuilt on the storage_attach pattern (form-row/form-input killed), SMB listed
> first, NFS two-recipe guidance with live computed uid+100000. Agent v0.81.0: NFS `retry=0`
> (dead-NAS access 91 s→3.8 s), `ClassifyNetVerifyFailure` (Q4-verbatim), unprivileged journal read
> (systemd-journal group — host-install v1.13.0 adds it; NO new sudoers). Live-validated A–E on 9201
> vs an isolated sim NAS (all transcripts + red-proofs in REPORT.md); Route A proven in production
> (alien-uid squash → server-side 1060:1060). Authoritative doc:
> felhom.eu/documentation/controller/network-storage-nas.md. NOT published (0.81.0 not in Gitea /
> Day-0 manifest; Peti untouched — his rollout incl. the usermod one-liner comes with the floor bump).
> Gotcha for future sessions: the controller container is bridge-only — in-guest API tests need the
> CONTAINER IP + `Host: felhom.demo-felhom.eu` (127.0.0.1:8080 is stale advice).

> **2026-07-10 — v0.112.0: self-update without credentials (LIVE on 9201, pairs with hub v0.43.1).**
> Root cause on Peti's box: the updater refused without Git Sync creds, but the public package is
> anonymously pullable. `queryRegistry` now does the Docker v2 anonymous token dance when both creds are
> empty (realm/service parsed FROM the WWW-Authenticate header — never hardcoded); `pullImage` skips
> `docker login` credential-less; creds path byte-unchanged (private catalogs); half-configured pair =
> loud misconfig; denial = "registry denied anonymous access — a private registry requires Git Sync
> credentials". Settings panel gains the mode line "Registry: nyilvános (hitelesítés nélkül) /
> hitelesített" — credential-less is a supported mode, not an error. Red-proof green (old guard restored
> → anonymous tests fail with the old message). LIVE-PROVEN on the credential-less demo (git creds are
> quoted-empty): /api/selfupdate/check → ok, latest=0.112.0, no error; settings shows "nyilvános".
> PENDING OPERATOR: floor-bump Peti to 0.112.0, then delete his temp Git Sync creds → clean "nyilvános"
> check. runCommand/runCommandStdin are now package VARS (test seam).

> **2026-07-10 — v0.111.0: remote app-log diagnostics (LIVE on 9201, pairs with hub v0.43.0).** The
> telemetry scraper now attaches `LogIssue.Context` (±5 raw lines around the FIRST occurrence of each
> error-severity issue; ≤11 lines, ≤400 chars/line, 16KB/report budget dropping lowest-count first; warns
> carry none) and `metrics.RedactLine` sanitizes EVERY off-box context/tail line (password/token/api-key/
> authorization/bearer → `[REDACTED]`, 64-hex → `[REDACTED-HEX64]`). On-demand log tails ride the ACK pull
> pattern: hub ACK `log_tail_requests` → next report `log_tails` (200 lines via stacks.GetLogs /
> FetchContainerLogTail, ordered, ≤64KB/app newest-kept, redacted, consume-once drain). Hub v0.43.0 stores
> context (first-capture-wins + `context_customer` provenance), renders click-to-expand copyable issues,
> fixes the period filter on Known Issues, replaces issue deletion with DISMISSAL (`dismissed_at`,
> resurface only on `last_seen > dismissed_at`), adds `?customer=` filtered drill-down, and keeps the last
> 2 tails per app with an ordered viewer + .log download. All red-proofs green (capture, redaction,
> consume-once ×2, dismissal guard, range filter, context clobber). Live-proven on demo: synthetic error →
> hub row with ordered 11-line context and `password=[REDACTED]`. OPERATOR: one click ("Request log tail"
> on demo felhom-controller) completes the live tail round-trip — hub UI is password-gated, CC cannot.

> **2026-07-09 — v0.106.0: offsite provisioning SLICE 2 — the apply-bridge (pairs with hub v0.38.0).** On
> startup the controller reconciles the hub-served `offsite:` descriptor into a key-only offbox target:
> `internal/offsiteapply.Bridge.Reconcile` — verify-pin the box host key against `host_fingerprint` (NO blind
> TOFU) → generate keypair → consume the one-time password (`POST /api/v1/offsite/consume-password/{id}`,
> single-use, never logged) → `sshpass ssh-copy-id -s -f` install + verify → `Manager.ApplyOffsiteTarget`
> (fork-4 enable → `EscrowState="pending"`) → persist a descriptor-hash marker LAST. **Idempotent** (no
> re-consume of a spent password) + **fail-safe** (any step fails → nothing persisted, retry next restart;
> consumed-but-failed install = loud "reset on the hub"). Seams faked in tests; both red-proofs (no-TOFU,
> marker-after-success) green. `Dockerfile` + `sshpass`. **NOT yet live-applied** — supervised end-to-end
> (hub provision → controller apply) is the next runbook, gated on the hub's new scoped `HETZNER_TOKEN`.
> NEXT slices: SLICE 3 (escrow auto-confirm), SLICE 4 (soft-quota).

> **2026-07-09 — v0.105.0: fork-4 offsite password custody (pairs with agent v0.77.0).** The restic-offsite
> repo password now rides the **customer-R escrow** (age-under-R in the agent `IdentityBundle`; custody spike
> `febdc56`). Enable → controller pushes the password (`StageEscrowSecret` → agent `POST /escrow/stage-secret`)
> → `EscrowState="pending"`. **Atomicity gate:** no offsite RUN until `EscrowState="escrowed"` (operator
> `POST /backup/offbox/confirm-escrow` after the escrow ceremony) — so no un-recoverable offsite ciphertext
> exists. **DR:** `POST /backup/offbox/inject-password` pre-places the recovered password (honored by
> `WriteOffboxSecrets`). DR recipe gains non-secret `offsite_restic` coords (`DRResticCoord`); the SFTP key is
> regenerated at DR (not escrowed). Atomicity + inject companion red-proofs green. **NOT yet live-validated**
> — the supervised escrow ceremony (enable→stage→escrow-create→confirm→gated run) is operator-run; NEXT =
> hub-verified auto-confirm + customer-self-serve enable (provisioning task). Deployed to 9201; see REPORT.

> **2026-07-09 — v0.104.0: off-box discovery over inference + no-silent-success.** The Storage-Box spike
> found offbox reporting `ok`/0 snapshots while backing up nothing; DIAG pinned it: offbox resolved each
> toggled app's recovery unit via `AppNamespaceRoot`→`GetAppDrivePath`, which reads the app's *live*
> `app.yaml` `HDD_PATH` and **silently falls back to `systemDataPath`** when the app isn't deployed → it
> looked on the wrong drive. **Decision: DISCOVER, don't infer** — scan the durable storage registry
> (schedulable, non-decommissioned paths ∪ systemDataPath) for `backups/primary/<app>`, deployment-state
> independent; newest-by-manifest wins on drive churn. **Silent-success closed:** 0-of-N toggled → hard
> error + operator alert; partial → `ok` + customer `LastWarning`. Write paths + `AppNamespaceRoot`
> untouched. Unit suite + both companion red-proofs green. **NEXT:** supervised box re-provision + a real
> offbox→Storage-Box endpoint round-trip (this task did NOT re-point at the live box — spike creds were
> torn down). Deployed to 9201; see REPORT.md.

> **2026-07-07 — v0.103.0: F-C2-1 (LIVE on 9201).** The config loader ran `os.ExpandEnv` over the
> whole YAML before parse, silently corrupting a bcrypt `web.password_hash` (`$2a$10$…` → `"a0"`) — a
> silent auth-integrity bug. Removed both `ExpandEnv` calls (parse raw bytes); typed
> `FELHOM_WEB_PASSWORD_HASH` override unchanged. Live-proven: a bcrypt hash in controller.yaml now
> loads intact and login succeeds (pre-fix it corrupted → login fail). Behavior change: literal
> `${VAR}` in a value is now preserved verbatim (no repo config depends on the old expansion).

Last updated: 2026-07-06 (v0.102.0 — async restore family; F4 re-adjudicated + fixed)

> **2026-07-06 — v0.102.0: async restore family (F4 UX fix, LIVE on 9201).** All three restore surfaces
> (`/backup/restore`, `/backup/tier2/restore`, `/backup/offbox/restore`) blocked the HTTP request until
> completion → through cloudflared's 100s cap a customer got an error page while the restore succeeded
> (offbox worse: bounded on `r.Context()`, canceling the SFTP restore mid-flight). Now async (offboxRun
> shape): fast-path IsRunning refuse → background goroutine (offbox ctx off r.Context()→Background+30m) →
> instant redirect. New `GET /api/backup/restore-status` + mutex op-status (`opstatus.go`) + 3s-polling
> `backups.html` banner. Live-proven: restore POST 0.018s internal / **0.235s external (F4 tunnel)**, canary
> bit-identical, status transitions. Restore single-flight unchanged. OPEN: op-status is in-memory (no
> persistence, by design).

> **2026-07-06 — v0.101.0: no-mercy campaign findings.** F3: git subprocess deadline in
> `internal/sync/sync.go` (`gitCmdTimeout=120s`, `exec.CommandContext`) — a hung remote no longer
> wedges `syncing=true` until restart. F2 evidence gap: `agentapi.EjectDisk`/`Decommission` now use
> `postWithStatus` + `refusalError` so the agent's `"…refused (role: X)"` reaches the operator
> instead of a bare `HTTP 403`. Companion: catalog `d86e256` (F1 vaultwarden `_ENABLE_SMTP` boot-gate
> — fresh email-off deploys crash-looped; live-validated Scenarios A/B on 9201). F2 diagnosed to a
> verdict (REAL finding — `roleForMountPath` over-refuses an enrolled user-data drive that isn't a
> PVE storage; fail-safe direction; agent fix DEFERRED). Full triage:
> `felhom.eu/documentation/audits/CAMPAIGN-nomercy-2026-07-06.md` addendum. OPEN follow-ups: agent
> `roleForMountPath` fallback; the targeted P1–P3 campaign re-run for clean backup/restore coverage.

> **2026-07-05 — v0.100.0 (TASK C2): drill finding F2 CLOSED — one-click class-C file restore.**
> `POST /backup/tier2/restore` + "Fájlok visszaállítása" on the Tier-2 row: in-place, ADDITIVE-ONLY
> (`rsync -a --ignore-existing` from the recorded Tier-2 copy — never overwrites, never deletes).
> Serves "I deleted my files"; corruption/point-in-time stays offbox/operator. **The C-series
> (drill findings F1/F2/F3/O4) is now fully closed.** Reindex caveat (e.g. Nextcloud occ files:scan)
> documented in backup-architecture.md.

> **2026-07-05 — v0.99.0 restore-path fixes (TASK C1): drill findings F1/F3/O4 RESOLVED.**
> F1: `GET /api/backup/snapshots` implemented (`backup.ListRestorePoints`) — the restore panel
> populates and the restore button enables. F3: `runVolumeDumps` wired into the nightly/manual
> backup run (volume gate BEFORE DumpAppVolumesSafe; before unit capture). O4: unrecoverable
> resettable secrets get a generated replacement (`GenerateSecretForField` + `SetSecretGenerator`
> seam); data-key gate untouched. **F2 (one-click in-place class-C restore) remains OPEN → TASK C2.**
> O4 residual: restored volume tar with an OLD credential hash may still need a manual in-DB reset.

> **2026-07-04 — app-data restore drill** → see `felhom.eu/documentation/audits/DRILL-appdata-restore-2026-07-04.md`.
> Keep-side restore proven live on 9201 (class-A DB replay + fail-closed data-key gate + non-destruction). **F1 (HIGH): UI restore is dead — `/api/backup/snapshots` has no handler, so the restore button never enables.** F2: no one-click in-place class-C (HDD bind-mount) restore. F3: named-volume data never backed up (`DumpAppVolumes*` has no caller). *(F1/F3/O4 resolved in v0.99.0, see above.)*

> **2026-07-03 — CLAUDE.md slimmed to stable orientation** (full package map, verified 9201 deploy
> summary, no version-pinned state). Deep runbooks/design/testing doctrine now in the personal
> skills `felhom-build-deploy` / `felhom-ui-design` / `felhom-testing` (source: `felhom.eu/skills/`).

> **2026-07-03 — `REUSE.md` exists at the repo root** (canonical helpers / patterns / traps / seams, code-verified). Check it before writing new code; update it in the same commit that adds/changes a shared helper (maintenance rule now in CLAUDE.md).

> **2026-07-02 — v0.98.0 (deployed on 9201): Tárhely IA follow-up.** User feedback on D1: NAS-add and
> local-drive enrollment buttons sat side by side — confusing. `/storage` split into two subpages
> under the Tárhely nav item (nested sub-links, `.nav-links-nested`): **/storage = Meghajtók**
> (drives + agent view + wizards + manual add) and **/storage/network = Hálózati tárhely (NAS)**
> (NAS-megosztások list + add form; own openDialog copy). `networkStoragePageData` (page key
> `storage-network`) split out of `storagePageData`. No API/storage-semantics change.

> **2026-07-02 — v0.97.0 (deployed on 9201): TASK-D1 — settings split + Tárhely page (IA only).**
> The 1451-line `settings.html` monolith is split into four pages: `/storage` (new main-nav
> **Tárhely**: Adattárolók + NAS + unified agent drive view + wizard entry), `/settings` (Rendszer:
> konfig, verzió/frissítés, vezérlő+kiszolgáló újraindítás), `/settings/notifications` (Értesítések
> + Alkalmazás-email), `/settings/security` (Jelszó, Földrajzi korlátozás, Vészhelyzeti információk).
> Sidebar gains the Tárhely item + a "Beállítások" group with three sub-links. **No storage/agent/API
> behavior change** — pages moved, logic didn't; every `/api/storage/*` endpoint + POST action
> unchanged. **Decisions:** (a) storage flashes + wizards moved to `/storage`; old
> `/settings/storage/{init,attach}` **301** to `/storage/{init,attach}`. (b) The two duplicated drive
> views MERGED: registry cards render server-side, then JS enriches each connected user-data card in
> place from the agent `/api/disks` (role tag, durable-id, agent actions) joined on mount path, plus
> two groups — **Rendszermeghajtók** (system/backup, read-only, lock tag, NO actions) and **Nem
> regisztrált** (register only); agent-down → one warn note, registry cards still render. (c) Every
> native `confirm()`/`prompt()` on the four pages now routes through a light `.confirm-overlay`
> (`openDialog`; texts verbatim; type-to-confirm kept only where it existed) — the D0 tab-freeze is
> gone. **Contract for future UI:** new templates must pass `scripts/template_id_gate.py` (every JS
> element-ID resolves in its own template) and `scripts/emoji_gate.py` (0 emoji — the D0 grep gate
> false-negatived multibyte emoji on Windows; 8 survivors removed). `.badge-lock`/`.lock-ico` CSS
> deleted; other `.badge*` still used on secondary pages (D2 to finish). **Next: D2** (badge→tag on
> backups/secondary pages; storage.html is still ~700 lines — NAS-add partial candidate). NOT
> live-validated: agent-down degradation (don't stop the live agent — static/unit only); destructive
> storage ops via the moved overlay paths (endpoints unchanged; supervised session).

> **2026-07-02 — v0.96.0 (deployed on 9201): TASK-D0 — design system v2 re-skin (appearance only).**
> The whole customer UI (dashboard AND setup wizard) now renders in the approved v2 language: navy
> token palette (`--bg-0/1/2, --line, --text-1/2/3, --blue, --warn, --crit`), single 2px radius, no
> shadows; **exception-based status color** — nominal is blue/neutral, amber/red only on deviation;
> zone bars → the hairline `.meter` (neutral 70/85 ticks, warn/crit flag „Fogyóban a hely" /
> „Kritikusan kevés hely"); pills → `.tag` / `.metarow`. **Decisions:** (a) `stateColor` semantics
> changed — **stopped/exited = neutral, NOT red** (operator-approved; failures surface via unhealthy
> + alerts); restarting = warn; outputs are now `run/progress/warn/neutral/off` and
> `nominal/warn/crit` — any new template MUST use these suffixes (truth tables guarded by unit tests,
> `stateLabel` copy frozen byte-identical). (b) Fonts (PJS + JBM variable woff2, latin+latin-ext) and
> a 30-icon Lucide sprite are **vendored in the binary** (`/static/fonts/`, `templates/icons.html`)
> — no CDN; `fonts.googleapis.com`, `box-shadow`, 999px and the emoji set are banned strings (grep
> gate in REPORT §5, 143→0). (c) Setup wizard `handleCSS` bug fixed: it read a dataDir-derived path
> that never exists in the container → always served minimalCSS in production; now serves embedded
> `web.StyleCSS()`. (d) Found+fixed pre-existing v0.93.0 bug: `/backups` 500'd once an off-box backup
> had run (`OffboxTarget.LastRun` is an RFC3339 string fed to `timeAgo`; new `timeAgoStr`).
> Canonical reference: `felhom.eu/documentation/design/design-system.md`. **Next: D1** (settings IA
> split + Tárhely main-nav promotion; also migrate the remaining native `confirm()` dialogs to the
> type-to-confirm overlay and finish badge→tag markup on secondary pages). NOT live-validated: setup
> wizard visuals (unit-only), warn/crit meters on real hardware (demo healthy).

> **2026-07-01 — v0.95.0 (deployed on 9201): Impl-2b — raw-drive enrollment wizards.** Both enrollment
> wizards (`storage_init.html` / `storage_attach.html`) now fetch candidates from the agent's raw-device
> scan `GET /api/disks/candidates` (proxy `agentDiskCandidatesHandler` → agent Impl-2a) instead of the
> `Observe()`-based `/api/disks`, so a brand-new (non-PVE-storage) drive is discoverable + enrollable.
> `agentapi.ListCandidates` + `resolveEnrollUUID` (a raw candidate isn't in `/disks` → resolve its
> fs-UUID from the scan). Live end-to-end on felhom-pve: raw `/dev/sdd` (SD card) → wizard → format (Impl-1
> guard) / attach → mount → bind → intent → guest-bind → **"Aktív"** in the UI (3rd drive, no false
> detach). Needed a chain of AGENT fixes (v0.56–0.58: durableIDForMount / ReassertGuestBinds / HostReader
> wiring / /disks GuestPath+BoundUnderParent for registry rows) — the whole enroll+tracking machinery had
> assumed a PVE-storage drive. **Follow-ups:** `runStorageInit` should poll the agent's detached-format
> status (`/disks/format/status`) so a SLOW-device format completes in one flow (pre-existing); Impl-3
> shared-box operator format gate.

> **2026-06-30 — v0.94.0 (deployed on 9201): pull-based config-refresh.** The hub report ACK now also
> carries a per-customer **`config_version`** (hub v0.26.0; a stored counter bumped on every config save).
> `OnPushResponse` → `ConfigRefresher.Reconcile` (`internal/report/config_refresh.go`): on a change vs.
> `settings.applied_config_version`, `bootstrap.RefreshConfig` re-pulls `controller.yaml` (re-merging
> `local_api` from bootstrap.json; overwrites controller.yaml, never settings.json) → record → graceful
> self-restart (`api.GracefulSelfRestart`). First-run records baseline (no restart); unchanged = no-op
> (no storm); failed pull keeps config + retries. This is the box-pulls-config replacement for the hub's
> retired inbound "Push Config" — the hub never connects into the box. **Companion hub v0.26.0** also
> retired Trigger Update / Pull Config / Show Diff and dropped geo-disable's inbound notify (kept the
> hub→Cloudflare WAF removal), and fixed the stale `docker-setup.sh` setup command → host-install.
> Live-validated on 9201: first-run baseline (applied=1, no restart) → DB bump 1→2 → re-pull+self-restart
> (session_secret rotated, local_api preserved, RestartCount 0→1) → 4 cycles no-loop → apps stayed up.

> **2026-06-29 — Self-health arc complete (agent v0.48.0 + hub v0.22.1).** The hub now proactively
> watches every agent's served local-API **leaf fingerprint** fleet-wide: the agent reports
> `leaf_fingerprint` on its host report (v0.48.0), and the hub `HostLeafChecker` (`monitor/host_leaf.go`)
> raises `host_leaf_changed` when it changes (trust-on-first-report baseline; empty fp = unknown, never
> alerts; reads from report_json, no migration; hub-generated so no allowlist change). Independent of the
> controller channel-check. Live: a leaf regen raised `host_leaf_changed` (`82078fab…→60b5974d…`) AND the
> controller's `agent_channel_pin_mismatch` — complementary. NOTE: v0.22.0 shipped the checker but the
> main.go wiring never applied (concurrent file-touch); the LIVE TEST caught it (no event on regen) →
> fixed in v0.22.1. The original silent-multi-day-outage incident class is now caught from THREE angles
> (agent capabilities v0.44.0 / controller channel-check v0.90.0+F2 / hub leaf-fp v0.22.x) AND prevented
> (agent v0.46.0 loud-regenerate + host-install --preserve-state-from). Backlog: the served-fp-vs-pinned-fp
> authoritative cross-check; the test-run coverage gaps (capability→hub live, host-reboot doubling).

> **2026-06-29 — v0.91.0: F2 (channel-health born-down alerting).** The v0.90.0 channel-health check
> alerted only on a live up→down transition; a channel broken at **startup/reseed** (e.g. the
> controller boots right after a leaf regen → first observation is `pin_mismatch`) was dashboard-only,
> no operator email — forever. Fixed with an `alerted` flag in `channelhealth/checker.go`: a confirmed
> down coming from up/unseeded OR a reason-change re-arms then alerts once; steady-down no dup; recovery
> re-arms; debounce intact (transient born-down still N≥2; healthy first-obs silent). Live-validated:
> controller restarted into a regenerated-leaf channel → `[channel] DOWN (unseeded->down:pin_mismatch)`
> + `Event pushed: agent_channel_pin_mismatch` (~60s) — the capstone's missing half. Companions: hub
> v0.21.0 (same fix for HostCapabilityChecker/HostStalenessChecker — seed only healthy, born-degraded/
> stale emits on first Check, cooldown dedups), agent v0.46.0 (EnsureLeaf loud-WARNs a REGENERATED leaf
> + host-install `--preserve-state-from`/populated-host guard = the prevention for the original incident
> class). Self-health story now complete: agent watches its capabilities, controller watches its link,
> detection works at startup, and a reinstall can preserve the leaf. Backlog: F1 swap-rollback
> RestartCount hardening; full live reinstall (supervised); hub-side leaf-fp fleet comparison.

> **2026-06-29 — v0.90.0: controller→agent channel health-check (self-health slice).** A ~60s
> scheduler job (`internal/channelhealth`) probes the local-API channel via the PRODUCTION memoized
> client (`Server.ProbeAgentChannel` → `s.agentClient()` + GET /storage — NOT a fresh client, per the
> spike: self-heals, mirrors the disk UI, no transport leak). Classifies failures (spike Q1 map:
> pin_mismatch/unauthorized/unreachable/timeout/misconfigured/construction_error/unknown), **debounces**
> transient reasons (refused/timeout need N≥2 consecutive so the ~1s agent-restart blip doesn't page;
> pin/401/DNS/construction alert first-obs), seeds the first observation silently, and on a transition
> emits an **English operator-only** event (`Notifier.NotifyAgentChannelDown/Recovered`) + a Hungarian
> dashboard banner (`AlertManager.SetAgentChannelAlert`). Closes the R1-incident gap (channel was only
> checked once at startup). **Required a hub change** (felhom-hub v0.20.0): the `agent_channel_*` event
> types were rejected by the hub's `allowedEventTypes` allowlist (HTTP 400) — controller-pushed events
> are gated, unlike the hub-generated host_* ones. Live-validated: transient restart → no alert;
> sustained stop → debounce(1/2) then DOWN transition + dashboard banner; start → recovered. No agent
> change. Self-health story now: agent watches its own capabilities (v0.44.0) + controller watches its
> link to the agent (this). Backlog: hub-side leaf-fp comparison.

> **2026-06-29 — OPS (no code change): controller↔agent TLS leaf-pin mismatch RESOLVED via R1.**
> The 2026-06-28 root→non-root agent migration regenerated the agent's local-API leaf
> (`60b5974d…`→`ced34036…`) and truncated its token store, so the in-guest controller's pinned
> `agentapi` client failed every call (`TLS pin mismatch`) — disk UI dead AND the quiesce
> app-consistent-backup loop dead. Fix: restored the pre-migration `local-api.{crt,key}` **+**
> `local-tokens.log` from `/root/agent-backup-bundle-test/aside-var-lib/` on felhom-pve (R1 RUNBOOK,
> supervised). Agent now serves `60b5974d…` again (== controller pin + bootstrap.json seed) and the
> restored token store's 9201 entry matches the controller's current `local_api.token` — **zero
> in-guest change.** Gates passed: leaf loaded (not regenerated), wire fp `60b5974d…`, authenticated
> `GET /disks` 200/5-disks (bogus token 401), and the next quiesce cycle ran a real backup job.
> Rollback point kept at `/root/agent-rollback-20260629-150137/`. **OPEN (carried forward):** the
> migration/Day-0 path must carry these three files into the new `felhom-agent`-owned data dir (or
> treat their loss as a re-bootstrap trigger), else any re-migration reintroduces this. Full record:
> `felhom.eu/documentation/audits/SPIKE-agent-pin-mismatch-2026-06-29.md` (§9 Outcome).

> **2026-06-27 — v0.86.0 (deployed on 9201): Phase 2 managed updates — controller-version FLOOR.**
> On top of the Phase 1 opt-in "update to latest" button, the controller now honors an operator-enforced
> **minimum version (FLOOR)** delivered on the hub **report ACK** (`min_controller_version` +
> `latest_version`; `internal/report/pusher.go` `PushResponse`). `OnPushResponse` →
> `updater.SetFloor()` + `updater.MaybeAutoUpdate()` (rides the report cycle — no new timer). Below the
> floor → **auto-update to the floor** (reuses Phase 1 `performUpdate`: in-guest pull → agent swap →
> rollback; `initiatedBy="auto-floor"`). At/above floor → nothing (does NOT chase latest — that's the
> button). Guards: dev/no-agent/backup → skip; floor must be pullable (floor ≤ latest; floor>latest →
> warn+noop); one attempt per below-floor condition (in-mem + persisted state) → no flapping. Floor source
> + operator UI are hub-side (felhom-hub v0.15.0: per-customer override + global default + report ACK).
> **No agent change** (reuses Phase 1 `POST /controller/swap`). Day-0 now ships current (golden rebuilt at
> 0.85.1) AND stays current (floor) — the fleet-currency story is closed. Live: dogfood 0.85.1→0.86.0 via
> the button, then floor 0.87.0 → auto 0.86.0→0.87.0 (no click).


> **2026-06-26 — v0.84.0 (deployed on 9201): show an app's auto-generated first-login on its page.**
> General, catalog-driven mechanism: `.felhom.yml initial_credentials: {file, format json|regex|plain,
> username_key/password_key | username_pattern/password_pattern, note}`. The controller reads the file
> **live** from the container (`internal/stacks/initialcreds.go` `ReadInitialCredentials` + pure
> `parseInitialCreds`, unit-tested), never persists it, and renders a "Kezdeti belépési adatok" card on
> `/apps/{slug}` (masked password + reveal/copy). First consumer: crafty-controller (Crafty writes a
> random admin password to `/crafty/app/config/default-creds.txt`). Live-verified on 9201: card shows
> username `admin` + the real extracted password. Reuse for any future self-seeding app. Security: same
> exposure class as the existing post-deploy reveal / default_creds card — relies on the prod dashboard
> being auth-gated (demo public-unauth is the separate tracked issue). Backlog idea unchanged: a
> `backend_scheme` hint for TLS backends (v0.83.0 line).

> **2026-06-26 — v0.83.0 (deployed on 9201): scoped Traefik backend transport for self-signed HTTPS apps.**
> Crafty is the **first/only catalog app with an HTTPS backend** (self-signed TLS on `:8443`, no plain-HTTP
> port). The crafty healthcheck fix un-withheld its route, exposing a pre-existing 502 (Traefik proxied
> HTTP to the HTTPS backend). Since traefik v3 forbids `insecureSkipVerify` via Docker labels, the
> controller now renders a file-provider dynamic file `dynamic/serverstransports.yml` defining a **named**
> `insecure-skip-verify` transport (`infra.RenderServersTransports` + `stacks.ensureServersTransports`,
> written from `EnsureBaseStack` outside the traefik-running guard, write-if-changed, hot-loaded). Apps opt
> in per-service via catalog labels `scheme=https` + `serverstransport=insecure-skip-verify@file`.
> **No global `insecureSkipVerify`** — verification stays ON for every other backend (scoped Option B).
> Pattern to reuse for any future self-signed HTTPS backend. Live-verified: `minecraft.demo-felhom.eu`
> 502→302; filebrowser (HTTP) unaffected. Backlog: a `.felhom.yml backend_scheme` hint so the catalog
> convention sets these labels instead of hand-adding.

> **2026-06-22 — v0.76.0 (deployed on 9201): three campaign-#3 hardening fixes.**
> - **S1**: corrupt `settings.json` no longer crash-loops — `save()` writes a last-known-good `.bak`
>   (after the primary rename); `Load()` recovers from `.bak`, else preserves the corrupt file as
>   `*.corrupt-<ts>` + safe defaults. New `Settings.LoadWarning` → dashboard banner. (`main.go` Fatalf
>   now only on the IO-unreadable path.)
> - **F2**: `web.validStackName` gates `backupRestoreHandler` + `apiExportStart` (reject `/ \ .. NUL`)
>   before any restore/export — closes the traversal defense-in-depth gap (no escape had occurred, but
>   it relied on downstream map-lookups).
> - **S3**: `quiesce.readMarker` logs `[WARN]` + quarantines a corrupt marker to `*.corrupt-<ts>`
>   instead of silently dropping it.
> All three live-validated on 9201 (truncate settings → recover from .bak no crash-loop; traversal
> restore → rejected, /etc intact; corrupt quiesce marker → quarantined). Tests + red-proofs.
> Still-open campaign-#3 deferrals (NOT done): time-chaos on a dedicated VM, host reboot (supervised),
> `.fab` import compose-fuzzing. See `felhom.eu/documentation/tests/test-campaign-3-2026-06-22-findings.md`.

Last updated: 2026-06-22 (v0.75.0 — gate userdata MkdirAll on a live mountpoint)

> **2026-06-22 — v0.75.0 (deployed on 9201): userdata MkdirAll gated on a live mountpoint.**
> Both `MkdirAll`-into-`<drive>/userdata` sites — the deploy belt (`stacks.ensureUserdataMounts`) and
> the FileBrowser sync (`web.syncFileBrowserMounts`) — now skip when an **external** drive root (under
> `StableParentDir=/mnt/felhom-drives`, `!= sysDataPath`) is **not a live mountpoint** (`system.IsMountPoint`).
> Closes the campaign-#2 hazard where a drive-absent window produced `mkdir …/userdata: permission denied`
> + transient `Created` flapping AND could write app data onto the guest **rootfs** (shadowed when the
> drive returns). The app is held by `planDriveGates` instead. System/local path is never gated. New
> `Manager.isMountPoint` seam + pure `web.skipFileBrowserPath` helper for tests. Live-proven via a C6
> re-run: 0 permission-denied, 0 shadow dirs on the rootfs, apps recover on reconnect.
>
> **Boot-ordering design note (NOT implemented — decide separately):** the *boot-time* `mkdir …
> permission denied` is a different cause — **docker's** boot-restore auto-starts drive-backed
> containers (`restart: unless-stopped`) before the agent mounts the drives, so docker (not the belt)
> tries to create the bind source; `planDriveGates` recovers them after mount convergence (why a reboot
> ends healthy). Options: (A) accept + suppress (apps self-heal; lowest risk; maybe downgrade the
> boot-window log level) — recommended now; (B) drive-backed app containers don't docker-auto-start at
> boot (restart policy `no`/`on-failure`) so the controller's gate is the sole starter after mounts
> converge — cleaner but bigger (must cover crash-restart too). See
> `felhom.eu/documentation/tests/test-campaign-2-finding1-recovery-diagnosis.md` (boot-ordering observation).

Last updated: 2026-06-22 (v0.74.0 — controller→agent connection-leak fix)

> **2026-06-22 — v0.74.0 (deployed on demo guest 9201): fixed the controller→agent connection leak.**
> `Server.agentClient()` built a new `agentapi.Client` (new bare `http.Transport`, `IdleConnTimeout:0`)
> per call → one leaked idle ESTABLISHED socket per agent call to `192.168.0.162:8443`, exhausting the
> ephemeral source-port range after ~5 days of controller uptime → EADDRNOTAVAIL, which had taken down
> storage UI + host-metrics + whole-guest backup (found in the 2026-06-22 unattended campaign). Fix:
> memoize ONE shared client via `sync.Once` + harden the Transport (`MaxIdleConnsPerHost:2`,
> `IdleConnTimeout:90s`). Live-proven: idle sockets to `:8443` stay flat at 2 across a 120-call burst
> (pre-fix grew ~1/call → 132). Agent/firewall untouched. Diagnosis + campaign findings live in
> `felhom.eu/documentation/tests/unattended-test-campaign-2026-06-22-*.md`. **Separate open item:** the
> defense-in-depth host firewall rule scoping `:8443` to the guest bridge subnet is still absent
> (pve-firewall disabled) — to be closed independently.

Last updated: 2026-06-13 (v0.60.0 backlog-Medium cleanup)

> **Live version: controller v0.60.0** (deployed on demo guest 9201), agent **v0.30.0** (AGENT-001 deployed), hub **v0.11.0**.
> **NOTE:** the long-form sections far below this banner are stale (last full pass ~v0.16.1). Current
> state is the dated entries immediately below + `CHANGELOG.md` + the now-authoritative **central docs at
> `felhom.eu/documentation/controller/`** (code-verified) + auto-memory `MEMORY.md`.
>
> **2026-06-13 — v0.60.0 backlog-Medium cleanup (BUGHUNT reconcile):**
> - **M25** (Server.integrationMgr data race — constructor goroutine vs post-construction `SetIntegrationManager`):
>   **FIXED** via `atomic.Pointer`; `-race`-verified. **M4/M5/M6** verified already FIXED (no action).
> - **M18** (dump re-validation every 5 min — perf) and **M19** (`deriveStackName` misattribution — low
>   incidence) verified LIVE but cross-package-entangled → branches `fix/m18-dump-validation-cache` /
>   `fix/m19-stackname-crossref` (notes + fix plan, PENDING REVIEW, not deployed).
>
> **2026-06-13 — v0.59.0 audit fixes + documentation centralization:**
> - Fixed the validated 2026-06-13 audit findings (records: `felhom.eu/documentation/audits/`):
>   **CTRL-001** (`.fab` import path traversal — manifest segment validator, fail the parse);
>   **CTRL-T2-1** (ghost-deployed on crash — `app.yaml` now persists `deployed:false` until `compose up -d`
>   succeeds, flipped true only on success; in-memory flag still true during pull for UX);
>   **H10** (plaintext secret on encrypt failure — `SaveAppConfig` now fail-closed, returns an error);
>   **M2** (misleading lock on init-only `SetStackProvider` removed). All with regression tests.
> - **AGENT-001** (wrong-disk wipe TOCTOU) fixed on agent branch `fix/agent-001-wipe-durable-reresolve`
>   — PENDING REVIEW, NOT deployed (supervised merge+golden-rebake reserved; see `AGENT-001-FIX-NOTES.md`).
> - Built + deployed v0.59.0 to demo guest 9201 (bootstrap mechanism); verified healthy, dashboard 200.
> - Documentation centralized under `felhom.eu/documentation/` (controller subtree + audits + top index),
>   code-verified; `controller/README.md` banner points there; this CONTEXT banner refreshed.
>
> **2026-06-13 — v0.58.0 OS/Docker-data split prevention layer (Phase 2; Phase 1 = agent v0.29.0):**
> - OS rootfs + Docker data split onto separate local-lvm volumes (golden bakes 32G rootfs + 256G
>   /var/lib/docker, backup=1, **overlay2** so images live on the data vol). Infra protected by
>   PREVENTION not placement: `system.GetDockerVolumeHeadroom()` (statfs "/" = data vol) reserves
>   max(5GB,10%); `deployStack` refuses HTTP 507 below buffer; deploy.html banner+disable; monitor
>   (warn80/crit90) watches the same vol; log rotation baked into the golden daemon.json.
> - Live-validated by DESTROYING + RE-PROVISIONING 9201 from the split golden: split layout, overlay2,
>   images on data vol, lean rootfs, OS isolation (data vol 100% → rootfs healthy), deploy gate 507,
>   ActualBudget volume on data vol, hub config pull, external access via CF. RomM/USB re-enroll = the
>   documented final restore step (RomM data safe on host USB). See memory [[os-data-split]].
>
> **2026-06-13 — v0.57.0 UI fixes (Part A of the UI-fixes/storage-spike spec):**
> - A1: fixed the RIGHT storage list — `#host-storage-bars` (the JS-filled, agent-PVE-storage list:
>   `local`/`local-lvm`/`felhom-pbs`/`felhom-usb`), which reordered on every poll. Now
>   `enrichHostStorageTargets` sorts `/api/host-metrics` server-side + adds friendly Hungarian
>   labels/purpose. Display-only — PVE storage ids never renamed. (v0.56.0's 4C had sorted the OTHER,
>   server-rendered user-data list.)
> - A2: per-app Tier-2 config panel at `GET/POST /stacks/{name}/backup`; the dead-end "Beállítás" button
>   (was → deploy page) is repointed there. Pin a target drive / toggle Tier 2 off; prefs
>   (`UserDisabled`/`PreferredTarget`) persist on `CrossDriveBackup` and survive the runner's status
>   writes (`withTier2Prefs`). Always visible incl. single-SSD + non-HDD (PBS-context) apps.
> - Part B (storage OS/data split spike) = build-nothing; findings → `felhom-agent/REPORT-storage-split-spike.md`.
> - Live-validated on guest 9201; build/deploy = golden bootstrap (`/etc/felhom-controller-image` + restart `felhom-controller-bootstrap.service`).
>
> **2026-06-13 — v0.56.0 Phase 4: FileBrowser scoping + UI polish (SLICE COMPLETE):**
> - 4A: FileBrowser bind scoped to `<drive>/appdata` (recovery units + Tier 2 copies under `backups/`
>   NOT mounted → customer can't browse/delete the restore source). 4B: deploy storage step states
>   files-on-drive / DB-on-fast-SSD. 4C: `buildStorageBars` stable sort + purpose description on the
>   monitoring list (user-data drives only; agent local/local-lvm/pbs live on the storage page, not here).
> - Live-validated (9201): FileBrowser mount `/mnt/felhom-usb/appdata -> /srv/felhom-usb` (backups hidden);
>   deploy + monitoring text rendered. **All 5 phases (1, 2, 2b, 3, 4) shipped + live-validated, v0.52→v0.56.**
>
> **2026-06-13 — v0.55.0 Phase 3: auto off-drive Tier 2 (rootfs-headroom guard):**
> - `internal/backup/tier2.go`: rsync `-a --delete` of each HDD app's recovery unit + appdata → a
>   DIFFERENT physical disk (`<target>/backups/secondary/<app>/`). Auto target: prefer another registered
>   drive (off-disk via `system.SamePhysicalDevice`), else internal SSD for SMALL units only.
> - **Rootfs-headroom guard** (`tier2FitsHeadroom`, unit-tested): SSD = ~8G guest rootfs, so REFUSE
>   unless the unit fits leaving reserve = max(2G, 20%) free; honest "needs 2nd HDD" status when nothing
>   fits — never fills the rootfs. Status via surviving `settings.CrossDriveBackup`; "2. mentés" UI card
>   now populated (`buildAppBackupRows`). Daily `tier2-backup` 03:30 + `POST /api/backup/tier2`.
> - **Live-validated (9201):** happy path (RomM → SSD, off felhom-usb, 77KB, "[SSD: DB/config only]");
>   refuse path (1G userdata dummy → REFUSED with honest msg, rootfs not filled); UI card shows
>   "Sikeres → belső SSD (csak DB/konfiguráció)". Demo cleaned.
> - Next: Phase 4 (FileBrowser scoping + deploy-UI DB-on-SSD note + monitoring sort).
>
> **2026-06-13 — v0.53.0/v0.53.1 Phase 2: per-app recovery unit (capture side, SECRET-FREE):**
> - Each app's `backups/primary/<app>/` becomes a self-contained recovery unit: `compose/`
>   (docker-compose.yml + .felhom.yml + **secret-stripped** app.yaml) + db-dumps/ + volume-dumps/ +
>   `manifest.json` (image pins, secret env-var NAMES, data_key names, checksums, secret_source note).
> - **Secret-free by design.** Decided after reading the ACTUAL hub code: hub is zero-knowledge (no app
>   secrets); app.yaml + key live on the guest rootfs → in the PBS whole-guest snapshot. So the unit
>   stores no secret/data-key/image; restore recovers secrets from the guest's app.yaml (live/PBS),
>   regenerates nothing. `data_key` (DeployField.DataKey; AdventureLog SECRET_KEY marked) = fail-closed
>   restore annotation only.
> - Capture needs no decryption (non-secret env is plaintext; excludes secret-named + encrypted keys).
>   Wired into RunDBDumps AND the periodic RefreshCache (idempotent checksum-skip → no USB thrash).
> - **Deploy mechanism resolved:** controller in guest 9201 is golden/bootstrap-managed —
>   `felhom-controller-bootstrap.service` docker-runs the tag from `/etc/felhom-controller-image`
>   (gitea anon-pull). Deploy = build+push → anon-pull → update tag file → restart the service.
> - **Live-validated (9201):** RomM unit captured (images=3, secrets=3, data_keys=0), secret-leak grep
>   = NO_LEAK.
> - **v0.54.0 Phase 2b (restore-from-unit + fail-closed gate):** `RestoreFromRecoveryUnit` recreates an
>   app from its unit + secrets recovered from the GUEST's live app.yaml (`RecoverStackSecrets`,
>   `stacks.RedeployFromEnv`), regenerating nothing. `reconcileRestoreSecrets` (pure, unit-tested) is the
>   fail-closed gate: missing/empty data-key → REFUSE (needs PBS whole-guest restore); missing resettable
>   secret → warn+proceed. Wired into `/backup/restore`. Gate + orchestration + data_key parsing
>   unit/integration-tested; deployed v0.54.0 healthy.
> - **LIVE-validated (9201, AdventureLog):** unit manifest `data_key_env_vars:[SECRET_KEY]`
>   (catalog→manifest live); with SECRET_KEY made unrecoverable, `POST /backup/restore` REFUSED with the
>   exact fail-closed message BEFORE any compose-up. Demo has NO dashboard password → API open (auth+CSRF
>   skipped), driven via public URL. NOTE: full deploy-with-data→restore e2e blocked because AdventureLog
>   images don't fit the 8G guest rootfs ("no space left") — that's the Phase 3 rootfs-headroom concern
>   seen live. Demo left clean (AdventureLog reverted to not-deployed).
> - Next: Phase 3 (Tier 2 auto off-drive, rootfs-headroom guard), Phase 4 (FileBrowser + UI).
>
> **2026-06-13 — v0.52.0 Phase 1 GATE: deploy-side double-nest fix (catalog) + path-agreement test:**
> - The `felhom-data` double-nest lived in the **app-catalog compose templates**
>   (`${HDD_PATH}/felhom-data/appdata/<app>`), not in `deploy.go`. On a Model-A in-guest drive the mount
>   already IS the `felhom-data` namespace, so it double-nested on disk while the v0.51.0 backup helpers
>   resolved single-nested → divergence. Fixed all four HDD templates (romm, nextcloud, immich,
>   paperless-ngx) → `${HDD_PATH}/appdata/<app>`.
> - New `internal/stacks/hddpath_agreement_test.go` locks deploy-resolver (`ParseComposeHDDMounts`) ==
>   backup helper (`AppDataDir(NamespaceRoot(.,true))`). No controller runtime change → no image rebuild
>   (deployed stays 0.51.0, functionally current; golden not rebaked for a no-op).
> - **Live (guest 9201):** git-sync auto-delivered the fix to all four stack files; RomM migrated
>   (stop→move→verify→redeploy) from `/mnt/felhom-usb/felhom-data/appdata/romm` →
>   `/mnt/felhom-usb/appdata/romm`, healthy + HTTP 200, no data loss, old namespace empty. **GATE PASSED.**
> - Next: Phase 2 (per-app recovery unit), Phase 3 (auto-enabled off-drive Tier 2 w/ rootfs-headroom
>   guard), Phase 4 (FileBrowser scoping + deploy-UI DB-on-SSD note + monitoring sort).
>
> **2026-06-12 — storage UX polish (v0.45.0), pairs with felhom-agent v0.24.0:**
> - **Agent eject role-gate (Part A, felhom-agent v0.24.0):** `POST /disks/eject` now refuses to
>   unmount system/backup storage *at the agent* (fail-safe to protected on ambiguity) — the UI hiding
>   the button was never the control. Validated live on guest 9201 (eject `/var/lib/vz` → 403, no unmount).
> - **Controller (Part B, v0.45.0):** B1 deterministic `/api/disks` order (user-data→system→backup,
>   alpha within); B2 init wizard excludes mounted drives; B3 **Regisztrálás** primary action for a
>   mounted-but-unregistered user-data drive (`POST /api/storage/register`); B4 per-card purpose
>   descriptions + app-backing tags + tiering note (`local` & `local-lvm` both kept); B5 eject already
>   names affected apps. All validated live on guest 9201.
> - **Golden REBAKED** to controller 0.45.0 (`/root/build-golden.sh`; gitea allows anon pull, no creds);
>   archive `local:backup/vzdump-lxc-9100-2026_06_12-09_53_03.tar.zst`, build guest 9100 purged.

---

## About Viktor (project owner)

- Works at Deutsche Telekom (Budapest), building Felhom.eu as a side business
- Felhom.eu: managed home-server service for Hungarian households
- Technical but prefers pragmatic solutions over over-engineering
- Runs all infrastructure on Gitea (gitea.dooplex.hu), k3s cluster for management
- Customer deployments use Docker Compose (not Kubernetes) for simplicity

### felhom-controller (this repo)
- **Version:** v0.16.1
- **Phase 1:** ✅ COMPLETE — Stack Manager + Deploy Flow
- **Phase 2:** ✅ COMPLETE — Monitoring & Health (scheduler, CPU/temp, healthchecks.io pings)
- **Phase 3:** ✅ COMPLETE — Backups (DB dumps, restic integration, manual trigger, **dedicated backup page**)
- **Phase 4:** ✅ COMPLETE — Monitoring Page with Metrics Store (SQLite, Chart.js, system + container metrics)
- **Phase 5:** ✅ COMPLETE — Authentication, Persistence & Settings Page (settings.json, password change, session management)
- **Phase 6:** ✅ COMPLETE — Monitoring Warnings, Dashboard Alerts & Notification System
- **Phase 7:** ✅ COMPLETE — Storage Overview, Per-App Backup Toggles & Limited Restore
- **Phase A:** ✅ COMPLETE — Storage Paths Foundation (registry, auto-discovery, per-app HDD_PATH, deploy dropdown, health monitoring)
- **Phase B:** ✅ COMPLETE — Storage Management UI Polish & Health Severity Fix (flash messages, label editing, app details, FS info, deploy free space, backup context)
- **Phase C:** ✅ COMPLETE — Storage Init Wizard, Data Migration & Startup Fix (disk scan/format/mount wizard, rsync-based migration, startup pings)
- **v0.11.1 bugfix:** ✅ COMPLETE — Storage Scan: system disk detection via host fstab + blkid UUID resolution; FSType enrichment via `blkid -o export`
- **v0.11.2 bugfix:** ✅ COMPLETE — /host-dev mount for block device access; `HostDevicePath()` helper; all format/scan/safety ops use /host-dev
- **v0.11.3 bugfix:** ✅ COMPLETE — Added `fdisk` package to Dockerfile (provides `sfdisk`; not in `util-linux` on Debian bookworm)
- **v0.11.4 bugfix:** ✅ COMPLETE — FormatAndMount: fixed sfdisk (wipefs+force+`,,`), mount (explicit device path), mount propagation (rshared), ASCII label, smart partition skip, findmnt verification
- **v0.11.6:** ✅ COMPLETE — FileBrowser auto-mount sync (`syncFileBrowserMounts()`) + 3 UI fixes (badge color, progress bar, button text)
- **v0.11.7:** ✅ COMPLETE — Stale data cleanup + FileBrowser sync after migration + deploy page title fix
- **v0.11.8:** ✅ COMPLETE — Per-App Cross-Drive Backup (3-2-1 rule): rsync/restic to secondary drive, deploy page UI, backup page summary, scheduler jobs, API endpoints
- **v0.11.9:** ✅ COMPLETE — UI Polish Fixes: spacing, tooltip on "Módszer", status dot instead of disabled checkbox, progressive disclosure, emoji cleanup
- **First app deployed:** Paperless-ngx on demo-felhom.eu (2026-02-13)
- **Running on:** demo-felhom (N100 mini PC) at 192.168.0.162:8080, felhotest (Proxmox VM) at router.abonet.hu:33022
- **All Phase 1-5 features working:** deploy, start/stop/restart/update, logs, health-aware states, auth, monitoring, backups, backup detail page, system monitoring page, settings page

## Architecture decisions

| Decision | Rationale |
|----------|-----------|
| Go stdlib for web (no Gin/Echo) | Minimal dependencies, single binary, easy to embed templates |
| Templates as go:embed HTML/CSS files | Zero runtime file dependencies (compiled into binary), but each template is a separate editable file |
| Docker Compose for customers (not k8s) | Simpler troubleshooting, customers don't need k8s knowledge |
| k3s for management infra only | Viktor's own services (gitea, monitoring, website) run on k3s |
| Cloudflare Tunnel for remote access | No port forwarding needed, works behind any NAT |
| app.yaml per stack | Separates deploy config from compose files, survives git pulls |
| Password fields require explicit input | Prevents accidental empty-password deployments |
| Health-aware state from Docker Status field | Docker's State says "running" even for unhealthy containers |
| Memory limits via deploy.resources.limits | Prevents runaway containers; ~50% headroom over expected usage |
| System info from /proc/meminfo + statfs | No external dependencies, cheap to read on each page load |
| mem_request vs mem_limit (K8s-inspired) | Requests = expected usage (hard block), limits = peak (overcommit OK) |
| 384MB reserved for system | Prevents deploying apps that would starve the OS/controller |
| Logo SVG embedded as Go constant | Same approach as CSS/HTML — zero external file deps |
| Git sync via os/exec git CLI | No Go git library needed, git is in the container image |
| SHA-256 for content comparison | Only copy changed files, avoid unnecessary disk writes |
| 30s debounce on manual sync | Prevents spamming the git server |
| Orphan = deployed but not in catalog | Safe lifecycle: remove from catalog → mark orphaned → user deletes via UI |
| FileBrowser as infra (not catalog) | Needed even after apps deleted (user browses HDD data); deployed by setup script |
| Protected HDD paths | Safety net: never delete top-level HDD dirs (media, storage, Dokumentumok, appdata) |
| Central scheduler (not ad-hoc goroutines) | Single place to register/monitor all periodic tasks, graceful shutdown, skip-if-running |
| CPU sampling via background goroutine | /proc/stat delta needs two readings — collector runs every 5s, GetInfo() reads cached value |
| Temperature from /host/sys (Docker mount) | Container can't read host /sys directly — mount /sys:/host/sys:ro, try /host/sys first |
| Restic password auto-generated | No manual setup needed — generated on first backup run, stored in named volume |
| DB discovery via docker inspect | No config needed — discovers postgres/mariadb containers by image name + env vars |
| Backup orchestrator with running flag | Prevents concurrent backups, supports both scheduled and manual trigger |
| modernc.org/sqlite (pure Go) | No CGO/gcc needed in Docker build stage — keeps `CGO_ENABLED=0` static binary |
| AlertManager state-based refresh | Alerts regenerated every 5min from health report — no persistent storage needed, always reflects current state |
| Notification relay via hub | Controller → hub → Resend → email. Hub acts as central relay: knows customer email, handles Resend API. Controller only needs hub URL + API key |
| In-memory notification cooldowns | Per-event-type cooldown map (default 6h). Lost on restart = acceptable (better to re-notify than miss). No persistence needed |
| Health status change detection | Only notify on degradation (ok→warn, ok→fail, warn→fail). Avoids spam on flapping. First run records baseline, doesn't notify |
| Resend HTTP API (no SMTP) | Direct POST to api.resend.com — same pattern as website contact-mailer. Simpler than SMTP setup, good deliverability |
| Preferences sync on save + startup | Controller pushes prefs to hub (not pull). Startup sync handles hub DB rebuild. Local save always succeeds even if sync fails |
| Chart.js embedded locally | Customer hardware may not have internet — CDN not reliable for offline environments |
| StackDataProvider interface | backup package needs stack data but can't import stacks (circular). Interface in backup, thin adapter in main.go |
| Password sync to hub via report | Restic password in Docker named volume on SSD. Hub sync provides redundancy for disaster recovery |
| App backup via HDD mounts only | Docker volumes at /var/lib/docker/volumes/ not mounted in controller. HDD data is the important user data; DB in volumes covered by nightly dump |
| Restore uses running mutex | Prevents concurrent backup+restore on same restic repo. Reuses existing `m.running` flag |
| Storage paths registry in settings.json | Multi-storage support: each app's HDD_PATH from app.yaml is authoritative. Auto-discovery on startup avoids manual config. Registry enables UI management + health monitoring per path |
| /mnt:/mnt:rw mount in controller | Replaces per-path HDD_PATH mount. Enables multi-storage + restore writes. All customer HDD mounts are under /mnt/ by convention |
| Per-app HDD_PATH resolution (app.yaml > global) | App's own env HDD_PATH is Priority 1, registered storage paths as fallback. Eliminates dependency on global controller.yaml hdd_path |
| Mount-point detection via syscall.Stat_t.Dev | Compares device ID of path vs parent dir — reliable check that path is on separate filesystem. Prevents data writes to SSD |
| Health severity: mount-point = warning | Non-mount-point is informational, not a service failure. FAIL reserved for genuinely broken things. Avoids false alarms on demo/test environments |
| FS info via findmnt + sysfs | `findmnt -n -o SOURCE,FSTYPE --target <path>` for filesystem type/device. `/sys/block/<dev>/device/model` for disk model. Best-effort, returns nil on failure |
| Query param flash messages | Stateless, no session store needed. Consistent with backup page pattern. `?storage_msg=success&storage_detail=...` |
| StorageLabels map on stacks page | Separate map passed to template (not modifying Stack struct). Built from deployed apps' HDD_PATH → registered path label lookup |
| Metrics downsampling via SQL | Bucket-based AVG in GROUP BY keeps Chart.js responsive with up to 30 days of data |
| 60s metrics collection interval | Good balance of resolution vs. storage — ~44K rows/month for system metrics |
| /etc/os-release mounted read-only | Container can't read host OS info directly — mount to /host/etc/os-release:ro |

## Key file locations on demo-felhom

```
/opt/docker/felhom-controller/         # Controller compose + config
  ├── controller.yaml                  # Customer config (domain, auth, paths)
  ├── docker-compose.yml               # Controller's own compose
  └── data/                             # Controller persistent data (named volume)

/opt/docker/stacks/                    # All app stacks
  ├── traefik/                         # Reverse proxy (protected)
  ├── cloudflared/                     # Tunnel (protected)
  ├── paperless-ngx/                   # First deployed app ✅
  │   ├── docker-compose.yml
  │   ├── .felhom.yml                  # App metadata
  │   └── app.yaml                     # Deploy config (env vars, locked fields)
  └── whoami/                          # Test stack (not deployed)

/mnt/hdd_placeholder/storage/          # HDD storage for apps
  └── paperless/
      ├── consume/                     # Drop files here for OCR
      ├── media/                       # Processed documents
      └── export/                      # Backup exports
```

## Related repositories and their state

| Repository | Status | Notes |
|------------|--------|-------|
| felhom-controller | Active | This repo. Controller code + deploy scripts |
| app-catalog-felhom.eu | Active | 10 app templates, all with .felhom.yml metadata + memory limits |
| felhom.eu | Active | Website + hub/ subfolder (felhom-hub service) + k8s manifests |
| homelab-manifests | Stable | k3s cluster running (dooplex.hu services) |
| misc-scripts | Utility | collect-repo.sh, backup helpers |

## Gotchas & lessons learned

- `docker compose restart` ≠ `docker compose up -d` — restart doesn't pick up new images
- Go maps have random iteration order — always sort slices before displaying
- Docker `.State`="running" doesn't mean healthy — check `.Status` for "(health: starting)" / "(unhealthy)"
- Paperless-ngx needs `PAPERLESS_OCR_LANGUAGES` (plural) to install language packs, `PAPERLESS_OCR_LANGUAGE` (singular) to select
- In-memory Deployed flag must be set BEFORE `docker compose up -d` (not after) — compose can take 30-60s for image pulls, during which the UI would show a stale "Telepítés" button
- Cloudflare Tunnel handles *.demo-felhom.eu → Traefik handles Host()-based routing to containers
- BIOS "AC Power Recovery" must be enabled on N100 for auto-restart after power outage
- `docker compose up -d` returns exit 0 even when containers immediately crash-loop — need post-start status check to detect this
- When logging env vars for debugging, only log keys (not values) to avoid leaking secrets in log files
- Mealie image (`ghcr.io/mealie-recipes/mealie`) doesn't include wget/curl — use Python TCP socket check for healthcheck
- Mealie DB migrations on first start take ~40s (alembic) — use `start_period: 60s` to avoid premature unhealthy status
- Alpine-based images (filebrowser, vaultwarden) have wget via BusyBox — healthchecks with `wget --spider` work fine
- Deploy `sed` command to update image version must target only the `image:` line — naive `sed 's|name:OLD|name:NEW|'` also matches the service name line (e.g., `felhom-controller:` → `felhom-controller:0.2.12`), breaking YAML. Use `sudo sed -i 's|image:.*felhom-controller:[^ ]*|image: ...felhom-controller:NEW|'` or similar scoped pattern
- Hungarian quotation marks `„"` in YAML: `„` (U+201E) is safe inside YAML double-quoted strings, but the closing `"` must NOT be ASCII `"` (0x22) — it terminates the YAML string. Use `\"` escape or Unicode `"` (U+201D). This caused a silent parse failure for the entire `.felhom.yml` file
- Never silently swallow parse errors — always log them. Silent failures make debugging impossible (took a dedicated debug session to find a simple quoting issue)