felhom-controller/CONTEXT.md

CONTEXT.md — Project Memory

This file serves as persistent project memory across Claude Code sessions. It replaces the auto-generated "Memory" from the claude.ai Project. Update this file at the end of each working session with current state, recent decisions, and anything the next session needs to know.

Ask Claude Code: "Please update CONTEXT.md with what we did today"

Last updated: 2026-07-03 (docs: CLAUDE.md refreshed — stable orientation; runbooks live in the felhom skills)

2026-07-03 — CLAUDE.md slimmed to stable orientation (full package map, verified 9201 deploy summary, no version-pinned state). Deep runbooks/design/testing doctrine now in the personal skills felhom-build-deploy / felhom-ui-design / felhom-testing (source: felhom.eu/skills/).

2026-07-03 — REUSE.md exists at the repo root (canonical helpers / patterns / traps / seams, code-verified). Check it before writing new code; update it in the same commit that adds/changes a shared helper (maintenance rule now in CLAUDE.md).

2026-07-02 — v0.98.0 (deployed on 9201): Tárhely IA follow-up. User feedback on D1: NAS-add and local-drive enrollment buttons sat side by side — confusing. /storage split into two subpages under the Tárhely nav item (nested sub-links, .nav-links-nested): /storage = Meghajtók (drives + agent view + wizards + manual add) and /storage/network = Hálózati tárhely (NAS) (NAS-megosztások list + add form; own openDialog copy). networkStoragePageData (page key storage-network) split out of storagePageData. No API/storage-semantics change.

2026-07-02 — v0.97.0 (deployed on 9201): TASK-D1 — settings split + Tárhely page (IA only). The 1451-line settings.html monolith is split into four pages: /storage (new main-nav Tárhely: Adattárolók + NAS + unified agent drive view + wizard entry), /settings (Rendszer: konfig, verzió/frissítés, vezérlő+kiszolgáló újraindítás), /settings/notifications (Értesítések

• Alkalmazás-email), /settings/security (Jelszó, Földrajzi korlátozás, Vészhelyzeti információk). Sidebar gains the Tárhely item + a "Beállítások" group with three sub-links. No storage/agent/API behavior change — pages moved, logic didn't; every /api/storage/* endpoint + POST action unchanged. Decisions: (a) storage flashes + wizards moved to /storage; old /settings/storage/{init,attach} 301 to /storage/{init,attach}. (b) The two duplicated drive views MERGED: registry cards render server-side, then JS enriches each connected user-data card in place from the agent /api/disks (role tag, durable-id, agent actions) joined on mount path, plus two groups — Rendszermeghajtók (system/backup, read-only, lock tag, NO actions) and Nem regisztrált (register only); agent-down → one warn note, registry cards still render. (c) Every native confirm()/prompt() on the four pages now routes through a light .confirm-overlay (openDialog; texts verbatim; type-to-confirm kept only where it existed) — the D0 tab-freeze is gone. Contract for future UI: new templates must pass scripts/template_id_gate.py (every JS element-ID resolves in its own template) and scripts/emoji_gate.py (0 emoji — the D0 grep gate false-negatived multibyte emoji on Windows; 8 survivors removed). .badge-lock/.lock-ico CSS deleted; other .badge* still used on secondary pages (D2 to finish). Next: D2 (badge→tag on backups/secondary pages; storage.html is still ~700 lines — NAS-add partial candidate). NOT live-validated: agent-down degradation (don't stop the live agent — static/unit only); destructive storage ops via the moved overlay paths (endpoints unchanged; supervised session).

2026-07-02 — v0.96.0 (deployed on 9201): TASK-D0 — design system v2 re-skin (appearance only). The whole customer UI (dashboard AND setup wizard) now renders in the approved v2 language: navy token palette (--bg-0/1/2, --line, --text-1/2/3, --blue, --warn, --crit), single 2px radius, no shadows; exception-based status color — nominal is blue/neutral, amber/red only on deviation; zone bars → the hairline .meter (neutral 70/85 ticks, warn/crit flag „Fogyóban a hely" / „Kritikusan kevés hely"); pills → .tag / .metarow. Decisions: (a) stateColor semantics changed — stopped/exited = neutral, NOT red (operator-approved; failures surface via unhealthy

• alerts); restarting = warn; outputs are now run/progress/warn/neutral/off and nominal/warn/crit — any new template MUST use these suffixes (truth tables guarded by unit tests, stateLabel copy frozen byte-identical). (b) Fonts (PJS + JBM variable woff2, latin+latin-ext) and a 30-icon Lucide sprite are vendored in the binary (/static/fonts/, templates/icons.html) — no CDN; fonts.googleapis.com, box-shadow, 999px and the emoji set are banned strings (grep gate in REPORT §5, 143→0). (c) Setup wizard handleCSS bug fixed: it read a dataDir-derived path that never exists in the container → always served minimalCSS in production; now serves embedded web.StyleCSS(). (d) Found+fixed pre-existing v0.93.0 bug: /backups 500'd once an off-box backup had run (OffboxTarget.LastRun is an RFC3339 string fed to timeAgo; new timeAgoStr). Canonical reference: felhom.eu/documentation/design/design-system.md. Next: D1 (settings IA split + Tárhely main-nav promotion; also migrate the remaining native confirm() dialogs to the type-to-confirm overlay and finish badge→tag markup on secondary pages). NOT live-validated: setup wizard visuals (unit-only), warn/crit meters on real hardware (demo healthy).

2026-07-01 — v0.95.0 (deployed on 9201): Impl-2b — raw-drive enrollment wizards. Both enrollment wizards (storage_init.html / storage_attach.html) now fetch candidates from the agent's raw-device scan GET /api/disks/candidates (proxy agentDiskCandidatesHandler → agent Impl-2a) instead of the Observe()-based /api/disks, so a brand-new (non-PVE-storage) drive is discoverable + enrollable. agentapi.ListCandidates + resolveEnrollUUID (a raw candidate isn't in /disks → resolve its fs-UUID from the scan). Live end-to-end on felhom-pve: raw /dev/sdd (SD card) → wizard → format (Impl-1 guard) / attach → mount → bind → intent → guest-bind → "Aktív" in the UI (3rd drive, no false detach). Needed a chain of AGENT fixes (v0.56–0.58: durableIDForMount / ReassertGuestBinds / HostReader wiring / /disks GuestPath+BoundUnderParent for registry rows) — the whole enroll+tracking machinery had assumed a PVE-storage drive. Follow-ups: runStorageInit should poll the agent's detached-format status (/disks/format/status) so a SLOW-device format completes in one flow (pre-existing); Impl-3 shared-box operator format gate.

2026-06-30 — v0.94.0 (deployed on 9201): pull-based config-refresh. The hub report ACK now also carries a per-customer config_version (hub v0.26.0; a stored counter bumped on every config save). OnPushResponse → ConfigRefresher.Reconcile (internal/report/config_refresh.go): on a change vs. settings.applied_config_version, bootstrap.RefreshConfig re-pulls controller.yaml (re-merging local_api from bootstrap.json; overwrites controller.yaml, never settings.json) → record → graceful self-restart (api.GracefulSelfRestart). First-run records baseline (no restart); unchanged = no-op (no storm); failed pull keeps config + retries. This is the box-pulls-config replacement for the hub's retired inbound "Push Config" — the hub never connects into the box. Companion hub v0.26.0 also retired Trigger Update / Pull Config / Show Diff and dropped geo-disable's inbound notify (kept the hub→Cloudflare WAF removal), and fixed the stale docker-setup.sh setup command → host-install. Live-validated on 9201: first-run baseline (applied=1, no restart) → DB bump 1→2 → re-pull+self-restart (session_secret rotated, local_api preserved, RestartCount 0→1) → 4 cycles no-loop → apps stayed up.

2026-06-29 — Self-health arc complete (agent v0.48.0 + hub v0.22.1). The hub now proactively watches every agent's served local-API leaf fingerprint fleet-wide: the agent reports leaf_fingerprint on its host report (v0.48.0), and the hub HostLeafChecker (monitor/host_leaf.go) raises host_leaf_changed when it changes (trust-on-first-report baseline; empty fp = unknown, never alerts; reads from report_json, no migration; hub-generated so no allowlist change). Independent of the controller channel-check. Live: a leaf regen raised host_leaf_changed (82078fab…→60b5974d…) AND the controller's agent_channel_pin_mismatch — complementary. NOTE: v0.22.0 shipped the checker but the main.go wiring never applied (concurrent file-touch); the LIVE TEST caught it (no event on regen) → fixed in v0.22.1. The original silent-multi-day-outage incident class is now caught from THREE angles (agent capabilities v0.44.0 / controller channel-check v0.90.0+F2 / hub leaf-fp v0.22.x) AND prevented (agent v0.46.0 loud-regenerate + host-install --preserve-state-from). Backlog: the served-fp-vs-pinned-fp authoritative cross-check; the test-run coverage gaps (capability→hub live, host-reboot doubling).

2026-06-29 — v0.91.0: F2 (channel-health born-down alerting). The v0.90.0 channel-health check alerted only on a live up→down transition; a channel broken at startup/reseed (e.g. the controller boots right after a leaf regen → first observation is pin_mismatch) was dashboard-only, no operator email — forever. Fixed with an alerted flag in channelhealth/checker.go: a confirmed down coming from up/unseeded OR a reason-change re-arms then alerts once; steady-down no dup; recovery re-arms; debounce intact (transient born-down still N≥2; healthy first-obs silent). Live-validated: controller restarted into a regenerated-leaf channel → [channel] DOWN (unseeded->down:pin_mismatch)

• Event pushed: agent_channel_pin_mismatch (~60s) — the capstone's missing half. Companions: hub v0.21.0 (same fix for HostCapabilityChecker/HostStalenessChecker — seed only healthy, born-degraded/ stale emits on first Check, cooldown dedups), agent v0.46.0 (EnsureLeaf loud-WARNs a REGENERATED leaf
• host-install --preserve-state-from/populated-host guard = the prevention for the original incident class). Self-health story now complete: agent watches its capabilities, controller watches its link, detection works at startup, and a reinstall can preserve the leaf. Backlog: F1 swap-rollback RestartCount hardening; full live reinstall (supervised); hub-side leaf-fp fleet comparison.

2026-06-29 — v0.90.0: controller→agent channel health-check (self-health slice). A ~60s scheduler job (internal/channelhealth) probes the local-API channel via the PRODUCTION memoized client (Server.ProbeAgentChannel → s.agentClient() + GET /storage — NOT a fresh client, per the spike: self-heals, mirrors the disk UI, no transport leak). Classifies failures (spike Q1 map: pin_mismatch/unauthorized/unreachable/timeout/misconfigured/construction_error/unknown), debounces transient reasons (refused/timeout need N≥2 consecutive so the ~1s agent-restart blip doesn't page; pin/401/DNS/construction alert first-obs), seeds the first observation silently, and on a transition emits an English operator-only event (Notifier.NotifyAgentChannelDown/Recovered) + a Hungarian dashboard banner (AlertManager.SetAgentChannelAlert). Closes the R1-incident gap (channel was only checked once at startup). Required a hub change (felhom-hub v0.20.0): the agent_channel_* event types were rejected by the hub's allowedEventTypes allowlist (HTTP 400) — controller-pushed events are gated, unlike the hub-generated host_* ones. Live-validated: transient restart → no alert; sustained stop → debounce(1/2) then DOWN transition + dashboard banner; start → recovered. No agent change. Self-health story now: agent watches its own capabilities (v0.44.0) + controller watches its link to the agent (this). Backlog: hub-side leaf-fp comparison.

2026-06-29 — OPS (no code change): controller↔agent TLS leaf-pin mismatch RESOLVED via R1. The 2026-06-28 root→non-root agent migration regenerated the agent's local-API leaf (60b5974d…→ced34036…) and truncated its token store, so the in-guest controller's pinned agentapi client failed every call (TLS pin mismatch) — disk UI dead AND the quiesce app-consistent-backup loop dead. Fix: restored the pre-migration local-api.{crt,key} + local-tokens.log from /root/agent-backup-bundle-test/aside-var-lib/ on felhom-pve (R1 RUNBOOK, supervised). Agent now serves 60b5974d… again (== controller pin + bootstrap.json seed) and the restored token store's 9201 entry matches the controller's current local_api.token — zero in-guest change. Gates passed: leaf loaded (not regenerated), wire fp 60b5974d…, authenticated GET /disks 200/5-disks (bogus token 401), and the next quiesce cycle ran a real backup job. Rollback point kept at /root/agent-rollback-20260629-150137/. OPEN (carried forward): the migration/Day-0 path must carry these three files into the new felhom-agent-owned data dir (or treat their loss as a re-bootstrap trigger), else any re-migration reintroduces this. Full record: felhom.eu/documentation/audits/SPIKE-agent-pin-mismatch-2026-06-29.md (§9 Outcome).

2026-06-27 — v0.86.0 (deployed on 9201): Phase 2 managed updates — controller-version FLOOR. On top of the Phase 1 opt-in "update to latest" button, the controller now honors an operator-enforced minimum version (FLOOR) delivered on the hub report ACK (min_controller_version + latest_version; internal/report/pusher.go PushResponse). OnPushResponse → updater.SetFloor() + updater.MaybeAutoUpdate() (rides the report cycle — no new timer). Below the floor → auto-update to the floor (reuses Phase 1 performUpdate: in-guest pull → agent swap → rollback; initiatedBy="auto-floor"). At/above floor → nothing (does NOT chase latest — that's the button). Guards: dev/no-agent/backup → skip; floor must be pullable (floor ≤ latest; floor>latest → warn+noop); one attempt per below-floor condition (in-mem + persisted state) → no flapping. Floor source

• operator UI are hub-side (felhom-hub v0.15.0: per-customer override + global default + report ACK). No agent change (reuses Phase 1 POST /controller/swap). Day-0 now ships current (golden rebuilt at 0.85.1) AND stays current (floor) — the fleet-currency story is closed. Live: dogfood 0.85.1→0.86.0 via the button, then floor 0.87.0 → auto 0.86.0→0.87.0 (no click).

2026-06-26 — v0.84.0 (deployed on 9201): show an app's auto-generated first-login on its page. General, catalog-driven mechanism: .felhom.yml initial_credentials: {file, format json|regex|plain, username_key/password_key | username_pattern/password_pattern, note}. The controller reads the file live from the container (internal/stacks/initialcreds.go ReadInitialCredentials + pure parseInitialCreds, unit-tested), never persists it, and renders a "Kezdeti belépési adatok" card on /apps/{slug} (masked password + reveal/copy). First consumer: crafty-controller (Crafty writes a random admin password to /crafty/app/config/default-creds.txt). Live-verified on 9201: card shows username admin + the real extracted password. Reuse for any future self-seeding app. Security: same exposure class as the existing post-deploy reveal / default_creds card — relies on the prod dashboard being auth-gated (demo public-unauth is the separate tracked issue). Backlog idea unchanged: a backend_scheme hint for TLS backends (v0.83.0 line).

2026-06-26 — v0.83.0 (deployed on 9201): scoped Traefik backend transport for self-signed HTTPS apps. Crafty is the first/only catalog app with an HTTPS backend (self-signed TLS on :8443, no plain-HTTP port). The crafty healthcheck fix un-withheld its route, exposing a pre-existing 502 (Traefik proxied HTTP to the HTTPS backend). Since traefik v3 forbids insecureSkipVerify via Docker labels, the controller now renders a file-provider dynamic file dynamic/serverstransports.yml defining a named insecure-skip-verify transport (infra.RenderServersTransports + stacks.ensureServersTransports, written from EnsureBaseStack outside the traefik-running guard, write-if-changed, hot-loaded). Apps opt in per-service via catalog labels scheme=https + serverstransport=insecure-skip-verify@file. No global insecureSkipVerify — verification stays ON for every other backend (scoped Option B). Pattern to reuse for any future self-signed HTTPS backend. Live-verified: minecraft.demo-felhom.eu 502→302; filebrowser (HTTP) unaffected. Backlog: a .felhom.yml backend_scheme hint so the catalog convention sets these labels instead of hand-adding.

2026-06-22 — v0.76.0 (deployed on 9201): three campaign-#3 hardening fixes.

• S1: corrupt settings.json no longer crash-loops — save() writes a last-known-good .bak (after the primary rename); Load() recovers from .bak, else preserves the corrupt file as *.corrupt-<ts> + safe defaults. New Settings.LoadWarning → dashboard banner. (main.go Fatalf now only on the IO-unreadable path.)
• F2: web.validStackName gates backupRestoreHandler + apiExportStart (reject / \ .. NUL) before any restore/export — closes the traversal defense-in-depth gap (no escape had occurred, but it relied on downstream map-lookups).
• S3: quiesce.readMarker logs [WARN] + quarantines a corrupt marker to *.corrupt-<ts> instead of silently dropping it. All three live-validated on 9201 (truncate settings → recover from .bak no crash-loop; traversal restore → rejected, /etc intact; corrupt quiesce marker → quarantined). Tests + red-proofs. Still-open campaign-#3 deferrals (NOT done): time-chaos on a dedicated VM, host reboot (supervised), .fab import compose-fuzzing. See felhom.eu/documentation/tests/test-campaign-3-2026-06-22-findings.md.

Last updated: 2026-06-22 (v0.75.0 — gate userdata MkdirAll on a live mountpoint)

2026-06-22 — v0.75.0 (deployed on 9201): userdata MkdirAll gated on a live mountpoint. Both MkdirAll-into-<drive>/userdata sites — the deploy belt (stacks.ensureUserdataMounts) and the FileBrowser sync (web.syncFileBrowserMounts) — now skip when an external drive root (under StableParentDir=/mnt/felhom-drives, != sysDataPath) is not a live mountpoint (system.IsMountPoint). Closes the campaign-#2 hazard where a drive-absent window produced mkdir …/userdata: permission denied

• transient Created flapping AND could write app data onto the guest rootfs (shadowed when the drive returns). The app is held by planDriveGates instead. System/local path is never gated. New Manager.isMountPoint seam + pure web.skipFileBrowserPath helper for tests. Live-proven via a C6 re-run: 0 permission-denied, 0 shadow dirs on the rootfs, apps recover on reconnect.

Boot-ordering design note (NOT implemented — decide separately): the boot-time mkdir … permission denied is a different cause — docker's boot-restore auto-starts drive-backed containers (restart: unless-stopped) before the agent mounts the drives, so docker (not the belt) tries to create the bind source; planDriveGates recovers them after mount convergence (why a reboot ends healthy). Options: (A) accept + suppress (apps self-heal; lowest risk; maybe downgrade the boot-window log level) — recommended now; (B) drive-backed app containers don't docker-auto-start at boot (restart policy no/on-failure) so the controller's gate is the sole starter after mounts converge — cleaner but bigger (must cover crash-restart too). See felhom.eu/documentation/tests/test-campaign-2-finding1-recovery-diagnosis.md (boot-ordering observation).

Last updated: 2026-06-22 (v0.74.0 — controller→agent connection-leak fix)

2026-06-22 — v0.74.0 (deployed on demo guest 9201): fixed the controller→agent connection leak. Server.agentClient() built a new agentapi.Client (new bare http.Transport, IdleConnTimeout:0) per call → one leaked idle ESTABLISHED socket per agent call to 192.168.0.162:8443, exhausting the ephemeral source-port range after ~5 days of controller uptime → EADDRNOTAVAIL, which had taken down storage UI + host-metrics + whole-guest backup (found in the 2026-06-22 unattended campaign). Fix: memoize ONE shared client via sync.Once + harden the Transport (MaxIdleConnsPerHost:2, IdleConnTimeout:90s). Live-proven: idle sockets to :8443 stay flat at 2 across a 120-call burst (pre-fix grew ~1/call → 132). Agent/firewall untouched. Diagnosis + campaign findings live in felhom.eu/documentation/tests/unattended-test-campaign-2026-06-22-*.md. Separate open item: the defense-in-depth host firewall rule scoping :8443 to the guest bridge subnet is still absent (pve-firewall disabled) — to be closed independently.

Last updated: 2026-06-13 (v0.60.0 backlog-Medium cleanup)

Live version: controller v0.60.0 (deployed on demo guest 9201), agent v0.30.0 (AGENT-001 deployed), hub v0.11.0. NOTE: the long-form sections far below this banner are stale (last full pass ~v0.16.1). Current state is the dated entries immediately below + CHANGELOG.md + the now-authoritative central docs at felhom.eu/documentation/controller/ (code-verified) + auto-memory MEMORY.md.

2026-06-13 — v0.60.0 backlog-Medium cleanup (BUGHUNT reconcile):

• M25 (Server.integrationMgr data race — constructor goroutine vs post-construction SetIntegrationManager): FIXED via atomic.Pointer; -race-verified. M4/M5/M6 verified already FIXED (no action).
• M18 (dump re-validation every 5 min — perf) and M19 (deriveStackName misattribution — low incidence) verified LIVE but cross-package-entangled → branches fix/m18-dump-validation-cache / fix/m19-stackname-crossref (notes + fix plan, PENDING REVIEW, not deployed).

2026-06-13 — v0.59.0 audit fixes + documentation centralization:

• Fixed the validated 2026-06-13 audit findings (records: felhom.eu/documentation/audits/): CTRL-001 (.fab import path traversal — manifest segment validator, fail the parse); CTRL-T2-1 (ghost-deployed on crash — app.yaml now persists deployed:false until compose up -d succeeds, flipped true only on success; in-memory flag still true during pull for UX); H10 (plaintext secret on encrypt failure — SaveAppConfig now fail-closed, returns an error); M2 (misleading lock on init-only SetStackProvider removed). All with regression tests.
• AGENT-001 (wrong-disk wipe TOCTOU) fixed on agent branch fix/agent-001-wipe-durable-reresolve — PENDING REVIEW, NOT deployed (supervised merge+golden-rebake reserved; see AGENT-001-FIX-NOTES.md).
• Built + deployed v0.59.0 to demo guest 9201 (bootstrap mechanism); verified healthy, dashboard 200.
• Documentation centralized under felhom.eu/documentation/ (controller subtree + audits + top index), code-verified; controller/README.md banner points there; this CONTEXT banner refreshed.

2026-06-13 — v0.58.0 OS/Docker-data split prevention layer (Phase 2; Phase 1 = agent v0.29.0):

• OS rootfs + Docker data split onto separate local-lvm volumes (golden bakes 32G rootfs + 256G /var/lib/docker, backup=1, overlay2 so images live on the data vol). Infra protected by PREVENTION not placement: system.GetDockerVolumeHeadroom() (statfs "/" = data vol) reserves max(5GB,10%); deployStack refuses HTTP 507 below buffer; deploy.html banner+disable; monitor (warn80/crit90) watches the same vol; log rotation baked into the golden daemon.json.
• Live-validated by DESTROYING + RE-PROVISIONING 9201 from the split golden: split layout, overlay2, images on data vol, lean rootfs, OS isolation (data vol 100% → rootfs healthy), deploy gate 507, ActualBudget volume on data vol, hub config pull, external access via CF. RomM/USB re-enroll = the documented final restore step (RomM data safe on host USB). See memory os-data-split.

2026-06-13 — v0.57.0 UI fixes (Part A of the UI-fixes/storage-spike spec):

• A1: fixed the RIGHT storage list — #host-storage-bars (the JS-filled, agent-PVE-storage list: local/local-lvm/felhom-pbs/felhom-usb), which reordered on every poll. Now enrichHostStorageTargets sorts /api/host-metrics server-side + adds friendly Hungarian labels/purpose. Display-only — PVE storage ids never renamed. (v0.56.0's 4C had sorted the OTHER, server-rendered user-data list.)
• A2: per-app Tier-2 config panel at GET/POST /stacks/{name}/backup; the dead-end "Beállítás" button (was → deploy page) is repointed there. Pin a target drive / toggle Tier 2 off; prefs (UserDisabled/PreferredTarget) persist on CrossDriveBackup and survive the runner's status writes (withTier2Prefs). Always visible incl. single-SSD + non-HDD (PBS-context) apps.
• Part B (storage OS/data split spike) = build-nothing; findings → felhom-agent/REPORT-storage-split-spike.md.
• Live-validated on guest 9201; build/deploy = golden bootstrap (/etc/felhom-controller-image + restart felhom-controller-bootstrap.service).

2026-06-13 — v0.56.0 Phase 4: FileBrowser scoping + UI polish (SLICE COMPLETE):

• 4A: FileBrowser bind scoped to <drive>/appdata (recovery units + Tier 2 copies under backups/ NOT mounted → customer can't browse/delete the restore source). 4B: deploy storage step states files-on-drive / DB-on-fast-SSD. 4C: buildStorageBars stable sort + purpose description on the monitoring list (user-data drives only; agent local/local-lvm/pbs live on the storage page, not here).
• Live-validated (9201): FileBrowser mount /mnt/felhom-usb/appdata -> /srv/felhom-usb (backups hidden); deploy + monitoring text rendered. All 5 phases (1, 2, 2b, 3, 4) shipped + live-validated, v0.52→v0.56.

2026-06-13 — v0.55.0 Phase 3: auto off-drive Tier 2 (rootfs-headroom guard):

• internal/backup/tier2.go: rsync -a --delete of each HDD app's recovery unit + appdata → a DIFFERENT physical disk (<target>/backups/secondary/<app>/). Auto target: prefer another registered drive (off-disk via system.SamePhysicalDevice), else internal SSD for SMALL units only.
• Rootfs-headroom guard (tier2FitsHeadroom, unit-tested): SSD = ~8G guest rootfs, so REFUSE unless the unit fits leaving reserve = max(2G, 20%) free; honest "needs 2nd HDD" status when nothing fits — never fills the rootfs. Status via surviving settings.CrossDriveBackup; "2. mentés" UI card now populated (buildAppBackupRows). Daily tier2-backup 03:30 + POST /api/backup/tier2.
• Live-validated (9201): happy path (RomM → SSD, off felhom-usb, 77KB, "[SSD: DB/config only]"); refuse path (1G userdata dummy → REFUSED with honest msg, rootfs not filled); UI card shows "Sikeres → belső SSD (csak DB/konfiguráció)". Demo cleaned.
• Next: Phase 4 (FileBrowser scoping + deploy-UI DB-on-SSD note + monitoring sort).

2026-06-13 — v0.53.0/v0.53.1 Phase 2: per-app recovery unit (capture side, SECRET-FREE):

• Each app's backups/primary/<app>/ becomes a self-contained recovery unit: compose/ (docker-compose.yml + .felhom.yml + secret-stripped app.yaml) + db-dumps/ + volume-dumps/ + manifest.json (image pins, secret env-var NAMES, data_key names, checksums, secret_source note).
• Secret-free by design. Decided after reading the ACTUAL hub code: hub is zero-knowledge (no app secrets); app.yaml + key live on the guest rootfs → in the PBS whole-guest snapshot. So the unit stores no secret/data-key/image; restore recovers secrets from the guest's app.yaml (live/PBS), regenerates nothing. data_key (DeployField.DataKey; AdventureLog SECRET_KEY marked) = fail-closed restore annotation only.
• Capture needs no decryption (non-secret env is plaintext; excludes secret-named + encrypted keys). Wired into RunDBDumps AND the periodic RefreshCache (idempotent checksum-skip → no USB thrash).
• Deploy mechanism resolved: controller in guest 9201 is golden/bootstrap-managed — felhom-controller-bootstrap.service docker-runs the tag from /etc/felhom-controller-image (gitea anon-pull). Deploy = build+push → anon-pull → update tag file → restart the service.
• Live-validated (9201): RomM unit captured (images=3, secrets=3, data_keys=0), secret-leak grep = NO_LEAK.
• v0.54.0 Phase 2b (restore-from-unit + fail-closed gate): RestoreFromRecoveryUnit recreates an app from its unit + secrets recovered from the GUEST's live app.yaml (RecoverStackSecrets, stacks.RedeployFromEnv), regenerating nothing. reconcileRestoreSecrets (pure, unit-tested) is the fail-closed gate: missing/empty data-key → REFUSE (needs PBS whole-guest restore); missing resettable secret → warn+proceed. Wired into /backup/restore. Gate + orchestration + data_key parsing unit/integration-tested; deployed v0.54.0 healthy.
• LIVE-validated (9201, AdventureLog): unit manifest data_key_env_vars:[SECRET_KEY] (catalog→manifest live); with SECRET_KEY made unrecoverable, POST /backup/restore REFUSED with the exact fail-closed message BEFORE any compose-up. Demo has NO dashboard password → API open (auth+CSRF skipped), driven via public URL. NOTE: full deploy-with-data→restore e2e blocked because AdventureLog images don't fit the 8G guest rootfs ("no space left") — that's the Phase 3 rootfs-headroom concern seen live. Demo left clean (AdventureLog reverted to not-deployed).
• Next: Phase 3 (Tier 2 auto off-drive, rootfs-headroom guard), Phase 4 (FileBrowser + UI).

2026-06-13 — v0.52.0 Phase 1 GATE: deploy-side double-nest fix (catalog) + path-agreement test:

• The felhom-data double-nest lived in the app-catalog compose templates (${HDD_PATH}/felhom-data/appdata/<app>), not in deploy.go. On a Model-A in-guest drive the mount already IS the felhom-data namespace, so it double-nested on disk while the v0.51.0 backup helpers resolved single-nested → divergence. Fixed all four HDD templates (romm, nextcloud, immich, paperless-ngx) → ${HDD_PATH}/appdata/<app>.
• New internal/stacks/hddpath_agreement_test.go locks deploy-resolver (ParseComposeHDDMounts) == backup helper (AppDataDir(NamespaceRoot(.,true))). No controller runtime change → no image rebuild (deployed stays 0.51.0, functionally current; golden not rebaked for a no-op).
• Live (guest 9201): git-sync auto-delivered the fix to all four stack files; RomM migrated (stop→move→verify→redeploy) from /mnt/felhom-usb/felhom-data/appdata/romm → /mnt/felhom-usb/appdata/romm, healthy + HTTP 200, no data loss, old namespace empty. GATE PASSED.
• Next: Phase 2 (per-app recovery unit), Phase 3 (auto-enabled off-drive Tier 2 w/ rootfs-headroom guard), Phase 4 (FileBrowser scoping + deploy-UI DB-on-SSD note + monitoring sort).

2026-06-12 — storage UX polish (v0.45.0), pairs with felhom-agent v0.24.0:

• Agent eject role-gate (Part A, felhom-agent v0.24.0): POST /disks/eject now refuses to unmount system/backup storage at the agent (fail-safe to protected on ambiguity) — the UI hiding the button was never the control. Validated live on guest 9201 (eject /var/lib/vz → 403, no unmount).
• Controller (Part B, v0.45.0): B1 deterministic /api/disks order (user-data→system→backup, alpha within); B2 init wizard excludes mounted drives; B3 Regisztrálás primary action for a mounted-but-unregistered user-data drive (POST /api/storage/register); B4 per-card purpose descriptions + app-backing tags + tiering note (local & local-lvm both kept); B5 eject already names affected apps. All validated live on guest 9201.
• Golden REBAKED to controller 0.45.0 (/root/build-golden.sh; gitea allows anon pull, no creds); archive local:backup/vzdump-lxc-9100-2026_06_12-09_53_03.tar.zst, build guest 9100 purged.

---

About Viktor (project owner)

• Works at Deutsche Telekom (Budapest), building Felhom.eu as a side business
• Felhom.eu: managed home-server service for Hungarian households
• Technical but prefers pragmatic solutions over over-engineering
• Runs all infrastructure on Gitea (gitea.dooplex.hu), k3s cluster for management
• Customer deployments use Docker Compose (not Kubernetes) for simplicity

felhom-controller (this repo)

• Version: v0.16.1
• Phase 1: ✅ COMPLETE — Stack Manager + Deploy Flow
• Phase 2: ✅ COMPLETE — Monitoring & Health (scheduler, CPU/temp, healthchecks.io pings)
• Phase 3: ✅ COMPLETE — Backups (DB dumps, restic integration, manual trigger, dedicated backup page)
• Phase 4: ✅ COMPLETE — Monitoring Page with Metrics Store (SQLite, Chart.js, system + container metrics)
• Phase 5: ✅ COMPLETE — Authentication, Persistence & Settings Page (settings.json, password change, session management)
• Phase 6: ✅ COMPLETE — Monitoring Warnings, Dashboard Alerts & Notification System
• Phase 7: ✅ COMPLETE — Storage Overview, Per-App Backup Toggles & Limited Restore
• Phase A: ✅ COMPLETE — Storage Paths Foundation (registry, auto-discovery, per-app HDD_PATH, deploy dropdown, health monitoring)
• Phase B: ✅ COMPLETE — Storage Management UI Polish & Health Severity Fix (flash messages, label editing, app details, FS info, deploy free space, backup context)
• Phase C: ✅ COMPLETE — Storage Init Wizard, Data Migration & Startup Fix (disk scan/format/mount wizard, rsync-based migration, startup pings)
• v0.11.1 bugfix: ✅ COMPLETE — Storage Scan: system disk detection via host fstab + blkid UUID resolution; FSType enrichment via blkid -o export
• v0.11.2 bugfix: ✅ COMPLETE — /host-dev mount for block device access; HostDevicePath() helper; all format/scan/safety ops use /host-dev
• v0.11.3 bugfix: ✅ COMPLETE — Added fdisk package to Dockerfile (provides sfdisk; not in util-linux on Debian bookworm)
• v0.11.4 bugfix: ✅ COMPLETE — FormatAndMount: fixed sfdisk (wipefs+force+,,), mount (explicit device path), mount propagation (rshared), ASCII label, smart partition skip, findmnt verification
• v0.11.6: ✅ COMPLETE — FileBrowser auto-mount sync (syncFileBrowserMounts()) + 3 UI fixes (badge color, progress bar, button text)
• v0.11.7: ✅ COMPLETE — Stale data cleanup + FileBrowser sync after migration + deploy page title fix
• v0.11.8: ✅ COMPLETE — Per-App Cross-Drive Backup (3-2-1 rule): rsync/restic to secondary drive, deploy page UI, backup page summary, scheduler jobs, API endpoints
• v0.11.9: ✅ COMPLETE — UI Polish Fixes: spacing, tooltip on "Módszer", status dot instead of disabled checkbox, progressive disclosure, emoji cleanup
• First app deployed: Paperless-ngx on demo-felhom.eu (2026-02-13)
• Running on: demo-felhom (N100 mini PC) at 192.168.0.162:8080, felhotest (Proxmox VM) at router.abonet.hu:33022
• All Phase 1-5 features working: deploy, start/stop/restart/update, logs, health-aware states, auth, monitoring, backups, backup detail page, system monitoring page, settings page

Architecture decisions

Decision	Rationale
Go stdlib for web (no Gin/Echo)	Minimal dependencies, single binary, easy to embed templates
Templates as go:embed HTML/CSS files	Zero runtime file dependencies (compiled into binary), but each template is a separate editable file
Docker Compose for customers (not k8s)	Simpler troubleshooting, customers don't need k8s knowledge
k3s for management infra only	Viktor's own services (gitea, monitoring, website) run on k3s
Cloudflare Tunnel for remote access	No port forwarding needed, works behind any NAT
app.yaml per stack	Separates deploy config from compose files, survives git pulls
Password fields require explicit input	Prevents accidental empty-password deployments
Health-aware state from Docker Status field	Docker's State says "running" even for unhealthy containers
Memory limits via deploy.resources.limits	Prevents runaway containers; ~50% headroom over expected usage
System info from /proc/meminfo + statfs	No external dependencies, cheap to read on each page load
mem_request vs mem_limit (K8s-inspired)	Requests = expected usage (hard block), limits = peak (overcommit OK)
384MB reserved for system	Prevents deploying apps that would starve the OS/controller
Logo SVG embedded as Go constant	Same approach as CSS/HTML — zero external file deps
Git sync via os/exec git CLI	No Go git library needed, git is in the container image
SHA-256 for content comparison	Only copy changed files, avoid unnecessary disk writes
30s debounce on manual sync	Prevents spamming the git server
Orphan = deployed but not in catalog	Safe lifecycle: remove from catalog → mark orphaned → user deletes via UI
FileBrowser as infra (not catalog)	Needed even after apps deleted (user browses HDD data); deployed by setup script
Protected HDD paths	Safety net: never delete top-level HDD dirs (media, storage, Dokumentumok, appdata)
Central scheduler (not ad-hoc goroutines)	Single place to register/monitor all periodic tasks, graceful shutdown, skip-if-running
CPU sampling via background goroutine	/proc/stat delta needs two readings — collector runs every 5s, GetInfo() reads cached value
Temperature from /host/sys (Docker mount)	Container can't read host /sys directly — mount /sys:/host/sys:ro, try /host/sys first
Restic password auto-generated	No manual setup needed — generated on first backup run, stored in named volume
DB discovery via docker inspect	No config needed — discovers postgres/mariadb containers by image name + env vars
Backup orchestrator with running flag	Prevents concurrent backups, supports both scheduled and manual trigger
modernc.org/sqlite (pure Go)	No CGO/gcc needed in Docker build stage — keeps CGO_ENABLED=0 static binary
AlertManager state-based refresh	Alerts regenerated every 5min from health report — no persistent storage needed, always reflects current state
Notification relay via hub	Controller → hub → Resend → email. Hub acts as central relay: knows customer email, handles Resend API. Controller only needs hub URL + API key
In-memory notification cooldowns	Per-event-type cooldown map (default 6h). Lost on restart = acceptable (better to re-notify than miss). No persistence needed
Health status change detection	Only notify on degradation (ok→warn, ok→fail, warn→fail). Avoids spam on flapping. First run records baseline, doesn't notify
Resend HTTP API (no SMTP)	Direct POST to api.resend.com — same pattern as website contact-mailer. Simpler than SMTP setup, good deliverability
Preferences sync on save + startup	Controller pushes prefs to hub (not pull). Startup sync handles hub DB rebuild. Local save always succeeds even if sync fails
Chart.js embedded locally	Customer hardware may not have internet — CDN not reliable for offline environments
StackDataProvider interface	backup package needs stack data but can't import stacks (circular). Interface in backup, thin adapter in main.go
Password sync to hub via report	Restic password in Docker named volume on SSD. Hub sync provides redundancy for disaster recovery
App backup via HDD mounts only	Docker volumes at /var/lib/docker/volumes/ not mounted in controller. HDD data is the important user data; DB in volumes covered by nightly dump
Restore uses running mutex	Prevents concurrent backup+restore on same restic repo. Reuses existing m.running flag
Storage paths registry in settings.json	Multi-storage support: each app's HDD_PATH from app.yaml is authoritative. Auto-discovery on startup avoids manual config. Registry enables UI management + health monitoring per path
/mnt:/mnt:rw mount in controller	Replaces per-path HDD_PATH mount. Enables multi-storage + restore writes. All customer HDD mounts are under /mnt/ by convention
Per-app HDD_PATH resolution (app.yaml > global)	App's own env HDD_PATH is Priority 1, registered storage paths as fallback. Eliminates dependency on global controller.yaml hdd_path
Mount-point detection via syscall.Stat_t.Dev	Compares device ID of path vs parent dir — reliable check that path is on separate filesystem. Prevents data writes to SSD
Health severity: mount-point = warning	Non-mount-point is informational, not a service failure. FAIL reserved for genuinely broken things. Avoids false alarms on demo/test environments
FS info via findmnt + sysfs	findmnt -n -o SOURCE,FSTYPE --target <path> for filesystem type/device. /sys/block/<dev>/device/model for disk model. Best-effort, returns nil on failure
Query param flash messages	Stateless, no session store needed. Consistent with backup page pattern. ?storage_msg=success&storage_detail=...
StorageLabels map on stacks page	Separate map passed to template (not modifying Stack struct). Built from deployed apps' HDD_PATH → registered path label lookup
Metrics downsampling via SQL	Bucket-based AVG in GROUP BY keeps Chart.js responsive with up to 30 days of data
60s metrics collection interval	Good balance of resolution vs. storage — ~44K rows/month for system metrics
/etc/os-release mounted read-only	Container can't read host OS info directly — mount to /host/etc/os-release:ro

Key file locations on demo-felhom

/opt/docker/felhom-controller/         # Controller compose + config
  ├── controller.yaml                  # Customer config (domain, auth, paths)
  ├── docker-compose.yml               # Controller's own compose
  └── data/                             # Controller persistent data (named volume)

/opt/docker/stacks/                    # All app stacks
  ├── traefik/                         # Reverse proxy (protected)
  ├── cloudflared/                     # Tunnel (protected)
  ├── paperless-ngx/                   # First deployed app ✅
  │   ├── docker-compose.yml
  │   ├── .felhom.yml                  # App metadata
  │   └── app.yaml                     # Deploy config (env vars, locked fields)
  └── whoami/                          # Test stack (not deployed)

/mnt/hdd_placeholder/storage/          # HDD storage for apps
  └── paperless/
      ├── consume/                     # Drop files here for OCR
      ├── media/                       # Processed documents
      └── export/                      # Backup exports

Related repositories and their state

Repository	Status	Notes
felhom-controller	Active	This repo. Controller code + deploy scripts
app-catalog-felhom.eu	Active	10 app templates, all with .felhom.yml metadata + memory limits
felhom.eu	Active	Website + hub/ subfolder (felhom-hub service) + k8s manifests
homelab-manifests	Stable	k3s cluster running (dooplex.hu services)
misc-scripts	Utility	collect-repo.sh, backup helpers

Gotchas & lessons learned

• docker compose restart ≠ docker compose up -d — restart doesn't pick up new images
• Go maps have random iteration order — always sort slices before displaying
• Docker .State="running" doesn't mean healthy — check .Status for "(health: starting)" / "(unhealthy)"
• Paperless-ngx needs PAPERLESS_OCR_LANGUAGES (plural) to install language packs, PAPERLESS_OCR_LANGUAGE (singular) to select
• In-memory Deployed flag must be set BEFORE docker compose up -d (not after) — compose can take 30-60s for image pulls, during which the UI would show a stale "Telepítés" button
• Cloudflare Tunnel handles *.demo-felhom.eu → Traefik handles Host()-based routing to containers
• BIOS "AC Power Recovery" must be enabled on N100 for auto-restart after power outage
• docker compose up -d returns exit 0 even when containers immediately crash-loop — need post-start status check to detect this
• When logging env vars for debugging, only log keys (not values) to avoid leaking secrets in log files
• Mealie image (ghcr.io/mealie-recipes/mealie) doesn't include wget/curl — use Python TCP socket check for healthcheck
• Mealie DB migrations on first start take ~40s (alembic) — use start_period: 60s to avoid premature unhealthy status
• Alpine-based images (filebrowser, vaultwarden) have wget via BusyBox — healthchecks with wget --spider work fine
• Deploy sed command to update image version must target only the image: line — naive sed 's|name:OLD|name:NEW|' also matches the service name line (e.g., felhom-controller: → felhom-controller:0.2.12), breaking YAML. Use sudo sed -i 's|image:.*felhom-controller:[^ ]*|image: ...felhom-controller:NEW|' or similar scoped pattern
• Hungarian quotation marks „" in YAML: „ (U+201E) is safe inside YAML double-quoted strings, but the closing " must NOT be ASCII " (0x22) — it terminates the YAML string. Use \" escape or Unicode " (U+201D). This caused a silent parse failure for the entire .felhom.yml file
• Never silently swallow parse errors — always log them. Silent failures make debugging impossible (took a dedicated debug session to find a simple quoting issue)