felhom-agent/REPORT.md

REPORT — v0.40.0: SSD user-data volume (/mnt/sys_drive, mp1) — the third CT-volume split

Task: add a third CT volume (mp1 @ /mnt/sys_drive, backup=1) baked into the golden + grown at provision (-sysdata-grow), so the controller's system_data_path becomes a separate, PBS-covered mount and the "not a separate drive" warning clears with zero controller change. Option A (static CT volume, NOT the enrolled-drive intent machinery).

1. Baselines

• felhom-agent main @ b908b9a (= spec's b908b9a8e), version v0.39.0 → v0.40.0.
• Currently-deployed in-guest controller: felhom-controller:0.77.0 (spec guessed 0.76.0; actual is 0.77.0 — this is the CONTROLLER_IMAGE the golden must bake).
• felhom-controller untouched (it already expects /mnt/sys_drive + warns via system.IsMountPoint).

2. Files modified (felhom-agent only)

• configs/build-golden.sh — --mp1 ...,mp=/mnt/sys_drive,backup=1 (env GOLDEN_SYSDATA_GB=8); findmnt /mnt/sys_drive separate-mount guard; vzdump-inclusion guard now aborts if mp0 OR mp1 excluded (the B3 trap), with the mirrored WARN-if-unconfirmed.
• internal/reconcile/bringup.go — const DefaultSysDataMount = "mp1"; BringUpSpec.{SysDataGrowGB, SysDataMount}; new "4c" online grow-only ResizeLXC block mirroring the "4b" Docker-data grow.
• cmd/felhom-agent/main.go — -sysdata-grow / -sysdata-mount flags; bringUpSizing.{SysDataGrowGB, SysDataMount}; wired at all three sites (the two --selftest=bring-up dispatch/spec paths + the --selftest=provision spec); version 0.40.0; --selftest=provision help text updated.
• internal/reconcile/bringup_test.go — TestRunBringUp_StorageSplit_SysDataGrow + …_SysDataGrowZeroNoResize.
• CHANGELOG.md (v0.40.0 entry, newest on top), RUNBOOK-provisioning-storage.md (three-volume layout), this REPORT.md.

3. Commits pushed to main

• 459dad9 — v0.40.0: third CT volume + -sysdata-grow (code + tests + CHANGELOG + RUNBOOK).
• (REPORT.md commit follows.)

4. Tests + red-proofs

• go build ./... && go vet ./... && go test ./... — green (reconcile + cmd + all packages).
  • Pre-existing unrelated flake: internal/escrow/TestGenerateRecoveryCode_EntropyAndFormat (random 10/11-word count) — passes on re-run; not touched by this change.
• TestRunBringUp_StorageSplit_SysDataGrow: PASS — fake api records ResizeLXC(8051,"mp1","+42G") alongside mp0 +240G.
• TestRunBringUp_StorageSplit_SysDataGrowZeroNoResize: PASS — SysDataGrowGB=0 ⇒ no mp1 resize.
• Red-proof #1 (Go "4c" block): removed the "4c" grow block → …_SysDataGrow FAILED (got [{mp0 +240G}], no mp1 resize) → reverted → green. ✅
• reconcile test count: 127 RUN entries (subtests included), all pass.
• Red-proof #2 (shell vzdump guard): done live during the golden rebuild — flipped mp1 to backup=0, vzdump excluded it, the guard fired FATAL and aborted, then reverted. Detail in §6. ✅

5. Live deploy DONE — agent binary (§13-A)

• Built v0.40.0 on the build server (192.168.0.180, go1.26.0, -ldflags -X main.version=0.40.0).
• Deployed to felhom-pve: backed up prior binary to /usr/local/bin/felhom-agent.bak-0.39.0, installed, systemctl restart felhom-agent.
• Verified: felhom-agent --version → 0.40.0; service active; clean restart (local-api listening on 192.168.0.162:8443; only WARN is the expected intent-gated skip of the lingering ejected ce9d… drive on 9201 — documented prior behavior, not an error).

6. Live acceptance — DONE on felhom-pve / guest 9201 (§13-B → §13-E) ✅

Secrets were provided out-of-band (retrieval passphrase from the operator; the gitea registry creds reused from the build server's cached docker login) and staged root-only in /root/.golden-secrets.env. The operator authorized §13 B–E.

§13-B golden rebuild (root@felhom-pve, build guest 9100, controller 0.77.0):

• GOLDEN_SYSDATA_GB=8 ./build-golden.sh 9100 <debian-13-template> local-lvm local vmbr0 gitea.dooplex.hu/admin/felhom-controller:0.77.0.
• Live guards: /var/lib/docker is a separate mount AND /mnt/sys_drive is a separate mount: /dev/mapper/pve-vm--9100--disk--2 ext4. vzdump log: including mount point mp0 AND including mount point mp1. New golden volid: local:backup/vzdump-lxc-9100-2026_06_23-16_38_49.tar.zst (580 MB).
• Red-proof #2 (shell vzdump guard): flipped 9100's mp1 to backup=0, re-ran vzdump → log showed excluding volume mount point mp1 ('/mnt/sys_drive') from backup (disabled) → the exact guard snippet fired the FATAL and aborted. Reverted (red-proof archive deleted, build guest 9100 destroyed). ✅

§13-C CHECKPOINT — pct stop 9201; pct destroy 9201 --purge (operator-authorized clean reinstall; 24 app containers + the controller storage registry wiped; the two physical enrolled drives felhom-usb/felhom-flash survive on the host).

§13-D re-provision — felhom-agent v0.40.0 --selftest=provision -archive <new-golden> -vmid 9201 -customer-id demo-felhom -hub-password <out-of-band> -rootfs-grow 0 -datavol-grow 184 -sysdata-grow 42 → front half up boot+running in 1m11s (fresh MAC BC:24:11:4C:B8:42), back half minted the per-guest token + populated bootstrap mp9. Then pct reboot 9201 (activates the baked controller-bootstrap).

§13-E acceptance gates — ALL PASS:

• pct config 9201: mp0 …/var/lib/docker,backup=1,size=200G; mp1 …/mnt/sys_drive,backup=1, size=50G; mp8 /mnt/felhom-drives; mp9 …bootstrap,ro=1. (16+184=200; 8+42=50.) ✅
• GATE 1 (guest): pct exec 9201 -- findmnt /mnt/sys_drive → /dev/mapper/pve-vm--9201--disk--2 ext4 — a SEPARATE device (rootfs is …--disk--0). df: 50G (47G avail). ✅
• GATE 2 (container propagation): docker exec felhom-controller findmnt /mnt/sys_drive → same …--disk--2 — the mp propagated into the controller through -v /mnt:/mnt:rslave. ✅
• Warning cleared: controller selftest now logs [PASS] System data path: /mnt/sys_drive (previously the system.IsMountPoint WARN). After an agent daemon restart (the documented stale-token workaround) the controller's local-api channel verified and enumerated mount mp1 → /mnt/sys_drive (storage=local-lvm, … backup=true) — recognized + PBS-covered. ✅
• System healthy: [PASS] Hub connectivity hub.felhom.eu HTTP 200, [PASS] Git catalog: 52 app definitions, base infra (controller/traefik/cloudflared/filebrowser) all Up. ✅

Verification method: the warning-cleared check was done against the controller's server-side selftest (the exact system.IsMountPoint computation that drives the monitor page) and the local-api mount enumeration — not the browser-rendered page. The only uncovered residual is the purely client-side render of that monitor panel.

7. NOT yet live-validated / residual

• External-drive re-enroll after the clean reinstall: the destroy wiped the controller's felhom-controller-data volume (its storage registry), so [WARN] Storage paths: no storage paths registered — felhom-usb (uuid da9e7089) / felhom-flash (uuid 81a26531) need re-enrollment via the UI. Expected for a clean reinstall; out of scope for this task's acceptance (solely /mnt/sys_drive becoming a separate, warning-free, PBS-covered mount — all proven above).
• Client-side monitor-panel render of the cleared warning + the ~50G user-data bar — not browser- verified (server-side mechanism proven instead; see §6).
• The transient [WARN] local-api … HTTP 401 — channel not verified seen on first boot is the known provision-flow gap (agent in-memory token map stale until daemon restart); cleared here by restarting the agent. Not introduced by this task.

8. Observations (noted, NOT acted on)

• build-golden.sh CONTROLLER_IMAGE default is stale (:0.43.0); the live controller is 0.77.0. The golden rebuild passes the tag explicitly, so the stale default is cosmetic — left as-is per the minimal-change rule.
• felhom-agent/CLAUDE.md self-reports "Current: v0.31.0" — stale; authoritative version is the main.go var + CHANGELOG top (now 0.40.0). Left as-is (out of scope).
• Live 9201 uses mp3 for the shared drives parent (/mnt/felhom-drives), not the mp8 the spec cited — freeMountSlot picks the lowest free slot, so the exact number varies. The baked mp1 does not collide regardless (the guest is rebuilt from the new golden). No action needed.