felhom-agent/REPORT.md

REPORT — v0.40.0: SSD user-data volume (/mnt/sys_drive, mp1) — the third CT-volume split

Task: add a third CT volume (mp1 @ /mnt/sys_drive, backup=1) baked into the golden + grown at provision (-sysdata-grow), so the controller's system_data_path becomes a separate, PBS-covered mount and the "not a separate drive" warning clears with zero controller change. Option A (static CT volume, NOT the enrolled-drive intent machinery).

1. Baselines

• felhom-agent main @ b908b9a (= spec's b908b9a8e), version v0.39.0 → v0.40.0.
• Currently-deployed in-guest controller: felhom-controller:0.77.0 (spec guessed 0.76.0; actual is 0.77.0 — this is the CONTROLLER_IMAGE the golden must bake).
• felhom-controller untouched (it already expects /mnt/sys_drive + warns via system.IsMountPoint).

2. Files modified (felhom-agent only)

• configs/build-golden.sh — --mp1 ...,mp=/mnt/sys_drive,backup=1 (env GOLDEN_SYSDATA_GB=8); findmnt /mnt/sys_drive separate-mount guard; vzdump-inclusion guard now aborts if mp0 OR mp1 excluded (the B3 trap), with the mirrored WARN-if-unconfirmed.
• internal/reconcile/bringup.go — const DefaultSysDataMount = "mp1"; BringUpSpec.{SysDataGrowGB, SysDataMount}; new "4c" online grow-only ResizeLXC block mirroring the "4b" Docker-data grow.
• cmd/felhom-agent/main.go — -sysdata-grow / -sysdata-mount flags; bringUpSizing.{SysDataGrowGB, SysDataMount}; wired at all three sites (the two --selftest=bring-up dispatch/spec paths + the --selftest=provision spec); version 0.40.0; --selftest=provision help text updated.
• internal/reconcile/bringup_test.go — TestRunBringUp_StorageSplit_SysDataGrow + …_SysDataGrowZeroNoResize.
• CHANGELOG.md (v0.40.0 entry, newest on top), RUNBOOK-provisioning-storage.md (three-volume layout), this REPORT.md.

3. Commits pushed to main

• 459dad9 — v0.40.0: third CT volume + -sysdata-grow (code + tests + CHANGELOG + RUNBOOK).
• (REPORT.md commit follows.)

4. Tests + red-proofs

• go build ./... && go vet ./... && go test ./... — green (reconcile + cmd + all packages).
  • Pre-existing unrelated flake: internal/escrow/TestGenerateRecoveryCode_EntropyAndFormat (random 10/11-word count) — passes on re-run; not touched by this change.
• TestRunBringUp_StorageSplit_SysDataGrow: PASS — fake api records ResizeLXC(8051,"mp1","+42G") alongside mp0 +240G.
• TestRunBringUp_StorageSplit_SysDataGrowZeroNoResize: PASS — SysDataGrowGB=0 ⇒ no mp1 resize.
• Red-proof #1 (Go "4c" block): removed the "4c" grow block → …_SysDataGrow FAILED (got [{mp0 +240G}], no mp1 resize) → reverted → green. ✅
• reconcile test count: 127 RUN entries (subtests included), all pass.
• Red-proof #2 (shell vzdump guard): to be run live during the golden rebuild — temporarily flip mp1 to backup=0, confirm the guard ABORTS, revert. PENDING (needs golden rebuild — §6).

5. Live deploy DONE — agent binary (§13-A)

• Built v0.40.0 on the build server (192.168.0.180, go1.26.0, -ldflags -X main.version=0.40.0).
• Deployed to felhom-pve: backed up prior binary to /usr/local/bin/felhom-agent.bak-0.39.0, installed, systemctl restart felhom-agent.
• Verified: felhom-agent --version → 0.40.0; service active; clean restart (local-api listening on 192.168.0.162:8443; only WARN is the expected intent-gated skip of the lingering ejected ce9d… drive on 9201 — documented prior behavior, not an error).

6. Live acceptance — PENDING (blocked on operator-held secrets + an irreversible CHECKPOINT)

The golden rebuild (§13-B), the pct destroy 9201 CHECKPOINT (§13-C), and the re-provision (§13-D/E) were not executed because they require inputs CC does not hold:

1. REGISTRY_USER + REGISTRY_TOKEN — for the golden's one-time controller-image pull inside the build guest. Confirmed absent on-host (no /root/.docker/config.json, no saved build env).
2. DEMO_RETRIEVAL_PASSPHRASE (the -hub-password) — for the 9201 re-provision (the controller's hub config pull).
3. Operator confirmation for the irreversible pct stop 9201; pct destroy 9201 --purge (§13-C CHECKPOINT). Operator pre-authorized a clean reinstall in principle, but the destroy is gated.

Planned live run once provided (locked sizing per RUNBOOK = 32 rootfs / 200 docker-data / 50 user-data):

# B. rebuild golden (root@felhom-pve) — bake controller 0.77.0
GOLDEN_SYSDATA_GB=8 REGISTRY_USER=… REGISTRY_TOKEN=… \
  ./build-golden.sh 9100 <TEMPLATE> local-lvm local vmbr0 \
  gitea.dooplex.hu/admin/felhom-controller:0.77.0
#    → vzdump log MUST show BOTH "including mount point mp0" AND "… mp1"; capture the new volid.
#    → Red-proof #2 here (flip mp1 backup=0 → guard aborts → revert).
# C. CHECKPOINT: pct stop 9201; pct destroy 9201 --purge
# D. re-provision: felhom-agent --selftest=provision -archive <new-golden> -vmid 9201 \
#      -customer-id demo-felhom -hub-password <SECRET> -rootfs-grow 0 -datavol-grow 184 -sysdata-grow 42
#    then: pct reboot 9201
# E. accept: pct config 9201 (mp0,mp1 backup=1,mp8,mp9) ; pct exec 9201 -- findmnt /mnt/sys_drive ;
#    pct exec 9201 -- docker exec felhom-controller findmnt /mnt/sys_drive  (← propagation gate) ;
#    controller monitor: /mnt/sys_drive warning GONE, SSD user-data ~50G.

Current 9201 baseline (pre-change, confirms the warning state): rootfs 32G, mp0 docker-data 256G (backup=1), mp3 /mnt/felhom-drives, mp9 bootstrap — no mp1 / /mnt/sys_drive volume (so /mnt/sys_drive is on the rootfs and the controller warns). Memory 12288 MiB.

7. NOT yet live-validated / residual

• The §13-B golden rebuild, the §13-D/E re-provision + acceptance gates (the three findmnts + the controller-UI/API warning-cleared check), and shell-guard Red-proof #2 — all blocked on §6.
• External-drive re-enroll after the clean reinstall: the destroy wipes the controller's felhom-controller-data volume (its storage registry), so felhom-usb/felhom-flash will need re-enrollment via the UI. Expected for a clean reinstall; out of scope for this task's acceptance (solely /mnt/sys_drive becoming a separate, warning-free, PBS-covered mount).

8. Observations (noted, NOT acted on)

• build-golden.sh CONTROLLER_IMAGE default is stale (:0.43.0); the live controller is 0.77.0. The golden rebuild passes the tag explicitly, so the stale default is cosmetic — left as-is per the minimal-change rule.
• felhom-agent/CLAUDE.md self-reports "Current: v0.31.0" — stale; authoritative version is the main.go var + CHANGELOG top (now 0.40.0). Left as-is (out of scope).
• Live 9201 uses mp3 for the shared drives parent (/mnt/felhom-drives), not the mp8 the spec cited — freeMountSlot picks the lowest free slot, so the exact number varies. The baked mp1 does not collide regardless (the guest is rebuilt from the new golden). No action needed.