felhom-agent/REPORT.md

# REPORT — v0.40.0: SSD user-data volume (`/mnt/sys_drive`, mp1) — the third CT-volume split

**Task:** add a third CT volume (`mp1` @ `/mnt/sys_drive`, `backup=1`) baked into the golden + grown
at provision (`-sysdata-grow`), so the controller's `system_data_path` becomes a separate, PBS-covered
mount and the "not a separate drive" warning clears with **zero controller change**. Option A (static
CT volume, NOT the enrolled-drive intent machinery).

## 1. Baselines
- felhom-agent `main` @ `b908b9a` (= spec's `b908b9a8e`), version **v0.39.0** → **v0.40.0**.
- Currently-deployed in-guest controller: **`felhom-controller:0.77.0`** (spec guessed 0.76.0; actual
  is 0.77.0 — this is the `CONTROLLER_IMAGE` the golden must bake).
- felhom-controller **untouched** (it already expects `/mnt/sys_drive` + warns via
  `system.IsMountPoint`).

## 2. Files modified (felhom-agent only)
- `configs/build-golden.sh` — `--mp1 ...,mp=/mnt/sys_drive,backup=1` (env `GOLDEN_SYSDATA_GB=8`);
  `findmnt /mnt/sys_drive` separate-mount guard; vzdump-inclusion guard now aborts if **mp0 OR mp1**
  excluded (the B3 trap), with the mirrored WARN-if-unconfirmed.
- `internal/reconcile/bringup.go` — `const DefaultSysDataMount = "mp1"`; `BringUpSpec.{SysDataGrowGB,
  SysDataMount}`; new **"4c"** online grow-only `ResizeLXC` block mirroring the "4b" Docker-data grow.
- `cmd/felhom-agent/main.go` — `-sysdata-grow` / `-sysdata-mount` flags; `bringUpSizing.{SysDataGrowGB,
  SysDataMount}`; wired at all three sites (the two `--selftest=bring-up` dispatch/spec paths + the
  `--selftest=provision` spec); version `0.40.0`; `--selftest=provision` help text updated.
- `internal/reconcile/bringup_test.go` — `TestRunBringUp_StorageSplit_SysDataGrow` +
  `…_SysDataGrowZeroNoResize`.
- `CHANGELOG.md` (v0.40.0 entry, newest on top), `RUNBOOK-provisioning-storage.md` (three-volume
  layout), this `REPORT.md`.

## 3. Commits pushed to `main`
- `459dad9` — v0.40.0: third CT volume + `-sysdata-grow` (code + tests + CHANGELOG + RUNBOOK).
- (REPORT.md commit follows.)

## 4. Tests + red-proofs
- `go build ./... && go vet ./... && go test ./...` — **green** (reconcile + cmd + all packages).
  - Pre-existing unrelated flake: `internal/escrow/TestGenerateRecoveryCode_EntropyAndFormat`
    (random 10/11-word count) — passes on re-run; **not touched by this change**.
- `TestRunBringUp_StorageSplit_SysDataGrow`: PASS — fake api records `ResizeLXC(8051,"mp1","+42G")`
  alongside `mp0 +240G`.
- `TestRunBringUp_StorageSplit_SysDataGrowZeroNoResize`: PASS — `SysDataGrowGB=0` ⇒ **no** mp1 resize.
- **Red-proof #1 (Go "4c" block):** removed the "4c" grow block → `…_SysDataGrow` FAILED
  (`got [{mp0 +240G}]`, no mp1 resize) → reverted → green. ✅
- reconcile test count: 127 RUN entries (subtests included), all pass.
- **Red-proof #2 (shell vzdump guard):** done live during the golden rebuild — flipped `mp1` to
  `backup=0`, vzdump excluded it, the guard fired FATAL and aborted, then reverted. Detail in §6. ✅

## 5. Live deploy DONE — agent binary (§13-A)
- Built v0.40.0 on the build server (192.168.0.180, go1.26.0, `-ldflags -X main.version=0.40.0`).
- Deployed to felhom-pve: backed up prior binary to `/usr/local/bin/felhom-agent.bak-0.39.0`,
  installed, `systemctl restart felhom-agent`.
- **Verified:** `felhom-agent --version` → `0.40.0`; service `active`; clean restart (local-api
  listening on 192.168.0.162:8443; only WARN is the expected intent-gated skip of the lingering ejected
  `ce9d…` drive on 9201 — documented prior behavior, not an error).

## 6. Live acceptance — DONE on felhom-pve / guest 9201 (§13-B → §13-E) ✅
Secrets were provided out-of-band (retrieval passphrase from the operator; the gitea registry creds
reused from the build server's cached docker login) and staged root-only in
`/root/.golden-secrets.env`. The operator authorized §13 B–E.

**§13-B golden rebuild** (root@felhom-pve, build guest 9100, controller `0.77.0`):
- `GOLDEN_SYSDATA_GB=8 ./build-golden.sh 9100 <debian-13-template> local-lvm local vmbr0
  gitea.dooplex.hu/admin/felhom-controller:0.77.0`.
- Live guards: `/var/lib/docker is a separate mount` AND `/mnt/sys_drive is a separate mount:
  /dev/mapper/pve-vm--9100--disk--2 ext4`. vzdump log: **`including mount point mp0` AND `including
  mount point mp1`**. New golden volid: **`local:backup/vzdump-lxc-9100-2026_06_23-16_38_49.tar.zst`**
  (580 MB).
- **Red-proof #2 (shell vzdump guard):** flipped 9100's `mp1` to `backup=0`, re-ran vzdump → log showed
  `excluding volume mount point mp1 ('/mnt/sys_drive') from backup (disabled)` → the exact guard snippet
  fired the FATAL and aborted. Reverted (red-proof archive deleted, build guest 9100 destroyed). ✅

**§13-C CHECKPOINT** — `pct stop 9201; pct destroy 9201 --purge` (operator-authorized clean reinstall;
24 app containers + the controller storage registry wiped; the two physical enrolled drives
`felhom-usb`/`felhom-flash` survive on the host).

**§13-D re-provision** — `felhom-agent v0.40.0 --selftest=provision -archive <new-golden> -vmid 9201
-customer-id demo-felhom -hub-password <out-of-band> -rootfs-grow 0 -datavol-grow 184 -sysdata-grow 42`
→ front half up boot+running in 1m11s (fresh MAC `BC:24:11:4C:B8:42`), back half minted the per-guest
token + populated bootstrap mp9. Then `pct reboot 9201` (activates the baked controller-bootstrap).

**§13-E acceptance gates — ALL PASS:**
- `pct config 9201`: `mp0 …/var/lib/docker,backup=1,size=200G`; **`mp1 …/mnt/sys_drive,backup=1,
  size=50G`**; `mp8 /mnt/felhom-drives`; `mp9 …bootstrap,ro=1`. (16+184=200; 8+42=50.) ✅
- **GATE 1 (guest):** `pct exec 9201 -- findmnt /mnt/sys_drive` → `/dev/mapper/pve-vm--9201--disk--2
  ext4` — a SEPARATE device (rootfs is `…--disk--0`). `df`: **50G** (47G avail). ✅
- **GATE 2 (container propagation):** `docker exec felhom-controller findmnt /mnt/sys_drive` → same
  `…--disk--2` — the mp propagated into the controller through `-v /mnt:/mnt:rslave`. ✅
- **Warning cleared:** controller selftest now logs **`[PASS] System data path: /mnt/sys_drive`**
  (previously the `system.IsMountPoint` WARN). After an agent daemon restart (the documented
  stale-token workaround) the controller's local-api channel verified and enumerated
  **`mount mp1 → /mnt/sys_drive (storage=local-lvm, … backup=true)`** — recognized + PBS-covered. ✅
- System healthy: `[PASS] Hub connectivity hub.felhom.eu HTTP 200`, `[PASS] Git catalog: 52 app
  definitions`, base infra (controller/traefik/cloudflared/filebrowser) all Up. ✅

**Verification method:** the warning-cleared check was done against the controller's **server-side
selftest** (the exact `system.IsMountPoint` computation that drives the monitor page) and the local-api
mount enumeration — not the browser-rendered page. The only uncovered residual is the purely
client-side render of that monitor panel.

## 7. NOT yet live-validated / residual
- **External-drive re-enroll** after the clean reinstall: the destroy wiped the controller's
  `felhom-controller-data` volume (its storage registry), so `[WARN] Storage paths: no storage paths
  registered` — `felhom-usb` (uuid `da9e7089`) / `felhom-flash` (uuid `81a26531`) need re-enrollment via
  the UI. Expected for a clean reinstall; **out of scope** for this task's acceptance (solely
  `/mnt/sys_drive` becoming a separate, warning-free, PBS-covered mount — all proven above).
- **Client-side monitor-panel render** of the cleared warning + the ~50G user-data bar — not browser-
  verified (server-side mechanism proven instead; see §6).
- The transient `[WARN] local-api … HTTP 401 — channel not verified` seen on first boot is the known
  provision-flow gap (agent in-memory token map stale until daemon restart); cleared here by restarting
  the agent. Not introduced by this task.

## 8. Observations (noted, NOT acted on)
- `build-golden.sh` `CONTROLLER_IMAGE` default is stale (`:0.43.0`); the live controller is `0.77.0`.
  The golden rebuild passes the tag explicitly, so the stale default is cosmetic — left as-is per the
  minimal-change rule.
- `felhom-agent/CLAUDE.md` self-reports "Current: v0.31.0" — stale; authoritative version is the
  `main.go` var + CHANGELOG top (now `0.40.0`). Left as-is (out of scope).
- Live 9201 uses `mp3` for the shared drives parent (`/mnt/felhom-drives`), not the `mp8` the spec
  cited — `freeMountSlot` picks the lowest free slot, so the exact number varies. The baked `mp1` does
  not collide regardless (the guest is rebuilt from the new golden). No action needed.