@@ -1,118 +1,57 @@
# REPORT — v0.40 .0: SSD user-data volume (`/mnt/sys_drive`, mp1) — the third CT-volume split
# REPORT — v0.41 .0: provisioned customer guests auto-start after a host reboot (`onboot:1`, F3)
**Task : ** add a third CT volume ( `mp1` @ `/mnt/sys_drive` , `backup=1` ) baked into the golden + grown
at provision (`-sysdata-grow` ), so the controller's `system_data_path` becomes a separate, PBS-covered
mount and the "not a separate drive" warning clears with **zero controller change ** . Option A (static
CT volume, NOT the enrolled-drive intent machinery).
**Repo : ** `felhom-agent` · **Version: ** `v0.40.0` → * * `v0.41.0` ** · **Date: ** 2026-06-24
**Baseline: ** `main` @ `db95d51` (version var `0.40.0` ), trunk-based, no branches.
**Class: ** risky/supervised — touches the provisioning chain (back-half).
## 1. Baselines
- felhom-agent `main` @ `b908b9a` (= spec's `b908b9a8e` ), version **v0.39.0 ** → **v0.40.0 ** .
- Currently-deployed in-guest controller: * * `felhom-controller:0.77.0` ** (spec guessed 0.76.0; actual
is 0.77.0 — this is the `CONTROLLER_IMAGE` the golden must bake).
- felhom-controller **untouched ** (it already expects `/mnt/sys_drive` + warns via
`system.IsMountPoint` ).
## Finding (F3, from `TEST-REPORT-stable-path-sysdrive-restart-2026-06-23.md`)
## 2. Files modified (felhom-agent only)
- `configs/build-golden.sh` — `--mp1 ...,mp=/mnt/sys_drive,backup=1` (env `GOLDEN_SYSDATA_GB=8` );
`findmnt /mnt/sys_drive` separate-mount guard; vzdump-inclusion guard now aborts if **mp0 OR mp1 **
excluded (the B3 trap), with the mirrored WARN-if-unconfirm ed.
- `internal/reconcile/bringup.go` — `const DefaultSysDataMount = "mp1"` ; `BringUpSpec.{SysDataGrowGB,
SysDataMount}`; new **"4c"** online grow-only ` ResizeLXC` block mirroring the "4b" Docker-data grow.
- ` cmd/felhom-agent/main.go` — ` -sysdata-grow` / ` -sysdata-mount` flags; ` bringUpSizing.{SysDataGrowGB,
SysDataMount}`; wired at all three sites (the two ` --selftest=bring-up` dispatch/spec paths + the
` --selftest=provision` spec); version ` 0.40.0`; ` --selftest=provision` help text updated.
- ` internal/reconcile/bringup_test.go` — ` TestRunBringUp_StorageSplit_SysDataGrow` +
` …_SysDataGrowZeroNoResize`.
- ` CHANGELOG.md` (v0.40.0 entry, newest on top), ` RUNBOOK-provisioning-storage.md` (three-volume
layout), this ` REPORT.md`.
The golden bakes `--onboot 0` (`configs/build-golden.sh:63` , template safety) and the provision
back-half never overrode it → **every provisioned customer guest was `onboot:0` ** . After a host
reboot/power-cut the customer's whole home-server (controller + all apps) stays **down until a manual
`pct start` **. Confirmed live in the campaign (Phase 4.1: host rebooted → `pct status 9201` = stopp ed) .
## 3. Commits pushed to ` main `
- ` 459dad9` — v0.40.0: third CT volume + ` -sysdata-grow` (code + tests + CHANGELOG + RUNBOOK).
- (REPORT.md commit follows.)
## The fix
## 4. Tests + red-proofs
- ` go build ./... && go vet ./... && go test ./...` — **green** (reconcile + cmd + all packages).
- Pre-existing unrelated flake: ` internal/escrow/TestGenerateRecoveryCode_EntropyAndFormat`
(random 10/11-word count) — passes on re-run; **not touched by this change**.
- ` TestRunBringUp_StorageSplit_SysDataGrow`: PASS — fake api records ` ResizeLXC(8051,"mp1","+42G")`
alongside ` mp0 +240G`.
- ` TestRunBringUp_StorageSplit_SysDataGrowZeroNoResize`: PASS — ` SysDataGrowGB=0` ⇒ **no** mp1 resize.
- **Red-proof #1 (Go "4c" block):** removed the "4c" grow block → ` …_SysDataGrow` FAILED
(` got [{mp0 +240G}]`, no mp1 resize) → reverted → green. ✅
- reconcile test count: 127 RUN entries (subtests included), all pass.
- **Red-proof #2 (shell vzdump guard):** done live during the golden rebuild — flipped ` mp1` to
` backup=0`, vzdump excluded it, the guard fired FATAL and aborted, then reverted. Detail in §6. ✅
`internal/provision/backhalf.go` , `BackHalf.Provision` : added a fatal step right after the config-mount
attach (and before the guest-hook install):
## 5. Live deploy DONE — agent binary (§13-A)
- Built v0.40.0 on the build server (192.168.0.180, go1.26.0, ` -ldflags -X main.version=0.40.0`).
- Deployed to felhom-pve: backed up prior binary to ` /usr/local/bin/felhom-agent.bak-0.39.0`,
installed, ` systemctl restart felhom-agent`.
- **Verified:** ` felhom-agent --version` → ` 0.40.0`; service ` active`; clean restart (local-api
listening on 192.168.0.162:8443; only WARN is the expected intent-gated skip of the lingering ejected
` ce9d…` drive on 9201 — documented prior behavior, not an error).
``` go
if err := b . run ( ctx , "pct" , "set" , strconv . Itoa ( in . VMID ) , "-onboot" , "1" ) ; err != nil {
return Result { } , fmt . Errorf ( "provision: set onboot: %w" , err )
}
```
## 6. Liv e acceptance — DONE on felhom-pve / guest 9201 (§13-B → §13-E) ✅
Secrets were provided out-of-band (retrieval passphrase from the operator; the gitea registry creds
reused from the build server's cached docker login) and staged root-only in
` /r oot/.golden-secrets.env `. The operator authorized §13 B–
- **Locus = thb ack-half, not the golden.** `build-golden.sh` `--onboot 0` is **unchanged ** — a template
must not auto-start; `onboot` is a per-guest property the back-half is the right place to set.
- **Fatal**, like the config-mount attach (a guest that won't auto-recover is a provisioning defect).
- **No `startup `b oot-order/delay** (operator decision): the v0.75 mountpoint-gate already covers the
drive-bind race at boot (Phase 4.4). The agent never auto-`pct start` s anything — this is config only.
**§13-B golden rebuild** (root@felhom-pve, build guest 9100, controller ` 0.77.0`):
- ` GOLDEN_SYSDATA_GB=8 ./build-golden.sh 9100 <debian-13-template> local-lvm local vmbr0
gitea.dooplex.hu/admin/felhom-controller:0.77.0`.
- Live guards: ` /var/lib/docker is a separate mount` AND ` /mnt/sys_drive is a separate mount:
/dev/mapper/pve-vm--9100--disk--2 ext4`. vzdump log: **` including mount point mp0` AND ` including
mount point mp1`**. New golden volid: **` local:backup/vzdump-lxc-9100-2026_06_23-16_38_49.tar.zst`**
(580 MB).
- **Red-proof #2 (shell vzdump guard):** flipped 9100's ` mp1` to ` backup=0`, re-ran vzdump → log showed
` excluding volume mount point mp1 ('/mnt/sys_drive') from backup (disabled)` → the exact guard snippet
fired the FATAL and aborted. Reverted (red-proof archive deleted, build guest 9100 destroyed). ✅
## Test + red-proof
**§13-C CHECKPOINT** — ` pct stop 9201; pct destroy 9201 --purge` (operator-authorized clean reinstall;
24 app containers + the controller storage registry wiped; the two physical enrolled drive s
` felhom-usb`/` felhom-flash` survive on the host).
`internal/provision/backhalf_test.go` :
- Added `TestProvision_SetsOnbootOne` — asserts the exact `pct set 8200 -onboot 1` invocation wa s
recorded by the fake runner (via a new `recRunner.hasExact` helper, since several `pct` calls are now
recorded and `find` only returns the first).
- **Red-proof (run/confirm/revert):** removed the `b.run(... -onboot 1)` call → `TestProvision_SetsOnbootOne`
**FAILED ** (`expected 'pct set 8200 -onboot 1' to be issued` ) → restored the call → **green ** .
- Existing back-half tests stay green (chown-failure-stops still holds: onboot is after the config-mount
attach, so a chown failure never reaches it).
- Green gate: `go build ./...` ✓ · `go vet ./...` ✓ · `go test ./...` ✓ (all packages ok).
**§13-D re-provision** — ` felhom-agent v0.40.0 --selftest=provision -archive <new-golden> -vmid 9201
-customer-id demo-felhom -hub-password <out-of-band> -rootfs-grow 0 -datavol-grow 184 -sysdata-grow 42`
→ front half up boot+running in 1m11s (fresh MAC ` BC:24:11:4C:B8:42`), back half minted the per-guest
token + populated bootstrap mp9. Then ` pct reboot 9201` (activates the baked controller-bootstrap).
## Deploy + remediate + verify
**§13-E acceptance gates — ALL PASS:**
- ` pct config 9201 `: ` mp0 …/var/lib/docker, backup=1,size=200G `; **` mp1 …/mnt/sys_drive,backup=1,
size=50G`**; ` mp8 /mnt/felhom-drives`; ` mp9 …bootstrap,ro=1`. (16+184=200; 8+42=50.) ✅
- **GATE 1 (guest):** ` pct exec 9201 -- findmnt /mnt/sys_drive` → ` /dev/mapper/pve-vm--9201--disk--2
ext4` — a SEPARATE device (rootfs is ` …--disk--0`). ` df`: **50G** (47G avail). ✅
- **GATE 2 (container propagation):** ` docker exec felhom-controller findmnt /mnt/sys_drive` → same
` …--disk--2` — the mp propagated into the controller through ` -v /mnt:/mnt:rslave`. ✅
- **Warning cleared:** controller selftest now logs **` [PASS] System data path: /mnt/sys_drive`**
(previously the ` system.IsMountPoint` WARN). After an agent daemon restart (the documented
stale-token workaround) the controller's local-api channel verified and enumerated
**` mount mp1 → /mnt/sys_drive (storage=local-lvm, … backup=true)`** — recognized + PBS-covered. ✅
- System healthy: ` [PASS] Hub connectivity hub.felhom.eu HTTP 200`, ` [PASS] Git catalog: 52 app
definitions`, base infra (controller/traefik/cloudflared/filebrowser) all Up. ✅
**A. Build + deploy agent v0.41.0 ** on the build server ( `-ldflags -X main.version=0.41.0` ), scp to
felhom-pve, back up the prior binary, install, restart the service. — _ filled below _
**Verification method:** the warning-cleared check was done against the controller's **server-side
selftest** (the exac t `system.IsMountPoint ` computation that drives the monitor page) and the local-api
mount enumeration — not the browser-rendered page. The only uncovered residual is the purely
client-side render of that monitor panel.
**B. Demo remediation (9201 was provisioned pre-fix → `onboot:0`): **
`pct set 9201 -onboo t 1 ` (non-destructive, config-only) → `pct config 9201 | grep onboot` → _ filled below _
## 7. NOT yet live-validated / residual
- **External-drive re-enroll** after the clean reinstall: the destroy wiped the controller's
` felhom-controller-data ` volume (its storage registry), so ` [WARN] Storage paths: no storage pa ths
registered ` — ` felhom-usb ` (uuid ` da9e7089 `) / ` felhom-flash ` (uuid ` 81a26531 `) need re-enrollm ent via
the UI. Expected for a clean reinstall; **out of scope** for this task's acceptance (solely
` /mnt/sys_drive` becoming a separate, warning-free, PBS-covered mount — all proven above).
- **Client-side monitor-panel render** of the cleared warning + the ~50G user-data bar — not browser-
verified (server-side mechanism proven instead; see §6).
- The transient ` [WARN] local-api … HTTP 401 — channel not verified` seen on first boot is the known
provision-flow gap (agent in-memory token map stale until daemon restart); cleared here by restarting
the agent. Not introduced by this task.
**C. Back-half fix proper — NOT yet live-validated. ** The unit test + `pct config 9201 → onboot:1` cover
the immediate state, but the back-half * path itself * (a fresh provision emitting `onboot:1` with no
manual set) **awaits the deferred supervised capstone re-provision ** (destroy + re-provision 9201 wi th
the new agent) — operator-gated, needs the gold en volid + retrieval passphrase.
## 8. Observations (noted, NOT acted on)
- ` build-golden.sh ` ` CONTROLLER_IMAGE ` default is stale (` :0.43.0 `); the live controller is ` 0.77.0 ` .
The golden rebuild passes the tag explicitly, so the stale default is cosmetic — left as-is per the
minimal-change rule.
- ` felhom-agent/CLAUDE.md` self-reports "Current: v0.31.0" — stale; authoritative version is the
` main.go` var + CHANGELOG top (now ` 0.40.0`). Left as-is (out of scope).
- Live 9201 uses ` mp3` for the shared drives parent (` /mnt/felhom-drives`), not the ` mp8` the spec
cited — ` freeMountSlot` picks the lowest free slot, so the exact number varies. The baked ` mp1` does
not collide regardless (the guest is rebuilt from the new golden). No action needed.
**Optional (operator-gated): ** reboot felhom-pve → confirm 9201 auto-starts (`pct status 9201` = running
without a manual `pct start` ) — the exact Phase-4.1 failure, now expected to pass .
