REPORT: v0.40.0 third CT volume — code shipped + agent v0.40.0 deployed live; golden/re-provision pending operator secrets+CHECKPOINT

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PsnU2ASocYrvzqE82YDYW

REPORT.md

@@ -1,111 +1,100 @@
-# REPORT — DR-recipe completion: live PBS coord + drop the two unfillable drive fields
+# REPORT — v0.40.0: SSD user-data volume (`/mnt/sys_drive`, mp1) — the third CT-volume split
 
-**Repo:** `felhom-agent` · **Version:** v0.38.0 → **v0.39.0** · **Date:** 2026-06-16
-**Companion repo:** `felhom.eu/hub` v0.13.0 → **v0.13.1** (test-data + regression test only)
-**Controller:** untouched.
+**Task:** add a third CT volume (`mp1` @ `/mnt/sys_drive`, `backup=1`) baked into the golden + grown
+at provision (`-sysdata-grow`), so the controller's `system_data_path` becomes a separate, PBS-covered
+mount and the "not a separate drive" warning clears with **zero controller change**. Option A (static
+CT volume, NOT the enrolled-drive intent machinery).
 
-## 1. Confirmed baselines + commit hashes
+## 1. Baselines
+- felhom-agent `main` @ `b908b9a` (= spec's `b908b9a8e`), version **v0.39.0** → **v0.40.0**.
+- Currently-deployed in-guest controller: **`felhom-controller:0.77.0`** (spec guessed 0.76.0; actual
+  is 0.77.0 — this is the `CONTROLLER_IMAGE` the golden must bake).
+- felhom-controller **untouched** (it already expects `/mnt/sys_drive` + warns via
+  `system.IsMountPoint`).
 
-| Repo | Baseline | Target | Commit |
-|---|---|---|---|
-| felhom-agent | v0.38.0 (`8abc1b8`) | **v0.39.0** | **`fbe1130`** |
-| felhom-hub (`felhom.eu`) | v0.13.0 (`5f5e3c5`) | **v0.13.1** | **`149a3b0`** |
+## 2. Files modified (felhom-agent only)
+- `configs/build-golden.sh` — `--mp1 ...,mp=/mnt/sys_drive,backup=1` (env `GOLDEN_SYSDATA_GB=8`);
+  `findmnt /mnt/sys_drive` separate-mount guard; vzdump-inclusion guard now aborts if **mp0 OR mp1**
+  excluded (the B3 trap), with the mirrored WARN-if-unconfirmed.
+- `internal/reconcile/bringup.go` — `const DefaultSysDataMount = "mp1"`; `BringUpSpec.{SysDataGrowGB,
+  SysDataMount}`; new **"4c"** online grow-only `ResizeLXC` block mirroring the "4b" Docker-data grow.
+- `cmd/felhom-agent/main.go` — `-sysdata-grow` / `-sysdata-mount` flags; `bringUpSizing.{SysDataGrowGB,
+  SysDataMount}`; wired at all three sites (the two `--selftest=bring-up` dispatch/spec paths + the
+  `--selftest=provision` spec); version `0.40.0`; `--selftest=provision` help text updated.
+- `internal/reconcile/bringup_test.go` — `TestRunBringUp_StorageSplit_SysDataGrow` +
+  `…_SysDataGrowZeroNoResize`.
+- `CHANGELOG.md` (v0.40.0 entry, newest on top), `RUNBOOK-provisioning-storage.md` (three-volume
+  layout), this `REPORT.md`.
 
-Both pushed to `main`. Green gate (`go build ./... && go vet ./... && go test ./...`) passed in BOTH repos.
+## 3. Commits pushed to `main`
+- `459dad9` — v0.40.0: third CT volume + `-sysdata-grow` (code + tests + CHANGELOG + RUNBOOK).
+- (REPORT.md commit follows.)
 
-## 2. What changed
+## 4. Tests + red-proofs
+- `go build ./... && go vet ./... && go test ./...` — **green** (reconcile + cmd + all packages).
+  - Pre-existing unrelated flake: `internal/escrow/TestGenerateRecoveryCode_EntropyAndFormat`
+    (random 10/11-word count) — passes on re-run; **not touched by this change**.
+- `TestRunBringUp_StorageSplit_SysDataGrow`: PASS — fake api records `ResizeLXC(8051,"mp1","+42G")`
+  alongside `mp0 +240G`.
+- `TestRunBringUp_StorageSplit_SysDataGrowZeroNoResize`: PASS — `SysDataGrowGB=0` ⇒ **no** mp1 resize.
+- **Red-proof #1 (Go "4c" block):** removed the "4c" grow block → `…_SysDataGrow` FAILED
+  (`got [{mp0 +240G}]`, no mp1 resize) → reverted → green. ✅
+- reconcile test count: 127 RUN entries (subtests included), all pass.
+- **Red-proof #2 (shell vzdump guard):** to be run live during the golden rebuild — temporarily flip
+  `mp1` to `backup=0`, confirm the guard ABORTS, revert. **PENDING (needs golden rebuild — §6).**
 
-### Part 1 — live PBS coord (agent)
-- **`internal/pbs/report.go`** — added `SnapshotStore.Get(datastore) []hub.PBSSnapshot` (mutex-guarded copy; per-datastore last-known-good). The ONLY `SnapshotStore` change.
-- **`internal/pbs/live_reporter.go` (new)** — `LiveSnapshotReporter` implements `hub.PBSReporter`. Each collect: derive an 8 s child ctx (`DefaultLiveSnapshotTimeout`), resolve targets, and for each datastore do the cheap `Client.Snapshots()` list, convert via `Snapshot.ToHub()`, `store.Record(...)` (authoritative incl. empty), append. Per-datastore live error → append `store.Get(ds)` (last-known-good, store not clobbered). Targets-resolution error → `store.PBSSnapshots(ctx)` (full LKG aggregate). **List only — never triggers a verify.** A test seam (`listSnapshots` func field, default `liveListSnapshots`) lets tests inject a fake with no live PBS.
-- **`cmd/felhom-agent/main.go`** — hoisted `pbsTargets := pbsTargetsFromPVE(...)` and wired `pbs.NewLiveSnapshotReporter(pbsTargets, pbsStore, pbs.DefaultLiveSnapshotTimeout, logger)` into the collector in **both** `runDaemon` (verify loop reuses the SAME `pbsStore` + `pbsTargets`) **and** `runSelftestHub` (which previously passed a `nil` reporter — that is precisely why the live `--selftest=hub` showed `pbs_snapshots:[]`/no pbs coord). Intended side effect: `report.pbs_snapshots` is now live too.
+## 5. Live deploy DONE — agent binary (§13-A)
+- Built v0.40.0 on the build server (192.168.0.180, go1.26.0, `-ldflags -X main.version=0.40.0`).
+- Deployed to felhom-pve: backed up prior binary to `/usr/local/bin/felhom-agent.bak-0.39.0`,
+  installed, `systemctl restart felhom-agent`.
+- **Verified:** `felhom-agent --version` → `0.40.0`; service `active`; clean restart (local-api
+  listening on 192.168.0.162:8443; only WARN is the expected intent-gated skip of the lingering ejected
+  `ce9d…` drive on 9201 — documented prior behavior, not an error).
 
-### Part 2 — drop `role` + `restic_repo_coord` from the v1 host-half drive shape (agent)
-- **`internal/hub/dr_recipe.go`** — removed `Role` and `ResticRepoCoord` from `DRDrive` and the `Role: t.Role` literal. v1 drive shape is now `{durable_id, mount_path, intent, fs_type?, total_bytes}`. Header + `DRDrive` doc rewritten: role deferred (hub/operator manifest concept), restic coord reserved for a future offsite tier (none exists today), pbs coord resolved live. `isUserDataDrive`/`latestPBSCoord`/`DRPBSCoord` unchanged.
+## 6. Live acceptance — PENDING (blocked on operator-held secrets + an irreversible CHECKPOINT)
+The golden rebuild (§13-B), the `pct destroy 9201` CHECKPOINT (§13-C), and the re-provision (§13-D/E)
+were **not** executed because they require inputs CC does not hold:
+1. **`REGISTRY_USER` + `REGISTRY_TOKEN`** — for the golden's one-time controller-image pull inside the
+   build guest. Confirmed **absent on-host** (no `/root/.docker/config.json`, no saved build env).
+2. **`DEMO_RETRIEVAL_PASSPHRASE`** (the `-hub-password`) — for the 9201 re-provision (the controller's
+   hub config pull).
+3. **Operator confirmation** for the irreversible `pct stop 9201; pct destroy 9201 --purge` (§13-C
+   CHECKPOINT). Operator pre-authorized a clean reinstall in principle, but the destroy is gated.
 
-### Part 3 — goldens (agent + hub)
-- Agent golden `internal/hub/testdata/host-report.golden.json` — dropped `role` from `dr_recipe.drives[0]` (hand-maintained; no generator script exists — `configs/build-golden.sh` builds the LXC archive, not this JSON; the contract test validates key-sets only).
-- Hub `internal/api/testdata/host-report.golden.json` — re-synced **byte-identical** with the agent golden (it previously lacked the `dr_recipe` section entirely).
-- Hub `internal/store/testdata/dr-recipe.golden.json` + `drHostHalf` fixture — dropped `role` from `drives[0]`.
-
-## 3. Tests — results + demonstrated pre-fix companion failures
-
-All run with the mutation applied → confirmed FAIL → reverted → green.
-
-| Test | Result | Companion mutation → demonstrated failure |
-|---|---|---|
-| `pbs.TestLiveReporter_CoordPresentWithoutPriorVerify` (T1, load-bearing) | PASS | `PBSSnapshots` mutated to `return r.store.PBSSnapshots(ctx)` (no live read) → `want 2 live snapshots, got 0` + recipe `pbs` nil. Reverted. |
-| `pbs.TestLiveReporter_ErrorFallsBackToLastKnownGood` (T2) | PASS | dropped the LKG-append on error → `want the last-known-good snapshot on live error, got []`. Reverted. |
-| `pbs.TestLiveReporter_SuccessUpdatesStore` (T3) | PASS | (same no-live-read mutation) → `store not warmed`. Reverted. |
-| `pbs.TestLiveReporter_TargetsErrorReturnsAggregate` (T4) | PASS | — |
-| `pbs.TestLiveReporter_BoundedByTimeout` (T5) | PASS | dropped the LKG-append → blocked-lister path returned `[]` instead of LKG. Reverted. |
-| `pbs.TestLiveReporter_EmptySuccessIsAuthoritative` (T6) | PASS | (same no-live-read mutation) → stale snapshot survived instead of being overwritten. Reverted. |
-| `hub.TestDRRecipeHostHalf_V1DriveShape` (T7) | PASS | re-added `Role string \`json:"role"\`` to `DRDrive` → `v1 drive must NOT carry "role" key` AND `TestHostReport_ContractMatchesGolden` drift `struct keys = [... role ...]`. Reverted. |
-| `hub.TestDRRecipeHostHalf_NoSecrets` (T8) | PASS (unchanged) | — |
-| `hub.TestBuildDRRecipeHostHalf` / `..._NoPBS` / `TestHostReport_ContractMatchesGolden` | PASS (updated to v1 shape) | — |
-| `store.TestAssembleDRRecipe_V1DriveShape` (T9, hub) | PASS | fixture re-added `role` to `drives[0]` → `v1 drives must not carry role/restic_repo_coord`. Reverted. |
-| hub `store`/`api` full suites incl. golden byte-match | PASS | — |
-
-The companion mutations confirmed the live read (T1/T3/T6), the LKG fallback (T2/T5), and both repos' v1-shape guards (T7/T9) are each load-bearing — not hollow assertions.
-
-## 4. Deployed versions (proof)
-
-- Build: `go build -ldflags "-X main.version=0.39.0"` on 192.168.0.180 (go1.26.0) → sha256 `04b7a410…6bd6bd7`; same sha verified after scp to felhom-pve.
-- `felhom-pve`: backed up `/usr/local/bin/felhom-agent` → `.bak-0.38.0`, `install -m0755`, `systemctl restart`. `felhom-agent --version` → **`felhom-agent 0.39.0`**; `systemctl is-active` → **active**.
-- Controller: `gitea.dooplex.hu/admin/felhom-controller:0.73.0` (untouched).
-
-## 5. Live `--selftest=hub` `dr_recipe` block (post-deploy, immediately after restart — NO verify ran)
-
-```json
-"dr_recipe": {
-  "recipe_version": 1,
-  "guests": [
-    { "vmid": 9001, "cores": 2, "memory_bytes": 2147483648, "disk_bytes": 10737418240 },
-    { "vmid": 9999, "cores": 1, "memory_bytes": 268435456, "disk_bytes": 2147483648 },
-    { "vmid": 9201, "cores": 2, "memory_bytes": 12884901888, "disk_bytes": 33501757440 }
-  ],
-  "pbs": { "repo_id": "felhom-pbs", "namespace": "root", "latest_snapshot_id": "9201" },
-  "drives": [
-    { "durable_id": "uuid:81a26531-62d8-408d-812f-a178b1d35310", "mount_path": "/mnt/felhom-flash", "intent": "enrolled", "total_bytes": 125704151040 },
-    { "durable_id": "uuid:da9e7089-cf8e-4617-adcb-a377743fae00", "mount_path": "/mnt/felhom-usb",   "intent": "enrolled", "total_bytes": 983349346304 }
-  ],
-  "pve_storage": [
-    { "name": "local-lvm",    "type": "lvmthin",   "content": "images,rootdir" },
-    { "name": "felhom-pbs",   "type": "pbs",       "content": "backup" },
-    { "name": "felhom-flash", "type": "usb",       "content": "backup" },
-    { "name": "felhom-usb",   "type": "local-dir", "content": "backup" },
-    { "name": "local",        "type": "local",     "content": "backup,vztmpl,iso,import" }
-  ]
-}
+Planned live run once provided (locked sizing per RUNBOOK = 32 rootfs / 200 docker-data / 50 user-data):
 ```
-
-- `pbs` **present** — `repo_id=felhom-pbs`, `namespace=root`, `latest_snapshot_id=9201` (matches the latest `ct/9201 @ 2026-06-16T17:00:10Z` per `pvesm list felhom-pbs`). **This is the gap closed: present seconds after restart, no verify cycle.**
-- `drives[]` carry **no** `role` / `restic_repo_coord` — only `{durable_id, mount_path, intent, total_bytes}` for both externals.
-- `report.pbs_snapshots` is now **live-populated (6)**, was `[]`.
-
-## 6. Hub assembled recipe (post-push, the real `AssembleDRRecipe` over the live `hub.db`)
-
-Daemon's startup host-report landed in the hub DB (`dr_recipe` row `demo-felhom` / host `demo-felhom-01`, `updated_at 2026-06-16 18:28 UTC`). Assembled via a throwaway `store`-package test on 180 against a copy of `/data/hub.db` (dashboard-auth blocks curl; no password available):
-
-- Assembled recipe = **3329 B**, `recipe_version 1`, customer `demo-felhom` (Demo Ügyfél / demo-felhom.eu), **3 guests w/ sizing**, `pbs` present (`felhom-pbs/root/9201`), **2 drives clean** (no role/restic), 5 pve_storage, **14 apps** (app half from controller v0.73.0).
-- **Secret scan over the FULL assembled recipe AND both raw stored halves: 0 secret-shaped key hits** (regex `(?i)(password|secret|token|hash|passphrase|api[_-]?key|\bkey\b|enc:)`). Nothing to redact — the boundary this whole arc protects.
-
-> NOTE: the assembled-recipe operator PANEL (`customer_unified.html` / `GET /customers/{id}/dr-recipe.json`) was NOT browser-rendered (no dashboard password in this env). Verified the exact server pipeline + stored/assembled bytes, not the rendered HTML.
-
-## 7. Cross-repo golden checksum match
-
-`internal/hub/testdata/host-report.golden.json` (agent) and `hub/internal/api/testdata/host-report.golden.json` (hub) are **byte-identical**:
-
-```
-57f2a5e7154ece1be94a79fd327f2924762e70c3ee388d6e375df7c918b2f2b5  (both)
+# B. rebuild golden (root@felhom-pve) — bake controller 0.77.0
+GOLDEN_SYSDATA_GB=8 REGISTRY_USER=… REGISTRY_TOKEN=… \
+  ./build-golden.sh 9100 <TEMPLATE> local-lvm local vmbr0 \
+  gitea.dooplex.hu/admin/felhom-controller:0.77.0
+#    → vzdump log MUST show BOTH "including mount point mp0" AND "… mp1"; capture the new volid.
+#    → Red-proof #2 here (flip mp1 backup=0 → guard aborts → revert).
+# C. CHECKPOINT: pct stop 9201; pct destroy 9201 --purge
+# D. re-provision: felhom-agent --selftest=provision -archive <new-golden> -vmid 9201 \
+#      -customer-id demo-felhom -hub-password <SECRET> -rootfs-grow 0 -datavol-grow 184 -sysdata-grow 42
+#    then: pct reboot 9201
+# E. accept: pct config 9201 (mp0,mp1 backup=1,mp8,mp9) ; pct exec 9201 -- findmnt /mnt/sys_drive ;
+#    pct exec 9201 -- docker exec felhom-controller findmnt /mnt/sys_drive  (← propagation gate) ;
+#    controller monitor: /mnt/sys_drive warning GONE, SSD user-data ~50G.
 ```
+Current 9201 baseline (pre-change, confirms the warning state): rootfs 32G, `mp0` docker-data 256G
+(backup=1), `mp3` `/mnt/felhom-drives`, `mp9` bootstrap — **no `mp1` / `/mnt/sys_drive` volume** (so
+`/mnt/sys_drive` is on the rootfs and the controller warns). Memory 12288 MiB.
 
-## 8. Observations / BACKLOG (flagged, NOT acted on)
+## 7. NOT yet live-validated / residual
+- The §13-B golden rebuild, the §13-D/E re-provision + acceptance gates (the three `findmnt`s + the
+  controller-UI/API warning-cleared check), and shell-guard Red-proof #2 — **all blocked on §6**.
+- **External-drive re-enroll** after the clean reinstall: the destroy wipes the controller's
+  `felhom-controller-data` volume (its storage registry), so `felhom-usb`/`felhom-flash` will need
+  re-enrollment via the UI. Expected for a clean reinstall; **out of scope** for this task's acceptance
+  (solely `/mnt/sys_drive` becoming a separate, warning-free, PBS-covered mount).
 
-**Strategic gap — external-drive bulk data has no offsite / second-failure-domain backup.** PBS covers the
-guest rootfs (whole-CT), but the external user-data drives (felhom-flash 117 GiB, felhom-usb 916 GiB) are
-NOT in PBS. The controller's "cross-drive" / Tier-2 backup (8 apps, method `rsync`, daily, all `ok`) lands
-on `/mnt/sys_drive/felhom-data` — the **same internal SSD inside the same guest on the same host**. In a
-host-loss / drive-loss event that copy dies with the primary. This is exactly why `restic_repo_coord` named
-nothing real and was dropped from v1. **One BACKLOG item: design a genuine offsite bulk-data tier (restic
-or PBS-for-dirs to an off-host repo) for the external drives; only then re-introduce the drive coord.**
+## 8. Observations (noted, NOT acted on)
+- `build-golden.sh` `CONTROLLER_IMAGE` default is stale (`:0.43.0`); the live controller is `0.77.0`.
+  The golden rebuild passes the tag explicitly, so the stale default is cosmetic — left as-is per the
+  minimal-change rule.
+- `felhom-agent/CLAUDE.md` self-reports "Current: v0.31.0" — stale; authoritative version is the
+  `main.go` var + CHANGELOG top (now `0.40.0`). Left as-is (out of scope).
+- Live 9201 uses `mp3` for the shared drives parent (`/mnt/felhom-drives`), not the `mp8` the spec
+  cited — `freeMountSlot` picks the lowest free slot, so the exact number varies. The baked `mp1` does
+  not collide regardless (the guest is rebuilt from the new golden). No action needed.