REPORT: v0.39.0 DR-recipe completion (live PBS coord + v1 drive-shape) — live-validated

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

REPORT.md

@@ -1,56 +1,111 @@
-# REPORT — felhom-agent v0.38.0 (DR recipe: agent storage/guest/PBS half)
+# REPORT — DR-recipe completion: live PBS coord + drop the two unfillable drive fields
 
-**TASK — DR recipe slice (agent half).** Emit the secret-free reconstruction-recipe scaffolding the
-agent already owns, as an additive host-report section. Grounded in `SPIKE-dr-recipe-2026-06-16.md`.
+**Repo:** `felhom-agent` · **Version:** v0.38.0 → **v0.39.0** · **Date:** 2026-06-16
+**Companion repo:** `felhom.eu/hub` v0.13.0 → **v0.13.1** (test-data + regression test only)
+**Controller:** untouched.
 
-## What
+## 1. Confirmed baselines + commit hashes
 
-The recipe is the non-secret **re-provision plan** that complements escrow (keys) and PBS/restic
-(bytes): guest sizing, drive inventory (durable-id → role → mount → intent), PVE storage defs, and PBS
-coordinates — the host/guest/storage scaffolding the operator must rebuild on new hardware *before* the
-PBS bytes can land. The agent owns every fact (`StorageTarget`, `GuestSpec`, `PBSSnapshot`), so it emits
-its half additively in the existing host-report; the hub assembles it with the controller's app half.
+| Repo | Baseline | Target | Commit |
+|---|---|---|---|
+| felhom-agent | v0.38.0 (`8abc1b8`) | **v0.39.0** | **`fbe1130`** |
+| felhom-hub (`felhom.eu`) | v0.13.0 (`5f5e3c5`) | **v0.13.1** | **`149a3b0`** |
 
-## Implementation
+Both pushed to `main`. Green gate (`go build ./... && go vet ./... && go test ./...`) passed in BOTH repos.
 
-- `internal/hub/dr_recipe.go` — wire types + the pure `BuildDRRecipeHostHalf(guests, targets, pbs)`:
-  - `guests[]` ← each `Guest.Spec` (cores/memory/disk), skipping status-unknown guests.
-  - `drives[]` ← user-data external drives only (`isUserDataDrive`: usb/local-dir with a non-empty
-    durable-id + mount path) → `{durable_id, role, mount_path, intent=enrolled, total_bytes}`.
-  - `pve_storage[]` ← **every** storage target `{name, type, content}` (rebuild `storage.cfg`).
-  - `pbs` ← the latest snapshot's `{repo_id (the pbs storage id), namespace, latest_snapshot_id}`.
-- `HostReport.DRRecipe *DRRecipeHostHalf json:"dr_recipe"` (always set in `Collect()`, never null).
-- No new reads — derived from the just-collected report facts.
+## 2. What changed
 
-## Boundary (non-negotiable)
+### Part 1 — live PBS coord (agent)
+- **`internal/pbs/report.go`** — added `SnapshotStore.Get(datastore) []hub.PBSSnapshot` (mutex-guarded copy; per-datastore last-known-good). The ONLY `SnapshotStore` change.
+- **`internal/pbs/live_reporter.go` (new)** — `LiveSnapshotReporter` implements `hub.PBSReporter`. Each collect: derive an 8 s child ctx (`DefaultLiveSnapshotTimeout`), resolve targets, and for each datastore do the cheap `Client.Snapshots()` list, convert via `Snapshot.ToHub()`, `store.Record(...)` (authoritative incl. empty), append. Per-datastore live error → append `store.Get(ds)` (last-known-good, store not clobbered). Targets-resolution error → `store.PBSSnapshots(ctx)` (full LKG aggregate). **List only — never triggers a verify.** A test seam (`listSnapshots` func field, default `liveListSnapshots`) lets tests inject a fake with no live PBS.
+- **`cmd/felhom-agent/main.go`** — hoisted `pbsTargets := pbsTargetsFromPVE(...)` and wired `pbs.NewLiveSnapshotReporter(pbsTargets, pbsStore, pbs.DefaultLiveSnapshotTimeout, logger)` into the collector in **both** `runDaemon` (verify loop reuses the SAME `pbsStore` + `pbsTargets`) **and** `runSelftestHub` (which previously passed a `nil` reporter — that is precisely why the live `--selftest=hub` showed `pbs_snapshots:[]`/no pbs coord). Intended side effect: `report.pbs_snapshots` is now live too.
 
-Every field is an identifier / intent / size / coordinate. The PBS **key** stays in escrow, the access
-**token** in identity-escrow, the restic **password** in escrow — the recipe names only the coordinates
-the restore targets. This is the exact axis the retired infra-backup violated (it shipped
-`encryption_key_b64`/`restic_password`/`cf_api_token`).
+### Part 2 — drop `role` + `restic_repo_coord` from the v1 host-half drive shape (agent)
+- **`internal/hub/dr_recipe.go`** — removed `Role` and `ResticRepoCoord` from `DRDrive` and the `Role: t.Role` literal. v1 drive shape is now `{durable_id, mount_path, intent, fs_type?, total_bytes}`. Header + `DRDrive` doc rewritten: role deferred (hub/operator manifest concept), restic coord reserved for a future offsite tier (none exists today), pbs coord resolved live. `isUserDataDrive`/`latestPBSCoord`/`DRPBSCoord` unchanged.
 
-## Tests (non-hollow)
+### Part 3 — goldens (agent + hub)
+- Agent golden `internal/hub/testdata/host-report.golden.json` — dropped `role` from `dr_recipe.drives[0]` (hand-maintained; no generator script exists — `configs/build-golden.sh` builds the LXC archive, not this JSON; the contract test validates key-sets only).
+- Hub `internal/api/testdata/host-report.golden.json` — re-synced **byte-identical** with the agent golden (it previously lacked the `dr_recipe` section entirely).
+- Hub `internal/store/testdata/dr-recipe.golden.json` + `drHostHalf` fixture — dropped `role` from `drives[0]`.
 
-- `TestBuildDRRecipeHostHalf` — drives = only user-data; pve_storage = all targets; pbs = latest
-  snapshot; guests skip nil-spec.
-- `TestBuildDRRecipeHostHalf_NoPBS` — no snapshots → `pbs` omitted, slices non-nil.
-- `TestDRRecipeHostHalf_NoSecrets` — the boundary mirror: serialized half has NO key matching
-  `(?i)(password|secret|token|hash|passphrase|api[_-]?key|\bkey\b|enc:)`. (The load-bearing boundary
-  test, with a synthetic-secret app + allowlist red-proof, lives on the controller emitter.)
-- `dr_recipe` key-set + sub-array element key-sets added to `TestHostReport_ContractMatchesGolden`
-  (the cross-repo golden, byte-pinned with the hub's copy).
+## 3. Tests — results + demonstrated pre-fix companion failures
 
-## Versioning
+All run with the mutation applied → confirmed FAIL → reverted → green.
 
-`recipe_version=1`, carried in the section. Read is ignore-unknown (encoding/json default) for
-forward-compat, mirroring `storage_manifest`. Golden discipline: `host-report.golden.json` here must stay
-byte-identical to the hub's copy — manual checksum-diff on any wire change (the golden spans three repos).
+| Test | Result | Companion mutation → demonstrated failure |
+|---|---|---|
+| `pbs.TestLiveReporter_CoordPresentWithoutPriorVerify` (T1, load-bearing) | PASS | `PBSSnapshots` mutated to `return r.store.PBSSnapshots(ctx)` (no live read) → `want 2 live snapshots, got 0` + recipe `pbs` nil. Reverted. |
+| `pbs.TestLiveReporter_ErrorFallsBackToLastKnownGood` (T2) | PASS | dropped the LKG-append on error → `want the last-known-good snapshot on live error, got []`. Reverted. |
+| `pbs.TestLiveReporter_SuccessUpdatesStore` (T3) | PASS | (same no-live-read mutation) → `store not warmed`. Reverted. |
+| `pbs.TestLiveReporter_TargetsErrorReturnsAggregate` (T4) | PASS | — |
+| `pbs.TestLiveReporter_BoundedByTimeout` (T5) | PASS | dropped the LKG-append → blocked-lister path returned `[]` instead of LKG. Reverted. |
+| `pbs.TestLiveReporter_EmptySuccessIsAuthoritative` (T6) | PASS | (same no-live-read mutation) → stale snapshot survived instead of being overwritten. Reverted. |
+| `hub.TestDRRecipeHostHalf_V1DriveShape` (T7) | PASS | re-added `Role string \`json:"role"\`` to `DRDrive` → `v1 drive must NOT carry "role" key` AND `TestHostReport_ContractMatchesGolden` drift `struct keys = [... role ...]`. Reverted. |
+| `hub.TestDRRecipeHostHalf_NoSecrets` (T8) | PASS (unchanged) | — |
+| `hub.TestBuildDRRecipeHostHalf` / `..._NoPBS` / `TestHostReport_ContractMatchesGolden` | PASS (updated to v1 shape) | — |
+| `store.TestAssembleDRRecipe_V1DriveShape` (T9, hub) | PASS | fixture re-added `role` to `drives[0]` → `v1 drives must not carry role/restic_repo_coord`. Reverted. |
+| hub `store`/`api` full suites incl. golden byte-match | PASS | — |
 
-## Gate
+The companion mutations confirmed the live read (T1/T3/T6), the LKG fallback (T2/T5), and both repos' v1-shape guards (T7/T9) are each load-bearing — not hollow assertions.
 
-`go build`, `go vet`, `go test ./...` all green (local + build server). Deployed to felhom-pve.
+## 4. Deployed versions (proof)
 
-## Deferred (NOT in this slice)
+- Build: `go build -ldflags "-X main.version=0.39.0"` on 192.168.0.180 (go1.26.0) → sha256 `04b7a410…6bd6bd7`; same sha verified after scp to felhom-pve.
+- `felhom-pve`: backed up `/usr/local/bin/felhom-agent` → `.bak-0.38.0`, `install -m0755`, `systemctl restart`. `felhom-agent --version` → **`felhom-agent 0.39.0`**; `systemctl is-active` → **active**.
+- Controller: `gitea.dooplex.hu/admin/felhom-controller:0.73.0` (untouched).
 
-The agent-side `restore_directive` consumption / recovery-mode execution (`syncer.go:92`) — slice-10D.
-This slice only EMITS the recipe.
+## 5. Live `--selftest=hub` `dr_recipe` block (post-deploy, immediately after restart — NO verify ran)
+
+```json
+"dr_recipe": {
+  "recipe_version": 1,
+  "guests": [
+    { "vmid": 9001, "cores": 2, "memory_bytes": 2147483648, "disk_bytes": 10737418240 },
+    { "vmid": 9999, "cores": 1, "memory_bytes": 268435456, "disk_bytes": 2147483648 },
+    { "vmid": 9201, "cores": 2, "memory_bytes": 12884901888, "disk_bytes": 33501757440 }
+  ],
+  "pbs": { "repo_id": "felhom-pbs", "namespace": "root", "latest_snapshot_id": "9201" },
+  "drives": [
+    { "durable_id": "uuid:81a26531-62d8-408d-812f-a178b1d35310", "mount_path": "/mnt/felhom-flash", "intent": "enrolled", "total_bytes": 125704151040 },
+    { "durable_id": "uuid:da9e7089-cf8e-4617-adcb-a377743fae00", "mount_path": "/mnt/felhom-usb",   "intent": "enrolled", "total_bytes": 983349346304 }
+  ],
+  "pve_storage": [
+    { "name": "local-lvm",    "type": "lvmthin",   "content": "images,rootdir" },
+    { "name": "felhom-pbs",   "type": "pbs",       "content": "backup" },
+    { "name": "felhom-flash", "type": "usb",       "content": "backup" },
+    { "name": "felhom-usb",   "type": "local-dir", "content": "backup" },
+    { "name": "local",        "type": "local",     "content": "backup,vztmpl,iso,import" }
+  ]
+}
+```
+
+- `pbs` **present** — `repo_id=felhom-pbs`, `namespace=root`, `latest_snapshot_id=9201` (matches the latest `ct/9201 @ 2026-06-16T17:00:10Z` per `pvesm list felhom-pbs`). **This is the gap closed: present seconds after restart, no verify cycle.**
+- `drives[]` carry **no** `role` / `restic_repo_coord` — only `{durable_id, mount_path, intent, total_bytes}` for both externals.
+- `report.pbs_snapshots` is now **live-populated (6)**, was `[]`.
+
+## 6. Hub assembled recipe (post-push, the real `AssembleDRRecipe` over the live `hub.db`)
+
+Daemon's startup host-report landed in the hub DB (`dr_recipe` row `demo-felhom` / host `demo-felhom-01`, `updated_at 2026-06-16 18:28 UTC`). Assembled via a throwaway `store`-package test on 180 against a copy of `/data/hub.db` (dashboard-auth blocks curl; no password available):
+
+- Assembled recipe = **3329 B**, `recipe_version 1`, customer `demo-felhom` (Demo Ügyfél / demo-felhom.eu), **3 guests w/ sizing**, `pbs` present (`felhom-pbs/root/9201`), **2 drives clean** (no role/restic), 5 pve_storage, **14 apps** (app half from controller v0.73.0).
+- **Secret scan over the FULL assembled recipe AND both raw stored halves: 0 secret-shaped key hits** (regex `(?i)(password|secret|token|hash|passphrase|api[_-]?key|\bkey\b|enc:)`). Nothing to redact — the boundary this whole arc protects.
+
+> NOTE: the assembled-recipe operator PANEL (`customer_unified.html` / `GET /customers/{id}/dr-recipe.json`) was NOT browser-rendered (no dashboard password in this env). Verified the exact server pipeline + stored/assembled bytes, not the rendered HTML.
+
+## 7. Cross-repo golden checksum match
+
+`internal/hub/testdata/host-report.golden.json` (agent) and `hub/internal/api/testdata/host-report.golden.json` (hub) are **byte-identical**:
+
+```
+57f2a5e7154ece1be94a79fd327f2924762e70c3ee388d6e375df7c918b2f2b5  (both)
+```
+
+## 8. Observations / BACKLOG (flagged, NOT acted on)
+
+**Strategic gap — external-drive bulk data has no offsite / second-failure-domain backup.** PBS covers the
+guest rootfs (whole-CT), but the external user-data drives (felhom-flash 117 GiB, felhom-usb 916 GiB) are
+NOT in PBS. The controller's "cross-drive" / Tier-2 backup (8 apps, method `rsync`, daily, all `ok`) lands
+on `/mnt/sys_drive/felhom-data` — the **same internal SSD inside the same guest on the same host**. In a
+host-loss / drive-loss event that copy dies with the primary. This is exactly why `restic_repo_coord` named
+nothing real and was dropped from v1. **One BACKLOG item: design a genuine offsite bulk-data tier (restic
+or PBS-for-dirs to an off-host repo) for the external drives; only then re-introduce the drive coord.**