felhom-agent/REPORT.md

REPORT — DR-recipe completion: live PBS coord + drop the two unfillable drive fields

Repo: felhom-agent · Version: v0.38.0 → v0.39.0 · Date: 2026-06-16 Companion repo: felhom.eu/hub v0.13.0 → v0.13.1 (test-data + regression test only) Controller: untouched.

1. Confirmed baselines + commit hashes

Repo	Baseline	Target	Commit
felhom-agent	v0.38.0 (8abc1b8)	v0.39.0	fbe1130
felhom-hub (felhom.eu)	v0.13.0 (5f5e3c5)	v0.13.1	149a3b0

Both pushed to main. Green gate (go build ./... && go vet ./... && go test ./...) passed in BOTH repos.

2. What changed

Part 1 — live PBS coord (agent)

• internal/pbs/report.go — added SnapshotStore.Get(datastore) []hub.PBSSnapshot (mutex-guarded copy; per-datastore last-known-good). The ONLY SnapshotStore change.
• internal/pbs/live_reporter.go (new) — LiveSnapshotReporter implements hub.PBSReporter. Each collect: derive an 8 s child ctx (DefaultLiveSnapshotTimeout), resolve targets, and for each datastore do the cheap Client.Snapshots() list, convert via Snapshot.ToHub(), store.Record(...) (authoritative incl. empty), append. Per-datastore live error → append store.Get(ds) (last-known-good, store not clobbered). Targets-resolution error → store.PBSSnapshots(ctx) (full LKG aggregate). List only — never triggers a verify. A test seam (listSnapshots func field, default liveListSnapshots) lets tests inject a fake with no live PBS.
• cmd/felhom-agent/main.go — hoisted pbsTargets := pbsTargetsFromPVE(...) and wired pbs.NewLiveSnapshotReporter(pbsTargets, pbsStore, pbs.DefaultLiveSnapshotTimeout, logger) into the collector in both runDaemon (verify loop reuses the SAME pbsStore + pbsTargets) and runSelftestHub (which previously passed a nil reporter — that is precisely why the live --selftest=hub showed pbs_snapshots:[]/no pbs coord). Intended side effect: report.pbs_snapshots is now live too.

Part 2 — drop role + restic_repo_coord from the v1 host-half drive shape (agent)

• internal/hub/dr_recipe.go — removed Role and ResticRepoCoord from DRDrive and the Role: t.Role literal. v1 drive shape is now {durable_id, mount_path, intent, fs_type?, total_bytes}. Header + DRDrive doc rewritten: role deferred (hub/operator manifest concept), restic coord reserved for a future offsite tier (none exists today), pbs coord resolved live. isUserDataDrive/latestPBSCoord/DRPBSCoord unchanged.

Part 3 — goldens (agent + hub)

• Agent golden internal/hub/testdata/host-report.golden.json — dropped role from dr_recipe.drives[0] (hand-maintained; no generator script exists — configs/build-golden.sh builds the LXC archive, not this JSON; the contract test validates key-sets only).
• Hub internal/api/testdata/host-report.golden.json — re-synced byte-identical with the agent golden (it previously lacked the dr_recipe section entirely).
• Hub internal/store/testdata/dr-recipe.golden.json + drHostHalf fixture — dropped role from drives[0].

3. Tests — results + demonstrated pre-fix companion failures

All run with the mutation applied → confirmed FAIL → reverted → green.

Test	Result	Companion mutation → demonstrated failure
pbs.TestLiveReporter_CoordPresentWithoutPriorVerify (T1, load-bearing)	PASS	PBSSnapshots mutated to return r.store.PBSSnapshots(ctx) (no live read) → want 2 live snapshots, got 0 + recipe pbs nil. Reverted.
pbs.TestLiveReporter_ErrorFallsBackToLastKnownGood (T2)	PASS	dropped the LKG-append on error → want the last-known-good snapshot on live error, got []. Reverted.
pbs.TestLiveReporter_SuccessUpdatesStore (T3)	PASS	(same no-live-read mutation) → store not warmed. Reverted.
pbs.TestLiveReporter_TargetsErrorReturnsAggregate (T4)	PASS	—
pbs.TestLiveReporter_BoundedByTimeout (T5)	PASS	dropped the LKG-append → blocked-lister path returned [] instead of LKG. Reverted.
pbs.TestLiveReporter_EmptySuccessIsAuthoritative (T6)	PASS	(same no-live-read mutation) → stale snapshot survived instead of being overwritten. Reverted.
hub.TestDRRecipeHostHalf_V1DriveShape (T7)	PASS	re-added Role string \json:"role"`toDRDrive→v1 drive must NOT carry "role" keyANDTestHostReport_ContractMatchesGoldendriftstruct keys = [... role ...]`. Reverted.
hub.TestDRRecipeHostHalf_NoSecrets (T8)	PASS (unchanged)	—
hub.TestBuildDRRecipeHostHalf / ..._NoPBS / TestHostReport_ContractMatchesGolden	PASS (updated to v1 shape)	—
store.TestAssembleDRRecipe_V1DriveShape (T9, hub)	PASS	fixture re-added role to drives[0] → v1 drives must not carry role/restic_repo_coord. Reverted.
hub store/api full suites incl. golden byte-match	PASS	—

The companion mutations confirmed the live read (T1/T3/T6), the LKG fallback (T2/T5), and both repos' v1-shape guards (T7/T9) are each load-bearing — not hollow assertions.

4. Deployed versions (proof)

• Build: go build -ldflags "-X main.version=0.39.0" on 192.168.0.180 (go1.26.0) → sha256 04b7a410…6bd6bd7; same sha verified after scp to felhom-pve.
• felhom-pve: backed up /usr/local/bin/felhom-agent → .bak-0.38.0, install -m0755, systemctl restart. felhom-agent --version → felhom-agent 0.39.0; systemctl is-active → active.
• Controller: gitea.dooplex.hu/admin/felhom-controller:0.73.0 (untouched).

5. Live --selftest=hub dr_recipe block (post-deploy, immediately after restart — NO verify ran)

"dr_recipe": {
  "recipe_version": 1,
  "guests": [
    { "vmid": 9001, "cores": 2, "memory_bytes": 2147483648, "disk_bytes": 10737418240 },
    { "vmid": 9999, "cores": 1, "memory_bytes": 268435456, "disk_bytes": 2147483648 },
    { "vmid": 9201, "cores": 2, "memory_bytes": 12884901888, "disk_bytes": 33501757440 }
  ],
  "pbs": { "repo_id": "felhom-pbs", "namespace": "root", "latest_snapshot_id": "9201" },
  "drives": [
    { "durable_id": "uuid:81a26531-62d8-408d-812f-a178b1d35310", "mount_path": "/mnt/felhom-flash", "intent": "enrolled", "total_bytes": 125704151040 },
    { "durable_id": "uuid:da9e7089-cf8e-4617-adcb-a377743fae00", "mount_path": "/mnt/felhom-usb",   "intent": "enrolled", "total_bytes": 983349346304 }
  ],
  "pve_storage": [
    { "name": "local-lvm",    "type": "lvmthin",   "content": "images,rootdir" },
    { "name": "felhom-pbs",   "type": "pbs",       "content": "backup" },
    { "name": "felhom-flash", "type": "usb",       "content": "backup" },
    { "name": "felhom-usb",   "type": "local-dir", "content": "backup" },
    { "name": "local",        "type": "local",     "content": "backup,vztmpl,iso,import" }
  ]
}

• pbs present — repo_id=felhom-pbs, namespace=root, latest_snapshot_id=9201 (matches the latest ct/9201 @ 2026-06-16T17:00:10Z per pvesm list felhom-pbs). This is the gap closed: present seconds after restart, no verify cycle.
• drives[] carry no role / restic_repo_coord — only {durable_id, mount_path, intent, total_bytes} for both externals.
• report.pbs_snapshots is now live-populated (6), was [].

6. Hub assembled recipe (post-push, the real AssembleDRRecipe over the live hub.db)

Daemon's startup host-report landed in the hub DB (dr_recipe row demo-felhom / host demo-felhom-01, updated_at 2026-06-16 18:28 UTC). Assembled via a throwaway store-package test on 180 against a copy of /data/hub.db (dashboard-auth blocks curl; no password available):

• Assembled recipe = 3329 B, recipe_version 1, customer demo-felhom (Demo Ügyfél / demo-felhom.eu), 3 guests w/ sizing, pbs present (felhom-pbs/root/9201), 2 drives clean (no role/restic), 5 pve_storage, 14 apps (app half from controller v0.73.0).
• Secret scan over the FULL assembled recipe AND both raw stored halves: 0 secret-shaped key hits (regex (?i)(password|secret|token|hash|passphrase|api[_-]?key|\bkey\b|enc:)). Nothing to redact — the boundary this whole arc protects.

NOTE: the assembled-recipe operator PANEL (customer_unified.html / GET /customers/{id}/dr-recipe.json) was NOT browser-rendered (no dashboard password in this env). Verified the exact server pipeline + stored/assembled bytes, not the rendered HTML.

7. Cross-repo golden checksum match

internal/hub/testdata/host-report.golden.json (agent) and hub/internal/api/testdata/host-report.golden.json (hub) are byte-identical:

57f2a5e7154ece1be94a79fd327f2924762e70c3ee388d6e375df7c918b2f2b5  (both)

8. Observations / BACKLOG (flagged, NOT acted on)

Strategic gap — external-drive bulk data has no offsite / second-failure-domain backup. PBS covers the guest rootfs (whole-CT), but the external user-data drives (felhom-flash 117 GiB, felhom-usb 916 GiB) are NOT in PBS. The controller's "cross-drive" / Tier-2 backup (8 apps, method rsync, daily, all ok) lands on /mnt/sys_drive/felhom-data — the same internal SSD inside the same guest on the same host. In a host-loss / drive-loss event that copy dies with the primary. This is exactly why restic_repo_coord named nothing real and was dropped from v1. One BACKLOG item: design a genuine offsite bulk-data tier (restic or PBS-for-dirs to an off-host repo) for the external drives; only then re-introduce the drive coord.