felhom-agent/REPORT.md

REPORT — Slice 6 Phase B: PBS offsite tier (v0.6.0) (2026-06-09)

Overwrite-latest report (most recent significant work only). Cumulative history lives in CHANGELOG.md.

Outcome

Slice 6 is complete and live-validated; pushed as v0.6.0. Phase B adds the PBS offsite tier (doc 03 §8). The PBS spike made it small: backup-to-PBS and restore-from-PBS reuse Phase A unchanged (PBS is just a storage target + a volid), and the operator token needs no widening. The only new agent code is the verify capability + a small PBS-API client + PBSSnapshot reporting. Escrow + host-loss DR remain slices 7/10.

What landed

• internal/pbs — the PBS-API client (the agent's SECOND privileged external surface, slice-1 discipline): TLS fingerprint-pinned to the PBS leaf cert (a wrong fingerprint → rejected — unit-tested against a fake TLS server), token auth (PBSAPIToken=<id>:<secret>; id from the storage username, secret read at runtime from /etc/pve/priv/storage/<id>.pw, never logged — unit-asserted on the verify loop), typed, no shell. Verify (POST, ignore-verified=false so it actually re-reads), Snapshots, TaskStatus/WaitVerify (node from the UPID — the spike B4 gotcha), NodeFromUPID.
• The verify maintenance loop (pbs/verify.go) — the cheap, key-free, ciphertext-level integrity check (§8) on its own cadence (default 6h, the 5th daemon goroutine). NOT gated/journaled — maintenance/reporting like the slice-5 watchdog. Verify → poll → re-list → record per-snapshot verify_state; a failed verify is logged loudly.
• PBSSnapshot reporting filled (namespace/type/id/time/size/owner/protected/encrypted (from files[].crypt-mode)/verify_state/verify_upid). PBSReporter collector seam + in-memory SnapshotStore. Cross-repo golden (byte-identical) + bidirectional key-set tests; hub handler.go parses pbs_snapshots + logs a failed-verify [WARN] (deployed v0.7.3).
• Truthful backup mode — Backup.mode now reports the ACTUAL vzdump mode from the task log (PVE downgrades snapshot→stop for a stopped guest). proxmox.Storage.Username. config PBSVerifyCadence/PBSSecretDir. --selftest=pbs-verify.

Tests

go test ./... green; go test -race ./... green on the build server (the new verify goroutine). PBS client: fingerprint-pin enforcement (wrong→rejected, right→connects), token-header construction + never-logged (loop log captured, secret asserted absent), Verify POST→UPID, Snapshots→PBSSnapshot (incl. crypt-mode→encrypted, absent verification→none), NodeFromUPID, verify cadence (records ok/failed, disabled-by-negative), mode-actual parse, cross-repo golden + hub ingest.

Live PBS runbook (reusing the spike's DooPlex PBS)

• A — backup → PBS: --selftest=backup -vmid 9001 → felhom-pbs:backup/ct/9001/…, success, and mode:"stop" — the truthful-mode parse caught PVE's downgrade live.
• B — verify → ok: --selftest=pbs-verify triggered the verify, polled the task (node dooplex from the UPID), re-listed → all snapshots verify_state:"ok".
• C — restore-test from PBS: --selftest=restore-test picked the newest PBS snapshot (source_tier:"pbs" — derived from the storage type), restored → net-link-down → boot → verified running → torn down, pass, no leak.
• D — forced failed verify: corrupted a chunk in the datastore on DooPlex → first re-verify still showed ok because PBS's ignore-verified=true default skips already-verified snapshots. Fix landed (Verify now sends ignore-verified=false) → re-verify re-read the chunk, detected the corruption: all snapshots verify_state:"failed", agent logged ERROR … FAILED-verify snapshots. Then cleaned up: forgot the corrupted snapshots, GC'd the chunk, took a fresh backup, final verify → ok. Datastore left healthy.

Two gaps the runbook surfaced were fixed live (committed): the ignore-verified default (else corruption is never caught) and the hardcoded source_tier (now derived from the storage type).

Not done (flagged)

• End-to-end hub reflection of pbs_snapshots (daemon → hub.felhom.eu): the hub side (v0.7.3) is deployed and its parse + failed-verify WARN are unit-tested, but the live daemon→hub round-trip needs the demo host enrolled on the operator hub (a per-host key) — the same one-time enrollment deferred in slices 5/6A. Offered, not done unprompted.

End state / left in place

PBS server on DooPlex + the felhom-spike datastore (one healthy, verified snapshot) + the N100's encrypted felhom-pbs storage — all left up. Agent on the N100 is v0.6.0; config reverted (backup.local_backup_target=local, pbs verify cadence disabled). No leftover guests. No secrets committed — the token secret + encryption key live only under /etc/pve/priv/storage/ (0600), referenced by location/fingerprint.

Next

Slice 7 — provisioning + identity-reset + the golden base (§9): the unified bring-up primitive; restore-overwrite + decommission executors the gate already guards; escrow + host-loss DR.