docs(REPORT): slice 6 Phase B — PBS offsite tier, live-validated on DooPlex PBS

Live: backup→PBS (actual mode 'stop' caught), verify→ok (node from UPID), restore-test
from PBS (source_tier=pbs), and a forced failed-verify (corrupt chunk → ignore-verified=false
re-read → verify_state=failed → cleaned up + re-verified ok). Hub e2e reflection flagged
(needs host enrollment). Datastore left healthy.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

REPORT.md

@@ -1,78 +1,81 @@
-# REPORT — Slice 6 Phase A: backup + the self-restore-test (v0.6.0-rc1) (2026-06-09)
+# REPORT — Slice 6 Phase B: PBS offsite tier (v0.6.0) (2026-06-09)
 
 > Overwrite-latest report (most recent significant work only). Cumulative history lives in [CHANGELOG.md](CHANGELOG.md).
 
 ## Outcome
 
-**Slice 6 Phase A is implemented, live-validated on `demo-felhom`, and pushed as
-`v0.6.0-rc1`** — the guest-level backup layer + the journaled **self-restore-test** that
-closes "a backup you haven't restored isn't a backup" (doc 03 §8). Everything is **benign**
-(backup, restore-to-NEW, scratch teardown): it reuses the slice-4 classifier/gate/journal —
-no new destructive class, no new crypto. **Local target only**; PBS is Phase B. Restore is
-to a **NEW** guest only (no overwrite). Backups are **crash-consistent** (marked so). This is
-the hard checkpoint: awaiting validation before Phase B.
+**Slice 6 is complete and live-validated; pushed as `v0.6.0`.** Phase B adds the PBS offsite
+tier (doc 03 §8). The PBS spike made it small: backup-to-PBS and restore-from-PBS reuse Phase
+A **unchanged** (PBS is just a storage target + a volid), and the operator token needs no
+widening. The only new agent code is **the verify capability + a small PBS-API client +
+PBSSnapshot reporting**. Escrow + host-loss DR remain slices 7/10.
 
 ## What landed
 
-- **proxmox**: `DestroyLXC` (the scratch-teardown primitive), `VzdumpOptions.Notes` →
-  `notes-template`, `LatestBackupVolID` (resolve a produced archive — the task status carries
-  no result volid).
-- **The self-restore-test** (`reconcile.Engine.RunRestoreTest`): pick a free scratch VMID
-  (configured band, excludes 9999; full band → skip) → **journal a `Scratch`-owned entry
-  BEFORE any mutation** → restore-to-new → benign net **link-down** SetConfig (so the clone
-  can't conflict with a running source's MAC/IP — test-safety, NOT slice-7 identity reset) →
-  boot → verify **reaches `running`** → **defer teardown that ALWAYS runs** (benign
-  `ClassGuestDestroy` + agent-tagged-scratch provenance, through the gate).
-- **Crash-safe recovery** (`Recover`): a `Scratch` entry is resolved by **teardown**, not by
-  re-checking the restore sub-task's UPID — special-cased before the generic path (else a
-  leaked guest would be marked "succeeded"). Idempotent: already-gone → clean; list-unreadable
-  → left in-flight.
-- **`internal/backup`**: `BackupRunner` (vzdump + archive/size resolve + the **bulk-volume
-  gap** — a mountpoint is uncovered unless it carries an explicit `backup=1`, so an unset
-  `backup=` counts as uncovered, the safe DR direction), `PickRestoreCandidate`, an in-memory
-  `Store` (latest backup-per-target + latest restore-test) feeding the hub reporter seams, and
-  a cadence `Scheduler` (default 24h; the fourth daemon goroutine; disabled cleanly when
-  off/misconfigured).
-- **hub report**: `Backup` + `RestoreTest` filled (`PBSSnapshot` stays a Phase-B stub);
-  collector seams; cross-repo golden byte-identical + bidirectional key-set tests in both
-  repos; hub `handler.go` parses + persists them and **logs a FAILED restore-test
-  prominently** (the loudest DR signal).
-- **config** `BackupConfig` (band 990000–990009 default); **`--selftest=backup`** and
-  **`--selftest=restore-test`** (Recover-then restore→boot→verify→teardown).
+- **`internal/pbs` — the PBS-API client** (the agent's SECOND privileged external surface,
+  slice-1 discipline): TLS **fingerprint-pinned** to the PBS leaf cert (a wrong fingerprint →
+  rejected — unit-tested against a fake TLS server), **token auth**
+  (`PBSAPIToken=<id>:<secret>`; id from the storage `username`, secret read at runtime from
+  `/etc/pve/priv/storage/<id>.pw`, **never logged** — unit-asserted on the verify loop), typed,
+  no shell. `Verify` (POST, **`ignore-verified=false`** so it actually re-reads), `Snapshots`,
+  `TaskStatus`/`WaitVerify` (**node from the UPID** — the spike B4 gotcha), `NodeFromUPID`.
+- **The verify maintenance loop** (`pbs/verify.go`) — the cheap, key-free, ciphertext-level
+  integrity check (§8) on its own cadence (default 6h, the 5th daemon goroutine). NOT
+  gated/journaled — maintenance/reporting like the slice-5 watchdog. Verify → poll → re-list →
+  record per-snapshot `verify_state`; a failed verify is logged loudly.
+- **`PBSSnapshot` reporting** filled (namespace/type/id/time/size/owner/protected/`encrypted`
+  (from `files[].crypt-mode`)/`verify_state`/`verify_upid`). `PBSReporter` collector seam +
+  in-memory `SnapshotStore`. Cross-repo golden (byte-identical) + bidirectional key-set tests;
+  hub `handler.go` parses `pbs_snapshots` + logs a **failed-verify `[WARN]`** (deployed v0.7.3).
+- **Truthful backup mode** — `Backup.mode` now reports the ACTUAL vzdump mode from the task
+  log (PVE downgrades snapshot→stop for a stopped guest). `proxmox.Storage.Username`. config
+  `PBSVerifyCadence`/`PBSSecretDir`. **`--selftest=pbs-verify`**.
 
 ## Tests
 
-`go test ./...` green; **`go test -race ./...` green on the build server** (the scheduler
-goroutine). Notable unit tests: vzdump async UPID→WaitTask + bulk-gap (`backup!=1` incl.
-unset); restore-test pass + **teardown-on-failed-verify** + restore-failure-still-tears-down +
-full-band-skip; the DELETE wire contract (no body + force=1); and the headline **recover-
-leaked-scratch** trio (destroyed / already-gone-clean / list-unreadable-left-in-flight).
+`go test ./...` green; **`go test -race ./...` green on the build server** (the new verify
+goroutine). PBS client: **fingerprint-pin enforcement** (wrong→rejected, right→connects),
+token-header construction + **never-logged** (loop log captured, secret asserted absent),
+`Verify` POST→UPID, `Snapshots`→`PBSSnapshot` (incl. crypt-mode→`encrypted`, absent
+verification→`none`), `NodeFromUPID`, verify cadence (records ok/failed, disabled-by-negative),
+mode-actual parse, cross-repo golden + hub ingest.
 
-## Live validation on demo-felhom
+## Live PBS runbook (reusing the spike's DooPlex PBS)
 
-- **`--selftest=backup -vmid 9001`** → vzdump of the stopped spike-lxc to `local`: success,
-  **crash-consistent**, archive `local:backup/vzdump-lxc-9001-…tar.zst` (~934 MiB) resolved,
-  26 s. Bulk gap empty (9001 has no extra mountpoints).
-- **`--selftest=restore-test`** → restored that backup into scratch **990000** on local-lvm,
-  net link-down, booted, verified `running`, torn down (gate-benign) — **pass, 15 s, no leak**.
-- **Crash-recovery proven for real**: the first teardown hit a `DestroyLXC` bug (DELETE with
-  a form body → PVE HTTP 501; a booted guest also needs `force=1`), which **leaked scratch
-  990000**. The fix moved the flags to the query string + `force=1`, **no body** (regression-
-  tested). On the next run, `Recover` found the journaled in-flight `Scratch` entry, confirmed
-  990000 existed, and **destroyed the leaked guest** (`ScratchDestroyed:1`) through the benign
-  gate — then ran a clean test. `pct list` after: **no 99000x guests**. The accidental leak +
-  reap is the best possible live proof of the slice's load-bearing crash-safety path.
+- **A — backup → PBS**: `--selftest=backup -vmid 9001` → `felhom-pbs:backup/ct/9001/…`,
+  success, and **`mode:"stop"`** — the truthful-mode parse caught PVE's downgrade live.
+- **B — verify → ok**: `--selftest=pbs-verify` triggered the verify, polled the task (node
+  `dooplex` from the UPID), re-listed → all snapshots `verify_state:"ok"`.
+- **C — restore-test from PBS**: `--selftest=restore-test` picked the newest PBS snapshot
+  (`source_tier:"pbs"` — derived from the storage type), restored → net-link-down → boot →
+  verified `running` → torn down, pass, no leak.
+- **D — forced failed verify**: corrupted a chunk in the datastore on DooPlex → first re-verify
+  still showed `ok` because PBS's `ignore-verified=true` default **skips** already-verified
+  snapshots. **Fix landed** (`Verify` now sends `ignore-verified=false`) → re-verify re-read the
+  chunk, detected the corruption: all snapshots `verify_state:"failed"`, agent logged
+  `ERROR … FAILED-verify snapshots`. Then cleaned up: forgot the corrupted snapshots, GC'd the
+  chunk, took a fresh backup, final verify → **ok**. Datastore left healthy.
 
-## Notes / cleanup
+Two gaps the runbook surfaced were fixed live (committed): the `ignore-verified` default (else
+corruption is never caught) and the hardcoded `source_tier` (now derived from the storage type).
 
-- The demo config (`/root/.config/felhom-agent/agent.json`) gained a `backup` block with the
-  cadence **disabled** (`restore_test_cadence_seconds: -1`); the validation used the selftest
-  harness. The example config ships cadence default-ON (24h) with the 990000-band + local-lvm.
-- One test backup of 9001 remains on `local` (a valid artifact; ~934 MiB of ~88 GiB free).
-- The binary on felhom-pve is v0.6.0-rc1; token/TLS pin per memory `demo-felhom-live-agent`.
+## Not done (flagged)
 
-## Deferred → Phase B
+- **End-to-end hub reflection of `pbs_snapshots`** (daemon → `hub.felhom.eu`): the hub side
+  (v0.7.3) is deployed and its parse + failed-verify WARN are unit-tested, but the live
+  daemon→hub round-trip needs the demo host **enrolled** on the operator hub (a per-host key) —
+  the same one-time enrollment deferred in slices 5/6A. **Offered, not done unprompted.**
 
-PBS: the datastore on the USB, zero-knowledge key custody, restore-from-PBS, and the PBS
-integrity-verify as the lighter frequent check. Then slice 7 (provisioning + identity-reset +
-golden base, §9).
+## End state / left in place
+
+PBS server on DooPlex + the `felhom-spike` datastore (one healthy, verified snapshot) + the
+N100's encrypted `felhom-pbs` storage — all left up. Agent on the N100 is **v0.6.0**; config
+reverted (`backup.local_backup_target=local`, pbs verify cadence disabled). No leftover guests.
+No secrets committed — the token secret + encryption key live only under
+`/etc/pve/priv/storage/` (0600), referenced by location/fingerprint.
+
+## Next
+
+Slice 7 — provisioning + identity-reset + the golden base (§9): the unified bring-up
+primitive; restore-overwrite + decommission executors the gate already guards; escrow +
+host-loss DR.