From 697e67c85eb26fe67f79886753cb4670ce9b7fcc Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 9 Jun 2026 17:03:00 +0200 Subject: [PATCH] =?UTF-8?q?docs(REPORT):=20slice=206=20Phase=20B=20?= =?UTF-8?q?=E2=80=94=20PBS=20offsite=20tier,=20live-validated=20on=20DooPl?= =?UTF-8?q?ex=20PBS?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Live: backup→PBS (actual mode 'stop' caught), verify→ok (node from UPID), restore-test from PBS (source_tier=pbs), and a forced failed-verify (corrupt chunk → ignore-verified=false re-read → verify_state=failed → cleaned up + re-verified ok). Hub e2e reflection flagged (needs host enrollment). Datastore left healthy. Co-Authored-By: Claude Opus 4.8 (1M context) --- REPORT.md | 125 ++++++++++++++++++++++++++++-------------------------- 1 file changed, 64 insertions(+), 61 deletions(-) diff --git a/REPORT.md b/REPORT.md index 230ee80..d3a60e7 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,78 +1,81 @@ -# REPORT — Slice 6 Phase A: backup + the self-restore-test (v0.6.0-rc1) (2026-06-09) +# REPORT — Slice 6 Phase B: PBS offsite tier (v0.6.0) (2026-06-09) > Overwrite-latest report (most recent significant work only). Cumulative history lives in [CHANGELOG.md](CHANGELOG.md). ## Outcome -**Slice 6 Phase A is implemented, live-validated on `demo-felhom`, and pushed as -`v0.6.0-rc1`** — the guest-level backup layer + the journaled **self-restore-test** that -closes "a backup you haven't restored isn't a backup" (doc 03 §8). Everything is **benign** -(backup, restore-to-NEW, scratch teardown): it reuses the slice-4 classifier/gate/journal — -no new destructive class, no new crypto. **Local target only**; PBS is Phase B. Restore is -to a **NEW** guest only (no overwrite). Backups are **crash-consistent** (marked so). This is -the hard checkpoint: awaiting validation before Phase B. +**Slice 6 is complete and live-validated; pushed as `v0.6.0`.** Phase B adds the PBS offsite +tier (doc 03 §8). The PBS spike made it small: backup-to-PBS and restore-from-PBS reuse Phase +A **unchanged** (PBS is just a storage target + a volid), and the operator token needs no +widening. The only new agent code is **the verify capability + a small PBS-API client + +PBSSnapshot reporting**. Escrow + host-loss DR remain slices 7/10. ## What landed -- **proxmox**: `DestroyLXC` (the scratch-teardown primitive), `VzdumpOptions.Notes` → - `notes-template`, `LatestBackupVolID` (resolve a produced archive — the task status carries - no result volid). -- **The self-restore-test** (`reconcile.Engine.RunRestoreTest`): pick a free scratch VMID - (configured band, excludes 9999; full band → skip) → **journal a `Scratch`-owned entry - BEFORE any mutation** → restore-to-new → benign net **link-down** SetConfig (so the clone - can't conflict with a running source's MAC/IP — test-safety, NOT slice-7 identity reset) → - boot → verify **reaches `running`** → **defer teardown that ALWAYS runs** (benign - `ClassGuestDestroy` + agent-tagged-scratch provenance, through the gate). -- **Crash-safe recovery** (`Recover`): a `Scratch` entry is resolved by **teardown**, not by - re-checking the restore sub-task's UPID — special-cased before the generic path (else a - leaked guest would be marked "succeeded"). Idempotent: already-gone → clean; list-unreadable - → left in-flight. -- **`internal/backup`**: `BackupRunner` (vzdump + archive/size resolve + the **bulk-volume - gap** — a mountpoint is uncovered unless it carries an explicit `backup=1`, so an unset - `backup=` counts as uncovered, the safe DR direction), `PickRestoreCandidate`, an in-memory - `Store` (latest backup-per-target + latest restore-test) feeding the hub reporter seams, and - a cadence `Scheduler` (default 24h; the fourth daemon goroutine; disabled cleanly when - off/misconfigured). -- **hub report**: `Backup` + `RestoreTest` filled (`PBSSnapshot` stays a Phase-B stub); - collector seams; cross-repo golden byte-identical + bidirectional key-set tests in both - repos; hub `handler.go` parses + persists them and **logs a FAILED restore-test - prominently** (the loudest DR signal). -- **config** `BackupConfig` (band 990000–990009 default); **`--selftest=backup`** and - **`--selftest=restore-test`** (Recover-then restore→boot→verify→teardown). +- **`internal/pbs` — the PBS-API client** (the agent's SECOND privileged external surface, + slice-1 discipline): TLS **fingerprint-pinned** to the PBS leaf cert (a wrong fingerprint → + rejected — unit-tested against a fake TLS server), **token auth** + (`PBSAPIToken=:`; id from the storage `username`, secret read at runtime from + `/etc/pve/priv/storage/.pw`, **never logged** — unit-asserted on the verify loop), typed, + no shell. `Verify` (POST, **`ignore-verified=false`** so it actually re-reads), `Snapshots`, + `TaskStatus`/`WaitVerify` (**node from the UPID** — the spike B4 gotcha), `NodeFromUPID`. +- **The verify maintenance loop** (`pbs/verify.go`) — the cheap, key-free, ciphertext-level + integrity check (§8) on its own cadence (default 6h, the 5th daemon goroutine). NOT + gated/journaled — maintenance/reporting like the slice-5 watchdog. Verify → poll → re-list → + record per-snapshot `verify_state`; a failed verify is logged loudly. +- **`PBSSnapshot` reporting** filled (namespace/type/id/time/size/owner/protected/`encrypted` + (from `files[].crypt-mode`)/`verify_state`/`verify_upid`). `PBSReporter` collector seam + + in-memory `SnapshotStore`. Cross-repo golden (byte-identical) + bidirectional key-set tests; + hub `handler.go` parses `pbs_snapshots` + logs a **failed-verify `[WARN]`** (deployed v0.7.3). +- **Truthful backup mode** — `Backup.mode` now reports the ACTUAL vzdump mode from the task + log (PVE downgrades snapshot→stop for a stopped guest). `proxmox.Storage.Username`. config + `PBSVerifyCadence`/`PBSSecretDir`. **`--selftest=pbs-verify`**. ## Tests -`go test ./...` green; **`go test -race ./...` green on the build server** (the scheduler -goroutine). Notable unit tests: vzdump async UPID→WaitTask + bulk-gap (`backup!=1` incl. -unset); restore-test pass + **teardown-on-failed-verify** + restore-failure-still-tears-down + -full-band-skip; the DELETE wire contract (no body + force=1); and the headline **recover- -leaked-scratch** trio (destroyed / already-gone-clean / list-unreadable-left-in-flight). +`go test ./...` green; **`go test -race ./...` green on the build server** (the new verify +goroutine). PBS client: **fingerprint-pin enforcement** (wrong→rejected, right→connects), +token-header construction + **never-logged** (loop log captured, secret asserted absent), +`Verify` POST→UPID, `Snapshots`→`PBSSnapshot` (incl. crypt-mode→`encrypted`, absent +verification→`none`), `NodeFromUPID`, verify cadence (records ok/failed, disabled-by-negative), +mode-actual parse, cross-repo golden + hub ingest. -## Live validation on demo-felhom +## Live PBS runbook (reusing the spike's DooPlex PBS) -- **`--selftest=backup -vmid 9001`** → vzdump of the stopped spike-lxc to `local`: success, - **crash-consistent**, archive `local:backup/vzdump-lxc-9001-…tar.zst` (~934 MiB) resolved, - 26 s. Bulk gap empty (9001 has no extra mountpoints). -- **`--selftest=restore-test`** → restored that backup into scratch **990000** on local-lvm, - net link-down, booted, verified `running`, torn down (gate-benign) — **pass, 15 s, no leak**. -- **Crash-recovery proven for real**: the first teardown hit a `DestroyLXC` bug (DELETE with - a form body → PVE HTTP 501; a booted guest also needs `force=1`), which **leaked scratch - 990000**. The fix moved the flags to the query string + `force=1`, **no body** (regression- - tested). On the next run, `Recover` found the journaled in-flight `Scratch` entry, confirmed - 990000 existed, and **destroyed the leaked guest** (`ScratchDestroyed:1`) through the benign - gate — then ran a clean test. `pct list` after: **no 99000x guests**. The accidental leak + - reap is the best possible live proof of the slice's load-bearing crash-safety path. +- **A — backup → PBS**: `--selftest=backup -vmid 9001` → `felhom-pbs:backup/ct/9001/…`, + success, and **`mode:"stop"`** — the truthful-mode parse caught PVE's downgrade live. +- **B — verify → ok**: `--selftest=pbs-verify` triggered the verify, polled the task (node + `dooplex` from the UPID), re-listed → all snapshots `verify_state:"ok"`. +- **C — restore-test from PBS**: `--selftest=restore-test` picked the newest PBS snapshot + (`source_tier:"pbs"` — derived from the storage type), restored → net-link-down → boot → + verified `running` → torn down, pass, no leak. +- **D — forced failed verify**: corrupted a chunk in the datastore on DooPlex → first re-verify + still showed `ok` because PBS's `ignore-verified=true` default **skips** already-verified + snapshots. **Fix landed** (`Verify` now sends `ignore-verified=false`) → re-verify re-read the + chunk, detected the corruption: all snapshots `verify_state:"failed"`, agent logged + `ERROR … FAILED-verify snapshots`. Then cleaned up: forgot the corrupted snapshots, GC'd the + chunk, took a fresh backup, final verify → **ok**. Datastore left healthy. -## Notes / cleanup +Two gaps the runbook surfaced were fixed live (committed): the `ignore-verified` default (else +corruption is never caught) and the hardcoded `source_tier` (now derived from the storage type). -- The demo config (`/root/.config/felhom-agent/agent.json`) gained a `backup` block with the - cadence **disabled** (`restore_test_cadence_seconds: -1`); the validation used the selftest - harness. The example config ships cadence default-ON (24h) with the 990000-band + local-lvm. -- One test backup of 9001 remains on `local` (a valid artifact; ~934 MiB of ~88 GiB free). -- The binary on felhom-pve is v0.6.0-rc1; token/TLS pin per memory `demo-felhom-live-agent`. +## Not done (flagged) -## Deferred → Phase B +- **End-to-end hub reflection of `pbs_snapshots`** (daemon → `hub.felhom.eu`): the hub side + (v0.7.3) is deployed and its parse + failed-verify WARN are unit-tested, but the live + daemon→hub round-trip needs the demo host **enrolled** on the operator hub (a per-host key) — + the same one-time enrollment deferred in slices 5/6A. **Offered, not done unprompted.** -PBS: the datastore on the USB, zero-knowledge key custody, restore-from-PBS, and the PBS -integrity-verify as the lighter frequent check. Then slice 7 (provisioning + identity-reset + -golden base, §9). +## End state / left in place + +PBS server on DooPlex + the `felhom-spike` datastore (one healthy, verified snapshot) + the +N100's encrypted `felhom-pbs` storage — all left up. Agent on the N100 is **v0.6.0**; config +reverted (`backup.local_backup_target=local`, pbs verify cadence disabled). No leftover guests. +No secrets committed — the token secret + encryption key live only under +`/etc/pve/priv/storage/` (0600), referenced by location/fingerprint. + +## Next + +Slice 7 — provisioning + identity-reset + the golden base (§9): the unified bring-up +primitive; restore-overwrite + decommission executors the gate already guards; escrow + +host-loss DR.