admin
697e67c85e
Live: backup→PBS (actual mode 'stop' caught), verify→ok (node from UPID), restore-test from PBS (source_tier=pbs), and a forced failed-verify (corrupt chunk → ignore-verified=false re-read → verify_state=failed → cleaned up + re-verified ok). Hub e2e reflection flagged (needs host enrollment). Datastore left healthy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
82 lines
5.1 KiB
Markdown
82 lines
5.1 KiB
Markdown
# REPORT — Slice 6 Phase B: PBS offsite tier (v0.6.0) (2026-06-09)
|
|
|
|
> Overwrite-latest report (most recent significant work only). Cumulative history lives in [CHANGELOG.md](CHANGELOG.md).
|
|
|
|
## Outcome
|
|
|
|
**Slice 6 is complete and live-validated; pushed as `v0.6.0`.** Phase B adds the PBS offsite
|
|
tier (doc 03 §8). The PBS spike made it small: backup-to-PBS and restore-from-PBS reuse Phase
|
|
A **unchanged** (PBS is just a storage target + a volid), and the operator token needs no
|
|
widening. The only new agent code is **the verify capability + a small PBS-API client +
|
|
PBSSnapshot reporting**. Escrow + host-loss DR remain slices 7/10.
|
|
|
|
## What landed
|
|
|
|
- **`internal/pbs` — the PBS-API client** (the agent's SECOND privileged external surface,
|
|
slice-1 discipline): TLS **fingerprint-pinned** to the PBS leaf cert (a wrong fingerprint →
|
|
rejected — unit-tested against a fake TLS server), **token auth**
|
|
(`PBSAPIToken=<id>:<secret>`; id from the storage `username`, secret read at runtime from
|
|
`/etc/pve/priv/storage/<id>.pw`, **never logged** — unit-asserted on the verify loop), typed,
|
|
no shell. `Verify` (POST, **`ignore-verified=false`** so it actually re-reads), `Snapshots`,
|
|
`TaskStatus`/`WaitVerify` (**node from the UPID** — the spike B4 gotcha), `NodeFromUPID`.
|
|
- **The verify maintenance loop** (`pbs/verify.go`) — the cheap, key-free, ciphertext-level
|
|
integrity check (§8) on its own cadence (default 6h, the 5th daemon goroutine). NOT
|
|
gated/journaled — maintenance/reporting like the slice-5 watchdog. Verify → poll → re-list →
|
|
record per-snapshot `verify_state`; a failed verify is logged loudly.
|
|
- **`PBSSnapshot` reporting** filled (namespace/type/id/time/size/owner/protected/`encrypted`
|
|
(from `files[].crypt-mode`)/`verify_state`/`verify_upid`). `PBSReporter` collector seam +
|
|
in-memory `SnapshotStore`. Cross-repo golden (byte-identical) + bidirectional key-set tests;
|
|
hub `handler.go` parses `pbs_snapshots` + logs a **failed-verify `[WARN]`** (deployed v0.7.3).
|
|
- **Truthful backup mode** — `Backup.mode` now reports the ACTUAL vzdump mode from the task
|
|
log (PVE downgrades snapshot→stop for a stopped guest). `proxmox.Storage.Username`. config
|
|
`PBSVerifyCadence`/`PBSSecretDir`. **`--selftest=pbs-verify`**.
|
|
|
|
## Tests
|
|
|
|
`go test ./...` green; **`go test -race ./...` green on the build server** (the new verify
|
|
goroutine). PBS client: **fingerprint-pin enforcement** (wrong→rejected, right→connects),
|
|
token-header construction + **never-logged** (loop log captured, secret asserted absent),
|
|
`Verify` POST→UPID, `Snapshots`→`PBSSnapshot` (incl. crypt-mode→`encrypted`, absent
|
|
verification→`none`), `NodeFromUPID`, verify cadence (records ok/failed, disabled-by-negative),
|
|
mode-actual parse, cross-repo golden + hub ingest.
|
|
|
|
## Live PBS runbook (reusing the spike's DooPlex PBS)
|
|
|
|
- **A — backup → PBS**: `--selftest=backup -vmid 9001` → `felhom-pbs:backup/ct/9001/…`,
|
|
success, and **`mode:"stop"`** — the truthful-mode parse caught PVE's downgrade live.
|
|
- **B — verify → ok**: `--selftest=pbs-verify` triggered the verify, polled the task (node
|
|
`dooplex` from the UPID), re-listed → all snapshots `verify_state:"ok"`.
|
|
- **C — restore-test from PBS**: `--selftest=restore-test` picked the newest PBS snapshot
|
|
(`source_tier:"pbs"` — derived from the storage type), restored → net-link-down → boot →
|
|
verified `running` → torn down, pass, no leak.
|
|
- **D — forced failed verify**: corrupted a chunk in the datastore on DooPlex → first re-verify
|
|
still showed `ok` because PBS's `ignore-verified=true` default **skips** already-verified
|
|
snapshots. **Fix landed** (`Verify` now sends `ignore-verified=false`) → re-verify re-read the
|
|
chunk, detected the corruption: all snapshots `verify_state:"failed"`, agent logged
|
|
`ERROR … FAILED-verify snapshots`. Then cleaned up: forgot the corrupted snapshots, GC'd the
|
|
chunk, took a fresh backup, final verify → **ok**. Datastore left healthy.
|
|
|
|
Two gaps the runbook surfaced were fixed live (committed): the `ignore-verified` default (else
|
|
corruption is never caught) and the hardcoded `source_tier` (now derived from the storage type).
|
|
|
|
## Not done (flagged)
|
|
|
|
- **End-to-end hub reflection of `pbs_snapshots`** (daemon → `hub.felhom.eu`): the hub side
|
|
(v0.7.3) is deployed and its parse + failed-verify WARN are unit-tested, but the live
|
|
daemon→hub round-trip needs the demo host **enrolled** on the operator hub (a per-host key) —
|
|
the same one-time enrollment deferred in slices 5/6A. **Offered, not done unprompted.**
|
|
|
|
## End state / left in place
|
|
|
|
PBS server on DooPlex + the `felhom-spike` datastore (one healthy, verified snapshot) + the
|
|
N100's encrypted `felhom-pbs` storage — all left up. Agent on the N100 is **v0.6.0**; config
|
|
reverted (`backup.local_backup_target=local`, pbs verify cadence disabled). No leftover guests.
|
|
No secrets committed — the token secret + encryption key live only under
|
|
`/etc/pve/priv/storage/` (0600), referenced by location/fingerprint.
|
|
|
|
## Next
|
|
|
|
Slice 7 — provisioning + identity-reset + the golden base (§9): the unified bring-up
|
|
primitive; restore-overwrite + decommission executors the gate already guards; escrow +
|
|
host-loss DR.
|