From 7eea638b923533090c33a5bed3d821595a72a559 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 9 Jun 2026 18:59:03 +0200 Subject: [PATCH] report update --- REPORT.md | 176 ++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 112 insertions(+), 64 deletions(-) diff --git a/REPORT.md b/REPORT.md index d3a60e7..4e1fb4e 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,81 +1,129 @@ -# REPORT — Slice 6 Phase B: PBS offsite tier (v0.6.0) (2026-06-09) +# REPORT — RUNBOOK: hub enrollment round-trip + failed-PBS-verify validation (2026-06-09) -> Overwrite-latest report (most recent significant work only). Cumulative history lives in [CHANGELOG.md](CHANGELOG.md). +> Overwrite-latest report (most recent significant work only). Cumulative history lives in +> [CHANGELOG.md](CHANGELOG.md). This was an **operational validation run** (RUNBOOK, no code +> change) against the **live, deployed v0.7.4 hub** — it closes the "End-to-end hub reflection" +> gap that the prior slice-6-Phase-B report flagged as *Not done*. ## Outcome -**Slice 6 is complete and live-validated; pushed as `v0.6.0`.** Phase B adds the PBS offsite -tier (doc 03 §8). The PBS spike made it small: backup-to-PBS and restore-from-PBS reuse Phase -A **unchanged** (PBS is just a storage target + a volid), and the operator token needs no -widening. The only new agent code is **the verify capability + a small PBS-API client + -PBSSnapshot reporting**. Escrow + host-loss DR remain slices 7/10. +**All runbook objectives met.** The demo host (`demo-felhom`) was enrolled on the deployed hub +for the first time, the daemon→hub round-trip was confirmed end-to-end on **v0.7.4**, and the +**failed-PBS-verify `[WARN]`** — the loudest offsite-DR signal — was driven live by a real +corrupted chunk in a dedicated throwaway datastore, with the encrypted production datastore +(`felhom-spike`) proven untouched (blast radius contained). Teardown left the system clean; per +explicit decision the demo host is **kept enrolled as a persistent systemd service**. -## What landed +Components exercised live: agent `felhom-agent` **v0.6.0** (daemon, PBS verify loop, host-report +collector) ↔ hub `felhom-hub` **v0.7.4** (host-report auth/ingest, storage/guests/pbs/restore +reflection, failed-verify WARN). -- **`internal/pbs` — the PBS-API client** (the agent's SECOND privileged external surface, - slice-1 discipline): TLS **fingerprint-pinned** to the PBS leaf cert (a wrong fingerprint → - rejected — unit-tested against a fake TLS server), **token auth** - (`PBSAPIToken=:`; id from the storage `username`, secret read at runtime from - `/etc/pve/priv/storage/.pw`, **never logged** — unit-asserted on the verify loop), typed, - no shell. `Verify` (POST, **`ignore-verified=false`** so it actually re-reads), `Snapshots`, - `TaskStatus`/`WaitVerify` (**node from the UPID** — the spike B4 gotcha), `NodeFromUPID`. -- **The verify maintenance loop** (`pbs/verify.go`) — the cheap, key-free, ciphertext-level - integrity check (§8) on its own cadence (default 6h, the 5th daemon goroutine). NOT - gated/journaled — maintenance/reporting like the slice-5 watchdog. Verify → poll → re-list → - record per-snapshot `verify_state`; a failed verify is logged loudly. -- **`PBSSnapshot` reporting** filled (namespace/type/id/time/size/owner/protected/`encrypted` - (from `files[].crypt-mode`)/`verify_state`/`verify_upid`). `PBSReporter` collector seam + - in-memory `SnapshotStore`. Cross-repo golden (byte-identical) + bidirectional key-set tests; - hub `handler.go` parses `pbs_snapshots` + logs a **failed-verify `[WARN]`** (deployed v0.7.3). -- **Truthful backup mode** — `Backup.mode` now reports the ACTUAL vzdump mode from the task - log (PVE downgrades snapshot→stop for a stopped guest). `proxmox.Storage.Username`. config - `PBSVerifyCadence`/`PBSSecretDir`. **`--selftest=pbs-verify`**. +## Phase 0 — pre-flight (all green) -## Tests +- **Image gate:** ArgoCD app `felhom` **Synced + Healthy**; hub Deployment image + `…/felhom-hub:v0.7.4`; pod log `[INFO] felhom-hub v0.7.4 starting`. All three agree. +- **Customer config:** `demo-felhom` exists in `customer_configs` (confirmed via a non-mutating + `GET /api/v1/config/demo-felhom` probe → `401 invalid password`, i.e. row present). +- **Existing enrollment:** `hosts` table empty (`count(*)=0`) → minted fresh. +- **Agent config:** at `/root/.config/felhom-agent/agent.json` (no systemd unit; launched + manually); **no `hub` section**; cadences `restore_test=-1`, `pbs_verify=-1` (both disabled); + `local_backup_target=local`; `pbs_secret_dir=/etc/pve/priv/storage`. Backed up to + `agent.json.pre-runbook`. Scratch guest **9999** (`felhom-selftest-scratch`, LXC) present. -`go test ./...` green; **`go test -race ./...` green on the build server** (the new verify -goroutine). PBS client: **fingerprint-pin enforcement** (wrong→rejected, right→connects), -token-header construction + **never-logged** (loop log captured, secret asserted absent), -`Verify` POST→UPID, `Snapshots`→`PBSSnapshot` (incl. crypt-mode→`encrypted`, absent -verification→`none`), `NodeFromUPID`, verify cadence (records ok/failed, disabled-by-negative), -mode-actual parse, cross-repo golden + hub ingest. +## Phase A — happy-path round-trip (CHECKPOINT GREEN) -## Live PBS runbook (reusing the spike's DooPlex PBS) +- **A.1 mint:** `POST /api/v1/admin/hosts` (global-key-gated, via the deployed hub) → **201**, + `host_id=demo-felhom-01`, 64-hex per-host key. Key transferred to the host out-of-band (stdin + pipe → 0600 file → atomic config patch), never echoed. +- **A.2 config:** added the `hub` block (`url`, `host_id`, `api_key`) + test cadences + (poll 60, pbs_verify 120, restore_test 300). +- **A.3 seed:** `vzdump 9999 → local` (mode downgraded snapshot→stop, expected for a stopped CT). +- **A.4 daemon + reflection:** daemon up (`version=0.6.0 host_id=demo-felhom-01 + hub_url=https://hub.felhom.eu`); all loops started. The **18:17:07** host-report reflected, + confirmed both in hub logs and the hub store (`host_reports.report_json`): + - ✅ **host-report auth + ingest** (slice 3) — `2xx`, no "Unknown host_id". + - ✅ **storage_targets** (slice 5) — 4 targets incl. `felhom-pbs`. + - ✅ **guests** reflected. + - ✅ **pbs_snapshots** (slice-6 Phase B) — `felhom-spike` ct/9001 `verify_state:"ok"`, + **`encrypted:true`** → the encrypted-storage verify path observed `ok` live. + - ✅ **restore_tests** (slice-6 Phase A) — recorded and reflected. + - `backups` empty (expected — no `RecordBackup` caller until slice 10). -- **A — backup → PBS**: `--selftest=backup -vmid 9001` → `felhom-pbs:backup/ct/9001/…`, - success, and **`mode:"stop"`** — the truthful-mode parse caught PVE's downgrade live. -- **B — verify → ok**: `--selftest=pbs-verify` triggered the verify, polled the task (node - `dooplex` from the UPID), re-listed → all snapshots `verify_state:"ok"`. -- **C — restore-test from PBS**: `--selftest=restore-test` picked the newest PBS snapshot - (`source_tier:"pbs"` — derived from the storage type), restored → net-link-down → boot → - verified `running` → torn down, pass, no leak. -- **D — forced failed verify**: corrupted a chunk in the datastore on DooPlex → first re-verify - still showed `ok` because PBS's `ignore-verified=true` default **skips** already-verified - snapshots. **Fix landed** (`Verify` now sends `ignore-verified=false`) → re-verify re-read the - chunk, detected the corruption: all snapshots `verify_state:"failed"`, agent logged - `ERROR … FAILED-verify snapshots`. Then cleaned up: forgot the corrupted snapshots, GC'd the - chunk, took a fresh backup, final verify → **ok**. Datastore left healthy. + **Finding (benign false-negative):** the scheduled restore-test reports **`pass:false`** + because the restored scratch guest's `vzstart` returns `exitstatus "WARNINGS: 1"` — + the warning is cosmetic: `WARN: Systemd 257 detected. You may need to enable nesting.` + The guest boots fine; the agent's strict `WaitTask` (`exitstatus == "OK"`) treats any + WARNING as failure. This means **every restore-test of a modern-distro (Debian 13 / + systemd 257) LXC will false-fail**, generating spurious DR alarms. It did, usefully, also + exercise the hub's `[WARN] restore-test FAILED` path. *(See "Follow-ups".)* -Two gaps the runbook surfaced were fixed live (committed): the `ignore-verified` default (else -corruption is never caught) and the hardcoded `source_tier` (now derived from the storage type). +## Phase C — destructive failed-verify (fenced; the slice-6-Phase-B closure) -## Not done (flagged) +A dedicated throwaway datastore `verifyfail` (`/mnt/5_hdd/pbs-verifyfail`, **unencrypted**, one +backup ⇒ every chunk exclusive) isolated the blast radius. `felhom-spike` and the homelab +`/mnt/5_hdd/backup/` tree were never touched. -- **End-to-end hub reflection of `pbs_snapshots`** (daemon → `hub.felhom.eu`): the hub side - (v0.7.3) is deployed and its parse + failed-verify WARN are unit-tested, but the live - daemon→hub round-trip needs the demo host **enrolled** on the operator hub (a per-host key) — - the same one-time enrollment deferred in slices 5/6A. **Offered, not done unprompted.** +- **C.1–C.3:** created `verifyfail` + `DatastoreAdmin` ACLs on **both** `felhom@pbs` and the + token `felhom@pbs!n100` (PBS privsep intersection); registered it as a PVE `pbs` storage on the + demo host (token authed → status **active**); put **exactly one** backup (`ct/9999`) into it. +- **C.4:** baseline verify **OK**, then corrupted one chunk + (`.chunks/008f/008f536d…b121dd`, first byte `0x31→0xCE`, **size unchanged** → true digest + mismatch with no size change, the realistic bit-rot case). +- **C.5.1 — agent local read path:** `--selftest=pbs-verify` → agent's `Verify` + (`ignore-verified=false` re-read) detected it: `datastore=verifyfail failed=1 total=1`, the + `ct/9999` `PBSSnapshot` shows `verify_state:"failed"`, while `felhom-spike` ct/9001 stays + `ok`. Confirms `report.go` passes `verification.state` through and `verify.go` re-lists/counts. +- **C.5.2 — live wire → hub WARN:** the running daemon's verify loop caught it + (`18:24:09 … FAILED-verify snapshots`), and the **18:32:07** host-report carried it. Hub logged: -## End state / left in place + ``` + 18:32:07 [WARN] host demo-felhom-01 PBS verify FAILED: ct/9999 ns=root owner=felhom@pbs!n100 + 18:32:07 [INFO] host-report from demo-felhom-01 (3 guests, 5 storage targets, 0 backups, + 1 restore-tests, 2 pbs-snapshots, 4928 bytes) + ``` -PBS server on DooPlex + the `felhom-spike` datastore (one healthy, verified snapshot) + the -N100's encrypted `felhom-pbs` storage — all left up. Agent on the N100 is **v0.6.0**; config -reverted (`backup.local_backup_target=local`, pbs verify cadence disabled). No leftover guests. -No secrets committed — the token secret + encryption key live only under -`/etc/pve/priv/storage/` (0600), referenced by location/fingerprint. + Hub store confirmed containment: `verifyfail` ct/9999 `encrypted:false verify_state:"failed"`, + `felhom-spike` ct/9001 `encrypted:true verify_state:"ok"`. **Real corrupted chunk → agent + verify loop → wire → hub failed-verify WARN, end to end, on v0.7.4.** -## Next +## Teardown (clean; demo host kept enrolled by decision) -Slice 7 — provisioning + identity-reset + the golden base (§9): the unified bring-up -primitive; restore-overwrite + decommission executors the gate already guards; escrow + -host-loss DR. +- Daemon stopped; `verifyfail` PVE storage + `verifyfail.pw` removed; datastore destroyed + (`--destroy-data true`), ACLs deleted, `/mnt/5_hdd/pbs-verifyfail` removed. +- **Verified as-found:** only `felhom-spike` datastore remains (`verify` → **TASK OK**); + `/mnt/5_hdd/backup/` tree byte-identical to the Phase-0 listing; no orphaned ACLs; seeded + `local` backup of 9999 removed; stray temp/key files cleaned. +- **Enrollment retained (explicit choice "keep enrolled as a service"):** `hub` block kept; + production cadences set (**poll 900**, **pbs_verify 6h**, **restore_test -1/disabled** to avoid + the benign-warning spam); installed + enabled **`/etc/systemd/system/felhom-agent.service`** + (active, enabled, survives reboot). First service report landed clean at **18:55:32** + (1 pbs-snapshot, **no WARN**). + +## What this proved (and didn't) + +- **Proved live on v0.7.4:** slice-3 host-report auth + ingest; slice-5 storage_targets + reflection; slice-6-Phase-B `pbs_snapshots` reflection **including the failed-verify `[WARN]`** + via a real corrupted chunk; slice-6-Phase-A `restore_tests` reflection; the encrypted-storage + verify path observed `ok` on `felhom-spike`. +- **Did not prove:** `backups` reflection (no daemon `RecordBackup` caller — golden-only until + slice 10); the corruption test datastore was unencrypted (the verify *mechanism* is identical + server-side; the encrypted config path is exercised by the normal `felhom-spike` verify). + +## Follow-ups surfaced + +1. **Restore-test false-fail on benign LXC start warnings** (new, notable): the agent treats + `vzstart` `WARNINGS: n` as failure, so modern-distro LXC restore-tests always report + `pass:false`. Options: treat `WARNINGS` distinct from non-OK, set scratch-guest + `features: nesting=1` on restore, or whitelist the systemd-257 warning. Left **disabled** on + the demo host as a stopgap. +2. **`POST /api/v1/admin/hosts` is global-key-gated only** and reachable via the public hub URL — + anyone with the global key can mint a host. Lock down at the slice 7–8 auth cutover (doc 05 §11). +3. **Hub hardcodes `defaultHostPollSeconds = 900`** (`handler.go`) and the agent adopts it, + silently overriding the configured `poll_seconds` — worth making configurable / documenting. +4. **`backups` producer** still absent (accept it lands with slice-10 desired-state). + +## No secrets committed + +The hub global key, the per-host key, and the PBS token secret are referenced by location only +(hub ConfigMap `hub-config`; `/root/.config/felhom-agent/agent.json` 0600; +`/etc/pve/priv/storage/*.pw` 0600) — never echoed into logs or this report.