diff --git a/CHANGELOG.md b/CHANGELOG.md index d82a817..903d7a7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,34 @@ +## v0.76.0 — restore-test full-fidelity verification (GL-5b / go-live G12) (2026-07-08) + +Closes GL-5 finding #2's mirror image: the restore-test's live-source-config bind-override path +tripped the same PVE drop-unlisted-mountpoints rule DR did, so it boot-verified scratch guests +WITHOUT their storage mpN — weaker verification than it claimed, and GL-6 will lean on it. +**0.75.0 was superseded unpublished; this is the Day-0 manifest bump target.** + +- **Archive-derived params:** the restore-test now derives its restore params exactly like DR + bring-up — `ExtractArchiveConfig` + `drRestoreOverrides` (the ARCHIVE is the object under test, + not any live guest's current config): rootfs explicit, every storage mpN passed through (its + content genuinely extracted — full fidelity; the added runtime IS the verification), structural + binds → throwaway stand-ins. Unreadable archive config / unknown bind topology → refuse UP FRONT + (never restore a partial guest to "verify" it). +- **Mount-parity assert (the non-hollow core):** pre-start, the restored scratch's mpN set is + compared against the archive's — a missing, mispathed, undersized, or extra mpN FAILS the test + naming the delta. PVE rule (b) can never regress into a green light again. `MountParity` + ("ok"|"mismatch") + `MountInventory` ride the result and the hub wire record (additive keys — an + older hub ignores them). +- Dead code deleted with its tests: `bindMountOverrides`, `archiveVMID` + the volid regexes (the + restore-test was their only caller; a reachable dead lookalike is how the next bug happens). + `throwawayVolumeOverride`/`rootfsSizeGB`/`mountPathOf` live on under `drRestoreOverrides`. +- DR bring-up, 4d, KeepMAC, grows, caps: untouched (bringup.go had zero line changes — the §13 + DR re-run trigger did not fire). +- Tests: Scenario A (exact 5-param derivation from the archive + parity ok + inventory + teardown), + B (dropped mp0 → FAIL naming it, never started, still torn down; RED-PROOF: with the parity + assert removed the run passes silently — run→fail→revert), C (extract-failure + unknown-topology + refusals, no restore attempted), D (provision/DR bring-up tests all green, byte-untouched); + `mountParity` pure-function matrix (parity/missing/mispathed/undersized/extra/trivial). +- Live validation + publish: REPORT.md (full-fidelity runtime measured vs the ~3m data-less shape; + `AGENT_VERSION`/`AGENT_SHA256` recorded verbatim for the operator's manifest bump). + ## v0.75.0 — DR bring-up structural bind overrides + real-bind swap (GL-5 / go-live G8) (2026-07-08) Implements the verdict of `felhom.eu/documentation/audits/SPIKE-dr-bindmount-source-2026-07-07.md`: diff --git a/CONTEXT.md b/CONTEXT.md index 7dcf37c..2fda20a 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -5,6 +5,16 @@ ## Current +- **v0.76.0** (2026-07-08, LIVE on felhom-pve + **PUBLISHED sha `9828c5f7…f50b`** — THE Day-0 + manifest bump target; **0.75.0 superseded unpublished**) — **GL-5b / G12: restore-test + full-fidelity**. Params derive from the ARCHIVE's embedded config (`drRestoreOverrides`, same as + DR — the old live-source-config path verified the wrong object AND dropped storage mpN per PVE's + all-or-nothing rule; deleted with `bindMountOverrides`/`archiveVMID`). NEW mount-parity assert + (restored mpN vs archive; miss/mispath/undersize/extra = FAIL naming the delta) + `MountParity`/ + `MountInventory` on the wire record (additive). Live-proven: scratch 990000 ← 6.5GB 9201 archive, + parity ok, inventory mp0 200G+mp1 50G+2 throwaways, **3m4s local tier** (cheaper than feared); + rotated-out archive volid → clean up-front refusal (nice failure mode). bringup.go untouched. + - **v0.75.0** (2026-07-08, LIVE on felhom-pve) — **GL-5 / go-live G8: guest-loss DR bring-up actually restores** (closes the v0.74.0 OPEN item + SPIKE-dr-bindmount-source §8). DR passes the COMPLETE explicit restore param set derived from the archive's embedded config (NEW diff --git a/REPORT.md b/REPORT.md index f5c575a..7b66815 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,101 +1,90 @@ -# REPORT — v0.75.0: DR bring-up structural bind overrides + real-bind swap (GL-5 / G8) +# REPORT — v0.76.0: restore-test full-fidelity verification (GL-5b / G12) + publish -**Date:** 2026-07-08 · **Class:** risky/supervised implementation (agent) + one constrained live -validation. **Baseline:** `main` @ `4c408467` (v0.74.0) → `b3446213` + docs. Spec: GL-5 (go-live -G8), verdict of `felhom.eu/documentation/audits/SPIKE-dr-bindmount-source-2026-07-07.md`. -STOP honored: no restore into 9201, no `-keep`, no PBS-tier writes; guest 9201 untouched throughout. +**Date:** 2026-07-08 · **Class:** risky/supervised implementation + live run + deploy + publish. +**Baseline:** `main` @ `5a72a4b5` (v0.75.0, live) → `b1697874` + docs. Closes GL-5 finding #2's +mirror image (REPORT v0.75.0 observation: the restore-test boot-verified scratch guests WITHOUT +their storage mpN). STOP honored: no 9201 restore target, no `-keep`, no PBS writes, no manifest +change (operator's). + +## The operator values (Day-0 manifest — Configs → Day-0 artifacts) + +``` +AGENT_VERSION=0.76.0 +AGENT_SHA256=9828c5f75e719fb2e1fc3523f9c322b595a963ec1dcdf37ca42a96bedab7f50b +GOLDEN_VERSION=0.103.0 +GOLDEN_SHA256=8481e8a14e2aa0abe3831cc40e5aea4c32f2017e7561c15dca8a674df6a6026e +``` + +**0.75.0 was superseded UNPUBLISHED** (deliberate — one manifest bump, straight to 0.76.0). The +golden stays 0.103.0 (it ships the controller, not the agent — no re-bake). ## What shipped -1. **DR restore overrides** (`bringup.go`): `ModeDRGuestLoss` builds the restore params via - `drRestoreOverrides` from the archive's own embedded config; `ModeProvision` passes nil — - the regression contract (test + red-proof). -2. **Step 4d real-bind swap** after the grows, before start: mp9 host dir (`os.MkdirAll`, agent-owned, - idempotent — a same-host guest-loss still has bootstrap.json there, untouched), `mkdir -p` parent - dir + two `pct set` calls via the host runner (root-only bind mounts; sudoers lines 39 + 107 - already allowlist both shapes), one slot per call so C2 failures name the exact mpN; then the - displaced throwaways (`unusedN`) deleted in one config PUT. A scoped-token delete refusal logs - loudly + warns in the result — privileges never widened. Rollback envelope respected (C2 test: - mid-swap failure → compensating destroy, guest never started). -3. **Engine seam**: `EngineOptions.HostRunner` + `StateDir`; DR refuses up front on an API-only - engine. The bring-up selftest wires the back-half's ExecRunner shape and removes the scratch - vmid's mp9 host dir at teardown (§7 edge — never a real drive's bind source). -4. `proxmox.GuestConfig.Unused()` + `Client.ExtractArchiveConfig`. +- Restore-test params now derive from the **archive's own embedded config** via + `ExtractArchiveConfig` + `drRestoreOverrides` — the same derivation DR bring-up uses, because the + ARCHIVE is the object under test (the old path read the live source guest's config: wrong object, + and its bind-only overrides tripped PVE's drop-unlisted-mountpoints rule). Unreadable config / + unknown bind topology → refuse up front, no partial-guest "verification". +- **Mount-parity assert** (step 2b, pre-start): restored mpN set vs the archive's — missing, + mispathed, undersized, or extra mpN fails the test naming the delta. `MountParity` + + `MountInventory` added to the result and the hub wire record (additive JSON keys; older hubs + ignore them). Teardown still fires on a parity fail (launch-proven). +- Dead code deleted WITH its tests: `bindMountOverrides`, `archiveVMID`, the two volid regexes — + the restore-test was their only caller (§5's delete-or-justify: deleted; no dead lookalike stays + reachable). `throwawayVolumeOverride`/`rootfsSizeGB`/`mountPathOf` live on under + `drRestoreOverrides`. +- **Scenario D:** `bringup.go` had ZERO line changes — provision/DR behavior byte-identical, all + GL-5 tests green, the §13 DR re-run trigger did not fire. -## Two LIVE-DISCOVERED PVE constraints (neither in the spike — it never ran an override restore) +## Tests -PVE's explicit-params restore is **all-or-nothing**: +Scenario A (exact 5-param derivation from a 9201-shaped archive config; parity ok; 4-entry +inventory; scratch torn down), B (mp0 deleted from the restored config → FAIL naming `mp0 MISSING`, +guest never started, still torn down), C (extract-failure + unknown-topology refusals with zero +restore attempts), `mountParity` pure-function matrix (parity / missing / mispathed / undersized / +extra / trivially-empty). **Scenario-B red-proof** (run→fail→revert on the committed baseline): +with the 2b parity assert removed, the dropped-mount run PASSES silently — the test caught it +(FAIL), then reverted, all green. Full gate on Windows + 180/Linux: 23 packages ok. -- **(a)** any mpN param without an explicit `rootfs` → HTTP 500 `mount points configured, but - 'rootfs' not set` (the constraint restoretest.go:211 documents for the live-config path). The - first live run died here. -- **(b)** mountpoints NOT named in the params are **silently dropped**: the second live run came up - `boot+running` in 2m56s — *without mp0/mp1*. A DR guest without its Docker-data/user-data volumes - is a data-loss restore that LOOKS green (liveness alone cannot catch it). +## Live validation (felhom-pve) -Resolution: derive the COMPLETE param set from the archive's embedded config via -`ExtractArchiveConfig` — probed live first: **HTTP 200 under the scoped agent token** (the PVE -server holds the PBS key; the spike's candidate-1 rejection — never hand the agent key material — -holds). The spec's known-constants rule still governs the BIND layout (constants + an unknown-bind -refusal); the archive config supplies sizes/paths for rootfs + storage mpN pass-through. §12's -"no PBS blobs for the mount layout" is honored in spirit and letter — candidate 1 (agent-side -PBS-key blob read) stays rejected. +Deploy: backup `felhom-agent.bak-0.75.0` → install → restart: active, v0.76.0, zero errors, no +ce9d rebind. First run FAILED honestly — the GL-5-era archive had been rotated out by retention +and `ExtractArchiveConfig` refused up front ("volume does not exist"; exactly the fail-fast the +design wants). Re-run against the current newest archive +(`vzdump-lxc-9201-2026_07_08-09_38_15.tar.zst`, 6.5GB): -Deviation from §5 as specced: `bindMountOverrides` is not called (its is-a-bind filter reads live -configs and broke on non-Linux test runners for synthesized input); its FORMAT was extracted into -`throwawayVolumeOverride` — one source of the override format, used by both callers. Restore-test -behavior untouched. +- transcript: `full-fidelity restore params derived from the archive config … params=5`; +- pre-teardown `pct config 990000`: `mp0 …,mp=/var/lib/docker,backup=1,size=200G`, + `mp1 …,mp=/mnt/sys_drive,backup=1,size=50G`, `mp8`/`mp9` as 1G throwaways at the structural + paths, `rootfs …,size=32G`, no unusedN; +- record: `pass: true`, `mount_parity: "ok"`, `mount_inventory`: the 4 verified mpN, + `duration_seconds: 184.28`; +- scratch torn down; band clean. -## Tests (all green: `go build`, `go vet`, `go test ./...` — 23 pkgs, Windows + 180/Linux) +**Runtime, honestly:** 3m4s total on the LOCAL tier — cheaper than the ~7m the GL-5 DR contrast +suggested (that 7m23s included DR bring-up's longer boot/verify chain; the extraction cost itself +for this guest's actual content is ~2 extra minutes over a data-less restore). The PBS tier will +be slower (WAN + decrypt); its generous `RestoreTaskTimeout` already exists spec-side. No fidelity +knob added per §12 — the runtime is the verification. -Scenario A (exact 3-way overrides incl. rootfs from the extracted config + exact swap commands + -mp9 dir + `delete=unused0,unused1`); B provision-nil + runner-untouched; C2 mid-swap rollback -(error names mp9 + "mp8 already landed", destroy fired, never started); C3 older-archive-without-mp9 -(constants regardless, one unused deleted); DR-without-runner refusal (before any restore); -extract-failure refusal; 403-residue warn-not-fail; `drRestoreOverrides` (full 9201-shaped config → -5 exact params; unknown-bind / no-rootfs / sizeless-mpN refusals; snapshot sections never shadow). -**Red-proofs** (mutate → run → confirm FAIL → restore, against the committed baseline): RP-A -override synthesis removed → Scenario A FAILS; RP-B unconditional overrides → B FAILS. +## Publish (GL-1 pattern) -## Live validation (campaign-2 precedent; the §13 sequence) +The exact deployed bytes (`readlink /proc/$(pidof felhom-agent)/exe` → `/usr/local/bin/felhom-agent`, +`--version` = 0.76.0) published via `publish-agent.sh 0.76.0`: PUT 201, round-trip sha match, then +an ANONYMOUS fetch re-verify — sha identical end-to-end +(`9828c5f7…f50b` on the box, on 180, and via the anonymous customer-path proxy). Creds sourced on +the build server, never echoed. -Deployed to felhom-pve (backup `felhom-agent.bak-0.74.0`, install, restart; clean start, no drive -rebind of the ejected ce9d drive, zero errors). Then -`--selftest=bring-up -mode dr -archive local:backup/vzdump-lxc-9201-2026_07_07-18_57_34.tar.zst --vmid 9310` (scratch band, no `-keep`, auto-teardown): +## Observations -- **On v0.74.0 this exact op FAILED** at the restore POST (constraint (a) above; pre-fix it failed - with `restoring 'mp8' to bind mount is only possible for root` — the spike's finding). -- **v0.75.0 final run: PASS in 7m23s** (vs 2m56s for the data-less intermediate build — the - mp0/mp1 content extraction is real). Pre-teardown `pct config 9310`: - `mp0: local-lvm:vm-9310-disk-1,mp=/var/lib/docker,backup=1,size=200G`, - `mp1: …,mp=/mnt/sys_drive,backup=1,size=50G`, - `mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives`, - `mp9: /var/lib/felhom-agent/guests/9310/bootstrap,mp=/etc/felhom-bootstrap,ro=1`, - `rootfs: …,size=32G`, **zero unusedN** (agent log: `displaced throwaway volumes deleted - unused=[unused0 unused1]`). Boot+running, pool re-asserted, teardown clean, - `/var/lib/felhom-agent/guests/` back to `9201` only. +- The pinned-archive rotation (first live run) is a nice property in production: a restore-test + against a pruned volid refuses loudly instead of "verifying" nothing. The daemon's scheduled + restore-test resolves the newest archive itself, so cadence runs are unaffected. +- 9201's local backups ran 3× this morning (08:52/09:07/09:38) — the local cadence is busy; not a + GL-5b concern, worth a glance at retention math someday. -## The unusedN mechanism (spec Part 2 asked what was found) +## NOT live-validated -Replacing an mpN via `pct set` parks the displaced volume as `unusedN` in the config; deleting is a -config PUT `delete=unusedN` (destroys the volume). **Works under the scoped token** (VM.Config.Disk -+ Datastore.Allocate on the restore storage) — no 403, no privilege gap; the warn-not-fail fallback -exists but did not fire. - -## Observations (documented, not acted on) - -- The DR **selftest** hardcodes `KeepMAC=true`, so a scratch DR while the source guest is LIVE - briefly duplicates its MAC on the bridge (BC:24:11:A4:21:6C ran twice for ~5 min). Pre-existing, - supervised-only surface; candidate: a `-keep-mac=false` selftest flag. -- The **restore-test** has the same constraint-(b) exposure in reverse: it passes rootfs + bind - overrides only, so its scratch guests boot WITHOUT mp0/mp1 — its boot-verify is weaker than it - looks (it verifies the OS volume restores, not the data volumes). Out of GL-5 scope (restore-test - untouched by spec); candidate follow-up: reuse `drRestoreOverrides` there. -- The published/vouch-pending agent is **0.74.0** while felhom-pve now runs 0.75.0 — publish 0.75.0 - with (or before) the operator's Day-0 manifest bump so the vouched agent has working DR. - -## NOT live-validated (GL-6 / S5-family, supervised) - -The full customer-data DR drill (escrow consume, offsite tier, restore-into-service); C2's rollback -on a real host; a real drive re-enroll after DR (mp8 parent bind is in place for it). +C7-class parity firing on REAL drift (needs a deliberately mangled restore — GL-6 family); the PBS +tier's full-fidelity runtime; hub-side rendering of the new record fields (additive, ignored today).