docs: v0.76.0 CHANGELOG + REPORT + CONTEXT (GL-5b shipped, published)

AGENT_VERSION=0.76.0 AGENT_SHA256=9828c5f75e719fb2e1fc3523f9c322b595a963ec1dcdf37ca42a96bedab7f50b Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-08 09:55:19 +02:00
parent b1697874ec
commit 7f07393623
3 changed files with 115 additions and 85 deletions
@@ -1,3 +1,34 @@
+## v0.76.0 — restore-test full-fidelity verification (GL-5b / go-live G12) (2026-07-08)
+
+Closes GL-5 finding #2's mirror image: the restore-test's live-source-config bind-override path
+tripped the same PVE drop-unlisted-mountpoints rule DR did, so it boot-verified scratch guests
+WITHOUT their storage mpN — weaker verification than it claimed, and GL-6 will lean on it.
+**0.75.0 was superseded unpublished; this is the Day-0 manifest bump target.**
+
+- **Archive-derived params:** the restore-test now derives its restore params exactly like DR
+  bring-up — `ExtractArchiveConfig` + `drRestoreOverrides` (the ARCHIVE is the object under test,
+  not any live guest's current config): rootfs explicit, every storage mpN passed through (its
+  content genuinely extracted — full fidelity; the added runtime IS the verification), structural
+  binds → throwaway stand-ins. Unreadable archive config / unknown bind topology → refuse UP FRONT
+  (never restore a partial guest to "verify" it).
+- **Mount-parity assert (the non-hollow core):** pre-start, the restored scratch's mpN set is
+  compared against the archive's — a missing, mispathed, undersized, or extra mpN FAILS the test
+  naming the delta. PVE rule (b) can never regress into a green light again. `MountParity`
+  ("ok"|"mismatch") + `MountInventory` ride the result and the hub wire record (additive keys — an
+  older hub ignores them).
+- Dead code deleted with its tests: `bindMountOverrides`, `archiveVMID` + the volid regexes (the
+  restore-test was their only caller; a reachable dead lookalike is how the next bug happens).
+  `throwawayVolumeOverride`/`rootfsSizeGB`/`mountPathOf` live on under `drRestoreOverrides`.
+- DR bring-up, 4d, KeepMAC, grows, caps: untouched (bringup.go had zero line changes — the §13
+  DR re-run trigger did not fire).
+- Tests: Scenario A (exact 5-param derivation from the archive + parity ok + inventory + teardown),
+  B (dropped mp0 → FAIL naming it, never started, still torn down; RED-PROOF: with the parity
+  assert removed the run passes silently — run→fail→revert), C (extract-failure + unknown-topology
+  refusals, no restore attempted), D (provision/DR bring-up tests all green, byte-untouched);
+  `mountParity` pure-function matrix (parity/missing/mispathed/undersized/extra/trivial).
+- Live validation + publish: REPORT.md (full-fidelity runtime measured vs the ~3m data-less shape;
+  `AGENT_VERSION`/`AGENT_SHA256` recorded verbatim for the operator's manifest bump).
+
 ## v0.75.0 — DR bring-up structural bind overrides + real-bind swap (GL-5 / go-live G8) (2026-07-08)

 Implements the verdict of `felhom.eu/documentation/audits/SPIKE-dr-bindmount-source-2026-07-07.md`:
@@ -5,6 +5,16 @@

 ## Current

+- **v0.76.0** (2026-07-08, LIVE on felhom-pve + **PUBLISHED sha `9828c5f7…f50b`** — THE Day-0
+  manifest bump target; **0.75.0 superseded unpublished**) — **GL-5b / G12: restore-test
+  full-fidelity**. Params derive from the ARCHIVE's embedded config (`drRestoreOverrides`, same as
+  DR — the old live-source-config path verified the wrong object AND dropped storage mpN per PVE's
+  all-or-nothing rule; deleted with `bindMountOverrides`/`archiveVMID`). NEW mount-parity assert
+  (restored mpN vs archive; miss/mispath/undersize/extra = FAIL naming the delta) + `MountParity`/
+  `MountInventory` on the wire record (additive). Live-proven: scratch 990000 ← 6.5GB 9201 archive,
+  parity ok, inventory mp0 200G+mp1 50G+2 throwaways, **3m4s local tier** (cheaper than feared);
+  rotated-out archive volid → clean up-front refusal (nice failure mode). bringup.go untouched.
+
 - **v0.75.0** (2026-07-08, LIVE on felhom-pve) — **GL-5 / go-live G8: guest-loss DR bring-up
  actually restores** (closes the v0.74.0 OPEN item + SPIKE-dr-bindmount-source §8). DR passes the
  COMPLETE explicit restore param set derived from the archive's embedded config (NEW
@@ -1,101 +1,90 @@
-# REPORT — v0.75.0: DR bring-up structural bind overrides + real-bind swap (GL-5 / G8)
+# REPORT — v0.76.0: restore-test full-fidelity verification (GL-5b / G12) + publish

-**Date:** 2026-07-08 · **Class:** risky/supervised implementation (agent) + one constrained live
-validation. **Baseline:** `main` @ `4c408467` (v0.74.0) → `b3446213` + docs. Spec: GL-5 (go-live
-G8), verdict of `felhom.eu/documentation/audits/SPIKE-dr-bindmount-source-2026-07-07.md`.
-STOP honored: no restore into 9201, no `-keep`, no PBS-tier writes; guest 9201 untouched throughout.
+**Date:** 2026-07-08 · **Class:** risky/supervised implementation + live run + deploy + publish.
+**Baseline:** `main` @ `5a72a4b5` (v0.75.0, live) → `b1697874` + docs. Closes GL-5 finding #2's
+mirror image (REPORT v0.75.0 observation: the restore-test boot-verified scratch guests WITHOUT
+their storage mpN). STOP honored: no 9201 restore target, no `-keep`, no PBS writes, no manifest
+change (operator's).
+
+## The operator values (Day-0 manifest — Configs → Day-0 artifacts)
+
+```
+AGENT_VERSION=0.76.0
+AGENT_SHA256=9828c5f75e719fb2e1fc3523f9c322b595a963ec1dcdf37ca42a96bedab7f50b
+GOLDEN_VERSION=0.103.0
+GOLDEN_SHA256=8481e8a14e2aa0abe3831cc40e5aea4c32f2017e7561c15dca8a674df6a6026e
+```
+
+**0.75.0 was superseded UNPUBLISHED** (deliberate — one manifest bump, straight to 0.76.0). The
+golden stays 0.103.0 (it ships the controller, not the agent — no re-bake).

 ## What shipped

-1. **DR restore overrides** (`bringup.go`): `ModeDRGuestLoss` builds the restore params via
-   `drRestoreOverrides` from the archive's own embedded config; `ModeProvision` passes nil —
-   the regression contract (test + red-proof).
-2. **Step 4d real-bind swap** after the grows, before start: mp9 host dir (`os.MkdirAll`, agent-owned,
-   idempotent — a same-host guest-loss still has bootstrap.json there, untouched), `mkdir -p` parent
-   dir + two `pct set` calls via the host runner (root-only bind mounts; sudoers lines 39 + 107
-   already allowlist both shapes), one slot per call so C2 failures name the exact mpN; then the
-   displaced throwaways (`unusedN`) deleted in one config PUT. A scoped-token delete refusal logs
-   loudly + warns in the result — privileges never widened. Rollback envelope respected (C2 test:
-   mid-swap failure → compensating destroy, guest never started).
-3. **Engine seam**: `EngineOptions.HostRunner` + `StateDir`; DR refuses up front on an API-only
-   engine. The bring-up selftest wires the back-half's ExecRunner shape and removes the scratch
-   vmid's mp9 host dir at teardown (§7 edge — never a real drive's bind source).
-4. `proxmox.GuestConfig.Unused()` + `Client.ExtractArchiveConfig`.
+- Restore-test params now derive from the **archive's own embedded config** via
+  `ExtractArchiveConfig` + `drRestoreOverrides` — the same derivation DR bring-up uses, because the
+  ARCHIVE is the object under test (the old path read the live source guest's config: wrong object,
+  and its bind-only overrides tripped PVE's drop-unlisted-mountpoints rule). Unreadable config /
+  unknown bind topology → refuse up front, no partial-guest "verification".
+- **Mount-parity assert** (step 2b, pre-start): restored mpN set vs the archive's — missing,
+  mispathed, undersized, or extra mpN fails the test naming the delta. `MountParity` +
+  `MountInventory` added to the result and the hub wire record (additive JSON keys; older hubs
+  ignore them). Teardown still fires on a parity fail (launch-proven).
+- Dead code deleted WITH its tests: `bindMountOverrides`, `archiveVMID`, the two volid regexes —
+  the restore-test was their only caller (§5's delete-or-justify: deleted; no dead lookalike stays
+  reachable). `throwawayVolumeOverride`/`rootfsSizeGB`/`mountPathOf` live on under
+  `drRestoreOverrides`.
+- **Scenario D:** `bringup.go` had ZERO line changes — provision/DR behavior byte-identical, all
+  GL-5 tests green, the §13 DR re-run trigger did not fire.

-## Two LIVE-DISCOVERED PVE constraints (neither in the spike — it never ran an override restore)
+## Tests

-PVE's explicit-params restore is **all-or-nothing**:
+Scenario A (exact 5-param derivation from a 9201-shaped archive config; parity ok; 4-entry
+inventory; scratch torn down), B (mp0 deleted from the restored config → FAIL naming `mp0 MISSING`,
+guest never started, still torn down), C (extract-failure + unknown-topology refusals with zero
+restore attempts), `mountParity` pure-function matrix (parity / missing / mispathed / undersized /
+extra / trivially-empty). **Scenario-B red-proof** (run→fail→revert on the committed baseline):
+with the 2b parity assert removed, the dropped-mount run PASSES silently — the test caught it
+(FAIL), then reverted, all green. Full gate on Windows + 180/Linux: 23 packages ok.

- **(a)** any mpN param without an explicit `rootfs` → HTTP 500 `mount points configured, but
-  'rootfs' not set` (the constraint restoretest.go:211 documents for the live-config path). The
-  first live run died here.
- **(b)** mountpoints NOT named in the params are **silently dropped**: the second live run came up
-  `boot+running` in 2m56s — *without mp0/mp1*. A DR guest without its Docker-data/user-data volumes
-  is a data-loss restore that LOOKS green (liveness alone cannot catch it).
+## Live validation (felhom-pve)

-Resolution: derive the COMPLETE param set from the archive's embedded config via
-`ExtractArchiveConfig` — probed live first: **HTTP 200 under the scoped agent token** (the PVE
-server holds the PBS key; the spike's candidate-1 rejection — never hand the agent key material —
-holds). The spec's known-constants rule still governs the BIND layout (constants + an unknown-bind
-refusal); the archive config supplies sizes/paths for rootfs + storage mpN pass-through. §12's
-"no PBS blobs for the mount layout" is honored in spirit and letter — candidate 1 (agent-side
-PBS-key blob read) stays rejected.
+Deploy: backup `felhom-agent.bak-0.75.0` → install → restart: active, v0.76.0, zero errors, no
+ce9d rebind. First run FAILED honestly — the GL-5-era archive had been rotated out by retention
+and `ExtractArchiveConfig` refused up front ("volume does not exist"; exactly the fail-fast the
+design wants). Re-run against the current newest archive
+(`vzdump-lxc-9201-2026_07_08-09_38_15.tar.zst`, 6.5GB):

-Deviation from §5 as specced: `bindMountOverrides` is not called (its is-a-bind filter reads live
-configs and broke on non-Linux test runners for synthesized input); its FORMAT was extracted into
-`throwawayVolumeOverride` — one source of the override format, used by both callers. Restore-test
-behavior untouched.
+- transcript: `full-fidelity restore params derived from the archive config … params=5`;
+- pre-teardown `pct config 990000`: `mp0 …,mp=/var/lib/docker,backup=1,size=200G`,
+  `mp1 …,mp=/mnt/sys_drive,backup=1,size=50G`, `mp8`/`mp9` as 1G throwaways at the structural
+  paths, `rootfs …,size=32G`, no unusedN;
+- record: `pass: true`, `mount_parity: "ok"`, `mount_inventory`: the 4 verified mpN,
+  `duration_seconds: 184.28`;
+- scratch torn down; band clean.

-## Tests (all green: `go build`, `go vet`, `go test ./...` — 23 pkgs, Windows + 180/Linux)
+**Runtime, honestly:** 3m4s total on the LOCAL tier — cheaper than the ~7m the GL-5 DR contrast
+suggested (that 7m23s included DR bring-up's longer boot/verify chain; the extraction cost itself
+for this guest's actual content is ~2 extra minutes over a data-less restore). The PBS tier will
+be slower (WAN + decrypt); its generous `RestoreTaskTimeout` already exists spec-side. No fidelity
+knob added per §12 — the runtime is the verification.

-Scenario A (exact 3-way overrides incl. rootfs from the extracted config + exact swap commands +
-mp9 dir + `delete=unused0,unused1`); B provision-nil + runner-untouched; C2 mid-swap rollback
-(error names mp9 + "mp8 already landed", destroy fired, never started); C3 older-archive-without-mp9
-(constants regardless, one unused deleted); DR-without-runner refusal (before any restore);
-extract-failure refusal; 403-residue warn-not-fail; `drRestoreOverrides` (full 9201-shaped config →
-5 exact params; unknown-bind / no-rootfs / sizeless-mpN refusals; snapshot sections never shadow).
-**Red-proofs** (mutate → run → confirm FAIL → restore, against the committed baseline): RP-A
-override synthesis removed → Scenario A FAILS; RP-B unconditional overrides → B FAILS.
+## Publish (GL-1 pattern)

-## Live validation (campaign-2 precedent; the §13 sequence)
+The exact deployed bytes (`readlink /proc/$(pidof felhom-agent)/exe` → `/usr/local/bin/felhom-agent`,
+`--version` = 0.76.0) published via `publish-agent.sh 0.76.0`: PUT 201, round-trip sha match, then
+an ANONYMOUS fetch re-verify — sha identical end-to-end
+(`9828c5f7…f50b` on the box, on 180, and via the anonymous customer-path proxy). Creds sourced on
+the build server, never echoed.

-Deployed to felhom-pve (backup `felhom-agent.bak-0.74.0`, install, restart; clean start, no drive
-rebind of the ejected ce9d drive, zero errors). Then
-`--selftest=bring-up -mode dr -archive local:backup/vzdump-lxc-9201-2026_07_07-18_57_34.tar.zst
-vmid 9310` (scratch band, no `-keep`, auto-teardown):
+## Observations

- **On v0.74.0 this exact op FAILED** at the restore POST (constraint (a) above; pre-fix it failed
-  with `restoring 'mp8' to bind mount is only possible for root` — the spike's finding).
- **v0.75.0 final run: PASS in 7m23s** (vs 2m56s for the data-less intermediate build — the
-  mp0/mp1 content extraction is real). Pre-teardown `pct config 9310`:
-  `mp0: local-lvm:vm-9310-disk-1,mp=/var/lib/docker,backup=1,size=200G`,
-  `mp1: …,mp=/mnt/sys_drive,backup=1,size=50G`,
-  `mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives`,
-  `mp9: /var/lib/felhom-agent/guests/9310/bootstrap,mp=/etc/felhom-bootstrap,ro=1`,
-  `rootfs: …,size=32G`, **zero unusedN** (agent log: `displaced throwaway volumes deleted
-  unused=[unused0 unused1]`). Boot+running, pool re-asserted, teardown clean,
-  `/var/lib/felhom-agent/guests/` back to `9201` only.
+- The pinned-archive rotation (first live run) is a nice property in production: a restore-test
+  against a pruned volid refuses loudly instead of "verifying" nothing. The daemon's scheduled
+  restore-test resolves the newest archive itself, so cadence runs are unaffected.
+- 9201's local backups ran 3× this morning (08:52/09:07/09:38) — the local cadence is busy; not a
+  GL-5b concern, worth a glance at retention math someday.

-## The unusedN mechanism (spec Part 2 asked what was found)
+## NOT live-validated

-Replacing an mpN via `pct set` parks the displaced volume as `unusedN` in the config; deleting is a
-config PUT `delete=unusedN` (destroys the volume). **Works under the scoped token** (VM.Config.Disk
-+ Datastore.Allocate on the restore storage) — no 403, no privilege gap; the warn-not-fail fallback
-exists but did not fire.
-
-## Observations (documented, not acted on)
-
- The DR **selftest** hardcodes `KeepMAC=true`, so a scratch DR while the source guest is LIVE
-  briefly duplicates its MAC on the bridge (BC:24:11:A4:21:6C ran twice for ~5 min). Pre-existing,
-  supervised-only surface; candidate: a `-keep-mac=false` selftest flag.
- The **restore-test** has the same constraint-(b) exposure in reverse: it passes rootfs + bind
-  overrides only, so its scratch guests boot WITHOUT mp0/mp1 — its boot-verify is weaker than it
-  looks (it verifies the OS volume restores, not the data volumes). Out of GL-5 scope (restore-test
-  untouched by spec); candidate follow-up: reuse `drRestoreOverrides` there.
- The published/vouch-pending agent is **0.74.0** while felhom-pve now runs 0.75.0 — publish 0.75.0
-  with (or before) the operator's Day-0 manifest bump so the vouched agent has working DR.
-
-## NOT live-validated (GL-6 / S5-family, supervised)
-
-The full customer-data DR drill (escrow consume, offsite tier, restore-into-service); C2's rollback
-on a real host; a real drive re-enroll after DR (mp8 parent bind is in place for it).
+C7-class parity firing on REAL drift (needs a deliberately mangled restore — GL-6 family); the PBS
+tier's full-fidelity runtime; hub-side rendering of the new record fields (additive, ignored today).