docs: v0.75.0 CHANGELOG + REPORT + CONTEXT (GL-5 shipped, live-validated)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6

CHANGELOG.md

@@ -12,13 +12,18 @@ back-half adds them) — that asymmetry was the bug.
   whole point — no archive parse for the LAYOUT) via the shared `throwawayVolumeOverride` format
   helper (extracted from `bindMountOverrides`; the restore-test's is-a-bind FILTER reads live
   configs, which DR by definition has none of). ModeProvision passes nil — regression-contract test.
-- **LIVE-DISCOVERED PVE constraint (not in the spike — it never ran an override restore):** mpN
-  restore params without an explicit `rootfs` → HTTP 500 "mount points configured, but 'rootfs' not
-  set" (same rule restoretest.go:211 documents for the live-config path). Fixed with NEW
-  `Client.ExtractArchiveConfig` (GET `/nodes/{node}/vzdump/extractconfig` — **answers 200 under the
-  scoped agent token**, verified live; PBS keys stay server-side, the spike's candidate-1 rejection
-  holds): the archive's OWN embedded config supplies the rootfs SIZE only; unreadable/unparseable →
-  clean refusal before any restore.
+- **LIVE-DISCOVERED: PVE's explicit-params restore is ALL-OR-NOTHING** (neither half was in the
+  spike — it never ran an override restore). (a) mpN params without an explicit `rootfs` → HTTP 500
+  "mount points configured, but 'rootfs' not set" (same rule restoretest.go:211 documents).
+  (b) **Mountpoints NOT named in the params are silently DROPPED** — the first live run came up
+  boot+running WITHOUT its mp0/mp1 data volumes (2m56s; the customer's world did not ride along).
+  Fix: NEW `Client.ExtractArchiveConfig` (GET `/nodes/{node}/vzdump/extractconfig` — **answers 200
+  under the scoped agent token**, verified live; PBS keys stay server-side, the spike's candidate-1
+  rejection holds) + `drRestoreOverrides` derives the COMPLETE param set from the archive's own
+  embedded config: explicit rootfs, every storage-backed mpN passed through (size + path + backup
+  preserved → vzrestore extracts its content), structural binds → throwaways. Unknown bind mpN /
+  unparseable size / unreadable config → clean refusal before any restore. Snapshot sections never
+  shadow the current config.
 - **Step 4d — real-bind swap** (DR only, pre-start): mp9 bootstrap host dir created (idempotent —
   a same-host guest-loss still has bootstrap.json there, untouched), then mp8/mp9 set to the REAL
   binds via the host runner (`pct set` — bind mounts are root@pam-only, hence NOT the API; sudoers
@@ -35,8 +40,11 @@ back-half adds them) — that asymmetry was the bug.
   refusal; extract-failure refusal; 403-residue warn; archive rootfs parse (snapshot sections never
   shadow). Red-proofs: override synthesis reverted → A FAILS; unconditional overrides → B FAILS
   (both run→fail→revert).
-- Live validation (campaign-2-precedent scratch DR, §REPORT): see REPORT.md — the exact op that
-  failed on v0.74.0 now round-trips with correctly-wired binds and no residue.
+- Live validation (campaign-2-precedent scratch DR into vmid 9310 from a real 9201 local archive,
+  auto-teardown, guest 9201 untouched): pre-teardown `pct config` shows mp0 200G + mp1 50G restored
+  (7m23s — content genuinely extracted), mp8/mp9 = the REAL binds (exact back-half values), rootfs
+  32G explicit, ZERO unusedN; boot+running; teardown clean incl. the scratch mp9 host dir. On
+  v0.74.0 the same op failed at the restore POST. Full evidence: REPORT.md.
 
 ## v0.74.0 — pool membership re-asserted after restore-over-existing (campaign-2 R2) (2026-07-07)
 

CONTEXT.md

@@ -5,6 +5,20 @@
 
 ## Current
 
+- **v0.75.0** (2026-07-08, LIVE on felhom-pve) — **GL-5 / go-live G8: guest-loss DR bring-up
+  actually restores** (closes the v0.74.0 OPEN item + SPIKE-dr-bindmount-source §8). DR passes the
+  COMPLETE explicit restore param set derived from the archive's embedded config (NEW
+  `Client.ExtractArchiveConfig`, 200 under the scoped token) — **two live-discovered PVE rules: mpN
+  params need an explicit rootfs, AND unlisted mountpoints are silently DROPPED** (first run booted
+  without mp0/mp1!) — storage mpN passed through, structural mp8/mp9 → throwaways, then step 4d
+  swaps the REAL binds in via the host runner (root pct; new `EngineOptions.HostRunner`+`StateDir`
+  seam) and deletes the unusedN residue. Scratch-DR live-proven end-to-end (9310 from a real 9201
+  archive: mp0 200G + mp1 50G + real binds + no residue + clean teardown). Provision = nil
+  overrides (regression-tested). NOTE: published/vouch-pending agent is 0.74.0 — publish 0.75.0
+  before/with the manifest bump. OBSERVATION: the DR selftest hardcodes KeepMAC=true — a scratch DR
+  while the SOURCE guest is live briefly duplicates its MAC on the bridge (pre-existing; fine for
+  supervised runs, worth a -keep-mac flag someday). Full customer-data DR drill = GL-6/S5 family.
+
 - **2026-07-07 — v0.74.0 Gitea-PUBLISHED (RUNBOOK GL-1)** — the LIVE felhom-pve binary's exact
   bytes, sha256 `1ec3f58842edce1e…76af05`, anon-fetch-verified. This supersedes/closes every
   standing "publish 0.6x + Day-0 vouch" OPEN item below (0.64→0.73 were never published; 0.74.0

REPORT.md

@@ -1,47 +1,101 @@
-# REPORT — v0.74.0: re-assert pool membership after restore-over-existing (campaign-2 R2)
+# REPORT — v0.75.0: DR bring-up structural bind overrides + real-bind swap (GL-5 / G8)
 
-**Date:** 2026-07-07 · **Class:** implementation (agent). **Baseline:** `main` @ `e04b75e` (v0.73.0)
-→ `ca0b169`. Part B of the campaign-2 R1/R2 fix bundle (C = controller F-C2-1 v0.103.0; A = operator
-pool one-liner; D = DR-source spike; E = report correction).
+**Date:** 2026-07-08 · **Class:** risky/supervised implementation (agent) + one constrained live
+validation. **Baseline:** `main` @ `4c408467` (v0.74.0) → `b3446213` + docs. Spec: GL-5 (go-live
+G8), verdict of `felhom.eu/documentation/audits/SPIKE-dr-bindmount-source-2026-07-07.md`.
+STOP honored: no restore into 9201, no `-keep`, no PBS-tier writes; guest 9201 untouched throughout.
 
-## Root cause (the campaign report's R1 was a symptom)
-Pool membership is what lets the pool-scoped `FelhomAgentGuest` token reach a guest — the grant at
-`/pool/<pool>` applies only to pool MEMBERS. `pct restore --pool` sets membership at CREATE, but a
-restore OVER AN EXISTING VMID (the host-loss/finale path) never re-applies it, and no code re-added a
-guest to the pool. So every destroy-restore silently dropped membership → the guest lost `VM.Audit`
-→ the NEXT `--selftest=restore-test`/DR couldn't read the source config → its *existing, correct*
-`bindMountOverrides` never ran → raw bind mounts hit `pct restore` → "mp8 … only possible for root"
-(the campaign's mislabelled "R1"). The role + ACL were correct all along; **membership** was the fault.
+## What shipped
 
-## The fix
-- **`Client.PoolAddVMID(ctx, pool, vmid)`** (`internal/proxmox/mutate.go`): `PUT /pools/{pool}`
-  `vms={vmid}` — PVE-additive (merge, not replace; `delete=1` removes), idempotent (already-member
-  swallowed), needs `Pool.Allocate` (the token has it). Sync (no UPID).
-- **bring-up re-asserts** (`internal/reconcile/bringup.go`): after liveness is proven, if
-  `spec.Pool != ""`, call `PoolAddVMID`. A pool-add hiccup is surfaced LOUD + into `res.StartWarnings`
-  but must NOT flip a healthy running guest's verdict (membership matters for the NEXT op).
-- **B3 (scratch teardown 403) — diagnosed, no code:** `restoretest.go` already passes
-  `Pool: DefaultPool` for scratch restores → the campaign's `VM.Allocate` teardown 403 was a CASCADE
-  of the failed bind-mount restore (a half-built guest outside any pool), not an independent gap.
-- Role/ACL untouched (correct); `bindMountOverrides` untouched (correct — it just needed to run).
+1. **DR restore overrides** (`bringup.go`): `ModeDRGuestLoss` builds the restore params via
+   `drRestoreOverrides` from the archive's own embedded config; `ModeProvision` passes nil —
+   the regression contract (test + red-proof).
+2. **Step 4d real-bind swap** after the grows, before start: mp9 host dir (`os.MkdirAll`, agent-owned,
+   idempotent — a same-host guest-loss still has bootstrap.json there, untouched), `mkdir -p` parent
+   dir + two `pct set` calls via the host runner (root-only bind mounts; sudoers lines 39 + 107
+   already allowlist both shapes), one slot per call so C2 failures name the exact mpN; then the
+   displaced throwaways (`unusedN`) deleted in one config PUT. A scoped-token delete refusal logs
+   loudly + warns in the result — privileges never widened. Rollback envelope respected (C2 test:
+   mid-swap failure → compensating destroy, guest never started).
+3. **Engine seam**: `EngineOptions.HostRunner` + `StateDir`; DR refuses up front on an API-only
+   engine. The bring-up selftest wires the back-half's ExecRunner shape and removes the scratch
+   vmid's mp9 host dir at teardown (§7 edge — never a real drive's bind source).
+4. `proxmox.GuestConfig.Unused()` + `Client.ExtractArchiveConfig`.
 
-## Tests + red-proofs
-`pool_test.go`: `PoolAddVMID` PUT shape + idempotent-on-already-member + real-error-surfaces +
-validation. `bringup_test.go`: re-asserts when `Pool!=""` (**red-proof:** pre-fix no-call →
-`poolAdds=[]` FAIL, demonstrated + reverted), no-pool→no-call, pool-add-failure warns-but-passes
-(liveness wins, guest kept). Full gate `go build/vet/test ./...` = PASS.
+## Two LIVE-DISCOVERED PVE constraints (neither in the spike — it never ran an override restore)
 
-## Deploy + live acceptance (the headline)
-Built `-X main.version=0.74.0`, deployed to felhom-pve (`.bak-0.73.0` kept); `--version` 0.74.0,
-active, capabilities 56/56 degraded=0. **Part A** run (`pveum pool modify felhom --vms 9201`, per
-operator go): 9201 now a pool member; `VM.Audit`+`VM.Allocate`+`VM.Backup` present on `/vms/9201`.
-**Live `--selftest=restore-test` then PASSED for the first time:** `neutralizing source bind-mount
-mountpoints … bind_mounts=2` → scratch 990000 restored + **verified boot+running** + **torn down
-clean (no VM.Allocate 403)** in 4m35s. B3 confirmed as a cascade; restore-test needs no bind code.
+PVE's explicit-params restore is **all-or-nothing**:
 
-## Observations
-- DR `bring-up -mode dr` has a SEPARATE real gap (no overrides, source guest gone) — scoped in
-  `felhom.eu/documentation/audits/SPIKE-dr-bindmount-source-2026-07-07.md`; recommended fix is a
-  small known-constant override reusing `bindMountOverrides` (mp8/mp9 are structural constants).
-- The empty pool predated the campaign (07:41 403 before the 14:09 finale) — an earlier drill's
-  restore-over-existing; v0.74.0 stops the recurrence going forward.
+- **(a)** any mpN param without an explicit `rootfs` → HTTP 500 `mount points configured, but
+  'rootfs' not set` (the constraint restoretest.go:211 documents for the live-config path). The
+  first live run died here.
+- **(b)** mountpoints NOT named in the params are **silently dropped**: the second live run came up
+  `boot+running` in 2m56s — *without mp0/mp1*. A DR guest without its Docker-data/user-data volumes
+  is a data-loss restore that LOOKS green (liveness alone cannot catch it).
+
+Resolution: derive the COMPLETE param set from the archive's embedded config via
+`ExtractArchiveConfig` — probed live first: **HTTP 200 under the scoped agent token** (the PVE
+server holds the PBS key; the spike's candidate-1 rejection — never hand the agent key material —
+holds). The spec's known-constants rule still governs the BIND layout (constants + an unknown-bind
+refusal); the archive config supplies sizes/paths for rootfs + storage mpN pass-through. §12's
+"no PBS blobs for the mount layout" is honored in spirit and letter — candidate 1 (agent-side
+PBS-key blob read) stays rejected.
+
+Deviation from §5 as specced: `bindMountOverrides` is not called (its is-a-bind filter reads live
+configs and broke on non-Linux test runners for synthesized input); its FORMAT was extracted into
+`throwawayVolumeOverride` — one source of the override format, used by both callers. Restore-test
+behavior untouched.
+
+## Tests (all green: `go build`, `go vet`, `go test ./...` — 23 pkgs, Windows + 180/Linux)
+
+Scenario A (exact 3-way overrides incl. rootfs from the extracted config + exact swap commands +
+mp9 dir + `delete=unused0,unused1`); B provision-nil + runner-untouched; C2 mid-swap rollback
+(error names mp9 + "mp8 already landed", destroy fired, never started); C3 older-archive-without-mp9
+(constants regardless, one unused deleted); DR-without-runner refusal (before any restore);
+extract-failure refusal; 403-residue warn-not-fail; `drRestoreOverrides` (full 9201-shaped config →
+5 exact params; unknown-bind / no-rootfs / sizeless-mpN refusals; snapshot sections never shadow).
+**Red-proofs** (mutate → run → confirm FAIL → restore, against the committed baseline): RP-A
+override synthesis removed → Scenario A FAILS; RP-B unconditional overrides → B FAILS.
+
+## Live validation (campaign-2 precedent; the §13 sequence)
+
+Deployed to felhom-pve (backup `felhom-agent.bak-0.74.0`, install, restart; clean start, no drive
+rebind of the ejected ce9d drive, zero errors). Then
+`--selftest=bring-up -mode dr -archive local:backup/vzdump-lxc-9201-2026_07_07-18_57_34.tar.zst
+-vmid 9310` (scratch band, no `-keep`, auto-teardown):
+
+- **On v0.74.0 this exact op FAILED** at the restore POST (constraint (a) above; pre-fix it failed
+  with `restoring 'mp8' to bind mount is only possible for root` — the spike's finding).
+- **v0.75.0 final run: PASS in 7m23s** (vs 2m56s for the data-less intermediate build — the
+  mp0/mp1 content extraction is real). Pre-teardown `pct config 9310`:
+  `mp0: local-lvm:vm-9310-disk-1,mp=/var/lib/docker,backup=1,size=200G`,
+  `mp1: …,mp=/mnt/sys_drive,backup=1,size=50G`,
+  `mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives`,
+  `mp9: /var/lib/felhom-agent/guests/9310/bootstrap,mp=/etc/felhom-bootstrap,ro=1`,
+  `rootfs: …,size=32G`, **zero unusedN** (agent log: `displaced throwaway volumes deleted
+  unused=[unused0 unused1]`). Boot+running, pool re-asserted, teardown clean,
+  `/var/lib/felhom-agent/guests/` back to `9201` only.
+
+## The unusedN mechanism (spec Part 2 asked what was found)
+
+Replacing an mpN via `pct set` parks the displaced volume as `unusedN` in the config; deleting is a
+config PUT `delete=unusedN` (destroys the volume). **Works under the scoped token** (VM.Config.Disk
++ Datastore.Allocate on the restore storage) — no 403, no privilege gap; the warn-not-fail fallback
+exists but did not fire.
+
+## Observations (documented, not acted on)
+
+- The DR **selftest** hardcodes `KeepMAC=true`, so a scratch DR while the source guest is LIVE
+  briefly duplicates its MAC on the bridge (BC:24:11:A4:21:6C ran twice for ~5 min). Pre-existing,
+  supervised-only surface; candidate: a `-keep-mac=false` selftest flag.
+- The **restore-test** has the same constraint-(b) exposure in reverse: it passes rootfs + bind
+  overrides only, so its scratch guests boot WITHOUT mp0/mp1 — its boot-verify is weaker than it
+  looks (it verifies the OS volume restores, not the data volumes). Out of GL-5 scope (restore-test
+  untouched by spec); candidate follow-up: reuse `drRestoreOverrides` there.
+- The published/vouch-pending agent is **0.74.0** while felhom-pve now runs 0.75.0 — publish 0.75.0
+  with (or before) the operator's Day-0 manifest bump so the vouched agent has working DR.
+
+## NOT live-validated (GL-6 / S5-family, supervised)
+
+The full customer-data DR drill (escrow consume, offsite tier, restore-into-service); C2's rollback
+on a real host; a real drive re-enroll after DR (mp8 parent bind is in place for it).