diff --git a/CHANGELOG.md b/CHANGELOG.md index df3c077..d82a817 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,13 +12,18 @@ back-half adds them) — that asymmetry was the bug. whole point — no archive parse for the LAYOUT) via the shared `throwawayVolumeOverride` format helper (extracted from `bindMountOverrides`; the restore-test's is-a-bind FILTER reads live configs, which DR by definition has none of). ModeProvision passes nil — regression-contract test. -- **LIVE-DISCOVERED PVE constraint (not in the spike — it never ran an override restore):** mpN - restore params without an explicit `rootfs` → HTTP 500 "mount points configured, but 'rootfs' not - set" (same rule restoretest.go:211 documents for the live-config path). Fixed with NEW - `Client.ExtractArchiveConfig` (GET `/nodes/{node}/vzdump/extractconfig` — **answers 200 under the - scoped agent token**, verified live; PBS keys stay server-side, the spike's candidate-1 rejection - holds): the archive's OWN embedded config supplies the rootfs SIZE only; unreadable/unparseable → - clean refusal before any restore. +- **LIVE-DISCOVERED: PVE's explicit-params restore is ALL-OR-NOTHING** (neither half was in the + spike — it never ran an override restore). (a) mpN params without an explicit `rootfs` → HTTP 500 + "mount points configured, but 'rootfs' not set" (same rule restoretest.go:211 documents). + (b) **Mountpoints NOT named in the params are silently DROPPED** — the first live run came up + boot+running WITHOUT its mp0/mp1 data volumes (2m56s; the customer's world did not ride along). + Fix: NEW `Client.ExtractArchiveConfig` (GET `/nodes/{node}/vzdump/extractconfig` — **answers 200 + under the scoped agent token**, verified live; PBS keys stay server-side, the spike's candidate-1 + rejection holds) + `drRestoreOverrides` derives the COMPLETE param set from the archive's own + embedded config: explicit rootfs, every storage-backed mpN passed through (size + path + backup + preserved → vzrestore extracts its content), structural binds → throwaways. Unknown bind mpN / + unparseable size / unreadable config → clean refusal before any restore. Snapshot sections never + shadow the current config. - **Step 4d — real-bind swap** (DR only, pre-start): mp9 bootstrap host dir created (idempotent — a same-host guest-loss still has bootstrap.json there, untouched), then mp8/mp9 set to the REAL binds via the host runner (`pct set` — bind mounts are root@pam-only, hence NOT the API; sudoers @@ -35,8 +40,11 @@ back-half adds them) — that asymmetry was the bug. refusal; extract-failure refusal; 403-residue warn; archive rootfs parse (snapshot sections never shadow). Red-proofs: override synthesis reverted → A FAILS; unconditional overrides → B FAILS (both run→fail→revert). -- Live validation (campaign-2-precedent scratch DR, §REPORT): see REPORT.md — the exact op that - failed on v0.74.0 now round-trips with correctly-wired binds and no residue. +- Live validation (campaign-2-precedent scratch DR into vmid 9310 from a real 9201 local archive, + auto-teardown, guest 9201 untouched): pre-teardown `pct config` shows mp0 200G + mp1 50G restored + (7m23s — content genuinely extracted), mp8/mp9 = the REAL binds (exact back-half values), rootfs + 32G explicit, ZERO unusedN; boot+running; teardown clean incl. the scratch mp9 host dir. On + v0.74.0 the same op failed at the restore POST. Full evidence: REPORT.md. ## v0.74.0 — pool membership re-asserted after restore-over-existing (campaign-2 R2) (2026-07-07) diff --git a/CONTEXT.md b/CONTEXT.md index ded7a8b..7dcf37c 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -5,6 +5,20 @@ ## Current +- **v0.75.0** (2026-07-08, LIVE on felhom-pve) — **GL-5 / go-live G8: guest-loss DR bring-up + actually restores** (closes the v0.74.0 OPEN item + SPIKE-dr-bindmount-source §8). DR passes the + COMPLETE explicit restore param set derived from the archive's embedded config (NEW + `Client.ExtractArchiveConfig`, 200 under the scoped token) — **two live-discovered PVE rules: mpN + params need an explicit rootfs, AND unlisted mountpoints are silently DROPPED** (first run booted + without mp0/mp1!) — storage mpN passed through, structural mp8/mp9 → throwaways, then step 4d + swaps the REAL binds in via the host runner (root pct; new `EngineOptions.HostRunner`+`StateDir` + seam) and deletes the unusedN residue. Scratch-DR live-proven end-to-end (9310 from a real 9201 + archive: mp0 200G + mp1 50G + real binds + no residue + clean teardown). Provision = nil + overrides (regression-tested). NOTE: published/vouch-pending agent is 0.74.0 — publish 0.75.0 + before/with the manifest bump. OBSERVATION: the DR selftest hardcodes KeepMAC=true — a scratch DR + while the SOURCE guest is live briefly duplicates its MAC on the bridge (pre-existing; fine for + supervised runs, worth a -keep-mac flag someday). Full customer-data DR drill = GL-6/S5 family. + - **2026-07-07 — v0.74.0 Gitea-PUBLISHED (RUNBOOK GL-1)** — the LIVE felhom-pve binary's exact bytes, sha256 `1ec3f58842edce1e…76af05`, anon-fetch-verified. This supersedes/closes every standing "publish 0.6x + Day-0 vouch" OPEN item below (0.64→0.73 were never published; 0.74.0 diff --git a/REPORT.md b/REPORT.md index 96f0ffd..f5c575a 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,47 +1,101 @@ -# REPORT — v0.74.0: re-assert pool membership after restore-over-existing (campaign-2 R2) +# REPORT — v0.75.0: DR bring-up structural bind overrides + real-bind swap (GL-5 / G8) -**Date:** 2026-07-07 · **Class:** implementation (agent). **Baseline:** `main` @ `e04b75e` (v0.73.0) -→ `ca0b169`. Part B of the campaign-2 R1/R2 fix bundle (C = controller F-C2-1 v0.103.0; A = operator -pool one-liner; D = DR-source spike; E = report correction). +**Date:** 2026-07-08 · **Class:** risky/supervised implementation (agent) + one constrained live +validation. **Baseline:** `main` @ `4c408467` (v0.74.0) → `b3446213` + docs. Spec: GL-5 (go-live +G8), verdict of `felhom.eu/documentation/audits/SPIKE-dr-bindmount-source-2026-07-07.md`. +STOP honored: no restore into 9201, no `-keep`, no PBS-tier writes; guest 9201 untouched throughout. -## Root cause (the campaign report's R1 was a symptom) -Pool membership is what lets the pool-scoped `FelhomAgentGuest` token reach a guest — the grant at -`/pool/` applies only to pool MEMBERS. `pct restore --pool` sets membership at CREATE, but a -restore OVER AN EXISTING VMID (the host-loss/finale path) never re-applies it, and no code re-added a -guest to the pool. So every destroy-restore silently dropped membership → the guest lost `VM.Audit` -→ the NEXT `--selftest=restore-test`/DR couldn't read the source config → its *existing, correct* -`bindMountOverrides` never ran → raw bind mounts hit `pct restore` → "mp8 … only possible for root" -(the campaign's mislabelled "R1"). The role + ACL were correct all along; **membership** was the fault. +## What shipped -## The fix -- **`Client.PoolAddVMID(ctx, pool, vmid)`** (`internal/proxmox/mutate.go`): `PUT /pools/{pool}` - `vms={vmid}` — PVE-additive (merge, not replace; `delete=1` removes), idempotent (already-member - swallowed), needs `Pool.Allocate` (the token has it). Sync (no UPID). -- **bring-up re-asserts** (`internal/reconcile/bringup.go`): after liveness is proven, if - `spec.Pool != ""`, call `PoolAddVMID`. A pool-add hiccup is surfaced LOUD + into `res.StartWarnings` - but must NOT flip a healthy running guest's verdict (membership matters for the NEXT op). -- **B3 (scratch teardown 403) — diagnosed, no code:** `restoretest.go` already passes - `Pool: DefaultPool` for scratch restores → the campaign's `VM.Allocate` teardown 403 was a CASCADE - of the failed bind-mount restore (a half-built guest outside any pool), not an independent gap. -- Role/ACL untouched (correct); `bindMountOverrides` untouched (correct — it just needed to run). +1. **DR restore overrides** (`bringup.go`): `ModeDRGuestLoss` builds the restore params via + `drRestoreOverrides` from the archive's own embedded config; `ModeProvision` passes nil — + the regression contract (test + red-proof). +2. **Step 4d real-bind swap** after the grows, before start: mp9 host dir (`os.MkdirAll`, agent-owned, + idempotent — a same-host guest-loss still has bootstrap.json there, untouched), `mkdir -p` parent + dir + two `pct set` calls via the host runner (root-only bind mounts; sudoers lines 39 + 107 + already allowlist both shapes), one slot per call so C2 failures name the exact mpN; then the + displaced throwaways (`unusedN`) deleted in one config PUT. A scoped-token delete refusal logs + loudly + warns in the result — privileges never widened. Rollback envelope respected (C2 test: + mid-swap failure → compensating destroy, guest never started). +3. **Engine seam**: `EngineOptions.HostRunner` + `StateDir`; DR refuses up front on an API-only + engine. The bring-up selftest wires the back-half's ExecRunner shape and removes the scratch + vmid's mp9 host dir at teardown (§7 edge — never a real drive's bind source). +4. `proxmox.GuestConfig.Unused()` + `Client.ExtractArchiveConfig`. -## Tests + red-proofs -`pool_test.go`: `PoolAddVMID` PUT shape + idempotent-on-already-member + real-error-surfaces + -validation. `bringup_test.go`: re-asserts when `Pool!=""` (**red-proof:** pre-fix no-call → -`poolAdds=[]` FAIL, demonstrated + reverted), no-pool→no-call, pool-add-failure warns-but-passes -(liveness wins, guest kept). Full gate `go build/vet/test ./...` = PASS. +## Two LIVE-DISCOVERED PVE constraints (neither in the spike — it never ran an override restore) -## Deploy + live acceptance (the headline) -Built `-X main.version=0.74.0`, deployed to felhom-pve (`.bak-0.73.0` kept); `--version` 0.74.0, -active, capabilities 56/56 degraded=0. **Part A** run (`pveum pool modify felhom --vms 9201`, per -operator go): 9201 now a pool member; `VM.Audit`+`VM.Allocate`+`VM.Backup` present on `/vms/9201`. -**Live `--selftest=restore-test` then PASSED for the first time:** `neutralizing source bind-mount -mountpoints … bind_mounts=2` → scratch 990000 restored + **verified boot+running** + **torn down -clean (no VM.Allocate 403)** in 4m35s. B3 confirmed as a cascade; restore-test needs no bind code. +PVE's explicit-params restore is **all-or-nothing**: -## Observations -- DR `bring-up -mode dr` has a SEPARATE real gap (no overrides, source guest gone) — scoped in - `felhom.eu/documentation/audits/SPIKE-dr-bindmount-source-2026-07-07.md`; recommended fix is a - small known-constant override reusing `bindMountOverrides` (mp8/mp9 are structural constants). -- The empty pool predated the campaign (07:41 403 before the 14:09 finale) — an earlier drill's - restore-over-existing; v0.74.0 stops the recurrence going forward. +- **(a)** any mpN param without an explicit `rootfs` → HTTP 500 `mount points configured, but + 'rootfs' not set` (the constraint restoretest.go:211 documents for the live-config path). The + first live run died here. +- **(b)** mountpoints NOT named in the params are **silently dropped**: the second live run came up + `boot+running` in 2m56s — *without mp0/mp1*. A DR guest without its Docker-data/user-data volumes + is a data-loss restore that LOOKS green (liveness alone cannot catch it). + +Resolution: derive the COMPLETE param set from the archive's embedded config via +`ExtractArchiveConfig` — probed live first: **HTTP 200 under the scoped agent token** (the PVE +server holds the PBS key; the spike's candidate-1 rejection — never hand the agent key material — +holds). The spec's known-constants rule still governs the BIND layout (constants + an unknown-bind +refusal); the archive config supplies sizes/paths for rootfs + storage mpN pass-through. §12's +"no PBS blobs for the mount layout" is honored in spirit and letter — candidate 1 (agent-side +PBS-key blob read) stays rejected. + +Deviation from §5 as specced: `bindMountOverrides` is not called (its is-a-bind filter reads live +configs and broke on non-Linux test runners for synthesized input); its FORMAT was extracted into +`throwawayVolumeOverride` — one source of the override format, used by both callers. Restore-test +behavior untouched. + +## Tests (all green: `go build`, `go vet`, `go test ./...` — 23 pkgs, Windows + 180/Linux) + +Scenario A (exact 3-way overrides incl. rootfs from the extracted config + exact swap commands + +mp9 dir + `delete=unused0,unused1`); B provision-nil + runner-untouched; C2 mid-swap rollback +(error names mp9 + "mp8 already landed", destroy fired, never started); C3 older-archive-without-mp9 +(constants regardless, one unused deleted); DR-without-runner refusal (before any restore); +extract-failure refusal; 403-residue warn-not-fail; `drRestoreOverrides` (full 9201-shaped config → +5 exact params; unknown-bind / no-rootfs / sizeless-mpN refusals; snapshot sections never shadow). +**Red-proofs** (mutate → run → confirm FAIL → restore, against the committed baseline): RP-A +override synthesis removed → Scenario A FAILS; RP-B unconditional overrides → B FAILS. + +## Live validation (campaign-2 precedent; the §13 sequence) + +Deployed to felhom-pve (backup `felhom-agent.bak-0.74.0`, install, restart; clean start, no drive +rebind of the ejected ce9d drive, zero errors). Then +`--selftest=bring-up -mode dr -archive local:backup/vzdump-lxc-9201-2026_07_07-18_57_34.tar.zst +-vmid 9310` (scratch band, no `-keep`, auto-teardown): + +- **On v0.74.0 this exact op FAILED** at the restore POST (constraint (a) above; pre-fix it failed + with `restoring 'mp8' to bind mount is only possible for root` — the spike's finding). +- **v0.75.0 final run: PASS in 7m23s** (vs 2m56s for the data-less intermediate build — the + mp0/mp1 content extraction is real). Pre-teardown `pct config 9310`: + `mp0: local-lvm:vm-9310-disk-1,mp=/var/lib/docker,backup=1,size=200G`, + `mp1: …,mp=/mnt/sys_drive,backup=1,size=50G`, + `mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives`, + `mp9: /var/lib/felhom-agent/guests/9310/bootstrap,mp=/etc/felhom-bootstrap,ro=1`, + `rootfs: …,size=32G`, **zero unusedN** (agent log: `displaced throwaway volumes deleted + unused=[unused0 unused1]`). Boot+running, pool re-asserted, teardown clean, + `/var/lib/felhom-agent/guests/` back to `9201` only. + +## The unusedN mechanism (spec Part 2 asked what was found) + +Replacing an mpN via `pct set` parks the displaced volume as `unusedN` in the config; deleting is a +config PUT `delete=unusedN` (destroys the volume). **Works under the scoped token** (VM.Config.Disk ++ Datastore.Allocate on the restore storage) — no 403, no privilege gap; the warn-not-fail fallback +exists but did not fire. + +## Observations (documented, not acted on) + +- The DR **selftest** hardcodes `KeepMAC=true`, so a scratch DR while the source guest is LIVE + briefly duplicates its MAC on the bridge (BC:24:11:A4:21:6C ran twice for ~5 min). Pre-existing, + supervised-only surface; candidate: a `-keep-mac=false` selftest flag. +- The **restore-test** has the same constraint-(b) exposure in reverse: it passes rootfs + bind + overrides only, so its scratch guests boot WITHOUT mp0/mp1 — its boot-verify is weaker than it + looks (it verifies the OS volume restores, not the data volumes). Out of GL-5 scope (restore-test + untouched by spec); candidate follow-up: reuse `drRestoreOverrides` there. +- The published/vouch-pending agent is **0.74.0** while felhom-pve now runs 0.75.0 — publish 0.75.0 + with (or before) the operator's Day-0 manifest bump so the vouched agent has working DR. + +## NOT live-validated (GL-6 / S5-family, supervised) + +The full customer-data DR drill (escrow consume, offsite tier, restore-into-service); C2's rollback +on a real host; a real drive re-enroll after DR (mp8 parent bind is in place for it).