felhom-agent/REPORT.md

REPORT — v0.75.0: DR bring-up structural bind overrides + real-bind swap (GL-5 / G8)

Date: 2026-07-08 · Class: risky/supervised implementation (agent) + one constrained live validation. Baseline: main @ 4c408467 (v0.74.0) → b3446213 + docs. Spec: GL-5 (go-live G8), verdict of felhom.eu/documentation/audits/SPIKE-dr-bindmount-source-2026-07-07.md. STOP honored: no restore into 9201, no -keep, no PBS-tier writes; guest 9201 untouched throughout.

What shipped

1. DR restore overrides (bringup.go): ModeDRGuestLoss builds the restore params via drRestoreOverrides from the archive's own embedded config; ModeProvision passes nil — the regression contract (test + red-proof).
2. Step 4d real-bind swap after the grows, before start: mp9 host dir (os.MkdirAll, agent-owned, idempotent — a same-host guest-loss still has bootstrap.json there, untouched), mkdir -p parent dir + two pct set calls via the host runner (root-only bind mounts; sudoers lines 39 + 107 already allowlist both shapes), one slot per call so C2 failures name the exact mpN; then the displaced throwaways (unusedN) deleted in one config PUT. A scoped-token delete refusal logs loudly + warns in the result — privileges never widened. Rollback envelope respected (C2 test: mid-swap failure → compensating destroy, guest never started).
3. Engine seam: EngineOptions.HostRunner + StateDir; DR refuses up front on an API-only engine. The bring-up selftest wires the back-half's ExecRunner shape and removes the scratch vmid's mp9 host dir at teardown (§7 edge — never a real drive's bind source).
4. proxmox.GuestConfig.Unused() + Client.ExtractArchiveConfig.

Two LIVE-DISCOVERED PVE constraints (neither in the spike — it never ran an override restore)

PVE's explicit-params restore is all-or-nothing:

• (a) any mpN param without an explicit rootfs → HTTP 500 mount points configured, but 'rootfs' not set (the constraint restoretest.go:211 documents for the live-config path). The first live run died here.
• (b) mountpoints NOT named in the params are silently dropped: the second live run came up boot+running in 2m56s — without mp0/mp1. A DR guest without its Docker-data/user-data volumes is a data-loss restore that LOOKS green (liveness alone cannot catch it).

Resolution: derive the COMPLETE param set from the archive's embedded config via ExtractArchiveConfig — probed live first: HTTP 200 under the scoped agent token (the PVE server holds the PBS key; the spike's candidate-1 rejection — never hand the agent key material — holds). The spec's known-constants rule still governs the BIND layout (constants + an unknown-bind refusal); the archive config supplies sizes/paths for rootfs + storage mpN pass-through. §12's "no PBS blobs for the mount layout" is honored in spirit and letter — candidate 1 (agent-side PBS-key blob read) stays rejected.

Deviation from §5 as specced: bindMountOverrides is not called (its is-a-bind filter reads live configs and broke on non-Linux test runners for synthesized input); its FORMAT was extracted into throwawayVolumeOverride — one source of the override format, used by both callers. Restore-test behavior untouched.

Tests (all green: go build, go vet, go test ./... — 23 pkgs, Windows + 180/Linux)

Scenario A (exact 3-way overrides incl. rootfs from the extracted config + exact swap commands + mp9 dir + delete=unused0,unused1); B provision-nil + runner-untouched; C2 mid-swap rollback (error names mp9 + "mp8 already landed", destroy fired, never started); C3 older-archive-without-mp9 (constants regardless, one unused deleted); DR-without-runner refusal (before any restore); extract-failure refusal; 403-residue warn-not-fail; drRestoreOverrides (full 9201-shaped config → 5 exact params; unknown-bind / no-rootfs / sizeless-mpN refusals; snapshot sections never shadow). Red-proofs (mutate → run → confirm FAIL → restore, against the committed baseline): RP-A override synthesis removed → Scenario A FAILS; RP-B unconditional overrides → B FAILS.

Live validation (campaign-2 precedent; the §13 sequence)

Deployed to felhom-pve (backup felhom-agent.bak-0.74.0, install, restart; clean start, no drive rebind of the ejected ce9d drive, zero errors). Then --selftest=bring-up -mode dr -archive local:backup/vzdump-lxc-9201-2026_07_07-18_57_34.tar.zst -vmid 9310 (scratch band, no -keep, auto-teardown):

• On v0.74.0 this exact op FAILED at the restore POST (constraint (a) above; pre-fix it failed with restoring 'mp8' to bind mount is only possible for root — the spike's finding).
• v0.75.0 final run: PASS in 7m23s (vs 2m56s for the data-less intermediate build — the mp0/mp1 content extraction is real). Pre-teardown pct config 9310: mp0: local-lvm:vm-9310-disk-1,mp=/var/lib/docker,backup=1,size=200G, mp1: …,mp=/mnt/sys_drive,backup=1,size=50G, mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives, mp9: /var/lib/felhom-agent/guests/9310/bootstrap,mp=/etc/felhom-bootstrap,ro=1, rootfs: …,size=32G, zero unusedN (agent log: displaced throwaway volumes deleted unused=[unused0 unused1]). Boot+running, pool re-asserted, teardown clean, /var/lib/felhom-agent/guests/ back to 9201 only.

The unusedN mechanism (spec Part 2 asked what was found)

Replacing an mpN via pct set parks the displaced volume as unusedN in the config; deleting is a config PUT delete=unusedN (destroys the volume). Works under the scoped token (VM.Config.Disk

• Datastore.Allocate on the restore storage) — no 403, no privilege gap; the warn-not-fail fallback exists but did not fire.

Observations (documented, not acted on)

• The DR selftest hardcodes KeepMAC=true, so a scratch DR while the source guest is LIVE briefly duplicates its MAC on the bridge (BC:24:11:A4:21:6C ran twice for ~5 min). Pre-existing, supervised-only surface; candidate: a -keep-mac=false selftest flag.
• The restore-test has the same constraint-(b) exposure in reverse: it passes rootfs + bind overrides only, so its scratch guests boot WITHOUT mp0/mp1 — its boot-verify is weaker than it looks (it verifies the OS volume restores, not the data volumes). Out of GL-5 scope (restore-test untouched by spec); candidate follow-up: reuse drRestoreOverrides there.
• The published/vouch-pending agent is 0.74.0 while felhom-pve now runs 0.75.0 — publish 0.75.0 with (or before) the operator's Day-0 manifest bump so the vouched agent has working DR.

NOT live-validated (GL-6 / S5-family, supervised)

The full customer-data DR drill (escrow consume, offsite tier, restore-into-service); C2's rollback on a real host; a real drive re-enroll after DR (mp8 parent bind is in place for it).