felhom-agent/REPORT.md

# REPORT — v0.75.0: DR bring-up structural bind overrides + real-bind swap (GL-5 / G8)

**Date:** 2026-07-08 · **Class:** risky/supervised implementation (agent) + one constrained live
validation. **Baseline:** `main` @ `4c408467` (v0.74.0) → `b3446213` + docs. Spec: GL-5 (go-live
G8), verdict of `felhom.eu/documentation/audits/SPIKE-dr-bindmount-source-2026-07-07.md`.
STOP honored: no restore into 9201, no `-keep`, no PBS-tier writes; guest 9201 untouched throughout.

## What shipped

1. **DR restore overrides** (`bringup.go`): `ModeDRGuestLoss` builds the restore params via
   `drRestoreOverrides` from the archive's own embedded config; `ModeProvision` passes nil —
   the regression contract (test + red-proof).
2. **Step 4d real-bind swap** after the grows, before start: mp9 host dir (`os.MkdirAll`, agent-owned,
   idempotent — a same-host guest-loss still has bootstrap.json there, untouched), `mkdir -p` parent
   dir + two `pct set` calls via the host runner (root-only bind mounts; sudoers lines 39 + 107
   already allowlist both shapes), one slot per call so C2 failures name the exact mpN; then the
   displaced throwaways (`unusedN`) deleted in one config PUT. A scoped-token delete refusal logs
   loudly + warns in the result — privileges never widened. Rollback envelope respected (C2 test:
   mid-swap failure → compensating destroy, guest never started).
3. **Engine seam**: `EngineOptions.HostRunner` + `StateDir`; DR refuses up front on an API-only
   engine. The bring-up selftest wires the back-half's ExecRunner shape and removes the scratch
   vmid's mp9 host dir at teardown (§7 edge — never a real drive's bind source).
4. `proxmox.GuestConfig.Unused()` + `Client.ExtractArchiveConfig`.

## Two LIVE-DISCOVERED PVE constraints (neither in the spike — it never ran an override restore)

PVE's explicit-params restore is **all-or-nothing**:

- **(a)** any mpN param without an explicit `rootfs` → HTTP 500 `mount points configured, but
  'rootfs' not set` (the constraint restoretest.go:211 documents for the live-config path). The
  first live run died here.
- **(b)** mountpoints NOT named in the params are **silently dropped**: the second live run came up
  `boot+running` in 2m56s — *without mp0/mp1*. A DR guest without its Docker-data/user-data volumes
  is a data-loss restore that LOOKS green (liveness alone cannot catch it).

Resolution: derive the COMPLETE param set from the archive's embedded config via
`ExtractArchiveConfig` — probed live first: **HTTP 200 under the scoped agent token** (the PVE
server holds the PBS key; the spike's candidate-1 rejection — never hand the agent key material —
holds). The spec's known-constants rule still governs the BIND layout (constants + an unknown-bind
refusal); the archive config supplies sizes/paths for rootfs + storage mpN pass-through. §12's
"no PBS blobs for the mount layout" is honored in spirit and letter — candidate 1 (agent-side
PBS-key blob read) stays rejected.

Deviation from §5 as specced: `bindMountOverrides` is not called (its is-a-bind filter reads live
configs and broke on non-Linux test runners for synthesized input); its FORMAT was extracted into
`throwawayVolumeOverride` — one source of the override format, used by both callers. Restore-test
behavior untouched.

## Tests (all green: `go build`, `go vet`, `go test ./...` — 23 pkgs, Windows + 180/Linux)

Scenario A (exact 3-way overrides incl. rootfs from the extracted config + exact swap commands +
mp9 dir + `delete=unused0,unused1`); B provision-nil + runner-untouched; C2 mid-swap rollback
(error names mp9 + "mp8 already landed", destroy fired, never started); C3 older-archive-without-mp9
(constants regardless, one unused deleted); DR-without-runner refusal (before any restore);
extract-failure refusal; 403-residue warn-not-fail; `drRestoreOverrides` (full 9201-shaped config →
5 exact params; unknown-bind / no-rootfs / sizeless-mpN refusals; snapshot sections never shadow).
**Red-proofs** (mutate → run → confirm FAIL → restore, against the committed baseline): RP-A
override synthesis removed → Scenario A FAILS; RP-B unconditional overrides → B FAILS.

## Live validation (campaign-2 precedent; the §13 sequence)

Deployed to felhom-pve (backup `felhom-agent.bak-0.74.0`, install, restart; clean start, no drive
rebind of the ejected ce9d drive, zero errors). Then
`--selftest=bring-up -mode dr -archive local:backup/vzdump-lxc-9201-2026_07_07-18_57_34.tar.zst
-vmid 9310` (scratch band, no `-keep`, auto-teardown):

- **On v0.74.0 this exact op FAILED** at the restore POST (constraint (a) above; pre-fix it failed
  with `restoring 'mp8' to bind mount is only possible for root` — the spike's finding).
- **v0.75.0 final run: PASS in 7m23s** (vs 2m56s for the data-less intermediate build — the
  mp0/mp1 content extraction is real). Pre-teardown `pct config 9310`:
  `mp0: local-lvm:vm-9310-disk-1,mp=/var/lib/docker,backup=1,size=200G`,
  `mp1: …,mp=/mnt/sys_drive,backup=1,size=50G`,
  `mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives`,
  `mp9: /var/lib/felhom-agent/guests/9310/bootstrap,mp=/etc/felhom-bootstrap,ro=1`,
  `rootfs: …,size=32G`, **zero unusedN** (agent log: `displaced throwaway volumes deleted
  unused=[unused0 unused1]`). Boot+running, pool re-asserted, teardown clean,
  `/var/lib/felhom-agent/guests/` back to `9201` only.

## The unusedN mechanism (spec Part 2 asked what was found)

Replacing an mpN via `pct set` parks the displaced volume as `unusedN` in the config; deleting is a
config PUT `delete=unusedN` (destroys the volume). **Works under the scoped token** (VM.Config.Disk
+ Datastore.Allocate on the restore storage) — no 403, no privilege gap; the warn-not-fail fallback
exists but did not fire.

## Observations (documented, not acted on)

- The DR **selftest** hardcodes `KeepMAC=true`, so a scratch DR while the source guest is LIVE
  briefly duplicates its MAC on the bridge (BC:24:11:A4:21:6C ran twice for ~5 min). Pre-existing,
  supervised-only surface; candidate: a `-keep-mac=false` selftest flag.
- The **restore-test** has the same constraint-(b) exposure in reverse: it passes rootfs + bind
  overrides only, so its scratch guests boot WITHOUT mp0/mp1 — its boot-verify is weaker than it
  looks (it verifies the OS volume restores, not the data volumes). Out of GL-5 scope (restore-test
  untouched by spec); candidate follow-up: reuse `drRestoreOverrides` there.
- The published/vouch-pending agent is **0.74.0** while felhom-pve now runs 0.75.0 — publish 0.75.0
  with (or before) the operator's Day-0 manifest bump so the vouched agent has working DR.

## NOT live-validated (GL-6 / S5-family, supervised)

The full customer-data DR drill (escrow consume, offsite tier, restore-into-service); C2's rollback
on a real host; a real drive re-enroll after DR (mp8 parent bind is in place for it).