felhom.eu/documentation/audits/SPIKE-dr-bindmount-source-2026-07-07.md

SPIKE — where does the DR bind-mount layout come from when the source guest is gone? (2026-07-07)

Findings-only spike (no production code). Companion to campaign-2 R1/R2. Scope: bring-up -mode dr (true guest-loss DR) passes no MountOverrides (bringup.go:222), so it has no way to neutralize the source guest's bind-mount mountpoints — and unlike restore-test, the source guest is GONE, so GuestConfig(srcVMID) can't supply the layout. The open question is the authoritative, reachable source of the mpN config. Read-only probes on the demo (agent v0.74.0, pool healed).

Context — why restore-test works but DR doesn't (yet)

restore-test reads the mpN layout from the live source guest (GuestConfig(9201)), computes bindMountOverrides (restoretest.go:334), and passes them to RestoreLXC{MountOverrides:…}. That whole chain is correct and already shipped — the campaign proved it once the pool was healed (VM.Audit restored): the 2026-07-07 live run logged neutralizing source bind-mount mountpoints … bind_mounts=2 and passed (scratch 990000 restored+booted+verified+torn-down, 4m35s). DR is the only gap, and only because its source is gone.

Candidate sources — probed

1. PBS pct.conf.blob (fetch the config member standalone) — BLOCKED by encryption

The snapshot carries it (proxmox-backup-client snapshot files ct/9201/<iso> → catalog.pcat1 client.log index.json pct.conf root.pxar). The pool-scoped token (felhom@pbs!n100) authenticates and reaches the snapshot, but the fetch fails:

$ proxmox-backup-client restore ct/9201/2026-07-07T06:11:58Z pct.conf.blob - --repository <repo>
Error: missing key - manifest was created with key 01:36:e9:fe:e1:ee:3d:7a…

The datastore is encrypted, and the non-root agent does not hold the datastore key — PVE holds it root-only at /etc/pve/priv/storage/felhom-pbs.enc (that is why restore-test works via the PVE-API pct restore: PVE, as root, does the crypto; the agent never touches the key). Fetching pct.conf.blob directly would require giving the non-root agent the PBS encryption key — a real expansion of its crypto/trust surface. Not viable under the current (correct) trust model.

2. vzdump archive ./pct/config (local tar) — partial: local tier only

A local vzdump-lxc-<vmid>-*.tar.zst carries ./pct/config (plaintext, standalone-readable). But offsite/guest-loss DR restores from PBS, which is encrypted (candidate 1), and local vzdump archives are on the box being lost. So this helps only the local-archive restore-over-existing case (where the live config is usually still readable anyway). Not the offsite-DR answer.

3. Hub DR recipe Guests blob — not ready (schema gap), but structurally the cleanest

store/dr_recipe.go carries Guests json.RawMessage, but the host-half builder populates DRGuest{VMID, Cores, MemoryBytes, DiskBytes} (felhom-agent/internal/hub/dr_recipe.go) — sizing only, no mpN layout. So the recipe does NOT currently capture the bind-mount layout. It COULD: the host-half is built at report time, while the guest is ALIVE, so the agent can read the mpN layout then (no key, no archive parse) and store it. Cost: extend DRGuest + the report builder (agent) + the consumer (bring-up) across two repos. Robust, but the heaviest change — and it must be kept authoritative vs the archive actually being restored (a stale recipe → wrong overrides).

The finding that changes the recommendation — the binds are STRUCTURAL CONSTANTS

The Felhom golden topology's bind mounts are fixed, not per-customer:

mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives          # the drives parent bind
mp9: /var/lib/felhom-agent/guests/<vmid>/bootstrap,mp=/etc/felhom-bootstrap,ro=1   # bootstrap bind

• provision/backhalf.go:38 calls mp8 "the dedicated mpN for the single permanent parent bind"; capability/manifest.go:55 names parent-bind-mp8 as a fixed provision step (pct set <vmid> -mp8 /mnt/felhom-drives).
• mp9 is the bootstrap bind, always at the same in-guest path, host path templated only by vmid.

So DR bring-up does not need to read the archive's config at all to know which mpN are binds — the Felhom guest always has exactly these two, at known in-guest paths. A DR restore can pass a known, constant MountOverrides (mp8 → throwaway restoreStorage:1,mp=/mnt/felhom-drives,backup=0; mp9 handled the same or dropped) reusing the existing bindMountOverrides semantics — no PBS key, no archive parse, no cross-repo schema change. In a real guest-loss DR the customer's drives are re-enrolled separately (the bind is re-established by the enrollment/reconcile path), and the bootstrap is regenerated — so neutralizing them to throwaways at restore time is exactly right (same rationale as restore-test).

This aligns with the endgame in guesthook/heal.go: the intermediary-mount re-architecture makes /mnt/felhom-drives the ONLY bind source — and that is already realized (mp8 IS "the single permanent parent bind"). The problem is already structurally minimized to two known entries.

Recommendation

Wire a KNOWN-CONSTANT bind override into bring-up -mode dr — small, agent-only — do it as a near follow-up, not deferred.

• Implementation sketch (follow-up task, agent-only): in bringup.go, for the DR mode / Felhom golden topology, build MountOverrides for the known structural binds (mp8 /mnt/felhom-drives, mp9 bootstrap) and pass them to RestoreLXC — reuse restoretest.bindMountOverrides (lift it to a shared helper; do NOT reinvent). No source-config read, no PBS key, no recipe change.
• Prefer the constants over candidate 3's recipe-schema expansion unless/until a guest can carry non-standard binds (none today). If that ever changes, extend the recipe (candidate 3) — the agent captures mpN at report-time while alive.
• Do NOT pursue candidate 1 (would hand the non-root agent the PBS encryption key — trust-surface regression for no benefit).

Cost/benefit on timing

• For: the override is tiny (two known entries), agent-only, reuses shipped code, and aligns with the already-realized single-parent-bind endgame. It closes the one remaining DR gap.
• Against (mild): offsite guest-loss DR bring-up is not yet a shipped customer path, and the pilot-relevant primitive (restore-test) already works via the live-config read (pool healed). So it is not urgent.
• Verdict: low-cost, low-risk, structurally-aligned → worth doing as a small scoped follow-up (reuse bindMountOverrides, known constants), not a deferral and not a schema-heavy recipe change.

Not done here (spike constraints honored)

Read-only probes only; no production code; no destructive DR into 9201; the healed restore-test path was exercised (read-only observation of the working primitive). The follow-up implementation is a separate task gated on this finding.