# SPIKE — where does the DR bind-mount layout come from when the source guest is gone? (2026-07-07) > Findings-only spike (no production code). Companion to campaign-2 R1/R2. Scope: `bring-up > -mode dr` (true guest-loss DR) passes **no `MountOverrides`** (`bringup.go:222`), so it has no way > to neutralize the source guest's bind-mount mountpoints — and unlike `restore-test`, the source > guest is GONE, so `GuestConfig(srcVMID)` can't supply the layout. The open question is the > **authoritative, reachable source** of the mpN config. Read-only probes on the demo (agent > v0.74.0, pool healed). ## Context — why restore-test works but DR doesn't (yet) `restore-test` reads the mpN layout from the **live** source guest (`GuestConfig(9201)`), computes `bindMountOverrides` (`restoretest.go:334`), and passes them to `RestoreLXC{MountOverrides:…}`. That whole chain is **correct and already shipped** — the campaign proved it once the pool was healed (`VM.Audit` restored): the 2026-07-07 live run logged `neutralizing source bind-mount mountpoints … bind_mounts=2` and passed (scratch 990000 restored+booted+verified+torn-down, 4m35s). **DR is the only gap**, and only because its source is gone. ## Candidate sources — probed ### 1. PBS `pct.conf.blob` (fetch the config member standalone) — **BLOCKED by encryption** The snapshot carries it (`proxmox-backup-client snapshot files ct/9201/` → `catalog.pcat1 client.log index.json pct.conf root.pxar`). The pool-scoped token (`felhom@pbs!n100`) **authenticates and reaches** the snapshot, but the fetch fails: ``` $ proxmox-backup-client restore ct/9201/2026-07-07T06:11:58Z pct.conf.blob - --repository Error: missing key - manifest was created with key 01:36:e9:fe:e1:ee:3d:7a… ``` The datastore is **encrypted**, and the non-root agent does **not** hold the datastore key — PVE holds it root-only at `/etc/pve/priv/storage/felhom-pbs.enc` (that is *why* restore-test works via the PVE-API `pct restore`: PVE, as root, does the crypto; the agent never touches the key). Fetching `pct.conf.blob` directly would require giving the non-root agent the PBS encryption key — a real expansion of its crypto/trust surface. **Not viable under the current (correct) trust model.** ### 2. vzdump archive `./pct/config` (local tar) — **partial: local tier only** A local `vzdump-lxc--*.tar.zst` carries `./pct/config` (plaintext, standalone-readable). But **offsite/guest-loss DR restores from PBS**, which is encrypted (candidate 1), and local vzdump archives are on the box being lost. So this helps only the local-archive restore-over-existing case (where the live config is usually still readable anyway). Not the offsite-DR answer. ### 3. Hub DR recipe `Guests` blob — **not ready (schema gap), but structurally the cleanest** `store/dr_recipe.go` carries `Guests json.RawMessage`, but the host-half builder populates `DRGuest{VMID, Cores, MemoryBytes, DiskBytes}` (`felhom-agent/internal/hub/dr_recipe.go`) — **sizing only, no mpN layout**. So the recipe does NOT currently capture the bind-mount layout. It COULD: the host-half is built at **report time, while the guest is ALIVE**, so the agent can read the mpN layout then (no key, no archive parse) and store it. Cost: extend `DRGuest` + the report builder (agent) + the consumer (bring-up) across two repos. Robust, but the heaviest change — and it must be kept authoritative vs the archive actually being restored (a stale recipe → wrong overrides). ## The finding that changes the recommendation — the binds are STRUCTURAL CONSTANTS The Felhom golden topology's bind mounts are **fixed, not per-customer**: ``` mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives # the drives parent bind mp9: /var/lib/felhom-agent/guests//bootstrap,mp=/etc/felhom-bootstrap,ro=1 # bootstrap bind ``` - `provision/backhalf.go:38` calls **mp8 "the dedicated mpN for the single permanent parent bind"**; `capability/manifest.go:55` names `parent-bind-mp8` as a fixed provision step (`pct set -mp8 /mnt/felhom-drives`). - mp9 is the bootstrap bind, always at the same in-guest path, host path templated only by vmid. So DR bring-up does **not need to read the archive's config at all** to know which mpN are binds — the Felhom guest always has exactly these two, at known in-guest paths. A DR restore can pass a **known, constant** `MountOverrides` (mp8 → throwaway `restoreStorage:1,mp=/mnt/felhom-drives,backup=0`; mp9 handled the same or dropped) reusing the **existing `bindMountOverrides` semantics** — no PBS key, no archive parse, no cross-repo schema change. In a real guest-loss DR the customer's drives are re-enrolled separately (the bind is re-established by the enrollment/reconcile path), and the bootstrap is regenerated — so neutralizing them to throwaways at restore time is exactly right (same rationale as restore-test). This aligns with the **endgame in `guesthook/heal.go`**: the intermediary-mount re-architecture makes `/mnt/felhom-drives` the ONLY bind source — and that is already realized (mp8 IS "the single permanent parent bind"). The problem is already structurally minimized to two known entries. ## Recommendation **Wire a KNOWN-CONSTANT bind override into `bring-up -mode dr` — small, agent-only — do it as a near follow-up, not deferred.** - Implementation sketch (follow-up task, agent-only): in `bringup.go`, for the DR mode / Felhom golden topology, build `MountOverrides` for the known structural binds (mp8 `/mnt/felhom-drives`, mp9 bootstrap) and pass them to `RestoreLXC` — **reuse `restoretest.bindMountOverrides`** (lift it to a shared helper; do NOT reinvent). No source-config read, no PBS key, no recipe change. - Prefer the constants over candidate 3's recipe-schema expansion unless/until a guest can carry *non-standard* binds (none today). If that ever changes, extend the recipe (candidate 3) — the agent captures mpN at report-time while alive. - Do NOT pursue candidate 1 (would hand the non-root agent the PBS encryption key — trust-surface regression for no benefit). ### Cost/benefit on timing - **For:** the override is tiny (two known entries), agent-only, reuses shipped code, and aligns with the already-realized single-parent-bind endgame. It closes the one remaining DR gap. - **Against (mild):** offsite guest-loss DR bring-up is **not yet a shipped customer path**, and the pilot-relevant primitive (`restore-test`) already works via the live-config read (pool healed). So it is not *urgent*. - **Verdict:** low-cost, low-risk, structurally-aligned → worth doing as a **small scoped follow-up** (reuse `bindMountOverrides`, known constants), not a deferral and not a schema-heavy recipe change. ## Not done here (spike constraints honored) Read-only probes only; no production code; no destructive DR into 9201; the healed restore-test path was exercised (read-only observation of the working primitive). The follow-up implementation is a separate task gated on this finding.