felhom.eu/documentation/audits/SPIKE-dr-bindmount-source-2026-07-07.md

# SPIKE — where does the DR bind-mount layout come from when the source guest is gone? (2026-07-07)

> Findings-only spike (no production code). Companion to campaign-2 R1/R2. Scope: `bring-up
> -mode dr` (true guest-loss DR) passes **no `MountOverrides`** (`bringup.go:222`), so it has no way
> to neutralize the source guest's bind-mount mountpoints — and unlike `restore-test`, the source
> guest is GONE, so `GuestConfig(srcVMID)` can't supply the layout. The open question is the
> **authoritative, reachable source** of the mpN config. Read-only probes on the demo (agent
> v0.74.0, pool healed).

## Context — why restore-test works but DR doesn't (yet)

`restore-test` reads the mpN layout from the **live** source guest (`GuestConfig(9201)`), computes
`bindMountOverrides` (`restoretest.go:334`), and passes them to `RestoreLXC{MountOverrides:…}`. That
whole chain is **correct and already shipped** — the campaign proved it once the pool was healed
(`VM.Audit` restored): the 2026-07-07 live run logged `neutralizing source bind-mount mountpoints …
bind_mounts=2` and passed (scratch 990000 restored+booted+verified+torn-down, 4m35s). **DR is the
only gap**, and only because its source is gone.

## Candidate sources — probed

### 1. PBS `pct.conf.blob` (fetch the config member standalone) — **BLOCKED by encryption**
The snapshot carries it (`proxmox-backup-client snapshot files ct/9201/<iso>` →
`catalog.pcat1 client.log index.json pct.conf root.pxar`). The pool-scoped token
(`felhom@pbs!n100`) **authenticates and reaches** the snapshot, but the fetch fails:
```
$ proxmox-backup-client restore ct/9201/2026-07-07T06:11:58Z pct.conf.blob - --repository <repo>
Error: missing key - manifest was created with key 01:36:e9:fe:e1:ee:3d:7a…
```
The datastore is **encrypted**, and the non-root agent does **not** hold the datastore key — PVE
holds it root-only at `/etc/pve/priv/storage/felhom-pbs.enc` (that is *why* restore-test works via the
PVE-API `pct restore`: PVE, as root, does the crypto; the agent never touches the key). Fetching
`pct.conf.blob` directly would require giving the non-root agent the PBS encryption key — a real
expansion of its crypto/trust surface. **Not viable under the current (correct) trust model.**

### 2. vzdump archive `./pct/config` (local tar) — **partial: local tier only**
A local `vzdump-lxc-<vmid>-*.tar.zst` carries `./pct/config` (plaintext, standalone-readable). But
**offsite/guest-loss DR restores from PBS**, which is encrypted (candidate 1), and local vzdump
archives are on the box being lost. So this helps only the local-archive restore-over-existing case
(where the live config is usually still readable anyway). Not the offsite-DR answer.

### 3. Hub DR recipe `Guests` blob — **not ready (schema gap), but structurally the cleanest**
`store/dr_recipe.go` carries `Guests json.RawMessage`, but the host-half builder populates
`DRGuest{VMID, Cores, MemoryBytes, DiskBytes}` (`felhom-agent/internal/hub/dr_recipe.go`) — **sizing
only, no mpN layout**. So the recipe does NOT currently capture the bind-mount layout. It COULD:
the host-half is built at **report time, while the guest is ALIVE**, so the agent can read the mpN
layout then (no key, no archive parse) and store it. Cost: extend `DRGuest` + the report builder
(agent) + the consumer (bring-up) across two repos. Robust, but the heaviest change — and it must be
kept authoritative vs the archive actually being restored (a stale recipe → wrong overrides).

## The finding that changes the recommendation — the binds are STRUCTURAL CONSTANTS

The Felhom golden topology's bind mounts are **fixed, not per-customer**:
```
mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives          # the drives parent bind
mp9: /var/lib/felhom-agent/guests/<vmid>/bootstrap,mp=/etc/felhom-bootstrap,ro=1   # bootstrap bind
```
- `provision/backhalf.go:38` calls **mp8 "the dedicated mpN for the single permanent parent bind"**;
  `capability/manifest.go:55` names `parent-bind-mp8` as a fixed provision step
  (`pct set <vmid> -mp8 /mnt/felhom-drives`).
- mp9 is the bootstrap bind, always at the same in-guest path, host path templated only by vmid.

So DR bring-up does **not need to read the archive's config at all** to know which mpN are binds — the
Felhom guest always has exactly these two, at known in-guest paths. A DR restore can pass a **known,
constant** `MountOverrides` (mp8 → throwaway `restoreStorage:1,mp=/mnt/felhom-drives,backup=0`; mp9
handled the same or dropped) reusing the **existing `bindMountOverrides` semantics** — no PBS key, no
archive parse, no cross-repo schema change. In a real guest-loss DR the customer's drives are
re-enrolled separately (the bind is re-established by the enrollment/reconcile path), and the bootstrap
is regenerated — so neutralizing them to throwaways at restore time is exactly right (same rationale
as restore-test).

This aligns with the **endgame in `guesthook/heal.go`**: the intermediary-mount re-architecture makes
`/mnt/felhom-drives` the ONLY bind source — and that is already realized (mp8 IS "the single permanent
parent bind"). The problem is already structurally minimized to two known entries.

## Recommendation

**Wire a KNOWN-CONSTANT bind override into `bring-up -mode dr` — small, agent-only — do it as a near
follow-up, not deferred.**

- Implementation sketch (follow-up task, agent-only): in `bringup.go`, for the DR mode / Felhom golden
  topology, build `MountOverrides` for the known structural binds (mp8 `/mnt/felhom-drives`, mp9
  bootstrap) and pass them to `RestoreLXC` — **reuse `restoretest.bindMountOverrides`** (lift it to a
  shared helper; do NOT reinvent). No source-config read, no PBS key, no recipe change.
- Prefer the constants over candidate 3's recipe-schema expansion unless/until a guest can carry
  *non-standard* binds (none today). If that ever changes, extend the recipe (candidate 3) — the
  agent captures mpN at report-time while alive.
- Do NOT pursue candidate 1 (would hand the non-root agent the PBS encryption key — trust-surface
  regression for no benefit).

### Cost/benefit on timing
- **For:** the override is tiny (two known entries), agent-only, reuses shipped code, and aligns with
  the already-realized single-parent-bind endgame. It closes the one remaining DR gap.
- **Against (mild):** offsite guest-loss DR bring-up is **not yet a shipped customer path**, and the
  pilot-relevant primitive (`restore-test`) already works via the live-config read (pool healed). So
  it is not *urgent*.
- **Verdict:** low-cost, low-risk, structurally-aligned → worth doing as a **small scoped follow-up**
  (reuse `bindMountOverrides`, known constants), not a deferral and not a schema-heavy recipe change.

## Not done here (spike constraints honored)
Read-only probes only; no production code; no destructive DR into 9201; the healed restore-test path
was exercised (read-only observation of the working primitive). The follow-up implementation is a
separate task gated on this finding.