docs: campaign-2 R1/R2 root-cause correction + DR bind-mount source spike
Correction: campaign-2's "R1" (bind-mount restore needs root) was a SYMPTOM of an empty felhom pool -> agent lacked VM.Audit on 9201 -> restore-test's existing, correct bindMountOverrides never ran. Live-proven: pool healed (Part A) -> restore-test PASSES (neutralizes 2 binds, boot+running, clean teardown, 4m35s). Report rec #1 (make pct restore work on bind mounts) superseded. Fixes shipped: agent v0.74.0 (pool re-assertion), controller v0.103.0 (F-C2-1). Spike (SPIKE-dr-bindmount-source-2026-07-07): DR bring-up source options — PBS pct.conf.blob blocked by encryption under the non-root token; DR recipe lacks mpN; mp8/mp9 are STRUCTURAL CONSTANTS -> recommend a small known-constant override reusing bindMountOverrides. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
@@ -0,0 +1,102 @@
|
||||
# SPIKE — where does the DR bind-mount layout come from when the source guest is gone? (2026-07-07)
|
||||
|
||||
> Findings-only spike (no production code). Companion to campaign-2 R1/R2. Scope: `bring-up
|
||||
> -mode dr` (true guest-loss DR) passes **no `MountOverrides`** (`bringup.go:222`), so it has no way
|
||||
> to neutralize the source guest's bind-mount mountpoints — and unlike `restore-test`, the source
|
||||
> guest is GONE, so `GuestConfig(srcVMID)` can't supply the layout. The open question is the
|
||||
> **authoritative, reachable source** of the mpN config. Read-only probes on the demo (agent
|
||||
> v0.74.0, pool healed).
|
||||
|
||||
## Context — why restore-test works but DR doesn't (yet)
|
||||
|
||||
`restore-test` reads the mpN layout from the **live** source guest (`GuestConfig(9201)`), computes
|
||||
`bindMountOverrides` (`restoretest.go:334`), and passes them to `RestoreLXC{MountOverrides:…}`. That
|
||||
whole chain is **correct and already shipped** — the campaign proved it once the pool was healed
|
||||
(`VM.Audit` restored): the 2026-07-07 live run logged `neutralizing source bind-mount mountpoints …
|
||||
bind_mounts=2` and passed (scratch 990000 restored+booted+verified+torn-down, 4m35s). **DR is the
|
||||
only gap**, and only because its source is gone.
|
||||
|
||||
## Candidate sources — probed
|
||||
|
||||
### 1. PBS `pct.conf.blob` (fetch the config member standalone) — **BLOCKED by encryption**
|
||||
The snapshot carries it (`proxmox-backup-client snapshot files ct/9201/<iso>` →
|
||||
`catalog.pcat1 client.log index.json pct.conf root.pxar`). The pool-scoped token
|
||||
(`felhom@pbs!n100`) **authenticates and reaches** the snapshot, but the fetch fails:
|
||||
```
|
||||
$ proxmox-backup-client restore ct/9201/2026-07-07T06:11:58Z pct.conf.blob - --repository <repo>
|
||||
Error: missing key - manifest was created with key 01:36:e9:fe:e1:ee:3d:7a…
|
||||
```
|
||||
The datastore is **encrypted**, and the non-root agent does **not** hold the datastore key — PVE
|
||||
holds it root-only at `/etc/pve/priv/storage/felhom-pbs.enc` (that is *why* restore-test works via the
|
||||
PVE-API `pct restore`: PVE, as root, does the crypto; the agent never touches the key). Fetching
|
||||
`pct.conf.blob` directly would require giving the non-root agent the PBS encryption key — a real
|
||||
expansion of its crypto/trust surface. **Not viable under the current (correct) trust model.**
|
||||
|
||||
### 2. vzdump archive `./pct/config` (local tar) — **partial: local tier only**
|
||||
A local `vzdump-lxc-<vmid>-*.tar.zst` carries `./pct/config` (plaintext, standalone-readable). But
|
||||
**offsite/guest-loss DR restores from PBS**, which is encrypted (candidate 1), and local vzdump
|
||||
archives are on the box being lost. So this helps only the local-archive restore-over-existing case
|
||||
(where the live config is usually still readable anyway). Not the offsite-DR answer.
|
||||
|
||||
### 3. Hub DR recipe `Guests` blob — **not ready (schema gap), but structurally the cleanest**
|
||||
`store/dr_recipe.go` carries `Guests json.RawMessage`, but the host-half builder populates
|
||||
`DRGuest{VMID, Cores, MemoryBytes, DiskBytes}` (`felhom-agent/internal/hub/dr_recipe.go`) — **sizing
|
||||
only, no mpN layout**. So the recipe does NOT currently capture the bind-mount layout. It COULD:
|
||||
the host-half is built at **report time, while the guest is ALIVE**, so the agent can read the mpN
|
||||
layout then (no key, no archive parse) and store it. Cost: extend `DRGuest` + the report builder
|
||||
(agent) + the consumer (bring-up) across two repos. Robust, but the heaviest change — and it must be
|
||||
kept authoritative vs the archive actually being restored (a stale recipe → wrong overrides).
|
||||
|
||||
## The finding that changes the recommendation — the binds are STRUCTURAL CONSTANTS
|
||||
|
||||
The Felhom golden topology's bind mounts are **fixed, not per-customer**:
|
||||
```
|
||||
mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives # the drives parent bind
|
||||
mp9: /var/lib/felhom-agent/guests/<vmid>/bootstrap,mp=/etc/felhom-bootstrap,ro=1 # bootstrap bind
|
||||
```
|
||||
- `provision/backhalf.go:38` calls **mp8 "the dedicated mpN for the single permanent parent bind"**;
|
||||
`capability/manifest.go:55` names `parent-bind-mp8` as a fixed provision step
|
||||
(`pct set <vmid> -mp8 /mnt/felhom-drives`).
|
||||
- mp9 is the bootstrap bind, always at the same in-guest path, host path templated only by vmid.
|
||||
|
||||
So DR bring-up does **not need to read the archive's config at all** to know which mpN are binds — the
|
||||
Felhom guest always has exactly these two, at known in-guest paths. A DR restore can pass a **known,
|
||||
constant** `MountOverrides` (mp8 → throwaway `restoreStorage:1,mp=/mnt/felhom-drives,backup=0`; mp9
|
||||
handled the same or dropped) reusing the **existing `bindMountOverrides` semantics** — no PBS key, no
|
||||
archive parse, no cross-repo schema change. In a real guest-loss DR the customer's drives are
|
||||
re-enrolled separately (the bind is re-established by the enrollment/reconcile path), and the bootstrap
|
||||
is regenerated — so neutralizing them to throwaways at restore time is exactly right (same rationale
|
||||
as restore-test).
|
||||
|
||||
This aligns with the **endgame in `guesthook/heal.go`**: the intermediary-mount re-architecture makes
|
||||
`/mnt/felhom-drives` the ONLY bind source — and that is already realized (mp8 IS "the single permanent
|
||||
parent bind"). The problem is already structurally minimized to two known entries.
|
||||
|
||||
## Recommendation
|
||||
|
||||
**Wire a KNOWN-CONSTANT bind override into `bring-up -mode dr` — small, agent-only — do it as a near
|
||||
follow-up, not deferred.**
|
||||
|
||||
- Implementation sketch (follow-up task, agent-only): in `bringup.go`, for the DR mode / Felhom golden
|
||||
topology, build `MountOverrides` for the known structural binds (mp8 `/mnt/felhom-drives`, mp9
|
||||
bootstrap) and pass them to `RestoreLXC` — **reuse `restoretest.bindMountOverrides`** (lift it to a
|
||||
shared helper; do NOT reinvent). No source-config read, no PBS key, no recipe change.
|
||||
- Prefer the constants over candidate 3's recipe-schema expansion unless/until a guest can carry
|
||||
*non-standard* binds (none today). If that ever changes, extend the recipe (candidate 3) — the
|
||||
agent captures mpN at report-time while alive.
|
||||
- Do NOT pursue candidate 1 (would hand the non-root agent the PBS encryption key — trust-surface
|
||||
regression for no benefit).
|
||||
|
||||
### Cost/benefit on timing
|
||||
- **For:** the override is tiny (two known entries), agent-only, reuses shipped code, and aligns with
|
||||
the already-realized single-parent-bind endgame. It closes the one remaining DR gap.
|
||||
- **Against (mild):** offsite guest-loss DR bring-up is **not yet a shipped customer path**, and the
|
||||
pilot-relevant primitive (`restore-test`) already works via the live-config read (pool healed). So
|
||||
it is not *urgent*.
|
||||
- **Verdict:** low-cost, low-risk, structurally-aligned → worth doing as a **small scoped follow-up**
|
||||
(reuse `bindMountOverrides`, known constants), not a deferral and not a schema-heavy recipe change.
|
||||
|
||||
## Not done here (spike constraints honored)
|
||||
Read-only probes only; no production code; no destructive DR into 9201; the healed restore-test path
|
||||
was exercised (read-only observation of the working primitive). The follow-up implementation is a
|
||||
separate task gated on this finding.
|
||||
@@ -260,3 +260,55 @@ required auth.)
|
||||
H4 (fresh journal), H5 (real escrow-blob locate) — then a clean re-run of P3–P9 + D1 + escrow-DR.
|
||||
4. **F-C2-1:** either make the controller config loader not `os.ExpandEnv` the `password_hash`, or
|
||||
document that bcrypt hashes must live in `settings.json`.
|
||||
|
||||
---
|
||||
|
||||
# Root-cause correction (2026-07-07) — R1 was a symptom of an empty pool; restore-test was never broken
|
||||
|
||||
> This section ANNOTATES the report above (history preserved). Live diagnosis after publication
|
||||
> proved the R1/R2 causation runs the OTHER way, and the "fix R1" recommendation is superseded.
|
||||
|
||||
**What R1 actually was.** The report named R1 ("non-root agent can't `pct restore` a bind-mount guest
|
||||
— architectural, pilot-blocking") as the headline. That inverts symptom and cause. The real chain,
|
||||
every link verified live:
|
||||
|
||||
1. `pvesh get /pools/felhom` → `members: []`. **The felhom pool was EMPTY.**
|
||||
2. Pool-scoped ACL grants (`FelhomAgentGuest` @ `/pool/felhom`, carrying `VM.Audit`/`VM.Allocate`)
|
||||
reach only pool MEMBERS. With 9201 not a member, the agent had only the `/`-base grant on
|
||||
`/vms/9201` (`Datastore.Audit`/`SDN.Use`/`Sys.Audit`) — **no `VM.Audit`**.
|
||||
3. `restore-test` calls `GuestConfig(9201)` to read the mp layout → **403 VM.Audit** → falls to the
|
||||
"restore as-is" fallback.
|
||||
4. Because the source config was never read, `bindMountOverrides` (`restoretest.go:334`) — **which
|
||||
already exists and is correct** — never ran. Raw bind mounts reached PVE → `mp8 … only possible
|
||||
for root`.
|
||||
|
||||
**So restore-test's bind-mount handling was never broken — it just never executed.** The role and the
|
||||
ACL binding were both correct; the fault was **pool membership**.
|
||||
|
||||
**Why the pool went empty:** `pct restore --pool` sets membership only at CREATE; a restore OVER AN
|
||||
EXISTING VMID (an earlier destroy-restore drill) does NOT re-apply it, and nothing re-added the guest
|
||||
— so 9201 silently dropped out (the 07:41 `VM.Audit` 403 predates the 14:09 finale, consistent with
|
||||
an earlier drill).
|
||||
|
||||
**Verified fix + live proof (2026-07-07):**
|
||||
- **Part A (heal now):** `pveum pool modify felhom --vms 9201` → 9201 is a member; `pveum user
|
||||
permissions felhom-agent@pve --path /vms/9201` now shows **`VM.Audit` + `VM.Allocate` + `VM.Backup`**.
|
||||
- **Live restore-test then PASSED for the first time** (agent v0.74.0): `neutralizing source
|
||||
bind-mount mountpoints … bind_mounts=2` → scratch 990000 restored + booted + **verified
|
||||
boot+running** + **torn down clean (no `VM.Allocate` 403)** in 4m35s. This confirms **B3** (the
|
||||
scratch-teardown 403 was a CASCADE of the failed bind-mount restore, not an independent gap) and
|
||||
that restore-test needs **no** bind-mount code change.
|
||||
- **Part B (stop the regression):** `felhom-agent v0.74.0` re-asserts pool membership after any
|
||||
restore-with-`Pool` (`Client.PoolAddVMID` + a bring-up call, warn-not-fail). A destroy-restore no
|
||||
longer silently drops membership.
|
||||
|
||||
**Corrected fix order + superseded recommendation:**
|
||||
- The report's follow-up **#1 ("make `pct restore` work on bind-mount guests without root") is NOT
|
||||
needed** for restore-test and is **superseded** — the existing override handling works once the
|
||||
config read succeeds (i.e. once membership is present).
|
||||
- (A) pool one-liner heals today; (B) v0.74.0 stops the recurrence; restore-test unchanged.
|
||||
- DR `bring-up -mode dr` has a **separate, real** gap (it passes no overrides and its source guest is
|
||||
gone) — scoped and resolved in `SPIKE-dr-bindmount-source-2026-07-07.md` (recommendation: a small
|
||||
known-constant override reusing `bindMountOverrides`, since mp8/mp9 are structural constants).
|
||||
- **F-C2-1** is fixed in `felhom-controller v0.103.0` (config loader no longer `os.ExpandEnv`-corrupts
|
||||
a bcrypt hash; live-proven).
|
||||
|
||||
Reference in New Issue
Block a user