From 1a1e42addb7300160133023e7773879c5b081123 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 7 Jul 2026 19:08:03 +0200 Subject: [PATCH] docs: campaign-2 R1/R2 root-cause correction + DR bind-mount source spike MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Correction: campaign-2's "R1" (bind-mount restore needs root) was a SYMPTOM of an empty felhom pool -> agent lacked VM.Audit on 9201 -> restore-test's existing, correct bindMountOverrides never ran. Live-proven: pool healed (Part A) -> restore-test PASSES (neutralizes 2 binds, boot+running, clean teardown, 4m35s). Report rec #1 (make pct restore work on bind mounts) superseded. Fixes shipped: agent v0.74.0 (pool re-assertion), controller v0.103.0 (F-C2-1). Spike (SPIKE-dr-bindmount-source-2026-07-07): DR bring-up source options — PBS pct.conf.blob blocked by encryption under the non-root token; DR recipe lacks mpN; mp8/mp9 are STRUCTURAL CONSTANTS -> recommend a small known-constant override reusing bindMountOverrides. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 --- .../SPIKE-dr-bindmount-source-2026-07-07.md | 102 ++++++++++++++++++ documentation/tests/CAMPAIGN-2-2026-07-07.md | 52 +++++++++ 2 files changed, 154 insertions(+) create mode 100644 documentation/audits/SPIKE-dr-bindmount-source-2026-07-07.md diff --git a/documentation/audits/SPIKE-dr-bindmount-source-2026-07-07.md b/documentation/audits/SPIKE-dr-bindmount-source-2026-07-07.md new file mode 100644 index 0000000..119b946 --- /dev/null +++ b/documentation/audits/SPIKE-dr-bindmount-source-2026-07-07.md @@ -0,0 +1,102 @@ +# SPIKE — where does the DR bind-mount layout come from when the source guest is gone? (2026-07-07) + +> Findings-only spike (no production code). Companion to campaign-2 R1/R2. Scope: `bring-up +> -mode dr` (true guest-loss DR) passes **no `MountOverrides`** (`bringup.go:222`), so it has no way +> to neutralize the source guest's bind-mount mountpoints — and unlike `restore-test`, the source +> guest is GONE, so `GuestConfig(srcVMID)` can't supply the layout. The open question is the +> **authoritative, reachable source** of the mpN config. Read-only probes on the demo (agent +> v0.74.0, pool healed). + +## Context — why restore-test works but DR doesn't (yet) + +`restore-test` reads the mpN layout from the **live** source guest (`GuestConfig(9201)`), computes +`bindMountOverrides` (`restoretest.go:334`), and passes them to `RestoreLXC{MountOverrides:…}`. That +whole chain is **correct and already shipped** — the campaign proved it once the pool was healed +(`VM.Audit` restored): the 2026-07-07 live run logged `neutralizing source bind-mount mountpoints … +bind_mounts=2` and passed (scratch 990000 restored+booted+verified+torn-down, 4m35s). **DR is the +only gap**, and only because its source is gone. + +## Candidate sources — probed + +### 1. PBS `pct.conf.blob` (fetch the config member standalone) — **BLOCKED by encryption** +The snapshot carries it (`proxmox-backup-client snapshot files ct/9201/` → +`catalog.pcat1 client.log index.json pct.conf root.pxar`). The pool-scoped token +(`felhom@pbs!n100`) **authenticates and reaches** the snapshot, but the fetch fails: +``` +$ proxmox-backup-client restore ct/9201/2026-07-07T06:11:58Z pct.conf.blob - --repository +Error: missing key - manifest was created with key 01:36:e9:fe:e1:ee:3d:7a… +``` +The datastore is **encrypted**, and the non-root agent does **not** hold the datastore key — PVE +holds it root-only at `/etc/pve/priv/storage/felhom-pbs.enc` (that is *why* restore-test works via the +PVE-API `pct restore`: PVE, as root, does the crypto; the agent never touches the key). Fetching +`pct.conf.blob` directly would require giving the non-root agent the PBS encryption key — a real +expansion of its crypto/trust surface. **Not viable under the current (correct) trust model.** + +### 2. vzdump archive `./pct/config` (local tar) — **partial: local tier only** +A local `vzdump-lxc--*.tar.zst` carries `./pct/config` (plaintext, standalone-readable). But +**offsite/guest-loss DR restores from PBS**, which is encrypted (candidate 1), and local vzdump +archives are on the box being lost. So this helps only the local-archive restore-over-existing case +(where the live config is usually still readable anyway). Not the offsite-DR answer. + +### 3. Hub DR recipe `Guests` blob — **not ready (schema gap), but structurally the cleanest** +`store/dr_recipe.go` carries `Guests json.RawMessage`, but the host-half builder populates +`DRGuest{VMID, Cores, MemoryBytes, DiskBytes}` (`felhom-agent/internal/hub/dr_recipe.go`) — **sizing +only, no mpN layout**. So the recipe does NOT currently capture the bind-mount layout. It COULD: +the host-half is built at **report time, while the guest is ALIVE**, so the agent can read the mpN +layout then (no key, no archive parse) and store it. Cost: extend `DRGuest` + the report builder +(agent) + the consumer (bring-up) across two repos. Robust, but the heaviest change — and it must be +kept authoritative vs the archive actually being restored (a stale recipe → wrong overrides). + +## The finding that changes the recommendation — the binds are STRUCTURAL CONSTANTS + +The Felhom golden topology's bind mounts are **fixed, not per-customer**: +``` +mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives # the drives parent bind +mp9: /var/lib/felhom-agent/guests//bootstrap,mp=/etc/felhom-bootstrap,ro=1 # bootstrap bind +``` +- `provision/backhalf.go:38` calls **mp8 "the dedicated mpN for the single permanent parent bind"**; + `capability/manifest.go:55` names `parent-bind-mp8` as a fixed provision step + (`pct set -mp8 /mnt/felhom-drives`). +- mp9 is the bootstrap bind, always at the same in-guest path, host path templated only by vmid. + +So DR bring-up does **not need to read the archive's config at all** to know which mpN are binds — the +Felhom guest always has exactly these two, at known in-guest paths. A DR restore can pass a **known, +constant** `MountOverrides` (mp8 → throwaway `restoreStorage:1,mp=/mnt/felhom-drives,backup=0`; mp9 +handled the same or dropped) reusing the **existing `bindMountOverrides` semantics** — no PBS key, no +archive parse, no cross-repo schema change. In a real guest-loss DR the customer's drives are +re-enrolled separately (the bind is re-established by the enrollment/reconcile path), and the bootstrap +is regenerated — so neutralizing them to throwaways at restore time is exactly right (same rationale +as restore-test). + +This aligns with the **endgame in `guesthook/heal.go`**: the intermediary-mount re-architecture makes +`/mnt/felhom-drives` the ONLY bind source — and that is already realized (mp8 IS "the single permanent +parent bind"). The problem is already structurally minimized to two known entries. + +## Recommendation + +**Wire a KNOWN-CONSTANT bind override into `bring-up -mode dr` — small, agent-only — do it as a near +follow-up, not deferred.** + +- Implementation sketch (follow-up task, agent-only): in `bringup.go`, for the DR mode / Felhom golden + topology, build `MountOverrides` for the known structural binds (mp8 `/mnt/felhom-drives`, mp9 + bootstrap) and pass them to `RestoreLXC` — **reuse `restoretest.bindMountOverrides`** (lift it to a + shared helper; do NOT reinvent). No source-config read, no PBS key, no recipe change. +- Prefer the constants over candidate 3's recipe-schema expansion unless/until a guest can carry + *non-standard* binds (none today). If that ever changes, extend the recipe (candidate 3) — the + agent captures mpN at report-time while alive. +- Do NOT pursue candidate 1 (would hand the non-root agent the PBS encryption key — trust-surface + regression for no benefit). + +### Cost/benefit on timing +- **For:** the override is tiny (two known entries), agent-only, reuses shipped code, and aligns with + the already-realized single-parent-bind endgame. It closes the one remaining DR gap. +- **Against (mild):** offsite guest-loss DR bring-up is **not yet a shipped customer path**, and the + pilot-relevant primitive (`restore-test`) already works via the live-config read (pool healed). So + it is not *urgent*. +- **Verdict:** low-cost, low-risk, structurally-aligned → worth doing as a **small scoped follow-up** + (reuse `bindMountOverrides`, known constants), not a deferral and not a schema-heavy recipe change. + +## Not done here (spike constraints honored) +Read-only probes only; no production code; no destructive DR into 9201; the healed restore-test path +was exercised (read-only observation of the working primitive). The follow-up implementation is a +separate task gated on this finding. diff --git a/documentation/tests/CAMPAIGN-2-2026-07-07.md b/documentation/tests/CAMPAIGN-2-2026-07-07.md index 0448923..e6a49dc 100644 --- a/documentation/tests/CAMPAIGN-2-2026-07-07.md +++ b/documentation/tests/CAMPAIGN-2-2026-07-07.md @@ -260,3 +260,55 @@ required auth.) H4 (fresh journal), H5 (real escrow-blob locate) — then a clean re-run of P3–P9 + D1 + escrow-DR. 4. **F-C2-1:** either make the controller config loader not `os.ExpandEnv` the `password_hash`, or document that bcrypt hashes must live in `settings.json`. + +--- + +# Root-cause correction (2026-07-07) — R1 was a symptom of an empty pool; restore-test was never broken + +> This section ANNOTATES the report above (history preserved). Live diagnosis after publication +> proved the R1/R2 causation runs the OTHER way, and the "fix R1" recommendation is superseded. + +**What R1 actually was.** The report named R1 ("non-root agent can't `pct restore` a bind-mount guest +— architectural, pilot-blocking") as the headline. That inverts symptom and cause. The real chain, +every link verified live: + +1. `pvesh get /pools/felhom` → `members: []`. **The felhom pool was EMPTY.** +2. Pool-scoped ACL grants (`FelhomAgentGuest` @ `/pool/felhom`, carrying `VM.Audit`/`VM.Allocate`) + reach only pool MEMBERS. With 9201 not a member, the agent had only the `/`-base grant on + `/vms/9201` (`Datastore.Audit`/`SDN.Use`/`Sys.Audit`) — **no `VM.Audit`**. +3. `restore-test` calls `GuestConfig(9201)` to read the mp layout → **403 VM.Audit** → falls to the + "restore as-is" fallback. +4. Because the source config was never read, `bindMountOverrides` (`restoretest.go:334`) — **which + already exists and is correct** — never ran. Raw bind mounts reached PVE → `mp8 … only possible + for root`. + +**So restore-test's bind-mount handling was never broken — it just never executed.** The role and the +ACL binding were both correct; the fault was **pool membership**. + +**Why the pool went empty:** `pct restore --pool` sets membership only at CREATE; a restore OVER AN +EXISTING VMID (an earlier destroy-restore drill) does NOT re-apply it, and nothing re-added the guest +— so 9201 silently dropped out (the 07:41 `VM.Audit` 403 predates the 14:09 finale, consistent with +an earlier drill). + +**Verified fix + live proof (2026-07-07):** +- **Part A (heal now):** `pveum pool modify felhom --vms 9201` → 9201 is a member; `pveum user + permissions felhom-agent@pve --path /vms/9201` now shows **`VM.Audit` + `VM.Allocate` + `VM.Backup`**. +- **Live restore-test then PASSED for the first time** (agent v0.74.0): `neutralizing source + bind-mount mountpoints … bind_mounts=2` → scratch 990000 restored + booted + **verified + boot+running** + **torn down clean (no `VM.Allocate` 403)** in 4m35s. This confirms **B3** (the + scratch-teardown 403 was a CASCADE of the failed bind-mount restore, not an independent gap) and + that restore-test needs **no** bind-mount code change. +- **Part B (stop the regression):** `felhom-agent v0.74.0` re-asserts pool membership after any + restore-with-`Pool` (`Client.PoolAddVMID` + a bring-up call, warn-not-fail). A destroy-restore no + longer silently drops membership. + +**Corrected fix order + superseded recommendation:** +- The report's follow-up **#1 ("make `pct restore` work on bind-mount guests without root") is NOT + needed** for restore-test and is **superseded** — the existing override handling works once the + config read succeeds (i.e. once membership is present). +- (A) pool one-liner heals today; (B) v0.74.0 stops the recurrence; restore-test unchanged. +- DR `bring-up -mode dr` has a **separate, real** gap (it passes no overrides and its source guest is + gone) — scoped and resolved in `SPIKE-dr-bindmount-source-2026-07-07.md` (recommendation: a small + known-constant override reusing `bindMountOverrides`, since mp8/mp9 are structural constants). +- **F-C2-1** is fixed in `felhom-controller v0.103.0` (config loader no longer `os.ExpandEnv`-corrupts + a bcrypt hash; live-proven).