docs: campaign-2 R1/R2 root-cause correction + DR bind-mount source spike

Correction: campaign-2's "R1" (bind-mount restore needs root) was a SYMPTOM of an
empty felhom pool -> agent lacked VM.Audit on 9201 -> restore-test's existing,
correct bindMountOverrides never ran. Live-proven: pool healed (Part A) ->
restore-test PASSES (neutralizes 2 binds, boot+running, clean teardown, 4m35s).
Report rec #1 (make pct restore work on bind mounts) superseded. Fixes shipped:
agent v0.74.0 (pool re-assertion), controller v0.103.0 (F-C2-1).

Spike (SPIKE-dr-bindmount-source-2026-07-07): DR bring-up source options — PBS
pct.conf.blob blocked by encryption under the non-root token; DR recipe lacks mpN;
mp8/mp9 are STRUCTURAL CONSTANTS -> recommend a small known-constant override
reusing bindMountOverrides.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
2026-07-07 19:08:03 +02:00
parent 7714f1f6e4
commit 1a1e42addb
2 changed files with 154 additions and 0 deletions
@@ -260,3 +260,55 @@ required auth.)
H4 (fresh journal), H5 (real escrow-blob locate) — then a clean re-run of P3P9 + D1 + escrow-DR.
4. **F-C2-1:** either make the controller config loader not `os.ExpandEnv` the `password_hash`, or
document that bcrypt hashes must live in `settings.json`.
---
# Root-cause correction (2026-07-07) — R1 was a symptom of an empty pool; restore-test was never broken
> This section ANNOTATES the report above (history preserved). Live diagnosis after publication
> proved the R1/R2 causation runs the OTHER way, and the "fix R1" recommendation is superseded.
**What R1 actually was.** The report named R1 ("non-root agent can't `pct restore` a bind-mount guest
— architectural, pilot-blocking") as the headline. That inverts symptom and cause. The real chain,
every link verified live:
1. `pvesh get /pools/felhom` → `members: []`. **The felhom pool was EMPTY.**
2. Pool-scoped ACL grants (`FelhomAgentGuest` @ `/pool/felhom`, carrying `VM.Audit`/`VM.Allocate`)
reach only pool MEMBERS. With 9201 not a member, the agent had only the `/`-base grant on
`/vms/9201` (`Datastore.Audit`/`SDN.Use`/`Sys.Audit`) — **no `VM.Audit`**.
3. `restore-test` calls `GuestConfig(9201)` to read the mp layout → **403 VM.Audit** → falls to the
"restore as-is" fallback.
4. Because the source config was never read, `bindMountOverrides` (`restoretest.go:334`) — **which
already exists and is correct** — never ran. Raw bind mounts reached PVE → `mp8 … only possible
for root`.
**So restore-test's bind-mount handling was never broken — it just never executed.** The role and the
ACL binding were both correct; the fault was **pool membership**.
**Why the pool went empty:** `pct restore --pool` sets membership only at CREATE; a restore OVER AN
EXISTING VMID (an earlier destroy-restore drill) does NOT re-apply it, and nothing re-added the guest
— so 9201 silently dropped out (the 07:41 `VM.Audit` 403 predates the 14:09 finale, consistent with
an earlier drill).
**Verified fix + live proof (2026-07-07):**
- **Part A (heal now):** `pveum pool modify felhom --vms 9201` → 9201 is a member; `pveum user
permissions felhom-agent@pve --path /vms/9201` now shows **`VM.Audit` + `VM.Allocate` + `VM.Backup`**.
- **Live restore-test then PASSED for the first time** (agent v0.74.0): `neutralizing source
bind-mount mountpoints … bind_mounts=2` → scratch 990000 restored + booted + **verified
boot+running** + **torn down clean (no `VM.Allocate` 403)** in 4m35s. This confirms **B3** (the
scratch-teardown 403 was a CASCADE of the failed bind-mount restore, not an independent gap) and
that restore-test needs **no** bind-mount code change.
- **Part B (stop the regression):** `felhom-agent v0.74.0` re-asserts pool membership after any
restore-with-`Pool` (`Client.PoolAddVMID` + a bring-up call, warn-not-fail). A destroy-restore no
longer silently drops membership.
**Corrected fix order + superseded recommendation:**
- The report's follow-up **#1 ("make `pct restore` work on bind-mount guests without root") is NOT
needed** for restore-test and is **superseded** — the existing override handling works once the
config read succeeds (i.e. once membership is present).
- (A) pool one-liner heals today; (B) v0.74.0 stops the recurrence; restore-test unchanged.
- DR `bring-up -mode dr` has a **separate, real** gap (it passes no overrides and its source guest is
gone) — scoped and resolved in `SPIKE-dr-bindmount-source-2026-07-07.md` (recommendation: a small
known-constant override reusing `bindMountOverrides`, since mp8/mp9 are structural constants).
- **F-C2-1** is fixed in `felhom-controller v0.103.0` (config loader no longer `os.ExpandEnv`-corrupts
a bcrypt hash; live-proven).