docs: campaign-2 R1/R2 root-cause correction + DR bind-mount source spike
Correction: campaign-2's "R1" (bind-mount restore needs root) was a SYMPTOM of an empty felhom pool -> agent lacked VM.Audit on 9201 -> restore-test's existing, correct bindMountOverrides never ran. Live-proven: pool healed (Part A) -> restore-test PASSES (neutralizes 2 binds, boot+running, clean teardown, 4m35s). Report rec #1 (make pct restore work on bind mounts) superseded. Fixes shipped: agent v0.74.0 (pool re-assertion), controller v0.103.0 (F-C2-1). Spike (SPIKE-dr-bindmount-source-2026-07-07): DR bring-up source options — PBS pct.conf.blob blocked by encryption under the non-root token; DR recipe lacks mpN; mp8/mp9 are STRUCTURAL CONSTANTS -> recommend a small known-constant override reusing bindMountOverrides. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
@@ -260,3 +260,55 @@ required auth.)
|
||||
H4 (fresh journal), H5 (real escrow-blob locate) — then a clean re-run of P3–P9 + D1 + escrow-DR.
|
||||
4. **F-C2-1:** either make the controller config loader not `os.ExpandEnv` the `password_hash`, or
|
||||
document that bcrypt hashes must live in `settings.json`.
|
||||
|
||||
---
|
||||
|
||||
# Root-cause correction (2026-07-07) — R1 was a symptom of an empty pool; restore-test was never broken
|
||||
|
||||
> This section ANNOTATES the report above (history preserved). Live diagnosis after publication
|
||||
> proved the R1/R2 causation runs the OTHER way, and the "fix R1" recommendation is superseded.
|
||||
|
||||
**What R1 actually was.** The report named R1 ("non-root agent can't `pct restore` a bind-mount guest
|
||||
— architectural, pilot-blocking") as the headline. That inverts symptom and cause. The real chain,
|
||||
every link verified live:
|
||||
|
||||
1. `pvesh get /pools/felhom` → `members: []`. **The felhom pool was EMPTY.**
|
||||
2. Pool-scoped ACL grants (`FelhomAgentGuest` @ `/pool/felhom`, carrying `VM.Audit`/`VM.Allocate`)
|
||||
reach only pool MEMBERS. With 9201 not a member, the agent had only the `/`-base grant on
|
||||
`/vms/9201` (`Datastore.Audit`/`SDN.Use`/`Sys.Audit`) — **no `VM.Audit`**.
|
||||
3. `restore-test` calls `GuestConfig(9201)` to read the mp layout → **403 VM.Audit** → falls to the
|
||||
"restore as-is" fallback.
|
||||
4. Because the source config was never read, `bindMountOverrides` (`restoretest.go:334`) — **which
|
||||
already exists and is correct** — never ran. Raw bind mounts reached PVE → `mp8 … only possible
|
||||
for root`.
|
||||
|
||||
**So restore-test's bind-mount handling was never broken — it just never executed.** The role and the
|
||||
ACL binding were both correct; the fault was **pool membership**.
|
||||
|
||||
**Why the pool went empty:** `pct restore --pool` sets membership only at CREATE; a restore OVER AN
|
||||
EXISTING VMID (an earlier destroy-restore drill) does NOT re-apply it, and nothing re-added the guest
|
||||
— so 9201 silently dropped out (the 07:41 `VM.Audit` 403 predates the 14:09 finale, consistent with
|
||||
an earlier drill).
|
||||
|
||||
**Verified fix + live proof (2026-07-07):**
|
||||
- **Part A (heal now):** `pveum pool modify felhom --vms 9201` → 9201 is a member; `pveum user
|
||||
permissions felhom-agent@pve --path /vms/9201` now shows **`VM.Audit` + `VM.Allocate` + `VM.Backup`**.
|
||||
- **Live restore-test then PASSED for the first time** (agent v0.74.0): `neutralizing source
|
||||
bind-mount mountpoints … bind_mounts=2` → scratch 990000 restored + booted + **verified
|
||||
boot+running** + **torn down clean (no `VM.Allocate` 403)** in 4m35s. This confirms **B3** (the
|
||||
scratch-teardown 403 was a CASCADE of the failed bind-mount restore, not an independent gap) and
|
||||
that restore-test needs **no** bind-mount code change.
|
||||
- **Part B (stop the regression):** `felhom-agent v0.74.0` re-asserts pool membership after any
|
||||
restore-with-`Pool` (`Client.PoolAddVMID` + a bring-up call, warn-not-fail). A destroy-restore no
|
||||
longer silently drops membership.
|
||||
|
||||
**Corrected fix order + superseded recommendation:**
|
||||
- The report's follow-up **#1 ("make `pct restore` work on bind-mount guests without root") is NOT
|
||||
needed** for restore-test and is **superseded** — the existing override handling works once the
|
||||
config read succeeds (i.e. once membership is present).
|
||||
- (A) pool one-liner heals today; (B) v0.74.0 stops the recurrence; restore-test unchanged.
|
||||
- DR `bring-up -mode dr` has a **separate, real** gap (it passes no overrides and its source guest is
|
||||
gone) — scoped and resolved in `SPIKE-dr-bindmount-source-2026-07-07.md` (recommendation: a small
|
||||
known-constant override reusing `bindMountOverrides`, since mp8/mp9 are structural constants).
|
||||
- **F-C2-1** is fixed in `felhom-controller v0.103.0` (config loader no longer `os.ExpandEnv`-corrupts
|
||||
a bcrypt hash; live-proven).
|
||||
|
||||
Reference in New Issue
Block a user