felhom.eu/documentation/tests/CAMPAIGN-2-2026-07-07.md

CAMPAIGN 2 — DR primitives · self-update · escrow · storage · faults · endurance (2026-07-07)

Unattended full-system validation on the demo node (felhom-pve / guest 9201), run during the day 2026-07-07 (07:20–14:11 CEST). No production code changed. Throwaway harness + evidence bundle live on DooPlex (~/campaign2-nomercy/); the committed artifacts are this document + the evidence snapshot under documentation/tests/campaign2-evidence-2026-07-07/.

Headline — read this first. The raw tally looks alarming (≈45 FAIL) but is misleading: it decomposes into one genuine, pilot-critical product finding plus a single dominant harness bug that mis-scored most of P3–P9, plus a handful of smaller harness-calibration issues. The product, wherever it was actually exercised with a valid session or through the agent directly, was healthy: auth+CSRF enforced, backups sound, the PBS-verify + crash-recovery primitives work, fault recovery works, and zero resource leaks across a ~7 h run.

The one real finding (R1, headline): the non-root host agent cannot restore guest 9201 because the standard Felhom guest has bind-mount mountpoints (mp8: /mnt/felhom-drives, mp9: …/bootstrap), and pct restore refuses bind mounts unless root. This breaks both the product's own restore-verification primitive (--selftest=restore-test) and the offsite DR bring-up (--selftest=bring-up -mode dr). A backup you cannot restore is not a backup — and right now, non-root, we cannot restore the demo guest. This is exactly what "exercise the real primitives" was meant to surface.

---

Run header

Item	Value
Date	2026-07-07, 07:20–14:11 CEST
Host / Guest	felhom-pve (192.168.0.162), node demo-felhom / guest 9201 (LAN 192.168.0.127)
Controller / Agent / Hub	v0.102.0 (cd0ebd2) / v0.73.0 (e04b75e) / v0.35.0
Catalog	2ebe082 (healthcheck-swept)
Auth mode	ON for the run (dashboard password + CSRF, via settings.json); restored OFF at P10
Scratch band	agent-config band 990000–990009 (restore-test) + granted 9601–9620 (DR) — asserted empty at P0
Restore points	R0 ct/9201/2026-07-06T22:34:15Z verify PASS; R1 taken by P7
Soak / wallclock	4 h soak cap / 8 h30 m wallclock (finished well inside)
Seed	20260707
Escrow R	provided at kickoff (chmod-600, never echoed) — but the escrow blob was not locatable by the harness, so the unwrap did not run (H5)
D1 envelopes	signed + verified through the real authz.Verifier pipeline at P0; moved early (after P3) to stay inside the 10:07Z TTL — see note below

D1-early adaptation (logged): the envelopes had a 12 h TTL from the previous evening (expire 2026-07-07T10:07Z). The as-ordered D1 slot (after the 4 h soak) would have run past expiry. Because 0.73.1 is the same source tree as the deployed 0.73.0 (only the -X main.version string differs), running P4–P9 on 0.73.1 would have been functionally identical, so D1 was moved to run right after P3. (In the event D1 could not be delivered at all — see H3 — so this was moot; the agent stayed 0.73.0.)

---

Result tally (raw, then triaged)

Status	Raw
PASS	15
FAIL	45
SKIP	3
SOAK "breaks"	170 (all one artifact)

After triage the 45 FAIL decompose as:

• 1 genuine product finding (R1/R2, surfaced by 3 independent agent-side tests: T-SELFRESTORE, T-DR-BRINGUP, and contributing to the P9/DR chain).
• ~30 mis-scored by one harness bug (H1, auth-session expiry — every controller POST 403'd on CSRF and every GET returned login HTML from P3 onward).
• 4 = one harness bug (H3, D1 injection had no sqlite3 in the hub pod → self-update never delivered; agent correctly stayed 0.73.0).
• 2 = stale dryrun journal entries (H4).
• 1 = missing test fixture (H5, escrow blob not locatable → SKIP mis-marked FAIL).
• The remainder are the soak (H2) and derived/dependent skips.

---

THE genuine finding — R1 + R2 (pilot-critical)

R1 — non-root pct restore of a bind-mount guest fails ("only possible for root")

• Symptom (verbatim, 3 tests): unable to restore CT <vmid> - restoring 'mp8' to bind mount is only possible for root
  • T-SELFRESTORE — --selftest=restore-test into scratch 990000 → FAIL in 2.2 s (evidence/T-SELFRESTORE/restore-test.out, pass:false).
  • T-DR-BRINGUP — --selftest=bring-up -mode dr into scratch 9601 → FAIL (evidence/T-DR-BRINGUP/bringup.out).
• Root cause (confirmed): guest 9201's config carries bind-mount mountpoints:

  mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives     # the enrolled-drives bind
  mp9: /var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=1
  

  Proxmox pct restore refuses to restore a bind-mount entry unless it runs as root. The agent runs non-root (felhom-agent), by design (de-privilege). So the restore of the standard Felhom guest topology fails.
• Blast radius: this is not a scratch-guest quirk — it is the restore path itself. The product's own "is this backup actually restorable?" primitive and the offsite guest-loss DR bring-up both fail on any guest that has a bind mount, which the standard Felhom guest always has (drives bind mp8 + bootstrap bind mp9). This is the single most important thing to know before pilot.
• Not fixed (report-only). Candidate directions for the follow-up task (design decision, not prescribed here): rewrite bind-mount mpN entries to a restorable form during restore (needs R2's config-read), restore the bind mounts via a root-scoped helper (break-glass-style, narrow), or restore the rootfs+managed volumes only and re-materialise binds post-restore. Cross-check with the offsite-WG DR design docs before choosing.

R2 — agent token is missing VM.Audit (and VM.Allocate on scratch) — links into R1

• restore-test: could not read source config for mp overrides (restoring as-is) … GET /nodes/demo-felhom/lxc/9201/config -> HTTP 403: permission denied at /vms/9201 (missing privilege VM.Audit). Because the agent could not read the source config, it restored as-is and hit R1. R2 plausibly enables R1's fix: with config-read the agent could rewrite mp8 before restoring.
• DELETE /nodes/demo-felhom/lxc/990000 … missing privilege VM.Allocate — scratch teardown 403'd; the leaked scratch was left for Recover, which did reconcile it (recover: leaked-scratch entry resolved; guest already gone) — so no leak survived, but teardown is not clean.
• Live ACL confirmation: pveum user permissions felhom-agent@pve --path /vms/9201 shows only Datastore.Audit + Sys.Audit — not VM.Audit. The FelhomAgentGuest role does carry VM.Audit/VM.Allocate, but the grant on /pool/felhom is not resolving to VM.Audit at the per-guest path the restore path checks. (Related to the A1 pool-audit thread.)

---

Other real observations

• F-C2-1 (kickoff, confirmed): setting the dashboard password_hash in controller.yaml does not take effect — the config loader runs os.ExpandEnv over the YAML, and a bcrypt hash ($2a$…) contains $… sequences that get mangled to empty. Worked around by writing the hash to settings.json (which is not env-expanded). Product-relevant: any operator putting a bcrypt hash in controller.yaml gets silent auth-bypass. Report-only.
• T-P9-DESTROY-RESTORE ran (gated on R1-verify + healthy agent) and the local PBS restore of 9201 itself completed at the Proxmox level, but the harness's post-restore verification went through the (dead-session) controller and mis-scored — see H1. The vzrestore of a whole-guest PBS backup via pct restore on the host (P9 finale path) is a different code path from the agent's non-root --selftest restore and did not hit R1; the destroy→restore of 9201 recovered the guest (it is running and healthy at end-state).

---

Strong positive results (real signal — do not lose these under the FAIL count)

No resource leaks — the endurance headline (flight recorder, 351 samples over ~7 h)

Metric	Start	End	Min	Max	Verdict
agent RSS (KB)	22 116	19 936	17 924	22 792	stable, trended down
agent fd	79	54	—	83	no leak
agent sockets	70	46	—	74	no idle-socket leak (the agentapi precedent is absent)
controller fd	0*	15	0*	30	no runaway (*0 = sampler pgrep miss)
controller latency	—	—	0.008 s	0.079 s	responsive all run

The controller answered every flight-recorder probe in 8–79 ms across the whole run; only ~10 of 351 samples were 502/000 (during deliberate restarts/reboots), all recovered. (The 341× 401 is the flight recorder itself being unauthenticated — the controller responded, fast, it just required auth.)

What actually passed (valid-session or agent-direct — trustworthy verdicts)

• P1: all pages 200; vaultwarden fresh-deploys healthy (_ENABLE_SMTP=false, restarts=0 — F1 fix re-confirmed); canary plant; lifecycle; sync-idempotent; and the new auth+CSRF grant is enforced — T-AUTH-1 (wrong-pw refused, unauth API blocked), T-CSRF (tokenless POST → 403, tokened accepted), T-AUTH-2 all PASS.
• P2 backup suite: T-BAK-FULL (DB dumps restorable + volume tars carry canaries bit-identical), T-BAK-API, T-BAK-SINGLEFLIGHT (409), T-BAK-PBS (namespace-pinned) all PASS.
• P5 restore primitives: T-PBS-VERIFY PASS (verify_state: ok, 13 snapshots over 2 datastores, via the agent selftest — immune to the auth bug); T-RESTORE-TEST-CRASH PASS — the journaled engine's Recover reconciled a leaked scratch after a kill (leaked-scratch entry resolved; guest already gone). Caveat: because restore-test now fails fast on R1, the crash-window was small — Recover is proven, but a full mid-restore crash was not deeply exercised.
• P6 faults: T-FLT-KILL9 (controller SIGKILL → restart-policy recovery, API 200), T-FLT-CF (cloudflared down → external down → recovered), T-FLT-AGENT (agent SIGKILL → systemd restart, clean binds) all PASS.
• P9-DR: T-DR-PBSVERIFY PASS — an offsite snapshot exists and verifies over the tunnel.

---

Harness bugs found (the real story behind the FAIL count — product NOT implicated)

• H1 — auth-session expiry (DOMINANT). The harness logs in once (P1) and never re-authenticates. Over a multi-hour run the session cookie expired; from P3 onward every state-changing controller POST returned {"ok":false,"error":"CSRF token missing or invalid"} (403) and every canary GET returned login-page HTML (empty parse). Proof: evidence/T-STG-ENROLL/init1.txt (CSRF-403), evidence/T-RST-VW/restore_resp.txt (302→login), canary_check.txt (got= empty). Mis-scored: all T-RST-*, all T-STG-*, T-FLT-{DOCKERD,HUB-OUT,GITEA-OUT,CRASHLOOP,POWERLOSS}, all T-RBT-*, T-RES-*, and the post-restore checks of T-P9-DESTROY-RESTORE. Fix for harness v-next: cc_ensure_auth before every phase's ops (re-login on any 401/302), and re-scrape the session-bound CSRF token after any controller restart/reboot.
• H2 — soak responsiveness probe (cc_code / expects 200; auth → 302→login). 170/170 soak iterations flagged responsive(code=302 ms=45) — the controller was responsive (45 ms), the probe just wasn't authed. T-SOAK-LOOP FAIL is entirely this. Fix: probe an authed page (or accept 302).
• H3 — D1 injection has no sqlite3 in the hub pod. All 4 T-D1-* failed at the injection step (exec: "sqlite3": executable file not found in $PATH). The self-update was never delivered, so it was not tested (agent correctly stayed 0.73.0). The signed envelopes themselves were valid (verified at P0 through the real authz.Verifier). Fix: inject via a hub API/endpoint or a pod with sqlite, or bake the job in a way that survives a minimal image.
• H4 — dryrun polluted the journal. The pre-launch run.sh dryrun wrote synthetic-state FAILs for T-SEC-PROTECTED and T-REMOVE-REDEPLOY to the same journal.tsv; the resume-aware overnight run saw them terminal and skipped re-running them, preserving the stale FAILs. Fix: fresh journal per run (or a dryrun-scoped journal).
• H5 — escrow blob not locatable. T-DR-ESCROW → SKIP-reason: no escrow blob reachable (local+hub). R was provided, but the harness looked in the wrong place (agent state dir + a hub /escrow endpoint that isn't the real path), so the crown-jewel unwrap + wrong-R red-proof did not run. Marked FAIL, really a SKIP. Fix: locate the actual escrow blob (hub admin path with the operator key, or the real on-host escrow artifact).
• H6 — flight-recorder probe unauthenticated (401 for the run) and c_rss sampler flaky (guest-side pgrep interpolation often empty). Cosmetic — latency/leak signal still usable.

---

Restore-primitive evidence (the campaign's purpose — verdicts)

Primitive	Verdict	Evidence
--selftest=pbs-verify	WORKS	verify_state: ok, 13 snapshots / 2 datastores
journaled engine.Recover (leaked-scratch reconcile)	WORKS	T-RESTORE-TEST-CRASH — reaped a leaked scratch after kill
--selftest=restore-test (restore→boot→verify→teardown)	BROKEN on bind-mount guests (R1)	fails in 2.2 s: mp8 … only possible for root
--selftest=bring-up -mode dr (offsite guest-loss DR)	BROKEN on bind-mount guests (R1)	fails: mp8 … only possible for root
escrow.Consume(R) unwrap (+ wrong-R red-proof)	NOT RUN (H5 — blob not located)	—
local whole-guest PBS destroy→restore (host pct restore, P9)	guest recovered	9201 running + healthy at end-state

---

Delta vs Campaign 1

Dimension	Campaign 1 (2026-07-06)	Campaign 2 (2026-07-07)
Auth	OFF (XFAIL)	ON — enforcement proven (T-AUTH/T-CSRF PASS) … but broke the harness mid-run (H1)
Restore primitives	not exercised	exercised — surfaced R1 (the headline)
Resource leaks	not sampled	sampled — none (agent RSS/fd/sockets stable)
DR bring-up / escrow	not attempted	attempted — R1 blocks bring-up; escrow blocked by H5
Self-update (D1)	shipped v0.70.0 (prior)	not delivered this run (H3)
Dominant mis-score cause	~18 harness bugs (auth-off era)	1 dominant harness bug (auth-on session expiry)

---

What this campaign STILL does not prove

• The async restore family end-to-end through the UI under load — H1 killed the harness's restore tests. (It was proven in the P1–P3 re-run 2026-07-06: 0.018 s internal / 0.235 s external, canary bit-identical.)
• Storage F2 lifecycle at scale — all T-STG-* are H1 artifacts. (F2 eject/decommission were proven live in the v0.73.0 acceptance 2026-07-06.)
• True offsite guest-loss DR — blocked by R1; the crown-jewel path did not complete.
• Escrow recovery-code unwrap — not run (H5).
• Agent self-update — not delivered (H3).
• True CGNAT; multi-customer hub scale; real hardware failure.

---

End state (demo node healed)

• Guest 9201 running + healthy, controller v0.102.0 healthy, 25 apps up, 0 unhealthy.
• Agent v0.73.0 active (self-update never applied — H3 — so no version drift; the 0.73.1 artifact remains published in Gitea, dormant).
• Auth restored OFF (pre-campaign state; root page 200). Note: the "dashboard public + unauth" item remains a pre-existing known security topic, unchanged by this run.
• Notifications restored to the exact pre-state (email nagyfenyvesi.viktor@gmail.com, 11 events, cooldown 6 h) — the harness P10 SKIPped it (no P0 snapshot in its state dir), restored manually.
• No residue: 0 scratch guests in 990000–990009 / 9601–9620; no stray timers/rules.
• Controller auto-update was already false pre-campaign (unchanged).

Follow-ups (for project-Claude / a future task — not done here)

1. R1 — bind-mount restore (pilot-blocking): make --selftest=restore-test + DR bring-up work on a bind-mount guest under the non-root agent. Design decision — cross-check the DR docs.
2. R2 — grant the agent VM.Audit (+ VM.Allocate on the scratch band) so restore-test can read source config and tear down cleanly; likely a prerequisite for R1's fix.
3. Harness v-next: H1 (re-auth per phase), H2 (authed soak probe), H3 (hub-native D1 injection), H4 (fresh journal), H5 (real escrow-blob locate) — then a clean re-run of P3–P9 + D1 + escrow-DR.
4. F-C2-1: either make the controller config loader not os.ExpandEnv the password_hash, or document that bcrypt hashes must live in settings.json.

---

Root-cause correction (2026-07-07) — R1 was a symptom of an empty pool; restore-test was never broken

This section ANNOTATES the report above (history preserved). Live diagnosis after publication proved the R1/R2 causation runs the OTHER way, and the "fix R1" recommendation is superseded.

What R1 actually was. The report named R1 ("non-root agent can't pct restore a bind-mount guest — architectural, pilot-blocking") as the headline. That inverts symptom and cause. The real chain, every link verified live:

1. pvesh get /pools/felhom → members: []. The felhom pool was EMPTY.
2. Pool-scoped ACL grants (FelhomAgentGuest @ /pool/felhom, carrying VM.Audit/VM.Allocate) reach only pool MEMBERS. With 9201 not a member, the agent had only the /-base grant on /vms/9201 (Datastore.Audit/SDN.Use/Sys.Audit) — no VM.Audit.
3. restore-test calls GuestConfig(9201) to read the mp layout → 403 VM.Audit → falls to the "restore as-is" fallback.
4. Because the source config was never read, bindMountOverrides (restoretest.go:334) — which already exists and is correct — never ran. Raw bind mounts reached PVE → mp8 … only possible for root.

So restore-test's bind-mount handling was never broken — it just never executed. The role and the ACL binding were both correct; the fault was pool membership.

Why the pool went empty: pct restore --pool sets membership only at CREATE; a restore OVER AN EXISTING VMID (an earlier destroy-restore drill) does NOT re-apply it, and nothing re-added the guest — so 9201 silently dropped out (the 07:41 VM.Audit 403 predates the 14:09 finale, consistent with an earlier drill).

Verified fix + live proof (2026-07-07):

• Part A (heal now): pveum pool modify felhom --vms 9201 → 9201 is a member; pveum user permissions felhom-agent@pve --path /vms/9201 now shows VM.Audit + VM.Allocate + VM.Backup.
• Live restore-test then PASSED for the first time (agent v0.74.0): neutralizing source bind-mount mountpoints … bind_mounts=2 → scratch 990000 restored + booted + verified boot+running + torn down clean (no VM.Allocate 403) in 4m35s. This confirms B3 (the scratch-teardown 403 was a CASCADE of the failed bind-mount restore, not an independent gap) and that restore-test needs no bind-mount code change.
• Part B (stop the regression): felhom-agent v0.74.0 re-asserts pool membership after any restore-with-Pool (Client.PoolAddVMID + a bring-up call, warn-not-fail). A destroy-restore no longer silently drops membership.

Corrected fix order + superseded recommendation:

• The report's follow-up #1 ("make pct restore work on bind-mount guests without root") is NOT needed for restore-test and is superseded — the existing override handling works once the config read succeeds (i.e. once membership is present).
• (A) pool one-liner heals today; (B) v0.74.0 stops the recurrence; restore-test unchanged.
• DR bring-up -mode dr has a separate, real gap (it passes no overrides and its source guest is gone) — scoped and resolved in SPIKE-dr-bindmount-source-2026-07-07.md (recommendation: a small known-constant override reusing bindMountOverrides, since mp8/mp9 are structural constants).
• F-C2-1 is fixed in felhom-controller v0.103.0 (config loader no longer os.ExpandEnv-corrupts a bcrypt hash; live-proven).