docs(tests): Campaign 2 report + evidence — DR primitives, self-update, storage, faults, endurance
Full-system validation on demo 9201 (controller v0.102.0 / agent v0.73.0). Headline: ONE genuine pilot-critical finding — the non-root agent cannot pct-restore the standard Felhom guest because it has bind-mount mountpoints (mp8 /mnt/felhom-drives, mp9 bootstrap) and pct restore refuses bind mounts unless root. This breaks BOTH --selftest=restore-test AND the offsite DR bring-up (R1), linked to a missing VM.Audit/VM.Allocate on the agent token (R2). The large raw FAIL count is dominated by one harness bug (auth-session expiry mis-scoring P3-P9) plus D1-injection/soak-probe/ dryrun-journal/escrow-blob calibration issues; the product where actually exercised was healthy with ZERO resource leaks over ~7h. Demo node healed to pre-campaign state (auth off, notifications restored, no scratch leaks). No production code changed. Evidence snapshot + README under campaign2-evidence-2026-07-07/ (cookie jars scrubbed). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
@@ -0,0 +1,262 @@
|
||||
# CAMPAIGN 2 — DR primitives · self-update · escrow · storage · faults · endurance (2026-07-07)
|
||||
|
||||
> Unattended full-system validation on the demo node (felhom-pve / guest 9201), run during the day
|
||||
> 2026-07-07 (07:20–14:11 CEST). **No production code changed.** Throwaway harness + evidence bundle
|
||||
> live on DooPlex (`~/campaign2-nomercy/`); the committed artifacts are this document + the evidence
|
||||
> snapshot under `documentation/tests/campaign2-evidence-2026-07-07/`.
|
||||
>
|
||||
> **Headline — read this first.** The raw tally looks alarming (≈45 FAIL) but is **misleading**: it
|
||||
> decomposes into **one genuine, pilot-critical product finding** plus a **single dominant harness
|
||||
> bug** that mis-scored most of P3–P9, plus a handful of smaller harness-calibration issues. The
|
||||
> product, wherever it was actually exercised with a valid session or through the agent directly,
|
||||
> was **healthy**: auth+CSRF enforced, backups sound, the PBS-verify + crash-recovery primitives
|
||||
> work, fault recovery works, and **zero resource leaks** across a ~7 h run.
|
||||
>
|
||||
> **The one real finding (R1, headline):** the non-root host agent **cannot restore guest 9201**
|
||||
> because the standard Felhom guest has **bind-mount mountpoints** (`mp8: /mnt/felhom-drives`,
|
||||
> `mp9: …/bootstrap`), and `pct restore` refuses bind mounts unless root. This breaks **both** the
|
||||
> product's own restore-verification primitive (`--selftest=restore-test`) **and** the offsite DR
|
||||
> bring-up (`--selftest=bring-up -mode dr`). *A backup you cannot restore is not a backup* — and
|
||||
> right now, non-root, we cannot restore the demo guest. This is exactly what "exercise the real
|
||||
> primitives" was meant to surface.
|
||||
|
||||
---
|
||||
|
||||
## Run header
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| Date | 2026-07-07, 07:20–14:11 CEST |
|
||||
| Host / Guest | felhom-pve (192.168.0.162), node demo-felhom / guest 9201 (LAN 192.168.0.127) |
|
||||
| Controller / Agent / Hub | **v0.102.0** (`cd0ebd2`) / **v0.73.0** (`e04b75e`) / v0.35.0 |
|
||||
| Catalog | `2ebe082` (healthcheck-swept) |
|
||||
| Auth mode | **ON** for the run (dashboard password + CSRF, via `settings.json`); restored **OFF** at P10 |
|
||||
| Scratch band | agent-config band 990000–990009 (restore-test) + granted 9601–9620 (DR) — asserted empty at P0 |
|
||||
| Restore points | **R0** `ct/9201/2026-07-06T22:34:15Z` verify PASS; **R1** taken by P7 |
|
||||
| Soak / wallclock | 4 h soak cap / 8 h30 m wallclock (finished well inside) |
|
||||
| Seed | 20260707 |
|
||||
| Escrow R | provided at kickoff (chmod-600, never echoed) — but the escrow **blob was not locatable** by the harness, so the unwrap did not run (H5) |
|
||||
| D1 envelopes | signed + verified through the **real** `authz.Verifier` pipeline at P0; **moved early** (after P3) to stay inside the 10:07Z TTL — see note below |
|
||||
|
||||
**D1-early adaptation (logged):** the envelopes had a 12 h TTL from the previous evening (expire
|
||||
2026-07-07T10:07Z). The as-ordered D1 slot (after the 4 h soak) would have run past expiry. Because
|
||||
0.73.1 is the **same source tree** as the deployed 0.73.0 (only the `-X main.version` string differs),
|
||||
running P4–P9 on 0.73.1 would have been functionally identical, so D1 was moved to run right after P3.
|
||||
(In the event D1 could not be delivered at all — see H3 — so this was moot; the agent stayed 0.73.0.)
|
||||
|
||||
---
|
||||
|
||||
## Result tally (raw, then triaged)
|
||||
|
||||
| Status | Raw |
|
||||
|---|---|
|
||||
| PASS | 15 |
|
||||
| FAIL | 45 |
|
||||
| SKIP | 3 |
|
||||
| SOAK "breaks" | 170 (all one artifact) |
|
||||
|
||||
**After triage the 45 FAIL decompose as:**
|
||||
- **1 genuine product finding** (R1/R2, surfaced by 3 independent agent-side tests: T-SELFRESTORE,
|
||||
T-DR-BRINGUP, and contributing to the P9/DR chain).
|
||||
- **~30 mis-scored by one harness bug** (H1, auth-session expiry — every controller POST 403'd on
|
||||
CSRF and every GET returned login HTML from P3 onward).
|
||||
- **4 = one harness bug** (H3, D1 injection had no `sqlite3` in the hub pod → self-update never
|
||||
delivered; agent correctly stayed 0.73.0).
|
||||
- **2 = stale dryrun journal entries** (H4).
|
||||
- **1 = missing test fixture** (H5, escrow blob not locatable → SKIP mis-marked FAIL).
|
||||
- The remainder are the soak (H2) and derived/dependent skips.
|
||||
|
||||
---
|
||||
|
||||
## THE genuine finding — R1 + R2 (pilot-critical)
|
||||
|
||||
### R1 — non-root `pct restore` of a bind-mount guest fails ("only possible for root")
|
||||
- **Symptom (verbatim, 3 tests):**
|
||||
`unable to restore CT <vmid> - restoring 'mp8' to bind mount is only possible for root`
|
||||
- `T-SELFRESTORE` — `--selftest=restore-test` into scratch 990000 → **FAIL in 2.2 s**
|
||||
(`evidence/T-SELFRESTORE/restore-test.out`, `pass:false`).
|
||||
- `T-DR-BRINGUP` — `--selftest=bring-up -mode dr` into scratch 9601 → **FAIL**
|
||||
(`evidence/T-DR-BRINGUP/bringup.out`).
|
||||
- **Root cause (confirmed):** guest 9201's config carries **bind-mount mountpoints**:
|
||||
```
|
||||
mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives # the enrolled-drives bind
|
||||
mp9: /var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=1
|
||||
```
|
||||
Proxmox `pct restore` refuses to restore a bind-mount entry unless it runs as **root**. The agent
|
||||
runs **non-root** (`felhom-agent`), by design (de-privilege). So the restore of the *standard
|
||||
Felhom guest topology* fails.
|
||||
- **Blast radius:** this is not a scratch-guest quirk — it is the **restore path itself**. The
|
||||
product's own "is this backup actually restorable?" primitive and the offsite guest-loss DR
|
||||
bring-up **both fail on any guest that has a bind mount**, which the standard Felhom guest always
|
||||
has (drives bind mp8 + bootstrap bind mp9). **This is the single most important thing to know
|
||||
before pilot.**
|
||||
- **Not fixed** (report-only). Candidate directions for the follow-up task (design decision, not
|
||||
prescribed here): rewrite bind-mount `mpN` entries to a restorable form during restore (needs R2's
|
||||
config-read), restore the bind mounts via a root-scoped helper (break-glass-style, narrow), or
|
||||
restore the rootfs+managed volumes only and re-materialise binds post-restore. **Cross-check with
|
||||
the offsite-WG DR design docs before choosing.**
|
||||
|
||||
### R2 — agent token is missing `VM.Audit` (and `VM.Allocate` on scratch) — links into R1
|
||||
- `restore-test: could not read source config for mp overrides (restoring as-is)` …
|
||||
`GET /nodes/demo-felhom/lxc/9201/config -> HTTP 403: permission denied at /vms/9201 (missing
|
||||
privilege VM.Audit)`. Because the agent could not read the source config, it restored **as-is**
|
||||
and hit R1. **R2 plausibly *enables* R1's fix**: with config-read the agent could rewrite `mp8`
|
||||
before restoring.
|
||||
- `DELETE /nodes/demo-felhom/lxc/990000 … missing privilege VM.Allocate` — scratch teardown 403'd;
|
||||
the leaked scratch was **left for Recover**, which did reconcile it (`recover: leaked-scratch
|
||||
entry resolved; guest already gone`) — so **no leak survived**, but teardown is not clean.
|
||||
- **Live ACL confirmation:** `pveum user permissions felhom-agent@pve --path /vms/9201` shows only
|
||||
`Datastore.Audit` + `Sys.Audit` — **not** `VM.Audit`. The `FelhomAgentGuest` role *does* carry
|
||||
`VM.Audit`/`VM.Allocate`, but the grant on `/pool/felhom` is not resolving to `VM.Audit` at the
|
||||
per-guest path the restore path checks. (Related to the A1 pool-audit thread.)
|
||||
|
||||
---
|
||||
|
||||
## Other real observations
|
||||
|
||||
- **F-C2-1 (kickoff, confirmed):** setting the dashboard `password_hash` in **`controller.yaml`**
|
||||
does **not** take effect — the config loader runs `os.ExpandEnv` over the YAML, and a bcrypt hash
|
||||
(`$2a$…`) contains `$…` sequences that get mangled to empty. Worked around by writing the hash to
|
||||
**`settings.json`** (which is not env-expanded). Product-relevant: any operator putting a bcrypt
|
||||
hash in `controller.yaml` gets silent auth-bypass. Report-only.
|
||||
- **T-P9-DESTROY-RESTORE** ran (gated on R1-verify + healthy agent) and the local PBS restore of 9201
|
||||
*itself* completed at the Proxmox level, but the harness's **post-restore verification** went
|
||||
through the (dead-session) controller and mis-scored — see H1. The vzrestore of a *whole-guest*
|
||||
PBS backup via `pct restore` on the **host** (P9 finale path) is a different code path from the
|
||||
agent's non-root `--selftest` restore and did not hit R1; the destroy→restore of 9201 recovered
|
||||
the guest (it is running and healthy at end-state).
|
||||
|
||||
---
|
||||
|
||||
## Strong positive results (real signal — do not lose these under the FAIL count)
|
||||
|
||||
### No resource leaks — the endurance headline (flight recorder, 351 samples over ~7 h)
|
||||
| Metric | Start | End | Min | Max | Verdict |
|
||||
|---|---|---|---|---|---|
|
||||
| agent RSS (KB) | 22 116 | 19 936 | 17 924 | 22 792 | **stable, trended down** |
|
||||
| agent fd | 79 | 54 | — | 83 | no leak |
|
||||
| agent sockets | 70 | 46 | — | 74 | **no idle-socket leak** (the agentapi precedent is absent) |
|
||||
| controller fd | 0* | 15 | 0* | 30 | no runaway (*0 = sampler pgrep miss) |
|
||||
| controller latency | — | — | 0.008 s | 0.079 s | responsive all run |
|
||||
|
||||
The controller answered every flight-recorder probe in **8–79 ms** across the whole run; only ~10 of
|
||||
351 samples were 502/000 (during **deliberate** restarts/reboots), all recovered. (The 341× `401`
|
||||
is the flight recorder itself being unauthenticated — the controller *responded*, fast, it just
|
||||
required auth.)
|
||||
|
||||
### What actually passed (valid-session or agent-direct — trustworthy verdicts)
|
||||
- **P1:** all pages 200; **vaultwarden fresh-deploys healthy** (`_ENABLE_SMTP=false`, restarts=0 —
|
||||
F1 fix re-confirmed); canary plant; lifecycle; sync-idempotent; and the **new auth+CSRF grant is
|
||||
enforced** — `T-AUTH-1` (wrong-pw refused, unauth API blocked), `T-CSRF` (tokenless POST → 403,
|
||||
tokened accepted), `T-AUTH-2` all **PASS**.
|
||||
- **P2 backup suite:** `T-BAK-FULL` (DB dumps restorable + volume tars carry canaries bit-identical),
|
||||
`T-BAK-API`, `T-BAK-SINGLEFLIGHT` (409), `T-BAK-PBS` (namespace-pinned) all **PASS**.
|
||||
- **P5 restore primitives:** `T-PBS-VERIFY` **PASS** (`verify_state: ok`, 13 snapshots over 2
|
||||
datastores, via the agent selftest — immune to the auth bug); `T-RESTORE-TEST-CRASH` **PASS** —
|
||||
the journaled engine's **`Recover` reconciled a leaked scratch after a kill** (`leaked-scratch
|
||||
entry resolved; guest already gone`). *Caveat:* because restore-test now fails fast on R1, the
|
||||
crash-window was small — Recover is proven, but a full mid-restore crash was not deeply exercised.
|
||||
- **P6 faults:** `T-FLT-KILL9` (controller SIGKILL → restart-policy recovery, API 200),
|
||||
`T-FLT-CF` (cloudflared down → external down → recovered), `T-FLT-AGENT` (agent SIGKILL → systemd
|
||||
restart, clean binds) all **PASS**.
|
||||
- **P9-DR:** `T-DR-PBSVERIFY` **PASS** — an offsite snapshot exists and verifies over the tunnel.
|
||||
|
||||
---
|
||||
|
||||
## Harness bugs found (the real story behind the FAIL count — product NOT implicated)
|
||||
|
||||
- **H1 — auth-session expiry (DOMINANT).** The harness logs in **once** (P1) and never
|
||||
re-authenticates. Over a multi-hour run the session cookie expired; from **P3 onward** every
|
||||
state-changing controller POST returned `{"ok":false,"error":"CSRF token missing or invalid"}`
|
||||
(403) and every canary GET returned login-page HTML (empty parse). Proof:
|
||||
`evidence/T-STG-ENROLL/init1.txt` (CSRF-403), `evidence/T-RST-VW/restore_resp.txt` (302→login),
|
||||
`canary_check.txt` (`got=` empty). **Mis-scored:** all `T-RST-*`, all `T-STG-*`,
|
||||
`T-FLT-{DOCKERD,HUB-OUT,GITEA-OUT,CRASHLOOP,POWERLOSS}`, all `T-RBT-*`, `T-RES-*`, and the
|
||||
post-restore checks of `T-P9-DESTROY-RESTORE`. **Fix for harness v-next:** `cc_ensure_auth` before
|
||||
every phase's ops (re-login on any 401/302), and re-scrape the session-bound CSRF token after any
|
||||
controller restart/reboot.
|
||||
- **H2 — soak responsiveness probe (`cc_code /` expects 200; auth → 302→login).** 170/170 soak
|
||||
iterations flagged `responsive(code=302 ms=45)` — the controller *was* responsive (45 ms), the
|
||||
probe just wasn't authed. `T-SOAK-LOOP` FAIL is entirely this. Fix: probe an authed page (or
|
||||
accept 302).
|
||||
- **H3 — D1 injection has no `sqlite3` in the hub pod.** All 4 `T-D1-*` failed at the injection step
|
||||
(`exec: "sqlite3": executable file not found in $PATH`). The self-update was **never delivered**,
|
||||
so it was **not tested** (agent correctly stayed 0.73.0). The signed envelopes themselves were
|
||||
valid (verified at P0 through the real `authz.Verifier`). Fix: inject via a hub API/endpoint or a
|
||||
pod with sqlite, or bake the job in a way that survives a minimal image.
|
||||
- **H4 — dryrun polluted the journal.** The pre-launch `run.sh dryrun` wrote synthetic-state FAILs
|
||||
for `T-SEC-PROTECTED` and `T-REMOVE-REDEPLOY` to the **same** `journal.tsv`; the resume-aware
|
||||
overnight run saw them terminal and **skipped re-running them**, preserving the stale FAILs. Fix:
|
||||
fresh journal per run (or a dryrun-scoped journal).
|
||||
- **H5 — escrow blob not locatable.** `T-DR-ESCROW` → `SKIP-reason: no escrow blob reachable
|
||||
(local+hub)`. R was provided, but the harness looked in the wrong place (agent state dir + a hub
|
||||
`/escrow` endpoint that isn't the real path), so the **crown-jewel unwrap + wrong-R red-proof did
|
||||
not run**. Marked FAIL, really a SKIP. Fix: locate the actual escrow blob (hub admin path with the
|
||||
operator key, or the real on-host escrow artifact).
|
||||
- **H6 — flight-recorder probe unauthenticated** (401 for the run) and **`c_rss` sampler flaky**
|
||||
(guest-side `pgrep` interpolation often empty). Cosmetic — latency/leak signal still usable.
|
||||
|
||||
---
|
||||
|
||||
## Restore-primitive evidence (the campaign's purpose — verdicts)
|
||||
|
||||
| Primitive | Verdict | Evidence |
|
||||
|---|---|---|
|
||||
| `--selftest=pbs-verify` | **WORKS** | `verify_state: ok`, 13 snapshots / 2 datastores |
|
||||
| journaled `engine.Recover` (leaked-scratch reconcile) | **WORKS** | `T-RESTORE-TEST-CRASH` — reaped a leaked scratch after kill |
|
||||
| `--selftest=restore-test` (restore→boot→verify→teardown) | **BROKEN on bind-mount guests (R1)** | fails in 2.2 s: `mp8 … only possible for root` |
|
||||
| `--selftest=bring-up -mode dr` (offsite guest-loss DR) | **BROKEN on bind-mount guests (R1)** | fails: `mp8 … only possible for root` |
|
||||
| `escrow.Consume(R)` unwrap (+ wrong-R red-proof) | **NOT RUN** (H5 — blob not located) | — |
|
||||
| local whole-guest PBS destroy→restore (host `pct restore`, P9) | guest recovered | 9201 running + healthy at end-state |
|
||||
|
||||
---
|
||||
|
||||
## Delta vs Campaign 1
|
||||
|
||||
| Dimension | Campaign 1 (2026-07-06) | Campaign 2 (2026-07-07) |
|
||||
|---|---|---|
|
||||
| Auth | OFF (XFAIL) | **ON — enforcement proven** (T-AUTH/T-CSRF PASS) … but broke the harness mid-run (H1) |
|
||||
| Restore primitives | not exercised | **exercised — surfaced R1** (the headline) |
|
||||
| Resource leaks | not sampled | **sampled — none** (agent RSS/fd/sockets stable) |
|
||||
| DR bring-up / escrow | not attempted | attempted — R1 blocks bring-up; escrow blocked by H5 |
|
||||
| Self-update (D1) | shipped v0.70.0 (prior) | **not delivered** this run (H3) |
|
||||
| Dominant mis-score cause | ~18 harness bugs (auth-off era) | **1 dominant harness bug** (auth-on session expiry) |
|
||||
|
||||
---
|
||||
|
||||
## What this campaign STILL does not prove
|
||||
|
||||
- **The async restore family end-to-end through the UI under load** — H1 killed the harness's restore
|
||||
tests. (It *was* proven in the P1–P3 re-run 2026-07-06: 0.018 s internal / 0.235 s external,
|
||||
canary bit-identical.)
|
||||
- **Storage F2 lifecycle at scale** — all `T-STG-*` are H1 artifacts. (F2 eject/decommission were
|
||||
proven live in the v0.73.0 acceptance 2026-07-06.)
|
||||
- **True offsite guest-loss DR** — blocked by R1; the crown-jewel path did not complete.
|
||||
- **Escrow recovery-code unwrap** — not run (H5).
|
||||
- **Agent self-update** — not delivered (H3).
|
||||
- True CGNAT; multi-customer hub scale; real hardware failure.
|
||||
|
||||
---
|
||||
|
||||
## End state (demo node healed)
|
||||
|
||||
- **Guest 9201 running + healthy**, controller v0.102.0 healthy, **25 apps up, 0 unhealthy**.
|
||||
- **Agent v0.73.0 active** (self-update never applied — H3 — so no version drift; the 0.73.1 artifact
|
||||
remains published in Gitea, dormant).
|
||||
- **Auth restored OFF** (pre-campaign state; root page 200). *Note: the "dashboard public + unauth"
|
||||
item remains a pre-existing known security topic, unchanged by this run.*
|
||||
- **Notifications restored** to the exact pre-state (email `nagyfenyvesi.viktor@gmail.com`, 11 events,
|
||||
cooldown 6 h) — the harness P10 SKIPped it (no P0 snapshot in its state dir), restored manually.
|
||||
- **No residue:** 0 scratch guests in 990000–990009 / 9601–9620; no stray timers/rules.
|
||||
- Controller auto-update was already `false` pre-campaign (unchanged).
|
||||
|
||||
## Follow-ups (for project-Claude / a future task — not done here)
|
||||
|
||||
1. **R1 — bind-mount restore** (pilot-blocking): make `--selftest=restore-test` + DR bring-up work on
|
||||
a bind-mount guest under the non-root agent. Design decision — cross-check the DR docs.
|
||||
2. **R2 — grant the agent `VM.Audit`** (+ `VM.Allocate` on the scratch band) so restore-test can read
|
||||
source config and tear down cleanly; likely a prerequisite for R1's fix.
|
||||
3. **Harness v-next:** H1 (re-auth per phase), H2 (authed soak probe), H3 (hub-native D1 injection),
|
||||
H4 (fresh journal), H5 (real escrow-blob locate) — then a clean re-run of P3–P9 + D1 + escrow-DR.
|
||||
4. **F-C2-1:** either make the controller config loader not `os.ExpandEnv` the `password_hash`, or
|
||||
document that bcrypt hashes must live in `settings.json`.
|
||||
Reference in New Issue
Block a user