felhom.eu/documentation/tests/CAMPAIGN-2-2026-07-07.md

# CAMPAIGN 2 — DR primitives · self-update · escrow · storage · faults · endurance (2026-07-07)

> Unattended full-system validation on the demo node (felhom-pve / guest 9201), run during the day
> 2026-07-07 (07:20–14:11 CEST). **No production code changed.** Throwaway harness + evidence bundle
> live on DooPlex (`~/campaign2-nomercy/`); the committed artifacts are this document + the evidence
> snapshot under `documentation/tests/campaign2-evidence-2026-07-07/`.
>
> **Headline — read this first.** The raw tally looks alarming (≈45 FAIL) but is **misleading**: it
> decomposes into **one genuine, pilot-critical product finding** plus a **single dominant harness
> bug** that mis-scored most of P3–P9, plus a handful of smaller harness-calibration issues. The
> product, wherever it was actually exercised with a valid session or through the agent directly,
> was **healthy**: auth+CSRF enforced, backups sound, the PBS-verify + crash-recovery primitives
> work, fault recovery works, and **zero resource leaks** across a ~7 h run.
>
> **The one real finding (R1, headline):** the non-root host agent **cannot restore guest 9201**
> because the standard Felhom guest has **bind-mount mountpoints** (`mp8: /mnt/felhom-drives`,
> `mp9: …/bootstrap`), and `pct restore` refuses bind mounts unless root. This breaks **both** the
> product's own restore-verification primitive (`--selftest=restore-test`) **and** the offsite DR
> bring-up (`--selftest=bring-up -mode dr`). *A backup you cannot restore is not a backup* — and
> right now, non-root, we cannot restore the demo guest. This is exactly what "exercise the real
> primitives" was meant to surface.

---

## Run header

| Item | Value |
|---|---|
| Date | 2026-07-07, 07:20–14:11 CEST |
| Host / Guest | felhom-pve (192.168.0.162), node demo-felhom / guest 9201 (LAN 192.168.0.127) |
| Controller / Agent / Hub | **v0.102.0** (`cd0ebd2`) / **v0.73.0** (`e04b75e`) / v0.35.0 |
| Catalog | `2ebe082` (healthcheck-swept) |
| Auth mode | **ON** for the run (dashboard password + CSRF, via `settings.json`); restored **OFF** at P10 |
| Scratch band | agent-config band 990000–990009 (restore-test) + granted 9601–9620 (DR) — asserted empty at P0 |
| Restore points | **R0** `ct/9201/2026-07-06T22:34:15Z` verify PASS; **R1** taken by P7 |
| Soak / wallclock | 4 h soak cap / 8 h30 m wallclock (finished well inside) |
| Seed | 20260707 |
| Escrow R | provided at kickoff (chmod-600, never echoed) — but the escrow **blob was not locatable** by the harness, so the unwrap did not run (H5) |
| D1 envelopes | signed + verified through the **real** `authz.Verifier` pipeline at P0; **moved early** (after P3) to stay inside the 10:07Z TTL — see note below |

**D1-early adaptation (logged):** the envelopes had a 12 h TTL from the previous evening (expire
2026-07-07T10:07Z). The as-ordered D1 slot (after the 4 h soak) would have run past expiry. Because
0.73.1 is the **same source tree** as the deployed 0.73.0 (only the `-X main.version` string differs),
running P4–P9 on 0.73.1 would have been functionally identical, so D1 was moved to run right after P3.
(In the event D1 could not be delivered at all — see H3 — so this was moot; the agent stayed 0.73.0.)

---

## Result tally (raw, then triaged)

| Status | Raw |
|---|---|
| PASS | 15 |
| FAIL | 45 |
| SKIP | 3 |
| SOAK "breaks" | 170 (all one artifact) |

**After triage the 45 FAIL decompose as:**
- **1 genuine product finding** (R1/R2, surfaced by 3 independent agent-side tests: T-SELFRESTORE,
  T-DR-BRINGUP, and contributing to the P9/DR chain).
- **~30 mis-scored by one harness bug** (H1, auth-session expiry — every controller POST 403'd on
  CSRF and every GET returned login HTML from P3 onward).
- **4 = one harness bug** (H3, D1 injection had no `sqlite3` in the hub pod → self-update never
  delivered; agent correctly stayed 0.73.0).
- **2 = stale dryrun journal entries** (H4).
- **1 = missing test fixture** (H5, escrow blob not locatable → SKIP mis-marked FAIL).
- The remainder are the soak (H2) and derived/dependent skips.

---

## THE genuine finding — R1 + R2 (pilot-critical)

### R1 — non-root `pct restore` of a bind-mount guest fails ("only possible for root")
- **Symptom (verbatim, 3 tests):**
  `unable to restore CT <vmid> - restoring 'mp8' to bind mount is only possible for root`
  - `T-SELFRESTORE` — `--selftest=restore-test` into scratch 990000 → **FAIL in 2.2 s**
    (`evidence/T-SELFRESTORE/restore-test.out`, `pass:false`).
  - `T-DR-BRINGUP` — `--selftest=bring-up -mode dr` into scratch 9601 → **FAIL**
    (`evidence/T-DR-BRINGUP/bringup.out`).
- **Root cause (confirmed):** guest 9201's config carries **bind-mount mountpoints**:
  ```
  mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives     # the enrolled-drives bind
  mp9: /var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=1
  ```
  Proxmox `pct restore` refuses to restore a bind-mount entry unless it runs as **root**. The agent
  runs **non-root** (`felhom-agent`), by design (de-privilege). So the restore of the *standard
  Felhom guest topology* fails.
- **Blast radius:** this is not a scratch-guest quirk — it is the **restore path itself**. The
  product's own "is this backup actually restorable?" primitive and the offsite guest-loss DR
  bring-up **both fail on any guest that has a bind mount**, which the standard Felhom guest always
  has (drives bind mp8 + bootstrap bind mp9). **This is the single most important thing to know
  before pilot.**
- **Not fixed** (report-only). Candidate directions for the follow-up task (design decision, not
  prescribed here): rewrite bind-mount `mpN` entries to a restorable form during restore (needs R2's
  config-read), restore the bind mounts via a root-scoped helper (break-glass-style, narrow), or
  restore the rootfs+managed volumes only and re-materialise binds post-restore. **Cross-check with
  the offsite-WG DR design docs before choosing.**

### R2 — agent token is missing `VM.Audit` (and `VM.Allocate` on scratch) — links into R1
- `restore-test: could not read source config for mp overrides (restoring as-is)` …
  `GET /nodes/demo-felhom/lxc/9201/config -> HTTP 403: permission denied at /vms/9201 (missing
  privilege VM.Audit)`. Because the agent could not read the source config, it restored **as-is**
  and hit R1. **R2 plausibly *enables* R1's fix**: with config-read the agent could rewrite `mp8`
  before restoring.
- `DELETE /nodes/demo-felhom/lxc/990000 … missing privilege VM.Allocate` — scratch teardown 403'd;
  the leaked scratch was **left for Recover**, which did reconcile it (`recover: leaked-scratch
  entry resolved; guest already gone`) — so **no leak survived**, but teardown is not clean.
- **Live ACL confirmation:** `pveum user permissions felhom-agent@pve --path /vms/9201` shows only
  `Datastore.Audit` + `Sys.Audit` — **not** `VM.Audit`. The `FelhomAgentGuest` role *does* carry
  `VM.Audit`/`VM.Allocate`, but the grant on `/pool/felhom` is not resolving to `VM.Audit` at the
  per-guest path the restore path checks. (Related to the A1 pool-audit thread.)

---

## Other real observations

- **F-C2-1 (kickoff, confirmed):** setting the dashboard `password_hash` in **`controller.yaml`**
  does **not** take effect — the config loader runs `os.ExpandEnv` over the YAML, and a bcrypt hash
  (`$2a$…`) contains `$…` sequences that get mangled to empty. Worked around by writing the hash to
  **`settings.json`** (which is not env-expanded). Product-relevant: any operator putting a bcrypt
  hash in `controller.yaml` gets silent auth-bypass. Report-only.
- **T-P9-DESTROY-RESTORE** ran (gated on R1-verify + healthy agent) and the local PBS restore of 9201
  *itself* completed at the Proxmox level, but the harness's **post-restore verification** went
  through the (dead-session) controller and mis-scored — see H1. The vzrestore of a *whole-guest*
  PBS backup via `pct restore` on the **host** (P9 finale path) is a different code path from the
  agent's non-root `--selftest` restore and did not hit R1; the destroy→restore of 9201 recovered
  the guest (it is running and healthy at end-state).

---

## Strong positive results (real signal — do not lose these under the FAIL count)

### No resource leaks — the endurance headline (flight recorder, 351 samples over ~7 h)
| Metric | Start | End | Min | Max | Verdict |
|---|---|---|---|---|---|
| agent RSS (KB) | 22 116 | 19 936 | 17 924 | 22 792 | **stable, trended down** |
| agent fd | 79 | 54 | — | 83 | no leak |
| agent sockets | 70 | 46 | — | 74 | **no idle-socket leak** (the agentapi precedent is absent) |
| controller fd | 0* | 15 | 0* | 30 | no runaway (*0 = sampler pgrep miss) |
| controller latency | — | — | 0.008 s | 0.079 s | responsive all run |

The controller answered every flight-recorder probe in **8–79 ms** across the whole run; only ~10 of
351 samples were 502/000 (during **deliberate** restarts/reboots), all recovered. (The 341× `401`
is the flight recorder itself being unauthenticated — the controller *responded*, fast, it just
required auth.)

### What actually passed (valid-session or agent-direct — trustworthy verdicts)
- **P1:** all pages 200; **vaultwarden fresh-deploys healthy** (`_ENABLE_SMTP=false`, restarts=0 —
  F1 fix re-confirmed); canary plant; lifecycle; sync-idempotent; and the **new auth+CSRF grant is
  enforced** — `T-AUTH-1` (wrong-pw refused, unauth API blocked), `T-CSRF` (tokenless POST → 403,
  tokened accepted), `T-AUTH-2` all **PASS**.
- **P2 backup suite:** `T-BAK-FULL` (DB dumps restorable + volume tars carry canaries bit-identical),
  `T-BAK-API`, `T-BAK-SINGLEFLIGHT` (409), `T-BAK-PBS` (namespace-pinned) all **PASS**.
- **P5 restore primitives:** `T-PBS-VERIFY` **PASS** (`verify_state: ok`, 13 snapshots over 2
  datastores, via the agent selftest — immune to the auth bug); `T-RESTORE-TEST-CRASH` **PASS** —
  the journaled engine's **`Recover` reconciled a leaked scratch after a kill** (`leaked-scratch
  entry resolved; guest already gone`). *Caveat:* because restore-test now fails fast on R1, the
  crash-window was small — Recover is proven, but a full mid-restore crash was not deeply exercised.
- **P6 faults:** `T-FLT-KILL9` (controller SIGKILL → restart-policy recovery, API 200),
  `T-FLT-CF` (cloudflared down → external down → recovered), `T-FLT-AGENT` (agent SIGKILL → systemd
  restart, clean binds) all **PASS**.
- **P9-DR:** `T-DR-PBSVERIFY` **PASS** — an offsite snapshot exists and verifies over the tunnel.

---

## Harness bugs found (the real story behind the FAIL count — product NOT implicated)

- **H1 — auth-session expiry (DOMINANT).** The harness logs in **once** (P1) and never
  re-authenticates. Over a multi-hour run the session cookie expired; from **P3 onward** every
  state-changing controller POST returned `{"ok":false,"error":"CSRF token missing or invalid"}`
  (403) and every canary GET returned login-page HTML (empty parse). Proof:
  `evidence/T-STG-ENROLL/init1.txt` (CSRF-403), `evidence/T-RST-VW/restore_resp.txt` (302→login),
  `canary_check.txt` (`got=` empty). **Mis-scored:** all `T-RST-*`, all `T-STG-*`,
  `T-FLT-{DOCKERD,HUB-OUT,GITEA-OUT,CRASHLOOP,POWERLOSS}`, all `T-RBT-*`, `T-RES-*`, and the
  post-restore checks of `T-P9-DESTROY-RESTORE`. **Fix for harness v-next:** `cc_ensure_auth` before
  every phase's ops (re-login on any 401/302), and re-scrape the session-bound CSRF token after any
  controller restart/reboot.
- **H2 — soak responsiveness probe (`cc_code /` expects 200; auth → 302→login).** 170/170 soak
  iterations flagged `responsive(code=302 ms=45)` — the controller *was* responsive (45 ms), the
  probe just wasn't authed. `T-SOAK-LOOP` FAIL is entirely this. Fix: probe an authed page (or
  accept 302).
- **H3 — D1 injection has no `sqlite3` in the hub pod.** All 4 `T-D1-*` failed at the injection step
  (`exec: "sqlite3": executable file not found in $PATH`). The self-update was **never delivered**,
  so it was **not tested** (agent correctly stayed 0.73.0). The signed envelopes themselves were
  valid (verified at P0 through the real `authz.Verifier`). Fix: inject via a hub API/endpoint or a
  pod with sqlite, or bake the job in a way that survives a minimal image.
- **H4 — dryrun polluted the journal.** The pre-launch `run.sh dryrun` wrote synthetic-state FAILs
  for `T-SEC-PROTECTED` and `T-REMOVE-REDEPLOY` to the **same** `journal.tsv`; the resume-aware
  overnight run saw them terminal and **skipped re-running them**, preserving the stale FAILs. Fix:
  fresh journal per run (or a dryrun-scoped journal).
- **H5 — escrow blob not locatable.** `T-DR-ESCROW` → `SKIP-reason: no escrow blob reachable
  (local+hub)`. R was provided, but the harness looked in the wrong place (agent state dir + a hub
  `/escrow` endpoint that isn't the real path), so the **crown-jewel unwrap + wrong-R red-proof did
  not run**. Marked FAIL, really a SKIP. Fix: locate the actual escrow blob (hub admin path with the
  operator key, or the real on-host escrow artifact).
- **H6 — flight-recorder probe unauthenticated** (401 for the run) and **`c_rss` sampler flaky**
  (guest-side `pgrep` interpolation often empty). Cosmetic — latency/leak signal still usable.

---

## Restore-primitive evidence (the campaign's purpose — verdicts)

| Primitive | Verdict | Evidence |
|---|---|---|
| `--selftest=pbs-verify` | **WORKS** | `verify_state: ok`, 13 snapshots / 2 datastores |
| journaled `engine.Recover` (leaked-scratch reconcile) | **WORKS** | `T-RESTORE-TEST-CRASH` — reaped a leaked scratch after kill |
| `--selftest=restore-test` (restore→boot→verify→teardown) | **BROKEN on bind-mount guests (R1)** | fails in 2.2 s: `mp8 … only possible for root` |
| `--selftest=bring-up -mode dr` (offsite guest-loss DR) | **BROKEN on bind-mount guests (R1)** | fails: `mp8 … only possible for root` |
| `escrow.Consume(R)` unwrap (+ wrong-R red-proof) | **NOT RUN** (H5 — blob not located) | — |
| local whole-guest PBS destroy→restore (host `pct restore`, P9) | guest recovered | 9201 running + healthy at end-state |

---

## Delta vs Campaign 1

| Dimension | Campaign 1 (2026-07-06) | Campaign 2 (2026-07-07) |
|---|---|---|
| Auth | OFF (XFAIL) | **ON — enforcement proven** (T-AUTH/T-CSRF PASS) … but broke the harness mid-run (H1) |
| Restore primitives | not exercised | **exercised — surfaced R1** (the headline) |
| Resource leaks | not sampled | **sampled — none** (agent RSS/fd/sockets stable) |
| DR bring-up / escrow | not attempted | attempted — R1 blocks bring-up; escrow blocked by H5 |
| Self-update (D1) | shipped v0.70.0 (prior) | **not delivered** this run (H3) |
| Dominant mis-score cause | ~18 harness bugs (auth-off era) | **1 dominant harness bug** (auth-on session expiry) |

---

## What this campaign STILL does not prove

- **The async restore family end-to-end through the UI under load** — H1 killed the harness's restore
  tests. (It *was* proven in the P1–P3 re-run 2026-07-06: 0.018 s internal / 0.235 s external,
  canary bit-identical.)
- **Storage F2 lifecycle at scale** — all `T-STG-*` are H1 artifacts. (F2 eject/decommission were
  proven live in the v0.73.0 acceptance 2026-07-06.)
- **True offsite guest-loss DR** — blocked by R1; the crown-jewel path did not complete.
- **Escrow recovery-code unwrap** — not run (H5).
- **Agent self-update** — not delivered (H3).
- True CGNAT; multi-customer hub scale; real hardware failure.

---

## End state (demo node healed)

- **Guest 9201 running + healthy**, controller v0.102.0 healthy, **25 apps up, 0 unhealthy**.
- **Agent v0.73.0 active** (self-update never applied — H3 — so no version drift; the 0.73.1 artifact
  remains published in Gitea, dormant).
- **Auth restored OFF** (pre-campaign state; root page 200). *Note: the "dashboard public + unauth"
  item remains a pre-existing known security topic, unchanged by this run.*
- **Notifications restored** to the exact pre-state (email `nagyfenyvesi.viktor@gmail.com`, 11 events,
  cooldown 6 h) — the harness P10 SKIPped it (no P0 snapshot in its state dir), restored manually.
- **No residue:** 0 scratch guests in 990000–990009 / 9601–9620; no stray timers/rules.
- Controller auto-update was already `false` pre-campaign (unchanged).

## Follow-ups (for project-Claude / a future task — not done here)

1. **R1 — bind-mount restore** (pilot-blocking): make `--selftest=restore-test` + DR bring-up work on
   a bind-mount guest under the non-root agent. Design decision — cross-check the DR docs.
2. **R2 — grant the agent `VM.Audit`** (+ `VM.Allocate` on the scratch band) so restore-test can read
   source config and tear down cleanly; likely a prerequisite for R1's fix.
3. **Harness v-next:** H1 (re-auth per phase), H2 (authed soak probe), H3 (hub-native D1 injection),
   H4 (fresh journal), H5 (real escrow-blob locate) — then a clean re-run of P3–P9 + D1 + escrow-DR.
4. **F-C2-1:** either make the controller config loader not `os.ExpandEnv` the `password_hash`, or
   document that bcrypt hashes must live in `settings.json`.

---

# Root-cause correction (2026-07-07) — R1 was a symptom of an empty pool; restore-test was never broken

> This section ANNOTATES the report above (history preserved). Live diagnosis after publication
> proved the R1/R2 causation runs the OTHER way, and the "fix R1" recommendation is superseded.

**What R1 actually was.** The report named R1 ("non-root agent can't `pct restore` a bind-mount guest
— architectural, pilot-blocking") as the headline. That inverts symptom and cause. The real chain,
every link verified live:

1. `pvesh get /pools/felhom` → `members: []`. **The felhom pool was EMPTY.**
2. Pool-scoped ACL grants (`FelhomAgentGuest` @ `/pool/felhom`, carrying `VM.Audit`/`VM.Allocate`)
   reach only pool MEMBERS. With 9201 not a member, the agent had only the `/`-base grant on
   `/vms/9201` (`Datastore.Audit`/`SDN.Use`/`Sys.Audit`) — **no `VM.Audit`**.
3. `restore-test` calls `GuestConfig(9201)` to read the mp layout → **403 VM.Audit** → falls to the
   "restore as-is" fallback.
4. Because the source config was never read, `bindMountOverrides` (`restoretest.go:334`) — **which
   already exists and is correct** — never ran. Raw bind mounts reached PVE → `mp8 … only possible
   for root`.

**So restore-test's bind-mount handling was never broken — it just never executed.** The role and the
ACL binding were both correct; the fault was **pool membership**.

**Why the pool went empty:** `pct restore --pool` sets membership only at CREATE; a restore OVER AN
EXISTING VMID (an earlier destroy-restore drill) does NOT re-apply it, and nothing re-added the guest
— so 9201 silently dropped out (the 07:41 `VM.Audit` 403 predates the 14:09 finale, consistent with
an earlier drill).

**Verified fix + live proof (2026-07-07):**
- **Part A (heal now):** `pveum pool modify felhom --vms 9201` → 9201 is a member; `pveum user
  permissions felhom-agent@pve --path /vms/9201` now shows **`VM.Audit` + `VM.Allocate` + `VM.Backup`**.
- **Live restore-test then PASSED for the first time** (agent v0.74.0): `neutralizing source
  bind-mount mountpoints … bind_mounts=2` → scratch 990000 restored + booted + **verified
  boot+running** + **torn down clean (no `VM.Allocate` 403)** in 4m35s. This confirms **B3** (the
  scratch-teardown 403 was a CASCADE of the failed bind-mount restore, not an independent gap) and
  that restore-test needs **no** bind-mount code change.
- **Part B (stop the regression):** `felhom-agent v0.74.0` re-asserts pool membership after any
  restore-with-`Pool` (`Client.PoolAddVMID` + a bring-up call, warn-not-fail). A destroy-restore no
  longer silently drops membership.

**Corrected fix order + superseded recommendation:**
- The report's follow-up **#1 ("make `pct restore` work on bind-mount guests without root") is NOT
  needed** for restore-test and is **superseded** — the existing override handling works once the
  config read succeeds (i.e. once membership is present).
- (A) pool one-liner heals today; (B) v0.74.0 stops the recurrence; restore-test unchanged.
- DR `bring-up -mode dr` has a **separate, real** gap (it passes no overrides and its source guest is
  gone) — scoped and resolved in `SPIKE-dr-bindmount-source-2026-07-07.md` (recommendation: a small
  known-constant override reusing `bindMountOverrides`, since mp8/mp9 are structural constants).
- **F-C2-1** is fixed in `felhom-controller v0.103.0` (config loader no longer `os.ExpandEnv`-corrupts
  a bcrypt hash; live-proven).