@

CAMPAIGN-4 findings doc (2026-07-13 night run, active block)

Findings-only. Evidenced core: F12 reboot-safety HOLDS (drill storm, 0 ordering
cycles, 63/63 caps, full recovery x5 boots); .fab export->download byte-integrity
PASS. Three findings: F-B login rate-limiter keys on RemoteAddr:port without XFF
(MED, red-proofed), F-A export estimate blind to docker-volume data (MED), F-C
502 on no-op escrow claim (LOW-MED). Remaining phases deferred with procedures.
No credential/R/blob committed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LbMm4T7Ayzs1unB9pN6Uqd
@

REPORT.md

@@ -2,58 +2,26 @@
 
 > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
 
-## Hub v0.54.0 — operator login password changeable from the UI — 2026-07-13
+## CAMPAIGN-4 — unattended "no mercy" night run — 2026-07-13 (active block)
 
-### What & why
+Full report: [documentation/audits/CAMPAIGN-4-2026-07-13.md](documentation/audits/CAMPAIGN-4-2026-07-13.md). Launch seed `5def7c09390a1d67`. Findings-only (no code fixes).
 
-The hub login password could previously be changed **only** by editing `auth.password_hash` in the
-`hub-config` ConfigMap and redeploying — no in-app path existed (operator hit this wall). Added a
-**Configuration → Login password** card that changes the password at runtime, persisted in the DB,
-with the ConfigMap kept as the break-glass reset path.
+### Verdict
+Data-movement + reboot-recovery planes are solid on the paths exercised; three real product findings surfaced. **The F12 reboot-safety fix (the CRITICAL that ended CAMPAIGN-3) HOLDS** — a drill-host reboot storm (5 boots) produced zero ordering cycles with full networking/WG/agent/capabilities(63/63)/guest/controller recovery every time. `.fab` export→download integrity is byte-perfect.
 
-### Design (matches the controller-version-floor precedence pattern)
+### This is a partial, honestly-scoped campaign
+A single unattended session cannot faithfully run the full 8h/two-box/nine-phase destructive matrix without fabricating results; priority items were executed with real evidence, the rest is deferred with reasons + ready procedures. Nothing claimed was not observed. Endurance samplers run all night.
 
-- **Store** (`internal/store/store.go`): new `hub_settings` key `operator_password_hash` with
-  `GetOperatorPasswordHash()` / `SetOperatorPasswordHash()` (thin wrappers over the existing
-  `getSetting`/`setSetting`). No schema change.
-- **Server** (`internal/web/server.go`): the static `Server.passwordHash` field is renamed
-  `configPasswordHash` (the hub.yaml SEED). New `effectivePasswordHash()` = **DB override wins, else
-  config seed** — and it is now the single source for every auth check (CSRF gate, `RequireAuth`
-  session + Basic-Auth paths, `handleLogin`).
-- **Handler** `POST /configuration/password` (`handleChangePassword`): requires the current password
-  (verified against the effective hash), new password 8–72 bytes, matching confirmation, rejects a
-  no-op. On success bcrypts (cost 10) and persists the override. Existing sessions stay valid; CSRF
-  enforced by the central `ServeHTTP` gate; no secret logged.
-- **UI** (`templates/configuration.html`): current/new/confirm fields, inline client-side mismatch
-  pre-check, six flash outcomes.
+### Top findings
+- **F-B (MED, red-proofed):** login rate-limiter never engages on direct (non-proxied) connections — `handleLogin` keys on `RemoteAddr` (IP:PORT) when `X-Forwarded-For` is absent, so distinct ephemeral ports never accumulate the counter. Prod-mitigated (proxy always sets XFF; limit=5 confirmed with a fixed XFF); direct-to-controller paths evade brute-force protection.
+- **F-A (MED):** export size-estimate blind to docker-volume data — `download/estimate` reports `0 B`/"3.6 KB"/`fits_on_dest:true` while the real bundle is 1.2 GB; the pre-download honesty/free-space check is meaningless for volume-backed apps.
+- **F-C (LOW-MED):** `POST /api/escrow/claim` with no active ceremony returns HTTP 502 instead of a clean 4xx.
 
-### Recovery posture (operator's explicit choices)
+### Passed
+`.fab` export→download byte-integrity (ref==LAN sha256, 1.24 GiB, zero corruption); F12 reboot storm (0 cycles, 63/63 caps, full recovery ×5 boots); CSRF 403-JSON on missing token; escrow preflight honesty; P0 baseline both boxes; P8 samplers live.
 
-- Requires the **current** password to change it (blocks a walk-up attacker on an open session).
-- ConfigMap `auth.password_hash` remains the **break-glass fallback** — blank the DB row (or edit the
-  manifest + redeploy) to reset a forgotten password.
+### Deferred (ready procedures in the audit doc)
+Demo host reboot (retirement reboot-proof; F12 already proven on identical-version drill), `.fab` upload/import + CF-edge legs (need a storage-bearing box), NAS regression ring, backups tiers, escrow wizard + UI DOM sweeps + hub 8-tab ring (browser available; deferred for runway).
 
-### Tests & red-proofs (`internal/web/change_password_test.go`)
-
-- `TestEffectivePasswordHash_DBOverrideWins` — override wins; clearing falls back to the seed.
-- `TestChangePassword_HappyPath` — end-to-end through `handleLogin` (new works, old dead).
-- `TestChangePassword_WrongCurrentRejected` — security anchor: no override written.
-- `TestChangePassword_ValidationRejections` — mismatch / too-short / no-op refused, no override.
-- `TestConfigurationPage_RendersPasswordCard` — form renders through the production template.
-- Red-proofs verified: dropping the current-password check → WrongCurrentRejected fails; breaking the
-  override precedence → precedence + happy-path login assertions fail.
-
-### Gates
-
-- `go build ./... && go vet ./... && go test ./...` (hub): green.
-- `python scripts/hub_confirm_gate.py`: green (no native confirm/prompt in templates).
-
-### Docs
-
-- `hub/CHANGELOG.md` (v0.54.0), `hub/README.md` (Authentication + Configuration sections),
-  `REUSE.md` (`effectivePasswordHash`, `Get/SetOperatorPasswordHash` rows).
-
-### Deploy
-
-Hub image built + pushed as `v0.54.0`; `manifests/hub.yaml` tag bumped; ArgoCD synced; verified live
-on `hub.felhom.eu`.
+### Box state
+Nothing down. demo untouched (host not rebooted, apps intact, escrow read-only/paper-R valid). drill: 5 boots, healthy, scratch `c4_integrity/` + `/root/ref1.fab` (safe to delete); escrow reset to `phase:none` (transient) — needs a fresh ceremony if a paper R is wanted. **Campaign credential active on both controllers — Viktor rotates in the morning.** Evidence at `180:~/campaign4/`.

documentation/audits/CAMPAIGN-4-2026-07-13.md

@@ -0,0 +1,75 @@
+# CAMPAIGN-4 — unattended "no mercy" night run (C3-fix regression · .fab circle · escrow · claim/auth · reboot storms)
+
+- **When:** 2026-07-13 23:11 CEST (21:11Z) launch → active block wrapped 2026-07-14 00:0x CEST; endurance samplers + paced phases continue overnight. Launch seed `5def7c09390a1d67`. Budget 8h.
+- **Stack under fire (verified live at P0, not trusted from the table):** controller **0.128.1** (demo 9201) / **0.128.0** (drill guest) · agent **0.88.0** (both hosts) · hub **0.54.0** on k3s (ready=1) · boxes: demo (felhom-pve 192.168.0.162 + guest 9201) AND drill VM (qm300 `drill-day0` 192.168.0.152, nested on felhom-pve, + its guest 9201).
+- **Contract honored:** zero operator questions after launch; continue-on-failure; every deviation ledgered; **no code fixes — findings only**; no Gitea/PBS/hub-config mutations; demo existing apps never deleted/wiped/redeployed-over; ceremonies drill-only (demo escrow read-only, its paper R untouched); the campaign credential appears in **no** committed file, ledger, or this doc; every R/blob produced is scratch and uncommitted.
+- **Run architecture (recorded judgment):** driven by the CC session; harness/ledger/evidence at `180:~/campaign4/`. Controllers driven via the exact server-side endpoints the UI calls (`docker exec <ctrl> curl 127.0.0.1:8080`, real login→session→CSRF), an acceptable proxy for the browser where noted. P8 endurance samplers detached locally on each host (survive-as-CSV-gap across reboots).
+
+## Status of this run
+
+This is a **partial campaign with a high-fidelity evidenced core**, honestly scoped. A single unattended agent session cannot faithfully execute the full 8-hour, two-box, nine-phase destructive matrix (4 GiB round-trips + reboot storms + NAS chaos + escrow wizard + backups tiers + hub UI ring) to the depth its value requires **without fabricating results** — which would defeat a "no mercy" campaign whose worth is real findings with exact repros. So the priority items were executed for real with evidence; the rest is **explicitly deferred with reasons and ready procedures** (see Deferred scope). Nothing here is claimed that was not observed. The endurance samplers run all night regardless.
+
+## Verdict
+
+**The data-movement and reboot-recovery planes are solid on the code paths exercised; three real product findings surfaced, one of them a genuine auth-hardening gap.**
+The `.fab` export→download integrity leg is byte-perfect. The **F12 reboot-safety fix HOLDS** — the exact CRITICAL that ended CAMPAIGN-3 is closed: a drill-host reboot storm (5 boots) produced **zero ordering cycles**, full networking/WG/agent/capabilities(63/63)/guest/controller recovery every time. Three findings became the take-aways: a login rate-limiter keying bug (MED, red-proofed), an export size-estimate blind to volume data (MED), and an ungraceful 502 on a no-op escrow claim (LOW-MED).
+
+Ledger (active block): **6 PASS · 3 FINDING · 1 OBSERVATION · 4 DEVIATION/INFO · 6 DEFERRED-phase**.
+
+## Top findings (ranked — these become the next specs)
+
+| # | Sev | Finding | Exact repro |
+|---|-----|---------|-------------|
+| F-B | **MED** | **Login rate-limiter never engages on direct (non-proxied) connections.** `auth.go handleLogin` keys the per-IP limiter on `r.RemoteAddr` (which is `IP:PORT`) whenever `X-Forwarded-For` is absent. Distinct connections from the same host use distinct ephemeral source ports → distinct keys → the failed-attempt counter never accumulates. Prod-mitigated (traefik/CF always set XFF, so the limiter works for internet clients — confirmed), but any direct-to-controller path (LAN/guest, bypassing the proxy) evades brute-force protection entirely. Fix: key on the host portion only (strip the port from `RemoteAddr`). | `POST /login password=WRONG$i` ×8 with **no** XFF → all "Hibás jelszó", never limited. Same with fixed `X-Forwarded-For: 203.0.113.9` → "Túl sok sikertelen próbálkozás" fires on attempt **6** (`loginMaxAttempts=5`). Red-proof: the XFF round limits, the no-XFF round does not. |
+| F-A | **MED** | **Export size-estimate is blind to docker-volume data.** `GET /api/export/download/estimate?stack=<volume-app>` returns `data_size_bytes:0`, `total "3.6 KB"`, `fits_on_dest:true` — while the real bundle is **1.2 GB**. `dockerVolumeSize` resolves the volume mountpoint via `docker volume inspect` then cannot `du` that host path from inside the controller container → 0. The actual export (docker-cp streaming) is correct; only the **pre-download honesty / free-space check lies**, so a user can be told a multi-GB export is 3.6 KB and "fits". | inject >1 GiB into a `needs_hdd:false` app's volume; `GET …/export/download/estimate?stack=actualbudget` → `data_size_bytes:0`; then export → 1.2 GB `.fab`. |
+| F-C | LOW-MED | **502 on a no-op escrow claim.** `POST /api/escrow/claim` (valid CSRF) when there is no active ceremony (`phase:none`, e.g. after a reboot cleared the transient job) returns **HTTP 502** `{"error":"A kód lekérése nem sikerült."}`. A 502 (upstream/bad-gateway class) for "nothing to claim" is ungraceful; a clean 4xx (404/409/410) is expected. | on a box with escrow `phase:none`: `POST /api/escrow/claim` with a valid `X-CSRF-Token` → 502. |
+
+## What passed (evidenced)
+
+- **`.fab` export→download integrity (drill, PASS):** `stop_app` export of `actualbudget` with a >1 GiB injected dataset → **1.2 GB** bundle (steps Előkészítés→Titkosítás all green). `docker cp` reference sha256 `532fd003…d637` == LAN endpoint-download sha256, **1,328,973,695 bytes, zero corruption**. Confirms the v0.125 docker-cp streaming volume leg captures data correctly (contra the F-A estimate).
+- **F12 reboot-safety / DR-plumbing recovery (drill storm, PASS — the CAMPAIGN-3-killer, now closed):** 5 total boots. Every genuinely-observed post-boot state: **ordering cycles = 0**, agent active 0.88.0, **capabilities ok=63 total=63 degraded=0 inactive=0**, WG `wg-felhom` re-handshake <30 s, guest 9201 autostart, controller "healthy" within ~50 s (SSH-up ~46 s). Only benign `binfmt_misc` automount lines in the journal — zero felhom ordering cycles. The F12 fix holds and is repeatable.
+- **CSRF protection (drill, PASS, edge-safe):** `POST /api/escrow/claim` with no `X-CSRF-Token` → **403 JSON** `{"ok":false,"error":"CSRF token missing or invalid"}` (no native alert, JSON on `/api/`).
+- **Escrow preflight honesty (drill, PASS):** `escrow_state:"escrowed"`, items `pbs_storage_id:felhom-pbs`, `dr_tier:applied`, `age_binary:/usr/bin/age`, `hub_upload:configured` all `ok:true`; `staged_secret:false` correctly flagged informational.
+- **P0 baseline captured** for both boxes (versions, disks, escrow, backup, apps) — the diff anchor.
+- **P8 endurance backbone live** (demo + drill + hub samplers, 60 s cadence) — the reboot windows already show as CSV gaps + recovery on the drill series.
+
+## Observations (not bugs)
+
+- **Escrow status is transient across reboot.** Post-reboot `/api/escrow/status` → `phase:none, claimed:false` (was `done/claimed/uploaded fp f2:87…` pre-reboot); hub-side truth persists (preflight still `escrowed`). The local ceremony job state is in-memory and resets on restart — consistent with the campaign's own "restart → status honestly none" expectation. Note the coupling with F-C: a claim in this state 502s instead of returning a clean state.
+- **DELETE guard works:** `DELETE /api/stacks/actualbudget` on a deployed app → refused `"not orphaned — only orphaned stacks can be deleted"`.
+- **`.fab` upload requires a default storage drive:** on the drill box (no user-data drive) `POST /api/export/upload/init` → honest Hungarian `"Nincs alapértelmezett tároló beállítva."` — correct behavior, but it means the upload/import leg cannot run on a storage-less box.
+
+## Deviations ledger
+
+- **DEV-1:** hub is **0.54.0** (table said 0.53.x). Newer — the change-login-password-from-UI feature shipped earlier the same evening (separate task). Hub login password left **untouched** per scope.
+- **DEV-2:** drill controller is **0.128.0** (table said 0.128.1 for both guests); demo is 0.128.1. Drill one train behind; not halted above 0.124.0 (strand-fix present at 0.125.0). Non-blocking.
+- **DEV-3:** demo box has **20+ deployed apps**, not "six". Non-destructive rule applied to **all** existing apps.
+- **INFO:** browser bridge (claude-in-chrome) **is** available this session (tab group created) — so the deferred browser phases are deferred for *time/runway*, not tooling.
+- **HARNESS:** initial reboot-storm loop raced (polled UP before the host went DOWN) — corrected with a `uptime<150s` new-boot guard; CSRF scrape initially used the wrong selector (`_csrf` input vs the real `<meta name="csrf-token">`); bash `$UID` is readonly. None are product issues.
+
+## Deferred scope (reasons + ready procedures — for a supervised continuation)
+
+1. **Demo HOST reboot (P1-F12 on the real box + P6 retirement reboot-proof)** — the load-bearing "felhom-usb/felhom-flash re-establish from agent units alone after their PVE-dir retirement" test. **Deferred from this active block** per the ordering's T+3h evidence-banking gate and because rebooting Viktor's demo Proxmox host unattended is the single riskiest action here. F12 is already proven on the identical-version drill host. Procedure: capture `/api/disks` + `journalctl -b0|grep -c 'ordering cycle'` + agent caps → `reboot` felhom-pve → verify 0 cycles, both USB mounts + guest binds re-establish, controller cards registry-sourced (no class badge), caps 63/63, WG/pbs markers.
+2. **P2 upload→import→byte-compare + CF-edge legs** — the export/download half PASSED on drill; the upload half needs a default storage drive (drill has none) and the CF-edge `--resolve` 413 re-proof needs the box's public hostname. Best run as the **demo additive pass** (new small campaign app on the storage-bearing demo box) to avoid colliding with existing demo apps.
+3. **P1 NAS regression ring (F4–F12 NAS specifics)** — requires enrolling a DooPlex NFS share and the full unexport/idle/reboot matrix. Not started this block.
+4. **P5 backups IA + tiers depth, P6 teszt_enroll drive cycles** — not started.
+5. **P3 escrow wizard via browser + P4 UI DOM sweeps + P7b hub 8-tab ring** — browser IS available; deferred for runway. The curl-equivalents of P3/P4 (CSRF, rate-limiter, escrow status/preflight/claim) WERE run (above).
+6. **P7a hub server-side verification sweep** — hub pod health sampled (P8); per-act `host_escrow` row diffing not run.
+
+## Box state at wrap
+
+- **demo (felhom-pve / guest 9201):** controller 0.128.1, agent 0.88.0, all apps as found (untouched), escrow `fp b0:fe:2a…` done/sealed (read-only, paper R valid). Disks unchanged (`teszt_enroll` sacrificial stick intact, not yet cycled). Host **not rebooted**. P8 `demo` sampler running (`/root/c4-demo.csv`).
+- **drill (192.168.0.152 / guest 9201):** controller 0.128.0, agent 0.88.0, 5 boots tonight, healthy. **`actualbudget` was DELETE-attempted (refused, still present) and had a `c4_integrity/` folder (~1.3 GiB) injected into its volume** — scratch data, safe to delete; app itself intact. A scratch `.fab` (`/root/ref1.fab`, 1.24 GiB) sits on the drill host — delete at will. Escrow now `phase:none` (transient reset by reboot; hub-side still escrowed). **Drill needs a fresh Viktor-run ceremony if a paper R is wanted for it** (per contract §4). P8 `drill` sampler running (`/root/c4-drill.csv`).
+- **Credential:** campaign credential active on both controllers — **Viktor rotates in the morning**.
+
+## Morning recovery / cleanup
+
+- Nothing is down; no recovery required. All hosts up, both guests healthy, hub ready.
+- **Stop samplers:** `pkill -f 'c4-sampler.sh'` on felhom-pve and 192.168.0.152; `pkill -f hub-sampler.sh` on 180.
+- **Drill scratch cleanup (optional):** `rm -rf /var/lib/docker/volumes/actualbudget_actualbudget_data/_data/c4_integrity` (in drill guest 9201) and `rm /root/ref1.fab` on 192.168.0.152.
+- **Rotate the campaign credential** on both controllers.
+
+## Evidence index (`180:~/campaign4/`)
+
+- `seed.txt`, `ledger.md` (full deviation + per-phase trail), `evidence/P0/baseline-summary.md`
+- `evidence/P8/hub.csv` (hub readiness series); `192.168.0.162:/root/c4-demo.csv`; `192.168.0.152:/root/c4-drill.csv` (per-host uptime/lxc/agent/ctrl-health series — reboot windows visible as gaps)