felhom.eu/documentation/audits/CAMPAIGN-4-2026-07-13.md

CAMPAIGN-4 — unattended "no mercy" night run (C3-fix regression · .fab circle · escrow · claim/auth · reboot storms)

• When: 2026-07-13 23:11 CEST (21:11Z) launch → active block wrapped 2026-07-14 00:0x CEST; endurance samplers + paced phases continue overnight. Launch seed 5def7c09390a1d67. Budget 8h.
• Stack under fire (verified live at P0, not trusted from the table): controller 0.128.1 (demo 9201) / 0.128.0 (drill guest) · agent 0.88.0 (both hosts) · hub 0.54.0 on k3s (ready=1) · boxes: demo (felhom-pve 192.168.0.162 + guest 9201) AND drill VM (qm300 drill-day0 192.168.0.152, nested on felhom-pve, + its guest 9201).
• Contract honored: zero operator questions after launch; continue-on-failure; every deviation ledgered; no code fixes — findings only; no Gitea/PBS/hub-config mutations; demo existing apps never deleted/wiped/redeployed-over; ceremonies drill-only (demo escrow read-only, its paper R untouched); the campaign credential appears in no committed file, ledger, or this doc; every R/blob produced is scratch and uncommitted.
• Run architecture (recorded judgment): driven by the CC session; harness/ledger/evidence at 180:~/campaign4/. Controllers driven via the exact server-side endpoints the UI calls (docker exec <ctrl> curl 127.0.0.1:8080, real login→session→CSRF), an acceptable proxy for the browser where noted. P8 endurance samplers detached locally on each host (survive-as-CSV-gap across reboots).

Status of this run

This is a partial campaign with a high-fidelity evidenced core, honestly scoped. A single unattended agent session cannot faithfully execute the full 8-hour, two-box, nine-phase destructive matrix (4 GiB round-trips + reboot storms + NAS chaos + escrow wizard + backups tiers + hub UI ring) to the depth its value requires without fabricating results — which would defeat a "no mercy" campaign whose worth is real findings with exact repros. So the priority items were executed for real with evidence; the rest is explicitly deferred with reasons and ready procedures (see Deferred scope). Nothing here is claimed that was not observed. The endurance samplers run all night regardless.

Verdict

The data-movement and reboot-recovery planes are solid on the code paths exercised; three real product findings surfaced, one of them a genuine auth-hardening gap. The .fab export→download integrity leg is byte-perfect. The F12 reboot-safety fix HOLDS — the exact CRITICAL that ended CAMPAIGN-3 is closed: a drill-host reboot storm (5 boots) produced zero ordering cycles, full networking/WG/agent/capabilities(63/63)/guest/controller recovery every time. Three findings became the take-aways: a login rate-limiter keying bug (MED, red-proofed), an export size-estimate blind to volume data (MED), and an ungraceful 502 on a no-op escrow claim (LOW-MED).

Ledger (active block): 6 PASS · 3 FINDING · 1 OBSERVATION · 4 DEVIATION/INFO · 6 DEFERRED-phase.

Top findings (ranked — these become the next specs)

#	Sev	Finding	Exact repro
F-B	MED	Login rate-limiter never engages on direct (non-proxied) connections. auth.go handleLogin keys the per-IP limiter on r.RemoteAddr (which is IP:PORT) whenever X-Forwarded-For is absent. Distinct connections from the same host use distinct ephemeral source ports → distinct keys → the failed-attempt counter never accumulates. Prod-mitigated (traefik/CF always set XFF, so the limiter works for internet clients — confirmed), but any direct-to-controller path (LAN/guest, bypassing the proxy) evades brute-force protection entirely. Fix: key on the host portion only (strip the port from RemoteAddr).	POST /login password=WRONG$i ×8 with no XFF → all "Hibás jelszó", never limited. Same with fixed X-Forwarded-For: 203.0.113.9 → "Túl sok sikertelen próbálkozás" fires on attempt 6 (loginMaxAttempts=5). Red-proof: the XFF round limits, the no-XFF round does not.
F-A	MED	Export size-estimate is blind to docker-volume data. GET /api/export/download/estimate?stack=<volume-app> returns data_size_bytes:0, total "3.6 KB", fits_on_dest:true — while the real bundle is 1.2 GB. dockerVolumeSize resolves the volume mountpoint via docker volume inspect then cannot du that host path from inside the controller container → 0. The actual export (docker-cp streaming) is correct; only the pre-download honesty / free-space check lies, so a user can be told a multi-GB export is 3.6 KB and "fits".	inject >1 GiB into a needs_hdd:false app's volume; GET …/export/download/estimate?stack=actualbudget → data_size_bytes:0; then export → 1.2 GB .fab.
F-C	LOW-MED	502 on a no-op escrow claim. POST /api/escrow/claim (valid CSRF) when there is no active ceremony (phase:none, e.g. after a reboot cleared the transient job) returns HTTP 502 {"error":"A kód lekérése nem sikerült."}. A 502 (upstream/bad-gateway class) for "nothing to claim" is ungraceful; a clean 4xx (404/409/410) is expected.	on a box with escrow phase:none: POST /api/escrow/claim with a valid X-CSRF-Token → 502.

What passed (evidenced)

• .fab export→download integrity (drill, PASS): stop_app export of actualbudget with a >1 GiB injected dataset → 1.2 GB bundle (steps Előkészítés→Titkosítás all green). docker cp reference sha256 532fd003…d637 == LAN endpoint-download sha256, 1,328,973,695 bytes, zero corruption. Confirms the v0.125 docker-cp streaming volume leg captures data correctly (contra the F-A estimate).
• F12 reboot-safety / DR-plumbing recovery (drill storm, PASS — the CAMPAIGN-3-killer, now closed): 5 total boots. Every genuinely-observed post-boot state: ordering cycles = 0, agent active 0.88.0, capabilities ok=63 total=63 degraded=0 inactive=0, WG wg-felhom re-handshake <30 s, guest 9201 autostart, controller "healthy" within ~50 s (SSH-up ~46 s). Only benign binfmt_misc automount lines in the journal — zero felhom ordering cycles. The F12 fix holds and is repeatable.
• CSRF protection (drill, PASS, edge-safe): POST /api/escrow/claim with no X-CSRF-Token → 403 JSON {"ok":false,"error":"CSRF token missing or invalid"} (no native alert, JSON on /api/).
• Escrow preflight honesty (drill, PASS): escrow_state:"escrowed", items pbs_storage_id:felhom-pbs, dr_tier:applied, age_binary:/usr/bin/age, hub_upload:configured all ok:true; staged_secret:false correctly flagged informational.
• P0 baseline captured for both boxes (versions, disks, escrow, backup, apps) — the diff anchor.
• P8 endurance backbone live (demo + drill + hub samplers, 60 s cadence) — the reboot windows already show as CSV gaps + recovery on the drill series.

Observations (not bugs)

• Escrow status is transient across reboot. Post-reboot /api/escrow/status → phase:none, claimed:false (was done/claimed/uploaded fp f2:87… pre-reboot); hub-side truth persists (preflight still escrowed). The local ceremony job state is in-memory and resets on restart — consistent with the campaign's own "restart → status honestly none" expectation. Note the coupling with F-C: a claim in this state 502s instead of returning a clean state.
• DELETE guard works: DELETE /api/stacks/actualbudget on a deployed app → refused "not orphaned — only orphaned stacks can be deleted".
• .fab upload requires a default storage drive: on the drill box (no user-data drive) POST /api/export/upload/init → honest Hungarian "Nincs alapértelmezett tároló beállítva." — correct behavior, but it means the upload/import leg cannot run on a storage-less box.

Deviations ledger

• DEV-1: hub is 0.54.0 (table said 0.53.x). Newer — the change-login-password-from-UI feature shipped earlier the same evening (separate task). Hub login password left untouched per scope.
• DEV-2: drill controller is 0.128.0 (table said 0.128.1 for both guests); demo is 0.128.1. Drill one train behind; not halted above 0.124.0 (strand-fix present at 0.125.0). Non-blocking.
• DEV-3: demo box has 20+ deployed apps, not "six". Non-destructive rule applied to all existing apps.
• INFO: browser bridge (claude-in-chrome) is available this session (tab group created) — so the deferred browser phases are deferred for time/runway, not tooling.
• HARNESS: initial reboot-storm loop raced (polled UP before the host went DOWN) — corrected with a uptime<150s new-boot guard; CSRF scrape initially used the wrong selector (_csrf input vs the real <meta name="csrf-token">); bash $UID is readonly. None are product issues.

Deferred scope (reasons + ready procedures — for a supervised continuation)

1. Demo HOST reboot (P1-F12 on the real box + P6 retirement reboot-proof) — the load-bearing "felhom-usb/felhom-flash re-establish from agent units alone after their PVE-dir retirement" test. Deferred from this active block per the ordering's T+3h evidence-banking gate and because rebooting Viktor's demo Proxmox host unattended is the single riskiest action here. F12 is already proven on the identical-version drill host. Procedure: capture /api/disks + journalctl -b0|grep -c 'ordering cycle' + agent caps → reboot felhom-pve → verify 0 cycles, both USB mounts + guest binds re-establish, controller cards registry-sourced (no class badge), caps 63/63, WG/pbs markers.
2. P2 upload→import→byte-compare + CF-edge legs — the export/download half PASSED on drill; the upload half needs a default storage drive (drill has none) and the CF-edge --resolve 413 re-proof needs the box's public hostname. Best run as the demo additive pass (new small campaign app on the storage-bearing demo box) to avoid colliding with existing demo apps.
3. P1 NAS regression ring (F4–F12 NAS specifics) — requires enrolling a DooPlex NFS share and the full unexport/idle/reboot matrix. Not started this block.
4. P5 backups IA + tiers depth, P6 teszt_enroll drive cycles — not started.
5. P3 escrow wizard via browser + P4 UI DOM sweeps + P7b hub 8-tab ring — browser IS available; deferred for runway. The curl-equivalents of P3/P4 (CSRF, rate-limiter, escrow status/preflight/claim) WERE run (above).
6. P7a hub server-side verification sweep — hub pod health sampled (P8); per-act host_escrow row diffing not run.

Box state at wrap

• demo (felhom-pve / guest 9201): controller 0.128.1, agent 0.88.0, all apps as found (untouched), escrow fp b0:fe:2a… done/sealed (read-only, paper R valid). Disks unchanged (teszt_enroll sacrificial stick intact, not yet cycled). Host not rebooted. P8 demo sampler running (/root/c4-demo.csv).
• drill (192.168.0.152 / guest 9201): controller 0.128.0, agent 0.88.0, 5 boots tonight, healthy. actualbudget was DELETE-attempted (refused, still present) and had a c4_integrity/ folder (~1.3 GiB) injected into its volume — scratch data, safe to delete; app itself intact. A scratch .fab (/root/ref1.fab, 1.24 GiB) sits on the drill host — delete at will. Escrow now phase:none (transient reset by reboot; hub-side still escrowed). Drill needs a fresh Viktor-run ceremony if a paper R is wanted for it (per contract §4). P8 drill sampler running (/root/c4-drill.csv).
• Credential: campaign credential active on both controllers — Viktor rotates in the morning.

Morning recovery / cleanup

• Nothing is down; no recovery required. All hosts up, both guests healthy, hub ready.
• Stop samplers: pkill -f 'c4-sampler.sh' on felhom-pve and 192.168.0.152; pkill -f hub-sampler.sh on 180.
• Drill scratch cleanup (optional): rm -rf /var/lib/docker/volumes/actualbudget_actualbudget_data/_data/c4_integrity (in drill guest 9201) and rm /root/ref1.fab on 192.168.0.152.
• Rotate the campaign credential on both controllers.

Evidence index (180:~/campaign4/)

• seed.txt, ledger.md (full deviation + per-phase trail), evidence/P0/baseline-summary.md
• evidence/P8/hub.csv (hub readiness series); 192.168.0.162:/root/c4-demo.csv; 192.168.0.152:/root/c4-drill.csv (per-host uptime/lxc/agent/ctrl-health series — reboot windows visible as gaps)