felhom.eu/documentation/audits/CAMPAIGN-4-2026-07-13.md

CAMPAIGN-4 — unattended "no mercy" night run (C3-fix regression · .fab circle · escrow · claim/auth · reboot storms)

• When: 2026-07-13 23:11 CEST (21:11Z) launch → active block wrapped 2026-07-14 00:0x CEST; endurance samplers + paced phases continue overnight. Launch seed 5def7c09390a1d67. Budget 8h.
• Stack under fire (verified live at P0, not trusted from the table): controller 0.128.1 (demo 9201) / 0.128.0 (drill guest) · agent 0.88.0 (both hosts) · hub 0.54.0 on k3s (ready=1) · boxes: demo (felhom-pve 192.168.0.162 + guest 9201) AND drill VM (qm300 drill-day0 192.168.0.152, nested on felhom-pve, + its guest 9201).
• Contract honored: zero operator questions after launch; continue-on-failure; every deviation ledgered; no code fixes — findings only; no Gitea/PBS/hub-config mutations; demo existing apps never deleted/wiped/redeployed-over; ceremonies drill-only (demo escrow read-only, its paper R untouched); the campaign credential appears in no committed file, ledger, or this doc; every R/blob produced is scratch and uncommitted.
• Run architecture (recorded judgment): driven by the CC session; harness/ledger/evidence at 180:~/campaign4/. Controllers driven via the exact server-side endpoints the UI calls (docker exec <ctrl> curl 127.0.0.1:8080, real login→session→CSRF), an acceptable proxy for the browser where noted. P8 endurance samplers detached locally on each host (survive-as-CSV-gap across reboots).

Status of this run

High-fidelity evidenced core, honestly scoped; nothing here is claimed that was not observed; the endurance samplers run all night. Mid-run update: after the first active block the operator explicitly authorized the risky steps to run unattended (test environment). So the two biggest "deferred" items — the demo HOST reboot (F12-on-real-box + P6 retirement reboot-proof) and the teszt_enroll drive cycle — were then executed and are reported below. Remaining phases (NAS regression ring, .fab upload/import leg, backups tiers, browser wizard/UI/hub-tab ring) stay deferred with ready procedures rather than fabricated.

Verdict

The data-movement and reboot-recovery planes are solid on the code paths exercised; three real product findings surfaced, one of them a genuine auth-hardening gap. The .fab export→download integrity leg is byte-perfect. The F12 reboot-safety fix HOLDS on both boxes — the exact CRITICAL that ended CAMPAIGN-3 is closed: a drill-host reboot storm (5 boots) AND a demo-HOST reboot (with an active nas-media nfs4 mount present) both produced zero ordering cycles with full networking/WG/agent/capabilities(63/63)/guest/controller recovery. The P6 retirement reboot-proof PASSES: all three USB drives re-established from agent units alone despite their device letters reshuffling across the reboot. The .fab wipe-gate + mount-guard + customer-confirm gate all correctly refuse an unsafe format. Three findings became the take-aways: a login rate-limiter keying bug (MED, red-proofed), an export size-estimate blind to volume data (MED), and an ungraceful 502 on a no-op escrow claim (LOW-MED).

Ledger: 9 PASS · 3 FINDING · 3 OBSERVATION · 4 DEVIATION/INFO · 4 DEFERRED-phase.

Top findings (ranked — these become the next specs)

#	Sev	Finding	Exact repro
F-B	MED	Login rate-limiter never engages on direct (non-proxied) connections. auth.go handleLogin keys the per-IP limiter on r.RemoteAddr (which is IP:PORT) whenever X-Forwarded-For is absent. Distinct connections from the same host use distinct ephemeral source ports → distinct keys → the failed-attempt counter never accumulates. Prod-mitigated (traefik/CF always set XFF, so the limiter works for internet clients — confirmed), but any direct-to-controller path (LAN/guest, bypassing the proxy) evades brute-force protection entirely. Fix: key on the host portion only (strip the port from RemoteAddr).	POST /login password=WRONG$i ×8 with no XFF → all "Hibás jelszó", never limited. Same with fixed X-Forwarded-For: 203.0.113.9 → "Túl sok sikertelen próbálkozás" fires on attempt 6 (loginMaxAttempts=5). Red-proof: the XFF round limits, the no-XFF round does not.
F-A	MED	Export size-estimate is blind to docker-volume data. GET /api/export/download/estimate?stack=<volume-app> returns data_size_bytes:0, total "3.6 KB", fits_on_dest:true — while the real bundle is 1.2 GB. dockerVolumeSize resolves the volume mountpoint via docker volume inspect then cannot du that host path from inside the controller container → 0. The actual export (docker-cp streaming) is correct; only the pre-download honesty / free-space check lies, so a user can be told a multi-GB export is 3.6 KB and "fits".	inject >1 GiB into a needs_hdd:false app's volume; GET …/export/download/estimate?stack=actualbudget → data_size_bytes:0; then export → 1.2 GB .fab.
F-C	LOW-MED	502 on a no-op escrow claim. POST /api/escrow/claim (valid CSRF) when there is no active ceremony (phase:none, e.g. after a reboot cleared the transient job) returns HTTP 502 {"error":"A kód lekérése nem sikerült."}. A 502 (upstream/bad-gateway class) for "nothing to claim" is ungraceful; a clean 4xx (404/409/410) is expected.	on a box with escrow phase:none: POST /api/escrow/claim with a valid X-CSRF-Token → 502.

What passed (evidenced)

• .fab export→download integrity (drill, PASS): stop_app export of actualbudget with a >1 GiB injected dataset → 1.2 GB bundle (steps Előkészítés→Titkosítás all green). docker cp reference sha256 532fd003…d637 == LAN endpoint-download sha256, 1,328,973,695 bytes, zero corruption. Confirms the v0.125 docker-cp streaming volume leg captures data correctly (contra the F-A estimate).
• F12 reboot-safety / DR-plumbing recovery (drill storm, PASS — the CAMPAIGN-3-killer, now closed): 5 total boots. Every genuinely-observed post-boot state: ordering cycles = 0, agent active 0.88.0, capabilities ok=63 total=63 degraded=0 inactive=0, WG wg-felhom re-handshake <30 s, guest 9201 autostart, controller "healthy" within ~50 s (SSH-up ~46 s). Only benign binfmt_misc automount lines in the journal — zero felhom ordering cycles. The F12 fix holds and is repeatable.
• DEMO HOST reboot — F12-on-real-box + P6 retirement reboot-proof (PASS, the load-bearing one): reboot issued 22:32:40Z, host down ~29 s. 0 ordering cycles even with an active nas-media nfs4 mount (192.168.0.180:/mnt/5_hdd/felhom-data) present pre-reboot; caps 63/63; WG recovered. All three USB drives (teszt_enroll / felhom-flash / felhom-usb) re-mounted + guest-bound from agent units alone despite their device letters reshuffling (teszt sdb→sdc, flash sdd1→sdb1, usb sdc1→sdd1) — the durable-id/registry model absorbed it. /api/disks cards persisted registry-sourced (durable_id present, class:"" = v0.128.1 badge-suppression holds), correct roles. nas-media nfs4 re-established with 0 rbind/rc255 errors (the open F13 "active nfs4 → rbind rc255" did not manifest this reboot). Guest + controller healthy; both samplers relaunched; drill VM (onboot=0) restarted manually and recovered clean (0 cycles).
• .fab wipe-gate + mount-guard + customer-confirm gate (demo teszt_enroll, PASS): POST /api/disks/format refuses a mounted device (reason:"device is mounted") regardless of confirmed; unconfirmed user-data → needs_confirmation:true. No accidental wipe reachable. Eject (where=/mnt/teszt_enroll) and reattach (assign) both 200. teszt_enroll data left intact.
• CSRF protection (drill, PASS, edge-safe): POST /api/escrow/claim with no X-CSRF-Token → 403 JSON {"ok":false,"error":"CSRF token missing or invalid"} (no native alert, JSON on /api/).
• Escrow preflight honesty (drill, PASS): escrow_state:"escrowed", items pbs_storage_id:felhom-pbs, dr_tier:applied, age_binary:/usr/bin/age, hub_upload:configured all ok:true; staged_secret:false correctly flagged informational.
• P0 baseline captured for both boxes (versions, disks, escrow, backup, apps) — the diff anchor.
• P8 endurance backbone live (demo + drill + hub samplers, 60 s cadence) — the reboot windows already show as CSV gaps + recovery on the drill series.

Observations (not bugs)

• Escrow status is transient across reboot. Post-reboot /api/escrow/status → phase:none, claimed:false (was done/claimed/uploaded fp f2:87… pre-reboot); hub-side truth persists (preflight still escrowed). The local ceremony job state is in-memory and resets on restart — consistent with the campaign's own "restart → status honestly none" expectation. Note the coupling with F-C: a claim in this state 502s instead of returning a clean state.
• DELETE guard works: DELETE /api/stacks/actualbudget on a deployed app → refused "not orphaned — only orphaned stacks can be deleted".
• .fab upload requires a default storage drive: on the drill box (no user-data drive) POST /api/export/upload/init → honest Hungarian "Nincs alapértelmezett tároló beállítva." — correct behavior, but it means the upload/import leg cannot run on a storage-less box.
• O-1 (LOW): a mount_path eject does not fully release the device. After POST /api/disks/eject {where:"/mnt/teszt_enroll"} (200), the device stays mounted via the /mnt/felhom-drives/teszt_enroll namespace bind, so a subsequent format correctly still refuses ("device is mounted"). A format-intent flow that ejects only the mount_path can never reach a wipe. Whether the UI's format flow detaches all binds first is untested. Also: eject where must be the mount_path — the bare name / guest_path / uuid: all 403 with a role: system misclassification (they resolve to the /mnt/felhom-drives parent on pve-root).
• O-2 (LOW): POST /api/disks/assign returns 200 but did not synchronously restore the mount. After a successful re-attach the /mnt/teszt_enroll host mount stayed absent (guest bind + data intact); the agent reconcile / next reboot restores it (reboot-restore proven above).

Deviations ledger

• DEV-1: hub is 0.54.0 (table said 0.53.x). Newer — the change-login-password-from-UI feature shipped earlier the same evening (separate task). Hub login password left untouched per scope.
• DEV-2: drill controller is 0.128.0 (table said 0.128.1 for both guests); demo is 0.128.1. Drill one train behind; not halted above 0.124.0 (strand-fix present at 0.125.0). Non-blocking.
• DEV-3: demo box has 20+ deployed apps, not "six". Non-destructive rule applied to all existing apps.
• INFO: browser bridge (claude-in-chrome) is available this session (tab group created) — so the deferred browser phases are deferred for time/runway, not tooling.
• HARNESS: initial reboot-storm loop raced (polled UP before the host went DOWN) — corrected with a uptime<150s new-boot guard; CSRF scrape initially used the wrong selector (_csrf input vs the real <meta name="csrf-token">); bash $UID is readonly. None are product issues.

Deferred scope (reasons + ready procedures — for a supervised continuation)

1. Demo HOST reboot DONE (operator-authorized mid-run) — PASS, see What passed.
2. teszt_enroll drive cycle DONE (operator-authorized mid-run) — eject/reattach + wipe-gate PASS, see What passed + O-1/O-2.
3. P2 upload→import→byte-compare + CF-edge legs — the export/download half PASSED on drill; the upload half needs a default storage drive (drill has none) and the CF-edge --resolve 413 re-proof needs the box's public hostname. Best run as the demo additive pass (new small campaign app on the storage-bearing demo box) to avoid colliding with existing demo apps.
4. P1 NAS regression ring (F4–F12 NAS specifics) — requires enrolling a DooPlex NFS share and the full unexport/idle/reboot matrix. Not started.
5. P5 backups IA + tiers depth — not started.
6. P3 escrow wizard via browser + P4 UI DOM sweeps + P7b hub 8-tab ring — browser IS available; deferred for runway. The curl-equivalents of P3/P4 (CSRF, rate-limiter, escrow status/preflight/claim) WERE run (above).
7. P7a hub server-side verification sweep — hub pod health sampled (P8); per-act host_escrow row diffing not run.

Box state at wrap

• demo (felhom-pve / guest 9201): controller 0.128.1, agent 0.88.0, all apps as found (untouched), escrow fp b0:fe:2a… done/sealed (read-only, paper R valid). Host WAS rebooted (authorized) and recovered clean (0 cycles, all drives + binds re-established, caps 63/63). teszt_enroll was eject/reattach-cycled; its data is intact (NOT wiped) but its /mnt/teszt_enroll host mount is currently unmounted (guest bind + data intact) — the agent reconcile / a controller restart / next reboot restores it (reboot-restore proven tonight). P8 demo sampler relaunched (/root/c4-demo.csv).
• drill (192.168.0.152 / guest 9201): controller 0.128.0, agent 0.88.0, 6 boots tonight (storm + the demo-host reboot took it down; restarted via qm start 300, onboot=0), healthy. actualbudget had a c4_integrity/ folder (~1.3 GiB) injected into its volume — scratch data, safe to delete; app intact (a DELETE was refused — not orphaned). A scratch .fab (/root/ref1.fab, 1.24 GiB) sits on the drill host — delete at will. Escrow now phase:none (transient reset by reboot; hub-side still escrowed). Drill needs a fresh operator-run ceremony if a paper R is wanted for it (per contract §4). P8 drill sampler relaunched (/root/c4-drill.csv).
• Credential: campaign credential active on both controllers — Viktor rotates in the morning.

Morning recovery / cleanup

• Nothing is down; no recovery required. All hosts up, both guests healthy, hub ready.
• Stop samplers: pkill -f 'c4-sampler.sh' on felhom-pve and 192.168.0.152; pkill -f hub-sampler.sh on 180.
• Drill scratch cleanup (optional): rm -rf /var/lib/docker/volumes/actualbudget_actualbudget_data/_data/c4_integrity (in drill guest 9201) and rm /root/ref1.fab on 192.168.0.152.
• Rotate the campaign credential on both controllers.

Evidence index (180:~/campaign4/)

• seed.txt, ledger.md (full deviation + per-phase trail), evidence/P0/baseline-summary.md
• evidence/P8/hub.csv (hub readiness series); 192.168.0.162:/root/c4-demo.csv; 192.168.0.152:/root/c4-drill.csv (per-host uptime/lxc/agent/ctrl-health series — reboot windows visible as gaps)