felhom.eu/documentation/audits/CAMPAIGN-3-2026-07-11.md

CAMPAIGN-3 — unattended "no mercy" night run (NAS · deploy · backup · chaos · observability)

• When: 2026-07-11 22:09 → 2026-07-12 HH:MM CEST (launch seed 2868120970, 8h budget, wrap early — see host incident)
• Stack under fire: controller v0.117.0 · agent v0.84.0 · demo guest 9201 on felhom-pve · DooPlex NAS (exports only) · hub untouched
• Contract honored: zero operator questions; continue-on-failure; no hub/PBS/Gitea mutations; DooPlex services never stopped (exportfs on felhom paths + campaign temps only); no code fixes — findings only.
• Run architecture (recorded judgment): campaign driven by the CC session with harness/ledger/evidence at 180:~/campaign3/ (180 has no root SSH to pve; evidence spans 180-API + pve-host + guest vantage points). Phase 5 sampler + Phase 6 endurance ran detached on 180.

Verdict

The data plane is solid; the reboot/recovery plane is not. Refusal categories, verify-before-commit, rollback, single-flight, orphan handling, deploy-view truth, backup/restore round-trips and EIO-fast-fail all behaved. What failed is everything around re-arming NFS automounts after disturbance: three independent HIGH/CRITICAL findings (F9/F10/F11) mean any NAS hiccup or guest reboot with an idle share strands all NAS apps until a human intervenes — the exact gap the v0.84.0 ReassertNetworkMounts feature was meant to close. Plus one HIGH backup-integrity finding (F7) and the host itself failed to return from its reboot (physical attention needed).

Ledger (final): 30 PASS · 17 FAIL · 11 FINDING · 1 DISCREPANCY · 10 INFO (FAILs include endurance-loop iterations that honestly fired during reboot/outage/host-down windows).

Top findings (ranked — these become the next specs)

#	Sev	Finding	Exact repro
F10	CRITICAL	mount-start-limit-hit on a NAS automount is never re-armed by any platform path: agent-restart sweep no-op (F9), guest-reboot hook no-op, and the pre-start hook once blocked guest start entirely (rc255 while the limit was live → guest DOWN). All 4 NAS apps dead until manual systemctl reset-failed.	unexport share on NAS; wait 60s idle-timeout; access path 5×; pct reboot within the limit window
F11	HIGH	Guest reboot with an idle (armed-not-mounted) share: the autofs trigger does not propagate into the new container namespace. The post-start reassert detects it and logs a WARNING ("share NOT visible in guest after reassert (skip-active)") but does not remediate — the skip-active branch is the bug. Manual systemctl restart <automount> heals instantly. Every reboot with idle share = 4 NAS apps dead-at-boot (couples with fix-3: no alerting). Reproduced on reboots #2, #3, #4.	pct reboot 9201 while nas-media idle; watch hook WARNING + docker ps
F7	HIGH	Volume dumps are written in place, no tmp+rename: a mid-backup NAS cut left a 0-byte calibre tar replacing the last good 247M dump. Until the next successful run, the only "restore point" restores an empty volume (tier-1 restore is replace-semantics). Run-level success:false is the only signal; the artifact damage itself is silent. Next run self-heals.	POST /api/backup/run; exportfs -u the backup-target share at T+6s; inspect volume-dumps/*.tar
F9	HIGH	Agent-restart startup sweep produced zero reassert log lines and left the automount failed/disarmed — likely checks unit enablement, not runtime state.	get F10 state; systemctl restart felhom-agent; grep journal
F8	MED	Storage health stays idle + reachable:true through an export outage (probe is server-level, not export-level) — even across a reboot with NFS down the share badge shows benign "Készenlét" while stacks/dashboard show stub warnings. Spec expected unreachable; got neither silent nor correct: contradictory UI.	exportfs -u while idle; poll /api/storage/netstorage 5 min
F6	LOW	Volume-only apps (no HDD_PATH) back up to sys_drive with drive_label:"" (blank in UI) and get no tier-2 secondary — single-copy exposure.	GET /api/backup/snapshots?stack=actualbudget
F4	LOW	mapped_uid out of range (101000) passes controller validation, fails only at agent → generic agent_error with raw detail instead of a friendly 400.	add with mapped_uid:101000
F5	LOW	backups/primary/ retains stale dirs for apps since redeployed to another drive (invisible in snapshot list, disk residue).	redeploy app to other drive; inspect old drive
F2	LOW	Failed-add rollback removes unit files but leaves systemd runtime units in failed state (no reset-failed) + leftover mountpoint dirs.	any refused add past reachability
F1	LOW	netstorage/remove leaves the empty mountpoint dir on host+guest (stub-shaped residue; 10 accumulated during P1).	add+remove any share
F3	DISCREPANCY	Spec expected not_network_fs for a no-all_squash export; actual = not_writable (×2, deterministic, message even hints all_squash). Correct-by-design: not_network_fs is the deploy/stub-time branch — which WAS live-validated via the organic stub (below).	—

Host incident (HUMAN): felhom-pve did not return from the planned 4e reboot at 23:30 (no ping, SSH 22/8822 closed, controller 530 via Cloudflare). Polled every 5 min until 04:27 — never came back. Needs physical attention (power/console). Note this reboot was issued while the box was in a healed, settled state — the failure is host-level (BIOS/boot device/console prompt?), not something the campaign's guest chaos can explain. The 4e boot-ordering proof is therefore NOT RUN.

What passed (the good news)

• NAS add/remove lifecycle (~19 ops): happy NFS routeA-style 2.6s / prod-style 2.5s / SMB 2.5s; all 5 refusal categories ×2 with correct category + Hungarian copy in 2–5s (retry=0 holds everywhere); single-flight = exactly one 409; SMB password lands in a root-only credentials= file and is gone after remove; registry consistent after every cycle.
• Ring reconstruction (S7): every add fully reconstructable — phase transitions, capability gate: netstorage_verify=yes (source=version), probe verdict, timings.
• Orphan flow: controller restart mid-add → Árva megosztás row surfaced → removed via the real action → clean.
• Deploy wave: radarr/jellyfin/navidrome onto nas-media + seerr/actualbudget (volume-only) + privatebin; all running; deploy-view truth holds for HDD apps (selected option == stored path; volume-only apps have no HDD_PATH field — N/A recorded, not a bug).
• *Hardlink (arr pattern) works on NFSv4.1 (links=2, correct uid mapping) — the Synology-era caveat does not apply to the DooPlex export.
• Backup/restore: tier-1 31s / tier-2 5s / offsite restic cycle ~30s clean (quota 129.7MB/50GB); restore round-trips byte-identical on calibre-web (53 files, deleted+corrupted recovered, 25s) and actualbudget; tier-1 restore = replace-semantics (defined; tier-2 is the additive path).
• EIO behavior under outage: active read fails same-second (soft,timeo=50,retrans=2), D-count=0, containers survive, re-export recovers without remount; mid-backup cut fails clean in 14s, no hang.
• Stub machinery: the organic stub (outage→idle-timeout→access) fires the distinct stub badge (4 apps) AND the deploy-time 409 with the exact Hungarian refusal. The branch is now live-validated.
• Concurrency spice: deploy-during-add, backup race (200/409), rapid add/remove ×5, remove-during-backup — all clean.

Backup-vs-NAS policy discovery (docs finding, not pass/fail)

1. Tier-1 backs up volumes + config only; NAS-hosted HDD userdata (the 2.8G media tree) is excluded by design (jellyfin backup = 560K).
2. NAS apps' tier-1 artifacts land on the NAS itself (nas-media/backups/primary/) — tier-2 then copies them cross-drive to felhom-usb, which is what gets them off-NAS. During a NAS outage both tier-1 (target) and the app data are on the dead device.
3. Volume-only apps → sys_drive/felhom-data/backups/primary/, blank drive label, no tier-2 copy (F6).
4. Outage during backup: clean failure, plus F7's truncation window.

Fix-6 ring numbers (wrap horizon, measured)

• Idle: controller ring at 1000-cap with ~55 min horizon (21:20→22:15).
• Under campaign load: ~2.6 entries/s → cap horizon ≈ 6.5 min (23:04→23:08 = 618 entries). A refused flow older than ~6 min under load is already unreconstructable.
• Restart/reboot loss: controller restart at 22:26 wiped the ring (the interrupted orphan-add's trace was gone); both rings died on every reboot (in-memory, expected) — post-incident operators lose exactly the window that matters. The viewer's total field tracked entries correctly in all samples (no 1067/1000-style overshoot reproduced; the counter reads clean).
• Verdict for fix-6: the 1000-entry cap is too small under load and ring persistence across restart is the real gap.

Timings table

Operation	Measured
happy add (NFS/SMB)	2.5–2.6 s
refusals (all categories)	2–5 s
EIO after unexport (active read)	same second
backup fail-clean after cut	14 s
deploy → running	25–60 s
tier-1 / tier-2 / offsite run	31 s / 5 s / 30 s
restore round-trip	25 s
boot→all-healthy	never zero-touch with idle NAS share (F11); manual heal ~1 min

Deviations & judgment calls (all ledgered)

• Reboots #2/#3 spaced 5–6 min instead of 15 (fast phase pace; state verified settled between). #4 and host reboot properly spaced.
• Endurance-loop injections limited to campaign exports (not felhom-data) to keep the unattended loop from re-tripping F10 with no pve-side healer.
• App-level API records skipped for seerr/actualbudget (interactive auth setup — not trivial per spec).
• ?raw=1 on the debug ring does not exist (it belongs to per-app container logs); JSON /api/debug/logs used instead.
• One offsite cycle run (of ≤2 allowed); second budget unused.
• No midnight operator push about the host (expendable demo box, Peti unaffected); this report is the notification.

Box state at wrap

• felhom-pve: DOWN (since the 23:30 planned reboot; HUMAN: physical power/console needed).
• DooPlex/NAS: fully restored — /etc/exports and smb.conf restored from .bak-campaign3, md5 == baseline (3 exports); campaign SMB user/share/creds/dirs removed; felhom-data export live; all customer/homelab services untouched throughout (exportfs-only rail held).
• Guest 9201 (as last seen, 23:33, pre-host-reboot): 11 containers healthy — controller 0.117.0, calibre-web/radarr/jellyfin/navidrome (nas-media), seerr/actualbudget (volumes), privatebin (campaign temp — removal via the real flow was pre-empted by the host loss; remove after recovery), filebrowser + infra. Dummy media data (2.8G) + backups intact on the NAS.
• Hub: untouched all night (reports were flowing until host loss; row will show the host offline — correct).

Morning recovery runbook (operator)

1. Power/console felhom-pve; check why boot stalled (BIOS/GRUB/fsck).
2. After boot, expect F10/F11: NAS apps dead-at-boot. Heal: systemctl reset-failed 'mnt-felhom*', systemctl restart 'mnt-felhom\x2ddrives-nas\x2dmedia.automount', then in guest docker start radarr jellyfin navidrome calibre-web.
3. POST /api/stacks/privatebin/remove (campaign temp app) — or leave it, it's harmless.
4. Verify: no stub badge on /stacks, nas-media ok, hub row green, one manual POST /api/backup/run green.

Evidence

180:~/campaign3/ — ledger.tsv (66+ entries), master.log (every API call), evidence/ (per-scenario ring+shares+stacks snapshots), ring-samples.log. Guest-side manifests at 9201:/root/c3/. No secrets in any committed artifact; campaign SMB cred stored only on 180 and deleted at wrap.