felhom.eu/REPORT.md

felhom.eu — task reports

Overwrite this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in hub/CHANGELOG.md; the scripts history lives in scripts/CHANGELOG.md.

CAMPAIGN-3 — unattended "no mercy" night run (NAS · deploy · backup · chaos · observability) — 2026-07-11/12

Full report: documentation/audits/CAMPAIGN-3-2026-07-11.md. Run 22:09 → 04:27 CEST against the demo box (controller 0.117.0 / agent 0.84.0), operator-unattended, real surfaces only, no code changes. Ledger: 30 PASS · 17 FAIL · 11 FINDING · 1 DISCREPANCY (66+ scenario entries, evidence at 180:~/campaign3/).

Headlines

• ⚠ HUMAN, first thing: felhom-pve did not return from the planned 4e host reboot (23:30) — polled every 5 min until 04:27; no ping, no SSH (22/8822), controller 530. Needs physical power/console. Morning recovery runbook is in the report (§Box state) — after boot expect F10/F11 and heal with reset-failed + automount restart + docker start of the 4 NAS apps.
• CRITICAL F10 + HIGH F11/F9 — the reboot/recovery plane around NAS automounts is broken: mount-start-limit-hit is never re-armed by any heal path (once even blocked guest start → guest DOWN); guest reboot with an idle share leaves the autofs trigger unpropagated into the container — the post-start reassert logs its own WARNING and then skips the automount restart that provably heals ("skip-active" branch). Every such reboot = 4 NAS apps dead-at-boot. Reproduced on 3 of 3 guest reboots.
• HIGH F7 — backup dumps are written in place (no tmp+rename): a mid-backup NAS cut left a 0-byte tar replacing the last good 247M dump; in that window restore = empty volume. Next run self-heals; run-level success:false is the only signal.
• The data plane held: all 5 refusal categories ×2 correct + fast (2–5 s, retry=0), verify/rollback/single-flight/orphan flows clean, deploy-view truth holds, restore round-trips byte-identical, EIO same-second under outage, organic stub → badge + deploy-409 live-validated, hardlinks work on NFSv4.1.
• Fix-6 answered with numbers: ring cap horizon = ~55 min idle but ~6.5 min under load; every restart/reboot wipes both rings — persistence, not just size, is the gap.
• Policy discovery (docs): tier-1 = volumes+config only (NAS media userdata excluded by design); NAS apps' tier-1 lands on the NAS, tier-2 is what gets it off; volume-only apps back up to sys_drive with blank drive label and no tier-2 copy.

Box state / cleanup

DooPlex NAS restored md5-identical to baseline (exports + smb.conf; campaign user/share/dirs/creds removed); services never touched (exportfs-only rail held). Guest 9201 as last seen healthy (11 containers, dummy data + backups intact on NAS); privatebin (campaign temp) removal deferred to post-recovery. Hub untouched all night.