admin
f97f7803c7
journalctl proved boot -1 came up at 23:31 WITHOUT networking: the agent automount template's After/Wants=network-online.target cycles with local-fs.target and systemd deleted networking.service to break it; the 06:45 power-cycle boot hit the same cycle and deleted the automount instead. Every boot with an enrolled share is a coin flip until the template drops that ordering. Recovery executed: automount started, apps healed, privatebin removed via real flow, verification backup green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf
3.5 KiB
felhom.eu — task reports
Overwrite this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in hub/CHANGELOG.md; the scripts history lives in scripts/CHANGELOG.md.
CAMPAIGN-3 — unattended "no mercy" night run (NAS · deploy · backup · chaos · observability) — 2026-07-11/12
Full report: documentation/audits/CAMPAIGN-3-2026-07-11.md. Run 22:09 → 04:27 CEST against the demo box (controller 0.117.0 / agent 0.84.0), operator-unattended, real surfaces only, no code changes. Ledger (final incl. morning RCA/recovery): 31 PASS · 17 FAIL · 12 FINDING · 1 DISCREPANCY (70+ scenario entries, evidence at 180:~/campaign3/).
Headlines
- CRITICAL F12 — the overnight host loss, RCA closed (hardware exonerated): the agent's network-storage automount template (
After=+Wants=network-online.target, implicitlyBefore=local-fs.target) creates a boot ordering cycle; systemd breaks it by deleting an arbitrary job. Boot at 23:31 sacrificednetworking.service→ host up 7 hours with no network; the 06:45 power-cycle boot hit the same cycle and sacrificed the automount instead (NAS dead, healed manually). Every boot of a host with an enrolled share is a coin flip until the template drops the network-online ordering (_netdevon the.mountsuffices). 4e caught exactly what it was designed to catch. - CRITICAL F10 + HIGH F11/F9 — the reboot/recovery plane around NAS automounts is broken:
mount-start-limit-hitis never re-armed by any heal path (once even blocked guest start → guest DOWN); guest reboot with an idle share leaves the autofs trigger unpropagated into the container — the post-start reassert logs its own WARNING and then skips the automount restart that provably heals ("skip-active" branch). Every such reboot = 4 NAS apps dead-at-boot. Reproduced on 3 of 3 guest reboots. - HIGH F7 — backup dumps are written in place (no tmp+rename): a mid-backup NAS cut left a 0-byte tar replacing the last good 247M dump; in that window restore = empty volume. Next run self-heals; run-level
success:falseis the only signal. - The data plane held: all 5 refusal categories ×2 correct + fast (2–5 s, retry=0), verify/rollback/single-flight/orphan flows clean, deploy-view truth holds, restore round-trips byte-identical, EIO same-second under outage, organic stub → badge + deploy-409 live-validated, hardlinks work on NFSv4.1.
- Fix-6 answered with numbers: ring cap horizon = ~55 min idle but ~6.5 min under load; every restart/reboot wipes both rings — persistence, not just size, is the gap.
- Policy discovery (docs): tier-1 = volumes+config only (NAS media userdata excluded by design); NAS apps' tier-1 lands on the NAS, tier-2 is what gets it off; volume-only apps back up to sys_drive with blank drive label and no tier-2 copy.
Box state / cleanup (final, 06:53)
DooPlex NAS restored md5-identical to baseline (exports + smb.conf; campaign user/share/dirs/creds removed); services never touched (exportfs-only rail held). Host recovered post-power-cycle; guest 9201 in defined state: 6 wave apps deployed + healthy with data, privatebin stop+removed via the real flow, verification backup success:true, nas-media ok, stub 0. Hub untouched throughout. ⚠ Next host reboot re-rolls the F12 dice until the agent template is fixed (interim: systemd drop-in on the automount units).