admin
f97f7803c7
journalctl proved boot -1 came up at 23:31 WITHOUT networking: the agent automount template's After/Wants=network-online.target cycles with local-fs.target and systemd deleted networking.service to break it; the 06:45 power-cycle boot hit the same cycle and deleted the automount instead. Every boot with an enrolled share is a coin flip until the template drops that ordering. Recovery executed: automount started, apps healed, privatebin removed via real flow, verification backup green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf
21 lines
3.5 KiB
Markdown
21 lines
3.5 KiB
Markdown
# felhom.eu — task reports
|
||
|
||
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
|
||
|
||
## CAMPAIGN-3 — unattended "no mercy" night run (NAS · deploy · backup · chaos · observability) — 2026-07-11/12
|
||
|
||
**Full report: [`documentation/audits/CAMPAIGN-3-2026-07-11.md`](documentation/audits/CAMPAIGN-3-2026-07-11.md).** Run 22:09 → 04:27 CEST against the demo box (controller 0.117.0 / agent 0.84.0), operator-unattended, real surfaces only, no code changes. Ledger (final incl. morning RCA/recovery): **31 PASS · 17 FAIL · 12 FINDING · 1 DISCREPANCY** (70+ scenario entries, evidence at `180:~/campaign3/`).
|
||
|
||
### Headlines
|
||
|
||
- **CRITICAL F12 — the overnight host loss, RCA closed (hardware exonerated):** the agent's network-storage **automount** template (`After=`+`Wants=network-online.target`, implicitly `Before=local-fs.target`) creates a boot **ordering cycle**; systemd breaks it by deleting an arbitrary job. Boot at 23:31 sacrificed `networking.service` → host up **7 hours with no network**; the 06:45 power-cycle boot hit the same cycle and sacrificed the **automount** instead (NAS dead, healed manually). **Every boot of a host with an enrolled share is a coin flip until the template drops the network-online ordering** (`_netdev` on the `.mount` suffices). 4e caught exactly what it was designed to catch.
|
||
- **CRITICAL F10 + HIGH F11/F9 — the reboot/recovery plane around NAS automounts is broken:** `mount-start-limit-hit` is never re-armed by any heal path (once even blocked guest start → guest DOWN); guest reboot with an idle share leaves the autofs trigger unpropagated into the container — the post-start reassert **logs its own WARNING and then skips** the automount restart that provably heals ("skip-active" branch). Every such reboot = 4 NAS apps dead-at-boot. Reproduced on 3 of 3 guest reboots.
|
||
- **HIGH F7 — backup dumps are written in place (no tmp+rename):** a mid-backup NAS cut left a 0-byte tar *replacing* the last good 247M dump; in that window restore = empty volume. Next run self-heals; run-level `success:false` is the only signal.
|
||
- **The data plane held:** all 5 refusal categories ×2 correct + fast (2–5 s, retry=0), verify/rollback/single-flight/orphan flows clean, deploy-view truth holds, restore round-trips **byte-identical**, EIO same-second under outage, organic stub → badge + deploy-409 live-validated, hardlinks work on NFSv4.1.
|
||
- **Fix-6 answered with numbers:** ring cap horizon = ~55 min idle but **~6.5 min under load**; every restart/reboot wipes both rings — persistence, not just size, is the gap.
|
||
- **Policy discovery (docs):** tier-1 = volumes+config only (NAS media userdata excluded by design); NAS apps' tier-1 lands *on the NAS*, tier-2 is what gets it off; volume-only apps back up to sys_drive with blank drive label and **no tier-2 copy**.
|
||
|
||
### Box state / cleanup (final, 06:53)
|
||
|
||
DooPlex NAS restored **md5-identical to baseline** (exports + smb.conf; campaign user/share/dirs/creds removed); services never touched (exportfs-only rail held). Host recovered post-power-cycle; guest 9201 in **defined state**: 6 wave apps deployed + healthy with data, `privatebin` stop+removed via the real flow, verification backup `success:true`, nas-media `ok`, stub 0. Hub untouched throughout. ⚠ Next host reboot re-rolls the F12 dice until the agent template is fixed (interim: systemd drop-in on the automount units).
|