From f97f7803c75e9d95fe237ca91a9ecdec181622fc Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sun, 12 Jul 2026 06:55:40 +0200 Subject: [PATCH] =?UTF-8?q?CAMPAIGN-3:=20morning=20RCA=20=E2=80=94=20host?= =?UTF-8?q?=20loss=20=3D=20F12=20CRITICAL=20automount=20ordering=20cycle?= =?UTF-8?q?=20(hardware=20exonerated);=20box=20recovered=20to=20defined=20?= =?UTF-8?q?state?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit journalctl proved boot -1 came up at 23:31 WITHOUT networking: the agent automount template's After/Wants=network-online.target cycles with local-fs.target and systemd deleted networking.service to break it; the 06:45 power-cycle boot hit the same cycle and deleted the automount instead. Every boot with an enrolled share is a coin flip until the template drops that ordering. Recovery executed: automount started, apps healed, privatebin removed via real flow, verification backup green. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf --- CONTEXT.md | 23 +++++++++++-------- REPORT.md | 8 +++---- documentation/audits/CAMPAIGN-3-2026-07-11.md | 23 +++++++++++-------- 3 files changed, 31 insertions(+), 23 deletions(-) diff --git a/CONTEXT.md b/CONTEXT.md index cc9e3e9..6482399 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,16 +3,21 @@ > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. -- **2026-07-12 — CAMPAIGN-3 NIGHT RUN DONE; ⚠ DEMO HOST DOWN (HUMAN).** Unattended chaos campaign +- **2026-07-12 — CAMPAIGN-3 NIGHT RUN DONE; host recovered, RCA closed.** Unattended chaos campaign 22:09→04:27 vs demo box (ctrl 0.117.0/agent 0.84.0): data plane solid (refusals/verify/rollback/ - restore byte-identical/EIO instant/stub badge + deploy-409 live-validated), but **felhom-pve never - returned from the planned host reboot** (physical power/console needed; recovery runbook in the - audit) and the **NAS automount re-arm plane failed**: F10 CRITICAL (start-limit never re-armed, - once blocked guest start), F11/F9 HIGH (post-start reassert "skip-active" skips the healing - automount restart; agent-restart sweep silent) → 4 NAS apps dead-at-boot on 3/3 guest reboots; - F7 HIGH (in-place dump write → 0-byte tar replaced last good backup during outage window). - Fix-6 numbers: ring horizon 55min idle / **6.5min under load**, wiped on every restart. - Full ledger/evidence: `documentation/audits/CAMPAIGN-3-2026-07-11.md` + `180:~/campaign3/`. + restore byte-identical/EIO instant/stub badge + deploy-409 live-validated). The NAS automount + plane produced 2 CRITICAL + 3 HIGH: **F12 CRITICAL (= the overnight host loss, RCA'd next + morning): the agent automount template's `After/Wants=network-online.target` creates a boot + ordering cycle — systemd deletes an arbitrary job to break it; boot -1 sacrificed networking + (host dark 7h, hardware exonerated), boot 0 sacrificed the automount (NAS dead). Every host + boot with an enrolled share = coin flip until the template drops that ordering (`_netdev` on + the .mount suffices).** F10 CRITICAL (start-limit never re-armed, once blocked guest start), + F11/F9 HIGH (post-start reassert "skip-active" skips the healing automount restart; agent-restart + sweep silent) → 4 NAS apps dead-at-boot on 3/3 guest reboots; F7 HIGH (in-place dump write → + 0-byte tar replaced last good backup during outage window). Fix-6 numbers: ring horizon 55min + idle / **6.5min under load**, wiped on every restart. Box left in defined state (6 wave apps + healthy, temps removed, NAS baseline-identical). Full ledger/evidence: + `documentation/audits/CAMPAIGN-3-2026-07-11.md` + `180:~/campaign3/`. - **2026-07-11 — HUB v0.47.0 UI REORGANIZATION SHIPPED** (CHANGELOG v0.47.0; commits `9f29bf3` → `0daddcd` + docs). Five deliverables: **(1)** `.data-table td a:not(.btn)` button-contrast fix; **(2)** customer page = **8 hash tabs** (`#tab=…`, sticky summary strip, Events error badge, diff --git a/REPORT.md b/REPORT.md index f0068b4..b297510 100644 --- a/REPORT.md +++ b/REPORT.md @@ -4,17 +4,17 @@ ## CAMPAIGN-3 — unattended "no mercy" night run (NAS · deploy · backup · chaos · observability) — 2026-07-11/12 -**Full report: [`documentation/audits/CAMPAIGN-3-2026-07-11.md`](documentation/audits/CAMPAIGN-3-2026-07-11.md).** Run 22:09 → 04:27 CEST against the demo box (controller 0.117.0 / agent 0.84.0), operator-unattended, real surfaces only, no code changes. Ledger: **30 PASS · 17 FAIL · 11 FINDING · 1 DISCREPANCY** (66+ scenario entries, evidence at `180:~/campaign3/`). +**Full report: [`documentation/audits/CAMPAIGN-3-2026-07-11.md`](documentation/audits/CAMPAIGN-3-2026-07-11.md).** Run 22:09 → 04:27 CEST against the demo box (controller 0.117.0 / agent 0.84.0), operator-unattended, real surfaces only, no code changes. Ledger (final incl. morning RCA/recovery): **31 PASS · 17 FAIL · 12 FINDING · 1 DISCREPANCY** (70+ scenario entries, evidence at `180:~/campaign3/`). ### Headlines -- **⚠ HUMAN, first thing:** felhom-pve **did not return** from the planned 4e host reboot (23:30) — polled every 5 min until 04:27; no ping, no SSH (22/8822), controller 530. Needs physical power/console. Morning recovery runbook is in the report (§Box state) — after boot expect F10/F11 and heal with `reset-failed` + automount restart + `docker start` of the 4 NAS apps. +- **CRITICAL F12 — the overnight host loss, RCA closed (hardware exonerated):** the agent's network-storage **automount** template (`After=`+`Wants=network-online.target`, implicitly `Before=local-fs.target`) creates a boot **ordering cycle**; systemd breaks it by deleting an arbitrary job. Boot at 23:31 sacrificed `networking.service` → host up **7 hours with no network**; the 06:45 power-cycle boot hit the same cycle and sacrificed the **automount** instead (NAS dead, healed manually). **Every boot of a host with an enrolled share is a coin flip until the template drops the network-online ordering** (`_netdev` on the `.mount` suffices). 4e caught exactly what it was designed to catch. - **CRITICAL F10 + HIGH F11/F9 — the reboot/recovery plane around NAS automounts is broken:** `mount-start-limit-hit` is never re-armed by any heal path (once even blocked guest start → guest DOWN); guest reboot with an idle share leaves the autofs trigger unpropagated into the container — the post-start reassert **logs its own WARNING and then skips** the automount restart that provably heals ("skip-active" branch). Every such reboot = 4 NAS apps dead-at-boot. Reproduced on 3 of 3 guest reboots. - **HIGH F7 — backup dumps are written in place (no tmp+rename):** a mid-backup NAS cut left a 0-byte tar *replacing* the last good 247M dump; in that window restore = empty volume. Next run self-heals; run-level `success:false` is the only signal. - **The data plane held:** all 5 refusal categories ×2 correct + fast (2–5 s, retry=0), verify/rollback/single-flight/orphan flows clean, deploy-view truth holds, restore round-trips **byte-identical**, EIO same-second under outage, organic stub → badge + deploy-409 live-validated, hardlinks work on NFSv4.1. - **Fix-6 answered with numbers:** ring cap horizon = ~55 min idle but **~6.5 min under load**; every restart/reboot wipes both rings — persistence, not just size, is the gap. - **Policy discovery (docs):** tier-1 = volumes+config only (NAS media userdata excluded by design); NAS apps' tier-1 lands *on the NAS*, tier-2 is what gets it off; volume-only apps back up to sys_drive with blank drive label and **no tier-2 copy**. -### Box state / cleanup +### Box state / cleanup (final, 06:53) -DooPlex NAS restored **md5-identical to baseline** (exports + smb.conf; campaign user/share/dirs/creds removed); services never touched (exportfs-only rail held). Guest 9201 as last seen healthy (11 containers, dummy data + backups intact on NAS); `privatebin` (campaign temp) removal deferred to post-recovery. Hub untouched all night. +DooPlex NAS restored **md5-identical to baseline** (exports + smb.conf; campaign user/share/dirs/creds removed); services never touched (exportfs-only rail held). Host recovered post-power-cycle; guest 9201 in **defined state**: 6 wave apps deployed + healthy with data, `privatebin` stop+removed via the real flow, verification backup `success:true`, nas-media `ok`, stub 0. Hub untouched throughout. ⚠ Next host reboot re-rolls the F12 dice until the agent template is fixed (interim: systemd drop-in on the automount units). diff --git a/documentation/audits/CAMPAIGN-3-2026-07-11.md b/documentation/audits/CAMPAIGN-3-2026-07-11.md index 7514be6..a4bbbb7 100644 --- a/documentation/audits/CAMPAIGN-3-2026-07-11.md +++ b/documentation/audits/CAMPAIGN-3-2026-07-11.md @@ -8,14 +8,15 @@ ## Verdict **The data plane is solid; the reboot/recovery plane is not.** -Refusal categories, verify-before-commit, rollback, single-flight, orphan handling, deploy-view truth, backup/restore round-trips and EIO-fast-fail all behaved. What failed is everything around **re-arming NFS automounts after disturbance**: three independent HIGH/CRITICAL findings (F9/F10/F11) mean any NAS hiccup or guest reboot with an idle share strands all NAS apps until a human intervenes — the exact gap the v0.84.0 ReassertNetworkMounts feature was meant to close. Plus one HIGH backup-integrity finding (F7) and the host itself failed to return from its reboot (physical attention needed). +Refusal categories, verify-before-commit, rollback, single-flight, orphan handling, deploy-view truth, backup/restore round-trips and EIO-fast-fail all behaved. What failed is everything around **NFS automount units and their re-arming**: four independent CRITICAL/HIGH findings (F12/F10/F11/F9) mean a host reboot is a coin flip between losing networking and losing the NAS (F12), and any NAS hiccup or guest reboot with an idle share strands all NAS apps until a human intervenes — the exact gap the v0.84.0 ReassertNetworkMounts feature was meant to close. Plus one HIGH backup-integrity finding (F7). The overnight host loss was F12 in action — hardware exonerated by RCA. -Ledger (final): **30 PASS · 17 FAIL · 11 FINDING · 1 DISCREPANCY · 10 INFO** (FAILs include endurance-loop iterations that honestly fired during reboot/outage/host-down windows). +Ledger (final): **31 PASS · 17 FAIL · 12 FINDING · 1 DISCREPANCY · 11 INFO** (FAILs include endurance-loop iterations that honestly fired during reboot/outage/host-down windows). ## Top findings (ranked — these become the next specs) | # | Sev | Finding | Exact repro | |---|-----|---------|-------------| +| F12 | **CRITICAL** | Agent network-storage **automount** unit orders `After=`+`Wants=network-online.target` → boot **ordering cycle** with `local-fs.target`; systemd deletes an arbitrary job to break it — one boot lost **networking entirely** (host dark 7 h), the next boot lost the **automount** (NAS dead). Every boot with an enrolled share is a coin flip. Fix: remove network-online ordering from the automount (`_netdev` on the `.mount` suffices). | reboot a host with any enrolled network share; `journalctl -b \| grep "ordering cycle"` | | F10 | **CRITICAL** | `mount-start-limit-hit` on a NAS automount is **never re-armed** by any platform path: agent-restart sweep no-op (F9), guest-reboot hook no-op, and the **pre-start hook once blocked guest start entirely** (rc255 while the limit was live → guest DOWN). All 4 NAS apps dead until manual `systemctl reset-failed`. | unexport share on NAS; wait 60s idle-timeout; access path 5×; `pct reboot` within the limit window | | F11 | **HIGH** | Guest reboot with an **idle** (armed-not-mounted) share: the autofs trigger does **not** propagate into the new container namespace. The post-start reassert **detects it and logs a WARNING** ("share NOT visible in guest after reassert (skip-active)") **but does not remediate** — the skip-active branch is the bug. Manual `systemctl restart ` heals instantly. Every reboot with idle share = 4 NAS apps dead-at-boot (couples with fix-3: no alerting). Reproduced on reboots #2, #3, #4. | `pct reboot 9201` while nas-media idle; watch hook WARNING + `docker ps` | | F7 | **HIGH** | Volume dumps are written **in place, no tmp+rename**: a mid-backup NAS cut left a **0-byte calibre tar replacing the last good 247M dump**. Until the next successful run, the only "restore point" restores an **empty volume** (tier-1 restore is replace-semantics). Run-level `success:false` is the only signal; the artifact damage itself is silent. Next run self-heals. | `POST /api/backup/run`; `exportfs -u` the backup-target share at T+6s; inspect `volume-dumps/*.tar` | @@ -28,7 +29,7 @@ Ledger (final): **30 PASS · 17 FAIL · 11 FINDING · 1 DISCREPANCY · 10 INFO** | F1 | LOW | `netstorage/remove` leaves the empty mountpoint dir on host+guest (stub-shaped residue; 10 accumulated during P1). | add+remove any share | | F3 | DISCREPANCY | Spec expected `not_network_fs` for a no-all_squash export; actual = `not_writable` (×2, deterministic, message even hints all_squash). Correct-by-design: `not_network_fs` is the deploy/stub-time branch — which WAS live-validated via the organic stub (below). | — | -**Host incident (HUMAN):** felhom-pve did **not return** from the planned 4e reboot at 23:30 (no ping, SSH 22/8822 closed, controller 530 via Cloudflare). Polled every 5 min until **04:27 — never came back**. Needs physical attention (power/console). Note this reboot was issued while the box was in a healed, settled state — the failure is host-level (BIOS/boot device/console prompt?), not something the campaign's guest chaos can explain. The 4e boot-ordering proof is therefore **NOT RUN**. +**Host incident — RCA CLOSED 2026-07-12 06:48 (= finding F12, CRITICAL).** felhom-pve was unreachable from the 4e reboot (23:30) until the operator power-cycled it at 06:45. `journalctl --list-boots` proved the box **booted fine at 23:31:04 and ran all night — with no network**: the agent's automount unit template (`After=` + `Wants=network-online.target` on the **automount** unit, which is implicitly `Before=local-fs.target`) creates the boot ordering cycle `networking → local-fs → nas-media.automount → network-online → networking`. systemd breaks such a cycle by deleting an **arbitrary** job in it: boot `-1` deleted `networking.service` (host up, no network, 7 h); boot `0` after the power cycle hit the **same cycle** and deleted the **automount** instead (network up, NAS dead — recovered manually). **Every boot of a host with an enrolled network share is a coin flip.** Hardware exonerated; the power cycle fixed nothing but the dice. Fix for the spec: drop the network-online ordering from the automount unit — `_netdev` on the `.mount` already delays the actual mount until the network is up; the trigger needs nothing. (Interim host-side mitigation if needed before the agent fix: a systemd drop-in removing the ordering on the installed automount units.) The 4e boot-ordering proof therefore **ran and caught exactly what it was designed to catch.** ## What passed (the good news) @@ -78,18 +79,20 @@ Ledger (final): **30 PASS · 17 FAIL · 11 FINDING · 1 DISCREPANCY · 10 INFO** - One offsite cycle run (of ≤2 allowed); second budget unused. - No midnight operator push about the host (expendable demo box, Peti unaffected); this report is the notification. -## Box state at wrap +## Box state at wrap (final, 2026-07-12 06:53) -- **felhom-pve: DOWN** (since the 23:30 planned reboot; HUMAN: physical power/console needed). +- **felhom-pve: RECOVERED** — operator power-cycled 06:45; RCA = F12 ordering cycle (above), hardware exonerated. Automount manually started (boot 0's cycle-break had deleted its job), all apps healed. +- **Guest 9201: defined state reached** — 6 wave apps (radarr/jellyfin/navidrome on nas-media, seerr/actualbudget volumes, calibre-web) deployed + healthy with their data; `privatebin` (campaign temp) stop+removed via the real flow; verification backup run **success:true** (25 s); nas-media `ok`+mounted; stub badge 0; deployed-not-running: none. +- ⚠ **Until F12 is fixed, every reboot of this host is the coin flip** — next boot may come up without network again (interim: systemd drop-in, or fix the agent template). - **DooPlex/NAS: fully restored** — /etc/exports and smb.conf restored from `.bak-campaign3`, **md5 == baseline** (3 exports); campaign SMB user/share/creds/dirs removed; felhom-data export live; all customer/homelab services untouched throughout (exportfs-only rail held). - **Guest 9201 (as last seen, 23:33, pre-host-reboot):** 11 containers healthy — controller 0.117.0, calibre-web/radarr/jellyfin/navidrome (nas-media), seerr/actualbudget (volumes), privatebin (campaign temp — **removal via the real flow was pre-empted by the host loss**; remove after recovery), filebrowser + infra. Dummy media data (2.8G) + backups intact on the NAS. - **Hub:** untouched all night (reports were flowing until host loss; row will show the host offline — correct). -### Morning recovery runbook (operator) -1. Power/console felhom-pve; check why boot stalled (BIOS/GRUB/fsck). -2. After boot, **expect F10/F11**: NAS apps dead-at-boot. Heal: `systemctl reset-failed 'mnt-felhom*'`, `systemctl restart 'mnt-felhom\x2ddrives-nas\x2dmedia.automount'`, then in guest `docker start radarr jellyfin navidrome calibre-web`. -3. `POST /api/stacks/privatebin/remove` (campaign temp app) — or leave it, it's harmless. -4. Verify: no stub badge on /stacks, nas-media `ok`, hub row green, one manual `POST /api/backup/run` green. +### Morning recovery (EXECUTED 06:48–06:53) +1. ✔ Operator power-cycled; RCA established from `journalctl -b -1` (F12 ordering cycle; boot -1 sacrificed networking.service, boot 0 sacrificed the automount). +2. ✔ Automount started, NAS apps + all others healthy (11 containers). +3. ✔ privatebin stopped + removed via the real flow (`remove_hdd_data` + `remove_backups`). +4. ✔ Verified: stub 0, nas-media `ok`, deployed-not-running none, verification backup `success:true`. ## Evidence