admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

CAMPAIGN-3: morning RCA — host loss = F12 CRITICAL automount ordering cycle (hardware exonerated); box recovered to defined state

Browse Source 
journalctl proved boot -1 came up at 23:31 WITHOUT networking: the agent automount
template's After/Wants=network-online.target cycles with local-fs.target and systemd
deleted networking.service to break it; the 06:45 power-cycle boot hit the same cycle
and deleted the automount instead. Every boot with an enrolled share is a coin flip
until the template drops that ordering. Recovery executed: automount started, apps
healed, privatebin removed via real flow, verification backup green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf
This commit is contained in:
authentik Default Admin
2026-07-12 06:55:40 +02:00
parent bed9bd2c5a
commit f97f7803c7
3 changed files with 31 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CONTEXT.md
+14 -9
View File
@@ -3,16 +3,21 @@
> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
- **2026-07-12 — CAMPAIGN-3 NIGHT RUN DONE; ⚠ DEMO HOST DOWN (HUMAN).** Unattended chaos campaign
- **2026-07-12 — CAMPAIGN-3 NIGHT RUN DONE; host recovered, RCA closed.** Unattended chaos campaign
22:09→04:27 vs demo box (ctrl 0.117.0/agent 0.84.0): data plane solid (refusals/verify/rollback/
restore byte-identical/EIO instant/stub badge + deploy-409 live-validated), but **felhom-pve never
returned from the planned host reboot** (physical power/console needed; recovery runbook in the
audit) and the **NAS automount re-arm plane failed**: F10 CRITICAL (start-limit never re-armed,
once blocked guest start), F11/F9 HIGH (post-start reassert "skip-active" skips the healing
automount restart; agent-restart sweep silent) → 4 NAS apps dead-at-boot on 3/3 guest reboots;
F7 HIGH (in-place dump write → 0-byte tar replaced last good backup during outage window).
Fix-6 numbers: ring horizon 55min idle / **6.5min under load**, wiped on every restart.
Full ledger/evidence: `documentation/audits/CAMPAIGN-3-2026-07-11.md` + `180:~/campaign3/`.
restore byte-identical/EIO instant/stub badge + deploy-409 live-validated). The NAS automount
plane produced 2 CRITICAL + 3 HIGH: **F12 CRITICAL (= the overnight host loss, RCA'd next
morning): the agent automount template's `After/Wants=network-online.target` creates a boot
ordering cycle — systemd deletes an arbitrary job to break it; boot -1 sacrificed networking
(host dark 7h, hardware exonerated), boot 0 sacrificed the automount (NAS dead). Every host
boot with an enrolled share = coin flip until the template drops that ordering (`_netdev` on
the .mount suffices).** F10 CRITICAL (start-limit never re-armed, once blocked guest start),
F11/F9 HIGH (post-start reassert "skip-active" skips the healing automount restart; agent-restart
sweep silent) → 4 NAS apps dead-at-boot on 3/3 guest reboots; F7 HIGH (in-place dump write →
0-byte tar replaced last good backup during outage window). Fix-6 numbers: ring horizon 55min
idle / **6.5min under load**, wiped on every restart. Box left in defined state (6 wave apps
healthy, temps removed, NAS baseline-identical). Full ledger/evidence:
`documentation/audits/CAMPAIGN-3-2026-07-11.md` + `180:~/campaign3/`.
- **2026-07-11 — HUB v0.47.0 UI REORGANIZATION SHIPPED** (CHANGELOG v0.47.0; commits `9f29bf3` →
`0daddcd` + docs). Five deliverables: **(1)** `.data-table td a:not(.btn)` button-contrast fix;
**(2)** customer page = **8 hash tabs** (`#tab=…`, sticky summary strip, Events error badge,