felhom.eu/documentation/audits/AUDIT-power-outage-recovery-2026-07-22.md

AUDIT — Power-outage recovery, vacation site, 2026-07-22

Run: read-only forensic + recovery audit, executed from DooPlex 2026-07-22 19:38–20:00 CEST. Scope: felhom-pve (N100, demo-felhom-8363b5, tailnet 100.70.170.35) and demo-hp (HP t740, demo-hp-bb76ea, 100.76.96.79), their guests 9201, the hub plane, and the backup plane. Mode: strictly read-only on all live systems — nothing was restarted, redeployed, unlocked or triggered. The single state-affecting action was a designed one: a read-only ls that woke the idle Felhom-Share autofs trigger (its normal first-access path). Evidence: DooPlex ~/outage-20260722/evidence/ — every claim below cites a file there. Timestamps: normalized to CEST. Hosts log CEST; guests, the hub DB and Cloudflare log UTC (CEST−2); the hub pod log prints CEST. Watch for this when re-reading raw evidence. Access paths used (p0-access.txt): felhom-pve = operator key, root@tailnet. demo-hp = key-less by design → hub-vaulted G1 break-glass (host_recovery/demo-hp-bb76ea, file→file, shredded after). Both worked on the first path; the OOB belt units however are absent — see F9.

Related: AUDIT-vacation-remote-ops-2026-07-20.md (F1–F7 of this arc; F-numbers continue at F8).

---

1. Verdict on cause: simultaneous external power cut — CC session exonerated

• Both previous-boot journals end abruptly mid-routine-agent-activity — felhom-pve at 14:41:31, demo-hp at 14:41:15 — with no Reached target Power-Off cascade (p1-felhom-pve-journal-prevboot-tail.txt, p1-demo-hp-journal-prevboot-tail.txt).
• last -x marks the previous boot crash on both hosts (p1-*-forensic.txt).
• The 14:00–15:00 sudo/sshd window contains zero non-agent entries on either host — 4 569 resp. 4 533 journal lines, all felhom-agent sudo routine; no SSH login, no poweroff/shutdown/ pct/qm invocation (p1-*-forensic.txt).
• Both boxes show the classic dirty-shutdown recovery trio on the next boot: journald system.journal corrupted or uncleanly shut down, renaming and replacing; ESP FAT Dirty bit is set … Automatically removing; ext4 orphan-inode cleanup (5 inodes on felhom-pve dm-7, 1 on demo-hp dm-7). NTP synchronized on both after boot.

Verdict: site-wide power cut at ~14:41–14:42 CEST hitting both boxes (and the router and the NAS) simultaneously. No software or operator action involved; the earlier CC session is ruled out.

2. Timeline (CEST)

Time	Event	Evidence
~14:27–14:29	Last reports received by hub (host_reports 14:26:54 / 14:27:54; controller reports 14:28:50 / 14:29:13 — N100 / hp)	p5-report-gap.txt
14:40–14:41	Last metrics rows (guest DBs): 14:40:49 N100, 14:40:12 hp	p3-*-metrics-quickcheck.txt
14:41–14:42	Power cut. Journals end 14:41:31 / 14:41:15; router reboots (operator-attested, uptime counter) — power itself returned within minutes; both miniPCs stayed OFF	p1-*
14:57:09 / 14:58:09	hub host_stale (N100 / hp) + operator email sent	p5-hub-log-grep.txt, p5-hub-notification-log.txt
14:59:09 / 15:00:09	hub node_stale + operator email sent	”
15:27:09 / 15:28:09	hub host_down + operator email sent	”
15:29:09 / 15:30:09	hub node_down + operator email; customer email (doodoo21@freemail.hu) for demo-felhom	”
~19:10–19:13	Viktor powers both boxes on manually (attested)	—
19:13:52 / 19:13:53	T1: kernel boot (N100 / hp, uptime -s)	p1-*-forensic.txt
19:14:04–19:14:21	Host plane up on both: tailscaled, wg-quick, felhom-agent 0.93.0, first host_report at hub (hp 19:14:12, N100 19:14:21)	p2-*, p4-*, p5-report-gap.txt
19:14:11 / 19:14:24	PBS-over-wg verify cycle complete (hp / N100) — WireGuard path proven live	p6-*-backup.txt
19:15:09	hub host_recovered (both, same tick)	p5-hub-events.txt
19:16:23 / 19:16:30	Guest 9201 boots (hp / N100; onboot: 1, PVE-started)	p3-*-guests, p3-*-docker-ps
19:16:33–19:16:40	cloudflared re-registers 4/4 tunnel connections on both guests	p4-*-tunnels.txt
19:16:34 / 19:16:38	First post-outage metrics row (hp / N100)	p3-*-metrics-quickcheck.txt
19:16:40 / 19:16:44	controller_started events; first controller reports at hub 19:16:41 / 19:16:44	p5-hub-events.txt, p5-report-gap.txt
19:17:09	hub node_recovered (both) — full platform recovery, Δ ≈ 3 m 15 s from power-on	p5-hub-events.txt
19:38–19:47	Audit probes: tailscale both active-direct; edge-pinned HTTPS 302 on both tunnels; wg handshakes fresh; NAS share wakes + mounts on first access	p0-, p4-external-, p3-demo-hp-nas-wake.txt

Total service outage ≈ 4 h 35 m, of which ~4 h 30 m was boxes sitting powered off (F8) and ~3 m 15 s was actual recovery work.

3. Self-healing scorecard

Every component recovered unaided; Δ measured from T1 (19:13:52/53).

Component	N100 Δ	demo-hp Δ	Unaided	Evidence
Kernel boot + fs recovery (journal replay, orphan cleanup, ESP dirty bit)	+3 s	+5 s	yes	p1-*-forensic.txt
tailscaled (control + direct conns)	~+21 s	~+11 s	yes	p4-*, p0-tailscale-status.txt
wg-quick@wg-felhom (iface up)	+22 s	+12 s	yes	p4-*-tunnels.txt
felhom-agent 0.93.0 active	+26 s	+16 s	yes	p2-*-host.txt
First host_report at hub	+29 s	+19 s	yes	p5-report-gap.txt
PBS reachable over wg (verify cycle OK)	+32 s	+18 s	yes	p6-*-backup.txt
Agent local API bound (192.168.0.162 / .87:8443)	+59 s	+19 s	yes	p2-*-f1-check.txt
Guest 9201 (onboot=1)	+2 m 38 s	+2 m 30 s	yes	p3-*
Docker + all stacks (12 resp. 7 containers, 0 exited, 0 unhealthy; controller surface: every deployed stack running)	~+3 m	~+3 m	yes	p3-*-docker-ps.txt, p3-*-stacks-parsed.txt
cloudflared 4/4 connections	+2 m 48 s	+2 m 42 s	yes	p4-*-tunnels.txt
First controller report at hub	+2 m 52 s	+2 m 48 s	yes	p5-report-gap.txt
Hub *_recovered events	+1 m 17 s (host) / +3 m 17 s (node)	same ticks	yes	p5-hub-events.txt
Felhom-Share NAS (demo-hp): idle autofs trigger restored at boot; first access mounts CIFS //192.168.0.104/Share; FileBrowser bind live; content intact	on first access	—	yes	p3-demo-hp-nas-mounts.txt, p3-demo-hp-nas-wake.txt
External URLs via pinned Cloudflare edge: 302 in 0.14–0.16 s	✔ at audit	✔	yes	p4-external-edge-probes.txt

F1 regression check (load-bearing): F1 did NOT recur. felhom-pve came back on its static 192.168.0.162; demo-hp re-acquired its previous DHCP 192.168.0.87 despite the router reboot; agent local-API bound cleanly on both, zero bind errors in the journals (p2-*-f1-check.txt). The static-IP mitigation from AUDIT-vacation-remote-ops F1 held under its natural trigger.

Dead-man's-switch verdict: fired exactly on design schedule. The runbook's ~15:12/~15:42 expectation measured from the outage instant; the design (30 m stale / 60 m down, hub-config stale_threshold: "30m", staleness.go downAfter = 2×) measures from the last received report (~14:27–14:29). Observed firings are correct to the minute (checker ticks at :09). All 9 dispatches logged status=sent, zero errors: 8 operator + 1 customer (demo-felhom/node_down — the only customer with notification prefs, see F12). *_recovered events carry severity info, which the dispatcher intentionally does not email (dispatcher.go: "info is an intentional non-notify (status/recovery events)") — see F11.

Data-integrity scars: confined to what the OS self-healed (§1). PRAGMA quick_check = ok on both metrics DBs (checked on copies); metrics gap cleanly bounded 14:40 → 19:16. Offbox restic repo consistent, 14 snapshots, latest 04:16 CEST (pre-outage), zero locks (read-only restic list locks). No backup job was running at the cut — guest backup windows are nightly (tier2 03:30, offbox 04:15 CEST), no window fell inside 14:42–19:15, catch-up is automatic tonight (p6-*).

4. Findings (continuing the vacation arc; F1–F7 in AUDIT-vacation-remote-ops-2026-07-20.md)

#	Finding	Severity	Evidence	Disposition
F8	Site power loss; no auto-power-on. Power itself returned within minutes (router rebooted at ~14:42 and stayed up); both miniPCs remained off ~4.5 h until manual power-on. For a paying customer this converts a power blip into a half-day outage ending only when someone is physically present.	HIGH	p1-*; router uptime operator-attested	mitigated-on-site — Viktor set BIOS AC-power-on on both boxes (attested; not OS-verifiable). roadmap-candidate: make BIOS "restore on AC power" a provisioning-checklist item + a host-install doc requirement for every fleet box.
F9	H1 OOB belt only partially installed on the current fleet. felhom-mgmt-watchdog.timer is live on both boxes (heal marker absent = no heal needed), but felhom-sshd.service and felhom-oob-nft.service (TASK H1, present in felhom-agent/configs/) are installed on neither box; operator access rides stock sshd :22 + tailscale + G1 break-glass. Pre-existing (both boxes provisioned via the universal ISO), surfaced by this audit's access preflight.	MEDIUM	p2-felhom-pve-ssh-belt.txt, p2-*-host.txt	RESOLVED 2026-07-23 (ISO train v1.25.0). Ruling: install everywhere. host-install grows the belt as a DEFAULT appliance leg (--no-oob opts out; byo still refuses — deliberate). Phase-0 confirmed the omission was NOT a coded exclusion, just --enable-oob never passed by the universal ISO. Belt installed + oob.enabled on BOTH live boxes; login PROVEN end-to-end on felhom-pve (felhom-op@demo-felhom over wg-felhom → belt); also re-anchored the orphaned operator identity to the operator's real machine. See REPORT.md (2026-07-23) + operations/nodes.md.
F10	demo-hp backup tiers incomplete. Tier-2 dump tree exists (paperless-ngx, fresh), but no offbox target is configured and the PBS DR datastore holds 0 snapshots for it. A power event with disk damage would have had no off-box recovery path for that guest. Pre-existing (box added 07-21), not outage-caused.	MEDIUM	p6-demo-hp-offbox-locks.txt (NO-OFFBOX), p6-demo-hp-backup.txt (snapshots=0), p6-demo-hp-tier2.txt	OFFSITE LEG RESOLVED 2026-07-23 — root cause was NOT "provisioning unfinished": the day-0 managed update killed the apply-bridge after password-consume, burning the one-shot credential (full diagnosis, designed-path repair via operator Re-issue, escrow ceremony, and a byte-identical offsite restore round-trip: DIAG-f10-demo-hp-offsite-2026-07-23.md; product rows minted R-70 visibility + R-71 the race). The PBS-DR-snapshot half STAYS OPEN pending F13 (cadence ruling) and the deliberate DR ceremony R-moment on demo-hp.
F11	Recovery is silent. host_recovered/node_recovered fired correctly but carry severity info, which the dispatcher deliberately does not email — the operator/customer only learns of recovery by looking. During a real customer outage the "it's back" signal is arguably the second-most valuable email. Working-as-coded, so a product question, not a defect.	LOW	p5-hub-events.txt; hub/internal/notify/dispatcher.go severityNotifies	needs-ruling — opt-in recovery notifications (operator at least)?
F12	demo-hp has no customer notification prefs (customer_notifications has no row for it) → its "customer" received no node_down email and never would. Only demo-felhom is wired (doodoo21@freemail.hu). Pre-existing demo-box config gap; on a real onboarding this must not be skippable.	LOW	p5-customer-notification-prefs.txt	roadmap-candidate — make notification-prefs setup a claim/onboarding step, not an optional settings page.
F13	PBS DR tier has no visible backup cadence. /etc/pve/jobs.cfg is empty on both hosts; N100's DR datastore holds a single CT-9201 snapshot from 2026-07-18 (S8-era), demo-hp none. The agent's 6 h verify loop runs, but nothing appears to create periodic DR snapshots. If DR snapshots are meant to be on-demand only, fine — but then the "DR tier" freshness expectation should be documented; if they're meant to be periodic, the scheduler is missing.	LOW	p6-*-backup.txt	needs-ruling — clarify intended DR-tier cadence; roadmap if periodic.

5. Explicitly NOT validated (would have required mutation, or is operator-side)

• BIOS auto-power-on: not OS-verifiable; recorded as operator-attested. Its first real proof will be the next power event (or a deliberate drill — candidate for R-55's reboot leg).
• Email delivery to the Gmail inbox: the hub logged all 9 sends as sent (Resend accepted); whether they landed at ~14:57–15:30 CEST is Viktor-side (inbox + Resend dashboard). Question for Viktor: did the 8 operator mails + 1 customer mail arrive? If not, that is a delivery finding (dispatcher is proven good).
• Cloudflare dashboard tunnel state: operator-attested; this audit proved the stronger fact (edge-pinned 302 end-to-end on both live tunnels).
• demo-hp offbox restic: nothing to check — no offbox exists (F10).
• UI click-throughs: no browser on DooPlex; controller state was validated on its own API surface (endpoint-level, per standing convention).

6. Observations (out of scope, not acted on)

• demo-vm tunnel will not recover — and that is correct. Its owner (the nested appliance) no longer exists: the three ISO-train drill VMs 9310/9311/9312 were destroyed cleanly this morning 11:01–11:43 CEST, before the outage (p3-felhom-pve-vm-absence.txt). The tunnel's Down state is permanent until the dead hub/Cloudflare records are discarded (already pending as operator cleanup from the ISO train; hub-side ghost-delete = R-62 arc).
• peti-felhom is silent since 2026-07-15 08:39 UTC (last report in the hub DB; hub seeded it down at its 09:25 restart, so no fresh event fired). Matches the petifelhom tunnel Down in Viktor's screenshot. His site, his box — noted for the planned Friday reinstall, nothing touched.
• Controllers restarted 14:13–14:14 CEST (controller_started events 1701/1702) — that was the 0.160.0 publish train, 28 minutes before the cut; unrelated to the outage, but a reminder that the fleet had been on 0.160.0 for less than half an hour when it took its first dirty shutdown — and came back clean.
• Hub pod log prints CEST while the hub DB stores UTC; this audit normalized everything to CEST.

7. Bottom line

The platform passed its first real, unplanned, site-wide dirty-shutdown drill. Every layer — filesystems, host plane, WireGuard, tailscale, guests, Docker stacks, cloudflared, NAS automount, hub staleness detection, notifications, backup integrity — recovered without a single manual intervention, in ~3 m 15 s of actual work after power returned. The entire 4.5 h of customer-visible downtime was attributable to one thing: nobody could press the power button (F8, now mitigated). The dead-man's-switch pipeline fired on schedule and the emails dispatched; the open questions are delivery-side (inbox) and product-side (silent recovery, F11).