docs(audits): power-outage recovery audit 2026-07-22 — power cut confirmed, full self-heal in 3m15s, dead-man's-switch fired on schedule; F8-F13

documentation/audits/AUDIT-power-outage-recovery-2026-07-22.md

@@ -0,0 +1,154 @@
+# AUDIT — Power-outage recovery, vacation site, 2026-07-22
+
+**Run:** read-only forensic + recovery audit, executed from DooPlex 2026-07-22 19:38–20:00 CEST.
+**Scope:** felhom-pve (N100, `demo-felhom-8363b5`, tailnet 100.70.170.35) and demo-hp (HP t740,
+`demo-hp-bb76ea`, 100.76.96.79), their guests 9201, the hub plane, and the backup plane.
+**Mode:** strictly read-only on all live systems — nothing was restarted, redeployed, unlocked or
+triggered. The single state-affecting action was a designed one: a read-only `ls` that woke the
+idle Felhom-Share autofs trigger (its normal first-access path).
+**Evidence:** DooPlex `~/outage-20260722/evidence/` — every claim below cites a file there.
+**Timestamps:** normalized to **CEST**. Hosts log CEST; guests, the hub **DB** and Cloudflare log
+UTC (CEST−2); the hub **pod log** prints CEST. Watch for this when re-reading raw evidence.
+**Access paths used (p0-access.txt):** felhom-pve = operator key, root@tailnet. demo-hp = key-less
+by design → hub-vaulted G1 break-glass (`host_recovery/demo-hp-bb76ea`, file→file, shredded after).
+Both worked on the first path; the OOB *belt* units however are absent — see **F9**.
+
+Related: `AUDIT-vacation-remote-ops-2026-07-20.md` (F1–F7 of this arc; F-numbers continue at F8).
+
+---
+
+## 1. Verdict on cause: simultaneous external power cut — CC session exonerated
+
+- Both previous-boot journals end **abruptly mid-routine-agent-activity** — felhom-pve at
+  **14:41:31**, demo-hp at **14:41:15** — with no `Reached target Power-Off` cascade
+  (p1-felhom-pve-journal-prevboot-tail.txt, p1-demo-hp-journal-prevboot-tail.txt).
+- `last -x` marks the previous boot **`crash`** on both hosts (p1-\*-forensic.txt).
+- The 14:00–15:00 sudo/sshd window contains **zero non-agent entries** on either host — 4 569 resp.
+  4 533 journal lines, all `felhom-agent` sudo routine; no SSH login, no `poweroff`/`shutdown`/
+  `pct`/`qm` invocation (p1-\*-forensic.txt).
+- Both boxes show the classic dirty-shutdown recovery trio on the next boot: journald
+  `system.journal corrupted or uncleanly shut down, renaming and replacing`; ESP FAT `Dirty bit is
+  set … Automatically removing`; ext4 orphan-inode cleanup (5 inodes on felhom-pve dm-7, 1 on
+  demo-hp dm-7). NTP synchronized on both after boot.
+
+**Verdict:** site-wide power cut at ~14:41–14:42 CEST hitting both boxes (and the router and the
+NAS) simultaneously. No software or operator action involved; the earlier CC session is ruled out.
+
+## 2. Timeline (CEST)
+
+| Time | Event | Evidence |
+|---|---|---|
+| ~14:27–14:29 | Last reports received by hub (host_reports 14:26:54 / 14:27:54; controller reports 14:28:50 / 14:29:13 — N100 / hp) | p5-report-gap.txt |
+| 14:40–14:41 | Last metrics rows (guest DBs): 14:40:49 N100, 14:40:12 hp | p3-\*-metrics-quickcheck.txt |
+| **14:41–14:42** | **Power cut.** Journals end 14:41:31 / 14:41:15; router reboots (operator-attested, uptime counter) — power itself returned within minutes; **both miniPCs stayed OFF** | p1-\* |
+| 14:57:09 / 14:58:09 | hub `host_stale` (N100 / hp) + operator email sent | p5-hub-log-grep.txt, p5-hub-notification-log.txt |
+| 14:59:09 / 15:00:09 | hub `node_stale` + operator email sent | ” |
+| 15:27:09 / 15:28:09 | hub `host_down` + operator email sent | ” |
+| 15:29:09 / 15:30:09 | hub `node_down` + operator email; **customer email** (doodoo21@freemail.hu) for demo-felhom | ” |
+| ~19:10–19:13 | Viktor powers both boxes on manually (attested) | — |
+| **19:13:52 / 19:13:53** | **T1: kernel boot** (N100 / hp, `uptime -s`) | p1-\*-forensic.txt |
+| 19:14:04–19:14:21 | Host plane up on both: tailscaled, wg-quick, felhom-agent 0.93.0, first host_report at hub (hp 19:14:12, N100 19:14:21) | p2-\*, p4-\*, p5-report-gap.txt |
+| 19:14:11 / 19:14:24 | PBS-over-wg verify cycle complete (hp / N100) — WireGuard path proven live | p6-\*-backup.txt |
+| 19:15:09 | hub `host_recovered` (both, same tick) | p5-hub-events.txt |
+| 19:16:23 / 19:16:30 | Guest 9201 boots (hp / N100; `onboot: 1`, PVE-started) | p3-\*-guests, p3-\*-docker-ps |
+| 19:16:33–19:16:40 | cloudflared re-registers **4/4 tunnel connections** on both guests | p4-\*-tunnels.txt |
+| 19:16:34 / 19:16:38 | First post-outage metrics row (hp / N100) | p3-\*-metrics-quickcheck.txt |
+| 19:16:40 / 19:16:44 | `controller_started` events; first controller reports at hub 19:16:41 / 19:16:44 | p5-hub-events.txt, p5-report-gap.txt |
+| **19:17:09** | hub `node_recovered` (both) — **full platform recovery, Δ ≈ 3 m 15 s from power-on** | p5-hub-events.txt |
+| 19:38–19:47 | Audit probes: tailscale both active-direct; edge-pinned HTTPS 302 on both tunnels; wg handshakes fresh; NAS share wakes + mounts on first access | p0-, p4-external-, p3-demo-hp-nas-wake.txt |
+
+Total service outage ≈ **4 h 35 m**, of which ~4 h 30 m was *boxes sitting powered off* (F8) and
+~3 m 15 s was actual recovery work.
+
+## 3. Self-healing scorecard
+
+Every component recovered **unaided**; Δ measured from T1 (19:13:52/53).
+
+| Component | N100 Δ | demo-hp Δ | Unaided | Evidence |
+|---|---|---|---|---|
+| Kernel boot + fs recovery (journal replay, orphan cleanup, ESP dirty bit) | +3 s | +5 s | yes | p1-\*-forensic.txt |
+| tailscaled (control + direct conns) | ~+21 s | ~+11 s | yes | p4-\*, p0-tailscale-status.txt |
+| wg-quick@wg-felhom (iface up) | +22 s | +12 s | yes | p4-\*-tunnels.txt |
+| felhom-agent 0.93.0 active | +26 s | +16 s | yes | p2-\*-host.txt |
+| First host_report at hub | +29 s | +19 s | yes | p5-report-gap.txt |
+| PBS reachable over wg (verify cycle OK) | +32 s | +18 s | yes | p6-\*-backup.txt |
+| Agent local API bound (192.168.0.162 / .87:8443) | +59 s | +19 s | yes | p2-\*-f1-check.txt |
+| Guest 9201 (onboot=1) | +2 m 38 s | +2 m 30 s | yes | p3-\* |
+| Docker + all stacks (12 resp. 7 containers, 0 exited, 0 unhealthy; controller surface: every deployed stack `running`) | ~+3 m | ~+3 m | yes | p3-\*-docker-ps.txt, p3-\*-stacks-parsed.txt |
+| cloudflared 4/4 connections | +2 m 48 s | +2 m 42 s | yes | p4-\*-tunnels.txt |
+| First controller report at hub | +2 m 52 s | +2 m 48 s | yes | p5-report-gap.txt |
+| Hub `*_recovered` events | +1 m 17 s (host) / +3 m 17 s (node) | same ticks | yes | p5-hub-events.txt |
+| Felhom-Share NAS (demo-hp): idle autofs trigger restored at boot; first access mounts CIFS `//192.168.0.104/Share`; FileBrowser bind live; content intact | on first access | — | yes | p3-demo-hp-nas-mounts.txt, p3-demo-hp-nas-wake.txt |
+| External URLs via pinned Cloudflare edge: 302 in 0.14–0.16 s | ✔ at audit | ✔ | yes | p4-external-edge-probes.txt |
+
+**F1 regression check (load-bearing): F1 did NOT recur.** felhom-pve came back on its static
+192.168.0.162; demo-hp re-acquired its previous DHCP 192.168.0.87 despite the router reboot; agent
+local-API bound cleanly on both, zero bind errors in the journals (p2-\*-f1-check.txt). The static-IP
+mitigation from AUDIT-vacation-remote-ops F1 held under its natural trigger.
+
+**Dead-man's-switch verdict: fired exactly on design schedule.** The runbook's ~15:12/~15:42
+expectation measured from the outage instant; the design (30 m stale / 60 m down,
+`hub-config` `stale_threshold: "30m"`, `staleness.go` `downAfter = 2×`) measures from the **last
+received report** (~14:27–14:29). Observed firings are correct to the minute (checker ticks at
+:09). All 9 dispatches logged `status=sent`, zero errors: 8 operator + 1 customer
+(demo-felhom/node_down — the only customer with notification prefs, see F12).
+`*_recovered` events carry severity `info`, which the dispatcher intentionally does not email
+(dispatcher.go: "info is an intentional non-notify (status/recovery events)") — see F11.
+
+**Data-integrity scars:** confined to what the OS self-healed (§1). `PRAGMA quick_check` = `ok` on
+both metrics DBs (checked on copies); metrics gap cleanly bounded 14:40 → 19:16. Offbox restic
+repo consistent, 14 snapshots, latest 04:16 CEST (pre-outage), **zero locks** (read-only
+`restic list locks`). No backup job was running at the cut — guest backup windows are nightly
+(tier2 03:30, offbox 04:15 CEST), no window fell inside 14:42–19:15, catch-up is automatic tonight
+(p6-\*).
+
+## 4. Findings (continuing the vacation arc; F1–F7 in AUDIT-vacation-remote-ops-2026-07-20.md)
+
+| # | Finding | Severity | Evidence | Disposition |
+|---|---|---|---|---|
+| **F8** | **Site power loss; no auto-power-on.** Power itself returned within minutes (router rebooted at ~14:42 and stayed up); both miniPCs remained off ~4.5 h until manual power-on. For a paying customer this converts a power blip into a half-day outage ending only when someone is physically present. | HIGH | p1-\*; router uptime operator-attested | **mitigated-on-site** — Viktor set BIOS AC-power-on on both boxes (attested; not OS-verifiable). **roadmap-candidate:** make BIOS "restore on AC power" a provisioning-checklist item + a host-install doc requirement for every fleet box. |
+| **F9** | **H1 OOB belt only partially installed on the current fleet.** `felhom-mgmt-watchdog.timer` is live on both boxes (heal marker absent = no heal needed), but `felhom-sshd.service` and `felhom-oob-nft.service` (TASK H1, present in `felhom-agent/configs/`) are installed on **neither** box; operator access rides stock sshd :22 + tailscale + G1 break-glass. Pre-existing (both boxes provisioned via the universal ISO), surfaced by this audit's access preflight. | MEDIUM | p2-felhom-pve-ssh-belt.txt, p2-\*-host.txt | **needs-ruling** — was H1 intentionally dropped from the universal-ISO provisioning path, or should host-install grow the belt? |
+| **F10** | **demo-hp backup tiers incomplete.** Tier-2 dump tree exists (paperless-ngx, fresh), but **no offbox target is configured** and the PBS DR datastore holds **0 snapshots** for it. A power event with disk damage would have had no off-box recovery path for that guest. Pre-existing (box added 07-21), not outage-caused. | MEDIUM | p6-demo-hp-offbox-locks.txt (`NO-OFFBOX`), p6-demo-hp-backup.txt (`snapshots=0`), p6-demo-hp-tier2.txt | **roadmap-candidate** — finish demo-hp tier provisioning (offbox + DR) as part of the pilot hardening pass. |
+| **F11** | **Recovery is silent.** `host_recovered`/`node_recovered` fired correctly but carry severity `info`, which the dispatcher deliberately does not email — the operator/customer only learns of recovery by looking. During a real customer outage the "it's back" signal is arguably the second-most valuable email. Working-as-coded, so a product question, not a defect. | LOW | p5-hub-events.txt; `hub/internal/notify/dispatcher.go` severityNotifies | **needs-ruling** — opt-in recovery notifications (operator at least)? |
+| **F12** | **demo-hp has no customer notification prefs** (`customer_notifications` has no row for it) → its "customer" received no node_down email and never would. Only demo-felhom is wired (doodoo21@freemail.hu). Pre-existing demo-box config gap; on a real onboarding this must not be skippable. | LOW | p5-customer-notification-prefs.txt | **roadmap-candidate** — make notification-prefs setup a claim/onboarding step, not an optional settings page. |
+| **F13** | **PBS DR tier has no visible backup cadence.** `/etc/pve/jobs.cfg` is empty on both hosts; N100's DR datastore holds a single CT-9201 snapshot from **2026-07-18** (S8-era), demo-hp none. The agent's 6 h *verify* loop runs, but nothing appears to *create* periodic DR snapshots. If DR snapshots are meant to be on-demand only, fine — but then the "DR tier" freshness expectation should be documented; if they're meant to be periodic, the scheduler is missing. | LOW | p6-\*-backup.txt | **needs-ruling** — clarify intended DR-tier cadence; roadmap if periodic. |
+
+## 5. Explicitly NOT validated (would have required mutation, or is operator-side)
+
+- **BIOS auto-power-on**: not OS-verifiable; recorded as operator-attested. Its first real proof
+  will be the next power event (or a deliberate drill — candidate for R-55's reboot leg).
+- **Email delivery to the Gmail inbox**: the hub logged all 9 sends as `sent` (Resend accepted);
+  whether they **landed** at ~14:57–15:30 CEST is Viktor-side (inbox + Resend dashboard).
+  **Question for Viktor:** did the 8 operator mails + 1 customer mail arrive? If not, that is a
+  *delivery* finding (dispatcher is proven good).
+- **Cloudflare dashboard tunnel state**: operator-attested; this audit proved the stronger fact
+  (edge-pinned 302 end-to-end on both live tunnels).
+- **demo-hp offbox restic**: nothing to check — no offbox exists (F10).
+- **UI click-throughs**: no browser on DooPlex; controller state was validated on its own API
+  surface (endpoint-level, per standing convention).
+
+## 6. Observations (out of scope, not acted on)
+
+- **`demo-vm` tunnel will not recover — and that is correct.** Its owner (the nested appliance) no
+  longer exists: the three ISO-train drill VMs 9310/9311/9312 were destroyed cleanly this morning
+  11:01–11:43 CEST, *before* the outage (p3-felhom-pve-vm-absence.txt). The tunnel's Down state is
+  permanent until the dead hub/Cloudflare records are discarded (already pending as operator
+  cleanup from the ISO train; hub-side ghost-delete = R-62 arc).
+- **peti-felhom is silent since 2026-07-15 08:39 UTC** (last report in the hub DB; hub seeded it
+  `down` at its 09:25 restart, so no fresh event fired). Matches the petifelhom tunnel Down in
+  Viktor's screenshot. His site, his box — noted for the planned Friday reinstall, nothing touched.
+- **Controllers restarted 14:13–14:14 CEST** (`controller_started` events 1701/1702) — that was the
+  0.160.0 publish train, 28 minutes before the cut; unrelated to the outage, but a reminder that
+  the fleet had been on 0.160.0 for less than half an hour when it took its first dirty shutdown —
+  and came back clean.
+- Hub pod log prints CEST while the hub DB stores UTC; this audit normalized everything to CEST.
+
+## 7. Bottom line
+
+The platform passed its first real, unplanned, site-wide dirty-shutdown drill. Every layer —
+filesystems, host plane, WireGuard, tailscale, guests, Docker stacks, cloudflared, NAS automount,
+hub staleness detection, notifications, backup integrity — recovered without a single manual
+intervention, in ~3 m 15 s of actual work after power returned. The entire 4.5 h of customer-visible
+downtime was attributable to one thing: nobody could press the power button (F8, now mitigated).
+The dead-man's-switch pipeline fired on schedule and the emails dispatched; the open questions are
+delivery-side (inbox) and product-side (silent recovery, F11).