docs(audits): power-outage recovery audit 2026-07-22 — power cut confirmed, full self-heal in 3m15s, dead-man's-switch fired on schedule; F8-F13

REPORT.md

@@ -1,102 +1,53 @@
-# REPORT — scripts/ISO train v1.24.0: R-59 + R-60 + R-61 (2026-07-22)
+# REPORT — Power-outage recovery audit, vacation site (2026-07-22)
 
-**Task:** the three HP-t740 findings — the box that can never call home (R-59 refuse-loudly gate),
+**Task:** RUNBOOK-driven, strictly read-only forensic + recovery audit after the 14:41–42 CEST
-the first-boot NIC sweep self-heal (R-60), and the knowable baked root password (R-61 slice 1) —
+site-wide power cut at the vacation site (felhom-pve N100 + demo-hp t740, ~4.5 h down, manual
-implemented in `scripts/iso/`, spike-proven, and driven end-to-end on a nested wrong-NIC install.
+power-on ~19:13). No code, no fixes, no restarts — diagnosis only. Deliverable:
-No hub/agent/controller/catalog changes; golden image and floors untouched.
+`documentation/audits/AUDIT-power-outage-recovery-2026-07-22.md`; raw evidence on DooPlex
+`~/outage-20260722/evidence/`.
 
-## Baselines
+## Verdict
 
-- felhom.eu `main` at start: `1613bc7` (scripts CHANGELOG head = build-hub.sh v1.23.0); clean-tree
+- **Cause:** simultaneous external power cut on both boxes — both prev-boot journals end abruptly
-  gate held before both ISO builds.
+  mid-agent-routine, `last -x` says `crash`, zero non-agent sudo/sshd entries in the window. The
-- Target: scripts train **v1.24.0** (`ISO_VERSION="1.24.0"` in `build-felhom-iso.sh`).
+  earlier CC session is exonerated.
+- **Recovery:** every layer self-healed unaided in **~3 m 15 s** after power-on — fs journal
+  replay, agent 0.93.0 (+26 s), wg + PBS-over-wg (+32 s), tailscale, guest 9201 (onboot=1,
+  +2 m 38 s), all stacks healthy on the controllers' own surfaces, cloudflared 4/4 (+2 m 48 s),
+  hub `*_recovered` (+3 m 17 s). Edge-pinned public probes 302. Felhom-Share autofs woke + CIFS
+  mounted on first access (first unplanned re-run of the R-67 dead-NAS proof — passed).
+- **Dead-man's-switch live fire:** host/node stale + down fired **exactly on the 30 m/60 m design
+  schedule** (measured from last received report, not from the outage instant), all 9 notification
+  dispatches `sent` (8 operator + 1 customer). Recovery events are severity `info` → intentionally
+  no email (flagged as product question F11).
+- **F1 regression:** did NOT recur — static .162 held, demo-hp re-acquired .87, local API bound
+  cleanly on both.
+- **Scars:** only OS-self-healed ones (journald rotate, ESP dirty bit, orphan inodes). Metrics DBs
+  `quick_check` ok, gap 14:40→19:16. Offbox restic: 14 snapshots, latest pre-outage 04:16, zero
+  stale locks. No backup window fell inside the outage; catch-up is tonight 03:30/04:15.
 
-## Commits (this train, in order)
+## Findings (F8+ continuing the vacation arc)
 
-| Hash | What |
+- **F8 (HIGH, mitigated-on-site):** no auto-power-on — power was back in minutes, boxes sat off
-|---|---|
+  4.5 h. BIOS AC-power-on now set (attested); roadmap-candidate: provisioning-checklist item.
-| `9e29295` | R-61 slice 1: rootpw sibling file (0600) + manifest pointer; `test/rootpw-emission.sh`; OUT_ISO hoist; `FELHOM_ISO_KEEP_WORK` escape |
+- **F9 (MEDIUM, needs-ruling):** H1 OOB belt (`felhom-sshd` + `felhom-oob-nft`) installed on
-| `ad1d26a` | Harness catches up with the v1.21.0 in-script pairing wait (old scenarios would hang); assistant image + python3 |
+  NEITHER fleet box — only the mgmt-watchdog timer; access is stock sshd :22 + break-glass.
-| `c043ba9` | Phase-0 spike doc — sweep mechanics GO, F-P1..F-P8 (STOP-0 discharged before gate code) |
+- **F10 (MEDIUM, roadmap):** demo-hp has tier-2 only — no offbox target, 0 PBS DR snapshots.
-| `cf1862c` | R-59+R-60: `network_gate` + `sweep_nics` + Hungarian console screen; 4 harness scenarios; 3 red-proofs run |
+- **F11 (LOW, needs-ruling):** recovery is silent (`info` severity never emails).
-| `699325b` | R-60 drill fix (F-P9): flush vmbr0 before the bounded dhclient — the baked fallback default route otherwise poisons the hub probe |
+- **F12 (LOW, roadmap):** demo-hp has no customer notification prefs row — node_down customer
-| (this commit) | v1.24.0 CHANGELOG, README, ROADMAP (R-59/60/61 SHIPPED + new R-62), onboarding runbook, capability map, spike-doc drill section + screen capture, this REPORT |
+  email impossible there.
+- **F13 (LOW, needs-ruling):** PBS DR tier shows no snapshot cadence (jobs.cfg empty; single
+  07-18 snapshot on N100, none on demo-hp) — intended?
 
-## Harness results
+## Observations
 
-- `test/bootstrap-modes.sh` (in `felhom-iso-assistant:trixie`): **40/40 green** — D (direct,
+demo-vm tunnel Down is permanent-correct (nested drill VMs 9310–9312 destroyed pre-outage this
-  zero appliance calls, G1 zero-new-behavior invariant incl. zero sleeps + fixture untouched),
+morning; record discard pending). peti-felhom silent since 07-15 08:39 UTC (his site, untouched).
-  P (v1.21.0 single-invocation pairing loop), 410 crash-window, G2 sweep-success
+Controllers had been on 0.160.0 for only ~28 min when the cut hit — first dirty shutdown on the
-  (re-point + persist `inet dhcp` + `.felhom-bak` + proceed), G3 sweep-fail (screen content +
+new version came back clean.
-  **byte-identical** interfaces + no bak), G4 state.json-present (sweep **never** invoked by
-  fake call-count, interfaces untouched).
-- `test/rootpw-emission.sh`: **11/11 green** — dry-run emission, 0600, plaintext↔answer-hash
-  cross-check (`openssl passwd -6 -salt`), plaintext absent from build output, manifest-heredoc
-  pointer/no-leak guard.
-- `shellcheck -S warning felhom-bootstrap.sh`: clean.
 
-## Red-proofs (all four run, failed exactly on their guarding assertion, restored)
+**Open question for Viktor:** did the ~14:57–15:30 CEST alert emails actually land in the inbox?
+Hub-side they are all `sent`; delivery is inbox/Resend-dashboard-side.
 
-1. **Manifest plaintext leak** (R-61): `root-password-plain : ${ROOT_PLAIN}` injected into the
+**Not validated (needs-mutation or operator-side):** BIOS setting itself, inbox delivery,
-   manifest heredoc → `manifest never references ROOT_PLAIN` FAILED → restored.
+Cloudflare dashboard state, UI click-throughs.
-2. **Restore-on-failure dropped** (R-60): sweep's byte-identical restore removed → G3
-   `interfaces BYTE-IDENTICAL after failed sweep` FAILED → restored.
-3. **State gate removed** (R-60): `[[ ! -e $STATE_FILE ]]` → `true` → G4 zero-ifreload/zero-
-   dhclient FAILED → restored.
-4. **Hub-reachable early return removed** (R-59): all five G1 invariants FAILED → restored.
-
-## Phase-0 spike (STOP-0 honored)
-
-`documentation/audits/SPIKE-firstboot-nic-sweep-2026-07-22.md` — written and committed BEFORE the
-gate code. Verdict GO; key inputs: success leg ≈ 4 s, dead candidate = exactly its DHCP bound;
-unused NICs are admin-DOWN (raise before carrier read); ANY HTTP status = reachable (hub answers
-302); trigger on hub-unreachable, never the fallback signature (the drill found the second
-wrong-NIC variant: the installer picks the NIC that LEASES and bakes the lease as static).
-
-## Nested end-to-end drill (evidence in the spike doc)
-
-- **Round 1** caught a real defect the harness could not: the fallback bake includes a **default
-  route via 192.168.100.1**, and dhclient-script never replaces an existing default route → the
-  sweep's hub probe rode the dead gateway and the gate oscillated. Fixed (`699325b`, flush before
-  dhclient), validated by hot-patch (heal in 33 s), then the ISO was rebuilt.
-- **Round 2, virgin final ISO, zero patching:** dead-NIC install → fallback baked → **R-59 screen
-  on the console** (capture committed: `SPIKE-firstboot-nic-sweep-2026-07-22-screen.png`) →
-  failed-sweep restore proven safe (flushed static re-applied by ifreload) → cable move →
-  **self-heal + hub registration in 23 s unaided** (persisted `inet dhcp` + winner port, original
-  in `interfaces.felhom-bak`, box at a proper DHCP address, pairing banner + code on console).
-- **R-61 lockout closed, proven:** the build's `.rootpw.txt` password recomputed against the
-  installed box's `/etc/shadow` root hash → exact match (computed on the build host; the
-  plaintext never left it). Value NOT included here by design.
-- Validation method note: no browser on DooPlex — console proof is QEMU screendump; hub-side
-  proof is the box's own registration/pairing-code delivery (endpoint-level).
-
-## Artifacts
-
-- Final drill ISO: `felhom-pve-9.2-1-v1.24.0-nested-probe-generic.iso`, sha256
-  `c25cdbbf35d2134b0bf278c4b069a3d5849c0960cf0a5e4500e52638489c8947`, built from clean tree at
-  `699325b` with an **uncommitted** key-bearing probe profile (nested-vm + ops SSH key — never
-  committed, per the armed-profile rule). Its `.rootpw.txt` exists beside it (0600) — verified,
-  value stored out-of-band only.
-- Shipping rebuilds for real profiles (hp/n100/generic) happen at next need; the committed
-  pipeline builds them from `main` as usual.
-
-## Cleanup verification
-
-Nested VMs 9310/9311/9312 destroyed; throwaway `vmbr9` (+ its temporary host address) deleted;
-transferred ISO + screendumps removed from felhom-pve; `brctl show` back to exactly `enp1s0` +
-`veth9201i0` on vmbr0; **`/etc/network/interfaces` on felhom-pve diffed BYTE-IDENTICAL against
-the pre-task copy** (sha `c7ace087…`). Residue by design: **three unclaimed-appliance
-registrations at the production hub** (all three source machines destroyed) — operator may
-discard them from the Unclaimed list.
-
-## Observations / follow-ups
-
-- **F-P8 (out of scope, noted):** the pairing loop's register-failure line prints every ~42 s on
-  a no-network box; v1.21.0 quieted the 204 wait but not this path. The gate makes it mostly moot
-  (it holds before register), but the line exists for a box that loses network AFTER install
-  start (state.json present → screen + retry, register never reached).
-- The console font renders Hungarian long vowels (ő/ű) approximately on the text console —
-  same behavior as the existing pairing banner; acceptable.
-- `qm` screendump (`qm monitor` → `screendump`) is a serviceable no-browser console-capture
-  method for nested drills — worth remembering for future runbooks.
-- R-62 (hub delete-dialog cosmetics) filed on the ROADMAP as agreed — docs-only here.

documentation/audits/AUDIT-power-outage-recovery-2026-07-22.md

@@ -0,0 +1,154 @@
+# AUDIT — Power-outage recovery, vacation site, 2026-07-22
+
+**Run:** read-only forensic + recovery audit, executed from DooPlex 2026-07-22 19:38–20:00 CEST.
+**Scope:** felhom-pve (N100, `demo-felhom-8363b5`, tailnet 100.70.170.35) and demo-hp (HP t740,
+`demo-hp-bb76ea`, 100.76.96.79), their guests 9201, the hub plane, and the backup plane.
+**Mode:** strictly read-only on all live systems — nothing was restarted, redeployed, unlocked or
+triggered. The single state-affecting action was a designed one: a read-only `ls` that woke the
+idle Felhom-Share autofs trigger (its normal first-access path).
+**Evidence:** DooPlex `~/outage-20260722/evidence/` — every claim below cites a file there.
+**Timestamps:** normalized to **CEST**. Hosts log CEST; guests, the hub **DB** and Cloudflare log
+UTC (CEST−2); the hub **pod log** prints CEST. Watch for this when re-reading raw evidence.
+**Access paths used (p0-access.txt):** felhom-pve = operator key, root@tailnet. demo-hp = key-less
+by design → hub-vaulted G1 break-glass (`host_recovery/demo-hp-bb76ea`, file→file, shredded after).
+Both worked on the first path; the OOB *belt* units however are absent — see **F9**.
+
+Related: `AUDIT-vacation-remote-ops-2026-07-20.md` (F1–F7 of this arc; F-numbers continue at F8).
+
+---
+
+## 1. Verdict on cause: simultaneous external power cut — CC session exonerated
+
+- Both previous-boot journals end **abruptly mid-routine-agent-activity** — felhom-pve at
+  **14:41:31**, demo-hp at **14:41:15** — with no `Reached target Power-Off` cascade
+  (p1-felhom-pve-journal-prevboot-tail.txt, p1-demo-hp-journal-prevboot-tail.txt).
+- `last -x` marks the previous boot **`crash`** on both hosts (p1-\*-forensic.txt).
+- The 14:00–15:00 sudo/sshd window contains **zero non-agent entries** on either host — 4 569 resp.
+  4 533 journal lines, all `felhom-agent` sudo routine; no SSH login, no `poweroff`/`shutdown`/
+  `pct`/`qm` invocation (p1-\*-forensic.txt).
+- Both boxes show the classic dirty-shutdown recovery trio on the next boot: journald
+  `system.journal corrupted or uncleanly shut down, renaming and replacing`; ESP FAT `Dirty bit is
+  set … Automatically removing`; ext4 orphan-inode cleanup (5 inodes on felhom-pve dm-7, 1 on
+  demo-hp dm-7). NTP synchronized on both after boot.
+
+**Verdict:** site-wide power cut at ~14:41–14:42 CEST hitting both boxes (and the router and the
+NAS) simultaneously. No software or operator action involved; the earlier CC session is ruled out.
+
+## 2. Timeline (CEST)
+
+| Time | Event | Evidence |
+|---|---|---|
+| ~14:27–14:29 | Last reports received by hub (host_reports 14:26:54 / 14:27:54; controller reports 14:28:50 / 14:29:13 — N100 / hp) | p5-report-gap.txt |
+| 14:40–14:41 | Last metrics rows (guest DBs): 14:40:49 N100, 14:40:12 hp | p3-\*-metrics-quickcheck.txt |
+| **14:41–14:42** | **Power cut.** Journals end 14:41:31 / 14:41:15; router reboots (operator-attested, uptime counter) — power itself returned within minutes; **both miniPCs stayed OFF** | p1-\* |
+| 14:57:09 / 14:58:09 | hub `host_stale` (N100 / hp) + operator email sent | p5-hub-log-grep.txt, p5-hub-notification-log.txt |
+| 14:59:09 / 15:00:09 | hub `node_stale` + operator email sent | ” |
+| 15:27:09 / 15:28:09 | hub `host_down` + operator email sent | ” |
+| 15:29:09 / 15:30:09 | hub `node_down` + operator email; **customer email** (doodoo21@freemail.hu) for demo-felhom | ” |
+| ~19:10–19:13 | Viktor powers both boxes on manually (attested) | — |
+| **19:13:52 / 19:13:53** | **T1: kernel boot** (N100 / hp, `uptime -s`) | p1-\*-forensic.txt |
+| 19:14:04–19:14:21 | Host plane up on both: tailscaled, wg-quick, felhom-agent 0.93.0, first host_report at hub (hp 19:14:12, N100 19:14:21) | p2-\*, p4-\*, p5-report-gap.txt |
+| 19:14:11 / 19:14:24 | PBS-over-wg verify cycle complete (hp / N100) — WireGuard path proven live | p6-\*-backup.txt |
+| 19:15:09 | hub `host_recovered` (both, same tick) | p5-hub-events.txt |
+| 19:16:23 / 19:16:30 | Guest 9201 boots (hp / N100; `onboot: 1`, PVE-started) | p3-\*-guests, p3-\*-docker-ps |
+| 19:16:33–19:16:40 | cloudflared re-registers **4/4 tunnel connections** on both guests | p4-\*-tunnels.txt |
+| 19:16:34 / 19:16:38 | First post-outage metrics row (hp / N100) | p3-\*-metrics-quickcheck.txt |
+| 19:16:40 / 19:16:44 | `controller_started` events; first controller reports at hub 19:16:41 / 19:16:44 | p5-hub-events.txt, p5-report-gap.txt |
+| **19:17:09** | hub `node_recovered` (both) — **full platform recovery, Δ ≈ 3 m 15 s from power-on** | p5-hub-events.txt |
+| 19:38–19:47 | Audit probes: tailscale both active-direct; edge-pinned HTTPS 302 on both tunnels; wg handshakes fresh; NAS share wakes + mounts on first access | p0-, p4-external-, p3-demo-hp-nas-wake.txt |
+
+Total service outage ≈ **4 h 35 m**, of which ~4 h 30 m was *boxes sitting powered off* (F8) and
+~3 m 15 s was actual recovery work.
+
+## 3. Self-healing scorecard
+
+Every component recovered **unaided**; Δ measured from T1 (19:13:52/53).
+
+| Component | N100 Δ | demo-hp Δ | Unaided | Evidence |
+|---|---|---|---|---|
+| Kernel boot + fs recovery (journal replay, orphan cleanup, ESP dirty bit) | +3 s | +5 s | yes | p1-\*-forensic.txt |
+| tailscaled (control + direct conns) | ~+21 s | ~+11 s | yes | p4-\*, p0-tailscale-status.txt |
+| wg-quick@wg-felhom (iface up) | +22 s | +12 s | yes | p4-\*-tunnels.txt |
+| felhom-agent 0.93.0 active | +26 s | +16 s | yes | p2-\*-host.txt |
+| First host_report at hub | +29 s | +19 s | yes | p5-report-gap.txt |
+| PBS reachable over wg (verify cycle OK) | +32 s | +18 s | yes | p6-\*-backup.txt |
+| Agent local API bound (192.168.0.162 / .87:8443) | +59 s | +19 s | yes | p2-\*-f1-check.txt |
+| Guest 9201 (onboot=1) | +2 m 38 s | +2 m 30 s | yes | p3-\* |
+| Docker + all stacks (12 resp. 7 containers, 0 exited, 0 unhealthy; controller surface: every deployed stack `running`) | ~+3 m | ~+3 m | yes | p3-\*-docker-ps.txt, p3-\*-stacks-parsed.txt |
+| cloudflared 4/4 connections | +2 m 48 s | +2 m 42 s | yes | p4-\*-tunnels.txt |
+| First controller report at hub | +2 m 52 s | +2 m 48 s | yes | p5-report-gap.txt |
+| Hub `*_recovered` events | +1 m 17 s (host) / +3 m 17 s (node) | same ticks | yes | p5-hub-events.txt |
+| Felhom-Share NAS (demo-hp): idle autofs trigger restored at boot; first access mounts CIFS `//192.168.0.104/Share`; FileBrowser bind live; content intact | on first access | — | yes | p3-demo-hp-nas-mounts.txt, p3-demo-hp-nas-wake.txt |
+| External URLs via pinned Cloudflare edge: 302 in 0.14–0.16 s | ✔ at audit | ✔ | yes | p4-external-edge-probes.txt |
+
+**F1 regression check (load-bearing): F1 did NOT recur.** felhom-pve came back on its static
+192.168.0.162; demo-hp re-acquired its previous DHCP 192.168.0.87 despite the router reboot; agent
+local-API bound cleanly on both, zero bind errors in the journals (p2-\*-f1-check.txt). The static-IP
+mitigation from AUDIT-vacation-remote-ops F1 held under its natural trigger.
+
+**Dead-man's-switch verdict: fired exactly on design schedule.** The runbook's ~15:12/~15:42
+expectation measured from the outage instant; the design (30 m stale / 60 m down,
+`hub-config` `stale_threshold: "30m"`, `staleness.go` `downAfter = 2×`) measures from the **last
+received report** (~14:27–14:29). Observed firings are correct to the minute (checker ticks at
+:09). All 9 dispatches logged `status=sent`, zero errors: 8 operator + 1 customer
+(demo-felhom/node_down — the only customer with notification prefs, see F12).
+`*_recovered` events carry severity `info`, which the dispatcher intentionally does not email
+(dispatcher.go: "info is an intentional non-notify (status/recovery events)") — see F11.
+
+**Data-integrity scars:** confined to what the OS self-healed (§1). `PRAGMA quick_check` = `ok` on
+both metrics DBs (checked on copies); metrics gap cleanly bounded 14:40 → 19:16. Offbox restic
+repo consistent, 14 snapshots, latest 04:16 CEST (pre-outage), **zero locks** (read-only
+`restic list locks`). No backup job was running at the cut — guest backup windows are nightly
+(tier2 03:30, offbox 04:15 CEST), no window fell inside 14:42–19:15, catch-up is automatic tonight
+(p6-\*).
+
+## 4. Findings (continuing the vacation arc; F1–F7 in AUDIT-vacation-remote-ops-2026-07-20.md)
+
+| # | Finding | Severity | Evidence | Disposition |
+|---|---|---|---|---|
+| **F8** | **Site power loss; no auto-power-on.** Power itself returned within minutes (router rebooted at ~14:42 and stayed up); both miniPCs remained off ~4.5 h until manual power-on. For a paying customer this converts a power blip into a half-day outage ending only when someone is physically present. | HIGH | p1-\*; router uptime operator-attested | **mitigated-on-site** — Viktor set BIOS AC-power-on on both boxes (attested; not OS-verifiable). **roadmap-candidate:** make BIOS "restore on AC power" a provisioning-checklist item + a host-install doc requirement for every fleet box. |
+| **F9** | **H1 OOB belt only partially installed on the current fleet.** `felhom-mgmt-watchdog.timer` is live on both boxes (heal marker absent = no heal needed), but `felhom-sshd.service` and `felhom-oob-nft.service` (TASK H1, present in `felhom-agent/configs/`) are installed on **neither** box; operator access rides stock sshd :22 + tailscale + G1 break-glass. Pre-existing (both boxes provisioned via the universal ISO), surfaced by this audit's access preflight. | MEDIUM | p2-felhom-pve-ssh-belt.txt, p2-\*-host.txt | **needs-ruling** — was H1 intentionally dropped from the universal-ISO provisioning path, or should host-install grow the belt? |
+| **F10** | **demo-hp backup tiers incomplete.** Tier-2 dump tree exists (paperless-ngx, fresh), but **no offbox target is configured** and the PBS DR datastore holds **0 snapshots** for it. A power event with disk damage would have had no off-box recovery path for that guest. Pre-existing (box added 07-21), not outage-caused. | MEDIUM | p6-demo-hp-offbox-locks.txt (`NO-OFFBOX`), p6-demo-hp-backup.txt (`snapshots=0`), p6-demo-hp-tier2.txt | **roadmap-candidate** — finish demo-hp tier provisioning (offbox + DR) as part of the pilot hardening pass. |
+| **F11** | **Recovery is silent.** `host_recovered`/`node_recovered` fired correctly but carry severity `info`, which the dispatcher deliberately does not email — the operator/customer only learns of recovery by looking. During a real customer outage the "it's back" signal is arguably the second-most valuable email. Working-as-coded, so a product question, not a defect. | LOW | p5-hub-events.txt; `hub/internal/notify/dispatcher.go` severityNotifies | **needs-ruling** — opt-in recovery notifications (operator at least)? |
+| **F12** | **demo-hp has no customer notification prefs** (`customer_notifications` has no row for it) → its "customer" received no node_down email and never would. Only demo-felhom is wired (doodoo21@freemail.hu). Pre-existing demo-box config gap; on a real onboarding this must not be skippable. | LOW | p5-customer-notification-prefs.txt | **roadmap-candidate** — make notification-prefs setup a claim/onboarding step, not an optional settings page. |
+| **F13** | **PBS DR tier has no visible backup cadence.** `/etc/pve/jobs.cfg` is empty on both hosts; N100's DR datastore holds a single CT-9201 snapshot from **2026-07-18** (S8-era), demo-hp none. The agent's 6 h *verify* loop runs, but nothing appears to *create* periodic DR snapshots. If DR snapshots are meant to be on-demand only, fine — but then the "DR tier" freshness expectation should be documented; if they're meant to be periodic, the scheduler is missing. | LOW | p6-\*-backup.txt | **needs-ruling** — clarify intended DR-tier cadence; roadmap if periodic. |
+
+## 5. Explicitly NOT validated (would have required mutation, or is operator-side)
+
+- **BIOS auto-power-on**: not OS-verifiable; recorded as operator-attested. Its first real proof
+  will be the next power event (or a deliberate drill — candidate for R-55's reboot leg).
+- **Email delivery to the Gmail inbox**: the hub logged all 9 sends as `sent` (Resend accepted);
+  whether they **landed** at ~14:57–15:30 CEST is Viktor-side (inbox + Resend dashboard).
+  **Question for Viktor:** did the 8 operator mails + 1 customer mail arrive? If not, that is a
+  *delivery* finding (dispatcher is proven good).
+- **Cloudflare dashboard tunnel state**: operator-attested; this audit proved the stronger fact
+  (edge-pinned 302 end-to-end on both live tunnels).
+- **demo-hp offbox restic**: nothing to check — no offbox exists (F10).
+- **UI click-throughs**: no browser on DooPlex; controller state was validated on its own API
+  surface (endpoint-level, per standing convention).
+
+## 6. Observations (out of scope, not acted on)
+
+- **`demo-vm` tunnel will not recover — and that is correct.** Its owner (the nested appliance) no
+  longer exists: the three ISO-train drill VMs 9310/9311/9312 were destroyed cleanly this morning
+  11:01–11:43 CEST, *before* the outage (p3-felhom-pve-vm-absence.txt). The tunnel's Down state is
+  permanent until the dead hub/Cloudflare records are discarded (already pending as operator
+  cleanup from the ISO train; hub-side ghost-delete = R-62 arc).
+- **peti-felhom is silent since 2026-07-15 08:39 UTC** (last report in the hub DB; hub seeded it
+  `down` at its 09:25 restart, so no fresh event fired). Matches the petifelhom tunnel Down in
+  Viktor's screenshot. His site, his box — noted for the planned Friday reinstall, nothing touched.
+- **Controllers restarted 14:13–14:14 CEST** (`controller_started` events 1701/1702) — that was the
+  0.160.0 publish train, 28 minutes before the cut; unrelated to the outage, but a reminder that
+  the fleet had been on 0.160.0 for less than half an hour when it took its first dirty shutdown —
+  and came back clean.
+- Hub pod log prints CEST while the hub DB stores UTC; this audit normalized everything to CEST.
+
+## 7. Bottom line
+
+The platform passed its first real, unplanned, site-wide dirty-shutdown drill. Every layer —
+filesystems, host plane, WireGuard, tailscale, guests, Docker stacks, cloudflared, NAS automount,
+hub staleness detection, notifications, backup integrity — recovered without a single manual
+intervention, in ~3 m 15 s of actual work after power returned. The entire 4.5 h of customer-visible
+downtime was attributable to one thing: nobody could press the power button (F8, now mitigated).
+The dead-man's-switch pipeline fired on schedule and the emails dispatched; the open questions are
+delivery-side (inbox) and product-side (silent recovery, F11).