diff --git a/REPORT.md b/REPORT.md index 99f52d4..b4b163c 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,102 +1,53 @@ -# REPORT — scripts/ISO train v1.24.0: R-59 + R-60 + R-61 (2026-07-22) +# REPORT — Power-outage recovery audit, vacation site (2026-07-22) -**Task:** the three HP-t740 findings — the box that can never call home (R-59 refuse-loudly gate), -the first-boot NIC sweep self-heal (R-60), and the knowable baked root password (R-61 slice 1) — -implemented in `scripts/iso/`, spike-proven, and driven end-to-end on a nested wrong-NIC install. -No hub/agent/controller/catalog changes; golden image and floors untouched. +**Task:** RUNBOOK-driven, strictly read-only forensic + recovery audit after the 14:41–42 CEST +site-wide power cut at the vacation site (felhom-pve N100 + demo-hp t740, ~4.5 h down, manual +power-on ~19:13). No code, no fixes, no restarts — diagnosis only. Deliverable: +`documentation/audits/AUDIT-power-outage-recovery-2026-07-22.md`; raw evidence on DooPlex +`~/outage-20260722/evidence/`. -## Baselines +## Verdict -- felhom.eu `main` at start: `1613bc7` (scripts CHANGELOG head = build-hub.sh v1.23.0); clean-tree - gate held before both ISO builds. -- Target: scripts train **v1.24.0** (`ISO_VERSION="1.24.0"` in `build-felhom-iso.sh`). +- **Cause:** simultaneous external power cut on both boxes — both prev-boot journals end abruptly + mid-agent-routine, `last -x` says `crash`, zero non-agent sudo/sshd entries in the window. The + earlier CC session is exonerated. +- **Recovery:** every layer self-healed unaided in **~3 m 15 s** after power-on — fs journal + replay, agent 0.93.0 (+26 s), wg + PBS-over-wg (+32 s), tailscale, guest 9201 (onboot=1, + +2 m 38 s), all stacks healthy on the controllers' own surfaces, cloudflared 4/4 (+2 m 48 s), + hub `*_recovered` (+3 m 17 s). Edge-pinned public probes 302. Felhom-Share autofs woke + CIFS + mounted on first access (first unplanned re-run of the R-67 dead-NAS proof — passed). +- **Dead-man's-switch live fire:** host/node stale + down fired **exactly on the 30 m/60 m design + schedule** (measured from last received report, not from the outage instant), all 9 notification + dispatches `sent` (8 operator + 1 customer). Recovery events are severity `info` → intentionally + no email (flagged as product question F11). +- **F1 regression:** did NOT recur — static .162 held, demo-hp re-acquired .87, local API bound + cleanly on both. +- **Scars:** only OS-self-healed ones (journald rotate, ESP dirty bit, orphan inodes). Metrics DBs + `quick_check` ok, gap 14:40→19:16. Offbox restic: 14 snapshots, latest pre-outage 04:16, zero + stale locks. No backup window fell inside the outage; catch-up is tonight 03:30/04:15. -## Commits (this train, in order) +## Findings (F8+ continuing the vacation arc) -| Hash | What | -|---|---| -| `9e29295` | R-61 slice 1: rootpw sibling file (0600) + manifest pointer; `test/rootpw-emission.sh`; OUT_ISO hoist; `FELHOM_ISO_KEEP_WORK` escape | -| `ad1d26a` | Harness catches up with the v1.21.0 in-script pairing wait (old scenarios would hang); assistant image + python3 | -| `c043ba9` | Phase-0 spike doc — sweep mechanics GO, F-P1..F-P8 (STOP-0 discharged before gate code) | -| `cf1862c` | R-59+R-60: `network_gate` + `sweep_nics` + Hungarian console screen; 4 harness scenarios; 3 red-proofs run | -| `699325b` | R-60 drill fix (F-P9): flush vmbr0 before the bounded dhclient — the baked fallback default route otherwise poisons the hub probe | -| (this commit) | v1.24.0 CHANGELOG, README, ROADMAP (R-59/60/61 SHIPPED + new R-62), onboarding runbook, capability map, spike-doc drill section + screen capture, this REPORT | +- **F8 (HIGH, mitigated-on-site):** no auto-power-on — power was back in minutes, boxes sat off + 4.5 h. BIOS AC-power-on now set (attested); roadmap-candidate: provisioning-checklist item. +- **F9 (MEDIUM, needs-ruling):** H1 OOB belt (`felhom-sshd` + `felhom-oob-nft`) installed on + NEITHER fleet box — only the mgmt-watchdog timer; access is stock sshd :22 + break-glass. +- **F10 (MEDIUM, roadmap):** demo-hp has tier-2 only — no offbox target, 0 PBS DR snapshots. +- **F11 (LOW, needs-ruling):** recovery is silent (`info` severity never emails). +- **F12 (LOW, roadmap):** demo-hp has no customer notification prefs row — node_down customer + email impossible there. +- **F13 (LOW, needs-ruling):** PBS DR tier shows no snapshot cadence (jobs.cfg empty; single + 07-18 snapshot on N100, none on demo-hp) — intended? -## Harness results +## Observations -- `test/bootstrap-modes.sh` (in `felhom-iso-assistant:trixie`): **40/40 green** — D (direct, - zero appliance calls, G1 zero-new-behavior invariant incl. zero sleeps + fixture untouched), - P (v1.21.0 single-invocation pairing loop), 410 crash-window, G2 sweep-success - (re-point + persist `inet dhcp` + `.felhom-bak` + proceed), G3 sweep-fail (screen content + - **byte-identical** interfaces + no bak), G4 state.json-present (sweep **never** invoked by - fake call-count, interfaces untouched). -- `test/rootpw-emission.sh`: **11/11 green** — dry-run emission, 0600, plaintext↔answer-hash - cross-check (`openssl passwd -6 -salt`), plaintext absent from build output, manifest-heredoc - pointer/no-leak guard. -- `shellcheck -S warning felhom-bootstrap.sh`: clean. +demo-vm tunnel Down is permanent-correct (nested drill VMs 9310–9312 destroyed pre-outage this +morning; record discard pending). peti-felhom silent since 07-15 08:39 UTC (his site, untouched). +Controllers had been on 0.160.0 for only ~28 min when the cut hit — first dirty shutdown on the +new version came back clean. -## Red-proofs (all four run, failed exactly on their guarding assertion, restored) +**Open question for Viktor:** did the ~14:57–15:30 CEST alert emails actually land in the inbox? +Hub-side they are all `sent`; delivery is inbox/Resend-dashboard-side. -1. **Manifest plaintext leak** (R-61): `root-password-plain : ${ROOT_PLAIN}` injected into the - manifest heredoc → `manifest never references ROOT_PLAIN` FAILED → restored. -2. **Restore-on-failure dropped** (R-60): sweep's byte-identical restore removed → G3 - `interfaces BYTE-IDENTICAL after failed sweep` FAILED → restored. -3. **State gate removed** (R-60): `[[ ! -e $STATE_FILE ]]` → `true` → G4 zero-ifreload/zero- - dhclient FAILED → restored. -4. **Hub-reachable early return removed** (R-59): all five G1 invariants FAILED → restored. - -## Phase-0 spike (STOP-0 honored) - -`documentation/audits/SPIKE-firstboot-nic-sweep-2026-07-22.md` — written and committed BEFORE the -gate code. Verdict GO; key inputs: success leg ≈ 4 s, dead candidate = exactly its DHCP bound; -unused NICs are admin-DOWN (raise before carrier read); ANY HTTP status = reachable (hub answers -302); trigger on hub-unreachable, never the fallback signature (the drill found the second -wrong-NIC variant: the installer picks the NIC that LEASES and bakes the lease as static). - -## Nested end-to-end drill (evidence in the spike doc) - -- **Round 1** caught a real defect the harness could not: the fallback bake includes a **default - route via 192.168.100.1**, and dhclient-script never replaces an existing default route → the - sweep's hub probe rode the dead gateway and the gate oscillated. Fixed (`699325b`, flush before - dhclient), validated by hot-patch (heal in 33 s), then the ISO was rebuilt. -- **Round 2, virgin final ISO, zero patching:** dead-NIC install → fallback baked → **R-59 screen - on the console** (capture committed: `SPIKE-firstboot-nic-sweep-2026-07-22-screen.png`) → - failed-sweep restore proven safe (flushed static re-applied by ifreload) → cable move → - **self-heal + hub registration in 23 s unaided** (persisted `inet dhcp` + winner port, original - in `interfaces.felhom-bak`, box at a proper DHCP address, pairing banner + code on console). -- **R-61 lockout closed, proven:** the build's `.rootpw.txt` password recomputed against the - installed box's `/etc/shadow` root hash → exact match (computed on the build host; the - plaintext never left it). Value NOT included here by design. -- Validation method note: no browser on DooPlex — console proof is QEMU screendump; hub-side - proof is the box's own registration/pairing-code delivery (endpoint-level). - -## Artifacts - -- Final drill ISO: `felhom-pve-9.2-1-v1.24.0-nested-probe-generic.iso`, sha256 - `c25cdbbf35d2134b0bf278c4b069a3d5849c0960cf0a5e4500e52638489c8947`, built from clean tree at - `699325b` with an **uncommitted** key-bearing probe profile (nested-vm + ops SSH key — never - committed, per the armed-profile rule). Its `.rootpw.txt` exists beside it (0600) — verified, - value stored out-of-band only. -- Shipping rebuilds for real profiles (hp/n100/generic) happen at next need; the committed - pipeline builds them from `main` as usual. - -## Cleanup verification - -Nested VMs 9310/9311/9312 destroyed; throwaway `vmbr9` (+ its temporary host address) deleted; -transferred ISO + screendumps removed from felhom-pve; `brctl show` back to exactly `enp1s0` + -`veth9201i0` on vmbr0; **`/etc/network/interfaces` on felhom-pve diffed BYTE-IDENTICAL against -the pre-task copy** (sha `c7ace087…`). Residue by design: **three unclaimed-appliance -registrations at the production hub** (all three source machines destroyed) — operator may -discard them from the Unclaimed list. - -## Observations / follow-ups - -- **F-P8 (out of scope, noted):** the pairing loop's register-failure line prints every ~42 s on - a no-network box; v1.21.0 quieted the 204 wait but not this path. The gate makes it mostly moot - (it holds before register), but the line exists for a box that loses network AFTER install - start (state.json present → screen + retry, register never reached). -- The console font renders Hungarian long vowels (ő/ű) approximately on the text console — - same behavior as the existing pairing banner; acceptable. -- `qm` screendump (`qm monitor` → `screendump`) is a serviceable no-browser console-capture - method for nested drills — worth remembering for future runbooks. -- R-62 (hub delete-dialog cosmetics) filed on the ROADMAP as agreed — docs-only here. +**Not validated (needs-mutation or operator-side):** BIOS setting itself, inbox delivery, +Cloudflare dashboard state, UI click-throughs. diff --git a/documentation/audits/AUDIT-power-outage-recovery-2026-07-22.md b/documentation/audits/AUDIT-power-outage-recovery-2026-07-22.md new file mode 100644 index 0000000..1522682 --- /dev/null +++ b/documentation/audits/AUDIT-power-outage-recovery-2026-07-22.md @@ -0,0 +1,154 @@ +# AUDIT — Power-outage recovery, vacation site, 2026-07-22 + +**Run:** read-only forensic + recovery audit, executed from DooPlex 2026-07-22 19:38–20:00 CEST. +**Scope:** felhom-pve (N100, `demo-felhom-8363b5`, tailnet 100.70.170.35) and demo-hp (HP t740, +`demo-hp-bb76ea`, 100.76.96.79), their guests 9201, the hub plane, and the backup plane. +**Mode:** strictly read-only on all live systems — nothing was restarted, redeployed, unlocked or +triggered. The single state-affecting action was a designed one: a read-only `ls` that woke the +idle Felhom-Share autofs trigger (its normal first-access path). +**Evidence:** DooPlex `~/outage-20260722/evidence/` — every claim below cites a file there. +**Timestamps:** normalized to **CEST**. Hosts log CEST; guests, the hub **DB** and Cloudflare log +UTC (CEST−2); the hub **pod log** prints CEST. Watch for this when re-reading raw evidence. +**Access paths used (p0-access.txt):** felhom-pve = operator key, root@tailnet. demo-hp = key-less +by design → hub-vaulted G1 break-glass (`host_recovery/demo-hp-bb76ea`, file→file, shredded after). +Both worked on the first path; the OOB *belt* units however are absent — see **F9**. + +Related: `AUDIT-vacation-remote-ops-2026-07-20.md` (F1–F7 of this arc; F-numbers continue at F8). + +--- + +## 1. Verdict on cause: simultaneous external power cut — CC session exonerated + +- Both previous-boot journals end **abruptly mid-routine-agent-activity** — felhom-pve at + **14:41:31**, demo-hp at **14:41:15** — with no `Reached target Power-Off` cascade + (p1-felhom-pve-journal-prevboot-tail.txt, p1-demo-hp-journal-prevboot-tail.txt). +- `last -x` marks the previous boot **`crash`** on both hosts (p1-\*-forensic.txt). +- The 14:00–15:00 sudo/sshd window contains **zero non-agent entries** on either host — 4 569 resp. + 4 533 journal lines, all `felhom-agent` sudo routine; no SSH login, no `poweroff`/`shutdown`/ + `pct`/`qm` invocation (p1-\*-forensic.txt). +- Both boxes show the classic dirty-shutdown recovery trio on the next boot: journald + `system.journal corrupted or uncleanly shut down, renaming and replacing`; ESP FAT `Dirty bit is + set … Automatically removing`; ext4 orphan-inode cleanup (5 inodes on felhom-pve dm-7, 1 on + demo-hp dm-7). NTP synchronized on both after boot. + +**Verdict:** site-wide power cut at ~14:41–14:42 CEST hitting both boxes (and the router and the +NAS) simultaneously. No software or operator action involved; the earlier CC session is ruled out. + +## 2. Timeline (CEST) + +| Time | Event | Evidence | +|---|---|---| +| ~14:27–14:29 | Last reports received by hub (host_reports 14:26:54 / 14:27:54; controller reports 14:28:50 / 14:29:13 — N100 / hp) | p5-report-gap.txt | +| 14:40–14:41 | Last metrics rows (guest DBs): 14:40:49 N100, 14:40:12 hp | p3-\*-metrics-quickcheck.txt | +| **14:41–14:42** | **Power cut.** Journals end 14:41:31 / 14:41:15; router reboots (operator-attested, uptime counter) — power itself returned within minutes; **both miniPCs stayed OFF** | p1-\* | +| 14:57:09 / 14:58:09 | hub `host_stale` (N100 / hp) + operator email sent | p5-hub-log-grep.txt, p5-hub-notification-log.txt | +| 14:59:09 / 15:00:09 | hub `node_stale` + operator email sent | ” | +| 15:27:09 / 15:28:09 | hub `host_down` + operator email sent | ” | +| 15:29:09 / 15:30:09 | hub `node_down` + operator email; **customer email** (doodoo21@freemail.hu) for demo-felhom | ” | +| ~19:10–19:13 | Viktor powers both boxes on manually (attested) | — | +| **19:13:52 / 19:13:53** | **T1: kernel boot** (N100 / hp, `uptime -s`) | p1-\*-forensic.txt | +| 19:14:04–19:14:21 | Host plane up on both: tailscaled, wg-quick, felhom-agent 0.93.0, first host_report at hub (hp 19:14:12, N100 19:14:21) | p2-\*, p4-\*, p5-report-gap.txt | +| 19:14:11 / 19:14:24 | PBS-over-wg verify cycle complete (hp / N100) — WireGuard path proven live | p6-\*-backup.txt | +| 19:15:09 | hub `host_recovered` (both, same tick) | p5-hub-events.txt | +| 19:16:23 / 19:16:30 | Guest 9201 boots (hp / N100; `onboot: 1`, PVE-started) | p3-\*-guests, p3-\*-docker-ps | +| 19:16:33–19:16:40 | cloudflared re-registers **4/4 tunnel connections** on both guests | p4-\*-tunnels.txt | +| 19:16:34 / 19:16:38 | First post-outage metrics row (hp / N100) | p3-\*-metrics-quickcheck.txt | +| 19:16:40 / 19:16:44 | `controller_started` events; first controller reports at hub 19:16:41 / 19:16:44 | p5-hub-events.txt, p5-report-gap.txt | +| **19:17:09** | hub `node_recovered` (both) — **full platform recovery, Δ ≈ 3 m 15 s from power-on** | p5-hub-events.txt | +| 19:38–19:47 | Audit probes: tailscale both active-direct; edge-pinned HTTPS 302 on both tunnels; wg handshakes fresh; NAS share wakes + mounts on first access | p0-, p4-external-, p3-demo-hp-nas-wake.txt | + +Total service outage ≈ **4 h 35 m**, of which ~4 h 30 m was *boxes sitting powered off* (F8) and +~3 m 15 s was actual recovery work. + +## 3. Self-healing scorecard + +Every component recovered **unaided**; Δ measured from T1 (19:13:52/53). + +| Component | N100 Δ | demo-hp Δ | Unaided | Evidence | +|---|---|---|---|---| +| Kernel boot + fs recovery (journal replay, orphan cleanup, ESP dirty bit) | +3 s | +5 s | yes | p1-\*-forensic.txt | +| tailscaled (control + direct conns) | ~+21 s | ~+11 s | yes | p4-\*, p0-tailscale-status.txt | +| wg-quick@wg-felhom (iface up) | +22 s | +12 s | yes | p4-\*-tunnels.txt | +| felhom-agent 0.93.0 active | +26 s | +16 s | yes | p2-\*-host.txt | +| First host_report at hub | +29 s | +19 s | yes | p5-report-gap.txt | +| PBS reachable over wg (verify cycle OK) | +32 s | +18 s | yes | p6-\*-backup.txt | +| Agent local API bound (192.168.0.162 / .87:8443) | +59 s | +19 s | yes | p2-\*-f1-check.txt | +| Guest 9201 (onboot=1) | +2 m 38 s | +2 m 30 s | yes | p3-\* | +| Docker + all stacks (12 resp. 7 containers, 0 exited, 0 unhealthy; controller surface: every deployed stack `running`) | ~+3 m | ~+3 m | yes | p3-\*-docker-ps.txt, p3-\*-stacks-parsed.txt | +| cloudflared 4/4 connections | +2 m 48 s | +2 m 42 s | yes | p4-\*-tunnels.txt | +| First controller report at hub | +2 m 52 s | +2 m 48 s | yes | p5-report-gap.txt | +| Hub `*_recovered` events | +1 m 17 s (host) / +3 m 17 s (node) | same ticks | yes | p5-hub-events.txt | +| Felhom-Share NAS (demo-hp): idle autofs trigger restored at boot; first access mounts CIFS `//192.168.0.104/Share`; FileBrowser bind live; content intact | on first access | — | yes | p3-demo-hp-nas-mounts.txt, p3-demo-hp-nas-wake.txt | +| External URLs via pinned Cloudflare edge: 302 in 0.14–0.16 s | ✔ at audit | ✔ | yes | p4-external-edge-probes.txt | + +**F1 regression check (load-bearing): F1 did NOT recur.** felhom-pve came back on its static +192.168.0.162; demo-hp re-acquired its previous DHCP 192.168.0.87 despite the router reboot; agent +local-API bound cleanly on both, zero bind errors in the journals (p2-\*-f1-check.txt). The static-IP +mitigation from AUDIT-vacation-remote-ops F1 held under its natural trigger. + +**Dead-man's-switch verdict: fired exactly on design schedule.** The runbook's ~15:12/~15:42 +expectation measured from the outage instant; the design (30 m stale / 60 m down, +`hub-config` `stale_threshold: "30m"`, `staleness.go` `downAfter = 2×`) measures from the **last +received report** (~14:27–14:29). Observed firings are correct to the minute (checker ticks at +:09). All 9 dispatches logged `status=sent`, zero errors: 8 operator + 1 customer +(demo-felhom/node_down — the only customer with notification prefs, see F12). +`*_recovered` events carry severity `info`, which the dispatcher intentionally does not email +(dispatcher.go: "info is an intentional non-notify (status/recovery events)") — see F11. + +**Data-integrity scars:** confined to what the OS self-healed (§1). `PRAGMA quick_check` = `ok` on +both metrics DBs (checked on copies); metrics gap cleanly bounded 14:40 → 19:16. Offbox restic +repo consistent, 14 snapshots, latest 04:16 CEST (pre-outage), **zero locks** (read-only +`restic list locks`). No backup job was running at the cut — guest backup windows are nightly +(tier2 03:30, offbox 04:15 CEST), no window fell inside 14:42–19:15, catch-up is automatic tonight +(p6-\*). + +## 4. Findings (continuing the vacation arc; F1–F7 in AUDIT-vacation-remote-ops-2026-07-20.md) + +| # | Finding | Severity | Evidence | Disposition | +|---|---|---|---|---| +| **F8** | **Site power loss; no auto-power-on.** Power itself returned within minutes (router rebooted at ~14:42 and stayed up); both miniPCs remained off ~4.5 h until manual power-on. For a paying customer this converts a power blip into a half-day outage ending only when someone is physically present. | HIGH | p1-\*; router uptime operator-attested | **mitigated-on-site** — Viktor set BIOS AC-power-on on both boxes (attested; not OS-verifiable). **roadmap-candidate:** make BIOS "restore on AC power" a provisioning-checklist item + a host-install doc requirement for every fleet box. | +| **F9** | **H1 OOB belt only partially installed on the current fleet.** `felhom-mgmt-watchdog.timer` is live on both boxes (heal marker absent = no heal needed), but `felhom-sshd.service` and `felhom-oob-nft.service` (TASK H1, present in `felhom-agent/configs/`) are installed on **neither** box; operator access rides stock sshd :22 + tailscale + G1 break-glass. Pre-existing (both boxes provisioned via the universal ISO), surfaced by this audit's access preflight. | MEDIUM | p2-felhom-pve-ssh-belt.txt, p2-\*-host.txt | **needs-ruling** — was H1 intentionally dropped from the universal-ISO provisioning path, or should host-install grow the belt? | +| **F10** | **demo-hp backup tiers incomplete.** Tier-2 dump tree exists (paperless-ngx, fresh), but **no offbox target is configured** and the PBS DR datastore holds **0 snapshots** for it. A power event with disk damage would have had no off-box recovery path for that guest. Pre-existing (box added 07-21), not outage-caused. | MEDIUM | p6-demo-hp-offbox-locks.txt (`NO-OFFBOX`), p6-demo-hp-backup.txt (`snapshots=0`), p6-demo-hp-tier2.txt | **roadmap-candidate** — finish demo-hp tier provisioning (offbox + DR) as part of the pilot hardening pass. | +| **F11** | **Recovery is silent.** `host_recovered`/`node_recovered` fired correctly but carry severity `info`, which the dispatcher deliberately does not email — the operator/customer only learns of recovery by looking. During a real customer outage the "it's back" signal is arguably the second-most valuable email. Working-as-coded, so a product question, not a defect. | LOW | p5-hub-events.txt; `hub/internal/notify/dispatcher.go` severityNotifies | **needs-ruling** — opt-in recovery notifications (operator at least)? | +| **F12** | **demo-hp has no customer notification prefs** (`customer_notifications` has no row for it) → its "customer" received no node_down email and never would. Only demo-felhom is wired (doodoo21@freemail.hu). Pre-existing demo-box config gap; on a real onboarding this must not be skippable. | LOW | p5-customer-notification-prefs.txt | **roadmap-candidate** — make notification-prefs setup a claim/onboarding step, not an optional settings page. | +| **F13** | **PBS DR tier has no visible backup cadence.** `/etc/pve/jobs.cfg` is empty on both hosts; N100's DR datastore holds a single CT-9201 snapshot from **2026-07-18** (S8-era), demo-hp none. The agent's 6 h *verify* loop runs, but nothing appears to *create* periodic DR snapshots. If DR snapshots are meant to be on-demand only, fine — but then the "DR tier" freshness expectation should be documented; if they're meant to be periodic, the scheduler is missing. | LOW | p6-\*-backup.txt | **needs-ruling** — clarify intended DR-tier cadence; roadmap if periodic. | + +## 5. Explicitly NOT validated (would have required mutation, or is operator-side) + +- **BIOS auto-power-on**: not OS-verifiable; recorded as operator-attested. Its first real proof + will be the next power event (or a deliberate drill — candidate for R-55's reboot leg). +- **Email delivery to the Gmail inbox**: the hub logged all 9 sends as `sent` (Resend accepted); + whether they **landed** at ~14:57–15:30 CEST is Viktor-side (inbox + Resend dashboard). + **Question for Viktor:** did the 8 operator mails + 1 customer mail arrive? If not, that is a + *delivery* finding (dispatcher is proven good). +- **Cloudflare dashboard tunnel state**: operator-attested; this audit proved the stronger fact + (edge-pinned 302 end-to-end on both live tunnels). +- **demo-hp offbox restic**: nothing to check — no offbox exists (F10). +- **UI click-throughs**: no browser on DooPlex; controller state was validated on its own API + surface (endpoint-level, per standing convention). + +## 6. Observations (out of scope, not acted on) + +- **`demo-vm` tunnel will not recover — and that is correct.** Its owner (the nested appliance) no + longer exists: the three ISO-train drill VMs 9310/9311/9312 were destroyed cleanly this morning + 11:01–11:43 CEST, *before* the outage (p3-felhom-pve-vm-absence.txt). The tunnel's Down state is + permanent until the dead hub/Cloudflare records are discarded (already pending as operator + cleanup from the ISO train; hub-side ghost-delete = R-62 arc). +- **peti-felhom is silent since 2026-07-15 08:39 UTC** (last report in the hub DB; hub seeded it + `down` at its 09:25 restart, so no fresh event fired). Matches the petifelhom tunnel Down in + Viktor's screenshot. His site, his box — noted for the planned Friday reinstall, nothing touched. +- **Controllers restarted 14:13–14:14 CEST** (`controller_started` events 1701/1702) — that was the + 0.160.0 publish train, 28 minutes before the cut; unrelated to the outage, but a reminder that + the fleet had been on 0.160.0 for less than half an hour when it took its first dirty shutdown — + and came back clean. +- Hub pod log prints CEST while the hub DB stores UTC; this audit normalized everything to CEST. + +## 7. Bottom line + +The platform passed its first real, unplanned, site-wide dirty-shutdown drill. Every layer — +filesystems, host plane, WireGuard, tailscale, guests, Docker stacks, cloudflared, NAS automount, +hub staleness detection, notifications, backup integrity — recovered without a single manual +intervention, in ~3 m 15 s of actual work after power returned. The entire 4.5 h of customer-visible +downtime was attributable to one thing: nobody could press the power button (F8, now mitigated). +The dead-man's-switch pipeline fired on schedule and the emails dispatched; the open questions are +delivery-side (inbox) and product-side (silent recovery, F11).