docs(audits): power-outage recovery audit 2026-07-22 — power cut confirmed, full self-heal in 3m15s, dead-man's-switch fired on schedule; F8-F13
This commit is contained in:
@@ -1,102 +1,53 @@
|
||||
# REPORT — scripts/ISO train v1.24.0: R-59 + R-60 + R-61 (2026-07-22)
|
||||
# REPORT — Power-outage recovery audit, vacation site (2026-07-22)
|
||||
|
||||
**Task:** the three HP-t740 findings — the box that can never call home (R-59 refuse-loudly gate),
|
||||
the first-boot NIC sweep self-heal (R-60), and the knowable baked root password (R-61 slice 1) —
|
||||
implemented in `scripts/iso/`, spike-proven, and driven end-to-end on a nested wrong-NIC install.
|
||||
No hub/agent/controller/catalog changes; golden image and floors untouched.
|
||||
**Task:** RUNBOOK-driven, strictly read-only forensic + recovery audit after the 14:41–42 CEST
|
||||
site-wide power cut at the vacation site (felhom-pve N100 + demo-hp t740, ~4.5 h down, manual
|
||||
power-on ~19:13). No code, no fixes, no restarts — diagnosis only. Deliverable:
|
||||
`documentation/audits/AUDIT-power-outage-recovery-2026-07-22.md`; raw evidence on DooPlex
|
||||
`~/outage-20260722/evidence/`.
|
||||
|
||||
## Baselines
|
||||
## Verdict
|
||||
|
||||
- felhom.eu `main` at start: `1613bc7` (scripts CHANGELOG head = build-hub.sh v1.23.0); clean-tree
|
||||
gate held before both ISO builds.
|
||||
- Target: scripts train **v1.24.0** (`ISO_VERSION="1.24.0"` in `build-felhom-iso.sh`).
|
||||
- **Cause:** simultaneous external power cut on both boxes — both prev-boot journals end abruptly
|
||||
mid-agent-routine, `last -x` says `crash`, zero non-agent sudo/sshd entries in the window. The
|
||||
earlier CC session is exonerated.
|
||||
- **Recovery:** every layer self-healed unaided in **~3 m 15 s** after power-on — fs journal
|
||||
replay, agent 0.93.0 (+26 s), wg + PBS-over-wg (+32 s), tailscale, guest 9201 (onboot=1,
|
||||
+2 m 38 s), all stacks healthy on the controllers' own surfaces, cloudflared 4/4 (+2 m 48 s),
|
||||
hub `*_recovered` (+3 m 17 s). Edge-pinned public probes 302. Felhom-Share autofs woke + CIFS
|
||||
mounted on first access (first unplanned re-run of the R-67 dead-NAS proof — passed).
|
||||
- **Dead-man's-switch live fire:** host/node stale + down fired **exactly on the 30 m/60 m design
|
||||
schedule** (measured from last received report, not from the outage instant), all 9 notification
|
||||
dispatches `sent` (8 operator + 1 customer). Recovery events are severity `info` → intentionally
|
||||
no email (flagged as product question F11).
|
||||
- **F1 regression:** did NOT recur — static .162 held, demo-hp re-acquired .87, local API bound
|
||||
cleanly on both.
|
||||
- **Scars:** only OS-self-healed ones (journald rotate, ESP dirty bit, orphan inodes). Metrics DBs
|
||||
`quick_check` ok, gap 14:40→19:16. Offbox restic: 14 snapshots, latest pre-outage 04:16, zero
|
||||
stale locks. No backup window fell inside the outage; catch-up is tonight 03:30/04:15.
|
||||
|
||||
## Commits (this train, in order)
|
||||
## Findings (F8+ continuing the vacation arc)
|
||||
|
||||
| Hash | What |
|
||||
|---|---|
|
||||
| `9e29295` | R-61 slice 1: rootpw sibling file (0600) + manifest pointer; `test/rootpw-emission.sh`; OUT_ISO hoist; `FELHOM_ISO_KEEP_WORK` escape |
|
||||
| `ad1d26a` | Harness catches up with the v1.21.0 in-script pairing wait (old scenarios would hang); assistant image + python3 |
|
||||
| `c043ba9` | Phase-0 spike doc — sweep mechanics GO, F-P1..F-P8 (STOP-0 discharged before gate code) |
|
||||
| `cf1862c` | R-59+R-60: `network_gate` + `sweep_nics` + Hungarian console screen; 4 harness scenarios; 3 red-proofs run |
|
||||
| `699325b` | R-60 drill fix (F-P9): flush vmbr0 before the bounded dhclient — the baked fallback default route otherwise poisons the hub probe |
|
||||
| (this commit) | v1.24.0 CHANGELOG, README, ROADMAP (R-59/60/61 SHIPPED + new R-62), onboarding runbook, capability map, spike-doc drill section + screen capture, this REPORT |
|
||||
- **F8 (HIGH, mitigated-on-site):** no auto-power-on — power was back in minutes, boxes sat off
|
||||
4.5 h. BIOS AC-power-on now set (attested); roadmap-candidate: provisioning-checklist item.
|
||||
- **F9 (MEDIUM, needs-ruling):** H1 OOB belt (`felhom-sshd` + `felhom-oob-nft`) installed on
|
||||
NEITHER fleet box — only the mgmt-watchdog timer; access is stock sshd :22 + break-glass.
|
||||
- **F10 (MEDIUM, roadmap):** demo-hp has tier-2 only — no offbox target, 0 PBS DR snapshots.
|
||||
- **F11 (LOW, needs-ruling):** recovery is silent (`info` severity never emails).
|
||||
- **F12 (LOW, roadmap):** demo-hp has no customer notification prefs row — node_down customer
|
||||
email impossible there.
|
||||
- **F13 (LOW, needs-ruling):** PBS DR tier shows no snapshot cadence (jobs.cfg empty; single
|
||||
07-18 snapshot on N100, none on demo-hp) — intended?
|
||||
|
||||
## Harness results
|
||||
## Observations
|
||||
|
||||
- `test/bootstrap-modes.sh` (in `felhom-iso-assistant:trixie`): **40/40 green** — D (direct,
|
||||
zero appliance calls, G1 zero-new-behavior invariant incl. zero sleeps + fixture untouched),
|
||||
P (v1.21.0 single-invocation pairing loop), 410 crash-window, G2 sweep-success
|
||||
(re-point + persist `inet dhcp` + `.felhom-bak` + proceed), G3 sweep-fail (screen content +
|
||||
**byte-identical** interfaces + no bak), G4 state.json-present (sweep **never** invoked by
|
||||
fake call-count, interfaces untouched).
|
||||
- `test/rootpw-emission.sh`: **11/11 green** — dry-run emission, 0600, plaintext↔answer-hash
|
||||
cross-check (`openssl passwd -6 -salt`), plaintext absent from build output, manifest-heredoc
|
||||
pointer/no-leak guard.
|
||||
- `shellcheck -S warning felhom-bootstrap.sh`: clean.
|
||||
demo-vm tunnel Down is permanent-correct (nested drill VMs 9310–9312 destroyed pre-outage this
|
||||
morning; record discard pending). peti-felhom silent since 07-15 08:39 UTC (his site, untouched).
|
||||
Controllers had been on 0.160.0 for only ~28 min when the cut hit — first dirty shutdown on the
|
||||
new version came back clean.
|
||||
|
||||
## Red-proofs (all four run, failed exactly on their guarding assertion, restored)
|
||||
**Open question for Viktor:** did the ~14:57–15:30 CEST alert emails actually land in the inbox?
|
||||
Hub-side they are all `sent`; delivery is inbox/Resend-dashboard-side.
|
||||
|
||||
1. **Manifest plaintext leak** (R-61): `root-password-plain : ${ROOT_PLAIN}` injected into the
|
||||
manifest heredoc → `manifest never references ROOT_PLAIN` FAILED → restored.
|
||||
2. **Restore-on-failure dropped** (R-60): sweep's byte-identical restore removed → G3
|
||||
`interfaces BYTE-IDENTICAL after failed sweep` FAILED → restored.
|
||||
3. **State gate removed** (R-60): `[[ ! -e $STATE_FILE ]]` → `true` → G4 zero-ifreload/zero-
|
||||
dhclient FAILED → restored.
|
||||
4. **Hub-reachable early return removed** (R-59): all five G1 invariants FAILED → restored.
|
||||
|
||||
## Phase-0 spike (STOP-0 honored)
|
||||
|
||||
`documentation/audits/SPIKE-firstboot-nic-sweep-2026-07-22.md` — written and committed BEFORE the
|
||||
gate code. Verdict GO; key inputs: success leg ≈ 4 s, dead candidate = exactly its DHCP bound;
|
||||
unused NICs are admin-DOWN (raise before carrier read); ANY HTTP status = reachable (hub answers
|
||||
302); trigger on hub-unreachable, never the fallback signature (the drill found the second
|
||||
wrong-NIC variant: the installer picks the NIC that LEASES and bakes the lease as static).
|
||||
|
||||
## Nested end-to-end drill (evidence in the spike doc)
|
||||
|
||||
- **Round 1** caught a real defect the harness could not: the fallback bake includes a **default
|
||||
route via 192.168.100.1**, and dhclient-script never replaces an existing default route → the
|
||||
sweep's hub probe rode the dead gateway and the gate oscillated. Fixed (`699325b`, flush before
|
||||
dhclient), validated by hot-patch (heal in 33 s), then the ISO was rebuilt.
|
||||
- **Round 2, virgin final ISO, zero patching:** dead-NIC install → fallback baked → **R-59 screen
|
||||
on the console** (capture committed: `SPIKE-firstboot-nic-sweep-2026-07-22-screen.png`) →
|
||||
failed-sweep restore proven safe (flushed static re-applied by ifreload) → cable move →
|
||||
**self-heal + hub registration in 23 s unaided** (persisted `inet dhcp` + winner port, original
|
||||
in `interfaces.felhom-bak`, box at a proper DHCP address, pairing banner + code on console).
|
||||
- **R-61 lockout closed, proven:** the build's `.rootpw.txt` password recomputed against the
|
||||
installed box's `/etc/shadow` root hash → exact match (computed on the build host; the
|
||||
plaintext never left it). Value NOT included here by design.
|
||||
- Validation method note: no browser on DooPlex — console proof is QEMU screendump; hub-side
|
||||
proof is the box's own registration/pairing-code delivery (endpoint-level).
|
||||
|
||||
## Artifacts
|
||||
|
||||
- Final drill ISO: `felhom-pve-9.2-1-v1.24.0-nested-probe-generic.iso`, sha256
|
||||
`c25cdbbf35d2134b0bf278c4b069a3d5849c0960cf0a5e4500e52638489c8947`, built from clean tree at
|
||||
`699325b` with an **uncommitted** key-bearing probe profile (nested-vm + ops SSH key — never
|
||||
committed, per the armed-profile rule). Its `.rootpw.txt` exists beside it (0600) — verified,
|
||||
value stored out-of-band only.
|
||||
- Shipping rebuilds for real profiles (hp/n100/generic) happen at next need; the committed
|
||||
pipeline builds them from `main` as usual.
|
||||
|
||||
## Cleanup verification
|
||||
|
||||
Nested VMs 9310/9311/9312 destroyed; throwaway `vmbr9` (+ its temporary host address) deleted;
|
||||
transferred ISO + screendumps removed from felhom-pve; `brctl show` back to exactly `enp1s0` +
|
||||
`veth9201i0` on vmbr0; **`/etc/network/interfaces` on felhom-pve diffed BYTE-IDENTICAL against
|
||||
the pre-task copy** (sha `c7ace087…`). Residue by design: **three unclaimed-appliance
|
||||
registrations at the production hub** (all three source machines destroyed) — operator may
|
||||
discard them from the Unclaimed list.
|
||||
|
||||
## Observations / follow-ups
|
||||
|
||||
- **F-P8 (out of scope, noted):** the pairing loop's register-failure line prints every ~42 s on
|
||||
a no-network box; v1.21.0 quieted the 204 wait but not this path. The gate makes it mostly moot
|
||||
(it holds before register), but the line exists for a box that loses network AFTER install
|
||||
start (state.json present → screen + retry, register never reached).
|
||||
- The console font renders Hungarian long vowels (ő/ű) approximately on the text console —
|
||||
same behavior as the existing pairing banner; acceptable.
|
||||
- `qm` screendump (`qm monitor` → `screendump`) is a serviceable no-browser console-capture
|
||||
method for nested drills — worth remembering for future runbooks.
|
||||
- R-62 (hub delete-dialog cosmetics) filed on the ROADMAP as agreed — docs-only here.
|
||||
**Not validated (needs-mutation or operator-side):** BIOS setting itself, inbox delivery,
|
||||
Cloudflare dashboard state, UI click-throughs.
|
||||
|
||||
Reference in New Issue
Block a user