From ad8d072752da20ff6de3b8feb9d1abb3175ed775 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 14 Sep 2026 22:08:36 +0200 Subject: [PATCH] BIGNIGHT: alarm truth table draft (Phase 3-4, F1-F4) --- .../alarm-truth-table.md | 35 +++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 documentation/audits/evidence-bignight-2026-09-14/alarm-truth-table.md diff --git a/documentation/audits/evidence-bignight-2026-09-14/alarm-truth-table.md b/documentation/audits/evidence-bignight-2026-09-14/alarm-truth-table.md new file mode 100644 index 00000000..f1d05c7b --- /dev/null +++ b/documentation/audits/evidence-bignight-2026-09-14/alarm-truth-table.md @@ -0,0 +1,35 @@ +# Alarm truth table — BIGNIGHT 2026-09-14 + +Every alarm the hub recorded for `tester-1` tonight, and every fault with the alarm it should have raised. +Source: hub log (`alarms.sh`, CEST times), the operator mailbox and the customer mailbox (Gmail connector). +„Mail" = the hub logged `Operator email sent`. The customer mailbox (`tester1@felhom.eu`) received no alarm mail all night +(checked after F4). + +## A. Alarms that fired + +| CEST | event (severity) | trigger | mail | TRUE? | +|---|---|---|---|---| +| 19:59:43 | `node_recovered` (staleness down → ok) | new box reporting | operator | true | +| 20:14:35 … 20:45:42 | `app_deployed (info)` × 12 | Phase 3 deploys | — | true | +| 21:02:50 … 21:03:06 | `crossdrive_completed (info)` × 13 | Tier 2 run | — | true | +| 21:11:12 | `whole_guest_backup_failed (error)` | PBS tier of „Mentés most" (storage absent, R-511) | operator | **true** (the tier cannot succeed); says nothing of the local success | +| 21:31:25 | `controller_started (info)` | F1 power-on | — | true | +| 21:42:53 | `backup_failed (error)` „volume dump interrupted by a controller restart — 1 app(s) … restarted" | F2 | operator | **true** | +| 21:42:58 | `controller_started (info)` | F2 power-on | — | true | +| 21:54:50 | `controller_started (info)` | F3 power-on | — | true | +| 21:58:02 | `storage_disconnected (error)` „Meghajtó váratlanul leválasztva: Adatlemez" | F4 | operator | **true** | +| 21:58:15 | `app_start_failed (warning)` × 4 (Paperless-ngx, Jellyfin, Immich, Nextcloud) | F4 consequence | operator × 4 | true but redundant (R-521) | + +Hub WARN lines without an event or mail: `host tester-1-a61396 backup FAILED: target=felhom-pbs … does not exist` at +21:12:43 and 21:27:43 (the agent's own PBS attempts). + +## B. Faults and the alarm each should raise + +| fault | should raise | fired? | verdict | +|---|---|---|---| +| Paperless OOM on 20 uploads (Phase 3) | a dead/unhealthy-worker alarm | **no** | **MISSED** (R-514) | +| Tunnel 502 for every public name (Phase 2) | tunnel has no working route | **no** | **MISSED** (R-510; box logs `Request failed` every request) | +| F1 power cut 61 s | nothing (under 30 min staleness) | `controller_started` only | correct | +| F2 power cut during backup | interrupted backup | `backup_failed (error)` | correct; customer page silent (R-519) | +| F3 power cut during update | interrupted update | `controller_started` only | nothing recorded; untestable same-version (R-520) | +| F4 drive unplugged | drive lost | `storage_disconnected (error)` (+4 redundant) | correct |