BIGNIGHT: alarm truth table draft (Phase 3-4, F1-F4)
gates / gates (push) Successful in 21s

This commit is contained in:
2026-09-14 22:08:36 +02:00
parent 75b31c4562
commit ad8d072752
@@ -0,0 +1,35 @@
# Alarm truth table — BIGNIGHT 2026-09-14
Every alarm the hub recorded for `tester-1` tonight, and every fault with the alarm it should have raised.
Source: hub log (`alarms.sh`, CEST times), the operator mailbox and the customer mailbox (Gmail connector).
„Mail" = the hub logged `Operator email sent`. The customer mailbox (`tester1@felhom.eu`) received no alarm mail all night
(checked after F4).
## A. Alarms that fired
| CEST | event (severity) | trigger | mail | TRUE? |
|---|---|---|---|---|
| 19:59:43 | `node_recovered` (staleness down → ok) | new box reporting | operator | true |
| 20:14:35 … 20:45:42 | `app_deployed (info)` × 12 | Phase 3 deploys | — | true |
| 21:02:50 … 21:03:06 | `crossdrive_completed (info)` × 13 | Tier 2 run | — | true |
| 21:11:12 | `whole_guest_backup_failed (error)` | PBS tier of „Mentés most" (storage absent, R-511) | operator | **true** (the tier cannot succeed); says nothing of the local success |
| 21:31:25 | `controller_started (info)` | F1 power-on | — | true |
| 21:42:53 | `backup_failed (error)` „volume dump interrupted by a controller restart — 1 app(s) … restarted" | F2 | operator | **true** |
| 21:42:58 | `controller_started (info)` | F2 power-on | — | true |
| 21:54:50 | `controller_started (info)` | F3 power-on | — | true |
| 21:58:02 | `storage_disconnected (error)` „Meghajtó váratlanul leválasztva: Adatlemez" | F4 | operator | **true** |
| 21:58:15 | `app_start_failed (warning)` × 4 (Paperless-ngx, Jellyfin, Immich, Nextcloud) | F4 consequence | operator × 4 | true but redundant (R-521) |
Hub WARN lines without an event or mail: `host tester-1-a61396 backup FAILED: target=felhom-pbs … does not exist` at
21:12:43 and 21:27:43 (the agent's own PBS attempts).
## B. Faults and the alarm each should raise
| fault | should raise | fired? | verdict |
|---|---|---|---|
| Paperless OOM on 20 uploads (Phase 3) | a dead/unhealthy-worker alarm | **no** | **MISSED** (R-514) |
| Tunnel 502 for every public name (Phase 2) | tunnel has no working route | **no** | **MISSED** (R-510; box logs `Request failed` every request) |
| F1 power cut 61 s | nothing (under 30 min staleness) | `controller_started` only | correct |
| F2 power cut during backup | interrupted backup | `backup_failed (error)` | correct; customer page silent (R-519) |
| F3 power cut during update | interrupted update | `controller_started` only | nothing recorded; untestable same-version (R-520) |
| F4 drive unplugged | drive lost | `storage_disconnected (error)` (+4 redundant) | correct |