This commit is contained in:
@@ -0,0 +1,35 @@
|
||||
# Alarm truth table — BIGNIGHT 2026-09-14
|
||||
|
||||
Every alarm the hub recorded for `tester-1` tonight, and every fault with the alarm it should have raised.
|
||||
Source: hub log (`alarms.sh`, CEST times), the operator mailbox and the customer mailbox (Gmail connector).
|
||||
„Mail" = the hub logged `Operator email sent`. The customer mailbox (`tester1@felhom.eu`) received no alarm mail all night
|
||||
(checked after F4).
|
||||
|
||||
## A. Alarms that fired
|
||||
|
||||
| CEST | event (severity) | trigger | mail | TRUE? |
|
||||
|---|---|---|---|---|
|
||||
| 19:59:43 | `node_recovered` (staleness down → ok) | new box reporting | operator | true |
|
||||
| 20:14:35 … 20:45:42 | `app_deployed (info)` × 12 | Phase 3 deploys | — | true |
|
||||
| 21:02:50 … 21:03:06 | `crossdrive_completed (info)` × 13 | Tier 2 run | — | true |
|
||||
| 21:11:12 | `whole_guest_backup_failed (error)` | PBS tier of „Mentés most" (storage absent, R-511) | operator | **true** (the tier cannot succeed); says nothing of the local success |
|
||||
| 21:31:25 | `controller_started (info)` | F1 power-on | — | true |
|
||||
| 21:42:53 | `backup_failed (error)` „volume dump interrupted by a controller restart — 1 app(s) … restarted" | F2 | operator | **true** |
|
||||
| 21:42:58 | `controller_started (info)` | F2 power-on | — | true |
|
||||
| 21:54:50 | `controller_started (info)` | F3 power-on | — | true |
|
||||
| 21:58:02 | `storage_disconnected (error)` „Meghajtó váratlanul leválasztva: Adatlemez" | F4 | operator | **true** |
|
||||
| 21:58:15 | `app_start_failed (warning)` × 4 (Paperless-ngx, Jellyfin, Immich, Nextcloud) | F4 consequence | operator × 4 | true but redundant (R-521) |
|
||||
|
||||
Hub WARN lines without an event or mail: `host tester-1-a61396 backup FAILED: target=felhom-pbs … does not exist` at
|
||||
21:12:43 and 21:27:43 (the agent's own PBS attempts).
|
||||
|
||||
## B. Faults and the alarm each should raise
|
||||
|
||||
| fault | should raise | fired? | verdict |
|
||||
|---|---|---|---|
|
||||
| Paperless OOM on 20 uploads (Phase 3) | a dead/unhealthy-worker alarm | **no** | **MISSED** (R-514) |
|
||||
| Tunnel 502 for every public name (Phase 2) | tunnel has no working route | **no** | **MISSED** (R-510; box logs `Request failed` every request) |
|
||||
| F1 power cut 61 s | nothing (under 30 min staleness) | `controller_started` only | correct |
|
||||
| F2 power cut during backup | interrupted backup | `backup_failed (error)` | correct; customer page silent (R-519) |
|
||||
| F3 power cut during update | interrupted update | `controller_started` only | nothing recorded; untestable same-version (R-520) |
|
||||
| F4 drive unplugged | drive lost | `storage_disconnected (error)` (+4 redundant) | correct |
|
||||
Reference in New Issue
Block a user