From 70f1e0173690ac7ea2e3a3a125272f89df1b006f Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Thu, 17 Sep 2026 01:15:50 +0200 Subject: [PATCH] chaos night: alarm truth table extended to rounds 1-9 Nine rounds, 8 alarms fired, 8 true, 0 missing. Two design gaps (R-547, R-549). Also records what this night will probably NOT answer: the event-drop path has never been exercised, because no event was raised during any of the three internet cuts, and rounds 10-12 draw no further cut. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS --- .../alarm-truth-table.txt | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/documentation/audits/evidence-chaos-night-2026-09-17/alarm-truth-table.txt b/documentation/audits/evidence-chaos-night-2026-09-17/alarm-truth-table.txt index 9634391e..0e9b6ad2 100644 --- a/documentation/audits/evidence-chaos-night-2026-09-17/alarm-truth-table.txt +++ b/documentation/audits/evidence-chaos-night-2026-09-17/alarm-truth-table.txt @@ -12,12 +12,24 @@ R5 | docker restarted | controller_started (info) R6 | none (control) | whole_guest_backup_failed (error) | YES (1/1) | none - and it named the TIER ("local tier"), not "the backup". | | | | No alarm for the 22 apps the backup stopped: correct, those stops are suppressed. R7 | internet cut 10m | none | n/a | none - node_stale threshold is 30m, the cut was 10m. Nothing false raised either. +R8 | internet cut 10m | none | n/a | none - node_stale threshold is 30m, the cut was 10m. BUT the accident did not do what its + | (public path only) | | | name said: the hub sits on the LAN here, so the box never lost it. Instrument fault, mine. +R9 | internet cut 10m | none | n/a | none - and this time the hub REALLY was cut. The report was built, pushed 3x over 1m40.8s, + | (hub cut too) | | | then given up. Nothing queued - correct, a report is a snapshot. Gap to next report 29m59s + | | | | against a 30m threshold: one second inside the alarm. Filed as R-549 (P2). -## Running totals, rounds 1-7 +## Running totals, rounds 1-9 alarms fired: 8 - alarms TRUE: 8 (8/8 - no false alarm all night so far) + alarms TRUE: 8 (8/8 - no false alarm in nine rounds) alarms MISSING: 0 - design gaps found: 1 (R-547: a transient full disk is never mentioned to anyone) + design gaps found: 2 (R-547: a transient full disk is never mentioned to anyone) + (R-549: one failed report push spends the whole staleness budget) + +## STILL UNMEASURED after three internet cuts, and it must be said plainly + Events pushed while the hub is unreachable are retried 3x and then DROPPED PERMANENTLY, with no + queue. That path has never been exercised, because no event happened to be raised during any cut. + Rounds 10-12 draw hard reset, drive pulled and nothing - none of them cuts the hub. So unless an + event coincides with an outage by accident, this night will not answer it. Recorded, not hidden. ## Two traps avoided, recorded so the numbers can be trusted R3: "none fired" was checked TWICE, independently, after the fill was released.