From e45fb5e37f680eba682a3d27fc6c5b5b24aafe07 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Thu, 17 Sep 2026 00:28:57 +0200 Subject: [PATCH] chaos night: draft alarm truth table for rounds 1-7 Built from the verdicts recorded in each round's write-up. 8 alarms fired, 8 true, 0 missing. The one design gap (a transient full disk is never mentioned to anyone) is already filed as R-547. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS --- .../alarm-truth-table.txt | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 documentation/audits/evidence-chaos-night-2026-09-17/alarm-truth-table.txt diff --git a/documentation/audits/evidence-chaos-night-2026-09-17/alarm-truth-table.txt b/documentation/audits/evidence-chaos-night-2026-09-17/alarm-truth-table.txt new file mode 100644 index 00000000..9634391e --- /dev/null +++ b/documentation/audits/evidence-chaos-night-2026-09-17/alarm-truth-table.txt @@ -0,0 +1,25 @@ +# Alarm truth table — CHAOS NIGHT 2026-09-17 (draft, rounds 1-7; 8-12 appended as they close) +# Built from the verdicts recorded in each round's write-up, not from a fresh grep. +# Columns: round | accident | alarms that FIRED | all true? | any that SHOULD have fired and did not + +R1 | none (control) | app_start_failed; backup_run_failures; offbox_repo_orphaned | YES (3/3) | none +R2 | power cut mid-restore | controller_started (info) | YES (1/1) | none - 60s outage is under both thresholds (node_stale 30m, boot grace 90s) +R3 | system disk 96% / 10m | none | n/a | none MISSING, but see note: disk_critical is defined at >=95% and never fired, because + | | | | the fill-watch is a DAILY sweep (03:30) + one check ~90s after a controller start. + | | | | Predicted BEFORE the round, confirmed after. Filed as R-547 (P3), not a false alarm. +R4 | tunnel killed 10m | health_critical (error) -> health_recovered (info) | YES (2/2) | none +R5 | docker restarted | controller_started (info) | YES (1/1) | none - 90s boot grace covers the app restarts +R6 | none (control) | whole_guest_backup_failed (error) | YES (1/1) | none - and it named the TIER ("local tier"), not "the backup". + | | | | No alarm for the 22 apps the backup stopped: correct, those stops are suppressed. +R7 | internet cut 10m | none | n/a | none - node_stale threshold is 30m, the cut was 10m. Nothing false raised either. + +## Running totals, rounds 1-7 + alarms fired: 8 + alarms TRUE: 8 (8/8 - no false alarm all night so far) + alarms MISSING: 0 + design gaps found: 1 (R-547: a transient full disk is never mentioned to anyone) + +## Two traps avoided, recorded so the numbers can be trusted + R3: "none fired" was checked TWICE, independently, after the fill was released. + R5: the first alarm snapshot was taken 6 s after the controller start. An alarm cannot be + called missing by a measurement taken before it could possibly have fired. Re-measured.