From c3e1986aafcae1a005858a66169d1f48edb0dbb5 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Wed, 16 Sep 2026 23:20:27 +0200 Subject: [PATCH] CHAOS NIGHT: household repaired, headroom checked, round 2 armed The five apps my disk-full burst broke are removed with their data and redeployed cleanly, one at a time. Two distinct faults of mine, with different cures, and separating them is what made either fixable: - image layers written while the pool was full -> "invalid ELF header", exit 127; cured by dropping the image so compose re-pulls - my re-seed's FRESH database passwords over volumes initialised with the first set -> Postgres auth_failed / MariaDB "Access denied"; cured only by removing the app with its data and deploying once gokapi proves they are different: a new image left it Restarting(1), a new database made it healthy. Headroom measured so the disk-full failure cannot quietly repeat: pool 39% of 75.8G, docker filesystem 29%, data drive 1%, 4.3G guest RAM free. Round 2's script hardened before it runs unattended: its "steady" test now compares against the count it measured itself in the same round, instead of a hardcoded 24 that the rebuilt household might never reach. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS --- .../audits/DRILL-chaos-night-2026-09-17.md | 33 +++++++++++++++++++ .../phase0-recovery.txt | 19 +++++++++++ .../round-1-notes.txt | 16 +++++++++ .../round-1.txt | 8 +++++ 4 files changed, 76 insertions(+) diff --git a/documentation/audits/DRILL-chaos-night-2026-09-17.md b/documentation/audits/DRILL-chaos-night-2026-09-17.md index 2b6a502e..dbc414cb 100644 --- a/documentation/audits/DRILL-chaos-night-2026-09-17.md +++ b/documentation/audits/DRILL-chaos-night-2026-09-17.md @@ -173,6 +173,39 @@ container; identical byte-for-byte to a no-such-host control) and a crash-loopin layers corrupted while my thin pool stood at 100 %; „invalid ELF header", repaired by a re-pull). Detail in `evidence-chaos-night-2026-09-17/round-1-notes.txt`. + +### Phase 0 postscript — repairing my own damage, and three conclusions I had to retract + +Nine of the first ten deploys failed because I fired twelve at once onto a thin pool carved from a +32 GB disk, and the pool hit 100 %. Repairing that took the rest of Phase 0 and produced **two +distinct faults of mine, with different cures**, which only separating them made fixable: + +| fault | symptom | cure | +|---|---|---| +| image layers written while the pool was full | `php: … libxml2.so.2: **invalid ELF header**`, exit 127 crash loop | drop the image, let compose pull it again | +| my re-seed generated **fresh database passwords** over volumes already initialised with the first set | Postgres `auth_failed`, MariaDB „Access denied for user … (using password: YES)" | remove the app **with its data**, deploy once with one consistent secret set | + +A fresh image did not fix gokapi and a fresh database did — that is the evidence the two faults are +different things rather than one. + +**Three conclusions I wrote and then had to retract, each corrected where it stood:** +1. „the five 404s were my mistimed sweep" — wrong for four of them. A **negative control** (a + no-such-host request) returned the identical 404 of 19 bytes, and a positive control returned + 200/1200 bytes: traefik simply has **no route to an unhealthy container**. +2. „nextcloud is repaired" — wrong. The re-pull fixed the crash, and the app still could not reach its + database. The container reported **`healthy`** throughout, because the image's own healthcheck asks + whether Apache answers, not whether the application works. +3. „all the broken apps are corrupt layers" — wrong. bookstack logged a clean startup, gokapi logged + nothing at all, immich showed a Postgres auth failure. + +I also nearly filed a defect against the drive gate, which was working and logging at DEBUG while I +read a settings snapshot inside its 30-second tick. **An absent log line is not evidence.** + +None of this is a product defect and none of it is filed as one. What the product did throughout was +correct and legible: it refused to route to unhealthy containers, `app_start_failed` named the app +that was down, `backup_run_failures` said „1 of 12 … nextcloud", the memory guard refused the +eleventh and twelfth installs with both numbers quoted, and `storage_fill_critical` fired at 100 %. + ### Rounds 2-12 PENDING diff --git a/documentation/audits/evidence-chaos-night-2026-09-17/phase0-recovery.txt b/documentation/audits/evidence-chaos-night-2026-09-17/phase0-recovery.txt index c307f685..cc61c6d8 100644 --- a/documentation/audits/evidence-chaos-night-2026-09-17/phase0-recovery.txt +++ b/documentation/audits/evidence-chaos-night-2026-09-17/phase0-recovery.txt @@ -124,3 +124,22 @@ The bar appeared while the off-site copy was paused, told the household exactly disappeared **for good** when they did it — on a box that installed itself from the published ISO, with no one setting the scene for the test. „védi"/„védené" are both 0 on /backups/apps for now because no class-A app is installed yet; that sentence is checked again once they are. + +## Headroom before the rounds — so the disk-full failure cannot quietly repeat +Measured after all the repairs and re-pulls (2026-09-16 ~21:20Z): + + LVM thin pool `data` 75.81 g, **39.15 %** used (was 11.80 g at 100 % when it wedged) + guest / 32 G, 944 M used, 4 % + guest /var/lib/docker 69 G, 19 G used, **29 %** (all twelve apps' images) + data drive /mnt/felhom-drives/hdd_1 98 G, 361 M used, 1 % + guest memory 6144 MB total, **4373 MB available** + drill host /mnt/hdd_1 938 G, 57 G used, 7 % (834 G free) + +Every number that mattered when the box wedged now has room: the pool that filled is at two fifths, +the docker filesystem that held the corrupt layers is at under a third, and the guest has more than +4 GB of memory free with all twelve apps running. The eleven remaining rounds include image pulls +(none, as it happens, since the catalog bump was not pushed) and repeated app restarts, and none of +them can exhaust this. + +This check exists because the failure it guards against was not predicted — it was discovered by the +box remounting itself read-only. Measuring the headroom is cheaper than meeting the wall again. diff --git a/documentation/audits/evidence-chaos-night-2026-09-17/round-1-notes.txt b/documentation/audits/evidence-chaos-night-2026-09-17/round-1-notes.txt index 7d2282f5..94d1bb9a 100644 --- a/documentation/audits/evidence-chaos-night-2026-09-17/round-1-notes.txt +++ b/documentation/audits/evidence-chaos-night-2026-09-17/round-1-notes.txt @@ -180,3 +180,19 @@ spacing a 23:45 start still reaches round 12 by about 04:20. The queue behind the current job: **nextcloud, immich and bookstack** get the same clean remove-and-redeploy. They are done one at a time, never two repair jobs at once on the same box — concurrent stop/start on one controller is what produced half of tonight's confusion already. + +## The diagnosis is confirmed by the cure +gokapi had already had a fresh image pulled and was STILL `Restarting (1)`. After a clean removal +(with data) and a fresh deploy with one consistent set of secrets: + + gokapi **Up 47 seconds (healthy)** + paperless-ngx INSTALLED at 2026-09-16T21:17:15Z + paperless-postgres Up 19 seconds (healthy) + paperless-redis Up 19 seconds (healthy) + +A new image did not fix it and a new database did — which is the difference between the corrupt-layer +fault (nextcloud's exit 127) and the regenerated-password fault (everything else). Two faults, both +mine, with different cures, and it took separating them to fix either. + +Household loop through all of this: unit active, **4 sampled apps since the classifier fix, 0 +failures** — the box kept serving its other apps while five were being torn down and rebuilt. diff --git a/documentation/audits/evidence-chaos-night-2026-09-17/round-1.txt b/documentation/audits/evidence-chaos-night-2026-09-17/round-1.txt index dcde7539..426effca 100644 --- a/documentation/audits/evidence-chaos-night-2026-09-17/round-1.txt +++ b/documentation/audits/evidence-chaos-night-2026-09-17/round-1.txt @@ -64,3 +64,11 @@ was written. That is the documented rebuild behaviour, surfaced honestly with th 2026-09-16T21:16:35Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files": 2026-09-16T21:16:55Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files": 2026-09-16T21:17:15Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files": + 2026-09-16T21:17:35Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files": + 2026-09-16T21:17:55Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files": + 2026-09-16T21:18:15Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files": + 2026-09-16T21:18:35Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files": + 2026-09-16T21:18:55Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files": + 2026-09-16T21:19:15Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files": + 2026-09-16T21:19:35Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files": + 2026-09-16T21:19:56Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files":