From a046db7df7cf6382dea482270549087ee3df848d Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Wed, 16 Sep 2026 23:23:04 +0200 Subject: [PATCH] CHAOS NIGHT: household verified, and a seventh error of mine found by control Verified after the repairs: 26 containers running, all twelve apps deployed, ZERO auth-failure lines on the rebuilt DB-backed apps, and 10 of 12 front doors answering 200 - including cloud (nextcloud) and share (gokapi), both of which were broken an hour ago. The cures are confirmed at the front door, not by a health badge. bookstack diagnosed properly rather than guessed at: its healthcheck exits 22 (curl's "server returned an HTTP error"), a direct request to the container returns 500, its migrations completed cleanly and it has no auth failures. So neither the database nor the image is at fault - the app itself errors. The likely cause is mine: I passed APP_KEY=base64: plus 32 random alphanumerics, which is not a base64-encoded 32-byte key. The repair decodes the stored key and prints its true length BEFORE redeploying, so the hypothesis is confirmed or refuted in the evidence. Also recorded: my sixth slip, running docker over SSH on the VM instead of inside the guest, which printed a tidy table of "absent" and "0" that read like "nothing is wrong" and was produced by a shell with no docker at all. Six of my errors tonight share one shape - a command whose precondition failed, still printing a confident answer - and the same discipline caught every one: ask the box directly, with a control. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS --- .../phase0-recovery.txt | 50 +++++++++++++++++++ .../round-1.txt | 8 +++ 2 files changed, 58 insertions(+) diff --git a/documentation/audits/evidence-chaos-night-2026-09-17/phase0-recovery.txt b/documentation/audits/evidence-chaos-night-2026-09-17/phase0-recovery.txt index cc61c6d8..933ef0ef 100644 --- a/documentation/audits/evidence-chaos-night-2026-09-17/phase0-recovery.txt +++ b/documentation/audits/evidence-chaos-night-2026-09-17/phase0-recovery.txt @@ -143,3 +143,53 @@ them can exhaust this. This check exists because the failure it guards against was not predicted — it was discovered by the box remounting itself read-only. Measuring the headroom is cheaper than meeting the wall again. + +## The household, verified after the repairs (2026-09-16 21:21Z) + containers: **26 running, 26 total** — nothing stopped, nothing restarting + the box's own record: all **twelve** apps `deployed: true` + auth-failure lines on the rebuilt DB-backed apps: + bookstack 0 · nextcloud 0 · paperless-webserver 0 · immich-server 0 + front doors from DooPlex: **10 of 12 answer 200** + paste · share · recipes · inventory · status · travel · paperless · cloud · media · vault + still 404: wiki (bookstack `unhealthy` — no traefik route) and photos (immich, seconds old) + +The two cures are both confirmed by the front door, not by a badge: `cloud` (nextcloud) went from a +crash loop to **200**, and `share` (gokapi) from `Restarting (1)` to **200**. Zero auth failures +anywhere is the other half — the regenerated-password fault is gone, not merely quieter. + +## My SIXTH slip tonight, same shape as the others +The first attempt at this verification ran `docker …` over SSH **on the VM** instead of inside the +customer guest — I dropped the `pct exec 9201 --` wrapper. Every line came back as + bash: docker: command not found + bookstack absent auth-failure lines: 0 +i.e. a tidy table of „absent" and „0" that reads exactly like „nothing is wrong". The numbers were +produced by a shell that had no docker at all. + +That is now six errors of mine tonight sharing one shape: **a command whose precondition failed, still +printing a confident answer.** („all twelve deploys ACCEPTED" · „login ok (csrf 0)" · `head -12` that +hid a disk · a `pkill` that killed itself · the JSON parser that reported 0 entries for a 29 KB ring · +and this.) The discipline that caught every one of them was the same: **ask the box directly, with a +control, before believing my own reporting layer.** + +## bookstack: not the database, not the image — the app itself returns 500 +Measured rather than assumed, because two wrong explanations had already been written tonight: + + healthcheck definition : ["CMD","curl","-f","http://127.0.0.1:80"], interval 30s, retries 3 + health status : **unhealthy**, failing streak **5** + healthcheck exit code : **22** — curl's „the server returned an HTTP error", i.e. something IS + listening and answering, with a failure status + direct request to the container, bypassing traefik: **HTTP 500** + its log : migrations run to completion („… DONE" for each) then + „[custom-init] No custom files found" / „[ls.io-init] done." + auth-failure lines : **0** — the database is fine + +So: the container is up, the database is reachable, migrations applied, and the web application +answers 500. Traefik then refuses it a route, which is why `wiki` reads 404 at the front door — the +same „no route to an unhealthy container" shape established earlier with controls. + +**The likely cause is mine, and it is checked before it is acted on:** I passed +`APP_KEY=base64:` + 32 random *alphanumerics*. Laravel expects `base64:` followed by a base64-encoded +**32-byte** key; 32 alphanumerics decode to something else entirely, and a wrong-length key is a 500 +on every request. The repair script decodes the stored key and prints its true length before +redeploying, so the hypothesis is confirmed or refuted in the evidence rather than assumed — and the +fresh deploy uses `base64:$(openssl rand -base64 32)`, which is the correct shape. diff --git a/documentation/audits/evidence-chaos-night-2026-09-17/round-1.txt b/documentation/audits/evidence-chaos-night-2026-09-17/round-1.txt index 426effca..968eee20 100644 --- a/documentation/audits/evidence-chaos-night-2026-09-17/round-1.txt +++ b/documentation/audits/evidence-chaos-night-2026-09-17/round-1.txt @@ -72,3 +72,11 @@ was written. That is the documented rebuild behaviour, surfaced honestly with th 2026-09-16T21:19:15Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files": 2026-09-16T21:19:35Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files": 2026-09-16T21:19:56Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files": + 2026-09-16T21:20:16Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files": + 2026-09-16T21:20:36Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files": + 2026-09-16T21:20:56Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files": + 2026-09-16T21:21:16Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files": + 2026-09-16T21:21:36Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files": + 2026-09-16T21:21:56Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files": + 2026-09-16T21:22:16Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files": + 2026-09-16T21:22:36Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files":