CHAOS NIGHT: household verified, and a seventh error of mine found by control

Verified after the repairs: 26 containers running, all twelve apps deployed,
ZERO auth-failure lines on the rebuilt DB-backed apps, and 10 of 12 front doors
answering 200 - including cloud (nextcloud) and share (gokapi), both of which
were broken an hour ago. The cures are confirmed at the front door, not by a
health badge.

bookstack diagnosed properly rather than guessed at: its healthcheck exits 22
(curl's "server returned an HTTP error"), a direct request to the container
returns 500, its migrations completed cleanly and it has no auth failures. So
neither the database nor the image is at fault - the app itself errors. The
likely cause is mine: I passed APP_KEY=base64: plus 32 random alphanumerics,
which is not a base64-encoded 32-byte key. The repair decodes the stored key
and prints its true length BEFORE redeploying, so the hypothesis is confirmed
or refuted in the evidence.

Also recorded: my sixth slip, running docker over SSH on the VM instead of
inside the guest, which printed a tidy table of "absent" and "0" that read like
"nothing is wrong" and was produced by a shell with no docker at all. Six of my
errors tonight share one shape - a command whose precondition failed, still
printing a confident answer - and the same discipline caught every one: ask the
box directly, with a control.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS

documentation/audits/evidence-chaos-night-2026-09-17/phase0-recovery.txt

@@ -143,3 +143,53 @@ them can exhaust this.
 
 This check exists because the failure it guards against was not predicted — it was discovered by the
 box remounting itself read-only. Measuring the headroom is cheaper than meeting the wall again.
+
+## The household, verified after the repairs (2026-09-16 21:21Z)
+    containers: **26 running, 26 total** — nothing stopped, nothing restarting
+    the box's own record: all **twelve** apps `deployed: true`
+    auth-failure lines on the rebuilt DB-backed apps:
+        bookstack 0 · nextcloud 0 · paperless-webserver 0 · immich-server 0
+    front doors from DooPlex: **10 of 12 answer 200**
+        paste · share · recipes · inventory · status · travel · paperless · cloud · media · vault
+        still 404: wiki (bookstack `unhealthy` — no traefik route) and photos (immich, seconds old)
+
+The two cures are both confirmed by the front door, not by a badge: `cloud` (nextcloud) went from a
+crash loop to **200**, and `share` (gokapi) from `Restarting (1)` to **200**. Zero auth failures
+anywhere is the other half — the regenerated-password fault is gone, not merely quieter.
+
+## My SIXTH slip tonight, same shape as the others
+The first attempt at this verification ran `docker …` over SSH **on the VM** instead of inside the
+customer guest — I dropped the `pct exec 9201 --` wrapper. Every line came back as
+    bash: docker: command not found
+    bookstack   absent   auth-failure lines: 0
+i.e. a tidy table of „absent" and „0" that reads exactly like „nothing is wrong". The numbers were
+produced by a shell that had no docker at all.
+
+That is now six errors of mine tonight sharing one shape: **a command whose precondition failed, still
+printing a confident answer.** („all twelve deploys ACCEPTED" · „login ok (csrf 0)" · `head -12` that
+hid a disk · a `pkill` that killed itself · the JSON parser that reported 0 entries for a 29 KB ring ·
+and this.) The discipline that caught every one of them was the same: **ask the box directly, with a
+control, before believing my own reporting layer.**
+
+## bookstack: not the database, not the image — the app itself returns 500
+Measured rather than assumed, because two wrong explanations had already been written tonight:
+
+    healthcheck definition : ["CMD","curl","-f","http://127.0.0.1:80"], interval 30s, retries 3
+    health status          : **unhealthy**, failing streak **5**
+    healthcheck exit code  : **22** — curl's „the server returned an HTTP error", i.e. something IS
+                             listening and answering, with a failure status
+    direct request to the container, bypassing traefik: **HTTP 500**
+    its log                : migrations run to completion („… DONE" for each) then
+                             „[custom-init] No custom files found" / „[ls.io-init] done."
+    auth-failure lines     : **0** — the database is fine
+
+So: the container is up, the database is reachable, migrations applied, and the web application
+answers 500. Traefik then refuses it a route, which is why `wiki` reads 404 at the front door — the
+same „no route to an unhealthy container" shape established earlier with controls.
+
+**The likely cause is mine, and it is checked before it is acted on:** I passed
+`APP_KEY=base64:` + 32 random *alphanumerics*. Laravel expects `base64:` followed by a base64-encoded
+**32-byte** key; 32 alphanumerics decode to something else entirely, and a wrong-length key is a 500
+on every request. The repair script decodes the stored key and prints its true length before
+redeploying, so the hypothesis is confirmed or refuted in the evidence rather than assumed — and the
+fresh deploy uses `base64:$(openssl rand -base64 32)`, which is the correct shape.

documentation/audits/evidence-chaos-night-2026-09-17/round-1.txt

@@ -72,3 +72,11 @@ was written. That is the documented rebuild behaviour, surfaced honestly with th
    2026-09-16T21:19:15Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files":
    2026-09-16T21:19:35Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files":
    2026-09-16T21:19:56Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files":
+   2026-09-16T21:20:16Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files":
+   2026-09-16T21:20:36Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files":
+   2026-09-16T21:20:56Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files":
+   2026-09-16T21:21:16Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files":
+   2026-09-16T21:21:36Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files":
+   2026-09-16T21:21:56Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files":
+   2026-09-16T21:22:16Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files":
+   2026-09-16T21:22:36Z {"last_duration":"1m45s","last_error":"","last_run":"2026-09-16T21:09:00Z","orphaned":true,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files":