From cf8dec84807d57ee46926323984c45cf43043dd5 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Wed, 23 Sep 2026 22:04:37 +0200 Subject: [PATCH] night 2026-09-23: the chaos schedule (seed 20260923) and its app table, written before round 1 Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS --- .../audits/DRILL-night-2026-09-23.md | 114 ++++++++++++++++++ .../audits/night-2026-09-23/C-moves.txt | 1 + 2 files changed, 115 insertions(+) create mode 100644 documentation/audits/DRILL-night-2026-09-23.md diff --git a/documentation/audits/DRILL-night-2026-09-23.md b/documentation/audits/DRILL-night-2026-09-23.md new file mode 100644 index 00000000..149052f3 --- /dev/null +++ b/documentation/audits/DRILL-night-2026-09-23.md @@ -0,0 +1,114 @@ +# DRILL — night shift 2026-09-23: DooPlex guarded from its own tests, the test record, the proven moves, the chaos hour + +Evidence: `audits/night-2026-09-23/` (`PROGRESS.md` is the step log). Architecture read first and named: +`architecture/09-update-architecture.md` (§3 decisions 11–20, §6.1a, §6.4 parts 4–6, the verdict record), +`07-backup-architecture.md` §6, `08-alarm-ladder.md`. **Method: endpoint-level** — every product act is +the endpoint the UI invokes; pages are fetched as HTML in both languages; no browser. + +## Not done, or changed + + + +## Three lines + + + +--- + +## Part A — controller v0.267.0 (`80e6ad8c4772`, CI job 919 success) + +| row | what shipped | red-proof (seen failing, then restored) | evidence | +|---|---|---|---| +| **R-650** | `internal/dockerexec`: every docker exec (77 sites) goes through it; under `go test` a real docker is refused with an error naming the command, unless `FELHOM_TEST_REAL_DOCKER=1` or the binary is a stub under the temp dir. `TestR650_NoBareDockerExec` sweeps the tree. | guard disabled → the decoy's `docker ps -a` runs (`got `); one bare call put back → the sweep names `dbdump.go:140` | `A1-r650-*.txt` | +| R-650 sweep | **8 `api` tests FAILED on the refusal** — they built a real `stacks.Manager` and ran `docker ps` on DooPlex; `web` ran 67× `docker compose version`, 19× `docker ps`, 19× `docker info`, 10× `docker inspect felhom-samba`, 2× `docker exec felhom-samba`; `stacks` ran **1× `docker-compose down`**, 11× `compose ps`; `backup` 46× `docker ps`; `appexport` 7×; `system` 1×. `api`/`stacks`/`web` now run under `RunWithStub` (TestMain); the other three pass on the refusal. | — | `A1-r650-sweep.txt` | +| **R-640** | `appbackup.CheckDumpComplete` (the engine's end marker, last 4 KiB, gzip-aware). The unit restore and the off-site restore refuse a cut-off copy **before the first mutation**; every replay checks again before any load, whatever the import seam is. | replay check removed → `a cut-off copy was LOADED`; unit gate removed → `stopped=true calls=[stop recreate startsvc:immich-postgres start]`; off-site gate removed → only the replay check caught it, after a stop and a rollback | `A2-r640-redproof.txt` | +| **R-626** | **measured, not reproduced** on v0.266.0: navidrome deployed and removed through the product; `docker events` 390 s — the remove's kill/stop/die/destroy seen (the positive observable), **no create**; controller restarted at +150 s; guest rebooted after → 0 containers, 0 volumes at every check. My script first said `came_back: true` — it had counted the catalog-mirror directory every catalog app has; corrected in the verdict file, both values kept. Leftover found: `applied-compose.yml` + `applied-meta/` stay after a remove (R-651). | — | `A3-r626-*` | +| **R-499** | the Tier-2 page's system-disk sentence has four branches (own drive / same disk / drive gone / cannot ask); „(PBS)" and „nincs külön teendő" only where true. Live on 9202 (no agent): the `unknown` branch in hu + en, matching `/api/storage/backup-target` `known:false`. | handler not passing the fact → 3 branches missing; the old sentence in the same-disk branch → `promises the backup protects this app` | `A4-*`, `A7-*` | +| **R-518** | the backup button states the measured stop (≈ 8 min on a 12-app box), both languages, page + confirm. **The brief's „csak néhány másodpercre" was already gone** (v0.243.0); the vague „általában néhány perc" is replaced. | bundles back to v0.243.0 → measured downtime appears 0 times | `A5-*` | + +Gates: `go build/vet` rc 0; `go test -count=1 ./...` rc 0 (31 packages); `controller_gates.py` all OK. +Deployed to **9202 only** (`A6-*`); the floor waits for Part E. + +## Part B — the test record and its gate (catalog `6db08a5`, CI job 920 success) + +**Spike first (15 min):** a drill navidrome carrying a top-level `update_ladder:` (one JSON entry per +line) on 9202 at **v0.266.0 and v0.267.0**: synced, deploy-fields read, deployed `running`, probe +`API GET :4533/ping → 200`, badge „Naprakész" / "Up to date", no YAML error. Source agrees (no +`KnownFields` anywhere). The brief's claim holds. `B1-spike-*`. + +**The record** (`scripts/ladder.py`): `from`/`to` per service, `digest` per `to` ref (sha256 from +`scripts/image_digest.py` — stdlib, equals Docker's `RepoDigests` for `privatebin/pdo:2.0.6` on 9202), +`verdict` (proven | unrecorded), `tested_at`, `harness_version`, `evidence`, `box_evidence`, +`memory_peak_pct` (+ `memory_basis`, `memory_cgroup_peak_pct`), `marks` {files_may_change, needs_person, +memory_tight}. Written ONLY by `upgrade-test.py --write-ladder` (bench AND box `proven`, template at +FROM, digests resolved) — `test_ladder_writer.py` 5/5. + +**The gate** — two rows of `catalog_gates.py`, both `--fast`: +`check-test-record.py` (static; runs in CI too: a ladder well-formed, continuous, and its newest step IS +the compose) and `check-test-record-move.py` (history + the registry for MOVED refs only: a move adds a +PROVEN, non-backfilled entry whose `from`/`to` are the compose before/after and whose digests the +registry still serves; `memory_tight` needs the limit raised in the same range). 16 decoy cases, both +directions. **Red-proofs:** the no-entry refusal removed → *a bare image move* and *the entry only in +README* both pass (rc 0, expected 1); `failed` allowed → both failed-verdict cases pass; the digest +comparison removed → *the registry now serves another digest* passes. `B2-gate-redproofs.txt`. +Live, on the first real move (romm): the gate judged it against the real registry (rc 0), and with a +wrong digest table it REFUSED all three services. + +**Backfill** — the 21 moves of 2026-09-22, each from the record its commit cited: + + + +## Part C — move every app that proves itself + + + +## Part D — the chaos hour + +**Written BEFORE round 1 (committed with this section).** Guest 9202, controller v0.267.0, drill catalog, +`update.health_timeout: 90s` (the drill setting — the product default is 5 min). Schedule drawn once by +`night-2026-09-23/chaos_schedule.py` from seed **20260923** (constraints in its docstring): + +seed 20260923, drawn in 3 attempt(s) + +| round | action | accident | at +s | +|---|---|---|---| +| 1 | install | none | 11 | +| 2 | good_update | box_backup | 46 | +| 3 | restore | controller_kill | 28 | +| 4 | fail_health | memory_hog | 43 | +| 5 | install | disk_fill | 28 | +| 6 | good_update | memory_hog | 46 | +| 7 | restore | power_cut | 56 | +| 8 | cutoff | box_backup | 23 | +| 9 | fail_health | controller_kill | 58 | +| 10 | remove | docker_restart | 10 | +| 11 | cutoff | two_updates | 51 | +| 12 | remove | none | 47 | + +**The app each round acts on** (fixed by rule in `chaos.py`, not drawn): + +| round | app | why this app | +|---|---|---| +| 1 install / 10 remove | actualbudget | a throwaway beside the six | +| 5 install / 12 remove | papra | a second throwaway | +| 2 good_update | docmost 0.95.0 → 0.96.0 | PostgreSQL, the BIG database (300 extra spaces) | +| 6 good_update | romm 5.3.0 → 5.3.1 | MariaDB | +| 3 restore | vikunja | volume-only (SQLite + files) | +| 7 restore | docmost | the big database, under a power cut | +| 4 fail_health | adventurelog v0.12.1 → v0.13.0, probe on a port it does not answer | the second PostgreSQL (PostGIS) | +| 9 fail_health | vikunja 2.5.0 → 2.6.0, wrong probe | volume-only, under a controller kill | +| 8 cutoff | navidrome 0.64.0 → 0.64.1, wrong probe, the undo copy's finished-marker removed during `verifying` | volume-only + a drive folder | +| 11 cutoff + two_updates | nextcloud 34.0.1 → 34.0.4 (the file-leg), and adventurelog's Update pressed at +51 s | two updates at once | + +Setup: the six at their FROM pins, each seeded through its front door (A); one whole-box backup on 9202 +(the restore rounds need a copy); seed B on docmost, romm and vikunja after it. + + + +## Teardown + + + +## Claims in the brief that turned out wrong, or held + + diff --git a/documentation/audits/night-2026-09-23/C-moves.txt b/documentation/audits/night-2026-09-23/C-moves.txt index ff8bd230..af5d252e 100644 --- a/documentation/audits/night-2026-09-23/C-moves.txt +++ b/documentation/audits/night-2026-09-23/C-moves.txt @@ -9,3 +9,4 @@ emby 4.11.0.1->4.11.0.3 moved in catalog 8d573ad at 21:55 kimai-db mariadb 11.6->11.8 moved in catalog 376b2c3 at 21:59 romm-db mariadb 11.4->11.8 moved in catalog 431cdec at 22:00 immich-machine-learning v3.0.3->v3.2.2 moved in catalog b82b7c6 at 22:01 +22:04 9202 images removed by name before Part D (54 test/old images, no prune): /var/lib/docker 86% -> 24%