night 2026-09-23: the chaos schedule (seed 20260923) and its app table, written before round 1
gates / gates (push) Successful in 26s
gates / gates (push) Successful in 26s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
@@ -0,0 +1,114 @@
|
||||
# DRILL — night shift 2026-09-23: DooPlex guarded from its own tests, the test record, the proven moves, the chaos hour
|
||||
|
||||
Evidence: `audits/night-2026-09-23/` (`PROGRESS.md` is the step log). Architecture read first and named:
|
||||
`architecture/09-update-architecture.md` (§3 decisions 11–20, §6.1a, §6.4 parts 4–6, the verdict record),
|
||||
`07-backup-architecture.md` §6, `08-alarm-ladder.md`. **Method: endpoint-level** — every product act is
|
||||
the endpoint the UI invokes; pages are fetched as HTML in both languages; no browser.
|
||||
|
||||
## Not done, or changed
|
||||
|
||||
<!-- FILLED AT THE END -->
|
||||
|
||||
## Three lines
|
||||
|
||||
<!-- FILLED AT THE END -->
|
||||
|
||||
---
|
||||
|
||||
## Part A — controller v0.267.0 (`80e6ad8c4772`, CI job 919 success)
|
||||
|
||||
| row | what shipped | red-proof (seen failing, then restored) | evidence |
|
||||
|---|---|---|---|
|
||||
| **R-650** | `internal/dockerexec`: every docker exec (77 sites) goes through it; under `go test` a real docker is refused with an error naming the command, unless `FELHOM_TEST_REAL_DOCKER=1` or the binary is a stub under the temp dir. `TestR650_NoBareDockerExec` sweeps the tree. | guard disabled → the decoy's `docker ps -a` runs (`got <nil>`); one bare call put back → the sweep names `dbdump.go:140` | `A1-r650-*.txt` |
|
||||
| R-650 sweep | **8 `api` tests FAILED on the refusal** — they built a real `stacks.Manager` and ran `docker ps` on DooPlex; `web` ran 67× `docker compose version`, 19× `docker ps`, 19× `docker info`, 10× `docker inspect felhom-samba`, 2× `docker exec felhom-samba`; `stacks` ran **1× `docker-compose down`**, 11× `compose ps`; `backup` 46× `docker ps`; `appexport` 7×; `system` 1×. `api`/`stacks`/`web` now run under `RunWithStub` (TestMain); the other three pass on the refusal. | — | `A1-r650-sweep.txt` |
|
||||
| **R-640** | `appbackup.CheckDumpComplete` (the engine's end marker, last 4 KiB, gzip-aware). The unit restore and the off-site restore refuse a cut-off copy **before the first mutation**; every replay checks again before any load, whatever the import seam is. | replay check removed → `a cut-off copy was LOADED`; unit gate removed → `stopped=true calls=[stop recreate startsvc:immich-postgres start]`; off-site gate removed → only the replay check caught it, after a stop and a rollback | `A2-r640-redproof.txt` |
|
||||
| **R-626** | **measured, not reproduced** on v0.266.0: navidrome deployed and removed through the product; `docker events` 390 s — the remove's kill/stop/die/destroy seen (the positive observable), **no create**; controller restarted at +150 s; guest rebooted after → 0 containers, 0 volumes at every check. My script first said `came_back: true` — it had counted the catalog-mirror directory every catalog app has; corrected in the verdict file, both values kept. Leftover found: `applied-compose.yml` + `applied-meta/` stay after a remove (R-651). | — | `A3-r626-*` |
|
||||
| **R-499** | the Tier-2 page's system-disk sentence has four branches (own drive / same disk / drive gone / cannot ask); „(PBS)" and „nincs külön teendő" only where true. Live on 9202 (no agent): the `unknown` branch in hu + en, matching `/api/storage/backup-target` `known:false`. | handler not passing the fact → 3 branches missing; the old sentence in the same-disk branch → `promises the backup protects this app` | `A4-*`, `A7-*` |
|
||||
| **R-518** | the backup button states the measured stop (≈ 8 min on a 12-app box), both languages, page + confirm. **The brief's „csak néhány másodpercre" was already gone** (v0.243.0); the vague „általában néhány perc" is replaced. | bundles back to v0.243.0 → measured downtime appears 0 times | `A5-*` |
|
||||
|
||||
Gates: `go build/vet` rc 0; `go test -count=1 ./...` rc 0 (31 packages); `controller_gates.py` all OK.
|
||||
Deployed to **9202 only** (`A6-*`); the floor waits for Part E.
|
||||
|
||||
## Part B — the test record and its gate (catalog `6db08a5`, CI job 920 success)
|
||||
|
||||
**Spike first (15 min):** a drill navidrome carrying a top-level `update_ladder:` (one JSON entry per
|
||||
line) on 9202 at **v0.266.0 and v0.267.0**: synced, deploy-fields read, deployed `running`, probe
|
||||
`API GET :4533/ping → 200`, badge „Naprakész" / "Up to date", no YAML error. Source agrees (no
|
||||
`KnownFields` anywhere). The brief's claim holds. `B1-spike-*`.
|
||||
|
||||
**The record** (`scripts/ladder.py`): `from`/`to` per service, `digest` per `to` ref (sha256 from
|
||||
`scripts/image_digest.py` — stdlib, equals Docker's `RepoDigests` for `privatebin/pdo:2.0.6` on 9202),
|
||||
`verdict` (proven | unrecorded), `tested_at`, `harness_version`, `evidence`, `box_evidence`,
|
||||
`memory_peak_pct` (+ `memory_basis`, `memory_cgroup_peak_pct`), `marks` {files_may_change, needs_person,
|
||||
memory_tight}. Written ONLY by `upgrade-test.py --write-ladder` (bench AND box `proven`, template at
|
||||
FROM, digests resolved) — `test_ladder_writer.py` 5/5.
|
||||
|
||||
**The gate** — two rows of `catalog_gates.py`, both `--fast`:
|
||||
`check-test-record.py` (static; runs in CI too: a ladder well-formed, continuous, and its newest step IS
|
||||
the compose) and `check-test-record-move.py` (history + the registry for MOVED refs only: a move adds a
|
||||
PROVEN, non-backfilled entry whose `from`/`to` are the compose before/after and whose digests the
|
||||
registry still serves; `memory_tight` needs the limit raised in the same range). 16 decoy cases, both
|
||||
directions. **Red-proofs:** the no-entry refusal removed → *a bare image move* and *the entry only in
|
||||
README* both pass (rc 0, expected 1); `failed` allowed → both failed-verdict cases pass; the digest
|
||||
comparison removed → *the registry now serves another digest* passes. `B2-gate-redproofs.txt`.
|
||||
Live, on the first real move (romm): the gate judged it against the real registry (rc 0), and with a
|
||||
wrong digest table it REFUSED all three services.
|
||||
|
||||
**Backfill** — the 21 moves of 2026-09-22, each from the record its commit cited:
|
||||
|
||||
<!-- BACKFILL TABLE -->
|
||||
|
||||
## Part C — move every app that proves itself
|
||||
|
||||
<!-- PART C -->
|
||||
|
||||
## Part D — the chaos hour
|
||||
|
||||
**Written BEFORE round 1 (committed with this section).** Guest 9202, controller v0.267.0, drill catalog,
|
||||
`update.health_timeout: 90s` (the drill setting — the product default is 5 min). Schedule drawn once by
|
||||
`night-2026-09-23/chaos_schedule.py` from seed **20260923** (constraints in its docstring):
|
||||
|
||||
seed 20260923, drawn in 3 attempt(s)
|
||||
|
||||
| round | action | accident | at +s |
|
||||
|---|---|---|---|
|
||||
| 1 | install | none | 11 |
|
||||
| 2 | good_update | box_backup | 46 |
|
||||
| 3 | restore | controller_kill | 28 |
|
||||
| 4 | fail_health | memory_hog | 43 |
|
||||
| 5 | install | disk_fill | 28 |
|
||||
| 6 | good_update | memory_hog | 46 |
|
||||
| 7 | restore | power_cut | 56 |
|
||||
| 8 | cutoff | box_backup | 23 |
|
||||
| 9 | fail_health | controller_kill | 58 |
|
||||
| 10 | remove | docker_restart | 10 |
|
||||
| 11 | cutoff | two_updates | 51 |
|
||||
| 12 | remove | none | 47 |
|
||||
|
||||
**The app each round acts on** (fixed by rule in `chaos.py`, not drawn):
|
||||
|
||||
| round | app | why this app |
|
||||
|---|---|---|
|
||||
| 1 install / 10 remove | actualbudget | a throwaway beside the six |
|
||||
| 5 install / 12 remove | papra | a second throwaway |
|
||||
| 2 good_update | docmost 0.95.0 → 0.96.0 | PostgreSQL, the BIG database (300 extra spaces) |
|
||||
| 6 good_update | romm 5.3.0 → 5.3.1 | MariaDB |
|
||||
| 3 restore | vikunja | volume-only (SQLite + files) |
|
||||
| 7 restore | docmost | the big database, under a power cut |
|
||||
| 4 fail_health | adventurelog v0.12.1 → v0.13.0, probe on a port it does not answer | the second PostgreSQL (PostGIS) |
|
||||
| 9 fail_health | vikunja 2.5.0 → 2.6.0, wrong probe | volume-only, under a controller kill |
|
||||
| 8 cutoff | navidrome 0.64.0 → 0.64.1, wrong probe, the undo copy's finished-marker removed during `verifying` | volume-only + a drive folder |
|
||||
| 11 cutoff + two_updates | nextcloud 34.0.1 → 34.0.4 (the file-leg), and adventurelog's Update pressed at +51 s | two updates at once |
|
||||
|
||||
Setup: the six at their FROM pins, each seeded through its front door (A); one whole-box backup on 9202
|
||||
(the restore rounds need a copy); seed B on docmost, romm and vikunja after it.
|
||||
|
||||
<!-- PART D ROUNDS -->
|
||||
|
||||
## Teardown
|
||||
|
||||
<!-- TEARDOWN -->
|
||||
|
||||
## Claims in the brief that turned out wrong, or held
|
||||
|
||||
<!-- CLAIMS -->
|
||||
@@ -9,3 +9,4 @@ emby 4.11.0.1->4.11.0.3 moved in catalog 8d573ad at 21:55
|
||||
kimai-db mariadb 11.6->11.8 moved in catalog 376b2c3 at 21:59
|
||||
romm-db mariadb 11.4->11.8 moved in catalog 431cdec at 22:00
|
||||
immich-machine-learning v3.0.3->v3.2.2 moved in catalog b82b7c6 at 22:01
|
||||
22:04 9202 images removed by name before Part D (54 test/old images, no prune): /var/lib/docker 86% -> 24%
|
||||
|
||||
Reference in New Issue
Block a user