felhom.eu/documentation/audits/DRILL-night-2026-09-24.md

DRILL — night 2026-09-24 (run 2026-09-24 from 11:07 CEST): the second drive brings a file app back whole; a crash loop is stopped; exact image fingerprints; automatic updates spiked; a chaos hour

Run 2026-09-24 from 11:07 CEST (daytime; the brief's clock times were applied relative to the start). Evidence: night-2026-09-24/ (PROGRESS.md is the step log). Controller v0.269.0 → v0.269.1, hub v0.123.0, catalog cf7cf84, floor 0.269.1 (MinAgent 0.131.0). Architecture read: 09 (§3 decisions 11–28, §6.4), 07 (the whole-copy table), 08 (the alarm ladder).

Not done, or changed

• Part D (move more apps) was NOT run. Its bar needs a bench machine on demo-hp that is created and destroyed tonight; this session's permission check refused host-level destroys (it refused a pct destroy), so a bench could not be torn down — a half-state the rules forbid. Part D is first in the brief's order of sacrifice. Apps moved: 0 of 0 tried. Part E's last step (Update on demo-hp 9201 the standing apps Part D moved) therefore had nothing to press.

• Two controller releases, not one — v0.269.1 fixes a defect v0.269.0's own live proof found (the sync moved an installed app's image digest, so a restart would have changed the image unguarded). Decided by CC unattended — operator may reverse. The floor went to 0.269.1.

• The crash-loop threshold is 6 restarts in 10 minutes, not the brief's 10 — measured: Docker's back-off caps a steady loop at ~1 restart/min. Decided by CC unattended — operator may reverse.

• Part E's "whole-box backup" accident was the box's own full backup run, not a vzdump: demo-hp's backup storage had ~4 GB free and its root was at 90 %.

• Part C step 1 was measured on demo-hp 9201's real night, not on 9202: 9202 has no off-site target, so its off-site leg never runs. 9202 gave the db-dump leg and the window move.

• demo-hp is damaged and needs the operator (R-672, P1): before tonight's run the agent's scheduled restore-test filled the host's thin pool; 9201's disks went READ-ONLY. The pool was freed (intervention 1); 9201 still needs stop + fsck + start, and did not take the 0.269.1 floor.

• Round 2's install and round 9's remove were broken by their accidents (R-681, R-682) — recorded, not fixed.

• A slip of mine: commit 4502af6 carried E/chaos-state.json with the passwords of three drill test accounts (wishlist, navidrome, n8n on scratch guest 9202). All three apps and their data were removed at teardown, so the accounts no longer exist; the file was deleted and ignored in f2234c0. History keeps it — not rewritten (a shared repo). The runner now keeps its state outside the evidence tree.

Interventions: 1 (an agent restart on demo-hp so its own Recover tore down the leaked restore-test guest). Apps moved: 0 of 0 tried. The result that matters most: a file app now comes back WHOLE from the second drive — proven four times, twice under a power cut or a nearly full disk — and nothing the household had was deleted or overwritten.

A1 — the whole restore from the second drive (decision 26, R-661) — PROVEN LIVE

• Spike (A1/00-spike*): the Tier-2 mirror keeps a file app's drive files as plain files under hdd/<rel> and userdata/<rel> — readable file by file. R-668 found on the way: the Tier-2 target was a removed folder on the app's own disk (the same-disk check failed open) — fixed, fails closed.
• File rules, tested (tier2_whole_test.go): the whole tree fingerprinted before and after; never delete, never overwrite a newer live file, bring back the missing, keep an older replaced file beside. Red-proofs: redproofs/A1-file-rules.txt, A1-r668.txt.
• Live, 9202 (A1/21-readback.json): restored 1, replaced 0, kept-newer 1, unchanged 137; 3/3 volumes, 1/1 database; 38 s; the account, the deleted file (back), the edited file (the household's newer copy kept) and an untouched file read back. From a hold (A1/30-*): the page named „második meghajtó … a beállításokat, az adatbázist és a fájlokat" / "second drive … the settings, the database and the files"; its restore cleared the hold in 33.8 s. Chaos rounds 3 and 11 repeated it under a power cut and a disk 1 GB above the floor.

A2 — keep-data-only Remove while support is informed (decision 27, R-666) — PROVEN LIVE

One-drive hold (the second-drive mirror moved aside): page hu „…A Felhom ügyfélszolgálatát értesítettük — addig ne indítsd újra, és ne töröld az adatait." / en "…Felhom support has been told — until then, do not restart the app or delete its data."; keep_data_only: true; Remove with data or with backups → 409 hu „Amíg az ügyfélszolgálat foglalkozik az alkalmazással, az adatai nem törölhetők — az alkalmazást az adatai megtartásával eltávolíthatod." / en "While support is working on this app, its data cannot be deleted — you can remove the app and keep its data."; the app stayed. Positive control: a hold WITH a whole copy → keep_data_only absent (A2/10-*). Found (R-669, proven): that hold was reached because the undo judged the right old version with the FAILED step's probe, left by the previous hold + restore.

A3 — the box stops a crash loop / OOM storm (decision 28, R-667) — PROVEN LIVE

gokapi (a real crash loop): stopped at „crash_loop — 6 in 10m0s", trip 1, page hu/en + Start, no „restore needed" badge (A3/21); Start lifted it, the fresh container restarted 9 times in 32 s, stopped again: „Az alkalmazást ismét leállítottuk … A Felhom ügyfélszolgálatát értesítettük." / "We stopped the app again … Felhom support has been told." (A3/30). Detector red-proofed (redproofs/A3-r667-detector.txt). Hub v0.123.0 routes app_stopped_unhealthy to the household and the operator (red-proofed, A3-hub.txt).

A4 — each step judged by its own file (R-664, R-665); R-662 removed — shipped, red-proofed

redproofs/A4-*. Live through Part C (three apps climbed with their own step files).

Part B — exact image fingerprints on the box — PROVEN LIVE (on v0.269.1)

• Compose accepts tag@digest for every template with a tested digest: 25 apps, 37 digests, 0 missing, 0 parse differences (B/01).
• A floating tag (redis:7-alpine) re-tested at a new digest: badge „Frissítés elérhető" (hu/en), Update pulled exactly that digest, badge „Naprakész", records digest-free (B/10, B/21).
• Found and fixed: v0.269.0's sync wrote the NEW digest into the RUNNING app's compose before Update — a restart would have changed the image with no backup and no undo. v0.269.1 CarryDigests keeps an installed app's digest; red-proofed (redproofs/B-sync-carry.txt); re-proven live (B/21: "the sync kept the running digest: True").

Part C — automatic updates, SPIKE — the build brief is 09 §6.4.2

Chain today: legs clock-scheduled, nothing waits. 9201's real night: off-site 04:15:05 → 04:18:33 (3 m 28 s), gate opens 04:30 → ~11 min for an update leg (decision 20's wait is required). "Off-site finished" signal: only the persisted offbox.LastStatus+LastRun (offbox.go:1003-1125), not written on five early returns — build: chain, do not poll. Gate interlock: an Options func beside WindowStartFn in quiesce.runOnce. Simulated night on 9202 (C/31): romm 3 steps 77/71/94 s; wishlist 56 s; navidrome 11/9 s; vikunja's failing step undone in 104 s (90 s drill timeout; ≈ 6–11 min at the 5 min default) and set aside; total 7 m 18 s. Found: stale steps-left after done (R-678), Update on a current app runs a full update (R-679), the box does not remember a failed step (R-680). Default of the per-box switch: ON (decision 12).

Part D — not run (see "Not done")

Part E — the schedule (committed before round 1)

Drawn by night-2026-09-24/tools/chaos_schedule.py, seed 20260924 (constraints: every action at least once; controller killed at most 3 times, ≥ 20 min apart; Start-after-unhealthy after a stop; Remove-on-held after a hold). Guest 9202 (scratch), controller v0.269.1, drill catalog.

seed 20260924, 12 rounds

round	action	accident
1	oom_app	power_cut
2	install	docker_restart
3	cutoff_undo_file_app	power_cut
4	failing_step	controller_killed
5	remove_held	none
6	failing_step	whole_box_backup
7	crash_loop	whole_box_backup
8	oom_app	disk_to_floor_plus_1g
9	remove_held	controller_killed
10	ladder_step	docker_restart
11	cutoff_undo_file_app	disk_to_floor_plus_1g
12	start_after_unhealthy	none

The app each round acts on — fixed here, not drawn (an app's state after round N decides what N+1 can do):

round	app	why
1 oom_app	chaosoom (drill-only: alpine, 32M cap, a 200 MB eater in a loop)	out-of-memory storm → the box must stop it
2 install	n8n at its ladder's FIRST from (2 steps behind), seeded through its front door	gives round 10 a step
3 cutoff_undo_file_app	nextcloud (the file app), a failing step (redis 7-alpine → 7.4.0-alpine, probe 8999), the NEWEST undo copy's marker cut at verifying → the hold must name the second drive → its whole restore, then the seeds + files read back	decision 26 under a power cut
4 failing_step	wishlist v0.67.1 → :latest with probe 8999 → undo	the undo under a controller kill
5 remove_held	gokapi (held since A3: the box stopped its crash loop) — Remove with data	Remove on a held app
6 failing_step	navidrome 0.64.1 → :latest with probe 8999 → undo	the undo under a backup run
7 crash_loop	chaoscrash (drill-only: alpine that exits after 3 s)	the box must stop it
8 oom_app	chaosoomb (a second eater)	the OOM stop under a full disk
9 remove_held	chaoscrash (held since round 7)	Remove on a held app under a controller kill
10 ladder_step	n8n, one step (2 → 1)	a good step under a docker restart
11 cutoff_undo_file_app	nextcloud again	decision 26 with the disk 1 GB above the floor (a disk refusal is a valid outcome)
12 start_after_unhealthy	chaosoomb (held since round 8) — Start → one more try → the repeat stop says support is told	decision 28's second half

Changed before round 1: the accident "a whole-box backup started on 9202" is the box's own full backup run (POST /api/backup/run, as on 2026-09-23), not a vzdump: demo-hp's backup storage had ~4 GB free and the host root was at 90 %, so a vzdump of 9202 could have filled the HOST (demo-hp also had R-672's full pool today). How accidents are fired: the runner arms the round and waits; the session fires each host-level accident as its own visible command (power cut = pct stop 9202; pct start 9202, kill -9 of the controller PID, systemctl restart docker in the guest, a fallocate to leave 3 GB free on /var/lib/docker).

The rounds

#	action × accident	app	steady	end state	data read back	events (from the box's own log)
1	oom_app × power cut	chaosoom	185 s	stopped, unhealthy_stop	—	app_oom (w), app_oom_storm (e), app_stopped_unhealthy (w)
2	install × docker restart	n8n	—	not installed, silently (R-681); reinstalled after, 211 s	—	app_deploy_started only
3	cutoff file app × power cut at verifying	nextcloud	204 s	the update RESUMED after boot → held (second drive) → whole restore → running	account ✓ files 2/2 ✓	app_update_held (e)
4	failing step × controller kill	wishlist	148 s	resumed → undone	✓	app_update_undone (w)
5	remove held × none	gokapi	27 s	removed, 0 leftovers	—	app_removed (i)
6	failing step × backup run	navidrome	110 s	undone (the backup skipped the updating app)	✓	app_update_undone (w)
7	crash loop × backup run	chaoscrash	116 s	stopped, unhealthy_stop	—	app_stopped_unhealthy (w)
8	oom_app × disk to floor + 1 GB	chaosoomb	102 s	stopped, unhealthy_stop	—	app_oom (w), app_stopped_unhealthy (w)
9	remove held × controller kill	chaoscrash	—	half-removed: 502 to the household, containers gone, still listed (R-682); a second Remove finished it	—	—
10	ladder step × docker restart	n8n	155 s	resumed → done, 2.31.3 → 2.40.5, steps 2 → 1	✓	—
11	cutoff file app × disk to floor + 1 GB	nextcloud	194 s	held (fresh 14:34 second-drive copy) → whole restore → running	account ✓ files 2/2 ✓	app_update_held (e)
12	Start after unhealthy × none	chaosoomb	125 s	one more try → stopped again, „support is told" hu/en	—	app_oom (w), app_oom_storm (e), app_stopped_unhealthy (w)

Controller kills: 2 of max 3, 20 min 32 s apart. Against 08: every stop sent app_stopped_unhealthy; every undo app_update_undone, every hold app_update_held. What should have fired and did not: round 2 (an install lost — nothing) and round 9 (a remove half-done — nothing); round 3's hold named an hour-old copy (R-683, watch). No user file was deleted or overwritten by any restore; no data was lost.

Teardown — three layers

• Machine (9202): tonight's 8 apps removed through the product (0 containers, 0 volumes, 0 undo copies); the three drive folders the product kept (R-442 on a box without drive access) removed by name; backup window back to 02:30 through the product's form; controller.yaml restored from .pre-night0924 (live catalog, no drill update: settings); controller stays on 0.269.1 (the floor). Standing before tonight and kept: privatebin, paperless-ngx, filebrowser. gokapi removed in round 5 (R-644's crash loop). F/10-12.
• Host (demo-hp): pct list 9201 + 9202 only (as at the start, minus R-672's leaked 990000); no bench was created; storage unchanged; the thin pool at 58.9 % after intervention 1. F/20.
• Hub: floor 0.269.1 with MinAgent 0.131.0, read back; N100 9201 arrived in ~16 s (managed floor SERVED … from declared); demo-hp 9201 did NOT arrive (read-only disks, R-672). Drill repo reset to live cf7cf8456f00, has_actions: false. F/13, F/30-31.

Claims in the brief that turned out wrong (or right), named

1. The Tier-2 mirror holds a file app's files in a form a restore can read file by file — TRUE (plain files under hdd/ and userdata/; the merge walked 139).
2. RestartCount survives the scans that reset restarting_since — TRUE per container run; it resets when the container is recreated (a Start after the box's stop), so the detector treats a drop as a new window.
3. 10 restarts in 10 minutes separates a crash loop from a slow start — WRONG: a steady loop runs at ~1 restart/min (gokapi 7 in 7 min) and would never reach 10; 6 chosen. No healthy first start in the drill evidence reached 6 (immich's broken first start reached 12 — R-676).
4. Docker Compose accepts tag@digest in every template's image line — TRUE (25/25). What the brief did not foresee: rendering digests in the SYNC moved a running app's image (fixed v0.269.1).
5. The chain signal can be measured on 9202 — WRONG: 9202 has no off-site target.
6. One release per repo could not hold (see "Not done").

Register

Before 335 rows / 679,393 B; after 344 rows / 685,662 B. Opened R-668…R-683 (16); closed R-661, R-662, R-664, R-665, R-666, R-667, R-668 (7, compressed into CLOSED-ITEMS).