felhom.eu/documentation/audits/d53-held-apps-2026-10-26

D53 — a held app reaches the operator: build and live proof (2026-10-26 evening)

Design: architecture/08-alarm-ladder.md §10. Ruling: D53 = operator-only reminders first (09 §3 decision 265). Code (commit only, not released): controller 5bd03ad (CI 2377 ✓), hub 7e57e824 (CI 2378 ✓).

Red proofs

• Controller internal/holdreport: with an inert Tick, all six behaviour tests failed; with the scheduler line commented out, the wiring test failed.
• Hub: six mutations, each turned its test red — redproof-hub.txt (one first attempt was only a compile error; it was redone with a compiling mutation, the last line).

Live proof — method

No hub deploy is allowed in this run, so the hub side ran as a test hub process on DooPlex (192.168.0.180:18090), built from a clean clone at 7e57e824, scratch database, a throwaway login, its own random global key, operator mail to d53-test@felhom.eu (the catch-all), and alerting.held_app_grace_override: 3m (the test configuration). Scratch guest 9202 ran a local-only controller image built from 5bd03ad (0.346.0-d53test, docker save/load, never pushed to the registry), pointed at the test hub for the test. Pages were read through the test hub's own login (the endpoints the operator's browser uses); the mail was read in the mailbox (a different channel from the hub log).

Live proof — what happened (UTC)

Time	What	Evidence
17:28	Both lists empty: „No app is held on this box." / „… on any box."	live/0-before_*.html
17:29:52	Test controller up; „Notifier enabled (hub: …:18090)", job hold-report every 1m	live/9202-controller-log.txt
17:30:52	The first tick reported four old holds 9202 has carried since 2026-09-24 (chaoscrash, chaosoom, chaosoomb, gokapi)	live/test-hub.log
17:33:35	Four operator reminders (one per app, per-app cooldown)	mailbox, live/notification_log.txt
17:35:41	Throwaway Radicale installed through the product	—
17:36:18–17:37:48	Seven kill -9 of its main process, 15 s apart	—
17:37:52	The guard stopped it: „crash_loop — 7 in 10m0s; STOPPING it"; hold unhealthy_stop	live/9202-controller-log.txt
17:38:52	app_held radicale reached the hub (next tick)	live/test-hub.log
17:41:34	One reminder after the 3-minute grace; mail „[Felhom] ⚠️ demo-hp: the app radicale is still held — a person must act" in the inbox at 17:41:35	mailbox
17:42	Both pages list 5 held apps, Radicale with its reason, sentence, and who may clear it	live/1-held_*.html
17:45:37	The household's Start (POST /api/stacks/radicale/start) lifted the hold	controller log
17:45:52	app_hold_cleared → the Radicale row went from both pages; no reminder after it	live/2-cleared_*.html, live/held_apps_final.txt

No household mail: every reminder's customer leg is skipped operator_only (live/notification_log.txt).

Teardown

• Machine (9202): Radicale removed through the product (no container, volume, app.yaml, data or subdomain left); config and image file put back byte-identical to the saved copies; the test's two settings traces (reported_holds, the Radicale stop time) removed while the controller was stopped; the test image removed; controller 0.343.0 healthy, hub off, exactly as found. The four old holds and an old PrivateBin record (below) were there before and stay.
• Host (demo-hp): nothing provisioned; the transfer file removed.
• Hub: the real hub was not touched. The test hub process stopped; its database, login, key and binary deleted. The local test image on DooPlex removed.

Observations

• FILED: R-1298 — 9202 carries leftovers made outside the product: four crash-loop holds since 2026-09-24 for apps that are not installed, and a PrivateBin app.yaml (2026-10-25) with no container, so the product reads it as installed. Once D53 is released, such orphan holds would be reminded weekly about apps that do not exist.
• NOT-A-FINDING: the test hub's log looked cut mid-line at 18:45:52 local — the sec wrapper holds back an unfinished part of a long-running child's output; the full line was there when the process ended.
• NOT-A-FINDING: my first switch of 9202 used systemctl start on the bootstrap oneshot, which was still active, so the controller was briefly not running (about one minute); restart is the right verb. Fixed in the run.

Secrets: the evidence was scanned for the Resend key, the hub token, the report key and 9202's password — 0 hits; the test hub's CSRF tokens are redacted.