# D53 — a held app reaches the operator: build and live proof (2026-10-26 evening)

**Design:** `architecture/08-alarm-ladder.md` §10. **Ruling:** D53 = operator-only reminders first (`09` §3 decision 265).
**Code (commit only, not released):** controller `5bd03ad` (CI 2377 ✓), hub `7e57e824` (CI 2378 ✓).

## Red proofs

- Controller `internal/holdreport`: with an inert `Tick`, all six behaviour tests failed; with the scheduler line
  commented out, the wiring test failed.
- Hub: six mutations, each turned its test red — `redproof-hub.txt` (one first attempt was only a compile error; it was
  redone with a compiling mutation, the last line).

## Live proof — method

No hub deploy is allowed in this run, so the hub side ran as a **test hub process** on DooPlex (`192.168.0.180:18090`),
built from a clean clone at `7e57e824`, scratch database, a throwaway login, its own random global key, operator mail to
`d53-test@felhom.eu` (the catch-all), and `alerting.held_app_grace_override: 3m` (the test configuration). Scratch guest
9202 ran a **local-only** controller image built from `5bd03ad` (`0.346.0-d53test`, `docker save`/`load`, never pushed to
the registry), pointed at the test hub for the test. Pages were read through the test hub's own login (the endpoints
the operator's browser uses); the mail was read in the mailbox (a different channel from the hub log).

## Live proof — what happened (UTC)

| Time | What | Evidence |
|---|---|---|
| 17:28 | Both lists empty: „No app is held on this box." / „… on any box." | `live/0-before_*.html` |
| 17:29:52 | Test controller up; „Notifier enabled (hub: …:18090)", job `hold-report` every 1m | `live/9202-controller-log.txt` |
| 17:30:52 | The first tick reported **four old holds 9202 has carried since 2026-09-24** (`chaoscrash`, `chaosoom`, `chaosoomb`, `gokapi`) | `live/test-hub.log` |
| 17:33:35 | Four operator reminders (one per app, per-app cooldown) | mailbox, `live/notification_log.txt` |
| 17:35:41 | Throwaway Radicale installed through the product | — |
| 17:36:18–17:37:48 | Seven `kill -9` of its main process, 15 s apart | — |
| 17:37:52 | The guard stopped it: „crash_loop — 7 in 10m0s; STOPPING it"; hold `unhealthy_stop` | `live/9202-controller-log.txt` |
| 17:38:52 | `app_held` radicale reached the hub (next tick) | `live/test-hub.log` |
| 17:41:34 | One reminder after the 3-minute grace; mail „[Felhom] ⚠️ demo-hp: the app radicale is still held — a person must act" in the inbox at 17:41:35 | mailbox |
| 17:42 | Both pages list 5 held apps, Radicale with its reason, sentence, and who may clear it | `live/1-held_*.html` |
| 17:45:37 | The household's Start (`POST /api/stacks/radicale/start`) lifted the hold | controller log |
| 17:45:52 | `app_hold_cleared` → the Radicale row went from both pages; no reminder after it | `live/2-cleared_*.html`, `live/held_apps_final.txt` |

**No household mail:** every reminder's customer leg is `skipped operator_only` (`live/notification_log.txt`).

## Teardown

- **Machine (9202):** Radicale removed through the product (no container, volume, `app.yaml`, data or subdomain left);
  config and image file put back byte-identical to the saved copies; the test's two settings traces (`reported_holds`, the
  Radicale stop time) removed while the controller was stopped; the test image removed; controller 0.343.0 healthy, hub off,
  exactly as found. The four old holds and an old PrivateBin record (below) were there before and stay.
- **Host (demo-hp):** nothing provisioned; the transfer file removed.
- **Hub:** the real hub was not touched. The test hub process stopped; its database, login, key and binary deleted. The
  local test image on DooPlex removed.

## Observations

- FILED: R-1298 — 9202 carries leftovers made outside the product: four crash-loop holds since 2026-09-24 for apps that
  are not installed, and a PrivateBin `app.yaml` (2026-10-25) with no container, so the product reads it as installed.
  Once D53 is released, such orphan holds would be reminded weekly about apps that do not exist.
- NOT-A-FINDING: the test hub's log looked cut mid-line at 18:45:52 local — the `sec` wrapper holds back an unfinished
  part of a long-running child's output; the full line was there when the process ended.
- NOT-A-FINDING: my first switch of 9202 used `systemctl start` on the bootstrap oneshot, which was still active, so the
  controller was briefly not running (about one minute); `restart` is the right verb. Fixed in the run.

Secrets: the evidence was scanned for the Resend key, the hub token, the report key and 9202's password — 0 hits; the
test hub's CSRF tokens are redacted.