# DRILL — R-329 + R-386: the alarm that fired but reached nobody, and the stop nobody heard (2026-08-23)

**Controller v0.222.0 → v0.223.0. Hub v0.106.0 → v0.107.0. Live leg on `demo-hp` (Tier 0), guest
9201. UNATTENDED.** Method: endpoint-level plus the hub's own SQLite records — no browser exists on
DooPlex. Guest and hub clocks are UTC; the hub pod logs CEST.

## Verdict

| Part | Outcome |
|---|---|
| 1.1 the severity word + sweep | ✅ **exactly one** bad severity in the whole controller |
| 1.2 the AST contract guard | ✅ and it found two dynamic sites the hand sweep missed |
| 1.3 customer toggle, default OFF | ✅ |
| 2 the hub says what it rewrote | ✅ hub v0.107.0, proven live |
| 3 ask the field that knows | ✅ proven live, both directions |
| 4 split the compound toggles | ✅ round trip byte-identical |
| 5 record the notification philosophy | ✅ recorded, **not implemented** |

**No halt condition fired.**

## The one number that says it all

Read from the live hub DB:

```
91 app_start_failed events stored, all-time
 0 notification_log rows before 2026-08-23 09:00   <- not one, ever, on any channel
```

After the fix, at 09:27:51: **one row — `warning` / `sent` / `operator`.**

## The two live pairs

**Scenario A** — a database dies, customer has not opted in:

```
events         : demo-hp  app_start_failed  warning  2026-08-23 09:27:51   <- v0.223.0
                 demo-hp  app_start_failed  info     2026-08-23 05:30:14   <- v0.222.0, coerced
notification_log: demo-hp  app_start_failed  warning  sent  operator  09:27:51
                 (no customer row)
```

**Scenario D** — an app stopped out of band, intent `running`:

```
2026/08/23 05:47:44  [deadapp] check alive: 40 scans since boot, 8 deployed app(s) evaluated, 0 currently down   <- v0.222.0
2026/08/23 09:34:51  [deadapp] check alive: 20 scans since boot, 8 deployed app(s) evaluated, 1 currently down   <- v0.223.0
```

Alarm fired **24 seconds** after the `docker compose stop`.

## Three things that had to be re-run, and why that matters

1. **Red-proof 5 passed first time — the mutation was INERT.** Changing `if next <= prev` to
   `if next < prev` in fillwatch does nothing, because an earlier `if next == prev { continue }` had
   already removed the equal case. The test was right to pass. Removing the guard outright convicts
   it. **Check the mutation applied before believing either verdict.**
2. **Scenario G was silently refused TWICE behind an HTTP 200.** The empty-email wipe guard declines
   the save and renders an error page — still `200`. The first run had no prefs at all; the second
   read the email with a single-line grep from a `<input>` that spans **three lines**, got `""`, and
   was refused again. Both times the before/after hashes matched — *because nothing was saved*, not
   because nothing changed. Fixed by asserting the refusal banner is **absent**. **A warning beside a
   success is read as a success.**
3. **The live Scenario A does NOT prove the customer gate**, and is not claimed to. `demo-hp` has no
   `customer_notifications` row at all, so the customer leg could not have delivered regardless. The
   toggle gate is proven by the unit tests, which configure prefs both ways. Stated rather than
   implied.

## Evidence index (`evidence/`)

| File | What it shows |
|---|---|
| `redproof-1-R329-emitter.txt` | severity back to `"warn"` → AST guard names file, line and value |
| `redproof-2-R386-intent.txt` | intent test reverted → `dead-app banner = []`, the live symptom |
| `redproof-3-part4-migration.txt` | no-op-save guard removed → the `defaults` case reorders |
| `redproof-4-R387-ingest.txt` | hub WARN removed → "the hub rewrote a severity and said nothing" |
| `redproof-5-fillwatch-consequence.txt` | the inert first attempt, and the effective one |
| `live-01`…`live-04` | Scenario A: stop, controller send, **hub records**, customer-leg control |
| `live-05-step4-absent-intent-count.txt` | **0 of 8** deployed apps carry an absent intent |
| `live-07`…`live-09` | Scenario D: out-of-band stop, alarm in 24 s, the heartbeat pair |
| `live-10`, `live-11` | Scenario C: UI Stop → intent `running`→`stopped`, 0 alarms / 9 scans |
| `live-12`, `live-13` | Scenario E: intent removed → suppressed **and** the log line names the app |
| `live-16-scenarioG-roundtrip.txt` | all three attempts, ending byte-identical (`10840f3a…`) |
| `live-18-golden-bake.txt` | golden 0.223.0 markers, each counted |
| `live-19-scenarioH-after.txt` | the new hub WARN line, live, with a silent `error` control |
| `live-06`, `live-14` | full controller-log windows (1041 and 4044 lines), pulled before each revert |

## Observations — noticed, recorded, NOT acted on

1. **The operator cooldown has no app identifier, and it now bites.** PrivateBin's event 4 minutes
   after BookStack's was logged `suppressed — operator cooldown 1h, key=demo-hp:app_start_failed`. So
   **only the first app-down per hour e-mails the operator.** This is R-182's known cooldown-key
   shape; it was harmless while the event was undeliverable and is not any more. Recorded, not fixed.
2. **The prompt's §6 premise was wrong and is corrected:** the hub's manifest **is** in version
   control, at `felhom.eu/manifests/hub.yaml:128`, and ArgoCD app `felhom` tracks
   `admin/felhom.eu.git` path `manifests`. No out-of-git deployment path exists.
3. **The golden-bake runbook still lacks `pveam update`** — second bake in a row to hit the stale
   index on the `virgin` snapshot, presenting as `400 … no such template`.
4. `internal/notify/notifier.go` carries **pre-existing** gofmt drift in a const block, confirmed by
   stashing this session's work. Not touched (§12).

## Teardown

Nothing provisioned. Every app restarted and confirmed healthy (17 containers). `privatebin`'s
`app.yaml` restored from its backup and the backup removed; its intent reads `running` again.
`demo-hp`'s notification settings restored to `enabled_events: null`, no e-mail — the state they were
in before the drill. The drill VM is powered off, its Gitea token shredded, and `drill.qcow2` reverted
to `virgin`. **Hub-side: two probe events (`backup_failed`, "R-387 scenario H probe"/"control") were
POSTed to the live hub for Scenario H and remain as event rows for customer `demo-hp`.** They are
inert records; named here rather than left for someone to find.