docs(R-329/R-386/R-387): the severity contract, the intent ruling, and Part 5 recorded

The alarm ladder gains the severity contract (the hub's vocabulary is exact, it coerces silently, and three things now hold it) and the intent test with its three-way ruling on unknown. Both marked [DESIGN] with the live measurements. Part 5 is RECORDED AND NOT IMPLEMENTED: the operator's notification philosophy, verbatim, marked plainly as direction rather than current behaviour, with the 12 -> 15 toggle growth as the argument. Filed as R-388, a product decision. R-329 and R-386 compressed into CLOSED-ITEMS with their rules kept and the full-text commit named. R-387 filed closed - including WHY the dispatcher branch was kept rather than deleted, which is evidence (three monitor checkers call ProcessEvent directly) and not caution. The drill record names three things that had to be re-run: an inert red-proof mutation, Scenario G refused twice behind an HTTP 200, and the live Scenario A NOT proving the customer gate because demo-hp has no prefs row at all. Register: OPEN 328325 -> 328132 B, CLOSED 71441 -> 74642 B.
2026-08-23 12:03:49 +02:00
parent 68a9f5475c
commit 2f7c9a6ce5
33 changed files with 5774 additions and 104 deletions
@@ -0,0 +1,110 @@
+# DRILL — R-329 + R-386: the alarm that fired but reached nobody, and the stop nobody heard (2026-08-23)
+
+**Controller v0.222.0 → v0.223.0. Hub v0.106.0 → v0.107.0. Live leg on `demo-hp` (Tier 0), guest
+9201. UNATTENDED.** Method: endpoint-level plus the hub's own SQLite records — no browser exists on
+DooPlex. Guest and hub clocks are UTC; the hub pod logs CEST.
+
+## Verdict
+
+| Part | Outcome |
+|---|---|
+| 1.1 the severity word + sweep | ✅ **exactly one** bad severity in the whole controller |
+| 1.2 the AST contract guard | ✅ and it found two dynamic sites the hand sweep missed |
+| 1.3 customer toggle, default OFF | ✅ |
+| 2 the hub says what it rewrote | ✅ hub v0.107.0, proven live |
+| 3 ask the field that knows | ✅ proven live, both directions |
+| 4 split the compound toggles | ✅ round trip byte-identical |
+| 5 record the notification philosophy | ✅ recorded, **not implemented** |
+
+**No halt condition fired.**
+
+## The one number that says it all
+
+Read from the live hub DB:
+
+```
+91 app_start_failed events stored, all-time
+ 0 notification_log rows before 2026-08-23 09:00   <- not one, ever, on any channel
+```
+
+After the fix, at 09:27:51: **one row — `warning` / `sent` / `operator`.**
+
+## The two live pairs
+
+**Scenario A** — a database dies, customer has not opted in:
+
+```
+events         : demo-hp  app_start_failed  warning  2026-08-23 09:27:51   <- v0.223.0
+                 demo-hp  app_start_failed  info     2026-08-23 05:30:14   <- v0.222.0, coerced
+notification_log: demo-hp  app_start_failed  warning  sent  operator  09:27:51
+                 (no customer row)
+```
+
+**Scenario D** — an app stopped out of band, intent `running`:
+
+```
+2026/08/23 05:47:44  [deadapp] check alive: 40 scans since boot, 8 deployed app(s) evaluated, 0 currently down   <- v0.222.0
+2026/08/23 09:34:51  [deadapp] check alive: 20 scans since boot, 8 deployed app(s) evaluated, 1 currently down   <- v0.223.0
+```
+
+Alarm fired **24 seconds** after the `docker compose stop`.
+
+## Three things that had to be re-run, and why that matters
+
+1. **Red-proof 5 passed first time — the mutation was INERT.** Changing `if next <= prev` to
+   `if next < prev` in fillwatch does nothing, because an earlier `if next == prev { continue }` had
+   already removed the equal case. The test was right to pass. Removing the guard outright convicts
+   it. **Check the mutation applied before believing either verdict.**
+2. **Scenario G was silently refused TWICE behind an HTTP 200.** The empty-email wipe guard declines
+   the save and renders an error page — still `200`. The first run had no prefs at all; the second
+   read the email with a single-line grep from a `<input>` that spans **three lines**, got `""`, and
+   was refused again. Both times the before/after hashes matched — *because nothing was saved*, not
+   because nothing changed. Fixed by asserting the refusal banner is **absent**. **A warning beside a
+   success is read as a success.**
+3. **The live Scenario A does NOT prove the customer gate**, and is not claimed to. `demo-hp` has no
+   `customer_notifications` row at all, so the customer leg could not have delivered regardless. The
+   toggle gate is proven by the unit tests, which configure prefs both ways. Stated rather than
+   implied.
+
+## Evidence index (`evidence/`)
+
+| File | What it shows |
+|---|---|
+| `redproof-1-R329-emitter.txt` | severity back to `"warn"` → AST guard names file, line and value |
+| `redproof-2-R386-intent.txt` | intent test reverted → `dead-app banner = []`, the live symptom |
+| `redproof-3-part4-migration.txt` | no-op-save guard removed → the `defaults` case reorders |
+| `redproof-4-R387-ingest.txt` | hub WARN removed → "the hub rewrote a severity and said nothing" |
+| `redproof-5-fillwatch-consequence.txt` | the inert first attempt, and the effective one |
+| `live-01`…`live-04` | Scenario A: stop, controller send, **hub records**, customer-leg control |
+| `live-05-step4-absent-intent-count.txt` | **0 of 8** deployed apps carry an absent intent |
+| `live-07`…`live-09` | Scenario D: out-of-band stop, alarm in 24 s, the heartbeat pair |
+| `live-10`, `live-11` | Scenario C: UI Stop → intent `running`→`stopped`, 0 alarms / 9 scans |
+| `live-12`, `live-13` | Scenario E: intent removed → suppressed **and** the log line names the app |
+| `live-16-scenarioG-roundtrip.txt` | all three attempts, ending byte-identical (`10840f3a…`) |
+| `live-18-golden-bake.txt` | golden 0.223.0 markers, each counted |
+| `live-19-scenarioH-after.txt` | the new hub WARN line, live, with a silent `error` control |
+| `live-06`, `live-14` | full controller-log windows (1041 and 4044 lines), pulled before each revert |
+
+## Observations — noticed, recorded, NOT acted on
+
+1. **The operator cooldown has no app identifier, and it now bites.** PrivateBin's event 4 minutes
+   after BookStack's was logged `suppressed — operator cooldown 1h, key=demo-hp:app_start_failed`. So
+   **only the first app-down per hour e-mails the operator.** This is R-182's known cooldown-key
+   shape; it was harmless while the event was undeliverable and is not any more. Recorded, not fixed.
+2. **The prompt's §6 premise was wrong and is corrected:** the hub's manifest **is** in version
+   control, at `felhom.eu/manifests/hub.yaml:128`, and ArgoCD app `felhom` tracks
+   `admin/felhom.eu.git` path `manifests`. No out-of-git deployment path exists.
+3. **The golden-bake runbook still lacks `pveam update`** — second bake in a row to hit the stale
+   index on the `virgin` snapshot, presenting as `400 … no such template`.
+4. `internal/notify/notifier.go` carries **pre-existing** gofmt drift in a const block, confirmed by
+   stashing this session's work. Not touched (§12).
+
+## Teardown
+
+Nothing provisioned. Every app restarted and confirmed healthy (17 containers). `privatebin`'s
+`app.yaml` restored from its backup and the backup removed; its intent reads `running` again.
+`demo-hp`'s notification settings restored to `enabled_events: null`, no e-mail — the state they were
+in before the drill. The drill VM is powered off, its Gitea token shredded, and `drill.qcow2` reverted
+to `virgin`. **Hub-side: two probe events (`backup_failed`, "R-387 scenario H probe"/"control") were
+POSTed to the live hub for Scenario H and remain as event rows for customer `demo-hp`.** They are
+inert records; named here rather than left for someone to find.
@@ -0,0 +1,10 @@
+=== LIVE WALK PRE-STATE (controller v0.223.0) ===
+2026-08-23T09:25:42Z
+gitea.dooplex.hu/admin/felhom-controller:0.223.0 Up About a minute (healthy)
+
+docmost	Up 4 hours (healthy)
+docmost-postgres	Up 4 hours (healthy)
+docmost-redis	Up 4 hours (healthy)
+privatebin	Up 4 hours (healthy)
+bookstack	Up 7 hours (healthy)
+bookstack-db	Up 4 hours (healthy)
@@ -0,0 +1,6 @@
+=== SCENARIO A — an app's database dies; customer has NOT opted in ===
+waiting out the 90s dead-app boot grace first...
+T0=2026-08-23T09:27:27Z
+2026-08-23T09:27:32Z
+bookstack	Up 7 hours (healthy)
+bookstack-db	Exited (0) 5 seconds ago
@@ -0,0 +1,4 @@
+-- the CONTROLLER only proves it SENT (quoted for the severity word) --
+2026/08/23 09:27:51 notifier.go:206: [DEBUG] PushEvent: type=app_start_failed severity=warning url=https://hub.felhom.eu/api/v1/event
+2026/08/23 09:27:51 notifier.go:232: [DEBUG] PushEvent: app_start_failed pushed OK (HTTP 200)
+2026/08/23 09:27:51 notifier.go:234: [INFO] Event pushed: app_start_failed (warning) — Telepített alkalmazás nem fut: BookStack
@@ -0,0 +1,14 @@
+=== SCENARIO A — THE HUB'S OWN RECORDS (not the controller's) ===
+
+-- 1. the STORED event: severity as the hub filed it --
+customer_id  event_type        severity  message                                   created_at         
+-----------  ----------------  --------  ----------------------------------------  -------------------
+demo-hp      app_start_failed  warning   Telepített alkalmazás nem fut: BookStack  2026-08-23 09:27:51
+demo-hp      app_start_failed  info      Telepített alkalmazás nem fut: BookStack  2026-08-23 05:30:14
+demo-hp      app_start_failed  info      Telepített alkalmazás nem fut: BookStack  2026-08-22 21:27:18
+demo-hp      app_start_failed  info      Telepített alkalmazás nem fut: Kimai      2026-08-22 14:19:12
+
+-- 2. the NOTIFICATION LOG: which channel actually delivered --
+customer_id  event_type        severity  status  channel   created_at         
+-----------  ----------------  --------  ------  --------  -------------------
+demo-hp      app_start_failed  warning   sent    operator  2026-08-23 09:27:51
@@ -0,0 +1,26 @@
+-- 3. IS THE CUSTOMER LEG EVEN CAPABLE? (or is 'no customer row' true for the wrong reason) --
+Error: in prepare, no such table: notification_prefs
+
+-- the customer leg DOES deliver other event types for this same customer: --
+event_type           severity  status   channel   created_at         
+-------------------  --------  -------  --------  -------------------
+backup_run_failures  error     skipped  customer  2026-08-22 22:20:32
+backup_run_failures  error     skipped  customer  2026-08-22 22:17:44
+backup_run_failures  error     skipped  customer  2026-08-22 21:11:19
+backup_run_failures  error     skipped  customer  2026-08-22 16:06:21
+backup_run_failures  error     skipped  customer  2026-08-21 21:20:36
+
+-- 4. HOW MANY notification rows did app_start_failed EVER produce before today? --
+0 rows before 2026-08-23 09:00
+91 app_start_failed EVENTS stored, all-time
+-- 3b. the customer's own preferences (does app_start_failed appear?) --
+-- 3b. the customer's own preferences (does app_start_failed appear?) --
+-- HONESTY CHECK: demo-hp has NO customer_notifications row at all --
+0 prefs rows for demo-hp
+
+-- why the customer channel logs 'skipped' for other types: --
+event_type           status   reason       
+-------------------  -------  -------------
+backup_run_failures  skipped  operator_only
+backup_run_failures  skipped  operator_only
+backup_run_failures  skipped  operator_only
@@ -0,0 +1,11 @@
+=== STEP 4 — THE ABSENT-INTENT COUNT ON demo-hp (deployed apps only) ===
+  bookstack          running
+  calibre-web        running
+  docmost            running
+  kimai              running
+  opengist           running
+  paperless-ngx      running
+  privatebin         running
+  romm               running
+
+DEPLOYED=8  running=8  stopped=0  ABSENT=0
@@ -0,0 +1,8 @@
+=== SCENARIO D — an app stopped OUT OF BAND, intent recorded as Running ===
+subject: privatebin (single container, desired_state: running) — R-386's measured-silent case
+desired_state: running
+T0=2026-08-23T09:31:27Z
+2026-08-23T09:31:33Z
+privatebin	Exited (0) 5 seconds ago
+bookstack	Up 7 hours (healthy)
+bookstack-db	Up 10 seconds (healthy)
@@ -0,0 +1,6 @@
+-- SCENARIO D verdict (T0=09:31:27Z) --
+2026-08-23T09:32:41Z
+privatebin	Exited (0) About a minute ago
+
+app_start_failed events since T0: 1
+2026/08/23 09:31:51 notifier.go:234: [INFO] Event pushed: app_start_failed (warning) — Telepített alkalmazás nem fut: PrivateBin
@@ -0,0 +1,17 @@
+-- SCENARIO D: the heartbeat, new beside last session's --
+LAST SESSION (v0.222.0, R-386 open, privatebin stopped out of band):
+  2026/08/23 05:5x  [deadapp] check alive: ... 8 deployed app(s) evaluated, 0 currently down
+  (9 scans, 0 events, 0 banner lines — evidence: DRILL-r384-.../live-18-sec4-verdict.txt)
+
+NOW (v0.223.0):
+2026/08/23 09:34:51 main.go:1731: [INFO] [deadapp] check alive: 20 scans since boot, 8 deployed app(s) evaluated, 1 currently down
+
+THE PAIR, exactly as logged — same box, same fixture (privatebin stopped out of band), same 8 apps:
+  2026/08/23 05:47:44  [deadapp] check alive: 40 scans since boot, 8 deployed app(s) evaluated, 0 currently down   <- v0.222.0
+  2026/08/23 09:34:51  [deadapp] check alive: 20 scans since boot, 8 deployed app(s) evaluated, 1 currently down   <- v0.223.0
+
+-- and the hub's own record for it --
+event_type        severity  status      channel   message                                    created_at         
+----------------  --------  ----------  --------  -----------------------------------------  -------------------
+app_start_failed  warning   suppressed  operator  Telepített alkalmazás nem fut: PrivateBin  2026-08-23 09:31:51
+app_start_failed  warning   sent        operator  Telepített alkalmazás nem fut: BookStack   2026-08-23 09:27:51
@@ -0,0 +1,11 @@
+=== SCENARIO C — the customer presses Stop in the interface ===
+-- intent BEFORE --
+desired_state: running
+-- restart it first, so the Stop is a real transition --
+{"ok":true,"message":"Stack privatebin start completed"}
+ start=200
+T_STOP=2026-08-23T09:36:13Z
+{"ok":true,"message":"Stack privatebin stop completed"}
+ stop=200
+-- intent AFTER --
+desired_state: stopped
@@ -0,0 +1,7 @@
+-- SCENARIO C verdict: 4 minutes past the Stop (T=09:36:13Z), every grace elapsed --
+2026-08-23T09:40:37Z
+
+app_start_failed since the Stop : 0
+deadapp scans since the Stop    : 9
+unknown-intent log lines        : 0
+
@@ -0,0 +1,7 @@
+=== SCENARIO E — an app with NO recorded intent (the legacy/upgrade population) ===
+demo-hp has ZERO such apps (all 8 read 'running'), so one is CREATED for the measurement:
+the desired_state key is removed from privatebin's app.yaml, exactly as a box upgraded from
+before R-166 would look. Backed up and restored afterwards. No code writer was added.
+desired_state line now: [0 found]
+T0=2026-08-23T09:40:55Z
+Up 30 seconds (healthy)
@@ -0,0 +1,11 @@
+-- SCENARIO E verdict --
+2026-08-23T09:51:48Z
+app_start_failed events : 0
+deadapp scans           : 21
+
+-- is privatebin actually being EVALUATED? (a suppression nobody reaches proves nothing) --
+privatebin deployed=True state=stopped
+
+-- waiting for the heartbeat scan (deadAppScans lags the log count by the boot grace) --
+2026/08/23 09:51:56 main.go:1731: [INFO] [deadapp] check alive: 20 scans since boot, 8 deployed app(s) evaluated, 0 currently down
+2026/08/23 09:51:56 main.go:1760: [INFO] [deadapp] 1 stopped app(s) have NO recorded customer intent, so their dead-app alarm is suppressed by the unknown-intent fallback (R-386): privatebin. This closes itself as each app is started or stopped through the interface.
@@ -0,0 +1,4 @@
+=== SCENARIO G — open the settings page and Save, changing NOTHING ===
+-- stored BEFORE --
+/var/lib/felhom/docker/volumes/felhom-controller-data/_data/data/settings.json
+/var/lib/docker/volumes/felhom-controller-data/_data/data/settings.json
@@ -0,0 +1,46 @@
+=== SCENARIO G — open the settings page and Save, changing NOTHING ===
+-- stored BEFORE (sha256 + value) --
+  enabled_events: null
+  sha256        : 74234e98afe7498fb5daf1f36ac2d78a
+  email set     : False
+
+-- RENDER the page, take exactly what it ticked, POST it back unchanged --
+  render=200
+  ticked boxes  : 10
+  save=200
+
+-- stored AFTER --
+  enabled_events: null
+  sha256        : 74234e98afe7498fb5daf1f36ac2d78a
+=== SCENARIO G — REDONE ===
+First attempt was INCONCLUSIVE and is reported: demo-hp has NO notification prefs, so the
+page rendered the DEFAULTS ticked and the save was refused by the empty-email wipe guard.
+Stored bytes were unchanged — but for the wrong reason. Seeding prefs makes it a real round trip.
+
+  seed=200
+  BEFORE: ["backup_failed", "db_dump_failed", "node_down", "disk_warning", "disk_critical", "expected_backup_missed", "expected_dbdump_missed"]
+  sha   : 10840f3a95bac168f0d7c79760998138
+
+  -- now RENDER and SAVE, changing nothing --
+  render=200
+  ticked: 7  email=
+  save=200
+  AFTER : ["backup_failed", "db_dump_failed", "node_down", "disk_warning", "disk_critical", "expected_backup_missed", "expected_dbdump_missed"]
+  sha   : 10840f3a95bac168f0d7c79760998138
+=== SCENARIO G — THIRD attempt, and the first CONCLUSIVE one ===
+Attempt 2 also refused: the email <input> spans THREE LINES, so a single-line grep read it as
+empty and the wipe guard declined the save — while still answering 200. A warning beside a
+success reads as a success; the sha matching meant NOTHING was saved, not that nothing changed.
+
+  BEFORE: ["backup_failed", "db_dump_failed", "node_down", "disk_warning", "disk_critical", "expected_backup_missed", "expected_dbdump_missed"]
+  sha   : 10840f3a95bac168f0d7c79760998138
+  render=200
+  email read : [drill@felhom.eu]   ticked: 7
+  HTTP=200
+  refusal banner present? : 0  (0 = a REAL save)
+  AFTER : ["backup_failed", "db_dump_failed", "node_down", "disk_warning", "disk_critical", "expected_backup_missed", "expected_dbdump_missed"]
+  sha   : 10840f3a95bac168f0d7c79760998138
+-- restoring demo-hp's notification settings to their pre-drill state (no email, no events) --
+  restore=200
+  enabled_events: null
+  email set     : False
@@ -0,0 +1,6 @@
+=== SCENARIO H — the hub receives an unknown severity ===
+NOTE: the LIVE hub is still v0.106.0 (the manifest bump is not synced yet), so this run is the
+BEFORE picture. It is repeated after the hub deploy.
+
+-- hub version now --
+gitea.dooplex.hu/admin/felhom-hub:0.106.0
@@ -0,0 +1,11 @@
+=== GOLDEN 0.223.0 — acceptance markers, each counted ===
+docker OK (overlay2   : 1
+including mount point : 2
+upload OK (HTTP 201)  : 1
+excluding (must be 0) : 0
+FATAL     (must be 0) : 0
+  docker OK (overlay2; data-root /var/lib/docker)
+GOLDEN_VERSION=0.223.0
+GOLDEN_SHA256=9eaf39ac39219b42ec9e6cbf890275febcdcc6f53325fe0c0f591d3431044f17
+-- round trip from the published URL --
+  ranged GET http=206
@@ -0,0 +1,9 @@
+=== SCENARIO H — LIVE hub v0.107.0, bad severity ===
+  api key length: 64
+  POST /event (severity=warn) http=200
+  POST /event (severity=error, control) http=200
+
+-- THE NEW LINE in the hub's own log --
+2026/08/23 11:59:07 [WARN] [api] Event from demo-hp: severity "warn" is not in {info,warning,error,critical} — coercing to "info", which severityNotifies DROPS, so this backup_failed alert will reach NOBODY. Fix the emitting controller; this event is stored but not routed.
+2026/08/23 11:59:07 [INFO] Event from demo-hp: backup_failed (info) — R-387 scenario H probe
+2026/08/23 11:59:07 [INFO] Event from demo-hp: backup_failed (error) — R-387 scenario H control
@@ -0,0 +1,30 @@
+=== END STATE — every app healthy, planted data untouched ===
+2026-08-23T09:59:32Z
+controller: gitea.dooplex.hu/admin/felhom-controller:0.223.0 Up 18 minutes (healthy)
+
+bookstack	Up 7 hours (healthy)
+bookstack-db	Up 28 minutes (healthy)
+calibre-web	Up 7 hours (healthy)
+docmost	Up 4 hours (healthy)
+docmost-postgres	Up 4 hours (healthy)
+docmost-redis	Up 4 hours (healthy)
+filebrowser	Up 42 hours (healthy)
+kimai	Up 7 hours (healthy)
+kimai-db	Up 7 hours (healthy)
+opengist	Up 7 hours (healthy)
+paperless-postgres	Up 7 hours (healthy)
+paperless-redis	Up 7 hours (healthy)
+paperless-webserver	Up 7 hours (healthy)
+privatebin	Up 7 minutes (healthy)
+romm	Up 7 hours (healthy)
+romm-db	Up 7 hours (healthy)
+romm-redis	Up 7 hours (healthy)
+traefik	Up 42 hours
+
+-- intents restored --
+  privatebin   desired_state: running
+  bookstack    desired_state: running
+  docmost      desired_state: running
+/root/privatebin-app.yaml.bak
+  (backup file still present — removing)
+  notification prefs: null email_set= False
@@ -0,0 +1,8 @@
+### RED-PROOF 1 (R-329) — mutation: app_start_failed severity back to "warn" ###
+### layer: the EMITTER — the last point at which the bad value still exists ###
+--- FAIL: TestR329_EveryEmittedSeverityIsInTheHubVocabulary (0.16s)
+    r329_severity_contract_test.go:203: /mnt/5_hdd/felhom.eu/git/felhom-controller/controller/internal/notify/notifier.go:561:30: emit(...) emits severity "warn", which is NOT in the hub's vocabulary {info, warning, error, critical}.
+    r329_severity_contract_test.go:138: checked 30 severity literals across the controller
+FAIL
+FAIL	gitea.dooplex.hu/admin/felhom-controller/internal/notify	0.163s
+FAIL
@@ -0,0 +1,11 @@
+### RED-PROOF 2 (R-386) — mutation: userStopped reverted to the state guess ###
+### layer: classifyRunStates — the single derivation point where the guess was made ###
+--- FAIL: TestR386_OutOfBandStopWithRunningIntentAlarms (0.00s)
+    r386_intent_test.go:51: dead-app banner = [], want exactly privatebin — nobody asked for this app to be stopped, so its being stopped is a fault (measured silent on demo-hp 2026-08-23)
+--- FAIL: TestR386_AbsentIntentSuppressesButIsAnnounced (0.00s)
+    r386_intent_test.go:94: IntentUnknown = false — the suppression happened but nothing records it, so an operator cannot answer 'how many apps am I blind to?'. A rule without a mechanism is not a rule
+--- FAIL: TestR386_TheSchedulerWiresBothHalves (0.01s)
+    r386_intent_test.go:243: stacks.DesiredStateOf is never called from main.go — the classifier is back to guessing from the state (R-386)
+FAIL
+FAIL	gitea.dooplex.hu/admin/felhom-controller/cmd/controller	0.023s
+FAIL
@@ -0,0 +1,8 @@
+### RED-PROOF 3 (Part 4) — mutation: sameEventSet no-op guard removed ###
+### layer: the SAVE handler — where a render-then-save would rewrite stored bytes ###
+--- FAIL: TestR329Part4_RoundTripIsByteIdentical (0.04s)
+    --- FAIL: TestR329Part4_RoundTripIsByteIdentical/defaults (0.01s)
+        r329_toggle_split_test.go:115: a no-op save CHANGED the stored settings.
+FAIL
+FAIL	gitea.dooplex.hu/admin/felhom-controller/internal/web	0.048s
+FAIL
@@ -0,0 +1,7 @@
+### RED-PROOF 4 (R-387) — mutation: the ingest WARN line removed ###
+### layer: INGEST — the last point at which the offending value still exists ###
+--- FAIL: TestR387_UnknownSeverityIsCoercedAndAnnounced (0.02s)
+    r387_severity_visibility_test.go:89: the hub rewrote a severity and said nothing — this is R-387, and it is how two features shipped undeliverable for months. log="[INFO] Event from c1: backup_failed (info) — test message\n"
+FAIL
+FAIL	gitea.dooplex.hu/admin/felhom-hub/internal/api	0.143s
+FAIL
@@ -0,0 +1,11 @@
+### RED-PROOF 5 (R-329, fillwatch) — REDONE ###
+### first attempt was INERT: an earlier 'if next == prev { continue }' makes 'next < prev'
+### and 'next <= prev' behave identically. Reported as a finding. ###
+### mutation: the de-escalation guard REMOVED, so BandOK reaches the notify seam ###
+### layer: Check() — the invariant lives THERE, not in Severity() ###
+--- FAIL: TestR329_FillwatchNeverEmitsTheEmptySeverity (0.00s)
+    r329_severity_test.go:62: notified with band ok → severity "" (event type ""): the hub coerces an unknown severity to "info" and then drops it, so this alert would reach NOBODY
+    r329_severity_test.go:67: notified with severity "", outside the hub vocabulary
+    r329_severity_test.go:70: 4 crossings notified, every one with a routable severity
+FAIL
+FAIL	gitea.dooplex.hu/admin/felhom-controller/internal/fillwatch	0.004s