hub v0.107.0: the hub rewrote a severity and said nothing (R-387); golden 0.223.0
gates / gates (push) Successful in 17s

One handler, two fields, opposite discipline. An unknown event_type is rejected
with a loud 400. An unknown severity was rewritten to "info" without a word -
and severityNotifies drops "info" before BOTH legs, so the event was stored,
answered 200, and mailed to nobody.

Two shipped features went out that way: DiskAlertKind.Severity emitted "warn"
until controller v0.215.0, app_start_failed until v0.223.0. Measured on the live
hub DB today: 91 app_start_failed events stored all-time, ZERO notification_log
rows before this session - not one, on any channel.

The mechanism built to catch this class was structurally blind to it: the
dispatcher's `unrecognized severity` line cannot execute for anything arriving
over the API, because the coercion one line earlier guarantees the value it
looks for cannot arrive.

The coercion STAYS - a rejected event is a lost event, and losing an alarm is
worse than mis-routing one. Only the silence is fixed: a WARN naming the
customer, the event type and the rejected value.

The dispatcher branch is KEPT, not deleted as dead, and the reason is evidence
rather than caution: cmd/hub/main.go wires dispatcher.ProcessEvent DIRECTLY as
the monitor.EventNotifyFunc for the staleness, host-staleness and offsite-box
checkers, which never pass through the handler. For those it is the only
severity guard there is. All 90 severity literals in internal/monitor are
already valid, so the guard is silent because the producers are correct.

Test count 702 -> 709. Red-proof seen failing: delete the WARN line and the
coercion test fails with "the hub rewrote a severity and said nothing".

Golden 0.223.0 baked and published (sha 9eaf39ac3921...), round-trip HTTP 206.
Vouching is the operator's act and was not done here.
This commit is contained in:
2026-08-23 11:57:26 +02:00
parent 55274d5ef3
commit 68a9f5475c
8 changed files with 762 additions and 2 deletions
+23 -1
View File
@@ -2118,10 +2118,32 @@ func (h *Handler) handleEvent(w http.ResponseWriter, r *http.Request) {
return
}
// Validate/default severity (exact-match lowercase; unknown values coerce to info)
// Validate/default severity (exact-match lowercase; unknown values coerce to info).
//
// R-387 — THE COERCION STAYS. THE SILENCE DOES NOT.
//
// Note the asymmetry two blocks up: an unknown event TYPE is rejected with a loud 400, while an
// unknown SEVERITY was rewritten without a word. The silent one is the one that hid a real defect
// for the whole life of two features — `DiskAlertKind.Severity` emitted "warn" until controller
// v0.215.0, and `app_start_failed` emitted it until v0.223.0. Both were stored, both were
// coerced here to "info", and "info" is dropped by severityNotifies — so both were mailed to
// NOBODY, on either leg, while every POST returned 200.
//
// The dispatcher has a line whose job is exactly this (`unrecognized severity %q`), and it can
// never execute, because this block guarantees the value it looks for cannot reach it. **The
// mechanism built to detect this class was structurally blind to it.**
//
// WHY NOT A 400. A rejected event is a LOST event, and losing an alarm is worse than mis-routing
// one — the same reasoning that made this a coercion in the first place. The producer is our own
// controller, so the fix belongs at the emitter; this line is how the emitter's mistake becomes
// VISIBLE the first time it happens instead of never.
switch payload.Severity {
case "info", "warning", "error", "critical":
default:
h.logger.Printf("[WARN] [api] Event from %s: severity %q is not in {info,warning,error,critical} "+
"— coercing to \"info\", which severityNotifies DROPS, so this %s alert will reach NOBODY. "+
"Fix the emitting controller; this event is stored but not routed.",
payload.CustomerID, payload.Severity, payload.EventType)
payload.Severity = "info"
}