hub v0.107.0: the hub rewrote a severity and said nothing (R-387); golden 0.223.0
gates / gates (push) Successful in 17s
gates / gates (push) Successful in 17s
One handler, two fields, opposite discipline. An unknown event_type is rejected with a loud 400. An unknown severity was rewritten to "info" without a word - and severityNotifies drops "info" before BOTH legs, so the event was stored, answered 200, and mailed to nobody. Two shipped features went out that way: DiskAlertKind.Severity emitted "warn" until controller v0.215.0, app_start_failed until v0.223.0. Measured on the live hub DB today: 91 app_start_failed events stored all-time, ZERO notification_log rows before this session - not one, on any channel. The mechanism built to catch this class was structurally blind to it: the dispatcher's `unrecognized severity` line cannot execute for anything arriving over the API, because the coercion one line earlier guarantees the value it looks for cannot arrive. The coercion STAYS - a rejected event is a lost event, and losing an alarm is worse than mis-routing one. Only the silence is fixed: a WARN naming the customer, the event type and the rejected value. The dispatcher branch is KEPT, not deleted as dead, and the reason is evidence rather than caution: cmd/hub/main.go wires dispatcher.ProcessEvent DIRECTLY as the monitor.EventNotifyFunc for the staleness, host-staleness and offsite-box checkers, which never pass through the handler. For those it is the only severity guard there is. All 90 severity literals in internal/monitor are already valid, so the guard is silent because the producers are correct. Test count 702 -> 709. Red-proof seen failing: delete the WARN line and the coercion test fails with "the hub rewrote a severity and said nothing". Golden 0.223.0 baked and published (sha 9eaf39ac3921...), round-trip HTTP 206. Vouching is the operator's act and was not done here.
This commit is contained in:
@@ -2118,10 +2118,32 @@ func (h *Handler) handleEvent(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
|
||||
// Validate/default severity (exact-match lowercase; unknown values coerce to info)
|
||||
// Validate/default severity (exact-match lowercase; unknown values coerce to info).
|
||||
//
|
||||
// R-387 — THE COERCION STAYS. THE SILENCE DOES NOT.
|
||||
//
|
||||
// Note the asymmetry two blocks up: an unknown event TYPE is rejected with a loud 400, while an
|
||||
// unknown SEVERITY was rewritten without a word. The silent one is the one that hid a real defect
|
||||
// for the whole life of two features — `DiskAlertKind.Severity` emitted "warn" until controller
|
||||
// v0.215.0, and `app_start_failed` emitted it until v0.223.0. Both were stored, both were
|
||||
// coerced here to "info", and "info" is dropped by severityNotifies — so both were mailed to
|
||||
// NOBODY, on either leg, while every POST returned 200.
|
||||
//
|
||||
// The dispatcher has a line whose job is exactly this (`unrecognized severity %q`), and it can
|
||||
// never execute, because this block guarantees the value it looks for cannot reach it. **The
|
||||
// mechanism built to detect this class was structurally blind to it.**
|
||||
//
|
||||
// WHY NOT A 400. A rejected event is a LOST event, and losing an alarm is worse than mis-routing
|
||||
// one — the same reasoning that made this a coercion in the first place. The producer is our own
|
||||
// controller, so the fix belongs at the emitter; this line is how the emitter's mistake becomes
|
||||
// VISIBLE the first time it happens instead of never.
|
||||
switch payload.Severity {
|
||||
case "info", "warning", "error", "critical":
|
||||
default:
|
||||
h.logger.Printf("[WARN] [api] Event from %s: severity %q is not in {info,warning,error,critical} "+
|
||||
"— coercing to \"info\", which severityNotifies DROPS, so this %s alert will reach NOBODY. "+
|
||||
"Fix the emitting controller; this event is stored but not routed.",
|
||||
payload.CustomerID, payload.Severity, payload.EventType)
|
||||
payload.Severity = "info"
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user