hub v0.78.0 — R-97a: whole-guest backup events, operator-only
internal/quiesce had no route to the hub at all: three failed whole-guest backups on 2026-07-27 produced zero events. Hub half of the fix. whole_guest_backup_failed / _recovered are allowlisted with NO customerMessages entry. Deliberately not backup_failed/backup_completed — those have customer Hungarian templates AND sit in demo-felhom's live enabled_events, so reusing them would email the customer that their backup failed while it is still retrying behind the R-88 breaker. The recovery joins recoveredPairedDownTypes because it is severity info and severityNotifies drops info — otherwise the operator hears it break and never hears it heal. Its customer leg is pairing-gated and can never fire. Operator cooldown gains a per-tier dimension from the event details, so one tier cannot mask another for an hour. Narrow: empty suffix unless a tier is sent, so no existing event type changes.
This commit is contained in:
@@ -7,6 +7,7 @@ import (
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
@@ -70,6 +71,16 @@ func priorityHeaders(severity string) map[string]string {
|
||||
var recoveredPairedDownTypes = map[string][]string{
|
||||
"node_recovered": {"node_stale", "node_down"},
|
||||
"host_recovered": {"host_stale", "host_down"},
|
||||
// R-97a. This branch runs BEFORE the severity gate, which is exactly why the recovery belongs
|
||||
// here: `whole_guest_backup_recovered` is severity "info", and severityNotifies drops "info", so
|
||||
// routing it normally would store the event and never mail it — the operator would be told the
|
||||
// tier broke and never told it healed, which is the half of Scenario B that matters.
|
||||
//
|
||||
// The customer leg needs no special handling: it is PAIRING-gated on a customer-channel "sent"
|
||||
// row for the down type, and `whole_guest_backup_failed` has no customerMessages entry and is in
|
||||
// nobody's enabled_events — so no such row can exist, and the customer correctly hears neither
|
||||
// edge. Operator hears both.
|
||||
"whole_guest_backup_recovered": {"whole_guest_backup_failed"},
|
||||
}
|
||||
|
||||
// severityNotifies reports whether a severity triggers email notifications. warning / error / critical
|
||||
@@ -225,12 +236,36 @@ func (d *Dispatcher) processRecovery(customerID, eventType, severity, message, d
|
||||
d.store.LogNotification(customerID, eventType, severity, message, "sent", "", "customer")
|
||||
}
|
||||
|
||||
// cooldownTierSuffix returns ":"+tier when the event's details carry a non-empty `tier`, else "".
|
||||
//
|
||||
// R-97a. The operator cooldown was keyed `customerID + ":" + eventType` alone, which is correct for
|
||||
// every event that describes ONE thing — but a whole-guest backup failure describes ONE TIER, and a
|
||||
// box has two. `felhom-pbs` failing at 09:00 would swallow `local` failing at 09:20 for the whole
|
||||
// hour, so the operator would be told about the offsite tier and never about the local one. That is
|
||||
// precisely the masking the per-tier signal exists to prevent.
|
||||
//
|
||||
// NARROW ON PURPOSE: the suffix is empty unless the producer opts in by sending a `tier`, so no
|
||||
// existing event type's cooldown behaviour changes. Widening the key for everything would, e.g.,
|
||||
// turn one hourly `app_start_failed` into one per app — a flood, not a fix.
|
||||
func cooldownTierSuffix(detailsJSON string) string {
|
||||
if detailsJSON == "" || !strings.Contains(detailsJSON, "\"tier\"") {
|
||||
return ""
|
||||
}
|
||||
var d struct {
|
||||
Tier string `json:"tier"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(detailsJSON), &d); err != nil || d.Tier == "" {
|
||||
return ""
|
||||
}
|
||||
return ":" + d.Tier
|
||||
}
|
||||
|
||||
func (d *Dispatcher) processOperator(customerID, eventType, severity, message, detailsJSON, source string) {
|
||||
if !d.operatorOn || d.operatorEmail == "" {
|
||||
return
|
||||
}
|
||||
|
||||
cooldownKey := customerID + ":" + eventType
|
||||
cooldownKey := customerID + ":" + eventType + cooldownTierSuffix(detailsJSON)
|
||||
d.mu.Lock()
|
||||
if last, ok := d.opCooldowns[cooldownKey]; ok && time.Since(last) < 1*time.Hour {
|
||||
d.mu.Unlock()
|
||||
|
||||
Reference in New Issue
Block a user