R-97: a failing backup is heard, and stops blaming the apps (v0.177.0)
R-97a: internal/quiesce had no route to the hub at all — three failed whole-guest backups on 2026-07-27 produced zero events. TierNotifier is a seam (not an import), wired by an init-only setter because main.go builds the notifier after the loop. Edge-triggered: the failure fires when the R-88 breaker ARMS, not per retry, and recovery rides recordSuccess's existing bool. Uses NEW operator-only event types; reusing backup_failed would have emailed the customer in Hungarian about a backup they cannot act on, since it has a customerMessages entry and is in live enabled_events. Requires hub >= v0.78.0. R-97b: v0.164.0's state filter cannot see an app caught MID-RESTART, which is how BookStack alarmed. The fix is a suppression window keyed to the quiesce CYCLE, consumed at the same single derivation point. 180s grace, derived from the deploy flow's 120s health timeout and Mealie's 60s start_period; it expires, so an app that genuinely fails to come back still alarms.
This commit is contained in:
@@ -94,6 +94,17 @@ type BackupDetails struct {
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// WholeGuestBackupDetails carries the TIER for a whole-guest (vzdump) backup event (R-97a).
|
||||
//
|
||||
// The `tier` field is load-bearing beyond display: the hub's operator cooldown is keyed
|
||||
// `customerID:eventType` plus this tier when present, so `local` failing does not get swallowed by
|
||||
// `felhom-pbs` having failed within the same hour. Rename it and the two tiers silently share one
|
||||
// cooldown again.
|
||||
type WholeGuestBackupDetails struct {
|
||||
Tier string `json:"tier"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// DBDumpDetails holds structured data for DB dump events.
|
||||
type DBDumpDetails struct {
|
||||
DatabaseCount int `json:"database_count,omitempty"`
|
||||
@@ -771,3 +782,26 @@ func statusRank(status string) int {
|
||||
return 0
|
||||
}
|
||||
}
|
||||
|
||||
// NotifyWholeGuestBackupFailed / ...Recovered — R-97a, the WHOLE-GUEST (vzdump) backup tier.
|
||||
//
|
||||
// OPERATOR-TIER ONLY, and that is why these are NOT `backup_failed`. `backup_failed` and
|
||||
// `backup_completed` both carry `customerMessages` entries in the hub AND sit in demo-felhom's live
|
||||
// `enabled_events`, so reusing them would email the CUSTOMER, in Hungarian, that their backup failed
|
||||
// — while it is still retrying behind the R-88 breaker. A customer can take no action on a failed
|
||||
// whole-guest backup; that is the same harm R-97b removes, re-introduced through the front door.
|
||||
//
|
||||
// These follow the R-85 precedent exactly: a type in the hub's `allowedEventTypes` with NO
|
||||
// `customerMessages` entry, so the dispatcher structurally cannot route it to a customer.
|
||||
//
|
||||
// HUB DEPENDENCY: both types MUST be present in the hub's allowedEventTypes or POST /event 400s
|
||||
// (the recorded allowlist gotcha). Do not deploy this controller ahead of that hub change.
|
||||
func (n *Notifier) NotifyWholeGuestBackupFailed(tier, message, errMsg string) {
|
||||
n.PushEvent("whole_guest_backup_failed", "error", message,
|
||||
WholeGuestBackupDetails{Tier: tier, Error: errMsg})
|
||||
}
|
||||
|
||||
func (n *Notifier) NotifyWholeGuestBackupRecovered(tier, message string) {
|
||||
n.PushEvent("whole_guest_backup_recovered", "info", message,
|
||||
WholeGuestBackupDetails{Tier: tier})
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user