R-97: a failing backup is heard, and stops blaming the apps (v0.177.0)
R-97a: internal/quiesce had no route to the hub at all — three failed whole-guest backups on 2026-07-27 produced zero events. TierNotifier is a seam (not an import), wired by an init-only setter because main.go builds the notifier after the loop. Edge-triggered: the failure fires when the R-88 breaker ARMS, not per retry, and recovery rides recordSuccess's existing bool. Uses NEW operator-only event types; reusing backup_failed would have emailed the customer in Hungarian about a backup they cannot act on, since it has a customerMessages entry and is in live enabled_events. Requires hub >= v0.78.0. R-97b: v0.164.0's state filter cannot see an app caught MID-RESTART, which is how BookStack alarmed. The fix is a suppression window keyed to the quiesce CYCLE, consumed at the same single derivation point. 180s grace, derived from the deploy flow's 120s health timeout and Mealie's 60s start_period; it expires, so an app that genuinely fails to come back still alarms.
This commit is contained in:
@@ -58,6 +58,26 @@ const (
|
||||
breakerMaxShift = 5
|
||||
)
|
||||
|
||||
// TierNotifier is the seam by which a whole-guest backup outcome reaches the hub (R-97a).
|
||||
//
|
||||
// WHY A SEAM AND NOT AN IMPORT: `internal/quiesce` deliberately keeps no dependency on
|
||||
// `internal/notify` — the same reason `windowStartFn` is injected rather than importing `settings`.
|
||||
// It is wired by an init-only setter (`SetTierNotifier`) because the notifier is constructed AFTER
|
||||
// the quiesce loop in main.go; nil means "not wired", which is the pre-provisioning case, not an
|
||||
// error.
|
||||
//
|
||||
// EDGE-TRIGGERED, ON PURPOSE. `BackupFailed` fires when the breaker ARMS — i.e. on the first failure
|
||||
// of a run — never on the retries behind it. The retry cadence is 15m/30m/1h/2h/4h and an event per
|
||||
// attempt is an inbox nobody reads. `BackupRecovered` fires when a tier that HAD been failing
|
||||
// succeeds, so the operator who was told it broke is also told it healed.
|
||||
//
|
||||
// OPERATOR-TIER ONLY. A customer can take no action on a failed whole-guest backup, and telling them
|
||||
// it failed while it is still retrying is alarming without being actionable.
|
||||
type TierNotifier interface {
|
||||
BackupFailed(tier, message, errMsg string)
|
||||
BackupRecovered(tier, message string)
|
||||
}
|
||||
|
||||
// breakerState is one tier's consecutive-failure record.
|
||||
type breakerState struct {
|
||||
failures int
|
||||
|
||||
Reference in New Issue
Block a user