R-97: a failing backup is heard, and stops blaming the apps (v0.177.0)

R-97a: internal/quiesce had no route to the hub at all — three failed whole-guest
backups on 2026-07-27 produced zero events. TierNotifier is a seam (not an import),
wired by an init-only setter because main.go builds the notifier after the loop.
Edge-triggered: the failure fires when the R-88 breaker ARMS, not per retry, and
recovery rides recordSuccess's existing bool. Uses NEW operator-only event types;
reusing backup_failed would have emailed the customer in Hungarian about a backup
they cannot act on, since it has a customerMessages entry and is in live
enabled_events. Requires hub >= v0.78.0.

R-97b: v0.164.0's state filter cannot see an app caught MID-RESTART, which is how
BookStack alarmed. The fix is a suppression window keyed to the quiesce CYCLE,
consumed at the same single derivation point. 180s grace, derived from the deploy
flow's 120s health timeout and Mealie's 60s start_period; it expires, so an app
that genuinely fails to come back still alarms.
This commit is contained in:
2026-07-27 17:01:41 +02:00
parent ccefff4f39
commit e9c99566b0
10 changed files with 559 additions and 11 deletions
+20
View File
@@ -58,6 +58,26 @@ const (
breakerMaxShift = 5
)
// TierNotifier is the seam by which a whole-guest backup outcome reaches the hub (R-97a).
//
// WHY A SEAM AND NOT AN IMPORT: `internal/quiesce` deliberately keeps no dependency on
// `internal/notify` — the same reason `windowStartFn` is injected rather than importing `settings`.
// It is wired by an init-only setter (`SetTierNotifier`) because the notifier is constructed AFTER
// the quiesce loop in main.go; nil means "not wired", which is the pre-provisioning case, not an
// error.
//
// EDGE-TRIGGERED, ON PURPOSE. `BackupFailed` fires when the breaker ARMS — i.e. on the first failure
// of a run — never on the retries behind it. The retry cadence is 15m/30m/1h/2h/4h and an event per
// attempt is an inbox nobody reads. `BackupRecovered` fires when a tier that HAD been failing
// succeeds, so the operator who was told it broke is also told it healed.
//
// OPERATOR-TIER ONLY. A customer can take no action on a failed whole-guest backup, and telling them
// it failed while it is still retrying is alarming without being actionable.
type TierNotifier interface {
BackupFailed(tier, message, errMsg string)
BackupRecovered(tier, message string)
}
// breakerState is one tier's consecutive-failure record.
type breakerState struct {
failures int