R-97: a failing backup is heard, and stops blaming the apps (v0.177.0)
R-97a: internal/quiesce had no route to the hub at all — three failed whole-guest backups on 2026-07-27 produced zero events. TierNotifier is a seam (not an import), wired by an init-only setter because main.go builds the notifier after the loop. Edge-triggered: the failure fires when the R-88 breaker ARMS, not per retry, and recovery rides recordSuccess's existing bool. Uses NEW operator-only event types; reusing backup_failed would have emailed the customer in Hungarian about a backup they cannot act on, since it has a customerMessages entry and is in live enabled_events. Requires hub >= v0.78.0. R-97b: v0.164.0's state filter cannot see an app caught MID-RESTART, which is how BookStack alarmed. The fix is a suppression window keyed to the quiesce CYCLE, consumed at the same single derivation point. 180s grace, derived from the deploy flow's 120s health timeout and Mealie's 60s start_period; it expires, so an app that genuinely fails to come back still alarms.
This commit is contained in:
@@ -0,0 +1,90 @@
|
||||
package quiesce
|
||||
|
||||
import "time"
|
||||
|
||||
// R-97b — a quiesce cycle must not make the customer think their apps broke.
|
||||
//
|
||||
// THE BUG: during the 2026-07-27 loop the ONLY customer-visible output was
|
||||
// `app_start_failed — "Telepített alkalmazás nem fut: BookStack"`, on the customer channel, in
|
||||
// Hungarian, during an outage the BACKUP SYSTEM ITSELF caused, with no indication why. That is worse
|
||||
// than silence: it tells the customer something is wrong with their app and hands them nothing to do.
|
||||
//
|
||||
// WHY v0.164.0's FILTER DOES NOT COVER THIS. That filter is state-based —
|
||||
// `IsDownState(st.State) && st.State != StateStopped` — and it suppresses DELIBERATELY STOPPED apps.
|
||||
// BookStack alarmed because the third quiesce cycle caught it MID-RESTART: starting, or up but not
|
||||
// yet healthy. Those are not `StateStopped`, and no state classification can tell "restarting because
|
||||
// a backup just stopped me" from "restarting because I keep crashing". The distinguishing fact is not
|
||||
// in the state at all — it is that *we* stopped it, and we know we did.
|
||||
//
|
||||
// So this is a SUPPRESSION WINDOW KEYED TO THE CYCLE, not a state test: the loop already tracks which
|
||||
// stacks it stopped (it must, to restart exactly those), and it knows when it unquiesced.
|
||||
//
|
||||
// ── THE TENSION, WHICH IS THE WHOLE DESIGN ───────────────────────────────────────────────────
|
||||
//
|
||||
// Suppress during the cycle and for a grace period after the restart — but an app that GENUINELY
|
||||
// fails to come back MUST still alarm. Permanent suppression would trade a loud false alarm for a
|
||||
// silent real one, which is the same over-correction as R-88's Scenario D. The grace window expires;
|
||||
// it does not latch.
|
||||
|
||||
// quiesceAlarmGrace is how long after unquiescing a stack stays exempt from app-down alarms.
|
||||
//
|
||||
// Derived from what a restarted app actually needs, not picked round:
|
||||
// - the deploy flow allows **120 s** for a stack to come up healthy — the project's own existing
|
||||
// answer to "how long is too long";
|
||||
// - the slowest catalog healthcheck start_period is Mealie's **60 s**, after which a couple of
|
||||
// check intervals must still elapse before a verdict is meaningful.
|
||||
//
|
||||
// 180 s clears both with margin. It is deliberately NOT longer: the app-state scan runs on its own
|
||||
// cadence, so an app that is genuinely dead alarms on the first scan after the window closes —
|
||||
// making the cost of this suppression a bounded DELAY in reporting a real failure, never its loss.
|
||||
const quiesceAlarmGrace = 180 * time.Second
|
||||
|
||||
// markQuiesced records stacks as exempt for the duration of the cycle. Expiry is set at unquiesce;
|
||||
// until then the entry is open-ended, because a cycle may legitimately run for hours (a first full
|
||||
// offsite snapshot) and an app stopped that whole time must not alarm halfway through.
|
||||
func (l *Loop) markQuiesced(names []string) {
|
||||
l.suppressMu.Lock()
|
||||
defer l.suppressMu.Unlock()
|
||||
if l.suppressed == nil {
|
||||
l.suppressed = map[string]time.Time{}
|
||||
}
|
||||
for _, n := range names {
|
||||
l.suppressed[n] = time.Time{} // zero = still quiesced, no expiry yet
|
||||
}
|
||||
}
|
||||
|
||||
// markUnquiesced starts the grace clock for the stacks this cycle restarted.
|
||||
func (l *Loop) markUnquiesced(names []string) {
|
||||
until := l.now().Add(quiesceAlarmGrace)
|
||||
l.suppressMu.Lock()
|
||||
defer l.suppressMu.Unlock()
|
||||
if l.suppressed == nil {
|
||||
return
|
||||
}
|
||||
for _, n := range names {
|
||||
l.suppressed[n] = until
|
||||
}
|
||||
}
|
||||
|
||||
// SuppressedStacks returns the set of stack names currently exempt from app-down alarms — those a
|
||||
// quiesce cycle stopped, plus those still inside the post-restart grace window.
|
||||
//
|
||||
// Nil-safe on a nil *Loop so the caller does not need a branch: a controller with no quiesce loop
|
||||
// (unprovisioned guest) suppresses nothing, which is the correct default.
|
||||
func (l *Loop) SuppressedStacks() map[string]bool {
|
||||
if l == nil {
|
||||
return nil
|
||||
}
|
||||
now := l.now()
|
||||
l.suppressMu.Lock()
|
||||
defer l.suppressMu.Unlock()
|
||||
out := make(map[string]bool, len(l.suppressed))
|
||||
for n, until := range l.suppressed {
|
||||
if until.IsZero() || now.Before(until) {
|
||||
out[n] = true
|
||||
continue
|
||||
}
|
||||
delete(l.suppressed, n) // expired — reap so the map cannot grow without bound
|
||||
}
|
||||
return out
|
||||
}
|
||||
Reference in New Issue
Block a user