admin
e9c99566b0
R-97a: internal/quiesce had no route to the hub at all — three failed whole-guest backups on 2026-07-27 produced zero events. TierNotifier is a seam (not an import), wired by an init-only setter because main.go builds the notifier after the loop. Edge-triggered: the failure fires when the R-88 breaker ARMS, not per retry, and recovery rides recordSuccess's existing bool. Uses NEW operator-only event types; reusing backup_failed would have emailed the customer in Hungarian about a backup they cannot act on, since it has a customerMessages entry and is in live enabled_events. Requires hub >= v0.78.0. R-97b: v0.164.0's state filter cannot see an app caught MID-RESTART, which is how BookStack alarmed. The fix is a suppression window keyed to the quiesce CYCLE, consumed at the same single derivation point. 180s grace, derived from the deploy flow's 120s health timeout and Mealie's 60s start_period; it expires, so an app that genuinely fails to come back still alarms.
91 lines
4.2 KiB
Go
91 lines
4.2 KiB
Go
package quiesce
|
|
|
|
import "time"
|
|
|
|
// R-97b — a quiesce cycle must not make the customer think their apps broke.
|
|
//
|
|
// THE BUG: during the 2026-07-27 loop the ONLY customer-visible output was
|
|
// `app_start_failed — "Telepített alkalmazás nem fut: BookStack"`, on the customer channel, in
|
|
// Hungarian, during an outage the BACKUP SYSTEM ITSELF caused, with no indication why. That is worse
|
|
// than silence: it tells the customer something is wrong with their app and hands them nothing to do.
|
|
//
|
|
// WHY v0.164.0's FILTER DOES NOT COVER THIS. That filter is state-based —
|
|
// `IsDownState(st.State) && st.State != StateStopped` — and it suppresses DELIBERATELY STOPPED apps.
|
|
// BookStack alarmed because the third quiesce cycle caught it MID-RESTART: starting, or up but not
|
|
// yet healthy. Those are not `StateStopped`, and no state classification can tell "restarting because
|
|
// a backup just stopped me" from "restarting because I keep crashing". The distinguishing fact is not
|
|
// in the state at all — it is that *we* stopped it, and we know we did.
|
|
//
|
|
// So this is a SUPPRESSION WINDOW KEYED TO THE CYCLE, not a state test: the loop already tracks which
|
|
// stacks it stopped (it must, to restart exactly those), and it knows when it unquiesced.
|
|
//
|
|
// ── THE TENSION, WHICH IS THE WHOLE DESIGN ───────────────────────────────────────────────────
|
|
//
|
|
// Suppress during the cycle and for a grace period after the restart — but an app that GENUINELY
|
|
// fails to come back MUST still alarm. Permanent suppression would trade a loud false alarm for a
|
|
// silent real one, which is the same over-correction as R-88's Scenario D. The grace window expires;
|
|
// it does not latch.
|
|
|
|
// quiesceAlarmGrace is how long after unquiescing a stack stays exempt from app-down alarms.
|
|
//
|
|
// Derived from what a restarted app actually needs, not picked round:
|
|
// - the deploy flow allows **120 s** for a stack to come up healthy — the project's own existing
|
|
// answer to "how long is too long";
|
|
// - the slowest catalog healthcheck start_period is Mealie's **60 s**, after which a couple of
|
|
// check intervals must still elapse before a verdict is meaningful.
|
|
//
|
|
// 180 s clears both with margin. It is deliberately NOT longer: the app-state scan runs on its own
|
|
// cadence, so an app that is genuinely dead alarms on the first scan after the window closes —
|
|
// making the cost of this suppression a bounded DELAY in reporting a real failure, never its loss.
|
|
const quiesceAlarmGrace = 180 * time.Second
|
|
|
|
// markQuiesced records stacks as exempt for the duration of the cycle. Expiry is set at unquiesce;
|
|
// until then the entry is open-ended, because a cycle may legitimately run for hours (a first full
|
|
// offsite snapshot) and an app stopped that whole time must not alarm halfway through.
|
|
func (l *Loop) markQuiesced(names []string) {
|
|
l.suppressMu.Lock()
|
|
defer l.suppressMu.Unlock()
|
|
if l.suppressed == nil {
|
|
l.suppressed = map[string]time.Time{}
|
|
}
|
|
for _, n := range names {
|
|
l.suppressed[n] = time.Time{} // zero = still quiesced, no expiry yet
|
|
}
|
|
}
|
|
|
|
// markUnquiesced starts the grace clock for the stacks this cycle restarted.
|
|
func (l *Loop) markUnquiesced(names []string) {
|
|
until := l.now().Add(quiesceAlarmGrace)
|
|
l.suppressMu.Lock()
|
|
defer l.suppressMu.Unlock()
|
|
if l.suppressed == nil {
|
|
return
|
|
}
|
|
for _, n := range names {
|
|
l.suppressed[n] = until
|
|
}
|
|
}
|
|
|
|
// SuppressedStacks returns the set of stack names currently exempt from app-down alarms — those a
|
|
// quiesce cycle stopped, plus those still inside the post-restart grace window.
|
|
//
|
|
// Nil-safe on a nil *Loop so the caller does not need a branch: a controller with no quiesce loop
|
|
// (unprovisioned guest) suppresses nothing, which is the correct default.
|
|
func (l *Loop) SuppressedStacks() map[string]bool {
|
|
if l == nil {
|
|
return nil
|
|
}
|
|
now := l.now()
|
|
l.suppressMu.Lock()
|
|
defer l.suppressMu.Unlock()
|
|
out := make(map[string]bool, len(l.suppressed))
|
|
for n, until := range l.suppressed {
|
|
if until.IsZero() || now.Before(until) {
|
|
out[n] = true
|
|
continue
|
|
}
|
|
delete(l.suppressed, n) // expired — reap so the map cannot grow without bound
|
|
}
|
|
return out
|
|
}
|