package quiesce import ( "sync" "time" ) // R-88 — the failure breaker. // // THE BUG THIS EXISTS TO KILL: before this, `internal/quiesce` had no consecutive-failure counter, // no backoff and no circuit breaker of any kind. The driver is a plain 5-minute ticker, so a tier // that was due and kept failing was re-quiesced every five minutes FOREVER — and a quiesce cycle // stops and restarts every customer app stack. Observed live on demo-felhom 2026-07-27: three full // stop/start cycles across eleven minutes (09:02:57, 09:07:58, 09:12:57 Budapest) against a PBS tier // that could not possibly succeed. It stopped after three only because PBS came back, not because // anything gave up. // // The harm is NOT the failing backup — it is the app outage taken to attempt it. So the breaker gates // the QUIESCE, not the backup: a tier in backoff is dropped from the due set before any stack is // stopped. // // ── WHAT THIS DELIBERATELY IS NOT ──────────────────────────────────────────────────────────── // // It is NOT permanent, and it must never become permanent. The cap bounds the retry INTERVAL; it // never stops retrying. A breaker that latches open is a silent backup outage, which is strictly // worse than the loop it replaces — the loop at least announced itself by stopping the apps. // // It is NOT global. State is per TARGET (`dueTier.target`), so a broken offsite tier cannot suppress // a healthy local one. Halting all backups because one tier is down would trade a narrow fault for a // total one. // // It does NOT gate the manual path. `TriggerNow` bypasses due-ness and the window gate, and it // bypasses this too — a human pressing „Mentés most" has made an explicit decision and must not be // deferred by a breaker built for the scheduler. Manual runs still RECORD their outcome (a manual // success clears the backoff, which is exactly what an operator fixing the tier expects). // // ── CRASH SAFETY: in-memory, ON PURPOSE ────────────────────────────────────────────────────── // // This state is deliberately NOT persisted. A controller restart clears it, so the next cycle // attempts the backup immediately. That is the direction this should fail in: forgetting a backoff // costs one extra attempt, whereas persisting it could carry a stale "this tier is broken" verdict // across a restart that actually fixed the tier. Do not "fix" this into persistence without deciding // which way you want it to fail — the cheap failure is the one we chose. const ( // breakerBaseDelay is the first backoff. It must exceed the poll interval (5m) by enough that the // thrash stops immediately: at 15m the very first failure already skips two ticks. breakerBaseDelay = 15 * time.Minute // breakerMaxDelay caps the interval. 4h is picked against two real constants rather than taste: // it sits well inside the SHORTEST tier cadence (local = 24h), so a tier that recovers still gets // several attempts within its own cadence; and it equals the width of the backup window gate // [W+2h, W+6h), so a tier at maximum backoff still gets at least one attempt inside any given // night's window instead of stepping over it entirely. breakerMaxDelay = 4 * time.Hour // breakerMaxShift bounds the doubling so a long-broken tier cannot overflow the shift. 15m << 5 // is already past the cap, so this is a guard, not a policy. breakerMaxShift = 5 ) // TierNotifier is the seam by which a whole-guest backup outcome reaches the hub (R-97a). // // WHY A SEAM AND NOT AN IMPORT: `internal/quiesce` deliberately keeps no dependency on // `internal/notify` — the same reason `windowStartFn` is injected rather than importing `settings`. // It is wired by an init-only setter (`SetTierNotifier`) because the notifier is constructed AFTER // the quiesce loop in main.go; nil means "not wired", which is the pre-provisioning case, not an // error. // // EDGE-TRIGGERED, ON PURPOSE. `BackupFailed` fires when the breaker ARMS — i.e. on the first failure // of a run — never on the retries behind it. The retry cadence is 15m/30m/1h/2h/4h and an event per // attempt is an inbox nobody reads. `BackupRecovered` fires when a tier that HAD been failing // succeeds, so the operator who was told it broke is also told it healed. // // OPERATOR-TIER ONLY. A customer can take no action on a failed whole-guest backup, and telling them // it failed while it is still retrying is alarming without being actionable. type TierNotifier interface { BackupFailed(tier, message, errMsg string) BackupRecovered(tier, message string) } // breakerState is one tier's consecutive-failure record. type breakerState struct { failures int until time.Time } // failureBreaker tracks consecutive backup failures per target and defers the quiesce accordingly. // The zero value is not usable — build it with newFailureBreaker. type failureBreaker struct { mu sync.Mutex states map[string]breakerState } func newFailureBreaker() *failureBreaker { return &failureBreaker{states: map[string]breakerState{}} } // backoffFor is the delay after n consecutive failures: 15m, 30m, 1h, 2h, 4h, then 4h forever. // PURE, so the schedule is a unit-testable contract rather than an emergent property of the loop. func backoffFor(failures int) time.Duration { if failures <= 0 { return 0 } shift := failures - 1 if shift > breakerMaxShift { shift = breakerMaxShift } d := breakerBaseDelay << uint(shift) if d > breakerMaxDelay { return breakerMaxDelay } return d } // blocked reports whether target is currently deferred, and until when. func (b *failureBreaker) blocked(target string, now time.Time) (time.Time, bool) { b.mu.Lock() defer b.mu.Unlock() st, ok := b.states[target] if !ok || st.until.IsZero() || !now.Before(st.until) { return time.Time{}, false } return st.until, true } // recordFailure increments the tier's consecutive-failure count and arms the next backoff. Returns // the new count and delay so the caller can log the deferral ONCE, at the moment it is armed — a // 5-minute loop that logged on every skipped tick would bury the signal it exists to raise. func (b *failureBreaker) recordFailure(target string, now time.Time) (int, time.Duration) { b.mu.Lock() defer b.mu.Unlock() st := b.states[target] st.failures++ d := backoffFor(st.failures) st.until = now.Add(d) b.states[target] = st return st.failures, d } // recordSuccess clears the tier's backoff. Returns true if there was one to clear, so the caller can // log the recovery without narrating every healthy backup. Scenario B: normal cadence resumes on the // very next cycle — a box that recovers carries no lingering penalty. func (b *failureBreaker) recordSuccess(target string) bool { b.mu.Lock() defer b.mu.Unlock() if _, ok := b.states[target]; !ok { return false } delete(b.states, target) return true } // failuresFor exposes the consecutive-failure count (tests + diagnosis). func (b *failureBreaker) failuresFor(target string) int { b.mu.Lock() defer b.mu.Unlock() return b.states[target].failures }