felhom-controller/controller/internal/quiesce/breaker.go

package quiesce

import (
	"sync"
	"time"
)

// R-88 — the failure breaker.
//
// THE BUG THIS EXISTS TO KILL: before this, `internal/quiesce` had no consecutive-failure counter,
// no backoff and no circuit breaker of any kind. The driver is a plain 5-minute ticker, so a tier
// that was due and kept failing was re-quiesced every five minutes FOREVER — and a quiesce cycle
// stops and restarts every customer app stack. Observed live on demo-felhom 2026-07-27: three full
// stop/start cycles across eleven minutes (09:02:57, 09:07:58, 09:12:57 Budapest) against a PBS tier
// that could not possibly succeed. It stopped after three only because PBS came back, not because
// anything gave up.
//
// The harm is NOT the failing backup — it is the app outage taken to attempt it. So the breaker gates
// the QUIESCE, not the backup: a tier in backoff is dropped from the due set before any stack is
// stopped.
//
// ── WHAT THIS DELIBERATELY IS NOT ────────────────────────────────────────────────────────────
//
// It is NOT permanent, and it must never become permanent. The cap bounds the retry INTERVAL; it
// never stops retrying. A breaker that latches open is a silent backup outage, which is strictly
// worse than the loop it replaces — the loop at least announced itself by stopping the apps.
//
// It is NOT global. State is per TARGET (`dueTier.target`), so a broken offsite tier cannot suppress
// a healthy local one. Halting all backups because one tier is down would trade a narrow fault for a
// total one.
//
// It does NOT gate the manual path. `TriggerNow` bypasses due-ness and the window gate, and it
// bypasses this too — a human pressing „Mentés most" has made an explicit decision and must not be
// deferred by a breaker built for the scheduler. Manual runs still RECORD their outcome (a manual
// success clears the backoff, which is exactly what an operator fixing the tier expects).
//
// ── CRASH SAFETY: in-memory, ON PURPOSE ──────────────────────────────────────────────────────
//
// This state is deliberately NOT persisted. A controller restart clears it, so the next cycle
// attempts the backup immediately. That is the direction this should fail in: forgetting a backoff
// costs one extra attempt, whereas persisting it could carry a stale "this tier is broken" verdict
// across a restart that actually fixed the tier. Do not "fix" this into persistence without deciding
// which way you want it to fail — the cheap failure is the one we chose.
const (
	// breakerBaseDelay is the first backoff. It must exceed the poll interval (5m) by enough that the
	// thrash stops immediately: at 15m the very first failure already skips two ticks.
	breakerBaseDelay = 15 * time.Minute

	// breakerMaxDelay caps the interval. 4h is picked against two real constants rather than taste:
	// it sits well inside the SHORTEST tier cadence (local = 24h), so a tier that recovers still gets
	// several attempts within its own cadence; and it equals the width of the backup window gate
	// [W+2h, W+6h), so a tier at maximum backoff still gets at least one attempt inside any given
	// night's window instead of stepping over it entirely.
	breakerMaxDelay = 4 * time.Hour

	// breakerMaxShift bounds the doubling so a long-broken tier cannot overflow the shift. 15m << 5
	// is already past the cap, so this is a guard, not a policy.
	breakerMaxShift = 5
)

// TierNotifier is the seam by which a whole-guest backup outcome reaches the hub (R-97a).
//
// WHY A SEAM AND NOT AN IMPORT: `internal/quiesce` deliberately keeps no dependency on
// `internal/notify` — the same reason `windowStartFn` is injected rather than importing `settings`.
// It is wired by an init-only setter (`SetTierNotifier`) because the notifier is constructed AFTER
// the quiesce loop in main.go; nil means "not wired", which is the pre-provisioning case, not an
// error.
//
// EDGE-TRIGGERED, ON PURPOSE. `BackupFailed` fires when the breaker ARMS — i.e. on the first failure
// of a run — never on the retries behind it. The retry cadence is 15m/30m/1h/2h/4h and an event per
// attempt is an inbox nobody reads. `BackupRecovered` fires when a tier that HAD been failing
// succeeds, so the operator who was told it broke is also told it healed.
//
// OPERATOR-TIER ONLY. A customer can take no action on a failed whole-guest backup, and telling them
// it failed while it is still retrying is alarming without being actionable.
type TierNotifier interface {
	BackupFailed(tier, message, errMsg string)
	BackupRecovered(tier, message string)
}

// breakerState is one tier's consecutive-failure record.
type breakerState struct {
	failures int
	until    time.Time
}

// failureBreaker tracks consecutive backup failures per target and defers the quiesce accordingly.
// The zero value is not usable — build it with newFailureBreaker.
type failureBreaker struct {
	mu     sync.Mutex
	states map[string]breakerState
}

func newFailureBreaker() *failureBreaker {
	return &failureBreaker{states: map[string]breakerState{}}
}

// backoffFor is the delay after n consecutive failures: 15m, 30m, 1h, 2h, 4h, then 4h forever.
// PURE, so the schedule is a unit-testable contract rather than an emergent property of the loop.
func backoffFor(failures int) time.Duration {
	if failures <= 0 {
		return 0
	}
	shift := failures - 1
	if shift > breakerMaxShift {
		shift = breakerMaxShift
	}
	d := breakerBaseDelay << uint(shift)
	if d > breakerMaxDelay {
		return breakerMaxDelay
	}
	return d
}

// blocked reports whether target is currently deferred, and until when.
func (b *failureBreaker) blocked(target string, now time.Time) (time.Time, bool) {
	b.mu.Lock()
	defer b.mu.Unlock()
	st, ok := b.states[target]
	if !ok || st.until.IsZero() || !now.Before(st.until) {
		return time.Time{}, false
	}
	return st.until, true
}

// recordFailure increments the tier's consecutive-failure count and arms the next backoff. Returns
// the new count and delay so the caller can log the deferral ONCE, at the moment it is armed — a
// 5-minute loop that logged on every skipped tick would bury the signal it exists to raise.
func (b *failureBreaker) recordFailure(target string, now time.Time) (int, time.Duration) {
	b.mu.Lock()
	defer b.mu.Unlock()
	st := b.states[target]
	st.failures++
	d := backoffFor(st.failures)
	st.until = now.Add(d)
	b.states[target] = st
	return st.failures, d
}

// recordSuccess clears the tier's backoff. Returns true if there was one to clear, so the caller can
// log the recovery without narrating every healthy backup. Scenario B: normal cadence resumes on the
// very next cycle — a box that recovers carries no lingering penalty.
func (b *failureBreaker) recordSuccess(target string) bool {
	b.mu.Lock()
	defer b.mu.Unlock()
	if _, ok := b.states[target]; !ok {
		return false
	}
	delete(b.states, target)
	return true
}

// failuresFor exposes the consecutive-failure count (tests + diagnosis).
func (b *failureBreaker) failuresFor(target string) int {
	b.mu.Lock()
	defer b.mu.Unlock()
	return b.states[target].failures
}