R-88 Part 1: a failing backup stops re-quiescing (v0.176.0)

internal/quiesce had no failure counter, no backoff and no breaker, and the driver
is a plain 5-minute ticker — so a tier that was due and kept failing stopped and
restarted every customer app stack every 5 minutes indefinitely. Live on
demo-felhom 2026-07-27: three cycles in eleven minutes against an unreachable PBS
tier; it ended only because PBS recovered.

The breaker gates the QUIESCE, not the backup — the harm was the outage taken to
attempt it, so backed-off tiers are dropped before any stack is stopped. Per
target (a broken offsite tier must not suppress a healthy local one), 15m→30m→
1h→2h→4h capped, reset on success, never permanent, never applied to TriggerNow,
and stillRunning is not a failure. State is in-memory on purpose: forgetting a
backoff costs one attempt; persisting one could outlive the fix.

Part 3 invariant recorded on scheduledRunAllowed — a missing value means UNKNOWN,
and only a positive 'never' may fire the safety valve. Fourth instance of the
class (hub v0.12.0, v0.73.0, R-81, R-88).

Part 2 (unknown != never) is NOT in this commit: the agent returns byte-identical
responses for 'read errored' and 'never backed up', so the controller cannot tell
them apart. That needs an agent wire change and is tracked separately.

controller/internal/quiesce/quiesce.go

@@ -100,6 +100,9 @@ type Loop struct {
 	mu sync.Mutex
 	// degradeOnce reports the pre-R-82 agent fallback exactly once per process (see tiers.go).
 	degradeOnce sync.Once
+	// breaker (R-88) defers the QUIESCE for a tier whose backups keep failing, so a broken target
+	// cannot stop the customer's apps every 5 minutes forever. Scheduled path only — see breaker.go.
+	breaker *failureBreaker
 }
 
 // New builds a Loop with sane defaults for any unset duration.
@@ -124,6 +127,7 @@ func New(o Options) *Loop {
 		poll: o.Poll, statusPoll: o.StatusPoll, maxQuiesce: o.MaxQuiesce,
 		logger: o.Logger, now: time.Now,
 		windowStartFn: o.WindowStartFn, cadence: o.Cadence,
+		breaker: newFailureBreaker(),
 	}
 }
 
@@ -189,6 +193,15 @@ func (l *Loop) runOnce(ctx context.Context) error {
 		return nil
 	}
 
+	// R-88 breaker — drop tiers whose backups keep failing, BEFORE anything is stopped. This is the
+	// gate that actually ends the app thrash: the harm was never the failing backup, it was the
+	// outage taken to attempt it. Per-tier, so a broken offsite tier leaves a healthy local one alone.
+	// SCHEDULED path only — TriggerNow never reaches here.
+	dueTiers = l.dropBackedOffTiers(dueTiers)
+	if len(dueTiers) == 0 {
+		return nil
+	}
+
 	// Window gate (Part 3) — SCHEDULED path only. TriggerNow calls quiesceAndPoll directly and is
 	// never gated. Disabled when no window fn is wired (pre-v0.168.0 behavior).
 	//
@@ -207,6 +220,42 @@ func (l *Loop) runOnce(ctx context.Context) error {
 	return l.quiesceAndPollTiers(ctx, dueTiers)
 }
 
+// noteTierFailure arms/extends the tier's backoff and announces the deferral exactly ONCE — here, at
+// the moment it is armed. Called from BOTH the scheduled and the manual path: a manual run that
+// fails is evidence about the tier too. Only the GATING is scheduler-only.
+func (l *Loop) noteTierFailure(target, label string) {
+	n, d := l.breaker.recordFailure(target, l.now())
+	l.logger.Printf("[WARN] [quiesce] tier %s has now failed %d time(s) in a row — deferring its next quiesce by %s (cap %s) so the apps are not stopped again for a backup that cannot succeed",
+		label, n, d, breakerMaxDelay)
+}
+
+// noteTierSuccess clears any backoff. Quiet unless there was something to clear — a line per healthy
+// backup would be noise, but a recovery is worth one.
+func (l *Loop) noteTierSuccess(target, label string) {
+	if l.breaker.recordSuccess(target) {
+		l.logger.Printf("[INFO] [quiesce] tier %s succeeded — clearing its failure backoff; normal cadence resumes", label)
+	}
+}
+
+// dropBackedOffTiers removes tiers currently inside an R-88 backoff window.
+//
+// Silent by design at INFO: the deferral is announced ONCE, when the backoff is armed in
+// quiesceAndPollTiers. Logging here would fire every 5 minutes for hours and bury the one line that
+// matters — the same "a repeating log is not a signal" problem the loop itself had.
+func (l *Loop) dropBackedOffTiers(tiers []dueTier) []dueTier {
+	now := l.now()
+	kept := make([]dueTier, 0, len(tiers))
+	for _, t := range tiers {
+		if until, blocked := l.breaker.blocked(t.target, now); blocked {
+			l.logger.Printf("[DEBUG] [quiesce] tier %s is in backoff after %d consecutive failure(s) — not quiescing until %s",
+				tierLabel(t.target), l.breaker.failuresFor(t.target), until.Format(time.RFC3339))
+			continue
+		}
+		kept = append(kept, t)
+	}
+	return kept
+}
+
 // oldestAge returns the largest (most overdue) age among the due tiers; nil when any tier has never
 // backed up (nil age = "never", which is maximally overdue and must win).
 func oldestAge(tiers []dueTier) *int64 {
@@ -341,6 +390,7 @@ func (l *Loop) quiesceAndPollTiers(ctx context.Context, tiers []dueTier) error {
 		jobID, err := l.startBackupOn(ctx, t.target)
 		if err != nil {
 			l.logger.Printf("[ERROR] [quiesce] start backup on tier %s: %v", label, err)
+			l.noteTierFailure(t.target, label)
 			if firstErr == nil {
 				firstErr = fmt.Errorf("start backup on %s: %w", label, err)
 			}
@@ -358,8 +408,15 @@ func (l *Loop) quiesceAndPollTiers(ctx context.Context, tiers []dueTier) error {
 		if perr != nil && firstErr == nil {
 			firstErr = perr
 		}
-		if phase == phaseFailed {
+		switch {
+		case phase == phaseFailed:
 			l.logger.Printf("[WARN] [quiesce] tier %s: backup job %s failed", label, jobID)
+			l.noteTierFailure(t.target, label)
+		case stillRunning:
+			// Neither outcome yet — a first full offsite snapshot legitimately runs for hours. It
+			// must NOT count as a failure, or a slow-but-healthy tier would back itself off.
+		default:
+			l.noteTierSuccess(t.target, label)
 		}
 		if stillRunning {
 			// The max-quiesce guard fired while THIS tier's backup is still going (a first full
@@ -463,6 +520,36 @@ const (
 // window [W+2h, W+6h); otherwise true ONLY if the safety valve holds — the newest successful backup is
 // missing (nil) or older than cadence+24h — so a box powered on only outside its window never starves.
 // An unparseable window fails OPEN (allow) rather than block backups forever.
+//
+// ── THE INVARIANT (R-88). READ THIS BEFORE TOUCHING THE nil BRANCH. ──────────────────────────
+//
+// A missing value means UNKNOWN. It does not mean zero, and it does not mean "never backed up".
+// Only a POSITIVE determination of "never backed up" may fire the safety valve.
+//
+// This project has now made the opposite mistake four times, in four different packages:
+// hub v0.12.0, hub v0.73.0, R-81 (hub `assessBackupFreshness`), and R-88 (here). Each time, the
+// absence of a signal was read as a specific value, and each time the fix was the same shape:
+// give "unknown" its own representation instead of letting it collapse into a real answer.
+//
+// ── WHAT IS AND IS NOT FIXED HERE ────────────────────────────────────────────────────────────
+//
+// The nil branch below STILL fires the valve, and that is currently correct-by-necessity, not by
+// design: the controller cannot yet tell the two apart. The agent's `/backup/due` returns
+// BYTE-IDENTICAL responses for "the storage read errored" and "there has genuinely never been a
+// backup" — same `Due: true`, same `Reason: "no successful backup recorded yet"`, same nil
+// `AgeSecs`. The root cause is agent-side: `newestArchiveOn` (localapi/server.go) documents that
+// errors "degrade to unknown, never to no-backup", but its `(time.Time, bool)` signature cannot
+// represent unknown, so the error collapses into a positive claim of "never".
+//
+// Distinguishing them needs a new field on `/backup/due` plus a compat rule in both directions →
+// tracked as its own task (R-88 Part 2, agent-side). Until then the R-88 BREAKER is what bounds the
+// damage: an unknown-driven cycle may still run once outside the window, but it can no longer repeat
+// every 5 minutes.
+//
+// DO NOT "fix" this by deleting the nil branch. Scenario D — a genuinely never-backed-up box that is
+// only ever powered on outside its window — depends on it, and TestContract_NeverBackedUp_RunsOutside
+// -TheWindow will fail if you do. Silencing the valve would trade a loud bug for a silent one: a box
+// that never backs up at all, with nobody noticing for weeks.
 func scheduledRunAllowed(now time.Time, windowStart string, lastAgeSecs *int64, cadence time.Duration) bool {
 	startMin, err := backupwindow.ParseHHMM(windowStart)
 	if err != nil {