R-88 Part 1: a failing backup stops re-quiescing (v0.176.0)

internal/quiesce had no failure counter, no backoff and no breaker, and the driver
is a plain 5-minute ticker — so a tier that was due and kept failing stopped and
restarted every customer app stack every 5 minutes indefinitely. Live on
demo-felhom 2026-07-27: three cycles in eleven minutes against an unreachable PBS
tier; it ended only because PBS recovered.

The breaker gates the QUIESCE, not the backup — the harm was the outage taken to
attempt it, so backed-off tiers are dropped before any stack is stopped. Per
target (a broken offsite tier must not suppress a healthy local one), 15m→30m→
1h→2h→4h capped, reset on success, never permanent, never applied to TriggerNow,
and stillRunning is not a failure. State is in-memory on purpose: forgetting a
backoff costs one attempt; persisting one could outlive the fix.

Part 3 invariant recorded on scheduledRunAllowed — a missing value means UNKNOWN,
and only a positive 'never' may fire the safety valve. Fourth instance of the
class (hub v0.12.0, v0.73.0, R-81, R-88).

Part 2 (unknown != never) is NOT in this commit: the agent returns byte-identical
responses for 'read errored' and 'never backed up', so the controller cannot tell
them apart. That needs an agent wire change and is tracked separately.
This commit is contained in:
2026-07-27 16:21:58 +02:00
parent 3f0420ff9c
commit 32200c7b5f
6 changed files with 677 additions and 1 deletions
+50
View File
@@ -1,5 +1,55 @@
## Changelog
### v0.176.0 — R-88 Part 1: a failing backup stops re-quiescing (2026-07-27)
**The apps were being stopped and restarted every five minutes for a backup that could not
succeed.** Observed live on demo-felhom 2026-07-27: three full quiesce cycles at 09:02:57, 09:07:58
and 09:12:57 Budapest — each stopping and restarting all four customer app stacks
(`bookstack calibre-web docmost immich`, ~19 s down per cycle) against a PBS tier that was
unreachable. It stopped after three only because PBS came back, **not** because anything gave up:
`internal/quiesce` had no consecutive-failure counter, no backoff and no circuit breaker of any
kind, and the driver is a plain 5-minute ticker. Had the outage lasted, so would the loop.
**The failure breaker** (`internal/quiesce/breaker.go`). Consecutive failures are tracked **per
target**; a tier inside its backoff is dropped from the due set **before any stack is stopped**
the gate is on the QUIESCE, not the backup, because the harm was never the failing backup but the
outage taken to attempt it. Backoff is `15m → 30m → 1h → 2h → 4h`, then 4h forever.
The cap is picked against two real constants rather than taste: 4h sits well inside the shortest
tier cadence (local = 24h), so a recovered tier still gets several attempts within its own cadence;
and it equals the width of the backup window gate `[W+2h, W+6h)`, so a tier at maximum backoff still
gets at least one attempt inside any given night's window instead of stepping over it.
Deliberately bounded in four ways, each with a test:
- **Never permanent.** The cap bounds the retry INTERVAL; it never stops retrying. A latched breaker
is a silent backup outage — strictly worse than the loop, which at least announced itself.
- **Never global.** One broken tier cannot suppress a healthy one.
- **Never gates `TriggerNow`.** A human pressing „Mentés most" is not deferred by a scheduler's
safety net. Manual runs still RECORD their outcome, so a manual success clears the backoff.
- **`stillRunning` is not a failure.** A first full offsite snapshot legitimately runs for hours.
State is **in-memory on purpose** — a restart forgets the backoff and re-attempts, which is the
cheap direction to fail; persisting it could carry a stale "this tier is broken" verdict across the
very restart that fixed it.
**The invariant, written where it will be read** (`scheduledRunAllowed`). A missing value means
UNKNOWN — not zero, not "never". Only a POSITIVE determination of "never backed up" may fire the
safety valve. This is the **fourth** instance of the same class (hub v0.12.0, hub v0.73.0, R-81, and
this), so the comment names all four and `TestContract_NeverBackedUp_RunsOutsideTheWindow` pins the
half that a careless fix would break.
**NOT fixed here, and deliberately so — R-88 Part 2 (agent-side).** The nil branch still fires the
valve, because the controller *cannot tell the two apart*: the agent's `/backup/due` returns
byte-identical responses for "the storage read errored" and "there has genuinely never been a
backup" — same `Due: true`, same `Reason: "no successful backup recorded yet"`, same nil `AgeSecs`.
Root cause is `localapi/server.go`'s `newestArchiveOn`, whose comment promises errors "degrade to
unknown, never to no-backup" while its `(time.Time, bool)` signature cannot represent unknown.
Splitting them needs a wire change plus a compat rule in both directions → its own task. Until then
the breaker bounds the damage: an unknown-driven cycle may still run once outside the window, but it
can no longer repeat.
Tests +11 (7 breaker, 4 contract). Red-proofs observed for Scenarios A, D and F.
### v0.175.0 — R-82: a tier that overruns the quiesce bound defers the rest (2026-07-26)
Operator ruling 2026-07-26: *"let the first backup run as long as needed; other backups shouldn't