R-88 Part 1: a failing backup stops re-quiescing (v0.176.0)
internal/quiesce had no failure counter, no backoff and no breaker, and the driver is a plain 5-minute ticker — so a tier that was due and kept failing stopped and restarted every customer app stack every 5 minutes indefinitely. Live on demo-felhom 2026-07-27: three cycles in eleven minutes against an unreachable PBS tier; it ended only because PBS recovered. The breaker gates the QUIESCE, not the backup — the harm was the outage taken to attempt it, so backed-off tiers are dropped before any stack is stopped. Per target (a broken offsite tier must not suppress a healthy local one), 15m→30m→ 1h→2h→4h capped, reset on success, never permanent, never applied to TriggerNow, and stillRunning is not a failure. State is in-memory on purpose: forgetting a backoff costs one attempt; persisting one could outlive the fix. Part 3 invariant recorded on scheduledRunAllowed — a missing value means UNKNOWN, and only a positive 'never' may fire the safety valve. Fourth instance of the class (hub v0.12.0, v0.73.0, R-81, R-88). Part 2 (unknown != never) is NOT in this commit: the agent returns byte-identical responses for 'read errored' and 'never backed up', so the controller cannot tell them apart. That needs an agent wire change and is tracked separately.
This commit is contained in:
@@ -1,5 +1,55 @@
|
||||
## Changelog
|
||||
|
||||
### v0.176.0 — R-88 Part 1: a failing backup stops re-quiescing (2026-07-27)
|
||||
|
||||
**The apps were being stopped and restarted every five minutes for a backup that could not
|
||||
succeed.** Observed live on demo-felhom 2026-07-27: three full quiesce cycles at 09:02:57, 09:07:58
|
||||
and 09:12:57 Budapest — each stopping and restarting all four customer app stacks
|
||||
(`bookstack calibre-web docmost immich`, ~19 s down per cycle) against a PBS tier that was
|
||||
unreachable. It stopped after three only because PBS came back, **not** because anything gave up:
|
||||
`internal/quiesce` had no consecutive-failure counter, no backoff and no circuit breaker of any
|
||||
kind, and the driver is a plain 5-minute ticker. Had the outage lasted, so would the loop.
|
||||
|
||||
**The failure breaker** (`internal/quiesce/breaker.go`). Consecutive failures are tracked **per
|
||||
target**; a tier inside its backoff is dropped from the due set **before any stack is stopped** —
|
||||
the gate is on the QUIESCE, not the backup, because the harm was never the failing backup but the
|
||||
outage taken to attempt it. Backoff is `15m → 30m → 1h → 2h → 4h`, then 4h forever.
|
||||
|
||||
The cap is picked against two real constants rather than taste: 4h sits well inside the shortest
|
||||
tier cadence (local = 24h), so a recovered tier still gets several attempts within its own cadence;
|
||||
and it equals the width of the backup window gate `[W+2h, W+6h)`, so a tier at maximum backoff still
|
||||
gets at least one attempt inside any given night's window instead of stepping over it.
|
||||
|
||||
Deliberately bounded in four ways, each with a test:
|
||||
- **Never permanent.** The cap bounds the retry INTERVAL; it never stops retrying. A latched breaker
|
||||
is a silent backup outage — strictly worse than the loop, which at least announced itself.
|
||||
- **Never global.** One broken tier cannot suppress a healthy one.
|
||||
- **Never gates `TriggerNow`.** A human pressing „Mentés most" is not deferred by a scheduler's
|
||||
safety net. Manual runs still RECORD their outcome, so a manual success clears the backoff.
|
||||
- **`stillRunning` is not a failure.** A first full offsite snapshot legitimately runs for hours.
|
||||
|
||||
State is **in-memory on purpose** — a restart forgets the backoff and re-attempts, which is the
|
||||
cheap direction to fail; persisting it could carry a stale "this tier is broken" verdict across the
|
||||
very restart that fixed it.
|
||||
|
||||
**The invariant, written where it will be read** (`scheduledRunAllowed`). A missing value means
|
||||
UNKNOWN — not zero, not "never". Only a POSITIVE determination of "never backed up" may fire the
|
||||
safety valve. This is the **fourth** instance of the same class (hub v0.12.0, hub v0.73.0, R-81, and
|
||||
this), so the comment names all four and `TestContract_NeverBackedUp_RunsOutsideTheWindow` pins the
|
||||
half that a careless fix would break.
|
||||
|
||||
**NOT fixed here, and deliberately so — R-88 Part 2 (agent-side).** The nil branch still fires the
|
||||
valve, because the controller *cannot tell the two apart*: the agent's `/backup/due` returns
|
||||
byte-identical responses for "the storage read errored" and "there has genuinely never been a
|
||||
backup" — same `Due: true`, same `Reason: "no successful backup recorded yet"`, same nil `AgeSecs`.
|
||||
Root cause is `localapi/server.go`'s `newestArchiveOn`, whose comment promises errors "degrade to
|
||||
unknown, never to no-backup" while its `(time.Time, bool)` signature cannot represent unknown.
|
||||
Splitting them needs a wire change plus a compat rule in both directions → its own task. Until then
|
||||
the breaker bounds the damage: an unknown-driven cycle may still run once outside the window, but it
|
||||
can no longer repeat.
|
||||
|
||||
Tests +11 (7 breaker, 4 contract). Red-proofs observed for Scenarios A, D and F.
|
||||
|
||||
### v0.175.0 — R-82: a tier that overruns the quiesce bound defers the rest (2026-07-26)
|
||||
|
||||
Operator ruling 2026-07-26: *"let the first backup run as long as needed; other backups shouldn't
|
||||
|
||||
Reference in New Issue
Block a user