R-97: a failing backup is heard, and stops blaming the apps (v0.177.0)
R-97a: internal/quiesce had no route to the hub at all — three failed whole-guest backups on 2026-07-27 produced zero events. TierNotifier is a seam (not an import), wired by an init-only setter because main.go builds the notifier after the loop. Edge-triggered: the failure fires when the R-88 breaker ARMS, not per retry, and recovery rides recordSuccess's existing bool. Uses NEW operator-only event types; reusing backup_failed would have emailed the customer in Hungarian about a backup they cannot act on, since it has a customerMessages entry and is in live enabled_events. Requires hub >= v0.78.0. R-97b: v0.164.0's state filter cannot see an app caught MID-RESTART, which is how BookStack alarmed. The fix is a suppression window keyed to the quiesce CYCLE, consumed at the same single derivation point. 180s grace, derived from the deploy flow's 120s health timeout and Mealie's 60s start_period; it expires, so an app that genuinely fails to come back still alarms.
This commit is contained in:
@@ -1,5 +1,51 @@
|
||||
## Changelog
|
||||
|
||||
### v0.177.0 — R-97: a failing backup is HEARD, and stops blaming the apps (2026-07-27) — MinAgent unchanged; requires hub >= v0.78.0
|
||||
|
||||
**R-97a — the whole-guest tier had no route to the hub.** `internal/quiesce` did not import
|
||||
`internal/notify` at all, so on 2026-07-27 three failed whole-guest backups and twelve app-stack
|
||||
stop/starts produced **zero** events. `NotifyBackupFailed` existed and the hub allowlisted
|
||||
`backup_failed`; only the wiring was missing — the inert-seam shape this project has now hit five times.
|
||||
|
||||
This got MORE urgent when R-88 shipped, not less. Before the breaker a failing backup retried every
|
||||
5 minutes: harmful, but loud enough to notice. Now it backs off to 4h and goes quiet, leaving the
|
||||
hub's deadline monitor as the only signal — **~26h for local, ~8 days for PBS**, a full cycle of the
|
||||
weekly tier. This trades that delay for an immediate one.
|
||||
|
||||
`quiesce.TierNotifier` is a seam, not an import (same reason `windowStartFn` is injected), wired by
|
||||
the init-only `SetTierNotifier` because main.go builds the notifier *after* the loop. It is
|
||||
**edge-triggered**: `BackupFailed` fires when the breaker ARMS — the first failure of a run, never
|
||||
the retries behind it — and `BackupRecovered` on `recordSuccess`'s existing bool, so an operator told
|
||||
a tier broke is also told it healed.
|
||||
|
||||
**New OPERATOR-ONLY event types**, `whole_guest_backup_failed` / `_recovered` (hub v0.78.0).
|
||||
Deliberately NOT `backup_failed`: that type carries a customer-facing Hungarian template **and** sits
|
||||
in demo-felhom's live `enabled_events`, so reusing it would have emailed the CUSTOMER
|
||||
„A biztonsági mentés sikertelen" while the backup was still retrying. The tier travels in
|
||||
`WholeGuestBackupDetails.Tier`, which is load-bearing — the hub keys its per-tier operator cooldown
|
||||
on it, so `local` failing is not swallowed by `felhom-pbs` having failed within the hour.
|
||||
|
||||
**R-97b — stop telling the customer their app is broken when WE stopped it.** During the loop the only
|
||||
customer-visible output was `app_start_failed — „Telepített alkalmazás nem fut: BookStack"`: customer
|
||||
channel, Hungarian, during an outage the backup system itself caused, with no indication why.
|
||||
|
||||
**v0.164.0's filter does not cover this.** That predicate is state-based
|
||||
(`IsDownState(st.State) && st.State != StateStopped`) and suppresses *deliberately stopped* apps.
|
||||
BookStack alarmed because the third cycle caught it **mid-restart** — starting, or up but not yet
|
||||
healthy — which is not `StateStopped`. No state classification can tell "restarting because a backup
|
||||
stopped me" from "restarting because I keep crashing"; the distinguishing fact is that *we* stopped
|
||||
it, and we know we did. So the fix is a **suppression window keyed to the cycle**, consumed at the
|
||||
same single derivation point (`classifyRunStates`) that already computes both the banner dead-list
|
||||
and the notifier Down-set — still one place.
|
||||
|
||||
**The grace window is 180 s**, derived rather than picked round: the deploy flow already allows
|
||||
**120 s** for a stack to come up healthy, and the slowest catalog healthcheck start_period is Mealie's
|
||||
**60 s**, after which a couple of check intervals must still elapse. It **expires** — an app that
|
||||
genuinely fails to come back alarms on the first scan after the window closes. Permanent suppression
|
||||
would trade a loud false alarm for a silent real one, which is R-88's Scenario D in a new costume.
|
||||
|
||||
Tests +9 (8 quiesce + 1 wiring reachability). Red-proofs observed for Scenarios C, E and F.
|
||||
|
||||
### v0.176.0 — R-88 Part 1: a failing backup stops re-quiescing (2026-07-27)
|
||||
|
||||
**The apps were being stopped and restarted every five minutes for a backup that could not
|
||||
|
||||
Reference in New Issue
Block a user