R-97: a failing backup is heard, and stops blaming the apps (v0.177.0)

R-97a: internal/quiesce had no route to the hub at all — three failed whole-guest
backups on 2026-07-27 produced zero events. TierNotifier is a seam (not an import),
wired by an init-only setter because main.go builds the notifier after the loop.
Edge-triggered: the failure fires when the R-88 breaker ARMS, not per retry, and
recovery rides recordSuccess's existing bool. Uses NEW operator-only event types;
reusing backup_failed would have emailed the customer in Hungarian about a backup
they cannot act on, since it has a customerMessages entry and is in live
enabled_events. Requires hub >= v0.78.0.

R-97b: v0.164.0's state filter cannot see an app caught MID-RESTART, which is how
BookStack alarmed. The fix is a suppression window keyed to the quiesce CYCLE,
consumed at the same single derivation point. 180s grace, derived from the deploy
flow's 120s health timeout and Mealie's 60s start_period; it expires, so an app
that genuinely fails to come back still alarms.
This commit is contained in:
2026-07-27 17:01:41 +02:00
parent ccefff4f39
commit e9c99566b0
10 changed files with 559 additions and 11 deletions
+46
View File
@@ -1,5 +1,51 @@
## Changelog
### v0.177.0 — R-97: a failing backup is HEARD, and stops blaming the apps (2026-07-27) — MinAgent unchanged; requires hub >= v0.78.0
**R-97a — the whole-guest tier had no route to the hub.** `internal/quiesce` did not import
`internal/notify` at all, so on 2026-07-27 three failed whole-guest backups and twelve app-stack
stop/starts produced **zero** events. `NotifyBackupFailed` existed and the hub allowlisted
`backup_failed`; only the wiring was missing — the inert-seam shape this project has now hit five times.
This got MORE urgent when R-88 shipped, not less. Before the breaker a failing backup retried every
5 minutes: harmful, but loud enough to notice. Now it backs off to 4h and goes quiet, leaving the
hub's deadline monitor as the only signal — **~26h for local, ~8 days for PBS**, a full cycle of the
weekly tier. This trades that delay for an immediate one.
`quiesce.TierNotifier` is a seam, not an import (same reason `windowStartFn` is injected), wired by
the init-only `SetTierNotifier` because main.go builds the notifier *after* the loop. It is
**edge-triggered**: `BackupFailed` fires when the breaker ARMS — the first failure of a run, never
the retries behind it — and `BackupRecovered` on `recordSuccess`'s existing bool, so an operator told
a tier broke is also told it healed.
**New OPERATOR-ONLY event types**, `whole_guest_backup_failed` / `_recovered` (hub v0.78.0).
Deliberately NOT `backup_failed`: that type carries a customer-facing Hungarian template **and** sits
in demo-felhom's live `enabled_events`, so reusing it would have emailed the CUSTOMER
„A biztonsági mentés sikertelen" while the backup was still retrying. The tier travels in
`WholeGuestBackupDetails.Tier`, which is load-bearing — the hub keys its per-tier operator cooldown
on it, so `local` failing is not swallowed by `felhom-pbs` having failed within the hour.
**R-97b — stop telling the customer their app is broken when WE stopped it.** During the loop the only
customer-visible output was `app_start_failed — „Telepített alkalmazás nem fut: BookStack"`: customer
channel, Hungarian, during an outage the backup system itself caused, with no indication why.
**v0.164.0's filter does not cover this.** That predicate is state-based
(`IsDownState(st.State) && st.State != StateStopped`) and suppresses *deliberately stopped* apps.
BookStack alarmed because the third cycle caught it **mid-restart** — starting, or up but not yet
healthy — which is not `StateStopped`. No state classification can tell "restarting because a backup
stopped me" from "restarting because I keep crashing"; the distinguishing fact is that *we* stopped
it, and we know we did. So the fix is a **suppression window keyed to the cycle**, consumed at the
same single derivation point (`classifyRunStates`) that already computes both the banner dead-list
and the notifier Down-set — still one place.
**The grace window is 180 s**, derived rather than picked round: the deploy flow already allows
**120 s** for a stack to come up healthy, and the slowest catalog healthcheck start_period is Mealie's
**60 s**, after which a couple of check intervals must still elapse. It **expires** — an app that
genuinely fails to come back alarms on the first scan after the window closes. Permanent suppression
would trade a loud false alarm for a silent real one, which is R-88's Scenario D in a new costume.
Tests +9 (8 quiesce + 1 wiring reachability). Red-proofs observed for Scenarios C, E and F.
### v0.176.0 — R-88 Part 1: a failing backup stops re-quiescing (2026-07-27)
**The apps were being stopped and restarted every five minutes for a backup that could not