R-97: a failing backup is heard, and stops blaming the apps (v0.177.0)

R-97a: internal/quiesce had no route to the hub at all — three failed whole-guest
backups on 2026-07-27 produced zero events. TierNotifier is a seam (not an import),
wired by an init-only setter because main.go builds the notifier after the loop.
Edge-triggered: the failure fires when the R-88 breaker ARMS, not per retry, and
recovery rides recordSuccess's existing bool. Uses NEW operator-only event types;
reusing backup_failed would have emailed the customer in Hungarian about a backup
they cannot act on, since it has a customerMessages entry and is in live
enabled_events. Requires hub >= v0.78.0.

R-97b: v0.164.0's state filter cannot see an app caught MID-RESTART, which is how
BookStack alarmed. The fix is a suppression window keyed to the quiesce CYCLE,
consumed at the same single derivation point. 180s grace, derived from the deploy
flow's 120s health timeout and Mealie's 60s start_period; it expires, so an app
that genuinely fails to come back still alarms.
This commit is contained in:
2026-07-27 17:01:41 +02:00
parent ccefff4f39
commit e9c99566b0
10 changed files with 559 additions and 11 deletions
+34
View File
@@ -94,6 +94,17 @@ type BackupDetails struct {
Error string `json:"error,omitempty"`
}
// WholeGuestBackupDetails carries the TIER for a whole-guest (vzdump) backup event (R-97a).
//
// The `tier` field is load-bearing beyond display: the hub's operator cooldown is keyed
// `customerID:eventType` plus this tier when present, so `local` failing does not get swallowed by
// `felhom-pbs` having failed within the same hour. Rename it and the two tiers silently share one
// cooldown again.
type WholeGuestBackupDetails struct {
Tier string `json:"tier"`
Error string `json:"error,omitempty"`
}
// DBDumpDetails holds structured data for DB dump events.
type DBDumpDetails struct {
DatabaseCount int `json:"database_count,omitempty"`
@@ -771,3 +782,26 @@ func statusRank(status string) int {
return 0
}
}
// NotifyWholeGuestBackupFailed / ...Recovered — R-97a, the WHOLE-GUEST (vzdump) backup tier.
//
// OPERATOR-TIER ONLY, and that is why these are NOT `backup_failed`. `backup_failed` and
// `backup_completed` both carry `customerMessages` entries in the hub AND sit in demo-felhom's live
// `enabled_events`, so reusing them would email the CUSTOMER, in Hungarian, that their backup failed
// — while it is still retrying behind the R-88 breaker. A customer can take no action on a failed
// whole-guest backup; that is the same harm R-97b removes, re-introduced through the front door.
//
// These follow the R-85 precedent exactly: a type in the hub's `allowedEventTypes` with NO
// `customerMessages` entry, so the dispatcher structurally cannot route it to a customer.
//
// HUB DEPENDENCY: both types MUST be present in the hub's allowedEventTypes or POST /event 400s
// (the recorded allowlist gotcha). Do not deploy this controller ahead of that hub change.
func (n *Notifier) NotifyWholeGuestBackupFailed(tier, message, errMsg string) {
n.PushEvent("whole_guest_backup_failed", "error", message,
WholeGuestBackupDetails{Tier: tier, Error: errMsg})
}
func (n *Notifier) NotifyWholeGuestBackupRecovered(tier, message string) {
n.PushEvent("whole_guest_backup_recovered", "info", message,
WholeGuestBackupDetails{Tier: tier})
}