v0.194.0 — one operator email per backup run, and nothing dropped without a trace (R-182)
gates / gates (push) Successful in 8s
gates / gates (push) Successful in 8s
MEASURED, not supposed. On 2026-08-03 nine per-app recovery_unit_capture_failed events reached the hub and TWO operator emails went out. The hub's operator cooldown key is customerID:eventType(+tier) and that event carries `app` but no `tier`, so the key held no app identifier: the first refused app took the hour's slot and every other app's failure was discarded BEFORE anything was written down, leaving no row on any channel. The obvious fix — put `app` in the key — was ruled against: on a full disk it produces one email per app, the volume problem wearing the correctness problem's clothes. internal/backup/runsummary.go: a per-run collector with exactly admissionSet's lifetime, fed by all three write legs, emitting backup_run_failures ONCE at the end and only when something failed. A clean run emits nothing. The per-app event stays and becomes the RECORD — the hub routes it record-only, stored and logged every time, never competing for an email slot. The record and the notification are now different things. Deliberate skips (disconnected, decommissioned) are excluded: they have their own alert, and a nightly email about an unplugged drive is one the operator learns to ignore. A manual run always reports: the digest carries a unique run_id the cooldown cannot collapse. Someone pressing the button is actively trying to get a backup. THE PERIODIC SWEEP GETS A DIGEST TOO. With the per-app event now record-only, a capture failure found between runs would be recorded and never notified — a new silence introduced while closing one. That path emits a digest with NO run_id, so the ordinary 1-hour cooldown caps it exactly as before while the mail now lists every failing app instead of whichever was first. A refusal is recorded ONCE, where the verdict is taken, not at the three legs that consult it — R-181's contract is one verdict per app per run. Noting it per leg listed one refused app three times and produced "2 of 1 apps failed". Found by the digest's own test, not in review. Silence is safe because the hub's deadline check raises expected_backup_missed from report freshness, independently of any mail this box sends (monitor/deadline.go:396,417). Confirmed, not assumed. 7 new tests, 4 red-proofs. The main.go seam walk did NOT fail on its first attempt — the AST test walked the backup package and not main.go; the test was fixed and the mutation re-run rather than the pass recorded.
This commit is contained in:
@@ -330,6 +330,44 @@ func (n *Notifier) NotifyRecoveryUnitCaptureFailed(message string, d RecoveryUni
|
||||
n.PushEvent("recovery_unit_capture_failed", "error", message, d)
|
||||
}
|
||||
|
||||
// RunFailureDetail is one app's failed leg inside a backup run digest.
|
||||
type RunFailureDetail struct {
|
||||
App string `json:"app"`
|
||||
Leg string `json:"leg"`
|
||||
Reason string `json:"reason"`
|
||||
}
|
||||
|
||||
// BackupRunFailuresDetails is the per-RUN digest payload (R-182). App NAMES, leg names, reasons and
|
||||
// byte figures only — never an env value (§9.4).
|
||||
type BackupRunFailuresDetails struct {
|
||||
// RunID makes the hub's 1-hour operator cooldown unable to collapse two real runs into one
|
||||
// e-mail. EMPTY on the periodic refresh sweep, deliberately: that path can fire on every status
|
||||
// poll, so it must fall under the ordinary cooldown instead.
|
||||
RunID string `json:"run_id,omitempty"`
|
||||
RunKind string `json:"run_kind"`
|
||||
Failed int `json:"failed"`
|
||||
Attempted int `json:"attempted"`
|
||||
TargetPath string `json:"target_path,omitempty"`
|
||||
UsedGB float64 `json:"used_gb,omitempty"`
|
||||
AvailGB float64 `json:"avail_gb,omitempty"`
|
||||
TotalGB float64 `json:"total_gb,omitempty"`
|
||||
UsedPercent float64 `json:"used_percent,omitempty"`
|
||||
SpaceKnown bool `json:"space_known"`
|
||||
Apps []RunFailureDetail `json:"apps"`
|
||||
}
|
||||
|
||||
// NotifyBackupRunFailures sends the ONE operator digest for a backup run in which something failed
|
||||
// (R-182). It is the NOTIFICATION; the per-app `recovery_unit_capture_failed` events are the RECORD,
|
||||
// and the hub routes those record-only so they never compete for an e-mail slot.
|
||||
//
|
||||
// OPERATOR-TIER, and for the same reason as its per-app sibling: a customer can act on a full disk
|
||||
// (that is the fill warning, which fires first and IS customer-facing) but not on a list of which
|
||||
// apps' backups failed and why. `notify.operatorOnlyEvents` in the hub enforces that — NOT the
|
||||
// absence of a customerMessages entry, which is a fallback rather than a block (v0.78.0).
|
||||
func (n *Notifier) NotifyBackupRunFailures(message string, d BackupRunFailuresDetails) {
|
||||
n.PushEvent("backup_run_failures", "error", message, d)
|
||||
}
|
||||
|
||||
// NotifyOffboxEnlargeBlocked sends a WARNING (not a failure) when an app's enlarged offsite push was
|
||||
// refused by the pre-push quota gate — its config+DB were still saved. Customer-facing (Hungarian
|
||||
// body). NOTE: the event type "offbox_enlarge_blocked" must be added to the hub's allowedEventTypes +
|
||||
|
||||
Reference in New Issue
Block a user