v0.194.0 — one operator email per backup run, and nothing dropped without a trace (R-182)
gates / gates (push) Successful in 8s
gates / gates (push) Successful in 8s
MEASURED, not supposed. On 2026-08-03 nine per-app recovery_unit_capture_failed events reached the hub and TWO operator emails went out. The hub's operator cooldown key is customerID:eventType(+tier) and that event carries `app` but no `tier`, so the key held no app identifier: the first refused app took the hour's slot and every other app's failure was discarded BEFORE anything was written down, leaving no row on any channel. The obvious fix — put `app` in the key — was ruled against: on a full disk it produces one email per app, the volume problem wearing the correctness problem's clothes. internal/backup/runsummary.go: a per-run collector with exactly admissionSet's lifetime, fed by all three write legs, emitting backup_run_failures ONCE at the end and only when something failed. A clean run emits nothing. The per-app event stays and becomes the RECORD — the hub routes it record-only, stored and logged every time, never competing for an email slot. The record and the notification are now different things. Deliberate skips (disconnected, decommissioned) are excluded: they have their own alert, and a nightly email about an unplugged drive is one the operator learns to ignore. A manual run always reports: the digest carries a unique run_id the cooldown cannot collapse. Someone pressing the button is actively trying to get a backup. THE PERIODIC SWEEP GETS A DIGEST TOO. With the per-app event now record-only, a capture failure found between runs would be recorded and never notified — a new silence introduced while closing one. That path emits a digest with NO run_id, so the ordinary 1-hour cooldown caps it exactly as before while the mail now lists every failing app instead of whichever was first. A refusal is recorded ONCE, where the verdict is taken, not at the three legs that consult it — R-181's contract is one verdict per app per run. Noting it per leg listed one refused app three times and produced "2 of 1 apps failed". Found by the digest's own test, not in review. Silence is safe because the hub's deadline check raises expected_backup_missed from report freshness, independently of any mail this box sends (monitor/deadline.go:396,417). Confirmed, not assumed. 7 new tests, 4 red-proofs. The main.go seam walk did NOT fail on its first attempt — the AST test walked the backup package and not main.go; the test was fixed and the mutation re-run rather than the pass recorded.
This commit is contained in:
@@ -744,6 +744,25 @@ func main() {
|
||||
"Recovery unit capture FAILED for %q — the app has no fresh local (Tier-1) backup, and Tier-2/Tier-3 have nothing to copy. %s. Error: %v",
|
||||
stackName, usage.String(), err), d)
|
||||
})
|
||||
// R-182: the ONE operator digest per backup run. The per-app events above are the RECORD
|
||||
// (the hub routes them record-only); this is the NOTIFICATION, sent once at the end of a run
|
||||
// and only when something actually failed. A clean run sends nothing — and that silence is
|
||||
// safe because the hub's own daily deadline check raises expected_backup_missed from report
|
||||
// freshness, independently of any mail this box chooses to send.
|
||||
backupMgr.SetRunSummaryNotify(func(rs backup.RunSummary) {
|
||||
d := notify.BackupRunFailuresDetails{
|
||||
RunID: rs.RunID, RunKind: rs.RunKind,
|
||||
Failed: rs.Failed, Attempted: rs.Attempted,
|
||||
}
|
||||
for _, a := range rs.Apps {
|
||||
d.Apps = append(d.Apps, notify.RunFailureDetail{App: a.App, Leg: a.Leg, Reason: a.Reason})
|
||||
}
|
||||
if rs.Usage != nil {
|
||||
d.TargetPath, d.UsedGB, d.AvailGB = rs.Usage.Path, rs.Usage.UsedGB, rs.Usage.AvailGB
|
||||
d.TotalGB, d.UsedPercent, d.SpaceKnown = rs.Usage.TotalGB, rs.Usage.UsedPercent, true
|
||||
}
|
||||
notifier.NotifyBackupRunFailures(rs.Message, d)
|
||||
})
|
||||
// 3a: the pre-push enlargement gate blocked an app's userdata push (config+DB still saved). Edge-
|
||||
// triggered by the engine (only NEW blocks notify), so the hub's per-event-type cooldown suffices —
|
||||
// no controller-side timer (the hub owns cooldown).
|
||||
|
||||
Reference in New Issue
Block a user