v0.194.0 — one operator email per backup run, and nothing dropped without a trace (R-182)
gates / gates (push) Successful in 8s
gates / gates (push) Successful in 8s
MEASURED, not supposed. On 2026-08-03 nine per-app recovery_unit_capture_failed events reached the hub and TWO operator emails went out. The hub's operator cooldown key is customerID:eventType(+tier) and that event carries `app` but no `tier`, so the key held no app identifier: the first refused app took the hour's slot and every other app's failure was discarded BEFORE anything was written down, leaving no row on any channel. The obvious fix — put `app` in the key — was ruled against: on a full disk it produces one email per app, the volume problem wearing the correctness problem's clothes. internal/backup/runsummary.go: a per-run collector with exactly admissionSet's lifetime, fed by all three write legs, emitting backup_run_failures ONCE at the end and only when something failed. A clean run emits nothing. The per-app event stays and becomes the RECORD — the hub routes it record-only, stored and logged every time, never competing for an email slot. The record and the notification are now different things. Deliberate skips (disconnected, decommissioned) are excluded: they have their own alert, and a nightly email about an unplugged drive is one the operator learns to ignore. A manual run always reports: the digest carries a unique run_id the cooldown cannot collapse. Someone pressing the button is actively trying to get a backup. THE PERIODIC SWEEP GETS A DIGEST TOO. With the per-app event now record-only, a capture failure found between runs would be recorded and never notified — a new silence introduced while closing one. That path emits a digest with NO run_id, so the ordinary 1-hour cooldown caps it exactly as before while the mail now lists every failing app instead of whichever was first. A refusal is recorded ONCE, where the verdict is taken, not at the three legs that consult it — R-181's contract is one verdict per app per run. Noting it per leg listed one refused app three times and produced "2 of 1 apps failed". Found by the digest's own test, not in review. Silence is safe because the hub's deadline check raises expected_backup_missed from report freshness, independently of any mail this box sends (monitor/deadline.go:396,417). Confirmed, not assumed. 7 new tests, 4 red-proofs. The main.go seam walk did NOT fail on its first attempt — the AST test walked the backup package and not main.go; the test was fixed and the mutation re-run rather than the pass recorded.
This commit is contained in:
@@ -1,5 +1,55 @@
|
||||
## Changelog
|
||||
|
||||
### v0.194.0 — one operator e-mail per backup run, and nothing dropped without a trace (2026-08-03, R-182) — MinAgent: none
|
||||
|
||||
**The defect, measured rather than supposed.** On 2026-08-03 nine per-app
|
||||
`recovery_unit_capture_failed` events reached the hub and **two operator e-mails went out**. The hub's
|
||||
operator cooldown key is `customerID + ":" + eventType + tier-suffix`, and that event carries `app`
|
||||
but **no `tier`** — so the key held no app identifier. The first refused app took the hour's slot and
|
||||
**every other app's failure was discarded before anything was written down**, leaving no row on any
|
||||
channel. A machine deciding not to tell you and nothing happening at all looked identical.
|
||||
|
||||
**The obvious fix was ruled against, and the reason is worth keeping.** Putting `app` into the key
|
||||
fixes the swallowing by producing **one e-mail per failing app**, which on a full disk is a dozen —
|
||||
the volume problem wearing the correctness problem's clothes.
|
||||
|
||||
**What ships instead: ONE digest per run, and every failure recorded when it happens.**
|
||||
|
||||
- **`internal/backup/runsummary.go`** — a per-run collector with exactly `admissionSet`'s lifetime
|
||||
(created where the run begins, cleared when it ends), fed by all three write legs. It emits
|
||||
`backup_run_failures` once at the end, **only when something failed**. A clean run emits nothing —
|
||||
not an empty digest.
|
||||
- **The per-app event stays and becomes the RECORD.** The hub now routes it *record-only*: stored and
|
||||
written to the notification log every time, never competing for an e-mail slot. The record and the
|
||||
notification are now different things, which is the durable half of this change.
|
||||
- **Deliberate skips are not failures.** A disconnected or decommissioned drive has its own alert and
|
||||
is excluded, because a nightly e-mail about an unplugged drive is one the operator learns to ignore.
|
||||
- **A manual run always reports**, even if the nightly one already wrote that hour: the digest carries
|
||||
a unique `run_id` that the hub's cooldown cannot collapse. Someone pressing the button is actively
|
||||
trying to get a backup.
|
||||
|
||||
**THE PERIODIC SWEEP GETS A DIGEST TOO, and that is not symmetry for its own sake.** `GetFullStatus`
|
||||
captures units outside any run. With the per-app event now record-only, a capture failure found
|
||||
between runs would have been recorded and **never notified** — a new silence introduced while closing
|
||||
one. So that path emits a digest as well, deliberately with **no `run_id`**, so the ordinary 1-hour
|
||||
cooldown caps it exactly as before while the mail now lists *every* failing app instead of whichever
|
||||
one happened to be first.
|
||||
|
||||
**A refusal is recorded ONCE, where the verdict is taken**, not at each of the three legs that consult
|
||||
it — R-181's contract is that one verdict covers all three. Noting it per leg listed a single refused
|
||||
app three times and produced counts like *"2 of 1 apps failed"*. **Found by the digest's own test, not
|
||||
in review.**
|
||||
|
||||
**Why silence is safe, checked rather than assumed.** A digest is only safe if the absence of a mail
|
||||
cannot mean "the run never finished". It cannot: the hub's daily deadline check raises
|
||||
`expected_backup_missed` / `expected_dbdump_missed` (`hub/internal/monitor/deadline.go:396,417`) from
|
||||
the box's **report freshness and stored events**, independently of any mail this box chooses to send.
|
||||
|
||||
**Tests: 7 new, plus 4 red-proofs demonstrated failing then restored.** One of them —
|
||||
the `main.go` seam walk — **did not fail on its first attempt**, because the AST test walked the
|
||||
backup package and not `main.go`; the test was fixed and the mutation re-run rather than the pass
|
||||
being recorded.
|
||||
|
||||
### v0.193.1 — the refusal's size estimate is rendered in bytes, not `0.00 GiB` (2026-08-03, R-181 follow-on)
|
||||
|
||||
**Found by the live proof run for v0.193.0, not by review.** The refusal message printed the estimate
|
||||
|
||||
Reference in New Issue
Block a user