hub v0.75.0: R-81 — "no signal" is not "bad signal" (anchor the backup deadline check)
Third instance of one class (hub v0.12.0, v0.73.0, this), fixed as a class. On 2026-07-26 03:00 UTC expected_backup_missed fired on demo-felhom, demo-hp and drill-r50 at once; the demo-felhom one reached the CUSTOMER channel claiming "newest backup is 176h0m0s old". Nothing was wrong — three vzdump archives were on disk. Cause: the agent backup store is in-memory, so the R-50 fleet restart emptied `backups` until the next run, and the hub read empty as "no backup exists". - assessBackupFreshness returns OK/UNKNOWN/MISSED instead of `missed bool`; absence is UNKNOWN until it outlives an anchored window. Still pure. - store.GetHostReportsSince + monitor.newestBackupEvidence read the hubs own retained history (bounded 7-day lookback, early-exit on fresh evidence) — "when did I last SEE evidence of a backup?" The anchor was free: the hub already retains 90 days. No agent change, no new persisted state. - store.GetFirstHostReportAt anchors absence at first contact, reusing the existing 26h threshold as the grace (no new knob, the v0.73.0 shape). - Deferrals logged + counted; reason strings kept distinct. - backupStaleAfter untouched; landmine recorded (a weekly PBS snapshot would alarm six days in seven) and owned by R-82. Tests 493->508. Red-proofs A/B/C observed and restored; A reproduces the live message verbatim. Replayed the real 03:00 reports (600/417/77 rows): all three now silent. Source: documentation/audits/DIAG-backup-missed-2026-07-26.md
This commit is contained in:
Claude Code
@@ -14,8 +14,27 @@ import (
|
||||
// deadline check raises expected_backup_missed. 26h covers an evening backup schedule
|
||||
// (e.g. ~18:00–22:00) plus headroom, so a healthy once-daily cadence never trips the
|
||||
// early-morning check.
|
||||
//
|
||||
// R-81 also reuses it as the ABSENCE window (see assessBackupFreshness): the existing
|
||||
// threshold, anchored at first-contact, IS the newborn grace — no new knob, exactly as
|
||||
// v0.73.0 reused offsite staleAfter for its never-ran anchor.
|
||||
//
|
||||
// ⚠️ LANDMINE — dependency on R-82 (the backup target split). This constant is applied to
|
||||
// whichever tier is NEWEST, PBS or vzdump, with no tier-awareness. Today a daily vzdump
|
||||
// always wins, so PBS's own age is invisible here and 26h is harmless. The moment PBS moves
|
||||
// to a WEEKLY cadence, a perfectly healthy weekly snapshot is >26h old six days in seven and
|
||||
// this constant alarms on it. Fixing that means per-tier thresholds, which cannot be built
|
||||
// before the per-tier cadence config exists (`local_backup_target` is a single target and
|
||||
// BackupCadence() a single 24h window today). Do NOT pre-build it — R-82 owns both halves.
|
||||
const backupStaleAfter = 26 * time.Hour
|
||||
|
||||
// backupEvidenceLookback bounds how far back the hub looks for evidence that a backup ever
|
||||
// happened, when the LATEST report carries none. Generous against any plausible daily cadence
|
||||
// (and against an agent that stayed restarted for days), bounded so the cold path can't turn
|
||||
// into a full-retention scan of every report the hub holds. Beyond this the verdict is
|
||||
// "no evidence in the lookback", which is a fault in its own right once the anchor elapsed.
|
||||
const backupEvidenceLookback = 7 * 24 * time.Hour
|
||||
|
||||
// hostReportBackups is the minimal slice of an agent host-report the deadline check
|
||||
// reads to judge backup freshness (pbs_snapshots is the offsite-DR signal; backups is
|
||||
// the local vzdump fallback). Mirrors the agent's hub.PBSSnapshot / hub.Backup wire
|
||||
@@ -31,28 +50,86 @@ type hostReportBackups struct {
|
||||
} `json:"backups"`
|
||||
}
|
||||
|
||||
// backupVerdict is the three-valued outcome of the freshness policy. The middle value is the
|
||||
// whole point of R-81: "I have no evidence" is NOT "the backup failed".
|
||||
type backupVerdict int
|
||||
|
||||
const (
|
||||
verdictOK backupVerdict = iota // positive evidence of a recent backup
|
||||
verdictUnknown // no evidence yet, and the anchored window has not elapsed
|
||||
verdictMissed // positive evidence of a problem — alarm
|
||||
)
|
||||
|
||||
// backupAssessment is the verdict for one customer's offsite backup health.
|
||||
type backupAssessment struct {
|
||||
missed bool // raise expected_backup_missed
|
||||
reason string // human-readable cause (event message + logs)
|
||||
verdict backupVerdict
|
||||
reason string // human-readable cause (event message + logs)
|
||||
}
|
||||
|
||||
// assessBackupFreshness decides whether a customer's latest host-report shows a healthy,
|
||||
// recent backup. Pure (now is injected) so the policy is unit-tested. Only POSITIVE
|
||||
// evidence of a problem fires an alarm:
|
||||
// - no PBS snapshot AND no successful vzdump in the report → missed ("no backup recorded")
|
||||
// - newest backup older than backupStaleAfter → missed ("stale")
|
||||
// - the newest PBS snapshot's verify_state is "failed" → missed ("verify failed")
|
||||
// missed reports whether this assessment should raise expected_backup_missed.
|
||||
func (a backupAssessment) missed() bool { return a.verdict == verdictMissed }
|
||||
|
||||
// backupEvidence is the hub-history half of the freshness policy, resolved by the caller so
|
||||
// assessBackupFreshness stays PURE (see its doc comment on why that matters).
|
||||
type backupEvidence struct {
|
||||
// newestSeen is the newest backup evidence found across the retained host-report window
|
||||
// (PBS backup_time or successful vzdump started_at), regardless of whether the LATEST
|
||||
// report still carries it. haveSeen is false when the window held none.
|
||||
newestSeen time.Time
|
||||
haveSeen bool
|
||||
|
||||
// firstReportAt is when the hub first saw ANY host-report from this customer — the
|
||||
// observation anchor. Zero when unknown, which the policy treats as "cannot defer"
|
||||
// (fail toward visibility, matching the v0.73.0 zero-anchor branch).
|
||||
firstReportAt time.Time
|
||||
}
|
||||
|
||||
// assessBackupFreshness decides whether a customer's backups are healthy. Pure (now and the
|
||||
// hub-history evidence are injected) so the policy is unit-tested — that purity is why the
|
||||
// 2026-07-26 incident could be diagnosed at all, and why the fix below is provable.
|
||||
//
|
||||
// ── THE INVARIANT (R-81) ──────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// Only POSITIVE EVIDENCE OF A PROBLEM raises an alarm. ABSENCE OF A SIGNAL IS UNKNOWN,
|
||||
// and becomes a fault only once that absence has persisted beyond an ANCHORED window.
|
||||
//
|
||||
// This monitor family has made the same mistake three times, and it is written down here so
|
||||
// the fourth is harder:
|
||||
// - hub v0.12.0 — `expected_backup_missed` fired daily for every healthy customer, because
|
||||
// it looked for a `backup_completed` event that no component emits anymore.
|
||||
// - hub v0.73.0 — `offsite_stale` fired minutes after a HEALTHY repair, because the
|
||||
// never-ran branch had no time anchor. Fixed by anchoring, not by silence.
|
||||
// - R-81 (this) — `expected_backup_missed` fired on three boxes at once on 2026-07-26,
|
||||
// because an agent restart empties the host-report `backups` array (the agent's store is
|
||||
// in-memory) and empty was read as "no backup exists". The vzdump had in fact run.
|
||||
//
|
||||
// Note the shape of the fix in all three: NOT silence. Silence is the opposite failure — a
|
||||
// box that genuinely never backs up would then alarm never, which is strictly worse than
|
||||
// crying wolf. Absence is deferred, then alarmed on, with its own distinct reason string.
|
||||
//
|
||||
// ── THE BRANCHES ──────────────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// unparseable latest report → missed ("could not be parsed")
|
||||
// newest evidence (report OR window) older than staleAfter → missed ("newest backup is Xh old")
|
||||
// newest PBS snapshot's verify_state == "failed" → missed ("failed verification")
|
||||
// NO evidence anywhere, anchor NOT yet elapsed → UNKNOWN, silent + logged
|
||||
// NO evidence anywhere, anchor elapsed → missed ("no backup evidence in …")
|
||||
// otherwise → ok
|
||||
//
|
||||
// Each failure mode keeps its OWN reason string. That is not polish: the entire 2026-07-26
|
||||
// diagnosis turned on reading the exact string, and collapsing them would have made it
|
||||
// impossible. In particular "absence over time" and "a timestamp that is too old" are
|
||||
// different faults with different causes, and must never share a message.
|
||||
//
|
||||
// A fresh-but-not-yet-verified snapshot (verify_state "none"/"") is NOT treated as a
|
||||
// failure: PBS verification runs on its own cadence, so a snapshot taken hours before the
|
||||
// 03:00 check may legitimately be unverified. Alarming on that would re-introduce exactly
|
||||
// the daily false alarm this repoint removes (hence "failed" only, not "≠ ok").
|
||||
func assessBackupFreshness(reportJSON string, now time.Time) backupAssessment {
|
||||
// the daily false alarm the v0.12.0 repoint removed (hence "failed" only, not "≠ ok").
|
||||
func assessBackupFreshness(reportJSON string, ev backupEvidence, now time.Time) backupAssessment {
|
||||
var hr hostReportBackups
|
||||
if err := json.Unmarshal([]byte(reportJSON), &hr); err != nil {
|
||||
// Unparseable report → can't confirm a backup. Surface it rather than swallow it.
|
||||
return backupAssessment{missed: true, reason: "latest host-report could not be parsed"}
|
||||
return backupAssessment{verdict: verdictMissed, reason: "latest host-report could not be parsed"}
|
||||
}
|
||||
|
||||
var newestPBS time.Time
|
||||
@@ -86,21 +163,92 @@ func assessBackupFreshness(reportJSON string, now time.Time) backupAssessment {
|
||||
}
|
||||
}
|
||||
|
||||
if !havePBS && !haveVzdump {
|
||||
return backupAssessment{missed: true, reason: "no PBS snapshot or successful backup in the latest host-report"}
|
||||
// The newest evidence the LATEST report itself carries.
|
||||
newest := newestPBS
|
||||
haveNewest := havePBS
|
||||
if haveVzdump && (!haveNewest || newestVzdump.After(newest)) {
|
||||
newest, haveNewest = newestVzdump, true
|
||||
}
|
||||
|
||||
newest := newestPBS
|
||||
if haveVzdump && (!havePBS || newestVzdump.After(newest)) {
|
||||
newest = newestVzdump
|
||||
// R-81: fold in what the hub REMEMBERS. The agent's store is point-in-time and forgets
|
||||
// across a restart; the hub's retained host-reports do not. An empty array in the latest
|
||||
// report therefore says nothing on its own — the question is when evidence was last SEEN,
|
||||
// not whether this one report happens to carry it.
|
||||
if ev.haveSeen && (!haveNewest || ev.newestSeen.After(newest)) {
|
||||
newest, haveNewest = ev.newestSeen, true
|
||||
}
|
||||
|
||||
if !haveNewest {
|
||||
// ABSENCE. Not a failure by itself — see the invariant above. It becomes one only
|
||||
// once it has outlived the existing threshold, counted from first contact (the point
|
||||
// at which a backup first became possible to observe).
|
||||
if ev.firstReportAt.IsZero() {
|
||||
// No anchor to defer against — fail toward visibility, as v0.73.0 does for the
|
||||
// legacy zero-anchor shape. Distinct string: this is an unanchored absence.
|
||||
return backupAssessment{
|
||||
verdict: verdictMissed,
|
||||
reason: "no backup evidence in any retained host-report, and no first-contact anchor to defer against",
|
||||
}
|
||||
}
|
||||
watched := now.Sub(ev.firstReportAt)
|
||||
if watched <= backupStaleAfter {
|
||||
return backupAssessment{
|
||||
verdict: verdictUnknown,
|
||||
reason: fmt.Sprintf("no backup evidence yet, but only watching for %s (grace %s since first contact %s) — newborn host, not a fault",
|
||||
watched.Round(time.Hour), backupStaleAfter, ev.firstReportAt.Format(time.RFC3339)),
|
||||
}
|
||||
}
|
||||
return backupAssessment{
|
||||
verdict: verdictMissed,
|
||||
reason: fmt.Sprintf("no backup evidence in any host-report for %s (limit %s, first contact %s, lookback %s)",
|
||||
watched.Round(time.Hour), backupStaleAfter, ev.firstReportAt.Format(time.RFC3339), backupEvidenceLookback),
|
||||
}
|
||||
}
|
||||
|
||||
if age := now.Sub(newest); age > backupStaleAfter {
|
||||
return backupAssessment{missed: true, reason: fmt.Sprintf("newest backup is %s old (limit %s)", age.Round(time.Hour), backupStaleAfter)}
|
||||
return backupAssessment{verdict: verdictMissed, reason: fmt.Sprintf("newest backup is %s old (limit %s)", age.Round(time.Hour), backupStaleAfter)}
|
||||
}
|
||||
if havePBS && newestPBSVerify == "failed" {
|
||||
return backupAssessment{missed: true, reason: "newest PBS snapshot failed verification"}
|
||||
return backupAssessment{verdict: verdictMissed, reason: "newest PBS snapshot failed verification"}
|
||||
}
|
||||
return backupAssessment{missed: false}
|
||||
return backupAssessment{verdict: verdictOK}
|
||||
}
|
||||
|
||||
// newestBackupEvidence scans the customer's retained host-reports (newest first) for the most
|
||||
// recent backup evidence — a PBS snapshot backup_time or a SUCCESSFUL vzdump started_at —
|
||||
// regardless of whether the latest report still carries it.
|
||||
//
|
||||
// Cost discipline: it stops as soon as it has found evidence FRESH enough that nothing older
|
||||
// could change the verdict, so the healthy path reads one row. Only the genuinely-broken path
|
||||
// (no fresh evidence anywhere) walks the full lookback, and that is bounded by
|
||||
// backupEvidenceLookback rather than by retention.
|
||||
func newestBackupEvidence(rows []store.HostReportRow, now time.Time) (time.Time, bool) {
|
||||
var newest time.Time
|
||||
have := false
|
||||
for _, r := range rows {
|
||||
var hr hostReportBackups
|
||||
if err := json.Unmarshal([]byte(r.ReportJSON), &hr); err != nil {
|
||||
continue // a single malformed retained report must not blind the scan
|
||||
}
|
||||
for _, ps := range hr.PBSSnapshots {
|
||||
if t, ok := parseBackupTime(ps.BackupTime); ok && (!have || t.After(newest)) {
|
||||
newest, have = t, true
|
||||
}
|
||||
}
|
||||
for _, b := range hr.Backups {
|
||||
if !b.Success {
|
||||
continue
|
||||
}
|
||||
if t, ok := parseBackupTime(b.StartedAt); ok && (!have || t.After(newest)) {
|
||||
newest, have = t, true
|
||||
}
|
||||
}
|
||||
// Fresh evidence found → the age branch cannot fire and nothing older matters.
|
||||
if have && now.Sub(newest) <= backupStaleAfter {
|
||||
break
|
||||
}
|
||||
}
|
||||
return newest, have
|
||||
}
|
||||
|
||||
// parseBackupTime parses an RFC3339 timestamp from a host-report and normalizes to UTC.
|
||||
@@ -153,7 +301,7 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
|
||||
midnightBudapest := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, budapest)
|
||||
sinceUTC := midnightBudapest.UTC()
|
||||
|
||||
var backupMissed, dbdumpMissed, skipped int
|
||||
var backupMissed, dbdumpMissed, skipped, deferred int
|
||||
|
||||
for _, id := range customerIDs {
|
||||
// Skip nodes that are down — they already have staleness events
|
||||
@@ -179,7 +327,26 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
|
||||
// has no PBS data to judge here and must not emit a daily backup alarm of its
|
||||
// own. (The DB-dump half below still applies.)
|
||||
default:
|
||||
if a := assessBackupFreshness(reportJSON, time.Now().UTC()); a.missed {
|
||||
nowUTC := time.Now().UTC()
|
||||
|
||||
// R-81: resolve the hub-history half BEFORE judging. A read failure here must not
|
||||
// invent a fault — it degrades to "the latest report is all I know", which is the
|
||||
// pre-R-81 behaviour, and is logged rather than swallowed.
|
||||
var ev backupEvidence
|
||||
if rows, rerr := s.GetHostReportsSince(id, nowUTC.Add(-backupEvidenceLookback)); rerr != nil {
|
||||
logger.Printf("[WARN] Deadline check: failed to read host-report window for %s: %v", id, rerr)
|
||||
} else {
|
||||
ev.newestSeen, ev.haveSeen = newestBackupEvidence(rows, nowUTC)
|
||||
}
|
||||
if first, ferr := s.GetFirstHostReportAt(id); ferr != nil {
|
||||
logger.Printf("[WARN] Deadline check: failed to read first host-report for %s: %v", id, ferr)
|
||||
} else {
|
||||
ev.firstReportAt = first
|
||||
}
|
||||
|
||||
a := assessBackupFreshness(reportJSON, ev, nowUTC)
|
||||
switch a.verdict {
|
||||
case verdictMissed:
|
||||
msg := "No fresh verified backup: " + a.reason
|
||||
if _, err := s.SaveEvent(id, "expected_backup_missed", "error", msg, "{}", "hub"); err != nil {
|
||||
logger.Printf("[WARN] Failed to save expected_backup_missed for %s: %v", id, err)
|
||||
@@ -187,6 +354,13 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
|
||||
onEvent(id, "expected_backup_missed", "error", msg, "{}", "hub")
|
||||
}
|
||||
backupMissed++
|
||||
case verdictUnknown:
|
||||
// Make the deferral VISIBLE (the v0.73.0 Part-7 precedent): a quiet check must
|
||||
// never be indistinguishable from a check that did not run. This check fires
|
||||
// once daily, so this is at most one line per customer per day — not spam, and
|
||||
// it is the line that proves the deferral happened rather than an error.
|
||||
logger.Printf("[INFO] Deadline check: %s backup verdict UNKNOWN (no alarm) — %s", id, a.reason)
|
||||
deferred++
|
||||
}
|
||||
}
|
||||
|
||||
@@ -204,6 +378,6 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
|
||||
}
|
||||
}
|
||||
|
||||
logger.Printf("[INFO] Deadline check: %d customers, %d backup missed, %d dbdump missed, %d skipped (down)",
|
||||
len(customerIDs), backupMissed, dbdumpMissed, skipped)
|
||||
logger.Printf("[INFO] Deadline check: %d customers, %d backup missed, %d backup unknown (deferred), %d dbdump missed, %d skipped (down)",
|
||||
len(customerIDs), backupMissed, deferred, dbdumpMissed, skipped)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user