hub v0.75.0: R-81 — "no signal" is not "bad signal" (anchor the backup deadline check)
Third instance of one class (hub v0.12.0, v0.73.0, this), fixed as a class. On 2026-07-26 03:00 UTC expected_backup_missed fired on demo-felhom, demo-hp and drill-r50 at once; the demo-felhom one reached the CUSTOMER channel claiming "newest backup is 176h0m0s old". Nothing was wrong — three vzdump archives were on disk. Cause: the agent backup store is in-memory, so the R-50 fleet restart emptied `backups` until the next run, and the hub read empty as "no backup exists". - assessBackupFreshness returns OK/UNKNOWN/MISSED instead of `missed bool`; absence is UNKNOWN until it outlives an anchored window. Still pure. - store.GetHostReportsSince + monitor.newestBackupEvidence read the hubs own retained history (bounded 7-day lookback, early-exit on fresh evidence) — "when did I last SEE evidence of a backup?" The anchor was free: the hub already retains 90 days. No agent change, no new persisted state. - store.GetFirstHostReportAt anchors absence at first contact, reusing the existing 26h threshold as the grace (no new knob, the v0.73.0 shape). - Deferrals logged + counted; reason strings kept distinct. - backupStaleAfter untouched; landmine recorded (a weekly PBS snapshot would alarm six days in seven) and owned by R-82. Tests 493->508. Red-proofs A/B/C observed and restored; A reproduces the live message verbatim. Replayed the real 03:00 reports (600/417/77 rows): all three now silent. Source: documentation/audits/DIAG-backup-missed-2026-07-26.md
This commit is contained in:
@@ -2825,6 +2825,83 @@ func (s *Store) GetLatestHostReportJSON(customerID string) (string, error) {
|
||||
return j, nil
|
||||
}
|
||||
|
||||
// HostReportRow is one retained host-report: when the hub received it, and its payload.
|
||||
type HostReportRow struct {
|
||||
ReceivedAt time.Time
|
||||
ReportJSON string
|
||||
}
|
||||
|
||||
// GetHostReportsSince returns the customer's retained host-reports received at or after
|
||||
// `since`, NEWEST FIRST. The hub keeps ~retention.max_days of history (90 by default), which
|
||||
// is what makes it possible to ask "when did I last SEE evidence of a backup?" rather than
|
||||
// only "what does the latest report say?".
|
||||
//
|
||||
// R-81: this is the anchor source for the backup-deadline check. The agent's backup record
|
||||
// store is IN-MEMORY (felhom-agent/internal/backup/store.go — "lost on restart; the cadence
|
||||
// re-populates"), so a restart empties `backups` in every report until the next backup runs.
|
||||
// The hub has memory the agent does not; reading across the window is what turns that blind
|
||||
// window from a false alarm into a correctly-silent verdict. Newest-first so the caller can
|
||||
// stop as soon as it has seen enough (see monitor.newestBackupEvidence).
|
||||
func (s *Store) GetHostReportsSince(customerID string, since time.Time) ([]HostReportRow, error) {
|
||||
rows, err := s.db.Query(
|
||||
`SELECT received_at, report_json FROM host_reports
|
||||
WHERE customer_id = ? AND received_at >= ?
|
||||
ORDER BY received_at DESC`,
|
||||
customerID, since.UTC().Format("2006-01-02 15:04:05"),
|
||||
)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var out []HostReportRow
|
||||
for rows.Next() {
|
||||
var at, j string
|
||||
if err := rows.Scan(&at, &j); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, HostReportRow{ReceivedAt: parseSQLiteTime(at).UTC(), ReportJSON: j})
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// GetFirstHostReportAt returns when the hub received its FIRST retained host-report for the
|
||||
// customer, or (zero, nil) when it holds none.
|
||||
//
|
||||
// R-81: this is the observation anchor. "No backup evidence anywhere" is only meaningful
|
||||
// relative to how long the hub has been watching — a box registered an hour ago has no
|
||||
// evidence yet and is NOT failing. Same shape as the v0.73.0 offsite never-ran anchor:
|
||||
// absence becomes a fault only once it has outlived the existing threshold, measured from a
|
||||
// point where evidence first became POSSIBLE.
|
||||
//
|
||||
// Caveat, deliberately accepted: retention prunes at max_days, so for a host older than the
|
||||
// window this returns the prune horizon rather than true first-contact. That only makes the
|
||||
// anchor MORE conservative for long-lived hosts (the window has long since elapsed either
|
||||
// way), and it never shortens a newborn's grace.
|
||||
func (s *Store) GetFirstHostReportAt(customerID string) (time.Time, error) {
|
||||
var at string
|
||||
err := s.db.QueryRow(
|
||||
`SELECT received_at FROM host_reports WHERE customer_id = ? ORDER BY received_at ASC LIMIT 1`,
|
||||
customerID,
|
||||
).Scan(&at)
|
||||
if err == sql.ErrNoRows {
|
||||
return time.Time{}, nil
|
||||
}
|
||||
if err != nil {
|
||||
return time.Time{}, err
|
||||
}
|
||||
return parseSQLiteTime(at).UTC(), nil
|
||||
}
|
||||
|
||||
// SetHostReportsReceivedAtForTest back-dates ALL of a customer's existing host_reports rows
|
||||
// to the given SQLite datetime string, so anchor/window behaviour is testable without
|
||||
// sleeping. TEST-ONLY — mirrors SetOneTimeSecretTimesForTest. Rows saved AFTER the call keep
|
||||
// datetime('now'), which is how a test stages "old reports, then a fresh one".
|
||||
func (s *Store) SetHostReportsReceivedAtForTest(customerID, sqliteTime string) error {
|
||||
_, err := s.db.Exec(`UPDATE host_reports SET received_at = ? WHERE customer_id = ?`, sqliteTime, customerID)
|
||||
return err
|
||||
}
|
||||
|
||||
// UpsertGuestFromReport upserts the REALITY columns of a guest. On conflict it
|
||||
// must NOT clobber the inert columns (api_key / desired_spec_json).
|
||||
func (s *Store) UpsertGuestFromReport(g *Guest) error {
|
||||
|
||||
Reference in New Issue
Block a user