R-81 merged every backup signal into one 'newest' against a single 26h limit.
backupStaleAfter's own comment recorded why that stops being right under a
weekly offsite tier. Each tier is now judged against its own threshold;
R-81's structure (three verdicts, anchored absence, distinct reasons) and its
boundary test are preserved intact.
- offsiteBackupStaleAfter = 8d (7d cadence + headroom); backupStaleAfter keeps
26h and now names the HOST tier only
- splitTiers / assessTier / newestBackupEvidenceByTier
Slice-A.4 rule implemented: a PBS-targeted vzdump appears in BOTH arrays, so
classification is by TARGET TYPE (target_id -> storage_targets[].name -> type),
never by array membership — otherwise a PBS backup makes a stale host tier look
fresh. storage_targets is used rather than pbs_dr.storage_id because the latter
is null on a box with a PBS storage but no DR descriptor.
A tier is only judged when the box HAS it, else every box without an offsite
tier would alarm once the anchor elapsed — R-81's mistake one level down. With
neither tier identifiable (old agent) the pre-Slice-C path runs unchanged.
Intended behaviour change: a 30h offsite snapshot no longer alarms. Three
fixtures asserted the merged threshold; each still asserts an alarm at the
correct limit. No assertion was weakened.
RECORDED LIMITATION: the hub infers 'PBS => weekly' from storage type.
defaultBackupTarget is felhom-pbs, so a box that never sets local_backup_target
would run PBS as its DAILY tier and be judged against 8 days — 7 days of
blindness. No box is in that shape today; the real fix is the agent reporting
per-tier cadences. Own task.
Red-proof observed. Replayed live: demo-felhom OK, demo-hp UNKNOWN (defers
correctly), drill-r50 MISSED (true positive). No customer email would be sent.
Third instance of one class (hub v0.12.0, v0.73.0, this), fixed as a class.
On 2026-07-26 03:00 UTC expected_backup_missed fired on demo-felhom, demo-hp
and drill-r50 at once; the demo-felhom one reached the CUSTOMER channel
claiming "newest backup is 176h0m0s old". Nothing was wrong — three vzdump
archives were on disk. Cause: the agent backup store is in-memory, so the
R-50 fleet restart emptied `backups` until the next run, and the hub read
empty as "no backup exists".
- assessBackupFreshness returns OK/UNKNOWN/MISSED instead of `missed bool`;
absence is UNKNOWN until it outlives an anchored window. Still pure.
- store.GetHostReportsSince + monitor.newestBackupEvidence read the hubs own
retained history (bounded 7-day lookback, early-exit on fresh evidence) —
"when did I last SEE evidence of a backup?" The anchor was free: the hub
already retains 90 days. No agent change, no new persisted state.
- store.GetFirstHostReportAt anchors absence at first contact, reusing the
existing 26h threshold as the grace (no new knob, the v0.73.0 shape).
- Deferrals logged + counted; reason strings kept distinct.
- backupStaleAfter untouched; landmine recorded (a weekly PBS snapshot would
alarm six days in seven) and owned by R-82.
Tests 493->508. Red-proofs A/B/C observed and restored; A reproduces the live
message verbatim. Replayed the real 03:00 reports (600/417/77 rows): all
three now silent.
Source: documentation/audits/DIAG-backup-missed-2026-07-26.md
Phase-1 of SPIKE-infra-backup-2026-06-15. The infra-backup mechanism was dead
since slice 8C yet stored plaintext customer secrets at rest (app-secret key,
restic password, Cloudflare tokens) — a zero-knowledge violation — and its
absence made the daily expected_backup_missed email fire for healthy customers.
- Repoint monitor.CheckBackupDeadlines backup half to the agent host-report's
PBS snapshots (+vzdump): alarm only on no-backup / >26h stale / verify failed.
Keep the db_dump half. No host-report → no backup alarm (liveness owns that).
New store.GetLatestHostReportJSON. Tests incl. a companion that fails pre-fix.
- Remove the infra-backup endpoints, store methods/types, and operator panel;
/recovery now returns config_yaml only.
- migrate(): DROP infra_backup_versions/infra_backups + VACUUM (+wal_checkpoint)
to physically reclaim the plaintext pages, gated on table existence.
Flagged out-of-scope: exposed creds need operator rotation; legacy reports table
holds historical plaintext restic_password rows (separate leak, not purged here).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude Code
admin