hub v0.76.0 — R-82 Slice C: tier-aware backup thresholds
R-81 merged every backup signal into one 'newest' against a single 26h limit. backupStaleAfter's own comment recorded why that stops being right under a weekly offsite tier. Each tier is now judged against its own threshold; R-81's structure (three verdicts, anchored absence, distinct reasons) and its boundary test are preserved intact. - offsiteBackupStaleAfter = 8d (7d cadence + headroom); backupStaleAfter keeps 26h and now names the HOST tier only - splitTiers / assessTier / newestBackupEvidenceByTier Slice-A.4 rule implemented: a PBS-targeted vzdump appears in BOTH arrays, so classification is by TARGET TYPE (target_id -> storage_targets[].name -> type), never by array membership — otherwise a PBS backup makes a stale host tier look fresh. storage_targets is used rather than pbs_dr.storage_id because the latter is null on a box with a PBS storage but no DR descriptor. A tier is only judged when the box HAS it, else every box without an offsite tier would alarm once the anchor elapsed — R-81's mistake one level down. With neither tier identifiable (old agent) the pre-Slice-C path runs unchanged. Intended behaviour change: a 30h offsite snapshot no longer alarms. Three fixtures asserted the merged threshold; each still asserts an alarm at the correct limit. No assertion was weakened. RECORDED LIMITATION: the hub infers 'PBS => weekly' from storage type. defaultBackupTarget is felhom-pbs, so a box that never sets local_backup_target would run PBS as its DAILY tier and be judged against 8 days — 7 days of blindness. No box is in that shape today; the real fix is the agent reporting per-tier cadences. Own task. Red-proof observed. Replayed live: demo-felhom OK, demo-hp UNKNOWN (defers correctly), drill-r50 MISSED (true positive). No customer email would be sent.
This commit is contained in:
@@ -45,9 +45,19 @@ type hostReportBackups struct {
|
||||
VerifyState string `json:"verify_state"`
|
||||
} `json:"pbs_snapshots"`
|
||||
Backups []struct {
|
||||
// TargetID is the SLICE-C discriminator: a backups[] entry belongs to whichever tier its
|
||||
// target storage belongs to, never to "the local tier" by virtue of being in this array.
|
||||
TargetID string `json:"target_id"`
|
||||
StartedAt string `json:"started_at"`
|
||||
Success bool `json:"success"`
|
||||
} `json:"backups"`
|
||||
// StorageTargets carries the target TYPE, which is how a backups[] entry is attributed to a
|
||||
// tier (target_id → name → type == "pbs"). See pbsTargetSet.
|
||||
StorageTargets []struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
Content string `json:"content"`
|
||||
} `json:"storage_targets"`
|
||||
}
|
||||
|
||||
// backupVerdict is the three-valued outcome of the freshness policy. The middle value is the
|
||||
@@ -78,6 +88,15 @@ type backupEvidence struct {
|
||||
newestSeen time.Time
|
||||
haveSeen bool
|
||||
|
||||
// Per-tier window evidence (R-82 Slice C). A single "newest across everything" would let a
|
||||
// fresh daily host backup satisfy the OFFSITE tier's lookup — the same error Slice A fixed
|
||||
// agent-side. newestSeen/haveSeen above remain the pre-Slice-C combined values, used only by
|
||||
// the no-tier fallback path.
|
||||
newestHost time.Time
|
||||
haveHost bool
|
||||
newestOffsite time.Time
|
||||
haveOffsite bool
|
||||
|
||||
// firstReportAt is when the hub first saw ANY host-report from this customer — the
|
||||
// observation anchor. Zero when unknown, which the policy treats as "cannot defer"
|
||||
// (fail toward visibility, matching the v0.73.0 zero-anchor branch).
|
||||
@@ -163,6 +182,22 @@ func assessBackupFreshness(reportJSON string, ev backupEvidence, now time.Time)
|
||||
}
|
||||
}
|
||||
|
||||
// ── R-82 Slice C: TIER-AWARE assessment ───────────────────────────────────────────────────
|
||||
// Judge each tier against ITS OWN threshold. Falls through to the pre-Slice-C combined logic
|
||||
// below only when NEITHER tier is identifiable — an old agent whose report carries no
|
||||
// storage_targets and no target_id — so nothing regresses on a fleet mid-upgrade.
|
||||
hostView, offView := splitTiers(hr)
|
||||
if hostView.expected || offView.expected {
|
||||
var as []backupAssessment
|
||||
if hostView.expected {
|
||||
as = append(as, assessTier(tierHost, hostView, ev.newestHost, ev.haveHost, ev.firstReportAt, now))
|
||||
}
|
||||
if offView.expected {
|
||||
as = append(as, assessTier(tierOffsite, offView, ev.newestOffsite, ev.haveOffsite, ev.firstReportAt, now))
|
||||
}
|
||||
return worst(as...)
|
||||
}
|
||||
|
||||
// The newest evidence the LATEST report itself carries.
|
||||
newest := newestPBS
|
||||
haveNewest := havePBS
|
||||
@@ -337,6 +372,16 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
|
||||
logger.Printf("[WARN] Deadline check: failed to read host-report window for %s: %v", id, rerr)
|
||||
} else {
|
||||
ev.newestSeen, ev.haveSeen = newestBackupEvidence(rows, nowUTC)
|
||||
// Slice C: per-tier window evidence. Which tiers to look for comes from the LATEST
|
||||
// report, so a box with no offsite tier never pays for scanning one.
|
||||
var latest hostReportBackups
|
||||
wantHost, wantOffsite := true, true
|
||||
if json.Unmarshal([]byte(reportJSON), &latest) == nil {
|
||||
h, o := splitTiers(latest)
|
||||
wantHost, wantOffsite = h.expected, o.expected
|
||||
}
|
||||
ev.newestHost, ev.haveHost, ev.newestOffsite, ev.haveOffsite =
|
||||
newestBackupEvidenceByTier(rows, wantHost, wantOffsite, nowUTC)
|
||||
}
|
||||
if first, ferr := s.GetFirstHostReportAt(id); ferr != nil {
|
||||
logger.Printf("[WARN] Deadline check: failed to read first host-report for %s: %v", id, ferr)
|
||||
|
||||
Reference in New Issue
Block a user