hub v0.91.0 — the staleness window learns each tier's own rhythm (R-86 Part 2)
gates / gates (push) Successful in 7s
gates / gates (push) Successful in 7s
Ships WITH agent v0.121.0, not after it. The agent now proves a tier once per
ARCHIVE GENERATION, so a weekly tier is proved weekly — in perfect health. The
flat 7-day restoreProvenStaleAfter derived its number from the 24h cadence R-86
removes, and a healthy weekly tier's proof age reaches EXACTLY 168h just before
its next proof: it sat ON the line, so any ordinary delay tipped it into a
nightly alarm about a working system.
restoreProvenWindow(tier, observed, ok):
- the tier's own archive interval, OBSERVED from reports the hub already holds
(pbs_snapshots + successful backups attributed by TARGET TYPE, slice A.4)
- x4 generations = the same tolerance the flat constant expressed
- floored at 7d (never tighter than before), capped at 12d (strictly inside the
2-week offsite retention)
- falls back to the DECLARED rhythm (26h host / 8d offsite — the thresholds the
backup-freshness checker already uses) when history is too short to observe
one; falling back to the FLOOR would recreate the false alarm on a fresh box
Kept: absence is UNKNOWN until the anchored window passes; the signal stays
edge-triggered; failed and stale remain distinct events. Every reason string now
states the window it was judged against (R-100's corollary).
Also backfills the missing v0.90.1 CHANGELOG entry (deployed since f21e7ca), and
records the operator's 2026-08-03 ruling that ep0 is Tier 2 / protected.
This commit is contained in:
@@ -32,15 +32,87 @@ import (
|
||||
// is a NEW monitor written straight after the third, so it copies R-81's verdict structure rather
|
||||
// than re-deriving it. A tier never proven on a newborn box is UNKNOWN, never FAILED.
|
||||
|
||||
// restoreProvenStaleAfter is how long a tier may go unproven before it is called stale.
|
||||
// ── HOW LONG MAY A TIER GO UNPROVEN? (R-86 Part 2) ───────────────────────────────────────────
|
||||
//
|
||||
// Derivation, not a guess: the restore-test cadence is 24h and rotation is oldest-first across two
|
||||
// tiers, so each tier is proven roughly every 2 days. 7 days therefore tolerates ~3 consecutive
|
||||
// missed opportunities before alarming — loud enough to matter, quiet enough not to fire on one
|
||||
// skipped cycle (a deferral behind a long backup is normal, not a fault). It is also comfortably
|
||||
// inside the 2-week offsite retention (operator ruling 2026-07-26), so a tier is never reported
|
||||
// stale against an archive that is about to be pruned anyway.
|
||||
const restoreProvenStaleAfter = 7 * 24 * time.Hour
|
||||
// This was one flat constant, 7 days, and its comment derived that number like this:
|
||||
//
|
||||
// "the restore-test cadence is 24h and rotation is oldest-first across two tiers, so each tier is
|
||||
// proven roughly every 2 days. 7 days therefore tolerates ~3 consecutive missed opportunities."
|
||||
//
|
||||
// **That premise is exactly what R-86 removed.** The agent no longer tests on an interval at all: a
|
||||
// tier is tested once per ARCHIVE GENERATION — when it holds a settled archive that has not been
|
||||
// proven. A tier backed up weekly is therefore proved weekly, by design and in perfect health, and
|
||||
// against a flat 7-day window it would sit on the line and alarm every night about a system that is
|
||||
// working. Shipping the agent's half alone would have converted the improvement into a false alarm,
|
||||
// which is why the two ship together.
|
||||
//
|
||||
// The window is now derived from **the tier's own backup rhythm**, which the hub can observe from
|
||||
// the reports it already receives, and it keeps everything the constant had earned:
|
||||
//
|
||||
// - absence is UNKNOWN until an anchored window has passed (R-81's structure, untouched);
|
||||
// - the signal stays edge-triggered;
|
||||
// - it never exceeds the offsite retention, so a tier is never called stale against an archive
|
||||
// that is about to be pruned;
|
||||
// - and it is never TIGHTER than the 7 days that were already tolerated.
|
||||
const (
|
||||
// restoreProvenGenerations is how many archive generations may pass unproven before alarming.
|
||||
// 4 = the settle lag's own generation plus ~3 missed opportunities — deliberately the same
|
||||
// tolerance the flat constant expressed, so the change is to the RHYTHM, not to the patience.
|
||||
restoreProvenGenerations = 4
|
||||
|
||||
// restoreProvenWindowFloor is the shortest window that may be applied to any tier. It is the
|
||||
// old constant, kept as a FLOOR rather than deleted: a daily tier computes 4 days from its own
|
||||
// rhythm, and tightening a live threshold is not what this task is for. A deferral behind a
|
||||
// long backup is normal, not a fault.
|
||||
restoreProvenWindowFloor = 7 * 24 * time.Hour
|
||||
|
||||
// restoreProvenWindowCap keeps the window strictly inside the 2-week offsite retention
|
||||
// (operator ruling 2026-07-26) with two days to spare. Beyond it the hub would be judging a
|
||||
// tier against an archive PBS has already pruned — an alarm nobody can act on, and the bound
|
||||
// the old constant respected in its own way.
|
||||
restoreProvenWindowCap = 12 * 24 * time.Hour
|
||||
|
||||
// restoreWindowRead is how far back the hub reads host-reports for this check: far enough to
|
||||
// find proof anywhere inside the widest window, and to see at least two archive generations of
|
||||
// a WEEKLY tier so its rhythm is observable at all.
|
||||
restoreWindowRead = 2 * restoreProvenWindowFloor
|
||||
)
|
||||
|
||||
// declaredArchiveInterval is the rhythm the hub ALREADY attributes to a tier — the same thresholds
|
||||
// the backup-freshness checker judges it against (deadline.go / deadline_tiers.go). It is the
|
||||
// fallback when a box's history is too short to observe a rhythm, and it is the right fallback
|
||||
// precisely because it is not a second opinion: if these two checkers disagreed about how often a
|
||||
// tier is expected to receive an archive, one of them would be alarming on the other's model.
|
||||
//
|
||||
// It is stated per RESTORE tier name ("local"/"pbs" — what the agent reports as source_tier), which
|
||||
// is the same split the backup tiers use under different names ("host"/"offsite").
|
||||
func declaredArchiveInterval(tier string) time.Duration {
|
||||
if tier == "pbs" {
|
||||
return offsiteBackupStaleAfter // 8 days: the weekly cadence plus a day of headroom
|
||||
}
|
||||
return backupStaleAfter // 26 hours: the daily cadence plus headroom
|
||||
}
|
||||
|
||||
// restoreProvenWindow is how long THIS tier may go unproven, given its observed archive interval.
|
||||
//
|
||||
// observedOK=false means the box's retained history did not contain two archive generations for
|
||||
// this tier, so the declared rhythm is used. That fallback matters most for exactly the tier this
|
||||
// task is about: a fresh box with a weekly offsite tier has one snapshot and no observable
|
||||
// interval, and falling back to the FLOOR there would recreate the false alarm.
|
||||
func restoreProvenWindow(tier string, observed time.Duration, observedOK bool) time.Duration {
|
||||
interval := declaredArchiveInterval(tier)
|
||||
if observedOK && observed > 0 {
|
||||
interval = observed
|
||||
}
|
||||
w := time.Duration(restoreProvenGenerations) * interval
|
||||
if w < restoreProvenWindowFloor {
|
||||
w = restoreProvenWindowFloor
|
||||
}
|
||||
if w > restoreProvenWindowCap {
|
||||
w = restoreProvenWindowCap
|
||||
}
|
||||
return w
|
||||
}
|
||||
|
||||
// Event types. Operator-tier only — see the dispatcher note in RestoreTestChecker.
|
||||
const (
|
||||
@@ -156,12 +228,13 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
|
||||
return
|
||||
}
|
||||
|
||||
rows, err := c.store.GetHostReportsSince(customerID, now.Add(-2*restoreProvenStaleAfter))
|
||||
rows, err := c.store.GetHostReportsSince(customerID, now.Add(-restoreWindowRead))
|
||||
if err != nil {
|
||||
c.logger.Printf("[WARN] restore-test check: window read failed for %s: %v", customerID, err)
|
||||
return
|
||||
}
|
||||
proven := lastProvenPerTier(rows)
|
||||
intervals := observedArchiveIntervals(rows)
|
||||
|
||||
first, ferr := c.store.GetFirstHostReportAt(customerID)
|
||||
if ferr != nil {
|
||||
@@ -170,7 +243,9 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
|
||||
}
|
||||
|
||||
for _, tier := range tiers {
|
||||
v := assessRestoreProven(tier, proven[tier], first, now)
|
||||
observed, observedOK := intervals[tier]
|
||||
window := restoreProvenWindow(tier, observed, observedOK)
|
||||
v := assessRestoreProven(tier, proven[tier], first, now, window)
|
||||
key := customerID + "|" + tier
|
||||
c.mu.Lock()
|
||||
prev := c.staleStates[key]
|
||||
@@ -192,37 +267,120 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
|
||||
}
|
||||
}
|
||||
|
||||
// assessRestoreProven is the per-tier verdict. PURE (now injected) so the policy is unit-tested —
|
||||
// the property that made R-81 provable, kept deliberately.
|
||||
// assessRestoreProven is the per-tier verdict. PURE (now and the window injected) so the policy is
|
||||
// unit-tested — the property that made R-81 provable, kept deliberately.
|
||||
//
|
||||
// no proof, anchor NOT elapsed → UNKNOWN (newborn box; never an alarm)
|
||||
// no proof, anchor elapsed → MISSED
|
||||
// proof older than the window → MISSED
|
||||
// otherwise → OK
|
||||
func assessRestoreProven(tier string, provenAt, firstReportAt, now time.Time) backupAssessment {
|
||||
//
|
||||
// `window` is now the TIER'S OWN (R-86 Part 2) rather than one constant for every tier, and every
|
||||
// reason string states the window it was judged against. That is R-100's corollary applied here:
|
||||
// when a verdict changes what it counts from, the alarm text has to change with it, or an operator
|
||||
// reads "limit 168h" under a tier that was actually judged at 288h and dismisses a true alarm.
|
||||
func assessRestoreProven(tier string, provenAt, firstReportAt, now time.Time, window time.Duration) backupAssessment {
|
||||
if provenAt.IsZero() {
|
||||
if firstReportAt.IsZero() {
|
||||
return backupAssessment{verdict: verdictMissed,
|
||||
reason: fmt.Sprintf("%s tier: never restore-proven, and no first-contact anchor to defer against", tier)}
|
||||
}
|
||||
watched := now.Sub(firstReportAt)
|
||||
if watched <= restoreProvenStaleAfter {
|
||||
if watched <= window {
|
||||
return backupAssessment{verdict: verdictUnknown,
|
||||
reason: fmt.Sprintf("%s tier: not restore-proven yet, but only watching for %s (grace %s since first contact %s) — newborn, not a fault",
|
||||
tier, watched.Round(time.Hour), restoreProvenStaleAfter, firstReportAt.Format(time.RFC3339))}
|
||||
tier, watched.Round(time.Hour), window, firstReportAt.Format(time.RFC3339))}
|
||||
}
|
||||
return backupAssessment{verdict: verdictMissed,
|
||||
reason: fmt.Sprintf("%s tier: NEVER successfully restore-proven in %s of watching (limit %s) — the tier is unverified, not known-broken",
|
||||
tier, watched.Round(time.Hour), restoreProvenStaleAfter)}
|
||||
reason: fmt.Sprintf("%s tier: NEVER successfully restore-proven in %s of watching (limit %s, this tier's own backup rhythm) — the tier is unverified, not known-broken",
|
||||
tier, watched.Round(time.Hour), window)}
|
||||
}
|
||||
if age := now.Sub(provenAt); age > restoreProvenStaleAfter {
|
||||
if age := now.Sub(provenAt); age > window {
|
||||
return backupAssessment{verdict: verdictMissed,
|
||||
reason: fmt.Sprintf("%s tier: last successful restore-test was %s ago (limit %s) — the tier is unverified, not known-broken",
|
||||
tier, age.Round(time.Hour), restoreProvenStaleAfter)}
|
||||
reason: fmt.Sprintf("%s tier: last successful restore-test was %s ago (limit %s, this tier's own backup rhythm) — the tier is unverified, not known-broken",
|
||||
tier, age.Round(time.Hour), window)}
|
||||
}
|
||||
return backupAssessment{verdict: verdictOK}
|
||||
}
|
||||
|
||||
// observedArchiveIntervals estimates how often each RESTORE tier actually receives an archive, from
|
||||
// the host-reports the hub already holds. Keyed by restore-tier name ("local"/"pbs").
|
||||
//
|
||||
// Evidence is every distinct archive timestamp in the window: `pbs_snapshots[]` for the offsite
|
||||
// tier (PBS enumerates its whole retention in each report, so one report usually settles the
|
||||
// question) and successful `backups[]` records attributed by TARGET TYPE for both tiers — the
|
||||
// slice-A.4 rule, because a PBS-targeted vzdump appears in BOTH arrays and classifying by array
|
||||
// membership would attribute an offsite archive to the host tier.
|
||||
//
|
||||
// The estimate is the MEAN gap: (newest − oldest) / (generations − 1). It needs two generations;
|
||||
// with fewer, ok=false and the caller falls back to the declared rhythm. It is deliberately crude,
|
||||
// and can afford to be: restoreProvenWindow clamps the result between a 7-day floor and a 12-day
|
||||
// cap, so the only discrimination this has to get right is "roughly daily" versus "several days or
|
||||
// slower" — which is exactly the distinction that turns a healthy weekly tier into a false alarm.
|
||||
func observedArchiveIntervals(rows []store.HostReportRow) map[string]time.Duration {
|
||||
seen := map[string]map[int64]struct{}{ // tier → set of archive unix times
|
||||
"local": {},
|
||||
"pbs": {},
|
||||
}
|
||||
add := func(tier string, t time.Time) {
|
||||
if t.IsZero() {
|
||||
return
|
||||
}
|
||||
seen[tier][t.UTC().Unix()] = struct{}{}
|
||||
}
|
||||
|
||||
for _, r := range rows {
|
||||
var hr hostReportBackups
|
||||
if json.Unmarshal([]byte(r.ReportJSON), &hr) != nil {
|
||||
continue // one malformed retained report must not blind the scan
|
||||
}
|
||||
pbs := pbsTargetSet(hr)
|
||||
for _, ps := range hr.PBSSnapshots {
|
||||
if t, ok := parseBackupTime(ps.BackupTime); ok {
|
||||
add("pbs", t)
|
||||
}
|
||||
}
|
||||
for _, b := range hr.Backups {
|
||||
if !b.Success {
|
||||
continue
|
||||
}
|
||||
t, ok := parseBackupTime(b.StartedAt)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if pbs[b.TargetID] {
|
||||
add("pbs", t)
|
||||
} else {
|
||||
add("local", t)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
out := map[string]time.Duration{}
|
||||
for tier, set := range seen {
|
||||
if len(set) < 2 {
|
||||
continue // not observable — the caller uses the declared rhythm
|
||||
}
|
||||
var oldest, newest int64
|
||||
first := true
|
||||
for ts := range set {
|
||||
if first || ts < oldest {
|
||||
oldest = ts
|
||||
}
|
||||
if first || ts > newest {
|
||||
newest = ts
|
||||
}
|
||||
first = false
|
||||
}
|
||||
span := time.Duration(newest-oldest) * time.Second
|
||||
if span <= 0 {
|
||||
continue
|
||||
}
|
||||
out[tier] = span / time.Duration(len(set)-1)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// expectedRestoreTiers names the tiers this box actually HAS, so a box without an offsite tier is
|
||||
// never reported stale for one. Same gate as Slice C's `expected`, and for the same reason: without
|
||||
// it every box lacking a tier would alarm once the anchor elapsed — absence-is-not-failure,
|
||||
|
||||
Reference in New Issue
Block a user