hub v0.77.0 — R-85 Part 2: a restore-test result becomes a SIGNAL
A failed restore-test was a [WARN] line in the ingest handler and nothing else — no event, no notification, no gauge. True for the LOCAL tier that was already being tested, so the loudest DR signal this system produces was inaudible. Rotating tiers without this would only mean two tiers can fail silently instead of one. Two signals, deliberately NOT merged: restore_test_failed (error) — a run completed and did NOT pass restore_test_stale (warning) — a tier not PROVEN within its interval Merging them collapses 'your DR is broken' into 'your DR is unverified', and the second is the one that quietly becomes the first. The staleness wording says 'unverified, not known-broken' and a test asserts that phrasing. Anchored per R-81, not re-derived: a never-proven tier on a newborn box is UNKNOWN, not FAILED, until the window elapses. This family has made the opposite mistake three times; this monitor was written straight after the third, so it copies R-81's structure rather than inventing a fourth shape. restoreProvenStaleAfter = 7d is derived: oldest-first over two tiers at a 24h cadence proves each ~every 2 days, so 7d tolerates ~3 missed opportunities and sits inside the 2-week offsite retention. Per-tier proof comes from the hub's retained host-report window — the agent reports only its latest run, so the latest report alone cannot answer 'when was the OTHER tier last proven?'. Reused R-81's mechanism instead of a wire change. Both types registered in allowedEventTypes (R-77's inert-seam lesson) and operator-tier only — no customerMessages entry. FIXED a time bomb I introduced in Slice C: the restart-blind-window test hard-coded 2026-07-18T18:31:06Z while comparing against the real clock. Harmless under one 26h threshold; once the offsite tier got an 8-day limit it passed all day and began failing at 18:31 UTC, exactly 8 days later. Now relative. Red-proofs B and D observed. Full suite green (17 packages, rc=0).
This commit is contained in:
@@ -374,15 +374,27 @@ func TestCheckBackupDeadlines_RestartBlindWindow_NoEvent(t *testing.T) {
|
||||
}{{rfc(-20 * time.Hour), true}}
|
||||
|
||||
// Pre-restart report: carries the vzdump. Back-dated so it is not the latest.
|
||||
pre := hostReportJSON(t, [][2]string{{"2026-07-18T18:31:06Z", "ok"}}, okVz)
|
||||
//
|
||||
// The offsite snapshot is RELATIVE and FRESH on purpose. It used to be the literal
|
||||
// `2026-07-18T18:31:06Z` from the incident, which was harmless while ONE 26h threshold covered
|
||||
// every tier — but Slice C gave the offsite tier its own 8-day limit, and this test compares
|
||||
// against the REAL clock (runDeadline uses time.Now). So the fixture quietly became a TIME BOMB:
|
||||
// it passed all day on 2026-07-26 and began failing at 18:31 UTC, exactly 8 days after the
|
||||
// hard-coded instant. A test that passes at commit time and fails hours later is worse than one
|
||||
// that fails immediately, because it lands on whoever is next in the file.
|
||||
//
|
||||
// This test is about the HOST tier's restart blind window; the offsite tier must be healthy so
|
||||
// it cannot contribute to the verdict.
|
||||
pre := hostReportJSON(t, [][2]string{{rfc(-2 * time.Hour), "ok"}}, okVz)
|
||||
if err := st.SaveHostReport("h1", "c1", []byte(pre), store.HostReportDenorm{}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.SetHostReportsReceivedAtForTest("c1", sqliteAgo(15*time.Hour)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Post-restart report: PBS only, 176h stale — exactly demo-felhom's 07-26 shape.
|
||||
post := hostReportJSON(t, [][2]string{{"2026-07-18T18:31:06Z", "ok"}}, nil)
|
||||
// Post-restart report: offsite only (the agent's in-memory `backups` was wiped by the restart)
|
||||
// — demo-felhom's 07-26 shape, with the offsite snapshot kept fresh per the note above.
|
||||
post := hostReportJSON(t, [][2]string{{rfc(-2 * time.Hour), "ok"}}, nil)
|
||||
if err := st.SaveHostReport("h1", "c1", []byte(post), store.HostReportDenorm{}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user