R-86: restore-test follows the backup, not the clock (v0.121.0)
gates / gates (push) Failing after 7s

The ticker survives as the EVALUATION interval only. A tier is DUE when its
newest archive that has settled for `settle` (default 24h) has not been proven:
daily tier -> proved daily on yesterday's archive, weekly tier -> weekly on its
own, newborn -> UNKNOWN.

The trap avoided: the literal reading ("newest archive is >= 24h old") is NEVER
true on a daily tier, so it silently switches restore-testing off where it
matters most. Red-proved at 0 runs over 5 simulated days.

- state records WHICH archive was proven; legacy files keep their time and yield
  no proven archive (each tier due once after the upgrade, deliberately)
- two knobs replace one: restore_test_eval_interval_seconds (6h, measured) and
  restore_test_settle_seconds (24h). The old cadence key keeps its DISABLE
  meaning verbatim and now seeds the settle lag, with a start-up WARN.
- due-check runs BEFORE the heavy-op gate (a frequent poll must not make a
  starting backup record a failure, F-A1)
- candidate picker skips implausible archives (a phantom would be due forever)
- new read-only --selftest=restore-test-due prints the verdict + its cost
This commit is contained in:
2026-08-03 14:54:57 +02:00
parent 1b14cfd0b4
commit 4618169036
13 changed files with 1273 additions and 123 deletions
+24 -3
View File
@@ -135,10 +135,11 @@ func TestBackup_VzdumpFailureReturnsFailedRecord(t *testing.T) {
}
func TestPickRestoreCandidate_NewestOrEmpty(t *testing.T) {
const big = 4 << 30 // a plausible whole-guest archive
api := &fakeBackupAPI{content: []proxmox.StorageContent{
{VolID: "a", Content: "backup", CTime: 10},
{VolID: "b", Content: "backup", CTime: 99},
{VolID: "iso", Content: "iso", CTime: 999}, // not a backup → ignored
{VolID: "a", Content: "backup", CTime: 10, Size: big},
{VolID: "b", Content: "backup", CTime: 99, Size: big},
{VolID: "iso", Content: "iso", CTime: 999, Size: big}, // not a backup → ignored
}}
r := NewBackupRunner(api, "local", "", "", "", quiet())
vol, err := r.PickRestoreCandidate(context.Background())
@@ -152,6 +153,26 @@ func TestPickRestoreCandidate_NewestOrEmpty(t *testing.T) {
}
}
// R-86: the NEWEST entry is not a candidate if it cannot be a complete archive. An incomplete
// artefact (F-CRIT-2's 1-byte phantom, which server-side prune does not collect) would otherwise be
// picked forever, fail its restore forever, never earn proof, and so leave the tier due at every
// evaluation — turning the evaluation interval into the retry rate for a multi-GB restore.
//
// COMPANION RED-PROOF (observed): drop the `archivePlausiblyComplete` guard from
// PickSettledRestoreCandidateOn and this fails with
// `pick = "phantom" want the newest COMPLETE archive 'real'`.
func TestPickRestoreCandidate_SkipsImplausibleArchives(t *testing.T) {
api := &fakeBackupAPI{content: []proxmox.StorageContent{
{VolID: "real", Content: "backup", CTime: 10, Size: 4 << 30},
{VolID: "phantom", Content: "backup", CTime: 99, Size: 1}, // newest, and impossible
}}
r := NewBackupRunner(api, "local", "", "", "", quiet())
vol, err := r.PickRestoreCandidate(context.Background())
if err != nil || vol != "real" {
t.Fatalf("pick = %q,%v want the newest COMPLETE archive 'real'", vol, err)
}
}
// --- scheduler ---
type fakeRTRunner struct {
+142
View File
@@ -0,0 +1,142 @@
package backup
import (
"context"
"fmt"
"time"
)
// R-86 — a restore-test follows the BACKUP, not the clock.
//
// ── WHAT WAS WRONG ───────────────────────────────────────────────────────────────────────────
//
// The trigger was `time.NewTicker(cadence)` started at daemon start, and the tier was chosen by
// oldest-proven rotation. Its phase was therefore the PROCESS'S UPTIME: agent deploys are routine,
// so the test drifted to an arbitrary time of day every week; a fresh archive could sit unproven
// while an older one was re-tested; and a weekly tier was tested on the same rhythm as a daily one,
// sometimes twice on the same archive.
//
// ── THE RULE, AND THE TRAP IN ITS OBVIOUS FORM ───────────────────────────────────────────────
//
// R-86's ask reads "test a tier ~24 h after its own newest archive". Implemented literally —
// *"due when the newest archive is at least `settle` old"* — a DAILY tier is NEVER due: a new
// archive lands every day, so the newest archive's age resets to zero long before it reaches 24 h.
// The naive rule silently switches restore-testing off for the tier that matters most, and it is
// the version a reasonable person would write. It has a red-proof of its own
// (TestDue_NaiveNewestArchiveAgeRuleNeverFiresOnADailyTier).
//
// The rule implemented here:
//
// Let A = the newest archive on this tier that is at least `settle` old.
// The tier is DUE when A exists and A HAS NOT ALREADY BEEN PROVEN.
//
// daily tier → A is yesterday's archive; a new one settles each day → proved once per day
// weekly tier → A is last week's until the next settles → proved once per week
// newborn tier → A does not exist → UNKNOWN, never a fault
//
// Per-archive due-ness IS the pacing: one test per archive generation and no more. There is
// deliberately no second rate limiter on top of it (§8.4) — two independent pacing mechanisms
// produce a cadence nobody can predict from either.
//
// ── WHAT DID NOT CHANGE ──────────────────────────────────────────────────────────────────────
//
// The one-heavy-operation gate, the success-only proof credit, the oldest-proven ordering (now the
// tie-break between two DUE tiers), the restore-test itself, its journal and its scratch band. Only
// the trigger changed.
// DueVerdict is one tier's due-ness, and the evidence for it. Every field is logged: a due-check
// that cannot say WHY is a quiet path, and quiet paths are what this monitor family keeps shipping.
type DueVerdict struct {
Target string // the tier's storage target id
// Due is true only when Archive is set and has not been proven.
Due bool
// Archive is the settled candidate A ("" when the tier holds none).
Archive string
// Landed is when A landed on the tier (zero when Archive is "").
Landed time.Time
// ProvenArchive is what the state says was last proven on this tier ("" = nothing/legacy).
ProvenArchive string
// Err is a candidate-lookup failure. A tier whose archives cannot be listed is UNKNOWN — it is
// NEVER reported as "not due", which would silently retire a tier the moment its storage
// stopped answering. Due stays false (we have no archive to test) and the error travels.
Err error
// Reason is the one-line human account of this verdict.
Reason string
}
// String renders a verdict for the operator log / selftest output.
func (v DueVerdict) String() string {
return fmt.Sprintf("tier=%s due=%v archive=%q reason=%s", v.Target, v.Due, v.Archive, v.Reason)
}
// EvaluateDue returns the due verdict for every configured tier, ordered oldest-proven first.
//
// Ordering is the R-85 rotation, demoted to a TIE-BREAK: it no longer decides whether a test
// happens (due-ness does), only which of several due tiers goes first. Keeping it means a tier can
// still never be starved — a tier that has waited longest is served first — and keeping it as the
// order rather than as the trigger is the whole of this change.
func (s *Scheduler) EvaluateDue(ctx context.Context) []DueVerdict {
if !s.rotating() {
return nil
}
order := s.tiers
if s.rtState != nil {
order = s.rtState.OldestFirst(s.tiers)
}
cutoff := s.settleCutoff()
out := make([]DueVerdict, 0, len(order))
for _, target := range order {
out = append(out, s.evaluateTier(ctx, target, cutoff))
}
return out
}
// settleCutoff is the newest landing time an archive may have and still count as settled.
func (s *Scheduler) settleCutoff() time.Time {
if s.settle <= 0 {
return time.Time{} // no settle requirement configured → any archive is a candidate
}
return s.now().Add(-s.settle)
}
// evaluateTier is the per-tier due-check. PURE given the picker and the state, so the rule is
// unit-tested directly rather than inferred from whether a fake runner happened to be called.
func (s *Scheduler) evaluateTier(ctx context.Context, target string, cutoff time.Time) DueVerdict {
v := DueVerdict{Target: target}
archive, landed, err := s.tierPick(ctx, target, cutoff)
if err != nil {
// UNKNOWN, never "not due", and never silent.
v.Err = err
v.Reason = fmt.Sprintf("candidate lookup FAILED (%v) — tier is unknown this evaluation, not proven and not dismissed", err)
return v
}
v.Archive, v.Landed = archive, landed
if archive == "" {
v.Reason = "no settled archive yet — nothing to prove (newborn or still settling)"
return v
}
proven, ok := "", false
if s.rtState != nil {
proven, ok = s.rtState.ProvenArchive(target)
}
v.ProvenArchive = proven
if ok && proven == archive {
v.Reason = fmt.Sprintf("newest settled archive (landed %s) is already proven", landed.Format(time.RFC3339))
return v
}
v.Due = true
switch {
case !ok && proven == "":
v.Reason = fmt.Sprintf("newest settled archive (landed %s) has not been proven; nothing proven on this tier yet", landed.Format(time.RFC3339))
default:
v.Reason = fmt.Sprintf("newest settled archive (landed %s) has not been proven (last proven archive was a different one)", landed.Format(time.RFC3339))
}
return v
}
// EvaluateDueTier is EvaluateDue for ONE named tier — the selftest's per-tier cost probe, so the
// WAN leg of an offsite lookup is attributable rather than buried in an aggregate.
func (s *Scheduler) EvaluateDueTier(ctx context.Context, target string) DueVerdict {
return s.evaluateTier(ctx, target, s.settleCutoff())
}
+458
View File
@@ -0,0 +1,458 @@
package backup
import (
"context"
"errors"
"fmt"
"os"
"path/filepath"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
)
// R-86 — the restore-test follows the BACKUP, not the clock.
//
// Every test here DRIVES time (`s.now` is injected and stepped) rather than waiting for it. A test
// that slept could not say anything about a 24-hour rule in under 24 hours, and one that only
// asserted "no error" would pass against a scheduler that never ran anything at all — which is
// precisely the failure mode §8.1's trap produces. So the assertions are: did a test run, on WHICH
// archive, and did a second evaluation correctly run NOTHING.
// ── the fake tier storage ────────────────────────────────────────────────────────────────────
// archiveStub is one archive on a tier: its volid and when it landed.
type archiveStub struct {
volid string
landed time.Time
}
// tierStorage is a TierPicker over per-tier archive lists. It implements the SAME contract as the
// production picker (*BackupRunner).PickSettledRestoreCandidateOn — newest archive that landed at
// or before the cutoff — which is itself covered against a fake PVE API in backup_test.go, and
// end-to-end by the live run. Naming the seam explicitly: everything below is true up to this
// picker; that the real picker obeys the same rule is asserted there, not here.
type tierStorage struct {
archives map[string][]archiveStub
err map[string]error // target → lookup failure
}
func (ts *tierStorage) pick(_ context.Context, target string, notAfter time.Time) (string, time.Time, error) {
if e, ok := ts.err[target]; ok && e != nil {
return "", time.Time{}, e
}
var best archiveStub
for _, a := range ts.archives[target] {
if !notAfter.IsZero() && a.landed.After(notAfter) {
continue // not settled yet
}
if best.volid == "" || a.landed.After(best.landed) {
best = a
}
}
return best.volid, best.landed, nil
}
// dueHarness is a scheduler with a driven clock over a fake tier storage.
type dueHarness struct {
s *Scheduler
rr *rotRunner
st *RestoreTestState
ts *tierStorage
clock time.Time
path string
}
func newDueHarness(t *testing.T, start time.Time, settle time.Duration, pass bool, tiers []string, ts *tierStorage) *dueHarness {
t.Helper()
return newDueHarnessAt(t, filepath.Join(t.TempDir(), "rt.json"), start, settle, pass, tiers, ts)
}
func newDueHarnessAt(t *testing.T, statePath string, start time.Time, settle time.Duration, pass bool, tiers []string, ts *tierStorage) *dueHarness {
t.Helper()
h := &dueHarness{rr: &rotRunner{pass: pass}, ts: ts, clock: start, path: statePath}
h.st = NewRestoreTestState(statePath)
h.s = NewScheduler(SchedulerOptions{
Runner: h.rr,
Store: NewStore(),
Spec: func(_ context.Context, archive string) reconcile.RestoreTestSpec {
return reconcile.RestoreTestSpec{RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009}
},
Cadence: time.Hour,
Settle: settle,
Logger: quiet(),
Tiers: tiers,
TierPick: ts.pick,
State: h.st,
InFlight: &InFlight{},
})
h.s.now = func() time.Time { return h.clock }
return h
}
// advance steps the clock by step, evaluating once at every step — the scheduler's real shape.
func (h *dueHarness) advance(step, total time.Duration) {
for elapsed := time.Duration(0); elapsed < total; elapsed += step {
h.clock = h.clock.Add(step)
h.s.tick(context.Background())
}
}
var day0 = time.Date(2026, 8, 1, 2, 0, 0, 0, time.UTC)
// dailyArchives lands one archive a day at 02:00 for n days, starting at day0.
func dailyArchives(tier string, n int) []archiveStub {
out := make([]archiveStub, 0, n)
for d := 0; d < n; d++ {
out = append(out, archiveStub{
volid: fmt.Sprintf("%s:backup/vzdump-lxc-9201-day%d.tar.zst", tier, d),
landed: day0.AddDate(0, 0, d),
})
}
return out
}
// ── SCENARIO A — a daily tier is proved daily, on its own archive ────────────────────────────
//
// THE TRAP THIS PINS (§8.1). R-86 reads "trigger a tier ~24 h after its own newest archive", and
// the literal implementation of that — *due when the newest archive is at least `settle` old* — is
// NEVER true on a daily tier: a new archive lands every day, so the newest archive's age resets to
// zero long before it reaches 24 h. The literal reading silently switches restore-testing OFF for
// the tier that matters most.
//
// COMPANION RED-PROOF (observed 2026-08-03). In Scheduler.evaluateTier, the per-archive comparison
// was replaced by the naive age rule:
//
// - if ok && proven == archive { … not due … }
// + if s.now().Sub(landed) < s.settle { … not due … } // and the proven-archive check deleted
//
// and the picker cutoff was removed (`cutoff := time.Time{}`), i.e. exactly "is the newest archive
// old enough". Result:
//
// --- FAIL: TestDue_DailyTierIsProvedDailyOnItsOwnArchive
// restoretest_due_test.go: a daily tier must be proved once per day; got 0 run(s) over 5 days
//
// Zero runs — restore-testing off. Restored immediately afterwards.
func TestDue_DailyTierIsProvedDailyOnItsOwnArchive(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 6)}}
h := newDueHarness(t, day0.Add(time.Hour), 24*time.Hour, true, []string{"local"}, ts)
// Five days, evaluated hourly.
h.advance(time.Hour, 5*24*time.Hour)
got := h.rr.seen()
if len(got) != 5 {
t.Fatalf("a daily tier must be proved once per day; got %d run(s) over 5 days: %v", len(got), got)
}
// And each run must be on the archive that settled that day — day0's on day 1, and so on.
for i, a := range got {
want := fmt.Sprintf("local:backup/vzdump-lxc-9201-day%d.tar.zst", i)
if a != want {
t.Fatalf("run %d tested %q, want %q — the test is not following the archive", i+1, a, want)
}
}
// The newest archive is NEVER the one tested: it has not settled.
if last := got[len(got)-1]; last == "local:backup/vzdump-lxc-9201-day5.tar.zst" {
t.Fatal("the still-settling archive was tested — the settle cutoff is not being applied")
}
}
// ── SCENARIO B — a weekly tier is proved weekly, not every other day ─────────────────────────
func TestDue_WeeklyTierIsProvedOncePerArchive(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": {
{volid: "felhom-pbs:backup/ct/9201/w0", landed: day0},
{volid: "felhom-pbs:backup/ct/9201/w1", landed: day0.AddDate(0, 0, 7)},
{volid: "felhom-pbs:backup/ct/9201/w2", landed: day0.AddDate(0, 0, 14)},
}}}
h := newDueHarness(t, day0.Add(time.Hour), 24*time.Hour, true, []string{"felhom-pbs"}, ts)
// Three weeks, evaluated every 6 hours — 84 evaluations.
h.advance(6*time.Hour, 21*24*time.Hour)
got := h.rr.seen()
want := []string{
"felhom-pbs:backup/ct/9201/w0",
"felhom-pbs:backup/ct/9201/w1",
"felhom-pbs:backup/ct/9201/w2",
}
if len(got) != len(want) {
t.Fatalf("a weekly tier must be proved ONCE PER ARCHIVE (3 archives over 3 weeks); got %d run(s): %v", len(got), got)
}
for i := range want {
if got[i] != want[i] {
t.Fatalf("run %d tested %q, want %q", i+1, got[i], want[i])
}
}
}
// ── SCENARIO C — an agent restart does not change the schedule ───────────────────────────────
//
// This is the defect a person actually notices: today every deploy restarts the ticker, so a
// restore-test runs one interval after each deploy regardless of what has already been proven.
//
// COMPANION RED-PROOF (observed 2026-08-03): revert the state to per-tier TIME by making
// ProvenArchive ignore the stored archive —
//
// - if !ok || p.Archive == "" { return "", false }
// + return "", false // per-tier time only, the pre-R-86 state
//
// → --- FAIL: TestDue_RestartRunsNothing
// restoretest_due_test.go:226: an agent restart must not trigger a restore-test; 2 restart(s)
// produced 4 run(s)
//
// Four: the same already-proven archive re-tested on EVERY evaluation after EVERY restart, which is
// today's behaviour with the ticker's phase reset by the deploy. Restored.
func TestDue_RestartRunsNothing(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "rt.json")
ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 2)}}
start := day0.AddDate(0, 0, 1).Add(time.Hour) // day 1, 03:00 — day0's archive has settled
h := newDueHarnessAt(t, path, start, 24*time.Hour, true, []string{"local"}, ts)
h.s.tick(context.Background())
if n := len(h.rr.seen()); n != 1 {
t.Fatalf("precondition: the settled archive should have been proved once; got %d run(s)", n)
}
// --- two restarts: brand-new scheduler + brand-new state object over the SAME file ---
total := 0
for i := 0; i < 2; i++ {
h2 := newDueHarnessAt(t, path, start.Add(time.Duration(i+1)*time.Hour), 24*time.Hour, true, []string{"local"}, ts)
h2.s.tick(context.Background())
h2.s.tick(context.Background())
total += len(h2.rr.seen())
}
if total != 0 {
t.Fatalf("an agent restart must not trigger a restore-test; 2 restart(s) produced %d run(s)", total)
}
}
// ── SCENARIO D — a new archive makes a tier due even if it was tested yesterday ──────────────
func TestDue_NewSettledArchiveMakesAProvedTierDueAgain(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 2)}}
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, true, []string{"local"}, ts)
h.s.tick(context.Background()) // proves day0's archive
h.s.tick(context.Background()) // nothing new has settled → nothing
if n := len(h.rr.seen()); n != 1 {
t.Fatalf("want exactly 1 run before the new archive settles, got %d: %v", n, h.rr.seen())
}
// Day 2, 03:00 — day1's archive has now settled.
h.clock = day0.AddDate(0, 0, 2).Add(time.Hour)
h.s.tick(context.Background())
got := h.rr.seen()
if len(got) != 2 {
t.Fatalf("a newly settled archive must make the tier due again; got %v", got)
}
if got[1] != "local:backup/vzdump-lxc-9201-day1.tar.zst" {
t.Fatalf("the NEW archive must be the one tested; got %q", got[1])
}
}
// ── SCENARIO E — a failing tier keeps being retried, and earns no proof ──────────────────────
//
// COMPANION RED-PROOF (observed 2026-08-03): give credit on failure in Scheduler.tick —
//
// - if rt.Pass && s.rtState != nil && target != "" {
// + if s.rtState != nil && target != "" {
//
// → --- FAIL: TestDue_FailingTierIsRetriedAndNeverProven
// restoretest_due_test.go: a failing tier must keep being retried; got 1 run(s) over 3
// evaluations
//
// A single failure would have retired the archive as proven — a permanently broken DR tier looking
// freshly verified, which is the loudest signal this system produces going silent. Restored.
func TestDue_FailingTierIsRetriedAndNeverProven(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 1)}}
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, false, []string{"local"}, ts)
for i := 0; i < 3; i++ {
h.s.tick(context.Background())
}
got := h.rr.seen()
if len(got) != 3 {
t.Fatalf("a failing tier must keep being retried; got %d run(s) over 3 evaluations: %v", len(got), got)
}
if _, ok := h.st.ProvenArchive("local"); ok {
t.Fatal("a FAILED restore-test must not record the archive as proven")
}
if _, ok := h.st.LastSuccess("local"); ok {
t.Fatal("a FAILED restore-test must not stamp the tier as proven")
}
}
// ── SCENARIO F — two tiers due at once do not run at once ────────────────────────────────────
func TestDue_TwoDueTiersRunOneAtATime(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{
"local": {{volid: "local:backup/a.tar.zst", landed: day0}},
"felhom-pbs": {{volid: "felhom-pbs:backup/ct/9201/a", landed: day0}},
}}
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts)
// Both tiers are due at this instant.
due := h.s.EvaluateDue(context.Background())
if len(due) != 2 || !due[0].Due || !due[1].Due {
t.Fatalf("precondition: both tiers should be due; got %v", due)
}
h.s.tick(context.Background())
if n := len(h.rr.seen()); n != 1 {
t.Fatalf("ONE evaluation must start ONE restore-test, never two multi-GB restores over one link; got %d: %v", n, h.rr.seen())
}
// The other tier was DEFERRED, not cancelled: it is still due and runs on the next evaluation.
h.s.tick(context.Background())
got := h.rr.seen()
if len(got) != 2 || got[0] == got[1] {
t.Fatalf("the deferred tier must run on the NEXT evaluation, on its own archive; got %v", got)
}
}
// The heavy-operation gate still holds, and a tier deferred behind a backup stays DUE.
func TestDue_DeferredBehindABackupStaysDue(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{"local": {{volid: "local:backup/a.tar.zst", landed: day0}}}}
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, true, []string{"local"}, ts)
gate := &InFlight{}
h.s.inFlight = gate
release, _, _ := gate.TryAcquire("backup:felhom-pbs")
h.s.tick(context.Background())
if n := len(h.rr.seen()); n != 0 {
t.Fatalf("the restore-test must DEFER while a backup holds the gate; got %d run(s)", n)
}
if due := h.s.EvaluateDue(context.Background()); !due[0].Due {
t.Fatal("a deferred tier must remain DUE — deferral is not dismissal")
}
release()
h.s.tick(context.Background())
if n := len(h.rr.seen()); n != 1 {
t.Fatalf("must resume once the gate frees; got %d run(s)", n)
}
}
// ── SCENARIO H — a newborn box is UNKNOWN, not stale and not a fault ─────────────────────────
func TestDue_NewbornTierIsNotDueAndNotAnError(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": nil}}
h := newDueHarness(t, day0, 24*time.Hour, true, []string{"felhom-pbs"}, ts)
due := h.s.EvaluateDue(context.Background())
if len(due) != 1 {
t.Fatalf("want one verdict, got %v", due)
}
v := due[0]
if v.Due || v.Err != nil || v.Archive != "" {
t.Fatalf("a tier with no archive is UNKNOWN — not due, not an error; got %+v", v)
}
if v.Reason == "" {
t.Fatal("every verdict must carry a reason — a due-check that cannot say why is a quiet path")
}
h.s.tick(context.Background())
if n := len(h.rr.seen()); n != 0 {
t.Fatalf("a newborn tier must not be restore-tested; got %d run(s)", n)
}
}
// An archive that exists but has NOT settled yet is not a candidate — and that is not an error.
func TestDue_UnsettledArchiveIsNotACandidate(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{"local": {{volid: "local:backup/fresh.tar.zst", landed: day0}}}}
h := newDueHarness(t, day0.Add(2*time.Hour), 24*time.Hour, true, []string{"local"}, ts)
if v := h.s.EvaluateDue(context.Background())[0]; v.Due || v.Archive != "" {
t.Fatalf("an archive 2h old must not be a candidate under a 24h settle lag; got %+v", v)
}
h.s.tick(context.Background())
if n := len(h.rr.seen()); n != 0 {
t.Fatalf("nothing settled → no run; got %d", n)
}
}
// A tier whose archives cannot be LISTED is UNKNOWN — never silently "not due", and never silent.
// Treating a lookup failure as "not due" would retire a tier the moment its storage stopped
// answering, which is the same absence-is-not-evidence error this monitor family keeps making.
func TestDue_LookupFailureIsUnknownNotNotDue(t *testing.T) {
boom := errors.New("storage unreachable")
ts := &tierStorage{
archives: map[string][]archiveStub{"local": {{volid: "local:backup/a.tar.zst", landed: day0}}},
err: map[string]error{"felhom-pbs": boom},
}
h := newDueHarness(t, day0.AddDate(0, 0, 1), 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts)
var pbs DueVerdict
for _, v := range h.s.EvaluateDue(context.Background()) {
if v.Target == "felhom-pbs" {
pbs = v
}
}
if pbs.Err == nil {
t.Fatal("a lookup failure must travel in the verdict, not be swallowed")
}
if pbs.Due {
t.Fatal("a tier we could not list must not be reported DUE — we have no archive to test")
}
if pbs.Reason == "" {
t.Fatal("the failure must be explained, not merely flagged")
}
// And the OTHER tier still runs: one tier's storage being unreadable must not cost the other
// tier its proof.
h.s.tick(context.Background())
if got := h.rr.seen(); len(got) != 1 || got[0] != "local:backup/a.tar.zst" {
t.Fatalf("the readable tier must still be proved; got %v", got)
}
}
// ── the state's migration (§8.2) ─────────────────────────────────────────────────────────────
// A pre-R-86 state file carries a TIME and no archive. It must keep its time (rotation ordering
// survives the upgrade) and yield NO proven archive, so each tier is due exactly once. Reading a
// legacy time as proof of the CURRENT archive would mark an unproven archive proven — a guarantee
// invented by a migration.
func TestRestoreTestState_LegacyFileMigratesToNothingProven(t *testing.T) {
path := filepath.Join(t.TempDir(), "rt.json")
legacy := `{"local":"2026-08-01T02:00:00Z","felhom-pbs":"2026-07-30T02:00:00Z"}`
if err := writeFileForTest(path, legacy); err != nil {
t.Fatal(err)
}
st := NewRestoreTestState(path)
if _, ok := st.ProvenArchive("local"); ok {
t.Fatal("a legacy record names no archive — it must NOT be read as proof of the current one")
}
at, ok := st.LastSuccess("local")
if !ok || !at.Equal(time.Date(2026, 8, 1, 2, 0, 0, 0, time.UTC)) {
t.Fatalf("the legacy TIME must survive (rotation ordering depends on it); got %v ok=%v", at, ok)
}
// Ordering still works off the legacy times.
if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "felhom-pbs" {
t.Fatalf("oldest-first must still order legacy records; got %v", got)
}
}
// The new shape round-trips, archive and all.
func TestRestoreTestState_ArchiveRoundTrips(t *testing.T) {
path := filepath.Join(t.TempDir(), "rt.json")
now := time.Now().UTC().Truncate(time.Second)
st := NewRestoreTestState(path)
if err := st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/x", now); err != nil {
t.Fatal(err)
}
re := NewRestoreTestState(path)
got, ok := re.ProvenArchive("felhom-pbs")
if !ok || got != "felhom-pbs:backup/ct/9201/x" {
t.Fatalf("the proven ARCHIVE must survive a restart; got %q ok=%v", got, ok)
}
at, ok := re.LastSuccess("felhom-pbs")
if !ok || !at.Equal(now) {
t.Fatalf("the proven TIME must survive too; got %v ok=%v", at, ok)
}
}
// writeFileForTest is a tiny helper so the legacy-migration fixture reads clearly above.
func writeFileForTest(path, content string) error {
return os.WriteFile(path, []byte(content), 0o600)
}
+83 -19
View File
@@ -28,42 +28,92 @@ import (
//
// Only SUCCESS is recorded. A failed run must not satisfy rotation, or a tier that fails every time
// would look freshly proven and stop being retried — the same "a failure satisfies the cadence"
// trap the backup due-check avoids.
// trap the backup due-check avoids. R-86 keeps that property unchanged and gives it a second job:
// the due-check reads this state, so a failure that recorded proof would ALSO stop the tier from
// ever becoming due again. The rule earns its keep twice now.
//
// R-86 (1.2) — WHICH ARCHIVE, not just when.
//
// A timestamp alone cannot answer the question the due-check asks. "This tier passed at 04:00" is
// consistent both with "yesterday's archive is proven" and with "an archive from a week ago is
// proven and nothing since has been looked at". Restore-testing is now per ARCHIVE GENERATION —
// a tier is due once it holds a settled archive that has not been proven — so the identity of the
// proven archive is the state, and the time is metadata (rotation ordering, operator reporting).
//
// This is the same class as the workspace rule "a timestamp records an ATTEMPT, not a RESULT":
// here it records a result, but not WHICH result, and that is just as unable to answer the question
// being asked of it.
type RestoreTestState struct {
path string
mu sync.Mutex
last map[string]time.Time // target id → last SUCCESSFUL restore-test (UTC)
last map[string]provenTier // target id → what was last PROVEN on that tier
}
// provenTier is one tier's proof: the archive that passed, and when it passed.
type provenTier struct {
Archive string // volid of the archive that PASSED; "" = a legacy record with no archive
At time.Time // when that run passed (UTC)
}
// provenTierJSON is the on-disk shape (R-86). The legacy shape was a bare RFC3339 STRING per
// target; both are read, only this one is written — see NewRestoreTestState.
type provenTierJSON struct {
Archive string `json:"archive"`
ProvenAt string `json:"proven_at"`
}
// NewRestoreTestState opens (or creates) the state at path. A missing or unreadable file is NOT an
// error: it degrades to "nothing proven yet", which is the correct starting point and keeps a
// corrupt file from wedging the daemon.
//
// MIGRATION (R-86). The pre-R-86 file is `{"<target>": "<RFC3339>"}` — a time and no archive. A
// legacy record keeps its TIME (rotation ordering survives a deploy, which is why the file exists
// at all) but yields NO proven archive, so every tier is due exactly once on first evaluation after
// the upgrade. One extra restore-test per tier, once, is the safe direction: the alternative is to
// read a legacy time as proof of whatever archive happens to be current, which would mark an
// unproven archive proven — inventing a guarantee out of a migration.
func NewRestoreTestState(path string) *RestoreTestState {
s := &RestoreTestState{path: path, last: map[string]time.Time{}}
s := &RestoreTestState{path: path, last: map[string]provenTier{}}
data, err := os.ReadFile(path)
if err != nil {
return s
}
var raw map[string]string
var raw map[string]json.RawMessage
if json.Unmarshal(data, &raw) != nil {
return s
}
for target, ts := range raw {
if t, perr := time.Parse(time.RFC3339, ts); perr == nil {
s.last[target] = t.UTC()
for target, msg := range raw {
// Legacy shape: a bare RFC3339 string.
var legacy string
if json.Unmarshal(msg, &legacy) == nil {
if t, perr := time.Parse(time.RFC3339, legacy); perr == nil {
s.last[target] = provenTier{At: t.UTC()} // no archive → due once, deliberately
}
continue
}
var cur provenTierJSON
if json.Unmarshal(msg, &cur) != nil {
continue // one unreadable entry must not lose the others
}
t, perr := time.Parse(time.RFC3339, cur.ProvenAt)
if perr != nil {
continue
}
s.last[target] = provenTier{Archive: cur.Archive, At: t.UTC()}
}
return s
}
// RecordSuccess stamps a tier as proven at t. Only call this for a PASSING restore-test.
func (s *RestoreTestState) RecordSuccess(target string, t time.Time) error {
// RecordSuccess stamps a tier as proven at t, naming the ARCHIVE that passed. Only call this for a
// PASSING restore-test — the archive is what makes the tier not-due, so recording one for a failed
// run would retire the archive unproven.
func (s *RestoreTestState) RecordSuccess(target, archive string, t time.Time) error {
if target == "" {
return nil
}
s.mu.Lock()
defer s.mu.Unlock()
s.last[target] = t.UTC()
s.last[target] = provenTier{Archive: archive, At: t.UTC()}
return s.saveLocked()
}
@@ -71,17 +121,30 @@ func (s *RestoreTestState) RecordSuccess(target string, t time.Time) error {
func (s *RestoreTestState) LastSuccess(target string) (time.Time, bool) {
s.mu.Lock()
defer s.mu.Unlock()
t, ok := s.last[target]
return t, ok
p, ok := s.last[target]
return p.At, ok
}
// Snapshot returns a copy of the whole map — for the host-report gauge.
// ProvenArchive returns the archive last PROVEN on this tier (ok=false = none — either never tested,
// or a legacy record carrying only a time). It is the due-check's whole question: an archive that is
// not this one has not been proven.
func (s *RestoreTestState) ProvenArchive(target string) (string, bool) {
s.mu.Lock()
defer s.mu.Unlock()
p, ok := s.last[target]
if !ok || p.Archive == "" {
return "", false
}
return p.Archive, true
}
// Snapshot returns a copy of the last-proven TIMES — for the host-report gauge.
func (s *RestoreTestState) Snapshot() map[string]time.Time {
s.mu.Lock()
defer s.mu.Unlock()
out := make(map[string]time.Time, len(s.last))
for k, v := range s.last {
out[k] = v
out[k] = v.At
}
return out
}
@@ -100,8 +163,9 @@ func (s *RestoreTestState) OldestFirst(targets []string) []string {
defer s.mu.Unlock()
out := append([]string(nil), targets...)
sort.SliceStable(out, func(i, j int) bool {
ti, oki := s.last[out[i]]
tj, okj := s.last[out[j]]
pi, oki := s.last[out[i]]
pj, okj := s.last[out[j]]
ti, tj := pi.At, pj.At
switch {
case !oki && !okj:
return out[i] < out[j] // both never proven → deterministic
@@ -119,9 +183,9 @@ func (s *RestoreTestState) OldestFirst(targets []string) []string {
}
func (s *RestoreTestState) saveLocked() error {
raw := make(map[string]string, len(s.last))
for target, t := range s.last {
raw[target] = t.UTC().Format(time.RFC3339)
raw := make(map[string]provenTierJSON, len(s.last))
for target, p := range s.last {
raw[target] = provenTierJSON{Archive: p.Archive, ProvenAt: p.At.UTC().Format(time.RFC3339)}
}
data, err := json.MarshalIndent(raw, "", " ")
if err != nil {
+36 -17
View File
@@ -39,10 +39,19 @@ func (r *rotRunner) seen() []string {
return append([]string(nil), r.archives...)
}
// testLanded is a landing time old enough to be settled under any cutoff these tests use. R-86
// widened the TierPicker seam with the archive's landing time; the rotation tests below are about
// tier ORDER and the heavy-operation gate, not about settling, so they hold it constant.
var testLanded = time.Date(2026, 7, 1, 0, 0, 0, 0, time.UTC)
// archiveFor is a TierPicker over a fixed map: target → archive ("" = that tier holds none).
func archiveFor(m map[string]string) TierPicker {
return func(_ context.Context, target string) (string, error) {
return m[target], nil
return func(_ context.Context, target string, _ time.Time) (string, time.Time, error) {
a := m[target]
if a == "" {
return "", time.Time{}, nil
}
return a, testLanded, nil
}
}
@@ -63,14 +72,24 @@ func rotScheduler(t *testing.T, rr *rotRunner, st *RestoreTestState, pick TierPi
})
}
// ── SCENARIO A — both tiers get tested across consecutive cadences ───────────────────────────
// ── SCENARIO A — both tiers get tested, each ONCE per archive ────────────────────────────────
//
// R-86 CHANGED THIS TEST'S CONTRACT, deliberately, and the old assertion is worth recording because
// it was a faithful statement of the defect. It read:
//
// 4 ticks → 4 runs, and consecutive runs must hit different tiers
//
// i.e. every tick produced a heavy restore-test, because the ticker WAS the trigger. Under R-86 a
// tick is an EVALUATION: both tiers are still exercised (rotation is intact), but a tier whose
// newest settled archive is already proven is not re-tested just because time passed. So the
// assertion is now 2 runs across 4 evaluations — one per tier, one per archive — which is a
// STRICTLY STRONGER statement: it pins both the coverage R-85 won and the pacing R-86 adds.
//
// COMPANION RED-PROOF (observed): restore the single-target picker — set `Tiers`/`TierPick` to nil
// so `pickForThisRun` falls back to `s.pick` on the primary runner — and this fails with
// "both tiers must be exercised across 4 cadences; got [local:… local:… local:… local:…]",
// i.e. the offsite tier never appears. That is today's behaviour, and it is why demo-hp's DR tier
// went unproven for its entire existence.
func TestRotation_BothTiersExercisedAcrossCadences(t *testing.T) {
// "both tiers must be exercised; got [local:…]", i.e. the offsite tier never appears. That is
// pre-R-85 behaviour, and it is why demo-hp's DR tier went unproven for its entire existence.
func TestRotation_BothTiersExercisedOncePerArchive(t *testing.T) {
rr := &rotRunner{pass: true}
st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
s := rotScheduler(t, rr, st, archiveFor(map[string]string{
@@ -94,14 +113,14 @@ func TestRotation_BothTiersExercisedAcrossCadences(t *testing.T) {
}
}
if !sawLocal || !sawPBS {
t.Fatalf("both tiers must be exercised across 4 cadences; got %v", got)
t.Fatalf("both tiers must be exercised; got %v", got)
}
// Oldest-first must ALTERNATE, not clump — otherwise one tier is starved between visits.
if len(got) != 4 {
t.Fatalf("want 4 runs, got %d: %v", len(got), got)
// Exactly one run per tier: the archives never changed, so nothing became due a second time.
if len(got) != 2 {
t.Fatalf("want 2 runs across 4 evaluations (one per archive generation), got %d: %v", len(got), got)
}
if got[0] == got[1] {
t.Fatalf("consecutive runs hit the same tier — oldest-first is not rotating: %v", got)
t.Fatalf("the two runs must be different tiers — oldest-first is not ordering due tiers: %v", got)
}
}
@@ -284,11 +303,11 @@ func TestOldestFirst_Ordering(t *testing.T) {
t.Fatalf("unexpected: %v", got)
}
}
_ = st.RecordSuccess("local", now)
_ = st.RecordSuccess("local", "local:backup/a.tar.zst", now)
if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "felhom-pbs" {
t.Fatalf("a never-proven tier must sort before a proven one; got %v", got)
}
_ = st.RecordSuccess("felhom-pbs", now.Add(time.Hour))
_ = st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/b", now.Add(time.Hour))
if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "local" {
t.Fatalf("the least recently proven must sort first; got %v", got)
}
@@ -299,8 +318,8 @@ func TestOldestFirst_Ordering(t *testing.T) {
func TestOldestFirst_DeterministicOnTies(t *testing.T) {
st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
now := time.Now().UTC()
_ = st.RecordSuccess("b-tier", now)
_ = st.RecordSuccess("a-tier", now)
_ = st.RecordSuccess("b-tier", "b:archive", now)
_ = st.RecordSuccess("a-tier", "a:archive", now)
for i := 0; i < 20; i++ {
if got := st.OldestFirst([]string{"b-tier", "a-tier"}); got[0] != "a-tier" {
t.Fatalf("tie-break must be deterministic; iteration %d gave %v", i, got)
@@ -315,7 +334,7 @@ func TestRestoreTestState_PersistenceAndCorruption(t *testing.T) {
now := time.Now().UTC().Truncate(time.Second)
st := NewRestoreTestState(path)
if err := st.RecordSuccess("felhom-pbs", now); err != nil {
if err := st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/x", now); err != nil {
t.Fatal(err)
}
reopened := NewRestoreTestState(path)
+42 -4
View File
@@ -260,21 +260,59 @@ func (r *BackupRunner) PickRestoreCandidate(ctx context.Context) (string, error)
// restore is not an error** — a brand-new offsite tier legitimately has nothing yet, and turning
// that into a failure would make every fresh box look broken for its first week.
func (r *BackupRunner) PickRestoreCandidateOn(ctx context.Context, target string) (string, error) {
archive, _, err := r.PickSettledRestoreCandidateOn(ctx, target, time.Time{})
return archive, err
}
// PickSettledRestoreCandidateOn is the R-86 due-check's picker: the newest archive on target that
// landed AT OR BEFORE notAfter (the settle cutoff), with the time it landed. A zero notAfter means
// "no cutoff" — that is the pre-R-86 behaviour, which is why PickRestoreCandidateOn is now a
// one-line call into this and its contract is untouched (one scan, one owner).
//
// WHY A CUTOFF AT ALL. An archive that landed minutes ago may still be settling — R-71a's
// settle-gate exists because the offsite tier's day-0 consume raced its own floor update — and
// restore-testing the archive a backup is still writing proves nothing about the backup that
// finished. The due-check therefore asks about the newest SETTLED archive, and §8.1's rule is built
// on that: the tier is due when a settled archive exists that has not been proven.
//
// The plausibility floor is applied here and not in the old path on purpose. Under R-86 the picked
// archive becomes the tier's due-ness: an incomplete 1-byte phantom (F-CRIT-2's artefact — server
// prune does NOT collect it) would be selected forever, fail its restore forever, never earn proof,
// and so make the tier due at EVERY evaluation. Skipping it is what keeps the retry rate bounded by
// the archive generation rather than by the evaluation interval.
//
// Contract preserved: ("", zero, nil) when the storage holds no eligible archive. **A tier with
// nothing to restore is not an error** — a brand-new offsite tier legitimately has nothing yet, and
// turning that into a failure would make every fresh box look broken for its first week.
func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target string, notAfter time.Time) (string, time.Time, error) {
if target == "" {
return "", nil
return "", time.Time{}, nil
}
contents, err := r.api.StorageContent(ctx, target)
if err != nil {
return "", err
return "", time.Time{}, err
}
var best string
var bestCTime int64 = -1
for _, e := range contents {
if e.Content == "backup" && e.CTime > bestCTime {
if e.Content != "backup" {
continue
}
if !notAfter.IsZero() && e.CTime > notAfter.Unix() {
continue // not settled yet — a newer archive is not a reason to re-prove an older one
}
if ok, why := archivePlausiblyComplete(e); !ok {
r.warnRejectedArchiveOnce(e, why)
continue
}
if e.CTime > bestCTime {
bestCTime, best = e.CTime, e.VolID
}
}
return best, nil
if best == "" {
return "", time.Time{}, nil
}
return best, time.Unix(bestCTime, 0).UTC(), nil
}
// latestArchive finds the newest backup archive volid + size for vmid on the target.
+107 -59
View File
@@ -32,22 +32,31 @@ type CandidatePicker func(ctx context.Context) (string, error)
// PBS archive was classified "local" and got the 10-minute local wait.
type SpecBuilder func(ctx context.Context, archive string) reconcile.RestoreTestSpec
// TierPicker resolves the newest archive on a NAMED tier, or "" when that tier holds none.
// (*BackupRunner).PickRestoreCandidateOn satisfies it. "" must NOT be an error — a brand-new
// offsite tier legitimately has nothing to restore yet.
type TierPicker func(ctx context.Context, target string) (string, error)
// TierPicker resolves the newest archive on a NAMED tier that landed AT OR BEFORE notAfter (the
// settle cutoff), together with when it landed. (*BackupRunner).PickSettledRestoreCandidateOn
// satisfies it. A zero notAfter means "no settle requirement".
//
// R-86 widened this seam from (target) → archive. The landing time is what makes the due-check's
// verdict explainable — "archive X, which landed at T, has not been proven" — and the cutoff is
// what makes the rule per-ARCHIVE-GENERATION instead of per-interval. "" must NOT be an error: a
// brand-new offsite tier legitimately has nothing to restore yet.
type TierPicker func(ctx context.Context, target string, notAfter time.Time) (archive string, landed time.Time, err error)
// Scheduler runs the self-restore-test on an agent-internal cadence. It is the fourth daemon
// goroutine; it does real restore→boot→destroy, so it only runs when the cadence is enabled
// AND a valid scratch band is configured (validated by the caller before construction).
type Scheduler struct {
runner RestoreTestRunner
pick CandidatePicker
store *Store
spec SpecBuilder // R-85: evaluated PER RUN, never frozen at construction
runner RestoreTestRunner
pick CandidatePicker
store *Store
spec SpecBuilder // R-85: evaluated PER RUN, never frozen at construction
// cadence is the EVALUATION interval (R-86) — how often "is anything due?" is asked. It is no
// longer the thing that decides a test happens; see restoretest_due.go.
cadence time.Duration
logger *slog.Logger
now func() time.Time
// settle is how long an archive must have sat before it is a candidate (R-86).
settle time.Duration
logger *slog.Logger
now func() time.Time
// R-85 tier rotation. All optional: without them the scheduler behaves exactly as before
// (single tier via `pick`), which keeps every existing caller and test working untouched.
@@ -64,9 +73,14 @@ type SchedulerOptions struct {
Store *Store
// Spec builds the run's spec (RestoreStorage, ScratchMin/Max, SourceTier, timeouts) from the
// picked archive. Called ONCE PER RUN — see SpecBuilder for why it is not a value.
Spec SpecBuilder
Cadence time.Duration // 0 → disabled
Logger *slog.Logger
Spec SpecBuilder
// Cadence is the EVALUATION interval — how often due-ness is asked, NOT how often a test runs
// (R-86). 0 → disabled.
Cadence time.Duration
// Settle is how long an archive must have sat before it is a restore-test candidate (R-86).
// 0 → no settle requirement (any archive is a candidate).
Settle time.Duration
Logger *slog.Logger
// R-85 (all optional — omit for the pre-R-85 single-tier behaviour):
// Tiers are the configured tier target ids (primary first); TierPick resolves an archive on a
@@ -89,6 +103,7 @@ func NewScheduler(opts SchedulerOptions) *Scheduler {
store: opts.Store,
spec: opts.Spec,
cadence: opts.Cadence,
settle: opts.Settle,
logger: logger,
now: func() time.Time { return time.Now().UTC() },
tiers: append([]string(nil), opts.Tiers...),
@@ -98,9 +113,19 @@ func NewScheduler(opts SchedulerOptions) *Scheduler {
}
}
// Run fires a restore-test on the cadence until ctx is cancelled. A 0 cadence disables it
// (the goroutine just waits for shutdown). It does NOT fire immediately on start (a restore
// is heavy; the first runs one interval in) — on-demand runs use the selftest harness.
// Run EVALUATES due-ness on the interval until ctx is cancelled, and runs a restore-test only when
// a tier is actually due (R-86). A 0 interval disables it (the goroutine just waits for shutdown).
//
// The ticker survives as the evaluation interval and nothing else. It is emphatically NOT the
// trigger any more: its phase is the process's uptime, and agent deploys reset it, which is exactly
// the defect R-86 removes. What decides that a test happens is `EvaluateDue`.
//
// It still does NOT evaluate immediately on start — the first evaluation is one interval in. That
// is an EARNED restraint, kept deliberately: a restore is heavy, agent restarts are routine, and a
// crash-loop that evaluated at start would hammer a permanently-failing tier as fast as it could
// restart. Due-ness does not expire while we wait, so the only cost is up to one interval of
// latency on a tier that just became due. On-demand runs use `--selftest=restore-test`.
//
// Returns nil on ctx cancellation.
func (s *Scheduler) Run(ctx context.Context) error {
if s.cadence <= 0 || s.runner == nil || s.spec == nil || (s.pick == nil && !s.rotating()) {
@@ -108,7 +133,8 @@ func (s *Scheduler) Run(ctx context.Context) error {
<-ctx.Done()
return nil
}
s.logger.Info("backup: restore-test scheduler starting", "cadence", s.cadence)
s.logger.Info("backup: restore-test scheduler starting (per-archive due-check)",
"eval_interval", s.cadence, "settle", s.settle)
t := time.NewTicker(s.cadence)
defer t.Stop()
for {
@@ -122,8 +148,12 @@ func (s *Scheduler) Run(ctx context.Context) error {
}
}
// tick runs one scheduled restore-test: pick a backup → run → record. No-ops cleanly when
// no backup exists yet. Deterministic given s.now — tests call it directly.
// tick is ONE EVALUATION: gate → due-check → run the first due tier → record which archive was
// proven. No-ops cleanly when nothing is due, when no backup exists yet, or when a heavy operation
// is already in flight. Deterministic given s.now — tests call it directly.
//
// One run per evaluation, by construction (Scenario F): a second due tier is left DUE and picked up
// by the next evaluation. Deferred, never cancelled, and never two multi-GB restores over one link.
func (s *Scheduler) tick(ctx context.Context) {
if s.spec == nil {
// Defensive: Run() already refuses to start without a SpecBuilder, but tick is also
@@ -132,27 +162,40 @@ func (s *Scheduler) tick(ctx context.Context) {
s.logger.Error("backup: restore-test has no spec builder — skipping (this is a wiring bug)")
return
}
// Scenario F: join the one-heavy-operation-at-a-time gate. A restore-test PULLS a multi-GB
// archive over the same tunnel an offsite backup PUSHES one; running both saturates the link and
// drives each toward its timeout, which is how a healthy tier gets recorded as failed. DEFER —
// never cancel what is already running: a deferred restore-test costs hours of coverage, a
// cancelled backup costs the backup.
release, busy, ok := s.inFlight.TryAcquire("restore-test")
if !ok {
s.logger.Info("backup: restore-test deferred — a heavy operation is already in flight", "busy", busy)
return
}
defer release()
// The due-check runs BEFORE the gate is taken, and that ORDER is load-bearing under R-86.
//
// It used to be the other way round, and correctly so: the gate was held for one heavy run a
// day, and the candidate lookup rode along inside it. Evaluations are now frequent, and the
// lookup is a storage listing that for the offsite tier crosses the WAN. Holding the
// one-heavy-operation gate for a read that answers "nothing to do" would open a small window at
// EVERY evaluation in which a starting backup cannot acquire — and a backup that cannot acquire
// does not merely wait, it records a failure and pages the operator (F-A1). A cheap poll must
// not be able to manufacture that.
//
// Nothing is lost by checking first: due-ness does not expire, and the gate is still taken
// before anything heavy begins.
archive, target, err := s.pickForThisRun(ctx)
if err != nil {
s.logger.Warn("backup: restore-test could not pick a candidate; skipping", "err", err)
return
}
if archive == "" {
s.logger.Info("backup: restore-test skipped; no backup available yet")
s.logger.Debug("backup: restore-test not due this evaluation")
return
}
// Scenario F: join the one-heavy-operation-at-a-time gate. A restore-test PULLS a multi-GB
// archive over the same tunnel an offsite backup PUSHES one; running both saturates the link and
// drives each toward its timeout, which is how a healthy tier gets recorded as failed. DEFER —
// never cancel what is already running: a deferred restore-test costs hours of coverage, a
// cancelled backup costs the backup. A deferred tier stays DUE, so the next evaluation retries it.
release, busy, ok := s.inFlight.TryAcquire("restore-test")
if !ok {
s.logger.Info("backup: restore-test deferred — a heavy operation is already in flight",
"busy", busy, "target", target, "archive", archive)
return
}
defer release()
// R-85: build the spec for THIS run, from THIS archive. Never a frozen value.
spec := s.spec(ctx, archive)
spec.Archive = archive
@@ -165,8 +208,10 @@ func (s *Scheduler) tick(ctx context.Context) {
// Rotation credit is given ONLY on success. A failing tier must keep sorting first, or a tier
// that fails every time would look freshly proven and quietly stop being retried.
if rt.Pass && s.rtState != nil && target != "" {
if err := s.rtState.RecordSuccess(target, s.now()); err != nil {
s.logger.Warn("backup: could not persist the restore-test rotation state", "target", target, "err", err)
// R-86: the ARCHIVE is recorded, not merely the time — that is what makes the tier
// not-due until a NEWER archive settles, and what makes a proof survive a restart.
if err := s.rtState.RecordSuccess(target, archive, s.now()); err != nil {
s.logger.Warn("backup: could not persist the restore-test proof state", "target", target, "err", err)
}
}
switch {
@@ -189,46 +234,49 @@ func (s *Scheduler) tick(ctx context.Context) {
// rotating reports whether multi-tier rotation is wired.
func (s *Scheduler) rotating() bool { return len(s.tiers) > 0 && s.tierPick != nil }
// pickForThisRun chooses the tier and its newest archive.
// pickForThisRun chooses the tier to test THIS evaluation: the first DUE tier, in oldest-proven
// order.
//
// OLDEST-FIRST (operator ruling 2026-07-26, Option 1): the tier whose last SUCCESSFUL restore-test
// is oldest goes first, never-proven first of all. Self-balancing, no config knob, and it naturally
// prioritises a tier that has never been proven — which on this fleet was the offsite tier, unproven
// for its entire existence while reporting `applied`.
// R-86 changed what this answers. It used to answer "whose turn is it?", and the answer was always
// somebody's — the ticker had fired, so a test was going to happen. It now answers "is anything
// due?", and "nothing" is a normal, frequent and correct answer.
//
// A tier with no archives is SKIPPED, not failed, and the next tier is tried. Skipping to a testable
// tier is strictly better than burning the whole cadence: a brand-new offsite tier has nothing to
// restore yet, and that is normal, not broken. It cannot starve the empty tier either — as soon as
// it has an archive it still sorts first, because it is still the least recently proven.
// OLDEST-FIRST (operator ruling 2026-07-26, Option 1) survives as the ORDER among due tiers: the
// tier whose last successful restore-test is oldest goes first, never-proven first of all. It is
// self-balancing, needs no config knob, and it still cannot starve a tier — but it no longer decides
// that a test happens at all.
//
// Returns ("", "", nil) when nothing anywhere is testable.
// A tier with no settled archive is SKIPPED, not failed — a brand-new offsite tier has nothing to
// restore yet, and that is normal, not broken. A tier whose archives cannot be LISTED is likewise
// skipped, loudly, and its error is returned only when no other tier was testable: one tier's
// storage being unreadable must not cost the other tier its proof, and must not be silent either.
//
// Returns ("", "", nil) when nothing anywhere is due.
func (s *Scheduler) pickForThisRun(ctx context.Context) (archive, target string, err error) {
if !s.rotating() {
// Pre-R-85 single-tier path (tests and any caller that wires only `Pick`): there is no tier
// identity and no persisted proof here, so there is nothing to compare an archive against
// and no due-check is possible. It runs on every evaluation, exactly as it always did.
a, perr := s.pick(ctx)
return a, "", perr // pre-R-85 single-tier path; no rotation credit to record
}
order := s.tiers
if s.rtState != nil {
order = s.rtState.OldestFirst(s.tiers)
return a, "", perr
}
var firstErr error
for _, t := range order {
a, perr := s.tierPick(ctx, t)
if perr != nil {
// One tier's storage being unreadable must not block the others.
for _, v := range s.EvaluateDue(ctx) {
if v.Err != nil {
s.logger.Warn("backup: restore-test candidate lookup failed for a tier; trying the next",
"target", t, "err", perr)
"target", v.Target, "err", v.Err)
if firstErr == nil {
firstErr = perr
firstErr = v.Err
}
continue
}
if a == "" {
s.logger.Debug("backup: restore-test tier has no archive yet; trying the next", "target", t)
if !v.Due {
s.logger.Debug("backup: restore-test tier is not due", "target", v.Target, "reason", v.Reason)
continue
}
s.logger.Info("backup: restore-test tier selected (oldest-proven first)", "target", t, "archive", a)
return a, t, nil
s.logger.Info("backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)",
"target", v.Target, "archive", v.Archive, "landed", v.Landed.Format(time.RFC3339), "reason", v.Reason)
return v.Archive, v.Target, nil
}
if firstErr != nil {
return "", "", firstErr
+95 -13
View File
@@ -333,9 +333,23 @@ type BackupConfig struct {
LocalBackupTarget string `json:"local_backup_target"`
// RestoreStorage is where a restore-test's restored rootfs lands, e.g. "local-lvm".
RestoreStorage string `json:"restore_storage"`
// RestoreTestCadenceSeconds is the self-restore-test interval; 0 → default (24h).
// Set negative to DISABLE the automatic cadence (on-demand selftest still works).
// RestoreTestCadenceSeconds is the LEGACY restore-test knob, retained for one meaning only:
// NEGATIVE still DISABLES the automatic restore-test entirely (on-demand selftest still works),
// and 0 still means "use the default". It no longer sets how often a test runs — R-86 replaced
// the interval trigger with a per-archive due-check — so a positive value now seeds
// RestoreTestSettleSeconds instead (see RestoreTestSettle). Prefer the two explicit keys below.
RestoreTestCadenceSeconds int `json:"restore_test_cadence_seconds"`
// RestoreTestEvalIntervalSeconds is how often the scheduler ASKS whether any tier is due
// (R-86); 0 → default. It is not how often a test runs: a tier is tested once per archive
// generation no matter how often it is asked. This interval sets two things — the latency
// between an archive settling and its proof, and the retry rate of a tier whose restore-test
// keeps failing. See defaultRestoreTestEvalInterval for the measurement it was chosen from.
RestoreTestEvalIntervalSeconds int `json:"restore_test_eval_interval_seconds"`
// RestoreTestSettleSeconds is how long an archive must have sat on its tier before it is a
// restore-test candidate (R-86); 0 → default (24h), negative → 0 (no settle requirement).
// Restore-testing an archive a backup is still writing proves nothing about the backup that
// finished — this is the same settle discipline R-71a's gate applies to the offsite consume.
RestoreTestSettleSeconds int `json:"restore_test_settle_seconds"`
// ScratchVMIDMin/Max bound the throwaway restore-test scratch-guest VMID band. The
// restore-test refuses to run unless this is a valid band (min>0, max>=min); 9999 is
// always excluded. Defaults to 990000990009.
@@ -545,26 +559,92 @@ func (b BackupConfig) BackupTarget() string {
return defaultBackupTarget
}
// Default scratch VMID band + restore-test cadence.
// Default scratch VMID band + the two R-86 restore-test knobs.
const (
defaultScratchVMIDMin = 990000
defaultScratchVMIDMax = 990009
defaultRestoreTestCadence = 24 * time.Hour
defaultScratchVMIDMin = 990000
defaultScratchVMIDMax = 990009
// defaultRestoreTestEvalInterval is how often due-ness is ASKED. It is bounded from BOTH sides,
// and neither bound alone would have picked it:
//
// FLOOR — what one evaluation costs. MEASURED on demo-felhom, 2026-08-03 (R-86 Part 1.4), via
// --selftest=restore-test-due and by timing the underlying API call directly. One evaluation
// is one storage-content listing per tier:
//
// local dir storage (3 archives) ....... 18 ms (18.7 / 18.3 / 18.5)
// PBS tier, WAN to ep0 (2 snapshots) ... 392 ms (375 / 378 / 424)
// both tiers together .................. 430 ms
//
// So cost does NOT set this: even at one evaluation a minute the offsite leg would be ~0.7 %
// of a WAN link's time and ~9 minutes of ep0's day. Worth writing down anyway, because the
// number that would have forbidden a frequent poll is the one nobody measures.
//
// CEILING — the retry rate of a FAILING tier. Under a per-archive due-check a tier whose
// restore-test keeps failing stays due, so the evaluation interval IS its retry interval, and
// a retry is a multi-GB restore. Every few minutes would be an incident of its own; the old
// timer retried a broken tier once a day.
//
// 6h sits between them: four heavy retries a day at the very worst, latency from settle to
// proof of at most 6h against a 24h settle lag (so a daily tier is still proved daily), and no
// second rate limiter anywhere — the pacing remains one test per archive generation.
defaultRestoreTestEvalInterval = 6 * time.Hour
// defaultRestoreTestSettle is how long an archive must sit before it may be restore-tested.
// 24h is R-86's own figure ("~24 h after its own newest archive") and it is what makes the
// candidate on a daily tier YESTERDAY's archive rather than the one still being written.
defaultRestoreTestSettle = 24 * time.Hour
)
// RestoreTestCadence returns the configured restore-test interval: a positive value as-is,
// 0 → 24h default, negative → 0 (disabled).
func (b BackupConfig) RestoreTestCadence() time.Duration {
// RestoreTestEvalInterval returns how often the scheduler evaluates due-ness (R-86): a positive
// value as-is, 0 → the measured default, negative → 0 (disabled).
//
// The LEGACY `restore_test_cadence_seconds` keeps exactly one power here, the one a box may be
// relying on: a NEGATIVE value still disables the automatic restore-test outright. It no longer
// sets the interval, because the interval no longer decides that a test happens.
func (b BackupConfig) RestoreTestEvalInterval() time.Duration {
if b.RestoreTestCadenceSeconds < 0 {
return 0 // legacy DISABLE — preserved verbatim
}
switch {
case b.RestoreTestCadenceSeconds > 0:
return time.Duration(b.RestoreTestCadenceSeconds) * time.Second
case b.RestoreTestCadenceSeconds < 0:
case b.RestoreTestEvalIntervalSeconds > 0:
return time.Duration(b.RestoreTestEvalIntervalSeconds) * time.Second
case b.RestoreTestEvalIntervalSeconds < 0:
return 0 // disabled
default:
return defaultRestoreTestCadence
return defaultRestoreTestEvalInterval
}
}
// RestoreTestSettle returns how long an archive must have sat before it is a restore-test
// candidate (R-86): a positive value as-is, negative → 0 (no settle requirement), 0 → the default.
//
// WHAT HAPPENED TO THE OLD KEY. A box that set `restore_test_cadence_seconds` to a positive value
// was expressing "how long may pass between a backup and the confidence that it restores". That
// quantity survives R-86 as the SETTLE LAG, so a positive legacy value seeds this rather than being
// dropped or silently repurposed as the evaluation interval — and the daemon says so at start-up
// (see RestoreTestLegacyCadenceInUse). It is deliberately not carried into the evaluation interval:
// a box that set 72h to spare a weak endpoint would otherwise get a 72h-latency due-check, whereas
// what it actually wanted — fewer heavy restores — is what per-archive due-ness already gives it.
func (b BackupConfig) RestoreTestSettle() time.Duration {
switch {
case b.RestoreTestSettleSeconds > 0:
return time.Duration(b.RestoreTestSettleSeconds) * time.Second
case b.RestoreTestSettleSeconds < 0:
return 0 // explicitly no settle requirement
case b.RestoreTestCadenceSeconds > 0:
return time.Duration(b.RestoreTestCadenceSeconds) * time.Second // legacy seeding
default:
return defaultRestoreTestSettle
}
}
// RestoreTestLegacyCadenceInUse reports whether the deprecated key is what is deciding the settle
// lag, so the daemon can name both replacements ONCE at start-up. A config key that changed meaning
// without saying so is exactly the silent repurposing §8.3 forbids.
func (b BackupConfig) RestoreTestLegacyCadenceInUse() bool {
return b.RestoreTestCadenceSeconds > 0 && b.RestoreTestSettleSeconds == 0
}
// PBSVerifyCadence returns the verify-loop interval: positive as-is, 0 → 6h default,
// negative → 0 (disabled).
func (b BackupConfig) PBSVerifyCadence() time.Duration {
@@ -810,6 +890,8 @@ func applyEnv(cfg *Config) {
cfg.Backup.RestoreStorage = v
}
cfg.Backup.RestoreTestCadenceSeconds = envInt("FELHOM_AGENT_BACKUP_RESTORE_TEST_CADENCE_SECONDS", cfg.Backup.RestoreTestCadenceSeconds)
cfg.Backup.RestoreTestEvalIntervalSeconds = envInt("FELHOM_AGENT_BACKUP_RESTORE_TEST_EVAL_INTERVAL_SECONDS", cfg.Backup.RestoreTestEvalIntervalSeconds)
cfg.Backup.RestoreTestSettleSeconds = envInt("FELHOM_AGENT_BACKUP_RESTORE_TEST_SETTLE_SECONDS", cfg.Backup.RestoreTestSettleSeconds)
}
// envInt overlays an int env var, keeping cur (with a stderr warning) on parse