R-85 Phase 1: the restore-test spec is built PER RUN, not frozen at daemon start

SchedulerOptions.Spec was a VALUE produced by an immediately-invoked function
at daemon start, so storageTier() and restoreTaskTimeout() were evaluated once
and reused for every run for the process lifetime. Nothing tier-varying was
expressible (the offsite tier could never be scheduled), and it was a latent
staleness bug besides: a storage-type or config change did not take effect
until restart.

- backup.SpecBuilder: func(ctx, archive) RestoreTestSpec, called once per run.
  The archive is passed because the tier MUST come from it (v0.100.0 rule) —
  config-derived is what classified a PBS archive as 'local' and killed a
  14.46 GB WAN restore at the 10-minute local bound.
- A nil spec builder SKIPS loudly instead of panicking: a wiring bug must cost a
  restore-test, never the daemon goroutine.

Red-proof observed. Full suite green (29 packages, rc=0).
This commit is contained in:
Claude Code
2026-07-26 20:47:28 +02:00
parent edde8a01ca
commit 765d8b3168
5 changed files with 204 additions and 14 deletions
+32 -8
View File
@@ -18,6 +18,20 @@ type RestoreTestRunner interface {
// there is none yet (the tick then no-ops).
type CandidatePicker func(ctx context.Context) (string, error)
// SpecBuilder yields the RestoreTestSpec for ONE run, given the archive that was picked.
//
// R-85 (1.1): this REPLACES a frozen spec value. It used to be built by an immediately-invoked
// function at daemon start, so `storageTier()` and `restoreTaskTimeout()` were evaluated ONCE and
// the resulting value reused for every run for the lifetime of the process. Two consequences:
// - nothing tier-varying was expressible at all (the offsite tier could never be scheduled), and
// - it was a latent staleness bug in its own right — a storage-type or config change did not take
// effect until the daemon restarted.
//
// The archive is passed in because the tier MUST be derived from it (the v0.100.0 rule), never from
// the configured target: deriving it from config is what produced the 600 s false failure when a
// PBS archive was classified "local" and got the 10-minute local wait.
type SpecBuilder func(ctx context.Context, archive string) reconcile.RestoreTestSpec
// Scheduler runs the self-restore-test on an agent-internal cadence. It is the fourth daemon
// goroutine; it does real restore→boot→destroy, so it only runs when the cadence is enabled
// AND a valid scratch band is configured (validated by the caller before construction).
@@ -25,7 +39,7 @@ type Scheduler struct {
runner RestoreTestRunner
pick CandidatePicker
store *Store
spec reconcile.RestoreTestSpec // archive is filled per-tick
spec SpecBuilder // R-85: evaluated PER RUN, never frozen at construction
cadence time.Duration
logger *slog.Logger
now func() time.Time
@@ -33,11 +47,13 @@ type Scheduler struct {
// SchedulerOptions configures a Scheduler.
type SchedulerOptions struct {
Runner RestoreTestRunner
Pick CandidatePicker
Store *Store
Spec reconcile.RestoreTestSpec // RestoreStorage, ScratchMin/Max, SourceTier, BootTimeout
Cadence time.Duration // 0 → disabled
Runner RestoreTestRunner
Pick CandidatePicker
Store *Store
// Spec builds the run's spec (RestoreStorage, ScratchMin/Max, SourceTier, timeouts) from the
// picked archive. Called ONCE PER RUN — see SpecBuilder for why it is not a value.
Spec SpecBuilder
Cadence time.Duration // 0 → disabled
Logger *slog.Logger
}
@@ -63,7 +79,7 @@ func NewScheduler(opts SchedulerOptions) *Scheduler {
// is heavy; the first runs one interval in) — on-demand runs use the selftest harness.
// Returns nil on ctx cancellation.
func (s *Scheduler) Run(ctx context.Context) error {
if s.cadence <= 0 || s.runner == nil || s.pick == nil {
if s.cadence <= 0 || s.runner == nil || s.pick == nil || s.spec == nil {
s.logger.Info("backup: restore-test cadence disabled")
<-ctx.Done()
return nil
@@ -85,6 +101,13 @@ func (s *Scheduler) Run(ctx context.Context) error {
// tick runs one scheduled restore-test: pick a backup → run → record. No-ops cleanly when
// no backup exists yet. Deterministic given s.now — tests call it directly.
func (s *Scheduler) tick(ctx context.Context) {
if s.spec == nil {
// Defensive: Run() already refuses to start without a SpecBuilder, but tick is also
// reachable directly. Skipping loudly beats panicking the daemon goroutine — a missing
// spec must cost a restore-test, never the agent.
s.logger.Error("backup: restore-test has no spec builder — skipping (this is a wiring bug)")
return
}
archive, err := s.pick(ctx)
if err != nil {
s.logger.Warn("backup: restore-test could not pick a candidate; skipping", "err", err)
@@ -94,7 +117,8 @@ func (s *Scheduler) tick(ctx context.Context) {
s.logger.Info("backup: restore-test skipped; no backup available yet")
return
}
spec := s.spec
// R-85: build the spec for THIS run, from THIS archive. Never a frozen value.
spec := s.spec(ctx, archive)
spec.Archive = archive
res := s.runner.RunRestoreTest(ctx, spec)
if res.Skipped {