R-85 Phase 1: the restore-test spec is built PER RUN, not frozen at daemon start
SchedulerOptions.Spec was a VALUE produced by an immediately-invoked function at daemon start, so storageTier() and restoreTaskTimeout() were evaluated once and reused for every run for the process lifetime. Nothing tier-varying was expressible (the offsite tier could never be scheduled), and it was a latent staleness bug besides: a storage-type or config change did not take effect until restart. - backup.SpecBuilder: func(ctx, archive) RestoreTestSpec, called once per run. The archive is passed because the tier MUST come from it (v0.100.0 rule) — config-derived is what classified a PBS archive as 'local' and killed a 14.46 GB WAN restore at the 10-minute local bound. - A nil spec builder SKIPS loudly instead of panicking: a wiring bug must cost a restore-test, never the daemon goroutine. Red-proof observed. Full suite green (29 packages, rc=0).
This commit is contained in:
@@ -18,6 +18,20 @@ type RestoreTestRunner interface {
|
||||
// there is none yet (the tick then no-ops).
|
||||
type CandidatePicker func(ctx context.Context) (string, error)
|
||||
|
||||
// SpecBuilder yields the RestoreTestSpec for ONE run, given the archive that was picked.
|
||||
//
|
||||
// R-85 (1.1): this REPLACES a frozen spec value. It used to be built by an immediately-invoked
|
||||
// function at daemon start, so `storageTier()` and `restoreTaskTimeout()` were evaluated ONCE and
|
||||
// the resulting value reused for every run for the lifetime of the process. Two consequences:
|
||||
// - nothing tier-varying was expressible at all (the offsite tier could never be scheduled), and
|
||||
// - it was a latent staleness bug in its own right — a storage-type or config change did not take
|
||||
// effect until the daemon restarted.
|
||||
//
|
||||
// The archive is passed in because the tier MUST be derived from it (the v0.100.0 rule), never from
|
||||
// the configured target: deriving it from config is what produced the 600 s false failure when a
|
||||
// PBS archive was classified "local" and got the 10-minute local wait.
|
||||
type SpecBuilder func(ctx context.Context, archive string) reconcile.RestoreTestSpec
|
||||
|
||||
// Scheduler runs the self-restore-test on an agent-internal cadence. It is the fourth daemon
|
||||
// goroutine; it does real restore→boot→destroy, so it only runs when the cadence is enabled
|
||||
// AND a valid scratch band is configured (validated by the caller before construction).
|
||||
@@ -25,7 +39,7 @@ type Scheduler struct {
|
||||
runner RestoreTestRunner
|
||||
pick CandidatePicker
|
||||
store *Store
|
||||
spec reconcile.RestoreTestSpec // archive is filled per-tick
|
||||
spec SpecBuilder // R-85: evaluated PER RUN, never frozen at construction
|
||||
cadence time.Duration
|
||||
logger *slog.Logger
|
||||
now func() time.Time
|
||||
@@ -33,11 +47,13 @@ type Scheduler struct {
|
||||
|
||||
// SchedulerOptions configures a Scheduler.
|
||||
type SchedulerOptions struct {
|
||||
Runner RestoreTestRunner
|
||||
Pick CandidatePicker
|
||||
Store *Store
|
||||
Spec reconcile.RestoreTestSpec // RestoreStorage, ScratchMin/Max, SourceTier, BootTimeout
|
||||
Cadence time.Duration // 0 → disabled
|
||||
Runner RestoreTestRunner
|
||||
Pick CandidatePicker
|
||||
Store *Store
|
||||
// Spec builds the run's spec (RestoreStorage, ScratchMin/Max, SourceTier, timeouts) from the
|
||||
// picked archive. Called ONCE PER RUN — see SpecBuilder for why it is not a value.
|
||||
Spec SpecBuilder
|
||||
Cadence time.Duration // 0 → disabled
|
||||
Logger *slog.Logger
|
||||
}
|
||||
|
||||
@@ -63,7 +79,7 @@ func NewScheduler(opts SchedulerOptions) *Scheduler {
|
||||
// is heavy; the first runs one interval in) — on-demand runs use the selftest harness.
|
||||
// Returns nil on ctx cancellation.
|
||||
func (s *Scheduler) Run(ctx context.Context) error {
|
||||
if s.cadence <= 0 || s.runner == nil || s.pick == nil {
|
||||
if s.cadence <= 0 || s.runner == nil || s.pick == nil || s.spec == nil {
|
||||
s.logger.Info("backup: restore-test cadence disabled")
|
||||
<-ctx.Done()
|
||||
return nil
|
||||
@@ -85,6 +101,13 @@ func (s *Scheduler) Run(ctx context.Context) error {
|
||||
// tick runs one scheduled restore-test: pick a backup → run → record. No-ops cleanly when
|
||||
// no backup exists yet. Deterministic given s.now — tests call it directly.
|
||||
func (s *Scheduler) tick(ctx context.Context) {
|
||||
if s.spec == nil {
|
||||
// Defensive: Run() already refuses to start without a SpecBuilder, but tick is also
|
||||
// reachable directly. Skipping loudly beats panicking the daemon goroutine — a missing
|
||||
// spec must cost a restore-test, never the agent.
|
||||
s.logger.Error("backup: restore-test has no spec builder — skipping (this is a wiring bug)")
|
||||
return
|
||||
}
|
||||
archive, err := s.pick(ctx)
|
||||
if err != nil {
|
||||
s.logger.Warn("backup: restore-test could not pick a candidate; skipping", "err", err)
|
||||
@@ -94,7 +117,8 @@ func (s *Scheduler) tick(ctx context.Context) {
|
||||
s.logger.Info("backup: restore-test skipped; no backup available yet")
|
||||
return
|
||||
}
|
||||
spec := s.spec
|
||||
// R-85: build the spec for THIS run, from THIS archive. Never a frozen value.
|
||||
spec := s.spec(ctx, archive)
|
||||
spec.Archive = archive
|
||||
res := s.runner.RunRestoreTest(ctx, spec)
|
||||
if res.Skipped {
|
||||
|
||||
Reference in New Issue
Block a user