R-86: restore-test follows the backup, not the clock (v0.121.0)
gates / gates (push) Failing after 7s

The ticker survives as the EVALUATION interval only. A tier is DUE when its
newest archive that has settled for `settle` (default 24h) has not been proven:
daily tier -> proved daily on yesterday's archive, weekly tier -> weekly on its
own, newborn -> UNKNOWN.

The trap avoided: the literal reading ("newest archive is >= 24h old") is NEVER
true on a daily tier, so it silently switches restore-testing off where it
matters most. Red-proved at 0 runs over 5 simulated days.

- state records WHICH archive was proven; legacy files keep their time and yield
  no proven archive (each tier due once after the upgrade, deliberately)
- two knobs replace one: restore_test_eval_interval_seconds (6h, measured) and
  restore_test_settle_seconds (24h). The old cadence key keeps its DISABLE
  meaning verbatim and now seeds the settle lag, with a start-up WARN.
- due-check runs BEFORE the heavy-op gate (a frequent poll must not make a
  starting backup record a failure, F-A1)
- candidate picker skips implausible archives (a phantom would be due forever)
- new read-only --selftest=restore-test-due prints the verdict + its cost
This commit is contained in:
2026-08-03 14:54:57 +02:00
parent 1b14cfd0b4
commit 4618169036
13 changed files with 1273 additions and 123 deletions
+95 -13
View File
@@ -333,9 +333,23 @@ type BackupConfig struct {
LocalBackupTarget string `json:"local_backup_target"`
// RestoreStorage is where a restore-test's restored rootfs lands, e.g. "local-lvm".
RestoreStorage string `json:"restore_storage"`
// RestoreTestCadenceSeconds is the self-restore-test interval; 0 → default (24h).
// Set negative to DISABLE the automatic cadence (on-demand selftest still works).
// RestoreTestCadenceSeconds is the LEGACY restore-test knob, retained for one meaning only:
// NEGATIVE still DISABLES the automatic restore-test entirely (on-demand selftest still works),
// and 0 still means "use the default". It no longer sets how often a test runs — R-86 replaced
// the interval trigger with a per-archive due-check — so a positive value now seeds
// RestoreTestSettleSeconds instead (see RestoreTestSettle). Prefer the two explicit keys below.
RestoreTestCadenceSeconds int `json:"restore_test_cadence_seconds"`
// RestoreTestEvalIntervalSeconds is how often the scheduler ASKS whether any tier is due
// (R-86); 0 → default. It is not how often a test runs: a tier is tested once per archive
// generation no matter how often it is asked. This interval sets two things — the latency
// between an archive settling and its proof, and the retry rate of a tier whose restore-test
// keeps failing. See defaultRestoreTestEvalInterval for the measurement it was chosen from.
RestoreTestEvalIntervalSeconds int `json:"restore_test_eval_interval_seconds"`
// RestoreTestSettleSeconds is how long an archive must have sat on its tier before it is a
// restore-test candidate (R-86); 0 → default (24h), negative → 0 (no settle requirement).
// Restore-testing an archive a backup is still writing proves nothing about the backup that
// finished — this is the same settle discipline R-71a's gate applies to the offsite consume.
RestoreTestSettleSeconds int `json:"restore_test_settle_seconds"`
// ScratchVMIDMin/Max bound the throwaway restore-test scratch-guest VMID band. The
// restore-test refuses to run unless this is a valid band (min>0, max>=min); 9999 is
// always excluded. Defaults to 990000990009.
@@ -545,26 +559,92 @@ func (b BackupConfig) BackupTarget() string {
return defaultBackupTarget
}
// Default scratch VMID band + restore-test cadence.
// Default scratch VMID band + the two R-86 restore-test knobs.
const (
defaultScratchVMIDMin = 990000
defaultScratchVMIDMax = 990009
defaultRestoreTestCadence = 24 * time.Hour
defaultScratchVMIDMin = 990000
defaultScratchVMIDMax = 990009
// defaultRestoreTestEvalInterval is how often due-ness is ASKED. It is bounded from BOTH sides,
// and neither bound alone would have picked it:
//
// FLOOR — what one evaluation costs. MEASURED on demo-felhom, 2026-08-03 (R-86 Part 1.4), via
// --selftest=restore-test-due and by timing the underlying API call directly. One evaluation
// is one storage-content listing per tier:
//
// local dir storage (3 archives) ....... 18 ms (18.7 / 18.3 / 18.5)
// PBS tier, WAN to ep0 (2 snapshots) ... 392 ms (375 / 378 / 424)
// both tiers together .................. 430 ms
//
// So cost does NOT set this: even at one evaluation a minute the offsite leg would be ~0.7 %
// of a WAN link's time and ~9 minutes of ep0's day. Worth writing down anyway, because the
// number that would have forbidden a frequent poll is the one nobody measures.
//
// CEILING — the retry rate of a FAILING tier. Under a per-archive due-check a tier whose
// restore-test keeps failing stays due, so the evaluation interval IS its retry interval, and
// a retry is a multi-GB restore. Every few minutes would be an incident of its own; the old
// timer retried a broken tier once a day.
//
// 6h sits between them: four heavy retries a day at the very worst, latency from settle to
// proof of at most 6h against a 24h settle lag (so a daily tier is still proved daily), and no
// second rate limiter anywhere — the pacing remains one test per archive generation.
defaultRestoreTestEvalInterval = 6 * time.Hour
// defaultRestoreTestSettle is how long an archive must sit before it may be restore-tested.
// 24h is R-86's own figure ("~24 h after its own newest archive") and it is what makes the
// candidate on a daily tier YESTERDAY's archive rather than the one still being written.
defaultRestoreTestSettle = 24 * time.Hour
)
// RestoreTestCadence returns the configured restore-test interval: a positive value as-is,
// 0 → 24h default, negative → 0 (disabled).
func (b BackupConfig) RestoreTestCadence() time.Duration {
// RestoreTestEvalInterval returns how often the scheduler evaluates due-ness (R-86): a positive
// value as-is, 0 → the measured default, negative → 0 (disabled).
//
// The LEGACY `restore_test_cadence_seconds` keeps exactly one power here, the one a box may be
// relying on: a NEGATIVE value still disables the automatic restore-test outright. It no longer
// sets the interval, because the interval no longer decides that a test happens.
func (b BackupConfig) RestoreTestEvalInterval() time.Duration {
if b.RestoreTestCadenceSeconds < 0 {
return 0 // legacy DISABLE — preserved verbatim
}
switch {
case b.RestoreTestCadenceSeconds > 0:
return time.Duration(b.RestoreTestCadenceSeconds) * time.Second
case b.RestoreTestCadenceSeconds < 0:
case b.RestoreTestEvalIntervalSeconds > 0:
return time.Duration(b.RestoreTestEvalIntervalSeconds) * time.Second
case b.RestoreTestEvalIntervalSeconds < 0:
return 0 // disabled
default:
return defaultRestoreTestCadence
return defaultRestoreTestEvalInterval
}
}
// RestoreTestSettle returns how long an archive must have sat before it is a restore-test
// candidate (R-86): a positive value as-is, negative → 0 (no settle requirement), 0 → the default.
//
// WHAT HAPPENED TO THE OLD KEY. A box that set `restore_test_cadence_seconds` to a positive value
// was expressing "how long may pass between a backup and the confidence that it restores". That
// quantity survives R-86 as the SETTLE LAG, so a positive legacy value seeds this rather than being
// dropped or silently repurposed as the evaluation interval — and the daemon says so at start-up
// (see RestoreTestLegacyCadenceInUse). It is deliberately not carried into the evaluation interval:
// a box that set 72h to spare a weak endpoint would otherwise get a 72h-latency due-check, whereas
// what it actually wanted — fewer heavy restores — is what per-archive due-ness already gives it.
func (b BackupConfig) RestoreTestSettle() time.Duration {
switch {
case b.RestoreTestSettleSeconds > 0:
return time.Duration(b.RestoreTestSettleSeconds) * time.Second
case b.RestoreTestSettleSeconds < 0:
return 0 // explicitly no settle requirement
case b.RestoreTestCadenceSeconds > 0:
return time.Duration(b.RestoreTestCadenceSeconds) * time.Second // legacy seeding
default:
return defaultRestoreTestSettle
}
}
// RestoreTestLegacyCadenceInUse reports whether the deprecated key is what is deciding the settle
// lag, so the daemon can name both replacements ONCE at start-up. A config key that changed meaning
// without saying so is exactly the silent repurposing §8.3 forbids.
func (b BackupConfig) RestoreTestLegacyCadenceInUse() bool {
return b.RestoreTestCadenceSeconds > 0 && b.RestoreTestSettleSeconds == 0
}
// PBSVerifyCadence returns the verify-loop interval: positive as-is, 0 → 6h default,
// negative → 0 (disabled).
func (b BackupConfig) PBSVerifyCadence() time.Duration {
@@ -810,6 +890,8 @@ func applyEnv(cfg *Config) {
cfg.Backup.RestoreStorage = v
}
cfg.Backup.RestoreTestCadenceSeconds = envInt("FELHOM_AGENT_BACKUP_RESTORE_TEST_CADENCE_SECONDS", cfg.Backup.RestoreTestCadenceSeconds)
cfg.Backup.RestoreTestEvalIntervalSeconds = envInt("FELHOM_AGENT_BACKUP_RESTORE_TEST_EVAL_INTERVAL_SECONDS", cfg.Backup.RestoreTestEvalIntervalSeconds)
cfg.Backup.RestoreTestSettleSeconds = envInt("FELHOM_AGENT_BACKUP_RESTORE_TEST_SETTLE_SECONDS", cfg.Backup.RestoreTestSettleSeconds)
}
// envInt overlays an int env var, keeping cur (with a stderr warning) on parse