R-86: restore-test follows the backup, not the clock (v0.121.0)
gates / gates (push) Failing after 7s
gates / gates (push) Failing after 7s
The ticker survives as the EVALUATION interval only. A tier is DUE when its
newest archive that has settled for `settle` (default 24h) has not been proven:
daily tier -> proved daily on yesterday's archive, weekly tier -> weekly on its
own, newborn -> UNKNOWN.
The trap avoided: the literal reading ("newest archive is >= 24h old") is NEVER
true on a daily tier, so it silently switches restore-testing off where it
matters most. Red-proved at 0 runs over 5 simulated days.
- state records WHICH archive was proven; legacy files keep their time and yield
no proven archive (each tier due once after the upgrade, deliberately)
- two knobs replace one: restore_test_eval_interval_seconds (6h, measured) and
restore_test_settle_seconds (24h). The old cadence key keeps its DISABLE
meaning verbatim and now seeds the settle lag, with a start-up WARN.
- due-check runs BEFORE the heavy-op gate (a frequent poll must not make a
starting backup record a failure, F-A1)
- candidate picker skips implausible archives (a phantom would be due forever)
- new read-only --selftest=restore-test-due prints the verdict + its cost
This commit is contained in:
+95
-13
@@ -333,9 +333,23 @@ type BackupConfig struct {
|
||||
LocalBackupTarget string `json:"local_backup_target"`
|
||||
// RestoreStorage is where a restore-test's restored rootfs lands, e.g. "local-lvm".
|
||||
RestoreStorage string `json:"restore_storage"`
|
||||
// RestoreTestCadenceSeconds is the self-restore-test interval; 0 → default (24h).
|
||||
// Set negative to DISABLE the automatic cadence (on-demand selftest still works).
|
||||
// RestoreTestCadenceSeconds is the LEGACY restore-test knob, retained for one meaning only:
|
||||
// NEGATIVE still DISABLES the automatic restore-test entirely (on-demand selftest still works),
|
||||
// and 0 still means "use the default". It no longer sets how often a test runs — R-86 replaced
|
||||
// the interval trigger with a per-archive due-check — so a positive value now seeds
|
||||
// RestoreTestSettleSeconds instead (see RestoreTestSettle). Prefer the two explicit keys below.
|
||||
RestoreTestCadenceSeconds int `json:"restore_test_cadence_seconds"`
|
||||
// RestoreTestEvalIntervalSeconds is how often the scheduler ASKS whether any tier is due
|
||||
// (R-86); 0 → default. It is not how often a test runs: a tier is tested once per archive
|
||||
// generation no matter how often it is asked. This interval sets two things — the latency
|
||||
// between an archive settling and its proof, and the retry rate of a tier whose restore-test
|
||||
// keeps failing. See defaultRestoreTestEvalInterval for the measurement it was chosen from.
|
||||
RestoreTestEvalIntervalSeconds int `json:"restore_test_eval_interval_seconds"`
|
||||
// RestoreTestSettleSeconds is how long an archive must have sat on its tier before it is a
|
||||
// restore-test candidate (R-86); 0 → default (24h), negative → 0 (no settle requirement).
|
||||
// Restore-testing an archive a backup is still writing proves nothing about the backup that
|
||||
// finished — this is the same settle discipline R-71a's gate applies to the offsite consume.
|
||||
RestoreTestSettleSeconds int `json:"restore_test_settle_seconds"`
|
||||
// ScratchVMIDMin/Max bound the throwaway restore-test scratch-guest VMID band. The
|
||||
// restore-test refuses to run unless this is a valid band (min>0, max>=min); 9999 is
|
||||
// always excluded. Defaults to 990000–990009.
|
||||
@@ -545,26 +559,92 @@ func (b BackupConfig) BackupTarget() string {
|
||||
return defaultBackupTarget
|
||||
}
|
||||
|
||||
// Default scratch VMID band + restore-test cadence.
|
||||
// Default scratch VMID band + the two R-86 restore-test knobs.
|
||||
const (
|
||||
defaultScratchVMIDMin = 990000
|
||||
defaultScratchVMIDMax = 990009
|
||||
defaultRestoreTestCadence = 24 * time.Hour
|
||||
defaultScratchVMIDMin = 990000
|
||||
defaultScratchVMIDMax = 990009
|
||||
|
||||
// defaultRestoreTestEvalInterval is how often due-ness is ASKED. It is bounded from BOTH sides,
|
||||
// and neither bound alone would have picked it:
|
||||
//
|
||||
// FLOOR — what one evaluation costs. MEASURED on demo-felhom, 2026-08-03 (R-86 Part 1.4), via
|
||||
// --selftest=restore-test-due and by timing the underlying API call directly. One evaluation
|
||||
// is one storage-content listing per tier:
|
||||
//
|
||||
// local dir storage (3 archives) ....... 18 ms (18.7 / 18.3 / 18.5)
|
||||
// PBS tier, WAN to ep0 (2 snapshots) ... 392 ms (375 / 378 / 424)
|
||||
// both tiers together .................. 430 ms
|
||||
//
|
||||
// So cost does NOT set this: even at one evaluation a minute the offsite leg would be ~0.7 %
|
||||
// of a WAN link's time and ~9 minutes of ep0's day. Worth writing down anyway, because the
|
||||
// number that would have forbidden a frequent poll is the one nobody measures.
|
||||
//
|
||||
// CEILING — the retry rate of a FAILING tier. Under a per-archive due-check a tier whose
|
||||
// restore-test keeps failing stays due, so the evaluation interval IS its retry interval, and
|
||||
// a retry is a multi-GB restore. Every few minutes would be an incident of its own; the old
|
||||
// timer retried a broken tier once a day.
|
||||
//
|
||||
// 6h sits between them: four heavy retries a day at the very worst, latency from settle to
|
||||
// proof of at most 6h against a 24h settle lag (so a daily tier is still proved daily), and no
|
||||
// second rate limiter anywhere — the pacing remains one test per archive generation.
|
||||
defaultRestoreTestEvalInterval = 6 * time.Hour
|
||||
|
||||
// defaultRestoreTestSettle is how long an archive must sit before it may be restore-tested.
|
||||
// 24h is R-86's own figure ("~24 h after its own newest archive") and it is what makes the
|
||||
// candidate on a daily tier YESTERDAY's archive rather than the one still being written.
|
||||
defaultRestoreTestSettle = 24 * time.Hour
|
||||
)
|
||||
|
||||
// RestoreTestCadence returns the configured restore-test interval: a positive value as-is,
|
||||
// 0 → 24h default, negative → 0 (disabled).
|
||||
func (b BackupConfig) RestoreTestCadence() time.Duration {
|
||||
// RestoreTestEvalInterval returns how often the scheduler evaluates due-ness (R-86): a positive
|
||||
// value as-is, 0 → the measured default, negative → 0 (disabled).
|
||||
//
|
||||
// The LEGACY `restore_test_cadence_seconds` keeps exactly one power here, the one a box may be
|
||||
// relying on: a NEGATIVE value still disables the automatic restore-test outright. It no longer
|
||||
// sets the interval, because the interval no longer decides that a test happens.
|
||||
func (b BackupConfig) RestoreTestEvalInterval() time.Duration {
|
||||
if b.RestoreTestCadenceSeconds < 0 {
|
||||
return 0 // legacy DISABLE — preserved verbatim
|
||||
}
|
||||
switch {
|
||||
case b.RestoreTestCadenceSeconds > 0:
|
||||
return time.Duration(b.RestoreTestCadenceSeconds) * time.Second
|
||||
case b.RestoreTestCadenceSeconds < 0:
|
||||
case b.RestoreTestEvalIntervalSeconds > 0:
|
||||
return time.Duration(b.RestoreTestEvalIntervalSeconds) * time.Second
|
||||
case b.RestoreTestEvalIntervalSeconds < 0:
|
||||
return 0 // disabled
|
||||
default:
|
||||
return defaultRestoreTestCadence
|
||||
return defaultRestoreTestEvalInterval
|
||||
}
|
||||
}
|
||||
|
||||
// RestoreTestSettle returns how long an archive must have sat before it is a restore-test
|
||||
// candidate (R-86): a positive value as-is, negative → 0 (no settle requirement), 0 → the default.
|
||||
//
|
||||
// WHAT HAPPENED TO THE OLD KEY. A box that set `restore_test_cadence_seconds` to a positive value
|
||||
// was expressing "how long may pass between a backup and the confidence that it restores". That
|
||||
// quantity survives R-86 as the SETTLE LAG, so a positive legacy value seeds this rather than being
|
||||
// dropped or silently repurposed as the evaluation interval — and the daemon says so at start-up
|
||||
// (see RestoreTestLegacyCadenceInUse). It is deliberately not carried into the evaluation interval:
|
||||
// a box that set 72h to spare a weak endpoint would otherwise get a 72h-latency due-check, whereas
|
||||
// what it actually wanted — fewer heavy restores — is what per-archive due-ness already gives it.
|
||||
func (b BackupConfig) RestoreTestSettle() time.Duration {
|
||||
switch {
|
||||
case b.RestoreTestSettleSeconds > 0:
|
||||
return time.Duration(b.RestoreTestSettleSeconds) * time.Second
|
||||
case b.RestoreTestSettleSeconds < 0:
|
||||
return 0 // explicitly no settle requirement
|
||||
case b.RestoreTestCadenceSeconds > 0:
|
||||
return time.Duration(b.RestoreTestCadenceSeconds) * time.Second // legacy seeding
|
||||
default:
|
||||
return defaultRestoreTestSettle
|
||||
}
|
||||
}
|
||||
|
||||
// RestoreTestLegacyCadenceInUse reports whether the deprecated key is what is deciding the settle
|
||||
// lag, so the daemon can name both replacements ONCE at start-up. A config key that changed meaning
|
||||
// without saying so is exactly the silent repurposing §8.3 forbids.
|
||||
func (b BackupConfig) RestoreTestLegacyCadenceInUse() bool {
|
||||
return b.RestoreTestCadenceSeconds > 0 && b.RestoreTestSettleSeconds == 0
|
||||
}
|
||||
|
||||
// PBSVerifyCadence returns the verify-loop interval: positive as-is, 0 → 6h default,
|
||||
// negative → 0 (disabled).
|
||||
func (b BackupConfig) PBSVerifyCadence() time.Duration {
|
||||
@@ -810,6 +890,8 @@ func applyEnv(cfg *Config) {
|
||||
cfg.Backup.RestoreStorage = v
|
||||
}
|
||||
cfg.Backup.RestoreTestCadenceSeconds = envInt("FELHOM_AGENT_BACKUP_RESTORE_TEST_CADENCE_SECONDS", cfg.Backup.RestoreTestCadenceSeconds)
|
||||
cfg.Backup.RestoreTestEvalIntervalSeconds = envInt("FELHOM_AGENT_BACKUP_RESTORE_TEST_EVAL_INTERVAL_SECONDS", cfg.Backup.RestoreTestEvalIntervalSeconds)
|
||||
cfg.Backup.RestoreTestSettleSeconds = envInt("FELHOM_AGENT_BACKUP_RESTORE_TEST_SETTLE_SECONDS", cfg.Backup.RestoreTestSettleSeconds)
|
||||
}
|
||||
|
||||
// envInt overlays an int env var, keeping cur (with a stderr warning) on parse
|
||||
|
||||
Reference in New Issue
Block a user