v0.98.0 — R-82 Slice A fix: per-tier vzdump wait bound (the 30-minute false failure)
Found by live validation on demo-felhom, not by review. The first real PBS-targeted backup ran past the runner's hard-coded 30-minute WaitTask bound. The agent stopped waiting and recorded success=false WHILE THE VZDUMP KEPT RUNNING (still running 72 min later, 2.4 GB uploaded). Consequences: the tier stays permanently due, the next attempt collides with the guest lock the live vzdump holds, and the hub sees a DR tier that never succeeds — R-82's 'applied and empty' fault re-created by a timeout. Measured: ~33 MB/min over wg to Hetzner, so a first FULL ~10 GB snapshot projects to ~5h. - BackupTargetConfig.WaitTimeoutSeconds: per-tier bound. Primary 30m UNCHANGED (a local vzdump hanging 30m IS a real fault); additional tier 6h, sized from the measurement. - backup.NewBackupRunnerWithWait: per-instance (per-tier) bound. NewBackupRunner keeps its signature, so restore-test/selftest are untouched. - localapi.BackupTier.WaitTimeout: the fire-and-forget context is sized from the tier, not a fixed 2h. BOTH bounds had to move — a 6h runner bound under a 2h outer context reproduces the same false failure four hours later. Same direction as restore_test_pbs_restore_timeout_seconds: when in doubt wait LONGER. A slow backup is a slow backup; a false timeout is a corrupt status plus lock contention. Red-proof observed and restored; full suite green.
This commit is contained in:
@@ -148,3 +148,35 @@ func TestBackupTiers_PrimaryRetentionClampUnchanged(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// R-82 live-failure regression (2026-07-26): the runner hard-coded a 30-minute vzdump wait, which
|
||||
// is right for a local vzdump and wrong for an offsite PBS upload. The first full ~10 GB PBS
|
||||
// snapshot on demo-felhom ran past 30 min; the agent gave up waiting and recorded success=false
|
||||
// WHILE THE BACKUP WAS STILL RUNNING — a false failure that leaves the tier permanently "due" and
|
||||
// makes the next attempt collide with the guest lock vzdump still holds.
|
||||
func TestBackupTiers_WaitTimeoutIsPerTier(t *testing.T) {
|
||||
b := BackupConfig{
|
||||
LocalBackupTarget: "local",
|
||||
ExtraTargets: []BackupTargetConfig{{TargetID: "felhom-pbs", CadenceSeconds: 604800}},
|
||||
}
|
||||
tiers, _ := b.BackupTiers()
|
||||
if len(tiers) != 2 {
|
||||
t.Fatalf("got %+v", tiers)
|
||||
}
|
||||
if tiers[0].WaitTimeout != 30*time.Minute {
|
||||
t.Fatalf("the PRIMARY must keep the historical 30m wait (unchanged behaviour); got %s", tiers[0].WaitTimeout)
|
||||
}
|
||||
if tiers[1].WaitTimeout != 6*time.Hour {
|
||||
t.Fatalf("an offsite tier must default to a GENEROUS wait — a false timeout is worse than a slow pass; got %s", tiers[1].WaitTimeout)
|
||||
}
|
||||
// And it must be overridable per tier.
|
||||
b.ExtraTargets[0].WaitTimeoutSeconds = 3600
|
||||
tiers, _ = b.BackupTiers()
|
||||
if tiers[1].WaitTimeout != time.Hour {
|
||||
t.Fatalf("wait_timeout_seconds must override; got %s", tiers[1].WaitTimeout)
|
||||
}
|
||||
// The two tiers must NOT share one bound.
|
||||
if tiers[0].WaitTimeout == tiers[1].WaitTimeout {
|
||||
t.Fatalf("wait bounds are shared between tiers — the whole point is that they differ: %+v", tiers)
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user