v0.98.0 — R-82 Slice A fix: per-tier vzdump wait bound (the 30-minute false failure)

Found by live validation on demo-felhom, not by review.

The first real PBS-targeted backup ran past the runner's hard-coded 30-minute
WaitTask bound. The agent stopped waiting and recorded success=false WHILE THE
VZDUMP KEPT RUNNING (still running 72 min later, 2.4 GB uploaded). Consequences:
the tier stays permanently due, the next attempt collides with the guest lock
the live vzdump holds, and the hub sees a DR tier that never succeeds — R-82's
'applied and empty' fault re-created by a timeout.

Measured: ~33 MB/min over wg to Hetzner, so a first FULL ~10 GB snapshot
projects to ~5h.

- BackupTargetConfig.WaitTimeoutSeconds: per-tier bound. Primary 30m UNCHANGED
  (a local vzdump hanging 30m IS a real fault); additional tier 6h, sized from
  the measurement.
- backup.NewBackupRunnerWithWait: per-instance (per-tier) bound.
  NewBackupRunner keeps its signature, so restore-test/selftest are untouched.
- localapi.BackupTier.WaitTimeout: the fire-and-forget context is sized from the
  tier, not a fixed 2h. BOTH bounds had to move — a 6h runner bound under a 2h
  outer context reproduces the same false failure four hours later.

Same direction as restore_test_pbs_restore_timeout_seconds: when in doubt wait
LONGER. A slow backup is a slow backup; a false timeout is a corrupt status
plus lock contention.

Red-proof observed and restored; full suite green.
This commit is contained in:
Claude Code
2026-07-26 14:53:24 +02:00
parent 68bcebe493
commit a667c269c7
7 changed files with 155 additions and 21 deletions
+32
View File
@@ -148,3 +148,35 @@ func TestBackupTiers_PrimaryRetentionClampUnchanged(t *testing.T) {
}
}
}
// R-82 live-failure regression (2026-07-26): the runner hard-coded a 30-minute vzdump wait, which
// is right for a local vzdump and wrong for an offsite PBS upload. The first full ~10 GB PBS
// snapshot on demo-felhom ran past 30 min; the agent gave up waiting and recorded success=false
// WHILE THE BACKUP WAS STILL RUNNING — a false failure that leaves the tier permanently "due" and
// makes the next attempt collide with the guest lock vzdump still holds.
func TestBackupTiers_WaitTimeoutIsPerTier(t *testing.T) {
b := BackupConfig{
LocalBackupTarget: "local",
ExtraTargets: []BackupTargetConfig{{TargetID: "felhom-pbs", CadenceSeconds: 604800}},
}
tiers, _ := b.BackupTiers()
if len(tiers) != 2 {
t.Fatalf("got %+v", tiers)
}
if tiers[0].WaitTimeout != 30*time.Minute {
t.Fatalf("the PRIMARY must keep the historical 30m wait (unchanged behaviour); got %s", tiers[0].WaitTimeout)
}
if tiers[1].WaitTimeout != 6*time.Hour {
t.Fatalf("an offsite tier must default to a GENEROUS wait — a false timeout is worse than a slow pass; got %s", tiers[1].WaitTimeout)
}
// And it must be overridable per tier.
b.ExtraTargets[0].WaitTimeoutSeconds = 3600
tiers, _ = b.BackupTiers()
if tiers[1].WaitTimeout != time.Hour {
t.Fatalf("wait_timeout_seconds must override; got %s", tiers[1].WaitTimeout)
}
// The two tiers must NOT share one bound.
if tiers[0].WaitTimeout == tiers[1].WaitTimeout {
t.Fatalf("wait bounds are shared between tiers — the whole point is that they differ: %+v", tiers)
}
}