v0.98.0 — R-82 Slice A fix: per-tier vzdump wait bound (the 30-minute false failure)

Found by live validation on demo-felhom, not by review.

The first real PBS-targeted backup ran past the runner's hard-coded 30-minute
WaitTask bound. The agent stopped waiting and recorded success=false WHILE THE
VZDUMP KEPT RUNNING (still running 72 min later, 2.4 GB uploaded). Consequences:
the tier stays permanently due, the next attempt collides with the guest lock
the live vzdump holds, and the hub sees a DR tier that never succeeds — R-82's
'applied and empty' fault re-created by a timeout.

Measured: ~33 MB/min over wg to Hetzner, so a first FULL ~10 GB snapshot
projects to ~5h.

- BackupTargetConfig.WaitTimeoutSeconds: per-tier bound. Primary 30m UNCHANGED
  (a local vzdump hanging 30m IS a real fault); additional tier 6h, sized from
  the measurement.
- backup.NewBackupRunnerWithWait: per-instance (per-tier) bound.
  NewBackupRunner keeps its signature, so restore-test/selftest are untouched.
- localapi.BackupTier.WaitTimeout: the fire-and-forget context is sized from the
  tier, not a fixed 2h. BOTH bounds had to move — a 6h runner bound under a 2h
  outer context reproduces the same false failure four hours later.

Same direction as restore_test_pbs_restore_timeout_seconds: when in doubt wait
LONGER. A slow backup is a slow backup; a false timeout is a corrupt status
plus lock contention.

Red-proof observed and restored; full suite green.
This commit is contained in:
Claude Code
2026-07-26 14:53:24 +02:00
parent 68bcebe493
commit a667c269c7
7 changed files with 155 additions and 21 deletions
+13 -3
View File
@@ -51,8 +51,12 @@ type BackupService interface {
type BackupTier struct {
TargetID string
Cadence time.Duration
Primary bool
Service BackupService
// WaitTimeout bounds the fire-and-forget backup context. It MUST be >= the runner's own wait
// bound, or the outer context cancels first and the tier reports a false failure while the
// vzdump keeps running (observed live 2026-07-26 with a fixed 2h outer bound).
WaitTimeout time.Duration
Primary bool
Service BackupService
}
// BackupStore records + reads the latest backup/restore-test state. Satisfied by *backup.Store.
@@ -734,7 +738,13 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
base = context.Background()
}
go func() {
bctx, cancel := context.WithTimeout(base, 2*time.Hour)
// Outer bound = the tier's own wait bound + headroom for the pre/post work around WaitTask.
// A fixed 2h here would silently cap a 6h offsite tier.
outer := tier.WaitTimeout
if outer <= 0 {
outer = 2 * time.Hour
}
bctx, cancel := context.WithTimeout(base, outer+15*time.Minute)
defer cancel()
// 8B.2: flip the job to `snapshotted` when the storage snapshot is taken, so the
// controller resumes its app early (snapshot mode only; in stop mode this never fires).