v0.98.0 — R-82 Slice A fix: per-tier vzdump wait bound (the 30-minute false failure)
Found by live validation on demo-felhom, not by review. The first real PBS-targeted backup ran past the runner's hard-coded 30-minute WaitTask bound. The agent stopped waiting and recorded success=false WHILE THE VZDUMP KEPT RUNNING (still running 72 min later, 2.4 GB uploaded). Consequences: the tier stays permanently due, the next attempt collides with the guest lock the live vzdump holds, and the hub sees a DR tier that never succeeds — R-82's 'applied and empty' fault re-created by a timeout. Measured: ~33 MB/min over wg to Hetzner, so a first FULL ~10 GB snapshot projects to ~5h. - BackupTargetConfig.WaitTimeoutSeconds: per-tier bound. Primary 30m UNCHANGED (a local vzdump hanging 30m IS a real fault); additional tier 6h, sized from the measurement. - backup.NewBackupRunnerWithWait: per-instance (per-tier) bound. NewBackupRunner keeps its signature, so restore-test/selftest are untouched. - localapi.BackupTier.WaitTimeout: the fire-and-forget context is sized from the tier, not a fixed 2h. BOTH bounds had to move — a 6h runner bound under a 2h outer context reproduces the same false failure four hours later. Same direction as restore_test_pbs_restore_timeout_seconds: when in doubt wait LONGER. A slow backup is a slow backup; a false timeout is a corrupt status plus lock contention. Red-proof observed and restored; full suite green.
This commit is contained in:
@@ -51,8 +51,12 @@ type BackupService interface {
|
||||
type BackupTier struct {
|
||||
TargetID string
|
||||
Cadence time.Duration
|
||||
Primary bool
|
||||
Service BackupService
|
||||
// WaitTimeout bounds the fire-and-forget backup context. It MUST be >= the runner's own wait
|
||||
// bound, or the outer context cancels first and the tier reports a false failure while the
|
||||
// vzdump keeps running (observed live 2026-07-26 with a fixed 2h outer bound).
|
||||
WaitTimeout time.Duration
|
||||
Primary bool
|
||||
Service BackupService
|
||||
}
|
||||
|
||||
// BackupStore records + reads the latest backup/restore-test state. Satisfied by *backup.Store.
|
||||
@@ -734,7 +738,13 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
|
||||
base = context.Background()
|
||||
}
|
||||
go func() {
|
||||
bctx, cancel := context.WithTimeout(base, 2*time.Hour)
|
||||
// Outer bound = the tier's own wait bound + headroom for the pre/post work around WaitTask.
|
||||
// A fixed 2h here would silently cap a 6h offsite tier.
|
||||
outer := tier.WaitTimeout
|
||||
if outer <= 0 {
|
||||
outer = 2 * time.Hour
|
||||
}
|
||||
bctx, cancel := context.WithTimeout(base, outer+15*time.Minute)
|
||||
defer cancel()
|
||||
// 8B.2: flip the job to `snapshotted` when the storage snapshot is taken, so the
|
||||
// controller resumes its app early (snapshot mode only; in stop mode this never fires).
|
||||
|
||||
Reference in New Issue
Block a user