v0.98.0 — R-82 Slice A fix: per-tier vzdump wait bound (the 30-minute false failure)
Found by live validation on demo-felhom, not by review. The first real PBS-targeted backup ran past the runner's hard-coded 30-minute WaitTask bound. The agent stopped waiting and recorded success=false WHILE THE VZDUMP KEPT RUNNING (still running 72 min later, 2.4 GB uploaded). Consequences: the tier stays permanently due, the next attempt collides with the guest lock the live vzdump holds, and the hub sees a DR tier that never succeeds — R-82's 'applied and empty' fault re-created by a timeout. Measured: ~33 MB/min over wg to Hetzner, so a first FULL ~10 GB snapshot projects to ~5h. - BackupTargetConfig.WaitTimeoutSeconds: per-tier bound. Primary 30m UNCHANGED (a local vzdump hanging 30m IS a real fault); additional tier 6h, sized from the measurement. - backup.NewBackupRunnerWithWait: per-instance (per-tier) bound. NewBackupRunner keeps its signature, so restore-test/selftest are untouched. - localapi.BackupTier.WaitTimeout: the fire-and-forget context is sized from the tier, not a fixed 2h. BOTH bounds had to move — a 6h runner bound under a 2h outer context reproduces the same false failure four hours later. Same direction as restore_test_pbs_restore_timeout_seconds: when in doubt wait LONGER. A slow backup is a slow backup; a false timeout is a corrupt status plus lock contention. Red-proof observed and restored; full suite green.
This commit is contained in:
@@ -1,3 +1,42 @@
|
||||
## v0.98.0 — R-82 Slice A fix: per-tier vzdump wait bound (the 30-minute false failure) (2026-07-26)
|
||||
|
||||
**Found by live validation on demo-felhom, not by review.** The first real PBS-targeted backup ran
|
||||
past the runner's hard-coded 30-minute `WaitTask` bound. The agent stopped waiting, recorded
|
||||
`success:false` — **while the vzdump kept running** (still running 72 minutes later, 2.4 GB
|
||||
uploaded). That is not "the backup didn't happen"; it is worse:
|
||||
|
||||
- the tier stays permanently **due** (a failed backup never satisfies a cadence),
|
||||
- the next attempt collides with the **guest lock** the live vzdump still holds,
|
||||
- the hub sees a DR tier that never succeeds — R-82's "applied and empty" fault, re-created by a
|
||||
timeout,
|
||||
- the recorded `Backup` says `success:false, size_bytes:0` for a backup that may yet complete.
|
||||
|
||||
30 minutes is right for a LOCAL vzdump (minutes) and simply wrong for an offsite upload. Measured on
|
||||
demo-felhom: ~33 MB/min over the wg link to Hetzner ⇒ a first FULL ~10 GB snapshot projects to ≈5 h.
|
||||
|
||||
### Changed
|
||||
- **`BackupTargetConfig.WaitTimeoutSeconds`** — per-tier vzdump wait bound. Defaults:
|
||||
**primary 30 m (UNCHANGED)**, additional tier **6 h**. The asymmetry is the point: the primary is
|
||||
the local tier where a 30-minute hang IS a genuine fault worth surfacing; an additional tier is by
|
||||
construction the offsite one, where the binding constraint is uplink speed, not health. 6 h is
|
||||
sized from the measurement above, not guessed.
|
||||
- **`backup.NewBackupRunnerWithWait`** — the runner's wait bound is per-instance (i.e. per tier).
|
||||
`NewBackupRunner` keeps its signature and delegates with 0 ⇒ 30 m, so every other caller
|
||||
(restore-test, selftest) is untouched.
|
||||
- **`localapi.BackupTier.WaitTimeout`** — the fire-and-forget backup context is now sized from the
|
||||
tier instead of a fixed 2 h. **Both bounds had to move**: a 6 h runner bound under a 2 h outer
|
||||
context would have reproduced the same false failure four hours later.
|
||||
|
||||
Same reasoning, and the same direction, as `restore_test_pbs_restore_timeout_seconds` on the restore
|
||||
side: **when in doubt wait LONGER.** A slow backup is a slow backup; a false timeout is a corrupt
|
||||
status plus lock contention.
|
||||
|
||||
### Tests
|
||||
`TestBackupTiers_WaitTimeoutIsPerTier` pins the asymmetric defaults, the override, and that the two
|
||||
tiers do NOT share one bound. Red-proof observed: setting the extra tier's default back to 30 m
|
||||
fails with `an offsite tier must default to a GENEROUS wait — a false timeout is worse than a slow
|
||||
pass; got 30m0s`. Restored; full suite green.
|
||||
|
||||
## v0.97.0 — R-82 Slice A: per-target backup tiers (local daily + PBS weekly) (2026-07-26)
|
||||
|
||||
Additive; **MinAgent floor rises** for the multi-tier contract (a controller that wants per-tier
|
||||
|
||||
Reference in New Issue
Block a user