v0.98.0 — R-82 Slice A fix: per-tier vzdump wait bound (the 30-minute false failure)

Found by live validation on demo-felhom, not by review.

The first real PBS-targeted backup ran past the runner's hard-coded 30-minute
WaitTask bound. The agent stopped waiting and recorded success=false WHILE THE
VZDUMP KEPT RUNNING (still running 72 min later, 2.4 GB uploaded). Consequences:
the tier stays permanently due, the next attempt collides with the guest lock
the live vzdump holds, and the hub sees a DR tier that never succeeds — R-82's
'applied and empty' fault re-created by a timeout.

Measured: ~33 MB/min over wg to Hetzner, so a first FULL ~10 GB snapshot
projects to ~5h.

- BackupTargetConfig.WaitTimeoutSeconds: per-tier bound. Primary 30m UNCHANGED
  (a local vzdump hanging 30m IS a real fault); additional tier 6h, sized from
  the measurement.
- backup.NewBackupRunnerWithWait: per-instance (per-tier) bound.
  NewBackupRunner keeps its signature, so restore-test/selftest are untouched.
- localapi.BackupTier.WaitTimeout: the fire-and-forget context is sized from the
  tier, not a fixed 2h. BOTH bounds had to move — a 6h runner bound under a 2h
  outer context reproduces the same false failure four hours later.

Same direction as restore_test_pbs_restore_timeout_seconds: when in doubt wait
LONGER. A slow backup is a slow backup; a false timeout is a corrupt status
plus lock contention.

Red-proof observed and restored; full suite green.
This commit is contained in:
Claude Code
2026-07-26 14:53:24 +02:00
parent 68bcebe493
commit a667c269c7
7 changed files with 155 additions and 21 deletions
+39
View File
@@ -1,3 +1,42 @@
## v0.98.0 — R-82 Slice A fix: per-tier vzdump wait bound (the 30-minute false failure) (2026-07-26)
**Found by live validation on demo-felhom, not by review.** The first real PBS-targeted backup ran
past the runner's hard-coded 30-minute `WaitTask` bound. The agent stopped waiting, recorded
`success:false`**while the vzdump kept running** (still running 72 minutes later, 2.4 GB
uploaded). That is not "the backup didn't happen"; it is worse:
- the tier stays permanently **due** (a failed backup never satisfies a cadence),
- the next attempt collides with the **guest lock** the live vzdump still holds,
- the hub sees a DR tier that never succeeds — R-82's "applied and empty" fault, re-created by a
timeout,
- the recorded `Backup` says `success:false, size_bytes:0` for a backup that may yet complete.
30 minutes is right for a LOCAL vzdump (minutes) and simply wrong for an offsite upload. Measured on
demo-felhom: ~33 MB/min over the wg link to Hetzner ⇒ a first FULL ~10 GB snapshot projects to ≈5 h.
### Changed
- **`BackupTargetConfig.WaitTimeoutSeconds`** — per-tier vzdump wait bound. Defaults:
**primary 30 m (UNCHANGED)**, additional tier **6 h**. The asymmetry is the point: the primary is
the local tier where a 30-minute hang IS a genuine fault worth surfacing; an additional tier is by
construction the offsite one, where the binding constraint is uplink speed, not health. 6 h is
sized from the measurement above, not guessed.
- **`backup.NewBackupRunnerWithWait`** — the runner's wait bound is per-instance (i.e. per tier).
`NewBackupRunner` keeps its signature and delegates with 0 ⇒ 30 m, so every other caller
(restore-test, selftest) is untouched.
- **`localapi.BackupTier.WaitTimeout`** — the fire-and-forget backup context is now sized from the
tier instead of a fixed 2 h. **Both bounds had to move**: a 6 h runner bound under a 2 h outer
context would have reproduced the same false failure four hours later.
Same reasoning, and the same direction, as `restore_test_pbs_restore_timeout_seconds` on the restore
side: **when in doubt wait LONGER.** A slow backup is a slow backup; a false timeout is a corrupt
status plus lock contention.
### Tests
`TestBackupTiers_WaitTimeoutIsPerTier` pins the asymmetric defaults, the override, and that the two
tiers do NOT share one bound. Red-proof observed: setting the extra tier's default back to 30 m
fails with `an offsite tier must default to a GENEROUS wait — a false timeout is worse than a slow
pass; got 30m0s`. Restored; full suite green.
## v0.97.0 — R-82 Slice A: per-target backup tiers (local daily + PBS weekly) (2026-07-26)
Additive; **MinAgent floor rises** for the multi-tier contract (a controller that wants per-tier