Claude Code
a667c269c7
Found by live validation on demo-felhom, not by review. The first real PBS-targeted backup ran past the runner's hard-coded 30-minute WaitTask bound. The agent stopped waiting and recorded success=false WHILE THE VZDUMP KEPT RUNNING (still running 72 min later, 2.4 GB uploaded). Consequences: the tier stays permanently due, the next attempt collides with the guest lock the live vzdump holds, and the hub sees a DR tier that never succeeds — R-82's 'applied and empty' fault re-created by a timeout. Measured: ~33 MB/min over wg to Hetzner, so a first FULL ~10 GB snapshot projects to ~5h. - BackupTargetConfig.WaitTimeoutSeconds: per-tier bound. Primary 30m UNCHANGED (a local vzdump hanging 30m IS a real fault); additional tier 6h, sized from the measurement. - backup.NewBackupRunnerWithWait: per-instance (per-tier) bound. NewBackupRunner keeps its signature, so restore-test/selftest are untouched. - localapi.BackupTier.WaitTimeout: the fire-and-forget context is sized from the tier, not a fixed 2h. BOTH bounds had to move — a 6h runner bound under a 2h outer context reproduces the same false failure four hours later. Same direction as restore_test_pbs_restore_timeout_seconds: when in doubt wait LONGER. A slow backup is a slow backup; a false timeout is a corrupt status plus lock contention. Red-proof observed and restored; full suite green.
60 lines
1.9 KiB
Go
60 lines
1.9 KiB
Go
package localapi
|
|
|
|
import "time"
|
|
|
|
// normalizeBackupTiers resolves the tier list the Server serves.
|
|
//
|
|
// Contract (R-82), and the reason this is a named function rather than inline setup: the UNTARGETED
|
|
// local-API endpoints must keep behaving exactly as they did before multi-tier existed, forever.
|
|
// That property lives here.
|
|
//
|
|
// - tiers == nil → synthesize ONE tier from the legacy (Backups, BackupCadence) pair and mark it
|
|
// primary. This is the pre-R-82 shape; every existing caller and test hits this path.
|
|
// - tiers supplied → keep order but hoist the primary to the front; if none is marked primary,
|
|
// the FIRST becomes primary (a tier list with no primary would leave untargeted requests with
|
|
// nothing to act on, which would silently stop backups).
|
|
// - tiers with a nil Service are dropped: a tier with no runner cannot back anything up, and
|
|
// advertising it would be an "applied and empty" tier — the exact fault R-82 exists to fix.
|
|
func normalizeBackupTiers(tiers []BackupTier, legacy BackupService, cadence time.Duration) []BackupTier {
|
|
usable := make([]BackupTier, 0, len(tiers))
|
|
for _, t := range tiers {
|
|
if t.Service == nil || t.TargetID == "" {
|
|
continue
|
|
}
|
|
if t.Cadence <= 0 {
|
|
t.Cadence = cadence
|
|
}
|
|
if t.WaitTimeout <= 0 {
|
|
t.WaitTimeout = 2 * time.Hour
|
|
}
|
|
usable = append(usable, t)
|
|
}
|
|
if len(usable) == 0 {
|
|
if legacy == nil {
|
|
return nil
|
|
}
|
|
return []BackupTier{{TargetID: "", Cadence: cadence, WaitTimeout: 2 * time.Hour, Primary: true, Service: legacy}}
|
|
}
|
|
primary := -1
|
|
for i, t := range usable {
|
|
if t.Primary {
|
|
primary = i
|
|
break
|
|
}
|
|
}
|
|
if primary < 0 {
|
|
primary = 0
|
|
}
|
|
out := make([]BackupTier, 0, len(usable))
|
|
usable[primary].Primary = true
|
|
out = append(out, usable[primary])
|
|
for i, t := range usable {
|
|
if i == primary {
|
|
continue
|
|
}
|
|
t.Primary = false
|
|
out = append(out, t)
|
|
}
|
|
return out
|
|
}
|