v0.99.0 — R-82 operator rulings: 2-week offsite retention + one backup at a time
Ruling 1 (2 weeks of weekly offsite backups): localPruneSpec's blanket PBS refusal is now scoped — an ADDITIONAL tier with an explicit keep_last may prune its PBS target. The refusal still applies in full to the PRIMARY tier, because BackupTarget() defaults to felhom-pbs and KeepLast() defaults to 3, so a box with neither key set would silently prune its offsite DR to 3 restore points. An additional tier cannot have that accident (keep_last defaults to 0). Ruling 3 (first backup runs as long as needed; nothing else starts until done): - additional-tier wait bound 6h -> 12h (measured ~33 MB/min => ~5h for a first full 10 GB snapshot; 12h gives margin but stays bounded so a hung task still surfaces) - ONE BACKUP AT A TIME PER GUEST across all tiers: POST /backup returns 409 when a DIFFERENT tier is in flight, naming the busy tier, with NO data object so nothing is parseable as the caller's own job. Same tier still returns that job (202, unchanged). - snapshotted now counts as in-flight, not just running — after the snapshot the vzdump is still uploading and holding the lock. The old check left a window where a second POST started a real second vzdump. Latent bug, closed. Full suite green (29 packages); red-proof observed and restored.
This commit is contained in:
@@ -705,17 +705,34 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
|
||||
}
|
||||
key := backupJobKey{vmid: vmid, target: tier.TargetID}
|
||||
|
||||
// Single-flight per guest PER TIER: if a backup is already running for this guest ON THIS
|
||||
// TIER, return that job (don't start a second concurrent vzdump to the same target). A
|
||||
// DIFFERENT tier is a different job — that is what lets the weekly night run both backups
|
||||
// inside one quiesce window without the second call being handed the first one's id.
|
||||
// ONE BACKUP AT A TIME PER GUEST, ACROSS ALL TIERS (operator ruling 2026-07-26: "other backup
|
||||
// shouldn't start until finished"). vzdump takes a guest lock, so a concurrent second backup
|
||||
// could not succeed anyway — but without this guard it would be ATTEMPTED, fail on the lock, and
|
||||
// record a spurious failure that leaves the tier permanently due.
|
||||
//
|
||||
// Two distinct cases, deliberately answered differently:
|
||||
// - SAME tier already in flight → return THAT job (202). Idempotent: the caller re-polls it.
|
||||
// - DIFFERENT tier in flight → 409. Not a new job, and NOT the other tier's job either —
|
||||
// handing back a foreign job id is how a caller comes to believe its own backup ran.
|
||||
//
|
||||
// "In flight" includes `snapshotted`, not just `running`: after the storage snapshot the vzdump
|
||||
// is still uploading and still holding the lock. Checking only `running` (the pre-R-82 code)
|
||||
// left a window where a second POST would start a real second vzdump.
|
||||
s.jobsMu.Lock()
|
||||
if cur := s.jobs[key]; cur != nil && cur.Phase == PhaseRunning {
|
||||
if cur := s.jobs[key]; cur != nil && backupInFlight(cur.Phase) {
|
||||
job := *cur
|
||||
s.jobsMu.Unlock()
|
||||
writeStatus(w, http.StatusAccepted, true, BackupResponse{VMID: vmid, JobID: job.JobID, Phase: job.Phase, Target: echo}, "")
|
||||
return
|
||||
}
|
||||
if busyTarget, busyJob, busy := s.otherTierInFlight(vmid, tier.TargetID); busy {
|
||||
s.jobsMu.Unlock()
|
||||
s.logger.Info("local-api: backup refused — another tier is still in flight",
|
||||
"vmid", vmid, "requested_target", tier.TargetID, "busy_target", busyTarget, "busy_job", busyJob)
|
||||
writeStatus(w, http.StatusConflict, false, nil,
|
||||
"a backup is already in flight on target "+busyTarget+" (job "+busyJob+") — only one backup runs at a time per guest")
|
||||
return
|
||||
}
|
||||
// Job ids must be unique PER TIER, and by construction rather than by clock luck: two tiers
|
||||
// started inside the same nanosecond (the weekly both-due night, or any injected clock) would
|
||||
// otherwise collide and hand the second caller the first tier's id. The PRIMARY keeps the
|
||||
@@ -770,6 +787,26 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
|
||||
writeStatus(w, http.StatusAccepted, true, BackupResponse{VMID: vmid, JobID: jobID, Phase: PhaseRunning}, "")
|
||||
}
|
||||
|
||||
// backupInFlight reports whether a phase means "this backup still holds the guest".
|
||||
// `snapshotted` counts: the storage snapshot is taken but the vzdump is still uploading.
|
||||
func backupInFlight(phase string) bool {
|
||||
return phase == PhaseRunning || phase == PhaseSnapshotted
|
||||
}
|
||||
|
||||
// otherTierInFlight reports whether a DIFFERENT tier has an in-flight backup for this guest.
|
||||
// Caller must hold s.jobsMu.
|
||||
func (s *Server) otherTierInFlight(vmid int, target string) (busyTarget, busyJob string, busy bool) {
|
||||
for k, j := range s.jobs {
|
||||
if k.vmid != vmid || k.target == target || j == nil {
|
||||
continue
|
||||
}
|
||||
if backupInFlight(j.Phase) {
|
||||
return k.target, j.JobID, true
|
||||
}
|
||||
}
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
// markSnapshotted flips the guest's running job to the `snapshotted` phase (8B.2) — only if it is
|
||||
// still the current job and still running (don't regress done/failed, and don't touch a newer job).
|
||||
func (s *Server) markSnapshotted(key backupJobKey, jobID string) {
|
||||
|
||||
Reference in New Issue
Block a user