v0.99.0 — R-82 operator rulings: 2-week offsite retention + one backup at a time

Ruling 1 (2 weeks of weekly offsite backups): localPruneSpec's blanket PBS
refusal is now scoped — an ADDITIONAL tier with an explicit keep_last may
prune its PBS target. The refusal still applies in full to the PRIMARY tier,
because BackupTarget() defaults to felhom-pbs and KeepLast() defaults to 3, so
a box with neither key set would silently prune its offsite DR to 3 restore
points. An additional tier cannot have that accident (keep_last defaults to 0).

Ruling 3 (first backup runs as long as needed; nothing else starts until done):
- additional-tier wait bound 6h -> 12h (measured ~33 MB/min => ~5h for a first
  full 10 GB snapshot; 12h gives margin but stays bounded so a hung task still
  surfaces)
- ONE BACKUP AT A TIME PER GUEST across all tiers: POST /backup returns 409
  when a DIFFERENT tier is in flight, naming the busy tier, with NO data object
  so nothing is parseable as the caller's own job. Same tier still returns that
  job (202, unchanged).
- snapshotted now counts as in-flight, not just running — after the snapshot the
  vzdump is still uploading and holding the lock. The old check left a window
  where a second POST started a real second vzdump. Latent bug, closed.

Full suite green (29 packages); red-proof observed and restored.
This commit is contained in:
Claude Code
2026-07-26 15:05:54 +02:00
parent a667c269c7
commit 3d955e4edd
7 changed files with 203 additions and 30 deletions
+42 -5
View File
@@ -705,17 +705,34 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
}
key := backupJobKey{vmid: vmid, target: tier.TargetID}
// Single-flight per guest PER TIER: if a backup is already running for this guest ON THIS
// TIER, return that job (don't start a second concurrent vzdump to the same target). A
// DIFFERENT tier is a different job — that is what lets the weekly night run both backups
// inside one quiesce window without the second call being handed the first one's id.
// ONE BACKUP AT A TIME PER GUEST, ACROSS ALL TIERS (operator ruling 2026-07-26: "other backup
// shouldn't start until finished"). vzdump takes a guest lock, so a concurrent second backup
// could not succeed anyway — but without this guard it would be ATTEMPTED, fail on the lock, and
// record a spurious failure that leaves the tier permanently due.
//
// Two distinct cases, deliberately answered differently:
// - SAME tier already in flight → return THAT job (202). Idempotent: the caller re-polls it.
// - DIFFERENT tier in flight → 409. Not a new job, and NOT the other tier's job either —
// handing back a foreign job id is how a caller comes to believe its own backup ran.
//
// "In flight" includes `snapshotted`, not just `running`: after the storage snapshot the vzdump
// is still uploading and still holding the lock. Checking only `running` (the pre-R-82 code)
// left a window where a second POST would start a real second vzdump.
s.jobsMu.Lock()
if cur := s.jobs[key]; cur != nil && cur.Phase == PhaseRunning {
if cur := s.jobs[key]; cur != nil && backupInFlight(cur.Phase) {
job := *cur
s.jobsMu.Unlock()
writeStatus(w, http.StatusAccepted, true, BackupResponse{VMID: vmid, JobID: job.JobID, Phase: job.Phase, Target: echo}, "")
return
}
if busyTarget, busyJob, busy := s.otherTierInFlight(vmid, tier.TargetID); busy {
s.jobsMu.Unlock()
s.logger.Info("local-api: backup refused — another tier is still in flight",
"vmid", vmid, "requested_target", tier.TargetID, "busy_target", busyTarget, "busy_job", busyJob)
writeStatus(w, http.StatusConflict, false, nil,
"a backup is already in flight on target "+busyTarget+" (job "+busyJob+") — only one backup runs at a time per guest")
return
}
// Job ids must be unique PER TIER, and by construction rather than by clock luck: two tiers
// started inside the same nanosecond (the weekly both-due night, or any injected clock) would
// otherwise collide and hand the second caller the first tier's id. The PRIMARY keeps the
@@ -770,6 +787,26 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
writeStatus(w, http.StatusAccepted, true, BackupResponse{VMID: vmid, JobID: jobID, Phase: PhaseRunning}, "")
}
// backupInFlight reports whether a phase means "this backup still holds the guest".
// `snapshotted` counts: the storage snapshot is taken but the vzdump is still uploading.
func backupInFlight(phase string) bool {
return phase == PhaseRunning || phase == PhaseSnapshotted
}
// otherTierInFlight reports whether a DIFFERENT tier has an in-flight backup for this guest.
// Caller must hold s.jobsMu.
func (s *Server) otherTierInFlight(vmid int, target string) (busyTarget, busyJob string, busy bool) {
for k, j := range s.jobs {
if k.vmid != vmid || k.target == target || j == nil {
continue
}
if backupInFlight(j.Phase) {
return k.target, j.JobID, true
}
}
return "", "", false
}
// markSnapshotted flips the guest's running job to the `snapshotted` phase (8B.2) — only if it is
// still the current job and still running (don't regress done/failed, and don't touch a newer job).
func (s *Server) markSnapshotted(key backupJobKey, jobID string) {