R-85 Phase 2: tier rotation, persisted state, one heavy op at a time

The scheduler could only ever see cfg.Backup.BackupTarget(), so the offsite
tier's archives were never candidates — which is why demo-hp's DR tier reported
'applied' with zero snapshots for five days and nobody noticed.

Selection: oldest-first (operator ruling, Option 1). Never-proven sorts first,
which is where the offsite tier starts. Ties break on target id so ordering is
deterministic rather than following Go's randomised map order. Rotation credit
only on SUCCESS — a permanently failing tier must keep sorting first, not look
freshly proven and stop being retried.

- backup.RestoreTestState: persisted last-success per tier (atomic tmp+rename).
  This genuinely needs persistence unlike R-84: R-84 had ground truth to consult
  (the archive is still on the storage), whereas a restore-test destroys its
  scratch and leaves no artifact. Corrupt/missing file -> 'nothing proven'.
- backup.InFlight: host-wide one-heavy-op gate shared with the local-API backup
  path. A LINK concern, not a lock one — an offsite restore pulls multi-GB over
  the same tunnel a backup pushes one, and at ~33 MB/min both drift toward
  timeout, which is how a healthy tier gets recorded as failed. Callers DEFER,
  never cancel.
- PickRestoreCandidateOn: newest archive on a named tier; '' is not an error, or
  every fresh box looks broken for its first week.
- An empty tier is skipped and the next tried; it cannot starve, since it is
  still least-recently-proven once it has an archive.
- POST /backup joins the gate (409 naming the holder).

Red-proofs A/E/F observed with the documented text. Full suite green (29
packages, rc=0).
This commit is contained in:
Claude Code
2026-07-26 21:00:42 +02:00
parent 765d8b3168
commit 043c7622bc
8 changed files with 743 additions and 18 deletions
+23 -2
View File
@@ -16,6 +16,7 @@ import (
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
"gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
applog "gitea.dooplex.hu/admin/felhom-agent/internal/log"
@@ -127,6 +128,9 @@ type Options struct {
// BackupTiers (R-82) is the resolved multi-tier policy, primary first. OPTIONAL: nil → one tier
// synthesized from Backups + BackupCadence, i.e. exactly the pre-R-82 behaviour.
BackupTiers []BackupTier
// InFlight (R-85) is the host-wide one-heavy-operation gate shared with the restore-test
// scheduler. OPTIONAL: nil → no cross-gating (pre-R-85 behaviour). See backup.InFlight.
InFlight *backup.InFlight
// Disk management (slice 8C) — OPTIONAL. When Disks + DiskGate are set, the /disks endpoints
// are served; otherwise they report "not configured". DiskGate authorizes the destructive
// (data-bearing) format path; Guests lists guests for the eject dependent-warning.
@@ -240,8 +244,10 @@ type Server struct {
// caller supplies no tiers it holds exactly one, synthesized from Backups+BackupCadence, which
// is the pre-R-82 shape.
tiers []BackupTier
logger *slog.Logger
now func() time.Time
// inFlight (R-85) is shared with the restore-test scheduler so the two never run together.
inFlight *backup.InFlight
logger *slog.Logger
now func() time.Time
disks DiskOps // slice 8C (optional)
diskGate StorageGate // slice 8C (optional)
@@ -391,6 +397,7 @@ func NewServer(o Options) (*Server, error) {
// (and every existing test) keeps working untouched. Exactly one tier is marked primary, and
// the primary is always first, because that is what the untargeted endpoints act on.
s.tiers = normalizeBackupTiers(o.BackupTiers, o.Backups, cadence)
s.inFlight = o.InFlight
if s.backups == nil && len(s.tiers) > 0 {
s.backups = s.tiers[0].Service
}
@@ -750,6 +757,19 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
// otherwise collide and hand the second caller the first tier's id. The PRIMARY keeps the
// pre-R-82 format byte-for-byte — an old controller stores this string and polls with it — so
// only the additive tiers carry the target segment.
// R-85 Scenario F: a backup and a restore-test must never run together — both move multi-GB over
// the same tunnel. Acquired here (still holding jobsMu is fine: TryAcquire never blocks) and
// released when the fire-and-forget goroutine finishes.
release, busy, free := s.inFlight.TryAcquire("backup:" + tier.TargetID)
if !free {
s.jobsMu.Unlock()
s.logger.Info("local-api: backup refused — a heavy operation is already in flight",
"vmid", vmid, "requested_target", tier.TargetID, "busy", busy)
writeStatus(w, http.StatusConflict, false, nil,
"a heavy operation is already in flight ("+busy+") — only one runs at a time on this host")
return
}
jobID := "backup-" + strconv.Itoa(vmid) + "-" + strconv.FormatInt(s.now().UnixNano(), 10)
if !tier.Primary && tier.TargetID != "" {
jobID = "backup-" + strconv.Itoa(vmid) + "-" + tier.TargetID + "-" + strconv.FormatInt(s.now().UnixNano(), 10)
@@ -767,6 +787,7 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
base = context.Background()
}
go func() {
defer release() // R-85: free the host-wide gate when this backup finishes, however it ends
// Outer bound = the tier's own wait bound + headroom for the pre/post work around WaitTask.
// A fixed 2h here would silently cap a 6h offsite tier.
outer := tier.WaitTimeout