R-85 Phase 2: tier rotation, persisted state, one heavy op at a time
The scheduler could only ever see cfg.Backup.BackupTarget(), so the offsite tier's archives were never candidates — which is why demo-hp's DR tier reported 'applied' with zero snapshots for five days and nobody noticed. Selection: oldest-first (operator ruling, Option 1). Never-proven sorts first, which is where the offsite tier starts. Ties break on target id so ordering is deterministic rather than following Go's randomised map order. Rotation credit only on SUCCESS — a permanently failing tier must keep sorting first, not look freshly proven and stop being retried. - backup.RestoreTestState: persisted last-success per tier (atomic tmp+rename). This genuinely needs persistence unlike R-84: R-84 had ground truth to consult (the archive is still on the storage), whereas a restore-test destroys its scratch and leaves no artifact. Corrupt/missing file -> 'nothing proven'. - backup.InFlight: host-wide one-heavy-op gate shared with the local-API backup path. A LINK concern, not a lock one — an offsite restore pulls multi-GB over the same tunnel a backup pushes one, and at ~33 MB/min both drift toward timeout, which is how a healthy tier gets recorded as failed. Callers DEFER, never cancel. - PickRestoreCandidateOn: newest archive on a named tier; '' is not an error, or every fresh box looks broken for its first week. - An empty tier is skipped and the next tried; it cannot starve, since it is still least-recently-proven once it has an archive. - POST /backup joins the gate (409 naming the holder). Red-proofs A/E/F observed with the documented text. Full suite green (29 packages, rc=0).
This commit is contained in:
@@ -16,6 +16,7 @@ import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/escrow"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
applog "gitea.dooplex.hu/admin/felhom-agent/internal/log"
|
||||
@@ -127,6 +128,9 @@ type Options struct {
|
||||
// BackupTiers (R-82) is the resolved multi-tier policy, primary first. OPTIONAL: nil → one tier
|
||||
// synthesized from Backups + BackupCadence, i.e. exactly the pre-R-82 behaviour.
|
||||
BackupTiers []BackupTier
|
||||
// InFlight (R-85) is the host-wide one-heavy-operation gate shared with the restore-test
|
||||
// scheduler. OPTIONAL: nil → no cross-gating (pre-R-85 behaviour). See backup.InFlight.
|
||||
InFlight *backup.InFlight
|
||||
// Disk management (slice 8C) — OPTIONAL. When Disks + DiskGate are set, the /disks endpoints
|
||||
// are served; otherwise they report "not configured". DiskGate authorizes the destructive
|
||||
// (data-bearing) format path; Guests lists guests for the eject dependent-warning.
|
||||
@@ -240,8 +244,10 @@ type Server struct {
|
||||
// caller supplies no tiers it holds exactly one, synthesized from Backups+BackupCadence, which
|
||||
// is the pre-R-82 shape.
|
||||
tiers []BackupTier
|
||||
logger *slog.Logger
|
||||
now func() time.Time
|
||||
// inFlight (R-85) is shared with the restore-test scheduler so the two never run together.
|
||||
inFlight *backup.InFlight
|
||||
logger *slog.Logger
|
||||
now func() time.Time
|
||||
|
||||
disks DiskOps // slice 8C (optional)
|
||||
diskGate StorageGate // slice 8C (optional)
|
||||
@@ -391,6 +397,7 @@ func NewServer(o Options) (*Server, error) {
|
||||
// (and every existing test) keeps working untouched. Exactly one tier is marked primary, and
|
||||
// the primary is always first, because that is what the untargeted endpoints act on.
|
||||
s.tiers = normalizeBackupTiers(o.BackupTiers, o.Backups, cadence)
|
||||
s.inFlight = o.InFlight
|
||||
if s.backups == nil && len(s.tiers) > 0 {
|
||||
s.backups = s.tiers[0].Service
|
||||
}
|
||||
@@ -750,6 +757,19 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
|
||||
// otherwise collide and hand the second caller the first tier's id. The PRIMARY keeps the
|
||||
// pre-R-82 format byte-for-byte — an old controller stores this string and polls with it — so
|
||||
// only the additive tiers carry the target segment.
|
||||
// R-85 Scenario F: a backup and a restore-test must never run together — both move multi-GB over
|
||||
// the same tunnel. Acquired here (still holding jobsMu is fine: TryAcquire never blocks) and
|
||||
// released when the fire-and-forget goroutine finishes.
|
||||
release, busy, free := s.inFlight.TryAcquire("backup:" + tier.TargetID)
|
||||
if !free {
|
||||
s.jobsMu.Unlock()
|
||||
s.logger.Info("local-api: backup refused — a heavy operation is already in flight",
|
||||
"vmid", vmid, "requested_target", tier.TargetID, "busy", busy)
|
||||
writeStatus(w, http.StatusConflict, false, nil,
|
||||
"a heavy operation is already in flight ("+busy+") — only one runs at a time on this host")
|
||||
return
|
||||
}
|
||||
|
||||
jobID := "backup-" + strconv.Itoa(vmid) + "-" + strconv.FormatInt(s.now().UnixNano(), 10)
|
||||
if !tier.Primary && tier.TargetID != "" {
|
||||
jobID = "backup-" + strconv.Itoa(vmid) + "-" + tier.TargetID + "-" + strconv.FormatInt(s.now().UnixNano(), 10)
|
||||
@@ -767,6 +787,7 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
|
||||
base = context.Background()
|
||||
}
|
||||
go func() {
|
||||
defer release() // R-85: free the host-wide gate when this backup finishes, however it ends
|
||||
// Outer bound = the tier's own wait bound + headroom for the pre/post work around WaitTask.
|
||||
// A fixed 2h here would silently cap a 6h offsite tier.
|
||||
outer := tier.WaitTimeout
|
||||
|
||||
Reference in New Issue
Block a user