Claude Code
043c7622bc
The scheduler could only ever see cfg.Backup.BackupTarget(), so the offsite tier's archives were never candidates — which is why demo-hp's DR tier reported 'applied' with zero snapshots for five days and nobody noticed. Selection: oldest-first (operator ruling, Option 1). Never-proven sorts first, which is where the offsite tier starts. Ties break on target id so ordering is deterministic rather than following Go's randomised map order. Rotation credit only on SUCCESS — a permanently failing tier must keep sorting first, not look freshly proven and stop being retried. - backup.RestoreTestState: persisted last-success per tier (atomic tmp+rename). This genuinely needs persistence unlike R-84: R-84 had ground truth to consult (the archive is still on the storage), whereas a restore-test destroys its scratch and leaves no artifact. Corrupt/missing file -> 'nothing proven'. - backup.InFlight: host-wide one-heavy-op gate shared with the local-API backup path. A LINK concern, not a lock one — an offsite restore pulls multi-GB over the same tunnel a backup pushes one, and at ~33 MB/min both drift toward timeout, which is how a healthy tier gets recorded as failed. Callers DEFER, never cancel. - PickRestoreCandidateOn: newest archive on a named tier; '' is not an error, or every fresh box looks broken for its first week. - An empty tier is skipped and the next tried; it cannot starve, since it is still least-recently-proven once it has an archive. - POST /backup joins the gate (409 naming the holder). Red-proofs A/E/F observed with the documented text. Full suite green (29 packages, rc=0).
58 lines
2.0 KiB
Go
58 lines
2.0 KiB
Go
package backup
|
|
|
|
import "sync"
|
|
|
|
// InFlight is the host-wide "one heavy guest operation at a time" gate.
|
|
//
|
|
// R-85 (Scenario F). The operator's R-82 ruling was "one backup at a time per guest"; a restore-test
|
|
// must JOIN that single-flight rather than sit outside it. It is not a lock-contention concern —
|
|
// a restore-test uses a scratch VMID, so it never touches the live guest's vzdump lock. It is a
|
|
// LINK concern: an offsite restore PULLS a multi-GB archive while an offsite backup PUSHES one, over
|
|
// the same WireGuard tunnel. On the demo fleet that link runs at ~33 MB/min upstream; running both
|
|
// at once makes each slower and pushes both toward their timeouts, which is how a healthy tier ends
|
|
// up recorded as failed.
|
|
//
|
|
// It is deliberately host-wide and coarse rather than per-guest: these boxes carry one customer
|
|
// guest, and the resource being protected (the uplink) is shared by everything on the host anyway.
|
|
//
|
|
// The gate is ADVISORY in one direction only — it never cancels anything already running. A caller
|
|
// that cannot acquire DEFERS to its next cadence. Deferring a restore-test costs a few hours of
|
|
// coverage; cancelling a running backup costs the backup.
|
|
type InFlight struct {
|
|
mu sync.Mutex
|
|
what string // "" = idle
|
|
}
|
|
|
|
// TryAcquire claims the gate for `what`. ok=false means something else holds it, and `busy` names
|
|
// it — the name matters, because "deferred" with no reason is indistinguishable from "broken".
|
|
func (g *InFlight) TryAcquire(what string) (release func(), busy string, ok bool) {
|
|
if g == nil {
|
|
// Not wired (older call sites, tests) → no gating, previous behaviour.
|
|
return func() {}, "", true
|
|
}
|
|
g.mu.Lock()
|
|
defer g.mu.Unlock()
|
|
if g.what != "" {
|
|
return nil, g.what, false
|
|
}
|
|
g.what = what
|
|
var once sync.Once
|
|
return func() {
|
|
once.Do(func() {
|
|
g.mu.Lock()
|
|
g.what = ""
|
|
g.mu.Unlock()
|
|
})
|
|
}, "", true
|
|
}
|
|
|
|
// Busy reports what currently holds the gate ("" = idle).
|
|
func (g *InFlight) Busy() string {
|
|
if g == nil {
|
|
return ""
|
|
}
|
|
g.mu.Lock()
|
|
defer g.mu.Unlock()
|
|
return g.what
|
|
}
|