R-85 Phase 2: tier rotation, persisted state, one heavy op at a time
The scheduler could only ever see cfg.Backup.BackupTarget(), so the offsite tier's archives were never candidates — which is why demo-hp's DR tier reported 'applied' with zero snapshots for five days and nobody noticed. Selection: oldest-first (operator ruling, Option 1). Never-proven sorts first, which is where the offsite tier starts. Ties break on target id so ordering is deterministic rather than following Go's randomised map order. Rotation credit only on SUCCESS — a permanently failing tier must keep sorting first, not look freshly proven and stop being retried. - backup.RestoreTestState: persisted last-success per tier (atomic tmp+rename). This genuinely needs persistence unlike R-84: R-84 had ground truth to consult (the archive is still on the storage), whereas a restore-test destroys its scratch and leaves no artifact. Corrupt/missing file -> 'nothing proven'. - backup.InFlight: host-wide one-heavy-op gate shared with the local-API backup path. A LINK concern, not a lock one — an offsite restore pulls multi-GB over the same tunnel a backup pushes one, and at ~33 MB/min both drift toward timeout, which is how a healthy tier gets recorded as failed. Callers DEFER, never cancel. - PickRestoreCandidateOn: newest archive on a named tier; '' is not an error, or every fresh box looks broken for its first week. - An empty tier is skipped and the next tried; it cannot starve, since it is still least-recently-proven once it has an archive. - POST /backup joins the gate (409 naming the holder). Red-proofs A/E/F observed with the documented text. Full suite green (29 packages, rc=0).
This commit is contained in:
@@ -1,3 +1,51 @@
|
||||
## v0.104.0-dev — R-85 Phase 2: tier rotation, persisted state, one heavy op at a time (2026-07-26)
|
||||
|
||||
The scheduler could only ever see `cfg.Backup.BackupTarget()`, so the offsite tier's archives were
|
||||
never candidates. That is why demo-hp's DR tier reported `applied` with **zero snapshots for five
|
||||
days** and nobody noticed.
|
||||
|
||||
### Selection — oldest-first (operator ruling 2026-07-26, Option 1)
|
||||
The tier whose last **successful** restore-test is oldest goes first; **never-proven sorts first of
|
||||
all**, which is exactly where the offsite tier starts. Self-balancing, no new config knob, and each
|
||||
tier is covered every ~2 cadences — comfortably inside the 2-week offsite retention, so a tier is
|
||||
never proven against an archive that is about to be pruned. Ties break on target id, because two
|
||||
tiers proven in the same second would otherwise rotate by Go's randomised map order: untestable, and
|
||||
occasionally starving.
|
||||
|
||||
Rotation credit is given **only on success**. A tier that fails every time must keep sorting first —
|
||||
otherwise a permanently broken tier would look freshly proven and quietly stop being retried.
|
||||
|
||||
### Added
|
||||
- **`backup.RestoreTestState`** — last successful restore-test per tier, persisted (atomic
|
||||
tmp+rename). **This one genuinely needs persistence, unlike R-84**, and the difference is worth
|
||||
stating because they look alike: R-84 had a GROUND TRUTH to consult (the archive is still on the
|
||||
storage), so it never persisted anything. A restore-test destroys its scratch as its final act and
|
||||
leaves no artifact — "did we prove this tier restores?" exists only as remembered state. A corrupt
|
||||
or missing file degrades to "nothing proven", which is the correct starting point.
|
||||
- **`backup.InFlight`** — the host-wide one-heavy-operation gate, shared by the restore-test
|
||||
scheduler and the local-API backup path. Not a lock concern (the scratch VMID never touches the
|
||||
live guest's vzdump lock) but a **LINK** concern: an offsite restore PULLS multi-GB over the same
|
||||
tunnel an offsite backup PUSHES one. At the ~33 MB/min measured upstream, running both drives each
|
||||
toward its timeout — which is how a healthy tier ends up recorded as failed. A caller that cannot
|
||||
acquire **defers**; it never cancels what is already running.
|
||||
- **`BackupRunner.PickRestoreCandidateOn`** — newest archive on a NAMED tier. `""` + nil error when
|
||||
that tier holds none: **a tier with nothing to restore is not an error**, or every fresh box would
|
||||
look broken for its first week.
|
||||
|
||||
### Changed
|
||||
- A tier with no archive is **skipped and the next tier tried**, not left to burn the cadence. It
|
||||
cannot starve either — an empty tier is still the least recently proven, so it still sorts first
|
||||
the moment it has an archive.
|
||||
- `POST /backup` now also joins the gate: a **409** naming the holder when a restore-test is running.
|
||||
|
||||
### Tests
|
||||
+12. Red-proofs observed:
|
||||
- **A** — the single-target picker yields `both tiers must be exercised across 4 cadences; got [local:… local:… local:… local:…]`.
|
||||
- **E** — losing the state file yields `after a restart the OTHER tier must be next; got … twice (rotation state was lost)`.
|
||||
- **F** — removing the gate yields `the restore-test must DEFER while a backup holds the gate; concurrent operations = 2`. **The count is the assertion** — "both completed" would pass against a fully concurrent implementation.
|
||||
|
||||
Full suite green (29 packages, `rc=0`, vet unpiped).
|
||||
|
||||
## v0.104.0-dev — R-85 Phase 1: the restore-test spec is built PER RUN (2026-07-26)
|
||||
|
||||
Prerequisite for scheduling the offsite tier at all. Shipped on its own because it is independently
|
||||
|
||||
Reference in New Issue
Block a user