R-85 Phase 2: tier rotation, persisted state, one heavy op at a time

The scheduler could only ever see cfg.Backup.BackupTarget(), so the offsite
tier's archives were never candidates — which is why demo-hp's DR tier reported
'applied' with zero snapshots for five days and nobody noticed.

Selection: oldest-first (operator ruling, Option 1). Never-proven sorts first,
which is where the offsite tier starts. Ties break on target id so ordering is
deterministic rather than following Go's randomised map order. Rotation credit
only on SUCCESS — a permanently failing tier must keep sorting first, not look
freshly proven and stop being retried.

- backup.RestoreTestState: persisted last-success per tier (atomic tmp+rename).
  This genuinely needs persistence unlike R-84: R-84 had ground truth to consult
  (the archive is still on the storage), whereas a restore-test destroys its
  scratch and leaves no artifact. Corrupt/missing file -> 'nothing proven'.
- backup.InFlight: host-wide one-heavy-op gate shared with the local-API backup
  path. A LINK concern, not a lock one — an offsite restore pulls multi-GB over
  the same tunnel a backup pushes one, and at ~33 MB/min both drift toward
  timeout, which is how a healthy tier gets recorded as failed. Callers DEFER,
  never cancel.
- PickRestoreCandidateOn: newest archive on a named tier; '' is not an error, or
  every fresh box looks broken for its first week.
- An empty tier is skipped and the next tried; it cannot starve, since it is
  still least-recently-proven once it has an archive.
- POST /backup joins the gate (409 naming the holder).

Red-proofs A/E/F observed with the documented text. Full suite green (29
packages, rc=0).
This commit is contained in:
Claude Code
2026-07-26 21:00:42 +02:00
parent 765d8b3168
commit 043c7622bc
8 changed files with 743 additions and 18 deletions
+48
View File
@@ -1,3 +1,51 @@
## v0.104.0-dev — R-85 Phase 2: tier rotation, persisted state, one heavy op at a time (2026-07-26)
The scheduler could only ever see `cfg.Backup.BackupTarget()`, so the offsite tier's archives were
never candidates. That is why demo-hp's DR tier reported `applied` with **zero snapshots for five
days** and nobody noticed.
### Selection — oldest-first (operator ruling 2026-07-26, Option 1)
The tier whose last **successful** restore-test is oldest goes first; **never-proven sorts first of
all**, which is exactly where the offsite tier starts. Self-balancing, no new config knob, and each
tier is covered every ~2 cadences — comfortably inside the 2-week offsite retention, so a tier is
never proven against an archive that is about to be pruned. Ties break on target id, because two
tiers proven in the same second would otherwise rotate by Go's randomised map order: untestable, and
occasionally starving.
Rotation credit is given **only on success**. A tier that fails every time must keep sorting first —
otherwise a permanently broken tier would look freshly proven and quietly stop being retried.
### Added
- **`backup.RestoreTestState`** — last successful restore-test per tier, persisted (atomic
tmp+rename). **This one genuinely needs persistence, unlike R-84**, and the difference is worth
stating because they look alike: R-84 had a GROUND TRUTH to consult (the archive is still on the
storage), so it never persisted anything. A restore-test destroys its scratch as its final act and
leaves no artifact — "did we prove this tier restores?" exists only as remembered state. A corrupt
or missing file degrades to "nothing proven", which is the correct starting point.
- **`backup.InFlight`** — the host-wide one-heavy-operation gate, shared by the restore-test
scheduler and the local-API backup path. Not a lock concern (the scratch VMID never touches the
live guest's vzdump lock) but a **LINK** concern: an offsite restore PULLS multi-GB over the same
tunnel an offsite backup PUSHES one. At the ~33 MB/min measured upstream, running both drives each
toward its timeout — which is how a healthy tier ends up recorded as failed. A caller that cannot
acquire **defers**; it never cancels what is already running.
- **`BackupRunner.PickRestoreCandidateOn`** — newest archive on a NAMED tier. `""` + nil error when
that tier holds none: **a tier with nothing to restore is not an error**, or every fresh box would
look broken for its first week.
### Changed
- A tier with no archive is **skipped and the next tier tried**, not left to burn the cadence. It
cannot starve either — an empty tier is still the least recently proven, so it still sorts first
the moment it has an archive.
- `POST /backup` now also joins the gate: a **409** naming the holder when a restore-test is running.
### Tests
+12. Red-proofs observed:
- **A** — the single-target picker yields `both tiers must be exercised across 4 cadences; got [local:… local:… local:… local:…]`.
- **E** — losing the state file yields `after a restart the OTHER tier must be next; got … twice (rotation state was lost)`.
- **F** — removing the gate yields `the restore-test must DEFER while a backup holds the gate; concurrent operations = 2`. **The count is the assertion** — "both completed" would pass against a fully concurrent implementation.
Full suite green (29 packages, `rc=0`, vet unpiped).
## v0.104.0-dev — R-85 Phase 1: the restore-test spec is built PER RUN (2026-07-26)
Prerequisite for scheduling the offsite tier at all. Shipped on its own because it is independently