R-86: restore-test follows the backup, not the clock (v0.121.0)
gates / gates (push) Failing after 7s
gates / gates (push) Failing after 7s
The ticker survives as the EVALUATION interval only. A tier is DUE when its
newest archive that has settled for `settle` (default 24h) has not been proven:
daily tier -> proved daily on yesterday's archive, weekly tier -> weekly on its
own, newborn -> UNKNOWN.
The trap avoided: the literal reading ("newest archive is >= 24h old") is NEVER
true on a daily tier, so it silently switches restore-testing off where it
matters most. Red-proved at 0 runs over 5 simulated days.
- state records WHICH archive was proven; legacy files keep their time and yield
no proven archive (each tier due once after the upgrade, deliberately)
- two knobs replace one: restore_test_eval_interval_seconds (6h, measured) and
restore_test_settle_seconds (24h). The old cadence key keeps its DISABLE
meaning verbatim and now seeds the settle lag, with a start-up WARN.
- due-check runs BEFORE the heavy-op gate (a frequent poll must not make a
starting backup record a failure, F-A1)
- candidate picker skips implausible archives (a phantom would be due forever)
- new read-only --selftest=restore-test-due prints the verdict + its cost
This commit is contained in:
@@ -1,3 +1,64 @@
|
||||
## v0.121.0 — a restore-test proves each BACKUP, not the clock (2026-08-03, R-86)
|
||||
|
||||
**The trigger changed; the restore-test did not.** `Scheduler.Run` still has a ticker, but it is now
|
||||
the **evaluation interval** — how often "is anything due?" is asked — and no longer the thing that
|
||||
decides a test happens. What decides is a per-archive due-check
|
||||
(`internal/backup/restoretest_due.go`):
|
||||
|
||||
> Let **A** = the newest archive on this tier that has settled for at least `settle` (default 24 h).
|
||||
> The tier is **DUE** when **A** exists and **A has not already been proven**.
|
||||
|
||||
A daily tier is therefore proved once a day, on yesterday's archive; a weekly tier once a week, on
|
||||
its own; a newborn tier is UNKNOWN and never a fault. Per-archive due-ness IS the pacing — one test
|
||||
per archive generation and no more — so there is deliberately no second rate limiter on top of it.
|
||||
|
||||
**The trap this avoided, recorded because it is the version a reasonable person writes.** R-86's own
|
||||
wording ("~24 h after its own newest archive") implemented literally is *"due when the newest archive
|
||||
is ≥ 24 h old"* — and on a **daily** tier that is never true, because a new archive resets the
|
||||
newest-archive age to zero long before it reaches 24 h. The literal rule silently switches
|
||||
restore-testing OFF for the tier that matters most. It has its own red-proof, which was observed
|
||||
failing with **0 runs over 5 simulated days**.
|
||||
|
||||
**What the fix rests on**
|
||||
|
||||
- **The state records WHICH archive was proven** (`restoretest_state.go`), not merely when a tier last
|
||||
passed — a time cannot answer "have we proven *this* archive". A pre-R-86 state file keeps its time
|
||||
(rotation ordering survives the upgrade) and yields **no** proven archive, so each tier is due
|
||||
exactly once after the upgrade: one extra test per tier, once, which is the safe direction.
|
||||
- **Two knobs replace one, and the old one is not silently repurposed.**
|
||||
`restore_test_eval_interval_seconds` (how often due-ness is asked; default **6 h**) and
|
||||
`restore_test_settle_seconds` (how long an archive must sit; default **24 h**). The deprecated
|
||||
`restore_test_cadence_seconds` keeps its DISABLE meaning (negative) verbatim, and a positive value
|
||||
now seeds the **settle lag** — with a start-up WARN naming both replacements.
|
||||
- **6 h is bounded from both sides, not picked.** MEASURED cost of one evaluation on demo-felhom
|
||||
(Part 1.4): local dir storage **18 ms**, the PBS tier over the WAN to ep0 **392 ms**, both together
|
||||
**430 ms** — cheap enough for minutes, so cost is not the constraint. The **ceiling** is: a tier
|
||||
whose restore-test keeps failing stays due, so the evaluation interval is also its RETRY interval,
|
||||
and a retry is a multi-GB restore.
|
||||
- **The due-check runs BEFORE the heavy-operation gate is taken.** Evaluations are frequent now, and
|
||||
holding that gate for a read that answers "nothing to do" would open a window at every evaluation
|
||||
in which a starting backup cannot acquire — and a backup that cannot acquire records a failure and
|
||||
pages the operator (F-A1). Nothing heavy starts before the gate.
|
||||
- **The candidate picker skips implausible archives.** Under per-archive due-ness an incomplete
|
||||
1-byte phantom (F-CRIT-2's artefact, which server-side prune does not collect) would be picked
|
||||
forever, fail forever, never earn proof, and leave the tier due at EVERY evaluation — turning the
|
||||
evaluation interval into the retry rate for a multi-GB restore. `PickRestoreCandidateOn` now
|
||||
delegates to the settle-aware picker, so both callers agree.
|
||||
|
||||
**Unchanged, deliberately:** the restore-test itself (restore → boot → verify → destroy the scratch),
|
||||
its journal, crash recovery, the scratch VMID band, the one-heavy-operation gate, success-only proof
|
||||
credit, and oldest-proven ordering — which survives as the tie-break **between due tiers**.
|
||||
|
||||
**New:** `--selftest=restore-test-due` — read-only; prints the per-tier due verdict the scheduler
|
||||
would act on, with the measured cost of the lookup.
|
||||
|
||||
**Live finding, pre-existing and NOT caused by this change (filed as R-185):** on demo-felhom the
|
||||
agent's PVE token has no ACL on `/storage/felhom-backup`, so the API returns an EMPTY content listing
|
||||
for that storage (root sees three archives). The host tier has therefore never been restore-testable
|
||||
on that box, and both R-85's rotation and R-86's due-check report it indistinguishably from "newborn"
|
||||
("no settled archive yet"). Verified live against `local` (grant present → 3 archives) and
|
||||
`felhom-backup` (no grant → `{"data":[]}`).
|
||||
|
||||
## Releasing publishes, and an unreleasable version cannot pass CI (2026-08-03, R-115 + R-183) — **NO VERSION BUMP**
|
||||
|
||||
**No Go code changed, so nothing is bumped and nothing was built.** This is the release path and a
|
||||
|
||||
Reference in New Issue
Block a user