R-86: restore-test follows the backup, not the clock (v0.121.0)
gates / gates (push) Failing after 7s

The ticker survives as the EVALUATION interval only. A tier is DUE when its
newest archive that has settled for `settle` (default 24h) has not been proven:
daily tier -> proved daily on yesterday's archive, weekly tier -> weekly on its
own, newborn -> UNKNOWN.

The trap avoided: the literal reading ("newest archive is >= 24h old") is NEVER
true on a daily tier, so it silently switches restore-testing off where it
matters most. Red-proved at 0 runs over 5 simulated days.

- state records WHICH archive was proven; legacy files keep their time and yield
  no proven archive (each tier due once after the upgrade, deliberately)
- two knobs replace one: restore_test_eval_interval_seconds (6h, measured) and
  restore_test_settle_seconds (24h). The old cadence key keeps its DISABLE
  meaning verbatim and now seeds the settle lag, with a start-up WARN.
- due-check runs BEFORE the heavy-op gate (a frequent poll must not make a
  starting backup record a failure, F-A1)
- candidate picker skips implausible archives (a phantom would be due forever)
- new read-only --selftest=restore-test-due prints the verdict + its cost
This commit is contained in:
2026-08-03 14:54:57 +02:00
parent 1b14cfd0b4
commit 4618169036
13 changed files with 1273 additions and 123 deletions
+26
View File
@@ -5,6 +5,32 @@
## Current
- **2026-08-03 — v0.121.0 (R-86): the restore-test follows the BACKUP, not the clock.** The ticker is
now only the **evaluation interval**; a tier is **DUE** when its newest archive that has settled for
`settle` (default 24 h) **has not been proven**. Daily tier → proved daily on yesterday's archive;
weekly tier → weekly on its own; newborn → UNKNOWN. **The trap, so it is not reintroduced:** the
literal reading of R-86 — *"due when the newest archive is ≥ 24 h old"* — is NEVER true on a daily
tier (a new archive resets the age before it reaches the lag), so it switches restore-testing off
where it matters most. Red-proved at 0 runs over 5 simulated days.
- **The state now records WHICH archive was proven**, not just when a tier passed. A pre-R-86 file
keeps its time (ordering survives) and yields no proven archive → each tier is due once after the
upgrade, deliberately.
- **The old cadence key:** `restore_test_cadence_seconds` is DEPRECATED. Negative still DISABLES
(verbatim); a positive value now seeds the **settle lag** and the daemon WARNs once at start-up
naming `restore_test_eval_interval_seconds` (default 6 h) and `restore_test_settle_seconds`
(default 24 h). It is NOT carried into the evaluation interval.
- **6 h is bounded from both ends:** measured evaluation cost (local 18 ms, PBS-over-WAN 392 ms,
both 430 ms) says cost is irrelevant; the ceiling is that a FAILING tier stays due, so the
evaluation interval is also its retry interval for a multi-GB restore.
- The due-check now runs **before** the heavy-operation gate is taken (a frequent poll must not be
able to make a starting backup record a failure — F-A1), and the candidate picker skips archives
failing `archivePlausiblyComplete` (a phantom would be due forever and fail forever).
- New read-only `--selftest=restore-test-due` prints the per-tier verdict + its cost.
- **R-185 (filed, NOT fixed here):** on demo-felhom the agent token has no ACL on
`/storage/felhom-backup`, so its content listing comes back EMPTY (root sees 3 archives) — the
host tier has never been restore-testable there, and the due-check cannot distinguish that from
a newborn tier.
- **2026-07-28 — v0.107.0: F-REBOOT fixed — a guest rebooted mid-backup now comes back by itself.**
New `internal/localapi/guestpower.go`: a 60 s watchdog that starts a guest which is `onboot:1`,
stopped, unlocked, and has no vzdump in flight. It closes the two narrow gaps that let