R-100: offsite staleness counts from the last SUCCESS (hub v0.80.0)

isStale counted from last_run, written unconditionally on failure, so a nightly-failing
tier read as fresh forever. Now anchored on last_success with an explicit legacy degrade
(logged once) and the never-ran branch untouched. emitStale states the real reason.
This commit is contained in:
2026-07-28 13:17:04 +02:00
parent 6369570e8d
commit b505ee9125
4 changed files with 405 additions and 16 deletions
+41
View File
@@ -1,3 +1,44 @@
## v0.80.0 — R-100: staleness counts from the last SUCCESS (2026-07-28)
`OffsiteChecker.isStale` counted from `last_run`, which the controller writes **unconditionally** at the
end of every run including failures. It therefore asked *"how long since we last TRIED"* — so a tier
failing on every single night refreshed the clock nightly and read as perfectly fresh forever. It now
counts from **`last_success`** (controller v0.181.0).
**What the defect is NOT, corrected after checking.** The old comment here said *"a recent-but-failing
run is NOT stale (backup_failed owns that signal)"*, and that was **true**`backup_failed` does fire
for a failing offsite run, nightly, and reaches the operator (live hub DB: 5 operator sends). The real
defect is **defeated defence in depth**: this checker is the hub-side, *pull-based* net that exists to be
independent of controller-*pushed* events, and anchoring it on a field the failing controller keeps
refreshing made it depend on the very thing it backs up. F-HUB — this campaign's own finding, the hub
dropping an event under `SQLITE_BUSY` with no retry — is exactly that loss.
**Three branches, each deliberate:**
- **never ran** (no `last_run`) — unchanged v0.73.0 anchored behaviour. Still keyed on `last_run`, not
`last_success`, on purpose: `last_run` answers "has anything ever happened here", and a box whose
*first* run failed has a `last_run` and no `last_success` — that is a run, not a newborn.
- **legacy** (`last_run` set, no `last_success`) — degrades **explicitly** to the old `last_run`
behaviour, logged **once** per customer. Treating absence as failure would alarm the whole
un-upgraded fleet at once; treating it as success keeps the bug. Same degrade direction as R-88
Part 2's `age_state`.
- **anchored** — counts from `last_success`. `last_status` is deliberately **not** consulted:
"error ⇒ stale" pages on every transient blip, which is the F-A1 noise path. One bad night is
tolerated because the threshold simply keeps running from the last good run. `running` is a real wire
value (a report captured mid-run) and is likewise not a verdict.
**The alarm text had to change with the verdict.** `emitStale` still said `last run 8h ago` while firing
on a six-day-old success — a true alarm that reads as a false one. `staleAge` now separates the two
diagnoses: *"runs are happening and failing — check the error, not the schedule"* versus *"the offsite
leg is silently not running"*. `last_success` joins the event details.
Fixtures are the **real** wire shapes from 4000 live reports (`ok` ×2269, absent ×541 always with an
empty `last_run`, `error` ×27, `running` ×7), not invented JSON.
Red-proofs, all observed failing: restore the `last_run` anchor → `a tier that has not succeeded in 6
days reads as FRESH`; delete the never-ran branch → `a newborn box alarmed`; collapse to
`last_status == "error"``a single transient failure alarmed`; delete the legacy degrade → `a legacy
controller alarmed — that is a fleet-wide alarm storm on an un-upgraded fleet`.
# Felhom Hub — Changelog
## v0.79.0 — R-97c: make the operator-only claim TRUE (2026-07-27)