R-100: offsite staleness counts from the last SUCCESS (hub v0.80.0)
isStale counted from last_run, written unconditionally on failure, so a nightly-failing tier read as fresh forever. Now anchored on last_success with an explicit legacy degrade (logged once) and the never-ran branch untouched. emitStale states the real reason.
This commit is contained in:
@@ -1,3 +1,44 @@
|
||||
## v0.80.0 — R-100: staleness counts from the last SUCCESS (2026-07-28)
|
||||
|
||||
`OffsiteChecker.isStale` counted from `last_run`, which the controller writes **unconditionally** at the
|
||||
end of every run including failures. It therefore asked *"how long since we last TRIED"* — so a tier
|
||||
failing on every single night refreshed the clock nightly and read as perfectly fresh forever. It now
|
||||
counts from **`last_success`** (controller v0.181.0).
|
||||
|
||||
**What the defect is NOT, corrected after checking.** The old comment here said *"a recent-but-failing
|
||||
run is NOT stale (backup_failed owns that signal)"*, and that was **true** — `backup_failed` does fire
|
||||
for a failing offsite run, nightly, and reaches the operator (live hub DB: 5 operator sends). The real
|
||||
defect is **defeated defence in depth**: this checker is the hub-side, *pull-based* net that exists to be
|
||||
independent of controller-*pushed* events, and anchoring it on a field the failing controller keeps
|
||||
refreshing made it depend on the very thing it backs up. F-HUB — this campaign's own finding, the hub
|
||||
dropping an event under `SQLITE_BUSY` with no retry — is exactly that loss.
|
||||
|
||||
**Three branches, each deliberate:**
|
||||
- **never ran** (no `last_run`) — unchanged v0.73.0 anchored behaviour. Still keyed on `last_run`, not
|
||||
`last_success`, on purpose: `last_run` answers "has anything ever happened here", and a box whose
|
||||
*first* run failed has a `last_run` and no `last_success` — that is a run, not a newborn.
|
||||
- **legacy** (`last_run` set, no `last_success`) — degrades **explicitly** to the old `last_run`
|
||||
behaviour, logged **once** per customer. Treating absence as failure would alarm the whole
|
||||
un-upgraded fleet at once; treating it as success keeps the bug. Same degrade direction as R-88
|
||||
Part 2's `age_state`.
|
||||
- **anchored** — counts from `last_success`. `last_status` is deliberately **not** consulted:
|
||||
"error ⇒ stale" pages on every transient blip, which is the F-A1 noise path. One bad night is
|
||||
tolerated because the threshold simply keeps running from the last good run. `running` is a real wire
|
||||
value (a report captured mid-run) and is likewise not a verdict.
|
||||
|
||||
**The alarm text had to change with the verdict.** `emitStale` still said `last run 8h ago` while firing
|
||||
on a six-day-old success — a true alarm that reads as a false one. `staleAge` now separates the two
|
||||
diagnoses: *"runs are happening and failing — check the error, not the schedule"* versus *"the offsite
|
||||
leg is silently not running"*. `last_success` joins the event details.
|
||||
|
||||
Fixtures are the **real** wire shapes from 4000 live reports (`ok` ×2269, absent ×541 always with an
|
||||
empty `last_run`, `error` ×27, `running` ×7), not invented JSON.
|
||||
|
||||
Red-proofs, all observed failing: restore the `last_run` anchor → `a tier that has not succeeded in 6
|
||||
days reads as FRESH`; delete the never-ran branch → `a newborn box alarmed`; collapse to
|
||||
`last_status == "error"` → `a single transient failure alarmed`; delete the legacy degrade → `a legacy
|
||||
controller alarmed — that is a fleet-wide alarm storm on an un-upgraded fleet`.
|
||||
|
||||
# Felhom Hub — Changelog
|
||||
|
||||
## v0.79.0 — R-97c: make the operator-only claim TRUE (2026-07-27)
|
||||
|
||||
Reference in New Issue
Block a user