hub v0.75.0: R-81 — "no signal" is not "bad signal" (anchor the backup deadline check)
Third instance of one class (hub v0.12.0, v0.73.0, this), fixed as a class. On 2026-07-26 03:00 UTC expected_backup_missed fired on demo-felhom, demo-hp and drill-r50 at once; the demo-felhom one reached the CUSTOMER channel claiming "newest backup is 176h0m0s old". Nothing was wrong — three vzdump archives were on disk. Cause: the agent backup store is in-memory, so the R-50 fleet restart emptied `backups` until the next run, and the hub read empty as "no backup exists". - assessBackupFreshness returns OK/UNKNOWN/MISSED instead of `missed bool`; absence is UNKNOWN until it outlives an anchored window. Still pure. - store.GetHostReportsSince + monitor.newestBackupEvidence read the hubs own retained history (bounded 7-day lookback, early-exit on fresh evidence) — "when did I last SEE evidence of a backup?" The anchor was free: the hub already retains 90 days. No agent change, no new persisted state. - store.GetFirstHostReportAt anchors absence at first contact, reusing the existing 26h threshold as the grace (no new knob, the v0.73.0 shape). - Deferrals logged + counted; reason strings kept distinct. - backupStaleAfter untouched; landmine recorded (a weekly PBS snapshot would alarm six days in seven) and owned by R-82. Tests 493->508. Red-proofs A/B/C observed and restored; A reproduces the live message verbatim. Replayed the real 03:00 reports (600/417/77 rows): all three now silent. Source: documentation/audits/DIAG-backup-missed-2026-07-26.md
This commit is contained in:
Claude Code
@@ -1,5 +1,82 @@
|
||||
# Felhom Hub — Changelog
|
||||
|
||||
## v0.75.0 — R-81: "no signal" is not "bad signal" — the backup deadline check is ANCHORED (2026-07-26)
|
||||
|
||||
The third instance of one bug class, fixed as a class. `expected_backup_missed` fired on
|
||||
demo-felhom, demo-hp and drill-r50 simultaneously at 03:00 UTC, and the demo-felhom one reached the
|
||||
**customer** channel claiming `newest backup is 176h0m0s old`. Nothing was wrong: three vzdump
|
||||
archives were on disk (07-24, 07-25, 07-26). Full evidence:
|
||||
`documentation/audits/DIAG-backup-missed-2026-07-26.md`.
|
||||
|
||||
**Cause.** The agent's backup record store is IN-MEMORY
|
||||
(`felhom-agent/internal/backup/store.go` — *"lost on restart; the cadence re-populates"*). The R-50
|
||||
island migration restarted the fleet at 12:44 UTC; the next backup landed at 07:03 the following
|
||||
morning. In between, every host-report carried `backups: []`, and `assessBackupFreshness` read empty
|
||||
as *no backup exists*. For demo-felhom it then fell through to the only surviving evidence — a PBS
|
||||
snapshot from 07-18 — and reported its age as the customer's backup age.
|
||||
|
||||
**The class.** hub v0.12.0 (`expected_backup_missed` daily for every healthy customer — looked for
|
||||
an event nobody emits), hub v0.73.0 (`offsite_stale` minutes after a *healthy* repair — never-ran
|
||||
branch had no time anchor), and now this. All three: **absence of signal treated as evidence of
|
||||
failure.** The invariant is now written at the head of `assessBackupFreshness` with all three
|
||||
instances named, and pinned by a boundary test whose name says what it protects.
|
||||
|
||||
### Changed
|
||||
- **`assessBackupFreshness` returns a three-valued verdict** — `verdictOK` / `verdictUnknown` /
|
||||
`verdictMissed`, replacing `missed bool`. Absence is UNKNOWN, not a fault. It becomes a fault only
|
||||
once it outlives an anchored window. Still **pure** (`now` and the evidence are injected) — that
|
||||
purity is why the incident was diagnosable and why this fix is provable.
|
||||
- **The check now reads hub HISTORY, not just the latest report.** New
|
||||
`store.GetHostReportsSince` + `monitor.newestBackupEvidence` answer *"when did I last SEE evidence
|
||||
of a backup?"* across a bounded 7-day lookback (`backupEvidenceLookback`). The agent's store is
|
||||
point-in-time and forgets across a restart; the hub's retained reports (90 d) do not. **This is the
|
||||
whole fix for the 07-26 shape** — no agent change, no new persisted state, and semantically exactly
|
||||
the right question. The scan stops at the first sufficiently-fresh evidence, so the healthy path
|
||||
reads one row; only the genuinely-broken path walks the lookback.
|
||||
- **The absence anchor is first contact** — new `store.GetFirstHostReportAt`. Absence is graded
|
||||
against how long the hub has been *watching*, reusing the existing `backupStaleAfter` (26 h) as the
|
||||
grace exactly as v0.73.0 reused offsite `staleAfter`. **No new knob.** A zero anchor fails toward
|
||||
visibility (the v0.73.0 legacy-shape precedent).
|
||||
- **`CheckBackupDeadlines` logs the deferral.** A deferred UNKNOWN emits one INFO naming the reason,
|
||||
and the summary line gained a `backup unknown (deferred)` counter — so a quiet check is never
|
||||
indistinguishable from a check that did not run (v0.73.0 Part-7 precedent). At most one line per
|
||||
customer per day.
|
||||
- **Reason strings split, not collapsed.** Absence-over-time, unanchored absence, deferred-newborn,
|
||||
stale-timestamp, failed-verify and unparseable each keep a distinct message. The entire 07-26
|
||||
diagnosis turned on reading the exact string; a test enforces distinctness.
|
||||
|
||||
### NOT changed (deliberate)
|
||||
- `backupStaleAfter` stays 26 h, and no tier-awareness was built. ⚠️ **Landmine recorded in the
|
||||
constant's comment:** it applies to whichever tier is newest, so once PBS moves to a **weekly**
|
||||
cadence a healthy weekly snapshot is >26 h old six days in seven and this will alarm on it.
|
||||
Per-tier thresholds cannot be built before the per-tier cadence config exists — **R-82 owns both
|
||||
halves.** Building it now would be speculative generality.
|
||||
- `parseBackupTime` untouched — the agent emits clean RFC3339 `Z` and the parse branch is not
|
||||
implicated. Its silent `continue` on an unparseable timestamp is a **latent member of this same
|
||||
class** and is recorded as an observation only.
|
||||
- The DB-dump half of `CheckBackupDeadlines` is event-based and correct — untouched.
|
||||
- The customer-facing Hungarian copy (`notify/templates.go:106`) is untouched here. The DIAG found it
|
||||
overstates scope (it reads as *all* backups failed, but this check only covers the host/PBS tier);
|
||||
that copy change was not in this task's scope.
|
||||
- The agent's in-memory `Store` is the *cause*; making the host-report truthful rather than merely
|
||||
defensively interpreted is **R-84**.
|
||||
|
||||
### Tests
|
||||
508 total (was 493), +15 in `internal/monitor/deadline_anchor_test.go`. Companion red-proofs observed
|
||||
and restored for all three acceptance scenarios:
|
||||
- **A** (restart blind window must not alarm) — removing the history fold-in reproduces
|
||||
`newest backup is 176h0m0s old (limit 26h0m0s)`, **verbatim the message demo-felhom actually sent**.
|
||||
- **B** (a genuinely dead box must still alarm) — the naive "absence is always silent" fix fails
|
||||
`TestBackupFreshness_NoEvidenceBeyondAnchor_Alarms` and two contract rows. This is the test that
|
||||
makes A safe: a suite proving only A would pass against an implementation that never alarms.
|
||||
- **C** (a fresh box is not born failing) — the literal pre-fix branch fails four contract rows plus
|
||||
the end-to-end newborn case.
|
||||
|
||||
**Replayed against the real thing:** the actual host-reports the hub held at 2026-07-26 03:00 UTC
|
||||
(600 / 417 / 77 retained rows) fed through the new policy → demo-felhom **OK** (window evidence
|
||||
`2026-07-25T06:30:14Z`), demo-hp **OK** (`2026-07-25T10:23:31Z`), drill-r50 **UNKNOWN** (newborn,
|
||||
watched 17 h < 26 h grace). **All three silent.**
|
||||
|
||||
## v0.74.0 — allow `local_api_endpoint_drift` (controller v0.173.0 / R-77) (2026-07-26)
|
||||
|
||||
One line in `allowedEventTypes`. It is **not optional**: `handleEvent` 400s an unknown `event_type`
|
||||
|
||||
Reference in New Issue
Block a user