hub v0.75.0: R-81 — "no signal" is not "bad signal" (anchor the backup deadline check)
Third instance of one class (hub v0.12.0, v0.73.0, this), fixed as a class. On 2026-07-26 03:00 UTC expected_backup_missed fired on demo-felhom, demo-hp and drill-r50 at once; the demo-felhom one reached the CUSTOMER channel claiming "newest backup is 176h0m0s old". Nothing was wrong — three vzdump archives were on disk. Cause: the agent backup store is in-memory, so the R-50 fleet restart emptied `backups` until the next run, and the hub read empty as "no backup exists". - assessBackupFreshness returns OK/UNKNOWN/MISSED instead of `missed bool`; absence is UNKNOWN until it outlives an anchored window. Still pure. - store.GetHostReportsSince + monitor.newestBackupEvidence read the hubs own retained history (bounded 7-day lookback, early-exit on fresh evidence) — "when did I last SEE evidence of a backup?" The anchor was free: the hub already retains 90 days. No agent change, no new persisted state. - store.GetFirstHostReportAt anchors absence at first contact, reusing the existing 26h threshold as the grace (no new knob, the v0.73.0 shape). - Deferrals logged + counted; reason strings kept distinct. - backupStaleAfter untouched; landmine recorded (a weekly PBS snapshot would alarm six days in seven) and owned by R-82. Tests 493->508. Red-proofs A/B/C observed and restored; A reproduces the live message verbatim. Replayed the real 03:00 reports (600/417/77 rows): all three now silent. Source: documentation/audits/DIAG-backup-missed-2026-07-26.md
This commit is contained in:
Claude Code
+39
@@ -3,6 +3,45 @@
|
||||
> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
|
||||
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
|
||||
|
||||
## Standing rulings
|
||||
|
||||
**S-1 — N.5 gains a third leg: architecture docs are same-session coupled (2026-07-26, R-81).**
|
||||
Any task that changes an **architectural contract** — tiers, targets, cadences, trust boundaries —
|
||||
updates the owning `documentation/architecture/*.md` in the **same session**, under exactly the same
|
||||
coupling rule that already binds the capability map and the ROADMAP. Origin: R-80/R-81 found
|
||||
`07-backup-architecture.md` still describing a backup model that the shipped code does not implement
|
||||
(single target, single cadence), while being cited as authoritative. A stale architecture doc is
|
||||
worse than a missing one, because it is trusted.
|
||||
|
||||
**S-2 — architecture docs carry an honest status header (2026-07-26, R-81).**
|
||||
Every `documentation/architecture/*.md` opens with the version it was **verified against** and the
|
||||
date. A doc more than a few trains behind its subject is marked **STALE** *in that header*, so a
|
||||
reader meets the warning before the content, not after acting on it. Origin:
|
||||
`07-backup-architecture.md` — DRAFT since 2026-07-14, verified against controller v0.132.0, now 41
|
||||
versions stale (live v0.173.0), and cited as authoritative throughout the R-80 diagnostic. Ratifying
|
||||
or retiring it is → **R-83**.
|
||||
|
||||
- **2026-07-26 — hub v0.75.0: R-81 SHIPPED — "no signal" is not "bad signal".** Third instance of
|
||||
one class (hub v0.12.0 `expected_backup_missed` daily-for-everyone; hub v0.73.0 `offsite_stale`
|
||||
minutes after a healthy repair; now this), fixed **as a class**, with the invariant written at the
|
||||
head of `assessBackupFreshness` naming all three. `assessBackupFreshness` returns
|
||||
OK/UNKNOWN/MISSED instead of `missed bool`; absence is UNKNOWN until it outlives an anchored
|
||||
window. **The anchor was free** — Phase-0 probe found the hub retains 90 days of host-reports, so
|
||||
`GetHostReportsSince` + `newestBackupEvidence` (7-day bounded lookback, early-exit on fresh
|
||||
evidence) answer *"when did I last SEE evidence of a backup?"* rather than *"what does the latest
|
||||
report say?"*; absence is graded from first contact (`GetFirstHostReportAt`) reusing the existing
|
||||
26 h threshold as the grace — **no new knob, no agent change, no new persisted state**.
|
||||
**Rulings:** absence is never silent (a genuinely dead box must still alarm — that is the half the
|
||||
naive fix breaks, and `TestBackupFreshness_NoEvidenceBeyondAnchor_Alarms` is what makes the
|
||||
suppression safe); reason strings stay distinct (the whole 07-26 diagnosis turned on reading the
|
||||
exact string); deferrals are logged so quiet ≠ not-checked. Tests 493→508; red-proofs A/B/C
|
||||
observed — A reproduces `newest backup is 176h0m0s old` **verbatim the mail demo-felhom sent**.
|
||||
**Replayed against the real 03:00 reports** (600/417/77 rows): all three now silent. `backupStaleAfter`
|
||||
deliberately untouched — ⚠️ it will alarm on a healthy WEEKLY PBS snapshot, landmine recorded,
|
||||
owned by **R-82**. The real finding behind R-80 stands: the **PBS/offsite-DR tier has no schedule
|
||||
at all** → R-82. Cause-fix (persist the agent store) → R-84. Evidence:
|
||||
`documentation/audits/DIAG-backup-missed-2026-07-26.md`, `hub/CHANGELOG.md` v0.75.0.
|
||||
|
||||
- **2026-07-23 — hub v0.72.0 + controller v0.161.0: R-70 SHIPPED + R-71(c) PARTIAL (the offsite
|
||||
last mile is visible; burned credentials self-heal).** One detector
|
||||
(`offsite.DeliveryStateFor` — secret-row timestamps × report offsite-presence), four consumers:
|
||||
|
||||
@@ -131,6 +131,7 @@
|
||||
| `(*Store).RequestLogBundle` / `PendingLogBundleRequest` / `SaveLogBundle` / `PurgeExpiredLogBundles` | hub/internal/store/logbundle.go | component (controller/agent) log pulls — the v0.46.0 sibling of logtail.go | box-component debug-ring pulls; gzip custody, newest-3, 72 h TTL on the 60 s sweep | scope = customer_id (controller/report ACK) vs host_id (agent/heartbeat envelope); `SaveLogBundle` runs the SECRET GATE fail-closed (blocked flag row, no payload) and clears the request in the same tx; `[REDACTED]`/checksums pass by design |
|
||||
| `upsertAppIssue` dismissal/context semantics | hub/internal/store/telemetry.go | ON CONFLICT CASE guards | Issue dismissal + first-capture-wins context | Un-dismiss ONLY on `excluded.last_seen > dismissed_at`; context adopted only while stored one is empty — do not "simplify" either CASE (red-proofed) |
|
||||
| `store.GuestID` | hub/internal/store/store.go (~L1268) | `(hostID string, vmid int) string` | Canonical guest primary key | Never hand-concatenate host+vmid. |
|
||||
| `(*Store).GetHostReportsSince` + `GetFirstHostReportAt` + `monitor.newestBackupEvidence` | hub/internal/store/store.go, hub/internal/monitor/deadline.go | `(customerID, since) ([]HostReportRow, error)`; `(customerID) (time.Time, error)`; `(rows, now) (time.Time, bool)` | **Asking "when did the hub last SEE evidence of X?" instead of "what does the latest report say?"** — the R-81 anchor. The agent's reporters are point-in-time and forget across a restart; the hub retains ~90 d of host-reports and does not. | The three go together: window scan + first-contact anchor + a bounded lookback (`backupEvidenceLookback`). **Never judge a report-derived absence on the LATEST report alone** — that is the bug class R-81 fixed for the third time. The scan early-exits on sufficiently-fresh evidence, so don't reorder rows away from newest-first. |
|
||||
| `scheduleDaily` | hub/cmd/hub/main.go (~L449) | `(ctx, name, "HH:MM", fn, logger)` | Daily jobs in Europe/Budapest (prune etc.) | Blocking — run as goroutine. `parseHM` returns 0,0 (midnight) on bad input. |
|
||||
|
||||
## 2. Canonical patterns (copy structure from THE named file)
|
||||
@@ -156,6 +157,7 @@
|
||||
|
||||
| Trap | Why it bites | Use instead |
|
||||
|---|---|---|
|
||||
| A plain `missed bool` for a report-derived absence (hub/internal/monitor/deadline.go) | Collapsing the three-valued verdict re-introduces one of TWO failure modes: absence→MISSED is the 2026-07-26 cry-wolf (three boxes alarmed at once, one reached a customer channel); absence→OK means a genuinely dead box alarms NEVER, which is strictly worse. Three instances of this class so far: hub v0.12.0, v0.73.0, R-81. | `backupAssessment{verdict: verdictOK|verdictUnknown|verdictMissed}` + an anchored window — copy the shape from `assessBackupFreshness`, not a bool. |
|
||||
| `(*Handler).handleNotify` + `formatNotificationEmail` + `sendResendEmail` (hub/internal/api/handler.go ~L1289/1624/1589) | Legacy pre-dispatcher notification trio: no cooldowns, no operator channel, no allowedEventTypes gate, duplicate Hungarian formatter. Controller path is FROZEN until slice-10 cutover. | `POST /api/v1/event` → `Dispatcher.ProcessEvent` + `notify.Format*Email` |
|
||||
| Severity `"critical"` POSTed to a PRE-v0.31.0 hub | Fixed in hub v0.31.0 (`handleEvent` now accepts critical). Older hubs coerce `critical` → `"info"`, which never notifies — silent alert loss. Case-variants (`"Critical"`) still coerce on every version. | Against an old hub send `warning`/`error`; otherwise lowercase `critical` is safe |
|
||||
| `compareVersions` for anything security-ish (hub/internal/web/server.go ~L571) | Returns 0 (equal) on unparseable input — a garbage version passes a floor check. `gitea.compareSemver` behaves differently (lexical fallback). | Validate input with `normalizeFloorInput` first; then compareVersions is safe |
|
||||
|
||||
@@ -111,7 +111,8 @@
|
||||
| Paired recovery notifications + prefs seeding at claim + priority headers (power-outage audit F11/F12/F14-light) | hub v0.71.0 | **IMPLEMENTED** (recovery leg **PARTIAL** until a live staleness cycle fires it) | `hub/CHANGELOG.md` v0.71.0; 17 tests + 4 red-proofs (`REPORT.md` 2026-07-22); Resend `headers` mechanism probed live (HTTP 200) pre-implementation; operator+customer `test` rows live-fired via the controller's own test endpoint | Recovery = explicit eventType branch, severity semantics frozen; customer gate = PAIRING (`notification_log` evidence), not `enabled_events`. Live legs pending: a natural `*_recovered` mail (next real staleness cycle or the reboot-drill arc — never fabricated by blocking reports) and seed-at-claim on a real claim (Peti Friday reinstall). F14-full (operator push channel, ntfy/Telegram) stays open → R-69 |
|
||||
| System + container metrics (SQLite, Chart.js, 30-day downsampling) | controller | **IMPLEMENTED** | metrics collection + `/monitoring` render present (page 200) | The cited `CAMPAIGN-2` T-RES-CPU/T-SOAK-LOOP are H1/H2 harness artifacts (auth-302), not metrics tests; SQLite/Chart.js/30-day downsampling validated in no campaign. Demoted |
|
||||
| Always-on debug rings + on-demand log-bundle pulls with TTL/custody | controller v0.116, agent v0.83, hub v0.46 | **PROVEN-LIVE** | debug rings live-exercised `CAMPAIGN-3` fix-6 (1000-cap ring, ~55min horizon under load) | The **log-bundle-pull TTL/custody** half is changelog-only (no dedicated observability audit doc); ring persistence across restart is a known gap |
|
||||
| Operator alerting (Healthchecks → monitoring@felhom.eu) | k3s, Resend | **IMPLEMENTED** | operator infra, stated in production since 02-04; no corpus validation doc | Per the status enum, no citation → not PROVEN-LIVE. Demoted pending an operator-cited live alert (candidate re-upgrade — see REPORT) |
|
||||
| Operator alerting (Healthchecks → monitoring@felhom.eu) | k3s, Resend | **IMPLEMENTED** | operator infra, stated in production since 02-04; no corpus validation doc |
|
||||
| Backup-deadline alerting (`expected_backup_missed`) is ANCHORED — absence of signal is UNKNOWN, not failure | hub v0.75.0 | **IMPLEMENTED** | `audits/DIAG-backup-missed-2026-07-26.md` + red-proofs A/B/C + replay of the real 2026-07-26 03:00 reports (all three silent) | **No row status flips** — this signal had a FALSE-POSITIVE class (three instances: hub v0.12.0, v0.73.0, R-81), now anchored at first contact and read across retained host-report history. Still unit-proven only, not live-fired at a real deadline. The *underlying* PBS/offsite-DR tier gap it exposed is → R-82. | Per the status enum, no citation → not PROVEN-LIVE. Demoted pending an operator-cited live alert (candidate re-upgrade — see REPORT) |
|
||||
|
||||
## G. Fleet & operator (hub)
|
||||
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -1,5 +1,82 @@
|
||||
# Felhom Hub — Changelog
|
||||
|
||||
## v0.75.0 — R-81: "no signal" is not "bad signal" — the backup deadline check is ANCHORED (2026-07-26)
|
||||
|
||||
The third instance of one bug class, fixed as a class. `expected_backup_missed` fired on
|
||||
demo-felhom, demo-hp and drill-r50 simultaneously at 03:00 UTC, and the demo-felhom one reached the
|
||||
**customer** channel claiming `newest backup is 176h0m0s old`. Nothing was wrong: three vzdump
|
||||
archives were on disk (07-24, 07-25, 07-26). Full evidence:
|
||||
`documentation/audits/DIAG-backup-missed-2026-07-26.md`.
|
||||
|
||||
**Cause.** The agent's backup record store is IN-MEMORY
|
||||
(`felhom-agent/internal/backup/store.go` — *"lost on restart; the cadence re-populates"*). The R-50
|
||||
island migration restarted the fleet at 12:44 UTC; the next backup landed at 07:03 the following
|
||||
morning. In between, every host-report carried `backups: []`, and `assessBackupFreshness` read empty
|
||||
as *no backup exists*. For demo-felhom it then fell through to the only surviving evidence — a PBS
|
||||
snapshot from 07-18 — and reported its age as the customer's backup age.
|
||||
|
||||
**The class.** hub v0.12.0 (`expected_backup_missed` daily for every healthy customer — looked for
|
||||
an event nobody emits), hub v0.73.0 (`offsite_stale` minutes after a *healthy* repair — never-ran
|
||||
branch had no time anchor), and now this. All three: **absence of signal treated as evidence of
|
||||
failure.** The invariant is now written at the head of `assessBackupFreshness` with all three
|
||||
instances named, and pinned by a boundary test whose name says what it protects.
|
||||
|
||||
### Changed
|
||||
- **`assessBackupFreshness` returns a three-valued verdict** — `verdictOK` / `verdictUnknown` /
|
||||
`verdictMissed`, replacing `missed bool`. Absence is UNKNOWN, not a fault. It becomes a fault only
|
||||
once it outlives an anchored window. Still **pure** (`now` and the evidence are injected) — that
|
||||
purity is why the incident was diagnosable and why this fix is provable.
|
||||
- **The check now reads hub HISTORY, not just the latest report.** New
|
||||
`store.GetHostReportsSince` + `monitor.newestBackupEvidence` answer *"when did I last SEE evidence
|
||||
of a backup?"* across a bounded 7-day lookback (`backupEvidenceLookback`). The agent's store is
|
||||
point-in-time and forgets across a restart; the hub's retained reports (90 d) do not. **This is the
|
||||
whole fix for the 07-26 shape** — no agent change, no new persisted state, and semantically exactly
|
||||
the right question. The scan stops at the first sufficiently-fresh evidence, so the healthy path
|
||||
reads one row; only the genuinely-broken path walks the lookback.
|
||||
- **The absence anchor is first contact** — new `store.GetFirstHostReportAt`. Absence is graded
|
||||
against how long the hub has been *watching*, reusing the existing `backupStaleAfter` (26 h) as the
|
||||
grace exactly as v0.73.0 reused offsite `staleAfter`. **No new knob.** A zero anchor fails toward
|
||||
visibility (the v0.73.0 legacy-shape precedent).
|
||||
- **`CheckBackupDeadlines` logs the deferral.** A deferred UNKNOWN emits one INFO naming the reason,
|
||||
and the summary line gained a `backup unknown (deferred)` counter — so a quiet check is never
|
||||
indistinguishable from a check that did not run (v0.73.0 Part-7 precedent). At most one line per
|
||||
customer per day.
|
||||
- **Reason strings split, not collapsed.** Absence-over-time, unanchored absence, deferred-newborn,
|
||||
stale-timestamp, failed-verify and unparseable each keep a distinct message. The entire 07-26
|
||||
diagnosis turned on reading the exact string; a test enforces distinctness.
|
||||
|
||||
### NOT changed (deliberate)
|
||||
- `backupStaleAfter` stays 26 h, and no tier-awareness was built. ⚠️ **Landmine recorded in the
|
||||
constant's comment:** it applies to whichever tier is newest, so once PBS moves to a **weekly**
|
||||
cadence a healthy weekly snapshot is >26 h old six days in seven and this will alarm on it.
|
||||
Per-tier thresholds cannot be built before the per-tier cadence config exists — **R-82 owns both
|
||||
halves.** Building it now would be speculative generality.
|
||||
- `parseBackupTime` untouched — the agent emits clean RFC3339 `Z` and the parse branch is not
|
||||
implicated. Its silent `continue` on an unparseable timestamp is a **latent member of this same
|
||||
class** and is recorded as an observation only.
|
||||
- The DB-dump half of `CheckBackupDeadlines` is event-based and correct — untouched.
|
||||
- The customer-facing Hungarian copy (`notify/templates.go:106`) is untouched here. The DIAG found it
|
||||
overstates scope (it reads as *all* backups failed, but this check only covers the host/PBS tier);
|
||||
that copy change was not in this task's scope.
|
||||
- The agent's in-memory `Store` is the *cause*; making the host-report truthful rather than merely
|
||||
defensively interpreted is **R-84**.
|
||||
|
||||
### Tests
|
||||
508 total (was 493), +15 in `internal/monitor/deadline_anchor_test.go`. Companion red-proofs observed
|
||||
and restored for all three acceptance scenarios:
|
||||
- **A** (restart blind window must not alarm) — removing the history fold-in reproduces
|
||||
`newest backup is 176h0m0s old (limit 26h0m0s)`, **verbatim the message demo-felhom actually sent**.
|
||||
- **B** (a genuinely dead box must still alarm) — the naive "absence is always silent" fix fails
|
||||
`TestBackupFreshness_NoEvidenceBeyondAnchor_Alarms` and two contract rows. This is the test that
|
||||
makes A safe: a suite proving only A would pass against an implementation that never alarms.
|
||||
- **C** (a fresh box is not born failing) — the literal pre-fix branch fails four contract rows plus
|
||||
the end-to-end newborn case.
|
||||
|
||||
**Replayed against the real thing:** the actual host-reports the hub held at 2026-07-26 03:00 UTC
|
||||
(600 / 417 / 77 retained rows) fed through the new policy → demo-felhom **OK** (window evidence
|
||||
`2026-07-25T06:30:14Z`), demo-hp **OK** (`2026-07-25T10:23:31Z`), drill-r50 **UNKNOWN** (newborn,
|
||||
watched 17 h < 26 h grace). **All three silent.**
|
||||
|
||||
## v0.74.0 — allow `local_api_endpoint_drift` (controller v0.173.0 / R-77) (2026-07-26)
|
||||
|
||||
One line in `allowedEventTypes`. It is **not optional**: `handleEvent` 400s an unknown `event_type`
|
||||
|
||||
@@ -14,8 +14,27 @@ import (
|
||||
// deadline check raises expected_backup_missed. 26h covers an evening backup schedule
|
||||
// (e.g. ~18:00–22:00) plus headroom, so a healthy once-daily cadence never trips the
|
||||
// early-morning check.
|
||||
//
|
||||
// R-81 also reuses it as the ABSENCE window (see assessBackupFreshness): the existing
|
||||
// threshold, anchored at first-contact, IS the newborn grace — no new knob, exactly as
|
||||
// v0.73.0 reused offsite staleAfter for its never-ran anchor.
|
||||
//
|
||||
// ⚠️ LANDMINE — dependency on R-82 (the backup target split). This constant is applied to
|
||||
// whichever tier is NEWEST, PBS or vzdump, with no tier-awareness. Today a daily vzdump
|
||||
// always wins, so PBS's own age is invisible here and 26h is harmless. The moment PBS moves
|
||||
// to a WEEKLY cadence, a perfectly healthy weekly snapshot is >26h old six days in seven and
|
||||
// this constant alarms on it. Fixing that means per-tier thresholds, which cannot be built
|
||||
// before the per-tier cadence config exists (`local_backup_target` is a single target and
|
||||
// BackupCadence() a single 24h window today). Do NOT pre-build it — R-82 owns both halves.
|
||||
const backupStaleAfter = 26 * time.Hour
|
||||
|
||||
// backupEvidenceLookback bounds how far back the hub looks for evidence that a backup ever
|
||||
// happened, when the LATEST report carries none. Generous against any plausible daily cadence
|
||||
// (and against an agent that stayed restarted for days), bounded so the cold path can't turn
|
||||
// into a full-retention scan of every report the hub holds. Beyond this the verdict is
|
||||
// "no evidence in the lookback", which is a fault in its own right once the anchor elapsed.
|
||||
const backupEvidenceLookback = 7 * 24 * time.Hour
|
||||
|
||||
// hostReportBackups is the minimal slice of an agent host-report the deadline check
|
||||
// reads to judge backup freshness (pbs_snapshots is the offsite-DR signal; backups is
|
||||
// the local vzdump fallback). Mirrors the agent's hub.PBSSnapshot / hub.Backup wire
|
||||
@@ -31,28 +50,86 @@ type hostReportBackups struct {
|
||||
} `json:"backups"`
|
||||
}
|
||||
|
||||
// backupVerdict is the three-valued outcome of the freshness policy. The middle value is the
|
||||
// whole point of R-81: "I have no evidence" is NOT "the backup failed".
|
||||
type backupVerdict int
|
||||
|
||||
const (
|
||||
verdictOK backupVerdict = iota // positive evidence of a recent backup
|
||||
verdictUnknown // no evidence yet, and the anchored window has not elapsed
|
||||
verdictMissed // positive evidence of a problem — alarm
|
||||
)
|
||||
|
||||
// backupAssessment is the verdict for one customer's offsite backup health.
|
||||
type backupAssessment struct {
|
||||
missed bool // raise expected_backup_missed
|
||||
reason string // human-readable cause (event message + logs)
|
||||
verdict backupVerdict
|
||||
reason string // human-readable cause (event message + logs)
|
||||
}
|
||||
|
||||
// assessBackupFreshness decides whether a customer's latest host-report shows a healthy,
|
||||
// recent backup. Pure (now is injected) so the policy is unit-tested. Only POSITIVE
|
||||
// evidence of a problem fires an alarm:
|
||||
// - no PBS snapshot AND no successful vzdump in the report → missed ("no backup recorded")
|
||||
// - newest backup older than backupStaleAfter → missed ("stale")
|
||||
// - the newest PBS snapshot's verify_state is "failed" → missed ("verify failed")
|
||||
// missed reports whether this assessment should raise expected_backup_missed.
|
||||
func (a backupAssessment) missed() bool { return a.verdict == verdictMissed }
|
||||
|
||||
// backupEvidence is the hub-history half of the freshness policy, resolved by the caller so
|
||||
// assessBackupFreshness stays PURE (see its doc comment on why that matters).
|
||||
type backupEvidence struct {
|
||||
// newestSeen is the newest backup evidence found across the retained host-report window
|
||||
// (PBS backup_time or successful vzdump started_at), regardless of whether the LATEST
|
||||
// report still carries it. haveSeen is false when the window held none.
|
||||
newestSeen time.Time
|
||||
haveSeen bool
|
||||
|
||||
// firstReportAt is when the hub first saw ANY host-report from this customer — the
|
||||
// observation anchor. Zero when unknown, which the policy treats as "cannot defer"
|
||||
// (fail toward visibility, matching the v0.73.0 zero-anchor branch).
|
||||
firstReportAt time.Time
|
||||
}
|
||||
|
||||
// assessBackupFreshness decides whether a customer's backups are healthy. Pure (now and the
|
||||
// hub-history evidence are injected) so the policy is unit-tested — that purity is why the
|
||||
// 2026-07-26 incident could be diagnosed at all, and why the fix below is provable.
|
||||
//
|
||||
// ── THE INVARIANT (R-81) ──────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// Only POSITIVE EVIDENCE OF A PROBLEM raises an alarm. ABSENCE OF A SIGNAL IS UNKNOWN,
|
||||
// and becomes a fault only once that absence has persisted beyond an ANCHORED window.
|
||||
//
|
||||
// This monitor family has made the same mistake three times, and it is written down here so
|
||||
// the fourth is harder:
|
||||
// - hub v0.12.0 — `expected_backup_missed` fired daily for every healthy customer, because
|
||||
// it looked for a `backup_completed` event that no component emits anymore.
|
||||
// - hub v0.73.0 — `offsite_stale` fired minutes after a HEALTHY repair, because the
|
||||
// never-ran branch had no time anchor. Fixed by anchoring, not by silence.
|
||||
// - R-81 (this) — `expected_backup_missed` fired on three boxes at once on 2026-07-26,
|
||||
// because an agent restart empties the host-report `backups` array (the agent's store is
|
||||
// in-memory) and empty was read as "no backup exists". The vzdump had in fact run.
|
||||
//
|
||||
// Note the shape of the fix in all three: NOT silence. Silence is the opposite failure — a
|
||||
// box that genuinely never backs up would then alarm never, which is strictly worse than
|
||||
// crying wolf. Absence is deferred, then alarmed on, with its own distinct reason string.
|
||||
//
|
||||
// ── THE BRANCHES ──────────────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// unparseable latest report → missed ("could not be parsed")
|
||||
// newest evidence (report OR window) older than staleAfter → missed ("newest backup is Xh old")
|
||||
// newest PBS snapshot's verify_state == "failed" → missed ("failed verification")
|
||||
// NO evidence anywhere, anchor NOT yet elapsed → UNKNOWN, silent + logged
|
||||
// NO evidence anywhere, anchor elapsed → missed ("no backup evidence in …")
|
||||
// otherwise → ok
|
||||
//
|
||||
// Each failure mode keeps its OWN reason string. That is not polish: the entire 2026-07-26
|
||||
// diagnosis turned on reading the exact string, and collapsing them would have made it
|
||||
// impossible. In particular "absence over time" and "a timestamp that is too old" are
|
||||
// different faults with different causes, and must never share a message.
|
||||
//
|
||||
// A fresh-but-not-yet-verified snapshot (verify_state "none"/"") is NOT treated as a
|
||||
// failure: PBS verification runs on its own cadence, so a snapshot taken hours before the
|
||||
// 03:00 check may legitimately be unverified. Alarming on that would re-introduce exactly
|
||||
// the daily false alarm this repoint removes (hence "failed" only, not "≠ ok").
|
||||
func assessBackupFreshness(reportJSON string, now time.Time) backupAssessment {
|
||||
// the daily false alarm the v0.12.0 repoint removed (hence "failed" only, not "≠ ok").
|
||||
func assessBackupFreshness(reportJSON string, ev backupEvidence, now time.Time) backupAssessment {
|
||||
var hr hostReportBackups
|
||||
if err := json.Unmarshal([]byte(reportJSON), &hr); err != nil {
|
||||
// Unparseable report → can't confirm a backup. Surface it rather than swallow it.
|
||||
return backupAssessment{missed: true, reason: "latest host-report could not be parsed"}
|
||||
return backupAssessment{verdict: verdictMissed, reason: "latest host-report could not be parsed"}
|
||||
}
|
||||
|
||||
var newestPBS time.Time
|
||||
@@ -86,21 +163,92 @@ func assessBackupFreshness(reportJSON string, now time.Time) backupAssessment {
|
||||
}
|
||||
}
|
||||
|
||||
if !havePBS && !haveVzdump {
|
||||
return backupAssessment{missed: true, reason: "no PBS snapshot or successful backup in the latest host-report"}
|
||||
// The newest evidence the LATEST report itself carries.
|
||||
newest := newestPBS
|
||||
haveNewest := havePBS
|
||||
if haveVzdump && (!haveNewest || newestVzdump.After(newest)) {
|
||||
newest, haveNewest = newestVzdump, true
|
||||
}
|
||||
|
||||
newest := newestPBS
|
||||
if haveVzdump && (!havePBS || newestVzdump.After(newest)) {
|
||||
newest = newestVzdump
|
||||
// R-81: fold in what the hub REMEMBERS. The agent's store is point-in-time and forgets
|
||||
// across a restart; the hub's retained host-reports do not. An empty array in the latest
|
||||
// report therefore says nothing on its own — the question is when evidence was last SEEN,
|
||||
// not whether this one report happens to carry it.
|
||||
if ev.haveSeen && (!haveNewest || ev.newestSeen.After(newest)) {
|
||||
newest, haveNewest = ev.newestSeen, true
|
||||
}
|
||||
|
||||
if !haveNewest {
|
||||
// ABSENCE. Not a failure by itself — see the invariant above. It becomes one only
|
||||
// once it has outlived the existing threshold, counted from first contact (the point
|
||||
// at which a backup first became possible to observe).
|
||||
if ev.firstReportAt.IsZero() {
|
||||
// No anchor to defer against — fail toward visibility, as v0.73.0 does for the
|
||||
// legacy zero-anchor shape. Distinct string: this is an unanchored absence.
|
||||
return backupAssessment{
|
||||
verdict: verdictMissed,
|
||||
reason: "no backup evidence in any retained host-report, and no first-contact anchor to defer against",
|
||||
}
|
||||
}
|
||||
watched := now.Sub(ev.firstReportAt)
|
||||
if watched <= backupStaleAfter {
|
||||
return backupAssessment{
|
||||
verdict: verdictUnknown,
|
||||
reason: fmt.Sprintf("no backup evidence yet, but only watching for %s (grace %s since first contact %s) — newborn host, not a fault",
|
||||
watched.Round(time.Hour), backupStaleAfter, ev.firstReportAt.Format(time.RFC3339)),
|
||||
}
|
||||
}
|
||||
return backupAssessment{
|
||||
verdict: verdictMissed,
|
||||
reason: fmt.Sprintf("no backup evidence in any host-report for %s (limit %s, first contact %s, lookback %s)",
|
||||
watched.Round(time.Hour), backupStaleAfter, ev.firstReportAt.Format(time.RFC3339), backupEvidenceLookback),
|
||||
}
|
||||
}
|
||||
|
||||
if age := now.Sub(newest); age > backupStaleAfter {
|
||||
return backupAssessment{missed: true, reason: fmt.Sprintf("newest backup is %s old (limit %s)", age.Round(time.Hour), backupStaleAfter)}
|
||||
return backupAssessment{verdict: verdictMissed, reason: fmt.Sprintf("newest backup is %s old (limit %s)", age.Round(time.Hour), backupStaleAfter)}
|
||||
}
|
||||
if havePBS && newestPBSVerify == "failed" {
|
||||
return backupAssessment{missed: true, reason: "newest PBS snapshot failed verification"}
|
||||
return backupAssessment{verdict: verdictMissed, reason: "newest PBS snapshot failed verification"}
|
||||
}
|
||||
return backupAssessment{missed: false}
|
||||
return backupAssessment{verdict: verdictOK}
|
||||
}
|
||||
|
||||
// newestBackupEvidence scans the customer's retained host-reports (newest first) for the most
|
||||
// recent backup evidence — a PBS snapshot backup_time or a SUCCESSFUL vzdump started_at —
|
||||
// regardless of whether the latest report still carries it.
|
||||
//
|
||||
// Cost discipline: it stops as soon as it has found evidence FRESH enough that nothing older
|
||||
// could change the verdict, so the healthy path reads one row. Only the genuinely-broken path
|
||||
// (no fresh evidence anywhere) walks the full lookback, and that is bounded by
|
||||
// backupEvidenceLookback rather than by retention.
|
||||
func newestBackupEvidence(rows []store.HostReportRow, now time.Time) (time.Time, bool) {
|
||||
var newest time.Time
|
||||
have := false
|
||||
for _, r := range rows {
|
||||
var hr hostReportBackups
|
||||
if err := json.Unmarshal([]byte(r.ReportJSON), &hr); err != nil {
|
||||
continue // a single malformed retained report must not blind the scan
|
||||
}
|
||||
for _, ps := range hr.PBSSnapshots {
|
||||
if t, ok := parseBackupTime(ps.BackupTime); ok && (!have || t.After(newest)) {
|
||||
newest, have = t, true
|
||||
}
|
||||
}
|
||||
for _, b := range hr.Backups {
|
||||
if !b.Success {
|
||||
continue
|
||||
}
|
||||
if t, ok := parseBackupTime(b.StartedAt); ok && (!have || t.After(newest)) {
|
||||
newest, have = t, true
|
||||
}
|
||||
}
|
||||
// Fresh evidence found → the age branch cannot fire and nothing older matters.
|
||||
if have && now.Sub(newest) <= backupStaleAfter {
|
||||
break
|
||||
}
|
||||
}
|
||||
return newest, have
|
||||
}
|
||||
|
||||
// parseBackupTime parses an RFC3339 timestamp from a host-report and normalizes to UTC.
|
||||
@@ -153,7 +301,7 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
|
||||
midnightBudapest := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, budapest)
|
||||
sinceUTC := midnightBudapest.UTC()
|
||||
|
||||
var backupMissed, dbdumpMissed, skipped int
|
||||
var backupMissed, dbdumpMissed, skipped, deferred int
|
||||
|
||||
for _, id := range customerIDs {
|
||||
// Skip nodes that are down — they already have staleness events
|
||||
@@ -179,7 +327,26 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
|
||||
// has no PBS data to judge here and must not emit a daily backup alarm of its
|
||||
// own. (The DB-dump half below still applies.)
|
||||
default:
|
||||
if a := assessBackupFreshness(reportJSON, time.Now().UTC()); a.missed {
|
||||
nowUTC := time.Now().UTC()
|
||||
|
||||
// R-81: resolve the hub-history half BEFORE judging. A read failure here must not
|
||||
// invent a fault — it degrades to "the latest report is all I know", which is the
|
||||
// pre-R-81 behaviour, and is logged rather than swallowed.
|
||||
var ev backupEvidence
|
||||
if rows, rerr := s.GetHostReportsSince(id, nowUTC.Add(-backupEvidenceLookback)); rerr != nil {
|
||||
logger.Printf("[WARN] Deadline check: failed to read host-report window for %s: %v", id, rerr)
|
||||
} else {
|
||||
ev.newestSeen, ev.haveSeen = newestBackupEvidence(rows, nowUTC)
|
||||
}
|
||||
if first, ferr := s.GetFirstHostReportAt(id); ferr != nil {
|
||||
logger.Printf("[WARN] Deadline check: failed to read first host-report for %s: %v", id, ferr)
|
||||
} else {
|
||||
ev.firstReportAt = first
|
||||
}
|
||||
|
||||
a := assessBackupFreshness(reportJSON, ev, nowUTC)
|
||||
switch a.verdict {
|
||||
case verdictMissed:
|
||||
msg := "No fresh verified backup: " + a.reason
|
||||
if _, err := s.SaveEvent(id, "expected_backup_missed", "error", msg, "{}", "hub"); err != nil {
|
||||
logger.Printf("[WARN] Failed to save expected_backup_missed for %s: %v", id, err)
|
||||
@@ -187,6 +354,13 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
|
||||
onEvent(id, "expected_backup_missed", "error", msg, "{}", "hub")
|
||||
}
|
||||
backupMissed++
|
||||
case verdictUnknown:
|
||||
// Make the deferral VISIBLE (the v0.73.0 Part-7 precedent): a quiet check must
|
||||
// never be indistinguishable from a check that did not run. This check fires
|
||||
// once daily, so this is at most one line per customer per day — not spam, and
|
||||
// it is the line that proves the deferral happened rather than an error.
|
||||
logger.Printf("[INFO] Deadline check: %s backup verdict UNKNOWN (no alarm) — %s", id, a.reason)
|
||||
deferred++
|
||||
}
|
||||
}
|
||||
|
||||
@@ -204,6 +378,6 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
|
||||
}
|
||||
}
|
||||
|
||||
logger.Printf("[INFO] Deadline check: %d customers, %d backup missed, %d dbdump missed, %d skipped (down)",
|
||||
len(customerIDs), backupMissed, dbdumpMissed, skipped)
|
||||
logger.Printf("[INFO] Deadline check: %d customers, %d backup missed, %d backup unknown (deferred), %d dbdump missed, %d skipped (down)",
|
||||
len(customerIDs), backupMissed, deferred, dbdumpMissed, skipped)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,435 @@
|
||||
package monitor
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"log"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
// R-81 — "no signal" is not "bad signal".
|
||||
//
|
||||
// Origin: 2026-07-26 03:00 UTC, expected_backup_missed fired on demo-felhom, demo-hp and
|
||||
// drill-r50 at once. The agent's backup record store is IN-MEMORY
|
||||
// (felhom-agent/internal/backup/store.go — "lost on restart; the cadence re-populates"), so
|
||||
// the R-50 island migration's fleet restart at 12:44 UTC emptied the host-report `backups`
|
||||
// array until the next backup at 07:03. The check read empty as "no backup exists". The
|
||||
// vzdump had in fact run: three archives were on disk (07-24, 07-25, 07-26).
|
||||
//
|
||||
// The fix is an ANCHOR, not silence — the v0.73.0 offsite never-ran shape. These tests pin
|
||||
// BOTH halves: absence must stop crying wolf (A, C) AND must still alarm when it is real (B).
|
||||
// A test suite that only proved A would pass against an implementation that never alarms,
|
||||
// which is strictly worse than the bug it replaces.
|
||||
|
||||
// evidenceAt builds a backupEvidence with hub-history evidence at `ago` before now, and a
|
||||
// first-contact anchor `watched` before now.
|
||||
func evidenceAt(now time.Time, ago, watched time.Duration) backupEvidence {
|
||||
return backupEvidence{
|
||||
newestSeen: now.Add(-ago),
|
||||
haveSeen: true,
|
||||
firstReportAt: now.Add(-watched),
|
||||
}
|
||||
}
|
||||
|
||||
// noEvidence builds a backupEvidence with NO backup evidence, watched for `watched`.
|
||||
func noEvidence(now time.Time, watched time.Duration) backupEvidence {
|
||||
return backupEvidence{firstReportAt: now.Add(-watched)}
|
||||
}
|
||||
|
||||
// ── Scenario A — the 2026-07-26 case must NOT alarm ──────────────────────────────────────
|
||||
|
||||
// COMPANION RED-PROOF (observed): removing the ev.haveSeen fold-in from
|
||||
// assessBackupFreshness (pre-R-81 shape: judge the latest report alone) makes this test fail
|
||||
// with:
|
||||
//
|
||||
// deadline_anchor_test.go: 07-26 shape must NOT alarm; got verdict=2 reason=
|
||||
// "newest backup is 176h0m0s old (limit 26h0m0s)"
|
||||
//
|
||||
// which is verbatim the message demo-felhom actually sent that morning. Restored after.
|
||||
func TestBackupFreshness_AgentRestartBlindWindow_NoAlarm(t *testing.T) {
|
||||
now := time.Date(2026, 7, 26, 3, 0, 0, 0, time.UTC)
|
||||
// The real demo-felhom shape: backups:[] (agent forgot), pbs_snapshots:[one from 07-18].
|
||||
report := `{"pbs_snapshots":[{"backup_time":"2026-07-18T18:31:06Z","verify_state":"ok"}],"backups":[]}`
|
||||
// The hub's retained window still holds the 07-25 06:30 vzdump — ~20.5h before the check.
|
||||
ev := evidenceAt(now, 20*time.Hour+30*time.Minute, 8*24*time.Hour)
|
||||
|
||||
got := assessBackupFreshness(report, ev, now)
|
||||
if got.missed() {
|
||||
t.Fatalf("07-26 shape must NOT alarm; got verdict=%d reason=%q", got.verdict, got.reason)
|
||||
}
|
||||
if got.verdict != verdictOK {
|
||||
t.Fatalf("evidence exists in the window → verdict must be OK, not a deferred UNKNOWN; got verdict=%d reason=%q", got.verdict, got.reason)
|
||||
}
|
||||
}
|
||||
|
||||
// The demo-hp / drill-r50 shape: BOTH arrays empty, nothing in the window either, but the
|
||||
// hub has been watching less than the threshold → UNKNOWN, not an alarm.
|
||||
func TestBackupFreshness_EmptyArraysWithinGrace_Unknown(t *testing.T) {
|
||||
now := time.Date(2026, 7, 26, 3, 0, 0, 0, time.UTC)
|
||||
got := assessBackupFreshness(`{"pbs_snapshots":[],"backups":[]}`, noEvidence(now, 10*time.Hour), now)
|
||||
if got.missed() {
|
||||
t.Fatalf("absence inside the anchored grace must NOT alarm; got reason=%q", got.reason)
|
||||
}
|
||||
if got.verdict != verdictUnknown {
|
||||
t.Fatalf("want verdictUnknown, got verdict=%d reason=%q", got.verdict, got.reason)
|
||||
}
|
||||
}
|
||||
|
||||
// ── Scenario B — a genuinely dead box MUST still alarm ───────────────────────────────────
|
||||
|
||||
// THE TEST THAT MAKES SCENARIO A SAFE. Without it, "return silent on absence" passes A and C.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): replacing the elapsed-anchor branch with an unconditional
|
||||
// `return backupAssessment{verdict: verdictUnknown, ...}` (the naive over-suppression fix)
|
||||
// makes this test fail with:
|
||||
//
|
||||
// deadline_anchor_test.go: a box with NO backup evidence for 240h MUST alarm; got verdict=1
|
||||
// reason="no backup evidence yet, but only watching for 240h0m0s ..."
|
||||
//
|
||||
// Restored after.
|
||||
func TestBackupFreshness_NoEvidenceBeyondAnchor_Alarms(t *testing.T) {
|
||||
now := time.Date(2026, 7, 26, 3, 0, 0, 0, time.UTC)
|
||||
got := assessBackupFreshness(`{"pbs_snapshots":[],"backups":[]}`, noEvidence(now, 240*time.Hour), now)
|
||||
if !got.missed() {
|
||||
t.Fatalf("a box with NO backup evidence for 240h MUST alarm; got verdict=%d reason=%q", got.verdict, got.reason)
|
||||
}
|
||||
// Scenario E: the reason must name absence-over-time, NOT bare absence and NOT staleness.
|
||||
if !strings.Contains(got.reason, "no backup evidence in any host-report for") {
|
||||
t.Fatalf("absence-over-time needs its own reason string; got %q", got.reason)
|
||||
}
|
||||
if strings.Contains(got.reason, "newest backup is") {
|
||||
t.Fatalf("absence must NOT reuse the stale-timestamp string; got %q", got.reason)
|
||||
}
|
||||
}
|
||||
|
||||
// A zero anchor (hub holds no first-contact time) must fail toward VISIBILITY, not silence —
|
||||
// the v0.73.0 legacy zero-anchor precedent.
|
||||
func TestBackupFreshness_NoEvidenceNoAnchor_Alarms(t *testing.T) {
|
||||
now := time.Date(2026, 7, 26, 3, 0, 0, 0, time.UTC)
|
||||
got := assessBackupFreshness(`{"pbs_snapshots":[],"backups":[]}`, backupEvidence{}, now)
|
||||
if !got.missed() {
|
||||
t.Fatalf("unanchored absence must fail toward visibility; got verdict=%d reason=%q", got.verdict, got.reason)
|
||||
}
|
||||
if !strings.Contains(got.reason, "no first-contact anchor") {
|
||||
t.Fatalf("unanchored absence needs its own reason string; got %q", got.reason)
|
||||
}
|
||||
}
|
||||
|
||||
// ── Scenario C — a fresh box is not born failing ─────────────────────────────────────────
|
||||
|
||||
// THE NAMED BOUNDARY CONTRACT (Part 2). "No evidence + no elapsed window → no alarm" is
|
||||
// pinned here as a contract with an obvious name, so re-introducing the bug requires deleting
|
||||
// a test that says what it is protecting. Exercises both sides of the boundary.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): pre-fix (`if !havePBS && !haveVzdump { return missed }`),
|
||||
// the 1-minute-old newborn fails with:
|
||||
//
|
||||
// deadline_anchor_test.go: CONTRACT VIOLATED: no evidence + no elapsed window must NOT
|
||||
// alarm (watched=1m0s, limit=26h0m0s); got reason="no PBS snapshot or successful backup in
|
||||
// the latest host-report"
|
||||
//
|
||||
// Restored after.
|
||||
func TestBackupFreshness_Contract_AbsenceIsUnknownUntilAnchorElapses(t *testing.T) {
|
||||
now := time.Date(2026, 7, 26, 3, 0, 0, 0, time.UTC)
|
||||
empty := `{"pbs_snapshots":[],"backups":[]}`
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
watched time.Duration
|
||||
wantMissed bool
|
||||
}{
|
||||
{"newborn, 1 minute", time.Minute, false},
|
||||
{"newborn, 1 hour", time.Hour, false},
|
||||
{"just inside the window", backupStaleAfter - time.Minute, false},
|
||||
{"exactly at the window", backupStaleAfter, false}, // <= is grace, not fault
|
||||
{"just outside the window", backupStaleAfter + time.Minute, true},
|
||||
{"long past the window", 30 * 24 * time.Hour, true},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
got := assessBackupFreshness(empty, noEvidence(now, c.watched), now)
|
||||
if got.missed() != c.wantMissed {
|
||||
if c.wantMissed {
|
||||
t.Fatalf("CONTRACT VIOLATED: absence beyond the window MUST alarm (watched=%s, limit=%s); got verdict=%d reason=%q",
|
||||
c.watched, backupStaleAfter, got.verdict, got.reason)
|
||||
}
|
||||
t.Fatalf("CONTRACT VIOLATED: no evidence + no elapsed window must NOT alarm (watched=%s, limit=%s); got reason=%q",
|
||||
c.watched, backupStaleAfter, got.reason)
|
||||
}
|
||||
if !c.wantMissed && got.verdict != verdictUnknown {
|
||||
t.Fatalf("deferred absence must be UNKNOWN (not OK) so it stays visible; got verdict=%d", got.verdict)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// ── Scenario D — the three existing behaviours are untouched ─────────────────────────────
|
||||
|
||||
// Pins the pre-R-81 outcomes byte-for-byte, INCLUDING the reason strings, under the new
|
||||
// signature. Hub-history evidence is present and fresh in each case so it cannot be the
|
||||
// thing producing the result — these must hold on the latest report's own merits.
|
||||
func TestBackupFreshness_ExistingBehavioursUnchanged(t *testing.T) {
|
||||
now := time.Date(2026, 7, 26, 3, 0, 0, 0, time.UTC)
|
||||
at := func(d time.Duration) string { return now.Add(d).Format(time.RFC3339) }
|
||||
anchor := 30 * 24 * time.Hour
|
||||
|
||||
t.Run("1 fresh verified PBS stays silent", func(t *testing.T) {
|
||||
r := `{"pbs_snapshots":[{"backup_time":"` + at(-3*time.Hour) + `","verify_state":"ok"}]}`
|
||||
got := assessBackupFreshness(r, evidenceAt(now, 3*time.Hour, anchor), now)
|
||||
if got.missed() || got.verdict != verdictOK {
|
||||
t.Fatalf("want silent OK; got verdict=%d reason=%q", got.verdict, got.reason)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("2 newest backup >26h still alarms with the same string", func(t *testing.T) {
|
||||
r := `{"pbs_snapshots":[{"backup_time":"` + at(-30*time.Hour) + `","verify_state":"ok"}]}`
|
||||
// Window evidence is ALSO 30h old — nothing fresher exists anywhere.
|
||||
got := assessBackupFreshness(r, evidenceAt(now, 30*time.Hour, anchor), now)
|
||||
if !got.missed() {
|
||||
t.Fatalf("stale backup must still alarm; got verdict=%d reason=%q", got.verdict, got.reason)
|
||||
}
|
||||
if got.reason != "newest backup is 30h0m0s old (limit 26h0m0s)" {
|
||||
t.Fatalf("stale reason string changed: %q", got.reason)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("3 failed verify still alarms with the same string", func(t *testing.T) {
|
||||
r := `{"pbs_snapshots":[{"backup_time":"` + at(-2*time.Hour) + `","verify_state":"failed"}]}`
|
||||
got := assessBackupFreshness(r, evidenceAt(now, 2*time.Hour, anchor), now)
|
||||
if !got.missed() {
|
||||
t.Fatalf("failed verify must still alarm; got verdict=%d reason=%q", got.verdict, got.reason)
|
||||
}
|
||||
if got.reason != "newest PBS snapshot failed verification" {
|
||||
t.Fatalf("verify-failed reason string changed: %q", got.reason)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("unparseable report still alarms", func(t *testing.T) {
|
||||
got := assessBackupFreshness(`not json`, evidenceAt(now, time.Hour, anchor), now)
|
||||
if !got.missed() || got.reason != "latest host-report could not be parsed" {
|
||||
t.Fatalf("unparseable behaviour changed: verdict=%d reason=%q", got.verdict, got.reason)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// Fresh hub-history evidence must NOT rescue a failed verify — behaviour 3 is about the
|
||||
// snapshot's integrity, not its age, so the anchor has no business suppressing it.
|
||||
func TestBackupFreshness_WindowEvidenceDoesNotRescueFailedVerify(t *testing.T) {
|
||||
now := time.Date(2026, 7, 26, 3, 0, 0, 0, time.UTC)
|
||||
r := `{"pbs_snapshots":[{"backup_time":"` + now.Add(-2*time.Hour).Format(time.RFC3339) + `","verify_state":"failed"}]}`
|
||||
got := assessBackupFreshness(r, evidenceAt(now, time.Minute, 30*24*time.Hour), now)
|
||||
if !got.missed() {
|
||||
t.Fatalf("fresh window evidence must not suppress a failed verify; got verdict=%d reason=%q", got.verdict, got.reason)
|
||||
}
|
||||
}
|
||||
|
||||
// ── Scenario E — reason strings stay diagnostic ──────────────────────────────────────────
|
||||
|
||||
// Every failure mode must produce a DISTINCT message. The 2026-07-26 diagnosis turned
|
||||
// entirely on reading the exact string; collapsing them would have made it impossible.
|
||||
func TestBackupFreshness_ReasonStringsAreDistinct(t *testing.T) {
|
||||
now := time.Date(2026, 7, 26, 3, 0, 0, 0, time.UTC)
|
||||
at := func(d time.Duration) string { return now.Add(d).Format(time.RFC3339) }
|
||||
|
||||
reasons := map[string]string{
|
||||
"unparseable": assessBackupFreshness(`nope`, backupEvidence{}, now).reason,
|
||||
"stale": assessBackupFreshness(`{"pbs_snapshots":[{"backup_time":"`+at(-40*time.Hour)+`","verify_state":"ok"}]}`, backupEvidence{}, now).reason,
|
||||
"verify failed": assessBackupFreshness(`{"pbs_snapshots":[{"backup_time":"`+at(-2*time.Hour)+`","verify_state":"failed"}]}`, evidenceAt(now, 2*time.Hour, 30*24*time.Hour), now).reason,
|
||||
"absence timed": assessBackupFreshness(`{"backups":[]}`, noEvidence(now, 240*time.Hour), now).reason,
|
||||
"absence unanch": assessBackupFreshness(`{"backups":[]}`, backupEvidence{}, now).reason,
|
||||
"absence grace": assessBackupFreshness(`{"backups":[]}`, noEvidence(now, time.Hour), now).reason,
|
||||
}
|
||||
seen := map[string]string{}
|
||||
for name, r := range reasons {
|
||||
if r == "" {
|
||||
t.Fatalf("%s produced an empty reason", name)
|
||||
}
|
||||
if prev, dup := seen[r]; dup {
|
||||
t.Fatalf("reason strings collapsed: %q and %q both produce %q", prev, name, r)
|
||||
}
|
||||
seen[r] = name
|
||||
}
|
||||
}
|
||||
|
||||
// ── newestBackupEvidence — the window scan ───────────────────────────────────────────────
|
||||
|
||||
func reportRow(t *testing.T, receivedAt time.Time, pbs []string, vzdump []struct {
|
||||
at string
|
||||
ok bool
|
||||
}) store.HostReportRow {
|
||||
t.Helper()
|
||||
type snap struct {
|
||||
BackupTime string `json:"backup_time"`
|
||||
VerifyState string `json:"verify_state"`
|
||||
}
|
||||
type bk struct {
|
||||
StartedAt string `json:"started_at"`
|
||||
Success bool `json:"success"`
|
||||
}
|
||||
p := struct {
|
||||
PBSSnapshots []snap `json:"pbs_snapshots"`
|
||||
Backups []bk `json:"backups"`
|
||||
}{}
|
||||
for _, s := range pbs {
|
||||
p.PBSSnapshots = append(p.PBSSnapshots, snap{BackupTime: s, VerifyState: "ok"})
|
||||
}
|
||||
for _, b := range vzdump {
|
||||
p.Backups = append(p.Backups, bk{StartedAt: b.at, Success: b.ok})
|
||||
}
|
||||
out, err := json.Marshal(p)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return store.HostReportRow{ReceivedAt: receivedAt, ReportJSON: string(out)}
|
||||
}
|
||||
|
||||
// The scan must reach PAST the empty reports the restart produced and find the vzdump that
|
||||
// an older report still carries. This is the mechanism behind Scenario A.
|
||||
func TestNewestBackupEvidence_ReachesPastEmptyReports(t *testing.T) {
|
||||
now := time.Date(2026, 7, 26, 3, 0, 0, 0, time.UTC)
|
||||
vz := func(at string, ok bool) []struct {
|
||||
at string
|
||||
ok bool
|
||||
} {
|
||||
return []struct {
|
||||
at string
|
||||
ok bool
|
||||
}{{at, ok}}
|
||||
}
|
||||
rows := []store.HostReportRow{
|
||||
// Newest first, as the store returns them. Post-restart reports carry nothing.
|
||||
reportRow(t, now.Add(-1*time.Minute), nil, nil),
|
||||
reportRow(t, now.Add(-1*time.Hour), nil, nil),
|
||||
reportRow(t, now.Add(-14*time.Hour), nil, nil),
|
||||
// The last pre-restart report still holds the 20.5h-old vzdump.
|
||||
reportRow(t, now.Add(-15*time.Hour), nil, vz(now.Add(-20*time.Hour-30*time.Minute).Format(time.RFC3339), true)),
|
||||
reportRow(t, now.Add(-40*time.Hour), nil, vz(now.Add(-44*time.Hour).Format(time.RFC3339), true)),
|
||||
}
|
||||
got, ok := newestBackupEvidence(rows, now)
|
||||
if !ok {
|
||||
t.Fatal("must find the vzdump carried by the pre-restart report")
|
||||
}
|
||||
if want := now.Add(-20*time.Hour - 30*time.Minute); !got.Equal(want) {
|
||||
t.Fatalf("newest evidence = %s, want %s", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// A FAILED vzdump is not evidence of a backup.
|
||||
func TestNewestBackupEvidence_IgnoresFailedVzdump(t *testing.T) {
|
||||
now := time.Date(2026, 7, 26, 3, 0, 0, 0, time.UTC)
|
||||
rows := []store.HostReportRow{
|
||||
reportRow(t, now.Add(-time.Hour), nil, []struct {
|
||||
at string
|
||||
ok bool
|
||||
}{{now.Add(-2 * time.Hour).Format(time.RFC3339), false}}),
|
||||
}
|
||||
if _, ok := newestBackupEvidence(rows, now); ok {
|
||||
t.Fatal("a failed vzdump must not count as evidence")
|
||||
}
|
||||
}
|
||||
|
||||
// One malformed retained report must not blind the scan to the good ones behind it.
|
||||
func TestNewestBackupEvidence_SkipsMalformedRows(t *testing.T) {
|
||||
now := time.Date(2026, 7, 26, 3, 0, 0, 0, time.UTC)
|
||||
rows := []store.HostReportRow{
|
||||
{ReceivedAt: now.Add(-time.Minute), ReportJSON: `{{{not json`},
|
||||
reportRow(t, now.Add(-2*time.Hour), []string{now.Add(-3 * time.Hour).Format(time.RFC3339)}, nil),
|
||||
}
|
||||
got, ok := newestBackupEvidence(rows, now)
|
||||
if !ok {
|
||||
t.Fatal("a malformed row must not blind the scan")
|
||||
}
|
||||
if want := now.Add(-3 * time.Hour); !got.Equal(want) {
|
||||
t.Fatalf("newest evidence = %s, want %s", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// No rows at all → no evidence, and emphatically not a zero timestamp treated as evidence.
|
||||
func TestNewestBackupEvidence_EmptyWindow(t *testing.T) {
|
||||
now := time.Date(2026, 7, 26, 3, 0, 0, 0, time.UTC)
|
||||
if _, ok := newestBackupEvidence(nil, now); ok {
|
||||
t.Fatal("an empty window must report no evidence")
|
||||
}
|
||||
}
|
||||
|
||||
// ── End-to-end through CheckBackupDeadlines ──────────────────────────────────────────────
|
||||
|
||||
// The full 2026-07-26 replay against the real store: an agent restart empties the array, the
|
||||
// hub's retained window still holds yesterday's vzdump → NO event.
|
||||
func TestCheckBackupDeadlines_RestartBlindWindow_NoEvent(t *testing.T) {
|
||||
st := newDeadlineStore(t)
|
||||
if _, err := st.SaveEvent("c1", "db_dump_completed", "info", "", "{}", "controller"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
okVz := []struct {
|
||||
at string
|
||||
ok bool
|
||||
}{{rfc(-20 * time.Hour), true}}
|
||||
|
||||
// Pre-restart report: carries the vzdump. Back-dated so it is not the latest.
|
||||
pre := hostReportJSON(t, [][2]string{{"2026-07-18T18:31:06Z", "ok"}}, okVz)
|
||||
if err := st.SaveHostReport("h1", "c1", []byte(pre), store.HostReportDenorm{}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.SetHostReportsReceivedAtForTest("c1", sqliteAgo(15*time.Hour)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Post-restart report: PBS only, 176h stale — exactly demo-felhom's 07-26 shape.
|
||||
post := hostReportJSON(t, [][2]string{{"2026-07-18T18:31:06Z", "ok"}}, nil)
|
||||
if err := st.SaveHostReport("h1", "c1", []byte(post), store.HostReportDenorm{}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
got := runDeadline(t, st)
|
||||
if has(got, "expected_backup_missed") {
|
||||
t.Fatalf("the 07-26 restart shape must NOT raise expected_backup_missed; got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The counterpart end-to-end: a host that has been reporting for days and has NEVER produced
|
||||
// a backup must still alarm through the real store path.
|
||||
func TestCheckBackupDeadlines_NeverBackedUpBeyondAnchor_Alarms(t *testing.T) {
|
||||
st := newDeadlineStore(t)
|
||||
st.SaveEvent("c1", "db_dump_completed", "info", "", "{}", "controller")
|
||||
empty := hostReportJSON(t, nil, nil)
|
||||
if err := st.SaveHostReport("h1", "c1", []byte(empty), store.HostReportDenorm{}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Back-date first contact well beyond the 26h anchor.
|
||||
if err := st.SetHostReportsReceivedAtForTest("c1", sqliteAgo(120*time.Hour)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.SaveHostReport("h1", "c1", []byte(empty), store.HostReportDenorm{}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
got := runDeadline(t, st)
|
||||
if !has(got, "expected_backup_missed") {
|
||||
t.Fatalf("a host with no backup for 120h MUST alarm; got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A newborn host must not alarm on its first morning — end-to-end.
|
||||
func TestCheckBackupDeadlines_NewbornHost_NoEvent(t *testing.T) {
|
||||
st := newDeadlineStore(t)
|
||||
st.SaveEvent("c1", "db_dump_completed", "info", "", "{}", "controller")
|
||||
empty := hostReportJSON(t, nil, nil)
|
||||
if err := st.SaveHostReport("h1", "c1", []byte(empty), store.HostReportDenorm{}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var logged strings.Builder
|
||||
onEvent := func(customerID, eventType, severity, message, detailsJSON, source string) {
|
||||
if customerID == "c1" && eventType == "expected_backup_missed" {
|
||||
t.Fatalf("a newborn host must not alarm; got %q", message)
|
||||
}
|
||||
}
|
||||
CheckBackupDeadlines(st, nil, onEvent, log.New(&logged, "", 0))
|
||||
|
||||
// The deferral must be VISIBLE — quiet must never look like "did not run".
|
||||
if !strings.Contains(logged.String(), "verdict UNKNOWN") {
|
||||
t.Fatalf("the deferred verdict must be logged; log was:\n%s", logged.String())
|
||||
}
|
||||
}
|
||||
@@ -171,6 +171,12 @@ func TestCheckBackupDeadlines_DbDumpHalfPreserved(t *testing.T) {
|
||||
}
|
||||
|
||||
// TestAssessBackupFreshness exercises the pure freshness policy directly.
|
||||
//
|
||||
// R-81: every case here passes a ZERO backupEvidence — no hub-history evidence and no
|
||||
// first-contact anchor. That is deliberate: it pins the latest-report-only behaviour
|
||||
// unchanged, and the "no snapshots and no backups" row exercises the UNANCHORED absence
|
||||
// branch (zero anchor → fail toward visibility, the v0.73.0 legacy-shape precedent).
|
||||
// The anchored branches have their own named tests below.
|
||||
func TestAssessBackupFreshness(t *testing.T) {
|
||||
now := time.Date(2026, 6, 16, 3, 0, 0, 0, time.UTC)
|
||||
at := func(d time.Duration) string { return now.Add(d).Format(time.RFC3339) }
|
||||
@@ -192,9 +198,9 @@ func TestAssessBackupFreshness(t *testing.T) {
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
got := assessBackupFreshness(c.report, now)
|
||||
if got.missed != c.wantMissed {
|
||||
t.Fatalf("missed=%v want=%v (reason=%q)", got.missed, c.wantMissed, got.reason)
|
||||
got := assessBackupFreshness(c.report, backupEvidence{}, now)
|
||||
if got.missed() != c.wantMissed {
|
||||
t.Fatalf("missed=%v want=%v (reason=%q)", got.missed(), c.wantMissed, got.reason)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
@@ -2825,6 +2825,83 @@ func (s *Store) GetLatestHostReportJSON(customerID string) (string, error) {
|
||||
return j, nil
|
||||
}
|
||||
|
||||
// HostReportRow is one retained host-report: when the hub received it, and its payload.
|
||||
type HostReportRow struct {
|
||||
ReceivedAt time.Time
|
||||
ReportJSON string
|
||||
}
|
||||
|
||||
// GetHostReportsSince returns the customer's retained host-reports received at or after
|
||||
// `since`, NEWEST FIRST. The hub keeps ~retention.max_days of history (90 by default), which
|
||||
// is what makes it possible to ask "when did I last SEE evidence of a backup?" rather than
|
||||
// only "what does the latest report say?".
|
||||
//
|
||||
// R-81: this is the anchor source for the backup-deadline check. The agent's backup record
|
||||
// store is IN-MEMORY (felhom-agent/internal/backup/store.go — "lost on restart; the cadence
|
||||
// re-populates"), so a restart empties `backups` in every report until the next backup runs.
|
||||
// The hub has memory the agent does not; reading across the window is what turns that blind
|
||||
// window from a false alarm into a correctly-silent verdict. Newest-first so the caller can
|
||||
// stop as soon as it has seen enough (see monitor.newestBackupEvidence).
|
||||
func (s *Store) GetHostReportsSince(customerID string, since time.Time) ([]HostReportRow, error) {
|
||||
rows, err := s.db.Query(
|
||||
`SELECT received_at, report_json FROM host_reports
|
||||
WHERE customer_id = ? AND received_at >= ?
|
||||
ORDER BY received_at DESC`,
|
||||
customerID, since.UTC().Format("2006-01-02 15:04:05"),
|
||||
)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var out []HostReportRow
|
||||
for rows.Next() {
|
||||
var at, j string
|
||||
if err := rows.Scan(&at, &j); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, HostReportRow{ReceivedAt: parseSQLiteTime(at).UTC(), ReportJSON: j})
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// GetFirstHostReportAt returns when the hub received its FIRST retained host-report for the
|
||||
// customer, or (zero, nil) when it holds none.
|
||||
//
|
||||
// R-81: this is the observation anchor. "No backup evidence anywhere" is only meaningful
|
||||
// relative to how long the hub has been watching — a box registered an hour ago has no
|
||||
// evidence yet and is NOT failing. Same shape as the v0.73.0 offsite never-ran anchor:
|
||||
// absence becomes a fault only once it has outlived the existing threshold, measured from a
|
||||
// point where evidence first became POSSIBLE.
|
||||
//
|
||||
// Caveat, deliberately accepted: retention prunes at max_days, so for a host older than the
|
||||
// window this returns the prune horizon rather than true first-contact. That only makes the
|
||||
// anchor MORE conservative for long-lived hosts (the window has long since elapsed either
|
||||
// way), and it never shortens a newborn's grace.
|
||||
func (s *Store) GetFirstHostReportAt(customerID string) (time.Time, error) {
|
||||
var at string
|
||||
err := s.db.QueryRow(
|
||||
`SELECT received_at FROM host_reports WHERE customer_id = ? ORDER BY received_at ASC LIMIT 1`,
|
||||
customerID,
|
||||
).Scan(&at)
|
||||
if err == sql.ErrNoRows {
|
||||
return time.Time{}, nil
|
||||
}
|
||||
if err != nil {
|
||||
return time.Time{}, err
|
||||
}
|
||||
return parseSQLiteTime(at).UTC(), nil
|
||||
}
|
||||
|
||||
// SetHostReportsReceivedAtForTest back-dates ALL of a customer's existing host_reports rows
|
||||
// to the given SQLite datetime string, so anchor/window behaviour is testable without
|
||||
// sleeping. TEST-ONLY — mirrors SetOneTimeSecretTimesForTest. Rows saved AFTER the call keep
|
||||
// datetime('now'), which is how a test stages "old reports, then a fresh one".
|
||||
func (s *Store) SetHostReportsReceivedAtForTest(customerID, sqliteTime string) error {
|
||||
_, err := s.db.Exec(`UPDATE host_reports SET received_at = ? WHERE customer_id = ?`, sqliteTime, customerID)
|
||||
return err
|
||||
}
|
||||
|
||||
// UpsertGuestFromReport upserts the REALITY columns of a guest. On conflict it
|
||||
// must NOT clobber the inert columns (api_key / desired_spec_json).
|
||||
func (s *Store) UpsertGuestFromReport(g *Guest) error {
|
||||
|
||||
Reference in New Issue
Block a user