# REPORT — R-81: "no signal" is not "bad signal" (hub v0.75.0) (2026-07-26) Class fix for the third instance of one bug: **absence of a signal treated as evidence of failure.** Input: `documentation/audits/DIAG-backup-missed-2026-07-26.md` (`add5b9b`). --- ## 1. Confirmed baselines | Repo | Start `main` | Version | → Shipped | |---|---|---|---| | felhom.eu (hub) | `add5b9b` | v0.74.0 | **v0.75.0** (`f5a5e2b` code+docs, `88b41ec` manifest pin) | | felhom-agent | `dfd5d73` | v0.96.0 | unchanged — **N/A this task** (cause-fix is R-84) | | felhom-controller | after R-77 | v0.173.0 | unchanged | Clean-tree gate passed before the build: `git status --porcelain` empty, `HEAD == origin/main`. --- ## 2. Phase-0 — the anchor probe **Question:** does the hub retain host-report HISTORY, or only the latest report? **Answer: HISTORY IS RETAINED — 90 days, indexed and queryable by customer + time.** - `host_reports` is an append-only table (`SaveHostReport` INSERTs; `GetLatestHostReportJSON` only ever read the newest row). - Retention: `cfg.Retention.MaxDays`, **90** both as the code default (`cmd/hub/main.go:631,654`) and as the live value in `manifests/hub.yaml` (`retention.max_days: 90`). `Prune` deletes `host_reports` older than that on the 04:30 job. - Index `idx_host_reports_customer ON host_reports(customer_id, received_at DESC)` — the exact access pattern needed, already present. - Live confirmation on the production DB: 664 / 436 / 96 retained rows for demo-felhom / demo-hp / drill-r50, spanning 7.63 / 4.64 / 0.91 days. **Branch taken: the free anchor.** No new persisted state, no `last_backup_seen` column, no agent change. The agent's store is *point-in-time*; the hub has *memory*. Judging backup evidence across the retained window is both the cheapest option and semantically the right question — **"when did I last SEE evidence of a backup?"** rather than "what does this one report happen to say?". --- ## 3. Files modified | File | Change | |---|---| | `hub/internal/monitor/deadline.go` | Three-valued verdict; hub-history fold-in; anchored absence; distinct reason strings; `newestBackupEvidence`; landmine comment on `backupStaleAfter`; new `backupEvidenceLookback` | | `hub/internal/store/store.go` | `HostReportRow`, `GetHostReportsSince`, `GetFirstHostReportAt`, `SetHostReportsReceivedAtForTest` | | `hub/internal/monitor/deadline_test.go` | Signature update; note explaining the zero-evidence rows | | `hub/internal/monitor/deadline_anchor_test.go` | **NEW** — 15 tests, scenarios A–E + the named boundary contract | | `hub/CHANGELOG.md` | v0.75.0 | | `CONTEXT.md` | New **Standing rulings** section (S-1, S-2) + the v0.75.0 entry | | `REUSE.md` | New canonical-helper row; new dangerous-lookalike row | | `documentation/backlog/ROADMAP.md` | R-80 closed, R-81 shipped, **R-82 / R-83 / R-84 added** | | `documentation/architecture/00-capability-map.md` | Note row — no status flips | | `manifests/hub.yaml` | image pin 0.74.0 → **0.75.0** | Commits: **`f5a5e2b`** (code + docs), **`88b41ec`** (manifest pin). Both on `main`, pushed. --- ## 4. The fix `assessBackupFreshness` returned `{missed bool, reason string}` and collapsed *absence of records* into *failure*. It now returns a three-valued verdict: ``` verdictOK positive evidence of a recent backup verdictUnknown no evidence yet, and the anchored window has not elapsed → silent, LOGGED verdictMissed positive evidence of a problem → alarm ``` Three inputs, all injected — the function stays **pure**: 1. the latest report (unchanged parsing), 2. `backupEvidence.newestSeen` — the newest backup evidence across a bounded 7-day window of retained reports (`GetHostReportsSince` + `newestBackupEvidence`), 3. `backupEvidence.firstReportAt` — first contact (`GetFirstHostReportAt`), the absence anchor. Absence is graded against the **existing** `backupStaleAfter` (26 h), exactly as v0.73.0 reused offsite `staleAfter` — **no new knob**. A zero anchor fails toward visibility (the v0.73.0 legacy-shape precedent). The window scan early-exits at the first sufficiently-fresh evidence, so the healthy path reads one row; only the genuinely-broken path walks the lookback. `CheckBackupDeadlines` logs each deferral (`verdict UNKNOWN`) and the summary line gained a `backup unknown (deferred)` counter — a quiet check must never be indistinguishable from one that did not run. The invariant is written at the head of `assessBackupFreshness`, naming all three instances (v0.12.0, v0.73.0, R-81), and pinned by a boundary test whose name states what it protects. --- ## 5. Test results **Full gate green:** `go build ./... && go vet ./... && go test ./...` — all rc=0, every package `ok`. **Test count: 493 → 508 (+15).** (Measured with `git grep -hE '^func Test'` at `HEAD` vs the working tree — a first attempt via `go test -list` after a `git stash` gave a bogus 434 because the stash broke compilation and silently dropped the whole package. Recorded because that is exactly the exit-code-lie class this project has been bitten by; likewise `go vet ... | head` reported rc=0 while vet was actually failing.) All 21 tests in `internal/monitor` covering this area pass, including the 6 pre-existing ones. ### The three companion red-proofs — applied, observed, restored **A — Scenario A (the 07-26 case must not alarm).** Removed the `ev.haveSeen` fold-in (pre-R-81 shape: judge the latest report alone): ``` --- FAIL: TestBackupFreshness_AgentRestartBlindWindow_NoAlarm (0.00s) deadline_anchor_test.go:61: 07-26 shape must NOT alarm; got verdict=2 reason="newest backup is 176h0m0s old (limit 26h0m0s)" --- FAIL: TestCheckBackupDeadlines_RestartBlindWindow_NoEvent (0.03s) deadline_anchor_test.go:388: the 07-26 restart shape must NOT raise expected_backup_missed; got [expected_backup_missed] ``` The reason string is **verbatim the message demo-felhom actually sent to the customer channel** that morning. Restored. **B — Scenario B (a genuinely dead box must still alarm).** Applied the *naive* fix — absence always returns UNKNOWN, never a fault: ``` --- FAIL: TestBackupFreshness_NoEvidenceBeyondAnchor_Alarms (0.00s) deadline_anchor_test.go:97: a box with NO backup evidence for 240h MUST alarm; got verdict=1 reason="no backup evidence yet, but only watching for 240h0m0s" --- FAIL: TestBackupFreshness_Contract_AbsenceIsUnknownUntilAnchorElapses/just_outside_the_window deadline_anchor_test.go:156: CONTRACT VIOLATED: absence beyond the window MUST alarm (watched=26h1m0s, limit=26h0m0s); got verdict=1 … --- FAIL: TestBackupFreshness_Contract_AbsenceIsUnknownUntilAnchorElapses/long_past_the_window deadline_anchor_test.go:156: CONTRACT VIOLATED: absence beyond the window MUST alarm (watched=720h0m0s, limit=26h0m0s); got verdict=1 … --- FAIL: TestCheckBackupDeadlines_NeverBackedUpBeyondAnchor_Alarms (0.03s) deadline_anchor_test.go:411: a host with no backup for 120h MUST alarm; got [] ``` **This is the proof that matters most.** A suite that only pinned Scenario A would have passed against this over-suppressed implementation — and over-suppression is strictly worse than the bug it replaces. Restored. **C — Scenario C (a fresh box is not born failing).** Restored the literal pre-fix branch (`if !havePBS && !haveVzdump { return missed }`): ``` --- FAIL: TestBackupFreshness_EmptyArraysWithinGrace_Unknown (0.00s) deadline_anchor_test.go:74: absence inside the anchored grace must NOT alarm; got reason="no PBS snapshot or successful backup in the latest host-report" --- FAIL: …/newborn,_1_minute deadline_anchor_test.go:159: CONTRACT VIOLATED: no evidence + no elapsed window must NOT alarm (watched=1m0s, limit=26h0m0s); got reason="no PBS snapshot or successful backup in the latest host-report" --- FAIL: …/newborn,_1_hour (watched=1h0m0s) --- FAIL: …/just_inside_the_window (watched=25h59m0s) --- FAIL: …/exactly_at_the_window (watched=26h0m0s) --- FAIL: TestCheckBackupDeadlines_NewbornHost_NoEvent (0.03s) deadline_anchor_test.go:426: a newborn host must not alarm; got "No fresh verified backup: no PBS snapshot or successful backup in the latest host-report" ``` Restored; full suite green after each. ### Scenario D and E - **D** — `TestBackupFreshness_ExistingBehavioursUnchanged` pins all three pre-existing outcomes **including their exact reason strings** (`"newest backup is 30h0m0s old (limit 26h0m0s)"`, `"newest PBS snapshot failed verification"`, `"latest host-report could not be parsed"`), each with fresh window evidence present so the result cannot be an artefact of the new input. Plus `TestBackupFreshness_WindowEvidenceDoesNotRescueFailedVerify` — the anchor must not suppress an integrity fault. - **E** — `TestBackupFreshness_ReasonStringsAreDistinct` fails if any two of the six failure modes ever produce the same message. --- ## 6. Replay of the 2026-07-26 shape — **actual reports, not a fixture** Two replays, both against the real thing. **(a) Real report payloads** extracted from the production hub DB (every retained report in the 7-day lookback at or before the check instant), fed through `newestBackupEvidence` + `assessBackupFreshness` at `now = 2026-07-26 03:00:00 UTC`: ``` demo-felhom rows=600 first=2026-07-18T16:30:51Z windowEvidence=2026-07-25T06:30:14Z -> OK demo-hp rows=417 first=2026-07-21T16:24:48Z windowEvidence=2026-07-25T10:23:31Z -> OK drill-r50 rows= 77 first=2026-07-25T09:50:55Z windowEvidence=none -> UNKNOWN "no backup evidence yet, but only watching for 17h0m0s (grace 26h0m0s since first contact 2026-07-25T09:50:55Z) — newborn host, not a fault" ``` **(b) Through the real store queries** — a copy of the live `hub.db` opened with `store.New`, driving `GetHostReportsSince` / `GetFirstHostReportAt` / `GetLatestHostReportJSON` for real. This was worth doing separately: the window query compares `received_at` as a **SQLite datetime string**, which is a genuine correctness risk that a hand-built fixture would not have exercised. Identical verdicts, plus: ``` peti-felhom NO host-report → deadline check skips the backup half entirely ``` **All three would now be silent. Zero events.** Both harnesses were throwaway and are deleted; the tree is clean. Only the DB *copy* was ever opened writable — the live `/data/hub.db` was not touched. Note what the verdicts say: demo-felhom and demo-hp are **OK** (real evidence was found, not suppressed), and drill-r50 is **UNKNOWN** (correctly deferred as a newborn, and it will alarm on its own if it is still backup-less after the grace). None of the three is silenced by a blanket rule. --- ## 7. Deployment | Step | Result | |---|---| | Image build + push | `gitea.dooplex.hu/admin/felhom-hub:0.75.0`, 25 MB, digest `sha256:a2ebaa50…10453` | | Manifest pin | `manifests/hub.yaml` → `0.75.0` (commit `88b41ec`) — **GitOps only**, no `kubectl set image` | | ArgoCD | hard-refresh + deliberate sync → **Synced / Healthy** | | Rollout | `deployment "hub" successfully rolled out` | | Pod | `hub-56946d5cb4-kwwn7` 1/1 Running | | **Running image (pod status, not spec)** | `felhom-hub:0.75.0` @ `sha256:a2ebaa50…10453` — **matches the pushed digest** | | Startup log | `[INFO] felhom-hub 0.75.0 starting` … `[INFO] deadline-check: next run at 2026-07-27 05:00 CEST (in 17h14m19s)` | The pin landed and matches — checked deliberately, since the DIAG found `hub.yaml` pinned at `0.73.1` while the CHANGELOG read `0.73.2`. --- ## 8. NOT yet live-validated — explicit 1. **The 05:00 CEST deadline check has not fired under v0.75.0.** Next run 2026-07-27 05:00 CEST. All evidence above is unit-proven + replayed against real data; the live firing is unobserved. 2. **The deferred-UNKNOWN INFO line has not appeared in a production log** — proven by unit test only. drill-r50's 26 h grace expires 2026-07-26 ~11:50 UTC, so by 03:00 on 07-27 it will instead **alarm** — correctly, and that will be the first real test of Scenario B in production. 3. **No genuinely-dead box has been observed alarming live.** The B path is unit-proven only. 4. **The cold path (full 7-day lookback scan) has not run in production.** Cost is bounded and the healthy path early-exits at one row, but the scan-heavy branch is untimed live. 5. **demo-hp and drill-r50 host-level state remains uncollected** (no SSH key; break-glass not used) — unchanged from the DIAG. --- ## 9. Observations — recorded, not acted on 1. **`parseBackupTime`'s silent `continue` is a latent member of the same class.** An unparseable timestamp is dropped without a trace, so a wire-format change on the agent side would present as "no backup" — the exact 07-26 failure, with no log line to catch it. Left untouched per scope (the agent emits clean RFC3339 `Z`, confirmed in the DIAG). Worth a WARN log when R-84 touches the agent side of this contract. 2. **The customer-facing Hungarian copy is unchanged.** `notify/templates.go:106` — *"A mai biztonsági mentés nem készült el a határidőig!"* — still overstates scope (it reads as *all* backups failed; this check only covers the host/PBS tier). Not in this task's Parts 1–3. The DIAG proposed it as fix #4; it needs a `felhom-ui-design` copy pass and the confirm gate. 3. **`GetFirstHostReportAt` returns the prune horizon, not true first contact, for hosts older than 90 days.** Accepted and documented at the function: it only makes the anchor *more* conservative for long-lived hosts (the window has elapsed either way) and never shortens a newborn's grace. 4. **The `backupStaleAfter` landmine is now written into the constant's comment** and carried as a named dependency in R-82. It will alarm on a healthy weekly PBS snapshot six days in seven. 5. **`retention.max_days: 90` is now load-bearing for a monitor**, not just for storage. Shortening it below `backupEvidenceLookback` would silently narrow the evidence window. Not currently guarded. 6. **The R-80 real finding stands untouched:** the PBS/offsite-DR tier has no schedule at all — demo-felhom still holds exactly one PBS snapshot (2026-07-18), demo-hp zero. R-81 fixes how the hub *reads* backup state; it does nothing about the DR tier being empty. **That is R-82, and it is the more serious of the two.** 7. The customer-channel suppression the DIAG recommended (removing `expected_backup_missed` from demo-felhom's `enabled_events`) was **not** applied — still an operator decision, and now much less pressing since the false-positive path is closed. --- ## 10. Scope discipline Not started, recorded only: **R-82** (backup target split — the real finding), **R-83** (ratify or retire `07-backup-architecture.md`), **R-84** (persist the agent's backup store — the cause-fix). No branches; every commit staged with explicit paths (never `git add -A`, per the shared-worktree rule); no `Co-Authored-By`.