Claude Code
269 lines
15 KiB
Markdown
269 lines
15 KiB
Markdown
# REPORT — R-81: "no signal" is not "bad signal" (hub v0.75.0) (2026-07-26)
|
||
|
||
Class fix for the third instance of one bug: **absence of a signal treated as evidence of failure.**
|
||
Input: `documentation/audits/DIAG-backup-missed-2026-07-26.md` (`add5b9b`).
|
||
|
||
---
|
||
|
||
## 1. Confirmed baselines
|
||
|
||
| Repo | Start `main` | Version | → Shipped |
|
||
|---|---|---|---|
|
||
| felhom.eu (hub) | `add5b9b` | v0.74.0 | **v0.75.0** (`f5a5e2b` code+docs, `88b41ec` manifest pin) |
|
||
| felhom-agent | `dfd5d73` | v0.96.0 | unchanged — **N/A this task** (cause-fix is R-84) |
|
||
| felhom-controller | after R-77 | v0.173.0 | unchanged |
|
||
|
||
Clean-tree gate passed before the build: `git status --porcelain` empty, `HEAD == origin/main`.
|
||
|
||
---
|
||
|
||
## 2. Phase-0 — the anchor probe
|
||
|
||
**Question:** does the hub retain host-report HISTORY, or only the latest report?
|
||
|
||
**Answer: HISTORY IS RETAINED — 90 days, indexed and queryable by customer + time.**
|
||
|
||
- `host_reports` is an append-only table (`SaveHostReport` INSERTs; `GetLatestHostReportJSON` only
|
||
ever read the newest row).
|
||
- Retention: `cfg.Retention.MaxDays`, **90** both as the code default (`cmd/hub/main.go:631,654`) and
|
||
as the live value in `manifests/hub.yaml` (`retention.max_days: 90`). `Prune` deletes
|
||
`host_reports` older than that on the 04:30 job.
|
||
- Index `idx_host_reports_customer ON host_reports(customer_id, received_at DESC)` — the exact access
|
||
pattern needed, already present.
|
||
- Live confirmation on the production DB: 664 / 436 / 96 retained rows for demo-felhom / demo-hp /
|
||
drill-r50, spanning 7.63 / 4.64 / 0.91 days.
|
||
|
||
**Branch taken: the free anchor.** No new persisted state, no `last_backup_seen` column, no agent
|
||
change. The agent's store is *point-in-time*; the hub has *memory*. Judging backup evidence across
|
||
the retained window is both the cheapest option and semantically the right question — **"when did I
|
||
last SEE evidence of a backup?"** rather than "what does this one report happen to say?".
|
||
|
||
---
|
||
|
||
## 3. Files modified
|
||
|
||
| File | Change |
|
||
|---|---|
|
||
| `hub/internal/monitor/deadline.go` | Three-valued verdict; hub-history fold-in; anchored absence; distinct reason strings; `newestBackupEvidence`; landmine comment on `backupStaleAfter`; new `backupEvidenceLookback` |
|
||
| `hub/internal/store/store.go` | `HostReportRow`, `GetHostReportsSince`, `GetFirstHostReportAt`, `SetHostReportsReceivedAtForTest` |
|
||
| `hub/internal/monitor/deadline_test.go` | Signature update; note explaining the zero-evidence rows |
|
||
| `hub/internal/monitor/deadline_anchor_test.go` | **NEW** — 15 tests, scenarios A–E + the named boundary contract |
|
||
| `hub/CHANGELOG.md` | v0.75.0 |
|
||
| `CONTEXT.md` | New **Standing rulings** section (S-1, S-2) + the v0.75.0 entry |
|
||
| `REUSE.md` | New canonical-helper row; new dangerous-lookalike row |
|
||
| `documentation/backlog/ROADMAP.md` | R-80 closed, R-81 shipped, **R-82 / R-83 / R-84 added** |
|
||
| `documentation/architecture/00-capability-map.md` | Note row — no status flips |
|
||
| `manifests/hub.yaml` | image pin 0.74.0 → **0.75.0** |
|
||
|
||
Commits: **`f5a5e2b`** (code + docs), **`88b41ec`** (manifest pin). Both on `main`, pushed.
|
||
|
||
---
|
||
|
||
## 4. The fix
|
||
|
||
`assessBackupFreshness` returned `{missed bool, reason string}` and collapsed *absence of records*
|
||
into *failure*. It now returns a three-valued verdict:
|
||
|
||
```
|
||
verdictOK positive evidence of a recent backup
|
||
verdictUnknown no evidence yet, and the anchored window has not elapsed → silent, LOGGED
|
||
verdictMissed positive evidence of a problem → alarm
|
||
```
|
||
|
||
Three inputs, all injected — the function stays **pure**:
|
||
|
||
1. the latest report (unchanged parsing),
|
||
2. `backupEvidence.newestSeen` — the newest backup evidence across a bounded 7-day window of retained
|
||
reports (`GetHostReportsSince` + `newestBackupEvidence`),
|
||
3. `backupEvidence.firstReportAt` — first contact (`GetFirstHostReportAt`), the absence anchor.
|
||
|
||
Absence is graded against the **existing** `backupStaleAfter` (26 h), exactly as v0.73.0 reused
|
||
offsite `staleAfter` — **no new knob**. A zero anchor fails toward visibility (the v0.73.0
|
||
legacy-shape precedent). The window scan early-exits at the first sufficiently-fresh evidence, so the
|
||
healthy path reads one row; only the genuinely-broken path walks the lookback.
|
||
|
||
`CheckBackupDeadlines` logs each deferral (`verdict UNKNOWN`) and the summary line gained a
|
||
`backup unknown (deferred)` counter — a quiet check must never be indistinguishable from one that
|
||
did not run.
|
||
|
||
The invariant is written at the head of `assessBackupFreshness`, naming all three instances
|
||
(v0.12.0, v0.73.0, R-81), and pinned by a boundary test whose name states what it protects.
|
||
|
||
---
|
||
|
||
## 5. Test results
|
||
|
||
**Full gate green:** `go build ./... && go vet ./... && go test ./...` — all rc=0, every package `ok`.
|
||
|
||
**Test count: 493 → 508 (+15).** (Measured with `git grep -hE '^func Test'` at `HEAD` vs the working
|
||
tree — a first attempt via `go test -list` after a `git stash` gave a bogus 434 because the stash
|
||
broke compilation and silently dropped the whole package. Recorded because that is exactly the
|
||
exit-code-lie class this project has been bitten by; likewise `go vet ... | head` reported rc=0 while
|
||
vet was actually failing.)
|
||
|
||
All 21 tests in `internal/monitor` covering this area pass, including the 6 pre-existing ones.
|
||
|
||
### The three companion red-proofs — applied, observed, restored
|
||
|
||
**A — Scenario A (the 07-26 case must not alarm).** Removed the `ev.haveSeen` fold-in (pre-R-81
|
||
shape: judge the latest report alone):
|
||
|
||
```
|
||
--- FAIL: TestBackupFreshness_AgentRestartBlindWindow_NoAlarm (0.00s)
|
||
deadline_anchor_test.go:61: 07-26 shape must NOT alarm; got verdict=2 reason="newest backup is 176h0m0s old (limit 26h0m0s)"
|
||
--- FAIL: TestCheckBackupDeadlines_RestartBlindWindow_NoEvent (0.03s)
|
||
deadline_anchor_test.go:388: the 07-26 restart shape must NOT raise expected_backup_missed; got [expected_backup_missed]
|
||
```
|
||
|
||
The reason string is **verbatim the message demo-felhom actually sent to the customer channel** that
|
||
morning. Restored.
|
||
|
||
**B — Scenario B (a genuinely dead box must still alarm).** Applied the *naive* fix — absence always
|
||
returns UNKNOWN, never a fault:
|
||
|
||
```
|
||
--- FAIL: TestBackupFreshness_NoEvidenceBeyondAnchor_Alarms (0.00s)
|
||
deadline_anchor_test.go:97: a box with NO backup evidence for 240h MUST alarm; got verdict=1 reason="no backup evidence yet, but only watching for 240h0m0s"
|
||
--- FAIL: TestBackupFreshness_Contract_AbsenceIsUnknownUntilAnchorElapses/just_outside_the_window
|
||
deadline_anchor_test.go:156: CONTRACT VIOLATED: absence beyond the window MUST alarm (watched=26h1m0s, limit=26h0m0s); got verdict=1 …
|
||
--- FAIL: TestBackupFreshness_Contract_AbsenceIsUnknownUntilAnchorElapses/long_past_the_window
|
||
deadline_anchor_test.go:156: CONTRACT VIOLATED: absence beyond the window MUST alarm (watched=720h0m0s, limit=26h0m0s); got verdict=1 …
|
||
--- FAIL: TestCheckBackupDeadlines_NeverBackedUpBeyondAnchor_Alarms (0.03s)
|
||
deadline_anchor_test.go:411: a host with no backup for 120h MUST alarm; got []
|
||
```
|
||
|
||
**This is the proof that matters most.** A suite that only pinned Scenario A would have passed
|
||
against this over-suppressed implementation — and over-suppression is strictly worse than the bug it
|
||
replaces. Restored.
|
||
|
||
**C — Scenario C (a fresh box is not born failing).** Restored the literal pre-fix branch
|
||
(`if !havePBS && !haveVzdump { return missed }`):
|
||
|
||
```
|
||
--- FAIL: TestBackupFreshness_EmptyArraysWithinGrace_Unknown (0.00s)
|
||
deadline_anchor_test.go:74: absence inside the anchored grace must NOT alarm; got reason="no PBS snapshot or successful backup in the latest host-report"
|
||
--- FAIL: …/newborn,_1_minute
|
||
deadline_anchor_test.go:159: CONTRACT VIOLATED: no evidence + no elapsed window must NOT alarm (watched=1m0s, limit=26h0m0s); got reason="no PBS snapshot or successful backup in the latest host-report"
|
||
--- FAIL: …/newborn,_1_hour (watched=1h0m0s)
|
||
--- FAIL: …/just_inside_the_window (watched=25h59m0s)
|
||
--- FAIL: …/exactly_at_the_window (watched=26h0m0s)
|
||
--- FAIL: TestCheckBackupDeadlines_NewbornHost_NoEvent (0.03s)
|
||
deadline_anchor_test.go:426: a newborn host must not alarm; got "No fresh verified backup: no PBS snapshot or successful backup in the latest host-report"
|
||
```
|
||
|
||
Restored; full suite green after each.
|
||
|
||
### Scenario D and E
|
||
|
||
- **D** — `TestBackupFreshness_ExistingBehavioursUnchanged` pins all three pre-existing outcomes
|
||
**including their exact reason strings** (`"newest backup is 30h0m0s old (limit 26h0m0s)"`,
|
||
`"newest PBS snapshot failed verification"`, `"latest host-report could not be parsed"`), each with
|
||
fresh window evidence present so the result cannot be an artefact of the new input. Plus
|
||
`TestBackupFreshness_WindowEvidenceDoesNotRescueFailedVerify` — the anchor must not suppress an
|
||
integrity fault.
|
||
- **E** — `TestBackupFreshness_ReasonStringsAreDistinct` fails if any two of the six failure modes
|
||
ever produce the same message.
|
||
|
||
---
|
||
|
||
## 6. Replay of the 2026-07-26 shape — **actual reports, not a fixture**
|
||
|
||
Two replays, both against the real thing.
|
||
|
||
**(a) Real report payloads** extracted from the production hub DB (every retained report in the
|
||
7-day lookback at or before the check instant), fed through `newestBackupEvidence` +
|
||
`assessBackupFreshness` at `now = 2026-07-26 03:00:00 UTC`:
|
||
|
||
```
|
||
demo-felhom rows=600 first=2026-07-18T16:30:51Z windowEvidence=2026-07-25T06:30:14Z -> OK
|
||
demo-hp rows=417 first=2026-07-21T16:24:48Z windowEvidence=2026-07-25T10:23:31Z -> OK
|
||
drill-r50 rows= 77 first=2026-07-25T09:50:55Z windowEvidence=none -> UNKNOWN
|
||
"no backup evidence yet, but only watching for 17h0m0s (grace 26h0m0s since first
|
||
contact 2026-07-25T09:50:55Z) — newborn host, not a fault"
|
||
```
|
||
|
||
**(b) Through the real store queries** — a copy of the live `hub.db` opened with `store.New`, driving
|
||
`GetHostReportsSince` / `GetFirstHostReportAt` / `GetLatestHostReportJSON` for real. This was worth
|
||
doing separately: the window query compares `received_at` as a **SQLite datetime string**, which is a
|
||
genuine correctness risk that a hand-built fixture would not have exercised. Identical verdicts, plus:
|
||
|
||
```
|
||
peti-felhom NO host-report → deadline check skips the backup half entirely
|
||
```
|
||
|
||
**All three would now be silent. Zero events.** Both harnesses were throwaway and are deleted; the
|
||
tree is clean. Only the DB *copy* was ever opened writable — the live `/data/hub.db` was not touched.
|
||
|
||
Note what the verdicts say: demo-felhom and demo-hp are **OK** (real evidence was found, not
|
||
suppressed), and drill-r50 is **UNKNOWN** (correctly deferred as a newborn, and it will alarm on its
|
||
own if it is still backup-less after the grace). None of the three is silenced by a blanket rule.
|
||
|
||
---
|
||
|
||
## 7. Deployment
|
||
|
||
| Step | Result |
|
||
|---|---|
|
||
| Image build + push | `gitea.dooplex.hu/admin/felhom-hub:0.75.0`, 25 MB, digest `sha256:a2ebaa50…10453` |
|
||
| Manifest pin | `manifests/hub.yaml` → `0.75.0` (commit `88b41ec`) — **GitOps only**, no `kubectl set image` |
|
||
| ArgoCD | hard-refresh + deliberate sync → **Synced / Healthy** |
|
||
| Rollout | `deployment "hub" successfully rolled out` |
|
||
| Pod | `hub-56946d5cb4-kwwn7` 1/1 Running |
|
||
| **Running image (pod status, not spec)** | `felhom-hub:0.75.0` @ `sha256:a2ebaa50…10453` — **matches the pushed digest** |
|
||
| Startup log | `[INFO] felhom-hub 0.75.0 starting` … `[INFO] deadline-check: next run at 2026-07-27 05:00 CEST (in 17h14m19s)` |
|
||
|
||
The pin landed and matches — checked deliberately, since the DIAG found `hub.yaml` pinned at
|
||
`0.73.1` while the CHANGELOG read `0.73.2`.
|
||
|
||
---
|
||
|
||
## 8. NOT yet live-validated — explicit
|
||
|
||
1. **The 05:00 CEST deadline check has not fired under v0.75.0.** Next run 2026-07-27 05:00 CEST. All
|
||
evidence above is unit-proven + replayed against real data; the live firing is unobserved.
|
||
2. **The deferred-UNKNOWN INFO line has not appeared in a production log** — proven by unit test only.
|
||
drill-r50's 26 h grace expires 2026-07-26 ~11:50 UTC, so by 03:00 on 07-27 it will instead
|
||
**alarm** — correctly, and that will be the first real test of Scenario B in production.
|
||
3. **No genuinely-dead box has been observed alarming live.** The B path is unit-proven only.
|
||
4. **The cold path (full 7-day lookback scan) has not run in production.** Cost is bounded and the
|
||
healthy path early-exits at one row, but the scan-heavy branch is untimed live.
|
||
5. **demo-hp and drill-r50 host-level state remains uncollected** (no SSH key; break-glass not used) —
|
||
unchanged from the DIAG.
|
||
|
||
---
|
||
|
||
## 9. Observations — recorded, not acted on
|
||
|
||
1. **`parseBackupTime`'s silent `continue` is a latent member of the same class.** An unparseable
|
||
timestamp is dropped without a trace, so a wire-format change on the agent side would present as
|
||
"no backup" — the exact 07-26 failure, with no log line to catch it. Left untouched per scope
|
||
(the agent emits clean RFC3339 `Z`, confirmed in the DIAG). Worth a WARN log when R-84 touches the
|
||
agent side of this contract.
|
||
2. **The customer-facing Hungarian copy is unchanged.** `notify/templates.go:106` — *"A mai
|
||
biztonsági mentés nem készült el a határidőig!"* — still overstates scope (it reads as *all*
|
||
backups failed; this check only covers the host/PBS tier). Not in this task's Parts 1–3. The DIAG
|
||
proposed it as fix #4; it needs a `felhom-ui-design` copy pass and the confirm gate.
|
||
3. **`GetFirstHostReportAt` returns the prune horizon, not true first contact, for hosts older than
|
||
90 days.** Accepted and documented at the function: it only makes the anchor *more* conservative
|
||
for long-lived hosts (the window has elapsed either way) and never shortens a newborn's grace.
|
||
4. **The `backupStaleAfter` landmine is now written into the constant's comment** and carried as a
|
||
named dependency in R-82. It will alarm on a healthy weekly PBS snapshot six days in seven.
|
||
5. **`retention.max_days: 90` is now load-bearing for a monitor**, not just for storage. Shortening it
|
||
below `backupEvidenceLookback` would silently narrow the evidence window. Not currently guarded.
|
||
6. **The R-80 real finding stands untouched:** the PBS/offsite-DR tier has no schedule at all —
|
||
demo-felhom still holds exactly one PBS snapshot (2026-07-18), demo-hp zero. R-81 fixes how the hub
|
||
*reads* backup state; it does nothing about the DR tier being empty. **That is R-82, and it is the
|
||
more serious of the two.**
|
||
7. The customer-channel suppression the DIAG recommended (removing `expected_backup_missed` from
|
||
demo-felhom's `enabled_events`) was **not** applied — still an operator decision, and now much less
|
||
pressing since the false-positive path is closed.
|
||
|
||
---
|
||
|
||
## 10. Scope discipline
|
||
|
||
Not started, recorded only: **R-82** (backup target split — the real finding), **R-83** (ratify or
|
||
retire `07-backup-architecture.md`), **R-84** (persist the agent's backup store — the cause-fix).
|
||
No branches; every commit staged with explicit paths (never `git add -A`, per the shared-worktree
|
||
rule); no `Co-Authored-By`.
|