From f5a5e2b911ad98b7dccff5a9fc024b6ab52c20e9 Mon Sep 17 00:00:00 2001 From: Claude Code Date: Sun, 26 Jul 2026 11:44:15 +0200 Subject: [PATCH] =?UTF-8?q?hub=20v0.75.0:=20R-81=20=E2=80=94=20"no=20signa?= =?UTF-8?q?l"=20is=20not=20"bad=20signal"=20(anchor=20the=20backup=20deadl?= =?UTF-8?q?ine=20check)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Third instance of one class (hub v0.12.0, v0.73.0, this), fixed as a class. On 2026-07-26 03:00 UTC expected_backup_missed fired on demo-felhom, demo-hp and drill-r50 at once; the demo-felhom one reached the CUSTOMER channel claiming "newest backup is 176h0m0s old". Nothing was wrong — three vzdump archives were on disk. Cause: the agent backup store is in-memory, so the R-50 fleet restart emptied `backups` until the next run, and the hub read empty as "no backup exists". - assessBackupFreshness returns OK/UNKNOWN/MISSED instead of `missed bool`; absence is UNKNOWN until it outlives an anchored window. Still pure. - store.GetHostReportsSince + monitor.newestBackupEvidence read the hubs own retained history (bounded 7-day lookback, early-exit on fresh evidence) — "when did I last SEE evidence of a backup?" The anchor was free: the hub already retains 90 days. No agent change, no new persisted state. - store.GetFirstHostReportAt anchors absence at first contact, reusing the existing 26h threshold as the grace (no new knob, the v0.73.0 shape). - Deferrals logged + counted; reason strings kept distinct. - backupStaleAfter untouched; landmine recorded (a weekly PBS snapshot would alarm six days in seven) and owned by R-82. Tests 493->508. Red-proofs A/B/C observed and restored; A reproduces the live message verbatim. Replayed the real 03:00 reports (600/417/77 rows): all three now silent. Source: documentation/audits/DIAG-backup-missed-2026-07-26.md --- CONTEXT.md | 39 ++ REUSE.md | 2 + .../architecture/00-capability-map.md | 3 +- documentation/backlog/ROADMAP.md | 6 +- hub/CHANGELOG.md | 77 ++++ hub/internal/monitor/deadline.go | 220 ++++++++- hub/internal/monitor/deadline_anchor_test.go | 435 ++++++++++++++++++ hub/internal/monitor/deadline_test.go | 12 +- hub/internal/store/store.go | 77 ++++ 9 files changed, 843 insertions(+), 28 deletions(-) create mode 100644 hub/internal/monitor/deadline_anchor_test.go diff --git a/CONTEXT.md b/CONTEXT.md index 9f759a7..750e96b 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,6 +3,45 @@ > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. +## Standing rulings + +**S-1 — N.5 gains a third leg: architecture docs are same-session coupled (2026-07-26, R-81).** +Any task that changes an **architectural contract** — tiers, targets, cadences, trust boundaries — +updates the owning `documentation/architecture/*.md` in the **same session**, under exactly the same +coupling rule that already binds the capability map and the ROADMAP. Origin: R-80/R-81 found +`07-backup-architecture.md` still describing a backup model that the shipped code does not implement +(single target, single cadence), while being cited as authoritative. A stale architecture doc is +worse than a missing one, because it is trusted. + +**S-2 — architecture docs carry an honest status header (2026-07-26, R-81).** +Every `documentation/architecture/*.md` opens with the version it was **verified against** and the +date. A doc more than a few trains behind its subject is marked **STALE** *in that header*, so a +reader meets the warning before the content, not after acting on it. Origin: +`07-backup-architecture.md` — DRAFT since 2026-07-14, verified against controller v0.132.0, now 41 +versions stale (live v0.173.0), and cited as authoritative throughout the R-80 diagnostic. Ratifying +or retiring it is → **R-83**. + +- **2026-07-26 — hub v0.75.0: R-81 SHIPPED — "no signal" is not "bad signal".** Third instance of + one class (hub v0.12.0 `expected_backup_missed` daily-for-everyone; hub v0.73.0 `offsite_stale` + minutes after a healthy repair; now this), fixed **as a class**, with the invariant written at the + head of `assessBackupFreshness` naming all three. `assessBackupFreshness` returns + OK/UNKNOWN/MISSED instead of `missed bool`; absence is UNKNOWN until it outlives an anchored + window. **The anchor was free** — Phase-0 probe found the hub retains 90 days of host-reports, so + `GetHostReportsSince` + `newestBackupEvidence` (7-day bounded lookback, early-exit on fresh + evidence) answer *"when did I last SEE evidence of a backup?"* rather than *"what does the latest + report say?"*; absence is graded from first contact (`GetFirstHostReportAt`) reusing the existing + 26 h threshold as the grace — **no new knob, no agent change, no new persisted state**. + **Rulings:** absence is never silent (a genuinely dead box must still alarm — that is the half the + naive fix breaks, and `TestBackupFreshness_NoEvidenceBeyondAnchor_Alarms` is what makes the + suppression safe); reason strings stay distinct (the whole 07-26 diagnosis turned on reading the + exact string); deferrals are logged so quiet ≠ not-checked. Tests 493→508; red-proofs A/B/C + observed — A reproduces `newest backup is 176h0m0s old` **verbatim the mail demo-felhom sent**. + **Replayed against the real 03:00 reports** (600/417/77 rows): all three now silent. `backupStaleAfter` + deliberately untouched — ⚠️ it will alarm on a healthy WEEKLY PBS snapshot, landmine recorded, + owned by **R-82**. The real finding behind R-80 stands: the **PBS/offsite-DR tier has no schedule + at all** → R-82. Cause-fix (persist the agent store) → R-84. Evidence: + `documentation/audits/DIAG-backup-missed-2026-07-26.md`, `hub/CHANGELOG.md` v0.75.0. + - **2026-07-23 — hub v0.72.0 + controller v0.161.0: R-70 SHIPPED + R-71(c) PARTIAL (the offsite last mile is visible; burned credentials self-heal).** One detector (`offsite.DeliveryStateFor` — secret-row timestamps × report offsite-presence), four consumers: diff --git a/REUSE.md b/REUSE.md index 89e8004..5c0653a 100644 --- a/REUSE.md +++ b/REUSE.md @@ -131,6 +131,7 @@ | `(*Store).RequestLogBundle` / `PendingLogBundleRequest` / `SaveLogBundle` / `PurgeExpiredLogBundles` | hub/internal/store/logbundle.go | component (controller/agent) log pulls — the v0.46.0 sibling of logtail.go | box-component debug-ring pulls; gzip custody, newest-3, 72 h TTL on the 60 s sweep | scope = customer_id (controller/report ACK) vs host_id (agent/heartbeat envelope); `SaveLogBundle` runs the SECRET GATE fail-closed (blocked flag row, no payload) and clears the request in the same tx; `[REDACTED]`/checksums pass by design | | `upsertAppIssue` dismissal/context semantics | hub/internal/store/telemetry.go | ON CONFLICT CASE guards | Issue dismissal + first-capture-wins context | Un-dismiss ONLY on `excluded.last_seen > dismissed_at`; context adopted only while stored one is empty — do not "simplify" either CASE (red-proofed) | | `store.GuestID` | hub/internal/store/store.go (~L1268) | `(hostID string, vmid int) string` | Canonical guest primary key | Never hand-concatenate host+vmid. | +| `(*Store).GetHostReportsSince` + `GetFirstHostReportAt` + `monitor.newestBackupEvidence` | hub/internal/store/store.go, hub/internal/monitor/deadline.go | `(customerID, since) ([]HostReportRow, error)`; `(customerID) (time.Time, error)`; `(rows, now) (time.Time, bool)` | **Asking "when did the hub last SEE evidence of X?" instead of "what does the latest report say?"** — the R-81 anchor. The agent's reporters are point-in-time and forget across a restart; the hub retains ~90 d of host-reports and does not. | The three go together: window scan + first-contact anchor + a bounded lookback (`backupEvidenceLookback`). **Never judge a report-derived absence on the LATEST report alone** — that is the bug class R-81 fixed for the third time. The scan early-exits on sufficiently-fresh evidence, so don't reorder rows away from newest-first. | | `scheduleDaily` | hub/cmd/hub/main.go (~L449) | `(ctx, name, "HH:MM", fn, logger)` | Daily jobs in Europe/Budapest (prune etc.) | Blocking — run as goroutine. `parseHM` returns 0,0 (midnight) on bad input. | ## 2. Canonical patterns (copy structure from THE named file) @@ -156,6 +157,7 @@ | Trap | Why it bites | Use instead | |---|---|---| +| A plain `missed bool` for a report-derived absence (hub/internal/monitor/deadline.go) | Collapsing the three-valued verdict re-introduces one of TWO failure modes: absence→MISSED is the 2026-07-26 cry-wolf (three boxes alarmed at once, one reached a customer channel); absence→OK means a genuinely dead box alarms NEVER, which is strictly worse. Three instances of this class so far: hub v0.12.0, v0.73.0, R-81. | `backupAssessment{verdict: verdictOK|verdictUnknown|verdictMissed}` + an anchored window — copy the shape from `assessBackupFreshness`, not a bool. | | `(*Handler).handleNotify` + `formatNotificationEmail` + `sendResendEmail` (hub/internal/api/handler.go ~L1289/1624/1589) | Legacy pre-dispatcher notification trio: no cooldowns, no operator channel, no allowedEventTypes gate, duplicate Hungarian formatter. Controller path is FROZEN until slice-10 cutover. | `POST /api/v1/event` → `Dispatcher.ProcessEvent` + `notify.Format*Email` | | Severity `"critical"` POSTed to a PRE-v0.31.0 hub | Fixed in hub v0.31.0 (`handleEvent` now accepts critical). Older hubs coerce `critical` → `"info"`, which never notifies — silent alert loss. Case-variants (`"Critical"`) still coerce on every version. | Against an old hub send `warning`/`error`; otherwise lowercase `critical` is safe | | `compareVersions` for anything security-ish (hub/internal/web/server.go ~L571) | Returns 0 (equal) on unparseable input — a garbage version passes a floor check. `gitea.compareSemver` behaves differently (lexical fallback). | Validate input with `normalizeFloorInput` first; then compareVersions is safe | diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index d244add..61fde26 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -111,7 +111,8 @@ | Paired recovery notifications + prefs seeding at claim + priority headers (power-outage audit F11/F12/F14-light) | hub v0.71.0 | **IMPLEMENTED** (recovery leg **PARTIAL** until a live staleness cycle fires it) | `hub/CHANGELOG.md` v0.71.0; 17 tests + 4 red-proofs (`REPORT.md` 2026-07-22); Resend `headers` mechanism probed live (HTTP 200) pre-implementation; operator+customer `test` rows live-fired via the controller's own test endpoint | Recovery = explicit eventType branch, severity semantics frozen; customer gate = PAIRING (`notification_log` evidence), not `enabled_events`. Live legs pending: a natural `*_recovered` mail (next real staleness cycle or the reboot-drill arc — never fabricated by blocking reports) and seed-at-claim on a real claim (Peti Friday reinstall). F14-full (operator push channel, ntfy/Telegram) stays open → R-69 | | System + container metrics (SQLite, Chart.js, 30-day downsampling) | controller | **IMPLEMENTED** | metrics collection + `/monitoring` render present (page 200) | The cited `CAMPAIGN-2` T-RES-CPU/T-SOAK-LOOP are H1/H2 harness artifacts (auth-302), not metrics tests; SQLite/Chart.js/30-day downsampling validated in no campaign. Demoted | | Always-on debug rings + on-demand log-bundle pulls with TTL/custody | controller v0.116, agent v0.83, hub v0.46 | **PROVEN-LIVE** | debug rings live-exercised `CAMPAIGN-3` fix-6 (1000-cap ring, ~55min horizon under load) | The **log-bundle-pull TTL/custody** half is changelog-only (no dedicated observability audit doc); ring persistence across restart is a known gap | -| Operator alerting (Healthchecks → monitoring@felhom.eu) | k3s, Resend | **IMPLEMENTED** | operator infra, stated in production since 02-04; no corpus validation doc | Per the status enum, no citation → not PROVEN-LIVE. Demoted pending an operator-cited live alert (candidate re-upgrade — see REPORT) | +| Operator alerting (Healthchecks → monitoring@felhom.eu) | k3s, Resend | **IMPLEMENTED** | operator infra, stated in production since 02-04; no corpus validation doc | +| Backup-deadline alerting (`expected_backup_missed`) is ANCHORED — absence of signal is UNKNOWN, not failure | hub v0.75.0 | **IMPLEMENTED** | `audits/DIAG-backup-missed-2026-07-26.md` + red-proofs A/B/C + replay of the real 2026-07-26 03:00 reports (all three silent) | **No row status flips** — this signal had a FALSE-POSITIVE class (three instances: hub v0.12.0, v0.73.0, R-81), now anchored at first contact and read across retained host-report history. Still unit-proven only, not live-fired at a real deadline. The *underlying* PBS/offsite-DR tier gap it exposed is → R-82. | Per the status enum, no citation → not PROVEN-LIVE. Demoted pending an operator-cited live alert (candidate re-upgrade — see REPORT) | ## G. Fleet & operator (hub) diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 34d0e52..7361dd0 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -108,7 +108,11 @@ | R-77 | **Endpoint-drift detection, samba protected-set gate, channel log honesty** | S | **SHIPPED (controller v0.173.0 + hub v0.74.0, 2026-07-26)** | Fixes the two defects in `audits/DIAG-agent-channel-2026-07-26.md`. `bootstrap.DetectEndpointDrift` names a `controller.yaml` vs `bootstrap.json` `local_api.endpoint` divergence with its own event type `local_api_endpoint_drift` and its own banner — and **writes nothing** (see R-78). `EffectiveProtected` now gates samba on `Enabled && UserSet`, mirroring both of `reconcileSambaAt`'s early returns, plus the doc-comment correction. Channel log distinguishes born-down (`unseeded->down`) from a real transition, logging-only. **Part 0 repaired both production boxes** (endpoint → `169.254.253.1:8443`; fingerprint+token agreed). Tests 951→959; red-proofs A/E/F. | | R-78 | **`local_api` authority ruling — auto-reconcile vs detect-only** | M | idea (deferred OUT of R-77 on purpose) | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | | R-79 | **`report.Issues` / `report.Warnings` are English on customer-facing surfaces** | M | idea | **Whole-surface, not a one-off** (DIAG §6): every producer is English — `"SSD/HDD disk usage critical"`, `"Docker: %v"`, `"Protected container not running: %s"`, and all six `Warnings` strings. They render on the customer's Hungarian dashboard, and the `health_critical` path has reached the **customer** email channel three times historically. Deliberately NOT bundled into R-77: a copy sweep across every producer would have buried two safety fixes in string churn, and the seam is not obvious — translate at the producer, or at the render/notification boundary where operator-English and customer-Hungarian already diverge? Pick the seam in a spike; the strings are mechanical after. | -| R-80 | **`expected_backup_missed` firing nightly on all three customers — one reaching a CUSTOMER channel** | M | idea — **likely outranks R-77** | 2026-07-26 03:00 fired for demo-felhom, demo-hp and drill-r50; the demo-felhom one went to the **customer** channel reporting `newest backup is 176h0m0s old` — **7.3 days**. **The causal link the DIAG hedged on is unlikely to be the whole story: 7.3 days materially exceeds the ~1.5-day channel outage (2026-07-25 12:44 → 2026-07-26 07:00), so backups were already stale for ~6 days BEFORE the channel broke.** The channel outage may compound it but cannot explain it. This is a customer-visible claim about their data protection and needs its own diagnostic: is it a real backup gap, a stale-report artefact, or a threshold/derivation defect? Do not assume it resolves with R-77's repair — verify against PBS snapshot reality. | +| R-80 | **`expected_backup_missed` false alarm — diagnosed + class-fixed** | M | **SHIPPED (hub v0.75.0, 2026-07-26)** | Diagnosed in `audits/DIAG-backup-missed-2026-07-26.md`, fixed as R-81. **The premise was wrong on both counts:** it fired ONCE (not nightly), and no real external customer was notified — the one customer-channel mail went to the operator's own mailbox; `peti-felhom` (the only real external customer) did not fire. **The 7.3 days did NOT predate the channel outage** — it is the age of the *PBS* snapshot, reached only as a fallback once the vzdump array went empty. Local vzdump was never stale (archives on disk 07-24/25/26). Cause: the agent's backup store is in-memory, so the R-50 fleet restart at 12:44 UTC emptied `backups` until the next backup at 07:03, and the hub read empty as "no backup exists". Fixed by anchoring (hub v0.75.0). **Left standing as a REAL finding → R-82:** the PBS/offsite-DR tier has no schedule at all. | +| R-81 | **"No signal" is not "bad signal" — anchor the backup deadline check** | S | **SHIPPED (hub v0.75.0, 2026-07-26)** | Third instance of one class (hub v0.12.0, v0.73.0, this). `assessBackupFreshness` now returns OK/UNKNOWN/MISSED; absence is UNKNOWN until it outlives an anchored window. The hub reads its own retained host-report history (`GetHostReportsSince` + `newestBackupEvidence`, 7-day bounded lookback) to answer "when did I last SEE evidence of a backup?", and anchors absence at first contact (`GetFirstHostReportAt`), reusing the existing 26 h threshold as the grace — no new knob, the v0.73.0 shape. Deferrals are logged; reason strings kept distinct. Tests 493→508; red-proofs A/B/C. **Not silence** — a genuinely dead box still alarms, which is the half the naive fix breaks. | +| R-82 | **The backup target split — local daily + PBS weekly is NOT EXPRESSIBLE today** | M | idea — **the real finding behind R-80** | `backup.local_backup_target` is a SINGLE target and `BackupCadence()` a SINGLE 24 h window, so a box cannot run local-daily and PBS-weekly at all. `felhom-host-install.sh:2291` pins every new box to `local`, leaving the DR tier provisioned, authenticating and **empty**: demo-felhom holds ONE PBS snapshot (2026-07-18, not from a recurring job), demo-hp has `pbs_dr` applied since 07-21 and **zero snapshots ever**, and there is no `/etc/pve/jobs.cfg`. The offsite-DR promise is currently unbacked on both demo boxes. Must also: (a) **move `backupStaleAfter`** — it is applied to whichever tier is newest with no tier-awareness, so a healthy WEEKLY snapshot would be >26 h old six days in seven and alarm (landmine recorded in the constant's comment, hub v0.75.0); (b) enumerate the state that exists ONLY in the whole-guest snapshot — guest rootfs, `controller.yaml`, `/etc/felhom-bootstrap`, systemd units, offbox SSH key, `settings.json` — and confirm none of it is catastrophic at 7 days old; (c) update `07-backup-architecture.md`. | +| R-83 | **Ratify or retire `07-backup-architecture.md`** | S | idea | DRAFT since 2026-07-14, verified against controller **v0.132.0** — now 41 versions stale (live is v0.173.0) — and cited as authoritative meanwhile. Either ratify it against current reality or mark it superseded; per the new CONTEXT ruling it must in any case carry an honest status header naming its verified-against version. | +| R-84 | **Persist the agent's backup `Store`** | S | idea — the CAUSE-fix behind R-81 | `felhom-agent/internal/backup/store.go` is in-memory by design ("lost on restart; the cadence re-populates"), so every agent restart blanks `backups` in the host-report until the next run. R-81 makes the hub interpret that correctly; this makes the report **truthful** rather than merely defensively interpreted. Either persist latest-per-target to the agent state dir (crash-safe journal/marker patterns already exist) or have the collector fall back to querying PVE for the newest archive on the target — the query path is arguably better, being ground truth rather than remembered state, and would also have surfaced the R-82 PBS gap. Small in code, but it is an agent deploy plus a MinAgent floor, so it is its own task. | | R-76 | **FileBrowser-created folders break the setgid chain, and a drop-zone's mode is not stable** | S | idea (surfaced by the R-75 spike, 2026-07-26) | Two related findings from `audits/SPIKE-catalog-data-paths-2026-07-26.md` P3/P5, both **pre-existing** and deliberately left alone by that spike. **(a)** FileBrowser Quantum 1.3.3 creates files `0644` and folders `0755` and does **not** propagate the setgid bit — even though the entrypoint wrapper's `umask 002` really is in effect (`/proc/1/status` `Umask: 0002`). Group inheritance itself works (a file uploaded into a 2775 group-100 dir landed group 100, not the process gid 1000), so the convention's *group* half holds and only its *mode* half is lost. The consequence is proven with a control: inside a UI-created `0755` folder a gid-1000 process's file landed group **1000**, while the identical write into the 2775 parent landed group **100**. So **any folder a customer creates through FileBrowser breaks the shared-group chain one level down.** Latent today — every userdata-touching catalog app that declares an identity declares uid/gid **1000**, the same uid FileBrowser runs as, so owner permissions mask it; it bites the day a content app runs as a different non-root uid with gid 1000. The comment at `infra/infra.go:156` is right that the image ignores `-e UMASK` but does not say the wrapper fails to achieve the intended mode either. **(b)** `import/calibre` is live on demo-felhom at `755 1000:1000` where every sibling is `2775 root:1000` — with `media/books` on the same box, same app, same deploy, at `2775` as the control, and no parser asymmetry (checked: `ParseComposeUserdataMounts` picks up both calibre-web binds). Consistent with the consuming app rewriting the mode of its own ingest dir after the deploy belt sets it; **not confirmed causally**. Together they mean **a drop-zone directory's mode is not stable against either the customer or the consuming app** — verify before building anything that assumes `import/*` stays 2775. Flips no capability-map row today (latent); would become customer-visible the moment a non-1000 content app enters the catalog | | R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → universal secret-free / operator-bind (option C) | XL | **SHIPPED + PHYSICALLY CLOSED (slices A+B+C; rehearsal executed 2026-07-18)** | **PHYSICAL CLOSURE 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the generic pairing ISO v1.20.0 (`--loader mkimage`, SB off) **booted the very AMI board that F1 blocked**, installed unattended, and the box self-registered as an unclaimed appliance the same second it first booted (16:17:14) → self-bind → credential → day-0 SUCCESS 16:32:32 → floor-lifted to current. **F1 is closed on physical hardware and the rehearsal dependency on this item is discharged.** Two residual notes stay open, neither blocking: PXE/network-boot is still unbuilt (the third F1 option, for boards where even USB-mkimage fails), and mkimage remains unsigned → **SB must be OFF**. The installer's GRUB menu still offers interactive installers → new item **R-38**. — Prior: **PHYSICAL RUN 2026-07-16 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`):** demo N100 reinstalled clean-slate from a pipeline ISO → chain reached **rc-0 first try on real hardware** (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. **F1 (HIGH, slice-B input):** this cheap AMI `AN3PLUS 0.01` firmware won't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`) — SB-off/shim-bypass don't help; worked around live with a `grub-mkimage` loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B — SHIPPED (scripts v1.18.0, 2026-07-17):** the F1 firmware fix is now a first-class pipeline mode `build-felhom-iso.sh --loader shim|mkimage` (default shim; `mkimage` = monolithic grub-mkimage loader from the ISO's own GRUB, recipe from the run evidence). RUNBOOK-B legs on nested VM 311 proved it: shim boots+installs under OVMF SB-enforcing + SeaBIOS; mkimage boots+installs under OVMF SB-off; mkimage under SB-enforcing FAILS `Access Denied` (unsigned → **SB must be OFF**, documented); surgery byte-identical payload. **Physical N100 boot on the real board still pending** → folds into the supervised rehearsal (R-1; an `n100-safety` match-nothing ISO is built + sha-recorded for a zero-risk pre-flight). **PXE/network-boot** (the third F1 option, for boards where even USB-mkimage fails) stays a deferred note under this item — not built. **Slice C — SHIPPED (hub v0.62.0 + scripts v1.19.0, 2026-07-17):** the GENERIC secret-free universal ISO (`build-felhom-iso.sh --pairing`). The box self-registers as an unclaimed appliance (keyed by SMBIOS-uuid + MAC set — the DMI-verdict tiebreaker), the operator BINDS it to a customer on the Hosts page, and the hub delivers customer-id + retrieval passphrase ONCE (`/api/v1/appliance/register` + one-shot poll, 404-no-oracle — all live-verified through the public ingress); the bootstrap then falls through to the slice-A direct path. **One unit, two modes** (direct = byte-identical, regression-proven zero-appliance-calls). Artifact proven secret-free (baked env = hub URL only; manifest `secret-bearing: no`). The **bind is operator-password-gated** → the live boot→register→bind→day-0 composition (with a Viktor-created drill customer) + the physical N100 boot fold into the supervised rehearsal — **which now runs the COMPLETE final product flow in one pass: RESET the demo → boot the generic ISO → bind → day-0**. SSH-host-key pinning: stored + fingerprints displayed (attaching to the host on bind = future, no clean hand-off surface today). **Customer-facing self-bind page = R-27 (future).** Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* | **SECOND-HARDWARE PROOF 2026-07-21 (slice C, demo-hp): the pairing flow is no longer a one-board result.** The universal secret-free ISO was booted on a completely different machine — an **HP t740 (Ryzen V1756B, AMI M42 firmware)** versus the N100 it was proven on — and the whole chain ran on **virgin hardware in one pass**: armed install → self-registration as an unclaimed appliance → operator bind → day-0, ending with a running customer guest 9201 and agent 0.92.1 checking in as host `demo-hp-bb76ea`. **Two things generalise from the second board specifically:** the **shim** loader booted with **Secure Boot ENABLED** (`mokutil --sb-state` → `SecureBoot enabled`), confirming the mkimage/SB-off dance is an N100-firmware workaround and NOT a Felhom requirement; and the exact-serial disk filter selected the SanDisk system SSD while leaving the box's **1TB NVMe untouched and unenrolled** (its prior NTFS partition is still intact, unmounted, in no LVM/ZFS) — the destructive path stayed inside its filter on hardware it had never seen. **Not clean, and the failures are filed:** the install got no DHCP on the 4-port NIC and baked a static fallback rather than aborting (**R-59**), which cost a cable move and a hand-repair; and the console was unreachable because the baked root password is unknowable (**R-61**) | R-29 | **The design-v2 green gates are not enforced anywhere — one has been RED for 16 releases.** `controller/scripts/docker_run_volume_path_gate.py` has failed continuously since **2026-07-14 (v0.129.0)** and nobody noticed until R-7b's close-out ran it by hand at v0.145.0. Two separable parts. **(a) The finding itself is benign and the fix is 3 lines.** The flagged call is `internal/appexport/estimate.go:179` `docker run --rm -v :/vol:ro alpine du` — a **NAMED-VOLUME** mount, i.e. daemon-side with no host path, which is the *safe* shape and byte-for-byte the same pattern as three entries already on the gate's ALLOWLIST (`export.go` `volName+":/vol"`, `backup.go` `volName+":/vol:ro"`, `restore.go` `volName+":/vol"`). It is NOT the v0.124.0 path-strand class the gate exists to catch — the author of the v0.129.0 F-A fix explicitly avoided that class (see the function's own comment) and simply never added the allowlist entry. So the fix is an ALLOWLIST addition WITH ITS WHY, **not** a docker-cp rewrite; anyone who 'fixes' this by rewriting the call has misread the gate. **(b) The systemic half is the real item:** the gates run only when a human remembers to run them, so a gate can sit red across 16 releases while every REPORT says 'green'. This is the SECOND instance of the class — cf. the v0.123.0 note *'Windows green gate silently red (read-only fsync)'*. Decide where they run (pre-push hook, `build.sh` step, or a CI job) and make a red gate block the train the way the Go green gate does. | S (a) / M (b) | idea | Origin: R-7b close-out, `felhom-controller` REPORT §4(f) — CC correctly left it alone as out-of-scope and pre-existing, and verified by stashing that it fails identically on the unmodified tree. Flips no capability-map row (engineering hygiene, no customer-visible behaviour). Affected gates to audit for the same rot: controller `template_id_gate` / `emoji_gate` / `native_confirm_gate` / `offbox_rename_gate` / `mojibake_gate` / `app_row_dedup_gate` / `docker_run_volume_path_gate`, hub `hub_confirm_gate`, manifests `manifest_bearer_gate`, website `site_gates`. **Do not bundle (a) into an unrelated feature commit** — it is a one-line behavioural claim about a mount's safety and deserves its own reviewed diff. **2026-07-18 rehearsal note:** the run's finding list independently re-raised "assign the pre-existing `docker_run_volume_path_gate` failure its ID so red stops normalizing" — **that is this item; no second ID was minted.** | diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index 87fcc5b..5dae086 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,5 +1,82 @@ # Felhom Hub — Changelog +## v0.75.0 — R-81: "no signal" is not "bad signal" — the backup deadline check is ANCHORED (2026-07-26) + +The third instance of one bug class, fixed as a class. `expected_backup_missed` fired on +demo-felhom, demo-hp and drill-r50 simultaneously at 03:00 UTC, and the demo-felhom one reached the +**customer** channel claiming `newest backup is 176h0m0s old`. Nothing was wrong: three vzdump +archives were on disk (07-24, 07-25, 07-26). Full evidence: +`documentation/audits/DIAG-backup-missed-2026-07-26.md`. + +**Cause.** The agent's backup record store is IN-MEMORY +(`felhom-agent/internal/backup/store.go` — *"lost on restart; the cadence re-populates"*). The R-50 +island migration restarted the fleet at 12:44 UTC; the next backup landed at 07:03 the following +morning. In between, every host-report carried `backups: []`, and `assessBackupFreshness` read empty +as *no backup exists*. For demo-felhom it then fell through to the only surviving evidence — a PBS +snapshot from 07-18 — and reported its age as the customer's backup age. + +**The class.** hub v0.12.0 (`expected_backup_missed` daily for every healthy customer — looked for +an event nobody emits), hub v0.73.0 (`offsite_stale` minutes after a *healthy* repair — never-ran +branch had no time anchor), and now this. All three: **absence of signal treated as evidence of +failure.** The invariant is now written at the head of `assessBackupFreshness` with all three +instances named, and pinned by a boundary test whose name says what it protects. + +### Changed +- **`assessBackupFreshness` returns a three-valued verdict** — `verdictOK` / `verdictUnknown` / + `verdictMissed`, replacing `missed bool`. Absence is UNKNOWN, not a fault. It becomes a fault only + once it outlives an anchored window. Still **pure** (`now` and the evidence are injected) — that + purity is why the incident was diagnosable and why this fix is provable. +- **The check now reads hub HISTORY, not just the latest report.** New + `store.GetHostReportsSince` + `monitor.newestBackupEvidence` answer *"when did I last SEE evidence + of a backup?"* across a bounded 7-day lookback (`backupEvidenceLookback`). The agent's store is + point-in-time and forgets across a restart; the hub's retained reports (90 d) do not. **This is the + whole fix for the 07-26 shape** — no agent change, no new persisted state, and semantically exactly + the right question. The scan stops at the first sufficiently-fresh evidence, so the healthy path + reads one row; only the genuinely-broken path walks the lookback. +- **The absence anchor is first contact** — new `store.GetFirstHostReportAt`. Absence is graded + against how long the hub has been *watching*, reusing the existing `backupStaleAfter` (26 h) as the + grace exactly as v0.73.0 reused offsite `staleAfter`. **No new knob.** A zero anchor fails toward + visibility (the v0.73.0 legacy-shape precedent). +- **`CheckBackupDeadlines` logs the deferral.** A deferred UNKNOWN emits one INFO naming the reason, + and the summary line gained a `backup unknown (deferred)` counter — so a quiet check is never + indistinguishable from a check that did not run (v0.73.0 Part-7 precedent). At most one line per + customer per day. +- **Reason strings split, not collapsed.** Absence-over-time, unanchored absence, deferred-newborn, + stale-timestamp, failed-verify and unparseable each keep a distinct message. The entire 07-26 + diagnosis turned on reading the exact string; a test enforces distinctness. + +### NOT changed (deliberate) +- `backupStaleAfter` stays 26 h, and no tier-awareness was built. ⚠️ **Landmine recorded in the + constant's comment:** it applies to whichever tier is newest, so once PBS moves to a **weekly** + cadence a healthy weekly snapshot is >26 h old six days in seven and this will alarm on it. + Per-tier thresholds cannot be built before the per-tier cadence config exists — **R-82 owns both + halves.** Building it now would be speculative generality. +- `parseBackupTime` untouched — the agent emits clean RFC3339 `Z` and the parse branch is not + implicated. Its silent `continue` on an unparseable timestamp is a **latent member of this same + class** and is recorded as an observation only. +- The DB-dump half of `CheckBackupDeadlines` is event-based and correct — untouched. +- The customer-facing Hungarian copy (`notify/templates.go:106`) is untouched here. The DIAG found it + overstates scope (it reads as *all* backups failed, but this check only covers the host/PBS tier); + that copy change was not in this task's scope. +- The agent's in-memory `Store` is the *cause*; making the host-report truthful rather than merely + defensively interpreted is **R-84**. + +### Tests +508 total (was 493), +15 in `internal/monitor/deadline_anchor_test.go`. Companion red-proofs observed +and restored for all three acceptance scenarios: +- **A** (restart blind window must not alarm) — removing the history fold-in reproduces + `newest backup is 176h0m0s old (limit 26h0m0s)`, **verbatim the message demo-felhom actually sent**. +- **B** (a genuinely dead box must still alarm) — the naive "absence is always silent" fix fails + `TestBackupFreshness_NoEvidenceBeyondAnchor_Alarms` and two contract rows. This is the test that + makes A safe: a suite proving only A would pass against an implementation that never alarms. +- **C** (a fresh box is not born failing) — the literal pre-fix branch fails four contract rows plus + the end-to-end newborn case. + +**Replayed against the real thing:** the actual host-reports the hub held at 2026-07-26 03:00 UTC +(600 / 417 / 77 retained rows) fed through the new policy → demo-felhom **OK** (window evidence +`2026-07-25T06:30:14Z`), demo-hp **OK** (`2026-07-25T10:23:31Z`), drill-r50 **UNKNOWN** (newborn, +watched 17 h < 26 h grace). **All three silent.** + ## v0.74.0 — allow `local_api_endpoint_drift` (controller v0.173.0 / R-77) (2026-07-26) One line in `allowedEventTypes`. It is **not optional**: `handleEvent` 400s an unknown `event_type` diff --git a/hub/internal/monitor/deadline.go b/hub/internal/monitor/deadline.go index ab61a83..ec18f13 100644 --- a/hub/internal/monitor/deadline.go +++ b/hub/internal/monitor/deadline.go @@ -14,8 +14,27 @@ import ( // deadline check raises expected_backup_missed. 26h covers an evening backup schedule // (e.g. ~18:00–22:00) plus headroom, so a healthy once-daily cadence never trips the // early-morning check. +// +// R-81 also reuses it as the ABSENCE window (see assessBackupFreshness): the existing +// threshold, anchored at first-contact, IS the newborn grace — no new knob, exactly as +// v0.73.0 reused offsite staleAfter for its never-ran anchor. +// +// ⚠️ LANDMINE — dependency on R-82 (the backup target split). This constant is applied to +// whichever tier is NEWEST, PBS or vzdump, with no tier-awareness. Today a daily vzdump +// always wins, so PBS's own age is invisible here and 26h is harmless. The moment PBS moves +// to a WEEKLY cadence, a perfectly healthy weekly snapshot is >26h old six days in seven and +// this constant alarms on it. Fixing that means per-tier thresholds, which cannot be built +// before the per-tier cadence config exists (`local_backup_target` is a single target and +// BackupCadence() a single 24h window today). Do NOT pre-build it — R-82 owns both halves. const backupStaleAfter = 26 * time.Hour +// backupEvidenceLookback bounds how far back the hub looks for evidence that a backup ever +// happened, when the LATEST report carries none. Generous against any plausible daily cadence +// (and against an agent that stayed restarted for days), bounded so the cold path can't turn +// into a full-retention scan of every report the hub holds. Beyond this the verdict is +// "no evidence in the lookback", which is a fault in its own right once the anchor elapsed. +const backupEvidenceLookback = 7 * 24 * time.Hour + // hostReportBackups is the minimal slice of an agent host-report the deadline check // reads to judge backup freshness (pbs_snapshots is the offsite-DR signal; backups is // the local vzdump fallback). Mirrors the agent's hub.PBSSnapshot / hub.Backup wire @@ -31,28 +50,86 @@ type hostReportBackups struct { } `json:"backups"` } +// backupVerdict is the three-valued outcome of the freshness policy. The middle value is the +// whole point of R-81: "I have no evidence" is NOT "the backup failed". +type backupVerdict int + +const ( + verdictOK backupVerdict = iota // positive evidence of a recent backup + verdictUnknown // no evidence yet, and the anchored window has not elapsed + verdictMissed // positive evidence of a problem — alarm +) + // backupAssessment is the verdict for one customer's offsite backup health. type backupAssessment struct { - missed bool // raise expected_backup_missed - reason string // human-readable cause (event message + logs) + verdict backupVerdict + reason string // human-readable cause (event message + logs) } -// assessBackupFreshness decides whether a customer's latest host-report shows a healthy, -// recent backup. Pure (now is injected) so the policy is unit-tested. Only POSITIVE -// evidence of a problem fires an alarm: -// - no PBS snapshot AND no successful vzdump in the report → missed ("no backup recorded") -// - newest backup older than backupStaleAfter → missed ("stale") -// - the newest PBS snapshot's verify_state is "failed" → missed ("verify failed") +// missed reports whether this assessment should raise expected_backup_missed. +func (a backupAssessment) missed() bool { return a.verdict == verdictMissed } + +// backupEvidence is the hub-history half of the freshness policy, resolved by the caller so +// assessBackupFreshness stays PURE (see its doc comment on why that matters). +type backupEvidence struct { + // newestSeen is the newest backup evidence found across the retained host-report window + // (PBS backup_time or successful vzdump started_at), regardless of whether the LATEST + // report still carries it. haveSeen is false when the window held none. + newestSeen time.Time + haveSeen bool + + // firstReportAt is when the hub first saw ANY host-report from this customer — the + // observation anchor. Zero when unknown, which the policy treats as "cannot defer" + // (fail toward visibility, matching the v0.73.0 zero-anchor branch). + firstReportAt time.Time +} + +// assessBackupFreshness decides whether a customer's backups are healthy. Pure (now and the +// hub-history evidence are injected) so the policy is unit-tested — that purity is why the +// 2026-07-26 incident could be diagnosed at all, and why the fix below is provable. +// +// ── THE INVARIANT (R-81) ────────────────────────────────────────────────────────────────── +// +// Only POSITIVE EVIDENCE OF A PROBLEM raises an alarm. ABSENCE OF A SIGNAL IS UNKNOWN, +// and becomes a fault only once that absence has persisted beyond an ANCHORED window. +// +// This monitor family has made the same mistake three times, and it is written down here so +// the fourth is harder: +// - hub v0.12.0 — `expected_backup_missed` fired daily for every healthy customer, because +// it looked for a `backup_completed` event that no component emits anymore. +// - hub v0.73.0 — `offsite_stale` fired minutes after a HEALTHY repair, because the +// never-ran branch had no time anchor. Fixed by anchoring, not by silence. +// - R-81 (this) — `expected_backup_missed` fired on three boxes at once on 2026-07-26, +// because an agent restart empties the host-report `backups` array (the agent's store is +// in-memory) and empty was read as "no backup exists". The vzdump had in fact run. +// +// Note the shape of the fix in all three: NOT silence. Silence is the opposite failure — a +// box that genuinely never backs up would then alarm never, which is strictly worse than +// crying wolf. Absence is deferred, then alarmed on, with its own distinct reason string. +// +// ── THE BRANCHES ────────────────────────────────────────────────────────────────────────── +// +// unparseable latest report → missed ("could not be parsed") +// newest evidence (report OR window) older than staleAfter → missed ("newest backup is Xh old") +// newest PBS snapshot's verify_state == "failed" → missed ("failed verification") +// NO evidence anywhere, anchor NOT yet elapsed → UNKNOWN, silent + logged +// NO evidence anywhere, anchor elapsed → missed ("no backup evidence in …") +// otherwise → ok +// +// Each failure mode keeps its OWN reason string. That is not polish: the entire 2026-07-26 +// diagnosis turned on reading the exact string, and collapsing them would have made it +// impossible. In particular "absence over time" and "a timestamp that is too old" are +// different faults with different causes, and must never share a message. // // A fresh-but-not-yet-verified snapshot (verify_state "none"/"") is NOT treated as a // failure: PBS verification runs on its own cadence, so a snapshot taken hours before the // 03:00 check may legitimately be unverified. Alarming on that would re-introduce exactly -// the daily false alarm this repoint removes (hence "failed" only, not "≠ ok"). -func assessBackupFreshness(reportJSON string, now time.Time) backupAssessment { +// the daily false alarm the v0.12.0 repoint removed (hence "failed" only, not "≠ ok"). +func assessBackupFreshness(reportJSON string, ev backupEvidence, now time.Time) backupAssessment { var hr hostReportBackups if err := json.Unmarshal([]byte(reportJSON), &hr); err != nil { // Unparseable report → can't confirm a backup. Surface it rather than swallow it. - return backupAssessment{missed: true, reason: "latest host-report could not be parsed"} + return backupAssessment{verdict: verdictMissed, reason: "latest host-report could not be parsed"} } var newestPBS time.Time @@ -86,21 +163,92 @@ func assessBackupFreshness(reportJSON string, now time.Time) backupAssessment { } } - if !havePBS && !haveVzdump { - return backupAssessment{missed: true, reason: "no PBS snapshot or successful backup in the latest host-report"} + // The newest evidence the LATEST report itself carries. + newest := newestPBS + haveNewest := havePBS + if haveVzdump && (!haveNewest || newestVzdump.After(newest)) { + newest, haveNewest = newestVzdump, true } - newest := newestPBS - if haveVzdump && (!havePBS || newestVzdump.After(newest)) { - newest = newestVzdump + // R-81: fold in what the hub REMEMBERS. The agent's store is point-in-time and forgets + // across a restart; the hub's retained host-reports do not. An empty array in the latest + // report therefore says nothing on its own — the question is when evidence was last SEEN, + // not whether this one report happens to carry it. + if ev.haveSeen && (!haveNewest || ev.newestSeen.After(newest)) { + newest, haveNewest = ev.newestSeen, true } + + if !haveNewest { + // ABSENCE. Not a failure by itself — see the invariant above. It becomes one only + // once it has outlived the existing threshold, counted from first contact (the point + // at which a backup first became possible to observe). + if ev.firstReportAt.IsZero() { + // No anchor to defer against — fail toward visibility, as v0.73.0 does for the + // legacy zero-anchor shape. Distinct string: this is an unanchored absence. + return backupAssessment{ + verdict: verdictMissed, + reason: "no backup evidence in any retained host-report, and no first-contact anchor to defer against", + } + } + watched := now.Sub(ev.firstReportAt) + if watched <= backupStaleAfter { + return backupAssessment{ + verdict: verdictUnknown, + reason: fmt.Sprintf("no backup evidence yet, but only watching for %s (grace %s since first contact %s) — newborn host, not a fault", + watched.Round(time.Hour), backupStaleAfter, ev.firstReportAt.Format(time.RFC3339)), + } + } + return backupAssessment{ + verdict: verdictMissed, + reason: fmt.Sprintf("no backup evidence in any host-report for %s (limit %s, first contact %s, lookback %s)", + watched.Round(time.Hour), backupStaleAfter, ev.firstReportAt.Format(time.RFC3339), backupEvidenceLookback), + } + } + if age := now.Sub(newest); age > backupStaleAfter { - return backupAssessment{missed: true, reason: fmt.Sprintf("newest backup is %s old (limit %s)", age.Round(time.Hour), backupStaleAfter)} + return backupAssessment{verdict: verdictMissed, reason: fmt.Sprintf("newest backup is %s old (limit %s)", age.Round(time.Hour), backupStaleAfter)} } if havePBS && newestPBSVerify == "failed" { - return backupAssessment{missed: true, reason: "newest PBS snapshot failed verification"} + return backupAssessment{verdict: verdictMissed, reason: "newest PBS snapshot failed verification"} } - return backupAssessment{missed: false} + return backupAssessment{verdict: verdictOK} +} + +// newestBackupEvidence scans the customer's retained host-reports (newest first) for the most +// recent backup evidence — a PBS snapshot backup_time or a SUCCESSFUL vzdump started_at — +// regardless of whether the latest report still carries it. +// +// Cost discipline: it stops as soon as it has found evidence FRESH enough that nothing older +// could change the verdict, so the healthy path reads one row. Only the genuinely-broken path +// (no fresh evidence anywhere) walks the full lookback, and that is bounded by +// backupEvidenceLookback rather than by retention. +func newestBackupEvidence(rows []store.HostReportRow, now time.Time) (time.Time, bool) { + var newest time.Time + have := false + for _, r := range rows { + var hr hostReportBackups + if err := json.Unmarshal([]byte(r.ReportJSON), &hr); err != nil { + continue // a single malformed retained report must not blind the scan + } + for _, ps := range hr.PBSSnapshots { + if t, ok := parseBackupTime(ps.BackupTime); ok && (!have || t.After(newest)) { + newest, have = t, true + } + } + for _, b := range hr.Backups { + if !b.Success { + continue + } + if t, ok := parseBackupTime(b.StartedAt); ok && (!have || t.After(newest)) { + newest, have = t, true + } + } + // Fresh evidence found → the age branch cannot fire and nothing older matters. + if have && now.Sub(newest) <= backupStaleAfter { + break + } + } + return newest, have } // parseBackupTime parses an RFC3339 timestamp from a host-report and normalizes to UTC. @@ -153,7 +301,7 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E midnightBudapest := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, budapest) sinceUTC := midnightBudapest.UTC() - var backupMissed, dbdumpMissed, skipped int + var backupMissed, dbdumpMissed, skipped, deferred int for _, id := range customerIDs { // Skip nodes that are down — they already have staleness events @@ -179,7 +327,26 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E // has no PBS data to judge here and must not emit a daily backup alarm of its // own. (The DB-dump half below still applies.) default: - if a := assessBackupFreshness(reportJSON, time.Now().UTC()); a.missed { + nowUTC := time.Now().UTC() + + // R-81: resolve the hub-history half BEFORE judging. A read failure here must not + // invent a fault — it degrades to "the latest report is all I know", which is the + // pre-R-81 behaviour, and is logged rather than swallowed. + var ev backupEvidence + if rows, rerr := s.GetHostReportsSince(id, nowUTC.Add(-backupEvidenceLookback)); rerr != nil { + logger.Printf("[WARN] Deadline check: failed to read host-report window for %s: %v", id, rerr) + } else { + ev.newestSeen, ev.haveSeen = newestBackupEvidence(rows, nowUTC) + } + if first, ferr := s.GetFirstHostReportAt(id); ferr != nil { + logger.Printf("[WARN] Deadline check: failed to read first host-report for %s: %v", id, ferr) + } else { + ev.firstReportAt = first + } + + a := assessBackupFreshness(reportJSON, ev, nowUTC) + switch a.verdict { + case verdictMissed: msg := "No fresh verified backup: " + a.reason if _, err := s.SaveEvent(id, "expected_backup_missed", "error", msg, "{}", "hub"); err != nil { logger.Printf("[WARN] Failed to save expected_backup_missed for %s: %v", id, err) @@ -187,6 +354,13 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E onEvent(id, "expected_backup_missed", "error", msg, "{}", "hub") } backupMissed++ + case verdictUnknown: + // Make the deferral VISIBLE (the v0.73.0 Part-7 precedent): a quiet check must + // never be indistinguishable from a check that did not run. This check fires + // once daily, so this is at most one line per customer per day — not spam, and + // it is the line that proves the deferral happened rather than an error. + logger.Printf("[INFO] Deadline check: %s backup verdict UNKNOWN (no alarm) — %s", id, a.reason) + deferred++ } } @@ -204,6 +378,6 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E } } - logger.Printf("[INFO] Deadline check: %d customers, %d backup missed, %d dbdump missed, %d skipped (down)", - len(customerIDs), backupMissed, dbdumpMissed, skipped) + logger.Printf("[INFO] Deadline check: %d customers, %d backup missed, %d backup unknown (deferred), %d dbdump missed, %d skipped (down)", + len(customerIDs), backupMissed, deferred, dbdumpMissed, skipped) } diff --git a/hub/internal/monitor/deadline_anchor_test.go b/hub/internal/monitor/deadline_anchor_test.go new file mode 100644 index 0000000..47ecaa2 --- /dev/null +++ b/hub/internal/monitor/deadline_anchor_test.go @@ -0,0 +1,435 @@ +package monitor + +import ( + "encoding/json" + "log" + "strings" + "testing" + "time" + + "gitea.dooplex.hu/admin/felhom-hub/internal/store" +) + +// R-81 — "no signal" is not "bad signal". +// +// Origin: 2026-07-26 03:00 UTC, expected_backup_missed fired on demo-felhom, demo-hp and +// drill-r50 at once. The agent's backup record store is IN-MEMORY +// (felhom-agent/internal/backup/store.go — "lost on restart; the cadence re-populates"), so +// the R-50 island migration's fleet restart at 12:44 UTC emptied the host-report `backups` +// array until the next backup at 07:03. The check read empty as "no backup exists". The +// vzdump had in fact run: three archives were on disk (07-24, 07-25, 07-26). +// +// The fix is an ANCHOR, not silence — the v0.73.0 offsite never-ran shape. These tests pin +// BOTH halves: absence must stop crying wolf (A, C) AND must still alarm when it is real (B). +// A test suite that only proved A would pass against an implementation that never alarms, +// which is strictly worse than the bug it replaces. + +// evidenceAt builds a backupEvidence with hub-history evidence at `ago` before now, and a +// first-contact anchor `watched` before now. +func evidenceAt(now time.Time, ago, watched time.Duration) backupEvidence { + return backupEvidence{ + newestSeen: now.Add(-ago), + haveSeen: true, + firstReportAt: now.Add(-watched), + } +} + +// noEvidence builds a backupEvidence with NO backup evidence, watched for `watched`. +func noEvidence(now time.Time, watched time.Duration) backupEvidence { + return backupEvidence{firstReportAt: now.Add(-watched)} +} + +// ── Scenario A — the 2026-07-26 case must NOT alarm ────────────────────────────────────── + +// COMPANION RED-PROOF (observed): removing the ev.haveSeen fold-in from +// assessBackupFreshness (pre-R-81 shape: judge the latest report alone) makes this test fail +// with: +// +// deadline_anchor_test.go: 07-26 shape must NOT alarm; got verdict=2 reason= +// "newest backup is 176h0m0s old (limit 26h0m0s)" +// +// which is verbatim the message demo-felhom actually sent that morning. Restored after. +func TestBackupFreshness_AgentRestartBlindWindow_NoAlarm(t *testing.T) { + now := time.Date(2026, 7, 26, 3, 0, 0, 0, time.UTC) + // The real demo-felhom shape: backups:[] (agent forgot), pbs_snapshots:[one from 07-18]. + report := `{"pbs_snapshots":[{"backup_time":"2026-07-18T18:31:06Z","verify_state":"ok"}],"backups":[]}` + // The hub's retained window still holds the 07-25 06:30 vzdump — ~20.5h before the check. + ev := evidenceAt(now, 20*time.Hour+30*time.Minute, 8*24*time.Hour) + + got := assessBackupFreshness(report, ev, now) + if got.missed() { + t.Fatalf("07-26 shape must NOT alarm; got verdict=%d reason=%q", got.verdict, got.reason) + } + if got.verdict != verdictOK { + t.Fatalf("evidence exists in the window → verdict must be OK, not a deferred UNKNOWN; got verdict=%d reason=%q", got.verdict, got.reason) + } +} + +// The demo-hp / drill-r50 shape: BOTH arrays empty, nothing in the window either, but the +// hub has been watching less than the threshold → UNKNOWN, not an alarm. +func TestBackupFreshness_EmptyArraysWithinGrace_Unknown(t *testing.T) { + now := time.Date(2026, 7, 26, 3, 0, 0, 0, time.UTC) + got := assessBackupFreshness(`{"pbs_snapshots":[],"backups":[]}`, noEvidence(now, 10*time.Hour), now) + if got.missed() { + t.Fatalf("absence inside the anchored grace must NOT alarm; got reason=%q", got.reason) + } + if got.verdict != verdictUnknown { + t.Fatalf("want verdictUnknown, got verdict=%d reason=%q", got.verdict, got.reason) + } +} + +// ── Scenario B — a genuinely dead box MUST still alarm ─────────────────────────────────── + +// THE TEST THAT MAKES SCENARIO A SAFE. Without it, "return silent on absence" passes A and C. +// +// COMPANION RED-PROOF (observed): replacing the elapsed-anchor branch with an unconditional +// `return backupAssessment{verdict: verdictUnknown, ...}` (the naive over-suppression fix) +// makes this test fail with: +// +// deadline_anchor_test.go: a box with NO backup evidence for 240h MUST alarm; got verdict=1 +// reason="no backup evidence yet, but only watching for 240h0m0s ..." +// +// Restored after. +func TestBackupFreshness_NoEvidenceBeyondAnchor_Alarms(t *testing.T) { + now := time.Date(2026, 7, 26, 3, 0, 0, 0, time.UTC) + got := assessBackupFreshness(`{"pbs_snapshots":[],"backups":[]}`, noEvidence(now, 240*time.Hour), now) + if !got.missed() { + t.Fatalf("a box with NO backup evidence for 240h MUST alarm; got verdict=%d reason=%q", got.verdict, got.reason) + } + // Scenario E: the reason must name absence-over-time, NOT bare absence and NOT staleness. + if !strings.Contains(got.reason, "no backup evidence in any host-report for") { + t.Fatalf("absence-over-time needs its own reason string; got %q", got.reason) + } + if strings.Contains(got.reason, "newest backup is") { + t.Fatalf("absence must NOT reuse the stale-timestamp string; got %q", got.reason) + } +} + +// A zero anchor (hub holds no first-contact time) must fail toward VISIBILITY, not silence — +// the v0.73.0 legacy zero-anchor precedent. +func TestBackupFreshness_NoEvidenceNoAnchor_Alarms(t *testing.T) { + now := time.Date(2026, 7, 26, 3, 0, 0, 0, time.UTC) + got := assessBackupFreshness(`{"pbs_snapshots":[],"backups":[]}`, backupEvidence{}, now) + if !got.missed() { + t.Fatalf("unanchored absence must fail toward visibility; got verdict=%d reason=%q", got.verdict, got.reason) + } + if !strings.Contains(got.reason, "no first-contact anchor") { + t.Fatalf("unanchored absence needs its own reason string; got %q", got.reason) + } +} + +// ── Scenario C — a fresh box is not born failing ───────────────────────────────────────── + +// THE NAMED BOUNDARY CONTRACT (Part 2). "No evidence + no elapsed window → no alarm" is +// pinned here as a contract with an obvious name, so re-introducing the bug requires deleting +// a test that says what it is protecting. Exercises both sides of the boundary. +// +// COMPANION RED-PROOF (observed): pre-fix (`if !havePBS && !haveVzdump { return missed }`), +// the 1-minute-old newborn fails with: +// +// deadline_anchor_test.go: CONTRACT VIOLATED: no evidence + no elapsed window must NOT +// alarm (watched=1m0s, limit=26h0m0s); got reason="no PBS snapshot or successful backup in +// the latest host-report" +// +// Restored after. +func TestBackupFreshness_Contract_AbsenceIsUnknownUntilAnchorElapses(t *testing.T) { + now := time.Date(2026, 7, 26, 3, 0, 0, 0, time.UTC) + empty := `{"pbs_snapshots":[],"backups":[]}` + + cases := []struct { + name string + watched time.Duration + wantMissed bool + }{ + {"newborn, 1 minute", time.Minute, false}, + {"newborn, 1 hour", time.Hour, false}, + {"just inside the window", backupStaleAfter - time.Minute, false}, + {"exactly at the window", backupStaleAfter, false}, // <= is grace, not fault + {"just outside the window", backupStaleAfter + time.Minute, true}, + {"long past the window", 30 * 24 * time.Hour, true}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + got := assessBackupFreshness(empty, noEvidence(now, c.watched), now) + if got.missed() != c.wantMissed { + if c.wantMissed { + t.Fatalf("CONTRACT VIOLATED: absence beyond the window MUST alarm (watched=%s, limit=%s); got verdict=%d reason=%q", + c.watched, backupStaleAfter, got.verdict, got.reason) + } + t.Fatalf("CONTRACT VIOLATED: no evidence + no elapsed window must NOT alarm (watched=%s, limit=%s); got reason=%q", + c.watched, backupStaleAfter, got.reason) + } + if !c.wantMissed && got.verdict != verdictUnknown { + t.Fatalf("deferred absence must be UNKNOWN (not OK) so it stays visible; got verdict=%d", got.verdict) + } + }) + } +} + +// ── Scenario D — the three existing behaviours are untouched ───────────────────────────── + +// Pins the pre-R-81 outcomes byte-for-byte, INCLUDING the reason strings, under the new +// signature. Hub-history evidence is present and fresh in each case so it cannot be the +// thing producing the result — these must hold on the latest report's own merits. +func TestBackupFreshness_ExistingBehavioursUnchanged(t *testing.T) { + now := time.Date(2026, 7, 26, 3, 0, 0, 0, time.UTC) + at := func(d time.Duration) string { return now.Add(d).Format(time.RFC3339) } + anchor := 30 * 24 * time.Hour + + t.Run("1 fresh verified PBS stays silent", func(t *testing.T) { + r := `{"pbs_snapshots":[{"backup_time":"` + at(-3*time.Hour) + `","verify_state":"ok"}]}` + got := assessBackupFreshness(r, evidenceAt(now, 3*time.Hour, anchor), now) + if got.missed() || got.verdict != verdictOK { + t.Fatalf("want silent OK; got verdict=%d reason=%q", got.verdict, got.reason) + } + }) + + t.Run("2 newest backup >26h still alarms with the same string", func(t *testing.T) { + r := `{"pbs_snapshots":[{"backup_time":"` + at(-30*time.Hour) + `","verify_state":"ok"}]}` + // Window evidence is ALSO 30h old — nothing fresher exists anywhere. + got := assessBackupFreshness(r, evidenceAt(now, 30*time.Hour, anchor), now) + if !got.missed() { + t.Fatalf("stale backup must still alarm; got verdict=%d reason=%q", got.verdict, got.reason) + } + if got.reason != "newest backup is 30h0m0s old (limit 26h0m0s)" { + t.Fatalf("stale reason string changed: %q", got.reason) + } + }) + + t.Run("3 failed verify still alarms with the same string", func(t *testing.T) { + r := `{"pbs_snapshots":[{"backup_time":"` + at(-2*time.Hour) + `","verify_state":"failed"}]}` + got := assessBackupFreshness(r, evidenceAt(now, 2*time.Hour, anchor), now) + if !got.missed() { + t.Fatalf("failed verify must still alarm; got verdict=%d reason=%q", got.verdict, got.reason) + } + if got.reason != "newest PBS snapshot failed verification" { + t.Fatalf("verify-failed reason string changed: %q", got.reason) + } + }) + + t.Run("unparseable report still alarms", func(t *testing.T) { + got := assessBackupFreshness(`not json`, evidenceAt(now, time.Hour, anchor), now) + if !got.missed() || got.reason != "latest host-report could not be parsed" { + t.Fatalf("unparseable behaviour changed: verdict=%d reason=%q", got.verdict, got.reason) + } + }) +} + +// Fresh hub-history evidence must NOT rescue a failed verify — behaviour 3 is about the +// snapshot's integrity, not its age, so the anchor has no business suppressing it. +func TestBackupFreshness_WindowEvidenceDoesNotRescueFailedVerify(t *testing.T) { + now := time.Date(2026, 7, 26, 3, 0, 0, 0, time.UTC) + r := `{"pbs_snapshots":[{"backup_time":"` + now.Add(-2*time.Hour).Format(time.RFC3339) + `","verify_state":"failed"}]}` + got := assessBackupFreshness(r, evidenceAt(now, time.Minute, 30*24*time.Hour), now) + if !got.missed() { + t.Fatalf("fresh window evidence must not suppress a failed verify; got verdict=%d reason=%q", got.verdict, got.reason) + } +} + +// ── Scenario E — reason strings stay diagnostic ────────────────────────────────────────── + +// Every failure mode must produce a DISTINCT message. The 2026-07-26 diagnosis turned +// entirely on reading the exact string; collapsing them would have made it impossible. +func TestBackupFreshness_ReasonStringsAreDistinct(t *testing.T) { + now := time.Date(2026, 7, 26, 3, 0, 0, 0, time.UTC) + at := func(d time.Duration) string { return now.Add(d).Format(time.RFC3339) } + + reasons := map[string]string{ + "unparseable": assessBackupFreshness(`nope`, backupEvidence{}, now).reason, + "stale": assessBackupFreshness(`{"pbs_snapshots":[{"backup_time":"`+at(-40*time.Hour)+`","verify_state":"ok"}]}`, backupEvidence{}, now).reason, + "verify failed": assessBackupFreshness(`{"pbs_snapshots":[{"backup_time":"`+at(-2*time.Hour)+`","verify_state":"failed"}]}`, evidenceAt(now, 2*time.Hour, 30*24*time.Hour), now).reason, + "absence timed": assessBackupFreshness(`{"backups":[]}`, noEvidence(now, 240*time.Hour), now).reason, + "absence unanch": assessBackupFreshness(`{"backups":[]}`, backupEvidence{}, now).reason, + "absence grace": assessBackupFreshness(`{"backups":[]}`, noEvidence(now, time.Hour), now).reason, + } + seen := map[string]string{} + for name, r := range reasons { + if r == "" { + t.Fatalf("%s produced an empty reason", name) + } + if prev, dup := seen[r]; dup { + t.Fatalf("reason strings collapsed: %q and %q both produce %q", prev, name, r) + } + seen[r] = name + } +} + +// ── newestBackupEvidence — the window scan ─────────────────────────────────────────────── + +func reportRow(t *testing.T, receivedAt time.Time, pbs []string, vzdump []struct { + at string + ok bool +}) store.HostReportRow { + t.Helper() + type snap struct { + BackupTime string `json:"backup_time"` + VerifyState string `json:"verify_state"` + } + type bk struct { + StartedAt string `json:"started_at"` + Success bool `json:"success"` + } + p := struct { + PBSSnapshots []snap `json:"pbs_snapshots"` + Backups []bk `json:"backups"` + }{} + for _, s := range pbs { + p.PBSSnapshots = append(p.PBSSnapshots, snap{BackupTime: s, VerifyState: "ok"}) + } + for _, b := range vzdump { + p.Backups = append(p.Backups, bk{StartedAt: b.at, Success: b.ok}) + } + out, err := json.Marshal(p) + if err != nil { + t.Fatal(err) + } + return store.HostReportRow{ReceivedAt: receivedAt, ReportJSON: string(out)} +} + +// The scan must reach PAST the empty reports the restart produced and find the vzdump that +// an older report still carries. This is the mechanism behind Scenario A. +func TestNewestBackupEvidence_ReachesPastEmptyReports(t *testing.T) { + now := time.Date(2026, 7, 26, 3, 0, 0, 0, time.UTC) + vz := func(at string, ok bool) []struct { + at string + ok bool + } { + return []struct { + at string + ok bool + }{{at, ok}} + } + rows := []store.HostReportRow{ + // Newest first, as the store returns them. Post-restart reports carry nothing. + reportRow(t, now.Add(-1*time.Minute), nil, nil), + reportRow(t, now.Add(-1*time.Hour), nil, nil), + reportRow(t, now.Add(-14*time.Hour), nil, nil), + // The last pre-restart report still holds the 20.5h-old vzdump. + reportRow(t, now.Add(-15*time.Hour), nil, vz(now.Add(-20*time.Hour-30*time.Minute).Format(time.RFC3339), true)), + reportRow(t, now.Add(-40*time.Hour), nil, vz(now.Add(-44*time.Hour).Format(time.RFC3339), true)), + } + got, ok := newestBackupEvidence(rows, now) + if !ok { + t.Fatal("must find the vzdump carried by the pre-restart report") + } + if want := now.Add(-20*time.Hour - 30*time.Minute); !got.Equal(want) { + t.Fatalf("newest evidence = %s, want %s", got, want) + } +} + +// A FAILED vzdump is not evidence of a backup. +func TestNewestBackupEvidence_IgnoresFailedVzdump(t *testing.T) { + now := time.Date(2026, 7, 26, 3, 0, 0, 0, time.UTC) + rows := []store.HostReportRow{ + reportRow(t, now.Add(-time.Hour), nil, []struct { + at string + ok bool + }{{now.Add(-2 * time.Hour).Format(time.RFC3339), false}}), + } + if _, ok := newestBackupEvidence(rows, now); ok { + t.Fatal("a failed vzdump must not count as evidence") + } +} + +// One malformed retained report must not blind the scan to the good ones behind it. +func TestNewestBackupEvidence_SkipsMalformedRows(t *testing.T) { + now := time.Date(2026, 7, 26, 3, 0, 0, 0, time.UTC) + rows := []store.HostReportRow{ + {ReceivedAt: now.Add(-time.Minute), ReportJSON: `{{{not json`}, + reportRow(t, now.Add(-2*time.Hour), []string{now.Add(-3 * time.Hour).Format(time.RFC3339)}, nil), + } + got, ok := newestBackupEvidence(rows, now) + if !ok { + t.Fatal("a malformed row must not blind the scan") + } + if want := now.Add(-3 * time.Hour); !got.Equal(want) { + t.Fatalf("newest evidence = %s, want %s", got, want) + } +} + +// No rows at all → no evidence, and emphatically not a zero timestamp treated as evidence. +func TestNewestBackupEvidence_EmptyWindow(t *testing.T) { + now := time.Date(2026, 7, 26, 3, 0, 0, 0, time.UTC) + if _, ok := newestBackupEvidence(nil, now); ok { + t.Fatal("an empty window must report no evidence") + } +} + +// ── End-to-end through CheckBackupDeadlines ────────────────────────────────────────────── + +// The full 2026-07-26 replay against the real store: an agent restart empties the array, the +// hub's retained window still holds yesterday's vzdump → NO event. +func TestCheckBackupDeadlines_RestartBlindWindow_NoEvent(t *testing.T) { + st := newDeadlineStore(t) + if _, err := st.SaveEvent("c1", "db_dump_completed", "info", "", "{}", "controller"); err != nil { + t.Fatal(err) + } + okVz := []struct { + at string + ok bool + }{{rfc(-20 * time.Hour), true}} + + // Pre-restart report: carries the vzdump. Back-dated so it is not the latest. + pre := hostReportJSON(t, [][2]string{{"2026-07-18T18:31:06Z", "ok"}}, okVz) + if err := st.SaveHostReport("h1", "c1", []byte(pre), store.HostReportDenorm{}); err != nil { + t.Fatal(err) + } + if err := st.SetHostReportsReceivedAtForTest("c1", sqliteAgo(15*time.Hour)); err != nil { + t.Fatal(err) + } + // Post-restart report: PBS only, 176h stale — exactly demo-felhom's 07-26 shape. + post := hostReportJSON(t, [][2]string{{"2026-07-18T18:31:06Z", "ok"}}, nil) + if err := st.SaveHostReport("h1", "c1", []byte(post), store.HostReportDenorm{}); err != nil { + t.Fatal(err) + } + + got := runDeadline(t, st) + if has(got, "expected_backup_missed") { + t.Fatalf("the 07-26 restart shape must NOT raise expected_backup_missed; got %v", got) + } +} + +// The counterpart end-to-end: a host that has been reporting for days and has NEVER produced +// a backup must still alarm through the real store path. +func TestCheckBackupDeadlines_NeverBackedUpBeyondAnchor_Alarms(t *testing.T) { + st := newDeadlineStore(t) + st.SaveEvent("c1", "db_dump_completed", "info", "", "{}", "controller") + empty := hostReportJSON(t, nil, nil) + if err := st.SaveHostReport("h1", "c1", []byte(empty), store.HostReportDenorm{}); err != nil { + t.Fatal(err) + } + // Back-date first contact well beyond the 26h anchor. + if err := st.SetHostReportsReceivedAtForTest("c1", sqliteAgo(120*time.Hour)); err != nil { + t.Fatal(err) + } + if err := st.SaveHostReport("h1", "c1", []byte(empty), store.HostReportDenorm{}); err != nil { + t.Fatal(err) + } + + got := runDeadline(t, st) + if !has(got, "expected_backup_missed") { + t.Fatalf("a host with no backup for 120h MUST alarm; got %v", got) + } +} + +// A newborn host must not alarm on its first morning — end-to-end. +func TestCheckBackupDeadlines_NewbornHost_NoEvent(t *testing.T) { + st := newDeadlineStore(t) + st.SaveEvent("c1", "db_dump_completed", "info", "", "{}", "controller") + empty := hostReportJSON(t, nil, nil) + if err := st.SaveHostReport("h1", "c1", []byte(empty), store.HostReportDenorm{}); err != nil { + t.Fatal(err) + } + var logged strings.Builder + onEvent := func(customerID, eventType, severity, message, detailsJSON, source string) { + if customerID == "c1" && eventType == "expected_backup_missed" { + t.Fatalf("a newborn host must not alarm; got %q", message) + } + } + CheckBackupDeadlines(st, nil, onEvent, log.New(&logged, "", 0)) + + // The deferral must be VISIBLE — quiet must never look like "did not run". + if !strings.Contains(logged.String(), "verdict UNKNOWN") { + t.Fatalf("the deferred verdict must be logged; log was:\n%s", logged.String()) + } +} diff --git a/hub/internal/monitor/deadline_test.go b/hub/internal/monitor/deadline_test.go index d27847a..6bef799 100644 --- a/hub/internal/monitor/deadline_test.go +++ b/hub/internal/monitor/deadline_test.go @@ -171,6 +171,12 @@ func TestCheckBackupDeadlines_DbDumpHalfPreserved(t *testing.T) { } // TestAssessBackupFreshness exercises the pure freshness policy directly. +// +// R-81: every case here passes a ZERO backupEvidence — no hub-history evidence and no +// first-contact anchor. That is deliberate: it pins the latest-report-only behaviour +// unchanged, and the "no snapshots and no backups" row exercises the UNANCHORED absence +// branch (zero anchor → fail toward visibility, the v0.73.0 legacy-shape precedent). +// The anchored branches have their own named tests below. func TestAssessBackupFreshness(t *testing.T) { now := time.Date(2026, 6, 16, 3, 0, 0, 0, time.UTC) at := func(d time.Duration) string { return now.Add(d).Format(time.RFC3339) } @@ -192,9 +198,9 @@ func TestAssessBackupFreshness(t *testing.T) { } for _, c := range cases { t.Run(c.name, func(t *testing.T) { - got := assessBackupFreshness(c.report, now) - if got.missed != c.wantMissed { - t.Fatalf("missed=%v want=%v (reason=%q)", got.missed, c.wantMissed, got.reason) + got := assessBackupFreshness(c.report, backupEvidence{}, now) + if got.missed() != c.wantMissed { + t.Fatalf("missed=%v want=%v (reason=%q)", got.missed(), c.wantMissed, got.reason) } }) } diff --git a/hub/internal/store/store.go b/hub/internal/store/store.go index 94acdb9..86830e8 100644 --- a/hub/internal/store/store.go +++ b/hub/internal/store/store.go @@ -2825,6 +2825,83 @@ func (s *Store) GetLatestHostReportJSON(customerID string) (string, error) { return j, nil } +// HostReportRow is one retained host-report: when the hub received it, and its payload. +type HostReportRow struct { + ReceivedAt time.Time + ReportJSON string +} + +// GetHostReportsSince returns the customer's retained host-reports received at or after +// `since`, NEWEST FIRST. The hub keeps ~retention.max_days of history (90 by default), which +// is what makes it possible to ask "when did I last SEE evidence of a backup?" rather than +// only "what does the latest report say?". +// +// R-81: this is the anchor source for the backup-deadline check. The agent's backup record +// store is IN-MEMORY (felhom-agent/internal/backup/store.go — "lost on restart; the cadence +// re-populates"), so a restart empties `backups` in every report until the next backup runs. +// The hub has memory the agent does not; reading across the window is what turns that blind +// window from a false alarm into a correctly-silent verdict. Newest-first so the caller can +// stop as soon as it has seen enough (see monitor.newestBackupEvidence). +func (s *Store) GetHostReportsSince(customerID string, since time.Time) ([]HostReportRow, error) { + rows, err := s.db.Query( + `SELECT received_at, report_json FROM host_reports + WHERE customer_id = ? AND received_at >= ? + ORDER BY received_at DESC`, + customerID, since.UTC().Format("2006-01-02 15:04:05"), + ) + if err != nil { + return nil, err + } + defer rows.Close() + + var out []HostReportRow + for rows.Next() { + var at, j string + if err := rows.Scan(&at, &j); err != nil { + return nil, err + } + out = append(out, HostReportRow{ReceivedAt: parseSQLiteTime(at).UTC(), ReportJSON: j}) + } + return out, rows.Err() +} + +// GetFirstHostReportAt returns when the hub received its FIRST retained host-report for the +// customer, or (zero, nil) when it holds none. +// +// R-81: this is the observation anchor. "No backup evidence anywhere" is only meaningful +// relative to how long the hub has been watching — a box registered an hour ago has no +// evidence yet and is NOT failing. Same shape as the v0.73.0 offsite never-ran anchor: +// absence becomes a fault only once it has outlived the existing threshold, measured from a +// point where evidence first became POSSIBLE. +// +// Caveat, deliberately accepted: retention prunes at max_days, so for a host older than the +// window this returns the prune horizon rather than true first-contact. That only makes the +// anchor MORE conservative for long-lived hosts (the window has long since elapsed either +// way), and it never shortens a newborn's grace. +func (s *Store) GetFirstHostReportAt(customerID string) (time.Time, error) { + var at string + err := s.db.QueryRow( + `SELECT received_at FROM host_reports WHERE customer_id = ? ORDER BY received_at ASC LIMIT 1`, + customerID, + ).Scan(&at) + if err == sql.ErrNoRows { + return time.Time{}, nil + } + if err != nil { + return time.Time{}, err + } + return parseSQLiteTime(at).UTC(), nil +} + +// SetHostReportsReceivedAtForTest back-dates ALL of a customer's existing host_reports rows +// to the given SQLite datetime string, so anchor/window behaviour is testable without +// sleeping. TEST-ONLY — mirrors SetOneTimeSecretTimesForTest. Rows saved AFTER the call keep +// datetime('now'), which is how a test stages "old reports, then a fresh one". +func (s *Store) SetHostReportsReceivedAtForTest(customerID, sqliteTime string) error { + _, err := s.db.Exec(`UPDATE host_reports SET received_at = ? WHERE customer_id = ?`, sqliteTime, customerID) + return err +} + // UpsertGuestFromReport upserts the REALITY columns of a guest. On conflict it // must NOT clobber the inert columns (api_key / desired_spec_json). func (s *Store) UpsertGuestFromReport(g *Guest) error {