hub v0.75.0: R-81 — "no signal" is not "bad signal" (anchor the backup deadline check)

Third instance of one class (hub v0.12.0, v0.73.0, this), fixed as a class.
On 2026-07-26 03:00 UTC expected_backup_missed fired on demo-felhom, demo-hp
and drill-r50 at once; the demo-felhom one reached the CUSTOMER channel
claiming "newest backup is 176h0m0s old". Nothing was wrong — three vzdump
archives were on disk. Cause: the agent backup store is in-memory, so the
R-50 fleet restart emptied `backups` until the next run, and the hub read
empty as "no backup exists".

- assessBackupFreshness returns OK/UNKNOWN/MISSED instead of `missed bool`;
  absence is UNKNOWN until it outlives an anchored window. Still pure.
- store.GetHostReportsSince + monitor.newestBackupEvidence read the hubs own
  retained history (bounded 7-day lookback, early-exit on fresh evidence) —
  "when did I last SEE evidence of a backup?" The anchor was free: the hub
  already retains 90 days. No agent change, no new persisted state.
- store.GetFirstHostReportAt anchors absence at first contact, reusing the
  existing 26h threshold as the grace (no new knob, the v0.73.0 shape).
- Deferrals logged + counted; reason strings kept distinct.
- backupStaleAfter untouched; landmine recorded (a weekly PBS snapshot would
  alarm six days in seven) and owned by R-82.

Tests 493->508. Red-proofs A/B/C observed and restored; A reproduces the live
message verbatim. Replayed the real 03:00 reports (600/417/77 rows): all
three now silent.

Source: documentation/audits/DIAG-backup-missed-2026-07-26.md
This commit is contained in:
Claude Code
2026-07-26 11:44:15 +02:00
parent add5b9bbbb
commit f5a5e2b911
9 changed files with 843 additions and 28 deletions
@@ -111,7 +111,8 @@
| Paired recovery notifications + prefs seeding at claim + priority headers (power-outage audit F11/F12/F14-light) | hub v0.71.0 | **IMPLEMENTED** (recovery leg **PARTIAL** until a live staleness cycle fires it) | `hub/CHANGELOG.md` v0.71.0; 17 tests + 4 red-proofs (`REPORT.md` 2026-07-22); Resend `headers` mechanism probed live (HTTP 200) pre-implementation; operator+customer `test` rows live-fired via the controller's own test endpoint | Recovery = explicit eventType branch, severity semantics frozen; customer gate = PAIRING (`notification_log` evidence), not `enabled_events`. Live legs pending: a natural `*_recovered` mail (next real staleness cycle or the reboot-drill arc — never fabricated by blocking reports) and seed-at-claim on a real claim (Peti Friday reinstall). F14-full (operator push channel, ntfy/Telegram) stays open → R-69 |
| System + container metrics (SQLite, Chart.js, 30-day downsampling) | controller | **IMPLEMENTED** | metrics collection + `/monitoring` render present (page 200) | The cited `CAMPAIGN-2` T-RES-CPU/T-SOAK-LOOP are H1/H2 harness artifacts (auth-302), not metrics tests; SQLite/Chart.js/30-day downsampling validated in no campaign. Demoted |
| Always-on debug rings + on-demand log-bundle pulls with TTL/custody | controller v0.116, agent v0.83, hub v0.46 | **PROVEN-LIVE** | debug rings live-exercised `CAMPAIGN-3` fix-6 (1000-cap ring, ~55min horizon under load) | The **log-bundle-pull TTL/custody** half is changelog-only (no dedicated observability audit doc); ring persistence across restart is a known gap |
| Operator alerting (Healthchecks → monitoring@felhom.eu) | k3s, Resend | **IMPLEMENTED** | operator infra, stated in production since 02-04; no corpus validation doc | Per the status enum, no citation → not PROVEN-LIVE. Demoted pending an operator-cited live alert (candidate re-upgrade — see REPORT) |
| Operator alerting (Healthchecks → monitoring@felhom.eu) | k3s, Resend | **IMPLEMENTED** | operator infra, stated in production since 02-04; no corpus validation doc |
| Backup-deadline alerting (`expected_backup_missed`) is ANCHORED — absence of signal is UNKNOWN, not failure | hub v0.75.0 | **IMPLEMENTED** | `audits/DIAG-backup-missed-2026-07-26.md` + red-proofs A/B/C + replay of the real 2026-07-26 03:00 reports (all three silent) | **No row status flips** — this signal had a FALSE-POSITIVE class (three instances: hub v0.12.0, v0.73.0, R-81), now anchored at first contact and read across retained host-report history. Still unit-proven only, not live-fired at a real deadline. The *underlying* PBS/offsite-DR tier gap it exposed is → R-82. | Per the status enum, no citation → not PROVEN-LIVE. Demoted pending an operator-cited live alert (candidate re-upgrade — see REPORT) |
## G. Fleet & operator (hub)
File diff suppressed because one or more lines are too long