hub v0.77.0 — R-85 Part 2: a restore-test result becomes a SIGNAL

A failed restore-test was a [WARN] line in the ingest handler and nothing else —
no event, no notification, no gauge. True for the LOCAL tier that was already
being tested, so the loudest DR signal this system produces was inaudible.
Rotating tiers without this would only mean two tiers can fail silently
instead of one.

Two signals, deliberately NOT merged:
  restore_test_failed (error)   — a run completed and did NOT pass
  restore_test_stale (warning)  — a tier not PROVEN within its interval
Merging them collapses 'your DR is broken' into 'your DR is unverified', and
the second is the one that quietly becomes the first. The staleness wording
says 'unverified, not known-broken' and a test asserts that phrasing.

Anchored per R-81, not re-derived: a never-proven tier on a newborn box is
UNKNOWN, not FAILED, until the window elapses. This family has made the
opposite mistake three times; this monitor was written straight after the third,
so it copies R-81's structure rather than inventing a fourth shape.

restoreProvenStaleAfter = 7d is derived: oldest-first over two tiers at a 24h
cadence proves each ~every 2 days, so 7d tolerates ~3 missed opportunities and
sits inside the 2-week offsite retention.

Per-tier proof comes from the hub's retained host-report window — the agent
reports only its latest run, so the latest report alone cannot answer 'when was
the OTHER tier last proven?'. Reused R-81's mechanism instead of a wire change.

Both types registered in allowedEventTypes (R-77's inert-seam lesson) and
operator-tier only — no customerMessages entry.

FIXED a time bomb I introduced in Slice C: the restart-blind-window test
hard-coded 2026-07-18T18:31:06Z while comparing against the real clock. Harmless
under one 26h threshold; once the offsite tier got an 8-day limit it passed all
day and began failing at 18:31 UTC, exactly 8 days later. Now relative.

Red-proofs B and D observed. Full suite green (17 packages, rc=0).
This commit is contained in:
Claude Code
2026-07-26 21:08:02 +02:00
parent b6d93e3fcb
commit ce4e03dcd8
6 changed files with 671 additions and 3 deletions
+58
View File
@@ -1,5 +1,63 @@
# Felhom Hub — Changelog
## v0.77.0 — R-85 Part 2: a restore-test result becomes a SIGNAL (2026-07-26)
Until now a failed restore-test was a `[WARN]` line in the ingest handler and nothing else — no
event, no notification, no gauge. That was true for the **local tier that was already being tested**,
so the loudest DR signal this system produces was, in practice, inaudible. Rotating tiers (agent
v0.104.0) without this would only mean two tiers can fail silently instead of one.
### Two signals, deliberately NOT merged
| event | meaning | severity |
|---|---|---|
| `restore_test_failed` | a run completed and did **not** pass — something is broken NOW | error |
| `restore_test_stale` | a tier has not been **proven** within its interval — nothing has necessarily broken; we no longer know | warning |
Merging them would collapse *"your DR is broken"* into *"your DR is unverified"*, and the second is
the one that quietly becomes the first. The staleness wording says **"unverified, not known-broken"**
and a test asserts that phrasing.
### Anchored, per R-81 — not re-derived
A tier never proven on a newborn box is **UNKNOWN**, not FAILED, until an anchored window elapses.
This monitor family has made the opposite mistake three times (hub v0.12.0, v0.73.0, R-81); this is a
NEW monitor written straight after the third, so it copies R-81's verdict structure and boundary-test
discipline rather than inventing a fourth shape. The deferral is logged once — a quiet check must
never be indistinguishable from one that did not run.
`restoreProvenStaleAfter = 7d` is derived, not guessed: a 24 h cadence rotating oldest-first across
two tiers proves each about every 2 days, so 7 days tolerates ~3 consecutive missed opportunities
before alarming — and sits comfortably inside the 2-week offsite retention, so a tier is never called
stale against an archive that is about to be pruned.
### How per-tier proof is recovered
The agent reports only its **latest** restore-test and its store is in-memory, so the latest report
alone cannot answer *"when was the OTHER tier last proven?"*. The hub's **retained host-report
window** can — the same R-81 mechanism, reused rather than re-solved with a wire change.
### Also
- Both types registered in `allowedEventTypes`. They are hub-generated, but that map is the
project's single register of legitimate event types, and R-77's lesson was that a type missing
from it ships as an inert seam.
- **Operator-tier only** — neither has a `customerMessages` entry, so the dispatcher cannot route it
to a customer. A customer can take no action on a failed restore-test, and *"a visszaállítási teszt
nem sikerült"* would frighten without informing. A persistently unproven DR tier may eventually
warrant a customer-visible statement; that needs copy review, not a side effect of this task.
- A tier the box does not HAVE is never reported stale (the Slice-C gate, same reasoning).
### Fixed — a time bomb introduced in Slice C
`TestCheckBackupDeadlines_RestartBlindWindow_NoEvent` hard-coded the literal incident timestamp
`2026-07-18T18:31:06Z` while comparing against the REAL clock. Harmless while one 26 h threshold
covered every tier; once Slice C gave the offsite tier an 8-day limit it became a bomb — the test
passed all day on 2026-07-26 and began failing at **18:31 UTC**, exactly 8 days after that instant.
Now relative. **A test that passes at commit time and fails hours later is worse than one that fails
immediately**, because it lands on whoever is next in the file.
### Tests
+10, full suite green (17 packages, `rc=0`, vet unpiped). Red-proofs observed:
- **B** — log-and-stop (the pre-R-85 shape) yields `a FAILED restore-test must EMIT an operator event; got 0 event(s)`. The assertion is that a NOTIFICATION IS EMITTED; the hollow version checks for a log line, which passes against exactly the code this replaces.
- **D** — removing the anchor yields `a newborn box must NOT alarm; got restore_test_stale: local tier: NEVER successfully restore-proven in 0s of watching`.
## v0.76.0 — R-82 Slice C: tier-aware backup thresholds (2026-07-26)
R-81 merged every backup signal into one "newest" and judged it against a single 26 h limit. That