hub v0.91.0 — the staleness window learns each tier's own rhythm (R-86 Part 2)
gates / gates (push) Successful in 7s

Ships WITH agent v0.121.0, not after it. The agent now proves a tier once per
ARCHIVE GENERATION, so a weekly tier is proved weekly — in perfect health. The
flat 7-day restoreProvenStaleAfter derived its number from the 24h cadence R-86
removes, and a healthy weekly tier's proof age reaches EXACTLY 168h just before
its next proof: it sat ON the line, so any ordinary delay tipped it into a
nightly alarm about a working system.

restoreProvenWindow(tier, observed, ok):
- the tier's own archive interval, OBSERVED from reports the hub already holds
  (pbs_snapshots + successful backups attributed by TARGET TYPE, slice A.4)
- x4 generations = the same tolerance the flat constant expressed
- floored at 7d (never tighter than before), capped at 12d (strictly inside the
  2-week offsite retention)
- falls back to the DECLARED rhythm (26h host / 8d offsite — the thresholds the
  backup-freshness checker already uses) when history is too short to observe
  one; falling back to the FLOOR would recreate the false alarm on a fresh box

Kept: absence is UNKNOWN until the anchored window passes; the signal stays
edge-triggered; failed and stale remain distinct events. Every reason string now
states the window it was judged against (R-100's corollary).

Also backfills the missing v0.90.1 CHANGELOG entry (deployed since f21e7ca), and
records the operator's 2026-08-03 ruling that ep0 is Tier 2 / protected.
This commit is contained in:
2026-08-03 15:03:35 +02:00
parent e34b614e5b
commit 323f45a5ef
4 changed files with 478 additions and 41 deletions
+18 -12
View File
@@ -29,20 +29,23 @@ prohibition as covering the act it names and nothing more.
|---|---|---|
| **0 — disposable. Reach here first.** | Exists to be broken; reinstalling is a routine afternoon, not an incident. **A drill that needs a victim uses one of these.** | `demo-hp` (t740), `demo-felhom` (N100) |
| **1 — create and destroy freely** | Throwaway VMs, guests, scratch customers — **hosted on a Tier 0 machine** | drill VMs, scratch guests |
| **2 — protected. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster** — and, by D-d, **nothing else** |
| **2 — protected. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster**, **`ep0`** (operator ruling 2026-08-03) — and nothing else |
**DooPlex is Tier 2 because it *is* the recovery chain** — hub, Gitea, registry, PBS, k3s + Longhorn.
Everything else rebuilds from it; it rebuilds from nothing. A bad moment in a DR drill there costs the
thing under test, the source of truth for it, and the backups, at once.
**`ep0` + the Hetzner Storage Boxes were Tier 2 until 2026-08-02 and are no longer** — D-d's protected
list names two machines and ep0 is not one of them. **That does not make them scratch, and the
difference is an act, not a tier** (see the rule above the table): ep0 holds the **PBS-DR datastore and
the restic copy of a real customer's data**, which is the only off-premises copy that exists, so
*deleting datastores, prune jobs, tunnel config or nftables rules* remains forbidden by what it would
destroy rather than by what tier it sits in. Reads are fine; it is still never a drill target.
**Flagged for the operator: D-d did not name ep0 either way.** Confirm it explicitly — this page has
read it the narrow way (not protected, but not wipeable) rather than assume the broad one.
**`ep0` is Tier 2 — PROTECTED. Operator ruling, 2026-08-03.** D-d named two protected machines and did
not name ep0 either way, so this page carried the question in writing for two days and read it the
narrow way meanwhile (not protected, but not wipeable). The ruling settles it and **extends D-d's
protected list to three machines**: DooPlex, Peti's cluster, ep0.
The reason it was never really in doubt: ep0 holds the **PBS-DR datastore and the restic copy of a
real customer's data**, which is the only off-premises copy that exists. So *deleting datastores,
prune jobs, tunnel config or nftables rules* was already forbidden by what it would destroy; the
ruling makes the classification say so plainly instead of leaving each session to re-derive it.
**Reads are fine** — including the ordinary off-site read a restore-test performs (R-86) — and it is
never a drill target. The Hetzner Storage Boxes ride the same reasoning.
**Standing ruling, 2026-07-25 (`operations/nodes.md`):** drill and build VMs live on the **t740** — not
felhom-pve, and **moved off DooPlex**. This page exists because that ruling sat where no session reads.
@@ -95,12 +98,15 @@ still shares a device with its guest, so a drive failure is **offsite-only recov
migrated, parked until the tester reinstalls (`PETI` in `backlog/OPEN-ITEMS.md`). Currently DOWN, no
enrolled host. No access route from DooPlex, and nothing here needs one.
### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **not protected by D-d; not scratch either**
### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **Tier 2, PROTECTED** (operator ruling 2026-08-03)
Reads are fine. It is the **offsite of last resort** (PBS-DR datastore, WireGuard hub, operator OOB
path) and — until 2026-08-03 — RAM-constrained (3.8 GB, R-90); it is now a **CX33 with 8 GB RAM and a
4 GiB swapfile**, which is what closed R-90. A very large restore is still worth watching. Do not delete datastores, prune
jobs, tunnel config or nftables rules; never a drill target. The Storage Boxes hold the restic copy —
4 GiB swapfile**, which is what closed R-90. A very large restore is still worth watching — the 8 GB
is comfortable, not unbounded, and the OOM that started R-90 was a 14.46 GB restore read against
3.8 GB. Do not delete datastores, prune jobs, tunnel config or nftables rules; never a drill target.
**The ordinary off-site READ a restore-test performs is permitted and unchanged by the ruling**
(R-86): the classification forbids destruction, not use. The Storage Boxes hold the restic copy —
customer documents and photos, on a credential that can still delete (R-95).
**Access: `ssh root@167.233.158.164` from DooPlex***not* `felhom-pve → 10.77.0.1`, the route that
produced a false "unreachable" verdict (standing rule 2).
+68
View File
@@ -1,3 +1,71 @@
## v0.91.0 — a tier's staleness window learns the tier's own rhythm (2026-08-03, R-86 Part 2)
**This ships WITH the agent's v0.121.0, not after it.** The agent now proves a tier once per ARCHIVE
GENERATION rather than on a 24h timer, so a tier backed up weekly is proved weekly — correctly, and
in perfect health. `restoreProvenStaleAfter` was a flat 7 days, and its own comment derived that
number from the cadence R-86 removes:
> *"the restore-test cadence is 24h and rotation is oldest-first across two tiers, so each tier is
> proven roughly every 2 days. 7 days therefore tolerates ~3 consecutive missed opportunities."*
A weekly tier proved weekly reaches a proof age of **exactly one interval** just before its next
proof — 168h against a 168h window, false by a hair — so it did not merely sit near that line, it sat
ON it, and any ordinary delay (a late backup, one deferral behind a running backup) tipped it into a
nightly alarm about a system that is working. Shipping the agent alone would have converted the
improvement into a false alarm.
**The window is now per tier**, `restoreProvenWindow(tier, observed, ok)`:
- **the tier's own archive interval**, observed from the host-reports the hub already holds —
`pbs_snapshots[]` plus successful `backups[]` attributed by TARGET TYPE (the slice-A.4 rule: a
PBS-targeted vzdump appears in both arrays, and classifying by array membership would attribute an
offsite archive to the host tier);
- **× 4 generations** — the settle generation plus ~3 missed opportunities, deliberately the same
tolerance the flat constant expressed. The change is to the RHYTHM, not to the patience;
- **floored at 7 days** — the old constant kept as a floor, so no tier is judged more tightly than
before;
- **capped at 12 days** — strictly inside the 2-week offsite retention with two days to spare, so a
tier is never called stale against an archive PBS has already pruned;
- **falling back to the DECLARED rhythm** when a box's history is too short to observe one:
`backupStaleAfter` (26h) for the host tier and `offsiteBackupStaleAfter` (8d) for the offsite tier
— the very thresholds the backup-freshness checker already judges those tiers against. A fresh box
with one snapshot has no observable interval, and falling back to the FLOOR there would recreate
the false alarm on exactly the tier this task is about.
**Kept, because both were earned:** absence is UNKNOWN until an anchored window has passed (R-81's
structure, untouched), and the stale signal stays edge-triggered. `restore_test_failed` and
`restore_test_stale` remain DISTINCT — one says your recovery is broken, the other says it is
unverified, and the second is the one that quietly becomes the first.
**Every reason string now states the window it was judged against** — R-100's corollary: when a
verdict changes what it counts from, the alarm text has to change with it, or an operator reads
"limit 168h" under a tier actually judged at 288h and dismisses a true alarm.
**The window READ is unchanged in cost** (14 days), which is both enough to find proof inside the
widest window and enough to see two generations of a weekly tier.
## v0.90.1 — the digest's per-app lines stop repeating the filesystem figures (2026-08-03, R-182)
**Backfilled 2026-08-03 (R-86 session).** This version was built, deployed and recorded in `REPORT.md`
and the R-182 row, but never given a CHANGELOG entry — and `REPORT.md` is overwritten every session,
so the per-repo history under-reported what was actually running. The deployed image has been
`felhom-hub:0.90.1` since `f21e7ca`.
Found by reading the first REAL digest, not by design: every app row ended with the same usage clause
the mail already prints once on its own Filesystem line. On a two-app box that is untidy; down a list
of a dozen it is the same forty characters twelve times, pushing the part that DIFFERS off a phone
screen at 07:00 — the only moment that mail has to work.
The reserve's refusal message is authored for a single-app alert where naming the filesystem is
right, so the message is unchanged and the DIGEST trims the duplicate when rendering.
`trimRepeatedUsage` removes ONLY an exact "— <target path>:" suffix, so an unrelated reason is
untouched and a reason that is nothing but the usage clause is left alone rather than emptied.
Also inverted the operator half of `TestRecoveryUnitCaptureFailed_NeverReachesTheCustomer`: it
required the operator to be e-mailed a per-app capture failure, which was correct when that event was
the only signal and is wrong now that it is the RECORD and the digest is the notification. The
customer-safety claim is unchanged — R-158's guarantee MOVED, it did not weaken.
## v0.90.0 — a dropped notification leaves a trace, and the backup digest arrives (2026-08-03, R-182)
**The smallest change on the board with the largest effect on trust: `processOperator`'s cooldown no
+178 -20
View File
@@ -32,15 +32,87 @@ import (
// is a NEW monitor written straight after the third, so it copies R-81's verdict structure rather
// than re-deriving it. A tier never proven on a newborn box is UNKNOWN, never FAILED.
// restoreProvenStaleAfter is how long a tier may go unproven before it is called stale.
// ── HOW LONG MAY A TIER GO UNPROVEN? (R-86 Part 2) ───────────────────────────────────────────
//
// Derivation, not a guess: the restore-test cadence is 24h and rotation is oldest-first across two
// tiers, so each tier is proven roughly every 2 days. 7 days therefore tolerates ~3 consecutive
// missed opportunities before alarming — loud enough to matter, quiet enough not to fire on one
// skipped cycle (a deferral behind a long backup is normal, not a fault). It is also comfortably
// inside the 2-week offsite retention (operator ruling 2026-07-26), so a tier is never reported
// stale against an archive that is about to be pruned anyway.
const restoreProvenStaleAfter = 7 * 24 * time.Hour
// This was one flat constant, 7 days, and its comment derived that number like this:
//
// "the restore-test cadence is 24h and rotation is oldest-first across two tiers, so each tier is
// proven roughly every 2 days. 7 days therefore tolerates ~3 consecutive missed opportunities."
//
// **That premise is exactly what R-86 removed.** The agent no longer tests on an interval at all: a
// tier is tested once per ARCHIVE GENERATION — when it holds a settled archive that has not been
// proven. A tier backed up weekly is therefore proved weekly, by design and in perfect health, and
// against a flat 7-day window it would sit on the line and alarm every night about a system that is
// working. Shipping the agent's half alone would have converted the improvement into a false alarm,
// which is why the two ship together.
//
// The window is now derived from **the tier's own backup rhythm**, which the hub can observe from
// the reports it already receives, and it keeps everything the constant had earned:
//
// - absence is UNKNOWN until an anchored window has passed (R-81's structure, untouched);
// - the signal stays edge-triggered;
// - it never exceeds the offsite retention, so a tier is never called stale against an archive
// that is about to be pruned;
// - and it is never TIGHTER than the 7 days that were already tolerated.
const (
// restoreProvenGenerations is how many archive generations may pass unproven before alarming.
// 4 = the settle lag's own generation plus ~3 missed opportunities — deliberately the same
// tolerance the flat constant expressed, so the change is to the RHYTHM, not to the patience.
restoreProvenGenerations = 4
// restoreProvenWindowFloor is the shortest window that may be applied to any tier. It is the
// old constant, kept as a FLOOR rather than deleted: a daily tier computes 4 days from its own
// rhythm, and tightening a live threshold is not what this task is for. A deferral behind a
// long backup is normal, not a fault.
restoreProvenWindowFloor = 7 * 24 * time.Hour
// restoreProvenWindowCap keeps the window strictly inside the 2-week offsite retention
// (operator ruling 2026-07-26) with two days to spare. Beyond it the hub would be judging a
// tier against an archive PBS has already pruned — an alarm nobody can act on, and the bound
// the old constant respected in its own way.
restoreProvenWindowCap = 12 * 24 * time.Hour
// restoreWindowRead is how far back the hub reads host-reports for this check: far enough to
// find proof anywhere inside the widest window, and to see at least two archive generations of
// a WEEKLY tier so its rhythm is observable at all.
restoreWindowRead = 2 * restoreProvenWindowFloor
)
// declaredArchiveInterval is the rhythm the hub ALREADY attributes to a tier — the same thresholds
// the backup-freshness checker judges it against (deadline.go / deadline_tiers.go). It is the
// fallback when a box's history is too short to observe a rhythm, and it is the right fallback
// precisely because it is not a second opinion: if these two checkers disagreed about how often a
// tier is expected to receive an archive, one of them would be alarming on the other's model.
//
// It is stated per RESTORE tier name ("local"/"pbs" — what the agent reports as source_tier), which
// is the same split the backup tiers use under different names ("host"/"offsite").
func declaredArchiveInterval(tier string) time.Duration {
if tier == "pbs" {
return offsiteBackupStaleAfter // 8 days: the weekly cadence plus a day of headroom
}
return backupStaleAfter // 26 hours: the daily cadence plus headroom
}
// restoreProvenWindow is how long THIS tier may go unproven, given its observed archive interval.
//
// observedOK=false means the box's retained history did not contain two archive generations for
// this tier, so the declared rhythm is used. That fallback matters most for exactly the tier this
// task is about: a fresh box with a weekly offsite tier has one snapshot and no observable
// interval, and falling back to the FLOOR there would recreate the false alarm.
func restoreProvenWindow(tier string, observed time.Duration, observedOK bool) time.Duration {
interval := declaredArchiveInterval(tier)
if observedOK && observed > 0 {
interval = observed
}
w := time.Duration(restoreProvenGenerations) * interval
if w < restoreProvenWindowFloor {
w = restoreProvenWindowFloor
}
if w > restoreProvenWindowCap {
w = restoreProvenWindowCap
}
return w
}
// Event types. Operator-tier only — see the dispatcher note in RestoreTestChecker.
const (
@@ -156,12 +228,13 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
return
}
rows, err := c.store.GetHostReportsSince(customerID, now.Add(-2*restoreProvenStaleAfter))
rows, err := c.store.GetHostReportsSince(customerID, now.Add(-restoreWindowRead))
if err != nil {
c.logger.Printf("[WARN] restore-test check: window read failed for %s: %v", customerID, err)
return
}
proven := lastProvenPerTier(rows)
intervals := observedArchiveIntervals(rows)
first, ferr := c.store.GetFirstHostReportAt(customerID)
if ferr != nil {
@@ -170,7 +243,9 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
}
for _, tier := range tiers {
v := assessRestoreProven(tier, proven[tier], first, now)
observed, observedOK := intervals[tier]
window := restoreProvenWindow(tier, observed, observedOK)
v := assessRestoreProven(tier, proven[tier], first, now, window)
key := customerID + "|" + tier
c.mu.Lock()
prev := c.staleStates[key]
@@ -192,37 +267,120 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
}
}
// assessRestoreProven is the per-tier verdict. PURE (now injected) so the policy is unit-tested —
// the property that made R-81 provable, kept deliberately.
// assessRestoreProven is the per-tier verdict. PURE (now and the window injected) so the policy is
// unit-tested — the property that made R-81 provable, kept deliberately.
//
// no proof, anchor NOT elapsed → UNKNOWN (newborn box; never an alarm)
// no proof, anchor elapsed → MISSED
// proof older than the window → MISSED
// otherwise → OK
func assessRestoreProven(tier string, provenAt, firstReportAt, now time.Time) backupAssessment {
//
// `window` is now the TIER'S OWN (R-86 Part 2) rather than one constant for every tier, and every
// reason string states the window it was judged against. That is R-100's corollary applied here:
// when a verdict changes what it counts from, the alarm text has to change with it, or an operator
// reads "limit 168h" under a tier that was actually judged at 288h and dismisses a true alarm.
func assessRestoreProven(tier string, provenAt, firstReportAt, now time.Time, window time.Duration) backupAssessment {
if provenAt.IsZero() {
if firstReportAt.IsZero() {
return backupAssessment{verdict: verdictMissed,
reason: fmt.Sprintf("%s tier: never restore-proven, and no first-contact anchor to defer against", tier)}
}
watched := now.Sub(firstReportAt)
if watched <= restoreProvenStaleAfter {
if watched <= window {
return backupAssessment{verdict: verdictUnknown,
reason: fmt.Sprintf("%s tier: not restore-proven yet, but only watching for %s (grace %s since first contact %s) — newborn, not a fault",
tier, watched.Round(time.Hour), restoreProvenStaleAfter, firstReportAt.Format(time.RFC3339))}
tier, watched.Round(time.Hour), window, firstReportAt.Format(time.RFC3339))}
}
return backupAssessment{verdict: verdictMissed,
reason: fmt.Sprintf("%s tier: NEVER successfully restore-proven in %s of watching (limit %s) — the tier is unverified, not known-broken",
tier, watched.Round(time.Hour), restoreProvenStaleAfter)}
reason: fmt.Sprintf("%s tier: NEVER successfully restore-proven in %s of watching (limit %s, this tier's own backup rhythm) — the tier is unverified, not known-broken",
tier, watched.Round(time.Hour), window)}
}
if age := now.Sub(provenAt); age > restoreProvenStaleAfter {
if age := now.Sub(provenAt); age > window {
return backupAssessment{verdict: verdictMissed,
reason: fmt.Sprintf("%s tier: last successful restore-test was %s ago (limit %s) — the tier is unverified, not known-broken",
tier, age.Round(time.Hour), restoreProvenStaleAfter)}
reason: fmt.Sprintf("%s tier: last successful restore-test was %s ago (limit %s, this tier's own backup rhythm) — the tier is unverified, not known-broken",
tier, age.Round(time.Hour), window)}
}
return backupAssessment{verdict: verdictOK}
}
// observedArchiveIntervals estimates how often each RESTORE tier actually receives an archive, from
// the host-reports the hub already holds. Keyed by restore-tier name ("local"/"pbs").
//
// Evidence is every distinct archive timestamp in the window: `pbs_snapshots[]` for the offsite
// tier (PBS enumerates its whole retention in each report, so one report usually settles the
// question) and successful `backups[]` records attributed by TARGET TYPE for both tiers — the
// slice-A.4 rule, because a PBS-targeted vzdump appears in BOTH arrays and classifying by array
// membership would attribute an offsite archive to the host tier.
//
// The estimate is the MEAN gap: (newest oldest) / (generations 1). It needs two generations;
// with fewer, ok=false and the caller falls back to the declared rhythm. It is deliberately crude,
// and can afford to be: restoreProvenWindow clamps the result between a 7-day floor and a 12-day
// cap, so the only discrimination this has to get right is "roughly daily" versus "several days or
// slower" — which is exactly the distinction that turns a healthy weekly tier into a false alarm.
func observedArchiveIntervals(rows []store.HostReportRow) map[string]time.Duration {
seen := map[string]map[int64]struct{}{ // tier → set of archive unix times
"local": {},
"pbs": {},
}
add := func(tier string, t time.Time) {
if t.IsZero() {
return
}
seen[tier][t.UTC().Unix()] = struct{}{}
}
for _, r := range rows {
var hr hostReportBackups
if json.Unmarshal([]byte(r.ReportJSON), &hr) != nil {
continue // one malformed retained report must not blind the scan
}
pbs := pbsTargetSet(hr)
for _, ps := range hr.PBSSnapshots {
if t, ok := parseBackupTime(ps.BackupTime); ok {
add("pbs", t)
}
}
for _, b := range hr.Backups {
if !b.Success {
continue
}
t, ok := parseBackupTime(b.StartedAt)
if !ok {
continue
}
if pbs[b.TargetID] {
add("pbs", t)
} else {
add("local", t)
}
}
}
out := map[string]time.Duration{}
for tier, set := range seen {
if len(set) < 2 {
continue // not observable — the caller uses the declared rhythm
}
var oldest, newest int64
first := true
for ts := range set {
if first || ts < oldest {
oldest = ts
}
if first || ts > newest {
newest = ts
}
first = false
}
span := time.Duration(newest-oldest) * time.Second
if span <= 0 {
continue
}
out[tier] = span / time.Duration(len(set)-1)
}
return out
}
// expectedRestoreTiers names the tiers this box actually HAS, so a box without an offsite tier is
// never reported stale for one. Same gate as Slice C's `expected`, and for the same reason: without
// it every box lacking a tier would alarm once the anchor elapsed — absence-is-not-failure,
+214 -9
View File
@@ -182,26 +182,33 @@ func TestRestoreTest_NewbornDoesNotAlarm(t *testing.T) {
}
}
// The boundary, pinned by name so a refactor has to delete an obviously-named contract.
// The boundary, pinned by name so a refactor has to delete an obviously-named contract. R-86 made
// the limit per-tier, so the anchor is now measured against THE TIER'S OWN window — here the local
// tier's, which clamps to the 7-day floor and so keeps this contract numerically identical to the
// one the flat constant expressed.
func TestRestoreTest_Contract_UnprovenIsUnknownUntilTheAnchorElapses(t *testing.T) {
now := time.Date(2026, 7, 26, 12, 0, 0, 0, time.UTC)
window := restoreProvenWindow("local", 24*time.Hour, true)
if window != restoreProvenWindowFloor {
t.Fatalf("precondition: a daily local tier must clamp to the floor; got %s", window)
}
cases := []struct {
name string
watched time.Duration
wantMissed bool
}{
{"newborn, 1h", time.Hour, false},
{"just inside", restoreProvenStaleAfter - time.Minute, false},
{"exactly at the limit", restoreProvenStaleAfter, false},
{"just outside", restoreProvenStaleAfter + time.Minute, true},
{"just inside", window - time.Minute, false},
{"exactly at the limit", window, false},
{"just outside", window + time.Minute, true},
{"long past", 30 * 24 * time.Hour, true},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
got := assessRestoreProven("pbs", time.Time{}, now.Add(-c.watched), now)
got := assessRestoreProven("local", time.Time{}, now.Add(-c.watched), now, window)
if got.missed() != c.wantMissed {
t.Fatalf("CONTRACT VIOLATED: unproven for %s (limit %s) → missed=%v, want %v (reason %q)",
c.watched, restoreProvenStaleAfter, got.missed(), c.wantMissed, got.reason)
c.watched, window, got.missed(), c.wantMissed, got.reason)
}
if !c.wantMissed && got.verdict != verdictUnknown {
t.Fatalf("a deferred tier must be UNKNOWN (visible), not OK; got verdict=%d", got.verdict)
@@ -215,15 +222,17 @@ func TestRestoreTest_Contract_UnprovenIsUnknownUntilTheAnchorElapses(t *testing.
func TestRestoreTest_StaleIsSeparateFromFailure(t *testing.T) {
now := time.Date(2026, 7, 26, 12, 0, 0, 0, time.UTC)
stale := assessRestoreProven("pbs", now.Add(-9*24*time.Hour), now.Add(-60*24*time.Hour), now)
// A DAILY tier judged on its own rhythm: the window clamps to the 7-day floor.
daily := restoreProvenWindow("local", 24*time.Hour, true)
stale := assessRestoreProven("local", now.Add(-9*24*time.Hour), now.Add(-60*24*time.Hour), now, daily)
if !stale.missed() {
t.Fatalf("a tier last proven 9 days ago (limit %s) must be stale; got %q", restoreProvenStaleAfter, stale.reason)
t.Fatalf("a daily tier last proven 9 days ago (limit %s) must be stale; got %q", daily, stale.reason)
}
// The wording must not read as "broken" — that is the other signal.
if !strings.Contains(stale.reason, "unverified, not known-broken") {
t.Fatalf("staleness must say UNVERIFIED, not broken — merging the two is the thing this avoids; got %q", stale.reason)
}
fresh := assessRestoreProven("pbs", now.Add(-2*24*time.Hour), now.Add(-60*24*time.Hour), now)
fresh := assessRestoreProven("local", now.Add(-2*24*time.Hour), now.Add(-60*24*time.Hour), now, daily)
if fresh.verdict != verdictOK {
t.Fatalf("a tier proven 2 days ago is fine; got verdict=%d reason=%q", fresh.verdict, fresh.reason)
}
@@ -283,3 +292,199 @@ func boolStr(b bool) string {
}
return "false"
}
// ── SCENARIO G — a healthy WEEKLY tier is never reported stale (R-86 Part 2) ─────────────────
//
// This is the test that pins the false alarm this change would otherwise have CREATED. The agent
// now proves a tier once per archive generation, so a weekly offsite tier is proved weekly — in
// perfect health. Against the old flat 7-day window it would sit on the line and alarm every night.
//
// COMPANION RED-PROOF (observed 2026-08-03): pin the window flat, as it was —
//
// - window := restoreProvenWindow(tier, observed, observedOK)
// - window := restoreProvenWindowFloor // the pre-R-86 flat 7 days
//
// → --- FAIL: TestRestoreTest_HealthyWeeklyTierIsNeverStale
//
// week 0: a weekly tier proved on its own archive must never be stale (proof age 172h0m0s,
// window 168h0m0s); verdict=2 reason="pbs tier: last successful restore-test was 172h0m0s ago
// (limit 168h0m0s, this tier's own backup rhythm) — the tier is unverified, not known-broken"
//
// Restored. The mutation is one line because the whole of Part 2 is one decision: whose rhythm.
//
// NOTE, because it is the finding this test nearly hid: the FIRST version of this fixture had NO
// jitter, and it PASSED under the mutation. A perfectly regular weekly tier's proof age reaches
// EXACTLY 168h just before the next proof, and `age > window` is false by a hair — a hollow test
// that would have shipped Part 1 and its false alarm together. The jitter below is what makes this
// a test, and it is also the truth about the old constant: a healthy weekly tier did not merely sit
// near the line, it sat ON it, so any ordinary delay tipped it over.
func TestRestoreTest_HealthyWeeklyTierIsNeverStale(t *testing.T) {
start := time.Date(2026, 6, 1, 3, 0, 0, 0, time.UTC)
firstContact := start.Add(-24 * time.Hour)
// The observable rhythm of a weekly tier, as the hub would compute it from the reports. No
// assertion about the window ITSELF here on purpose: that is the mechanism, and it is pinned in
// TestRestoreProvenWindow_Contract. What this test asserts is the CONSEQUENCE — does the alarm
// fire? — because R-97b proved a mechanism and shipped a broken consequence anyway.
weekly := restoreProvenWindow("pbs", 7*24*time.Hour, true)
// Walk several weeks of a HEALTHY tier, with the jitter a real one has: the backup does not land
// to the second, and a restore-test can be deferred one evaluation behind a running backup.
//
// The jitter is the point. A perfectly regular weekly tier's proof reaches an age of EXACTLY one
// interval (168h) just before the next proof, and against a flat 168h window `age > window` is
// false by a hair — so a regular fixture would pass against the very constant this task must
// change, and prove nothing. That is the brief's "sits exactly on that line": every real-world
// delay pushes it over, and the alarm is about a system that is working.
settle, evalLatency := 24*time.Hour, 6*time.Hour
archiveLate := []time.Duration{0, 4 * time.Hour, 2 * time.Hour, 6 * time.Hour, 0, 3 * time.Hour}
deferred := []time.Duration{0, 0, 6 * time.Hour, 0, 0, 6 * time.Hour} // one evaluation behind a backup
archiveAt := func(week int) time.Time {
return start.AddDate(0, 0, 7*week).Add(archiveLate[week])
}
provenAt := func(week int) time.Time {
return archiveAt(week).Add(settle + evalLatency + deferred[week])
}
var worst time.Duration
for week := 0; week+1 < len(archiveLate); week++ {
// The widest the proof's age ever gets: the instant before the NEXT week's proof lands.
now := provenAt(week + 1).Add(-time.Second)
age := now.Sub(provenAt(week))
if age > worst {
worst = age
}
v := assessRestoreProven("pbs", provenAt(week), firstContact, now, weekly)
if v.verdict != verdictOK {
t.Fatalf("week %d: a weekly tier proved on its own archive must never be stale (proof age %s, window %s); verdict=%d reason=%q",
week, age.Round(time.Hour), weekly, v.verdict, v.reason)
}
}
// The fixture must actually EXERCISE the boundary — a jitter-free walk would sit at exactly one
// interval and pass against a flat 7-day window, which is the hollow version of this test.
if worst <= restoreProvenWindowFloor {
t.Fatalf("this fixture never exceeds the old flat window (worst proof age %s) — it cannot detect the defect it exists for", worst)
}
}
// ...and a weekly tier that genuinely STOPS being proved must still alarm. A window that never
// fires is not a fix, it is a deletion.
func TestRestoreTest_WeeklyTierThatStopsBeingProvedStillAlarms(t *testing.T) {
now := time.Date(2026, 7, 26, 12, 0, 0, 0, time.UTC)
weekly := restoreProvenWindow("pbs", 7*24*time.Hour, true)
v := assessRestoreProven("pbs", now.Add(-weekly-time.Hour), now.Add(-90*24*time.Hour), now, weekly)
if !v.missed() {
t.Fatalf("a weekly tier unproven for longer than its own window MUST alarm; got verdict=%d reason=%q", v.verdict, v.reason)
}
if !strings.Contains(v.reason, weekly.String()) {
t.Fatalf("the alarm must state the window it was judged against (R-100's corollary); got %q", v.reason)
}
}
// The window's own contract: derived from the tier's rhythm, floored, capped, and never dependent
// on an unobservable history for the tier that would suffer most from a wrong answer.
func TestRestoreProvenWindow_Contract(t *testing.T) {
cases := []struct {
name string
tier string
observed time.Duration
observedOK bool
want time.Duration
}{
{"daily local clamps to the floor", "local", 24 * time.Hour, true, restoreProvenWindowFloor},
{"weekly pbs widens", "pbs", 7 * 24 * time.Hour, true, restoreProvenWindowCap},
{"3-day tier sits between", "pbs", 72 * time.Hour, true, 12 * 24 * time.Hour},
{"unobservable local falls back to its declared rhythm", "local", 0, false, restoreProvenWindowFloor},
{"unobservable pbs falls back WIDE, not to the floor", "pbs", 0, false, restoreProvenWindowCap},
{"a nonsense zero interval is ignored", "pbs", 0, true, restoreProvenWindowCap},
}
// The relationship Part 1 depends on: a weekly tier's window must be WIDER than a daily tier's,
// or proving weekly (which is now correct behaviour) alarms on itself.
if restoreProvenWindow("pbs", 7*24*time.Hour, true) <= restoreProvenWindow("local", 24*time.Hour, true) {
t.Fatal("a weekly tier must earn a wider window than a daily one — otherwise R-86's agent half alarms about itself")
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
got := restoreProvenWindow(c.tier, c.observed, c.observedOK)
if got != c.want {
t.Fatalf("window(%s, observed=%s ok=%v) = %s, want %s", c.tier, c.observed, c.observedOK, got, c.want)
}
if got < restoreProvenWindowFloor || got > restoreProvenWindowCap {
t.Fatalf("every window must stay inside [%s, %s]; got %s", restoreProvenWindowFloor, restoreProvenWindowCap, got)
}
})
}
}
// The rhythm must be OBSERVED from the reports, not assumed — including the slice-A.4 rule that a
// PBS-targeted vzdump appears in both arrays and must be attributed by TARGET TYPE.
func TestObservedArchiveIntervals_FromReports(t *testing.T) {
base := time.Date(2026, 7, 1, 2, 0, 0, 0, time.UTC)
mk := func(localAt []time.Time, pbsAt []time.Time) string {
type stg struct{ Name, Type, Content string }
type bk struct {
TargetID string `json:"target_id"`
Success bool `json:"success"`
StartedAt string `json:"started_at"`
}
type snap struct {
BackupTime string `json:"backup_time"`
}
payload := struct {
StorageTargets []struct {
Name string `json:"name"`
Type string `json:"type"`
Content string `json:"content"`
} `json:"storage_targets"`
Backups []bk `json:"backups"`
PBSSnapshots []snap `json:"pbs_snapshots"`
}{}
payload.StorageTargets = append(payload.StorageTargets, struct {
Name string `json:"name"`
Type string `json:"type"`
Content string `json:"content"`
}{"felhom-backup", "dir", "backup"}, struct {
Name string `json:"name"`
Type string `json:"type"`
Content string `json:"content"`
}{"felhom-pbs", "pbs", "backup"})
for _, at := range localAt {
payload.Backups = append(payload.Backups, bk{"felhom-backup", true, at.Format(time.RFC3339)})
}
for _, at := range pbsAt {
// The SAME archive appears as a vzdump record AND as a snapshot — slice A.4.
payload.Backups = append(payload.Backups, bk{"felhom-pbs", true, at.Format(time.RFC3339)})
payload.PBSSnapshots = append(payload.PBSSnapshots, snap{at.Format(time.RFC3339)})
}
b, err := json.Marshal(payload)
if err != nil {
t.Fatal(err)
}
return string(b)
}
rows := []store.HostReportRow{
{ReportJSON: mk(
[]time.Time{base, base.AddDate(0, 0, 1), base.AddDate(0, 0, 2)},
[]time.Time{base, base.AddDate(0, 0, 7)},
)},
{ReportJSON: `{{{malformed`}, // must not blind the scan
}
got := observedArchiveIntervals(rows)
if d, ok := got["local"]; !ok || d != 24*time.Hour {
t.Fatalf("a daily host tier must be observed as ~24h; got %s ok=%v", d, ok)
}
if d, ok := got["pbs"]; !ok || d != 7*24*time.Hour {
t.Fatalf("a weekly offsite tier must be observed as ~7d — and its vzdump record must NOT be "+
"counted into the host tier (slice A.4); got %s ok=%v", d, ok)
}
// One generation is not a rhythm: unobservable, so the caller falls back to the declared one.
single := []store.HostReportRow{{ReportJSON: mk(nil, []time.Time{base})}}
if d, ok := observedArchiveIntervals(single)["pbs"]; ok {
t.Fatalf("one archive cannot yield an interval; got %s", d)
}
}