hub v0.91.0 — the staleness window learns each tier's own rhythm (R-86 Part 2)
gates / gates (push) Successful in 7s
gates / gates (push) Successful in 7s
Ships WITH agent v0.121.0, not after it. The agent now proves a tier once per
ARCHIVE GENERATION, so a weekly tier is proved weekly — in perfect health. The
flat 7-day restoreProvenStaleAfter derived its number from the 24h cadence R-86
removes, and a healthy weekly tier's proof age reaches EXACTLY 168h just before
its next proof: it sat ON the line, so any ordinary delay tipped it into a
nightly alarm about a working system.
restoreProvenWindow(tier, observed, ok):
- the tier's own archive interval, OBSERVED from reports the hub already holds
(pbs_snapshots + successful backups attributed by TARGET TYPE, slice A.4)
- x4 generations = the same tolerance the flat constant expressed
- floored at 7d (never tighter than before), capped at 12d (strictly inside the
2-week offsite retention)
- falls back to the DECLARED rhythm (26h host / 8d offsite — the thresholds the
backup-freshness checker already uses) when history is too short to observe
one; falling back to the FLOOR would recreate the false alarm on a fresh box
Kept: absence is UNKNOWN until the anchored window passes; the signal stays
edge-triggered; failed and stale remain distinct events. Every reason string now
states the window it was judged against (R-100's corollary).
Also backfills the missing v0.90.1 CHANGELOG entry (deployed since f21e7ca), and
records the operator's 2026-08-03 ruling that ep0 is Tier 2 / protected.
This commit is contained in:
@@ -29,20 +29,23 @@ prohibition as covering the act it names and nothing more.
|
||||
|---|---|---|
|
||||
| **0 — disposable. Reach here first.** | Exists to be broken; reinstalling is a routine afternoon, not an incident. **A drill that needs a victim uses one of these.** | `demo-hp` (t740), `demo-felhom` (N100) |
|
||||
| **1 — create and destroy freely** | Throwaway VMs, guests, scratch customers — **hosted on a Tier 0 machine** | drill VMs, scratch guests |
|
||||
| **2 — protected. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster** — and, by D-d, **nothing else** |
|
||||
| **2 — protected. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster**, **`ep0`** (operator ruling 2026-08-03) — and nothing else |
|
||||
|
||||
**DooPlex is Tier 2 because it *is* the recovery chain** — hub, Gitea, registry, PBS, k3s + Longhorn.
|
||||
Everything else rebuilds from it; it rebuilds from nothing. A bad moment in a DR drill there costs the
|
||||
thing under test, the source of truth for it, and the backups, at once.
|
||||
|
||||
**`ep0` + the Hetzner Storage Boxes were Tier 2 until 2026-08-02 and are no longer** — D-d's protected
|
||||
list names two machines and ep0 is not one of them. **That does not make them scratch, and the
|
||||
difference is an act, not a tier** (see the rule above the table): ep0 holds the **PBS-DR datastore and
|
||||
the restic copy of a real customer's data**, which is the only off-premises copy that exists, so
|
||||
*deleting datastores, prune jobs, tunnel config or nftables rules* remains forbidden by what it would
|
||||
destroy rather than by what tier it sits in. Reads are fine; it is still never a drill target.
|
||||
**Flagged for the operator: D-d did not name ep0 either way.** Confirm it explicitly — this page has
|
||||
read it the narrow way (not protected, but not wipeable) rather than assume the broad one.
|
||||
**`ep0` is Tier 2 — PROTECTED. Operator ruling, 2026-08-03.** D-d named two protected machines and did
|
||||
not name ep0 either way, so this page carried the question in writing for two days and read it the
|
||||
narrow way meanwhile (not protected, but not wipeable). The ruling settles it and **extends D-d's
|
||||
protected list to three machines**: DooPlex, Peti's cluster, ep0.
|
||||
|
||||
The reason it was never really in doubt: ep0 holds the **PBS-DR datastore and the restic copy of a
|
||||
real customer's data**, which is the only off-premises copy that exists. So *deleting datastores,
|
||||
prune jobs, tunnel config or nftables rules* was already forbidden by what it would destroy; the
|
||||
ruling makes the classification say so plainly instead of leaving each session to re-derive it.
|
||||
**Reads are fine** — including the ordinary off-site read a restore-test performs (R-86) — and it is
|
||||
never a drill target. The Hetzner Storage Boxes ride the same reasoning.
|
||||
|
||||
**Standing ruling, 2026-07-25 (`operations/nodes.md`):** drill and build VMs live on the **t740** — not
|
||||
felhom-pve, and **moved off DooPlex**. This page exists because that ruling sat where no session reads.
|
||||
@@ -95,12 +98,15 @@ still shares a device with its guest, so a drive failure is **offsite-only recov
|
||||
migrated, parked until the tester reinstalls (`PETI` in `backlog/OPEN-ITEMS.md`). Currently DOWN, no
|
||||
enrolled host. No access route from DooPlex, and nothing here needs one.
|
||||
|
||||
### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **not protected by D-d; not scratch either**
|
||||
### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **Tier 2, PROTECTED** (operator ruling 2026-08-03)
|
||||
|
||||
Reads are fine. It is the **offsite of last resort** (PBS-DR datastore, WireGuard hub, operator OOB
|
||||
path) and — until 2026-08-03 — RAM-constrained (3.8 GB, R-90); it is now a **CX33 with 8 GB RAM and a
|
||||
4 GiB swapfile**, which is what closed R-90. A very large restore is still worth watching. Do not delete datastores, prune
|
||||
jobs, tunnel config or nftables rules; never a drill target. The Storage Boxes hold the restic copy —
|
||||
4 GiB swapfile**, which is what closed R-90. A very large restore is still worth watching — the 8 GB
|
||||
is comfortable, not unbounded, and the OOM that started R-90 was a 14.46 GB restore read against
|
||||
3.8 GB. Do not delete datastores, prune jobs, tunnel config or nftables rules; never a drill target.
|
||||
**The ordinary off-site READ a restore-test performs is permitted and unchanged by the ruling**
|
||||
(R-86): the classification forbids destruction, not use. The Storage Boxes hold the restic copy —
|
||||
customer documents and photos, on a credential that can still delete (R-95).
|
||||
**Access: `ssh root@167.233.158.164` from DooPlex** — *not* `felhom-pve → 10.77.0.1`, the route that
|
||||
produced a false "unreachable" verdict (standing rule 2).
|
||||
|
||||
@@ -1,3 +1,71 @@
|
||||
## v0.91.0 — a tier's staleness window learns the tier's own rhythm (2026-08-03, R-86 Part 2)
|
||||
|
||||
**This ships WITH the agent's v0.121.0, not after it.** The agent now proves a tier once per ARCHIVE
|
||||
GENERATION rather than on a 24h timer, so a tier backed up weekly is proved weekly — correctly, and
|
||||
in perfect health. `restoreProvenStaleAfter` was a flat 7 days, and its own comment derived that
|
||||
number from the cadence R-86 removes:
|
||||
|
||||
> *"the restore-test cadence is 24h and rotation is oldest-first across two tiers, so each tier is
|
||||
> proven roughly every 2 days. 7 days therefore tolerates ~3 consecutive missed opportunities."*
|
||||
|
||||
A weekly tier proved weekly reaches a proof age of **exactly one interval** just before its next
|
||||
proof — 168h against a 168h window, false by a hair — so it did not merely sit near that line, it sat
|
||||
ON it, and any ordinary delay (a late backup, one deferral behind a running backup) tipped it into a
|
||||
nightly alarm about a system that is working. Shipping the agent alone would have converted the
|
||||
improvement into a false alarm.
|
||||
|
||||
**The window is now per tier**, `restoreProvenWindow(tier, observed, ok)`:
|
||||
|
||||
- **the tier's own archive interval**, observed from the host-reports the hub already holds —
|
||||
`pbs_snapshots[]` plus successful `backups[]` attributed by TARGET TYPE (the slice-A.4 rule: a
|
||||
PBS-targeted vzdump appears in both arrays, and classifying by array membership would attribute an
|
||||
offsite archive to the host tier);
|
||||
- **× 4 generations** — the settle generation plus ~3 missed opportunities, deliberately the same
|
||||
tolerance the flat constant expressed. The change is to the RHYTHM, not to the patience;
|
||||
- **floored at 7 days** — the old constant kept as a floor, so no tier is judged more tightly than
|
||||
before;
|
||||
- **capped at 12 days** — strictly inside the 2-week offsite retention with two days to spare, so a
|
||||
tier is never called stale against an archive PBS has already pruned;
|
||||
- **falling back to the DECLARED rhythm** when a box's history is too short to observe one:
|
||||
`backupStaleAfter` (26h) for the host tier and `offsiteBackupStaleAfter` (8d) for the offsite tier
|
||||
— the very thresholds the backup-freshness checker already judges those tiers against. A fresh box
|
||||
with one snapshot has no observable interval, and falling back to the FLOOR there would recreate
|
||||
the false alarm on exactly the tier this task is about.
|
||||
|
||||
**Kept, because both were earned:** absence is UNKNOWN until an anchored window has passed (R-81's
|
||||
structure, untouched), and the stale signal stays edge-triggered. `restore_test_failed` and
|
||||
`restore_test_stale` remain DISTINCT — one says your recovery is broken, the other says it is
|
||||
unverified, and the second is the one that quietly becomes the first.
|
||||
|
||||
**Every reason string now states the window it was judged against** — R-100's corollary: when a
|
||||
verdict changes what it counts from, the alarm text has to change with it, or an operator reads
|
||||
"limit 168h" under a tier actually judged at 288h and dismisses a true alarm.
|
||||
|
||||
**The window READ is unchanged in cost** (14 days), which is both enough to find proof inside the
|
||||
widest window and enough to see two generations of a weekly tier.
|
||||
|
||||
## v0.90.1 — the digest's per-app lines stop repeating the filesystem figures (2026-08-03, R-182)
|
||||
|
||||
**Backfilled 2026-08-03 (R-86 session).** This version was built, deployed and recorded in `REPORT.md`
|
||||
and the R-182 row, but never given a CHANGELOG entry — and `REPORT.md` is overwritten every session,
|
||||
so the per-repo history under-reported what was actually running. The deployed image has been
|
||||
`felhom-hub:0.90.1` since `f21e7ca`.
|
||||
|
||||
Found by reading the first REAL digest, not by design: every app row ended with the same usage clause
|
||||
the mail already prints once on its own Filesystem line. On a two-app box that is untidy; down a list
|
||||
of a dozen it is the same forty characters twelve times, pushing the part that DIFFERS off a phone
|
||||
screen at 07:00 — the only moment that mail has to work.
|
||||
|
||||
The reserve's refusal message is authored for a single-app alert where naming the filesystem is
|
||||
right, so the message is unchanged and the DIGEST trims the duplicate when rendering.
|
||||
`trimRepeatedUsage` removes ONLY an exact "— <target path>:" suffix, so an unrelated reason is
|
||||
untouched and a reason that is nothing but the usage clause is left alone rather than emptied.
|
||||
|
||||
Also inverted the operator half of `TestRecoveryUnitCaptureFailed_NeverReachesTheCustomer`: it
|
||||
required the operator to be e-mailed a per-app capture failure, which was correct when that event was
|
||||
the only signal and is wrong now that it is the RECORD and the digest is the notification. The
|
||||
customer-safety claim is unchanged — R-158's guarantee MOVED, it did not weaken.
|
||||
|
||||
## v0.90.0 — a dropped notification leaves a trace, and the backup digest arrives (2026-08-03, R-182)
|
||||
|
||||
**The smallest change on the board with the largest effect on trust: `processOperator`'s cooldown no
|
||||
|
||||
@@ -32,15 +32,87 @@ import (
|
||||
// is a NEW monitor written straight after the third, so it copies R-81's verdict structure rather
|
||||
// than re-deriving it. A tier never proven on a newborn box is UNKNOWN, never FAILED.
|
||||
|
||||
// restoreProvenStaleAfter is how long a tier may go unproven before it is called stale.
|
||||
// ── HOW LONG MAY A TIER GO UNPROVEN? (R-86 Part 2) ───────────────────────────────────────────
|
||||
//
|
||||
// Derivation, not a guess: the restore-test cadence is 24h and rotation is oldest-first across two
|
||||
// tiers, so each tier is proven roughly every 2 days. 7 days therefore tolerates ~3 consecutive
|
||||
// missed opportunities before alarming — loud enough to matter, quiet enough not to fire on one
|
||||
// skipped cycle (a deferral behind a long backup is normal, not a fault). It is also comfortably
|
||||
// inside the 2-week offsite retention (operator ruling 2026-07-26), so a tier is never reported
|
||||
// stale against an archive that is about to be pruned anyway.
|
||||
const restoreProvenStaleAfter = 7 * 24 * time.Hour
|
||||
// This was one flat constant, 7 days, and its comment derived that number like this:
|
||||
//
|
||||
// "the restore-test cadence is 24h and rotation is oldest-first across two tiers, so each tier is
|
||||
// proven roughly every 2 days. 7 days therefore tolerates ~3 consecutive missed opportunities."
|
||||
//
|
||||
// **That premise is exactly what R-86 removed.** The agent no longer tests on an interval at all: a
|
||||
// tier is tested once per ARCHIVE GENERATION — when it holds a settled archive that has not been
|
||||
// proven. A tier backed up weekly is therefore proved weekly, by design and in perfect health, and
|
||||
// against a flat 7-day window it would sit on the line and alarm every night about a system that is
|
||||
// working. Shipping the agent's half alone would have converted the improvement into a false alarm,
|
||||
// which is why the two ship together.
|
||||
//
|
||||
// The window is now derived from **the tier's own backup rhythm**, which the hub can observe from
|
||||
// the reports it already receives, and it keeps everything the constant had earned:
|
||||
//
|
||||
// - absence is UNKNOWN until an anchored window has passed (R-81's structure, untouched);
|
||||
// - the signal stays edge-triggered;
|
||||
// - it never exceeds the offsite retention, so a tier is never called stale against an archive
|
||||
// that is about to be pruned;
|
||||
// - and it is never TIGHTER than the 7 days that were already tolerated.
|
||||
const (
|
||||
// restoreProvenGenerations is how many archive generations may pass unproven before alarming.
|
||||
// 4 = the settle lag's own generation plus ~3 missed opportunities — deliberately the same
|
||||
// tolerance the flat constant expressed, so the change is to the RHYTHM, not to the patience.
|
||||
restoreProvenGenerations = 4
|
||||
|
||||
// restoreProvenWindowFloor is the shortest window that may be applied to any tier. It is the
|
||||
// old constant, kept as a FLOOR rather than deleted: a daily tier computes 4 days from its own
|
||||
// rhythm, and tightening a live threshold is not what this task is for. A deferral behind a
|
||||
// long backup is normal, not a fault.
|
||||
restoreProvenWindowFloor = 7 * 24 * time.Hour
|
||||
|
||||
// restoreProvenWindowCap keeps the window strictly inside the 2-week offsite retention
|
||||
// (operator ruling 2026-07-26) with two days to spare. Beyond it the hub would be judging a
|
||||
// tier against an archive PBS has already pruned — an alarm nobody can act on, and the bound
|
||||
// the old constant respected in its own way.
|
||||
restoreProvenWindowCap = 12 * 24 * time.Hour
|
||||
|
||||
// restoreWindowRead is how far back the hub reads host-reports for this check: far enough to
|
||||
// find proof anywhere inside the widest window, and to see at least two archive generations of
|
||||
// a WEEKLY tier so its rhythm is observable at all.
|
||||
restoreWindowRead = 2 * restoreProvenWindowFloor
|
||||
)
|
||||
|
||||
// declaredArchiveInterval is the rhythm the hub ALREADY attributes to a tier — the same thresholds
|
||||
// the backup-freshness checker judges it against (deadline.go / deadline_tiers.go). It is the
|
||||
// fallback when a box's history is too short to observe a rhythm, and it is the right fallback
|
||||
// precisely because it is not a second opinion: if these two checkers disagreed about how often a
|
||||
// tier is expected to receive an archive, one of them would be alarming on the other's model.
|
||||
//
|
||||
// It is stated per RESTORE tier name ("local"/"pbs" — what the agent reports as source_tier), which
|
||||
// is the same split the backup tiers use under different names ("host"/"offsite").
|
||||
func declaredArchiveInterval(tier string) time.Duration {
|
||||
if tier == "pbs" {
|
||||
return offsiteBackupStaleAfter // 8 days: the weekly cadence plus a day of headroom
|
||||
}
|
||||
return backupStaleAfter // 26 hours: the daily cadence plus headroom
|
||||
}
|
||||
|
||||
// restoreProvenWindow is how long THIS tier may go unproven, given its observed archive interval.
|
||||
//
|
||||
// observedOK=false means the box's retained history did not contain two archive generations for
|
||||
// this tier, so the declared rhythm is used. That fallback matters most for exactly the tier this
|
||||
// task is about: a fresh box with a weekly offsite tier has one snapshot and no observable
|
||||
// interval, and falling back to the FLOOR there would recreate the false alarm.
|
||||
func restoreProvenWindow(tier string, observed time.Duration, observedOK bool) time.Duration {
|
||||
interval := declaredArchiveInterval(tier)
|
||||
if observedOK && observed > 0 {
|
||||
interval = observed
|
||||
}
|
||||
w := time.Duration(restoreProvenGenerations) * interval
|
||||
if w < restoreProvenWindowFloor {
|
||||
w = restoreProvenWindowFloor
|
||||
}
|
||||
if w > restoreProvenWindowCap {
|
||||
w = restoreProvenWindowCap
|
||||
}
|
||||
return w
|
||||
}
|
||||
|
||||
// Event types. Operator-tier only — see the dispatcher note in RestoreTestChecker.
|
||||
const (
|
||||
@@ -156,12 +228,13 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
|
||||
return
|
||||
}
|
||||
|
||||
rows, err := c.store.GetHostReportsSince(customerID, now.Add(-2*restoreProvenStaleAfter))
|
||||
rows, err := c.store.GetHostReportsSince(customerID, now.Add(-restoreWindowRead))
|
||||
if err != nil {
|
||||
c.logger.Printf("[WARN] restore-test check: window read failed for %s: %v", customerID, err)
|
||||
return
|
||||
}
|
||||
proven := lastProvenPerTier(rows)
|
||||
intervals := observedArchiveIntervals(rows)
|
||||
|
||||
first, ferr := c.store.GetFirstHostReportAt(customerID)
|
||||
if ferr != nil {
|
||||
@@ -170,7 +243,9 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
|
||||
}
|
||||
|
||||
for _, tier := range tiers {
|
||||
v := assessRestoreProven(tier, proven[tier], first, now)
|
||||
observed, observedOK := intervals[tier]
|
||||
window := restoreProvenWindow(tier, observed, observedOK)
|
||||
v := assessRestoreProven(tier, proven[tier], first, now, window)
|
||||
key := customerID + "|" + tier
|
||||
c.mu.Lock()
|
||||
prev := c.staleStates[key]
|
||||
@@ -192,37 +267,120 @@ func (c *RestoreTestChecker) checkStaleness(customerID, latestJSON string, now t
|
||||
}
|
||||
}
|
||||
|
||||
// assessRestoreProven is the per-tier verdict. PURE (now injected) so the policy is unit-tested —
|
||||
// the property that made R-81 provable, kept deliberately.
|
||||
// assessRestoreProven is the per-tier verdict. PURE (now and the window injected) so the policy is
|
||||
// unit-tested — the property that made R-81 provable, kept deliberately.
|
||||
//
|
||||
// no proof, anchor NOT elapsed → UNKNOWN (newborn box; never an alarm)
|
||||
// no proof, anchor elapsed → MISSED
|
||||
// proof older than the window → MISSED
|
||||
// otherwise → OK
|
||||
func assessRestoreProven(tier string, provenAt, firstReportAt, now time.Time) backupAssessment {
|
||||
//
|
||||
// `window` is now the TIER'S OWN (R-86 Part 2) rather than one constant for every tier, and every
|
||||
// reason string states the window it was judged against. That is R-100's corollary applied here:
|
||||
// when a verdict changes what it counts from, the alarm text has to change with it, or an operator
|
||||
// reads "limit 168h" under a tier that was actually judged at 288h and dismisses a true alarm.
|
||||
func assessRestoreProven(tier string, provenAt, firstReportAt, now time.Time, window time.Duration) backupAssessment {
|
||||
if provenAt.IsZero() {
|
||||
if firstReportAt.IsZero() {
|
||||
return backupAssessment{verdict: verdictMissed,
|
||||
reason: fmt.Sprintf("%s tier: never restore-proven, and no first-contact anchor to defer against", tier)}
|
||||
}
|
||||
watched := now.Sub(firstReportAt)
|
||||
if watched <= restoreProvenStaleAfter {
|
||||
if watched <= window {
|
||||
return backupAssessment{verdict: verdictUnknown,
|
||||
reason: fmt.Sprintf("%s tier: not restore-proven yet, but only watching for %s (grace %s since first contact %s) — newborn, not a fault",
|
||||
tier, watched.Round(time.Hour), restoreProvenStaleAfter, firstReportAt.Format(time.RFC3339))}
|
||||
tier, watched.Round(time.Hour), window, firstReportAt.Format(time.RFC3339))}
|
||||
}
|
||||
return backupAssessment{verdict: verdictMissed,
|
||||
reason: fmt.Sprintf("%s tier: NEVER successfully restore-proven in %s of watching (limit %s) — the tier is unverified, not known-broken",
|
||||
tier, watched.Round(time.Hour), restoreProvenStaleAfter)}
|
||||
reason: fmt.Sprintf("%s tier: NEVER successfully restore-proven in %s of watching (limit %s, this tier's own backup rhythm) — the tier is unverified, not known-broken",
|
||||
tier, watched.Round(time.Hour), window)}
|
||||
}
|
||||
if age := now.Sub(provenAt); age > restoreProvenStaleAfter {
|
||||
if age := now.Sub(provenAt); age > window {
|
||||
return backupAssessment{verdict: verdictMissed,
|
||||
reason: fmt.Sprintf("%s tier: last successful restore-test was %s ago (limit %s) — the tier is unverified, not known-broken",
|
||||
tier, age.Round(time.Hour), restoreProvenStaleAfter)}
|
||||
reason: fmt.Sprintf("%s tier: last successful restore-test was %s ago (limit %s, this tier's own backup rhythm) — the tier is unverified, not known-broken",
|
||||
tier, age.Round(time.Hour), window)}
|
||||
}
|
||||
return backupAssessment{verdict: verdictOK}
|
||||
}
|
||||
|
||||
// observedArchiveIntervals estimates how often each RESTORE tier actually receives an archive, from
|
||||
// the host-reports the hub already holds. Keyed by restore-tier name ("local"/"pbs").
|
||||
//
|
||||
// Evidence is every distinct archive timestamp in the window: `pbs_snapshots[]` for the offsite
|
||||
// tier (PBS enumerates its whole retention in each report, so one report usually settles the
|
||||
// question) and successful `backups[]` records attributed by TARGET TYPE for both tiers — the
|
||||
// slice-A.4 rule, because a PBS-targeted vzdump appears in BOTH arrays and classifying by array
|
||||
// membership would attribute an offsite archive to the host tier.
|
||||
//
|
||||
// The estimate is the MEAN gap: (newest − oldest) / (generations − 1). It needs two generations;
|
||||
// with fewer, ok=false and the caller falls back to the declared rhythm. It is deliberately crude,
|
||||
// and can afford to be: restoreProvenWindow clamps the result between a 7-day floor and a 12-day
|
||||
// cap, so the only discrimination this has to get right is "roughly daily" versus "several days or
|
||||
// slower" — which is exactly the distinction that turns a healthy weekly tier into a false alarm.
|
||||
func observedArchiveIntervals(rows []store.HostReportRow) map[string]time.Duration {
|
||||
seen := map[string]map[int64]struct{}{ // tier → set of archive unix times
|
||||
"local": {},
|
||||
"pbs": {},
|
||||
}
|
||||
add := func(tier string, t time.Time) {
|
||||
if t.IsZero() {
|
||||
return
|
||||
}
|
||||
seen[tier][t.UTC().Unix()] = struct{}{}
|
||||
}
|
||||
|
||||
for _, r := range rows {
|
||||
var hr hostReportBackups
|
||||
if json.Unmarshal([]byte(r.ReportJSON), &hr) != nil {
|
||||
continue // one malformed retained report must not blind the scan
|
||||
}
|
||||
pbs := pbsTargetSet(hr)
|
||||
for _, ps := range hr.PBSSnapshots {
|
||||
if t, ok := parseBackupTime(ps.BackupTime); ok {
|
||||
add("pbs", t)
|
||||
}
|
||||
}
|
||||
for _, b := range hr.Backups {
|
||||
if !b.Success {
|
||||
continue
|
||||
}
|
||||
t, ok := parseBackupTime(b.StartedAt)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if pbs[b.TargetID] {
|
||||
add("pbs", t)
|
||||
} else {
|
||||
add("local", t)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
out := map[string]time.Duration{}
|
||||
for tier, set := range seen {
|
||||
if len(set) < 2 {
|
||||
continue // not observable — the caller uses the declared rhythm
|
||||
}
|
||||
var oldest, newest int64
|
||||
first := true
|
||||
for ts := range set {
|
||||
if first || ts < oldest {
|
||||
oldest = ts
|
||||
}
|
||||
if first || ts > newest {
|
||||
newest = ts
|
||||
}
|
||||
first = false
|
||||
}
|
||||
span := time.Duration(newest-oldest) * time.Second
|
||||
if span <= 0 {
|
||||
continue
|
||||
}
|
||||
out[tier] = span / time.Duration(len(set)-1)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// expectedRestoreTiers names the tiers this box actually HAS, so a box without an offsite tier is
|
||||
// never reported stale for one. Same gate as Slice C's `expected`, and for the same reason: without
|
||||
// it every box lacking a tier would alarm once the anchor elapsed — absence-is-not-failure,
|
||||
|
||||
@@ -182,26 +182,33 @@ func TestRestoreTest_NewbornDoesNotAlarm(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// The boundary, pinned by name so a refactor has to delete an obviously-named contract.
|
||||
// The boundary, pinned by name so a refactor has to delete an obviously-named contract. R-86 made
|
||||
// the limit per-tier, so the anchor is now measured against THE TIER'S OWN window — here the local
|
||||
// tier's, which clamps to the 7-day floor and so keeps this contract numerically identical to the
|
||||
// one the flat constant expressed.
|
||||
func TestRestoreTest_Contract_UnprovenIsUnknownUntilTheAnchorElapses(t *testing.T) {
|
||||
now := time.Date(2026, 7, 26, 12, 0, 0, 0, time.UTC)
|
||||
window := restoreProvenWindow("local", 24*time.Hour, true)
|
||||
if window != restoreProvenWindowFloor {
|
||||
t.Fatalf("precondition: a daily local tier must clamp to the floor; got %s", window)
|
||||
}
|
||||
cases := []struct {
|
||||
name string
|
||||
watched time.Duration
|
||||
wantMissed bool
|
||||
}{
|
||||
{"newborn, 1h", time.Hour, false},
|
||||
{"just inside", restoreProvenStaleAfter - time.Minute, false},
|
||||
{"exactly at the limit", restoreProvenStaleAfter, false},
|
||||
{"just outside", restoreProvenStaleAfter + time.Minute, true},
|
||||
{"just inside", window - time.Minute, false},
|
||||
{"exactly at the limit", window, false},
|
||||
{"just outside", window + time.Minute, true},
|
||||
{"long past", 30 * 24 * time.Hour, true},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
got := assessRestoreProven("pbs", time.Time{}, now.Add(-c.watched), now)
|
||||
got := assessRestoreProven("local", time.Time{}, now.Add(-c.watched), now, window)
|
||||
if got.missed() != c.wantMissed {
|
||||
t.Fatalf("CONTRACT VIOLATED: unproven for %s (limit %s) → missed=%v, want %v (reason %q)",
|
||||
c.watched, restoreProvenStaleAfter, got.missed(), c.wantMissed, got.reason)
|
||||
c.watched, window, got.missed(), c.wantMissed, got.reason)
|
||||
}
|
||||
if !c.wantMissed && got.verdict != verdictUnknown {
|
||||
t.Fatalf("a deferred tier must be UNKNOWN (visible), not OK; got verdict=%d", got.verdict)
|
||||
@@ -215,15 +222,17 @@ func TestRestoreTest_Contract_UnprovenIsUnknownUntilTheAnchorElapses(t *testing.
|
||||
func TestRestoreTest_StaleIsSeparateFromFailure(t *testing.T) {
|
||||
now := time.Date(2026, 7, 26, 12, 0, 0, 0, time.UTC)
|
||||
|
||||
stale := assessRestoreProven("pbs", now.Add(-9*24*time.Hour), now.Add(-60*24*time.Hour), now)
|
||||
// A DAILY tier judged on its own rhythm: the window clamps to the 7-day floor.
|
||||
daily := restoreProvenWindow("local", 24*time.Hour, true)
|
||||
stale := assessRestoreProven("local", now.Add(-9*24*time.Hour), now.Add(-60*24*time.Hour), now, daily)
|
||||
if !stale.missed() {
|
||||
t.Fatalf("a tier last proven 9 days ago (limit %s) must be stale; got %q", restoreProvenStaleAfter, stale.reason)
|
||||
t.Fatalf("a daily tier last proven 9 days ago (limit %s) must be stale; got %q", daily, stale.reason)
|
||||
}
|
||||
// The wording must not read as "broken" — that is the other signal.
|
||||
if !strings.Contains(stale.reason, "unverified, not known-broken") {
|
||||
t.Fatalf("staleness must say UNVERIFIED, not broken — merging the two is the thing this avoids; got %q", stale.reason)
|
||||
}
|
||||
fresh := assessRestoreProven("pbs", now.Add(-2*24*time.Hour), now.Add(-60*24*time.Hour), now)
|
||||
fresh := assessRestoreProven("local", now.Add(-2*24*time.Hour), now.Add(-60*24*time.Hour), now, daily)
|
||||
if fresh.verdict != verdictOK {
|
||||
t.Fatalf("a tier proven 2 days ago is fine; got verdict=%d reason=%q", fresh.verdict, fresh.reason)
|
||||
}
|
||||
@@ -283,3 +292,199 @@ func boolStr(b bool) string {
|
||||
}
|
||||
return "false"
|
||||
}
|
||||
|
||||
// ── SCENARIO G — a healthy WEEKLY tier is never reported stale (R-86 Part 2) ─────────────────
|
||||
//
|
||||
// This is the test that pins the false alarm this change would otherwise have CREATED. The agent
|
||||
// now proves a tier once per archive generation, so a weekly offsite tier is proved weekly — in
|
||||
// perfect health. Against the old flat 7-day window it would sit on the line and alarm every night.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-08-03): pin the window flat, as it was —
|
||||
//
|
||||
// - window := restoreProvenWindow(tier, observed, observedOK)
|
||||
// - window := restoreProvenWindowFloor // the pre-R-86 flat 7 days
|
||||
//
|
||||
// → --- FAIL: TestRestoreTest_HealthyWeeklyTierIsNeverStale
|
||||
//
|
||||
// week 0: a weekly tier proved on its own archive must never be stale (proof age 172h0m0s,
|
||||
// window 168h0m0s); verdict=2 reason="pbs tier: last successful restore-test was 172h0m0s ago
|
||||
// (limit 168h0m0s, this tier's own backup rhythm) — the tier is unverified, not known-broken"
|
||||
//
|
||||
// Restored. The mutation is one line because the whole of Part 2 is one decision: whose rhythm.
|
||||
//
|
||||
// NOTE, because it is the finding this test nearly hid: the FIRST version of this fixture had NO
|
||||
// jitter, and it PASSED under the mutation. A perfectly regular weekly tier's proof age reaches
|
||||
// EXACTLY 168h just before the next proof, and `age > window` is false by a hair — a hollow test
|
||||
// that would have shipped Part 1 and its false alarm together. The jitter below is what makes this
|
||||
// a test, and it is also the truth about the old constant: a healthy weekly tier did not merely sit
|
||||
// near the line, it sat ON it, so any ordinary delay tipped it over.
|
||||
func TestRestoreTest_HealthyWeeklyTierIsNeverStale(t *testing.T) {
|
||||
start := time.Date(2026, 6, 1, 3, 0, 0, 0, time.UTC)
|
||||
firstContact := start.Add(-24 * time.Hour)
|
||||
|
||||
// The observable rhythm of a weekly tier, as the hub would compute it from the reports. No
|
||||
// assertion about the window ITSELF here on purpose: that is the mechanism, and it is pinned in
|
||||
// TestRestoreProvenWindow_Contract. What this test asserts is the CONSEQUENCE — does the alarm
|
||||
// fire? — because R-97b proved a mechanism and shipped a broken consequence anyway.
|
||||
weekly := restoreProvenWindow("pbs", 7*24*time.Hour, true)
|
||||
|
||||
// Walk several weeks of a HEALTHY tier, with the jitter a real one has: the backup does not land
|
||||
// to the second, and a restore-test can be deferred one evaluation behind a running backup.
|
||||
//
|
||||
// The jitter is the point. A perfectly regular weekly tier's proof reaches an age of EXACTLY one
|
||||
// interval (168h) just before the next proof, and against a flat 168h window `age > window` is
|
||||
// false by a hair — so a regular fixture would pass against the very constant this task must
|
||||
// change, and prove nothing. That is the brief's "sits exactly on that line": every real-world
|
||||
// delay pushes it over, and the alarm is about a system that is working.
|
||||
settle, evalLatency := 24*time.Hour, 6*time.Hour
|
||||
archiveLate := []time.Duration{0, 4 * time.Hour, 2 * time.Hour, 6 * time.Hour, 0, 3 * time.Hour}
|
||||
deferred := []time.Duration{0, 0, 6 * time.Hour, 0, 0, 6 * time.Hour} // one evaluation behind a backup
|
||||
|
||||
archiveAt := func(week int) time.Time {
|
||||
return start.AddDate(0, 0, 7*week).Add(archiveLate[week])
|
||||
}
|
||||
provenAt := func(week int) time.Time {
|
||||
return archiveAt(week).Add(settle + evalLatency + deferred[week])
|
||||
}
|
||||
|
||||
var worst time.Duration
|
||||
for week := 0; week+1 < len(archiveLate); week++ {
|
||||
// The widest the proof's age ever gets: the instant before the NEXT week's proof lands.
|
||||
now := provenAt(week + 1).Add(-time.Second)
|
||||
age := now.Sub(provenAt(week))
|
||||
if age > worst {
|
||||
worst = age
|
||||
}
|
||||
v := assessRestoreProven("pbs", provenAt(week), firstContact, now, weekly)
|
||||
if v.verdict != verdictOK {
|
||||
t.Fatalf("week %d: a weekly tier proved on its own archive must never be stale (proof age %s, window %s); verdict=%d reason=%q",
|
||||
week, age.Round(time.Hour), weekly, v.verdict, v.reason)
|
||||
}
|
||||
}
|
||||
// The fixture must actually EXERCISE the boundary — a jitter-free walk would sit at exactly one
|
||||
// interval and pass against a flat 7-day window, which is the hollow version of this test.
|
||||
if worst <= restoreProvenWindowFloor {
|
||||
t.Fatalf("this fixture never exceeds the old flat window (worst proof age %s) — it cannot detect the defect it exists for", worst)
|
||||
}
|
||||
}
|
||||
|
||||
// ...and a weekly tier that genuinely STOPS being proved must still alarm. A window that never
|
||||
// fires is not a fix, it is a deletion.
|
||||
func TestRestoreTest_WeeklyTierThatStopsBeingProvedStillAlarms(t *testing.T) {
|
||||
now := time.Date(2026, 7, 26, 12, 0, 0, 0, time.UTC)
|
||||
weekly := restoreProvenWindow("pbs", 7*24*time.Hour, true)
|
||||
|
||||
v := assessRestoreProven("pbs", now.Add(-weekly-time.Hour), now.Add(-90*24*time.Hour), now, weekly)
|
||||
if !v.missed() {
|
||||
t.Fatalf("a weekly tier unproven for longer than its own window MUST alarm; got verdict=%d reason=%q", v.verdict, v.reason)
|
||||
}
|
||||
if !strings.Contains(v.reason, weekly.String()) {
|
||||
t.Fatalf("the alarm must state the window it was judged against (R-100's corollary); got %q", v.reason)
|
||||
}
|
||||
}
|
||||
|
||||
// The window's own contract: derived from the tier's rhythm, floored, capped, and never dependent
|
||||
// on an unobservable history for the tier that would suffer most from a wrong answer.
|
||||
func TestRestoreProvenWindow_Contract(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
tier string
|
||||
observed time.Duration
|
||||
observedOK bool
|
||||
want time.Duration
|
||||
}{
|
||||
{"daily local clamps to the floor", "local", 24 * time.Hour, true, restoreProvenWindowFloor},
|
||||
{"weekly pbs widens", "pbs", 7 * 24 * time.Hour, true, restoreProvenWindowCap},
|
||||
{"3-day tier sits between", "pbs", 72 * time.Hour, true, 12 * 24 * time.Hour},
|
||||
{"unobservable local falls back to its declared rhythm", "local", 0, false, restoreProvenWindowFloor},
|
||||
{"unobservable pbs falls back WIDE, not to the floor", "pbs", 0, false, restoreProvenWindowCap},
|
||||
{"a nonsense zero interval is ignored", "pbs", 0, true, restoreProvenWindowCap},
|
||||
}
|
||||
// The relationship Part 1 depends on: a weekly tier's window must be WIDER than a daily tier's,
|
||||
// or proving weekly (which is now correct behaviour) alarms on itself.
|
||||
if restoreProvenWindow("pbs", 7*24*time.Hour, true) <= restoreProvenWindow("local", 24*time.Hour, true) {
|
||||
t.Fatal("a weekly tier must earn a wider window than a daily one — otherwise R-86's agent half alarms about itself")
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
got := restoreProvenWindow(c.tier, c.observed, c.observedOK)
|
||||
if got != c.want {
|
||||
t.Fatalf("window(%s, observed=%s ok=%v) = %s, want %s", c.tier, c.observed, c.observedOK, got, c.want)
|
||||
}
|
||||
if got < restoreProvenWindowFloor || got > restoreProvenWindowCap {
|
||||
t.Fatalf("every window must stay inside [%s, %s]; got %s", restoreProvenWindowFloor, restoreProvenWindowCap, got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The rhythm must be OBSERVED from the reports, not assumed — including the slice-A.4 rule that a
|
||||
// PBS-targeted vzdump appears in both arrays and must be attributed by TARGET TYPE.
|
||||
func TestObservedArchiveIntervals_FromReports(t *testing.T) {
|
||||
base := time.Date(2026, 7, 1, 2, 0, 0, 0, time.UTC)
|
||||
mk := func(localAt []time.Time, pbsAt []time.Time) string {
|
||||
type stg struct{ Name, Type, Content string }
|
||||
type bk struct {
|
||||
TargetID string `json:"target_id"`
|
||||
Success bool `json:"success"`
|
||||
StartedAt string `json:"started_at"`
|
||||
}
|
||||
type snap struct {
|
||||
BackupTime string `json:"backup_time"`
|
||||
}
|
||||
payload := struct {
|
||||
StorageTargets []struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
Content string `json:"content"`
|
||||
} `json:"storage_targets"`
|
||||
Backups []bk `json:"backups"`
|
||||
PBSSnapshots []snap `json:"pbs_snapshots"`
|
||||
}{}
|
||||
payload.StorageTargets = append(payload.StorageTargets, struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
Content string `json:"content"`
|
||||
}{"felhom-backup", "dir", "backup"}, struct {
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
Content string `json:"content"`
|
||||
}{"felhom-pbs", "pbs", "backup"})
|
||||
for _, at := range localAt {
|
||||
payload.Backups = append(payload.Backups, bk{"felhom-backup", true, at.Format(time.RFC3339)})
|
||||
}
|
||||
for _, at := range pbsAt {
|
||||
// The SAME archive appears as a vzdump record AND as a snapshot — slice A.4.
|
||||
payload.Backups = append(payload.Backups, bk{"felhom-pbs", true, at.Format(time.RFC3339)})
|
||||
payload.PBSSnapshots = append(payload.PBSSnapshots, snap{at.Format(time.RFC3339)})
|
||||
}
|
||||
b, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
rows := []store.HostReportRow{
|
||||
{ReportJSON: mk(
|
||||
[]time.Time{base, base.AddDate(0, 0, 1), base.AddDate(0, 0, 2)},
|
||||
[]time.Time{base, base.AddDate(0, 0, 7)},
|
||||
)},
|
||||
{ReportJSON: `{{{malformed`}, // must not blind the scan
|
||||
}
|
||||
|
||||
got := observedArchiveIntervals(rows)
|
||||
if d, ok := got["local"]; !ok || d != 24*time.Hour {
|
||||
t.Fatalf("a daily host tier must be observed as ~24h; got %s ok=%v", d, ok)
|
||||
}
|
||||
if d, ok := got["pbs"]; !ok || d != 7*24*time.Hour {
|
||||
t.Fatalf("a weekly offsite tier must be observed as ~7d — and its vzdump record must NOT be "+
|
||||
"counted into the host tier (slice A.4); got %s ok=%v", d, ok)
|
||||
}
|
||||
|
||||
// One generation is not a rhythm: unobservable, so the caller falls back to the declared one.
|
||||
single := []store.HostReportRow{{ReportJSON: mk(nil, []time.Time{base})}}
|
||||
if d, ok := observedArchiveIntervals(single)["pbs"]; ok {
|
||||
t.Fatalf("one archive cannot yield an interval; got %s", d)
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user