- OPEN-ITEMS: R-86 CLOSED with the trap in its own wording recorded (the literal reading is never true on a daily tier); R-87 re-ranked UP because R-86 built most of what it waited for; R-185 (the agent cannot list demo-felhom's host backup tier — a missing storage ACL, pre-existing), R-186 (a released binary's sha is not reproducible from its tag), R-187 (R-115's publish leg had never actually run) filed. R-184 was the highest ID in use. - ROADMAP: R-86 collapsed, keeping the reasoning and correcting the shape the row itself proposed — which would have been the never-fires version. - 07-backup-architecture: new contract section — restore-testing is per ARCHIVE GENERATION, with the trap and what did not change (S-1). - 00-capability-map: the unattended restore-proof row upgraded to PROVEN-LIVE on the 635 s due-triggered offsite run, with the restart and teardown evidence. - CONTEXT: S-17 (the rule, the trap, the config key, the hub's derivation) and S-18 (ep0 is Tier 2 — extends D-d's protected list to three machines). Numbered 17/18 because S-14 and S-15 were already duplicated in the file. - STATUS: rewritten for the operator, trimmed back to one screen.
8.5 KiB
REPORT — R-86 Part 2: the staleness window learns each tier's rhythm (2026-08-03)
Date: 2026-08-03 · Repo: felhom.eu hub v0.90.1 → v0.91.0 → v0.91.1
(323f45a, 046df30), manifests 687fedd, ff2655c. Ships with felhom-agent v0.121.0, not
after it. Ruling recorded: ep0 is Tier 2, protected.
1. Baselines, re-read on arrival
| Repo | main @ commit |
Version | Matched §1? |
|---|---|---|---|
felhom.eu |
e34b614e5b65 |
hub CHANGELOG top entry v0.90.0; manifests/hub.yaml ran 0.90.1 |
Yes — and the discrepancy was real |
The missing v0.90.1 CHANGELOG entry is backfilled. It was built, deployed and recorded in
REPORT.md + the R-182 row on 2026-08-03 (f21e7ca) but never given a CHANGELOG entry — and
REPORT.md is overwritten every session, so the per-repo history under-reported what was running.
2. Why Part 2 could not ship later
The agent now proves a tier once per archive generation, so a tier backed up weekly is proved
weekly — correctly, in perfect health. restoreProvenStaleAfter was a flat 7 days and its own comment
derived that number from the cadence R-86 removes.
The brief said a weekly tier would "sit exactly on that line". That is literally true, and the
arithmetic is worth stating: proofs land at archive + settle, so just before the next proof the
current one is exactly one interval old — 168 h against a 168 h window, and age > window is false
by a hair. It did not sit near the line, it sat on it. Any ordinary delay — a backup landing late,
one evaluation deferred behind a running backup — tips a healthy tier into a nightly alarm.
3. The window as derived
interval = declaredArchiveInterval(tier) // host 26h, offsite 8d
if observedOK && observed > interval { interval = observed } // observation may only WIDEN
window = clamp(4 * interval, floor 7d, cap 12d)
- 4 generations — the settle generation plus ~3 missed opportunities: deliberately the same tolerance the flat constant expressed. The change is to the rhythm, not to the patience.
- The declared rhythm is
backupStaleAfter(26 h) /offsiteBackupStaleAfter(8 d) — the very thresholds the backup-freshness checker already judges those tiers against. Not a second opinion: if the two checkers disagreed about how often a tier receives an archive, one would be alarming on the other's model. - The observation comes from reports the hub already holds:
pbs_snapshots[]plus successfulbackups[]attributed by target type (slice-A.4 — a PBS-targeted vzdump appears in both arrays, and classifying by array membership would attribute an offsite archive to the host tier). Mean gap over distinct archive times; fewer than two generations ⇒ unobservable. - Floor 7 d — the old constant kept as a floor, so no tier is judged more tightly than before.
- Cap 12 d — strictly inside the 2-week offsite retention with two days to spare, so a tier is never called stale against an archive PBS has already pruned.
| tier | interval used | window | proof age in health | verdict |
|---|---|---|---|---|
| daily host | 26 h declared | 4×26 h → floor 168 h | ≤ ~54 h | OK — numerically unchanged from before |
| weekly offsite | 8 d declared (7 d observed loses to it) | 4×8 d → cap 288 h | ≤ ~198 h | OK, 3.75 d of margin |
| newborn offsite | 8 d declared | 288 h | — | UNKNOWN until the anchor passes |
4. v0.91.1 — a flaw v0.91.0 shipped with, found by checking the live box
demo-felhom's offsite tier holds two retained snapshots, 2026-07-27T19:55:41Z and
2026-07-28T04:49:43Z — 8 h 54 m apart, because one is a healing artefact and the other a real
weekly run. A mean-gap estimate therefore reads a weekly tier as nine-hourly: ×4 = 36 h, the floor
lifts it to 168 h, and a weekly tier proved weekly reaches ~8.25 days of proof age. The false alarm
this whole task exists to prevent would have returned within a week, on the box it had just shipped
to.
Fixed by max(observed, declared). A gap shorter than the declared rhythm is routine and means
nothing (a retry, a manual run, a heal, a catch-up after an outage); a gap longer than it is real
information. Cost, stated rather than hidden: a tier that truly runs faster than its declared
rhythm gets a wider window than it needs, i.e. a slower restore_test_stale. Right direction for a
signal meaning unverified — broken now is restore_test_failed, immediate and untouched.
5. Kept, because it was earned
- Absence is UNKNOWN until an anchored window has passed (R-81's structure, unchanged).
- The stale signal stays edge-triggered (
staleStatesuntouched). restore_test_failedandrestore_test_stalestay DISTINCT — one says your recovery is broken, the other that it is unverified, and the second is the one that quietly becomes the first.- Every reason string now states the window it was judged against — R-100's corollary: when a verdict changes what it counts from, the alarm text must change with it, or an operator reads "limit 168h" under a tier judged at 288 h and dismisses a true alarm.
- The window READ is unchanged in cost (14 days) — enough to find proof inside the widest window and to see two generations of a weekly tier.
6. Tests and red-proofs
| Test | Asserts | Mutation | Observed |
|---|---|---|---|
TestRestoreTest_HealthyWeeklyTierIsNeverStale |
a healthy weekly tier with jitter is never stale across 6 weeks | window pinned flat at restoreProvenWindowFloor |
FAIL — week 0: … proof age 172h0m0s, window 168h0m0s … "limit 168h0m0s" |
TestRestoreProvenWindow_Contract |
floor, cap, declared fallback, observation may only widen | observed > interval → observed > 0 |
FAIL — window(pbs, observed=8h54m0s ok=true) = 168h0m0s, want 288h0m0s |
TestObservedArchiveIntervals_FromReports |
rhythm observed per tier; a PBS vzdump is not counted into the host tier | — | pass |
TestRestoreTest_WeeklyTierThatStopsBeingProvedStillAlarms |
a window that never fires is a deletion, not a fix | — | pass |
| existing R-85/R-81 suite | anchored UNKNOWN, edge-trigger, distinct events, newborn silence | — | pass, unchanged |
A hollow test caught by its own red-proof. The first Scenario-G fixture used a perfectly regular
weekly tier and PASSED under the flat-window mutation, because a regular tier's proof age lands on
exactly 168 h and > lets it through. It would have shipped Part 1 and its false alarm together while
looking like proof. The jitter is what makes it a test — and it is also the truth about the old
constant.
7. Deployment
| Step | Evidence |
|---|---|
| Images built + pushed | felhom-hub:0.91.0, then 0.91.1 |
| Manifest bumped in git | 687fedd, ff2655c — never kubectl set image |
| ArgoCD deliberate sync | sync=Synced health=Healthy |
| Running image | gitea.dooplex.hu/admin/felhom-hub:0.91.1 |
| Startup log | checkers initialized; Listening on :8080 |
8. Part 3 — the ep0 ruling, recorded
runbooks/target-selection.md had carried "D-d did not name ep0 either way. Confirm it explicitly"
for two days. Operator ruling, 2026-08-03: ep0 is protected. Recorded three ways — the tier table
names it, the standing question is gone, and its per-machine section is retitled Tier 2, PROTECTED.
CONTEXT.md S-18 records that this extends D-d's protected list to three machines (DooPlex,
Peti's cluster, ep0) and that it is a classification, not new prohibitions: destroying datastores,
prune jobs, tunnel config or nftables rules was already forbidden by what it would destroy, and the
ordinary off-site read a restore-test performs remains permitted — which this session then exercised.
9. Registers and docs
- R-86 → CLOSED, shipped + proven live, with the trap in its own wording recorded on the row.
- R-87 → re-ranked UP: R-86 built most of what it waited for. What remains is restic-specific.
- R-185 / R-186 / R-187 → filed (grep established 185–187 free; R-184 was the highest in use).
ROADMAP.mdR-86 collapsed, keeping the reasoning and correcting the shape the row itself proposed — which was the never-fires version.STATUS.mdrewritten for the operator, trimmed back to one screen.07-backup-architecture.md§3 carries the new contract (S-1);00-capability-map.md's restore-test row upgraded with its live citation;CONTEXT.mdgains S-17 (the rule + the trap + the config key) and S-18 (ep0).