Files
felhom.eu/REPORT.md
T
admin c102832892
gates / gates (push) Successful in 8s
docs: R-86 closed and proven live; ep0 recorded as protected; R-185/186/187 filed
- OPEN-ITEMS: R-86 CLOSED with the trap in its own wording recorded (the literal
  reading is never true on a daily tier); R-87 re-ranked UP because R-86 built
  most of what it waited for; R-185 (the agent cannot list demo-felhom's host
  backup tier — a missing storage ACL, pre-existing), R-186 (a released binary's
  sha is not reproducible from its tag), R-187 (R-115's publish leg had never
  actually run) filed. R-184 was the highest ID in use.
- ROADMAP: R-86 collapsed, keeping the reasoning and correcting the shape the row
  itself proposed — which would have been the never-fires version.
- 07-backup-architecture: new contract section — restore-testing is per ARCHIVE
  GENERATION, with the trap and what did not change (S-1).
- 00-capability-map: the unattended restore-proof row upgraded to PROVEN-LIVE on
  the 635 s due-triggered offsite run, with the restart and teardown evidence.
- CONTEXT: S-17 (the rule, the trap, the config key, the hub's derivation) and
  S-18 (ep0 is Tier 2 — extends D-d's protected list to three machines).
  Numbered 17/18 because S-14 and S-15 were already duplicated in the file.
- STATUS: rewritten for the operator, trimmed back to one screen.
2026-08-03 15:28:32 +02:00

8.5 KiB
Raw Blame History

REPORT — R-86 Part 2: the staleness window learns each tier's rhythm (2026-08-03)

Date: 2026-08-03 · Repo: felhom.eu hub v0.90.1 → v0.91.0 → v0.91.1 (323f45a, 046df30), manifests 687fedd, ff2655c. Ships with felhom-agent v0.121.0, not after it. Ruling recorded: ep0 is Tier 2, protected.

1. Baselines, re-read on arrival

Repo main @ commit Version Matched §1?
felhom.eu e34b614e5b65 hub CHANGELOG top entry v0.90.0; manifests/hub.yaml ran 0.90.1 Yes — and the discrepancy was real

The missing v0.90.1 CHANGELOG entry is backfilled. It was built, deployed and recorded in REPORT.md + the R-182 row on 2026-08-03 (f21e7ca) but never given a CHANGELOG entry — and REPORT.md is overwritten every session, so the per-repo history under-reported what was running.

2. Why Part 2 could not ship later

The agent now proves a tier once per archive generation, so a tier backed up weekly is proved weekly — correctly, in perfect health. restoreProvenStaleAfter was a flat 7 days and its own comment derived that number from the cadence R-86 removes.

The brief said a weekly tier would "sit exactly on that line". That is literally true, and the arithmetic is worth stating: proofs land at archive + settle, so just before the next proof the current one is exactly one interval old — 168 h against a 168 h window, and age > window is false by a hair. It did not sit near the line, it sat on it. Any ordinary delay — a backup landing late, one evaluation deferred behind a running backup — tips a healthy tier into a nightly alarm.

3. The window as derived

interval = declaredArchiveInterval(tier)                        // host 26h, offsite 8d
if observedOK && observed > interval { interval = observed }    // observation may only WIDEN
window   = clamp(4 * interval, floor 7d, cap 12d)
  • 4 generations — the settle generation plus ~3 missed opportunities: deliberately the same tolerance the flat constant expressed. The change is to the rhythm, not to the patience.
  • The declared rhythm is backupStaleAfter (26 h) / offsiteBackupStaleAfter (8 d) — the very thresholds the backup-freshness checker already judges those tiers against. Not a second opinion: if the two checkers disagreed about how often a tier receives an archive, one would be alarming on the other's model.
  • The observation comes from reports the hub already holds: pbs_snapshots[] plus successful backups[] attributed by target type (slice-A.4 — a PBS-targeted vzdump appears in both arrays, and classifying by array membership would attribute an offsite archive to the host tier). Mean gap over distinct archive times; fewer than two generations ⇒ unobservable.
  • Floor 7 d — the old constant kept as a floor, so no tier is judged more tightly than before.
  • Cap 12 d — strictly inside the 2-week offsite retention with two days to spare, so a tier is never called stale against an archive PBS has already pruned.
tier interval used window proof age in health verdict
daily host 26 h declared 4×26 h → floor 168 h ≤ ~54 h OK — numerically unchanged from before
weekly offsite 8 d declared (7 d observed loses to it) 4×8 d → cap 288 h ≤ ~198 h OK, 3.75 d of margin
newborn offsite 8 d declared 288 h UNKNOWN until the anchor passes

4. v0.91.1 — a flaw v0.91.0 shipped with, found by checking the live box

demo-felhom's offsite tier holds two retained snapshots, 2026-07-27T19:55:41Z and 2026-07-28T04:49:43Z8 h 54 m apart, because one is a healing artefact and the other a real weekly run. A mean-gap estimate therefore reads a weekly tier as nine-hourly: ×4 = 36 h, the floor lifts it to 168 h, and a weekly tier proved weekly reaches ~8.25 days of proof age. The false alarm this whole task exists to prevent would have returned within a week, on the box it had just shipped to.

Fixed by max(observed, declared). A gap shorter than the declared rhythm is routine and means nothing (a retry, a manual run, a heal, a catch-up after an outage); a gap longer than it is real information. Cost, stated rather than hidden: a tier that truly runs faster than its declared rhythm gets a wider window than it needs, i.e. a slower restore_test_stale. Right direction for a signal meaning unverifiedbroken now is restore_test_failed, immediate and untouched.

5. Kept, because it was earned

  • Absence is UNKNOWN until an anchored window has passed (R-81's structure, unchanged).
  • The stale signal stays edge-triggered (staleStates untouched).
  • restore_test_failed and restore_test_stale stay DISTINCT — one says your recovery is broken, the other that it is unverified, and the second is the one that quietly becomes the first.
  • Every reason string now states the window it was judged against — R-100's corollary: when a verdict changes what it counts from, the alarm text must change with it, or an operator reads "limit 168h" under a tier judged at 288 h and dismisses a true alarm.
  • The window READ is unchanged in cost (14 days) — enough to find proof inside the widest window and to see two generations of a weekly tier.

6. Tests and red-proofs

Test Asserts Mutation Observed
TestRestoreTest_HealthyWeeklyTierIsNeverStale a healthy weekly tier with jitter is never stale across 6 weeks window pinned flat at restoreProvenWindowFloor FAILweek 0: … proof age 172h0m0s, window 168h0m0s … "limit 168h0m0s"
TestRestoreProvenWindow_Contract floor, cap, declared fallback, observation may only widen observed > intervalobserved > 0 FAILwindow(pbs, observed=8h54m0s ok=true) = 168h0m0s, want 288h0m0s
TestObservedArchiveIntervals_FromReports rhythm observed per tier; a PBS vzdump is not counted into the host tier pass
TestRestoreTest_WeeklyTierThatStopsBeingProvedStillAlarms a window that never fires is a deletion, not a fix pass
existing R-85/R-81 suite anchored UNKNOWN, edge-trigger, distinct events, newborn silence pass, unchanged

A hollow test caught by its own red-proof. The first Scenario-G fixture used a perfectly regular weekly tier and PASSED under the flat-window mutation, because a regular tier's proof age lands on exactly 168 h and > lets it through. It would have shipped Part 1 and its false alarm together while looking like proof. The jitter is what makes it a test — and it is also the truth about the old constant.

7. Deployment

Step Evidence
Images built + pushed felhom-hub:0.91.0, then 0.91.1
Manifest bumped in git 687fedd, ff2655c — never kubectl set image
ArgoCD deliberate sync sync=Synced health=Healthy
Running image gitea.dooplex.hu/admin/felhom-hub:0.91.1
Startup log checkers initialized; Listening on :8080

8. Part 3 — the ep0 ruling, recorded

runbooks/target-selection.md had carried "D-d did not name ep0 either way. Confirm it explicitly" for two days. Operator ruling, 2026-08-03: ep0 is protected. Recorded three ways — the tier table names it, the standing question is gone, and its per-machine section is retitled Tier 2, PROTECTED. CONTEXT.md S-18 records that this extends D-d's protected list to three machines (DooPlex, Peti's cluster, ep0) and that it is a classification, not new prohibitions: destroying datastores, prune jobs, tunnel config or nftables rules was already forbidden by what it would destroy, and the ordinary off-site read a restore-test performs remains permitted — which this session then exercised.

9. Registers and docs

  • R-86 → CLOSED, shipped + proven live, with the trap in its own wording recorded on the row.
  • R-87 → re-ranked UP: R-86 built most of what it waited for. What remains is restic-specific.
  • R-185 / R-186 / R-187 → filed (grep established 185187 free; R-184 was the highest in use).
  • ROADMAP.md R-86 collapsed, keeping the reasoning and correcting the shape the row itself proposed — which was the never-fires version.
  • STATUS.md rewritten for the operator, trimmed back to one screen.
  • 07-backup-architecture.md §3 carries the new contract (S-1); 00-capability-map.md's restore-test row upgraded with its live citation; CONTEXT.md gains S-17 (the rule + the trap + the config key) and S-18 (ep0).