admin
c102832892
gates / gates (push) Successful in 8s
- OPEN-ITEMS: R-86 CLOSED with the trap in its own wording recorded (the literal reading is never true on a daily tier); R-87 re-ranked UP because R-86 built most of what it waited for; R-185 (the agent cannot list demo-felhom's host backup tier — a missing storage ACL, pre-existing), R-186 (a released binary's sha is not reproducible from its tag), R-187 (R-115's publish leg had never actually run) filed. R-184 was the highest ID in use. - ROADMAP: R-86 collapsed, keeping the reasoning and correcting the shape the row itself proposed — which would have been the never-fires version. - 07-backup-architecture: new contract section — restore-testing is per ARCHIVE GENERATION, with the trap and what did not change (S-1). - 00-capability-map: the unattended restore-proof row upgraded to PROVEN-LIVE on the 635 s due-triggered offsite run, with the restart and teardown evidence. - CONTEXT: S-17 (the rule, the trap, the config key, the hub's derivation) and S-18 (ep0 is Tier 2 — extends D-d's protected list to three machines). Numbered 17/18 because S-14 and S-15 were already duplicated in the file. - STATUS: rewritten for the operator, trimmed back to one screen.
131 lines
8.5 KiB
Markdown
131 lines
8.5 KiB
Markdown
# REPORT — R-86 Part 2: the staleness window learns each tier's rhythm (2026-08-03)
|
||
|
||
**Date:** 2026-08-03 · **Repo:** `felhom.eu` hub **v0.90.1 → v0.91.0 → v0.91.1**
|
||
(`323f45a`, `046df30`), manifests `687fedd`, `ff2655c`. Ships **with** `felhom-agent` v0.121.0, not
|
||
after it. Ruling recorded: **ep0 is Tier 2, protected**.
|
||
|
||
## 1. Baselines, re-read on arrival
|
||
|
||
| Repo | `main` @ commit | Version | Matched §1? |
|
||
|---|---|---|---|
|
||
| `felhom.eu` | `e34b614e5b65` | hub CHANGELOG top entry `v0.90.0`; `manifests/hub.yaml` ran **`0.90.1`** | **Yes — and the discrepancy was real** |
|
||
|
||
The missing `v0.90.1` CHANGELOG entry is backfilled. It was built, deployed and recorded in
|
||
`REPORT.md` + the R-182 row on 2026-08-03 (`f21e7ca`) but never given a CHANGELOG entry — and
|
||
`REPORT.md` is overwritten every session, so the per-repo history under-reported what was running.
|
||
|
||
## 2. Why Part 2 could not ship later
|
||
|
||
The agent now proves a tier **once per archive generation**, so a tier backed up weekly is proved
|
||
weekly — correctly, in perfect health. `restoreProvenStaleAfter` was a flat 7 days and its own comment
|
||
derived that number *from the cadence R-86 removes*.
|
||
|
||
The brief said a weekly tier would "sit exactly on that line". That is literally true, and the
|
||
arithmetic is worth stating: proofs land at `archive + settle`, so just before the next proof the
|
||
current one is **exactly one interval old** — 168 h against a 168 h window, and `age > window` is false
|
||
by a hair. It did not sit near the line, it sat **on** it. Any ordinary delay — a backup landing late,
|
||
one evaluation deferred behind a running backup — tips a healthy tier into a nightly alarm.
|
||
|
||
## 3. The window as derived
|
||
|
||
```
|
||
interval = declaredArchiveInterval(tier) // host 26h, offsite 8d
|
||
if observedOK && observed > interval { interval = observed } // observation may only WIDEN
|
||
window = clamp(4 * interval, floor 7d, cap 12d)
|
||
```
|
||
|
||
- **4 generations** — the settle generation plus ~3 missed opportunities: deliberately the same
|
||
tolerance the flat constant expressed. The change is to the *rhythm*, not to the patience.
|
||
- **The declared rhythm is `backupStaleAfter` (26 h) / `offsiteBackupStaleAfter` (8 d)** — the very
|
||
thresholds the backup-freshness checker already judges those tiers against. Not a second opinion:
|
||
if the two checkers disagreed about how often a tier receives an archive, one would be alarming on
|
||
the other's model.
|
||
- **The observation** comes from reports the hub already holds: `pbs_snapshots[]` plus successful
|
||
`backups[]` attributed by **target type** (slice-A.4 — a PBS-targeted vzdump appears in *both*
|
||
arrays, and classifying by array membership would attribute an offsite archive to the host tier).
|
||
Mean gap over distinct archive times; fewer than two generations ⇒ unobservable.
|
||
- **Floor 7 d** — the old constant kept as a floor, so no tier is judged more tightly than before.
|
||
- **Cap 12 d** — strictly inside the 2-week offsite retention with two days to spare, so a tier is
|
||
never called stale against an archive PBS has already pruned.
|
||
|
||
| tier | interval used | window | proof age in health | verdict |
|
||
|---|---|---|---|---|
|
||
| daily host | 26 h declared | 4×26 h → **floor 168 h** | ≤ ~54 h | OK — numerically unchanged from before |
|
||
| weekly offsite | 8 d declared (7 d observed loses to it) | 4×8 d → **cap 288 h** | ≤ ~198 h | OK, 3.75 d of margin |
|
||
| newborn offsite | 8 d declared | **288 h** | — | UNKNOWN until the anchor passes |
|
||
|
||
## 4. v0.91.1 — a flaw v0.91.0 shipped with, found by checking the live box
|
||
|
||
demo-felhom's offsite tier holds two retained snapshots, `2026-07-27T19:55:41Z` and
|
||
`2026-07-28T04:49:43Z` — **8 h 54 m apart**, because one is a healing artefact and the other a real
|
||
weekly run. A mean-gap estimate therefore reads a **weekly** tier as nine-hourly: ×4 = 36 h, the floor
|
||
lifts it to 168 h, and a weekly tier proved weekly reaches ~8.25 days of proof age. **The false alarm
|
||
this whole task exists to prevent would have returned within a week, on the box it had just shipped
|
||
to.**
|
||
|
||
Fixed by `max(observed, declared)`. A gap *shorter* than the declared rhythm is routine and means
|
||
nothing (a retry, a manual run, a heal, a catch-up after an outage); a gap *longer* than it is real
|
||
information. **Cost, stated rather than hidden:** a tier that truly runs faster than its declared
|
||
rhythm gets a wider window than it needs, i.e. a slower `restore_test_stale`. Right direction for a
|
||
signal meaning *unverified* — *broken now* is `restore_test_failed`, immediate and untouched.
|
||
|
||
## 5. Kept, because it was earned
|
||
|
||
- **Absence is UNKNOWN** until an anchored window has passed (R-81's structure, unchanged).
|
||
- **The stale signal stays edge-triggered** (`staleStates` untouched).
|
||
- **`restore_test_failed` and `restore_test_stale` stay DISTINCT** — one says your recovery is broken,
|
||
the other that it is unverified, and the second is the one that quietly becomes the first.
|
||
- **Every reason string now states the window it was judged against** — R-100's corollary: when a
|
||
verdict changes what it counts from, the alarm text must change with it, or an operator reads
|
||
"limit 168h" under a tier judged at 288 h and dismisses a true alarm.
|
||
- **The window READ is unchanged in cost** (14 days) — enough to find proof inside the widest window
|
||
and to see two generations of a weekly tier.
|
||
|
||
## 6. Tests and red-proofs
|
||
|
||
| Test | Asserts | Mutation | Observed |
|
||
|---|---|---|---|
|
||
| `TestRestoreTest_HealthyWeeklyTierIsNeverStale` | a healthy weekly tier **with jitter** is never stale across 6 weeks | window pinned flat at `restoreProvenWindowFloor` | **FAIL** — `week 0: … proof age 172h0m0s, window 168h0m0s … "limit 168h0m0s"` |
|
||
| `TestRestoreProvenWindow_Contract` | floor, cap, declared fallback, **observation may only widen** | `observed > interval` → `observed > 0` | **FAIL** — `window(pbs, observed=8h54m0s ok=true) = 168h0m0s, want 288h0m0s` |
|
||
| `TestObservedArchiveIntervals_FromReports` | rhythm observed per tier; a PBS vzdump is **not** counted into the host tier | — | pass |
|
||
| `TestRestoreTest_WeeklyTierThatStopsBeingProvedStillAlarms` | a window that never fires is a deletion, not a fix | — | pass |
|
||
| existing R-85/R-81 suite | anchored UNKNOWN, edge-trigger, distinct events, newborn silence | — | pass, unchanged |
|
||
|
||
**A hollow test caught by its own red-proof.** The first Scenario-G fixture used a perfectly regular
|
||
weekly tier and **PASSED under the flat-window mutation**, because a regular tier's proof age lands on
|
||
exactly 168 h and `>` lets it through. It would have shipped Part 1 and its false alarm together while
|
||
looking like proof. The jitter is what makes it a test — and it is also the truth about the old
|
||
constant.
|
||
|
||
## 7. Deployment
|
||
|
||
| Step | Evidence |
|
||
|---|---|
|
||
| Images built + pushed | `felhom-hub:0.91.0`, then `0.91.1` |
|
||
| Manifest bumped in git | `687fedd`, `ff2655c` — never `kubectl set image` |
|
||
| ArgoCD deliberate sync | `sync=Synced health=Healthy` |
|
||
| Running image | `gitea.dooplex.hu/admin/felhom-hub:0.91.1` |
|
||
| Startup log | checkers initialized; `Listening on :8080` |
|
||
|
||
## 8. Part 3 — the ep0 ruling, recorded
|
||
|
||
`runbooks/target-selection.md` had carried *"D-d did not name ep0 either way. Confirm it explicitly"*
|
||
for two days. **Operator ruling, 2026-08-03: ep0 is protected.** Recorded three ways — the tier table
|
||
names it, the standing question is gone, and its per-machine section is retitled **Tier 2, PROTECTED**.
|
||
`CONTEXT.md` S-18 records that this **extends D-d's protected list to three machines** (DooPlex,
|
||
Peti's cluster, ep0) and that it is a *classification*, not new prohibitions: destroying datastores,
|
||
prune jobs, tunnel config or nftables rules was already forbidden by what it would destroy, and **the
|
||
ordinary off-site read a restore-test performs remains permitted** — which this session then exercised.
|
||
|
||
## 9. Registers and docs
|
||
|
||
- **R-86 → CLOSED**, shipped + proven live, with the trap in its own wording recorded on the row.
|
||
- **R-87 → re-ranked UP**: R-86 built most of what it waited for. What remains is restic-specific.
|
||
- **R-185 / R-186 / R-187 → filed** (grep established 185–187 free; R-184 was the highest in use).
|
||
- `ROADMAP.md` R-86 collapsed, keeping the reasoning and **correcting the shape the row itself
|
||
proposed** — which was the never-fires version.
|
||
- `STATUS.md` rewritten for the operator, trimmed back to one screen.
|
||
- `07-backup-architecture.md` §3 carries the new contract (S-1); `00-capability-map.md`'s restore-test
|
||
row upgraded with its live citation; `CONTEXT.md` gains S-17 (the rule + the trap + the config key)
|
||
and S-18 (ep0).
|