Files
felhom.eu/REPORT.md
T

138 lines
9.0 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# REPORT — R-86 Part 2: the staleness window learns each tier's rhythm (2026-08-03)
> **A LATER SESSION THE SAME DAY (R-189 · R-188 · R-186) touched this repo for REGISTERS AND DOCS
> ONLY** — `OPEN-ITEMS.md`, `STATUS.md`, `CONTEXT.md` (S-19, S-20) and the capability map. **No hub
> code changed and the hub was not bumped**: the defect was the agent no longer sending
> `restore_tests[]`, not the hub failing to read them. That session's report is
> `felhom-agent/REPORT.md`. What follows is the last hub IMPLEMENTATION, v0.91.1, which is what this
> file is for.
**Date:** 2026-08-03 · **Repo:** `felhom.eu` hub **v0.90.1 → v0.91.0 → v0.91.1**
(`323f45a`, `046df30`), manifests `687fedd`, `ff2655c`. Ships **with** `felhom-agent` v0.121.0, not
after it. Ruling recorded: **ep0 is Tier 2, protected**.
## 1. Baselines, re-read on arrival
| Repo | `main` @ commit | Version | Matched §1? |
|---|---|---|---|
| `felhom.eu` | `e34b614e5b65` | hub CHANGELOG top entry `v0.90.0`; `manifests/hub.yaml` ran **`0.90.1`** | **Yes — and the discrepancy was real** |
The missing `v0.90.1` CHANGELOG entry is backfilled. It was built, deployed and recorded in
`REPORT.md` + the R-182 row on 2026-08-03 (`f21e7ca`) but never given a CHANGELOG entry — and
`REPORT.md` is overwritten every session, so the per-repo history under-reported what was running.
## 2. Why Part 2 could not ship later
The agent now proves a tier **once per archive generation**, so a tier backed up weekly is proved
weekly — correctly, in perfect health. `restoreProvenStaleAfter` was a flat 7 days and its own comment
derived that number *from the cadence R-86 removes*.
The brief said a weekly tier would "sit exactly on that line". That is literally true, and the
arithmetic is worth stating: proofs land at `archive + settle`, so just before the next proof the
current one is **exactly one interval old** — 168 h against a 168 h window, and `age > window` is false
by a hair. It did not sit near the line, it sat **on** it. Any ordinary delay — a backup landing late,
one evaluation deferred behind a running backup — tips a healthy tier into a nightly alarm.
## 3. The window as derived
```
interval = declaredArchiveInterval(tier) // host 26h, offsite 8d
if observedOK && observed > interval { interval = observed } // observation may only WIDEN
window = clamp(4 * interval, floor 7d, cap 12d)
```
- **4 generations** — the settle generation plus ~3 missed opportunities: deliberately the same
tolerance the flat constant expressed. The change is to the *rhythm*, not to the patience.
- **The declared rhythm is `backupStaleAfter` (26 h) / `offsiteBackupStaleAfter` (8 d)** — the very
thresholds the backup-freshness checker already judges those tiers against. Not a second opinion:
if the two checkers disagreed about how often a tier receives an archive, one would be alarming on
the other's model.
- **The observation** comes from reports the hub already holds: `pbs_snapshots[]` plus successful
`backups[]` attributed by **target type** (slice-A.4 — a PBS-targeted vzdump appears in *both*
arrays, and classifying by array membership would attribute an offsite archive to the host tier).
Mean gap over distinct archive times; fewer than two generations ⇒ unobservable.
- **Floor 7 d** — the old constant kept as a floor, so no tier is judged more tightly than before.
- **Cap 12 d** — strictly inside the 2-week offsite retention with two days to spare, so a tier is
never called stale against an archive PBS has already pruned.
| tier | interval used | window | proof age in health | verdict |
|---|---|---|---|---|
| daily host | 26 h declared | 4×26 h → **floor 168 h** | ≤ ~54 h | OK — numerically unchanged from before |
| weekly offsite | 8 d declared (7 d observed loses to it) | 4×8 d → **cap 288 h** | ≤ ~198 h | OK, 3.75 d of margin |
| newborn offsite | 8 d declared | **288 h** | — | UNKNOWN until the anchor passes |
## 4. v0.91.1 — a flaw v0.91.0 shipped with, found by checking the live box
demo-felhom's offsite tier holds two retained snapshots, `2026-07-27T19:55:41Z` and
`2026-07-28T04:49:43Z`**8 h 54 m apart**, because one is a healing artefact and the other a real
weekly run. A mean-gap estimate therefore reads a **weekly** tier as nine-hourly: ×4 = 36 h, the floor
lifts it to 168 h, and a weekly tier proved weekly reaches ~8.25 days of proof age. **The false alarm
this whole task exists to prevent would have returned within a week, on the box it had just shipped
to.**
Fixed by `max(observed, declared)`. A gap *shorter* than the declared rhythm is routine and means
nothing (a retry, a manual run, a heal, a catch-up after an outage); a gap *longer* than it is real
information. **Cost, stated rather than hidden:** a tier that truly runs faster than its declared
rhythm gets a wider window than it needs, i.e. a slower `restore_test_stale`. Right direction for a
signal meaning *unverified**broken now* is `restore_test_failed`, immediate and untouched.
## 5. Kept, because it was earned
- **Absence is UNKNOWN** until an anchored window has passed (R-81's structure, unchanged).
- **The stale signal stays edge-triggered** (`staleStates` untouched).
- **`restore_test_failed` and `restore_test_stale` stay DISTINCT** — one says your recovery is broken,
the other that it is unverified, and the second is the one that quietly becomes the first.
- **Every reason string now states the window it was judged against** — R-100's corollary: when a
verdict changes what it counts from, the alarm text must change with it, or an operator reads
"limit 168h" under a tier judged at 288 h and dismisses a true alarm.
- **The window READ is unchanged in cost** (14 days) — enough to find proof inside the widest window
and to see two generations of a weekly tier.
## 6. Tests and red-proofs
| Test | Asserts | Mutation | Observed |
|---|---|---|---|
| `TestRestoreTest_HealthyWeeklyTierIsNeverStale` | a healthy weekly tier **with jitter** is never stale across 6 weeks | window pinned flat at `restoreProvenWindowFloor` | **FAIL**`week 0: … proof age 172h0m0s, window 168h0m0s … "limit 168h0m0s"` |
| `TestRestoreProvenWindow_Contract` | floor, cap, declared fallback, **observation may only widen** | `observed > interval``observed > 0` | **FAIL**`window(pbs, observed=8h54m0s ok=true) = 168h0m0s, want 288h0m0s` |
| `TestObservedArchiveIntervals_FromReports` | rhythm observed per tier; a PBS vzdump is **not** counted into the host tier | — | pass |
| `TestRestoreTest_WeeklyTierThatStopsBeingProvedStillAlarms` | a window that never fires is a deletion, not a fix | — | pass |
| existing R-85/R-81 suite | anchored UNKNOWN, edge-trigger, distinct events, newborn silence | — | pass, unchanged |
**A hollow test caught by its own red-proof.** The first Scenario-G fixture used a perfectly regular
weekly tier and **PASSED under the flat-window mutation**, because a regular tier's proof age lands on
exactly 168 h and `>` lets it through. It would have shipped Part 1 and its false alarm together while
looking like proof. The jitter is what makes it a test — and it is also the truth about the old
constant.
## 7. Deployment
| Step | Evidence |
|---|---|
| Images built + pushed | `felhom-hub:0.91.0`, then `0.91.1` |
| Manifest bumped in git | `687fedd`, `ff2655c` — never `kubectl set image` |
| ArgoCD deliberate sync | `sync=Synced health=Healthy` |
| Running image | `gitea.dooplex.hu/admin/felhom-hub:0.91.1` |
| Startup log | checkers initialized; `Listening on :8080` |
## 8. Part 3 — the ep0 ruling, recorded
`runbooks/target-selection.md` had carried *"D-d did not name ep0 either way. Confirm it explicitly"*
for two days. **Operator ruling, 2026-08-03: ep0 is protected.** Recorded three ways — the tier table
names it, the standing question is gone, and its per-machine section is retitled **Tier 2, PROTECTED**.
`CONTEXT.md` S-18 records that this **extends D-d's protected list to three machines** (DooPlex,
Peti's cluster, ep0) and that it is a *classification*, not new prohibitions: destroying datastores,
prune jobs, tunnel config or nftables rules was already forbidden by what it would destroy, and **the
ordinary off-site read a restore-test performs remains permitted** — which this session then exercised.
## 9. Registers and docs
- **R-86 → CLOSED**, shipped + proven live, with the trap in its own wording recorded on the row.
- **R-87 → re-ranked UP**: R-86 built most of what it waited for. What remains is restic-specific.
- **R-185 / R-186 / R-187 → filed** (grep established 185187 free; R-184 was the highest in use).
- `ROADMAP.md` R-86 collapsed, keeping the reasoning and **correcting the shape the row itself
proposed** — which was the never-fires version.
- `STATUS.md` rewritten for the operator, trimmed back to one screen.
- `07-backup-architecture.md` §3 carries the new contract (S-1); `00-capability-map.md`'s restore-test
row upgraded with its live citation; `CONTEXT.md` gains S-17 (the rule + the trap + the config key)
and S-18 (ep0).