hub v0.91.0 — the staleness window learns each tier's own rhythm (R-86 Part 2)
gates / gates (push) Successful in 7s

Ships WITH agent v0.121.0, not after it. The agent now proves a tier once per
ARCHIVE GENERATION, so a weekly tier is proved weekly — in perfect health. The
flat 7-day restoreProvenStaleAfter derived its number from the 24h cadence R-86
removes, and a healthy weekly tier's proof age reaches EXACTLY 168h just before
its next proof: it sat ON the line, so any ordinary delay tipped it into a
nightly alarm about a working system.

restoreProvenWindow(tier, observed, ok):
- the tier's own archive interval, OBSERVED from reports the hub already holds
  (pbs_snapshots + successful backups attributed by TARGET TYPE, slice A.4)
- x4 generations = the same tolerance the flat constant expressed
- floored at 7d (never tighter than before), capped at 12d (strictly inside the
  2-week offsite retention)
- falls back to the DECLARED rhythm (26h host / 8d offsite — the thresholds the
  backup-freshness checker already uses) when history is too short to observe
  one; falling back to the FLOOR would recreate the false alarm on a fresh box

Kept: absence is UNKNOWN until the anchored window passes; the signal stays
edge-triggered; failed and stale remain distinct events. Every reason string now
states the window it was judged against (R-100's corollary).

Also backfills the missing v0.90.1 CHANGELOG entry (deployed since f21e7ca), and
records the operator's 2026-08-03 ruling that ep0 is Tier 2 / protected.
This commit is contained in:
2026-08-03 15:03:35 +02:00
parent e34b614e5b
commit 323f45a5ef
4 changed files with 478 additions and 41 deletions
+18 -12
View File
@@ -29,20 +29,23 @@ prohibition as covering the act it names and nothing more.
|---|---|---|
| **0 — disposable. Reach here first.** | Exists to be broken; reinstalling is a routine afternoon, not an incident. **A drill that needs a victim uses one of these.** | `demo-hp` (t740), `demo-felhom` (N100) |
| **1 — create and destroy freely** | Throwaway VMs, guests, scratch customers — **hosted on a Tier 0 machine** | drill VMs, scratch guests |
| **2 — protected. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster** — and, by D-d, **nothing else** |
| **2 — protected. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster**, **`ep0`** (operator ruling 2026-08-03) — and nothing else |
**DooPlex is Tier 2 because it *is* the recovery chain** — hub, Gitea, registry, PBS, k3s + Longhorn.
Everything else rebuilds from it; it rebuilds from nothing. A bad moment in a DR drill there costs the
thing under test, the source of truth for it, and the backups, at once.
**`ep0` + the Hetzner Storage Boxes were Tier 2 until 2026-08-02 and are no longer** — D-d's protected
list names two machines and ep0 is not one of them. **That does not make them scratch, and the
difference is an act, not a tier** (see the rule above the table): ep0 holds the **PBS-DR datastore and
the restic copy of a real customer's data**, which is the only off-premises copy that exists, so
*deleting datastores, prune jobs, tunnel config or nftables rules* remains forbidden by what it would
destroy rather than by what tier it sits in. Reads are fine; it is still never a drill target.
**Flagged for the operator: D-d did not name ep0 either way.** Confirm it explicitly — this page has
read it the narrow way (not protected, but not wipeable) rather than assume the broad one.
**`ep0` is Tier 2 — PROTECTED. Operator ruling, 2026-08-03.** D-d named two protected machines and did
not name ep0 either way, so this page carried the question in writing for two days and read it the
narrow way meanwhile (not protected, but not wipeable). The ruling settles it and **extends D-d's
protected list to three machines**: DooPlex, Peti's cluster, ep0.
The reason it was never really in doubt: ep0 holds the **PBS-DR datastore and the restic copy of a
real customer's data**, which is the only off-premises copy that exists. So *deleting datastores,
prune jobs, tunnel config or nftables rules* was already forbidden by what it would destroy; the
ruling makes the classification say so plainly instead of leaving each session to re-derive it.
**Reads are fine** — including the ordinary off-site read a restore-test performs (R-86) — and it is
never a drill target. The Hetzner Storage Boxes ride the same reasoning.
**Standing ruling, 2026-07-25 (`operations/nodes.md`):** drill and build VMs live on the **t740** — not
felhom-pve, and **moved off DooPlex**. This page exists because that ruling sat where no session reads.
@@ -95,12 +98,15 @@ still shares a device with its guest, so a drive failure is **offsite-only recov
migrated, parked until the tester reinstalls (`PETI` in `backlog/OPEN-ITEMS.md`). Currently DOWN, no
enrolled host. No access route from DooPlex, and nothing here needs one.
### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **not protected by D-d; not scratch either**
### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **Tier 2, PROTECTED** (operator ruling 2026-08-03)
Reads are fine. It is the **offsite of last resort** (PBS-DR datastore, WireGuard hub, operator OOB
path) and — until 2026-08-03 — RAM-constrained (3.8 GB, R-90); it is now a **CX33 with 8 GB RAM and a
4 GiB swapfile**, which is what closed R-90. A very large restore is still worth watching. Do not delete datastores, prune
jobs, tunnel config or nftables rules; never a drill target. The Storage Boxes hold the restic copy —
4 GiB swapfile**, which is what closed R-90. A very large restore is still worth watching — the 8 GB
is comfortable, not unbounded, and the OOM that started R-90 was a 14.46 GB restore read against
3.8 GB. Do not delete datastores, prune jobs, tunnel config or nftables rules; never a drill target.
**The ordinary off-site READ a restore-test performs is permitted and unchanged by the ruling**
(R-86): the classification forbids destruction, not use. The Storage Boxes hold the restic copy —
customer documents and photos, on a credential that can still delete (R-95).
**Access: `ssh root@167.233.158.164` from DooPlex***not* `felhom-pve → 10.77.0.1`, the route that
produced a false "unreachable" verdict (standing rule 2).