admin
323f45a5ef
gates / gates (push) Successful in 7s
Ships WITH agent v0.121.0, not after it. The agent now proves a tier once per
ARCHIVE GENERATION, so a weekly tier is proved weekly — in perfect health. The
flat 7-day restoreProvenStaleAfter derived its number from the 24h cadence R-86
removes, and a healthy weekly tier's proof age reaches EXACTLY 168h just before
its next proof: it sat ON the line, so any ordinary delay tipped it into a
nightly alarm about a working system.
restoreProvenWindow(tier, observed, ok):
- the tier's own archive interval, OBSERVED from reports the hub already holds
(pbs_snapshots + successful backups attributed by TARGET TYPE, slice A.4)
- x4 generations = the same tolerance the flat constant expressed
- floored at 7d (never tighter than before), capped at 12d (strictly inside the
2-week offsite retention)
- falls back to the DECLARED rhythm (26h host / 8d offsite — the thresholds the
backup-freshness checker already uses) when history is too short to observe
one; falling back to the FLOOR would recreate the false alarm on a fresh box
Kept: absence is UNKNOWN until the anchored window passes; the signal stays
edge-triggered; failed and stale remain distinct events. Every reason string now
states the window it was judged against (R-100's corollary).
Also backfills the missing v0.90.1 CHANGELOG entry (deployed since f21e7ca), and
records the operator's 2026-08-03 ruling that ep0 is Tier 2 / protected.
159 lines
11 KiB
Markdown
159 lines
11 KiB
Markdown
# Which box do I break? — target selection by blast radius
|
|
|
|
> Read before picking a machine for a drill, a destructive test, or a throwaway VM. It answers one
|
|
> question: *what is safe to lose.* Reachability is in `CLAUDE.md`; hardware in `operations/nodes.md`.
|
|
> Added 2026-07-30. **Revised 2026-08-02 by operator decision D-d** (`CONTEXT.md` S-5).
|
|
|
|
## The rule
|
|
|
|
> **Two machines are protected: `DooPlex` and Peti's box. Everything else is disposable.**
|
|
> Operator decision **D-d**, 2026-08-02. DooPlex because it holds Gitea, the hub, the backups and the
|
|
> registry — everything else rebuilds from it, and it rebuilds from nothing. Peti's box because there
|
|
> is a real person behind it. **Every other box, both demo boxes included, may be broken or
|
|
> reinstalled freely.**
|
|
>
|
|
> **This is a correction, not a relaxation.** The earlier posture was costing whole sessions to
|
|
> caution and pushing drills onto DooPlex — the one machine that should never host them. If you are
|
|
> weighing whether a demo box can take a destructive test: it can. **Reach for a Tier 0 box first and
|
|
> do not ask.**
|
|
>
|
|
> Still true: **start at Tier 0 and work down only if Tier 0 genuinely cannot host the work**, and a
|
|
> protected machine is used only when a task says so explicitly — never by inference from what was
|
|
> not forbidden. An absent fence is not permission. If no tier fits, **stop and ask.**
|
|
|
|
Fences name **acts**, not machines. "Do not destroy demo-hp's `drill-r50` fixture" and "do not use
|
|
demo-hp to host a throwaway VM" are unrelated; only the first has ever been meant. Read a per-machine
|
|
prohibition as covering the act it names and nothing more.
|
|
|
|
| Tier | Meaning | Machines |
|
|
|---|---|---|
|
|
| **0 — disposable. Reach here first.** | Exists to be broken; reinstalling is a routine afternoon, not an incident. **A drill that needs a victim uses one of these.** | `demo-hp` (t740), `demo-felhom` (N100) |
|
|
| **1 — create and destroy freely** | Throwaway VMs, guests, scratch customers — **hosted on a Tier 0 machine** | drill VMs, scratch guests |
|
|
| **2 — protected. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster**, **`ep0`** (operator ruling 2026-08-03) — and nothing else |
|
|
|
|
**DooPlex is Tier 2 because it *is* the recovery chain** — hub, Gitea, registry, PBS, k3s + Longhorn.
|
|
Everything else rebuilds from it; it rebuilds from nothing. A bad moment in a DR drill there costs the
|
|
thing under test, the source of truth for it, and the backups, at once.
|
|
|
|
**`ep0` is Tier 2 — PROTECTED. Operator ruling, 2026-08-03.** D-d named two protected machines and did
|
|
not name ep0 either way, so this page carried the question in writing for two days and read it the
|
|
narrow way meanwhile (not protected, but not wipeable). The ruling settles it and **extends D-d's
|
|
protected list to three machines**: DooPlex, Peti's cluster, ep0.
|
|
|
|
The reason it was never really in doubt: ep0 holds the **PBS-DR datastore and the restic copy of a
|
|
real customer's data**, which is the only off-premises copy that exists. So *deleting datastores,
|
|
prune jobs, tunnel config or nftables rules* was already forbidden by what it would destroy; the
|
|
ruling makes the classification say so plainly instead of leaving each session to re-derive it.
|
|
**Reads are fine** — including the ordinary off-site read a restore-test performs (R-86) — and it is
|
|
never a drill target. The Hetzner Storage Boxes ride the same reasoning.
|
|
|
|
**Standing ruling, 2026-07-25 (`operations/nodes.md`):** drill and build VMs live on the **t740** — not
|
|
felhom-pve, and **moved off DooPlex**. This page exists because that ruling sat where no session reads.
|
|
|
|
## Per machine — permitted / needs care / forbidden
|
|
|
|
### `demo-hp` — HP t740 · **Tier 0 · the designated drill + build VM host**
|
|
|
|
- **Freely:** host nested drill VMs; create/destroy guests and scratch customers; reinstall the box.
|
|
**This is the default answer to "where do I run this".**
|
|
- **Care:** `local-lvm` is a **thin pool, over-subscribed** (~144 GiB allocated over ~54 GiB) backing
|
|
live guest 9201 — filling it corrupts every guest. Put VM disks on a dir storage at
|
|
**`/mnt/nvme-1tb`, at its root** (a subdirectory fails the agent's `exactMount` check → storage reads
|
|
`disconnected` forever). **That warning is about one storage, not the box.** `/mnt/nvme-1tb` is also
|
|
the `felhom-backup` target and the enrolled user-data drive, so remove scratch storages when done.
|
|
- **Forbidden:** do not destroy or unblock **`drill-r50` (VM 300)** — the only drift fixture (R-93).
|
|
(Access: the docs say no baked SSH key and G1 break-glass, but a key authenticated on 2026-07-31 —
|
|
**R-129**, unresolved.)
|
|
|
|
### `demo-felhom` — N100 · **Tier 0**
|
|
|
|
- **Freely:** create/destroy guests and scratch customers; reinstall the box.
|
|
- **Care:** it carries the **PBS-DR / offsite tier** (demo-hp has none), so it is the Tier 0 box whose
|
|
backup chain a test can actually disturb. Prefer demo-hp, per the 2026-07-25 ruling.
|
|
|
|
**Both Tier 0 boxes — the shared backup-target fence is DOWNGRADED to a cost, 2026-08-02 (D-d).** It
|
|
read *"do not re-point either backup target"*, because these are the only two **correctly configured**
|
|
boxes and therefore the regression path new installer logic is measured against. D-d makes both boxes
|
|
freely breakable and reinstallable, which loses that reference just as thoroughly — so the fence was
|
|
inconsistent with the decision and is not kept as a prohibition. **What survives is the reason:**
|
|
re-pointing (or reinstalling) costs the reference configuration, so know that you are spending it and
|
|
put the box back. If both are spent at once there is no correctly-configured box left to compare
|
|
against.
|
|
|
|
### `DooPlex` — 192.168.0.180 · **Tier 2**
|
|
|
|
- **Freely:** build, push images, `sudo kubectl`, read anything. Work inside `/mnt/5_hdd/felhom.eu/`.
|
|
- **Care:** the golden-bake nested VM `drill/drill.qcow2` lives here and is an accepted exception **for
|
|
bakes** (last used 0.185.1, 2026-07-29). Using it as a *drill victim* is not covered by that exception
|
|
and contradicts the 2026-07-25 ruling. If touched, restore `qemu-img snapshot -a virgin`.
|
|
- **Forbidden:** no global Docker cleanup; do not touch k3s data dirs, Longhorn mounts, PBS datastores
|
|
or Gitea storage; no destructive disk/guest ops; never run CC here with prompts disabled; abort large
|
|
builds if `df -h /mnt/5_hdd /` shows either over 90 %. **Because it is the recovery chain and a live
|
|
k3s node.**
|
|
|
|
### `Peti's cluster` (`peti-felhom`) — **Tier 2**
|
|
|
|
**Do not touch, at all.** A real external pilot with a real person behind it; its whole-guest backup
|
|
still shares a device with its guest, so a drive failure is **offsite-only recovery**. Deliberately not
|
|
migrated, parked until the tester reinstalls (`PETI` in `backlog/OPEN-ITEMS.md`). Currently DOWN, no
|
|
enrolled host. No access route from DooPlex, and nothing here needs one.
|
|
|
|
### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **Tier 2, PROTECTED** (operator ruling 2026-08-03)
|
|
|
|
Reads are fine. It is the **offsite of last resort** (PBS-DR datastore, WireGuard hub, operator OOB
|
|
path) and — until 2026-08-03 — RAM-constrained (3.8 GB, R-90); it is now a **CX33 with 8 GB RAM and a
|
|
4 GiB swapfile**, which is what closed R-90. A very large restore is still worth watching — the 8 GB
|
|
is comfortable, not unbounded, and the OOM that started R-90 was a 14.46 GB restore read against
|
|
3.8 GB. Do not delete datastores, prune jobs, tunnel config or nftables rules; never a drill target.
|
|
**The ordinary off-site READ a restore-test performs is permitted and unchanged by the ruling**
|
|
(R-86): the classification forbids destruction, not use. The Storage Boxes hold the restic copy —
|
|
customer documents and photos, on a credential that can still delete (R-95).
|
|
**Access: `ssh root@167.233.158.164` from DooPlex** — *not* `felhom-pve → 10.77.0.1`, the route that
|
|
produced a false "unreachable" verdict (standing rule 2).
|
|
|
|
### Tier 1 — drill VMs, scratch guests, scratch customers
|
|
|
|
Create and destroy freely **on a Tier 0 host**. Two exceptions: **`drill-r50`** is a fixture, not
|
|
scratch; and scratch **customers** outlive their VMs in the hub — delete those too, or they accumulate
|
|
(`sess-c` and `sess-d` were both left behind before anyone noticed).
|
|
|
|
#### A fixture may prove a mechanism. Only a fresh box may prove a path.
|
|
|
|
Rebuilding from scratch every time is waste; reusing a box is legitimate — but not for every claim.
|
|
|
|
- **Reusable fixture** — a snapshot-reset VM on a Tier 0 host. Use it for **mechanism** work: payload
|
|
capture, fix cycles, anything whose claim is about *code behaviour*. Reset to `virgin` between runs.
|
|
Fast, repeatable, and the right default for iteration.
|
|
- **Fresh day-0 from the ISO** — required for any claim about the **install path, the golden image,
|
|
agent publish/vouch, or first-boot state**. Slow, and the only thing that catches the drift family:
|
|
**R-111** (the golden's agent 17 releases behind), **R-115** (an agent built and deployed but never
|
|
published), **R-120** (the golden a controller release behind).
|
|
|
|
**A fixture must record its provenance** — which golden, agent and controller it was built from, and
|
|
when — alongside the VM. A fixture whose versions drift silently is R-120's mechanism turned into a
|
|
permanent installation, and it is *worse* than no fixture, because it produces confident wrong results
|
|
quickly.
|
|
|
|
**Worked example.** R-116's closing run deliberately did a real day-0 from the v1.25.0 ISO on demo-hp
|
|
instead of reusing the standing fixture. **That is how R-120 surfaced** — the fresh box installed the
|
|
golden's controller, which is a release behind, and showed the customer the wrong absent-target message.
|
|
The fixture would have shown a controller nobody installs.
|
|
|
|
## Not established — unknown, not guessed
|
|
|
|
- **Hetzner Storage Boxes** (`storage-box-pool-1`, `PBS-storage-1` / u629193 / box 611421) — state taken
|
|
from `backlog/OPEN-ITEMS.md`; **no direct access attempted or confirmed.** Tier 2 by what they hold.
|
|
- **`felhotest`** (legacy, `ssh -p 33022 kisfenyo@router.abonet.hu`) — **`Connection refused`**
|
|
2026-07-30; that was the only route tried. Untiered; assume nothing.
|
|
- **Peti's cluster** hardware/storage layout — unverified. Tier 2 rests on the relationship, which
|
|
needs no verification.
|
|
|
|
## The gap this page closes
|
|
|
|
During the R-116 diagnosis session (2026-07-30) a nested-Proxmox drill ran **on DooPlex**, a Tier 2
|
|
machine. Not a newly provisioned VM — the standing golden-bake fixture, booted from `virgin` and
|
|
restored to it, so nothing was lost. But it was chosen because DooPlex was the only machine no spec had
|
|
fenced, while the ruling naming the t740 sat in `operations/nodes.md` with nothing pointing at it.
|
|
**The designation existed and was unreachable** — the same class as an absent signal read as a positive
|
|
one.
|