admin
e34b614e5b
gates / gates (push) Successful in 7s
R-182 CLOSED (controller v0.194.0 + hub v0.90.0/.1), proven live on demo-hp. The hub's notification_log for the run reads: two per-app failures RECORDED, one digest SENT naming both, and the customer channel SKIPPED with operator_only. Against the measured previous behaviour — two failures, one email naming one app, one leaving no trace anywhere. Scenario D proved itself on an event I had not planned: disk_critical alarmed on two filesystems, the second was collapsed by the cooldown, and that collapse is now visible WITH ITS KEY. Yesterday it would have left nothing at all. A gap the spec did not anticipate is recorded with its fix: the per-app event also fires from the periodic sweep, outside any run, so making it record-only would have created a NEW silence. The sweep emits a digest too, with no run_id, so it stays under the ordinary hourly cooldown. ep0: MEASURED on the box — 7757 MB (8 GB), 4 vCPU, and the 4 GiB swapfile SURVIVED the resize and is active (checked, because a resize is a stop/start). The 40 GB local disk is UNCHANGED, so no disk figure was touched anywhere. Five documents corrected — three of which the task's list did not name, found by searching. Two audit/evidence documents ANNOTATED, body untouched: they record what was true when written and that is their value. R-90 CLOSED. R-86 unblocked and re-ranked, stated honestly: 8 GB is comfortable, not unbounded — the original OOM was a 14.46 GB restore — so the restore-test cadence should still be paced, just not by fear of the endpoint. target-selection.md's "D-d did not name ep0 either way" is deliberately left standing. It is the operator's question, not CC's. STATUS.md 127 -> 83 lines, items rather than sentences.
153 lines
10 KiB
Markdown
153 lines
10 KiB
Markdown
# Which box do I break? — target selection by blast radius
|
|
|
|
> Read before picking a machine for a drill, a destructive test, or a throwaway VM. It answers one
|
|
> question: *what is safe to lose.* Reachability is in `CLAUDE.md`; hardware in `operations/nodes.md`.
|
|
> Added 2026-07-30. **Revised 2026-08-02 by operator decision D-d** (`CONTEXT.md` S-5).
|
|
|
|
## The rule
|
|
|
|
> **Two machines are protected: `DooPlex` and Peti's box. Everything else is disposable.**
|
|
> Operator decision **D-d**, 2026-08-02. DooPlex because it holds Gitea, the hub, the backups and the
|
|
> registry — everything else rebuilds from it, and it rebuilds from nothing. Peti's box because there
|
|
> is a real person behind it. **Every other box, both demo boxes included, may be broken or
|
|
> reinstalled freely.**
|
|
>
|
|
> **This is a correction, not a relaxation.** The earlier posture was costing whole sessions to
|
|
> caution and pushing drills onto DooPlex — the one machine that should never host them. If you are
|
|
> weighing whether a demo box can take a destructive test: it can. **Reach for a Tier 0 box first and
|
|
> do not ask.**
|
|
>
|
|
> Still true: **start at Tier 0 and work down only if Tier 0 genuinely cannot host the work**, and a
|
|
> protected machine is used only when a task says so explicitly — never by inference from what was
|
|
> not forbidden. An absent fence is not permission. If no tier fits, **stop and ask.**
|
|
|
|
Fences name **acts**, not machines. "Do not destroy demo-hp's `drill-r50` fixture" and "do not use
|
|
demo-hp to host a throwaway VM" are unrelated; only the first has ever been meant. Read a per-machine
|
|
prohibition as covering the act it names and nothing more.
|
|
|
|
| Tier | Meaning | Machines |
|
|
|---|---|---|
|
|
| **0 — disposable. Reach here first.** | Exists to be broken; reinstalling is a routine afternoon, not an incident. **A drill that needs a victim uses one of these.** | `demo-hp` (t740), `demo-felhom` (N100) |
|
|
| **1 — create and destroy freely** | Throwaway VMs, guests, scratch customers — **hosted on a Tier 0 machine** | drill VMs, scratch guests |
|
|
| **2 — protected. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster** — and, by D-d, **nothing else** |
|
|
|
|
**DooPlex is Tier 2 because it *is* the recovery chain** — hub, Gitea, registry, PBS, k3s + Longhorn.
|
|
Everything else rebuilds from it; it rebuilds from nothing. A bad moment in a DR drill there costs the
|
|
thing under test, the source of truth for it, and the backups, at once.
|
|
|
|
**`ep0` + the Hetzner Storage Boxes were Tier 2 until 2026-08-02 and are no longer** — D-d's protected
|
|
list names two machines and ep0 is not one of them. **That does not make them scratch, and the
|
|
difference is an act, not a tier** (see the rule above the table): ep0 holds the **PBS-DR datastore and
|
|
the restic copy of a real customer's data**, which is the only off-premises copy that exists, so
|
|
*deleting datastores, prune jobs, tunnel config or nftables rules* remains forbidden by what it would
|
|
destroy rather than by what tier it sits in. Reads are fine; it is still never a drill target.
|
|
**Flagged for the operator: D-d did not name ep0 either way.** Confirm it explicitly — this page has
|
|
read it the narrow way (not protected, but not wipeable) rather than assume the broad one.
|
|
|
|
**Standing ruling, 2026-07-25 (`operations/nodes.md`):** drill and build VMs live on the **t740** — not
|
|
felhom-pve, and **moved off DooPlex**. This page exists because that ruling sat where no session reads.
|
|
|
|
## Per machine — permitted / needs care / forbidden
|
|
|
|
### `demo-hp` — HP t740 · **Tier 0 · the designated drill + build VM host**
|
|
|
|
- **Freely:** host nested drill VMs; create/destroy guests and scratch customers; reinstall the box.
|
|
**This is the default answer to "where do I run this".**
|
|
- **Care:** `local-lvm` is a **thin pool, over-subscribed** (~144 GiB allocated over ~54 GiB) backing
|
|
live guest 9201 — filling it corrupts every guest. Put VM disks on a dir storage at
|
|
**`/mnt/nvme-1tb`, at its root** (a subdirectory fails the agent's `exactMount` check → storage reads
|
|
`disconnected` forever). **That warning is about one storage, not the box.** `/mnt/nvme-1tb` is also
|
|
the `felhom-backup` target and the enrolled user-data drive, so remove scratch storages when done.
|
|
- **Forbidden:** do not destroy or unblock **`drill-r50` (VM 300)** — the only drift fixture (R-93).
|
|
(Access: the docs say no baked SSH key and G1 break-glass, but a key authenticated on 2026-07-31 —
|
|
**R-129**, unresolved.)
|
|
|
|
### `demo-felhom` — N100 · **Tier 0**
|
|
|
|
- **Freely:** create/destroy guests and scratch customers; reinstall the box.
|
|
- **Care:** it carries the **PBS-DR / offsite tier** (demo-hp has none), so it is the Tier 0 box whose
|
|
backup chain a test can actually disturb. Prefer demo-hp, per the 2026-07-25 ruling.
|
|
|
|
**Both Tier 0 boxes — the shared backup-target fence is DOWNGRADED to a cost, 2026-08-02 (D-d).** It
|
|
read *"do not re-point either backup target"*, because these are the only two **correctly configured**
|
|
boxes and therefore the regression path new installer logic is measured against. D-d makes both boxes
|
|
freely breakable and reinstallable, which loses that reference just as thoroughly — so the fence was
|
|
inconsistent with the decision and is not kept as a prohibition. **What survives is the reason:**
|
|
re-pointing (or reinstalling) costs the reference configuration, so know that you are spending it and
|
|
put the box back. If both are spent at once there is no correctly-configured box left to compare
|
|
against.
|
|
|
|
### `DooPlex` — 192.168.0.180 · **Tier 2**
|
|
|
|
- **Freely:** build, push images, `sudo kubectl`, read anything. Work inside `/mnt/5_hdd/felhom.eu/`.
|
|
- **Care:** the golden-bake nested VM `drill/drill.qcow2` lives here and is an accepted exception **for
|
|
bakes** (last used 0.185.1, 2026-07-29). Using it as a *drill victim* is not covered by that exception
|
|
and contradicts the 2026-07-25 ruling. If touched, restore `qemu-img snapshot -a virgin`.
|
|
- **Forbidden:** no global Docker cleanup; do not touch k3s data dirs, Longhorn mounts, PBS datastores
|
|
or Gitea storage; no destructive disk/guest ops; never run CC here with prompts disabled; abort large
|
|
builds if `df -h /mnt/5_hdd /` shows either over 90 %. **Because it is the recovery chain and a live
|
|
k3s node.**
|
|
|
|
### `Peti's cluster` (`peti-felhom`) — **Tier 2**
|
|
|
|
**Do not touch, at all.** A real external pilot with a real person behind it; its whole-guest backup
|
|
still shares a device with its guest, so a drive failure is **offsite-only recovery**. Deliberately not
|
|
migrated, parked until the tester reinstalls (`PETI` in `backlog/OPEN-ITEMS.md`). Currently DOWN, no
|
|
enrolled host. No access route from DooPlex, and nothing here needs one.
|
|
|
|
### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **not protected by D-d; not scratch either**
|
|
|
|
Reads are fine. It is the **offsite of last resort** (PBS-DR datastore, WireGuard hub, operator OOB
|
|
path) and — until 2026-08-03 — RAM-constrained (3.8 GB, R-90); it is now a **CX33 with 8 GB RAM and a
|
|
4 GiB swapfile**, which is what closed R-90. A very large restore is still worth watching. Do not delete datastores, prune
|
|
jobs, tunnel config or nftables rules; never a drill target. The Storage Boxes hold the restic copy —
|
|
customer documents and photos, on a credential that can still delete (R-95).
|
|
**Access: `ssh root@167.233.158.164` from DooPlex** — *not* `felhom-pve → 10.77.0.1`, the route that
|
|
produced a false "unreachable" verdict (standing rule 2).
|
|
|
|
### Tier 1 — drill VMs, scratch guests, scratch customers
|
|
|
|
Create and destroy freely **on a Tier 0 host**. Two exceptions: **`drill-r50`** is a fixture, not
|
|
scratch; and scratch **customers** outlive their VMs in the hub — delete those too, or they accumulate
|
|
(`sess-c` and `sess-d` were both left behind before anyone noticed).
|
|
|
|
#### A fixture may prove a mechanism. Only a fresh box may prove a path.
|
|
|
|
Rebuilding from scratch every time is waste; reusing a box is legitimate — but not for every claim.
|
|
|
|
- **Reusable fixture** — a snapshot-reset VM on a Tier 0 host. Use it for **mechanism** work: payload
|
|
capture, fix cycles, anything whose claim is about *code behaviour*. Reset to `virgin` between runs.
|
|
Fast, repeatable, and the right default for iteration.
|
|
- **Fresh day-0 from the ISO** — required for any claim about the **install path, the golden image,
|
|
agent publish/vouch, or first-boot state**. Slow, and the only thing that catches the drift family:
|
|
**R-111** (the golden's agent 17 releases behind), **R-115** (an agent built and deployed but never
|
|
published), **R-120** (the golden a controller release behind).
|
|
|
|
**A fixture must record its provenance** — which golden, agent and controller it was built from, and
|
|
when — alongside the VM. A fixture whose versions drift silently is R-120's mechanism turned into a
|
|
permanent installation, and it is *worse* than no fixture, because it produces confident wrong results
|
|
quickly.
|
|
|
|
**Worked example.** R-116's closing run deliberately did a real day-0 from the v1.25.0 ISO on demo-hp
|
|
instead of reusing the standing fixture. **That is how R-120 surfaced** — the fresh box installed the
|
|
golden's controller, which is a release behind, and showed the customer the wrong absent-target message.
|
|
The fixture would have shown a controller nobody installs.
|
|
|
|
## Not established — unknown, not guessed
|
|
|
|
- **Hetzner Storage Boxes** (`storage-box-pool-1`, `PBS-storage-1` / u629193 / box 611421) — state taken
|
|
from `backlog/OPEN-ITEMS.md`; **no direct access attempted or confirmed.** Tier 2 by what they hold.
|
|
- **`felhotest`** (legacy, `ssh -p 33022 kisfenyo@router.abonet.hu`) — **`Connection refused`**
|
|
2026-07-30; that was the only route tried. Untiered; assume nothing.
|
|
- **Peti's cluster** hardware/storage layout — unverified. Tier 2 rests on the relationship, which
|
|
needs no verification.
|
|
|
|
## The gap this page closes
|
|
|
|
During the R-116 diagnosis session (2026-07-30) a nested-Proxmox drill ran **on DooPlex**, a Tier 2
|
|
machine. Not a newly provisioned VM — the standing golden-bake fixture, booted from `virgin` and
|
|
restored to it, so nothing was lost. But it was chosen because DooPlex was the only machine no spec had
|
|
fenced, while the ruling naming the t740 sat in `operations/nodes.md` with nothing pointing at it.
|
|
**The designation existed and was unreachable** — the same class as an absent signal read as a positive
|
|
one.
|