docs: write down which boxes are disposable (target selection by blast radius)
Nothing in the repo said which machines are safe to break. The host table gave access and role and stopped there, so a session needing a victim had to guess -- and the guessing inverted: the two boxes that exist to be broken were treated as sacred, and DooPlex (the recovery chain) got used because it was the only box no spec had fenced. New documentation/runbooks/target-selection.md -- one page, three tiers, and per machine what is freely permitted / needs care / forbidden, each carrying its REASON so a rule can be correctly narrowed later instead of ossifying. States the selection rule positively (start at Tier 0; a Tier 2 box only when a task says so explicitly; an absent fence is not permission) and that fences name ACTS, not machines -- demo-hp's over-subscribed local-lvm is one dangerous storage, not a dangerous box. CLAUDE.md: host table gains a Blast radius column, gains the missing demo-hp row (it was where the drill VMs ran and it was not in the table at all), and a pointer line to the new runbook. CORRECTION to the spec's problem statement: the designation was not missing. The 2026-07-25 operator ruling naming the t740 as drill+build VM host -- explicitly "moved off DooPlex" -- already existed in operations/nodes.md. It sat where no session reads at start, while the prohibitions were repeated in every task spec. The defect is reachability of the ruling, not its absence, and the R-116 drill on DooPlex contradicted a written ruling rather than filling a vacuum. CORRECTION to the R-116 record, same commit: the baseline claimed controller 0.186.0 on both demo boxes. Only felhom-pve was sampled and generalised; demo-hp re-checked directly runs 0.185.1, so the fleet is split and R-114's TargetAbsent branch is absent from demo-hp. Fixed in the audit table and REPORT-r116-diag. Docs only -- no code, no build, no deploy, no host reconfigured, no host renamed.
This commit is contained in:
@@ -108,10 +108,15 @@ Claude Code runs **on DooPlex (192.168.0.180, Debian 13, user `kisfenyo`)** —
|
||||
Repos in `/mnt/5_hdd/felhom.eu/git/`, build dirs in `/mnt/5_hdd/felhom.eu/build/`. `kubectl` and the
|
||||
image build/push are local commands; felhom-pve is one SSH hop.
|
||||
|
||||
| Host | Access | Role |
|
||||
|------|--------|------|
|
||||
| **DooPlex (this host)** | local — `/mnt/5_hdd/felhom.eu/{git,build}/` | Build + push images, `sudo kubectl` |
|
||||
| Demo Proxmox host (N100) | `ssh felhom-pve` — via Tailscale `100.70.170.35` (location-independent); `felhom-pve-lan` = LAN `192.168.0.162` fallback | pveum/pct + live Proxmox validation |
|
||||
| Host | Access | Role | Blast radius |
|
||||
|------|--------|------|--------------|
|
||||
| **DooPlex (this host)** | local — `/mnt/5_hdd/felhom.eu/{git,build}/` | Build + push images, `sudo kubectl` | **Tier 2 — precious.** It *is* the recovery chain (hub, Gitea, registry, PBS, k3s+Longhorn). **Never a drill target** |
|
||||
| Demo Proxmox host (N100) | `ssh felhom-pve` — via Tailscale `100.70.170.35` (location-independent); `felhom-pve-lan` = LAN `192.168.0.162` fallback | pveum/pct + live Proxmox validation | **Tier 0 — disposable** |
|
||||
| Demo Proxmox host (HP t740) | `ssh demo-hp` — via Tailscale `100.76.96.79`; `demo-hp-lan` = LAN `192.168.0.87` (ProxyJump `felhom-pve`). **No baked SSH key** — G1 break-glass password vaulted in the hub | **The designated drill + build VM host** (operator ruling 2026-07-25) | **Tier 0 — disposable. Reach here first** |
|
||||
|
||||
**Which box do I break?** → **`documentation/runbooks/target-selection.md`** — the tiers, and per
|
||||
machine what is freely permitted / needs care / forbidden, each with its reason. Read it before picking
|
||||
a machine for a drill, a destructive test or a throwaway VM.
|
||||
|
||||
The `felhom-pve` transport is Tailscale (the N100 is travel-portable) — topology, the accept-dns
|
||||
rule, the accept-routes spike result, rollback, and the vacation-day checklist live in
|
||||
|
||||
+9
-5
@@ -84,11 +84,15 @@ R-117 and R-118 added. The single register edit this session makes.
|
||||
|
||||
## Record correction
|
||||
|
||||
The brief's baseline `controller 0.185.1` is the version the **golden bakes**; both demo boxes actually
|
||||
run **0.186.0** (R-114 + R-112, 2026-07-29). Both are true of different things, and the distinction
|
||||
matters because R-114's `TargetAbsent` branch exists only in 0.186.0. Confirmed: agent **0.115.0**,
|
||||
controller **0.186.0** live, hub **0.81.0** (manifest pin and live pod image agree), host-install
|
||||
**1.22.0**, `felhom.eu` HEAD `c3ce4c7`.
|
||||
The brief's baseline `controller 0.185.1` is the version the **golden bakes**. **0.186.0** (R-114 +
|
||||
R-112, 2026-07-29) is what **demo-felhom** runs — **demo-hp is still on 0.185.1**, so the fleet is
|
||||
split, and R-114's `TargetAbsent` branch exists only on demo-felhom. Confirmed: agent **0.115.0**
|
||||
(felhom-pve) / **0.113.0** (demo-hp), hub **0.81.0** (manifest pin and live pod image agree),
|
||||
host-install **1.22.0**, `felhom.eu` HEAD `c3ce4c7`.
|
||||
|
||||
> **Correction, 2026-07-30.** As first written this section said 0.186.0 was what *both* demo boxes run.
|
||||
> That was wrong — only felhom-pve's guest was sampled and the result generalised to the fleet. demo-hp
|
||||
> re-checked directly → `0.185.1`. Fixed here and in the audit's baseline table.
|
||||
|
||||
## Teardown and fences
|
||||
|
||||
|
||||
@@ -14,15 +14,21 @@ All confirmed this run, not carried from the brief.
|
||||
| Artifact | Version | How confirmed |
|
||||
|---|---|---|
|
||||
| agent | **0.115.0** | `felhom-agent --version` on felhom-pve; `felhom-agent/CHANGELOG.md:1` = `## v0.115.0`; repo HEAD `a58239f6ded367fe27b4eb57cf2fcb0c54414b80` |
|
||||
| controller | **0.186.0** ⚠ | `pct exec 9201 -- docker ps` on felhom-pve → `felhom-controller:0.186.0`; `CHANGELOG.md:3` = `### v0.186.0`; HEAD `b331f18424b7cf8d0883e7709f74595808264080` |
|
||||
| controller | **0.186.0 on demo-felhom, 0.185.1 on demo-hp** ⚠ | `pct exec 9201 -- docker ps`: felhom-pve → `felhom-controller:0.186.0`, demo-hp → `felhom-controller:0.185.1` (both re-verified 2026-07-30); `CHANGELOG.md:3` = `### v0.186.0`; HEAD `b331f18424b7cf8d0883e7709f74595808264080` |
|
||||
| hub | **0.81.0** | `manifests/hub.yaml:128` pin **and** the live pod image (`kubectl -n felhom-system get pod hub-… -o jsonpath=…`) both `felhom-hub:0.81.0`; `hub/CHANGELOG.md:1` = `## v0.81.0` |
|
||||
| host-install | **1.22.0** | `scripts/felhom-host-install.sh:187` `SCRIPT_VERSION="1.22.0"` |
|
||||
| `felhom.eu` HEAD | `c3ce4c7b20ce4c0d3673590d710d13e5d28b68bb` | `git rev-parse HEAD` |
|
||||
|
||||
⚠ **The brief's `controller 0.185.1` is not what is deployed.** 0.185.1 is what the **golden bakes**
|
||||
(so it is what a fresh drill box installs, which is why Part 5 correctly recorded it); **0.186.0** is
|
||||
what both demo boxes actually run, shipped 2026-07-29 as R-114 + R-112. Both numbers are true of
|
||||
different things; the distinction matters because R-114's `TargetAbsent` branch exists only in 0.186.0.
|
||||
⚠ **The controller version is not one number.** 0.185.1 is what the **golden bakes** (so it is what a
|
||||
fresh drill box installs, which is why the brief and Part 5 both recorded it); **0.186.0** shipped
|
||||
2026-07-29 as R-114 + R-112 and is what **demo-felhom** runs — but **demo-hp is still on 0.185.1**, so
|
||||
the fleet is split. The distinction matters because R-114's `TargetAbsent` branch exists only in 0.186.0,
|
||||
i.e. **demo-hp does not have it**.
|
||||
|
||||
> **Correction, 2026-07-30.** As first written this row said 0.186.0 was what *both* demo boxes run.
|
||||
> That was wrong: only felhom-pve's guest was sampled, and the claim was generalised to the fleet from
|
||||
> one box. demo-hp re-checked directly → `felhom-controller:0.185.1`. Corrected in the same commit that
|
||||
> added `runbooks/target-selection.md`.
|
||||
|
||||
Hub host table, read this run: `demo-felhom-8363b5` agent 0.115.0 ONLINE · `demo-hp-bb76ea` agent
|
||||
0.113.0 ONLINE · `drill-r50-0a4f9a` DOWN · `sess-c-8158e4` DOWN · `sess-d-0452c4` **STALE**.
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
# Which box do I break? — target selection by blast radius
|
||||
|
||||
> Read before picking a machine for a drill, a destructive test, or a throwaway VM. It answers one
|
||||
> question: *what is safe to lose.* Reachability is in `CLAUDE.md`; hardware in `operations/nodes.md`.
|
||||
> Added 2026-07-30.
|
||||
|
||||
## The rule
|
||||
|
||||
> **Start at Tier 0 and work down only if Tier 0 genuinely cannot host the work.**
|
||||
> **A Tier 2 machine is used only when a task says so explicitly** — never by inference from what was
|
||||
> not forbidden. An absent fence is not permission. If no tier fits, **stop and ask.**
|
||||
|
||||
Fences name **acts**, not machines. "Do not re-target demo-hp's backup target" and "do not use demo-hp
|
||||
to host a throwaway VM" are unrelated; only the first has ever been meant. Read a per-machine
|
||||
prohibition as covering the act it names and nothing more.
|
||||
|
||||
| Tier | Meaning | Machines |
|
||||
|---|---|---|
|
||||
| **0 — disposable. Reach here first.** | Exists to be broken; reinstalling is a routine afternoon, not an incident. **A drill that needs a victim uses one of these.** | `demo-hp` (t740), `demo-felhom` (N100) |
|
||||
| **1 — create and destroy freely** | Throwaway VMs, guests, scratch customers — **hosted on a Tier 0 machine** | drill VMs, scratch guests |
|
||||
| **2 — precious. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster**, **ep0** + the Hetzner Storage Boxes |
|
||||
|
||||
**DooPlex is Tier 2 because it *is* the recovery chain** — hub, Gitea, registry, PBS, k3s + Longhorn.
|
||||
Everything else rebuilds from it; it rebuilds from nothing. A bad moment in a DR drill there costs the
|
||||
thing under test, the source of truth for it, and the backups, at once.
|
||||
|
||||
**Standing ruling, 2026-07-25 (`operations/nodes.md`):** drill and build VMs live on the **t740** — not
|
||||
felhom-pve, and **moved off DooPlex**. This page exists because that ruling sat where no session reads.
|
||||
|
||||
## Per machine — permitted / needs care / forbidden
|
||||
|
||||
### `demo-hp` — HP t740 · **Tier 0 · the designated drill + build VM host**
|
||||
|
||||
- **Freely:** host nested drill VMs; create/destroy guests and scratch customers; reinstall the box.
|
||||
**This is the default answer to "where do I run this".**
|
||||
- **Care:** `local-lvm` is a **thin pool, over-subscribed** (~144 GiB allocated over ~54 GiB) backing
|
||||
live guest 9201 — filling it corrupts every guest. Put VM disks on a dir storage at
|
||||
**`/mnt/nvme-1tb`, at its root** (a subdirectory fails the agent's `exactMount` check → storage reads
|
||||
`disconnected` forever). **That warning is about one storage, not the box.** `/mnt/nvme-1tb` is also
|
||||
the `felhom-backup` target and the enrolled user-data drive, so remove scratch storages when done.
|
||||
- **Forbidden:** do not re-target its backup configuration (below). Do not destroy or unblock
|
||||
**`drill-r50` (VM 300)** — the only drift fixture (R-93). No baked SSH key; access is the hub-vaulted
|
||||
G1 break-glass password (R-61).
|
||||
|
||||
### `demo-felhom` — N100 · **Tier 0**
|
||||
|
||||
- **Freely:** create/destroy guests and scratch customers; reinstall the box.
|
||||
- **Care:** it carries the **PBS-DR / offsite tier** (demo-hp has none), so it is the Tier 0 box whose
|
||||
backup chain a test can actually disturb. Prefer demo-hp, per the 2026-07-25 ruling.
|
||||
- **Forbidden:** do not re-target its backup configuration.
|
||||
|
||||
**Both Tier 0 boxes, one shared prohibition and its reason:** do not re-point either backup target.
|
||||
They are the only two **correctly configured** boxes, so they are the regression path new installer
|
||||
logic is measured against — losing that reference costs more than the box does.
|
||||
|
||||
### `DooPlex` — 192.168.0.180 · **Tier 2**
|
||||
|
||||
- **Freely:** build, push images, `sudo kubectl`, read anything. Work inside `/mnt/5_hdd/felhom.eu/`.
|
||||
- **Care:** the golden-bake nested VM `drill/drill.qcow2` lives here and is an accepted exception **for
|
||||
bakes** (last used 0.185.1, 2026-07-29). Using it as a *drill victim* is not covered by that exception
|
||||
and contradicts the 2026-07-25 ruling. If touched, restore `qemu-img snapshot -a virgin`.
|
||||
- **Forbidden:** no global Docker cleanup; do not touch k3s data dirs, Longhorn mounts, PBS datastores
|
||||
or Gitea storage; no destructive disk/guest ops; never run CC here with prompts disabled; abort large
|
||||
builds if `df -h /mnt/5_hdd /` shows either over 90 %. **Because it is the recovery chain and a live
|
||||
k3s node.**
|
||||
|
||||
### `Peti's cluster` (`peti-felhom`) — **Tier 2**
|
||||
|
||||
**Do not touch, at all.** A real external pilot with a real person behind it; its whole-guest backup
|
||||
still shares a device with its guest, so a drive failure is **offsite-only recovery**. Deliberately not
|
||||
migrated, parked until the tester reinstalls (`PETI` in `backlog/OPEN-ITEMS.md`). Currently DOWN, no
|
||||
enrolled host. No access route from DooPlex, and nothing here needs one.
|
||||
|
||||
### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **Tier 2**
|
||||
|
||||
Reads are fine. It is the **offsite of last resort** (PBS-DR datastore, WireGuard hub, operator OOB
|
||||
path) and RAM-constrained (3.8 GB, R-90) so a large restore can OOM it. Do not delete datastores, prune
|
||||
jobs, tunnel config or nftables rules; never a drill target. The Storage Boxes hold the restic copy —
|
||||
customer documents and photos, on a credential that can still delete (R-95).
|
||||
**Access: `ssh root@167.233.158.164` from DooPlex** — *not* `felhom-pve → 10.77.0.1`, the route that
|
||||
produced a false "unreachable" verdict (standing rule 2).
|
||||
|
||||
### Tier 1 — drill VMs, scratch guests, scratch customers
|
||||
|
||||
Create and destroy freely **on a Tier 0 host**. Two exceptions: **`drill-r50`** is a fixture, not
|
||||
scratch; and scratch **customers** outlive their VMs in the hub — delete those too, or they accumulate
|
||||
(`sess-c` and `sess-d` are both still there).
|
||||
|
||||
## Not established — unknown, not guessed
|
||||
|
||||
- **Hetzner Storage Boxes** (`storage-box-pool-1`, `PBS-storage-1` / u629193 / box 611421) — state taken
|
||||
from `backlog/OPEN-ITEMS.md`; **no direct access attempted or confirmed.** Tier 2 by what they hold.
|
||||
- **`felhotest`** (legacy, `ssh -p 33022 kisfenyo@router.abonet.hu`) — **`Connection refused`**
|
||||
2026-07-30; that was the only route tried. Untiered; assume nothing.
|
||||
- **Peti's cluster** hardware/storage layout — unverified. Tier 2 rests on the relationship, which
|
||||
needs no verification.
|
||||
|
||||
## The gap this page closes
|
||||
|
||||
During the R-116 diagnosis session (2026-07-30) a nested-Proxmox drill ran **on DooPlex**, a Tier 2
|
||||
machine. Not a newly provisioned VM — the standing golden-bake fixture, booted from `virgin` and
|
||||
restored to it, so nothing was lost. But it was chosen because DooPlex was the only machine no spec had
|
||||
fenced, while the ruling naming the t740 sat in `operations/nodes.md` with nothing pointing at it.
|
||||
**The designation existed and was unreachable** — the same class as an absent signal read as a positive
|
||||
one.
|
||||
Reference in New Issue
Block a user