diff --git a/CLAUDE.md b/CLAUDE.md index 19f4254..b904c01 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -108,10 +108,15 @@ Claude Code runs **on DooPlex (192.168.0.180, Debian 13, user `kisfenyo`)** — Repos in `/mnt/5_hdd/felhom.eu/git/`, build dirs in `/mnt/5_hdd/felhom.eu/build/`. `kubectl` and the image build/push are local commands; felhom-pve is one SSH hop. -| Host | Access | Role | -|------|--------|------| -| **DooPlex (this host)** | local — `/mnt/5_hdd/felhom.eu/{git,build}/` | Build + push images, `sudo kubectl` | -| Demo Proxmox host (N100) | `ssh felhom-pve` — via Tailscale `100.70.170.35` (location-independent); `felhom-pve-lan` = LAN `192.168.0.162` fallback | pveum/pct + live Proxmox validation | +| Host | Access | Role | Blast radius | +|------|--------|------|--------------| +| **DooPlex (this host)** | local — `/mnt/5_hdd/felhom.eu/{git,build}/` | Build + push images, `sudo kubectl` | **Tier 2 — precious.** It *is* the recovery chain (hub, Gitea, registry, PBS, k3s+Longhorn). **Never a drill target** | +| Demo Proxmox host (N100) | `ssh felhom-pve` — via Tailscale `100.70.170.35` (location-independent); `felhom-pve-lan` = LAN `192.168.0.162` fallback | pveum/pct + live Proxmox validation | **Tier 0 — disposable** | +| Demo Proxmox host (HP t740) | `ssh demo-hp` — via Tailscale `100.76.96.79`; `demo-hp-lan` = LAN `192.168.0.87` (ProxyJump `felhom-pve`). **No baked SSH key** — G1 break-glass password vaulted in the hub | **The designated drill + build VM host** (operator ruling 2026-07-25) | **Tier 0 — disposable. Reach here first** | + +**Which box do I break?** → **`documentation/runbooks/target-selection.md`** — the tiers, and per +machine what is freely permitted / needs care / forbidden, each with its reason. Read it before picking +a machine for a drill, a destructive test or a throwaway VM. The `felhom-pve` transport is Tailscale (the N100 is travel-portable) — topology, the accept-dns rule, the accept-routes spike result, rollback, and the vacation-day checklist live in diff --git a/REPORT-r116-diag.md b/REPORT-r116-diag.md index d5d527e..cb86300 100644 --- a/REPORT-r116-diag.md +++ b/REPORT-r116-diag.md @@ -84,11 +84,15 @@ R-117 and R-118 added. The single register edit this session makes. ## Record correction -The brief's baseline `controller 0.185.1` is the version the **golden bakes**; both demo boxes actually -run **0.186.0** (R-114 + R-112, 2026-07-29). Both are true of different things, and the distinction -matters because R-114's `TargetAbsent` branch exists only in 0.186.0. Confirmed: agent **0.115.0**, -controller **0.186.0** live, hub **0.81.0** (manifest pin and live pod image agree), host-install -**1.22.0**, `felhom.eu` HEAD `c3ce4c7`. +The brief's baseline `controller 0.185.1` is the version the **golden bakes**. **0.186.0** (R-114 + +R-112, 2026-07-29) is what **demo-felhom** runs — **demo-hp is still on 0.185.1**, so the fleet is +split, and R-114's `TargetAbsent` branch exists only on demo-felhom. Confirmed: agent **0.115.0** +(felhom-pve) / **0.113.0** (demo-hp), hub **0.81.0** (manifest pin and live pod image agree), +host-install **1.22.0**, `felhom.eu` HEAD `c3ce4c7`. + +> **Correction, 2026-07-30.** As first written this section said 0.186.0 was what *both* demo boxes run. +> That was wrong — only felhom-pve's guest was sampled and the result generalised to the fleet. demo-hp +> re-checked directly → `0.185.1`. Fixed here and in the audit's baseline table. ## Teardown and fences diff --git a/documentation/audits/DIAG-r116-disks-payload-2026-07-30.md b/documentation/audits/DIAG-r116-disks-payload-2026-07-30.md index 7bae28c..875ee82 100644 --- a/documentation/audits/DIAG-r116-disks-payload-2026-07-30.md +++ b/documentation/audits/DIAG-r116-disks-payload-2026-07-30.md @@ -14,15 +14,21 @@ All confirmed this run, not carried from the brief. | Artifact | Version | How confirmed | |---|---|---| | agent | **0.115.0** | `felhom-agent --version` on felhom-pve; `felhom-agent/CHANGELOG.md:1` = `## v0.115.0`; repo HEAD `a58239f6ded367fe27b4eb57cf2fcb0c54414b80` | -| controller | **0.186.0** ⚠ | `pct exec 9201 -- docker ps` on felhom-pve → `felhom-controller:0.186.0`; `CHANGELOG.md:3` = `### v0.186.0`; HEAD `b331f18424b7cf8d0883e7709f74595808264080` | +| controller | **0.186.0 on demo-felhom, 0.185.1 on demo-hp** ⚠ | `pct exec 9201 -- docker ps`: felhom-pve → `felhom-controller:0.186.0`, demo-hp → `felhom-controller:0.185.1` (both re-verified 2026-07-30); `CHANGELOG.md:3` = `### v0.186.0`; HEAD `b331f18424b7cf8d0883e7709f74595808264080` | | hub | **0.81.0** | `manifests/hub.yaml:128` pin **and** the live pod image (`kubectl -n felhom-system get pod hub-… -o jsonpath=…`) both `felhom-hub:0.81.0`; `hub/CHANGELOG.md:1` = `## v0.81.0` | | host-install | **1.22.0** | `scripts/felhom-host-install.sh:187` `SCRIPT_VERSION="1.22.0"` | | `felhom.eu` HEAD | `c3ce4c7b20ce4c0d3673590d710d13e5d28b68bb` | `git rev-parse HEAD` | -⚠ **The brief's `controller 0.185.1` is not what is deployed.** 0.185.1 is what the **golden bakes** -(so it is what a fresh drill box installs, which is why Part 5 correctly recorded it); **0.186.0** is -what both demo boxes actually run, shipped 2026-07-29 as R-114 + R-112. Both numbers are true of -different things; the distinction matters because R-114's `TargetAbsent` branch exists only in 0.186.0. +⚠ **The controller version is not one number.** 0.185.1 is what the **golden bakes** (so it is what a +fresh drill box installs, which is why the brief and Part 5 both recorded it); **0.186.0** shipped +2026-07-29 as R-114 + R-112 and is what **demo-felhom** runs — but **demo-hp is still on 0.185.1**, so +the fleet is split. The distinction matters because R-114's `TargetAbsent` branch exists only in 0.186.0, +i.e. **demo-hp does not have it**. + +> **Correction, 2026-07-30.** As first written this row said 0.186.0 was what *both* demo boxes run. +> That was wrong: only felhom-pve's guest was sampled, and the claim was generalised to the fleet from +> one box. demo-hp re-checked directly → `felhom-controller:0.185.1`. Corrected in the same commit that +> added `runbooks/target-selection.md`. Hub host table, read this run: `demo-felhom-8363b5` agent 0.115.0 ONLINE · `demo-hp-bb76ea` agent 0.113.0 ONLINE · `drill-r50-0a4f9a` DOWN · `sess-c-8158e4` DOWN · `sess-d-0452c4` **STALE**. diff --git a/documentation/runbooks/target-selection.md b/documentation/runbooks/target-selection.md new file mode 100644 index 0000000..45d7991 --- /dev/null +++ b/documentation/runbooks/target-selection.md @@ -0,0 +1,105 @@ +# Which box do I break? — target selection by blast radius + +> Read before picking a machine for a drill, a destructive test, or a throwaway VM. It answers one +> question: *what is safe to lose.* Reachability is in `CLAUDE.md`; hardware in `operations/nodes.md`. +> Added 2026-07-30. + +## The rule + +> **Start at Tier 0 and work down only if Tier 0 genuinely cannot host the work.** +> **A Tier 2 machine is used only when a task says so explicitly** — never by inference from what was +> not forbidden. An absent fence is not permission. If no tier fits, **stop and ask.** + +Fences name **acts**, not machines. "Do not re-target demo-hp's backup target" and "do not use demo-hp +to host a throwaway VM" are unrelated; only the first has ever been meant. Read a per-machine +prohibition as covering the act it names and nothing more. + +| Tier | Meaning | Machines | +|---|---|---| +| **0 — disposable. Reach here first.** | Exists to be broken; reinstalling is a routine afternoon, not an incident. **A drill that needs a victim uses one of these.** | `demo-hp` (t740), `demo-felhom` (N100) | +| **1 — create and destroy freely** | Throwaway VMs, guests, scratch customers — **hosted on a Tier 0 machine** | drill VMs, scratch guests | +| **2 — precious. Never a drill target.** | Losing it costs the recovery chain or a real relationship | **DooPlex**, **Peti's cluster**, **ep0** + the Hetzner Storage Boxes | + +**DooPlex is Tier 2 because it *is* the recovery chain** — hub, Gitea, registry, PBS, k3s + Longhorn. +Everything else rebuilds from it; it rebuilds from nothing. A bad moment in a DR drill there costs the +thing under test, the source of truth for it, and the backups, at once. + +**Standing ruling, 2026-07-25 (`operations/nodes.md`):** drill and build VMs live on the **t740** — not +felhom-pve, and **moved off DooPlex**. This page exists because that ruling sat where no session reads. + +## Per machine — permitted / needs care / forbidden + +### `demo-hp` — HP t740 · **Tier 0 · the designated drill + build VM host** + +- **Freely:** host nested drill VMs; create/destroy guests and scratch customers; reinstall the box. + **This is the default answer to "where do I run this".** +- **Care:** `local-lvm` is a **thin pool, over-subscribed** (~144 GiB allocated over ~54 GiB) backing + live guest 9201 — filling it corrupts every guest. Put VM disks on a dir storage at + **`/mnt/nvme-1tb`, at its root** (a subdirectory fails the agent's `exactMount` check → storage reads + `disconnected` forever). **That warning is about one storage, not the box.** `/mnt/nvme-1tb` is also + the `felhom-backup` target and the enrolled user-data drive, so remove scratch storages when done. +- **Forbidden:** do not re-target its backup configuration (below). Do not destroy or unblock + **`drill-r50` (VM 300)** — the only drift fixture (R-93). No baked SSH key; access is the hub-vaulted + G1 break-glass password (R-61). + +### `demo-felhom` — N100 · **Tier 0** + +- **Freely:** create/destroy guests and scratch customers; reinstall the box. +- **Care:** it carries the **PBS-DR / offsite tier** (demo-hp has none), so it is the Tier 0 box whose + backup chain a test can actually disturb. Prefer demo-hp, per the 2026-07-25 ruling. +- **Forbidden:** do not re-target its backup configuration. + +**Both Tier 0 boxes, one shared prohibition and its reason:** do not re-point either backup target. +They are the only two **correctly configured** boxes, so they are the regression path new installer +logic is measured against — losing that reference costs more than the box does. + +### `DooPlex` — 192.168.0.180 · **Tier 2** + +- **Freely:** build, push images, `sudo kubectl`, read anything. Work inside `/mnt/5_hdd/felhom.eu/`. +- **Care:** the golden-bake nested VM `drill/drill.qcow2` lives here and is an accepted exception **for + bakes** (last used 0.185.1, 2026-07-29). Using it as a *drill victim* is not covered by that exception + and contradicts the 2026-07-25 ruling. If touched, restore `qemu-img snapshot -a virgin`. +- **Forbidden:** no global Docker cleanup; do not touch k3s data dirs, Longhorn mounts, PBS datastores + or Gitea storage; no destructive disk/guest ops; never run CC here with prompts disabled; abort large + builds if `df -h /mnt/5_hdd /` shows either over 90 %. **Because it is the recovery chain and a live + k3s node.** + +### `Peti's cluster` (`peti-felhom`) — **Tier 2** + +**Do not touch, at all.** A real external pilot with a real person behind it; its whole-guest backup +still shares a device with its guest, so a drive failure is **offsite-only recovery**. Deliberately not +migrated, parked until the tester reinstalls (`PETI` in `backlog/OPEN-ITEMS.md`). Currently DOWN, no +enrolled host. No access route from DooPlex, and nothing here needs one. + +### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **Tier 2** + +Reads are fine. It is the **offsite of last resort** (PBS-DR datastore, WireGuard hub, operator OOB +path) and RAM-constrained (3.8 GB, R-90) so a large restore can OOM it. Do not delete datastores, prune +jobs, tunnel config or nftables rules; never a drill target. The Storage Boxes hold the restic copy — +customer documents and photos, on a credential that can still delete (R-95). +**Access: `ssh root@167.233.158.164` from DooPlex** — *not* `felhom-pve → 10.77.0.1`, the route that +produced a false "unreachable" verdict (standing rule 2). + +### Tier 1 — drill VMs, scratch guests, scratch customers + +Create and destroy freely **on a Tier 0 host**. Two exceptions: **`drill-r50`** is a fixture, not +scratch; and scratch **customers** outlive their VMs in the hub — delete those too, or they accumulate +(`sess-c` and `sess-d` are both still there). + +## Not established — unknown, not guessed + +- **Hetzner Storage Boxes** (`storage-box-pool-1`, `PBS-storage-1` / u629193 / box 611421) — state taken + from `backlog/OPEN-ITEMS.md`; **no direct access attempted or confirmed.** Tier 2 by what they hold. +- **`felhotest`** (legacy, `ssh -p 33022 kisfenyo@router.abonet.hu`) — **`Connection refused`** + 2026-07-30; that was the only route tried. Untiered; assume nothing. +- **Peti's cluster** hardware/storage layout — unverified. Tier 2 rests on the relationship, which + needs no verification. + +## The gap this page closes + +During the R-116 diagnosis session (2026-07-30) a nested-Proxmox drill ran **on DooPlex**, a Tier 2 +machine. Not a newly provisioned VM — the standing golden-bake fixture, booted from `virgin` and +restored to it, so nothing was lost. But it was chosen because DooPlex was the only machine no spec had +fenced, while the ruling naming the t740 sat in `operations/nodes.md` with nothing pointing at it. +**The designation existed and was unreachable** — the same class as an absent signal read as a positive +one.