diff --git a/documentation/PROMPT-TEMPLATE.md b/documentation/PROMPT-TEMPLATE.md index 9513726..3e213b9 100644 --- a/documentation/PROMPT-TEMPLATE.md +++ b/documentation/PROMPT-TEMPLATE.md @@ -377,7 +377,12 @@ an owner — a row nobody owns is how items got lost in the first place. - Do NOT [specific out-of-scope thing CC might build]. -- Do NOT touch [the untouchable: e.g. the operator-signed `DecommissionExecutor` / `classify.go`]. +- Do NOT [**the forbidden ACT**, and **its reason**: e.g. "modify the operator-signed + `DecommissionExecutor` — it is the signature boundary"]. **Fence the act, not the object.** A bare + "do not touch X" is read as covering every act on X, including ones nobody meant to restrict — that + is how "do not re-target demo-hp's backup target" became "do not use demo-hp at all", which pushed a + destructive drill onto the one machine holding the recovery chain. State the reason too: a rule whose + reason is recorded can be correctly narrowed later, and one without it becomes permanent by default. - Do NOT reuse [the dangerous lookalike from §5]. - Do NOT refactor nearby code, change passing tests, or create a branch. - Do NOT add "Co-Authored by..." anywhere @@ -391,6 +396,12 @@ an owner — a row nobody owns is how items got lost in the first place. **Clean-tree gate first:** `git status --porcelain` empty AND `HEAD` == `origin/main` in the repo being built. An unpushed change does not exist. +**If the task needs a machine to break — NAME IT.** A drill, destructive test or throwaway VM gets an +explicit target: *"run this on demo-hp"*. Do not leave it to be inferred from the prohibition list; +listing only what is off-limits leaves the most valuable unfenced machine as the residual choice, which +is exactly how a drill landed on DooPlex. Tiers and per-machine permitted/care/forbidden: +[`runbooks/target-selection.md`](runbooks/target-selection.md). + **Controller** (local build on DooPlex → guest via golden/bootstrap): ```bash FELHOM_ROOT=/mnt/5_hdd/felhom.eu diff --git a/documentation/operations/nodes.md b/documentation/operations/nodes.md index 048cf00..3c766e5 100644 --- a/documentation/operations/nodes.md +++ b/documentation/operations/nodes.md @@ -13,15 +13,30 @@ | Firmware | AMI AN3PLUS-class | **AMI M42 v01.10 (11/11/2020)** | | PVE node name | `demo-felhom` | `felhom-host` | | Customer | `demo-felhom` | `demo-hp` | -| Agent | 0.96.0 | 0.96.0 | +| Agent / controller | *not recorded here* — see the note below | *not recorded here* | | Control plane | **island** `169.254.253.1:8443` on `vmbr9` (R-50, migrated 2026-07-25) | **island** `169.254.253.1:8443` on `vmbr9` (R-50, migrated 2026-07-25) | | SSH alias | `felhom-pve` | **`demo-hp`** | | Tailnet | `100.70.170.35` | **`100.76.96.79`** | | Loader used to install | `mkimage` (unsigned, SB **off** — firmware workaround) | **`shim`, Secure Boot ENABLED** | +> **No component versions are recorded on this page — deliberately.** Agent, controller, hub and +> host-install versions change several times a day, so any number written here is wrong within hours and +> is then read as fact. **Ask the fleet instead:** the hub host list (`/hosts`) and customer list +> (`/configs`) carry the live agent and controller versions per box; `felhom-agent --version` on the +> node and `pct exec -- docker ps` in the guest are the authorities. Versions that must be pinned +> in writing belong in the per-repo `CHANGELOG.md` and the hub's Day-0 artifact manifest — not in an +> inventory. **The fleet is not uniform**: on 2026-07-30 the two boxes ran different agent *and* +> different controller versions, so a single number for "the fleet" would have been wrong regardless. + **Both are at the VACATION site** and travel home with the rest of the kit **~2026-08-02**. While -away, their LAN addresses are on that site's `192.168.0.0/24`: N100 `.147`/`.162`, HP `.87`. The -tailnet addresses are the stable ones — use those. +away, their LAN addresses are on that site's `192.168.0.0/24`: N100 `.147`/`.162`, HP `.87` — **re-check +rather than trusting these** (`ip -br addr show vmbr0`; the N100 read `.162` on 2026-07-30). The +tailnet addresses are the stable ones — use those. Direct LAN literals are **not** reachable from +DooPlex while the boxes are away (`felhom-pve-lan` → `No route to host`, 2026-07-30). + +**Which box is safe to break, and what may be done to each: +[`../runbooks/target-selection.md`](../runbooks/target-selection.md).** This page is *what the hardware +is*; that page is *what you may do to it*. ## demo-hp — the HP t740, in detail @@ -30,13 +45,23 @@ tailnet addresses are the stable ones — use those. | device | model | serial | role | |---|---|---|---| | `sda` | SanDisk X600 M.2 2280 SATA 128GB | `182195804614` | **system disk** — PVE, LVM (`pve-root` 39.6G, `pve-data` thin pool, guest 9201's three volumes) | -| `nvme0n1` | KXG50PNV1T02 NVMe TOSHIBA 1024GB | `58BS11AFT8MQ` | **PRESENT AND UNENROLLED — do not touch** | +| `nvme0n1` | KXG50PNV1T02 NVMe TOSHIBA 1024GB | `58BS11AFT8MQ` | **ENROLLED** — mounted `/mnt/nvme-1tb`, the enrolled user-data drive **and** the `felhom-backup` whole-guest backup target | -> **The NVMe is a future Tárhely candidate and is deliberately outside everything.** It still holds -> its previous **NTFS** partition, is not mounted, and appears in no LVM PV and no ZFS pool. The -> install ISO's exact-serial filter pinned `sda` only, which is why it survived. **It joins the -> product later through the normal Tárhely (storage) flow — never through the installer, and never -> by hand.** Anything that formats it before that decision is a mistake. +> **The NVMe joined the product on 2026-07-22, through the normal Tárhely flow, as intended.** Enrolled +> to guest 9201 (PUBLISH TRAIN, `pilot/RUNBOOK-publish-agent-0.93-2026-07-22.md`); the vzdump target was +> moved onto it by E-2a. Verified live 2026-07-30: `nvme0n1` → `/mnt/nvme-1tb`, and +> `dir: felhom-backup / path /mnt/nvme-1tb / is_mountpoint 1` in `storage.cfg`. +> +> **The "PRESENT AND UNENROLLED — do not touch" fence that stood here is RETRACTED**, and its reason is +> recorded so it is not mistaken for a live rule: the NVMe was outside everything because the install +> ISO's exact-serial filter pinned `sda` only, and it was to join *through the Tárhely flow rather than +> the installer or by hand*. **That condition was satisfied; the prohibition expired with it.** It stood +> for eight days after enrolment and contradicted the task specs that (correctly) sent drill-VM disks to +> `/mnt/nvme-1tb`. +> +> Still true, and now the operative caution: a dir storage there **must sit at the mountpoint root** (a +> subdirectory fails the agent's `exactMount` check → the storage reads `disconnected` forever), and it +> shares the device with the box's own backups — so remove scratch storages when done. ### NIC map — and the trap @@ -58,8 +83,8 @@ that looked installed and could never call home. Repaired on the console by brid Current, post-repair: `vmbr0` static `192.168.0.87/24`, gw `192.168.0.1`, bridge-port `enp2s0f0`. No trace of `192.168.100.2` remains. `wg-felhom` `10.77.0.3/32` up to the hub. Guest **9201 -`demo-hp`** running. Agent **0.96.0** (R-50 island: `local_api` on `169.254.253.1:8443`/`vmbr9`, -guest `eth1 169.254.253.2/30`, `lan_resolver.host_ip` pinned to `192.168.0.87`; `.bak-0.95.0` retained; caps 68/68 ok). +`demo-hp`** running. Agent config shape (R-50 island): `local_api` on `169.254.253.1:8443`/`vmbr9`, +guest `eth1 169.254.253.2/30`, `lan_resolver.host_ip` pinned to `192.168.0.87`. ### Designated drill + build VM host (operator ruling, 2026-07-25) @@ -71,8 +96,8 @@ relocation to the t740 is NOT ruled or implemented here.** drill appliance. **QEMU VM `300` = `drill-r50`**, a nested PVE-in-a-VM (8 GiB RAM, 4 vCPU cpu=host, 32 GiB local-lvm disk, OVMF/SB-off, one NIC on `vmbr0` DHCP). Installed from `felhom-pve-9.2-1-v1.25.0-nested-vm-generic-mkimage.iso` through the **real day-0** (self-register → -operator bind to scratch customer `drill-r50` → deliver → nested guest **9201** provisioned, controller -0.161.0 / agent 0.93.0 healthy). Its own break-glass root is vaulted in the hub +operator bind to scratch customer `drill-r50` → deliver → nested guest **9201** provisioned and healthy +on the then-current controller + agent). Its own break-glass root is vaulted in the hub `host_recovery/drill-r50-0a4f9a`; reach it as `root@192.168.0.176` **through `demo-hp`** (it has no key and no tailnet — it is a peer on demo-hp's LAN). It has snapshot **`r50pre`** (clean LAN-literal day-0). This VM was provisioned to unblock and run the **R-50 island-bridge empirical spike** @@ -152,6 +177,7 @@ Details, and the two hard rules that apply to any host node, in `operations/tail ## What is NOT enrolled here (deliberately) -- The **1TB NVMe** on demo-hp — see above. -- **No PBS datastore, no offsite target** on demo-hp yet; the DR tier is the N100's. +- **No PBS datastore, no offsite target** on demo-hp; the DR tier is the N100's. - **No second customer guest** beyond 9201. + +*(The 1TB NVMe was on this list until 2026-07-22. It is enrolled — see the disks table above.)* diff --git a/documentation/runbooks/workspace-CLAUDE.md b/documentation/runbooks/workspace-CLAUDE.md index 08de640..76ea98c 100644 --- a/documentation/runbooks/workspace-CLAUDE.md +++ b/documentation/runbooks/workspace-CLAUDE.md @@ -122,12 +122,22 @@ Local (this host): repos `/mnt/5_hdd/felhom.eu/git/`, build dirs `/mnt/5_hdd/felhom.eu/build/felhom-{controller,hub,agent}`, `sudo kubectl`, Go toolchain, Docker build+push to `gitea.dooplex.hu/admin/`. -| Host | Access | Use | -|---|---|---| -| **DooPlex (this host)** | local — Debian 13, `kisfenyo`, `/mnt/5_hdd/felhom.eu/` | build/push images, `sudo kubectl`, build+run the agent for tests | -| Demo Proxmox host `demo-felhom` | `ssh felhom-pve` (root@192.168.0.162, no sudo) | pveum/pct + live Proxmox validation | -| Demo guest 9201 | `ssh felhom-pve "pct exec 9201 -- ..."` | the live demo controller | -| felhotest (legacy) | `ssh -p 33022 kisfenyo@router.abonet.hu` | OLD /opt/docker compose mechanism | +| Host | Access | Use | Blast radius | +|---|---|---|---| +| **DooPlex (this host)** | local — Debian 13, `kisfenyo`, `/mnt/5_hdd/felhom.eu/` | build/push images, `sudo kubectl`, build+run the agent for tests | **Tier 2 — precious.** It *is* the recovery chain (hub, Gitea, registry, PBS, k3s+Longhorn). **Never a drill target** | +| Demo Proxmox host `demo-hp` (HP t740) | `ssh demo-hp` (tailnet `100.76.96.79`; **no baked key** — G1 break-glass password vaulted in the hub) | **the designated drill + build VM host** (operator ruling 2026-07-25) | **Tier 0 — disposable. Reach here first** | +| Demo Proxmox host `demo-felhom` (N100) | `ssh felhom-pve` (root, no sudo; tailnet `100.70.170.35`) | pveum/pct + live Proxmox validation | **Tier 0 — disposable** | +| Demo guest 9201 | `ssh felhom-pve "pct exec 9201 -- ..."` | the live demo controller | Tier 0 (rides its host) | +| felhotest (legacy) | `ssh -p 33022 kisfenyo@router.abonet.hu` — **`Connection refused` 2026-07-30** | OLD /opt/docker compose mechanism | untiered — assume nothing | + +**Which box do I break?** → **`felhom.eu/documentation/runbooks/target-selection.md`** — the tiers, and +per machine what is freely permitted / needs care / forbidden, each with its reason. Read it before +picking a machine for a drill, a destructive test or a throwaway VM. **A task that needs a victim names +one; an absent fence is not permission.** + +**Component versions are not recorded in any inventory doc** — agent/controller/hub versions change +several times a day and the fleet is not uniform. Ask the hub's `/hosts` + `/configs`, or +`felhom-agent --version` / `pct exec -- docker ps` on the box. The demo Proxmox host key changes on reprovision (N100) → refresh with `ssh-keygen -R 192.168.0.162` then connect with `-o StrictHostKeyChecking=accept-new`