From 8ec45f3b42b9dbeea187029522024a49c0eda1ad Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 21 Jul 2026 18:51:05 +0200 Subject: [PATCH] docs: node inventory for the two-host fleet; demo-hp on the tailnet New documentation/operations/nodes.md: HP t740 hardware, disks (SanDisk system SSD 182195804614; Toshiba 1TB NVMe 58BS11AFT8MQ PRESENT AND UNENROLLED, still NTFS, do not touch), the five-NIC map and the trap that cost the first install, and the access path - no SSH key is baked, auth is the hub-vaulted G1 break-glass password. tailscale.md gains demo-hp, the operator-lab-exception warning so a future product-shape audit does not conclude the product ships tailscale, and the --accept-dns evidence: the join omitted the flag, MagicDNS rewrote /etc/resolv.conf, and it was reverted. Harmless at the vacation site, would have bitten silently at home where split-horizon matters. OPEN: key expiry still enabled on demo-hp (2027-01-17) - needs an admin-console toggle or an API token; a pre-auth key cannot do it. --- CONTEXT.md | 17 +++++ documentation/operations/nodes.md | 104 ++++++++++++++++++++++++++ documentation/operations/tailscale.md | 39 +++++++++- 3 files changed, 158 insertions(+), 2 deletions(-) create mode 100644 documentation/operations/nodes.md diff --git a/CONTEXT.md b/CONTEXT.md index ba04946..58fb01c 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,6 +3,23 @@ > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. +- **2026-07-21 — THE FLEET IS TWO HOSTS.** A second Proxmox node exists: **`demo-hp-bb76ea`**, an + **HP t740 Thin Client** (Ryzen V1756B, 30 GiB, PVE node `felhom-host`, customer `demo-hp`), installed + from the armed universal ISO the same day. Both hosts run **agent 0.92.1**; the N100 is + `demo-felhom-8363b5`. **Full inventory: `documentation/operations/nodes.md`** — read it before + touching the HP, because two things there are load-bearing: the box's **1TB Toshiba NVMe is present + and UNENROLLED** (still carrying its old NTFS partition; a future Tárhely candidate that must join + through the storage flow, never the installer), and there is **no operator SSH key baked** on it, so + access is the hub-vaulted G1 break-glass password (R-61 is the fix for that lockout). + **Both nodes are at the VACATION site and travel home ~2026-08-02.** `demo-hp` joined the tailnet as + `100.76.96.79` — an **operator-lab exception**, not product shape; real customer boxes never get + tailscale. **Second-hardware proof, worth keeping:** the pairing/day-0 chain ran end to end on virgin + hardware it had never seen, the **shim loader booted with Secure Boot ENABLED** (so SB-off was an + N100-firmware workaround, not a Felhom requirement), and the box **self-lifted the floor + 0.153.0 → 0.156.0 during day-0**. The install was not clean, and the failures are filed: **R-59** + (no DHCP on the 4-port NIC → the installer baked a static fallback instead of aborting), **R-60** + (first-boot NIC sweep self-heal), **R-61** (baked root password unknowable). + - **2026-07-19 — N100 MADE LOCATION-INDEPENDENT via Tailscale.** `felhom-pve` (the N100) joined the tailnet as a host package (`100.70.170.35`, expiry disabled); DooPlex already advertised `192.168.0.0/24` via its GitOps k3s `admin-system/tailscale` pod (`100.107.87.53`). `ssh felhom-pve` diff --git a/documentation/operations/nodes.md b/documentation/operations/nodes.md new file mode 100644 index 0000000..5d8876b --- /dev/null +++ b/documentation/operations/nodes.md @@ -0,0 +1,104 @@ +# Fleet node inventory — the physical demo/lab hosts + +> Added 2026-07-21, when the fleet stopped being one box. Two Proxmox hosts now check in to the hub. +> This is the operator-facing inventory: what the hardware is, how to reach it, and what is +> deliberately NOT enrolled on it. + +## The fleet + +| | `demo-felhom-8363b5` | `demo-hp-bb76ea` | +|---|---|---| +| Hardware | N100 mini-PC | **HP t740 Thin Client** (s/n `8CN944035T`) | +| CPU / RAM | Intel N100 | **AMD Ryzen Embedded V1756B**, 8 threads / **30 GiB** | +| Firmware | AMI AN3PLUS-class | **AMI M42 v01.10 (11/11/2020)** | +| PVE node name | `demo-felhom` | `felhom-host` | +| Customer | `demo-felhom` | `demo-hp` | +| Agent | 0.92.1 | 0.92.1 | +| SSH alias | `felhom-pve` | **`demo-hp`** | +| Tailnet | `100.70.170.35` | **`100.76.96.79`** | +| Loader used to install | `mkimage` (unsigned, SB **off** — firmware workaround) | **`shim`, Secure Boot ENABLED** | + +**Both are at the VACATION site** and travel home with the rest of the kit **~2026-08-02**. While +away, their LAN addresses are on that site's `192.168.0.0/24`: N100 `.147`/`.162`, HP `.87`. The +tailnet addresses are the stable ones — use those. + +## demo-hp — the HP t740, in detail + +### Disks + +| device | model | serial | role | +|---|---|---|---| +| `sda` | SanDisk X600 M.2 2280 SATA 128GB | `182195804614` | **system disk** — PVE, LVM (`pve-root` 39.6G, `pve-data` thin pool, guest 9201's three volumes) | +| `nvme0n1` | KXG50PNV1T02 NVMe TOSHIBA 1024GB | `58BS11AFT8MQ` | **PRESENT AND UNENROLLED — do not touch** | + +> **The NVMe is a future Tárhely candidate and is deliberately outside everything.** It still holds +> its previous **NTFS** partition, is not mounted, and appears in no LVM PV and no ZFS pool. The +> install ISO's exact-serial filter pinned `sda` only, which is why it survived. **It joins the +> product later through the normal Tárhely (storage) flow — never through the installer, and never +> by hand.** Anything that formats it before that decision is a mistake. + +### NIC map — and the trap + +This board has **five wired interfaces**, and the obvious one is the wrong one: + +| interface | MAC | driver | what it is | state at this site | +|---|---|---|---|---| +| `enp1s0f0` | `a0:36:9f:5d:07:20` | `igb` | 4-port expansion card | no carrier, **no DHCP** | +| `enp1s0f1` | `a0:36:9f:5d:07:21` | `igb` | ″ | no carrier | +| `enp1s0f2` | `a0:36:9f:5d:07:22` | `igb` | ″ | no carrier | +| `enp1s0f3` | `a0:36:9f:5d:07:23` | `igb` | ″ | no carrier | +| **`enp2s0f0`** | `7c:d3:0a:77:d9:76` | `r8169` | **onboard port — the one that works** | carrier up, 1000 Mb, **this is `vmbr0`'s port** | +| `wlo1` | `24:ee:9a:e5:05:b0` | `iwlwifi` | wifi | unused | + +**This trap cost the first install.** The 4-port card got no lease, and instead of aborting the +installer baked its `192.168.100.2` fallback as a **static** `vmbr0` address and completed — a box +that looked installed and could never call home. Repaired on the console by bridging `vmbr0` to +`enp2s0f0`. Filed as **R-59** (must hard-abort) and **R-60** (first-boot NIC sweep self-heal). + +Current, post-repair: `vmbr0` static `192.168.0.87/24`, gw `192.168.0.1`, bridge-port `enp2s0f0`. +No trace of `192.168.100.2` remains. `wg-felhom` `10.77.0.3/32` up to the hub. Guest **9201 +`demo-hp`** running. + +### Access — there is no baked SSH key + +`ssh demo-hp` resolves to the tailnet address, but **no operator public key is on this box** — the +HP profile deliberately left `FELHOM_ROOT_SSH_KEY` blank. Authentication is the **G1 break-glass +root password vaulted in the hub**, `host_recovery` row `demo-hp-bb76ea` (set at day-0, +2026-07-21 16:24 UTC). + +Retrieval (operator-side, and **shred the copy** — that DB holds every host's secret): + +```bash +sudo kubectl -n felhom-system exec -- cat /data/hub.db > /tmp/x.db +python3 -c "import sqlite3;print(sqlite3.connect('/tmp/x.db').execute( + \"SELECT secret FROM host_recovery WHERE host_id='demo-hp-bb76ea'\").fetchone()[0])" +shred -u /tmp/x.db +``` + +Then `sshpass -e ssh root@demo-hp` (sshpass is on DooPlex, not on the nodes). + +**This is the lockout filed as R-61**: the ISO mints a throwaway root password per build and discards +the plaintext, so the console is unreachable without a working hub and network — precisely what you +may be trying to fix. Slice 1 is to emit the baked password into the build report. + +`demo-hp-lan` (`192.168.0.87` via `ProxyJump felhom-pve`) is the fallback while the box is away. + +## Tailscale on demo-hp is an OPERATOR-LAB EXCEPTION + +> **Read this before any product-shape audit.** `demo-hp` is **customer-shaped** — it is a normal +> appliance install with a real customer record (`demo-hp`), a real guest, and a real day-0. +> **Tailscale is not part of that shape.** It was installed by hand on 2026-07-21 purely so the +> operator can reach a lab box that lives on someone else's LAN. +> +> **Real customer boxes never get tailscale.** Their operator access is the WireGuard tunnel plus +> the H1 OOB path, and nothing else. If a future audit finds tailscale on `demo-hp` and concludes +> the product ships it — that conclusion is wrong, and this paragraph is the reason. The same +> exception already applies to `felhom-pve`. + +Details, and the two hard rules that apply to any host node, in `operations/tailscale.md`. + +## What is NOT enrolled here (deliberately) + +- The **1TB NVMe** on demo-hp — see above. +- **No PBS datastore, no offsite target** on demo-hp yet; the DR tier is the N100's. +- **No second customer guest** beyond 9201. diff --git a/documentation/operations/tailscale.md b/documentation/operations/tailscale.md index eb4f144..5d7eda2 100644 --- a/documentation/operations/tailscale.md +++ b/documentation/operations/tailscale.md @@ -1,4 +1,4 @@ -# Tailscale — location-independent access to felhom-pve (the N100) +# Tailscale — location-independent access to the demo nodes (N100 + HP t740) > Added 2026-07-19. Goal: the demo Proxmox host (N100, `felhom-pve`) can be taken to a different > LAN (e.g. a vacation home) and stay reachable from DooPlex, while its own dependencies keep @@ -10,6 +10,7 @@ |---|---|---|---|---| | DooPlex (192.168.0.180) | `dooplex` | `100.107.87.53` | **subnet router** advertising `192.168.0.0/24`; expiry disabled | **k3s pod** `admin-system/tailscale` (`hostNetwork`, GitOps/ArgoCD) — 157 d old, predates this task | | N100 (192.168.0.162) | `felhom-pve` | `100.70.170.35` | its own node; expiry disabled | **host package** `tailscale` (Debian trixie apt repo), `tailscaled.service` | +| HP t740 (`demo-hp-bb76ea`) | `demo-hp` | `100.76.96.79` | its own node; **expiry NOT yet disabled — see below** | **host package** `tailscale` 1.98.9 (Debian trixie apt repo), `tailscaled.service`, added 2026-07-21 | Tailnet: `nagyfenyvesi.viktor@gmail.com` (Tailscale Free). Both nodes run v1.98.x. @@ -18,11 +19,45 @@ the **direct LAN path** when both boxes are home (measured: `pong … via 192.16 and falls back to the tunnel / a DERP relay when the N100 is remote. `felhom-pve-lan` → `192.168.0.162` is the explicit LAN-only fallback. +## This is an OPERATOR-LAB EXCEPTION, not product shape + +**Real customer boxes never get tailscale.** Customer operator-access is the WireGuard tunnel plus +the H1 OOB path, full stop. Tailscale exists on `felhom-pve` and `demo-hp` only because those two are +lab machines that live on LANs the operator does not control. + +`demo-hp` is the confusing one: it is **customer-shaped** — a normal appliance install, a real +customer record, a real guest, a real day-0 — with tailscale bolted on afterwards by hand. A future +product-shape audit that finds it there must not conclude the product ships it. See +`operations/nodes.md`. + +## demo-hp joined 2026-07-21 + +`tailscale up --authkey=file:… --hostname=demo-hp --accept-routes=false`, then +`tailscale set --accept-dns=false` (see the hard rule below — it was needed, and why is worth +reading). Auth was a pre-auth key, passed as `--authkey=file:` and shredded immediately so it +never appeared in the box's process list. Verified from DooPlex: `ping` ~40 ms, direct SSH with no +ProxyJump, and the path is **direct** (`37.191.56.193:45127`), not a DERP relay. + +**OPEN: key expiry is still enabled on `demo-hp`** (expires `2027-01-17`), unlike the other two +nodes which have it disabled. Disabling it is a per-device **admin-console toggle** (Machines → +demo-hp → Disable key expiry) or an API call with a `tskey-api-…` token; a `tskey-auth-…` pre-auth +key cannot do it (verified: the API returns 401). Until it is toggled the node will drop off the +tailnet on that date and need re-authentication. + ## Hard rule: `--accept-dns=false` on the host node The N100's `tailscaled` runs with **`--accept-dns=false`** (`CorpDNS:false`). MagicDNS must never rewrite `/etc/resolv.conf`: the box relies on pi-hole (`192.168.0.250`) for split-horizon -(`gitea.dooplex.hu` → `192.168.0.180`). Verified byte-identical after `tailscale up`. (DooPlex's +(`gitea.dooplex.hu` → `192.168.0.180`). Verified byte-identical after `tailscale up`. + +**On `demo-hp` this rule was broken and then fixed, which is the useful part of the story.** The join +omitted `--accept-dns=false`, and MagicDNS promptly rewrote `/etc/resolv.conf` to +`nameserver 100.100.100.100` — exactly what this rule forbids. Nothing broke *at the vacation site*, +because there is no pi-hole there and both `gitea.dooplex.hu` and `hub.felhom.eu` resolve publicly +anyway; the damage would have appeared silently **when the box comes home**, where split-horizon is +what makes `gitea.dooplex.hu` → `192.168.0.180` local. `tailscale set --accept-dns=false` restored +`nameserver 192.168.0.1` immediately, with the tailnet and the agent unaffected. **Pass +`--accept-dns=false` at join time on any host node; do not rely on noticing afterwards.** (DooPlex's containerised node runs `TS_ACCEPT_DNS=true`, but because it is a pod with its own mount namespace that only rewrites the *container's* resolv.conf — the DooPlex host resolv.conf is untouched.)