Fleet node inventory — the physical demo/lab hosts

Added 2026-07-21, when the fleet stopped being one box. Two Proxmox hosts now check in to the hub. This is the operator-facing inventory: what the hardware is, how to reach it, and what is deliberately NOT enrolled on it.

The fleet

	`demo-felhom-8363b5`	`demo-hp-bb76ea`
Hardware	N100 mini-PC	HP t740 Thin Client (s/n `8CN944035T`)
CPU / RAM	Intel N100	AMD Ryzen Embedded V1756B, 8 threads / 30 GiB
Firmware	AMI AN3PLUS-class	AMI M42 v01.10 (11/11/2020)
PVE node name	`demo-felhom`	`felhom-host`
Customer	`demo-felhom`	`demo-hp`
Agent	0.92.1	0.92.1
SSH alias	`felhom-pve`	`demo-hp`
Tailnet	`100.70.170.35`	`100.76.96.79`
Loader used to install	`mkimage` (unsigned, SB off — firmware workaround)	`shim`, Secure Boot ENABLED

Both are at the VACATION site and travel home with the rest of the kit ~2026-08-02. While away, their LAN addresses are on that site's 192.168.0.0/24: N100 .147/.162, HP .87. The tailnet addresses are the stable ones — use those.

demo-hp — the HP t740, in detail

Disks

device	model	serial	role
`sda`	SanDisk X600 M.2 2280 SATA 128GB	`182195804614`	system disk — PVE, LVM (`pve-root` 39.6G, `pve-data` thin pool, guest 9201's three volumes)
`nvme0n1`	KXG50PNV1T02 NVMe TOSHIBA 1024GB	`58BS11AFT8MQ`	PRESENT AND UNENROLLED — do not touch

The NVMe is a future Tárhely candidate and is deliberately outside everything. It still holds its previous NTFS partition, is not mounted, and appears in no LVM PV and no ZFS pool. The install ISO's exact-serial filter pinned sda only, which is why it survived. It joins the product later through the normal Tárhely (storage) flow — never through the installer, and never by hand. Anything that formats it before that decision is a mistake.

NIC map — and the trap

This board has five wired interfaces, and the obvious one is the wrong one:

interface	MAC	driver	what it is	state at this site
`enp1s0f0`	`a0:36:9f:5d:07:20`	`igb`	4-port expansion card	no carrier, no DHCP
`enp1s0f1`	`a0:36:9f:5d:07:21`	`igb`	″	no carrier
`enp1s0f2`	`a0:36:9f:5d:07:22`	`igb`	″	no carrier
`enp1s0f3`	`a0:36:9f:5d:07:23`	`igb`	″	no carrier
`enp2s0f0`	`7c:d3:0a:77:d9:76`	`r8169`	onboard port — the one that works	carrier up, 1000 Mb, this is `vmbr0`'s port
`wlo1`	`24:ee:9a:e5:05:b0`	`iwlwifi`	wifi	unused

This trap cost the first install. The 4-port card got no lease, and instead of aborting the installer baked its 192.168.100.2 fallback as a static vmbr0 address and completed — a box that looked installed and could never call home. Repaired on the console by bridging vmbr0 to enp2s0f0. Filed as R-59 (must hard-abort) and R-60 (first-boot NIC sweep self-heal).

Current, post-repair: vmbr0 static 192.168.0.87/24, gw 192.168.0.1, bridge-port enp2s0f0. No trace of 192.168.100.2 remains. wg-felhom 10.77.0.3/32 up to the hub. Guest 9201 demo-hp running. Agent 0.93.0 (behind demo-felhom's 0.95.0 — publish when convenient).

Designated drill + build VM host (operator ruling, 2026-07-25)

Ruling: drill and build VMs are hosted on the t740 from now on — NOT on felhom-pve, and moving them off DooPlex (the production k3s node). This is a VM-HOSTING ruling only; the build-PIPELINE relocation to the t740 is NOT ruled or implemented here.

Current state (verified 2026-07-25): the ruling is forward-looking and not yet realized. The t740 hosts no drill/build VM yet — qm list is empty, /etc/pve/qemu-server/ is empty, it is a standalone PVE node running only its own LXC guest 9201. The historical drill appliance is still drill.qcow2 on DooPlex (/mnt/5_hdd/felhom.eu/drill/, ~18G, powered off — a golden-bake VM whose nested guest is purged after each bake). felhom-pve likewise has no drill VM. A nested-PVE drill VM (agent + a nested guest) must be provisioned on the t740 before the R-50 island-bridge empirical spike (audits/SPIKE-island-bridge-2026-07-25.md) can run its P2–P7 probes — that spike is blocked on exactly this. The t740's ~30 GB RAM / ~49 GB free local-lvm suit it as the drill host.

Access — there is no baked SSH key

ssh demo-hp resolves to the tailnet address, but no operator public key is on this box — the HP profile deliberately left FELHOM_ROOT_SSH_KEY blank. Authentication is the G1 break-glass root password vaulted in the hub, host_recovery row demo-hp-bb76ea (set at day-0, 2026-07-21 16:24 UTC).

Retrieval (operator-side, and shred the copy — that DB holds every host's secret):

sudo kubectl -n felhom-system exec <hub-pod> -- cat /data/hub.db > /tmp/x.db
python3 -c "import sqlite3;print(sqlite3.connect('/tmp/x.db').execute(
  \"SELECT secret FROM host_recovery WHERE host_id='demo-hp-bb76ea'\").fetchone()[0])"
shred -u /tmp/x.db

Then sshpass -e ssh root@demo-hp (sshpass is on DooPlex, not on the nodes).

This is the lockout filed as R-61: the ISO mints a throwaway root password per build and discards the plaintext, so the console is unreachable without a working hub and network — precisely what you may be trying to fix. Slice 1 is to emit the baked password into the build report.

demo-hp-lan (192.168.0.87 via ProxyJump felhom-pve) is the fallback while the box is away.

OOB belt (H1) — both boxes, since 2026-07-23 (ISO train v1.25.0)

The dedicated OOB sshd belt (TASK H1: felhom-sshd + the static inet felhom_oob table + felhom-op) is installed and active on BOTH fleet boxes — the F9 gap (belt on neither) is closed. From v1.25.0 host-install installs it by default on every appliance install (--no-oob opts out; byo still refuses).

Claimed port: 8822 on both (first-free from [8822,2222,8022,62222]; persisted per box).
Reachability: the wg-felhom offsite tunnel ONLY — the belt admits the operator /32 (10.77.0.250) over wg-felhom to 8822 and drops everything else; :22 and every other interface are untouched. tailscale does NOT reach the belt (wrong fabric, dropped by design).
Operator login (from the machine holding the wg-felhom operator tunnel + the registered oob_operator_ssh_pubkey): ssh -p 8822 felhom-op@10.77.0.2 (felhom-pve) / @10.77.0.3 (demo-hp). PROVEN live 2026-07-23 on felhom-pve (felhom-op@demo-felhom).
Operator tunnel: the Mac/Windows operator peer dials ep0.felhom.eu:443 (WireGuard), address 10.77.0.250/32, AllowedIPs 10.77.0.0/24, server pubkey f3d1ZI7…. ep0's forward chain (persisted in its /etc/nftables.conf) allows 10.77.0.250 → 10.77.0.2/.3. If a work-network blocks UDP/443, the RheinMetall-style firewalls pass UDP/51820 — a home/hotspot network works on 443.
Register/rotate the operator identity hub-side: PUT /api/v1/admin/wg/operator-peer (global key) with {pubkey, assigned_ip:"10.77.0.250", ssh_pubkey}; wgsync pushes it to ep0 and the SSH key flows to both boxes' felhom-op authorized_keys within a tick.

felhom-pve (the N100) — vault parity + access

felhom-pve has operator SSH-key access (over tailscale 100.70.170.35) AND, since 2026-07-23, G1 break-glass vault parity with demo-hp: its root@pam password is freshly rotated and vaulted in the hub host_recovery row demo-felhom-8363b5 (same PUT …/recovery-credential mechanism day-0 uses; verified retrievable + authenticating over :22). Retrieval + shred-the-copy recipe is identical to demo-hp's below (swap the host_id). So a lost N100 key is recoverable the same way as the key-less HP.

Tailscale on demo-hp is an OPERATOR-LAB EXCEPTION

Read this before any product-shape audit. demo-hp is customer-shaped — it is a normal appliance install with a real customer record (demo-hp), a real guest, and a real day-0. Tailscale is not part of that shape. It was installed by hand on 2026-07-21 purely so the operator can reach a lab box that lives on someone else's LAN.

Real customer boxes never get tailscale. Their operator access is the WireGuard tunnel plus the H1 OOB path, and nothing else. If a future audit finds tailscale on demo-hp and concludes the product ships it — that conclusion is wrong, and this paragraph is the reason. The same exception already applies to felhom-pve.

Details, and the two hard rules that apply to any host node, in operations/tailscale.md.

What is NOT enrolled here (deliberately)

The 1TB NVMe on demo-hp — see above.
No PBS datastore, no offsite target on demo-hp yet; the DR tier is the N100's.
No second customer guest beyond 9201.

8.8 KiB Raw Blame History Unescape Escape