felhom.eu/documentation/operations/nodes.md

# Fleet node inventory — the physical demo/lab hosts

> Added 2026-07-21, when the fleet stopped being one box. Two Proxmox hosts now check in to the hub.
> This is the operator-facing inventory: what the hardware is, how to reach it, and what is
> deliberately NOT enrolled on it.

## The fleet

| | `demo-felhom-8363b5` | `demo-hp-bb76ea` |
|---|---|---|
| Hardware | N100 mini-PC | **HP t740 Thin Client** (s/n `8CN944035T`) |
| CPU / RAM | Intel N100 | **AMD Ryzen Embedded V1756B**, 8 threads / **30 GiB** |
| Firmware | AMI AN3PLUS-class | **AMI M42 v01.10 (11/11/2020)** |
| PVE node name | `demo-felhom` | `felhom-host` |
| Customer | `demo-felhom` | `demo-hp` |
| Agent | 0.92.1 | 0.92.1 |
| SSH alias | `felhom-pve` | **`demo-hp`** |
| Tailnet | `100.70.170.35` | **`100.76.96.79`** |
| Loader used to install | `mkimage` (unsigned, SB **off** — firmware workaround) | **`shim`, Secure Boot ENABLED** |

**Both are at the VACATION site** and travel home with the rest of the kit **~2026-08-02**. While
away, their LAN addresses are on that site's `192.168.0.0/24`: N100 `.147`/`.162`, HP `.87`. The
tailnet addresses are the stable ones — use those.

## demo-hp — the HP t740, in detail

### Disks

| device | model | serial | role |
|---|---|---|---|
| `sda` | SanDisk X600 M.2 2280 SATA 128GB | `182195804614` | **system disk** — PVE, LVM (`pve-root` 39.6G, `pve-data` thin pool, guest 9201's three volumes) |
| `nvme0n1` | KXG50PNV1T02 NVMe TOSHIBA 1024GB | `58BS11AFT8MQ` | **PRESENT AND UNENROLLED — do not touch** |

> **The NVMe is a future Tárhely candidate and is deliberately outside everything.** It still holds
> its previous **NTFS** partition, is not mounted, and appears in no LVM PV and no ZFS pool. The
> install ISO's exact-serial filter pinned `sda` only, which is why it survived. **It joins the
> product later through the normal Tárhely (storage) flow — never through the installer, and never
> by hand.** Anything that formats it before that decision is a mistake.

### NIC map — and the trap

This board has **five wired interfaces**, and the obvious one is the wrong one:

| interface | MAC | driver | what it is | state at this site |
|---|---|---|---|---|
| `enp1s0f0` | `a0:36:9f:5d:07:20` | `igb` | 4-port expansion card | no carrier, **no DHCP** |
| `enp1s0f1` | `a0:36:9f:5d:07:21` | `igb` | ″ | no carrier |
| `enp1s0f2` | `a0:36:9f:5d:07:22` | `igb` | ″ | no carrier |
| `enp1s0f3` | `a0:36:9f:5d:07:23` | `igb` | ″ | no carrier |
| **`enp2s0f0`** | `7c:d3:0a:77:d9:76` | `r8169` | **onboard port — the one that works** | carrier up, 1000 Mb, **this is `vmbr0`'s port** |
| `wlo1` | `24:ee:9a:e5:05:b0` | `iwlwifi` | wifi | unused |

**This trap cost the first install.** The 4-port card got no lease, and instead of aborting the
installer baked its `192.168.100.2` fallback as a **static** `vmbr0` address and completed — a box
that looked installed and could never call home. Repaired on the console by bridging `vmbr0` to
`enp2s0f0`. Filed as **R-59** (must hard-abort) and **R-60** (first-boot NIC sweep self-heal).

Current, post-repair: `vmbr0` static `192.168.0.87/24`, gw `192.168.0.1`, bridge-port `enp2s0f0`.
No trace of `192.168.100.2` remains. `wg-felhom` `10.77.0.3/32` up to the hub. Guest **9201
`demo-hp`** running.

### Access — there is no baked SSH key

`ssh demo-hp` resolves to the tailnet address, but **no operator public key is on this box** — the
HP profile deliberately left `FELHOM_ROOT_SSH_KEY` blank. Authentication is the **G1 break-glass
root password vaulted in the hub**, `host_recovery` row `demo-hp-bb76ea` (set at day-0,
2026-07-21 16:24 UTC).

Retrieval (operator-side, and **shred the copy** — that DB holds every host's secret):

```bash
sudo kubectl -n felhom-system exec <hub-pod> -- cat /data/hub.db > /tmp/x.db
python3 -c "import sqlite3;print(sqlite3.connect('/tmp/x.db').execute(
  \"SELECT secret FROM host_recovery WHERE host_id='demo-hp-bb76ea'\").fetchone()[0])"
shred -u /tmp/x.db
```

Then `sshpass -e ssh root@demo-hp` (sshpass is on DooPlex, not on the nodes).

**This is the lockout filed as R-61**: the ISO mints a throwaway root password per build and discards
the plaintext, so the console is unreachable without a working hub and network — precisely what you
may be trying to fix. Slice 1 is to emit the baked password into the build report.

`demo-hp-lan` (`192.168.0.87` via `ProxyJump felhom-pve`) is the fallback while the box is away.

## OOB belt (H1) — both boxes, since 2026-07-23 (ISO train v1.25.0)

The dedicated OOB sshd belt (TASK H1: `felhom-sshd` + the static `inet felhom_oob` table + `felhom-op`)
is installed and **active on BOTH fleet boxes** — the F9 gap (belt on neither) is closed. From
v1.25.0 host-install installs it by default on every appliance install (`--no-oob` opts out; byo still
refuses).

- **Claimed port: `8822` on both** (first-free from `[8822,2222,8022,62222]`; persisted per box).
- **Reachability: the wg-felhom offsite tunnel ONLY** — the belt admits the operator `/32`
  (`10.77.0.250`) over `wg-felhom` to 8822 and drops everything else; `:22` and every other interface
  are untouched. **tailscale does NOT reach the belt** (wrong fabric, dropped by design).
- **Operator login** (from the machine holding the wg-felhom operator tunnel + the registered
  `oob_operator_ssh_pubkey`): `ssh -p 8822 felhom-op@10.77.0.2` (felhom-pve) / `@10.77.0.3` (demo-hp).
  **PROVEN live 2026-07-23** on felhom-pve (`felhom-op@demo-felhom`).
- **Operator tunnel**: the Mac/Windows operator peer dials `ep0.felhom.eu:443` (WireGuard), address
  `10.77.0.250/32`, AllowedIPs `10.77.0.0/24`, server pubkey `f3d1ZI7…`. ep0's `forward` chain
  (persisted in its `/etc/nftables.conf`) allows `10.77.0.250 → 10.77.0.2/.3`. If a work-network blocks
  UDP/443, the RheinMetall-style firewalls pass UDP/51820 — a home/hotspot network works on 443.
- Register/rotate the operator identity hub-side: `PUT /api/v1/admin/wg/operator-peer` (global key)
  with `{pubkey, assigned_ip:"10.77.0.250", ssh_pubkey}`; wgsync pushes it to ep0 and the SSH key flows
  to both boxes' `felhom-op` authorized_keys within a tick.

## felhom-pve (the N100) — vault parity + access

felhom-pve has **operator SSH-key access** (over tailscale `100.70.170.35`) AND, since 2026-07-23,
**G1 break-glass vault parity with demo-hp**: its root@pam password is freshly rotated and vaulted in
the hub `host_recovery` row **`demo-felhom-8363b5`** (same PUT `…/recovery-credential` mechanism day-0
uses; verified retrievable + authenticating over `:22`). Retrieval + shred-the-copy recipe is identical
to demo-hp's below (swap the host_id). So a lost N100 key is recoverable the same way as the key-less HP.

## Tailscale on demo-hp is an OPERATOR-LAB EXCEPTION

> **Read this before any product-shape audit.** `demo-hp` is **customer-shaped** — it is a normal
> appliance install with a real customer record (`demo-hp`), a real guest, and a real day-0.
> **Tailscale is not part of that shape.** It was installed by hand on 2026-07-21 purely so the
> operator can reach a lab box that lives on someone else's LAN.
>
> **Real customer boxes never get tailscale.** Their operator access is the WireGuard tunnel plus
> the H1 OOB path, and nothing else. If a future audit finds tailscale on `demo-hp` and concludes
> the product ships it — that conclusion is wrong, and this paragraph is the reason. The same
> exception already applies to `felhom-pve`.

Details, and the two hard rules that apply to any host node, in `operations/tailscale.md`.

## What is NOT enrolled here (deliberately)

- The **1TB NVMe** on demo-hp — see above.
- **No PBS datastore, no offsite target** on demo-hp yet; the DR tier is the N100's.
- **No second customer guest** beyond 9201.