From 914d51aa5f76128746c2b7d5646f9b5201d8deb8 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Thu, 16 Jul 2026 11:29:03 +0200 Subject: [PATCH] =?UTF-8?q?docs(spike):=20bare-metal=20Felhom=20ISO=20?= =?UTF-8?q?=E2=80=94=20R-21=20spiked=20(all=20mechanisms=20GREEN=20on=20ne?= =?UTF-8?q?sted=20virt)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Empirical spike of the Proxmox automated-installation path for R-21 (bare-metal customer-hardware ISO). All legs proven on nested VM 310 (felhom-pve): - S1 assistant 9.2.7 + ISO provenance; validate-answer exits 0 on failure (parse output) - S2/S2b zero-touch install SeaBIOS + UEFI incl. Secure Boot enforcing (no keypress/MOK) - S3 first-boot hook: fully-up = root + working pvesh/pct, exactly-once - S4 webhook payload enumerated (SMBIOS-UUID + MAC + host SSH keys) - S5 disk-filter installs only target (canary byte-identical), match-nothing fails safe - S6 from-url + cert-fingerprint fails CLOSED (red-proof) - S7 http answer-fetch identity; S8 no-net + invalid-answer failure modes Findings doc + ROADMAP R-21 status=spiked. No production code. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_017qDiBqKKQ5vPB5fXBqu7Kp --- .../audits/SPIKE-baremetal-iso-2026-07-16.md | 246 ++++++++++++++++++ documentation/backlog/ROADMAP.md | 1 + 2 files changed, 247 insertions(+) create mode 100644 documentation/audits/SPIKE-baremetal-iso-2026-07-16.md diff --git a/documentation/audits/SPIKE-baremetal-iso-2026-07-16.md b/documentation/audits/SPIKE-baremetal-iso-2026-07-16.md new file mode 100644 index 0000000..42893c6 --- /dev/null +++ b/documentation/audits/SPIKE-baremetal-iso-2026-07-16.md @@ -0,0 +1,246 @@ +# SPIKE — bare-metal Felhom ISO (Proxmox automated-installation mechanics), 2026-07-16 + +**Class:** spike (empirical mechanism validation before any production spec). **Output:** this findings +doc. **No production code shipped.** The only repo write is this file + one ROADMAP line (R-21 → spiked). + +**Verdict in one line:** every mechanism the bare-metal path depends on — the +`proxmox-auto-install-assistant` toolchain, zero-touch install (BIOS *and* UEFI, incl. Secure Boot), +the first-boot hook (privileges / ordering / exactly-once), the post-installation webhook, disk-filter +safety, and from-url + cert-fingerprint pinning — **works and fails-closed where it must**. R-21 is +technically green to spec on nested virt; the only residual unknowns need one real bare-metal run +(vendor DMI serials, real firmware quirks). *(The spike brief refers to this feature as "R-22"; it is +entered in the ROADMAP as **R-21** — the same item.)* + +## 0. Environment & baselines + +| Item | Value | +|---|---| +| felhom.eu | `main` @ `d202b43` (clean, confirmed at start) | +| Host (nested-VM + live validation) | felhom-pve 192.168.0.162, PVE **9.2.2**, kernel 7.0.2-6-pve; 7.5 GiB RAM free at start (≥5 GiB gate ✓) | +| Assistant / ISO build host | DooPlex 192.168.0.180 (Docker 29.2.1) | +| Spike VM | **310** `spike-baremetal-iso`, 4 GB / 2 vCPU, scsi0 32 GB (target) + scsi1 8 GB (data/canary), virtio-scsi-single; SMBIOS uuid `14a82ff6-3ab5-440d-ac40-7f9968408618`, NIC MAC `bc:24:11:98:10:0e` | +| Never touched | qm 300 (drill), 9000, LXC 9201, Peti, hub — honored | +| Throwaway password | `$6$…` crypt hash baked into answer files (spike-only; `# THROWAWAY` in the archived answer.toml) | +| Evidence root | `180:~/spike-baremetal/` (ISOs, answers, `evidence//`, listener/https logs) — kept | +| Spike VM final state | **stopped, kept** (per cleanup rule) pending acceptance | + +## S1 — toolchain + ISO provenance — **PROVEN** + +- **Assistant obtainable + pinnable per release:** `proxmox-auto-install-assistant 9.2.7` + (`proxmox-installer-common 9.2.7`) installed in a disposable `debian:trixie` Docker container on + DooPlex from the PVE **trixie** no-subscription repo. This is the intended `build-felhom-iso.sh` + host and it works. Pairing rule: install the assistant from the `pve no-subscription` + repo whose Debian codename matches the ISO generation (PVE 9.x = Debian 13 *trixie*); the assistant + point release (9.2.7) tracks the repo and pairs with the ISO by **major.minor** (9.2), not by the + ISO's point release (9.2-1). +- **ISO provenance:** `proxmox-ve_9.2-1.iso` sha256 + `4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c` — matches the official + `download.proxmox.com/iso/SHA256SUMS` **and** the copy already on felhom-pve. +- **Build cost (design input #6):** container image build 37.9 s / 243 MB; `prepare-iso` ~**11.5 s** + per prepared ISO (source local); prepared ISO 1,707,278,336 B (~1.1 MB over the 1,706,178,560 B + source). +- **`validate-answer` catches a broken file** — fed one with both `root-password` and + `root-password-hashed`: `` `global.root-password` and `global.root-password-hashed` cannot be set at + the same time / Error: Found issues in the answer file.`` **TRAP (load-bearing for the pipeline):** + it prints that to **stderr but returns exit code 0** — for *both* valid and invalid files. A + `build-felhom-iso.sh` gate **must parse the output text**, never `$?`. It is also **syntactic only**: + `disk-list=["sdz"]` (a non-existent disk) passed validation and only failed at install time (see S8b). +- Evidence: `evidence/S1-S8-key-results.txt`, `~/spike-baremetal/build/build.log`, `iso/SHA256SUMS`. + +## S2 — baseline zero-touch install (SeaBIOS) — **PROVEN** + +- **Given/When/Then:** prepared ISO (`--fetch-from iso`, keyless `answer-base.toml`) on VM 310 + (SeaBIOS, single 32 GB disk) → boot → automated install with **no keypress**, reboot into working PVE. +- Boot menu **auto-selected "Install Proxmox VE (Automated)"** with a 5 s countdown + (`evidence/S2/boot-8s.png`); unattended package extraction (`evidence/S2/mid-install.png`); **no + interactive prompt at any point**. Install wall-clock **≈ 3 min** (start 09:45:33 → booting installed + PVE by 09:48:48). +- Installed system reachable over IPv6 link-local: `22` OPEN (`SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u4`), + `8006` OPEN. TLS cert **CN=spike310.felhom.local** (configured fqdn applied); SAN carries the + **DHCP-assigned IPv4 192.168.0.115**. Boot-order `scsi0;ide2` fall-through: the post-install reboot + landed on the installed disk (no reinstall loop). +- Evidence: `evidence/S2/verdict.txt`, `boot-8s.png`, `mid-install.png`. + +## S2b — UEFI + Secure Boot — **PROVEN** (no caveat needed) + +- **UEFI (OVMF/q35, Secure Boot OFF, `pre-enrolled-keys=0`):** automated install identical, zero + keypress (`evidence/S2b/boot-uefi-installer.png`). Keyed SSH login worked; `/sys/firmware/efi` + present → **genuine UEFI boot**; `mokutil`: *SecureBoot disabled / Setup Mode*. + **Password mechanism proven non-hollow:** the installed `/etc/shadow` root hash equals the answer's + `root-password-hashed` **verbatim** → the password would work as configured. +- **Secure Boot ON (`pre-enrolled-keys=1`, MS 2023 certs, enforcing):** the PVE ISO's MS-signed shim + chain booted the automated installer (`evidence/S2b/sbon-18s.png`), the install completed, **and the + installed PVE booted under Secure Boot enforcing** — `mokutil: SecureBoot enabled`, `BOOTED_VIA=UEFI`, + lockdown=integrity, **zero keypress, no MOK-enrollment prompt**. → **R-21 needs no Secure-Boot + caveat**: customer mini-PCs with SB on by default install and boot fine. +- **Bonus finding:** the PVE installer writes a **dual BIOS+UEFI-bootable disk** — the S2 SeaBIOS + install also booted under OVMF. +- **Re-imaging gotcha (finding):** the first UEFI reinstall over a disk that still held the prior + install's **LVM PV signature** (only first/last-10 MiB wiped) **aborted**: *"unable to initialize + physical volume /dev/sda3 / low level installer returned early"*. The auto-installer's + "cleanup root-disks" step does **not** force-clear a pre-existing LVM PV. A full `blkdiscard` then + installed cleanly. → a customer disk with a prior LVM/Proxmox/Linux install may need an explicit + wipe before the automated install. + +## S3 — first-boot hook: privileges, ordering, exactly-once — **PROVEN** + +- **`--on-first-boot stub.sh`, `[first-boot] source="from-iso" ordering="fully-up"`.** Unit is + **`proxmox-first-boot-multi-user.service`** ("Fully Booted"), + `After=… pveproxy.service pmgproxy.service proxmox-backup-proxy.service`. +- **Runs as root (uid 0).** Under **fully-up**, `pvesh get /nodes` returned a **complete** node object + (status online, ssl_fingerprint present, uptime) `pvesh_rc=0`; `pct list` `pct_rc=0`; outbound + `curl` hit the DooPlex listener — `GET /firstboot?host=spike310&uid=0 from 192.168.0.115` (network + truly up). This is the ordering the real wrapper needs (it drives pveum/pct/pvesh). +- **Exactly-once mechanism:** unit has `ConditionPathExists=/var/lib/proxmox-first-boot/pending-first-boot-setup` + and `ExecStartPost=/usr/bin/rm … pending-first-boot-setup`. Empirically: after a reboot the hook log + is **byte-identical** (16 lines, 1 marker, sha256 unchanged), the unit is `inactive(dead)` with + *"ConditionPathExists … was not met"* (`ConditionResult=no`), and the listener shows **exactly 1** + firstboot hit. +- **Comparison leg — `ordering="network-online"`** (`proxmox-first-boot-network-online.service`): + pvesh returned **degraded** data — `'/etc/pve/nodes/spike310/pve-ssl.pem' does not exist!`, + `status:"unknown"`, `ssl_fingerprint:null` (pmxcfs/pveproxy not yet ready). Note `pvesh_rc=0` in + **both** orderings — the exit code lies; only **fully-up** yields a coherent PVE stack. +- Evidence: `evidence/S3/firstboot-network-online.log`, listener log, `evidence/S1-S8-key-results.txt`. + +## S4 — post-installation-webhook payload — **PROVEN** (full payload archived) + +- `[post-installation-webhook] url="http://…:8088/webhook"` (plain HTTP; no cert-fingerprint required). + Fired **near end of install, before reboot** (~2 m 42 s in). `POST … application/json`, `ureq/3.0.11`, + **5052 B**, schema **v1.2**. Full body: `evidence/S4/webhook-payload.json`. +- **Identity fields the hub's unclaimed-appliance record can key on:** + - `dmi.system.uuid` = `14a82ff6-…` — SMBIOS/product UUID (stable hardware id). + - `network-interfaces[].mac` = `bc:24:11:98:10:0e` (+ `is-management:true`, DHCP `address`). + - `machine-id` (regenerates per install — not a hardware id), `fqdn`, `boot-info.mode` ("bios"), + `product.version` 9.2.2, `debian-version` 13.5, `cpu-info.model` "Intel(R) N100". + - `disks[].udev-properties.ID_SERIAL / ID_SERIAL_SHORT / ID_PATH` — disk serials. + - **`ssh-public-host-keys`** (ecdsa/ed25519/rsa) — lets the hub pin the box's SSH host keys for + later TOFU-free access. +- **CAVEATS (design input #2):** + - On **nested virt** `dmi.baseboard` is `{}`, `dmi.chassis.serial` and `dmi.system.serial` are **empty**. + On real bare metal these carry vendor board/chassis/system serials — **unprovable here** (OPEN item). + - **Webhook `disks[].size` is unreliable:** the 8 GB second disk was reported as 32 GB (it echoed the + boot disk's size); the guest's own `lsblk` shows 8 GB. Key on disk **serials**, not `disks[].size`. + +## S5 — disk-filter safety (LOAD-BEARING) — **PROVEN** + +Two disks: sda=32 GB target, sdb=8 GB ext4 canary (`LABEL=FELHOM_CANARY`, +`UUID=1111…5555`, `canary.bin` sha256 `0d22f95b…`, `canary.txt` `42139528…`; `evidence/S5/canary-before.sha`). +Canary verified by **mounting the LV on the host and hashing**, not from installer output. + +| Sub-run | Selection | Result | +|---|---|---| +| (a) | `disk-list=["sda"]` | installed on sda; sdb canary `sha256 -c: OK` (byte-identical), UUID unchanged. **PROVEN** | +| (b) | `filter.ID_SERIAL_SHORT="drive-scsi0"` (matches only sda) | installed on sda (lsblk: sda=pve LVM, sdb=ext4 FELHOM_CANARY); canary `OK`. **PROVEN** | +| (c) | `filter.ID_SERIAL_SHORT="no-such-disk-zzz"` (matches nothing) | **`ERROR: Installation failed: filter did not match any device` → `Auto-installation failed (exit-code 1)` → aborts to installer root shell.** No interactive fallback, **no disk touched**, canary intact. `evidence/S5/matchnothing-console.png` | + +## S6 — first-boot from-url + cert-fingerprint pinning — **PROVEN (fails closed)** + +- Stub served over **HTTPS with a self-signed cert** from DooPlex; `[first-boot] source="from-url" + url="https://…:9443/stub.sh" cert-fingerprint= ordering="fully-up"`. The pin is checked at + **install time** (in the installer environment), not first boot. +- **Correct fingerprint:** installer fetched the stub — HTTPS log + `192.168.0.115 "GET /stub.sh HTTP/1.1" 200` — and it ran at first boot (uid 0, log present, unit active). +- **RED-PROOF — wrong fingerprint (`F6…`→`A6…`):** *"INFO: Fetching first-boot hook from https://…/stub.sh + .. ERROR: Autoinstaller setup error: io: unexpected error: **Fingerprint did not match!** → + Auto-installation failed (exit-code 1)"* — the **entire install aborts**; the stub was **never + fetched** (HTTPS log empty) and never ran. `evidence/S6/redproof-console.png`. +- **Verdict: pinning FAILS CLOSED** — fit for the wrapper download story (from-url is safe; from-iso is + not required for security). + +## S7 — HTTP answer-fetch identity (stretch) — **PROVEN** + +- `--fetch-from http --url http://…:8088/answer`. The installer **POSTs** for its answer: + `application/json`, `ureq/3.0.11`, `accept: application/json, application/toml;q=0.5`, schema **v1.0**, + **388 B**. Body carries: `dmi.system.uuid` `14a82ff6-…`, `network_interfaces[].mac` + `BC:24:11:98:10:0E`, `product`/`iso` info, and (empty on virt) DMI serials — **but no disk info yet** + (disks aren't enumerated at answer-fetch time). → option B (hub serves a per-customer answer) is + viable, keyed on **MAC + SMBIOS UUID**. Full request in the listener log after the `S7-HTTPFETCH` marker. + +## S8 — failure modes — **PROVEN** + +- **(a) No DHCP / no link at first boot** (host-side `qm reboot` with NIC `link_down=1`; hook status + read via the pending-flag over IPv6 link-local — no reboot needed): + - **fully-up:** **hook RUNS** — flag consumed, new marker, pvesh works locally, outbound `curl` fails + ("Could not connect"). Boot reaches login. → robust to "customer's router is off". + - **network-online:** **hook does NOT run** — flag persists, unit `inactive(dead)` (gated on + `network-online.target`, unsatisfied without a DHCP-configured link); boot still reaches login + (**no infinite hang**). Hook stays armed for a later boot once network returns. + - *(Method note: a first attempt set `link_down=1` before an ssh `reboot`, which then couldn't reach + the guest — the reboot never fired. Redone host-side. The two "*-nonet-console.png" captures are + from that stale boot and are superseded by the flag-check result in `evidence/S1-S8-key-results.txt`.)* +- **(b) Invalid answer — `disk-list=["sdz"]` (non-existent disk):** validate-answer **passed** it + (syntactic only); at install: *"ERROR: Installation failed: disk in 'disk-selection' not found → + Auto-installation failed (exit-code 1)"*, aborts to shell, no disk touched, **no interactive + fallback**. `evidence/S8/invalid-disk-console.png`. + +## 13. Evidence + +All under `180:~/spike-baremetal/evidence/`: per-leg console PNGs (S2/S2b/S5/S6/S8), the full webhook +JSON (`S4/webhook-payload.json`), the request-logging listener (`listener.log`) and HTTPS access log +(`https.log`), canary hashes (`S5/canary-before.sha`), and the consolidated +`S1-S8-key-results.txt`. Console captures were taken with QEMU-monitor `screendump` → PPM → PNG +(felhom-pve has no image tools; converted on DooPlex). Prepared ISOs + source ISO retained under +`180:~/spike-baremetal/iso/`. + +## 15. Design inputs for R-21 (bare-metal ISO) — *(the spike brief calls this feature "R-22")* + +1. **First-boot ordering → `fully-up`.** Proven: under fully-up, pvesh returns a *complete* node object + and pct works (S3); under network-online pvesh is degraded (`pve-ssl.pem` missing) and the hook does + not even run without a DHCP link (S8a). The wrapper (drives pveum/pct/pvesh) must use fully-up. The + exactly-once guard is `ConditionPathExists=…/pending-first-boot-setup` + `ExecStartPost=rm` — re-arm + by re-touching that flag if a redo is ever needed. +2. **Webhook keys for the unclaimed-appliance record:** `dmi.system.uuid` (SMBIOS/product UUID) + + management-NIC `mac` are the stable pair; also archive `ssh-public-host-keys` (host-key pinning), + `fqdn`, `machine-id` (per-install, not hardware), `cpu-info.model`, disk `ID_SERIAL`. **Do not key on + `disks[].size`** (unreliable — S4). Real board/chassis/system serials are empty on virt → confirm on + bare metal before relying on them. +3. **Disk-selection policy for alpha:** both `disk-list=["sda"]` and a udev `filter.*` glob install only + the intended disk and leave other disks **byte-identical** (S5 a/b, proven by host-side hashing). A + **match-nothing** filter and a **non-existent disk name** both **fail-safe**: abort with exit-code 1, + touch no disk, no interactive fallback (S5c, S8b). Recommendation: **single-disk mini-PC → a + size/model `filter.*` glob** (portable across identical hardware, self-documenting), with the + match-nothing abort as the built-in wrong-hardware guard. Add a pre-install **LVM/partition wipe** + step (or document it) — the installer does not force-clear a prior LVM PV (S2b re-imaging gotcha). +4. **UEFI / Secure Boot:** no caveat required. Automated install and the installed PVE both work under + UEFI with Secure Boot **enforcing** (pre-enrolled MS keys), zero keypress, no MOK prompt (S2b). A + single install image is BIOS+UEFI-bootable. +5. **Pinning verdict:** `from-url` + `cert-fingerprint` is **fit** — a wrong pin aborts the whole + install (fails closed, S6). The wrapper/stub may be fetched from a URL with a pinned fingerprint; + `from-iso` (baking) is not required for security. Note the cert-fingerprint format is the + colon-separated uppercase SHA256 (`openssl x509 -fingerprint -sha256`). +6. **Assistant↔ISO pairing + build cost → `build-felhom-iso.sh`:** install the assistant from the PVE + `no-subscription` repo of the ISO's Debian codename (PVE 9.x = trixie); pair by major.minor. Per + prepared ISO ≈ 11.5 s + ~1.7 GB artifact; the container build is ~40 s one-time. The pipeline runs + fine on DooPlex (the build host). **Gate on parsed `validate-answer` output, never `$?`** (exit 0 on + failure). Remember `validate-answer` is syntactic only — disk existence is a runtime check. +7. **Failure-mode table (customer-facing "what if it goes wrong"):** + + | Scenario | Behavior | Customer/operator signal | + |---|---|---| + | Wrong hardware — filter matches nothing | install aborts, exit 1, no disk touched | box never finishes install; stays on ISO/shell | + | Non-existent disk in answer | install aborts, exit 1, no disk touched | same | + | Prior LVM/OS on disk | install aborts ("cannot init PV") | needs a wipe step (see #3) | + | Wrong cert-fingerprint (from-url) | whole install aborts, exit 1 | box never finishes install | + | No internet at first boot (fully-up) | hook runs; net-dependent steps fail-and-retry | box installed, wrapper fired, retries when net returns | + | No internet at first boot (network-online) | hook does not run; boot OK | avoid this ordering | + +8. **OPEN — needs one real bare-metal run (not provable on nested virt):** + - Real vendor **DMI serials** (baseboard/chassis/system) in the webhook + answer-fetch payloads + (empty on QEMU) — decide whether to key on them. + - Real firmware quirks: vendor UEFI boot-entry handling, whether the disk-first→CD fall-through works + on a real board (worked on SeaBIOS and OVMF here), and Secure Boot with a *vendor's* (non-MS-clean) + key set. + - Real-NIC naming (this VM enumerated `ens18`) and multi-NIC `is-management` selection. + - Real DHCP behavior / captive networks; USB-stick boot media (we booted a virtual CD). + +9. **Observations (recorded, not acted on):** + - `validate-answer` exit-0-on-failure is arguably a Proxmox bug; regardless, the pipeline must parse output. + - The probe stub's `curl_rc` reflected the trailing pipe stage, not curl — rely on curl's error text. + - Webhook `disks[].size` mis-report (see #2) is a Proxmox data-quality issue worth reporting upstream. + - `machine-id` regenerates per install → not a durable hardware identity for re-imaged boxes. + +## Cleanup state + +Spike VM 310 **stopped, kept** (config intact, s8b ISO still attached). ISOs + evidence retained under +`180:~/spike-baremetal/`. DooPlex helper listeners (8088 answer/webhook, 9443 HTTPS, 8099 file-server) +were used only during the spike and may be stopped. Nothing else on either host was modified. diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 21a83ee..3a4a94c 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -49,6 +49,7 @@ | R-13 | OOB management arc: dual-use existing WireGuard + hub desired-state channel as mutual-repair | L | idea | | | R-14 | Headscale/WireGuard spike: Minecraft/gaming port connectivity (CGNAT-proof, sovereign DERP fallback) | M | idea | | | R-15 | Multi-user dashboard accounts (household members, roles) | L | idea | Single password is a stated alpha limitation (R-11) | +| R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → claim-code pairing (option C) | XL | **spiked** (2026-07-16) | Spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* | ## Absorbed / superseded notes in this folder