felhom.eu/documentation/audits/SPIKE-baremetal-iso-2026-07-16.md

SPIKE — bare-metal Felhom ISO (Proxmox automated-installation mechanics), 2026-07-16

Class: spike (empirical mechanism validation before any production spec). Output: this findings doc. No production code shipped. The only repo write is this file + one ROADMAP line (R-21 → spiked).

Verdict in one line: every mechanism the bare-metal path depends on — the proxmox-auto-install-assistant toolchain, zero-touch install (BIOS and UEFI, incl. Secure Boot), the first-boot hook (privileges / ordering / exactly-once), the post-installation webhook, disk-filter safety, and from-url + cert-fingerprint pinning — works and fails-closed where it must. R-21 is technically green to spec on nested virt; the only residual unknowns need one real bare-metal run (vendor DMI serials, real firmware quirks). (The spike brief refers to this feature as "R-22"; it is entered in the ROADMAP as R-21 — the same item.)

0. Environment & baselines

Item	Value
felhom.eu	main @ d202b43 (clean, confirmed at start)
Host (nested-VM + live validation)	felhom-pve 192.168.0.162, PVE 9.2.2, kernel 7.0.2-6-pve; 7.5 GiB RAM free at start (≥5 GiB gate ✓)
Assistant / ISO build host	DooPlex 192.168.0.180 (Docker 29.2.1)
Spike VM	310 spike-baremetal-iso, 4 GB / 2 vCPU, scsi0 32 GB (target) + scsi1 8 GB (data/canary), virtio-scsi-single; SMBIOS uuid 14a82ff6-3ab5-440d-ac40-7f9968408618, NIC MAC bc:24:11:98:10:0e
Never touched	qm 300 (drill), 9000, LXC 9201, Peti, hub — honored
Throwaway password	$6$… crypt hash baked into answer files (spike-only; # THROWAWAY in the archived answer.toml)
Evidence root	180:~/spike-baremetal/ (ISOs, answers, evidence/<leg>/, listener/https logs) — kept
Spike VM final state	stopped, kept (per cleanup rule) pending acceptance

S1 — toolchain + ISO provenance — PROVEN

• Assistant obtainable + pinnable per release: proxmox-auto-install-assistant 9.2.7 (proxmox-installer-common 9.2.7) installed in a disposable debian:trixie Docker container on DooPlex from the PVE trixie no-subscription repo. This is the intended build-felhom-iso.sh host and it works. Pairing rule: install the assistant from the pve <codename> no-subscription repo whose Debian codename matches the ISO generation (PVE 9.x = Debian 13 trixie); the assistant point release (9.2.7) tracks the repo and pairs with the ISO by major.minor (9.2), not by the ISO's point release (9.2-1).
• ISO provenance: proxmox-ve_9.2-1.iso sha256 4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c — matches the official download.proxmox.com/iso/SHA256SUMS and the copy already on felhom-pve.
• Build cost (design input #6): container image build 37.9 s / 243 MB; prepare-iso ~11.5 s per prepared ISO (source local); prepared ISO 1,707,278,336 B (~1.1 MB over the 1,706,178,560 B source).
• validate-answer catches a broken file — fed one with both root-password and root-password-hashed: `global.root-password` and `global.root-password-hashed` cannot be set at the same time / Error: Found issues in the answer file. TRAP (load-bearing for the pipeline): it prints that to stderr but returns exit code 0 — for both valid and invalid files. A build-felhom-iso.sh gate must parse the output text, never $?. It is also syntactic only: disk-list=["sdz"] (a non-existent disk) passed validation and only failed at install time (see S8b).
• Evidence: evidence/S1-S8-key-results.txt, ~/spike-baremetal/build/build.log, iso/SHA256SUMS.

S2 — baseline zero-touch install (SeaBIOS) — PROVEN

• Given/When/Then: prepared ISO (--fetch-from iso, keyless answer-base.toml) on VM 310 (SeaBIOS, single 32 GB disk) → boot → automated install with no keypress, reboot into working PVE.
• Boot menu auto-selected "Install Proxmox VE (Automated)" with a 5 s countdown (evidence/S2/boot-8s.png); unattended package extraction (evidence/S2/mid-install.png); no interactive prompt at any point. Install wall-clock ≈ 3 min (start 09:45:33 → booting installed PVE by 09:48:48).
• Installed system reachable over IPv6 link-local: 22 OPEN (SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u4), 8006 OPEN. TLS cert CN=spike310.felhom.local (configured fqdn applied); SAN carries the DHCP-assigned IPv4 192.168.0.115. Boot-order scsi0;ide2 fall-through: the post-install reboot landed on the installed disk (no reinstall loop).
• Evidence: evidence/S2/verdict.txt, boot-8s.png, mid-install.png.

S2b — UEFI + Secure Boot — PROVEN (no caveat needed)

• UEFI (OVMF/q35, Secure Boot OFF, pre-enrolled-keys=0): automated install identical, zero keypress (evidence/S2b/boot-uefi-installer.png). Keyed SSH login worked; /sys/firmware/efi present → genuine UEFI boot; mokutil: SecureBoot disabled / Setup Mode. Password mechanism proven non-hollow: the installed /etc/shadow root hash equals the answer's root-password-hashed verbatim → the password would work as configured.
• Secure Boot ON (pre-enrolled-keys=1, MS 2023 certs, enforcing): the PVE ISO's MS-signed shim chain booted the automated installer (evidence/S2b/sbon-18s.png), the install completed, and the installed PVE booted under Secure Boot enforcing — mokutil: SecureBoot enabled, BOOTED_VIA=UEFI, lockdown=integrity, zero keypress, no MOK-enrollment prompt. → R-21 needs no Secure-Boot caveat: customer mini-PCs with SB on by default install and boot fine.
• Bonus finding: the PVE installer writes a dual BIOS+UEFI-bootable disk — the S2 SeaBIOS install also booted under OVMF.
• Re-imaging gotcha (finding): the first UEFI reinstall over a disk that still held the prior install's LVM PV signature (only first/last-10 MiB wiped) aborted: "unable to initialize physical volume /dev/sda3 / low level installer returned early". The auto-installer's "cleanup root-disks" step does not force-clear a pre-existing LVM PV. A full blkdiscard then installed cleanly. → a customer disk with a prior LVM/Proxmox/Linux install may need an explicit wipe before the automated install.

S3 — first-boot hook: privileges, ordering, exactly-once — PROVEN

• --on-first-boot stub.sh, [first-boot] source="from-iso" ordering="fully-up". Unit is proxmox-first-boot-multi-user.service ("Fully Booted"), After=… pveproxy.service pmgproxy.service proxmox-backup-proxy.service.
• Runs as root (uid 0). Under fully-up, pvesh get /nodes returned a complete node object (status online, ssl_fingerprint present, uptime) pvesh_rc=0; pct list pct_rc=0; outbound curl hit the DooPlex listener — GET /firstboot?host=spike310&uid=0 from 192.168.0.115 (network truly up). This is the ordering the real wrapper needs (it drives pveum/pct/pvesh).
• Exactly-once mechanism: unit has ConditionPathExists=/var/lib/proxmox-first-boot/pending-first-boot-setup and ExecStartPost=/usr/bin/rm … pending-first-boot-setup. Empirically: after a reboot the hook log is byte-identical (16 lines, 1 marker, sha256 unchanged), the unit is inactive(dead) with "ConditionPathExists … was not met" (ConditionResult=no), and the listener shows exactly 1 firstboot hit.
• Comparison leg — ordering="network-online" (proxmox-first-boot-network-online.service): pvesh returned degraded data — '/etc/pve/nodes/spike310/pve-ssl.pem' does not exist!, status:"unknown", ssl_fingerprint:null (pmxcfs/pveproxy not yet ready). Note pvesh_rc=0 in both orderings — the exit code lies; only fully-up yields a coherent PVE stack.
• Evidence: evidence/S3/firstboot-network-online.log, listener log, evidence/S1-S8-key-results.txt.

S4 — post-installation-webhook payload — PROVEN (full payload archived)

• [post-installation-webhook] url="http://…:8088/webhook" (plain HTTP; no cert-fingerprint required). Fired near end of install, before reboot (~2 m 42 s in). POST … application/json, ureq/3.0.11, 5052 B, schema v1.2. Full body: evidence/S4/webhook-payload.json.
• Identity fields the hub's unclaimed-appliance record can key on:
  • dmi.system.uuid = 14a82ff6-… — SMBIOS/product UUID (stable hardware id).
  • network-interfaces[].mac = bc:24:11:98:10:0e (+ is-management:true, DHCP address).
  • machine-id (regenerates per install — not a hardware id), fqdn, boot-info.mode ("bios"), product.version 9.2.2, debian-version 13.5, cpu-info.model "Intel(R) N100".
  • disks[].udev-properties.ID_SERIAL / ID_SERIAL_SHORT / ID_PATH — disk serials.
  • ssh-public-host-keys (ecdsa/ed25519/rsa) — lets the hub pin the box's SSH host keys for later TOFU-free access.
• CAVEATS (design input #2):
  • On nested virt dmi.baseboard is {}, dmi.chassis.serial and dmi.system.serial are empty. On real bare metal these carry vendor board/chassis/system serials — unprovable here (OPEN item).
  • Webhook disks[].size is unreliable: the 8 GB second disk was reported as 32 GB (it echoed the boot disk's size); the guest's own lsblk shows 8 GB. Key on disk serials, not disks[].size.

S5 — disk-filter safety (LOAD-BEARING) — PROVEN

Two disks: sda=32 GB target, sdb=8 GB ext4 canary (LABEL=FELHOM_CANARY, UUID=1111…5555, canary.bin sha256 0d22f95b…, canary.txt 42139528…; evidence/S5/canary-before.sha). Canary verified by mounting the LV on the host and hashing, not from installer output.

Sub-run	Selection	Result
(a)	disk-list=["sda"]	installed on sda; sdb canary sha256 -c: OK (byte-identical), UUID unchanged. PROVEN
(b)	filter.ID_SERIAL_SHORT="drive-scsi0" (matches only sda)	installed on sda (lsblk: sda=pve LVM, sdb=ext4 FELHOM_CANARY); canary OK. PROVEN
(c)	filter.ID_SERIAL_SHORT="no-such-disk-zzz" (matches nothing)	ERROR: Installation failed: filter did not match any device → Auto-installation failed (exit-code 1) → aborts to installer root shell. No interactive fallback, no disk touched, canary intact. evidence/S5/matchnothing-console.png

S6 — first-boot from-url + cert-fingerprint pinning — PROVEN (fails closed)

• Stub served over HTTPS with a self-signed cert from DooPlex; [first-boot] source="from-url" url="https://…:9443/stub.sh" cert-fingerprint=<SHA256> ordering="fully-up". The pin is checked at install time (in the installer environment), not first boot.
• Correct fingerprint: installer fetched the stub — HTTPS log 192.168.0.115 "GET /stub.sh HTTP/1.1" 200 — and it ran at first boot (uid 0, log present, unit active).
• RED-PROOF — wrong fingerprint (F6…→A6…): "INFO: Fetching first-boot hook from https://…/stub.sh .. ERROR: Autoinstaller setup error: io: unexpected error: Fingerprint did not match! → Auto-installation failed (exit-code 1)" — the entire install aborts; the stub was never fetched (HTTPS log empty) and never ran. evidence/S6/redproof-console.png.
• Verdict: pinning FAILS CLOSED — fit for the wrapper download story (from-url is safe; from-iso is not required for security).

S7 — HTTP answer-fetch identity (stretch) — PROVEN

• --fetch-from http --url http://…:8088/answer. The installer POSTs for its answer: application/json, ureq/3.0.11, accept: application/json, application/toml;q=0.5, schema v1.0, 388 B. Body carries: dmi.system.uuid 14a82ff6-…, network_interfaces[].mac BC:24:11:98:10:0E, product/iso info, and (empty on virt) DMI serials — but no disk info yet (disks aren't enumerated at answer-fetch time). → option B (hub serves a per-customer answer) is viable, keyed on MAC + SMBIOS UUID. Full request in the listener log after the S7-HTTPFETCH marker.

S8 — failure modes — PROVEN

• (a) No DHCP / no link at first boot (host-side qm reboot with NIC link_down=1; hook status read via the pending-flag over IPv6 link-local — no reboot needed):
  • fully-up: hook RUNS — flag consumed, new marker, pvesh works locally, outbound curl fails ("Could not connect"). Boot reaches login. → robust to "customer's router is off".
  • network-online: hook does NOT run — flag persists, unit inactive(dead) (gated on network-online.target, unsatisfied without a DHCP-configured link); boot still reaches login (no infinite hang). Hook stays armed for a later boot once network returns.
  • (Method note: a first attempt set link_down=1 before an ssh reboot, which then couldn't reach the guest — the reboot never fired. Redone host-side. The two "-nonet-console.png" captures are from that stale boot and are superseded by the flag-check result in evidence/S1-S8-key-results.txt.)*
• (b) Invalid answer — disk-list=["sdz"] (non-existent disk): validate-answer passed it (syntactic only); at install: "ERROR: Installation failed: disk in 'disk-selection' not found → Auto-installation failed (exit-code 1)", aborts to shell, no disk touched, no interactive fallback. evidence/S8/invalid-disk-console.png.

13. Evidence

All under 180:~/spike-baremetal/evidence/: per-leg console PNGs (S2/S2b/S5/S6/S8), the full webhook JSON (S4/webhook-payload.json), the request-logging listener (listener.log) and HTTPS access log (https.log), canary hashes (S5/canary-before.sha), and the consolidated S1-S8-key-results.txt. Console captures were taken with QEMU-monitor screendump → PPM → PNG (felhom-pve has no image tools; converted on DooPlex). Prepared ISOs + source ISO retained under 180:~/spike-baremetal/iso/.

15. Design inputs for R-21 (bare-metal ISO) — (the spike brief calls this feature "R-22")

1. First-boot ordering → fully-up. Proven: under fully-up, pvesh returns a complete node object and pct works (S3); under network-online pvesh is degraded (pve-ssl.pem missing) and the hook does not even run without a DHCP link (S8a). The wrapper (drives pveum/pct/pvesh) must use fully-up. The exactly-once guard is ConditionPathExists=…/pending-first-boot-setup + ExecStartPost=rm — re-arm by re-touching that flag if a redo is ever needed.

2. Webhook keys for the unclaimed-appliance record: dmi.system.uuid (SMBIOS/product UUID) + management-NIC mac are the stable pair; also archive ssh-public-host-keys (host-key pinning), fqdn, machine-id (per-install, not hardware), cpu-info.model, disk ID_SERIAL. Do not key on disks[].size (unreliable — S4). Real board/chassis/system serials are empty on virt → confirm on bare metal before relying on them.

3. Disk-selection policy for alpha: both disk-list=["sda"] and a udev filter.* glob install only the intended disk and leave other disks byte-identical (S5 a/b, proven by host-side hashing). A match-nothing filter and a non-existent disk name both fail-safe: abort with exit-code 1, touch no disk, no interactive fallback (S5c, S8b). Recommendation: single-disk mini-PC → a size/model filter.* glob (portable across identical hardware, self-documenting), with the match-nothing abort as the built-in wrong-hardware guard. Add a pre-install LVM/partition wipe step (or document it) — the installer does not force-clear a prior LVM PV (S2b re-imaging gotcha).

4. UEFI / Secure Boot: no caveat required. Automated install and the installed PVE both work under UEFI with Secure Boot enforcing (pre-enrolled MS keys), zero keypress, no MOK prompt (S2b). A single install image is BIOS+UEFI-bootable.

5. Pinning verdict: from-url + cert-fingerprint is fit — a wrong pin aborts the whole install (fails closed, S6). The wrapper/stub may be fetched from a URL with a pinned fingerprint; from-iso (baking) is not required for security. Note the cert-fingerprint format is the colon-separated uppercase SHA256 (openssl x509 -fingerprint -sha256).

6. Assistant↔ISO pairing + build cost → build-felhom-iso.sh: install the assistant from the PVE no-subscription repo of the ISO's Debian codename (PVE 9.x = trixie); pair by major.minor. Per prepared ISO ≈ 11.5 s + ~1.7 GB artifact; the container build is ~40 s one-time. The pipeline runs fine on DooPlex (the build host). Gate on parsed validate-answer output, never $? (exit 0 on failure). Remember validate-answer is syntactic only — disk existence is a runtime check.

7. Failure-mode table (customer-facing "what if it goes wrong"):

   Scenario	Behavior	Customer/operator signal
   Wrong hardware — filter matches nothing	install aborts, exit 1, no disk touched	box never finishes install; stays on ISO/shell
   Non-existent disk in answer	install aborts, exit 1, no disk touched	same
   Prior LVM/OS on disk	install aborts ("cannot init PV")	needs a wipe step (see #3)
   Wrong cert-fingerprint (from-url)	whole install aborts, exit 1	box never finishes install
   No internet at first boot (fully-up)	hook runs; net-dependent steps fail-and-retry	box installed, wrapper fired, retries when net returns
   No internet at first boot (network-online)	hook does not run; boot OK	avoid this ordering

8. OPEN — needs one real bare-metal run (not provable on nested virt):

   • Real vendor DMI serials (baseboard/chassis/system) in the webhook + answer-fetch payloads (empty on QEMU) — decide whether to key on them.
   • Real firmware quirks: vendor UEFI boot-entry handling, whether the disk-first→CD fall-through works on a real board (worked on SeaBIOS and OVMF here), and Secure Boot with a vendor's (non-MS-clean) key set.
   • Real-NIC naming (this VM enumerated ens18) and multi-NIC is-management selection.
   • Real DHCP behavior / captive networks; USB-stick boot media (we booted a virtual CD).

9. Observations (recorded, not acted on):

   • validate-answer exit-0-on-failure is arguably a Proxmox bug; regardless, the pipeline must parse output.
   • The probe stub's curl_rc reflected the trailing pipe stage, not curl — rely on curl's error text.
   • Webhook disks[].size mis-report (see #2) is a Proxmox data-quality issue worth reporting upstream.
   • machine-id regenerates per install → not a durable hardware identity for re-imaged boxes.

Cleanup state

Spike VM 310 stopped, kept (config intact, s8b ISO still attached). ISOs + evidence retained under 180:~/spike-baremetal/. DooPlex helper listeners (8088 answer/webhook, 9443 HTTPS, 8099 file-server) were used only during the spike and may be stopped. Nothing else on either host was modified.