# SPIKE — bare-metal Felhom ISO (Proxmox automated-installation mechanics), 2026-07-16

**Class:** spike (empirical mechanism validation before any production spec). **Output:** this findings
doc. **No production code shipped.** The only repo write is this file + one ROADMAP line (R-21 → spiked).

**Verdict in one line:** every mechanism the bare-metal path depends on — the
`proxmox-auto-install-assistant` toolchain, zero-touch install (BIOS *and* UEFI, incl. Secure Boot),
the first-boot hook (privileges / ordering / exactly-once), the post-installation webhook, disk-filter
safety, and from-url + cert-fingerprint pinning — **works and fails-closed where it must**. R-21 is
technically green to spec on nested virt; the only residual unknowns need one real bare-metal run
(vendor DMI serials, real firmware quirks). *(The spike brief refers to this feature as "R-22"; it is
entered in the ROADMAP as **R-21** — the same item.)*

## 0. Environment & baselines

| Item | Value |
|---|---|
| felhom.eu | `main` @ `d202b43` (clean, confirmed at start) |
| Host (nested-VM + live validation) | felhom-pve 192.168.0.162, PVE **9.2.2**, kernel 7.0.2-6-pve; 7.5 GiB RAM free at start (≥5 GiB gate ✓) |
| Assistant / ISO build host | DooPlex 192.168.0.180 (Docker 29.2.1) |
| Spike VM | **310** `spike-baremetal-iso`, 4 GB / 2 vCPU, scsi0 32 GB (target) + scsi1 8 GB (data/canary), virtio-scsi-single; SMBIOS uuid `14a82ff6-3ab5-440d-ac40-7f9968408618`, NIC MAC `bc:24:11:98:10:0e` |
| Never touched | qm 300 (drill), 9000, LXC 9201, Peti, hub — honored |
| Throwaway password | `$6$…` crypt hash baked into answer files (spike-only; `# THROWAWAY` in the archived answer.toml) |
| Evidence root | `180:~/spike-baremetal/` (ISOs, answers, `evidence/<leg>/`, listener/https logs) — kept |
| Spike VM final state | **stopped, kept** (per cleanup rule) pending acceptance |

## S1 — toolchain + ISO provenance — **PROVEN**

- **Assistant obtainable + pinnable per release:** `proxmox-auto-install-assistant 9.2.7`
  (`proxmox-installer-common 9.2.7`) installed in a disposable `debian:trixie` Docker container on
  DooPlex from the PVE **trixie** no-subscription repo. This is the intended `build-felhom-iso.sh`
  host and it works. Pairing rule: install the assistant from the `pve <codename> no-subscription`
  repo whose Debian codename matches the ISO generation (PVE 9.x = Debian 13 *trixie*); the assistant
  point release (9.2.7) tracks the repo and pairs with the ISO by **major.minor** (9.2), not by the
  ISO's point release (9.2-1).
- **ISO provenance:** `proxmox-ve_9.2-1.iso` sha256
  `4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c` — matches the official
  `download.proxmox.com/iso/SHA256SUMS` **and** the copy already on felhom-pve.
- **Build cost (design input #6):** container image build 37.9 s / 243 MB; `prepare-iso` ~**11.5 s**
  per prepared ISO (source local); prepared ISO 1,707,278,336 B (~1.1 MB over the 1,706,178,560 B
  source).
- **`validate-answer` catches a broken file** — fed one with both `root-password` and
  `root-password-hashed`: `` `global.root-password` and `global.root-password-hashed` cannot be set at
  the same time / Error: Found issues in the answer file.`` **TRAP (load-bearing for the pipeline):**
  it prints that to **stderr but returns exit code 0** — for *both* valid and invalid files. A
  `build-felhom-iso.sh` gate **must parse the output text**, never `$?`. It is also **syntactic only**:
  `disk-list=["sdz"]` (a non-existent disk) passed validation and only failed at install time (see S8b).
- Evidence: `evidence/S1-S8-key-results.txt`, `~/spike-baremetal/build/build.log`, `iso/SHA256SUMS`.

## S2 — baseline zero-touch install (SeaBIOS) — **PROVEN**

- **Given/When/Then:** prepared ISO (`--fetch-from iso`, keyless `answer-base.toml`) on VM 310
  (SeaBIOS, single 32 GB disk) → boot → automated install with **no keypress**, reboot into working PVE.
- Boot menu **auto-selected "Install Proxmox VE (Automated)"** with a 5 s countdown
  (`evidence/S2/boot-8s.png`); unattended package extraction (`evidence/S2/mid-install.png`); **no
  interactive prompt at any point**. Install wall-clock **≈ 3 min** (start 09:45:33 → booting installed
  PVE by 09:48:48).
- Installed system reachable over IPv6 link-local: `22` OPEN (`SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u4`),
  `8006` OPEN. TLS cert **CN=spike310.felhom.local** (configured fqdn applied); SAN carries the
  **DHCP-assigned IPv4 192.168.0.115**. Boot-order `scsi0;ide2` fall-through: the post-install reboot
  landed on the installed disk (no reinstall loop).
- Evidence: `evidence/S2/verdict.txt`, `boot-8s.png`, `mid-install.png`.

## S2b — UEFI + Secure Boot — **PROVEN** (no caveat needed)

- **UEFI (OVMF/q35, Secure Boot OFF, `pre-enrolled-keys=0`):** automated install identical, zero
  keypress (`evidence/S2b/boot-uefi-installer.png`). Keyed SSH login worked; `/sys/firmware/efi`
  present → **genuine UEFI boot**; `mokutil`: *SecureBoot disabled / Setup Mode*.
  **Password mechanism proven non-hollow:** the installed `/etc/shadow` root hash equals the answer's
  `root-password-hashed` **verbatim** → the password would work as configured.
- **Secure Boot ON (`pre-enrolled-keys=1`, MS 2023 certs, enforcing):** the PVE ISO's MS-signed shim
  chain booted the automated installer (`evidence/S2b/sbon-18s.png`), the install completed, **and the
  installed PVE booted under Secure Boot enforcing** — `mokutil: SecureBoot enabled`, `BOOTED_VIA=UEFI`,
  lockdown=integrity, **zero keypress, no MOK-enrollment prompt**. → **R-21 needs no Secure-Boot
  caveat**: customer mini-PCs with SB on by default install and boot fine.
- **Bonus finding:** the PVE installer writes a **dual BIOS+UEFI-bootable disk** — the S2 SeaBIOS
  install also booted under OVMF.
- **Re-imaging gotcha (finding):** the first UEFI reinstall over a disk that still held the prior
  install's **LVM PV signature** (only first/last-10 MiB wiped) **aborted**: *"unable to initialize
  physical volume /dev/sda3 / low level installer returned early"*. The auto-installer's
  "cleanup root-disks" step does **not** force-clear a pre-existing LVM PV. A full `blkdiscard` then
  installed cleanly. → a customer disk with a prior LVM/Proxmox/Linux install may need an explicit
  wipe before the automated install.

## S3 — first-boot hook: privileges, ordering, exactly-once — **PROVEN**

- **`--on-first-boot stub.sh`, `[first-boot] source="from-iso" ordering="fully-up"`.** Unit is
  **`proxmox-first-boot-multi-user.service`** ("Fully Booted"),
  `After=… pveproxy.service pmgproxy.service proxmox-backup-proxy.service`.
- **Runs as root (uid 0).** Under **fully-up**, `pvesh get /nodes` returned a **complete** node object
  (status online, ssl_fingerprint present, uptime) `pvesh_rc=0`; `pct list` `pct_rc=0`; outbound
  `curl` hit the DooPlex listener — `GET /firstboot?host=spike310&uid=0 from 192.168.0.115` (network
  truly up). This is the ordering the real wrapper needs (it drives pveum/pct/pvesh).
- **Exactly-once mechanism:** unit has `ConditionPathExists=/var/lib/proxmox-first-boot/pending-first-boot-setup`
  and `ExecStartPost=/usr/bin/rm … pending-first-boot-setup`. Empirically: after a reboot the hook log
  is **byte-identical** (16 lines, 1 marker, sha256 unchanged), the unit is `inactive(dead)` with
  *"ConditionPathExists … was not met"* (`ConditionResult=no`), and the listener shows **exactly 1**
  firstboot hit.
- **Comparison leg — `ordering="network-online"`** (`proxmox-first-boot-network-online.service`):
  pvesh returned **degraded** data — `'/etc/pve/nodes/spike310/pve-ssl.pem' does not exist!`,
  `status:"unknown"`, `ssl_fingerprint:null` (pmxcfs/pveproxy not yet ready). Note `pvesh_rc=0` in
  **both** orderings — the exit code lies; only **fully-up** yields a coherent PVE stack.
- Evidence: `evidence/S3/firstboot-network-online.log`, listener log, `evidence/S1-S8-key-results.txt`.

## S4 — post-installation-webhook payload — **PROVEN** (full payload archived)

- `[post-installation-webhook] url="http://…:8088/webhook"` (plain HTTP; no cert-fingerprint required).
  Fired **near end of install, before reboot** (~2 m 42 s in). `POST … application/json`, `ureq/3.0.11`,
  **5052 B**, schema **v1.2**. Full body: `evidence/S4/webhook-payload.json`.
- **Identity fields the hub's unclaimed-appliance record can key on:**
  - `dmi.system.uuid` = `14a82ff6-…` — SMBIOS/product UUID (stable hardware id).
  - `network-interfaces[].mac` = `bc:24:11:98:10:0e` (+ `is-management:true`, DHCP `address`).
  - `machine-id` (regenerates per install — not a hardware id), `fqdn`, `boot-info.mode` ("bios"),
    `product.version` 9.2.2, `debian-version` 13.5, `cpu-info.model` "Intel(R) N100".
  - `disks[].udev-properties.ID_SERIAL / ID_SERIAL_SHORT / ID_PATH` — disk serials.
  - **`ssh-public-host-keys`** (ecdsa/ed25519/rsa) — lets the hub pin the box's SSH host keys for
    later TOFU-free access.
- **CAVEATS (design input #2):**
  - On **nested virt** `dmi.baseboard` is `{}`, `dmi.chassis.serial` and `dmi.system.serial` are **empty**.
    On real bare metal these carry vendor board/chassis/system serials — **unprovable here** (OPEN item).
  - **Webhook `disks[].size` is unreliable:** the 8 GB second disk was reported as 32 GB (it echoed the
    boot disk's size); the guest's own `lsblk` shows 8 GB. Key on disk **serials**, not `disks[].size`.

## S5 — disk-filter safety (LOAD-BEARING) — **PROVEN**

Two disks: sda=32 GB target, sdb=8 GB ext4 canary (`LABEL=FELHOM_CANARY`,
`UUID=1111…5555`, `canary.bin` sha256 `0d22f95b…`, `canary.txt` `42139528…`; `evidence/S5/canary-before.sha`).
Canary verified by **mounting the LV on the host and hashing**, not from installer output.

| Sub-run | Selection | Result |
|---|---|---|
| (a) | `disk-list=["sda"]` | installed on sda; sdb canary `sha256 -c: OK` (byte-identical), UUID unchanged. **PROVEN** |
| (b) | `filter.ID_SERIAL_SHORT="drive-scsi0"` (matches only sda) | installed on sda (lsblk: sda=pve LVM, sdb=ext4 FELHOM_CANARY); canary `OK`. **PROVEN** |
| (c) | `filter.ID_SERIAL_SHORT="no-such-disk-zzz"` (matches nothing) | **`ERROR: Installation failed: filter did not match any device` → `Auto-installation failed (exit-code 1)` → aborts to installer root shell.** No interactive fallback, **no disk touched**, canary intact. `evidence/S5/matchnothing-console.png` |

## S6 — first-boot from-url + cert-fingerprint pinning — **PROVEN (fails closed)**

- Stub served over **HTTPS with a self-signed cert** from DooPlex; `[first-boot] source="from-url"
  url="https://…:9443/stub.sh" cert-fingerprint=<SHA256> ordering="fully-up"`. The pin is checked at
  **install time** (in the installer environment), not first boot.
- **Correct fingerprint:** installer fetched the stub — HTTPS log
  `192.168.0.115 "GET /stub.sh HTTP/1.1" 200` — and it ran at first boot (uid 0, log present, unit active).
- **RED-PROOF — wrong fingerprint (`F6…`→`A6…`):** *"INFO: Fetching first-boot hook from https://…/stub.sh
  .. ERROR: Autoinstaller setup error: io: unexpected error: **Fingerprint did not match!** →
  Auto-installation failed (exit-code 1)"* — the **entire install aborts**; the stub was **never
  fetched** (HTTPS log empty) and never ran. `evidence/S6/redproof-console.png`.
- **Verdict: pinning FAILS CLOSED** — fit for the wrapper download story (from-url is safe; from-iso is
  not required for security).

## S7 — HTTP answer-fetch identity (stretch) — **PROVEN**

- `--fetch-from http --url http://…:8088/answer`. The installer **POSTs** for its answer:
  `application/json`, `ureq/3.0.11`, `accept: application/json, application/toml;q=0.5`, schema **v1.0**,
  **388 B**. Body carries: `dmi.system.uuid` `14a82ff6-…`, `network_interfaces[].mac`
  `BC:24:11:98:10:0E`, `product`/`iso` info, and (empty on virt) DMI serials — **but no disk info yet**
  (disks aren't enumerated at answer-fetch time). → option B (hub serves a per-customer answer) is
  viable, keyed on **MAC + SMBIOS UUID**. Full request in the listener log after the `S7-HTTPFETCH` marker.

## S8 — failure modes — **PROVEN**

- **(a) No DHCP / no link at first boot** (host-side `qm reboot` with NIC `link_down=1`; hook status
  read via the pending-flag over IPv6 link-local — no reboot needed):
  - **fully-up:** **hook RUNS** — flag consumed, new marker, pvesh works locally, outbound `curl` fails
    ("Could not connect"). Boot reaches login. → robust to "customer's router is off".
  - **network-online:** **hook does NOT run** — flag persists, unit `inactive(dead)` (gated on
    `network-online.target`, unsatisfied without a DHCP-configured link); boot still reaches login
    (**no infinite hang**). Hook stays armed for a later boot once network returns.
  - *(Method note: a first attempt set `link_down=1` before an ssh `reboot`, which then couldn't reach
    the guest — the reboot never fired. Redone host-side. The two "*-nonet-console.png" captures are
    from that stale boot and are superseded by the flag-check result in `evidence/S1-S8-key-results.txt`.)*
- **(b) Invalid answer — `disk-list=["sdz"]` (non-existent disk):** validate-answer **passed** it
  (syntactic only); at install: *"ERROR: Installation failed: disk in 'disk-selection' not found →
  Auto-installation failed (exit-code 1)"*, aborts to shell, no disk touched, **no interactive
  fallback**. `evidence/S8/invalid-disk-console.png`.

## 13. Evidence

All under `180:~/spike-baremetal/evidence/`: per-leg console PNGs (S2/S2b/S5/S6/S8), the full webhook
JSON (`S4/webhook-payload.json`), the request-logging listener (`listener.log`) and HTTPS access log
(`https.log`), canary hashes (`S5/canary-before.sha`), and the consolidated
`S1-S8-key-results.txt`. Console captures were taken with QEMU-monitor `screendump` → PPM → PNG
(felhom-pve has no image tools; converted on DooPlex). Prepared ISOs + source ISO retained under
`180:~/spike-baremetal/iso/`.

## 15. Design inputs for R-21 (bare-metal ISO) — *(the spike brief calls this feature "R-22")*

1. **First-boot ordering → `fully-up`.** Proven: under fully-up, pvesh returns a *complete* node object
   and pct works (S3); under network-online pvesh is degraded (`pve-ssl.pem` missing) and the hook does
   not even run without a DHCP link (S8a). The wrapper (drives pveum/pct/pvesh) must use fully-up. The
   exactly-once guard is `ConditionPathExists=…/pending-first-boot-setup` + `ExecStartPost=rm` — re-arm
   by re-touching that flag if a redo is ever needed.
2. **Webhook keys for the unclaimed-appliance record:** `dmi.system.uuid` (SMBIOS/product UUID) +
   management-NIC `mac` are the stable pair; also archive `ssh-public-host-keys` (host-key pinning),
   `fqdn`, `machine-id` (per-install, not hardware), `cpu-info.model`, disk `ID_SERIAL`. **Do not key on
   `disks[].size`** (unreliable — S4). Real board/chassis/system serials are empty on virt → confirm on
   bare metal before relying on them.
3. **Disk-selection policy for alpha:** both `disk-list=["sda"]` and a udev `filter.*` glob install only
   the intended disk and leave other disks **byte-identical** (S5 a/b, proven by host-side hashing). A
   **match-nothing** filter and a **non-existent disk name** both **fail-safe**: abort with exit-code 1,
   touch no disk, no interactive fallback (S5c, S8b). Recommendation: **single-disk mini-PC → a
   size/model `filter.*` glob** (portable across identical hardware, self-documenting), with the
   match-nothing abort as the built-in wrong-hardware guard. Add a pre-install **LVM/partition wipe**
   step (or document it) — the installer does not force-clear a prior LVM PV (S2b re-imaging gotcha).
4. **UEFI / Secure Boot:** no caveat required. Automated install and the installed PVE both work under
   UEFI with Secure Boot **enforcing** (pre-enrolled MS keys), zero keypress, no MOK prompt (S2b). A
   single install image is BIOS+UEFI-bootable.
5. **Pinning verdict:** `from-url` + `cert-fingerprint` is **fit** — a wrong pin aborts the whole
   install (fails closed, S6). The wrapper/stub may be fetched from a URL with a pinned fingerprint;
   `from-iso` (baking) is not required for security. Note the cert-fingerprint format is the
   colon-separated uppercase SHA256 (`openssl x509 -fingerprint -sha256`).
6. **Assistant↔ISO pairing + build cost → `build-felhom-iso.sh`:** install the assistant from the PVE
   `no-subscription` repo of the ISO's Debian codename (PVE 9.x = trixie); pair by major.minor. Per
   prepared ISO ≈ 11.5 s + ~1.7 GB artifact; the container build is ~40 s one-time. The pipeline runs
   fine on DooPlex (the build host). **Gate on parsed `validate-answer` output, never `$?`** (exit 0 on
   failure). Remember `validate-answer` is syntactic only — disk existence is a runtime check.
7. **Failure-mode table (customer-facing "what if it goes wrong"):**

   | Scenario | Behavior | Customer/operator signal |
   |---|---|---|
   | Wrong hardware — filter matches nothing | install aborts, exit 1, no disk touched | box never finishes install; stays on ISO/shell |
   | Non-existent disk in answer | install aborts, exit 1, no disk touched | same |
   | Prior LVM/OS on disk | install aborts ("cannot init PV") | needs a wipe step (see #3) |
   | Wrong cert-fingerprint (from-url) | whole install aborts, exit 1 | box never finishes install |
   | No internet at first boot (fully-up) | hook runs; net-dependent steps fail-and-retry | box installed, wrapper fired, retries when net returns |
   | No internet at first boot (network-online) | hook does not run; boot OK | avoid this ordering |

8. **OPEN — needs one real bare-metal run (not provable on nested virt):**
   - Real vendor **DMI serials** (baseboard/chassis/system) in the webhook + answer-fetch payloads
     (empty on QEMU) — decide whether to key on them.
   - Real firmware quirks: vendor UEFI boot-entry handling, whether the disk-first→CD fall-through works
     on a real board (worked on SeaBIOS and OVMF here), and Secure Boot with a *vendor's* (non-MS-clean)
     key set.
   - Real-NIC naming (this VM enumerated `ens18`) and multi-NIC `is-management` selection.
   - Real DHCP behavior / captive networks; USB-stick boot media (we booted a virtual CD).

9. **Observations (recorded, not acted on):**
   - `validate-answer` exit-0-on-failure is arguably a Proxmox bug; regardless, the pipeline must parse output.
   - The probe stub's `curl_rc` reflected the trailing pipe stage, not curl — rely on curl's error text.
   - Webhook `disks[].size` mis-report (see #2) is a Proxmox data-quality issue worth reporting upstream.
   - `machine-id` regenerates per install → not a durable hardware identity for re-imaged boxes.

## Cleanup state

Spike VM 310 **stopped, kept** (config intact, s8b ISO still attached). ISOs + evidence retained under
`180:~/spike-baremetal/`. DooPlex helper listeners (8088 answer/webhook, 9443 HTTPS, 8099 file-server)
were used only during the spike and may be stopped. Nothing else on either host was modified.