admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(spike): bare-metal Felhom ISO — R-21 spiked (all mechanisms GREEN on nested virt)

Browse Source 
Empirical spike of the Proxmox automated-installation path for R-21 (bare-metal
customer-hardware ISO). All legs proven on nested VM 310 (felhom-pve):
- S1 assistant 9.2.7 + ISO provenance; validate-answer exits 0 on failure (parse output)
- S2/S2b zero-touch install SeaBIOS + UEFI incl. Secure Boot enforcing (no keypress/MOK)
- S3 first-boot hook: fully-up = root + working pvesh/pct, exactly-once
- S4 webhook payload enumerated (SMBIOS-UUID + MAC + host SSH keys)
- S5 disk-filter installs only target (canary byte-identical), match-nothing fails safe
- S6 from-url + cert-fingerprint fails CLOSED (red-proof)
- S7 http answer-fetch identity; S8 no-net + invalid-answer failure modes
Findings doc + ROADMAP R-21 status=spiked. No production code.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017qDiBqKKQ5vPB5fXBqu7Kp
This commit is contained in:
authentik Default Admin
2026-07-16 11:29:03 +02:00
parent d202b43b95
commit 914d51aa5f
2 changed files with 247 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/audits/SPIKE-baremetal-iso-2026-07-16.md
+246
View File
@@ -0,0 +1,246 @@
# SPIKE — bare-metal Felhom ISO (Proxmox automated-installation mechanics), 2026-07-16
**Class:** spike (empirical mechanism validation before any production spec). **Output:** this findings
doc. **No production code shipped.** The only repo write is this file + one ROADMAP line (R-21 → spiked).
**Verdict in one line:** every mechanism the bare-metal path depends on — the
`proxmox-auto-install-assistant` toolchain, zero-touch install (BIOS *and* UEFI, incl. Secure Boot),
the first-boot hook (privileges / ordering / exactly-once), the post-installation webhook, disk-filter
safety, and from-url + cert-fingerprint pinning — **works and fails-closed where it must**. R-21 is
technically green to spec on nested virt; the only residual unknowns need one real bare-metal run
(vendor DMI serials, real firmware quirks). *(The spike brief refers to this feature as "R-22"; it is
entered in the ROADMAP as **R-21** — the same item.)*
## 0. Environment & baselines
| Item | Value |
|---|---|
| felhom.eu | `main` @ `d202b43` (clean, confirmed at start) |
| Host (nested-VM + live validation) | felhom-pve 192.168.0.162, PVE **9.2.2**, kernel 7.0.2-6-pve; 7.5 GiB RAM free at start (≥5 GiB gate ✓) |
| Assistant / ISO build host | DooPlex 192.168.0.180 (Docker 29.2.1) |
| Spike VM | **310** `spike-baremetal-iso`, 4 GB / 2 vCPU, scsi0 32 GB (target) + scsi1 8 GB (data/canary), virtio-scsi-single; SMBIOS uuid `14a82ff6-3ab5-440d-ac40-7f9968408618`, NIC MAC `bc:24:11:98:10:0e` |
| Never touched | qm 300 (drill), 9000, LXC 9201, Peti, hub — honored |
| Throwaway password | `$6$…` crypt hash baked into answer files (spike-only; `# THROWAWAY` in the archived answer.toml) |
| Evidence root | `180:~/spike-baremetal/` (ISOs, answers, `evidence/<leg>/`, listener/https logs) — kept |
| Spike VM final state | **stopped, kept** (per cleanup rule) pending acceptance |
## S1 — toolchain + ISO provenance — **PROVEN**
- **Assistant obtainable + pinnable per release:** `proxmox-auto-install-assistant 9.2.7`
(`proxmox-installer-common 9.2.7`) installed in a disposable `debian:trixie` Docker container on
DooPlex from the PVE **trixie** no-subscription repo. This is the intended `build-felhom-iso.sh`
host and it works. Pairing rule: install the assistant from the `pve <codename> no-subscription`
repo whose Debian codename matches the ISO generation (PVE 9.x = Debian 13 *trixie*); the assistant
point release (9.2.7) tracks the repo and pairs with the ISO by **major.minor** (9.2), not by the
ISO's point release (9.2-1).
- **ISO provenance:** `proxmox-ve_9.2-1.iso` sha256
`4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c` — matches the official
`download.proxmox.com/iso/SHA256SUMS` **and** the copy already on felhom-pve.
- **Build cost (design input #6):** container image build 37.9 s / 243 MB; `prepare-iso` ~**11.5 s**
per prepared ISO (source local); prepared ISO 1,707,278,336 B (~1.1 MB over the 1,706,178,560 B
source).
- **`validate-answer` catches a broken file** — fed one with both `root-password` and
`root-password-hashed`: `` `global.root-password` and `global.root-password-hashed` cannot be set at
the same time / Error: Found issues in the answer file.`` **TRAP (load-bearing for the pipeline):**
it prints that to **stderr but returns exit code 0** — for *both* valid and invalid files. A
`build-felhom-iso.sh` gate **must parse the output text**, never `$?`. It is also **syntactic only**:
`disk-list=["sdz"]` (a non-existent disk) passed validation and only failed at install time (see S8b).
- Evidence: `evidence/S1-S8-key-results.txt`, `~/spike-baremetal/build/build.log`, `iso/SHA256SUMS`.
## S2 — baseline zero-touch install (SeaBIOS) — **PROVEN**
- **Given/When/Then:** prepared ISO (`--fetch-from iso`, keyless `answer-base.toml`) on VM 310
(SeaBIOS, single 32 GB disk) → boot → automated install with **no keypress**, reboot into working PVE.
- Boot menu **auto-selected "Install Proxmox VE (Automated)"** with a 5 s countdown
(`evidence/S2/boot-8s.png`); unattended package extraction (`evidence/S2/mid-install.png`); **no
interactive prompt at any point**. Install wall-clock **≈ 3 min** (start 09:45:33 → booting installed
PVE by 09:48:48).
- Installed system reachable over IPv6 link-local: `22` OPEN (`SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u4`),
`8006` OPEN. TLS cert **CN=spike310.felhom.local** (configured fqdn applied); SAN carries the
**DHCP-assigned IPv4 192.168.0.115**. Boot-order `scsi0;ide2` fall-through: the post-install reboot
landed on the installed disk (no reinstall loop).
- Evidence: `evidence/S2/verdict.txt`, `boot-8s.png`, `mid-install.png`.
## S2b — UEFI + Secure Boot — **PROVEN** (no caveat needed)
- **UEFI (OVMF/q35, Secure Boot OFF, `pre-enrolled-keys=0`):** automated install identical, zero
keypress (`evidence/S2b/boot-uefi-installer.png`). Keyed SSH login worked; `/sys/firmware/efi`
present → **genuine UEFI boot**; `mokutil`: *SecureBoot disabled / Setup Mode*.
**Password mechanism proven non-hollow:** the installed `/etc/shadow` root hash equals the answer's
`root-password-hashed` **verbatim** → the password would work as configured.
- **Secure Boot ON (`pre-enrolled-keys=1`, MS 2023 certs, enforcing):** the PVE ISO's MS-signed shim
chain booted the automated installer (`evidence/S2b/sbon-18s.png`), the install completed, **and the
installed PVE booted under Secure Boot enforcing** — `mokutil: SecureBoot enabled`, `BOOTED_VIA=UEFI`,
lockdown=integrity, **zero keypress, no MOK-enrollment prompt**. → **R-21 needs no Secure-Boot
caveat**: customer mini-PCs with SB on by default install and boot fine.
- **Bonus finding:** the PVE installer writes a **dual BIOS+UEFI-bootable disk** — the S2 SeaBIOS
install also booted under OVMF.
- **Re-imaging gotcha (finding):** the first UEFI reinstall over a disk that still held the prior
install's **LVM PV signature** (only first/last-10 MiB wiped) **aborted**: *"unable to initialize
physical volume /dev/sda3 / low level installer returned early"*. The auto-installer's
"cleanup root-disks" step does **not** force-clear a pre-existing LVM PV. A full `blkdiscard` then
installed cleanly. → a customer disk with a prior LVM/Proxmox/Linux install may need an explicit
wipe before the automated install.
## S3 — first-boot hook: privileges, ordering, exactly-once — **PROVEN**
- **`--on-first-boot stub.sh`, `[first-boot] source="from-iso" ordering="fully-up"`.** Unit is
**`proxmox-first-boot-multi-user.service`** ("Fully Booted"),
`After=… pveproxy.service pmgproxy.service proxmox-backup-proxy.service`.
- **Runs as root (uid 0).** Under **fully-up**, `pvesh get /nodes` returned a **complete** node object
(status online, ssl_fingerprint present, uptime) `pvesh_rc=0`; `pct list` `pct_rc=0`; outbound
`curl` hit the DooPlex listener — `GET /firstboot?host=spike310&uid=0 from 192.168.0.115` (network
truly up). This is the ordering the real wrapper needs (it drives pveum/pct/pvesh).
- **Exactly-once mechanism:** unit has `ConditionPathExists=/var/lib/proxmox-first-boot/pending-first-boot-setup`
and `ExecStartPost=/usr/bin/rm … pending-first-boot-setup`. Empirically: after a reboot the hook log
is **byte-identical** (16 lines, 1 marker, sha256 unchanged), the unit is `inactive(dead)` with
*"ConditionPathExists … was not met"* (`ConditionResult=no`), and the listener shows **exactly 1**
firstboot hit.
- **Comparison leg — `ordering="network-online"`** (`proxmox-first-boot-network-online.service`):
pvesh returned **degraded** data — `'/etc/pve/nodes/spike310/pve-ssl.pem' does not exist!`,
`status:"unknown"`, `ssl_fingerprint:null` (pmxcfs/pveproxy not yet ready). Note `pvesh_rc=0` in
**both** orderings — the exit code lies; only **fully-up** yields a coherent PVE stack.
- Evidence: `evidence/S3/firstboot-network-online.log`, listener log, `evidence/S1-S8-key-results.txt`.
## S4 — post-installation-webhook payload — **PROVEN** (full payload archived)
- `[post-installation-webhook] url="http://…:8088/webhook"` (plain HTTP; no cert-fingerprint required).
Fired **near end of install, before reboot** (~2 m 42 s in). `POST … application/json`, `ureq/3.0.11`,
**5052 B**, schema **v1.2**. Full body: `evidence/S4/webhook-payload.json`.
- **Identity fields the hub's unclaimed-appliance record can key on:**
- `dmi.system.uuid` = `14a82ff6-…` — SMBIOS/product UUID (stable hardware id).
- `network-interfaces[].mac` = `bc:24:11:98:10:0e` (+ `is-management:true`, DHCP `address`).
- `machine-id` (regenerates per install — not a hardware id), `fqdn`, `boot-info.mode` ("bios"),
`product.version` 9.2.2, `debian-version` 13.5, `cpu-info.model` "Intel(R) N100".
- `disks[].udev-properties.ID_SERIAL / ID_SERIAL_SHORT / ID_PATH` — disk serials.
- **`ssh-public-host-keys`** (ecdsa/ed25519/rsa) — lets the hub pin the box's SSH host keys for
later TOFU-free access.
- **CAVEATS (design input #2):**
- On **nested virt** `dmi.baseboard` is `{}`, `dmi.chassis.serial` and `dmi.system.serial` are **empty**.
On real bare metal these carry vendor board/chassis/system serials — **unprovable here** (OPEN item).
- **Webhook `disks[].size` is unreliable:** the 8 GB second disk was reported as 32 GB (it echoed the
boot disk's size); the guest's own `lsblk` shows 8 GB. Key on disk **serials**, not `disks[].size`.
## S5 — disk-filter safety (LOAD-BEARING) — **PROVEN**
Two disks: sda=32 GB target, sdb=8 GB ext4 canary (`LABEL=FELHOM_CANARY`,
`UUID=1111…5555`, `canary.bin` sha256 `0d22f95b…`, `canary.txt` `42139528…`; `evidence/S5/canary-before.sha`).
Canary verified by **mounting the LV on the host and hashing**, not from installer output.
| Sub-run | Selection | Result |
|---|---|---|
| (a) | `disk-list=["sda"]` | installed on sda; sdb canary `sha256 -c: OK` (byte-identical), UUID unchanged. **PROVEN** |
| (b) | `filter.ID_SERIAL_SHORT="drive-scsi0"` (matches only sda) | installed on sda (lsblk: sda=pve LVM, sdb=ext4 FELHOM_CANARY); canary `OK`. **PROVEN** |
| (c) | `filter.ID_SERIAL_SHORT="no-such-disk-zzz"` (matches nothing) | **`ERROR: Installation failed: filter did not match any device` → `Auto-installation failed (exit-code 1)` → aborts to installer root shell.** No interactive fallback, **no disk touched**, canary intact. `evidence/S5/matchnothing-console.png` |
## S6 — first-boot from-url + cert-fingerprint pinning — **PROVEN (fails closed)**
- Stub served over **HTTPS with a self-signed cert** from DooPlex; `[first-boot] source="from-url"
url="https://…:9443/stub.sh" cert-fingerprint=<SHA256> ordering="fully-up"`. The pin is checked at
**install time** (in the installer environment), not first boot.
- **Correct fingerprint:** installer fetched the stub — HTTPS log
`192.168.0.115 "GET /stub.sh HTTP/1.1" 200` — and it ran at first boot (uid 0, log present, unit active).
- **RED-PROOF — wrong fingerprint (`F6…`→`A6…`):** *"INFO: Fetching first-boot hook from https://…/stub.sh
.. ERROR: Autoinstaller setup error: io: unexpected error: **Fingerprint did not match!** →
Auto-installation failed (exit-code 1)"* — the **entire install aborts**; the stub was **never
fetched** (HTTPS log empty) and never ran. `evidence/S6/redproof-console.png`.
- **Verdict: pinning FAILS CLOSED** — fit for the wrapper download story (from-url is safe; from-iso is
not required for security).
## S7 — HTTP answer-fetch identity (stretch) — **PROVEN**
- `--fetch-from http --url http://…:8088/answer`. The installer **POSTs** for its answer:
`application/json`, `ureq/3.0.11`, `accept: application/json, application/toml;q=0.5`, schema **v1.0**,
**388 B**. Body carries: `dmi.system.uuid` `14a82ff6-…`, `network_interfaces[].mac`
`BC:24:11:98:10:0E`, `product`/`iso` info, and (empty on virt) DMI serials — **but no disk info yet**
(disks aren't enumerated at answer-fetch time). → option B (hub serves a per-customer answer) is
viable, keyed on **MAC + SMBIOS UUID**. Full request in the listener log after the `S7-HTTPFETCH` marker.
## S8 — failure modes — **PROVEN**
- **(a) No DHCP / no link at first boot** (host-side `qm reboot` with NIC `link_down=1`; hook status
read via the pending-flag over IPv6 link-local — no reboot needed):
- **fully-up:** **hook RUNS** — flag consumed, new marker, pvesh works locally, outbound `curl` fails
("Could not connect"). Boot reaches login. → robust to "customer's router is off".
- **network-online:** **hook does NOT run** — flag persists, unit `inactive(dead)` (gated on
`network-online.target`, unsatisfied without a DHCP-configured link); boot still reaches login
(**no infinite hang**). Hook stays armed for a later boot once network returns.
- *(Method note: a first attempt set `link_down=1` before an ssh `reboot`, which then couldn't reach
the guest — the reboot never fired. Redone host-side. The two "*-nonet-console.png" captures are
from that stale boot and are superseded by the flag-check result in `evidence/S1-S8-key-results.txt`.)*
- **(b) Invalid answer — `disk-list=["sdz"]` (non-existent disk):** validate-answer **passed** it
(syntactic only); at install: *"ERROR: Installation failed: disk in 'disk-selection' not found →
Auto-installation failed (exit-code 1)"*, aborts to shell, no disk touched, **no interactive
fallback**. `evidence/S8/invalid-disk-console.png`.
## 13. Evidence
All under `180:~/spike-baremetal/evidence/`: per-leg console PNGs (S2/S2b/S5/S6/S8), the full webhook
JSON (`S4/webhook-payload.json`), the request-logging listener (`listener.log`) and HTTPS access log
(`https.log`), canary hashes (`S5/canary-before.sha`), and the consolidated
`S1-S8-key-results.txt`. Console captures were taken with QEMU-monitor `screendump` → PPM → PNG
(felhom-pve has no image tools; converted on DooPlex). Prepared ISOs + source ISO retained under
`180:~/spike-baremetal/iso/`.
## 15. Design inputs for R-21 (bare-metal ISO) — *(the spike brief calls this feature "R-22")*
1. **First-boot ordering → `fully-up`.** Proven: under fully-up, pvesh returns a *complete* node object
and pct works (S3); under network-online pvesh is degraded (`pve-ssl.pem` missing) and the hook does
not even run without a DHCP link (S8a). The wrapper (drives pveum/pct/pvesh) must use fully-up. The
exactly-once guard is `ConditionPathExists=…/pending-first-boot-setup` + `ExecStartPost=rm` — re-arm
by re-touching that flag if a redo is ever needed.
2. **Webhook keys for the unclaimed-appliance record:** `dmi.system.uuid` (SMBIOS/product UUID) +
management-NIC `mac` are the stable pair; also archive `ssh-public-host-keys` (host-key pinning),
`fqdn`, `machine-id` (per-install, not hardware), `cpu-info.model`, disk `ID_SERIAL`. **Do not key on
`disks[].size`** (unreliable — S4). Real board/chassis/system serials are empty on virt → confirm on
bare metal before relying on them.
3. **Disk-selection policy for alpha:** both `disk-list=["sda"]` and a udev `filter.*` glob install only
the intended disk and leave other disks **byte-identical** (S5 a/b, proven by host-side hashing). A
**match-nothing** filter and a **non-existent disk name** both **fail-safe**: abort with exit-code 1,
touch no disk, no interactive fallback (S5c, S8b). Recommendation: **single-disk mini-PC → a
size/model `filter.*` glob** (portable across identical hardware, self-documenting), with the
match-nothing abort as the built-in wrong-hardware guard. Add a pre-install **LVM/partition wipe**
step (or document it) — the installer does not force-clear a prior LVM PV (S2b re-imaging gotcha).
4. **UEFI / Secure Boot:** no caveat required. Automated install and the installed PVE both work under
UEFI with Secure Boot **enforcing** (pre-enrolled MS keys), zero keypress, no MOK prompt (S2b). A
single install image is BIOS+UEFI-bootable.
5. **Pinning verdict:** `from-url` + `cert-fingerprint` is **fit** — a wrong pin aborts the whole
install (fails closed, S6). The wrapper/stub may be fetched from a URL with a pinned fingerprint;
`from-iso` (baking) is not required for security. Note the cert-fingerprint format is the
colon-separated uppercase SHA256 (`openssl x509 -fingerprint -sha256`).
6. **Assistant↔ISO pairing + build cost → `build-felhom-iso.sh`:** install the assistant from the PVE
`no-subscription` repo of the ISO's Debian codename (PVE 9.x = trixie); pair by major.minor. Per
prepared ISO ≈ 11.5 s + ~1.7 GB artifact; the container build is ~40 s one-time. The pipeline runs
fine on DooPlex (the build host). **Gate on parsed `validate-answer` output, never `$?`** (exit 0 on
failure). Remember `validate-answer` is syntactic only — disk existence is a runtime check.
7. **Failure-mode table (customer-facing "what if it goes wrong"):**
| Scenario | Behavior | Customer/operator signal |
|---|---|---|
| Wrong hardware — filter matches nothing | install aborts, exit 1, no disk touched | box never finishes install; stays on ISO/shell |
| Non-existent disk in answer | install aborts, exit 1, no disk touched | same |
| Prior LVM/OS on disk | install aborts ("cannot init PV") | needs a wipe step (see #3) |
| Wrong cert-fingerprint (from-url) | whole install aborts, exit 1 | box never finishes install |
| No internet at first boot (fully-up) | hook runs; net-dependent steps fail-and-retry | box installed, wrapper fired, retries when net returns |
| No internet at first boot (network-online) | hook does not run; boot OK | avoid this ordering |
8. **OPEN — needs one real bare-metal run (not provable on nested virt):**
- Real vendor **DMI serials** (baseboard/chassis/system) in the webhook + answer-fetch payloads
(empty on QEMU) — decide whether to key on them.
- Real firmware quirks: vendor UEFI boot-entry handling, whether the disk-first→CD fall-through works
on a real board (worked on SeaBIOS and OVMF here), and Secure Boot with a *vendor's* (non-MS-clean)
key set.
- Real-NIC naming (this VM enumerated `ens18`) and multi-NIC `is-management` selection.
- Real DHCP behavior / captive networks; USB-stick boot media (we booted a virtual CD).
9. **Observations (recorded, not acted on):**
- `validate-answer` exit-0-on-failure is arguably a Proxmox bug; regardless, the pipeline must parse output.
- The probe stub's `curl_rc` reflected the trailing pipe stage, not curl — rely on curl's error text.
- Webhook `disks[].size` mis-report (see #2) is a Proxmox data-quality issue worth reporting upstream.
- `machine-id` regenerates per install → not a durable hardware identity for re-imaged boxes.
## Cleanup state
Spike VM 310 **stopped, kept** (config intact, s8b ISO still attached). ISOs + evidence retained under
`180:~/spike-baremetal/`. DooPlex helper listeners (8088 answer/webhook, 9443 HTTPS, 8099 file-server)
were used only during the spike and may be stopped. Nothing else on either host was modified.
documentation/backlog/ROADMAP.md
+1
View File
@@ -49,6 +49,7 @@
| R-13 | OOB management arc: dual-use existing WireGuard + hub desired-state channel as mutual-repair | L | idea | |
| R-14 | Headscale/WireGuard spike: Minecraft/gaming port connectivity (CGNAT-proof, sovereign DERP fallback) | M | idea | |
| R-15 | Multi-user dashboard accounts (household members, roles) | L | idea | Single password is a stated alpha limitation (R-11) |
| R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → claim-code pairing (option C) | XL | **spiked** (2026-07-16) | Spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* |
## Absorbed / superseded notes in this folder