admin
3172df1927
Closes N100 F1 (HIGH): cheap AMI (AN3PLUS 0.01-class) UEFI firmware can't relocate the ISO's stock signed GRUB from USB (relocation 0x0). The run's live grub-mkimage workaround is now a first-class pipeline mode. - build-felhom-iso.sh: --loader shim|mkimage (default shim, byte-for-byte unchanged; profile-settable FELHOM_LOADER; --loader wins). Loud banner + manifest loader:/grub-mkimage: fields + -mkimage filename suffix. - mkimage-surgery.sh (new): post-prepare-iso, in the assistant container. Builds a monolithic grub-mkimage loader from the ISO's own GRUB (module set from its grub.cfg; embedded search --fs-uuid -> configfile the real menu). Swaps it into the ISO9660 tree (real lowercase path) + the efi.img ESP; xorriso re-master preserves BIOS-hybrid + UEFI + GPT-ESP, drops only Apple HFS+/APM. Recipe from the N100 run evidence, not re-derived. - Dockerfile.assistant: grub-common + grub-efi-amd64-bin + mtools + dosfstools. profiles/n100.profile (new, mkimage + SB-off note). - Validated on nested VM 311 (RUNBOOK-B legs): leg1 shim boots+installs under OVMF SB-enforcing + SeaBIOS; leg2 mkimage boots+installs under SB-off; leg3 (red-proof) mkimage under SB-enforcing FAILS Access Denied (unsigned -> SB must be OFF); leg4 surgery byte-identical payload. bash -n + shellcheck clean. Physical N100 closure folds into the rehearsal (n100-safety match-nothing ISO built + sha-recorded, unbooted). PXE stays a deferred R-21 note.
166 lines
10 KiB
Markdown
166 lines
10 KiB
Markdown
# Felhom bare-metal ISO pipeline (R-21 slice A + B)
|
||
|
||
Turns the official Proxmox VE ISO into a **Felhom auto-install ISO** whose first-boot chain,
|
||
unattended, fetches and runs `felhom-host-install.sh` until the host is enrolled and a guest is
|
||
provisioned. Boot the ISO on blank hardware → walk away → managed Felhom host.
|
||
|
||
Grounding: every install/first-boot/webhook mechanism used here is proven in
|
||
`documentation/audits/SPIKE-baremetal-iso-2026-07-16.md` (S1–S8). This pipeline **composes** them;
|
||
it does **not** modify `felhom-host-install.sh` (which it only invokes over the public distribution
|
||
channel).
|
||
|
||
> **Scope:** slice A = the pipeline + the in-ISO bootstrap, validated on nested VM 310. The physical
|
||
> N100 run is **RUNBOOK B** (separate, supervised). The customer-facing claim/pairing that removes the
|
||
> secret from the ISO is **slice C** (not built here).
|
||
|
||
## Files
|
||
|
||
| File | Role |
|
||
|---|---|
|
||
| `Dockerfile.assistant` | build host: Debian trixie + `proxmox-auto-install-assistant` (paired to the ISO's PVE generation by Debian codename) + `xorriso` + `shellcheck` |
|
||
| `build-felhom-iso.sh` | the pipeline: render answer, mint throwaway root hash, **validate-answer output-parse gate**, render stub, `prepare-iso`, **optional `--loader mkimage` surgery**, emit ISO + sha256 + manifest |
|
||
| `mkimage-surgery.sh` | slice-B loader surgery (runs in the container, post-`prepare-iso`): builds a monolithic `grub-mkimage` UEFI loader and swaps it into the ISO's EFI boot path (F1 firmware fix) |
|
||
| `answer.toml.tmpl` | Proxmox answer template (`__FQDN__` / `__ROOT_HASH__` / `__ROOT_SSH_KEYS__` / `__DISK_SETUP__` placeholders; `[first-boot]` from-iso/fully-up) |
|
||
| `profiles/*.profile` | disk-selection + fqdn fragments (sourceable, no secret) |
|
||
| `stub-first-boot.sh` | the ONE first-boot executable (skeleton; build injects the bootstrap script/unit/env as base64) |
|
||
| `felhom-bootstrap.sh` | per-attempt: fetch host-install from the public channel → run it with the retrieval passphrase → on rc 0 write done-flag + disable |
|
||
| `felhom-bootstrap.service` | retry-forever unit (`Type=oneshot`, `Restart=on-failure`, `RestartSec=30`, `StartLimitIntervalSec=0`) |
|
||
|
||
## Build
|
||
|
||
```bash
|
||
# 1. build the assistant image once (rebuild when the target PVE generation changes)
|
||
docker build -f scripts/iso/Dockerfile.assistant -t felhom-iso-assistant:trixie scripts/iso
|
||
|
||
# 2. build an ISO (on the build host; PVE ISO pre-downloaded + its official sha256)
|
||
scripts/iso/build-felhom-iso.sh \
|
||
--pve-iso /path/proxmox-ve_9.2-1.iso \
|
||
--iso-sha256 4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c \
|
||
--profile scripts/iso/profiles/nested-vm.profile \
|
||
--bootstrap-env /secure/bootstrap.env \
|
||
--out ~/felhom-iso/out
|
||
```
|
||
|
||
Output: `felhom-pve-<pvever>-v<isover>-<profile>.iso` + `.sha256` + `.manifest.txt`.
|
||
|
||
The build **gates the answer on validate-answer's OUTPUT TEXT, never `$?`** — `validate-answer`
|
||
returns exit 0 even on a broken file (spike S1 trap; also `prepare-iso` was observed to exit 0 on a
|
||
bad answer). A broken answer aborts the build with the validator's message and produces **no ISO**.
|
||
Note `validate-answer` is *syntactic only*: disk existence is a runtime property (a non-existent disk
|
||
passes the build and fails-safe at install — spike S5c/S8b).
|
||
|
||
## Profile format
|
||
|
||
A sourceable shell fragment:
|
||
|
||
```sh
|
||
FELHOM_FQDN="felhom-host.local"
|
||
FELHOM_DISK_SETUP='[disk-setup]
|
||
filesystem = "ext4"
|
||
disk-list = ["sda"]'
|
||
# optional emergency/validation key baked into root's authorized_keys (blank -> not baked):
|
||
# FELHOM_ROOT_SSH_KEY="ssh-ed25519 AAAA... ops@felhom"
|
||
```
|
||
|
||
`FELHOM_DISK_SETUP` is any valid Proxmox `[disk-setup]` block — a `disk-list` OR a udev `filter.*`
|
||
glob. **A filter matching nothing, or a non-existent disk, fails-safe**: the installer aborts (exit
|
||
1, no disk touched) rather than installing on the wrong disk (spike S5c). Never commit a real key in
|
||
`FELHOM_ROOT_SSH_KEY` unless it is meant to ship in every ISO built from that profile.
|
||
|
||
A profile may also set `FELHOM_LOADER="shim"|"mkimage"` (see below); `--loader` on the command line
|
||
overrides it.
|
||
|
||
## Loader modes (`--loader shim|mkimage`, slice B)
|
||
|
||
The UEFI boot loader is selectable. Default **shim**; a profile can pin `FELHOM_LOADER`; `--loader`
|
||
wins over both.
|
||
|
||
- **`shim`** (default) — the stock output: the ISO's MS-signed shim → GRUB chain. **Secure Boot keeps
|
||
working** on compliant firmware (no MOK, no keypress — spike S2b). Use this everywhere it boots.
|
||
- **`mkimage`** — replaces the ISO's UEFI boot path with a **monolithic `grub-mkimage` loader** built
|
||
from the ISO's own GRUB modules. This is the fix for cheap AMI (`AN3PLUS 0.01`-class) mini-PC
|
||
firmware that **cannot UEFI-boot the ISO's stock GRUB from USB** (`relocation 0x0 is not implemented
|
||
yet`; SB-off and shim-bypass don't help) — the F1 finding of `tests/VALIDATION-n100-baremetal-2026-07-16.md`,
|
||
worked around live during that run and now first-class. The recipe (module set from the ISO's
|
||
`grub.cfg`, embedded `search --fs-uuid` → `configfile` the real menu) is reproduced from the run
|
||
evidence, not re-derived.
|
||
|
||
> **⚠ mkimage is UNSIGNED → the target board MUST have Secure Boot OFF.** With Secure Boot
|
||
> enforcing the firmware refuses the loader (`Access Denied`, no boot — proven, RUNBOOK-B leg 3).
|
||
> mkimage ISOs are named with a **`-mkimage`** suffix and the manifest's `loader:` line says so;
|
||
> never confuse them with a shim (SB-capable) build. shim mode's output is byte-for-byte the same as
|
||
> before this flag existed (the surgery is skipped entirely).
|
||
|
||
The surgery runs **after** `prepare-iso` and touches only the EFI boot path — the answer file and
|
||
the on-first-boot payload are provably byte-identical to the shim variant's (RUNBOOK-B leg 4).
|
||
|
||
## bootstrap-env (SECRET-BEARING)
|
||
|
||
The `--bootstrap-env` file becomes the in-ISO `/etc/felhom/bootstrap.env`:
|
||
|
||
```sh
|
||
FELHOM_CUSTOMER_ID=<hub customer id> # required
|
||
FELHOM_MODE=appliance # required (appliance|byo)
|
||
FELHOM_RETRIEVAL_PASSPHRASE=<customer passphrase> # required — SECRET
|
||
# optional:
|
||
# FELHOM_HUB_URL=https://hub.felhom.eu
|
||
# FELHOM_INSTALL_URL=https://felhom.eu/scripts/felhom-host-install.sh
|
||
# FELHOM_EXTRA_ARGS="--cores 4 --memory 8192" # profile-only flags, never secrets
|
||
```
|
||
|
||
**Why the ISO is secret-bearing (§4.4 finding):** the hub install-command's *distribution channel* is
|
||
the public `https://felhom.eu/scripts/felhom-host-install.sh` — no token, no secret in the command.
|
||
But `felhom-host-install.sh` requires the customer **retrieval passphrase** (normally a no-echo
|
||
prompt) to fetch the config and enroll the host. An *unattended* install must supply it via
|
||
`--passphrase-file`, so the ISO embeds it. **Rules for a secret-bearing ISO:**
|
||
|
||
- supervised / single-use only; **never distributed**; **delete after the run**.
|
||
- issue the passphrase with the shortest usable lifetime; rotate/retire after the install.
|
||
- the bootstrap `shred`s `bootstrap.env` on the box once host-install succeeds (reduces secret-at-rest).
|
||
- **slice C** removes this: the claim/pairing flow delivers the passphrase (or config) to the box
|
||
*after* the customer claims it, so the *distributed* ISO carries no secret.
|
||
|
||
## The first-boot chain
|
||
|
||
`stub-first-boot.sh` (exactly-once, `[first-boot]` from-iso/fully-up) is **dumb**: it lays down
|
||
`/usr/local/sbin/felhom-bootstrap.sh`, the systemd unit, and `/etc/felhom/bootstrap.env` (0600), then
|
||
enables + starts the unit. All fallible/network work lives in `felhom-bootstrap.service`, which
|
||
retries forever until `felhom-host-install.sh` exits 0, then writes `/etc/felhom/.bootstrap-done`,
|
||
disables itself, and scrubs the env. Retry-vs-resume: the first attempt is plain; any later attempt
|
||
that finds `/var/lib/felhom-install/state.json` adds `--resume` (safe per host-install v1.11.3 — its
|
||
producer steps re-run each pass).
|
||
|
||
## N100 profile (for RUNBOOK B)
|
||
|
||
Use the committed `profiles/n100.profile` template (cheap-board, `FELHOM_LOADER="mkimage"`). **A real
|
||
single-disk mini-PC uses a unique-serial udev filter, not `disk-list`:**
|
||
|
||
1. Boot once (or use a live env), harvest the target disk's udev id:
|
||
`udevadm info --query=property --name=/dev/sda | grep -E 'ID_SERIAL_SHORT|ID_MODEL|ID_WWN'`.
|
||
2. In the profile, set `filter.ID_SERIAL_SHORT = "<harvested>"` + `filter-match = "all"`. A
|
||
match-nothing filter fails-safe (won't wipe the wrong disk).
|
||
3. **Prior-LVM prerequisite:** the auto-installer does **not** force-clear a pre-existing LVM PV
|
||
signature — a disk with a previous Linux/Proxmox install aborts at "cannot init physical volume"
|
||
(spike S2b). Wipe first: `blkdiscard -f /dev/<disk>` (or `wipefs -a` + zero the PV region).
|
||
4. **Boot loader + Secure Boot (F1):** cheap AMI (`AN3PLUS`-class) firmware can't boot the stock GRUB
|
||
from USB, so `profiles/n100.profile` uses `--loader mkimage`. **That loader is unsigned → set
|
||
Secure Boot to OFF/Disabled in the BIOS** before booting the stick (with SB enforcing the board
|
||
refuses it: `Access Denied`, no boot). Firmware that boots the stock ISO fine needs no mkimage and
|
||
keeps Secure Boot working (default `shim`, spike S2b). *Optional pre-flight (operator, no risk):* a
|
||
`n100-safety` ISO — `--loader mkimage` + a **match-nothing** filter — boots the board, loads GRUB,
|
||
and the installer aborts touching nothing (S5c). It confirms the loader + SB-off setting before the
|
||
real install; its sha is recorded at build. Never boot the real install-armed ISO except in the
|
||
supervised rehearsal.
|
||
5. **BIOS "State After G3" → "Power On" (or "Last State")** before final sign-off — so the mini-PC
|
||
auto-powers-on after a mains loss (a managed appliance must come back without a physical press).
|
||
Set it in the BIOS power/ACPI menu; it is board-specific and cannot be scripted (operator step).
|
||
|
||
## Validation status (nested VM 310, 2026-07-16)
|
||
|
||
PASS: build gate + red-proof (C); disk-filter fail-safe through the pipeline (D); stub→unit→**real
|
||
public-channel fetch**→correct host-install invocation→retry; 2.3 resume decision; exactly-once stub;
|
||
no-network retry + auto-recovery (B). **Operator-gated (not live-proven here):** host-install rc-0
|
||
terminal success → hub-enrolled + guest-provisioned + done-flag + unit-disable — needs a properly
|
||
created drill customer (the hub create-UI is password-gated and a live-DB insert is unsafe). Also
|
||
out of scope: the physical N100 run (RUNBOOK B).
|