felhom.eu/scripts/iso/README.md

# Felhom bare-metal ISO pipeline (R-21 slice A + B)

Turns the official Proxmox VE ISO into a **Felhom auto-install ISO** whose first-boot chain,
unattended, fetches and runs `felhom-host-install.sh` until the host is enrolled and a guest is
provisioned. Boot the ISO on blank hardware → walk away → managed Felhom host.

Grounding: every install/first-boot/webhook mechanism used here is proven in
`documentation/audits/SPIKE-baremetal-iso-2026-07-16.md` (S1–S8). This pipeline **composes** them;
it does **not** modify `felhom-host-install.sh` (which it only invokes over the public distribution
channel).

> **Scope:** slice A = the pipeline + the in-ISO bootstrap, validated on nested VM 310. The physical
> N100 run is **RUNBOOK B** (separate, supervised). The customer-facing claim/pairing that removes the
> secret from the ISO is **slice C** (not built here).

## Files

| File | Role |
|---|---|
| `Dockerfile.assistant` | build host: Debian trixie + `proxmox-auto-install-assistant` (paired to the ISO's PVE generation by Debian codename) + `xorriso` + `shellcheck` |
| `build-felhom-iso.sh` | the pipeline: render answer, mint throwaway root hash, **validate-answer output-parse gate**, render stub, `prepare-iso`, **optional `--loader mkimage` surgery**, emit ISO + sha256 + manifest |
| `iso-repack.sh` | the post-`prepare-iso` repack (runs in the container). **v1.22.0/R-38:** Felhom GRUB branding + **single-entry menu surgery**; plus, in `--loader mkimage`, the slice-B monolithic `grub-mkimage` UEFI loader swap (F1 firmware fix). One extract→modify→re-master pass for both. *(was `mkimage-surgery.sh` ≤ v1.21.0)* |
| `grub/grub.cfg.tmpl` | the replacement GRUB menu: exactly **one** entry („Felhom telepítés", default, 5 s). The `linux`/`initrd` lines are lifted verbatim at repack time from the ISO's own *Install Proxmox VE (Automated)* entry, so a PVE bump tracks automatically |
| `grub/felhom-theme.txt` | gfxmenu theme — positions the menu in the clean lower third of the boot card (plain `background_image` cannot position the menu, so the text would land on the wordmark) |
| `grub/generate-grub-background.sh` | letterboxes `website/assets/og-image_2.png` onto a 1024×768 gfxterm canvas, phase-locking a continuation of the card's own background grid across the fill |
| `answer.toml.tmpl` | Proxmox answer template (`__FQDN__` / `__ROOT_HASH__` / `__ROOT_SSH_KEYS__` / `__DISK_SETUP__` placeholders; `[first-boot]` from-iso/fully-up) |
| `profiles/*.profile` | disk-selection + fqdn fragments (sourceable, no secret) |
| `stub-first-boot.sh` | the ONE first-boot executable (skeleton; build injects the bootstrap script/unit/env as base64) |
| `felhom-bootstrap.sh` | per-attempt: fetch host-install from the public channel → run it with the retrieval passphrase → on rc 0 write done-flag + disable |
| `felhom-bootstrap.service` | retry-forever unit (`Type=oneshot`, `Restart=on-failure`, `RestartSec=30`, `StartLimitIntervalSec=0`, **`TimeoutStartSec=infinity`** — v1.21.0/R-33: the PAIRING wait loops inside the script, so the unit sits in `activating` instead of failing every 30s; without the lifted timeout systemd would kill it at 90s and restore the console spam) |

## Build

```bash
# 1. build the assistant image once (rebuild when the target PVE generation changes)
docker build -f scripts/iso/Dockerfile.assistant -t felhom-iso-assistant:trixie scripts/iso

# 2. build an ISO (on the build host; PVE ISO pre-downloaded + its official sha256)
scripts/iso/build-felhom-iso.sh \
  --pve-iso   /path/proxmox-ve_9.2-1.iso \
  --iso-sha256 4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c \
  --profile   scripts/iso/profiles/nested-vm.profile \
  --bootstrap-env /secure/bootstrap.env \
  --out       /mnt/5_hdd/felhom.eu/felhom-iso/out
```
> On the DooPlex build server (180) the felhom working dirs were moved off the SSD to
> `/mnt/5_hdd/felhom.eu/` (2026-07-18): the input PVE ISO lives at
> `/mnt/5_hdd/felhom.eu/drill/proxmox-ve_9.2-1.iso`. The script's `--out` **default is now
> `/mnt/5_hdd/felhom.eu/felhom-iso/out`** (override with `$FELHOM_ISO_OUT` on other hosts), so on 180 you
> can omit `--out` entirely and output still lands on the HDD, never the SSD.

Output: `felhom-pve-<pvever>-v<isover>-<profile>.iso` + `.sha256` + `.manifest.txt`.

The build **gates the answer on validate-answer's OUTPUT TEXT, never `$?`** — `validate-answer`
returns exit 0 even on a broken file (spike S1 trap; also `prepare-iso` was observed to exit 0 on a
bad answer). A broken answer aborts the build with the validator's message and produces **no ISO**.
Note `validate-answer` is *syntactic only*: disk existence is a runtime property (a non-existent disk
passes the build and fails-safe at install — spike S5c/S8b).

## Profile format

A sourceable shell fragment:

```sh
FELHOM_FQDN="felhom-host.local"
FELHOM_DISK_SETUP='[disk-setup]
filesystem = "ext4"
disk-list = ["sda"]'
# optional emergency/validation key baked into root's authorized_keys (blank -> not baked):
# FELHOM_ROOT_SSH_KEY="ssh-ed25519 AAAA... ops@felhom"
```

`FELHOM_DISK_SETUP` is any valid Proxmox `[disk-setup]` block — a `disk-list` OR a udev `filter.*`
glob. **A filter matching nothing, or a non-existent disk, fails-safe**: the installer aborts (exit
1, no disk touched) rather than installing on the wrong disk (spike S5c). Never commit a real key in
`FELHOM_ROOT_SSH_KEY` unless it is meant to ship in every ISO built from that profile.

A profile may also set `FELHOM_LOADER="shim"|"mkimage"` (see below); `--loader` on the command line
overrides it.

## Loader modes (`--loader shim|mkimage`, slice B)

The UEFI boot loader is selectable. Default **shim**; a profile can pin `FELHOM_LOADER`; `--loader`
wins over both.

- **`shim`** (default) — the stock output: the ISO's MS-signed shim → GRUB chain. **Secure Boot keeps
  working** on compliant firmware (no MOK, no keypress — spike S2b). Use this everywhere it boots.
- **`mkimage`** — replaces the ISO's UEFI boot path with a **monolithic `grub-mkimage` loader** built
  from the ISO's own GRUB modules. This is the fix for cheap AMI (`AN3PLUS 0.01`-class) mini-PC
  firmware that **cannot UEFI-boot the ISO's stock GRUB from USB** (`relocation 0x0 is not implemented
  yet`; SB-off and shim-bypass don't help) — the F1 finding of `tests/VALIDATION-n100-baremetal-2026-07-16.md`,
  worked around live during that run and now first-class. The recipe (module set from the ISO's
  `grub.cfg`, embedded `search --fs-uuid` → `configfile` the real menu) is reproduced from the run
  evidence, not re-derived.

  > **⚠ mkimage is UNSIGNED → the target board MUST have Secure Boot OFF.** With Secure Boot
  > enforcing the firmware refuses the loader (`Access Denied`, no boot — proven, RUNBOOK-B leg 3).
  > mkimage ISOs are named with a **`-mkimage`** suffix and the manifest's `loader:` line says so;
  > never confuse them with a shim (SB-capable) build. shim mode's output is byte-for-byte the same as
  > before this flag existed (the surgery is skipped entirely).

  The surgery runs **after** `prepare-iso` and touches only the EFI boot path — the answer file and
  the on-first-boot payload are provably byte-identical to the shim variant's (RUNBOOK-B leg 4).

## bootstrap-env (SECRET-BEARING)

The `--bootstrap-env` file becomes the in-ISO `/etc/felhom/bootstrap.env`:

```sh
FELHOM_CUSTOMER_ID=<hub customer id>          # required
FELHOM_MODE=appliance                         # required (appliance|byo)
FELHOM_RETRIEVAL_PASSPHRASE=<customer passphrase>   # required — SECRET
# optional:
# FELHOM_HUB_URL=https://hub.felhom.eu
# FELHOM_INSTALL_URL=https://felhom.eu/scripts/felhom-host-install.sh
# FELHOM_EXTRA_ARGS="--cores 4 --memory 8192"   # profile-only flags, never secrets
```

**Why the ISO is secret-bearing (§4.4 finding):** the hub install-command's *distribution channel* is
the public `https://felhom.eu/scripts/felhom-host-install.sh` — no token, no secret in the command.
But `felhom-host-install.sh` requires the customer **retrieval passphrase** (normally a no-echo
prompt) to fetch the config and enroll the host. An *unattended* install must supply it via
`--passphrase-file`, so the ISO embeds it. **Rules for a secret-bearing ISO:**

- supervised / single-use only; **never distributed**; **delete after the run**.
- issue the passphrase with the shortest usable lifetime; rotate/retire after the install.
- the bootstrap `shred`s `bootstrap.env` on the box once host-install succeeds (reduces secret-at-rest).
- **slice C** removes this: the claim/pairing flow delivers the passphrase (or config) to the box
  *after* the customer claims it, so the *distributed* ISO carries no secret.

## The first-boot chain

`stub-first-boot.sh` (exactly-once, `[first-boot]` from-iso/fully-up) is **dumb**: it lays down
`/usr/local/sbin/felhom-bootstrap.sh`, the systemd unit, and `/etc/felhom/bootstrap.env` (0600), then
enables + starts the unit. All fallible/network work lives in `felhom-bootstrap.service`, which
retries forever until `felhom-host-install.sh` exits 0, then writes `/etc/felhom/.bootstrap-done`,
disables itself, and scrubs the env. Retry-vs-resume: the first attempt is plain; any later attempt
that finds `/var/lib/felhom-install/state.json` adds `--resume` (safe per host-install v1.11.3 — its
producer steps re-run each pass).

## HP mini profile (second pilot board, 2026-07-21)

`profiles/hp.profile` — same shape as the N100 template with ONE deliberate difference: **the loader
default stays at the stock signed `shim`**, so Secure Boot keeps working. `mkimage` exists only to
work around the N100's AMI firmware GRUB defect and is unsigned.

**PROVEN on this board (2026-07-21):** the safety boot loaded shim, fetched and parsed the answer file,
and the match-nothing filter refused with **zero disk writes**; the armed ISO then installed end to end.
**Secure Boot stayed ENABLED throughout** (`mokutil --sb-state` on the installed box → `SecureBoot
enabled`) — so SB-off is an N100-firmware workaround, not a Felhom requirement.
so the loader, Secure Boot setting, and network/answer path are all confirmed before anything
destructive existed on a stick. `enp1s0f0` auto-detect verified on-board.

**Arming it:** never edit the committed profile. Copy it to an UNCOMMITTED working profile (the
`n100-demo.profile` precedent — a real serial never goes in git), replace the placeholder serial, and
rebuild. Verify the armed filter by extracting `answer.toml` **from inside the built ISO**, and again
from the written stick — the profile is what you intended, the ISO is what you shipped:

```
osirrox -indev <iso> -extract /answer.toml /tmp/a.toml && sed -n '/^\[disk-setup\]/,$p' /tmp/a.toml
```

**t740 NIC TRAP — the 4-port card gets no DHCP; use the onboard port.** This board presents FIVE
wired NICs and the install picks wrong:

| interface | driver | what it is | at this site |
|---|---|---|---|
| `enp1s0f0`–`f3` | `igb` | the 4-port expansion card | **no DHCP lease** (no link) |
| `enp2s0f0` | `r8169` | **the onboard port** | leases fine, 1000 Mb |
| `wlo1` | `iwlwifi` | wifi | unused |

Plug the cable into the **onboard** port. If the install already happened on the wrong port, the
symptom is nasty: the installer does not abort, it bakes its **192.168.100.2 fallback as a STATIC
`vmbr0` address** and completes, so the box looks installed and can never reach the hub (R-59).
Repair on the console: point `bridge-ports` at `enp2s0f0` in `/etc/network/interfaces`, set the
correct address (or `dhcp`), `ifreload -a`. R-60 is the self-heal that would make this unnecessary.

**Confirm the serial is the system disk and not a data drive.** On this board the SanDisk X600 128GB
(`sda`) is the system disk; the 1TB NVMe is the future data drive and must stay OUTSIDE the filter —
it joins later through the normal Tárhely flow, not the installer.

> **The one known failure mode on a second-hand board — read step 3 below.** The auto-installer does
> NOT force-clear a pre-existing LVM PV signature. A disk carrying a previous Linux/Proxmox install
> aborts at *"cannot init physical volume"* (spike S2b). A previous Windows/NTFS install is fine. If
> the install aborts that way, wipe the target first (`blkdiscard -f /dev/<disk>`, or `wipefs -a` plus
> zeroing the PV region) from any live environment, then re-boot the same stick — the ISO does not
> need rebuilding.

## N100 profile (for RUNBOOK B)

Use the committed `profiles/n100.profile` template (cheap-board, `FELHOM_LOADER="mkimage"`). **A real
single-disk mini-PC uses a unique-serial udev filter, not `disk-list`:**

1. Boot once (or use a live env), harvest the target disk's udev id:
   `udevadm info --query=property --name=/dev/sda | grep -E 'ID_SERIAL_SHORT|ID_MODEL|ID_WWN'`.
2. In the profile, set `filter.ID_SERIAL_SHORT = "<harvested>"` + `filter-match = "all"`. A
   match-nothing filter fails-safe (won't wipe the wrong disk).
3. **Prior-LVM prerequisite:** the auto-installer does **not** force-clear a pre-existing LVM PV
   signature — a disk with a previous Linux/Proxmox install aborts at "cannot init physical volume"
   (spike S2b). Wipe first: `blkdiscard -f /dev/<disk>` (or `wipefs -a` + zero the PV region).
4. **Boot loader + Secure Boot (F1):** cheap AMI (`AN3PLUS`-class) firmware can't boot the stock GRUB
   from USB, so `profiles/n100.profile` uses `--loader mkimage`. **That loader is unsigned → set
   Secure Boot to OFF/Disabled in the BIOS** before booting the stick (with SB enforcing the board
   refuses it: `Access Denied`, no boot). Firmware that boots the stock ISO fine needs no mkimage and
   keeps Secure Boot working (default `shim`, spike S2b). *Optional pre-flight (operator, no risk):* a
   `n100-safety` ISO — `--loader mkimage` + a **match-nothing** filter — boots the board, loads GRUB,
   and the installer aborts touching nothing (S5c). It confirms the loader + SB-off setting before the
   real install; its sha is recorded at build. Never boot the real install-armed ISO except in the
   supervised rehearsal.
5. **BIOS "State After G3" → "Power On" (or "Last State")** before final sign-off — so the mini-PC
   auto-powers-on after a mains loss (a managed appliance must come back without a physical press).
   Set it in the BIOS power/ACPI menu; it is board-specific and cannot be scripted (operator step).

## Validation status (nested VM 310, 2026-07-16)

PASS: build gate + red-proof (C); disk-filter fail-safe through the pipeline (D); stub→unit→**real
public-channel fetch**→correct host-install invocation→retry; 2.3 resume decision; exactly-once stub;
no-network retry + auto-recovery (B). **Operator-gated (not live-proven here):** host-install rc-0
terminal success → hub-enrolled + guest-provisioned + done-flag + unit-disable — needs a properly
created drill customer (the hub create-UI is password-gated and a live-DB insert is unsafe). Also
out of scope: the physical N100 run (RUNBOOK B).