admin
a12c6f9730
Virgin-ISO nested drill closed the train: dead-NIC install baked the fallback (incl. the dead default gateway), the R-59 screen painted (capture committed beside the spike doc), the cable move healed + registered at the hub in 23s unaided, and the build's rootpw file matched the installed box's shadow hash. R-59 SHIPPED with the recorded deviation (first-boot gate; installer-initrd abort out of scope by operator ack). R-60 SHIPPED (spike + drill cited; F-P9 route-flush fix included). R-61 slice 1 SHIPPED. New R-62 row (hub delete-dialog cosmetics, XS). Capability map: new PROVEN-LIVE row (nested != metal, said so). Cleanup verified: felhom-pve interfaces byte-identical, bridge/VMs/ISO removed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UuFPHmHNrCJj1VhY6QdDMU
238 lines
16 KiB
Markdown
238 lines
16 KiB
Markdown
# Felhom bare-metal ISO pipeline (R-21 slice A + B)
|
||
|
||
Turns the official Proxmox VE ISO into a **Felhom auto-install ISO** whose first-boot chain,
|
||
unattended, fetches and runs `felhom-host-install.sh` until the host is enrolled and a guest is
|
||
provisioned. Boot the ISO on blank hardware → walk away → managed Felhom host.
|
||
|
||
Grounding: every install/first-boot/webhook mechanism used here is proven in
|
||
`documentation/audits/SPIKE-baremetal-iso-2026-07-16.md` (S1–S8). This pipeline **composes** them;
|
||
it does **not** modify `felhom-host-install.sh` (which it only invokes over the public distribution
|
||
channel).
|
||
|
||
> **Scope:** slice A = the pipeline + the in-ISO bootstrap, validated on nested VM 310. The physical
|
||
> N100 run is **RUNBOOK B** (separate, supervised). The customer-facing claim/pairing that removes the
|
||
> secret from the ISO is **slice C** (not built here).
|
||
|
||
## Files
|
||
|
||
| File | Role |
|
||
|---|---|
|
||
| `Dockerfile.assistant` | build host: Debian trixie + `proxmox-auto-install-assistant` (paired to the ISO's PVE generation by Debian codename) + `xorriso` + `shellcheck` |
|
||
| `build-felhom-iso.sh` | the pipeline: render answer, mint throwaway root hash, **validate-answer output-parse gate**, render stub, `prepare-iso`, **optional `--loader mkimage` surgery**, emit ISO + sha256 + manifest |
|
||
| `iso-repack.sh` | the post-`prepare-iso` repack (runs in the container). **v1.22.0/R-38:** Felhom GRUB branding + **single-entry menu surgery**; plus, in `--loader mkimage`, the slice-B monolithic `grub-mkimage` UEFI loader swap (F1 firmware fix). One extract→modify→re-master pass for both. *(was `mkimage-surgery.sh` ≤ v1.21.0)* |
|
||
| `grub/grub.cfg.tmpl` | the replacement GRUB menu: exactly **one** entry („Felhom telepítés", default, 5 s). The `linux`/`initrd` lines are lifted verbatim at repack time from the ISO's own *Install Proxmox VE (Automated)* entry, so a PVE bump tracks automatically |
|
||
| `grub/felhom-theme.txt` | gfxmenu theme — positions the menu in the clean lower third of the boot card (plain `background_image` cannot position the menu, so the text would land on the wordmark) |
|
||
| `grub/generate-grub-background.sh` | letterboxes `website/assets/og-image_2.png` onto a 1024×768 gfxterm canvas, phase-locking a continuation of the card's own background grid across the fill |
|
||
| `answer.toml.tmpl` | Proxmox answer template (`__FQDN__` / `__ROOT_HASH__` / `__ROOT_SSH_KEYS__` / `__DISK_SETUP__` placeholders; `[first-boot]` from-iso/fully-up) |
|
||
| `profiles/*.profile` | disk-selection + fqdn fragments (sourceable, no secret) |
|
||
| `stub-first-boot.sh` | the ONE first-boot executable (skeleton; build injects the bootstrap script/unit/env as base64) |
|
||
| `felhom-bootstrap.sh` | per-attempt: fetch host-install from the public channel → run it with the retrieval passphrase → on rc 0 write done-flag + disable |
|
||
| `felhom-bootstrap.service` | retry-forever unit (`Type=oneshot`, `Restart=on-failure`, `RestartSec=30`, `StartLimitIntervalSec=0`, **`TimeoutStartSec=infinity`** — v1.21.0/R-33: the PAIRING wait loops inside the script, so the unit sits in `activating` instead of failing every 30s; without the lifted timeout systemd would kill it at 90s and restore the console spam) |
|
||
|
||
## Build
|
||
|
||
```bash
|
||
# 1. build the assistant image once (rebuild when the target PVE generation changes)
|
||
docker build -f scripts/iso/Dockerfile.assistant -t felhom-iso-assistant:trixie scripts/iso
|
||
|
||
# 2. build an ISO (on the build host; PVE ISO pre-downloaded + its official sha256)
|
||
scripts/iso/build-felhom-iso.sh \
|
||
--pve-iso /path/proxmox-ve_9.2-1.iso \
|
||
--iso-sha256 4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c \
|
||
--profile scripts/iso/profiles/nested-vm.profile \
|
||
--bootstrap-env /secure/bootstrap.env \
|
||
--out /mnt/5_hdd/felhom.eu/felhom-iso/out
|
||
```
|
||
> On the DooPlex build server (180) the felhom working dirs were moved off the SSD to
|
||
> `/mnt/5_hdd/felhom.eu/` (2026-07-18): the input PVE ISO lives at
|
||
> `/mnt/5_hdd/felhom.eu/drill/proxmox-ve_9.2-1.iso`. The script's `--out` **default is now
|
||
> `/mnt/5_hdd/felhom.eu/felhom-iso/out`** (override with `$FELHOM_ISO_OUT` on other hosts), so on 180 you
|
||
> can omit `--out` entirely and output still lands on the HDD, never the SSD.
|
||
|
||
Output: `felhom-pve-<pvever>-v<isover>-<profile>.iso` + `.sha256` + `.manifest.txt` +
|
||
**`.rootpw.txt` (0600, R-61 slice 1)** — the minted throwaway root password in plaintext (one line:
|
||
password + ISO name + build date), the operator's console credential for every box installed from
|
||
that build. It is the SINGLE record of truth: never logged, never in the manifest (which carries
|
||
only a pointer — manifests get pasted into committed REPORTs), **never committed or pasted
|
||
anywhere**. A fixed well-known password stays rejected (operator ruling 2026-07-21): a pre-pairing
|
||
box sits on a stranger's LAN.
|
||
|
||
The build **gates the answer on validate-answer's OUTPUT TEXT, never `$?`** — `validate-answer`
|
||
returns exit 0 even on a broken file (spike S1 trap; also `prepare-iso` was observed to exit 0 on a
|
||
bad answer). A broken answer aborts the build with the validator's message and produces **no ISO**.
|
||
Note `validate-answer` is *syntactic only*: disk existence is a runtime property (a non-existent disk
|
||
passes the build and fails-safe at install — spike S5c/S8b).
|
||
|
||
## Profile format
|
||
|
||
A sourceable shell fragment:
|
||
|
||
```sh
|
||
FELHOM_FQDN="felhom-host.local"
|
||
FELHOM_DISK_SETUP='[disk-setup]
|
||
filesystem = "ext4"
|
||
disk-list = ["sda"]'
|
||
# optional emergency/validation key baked into root's authorized_keys (blank -> not baked):
|
||
# FELHOM_ROOT_SSH_KEY="ssh-ed25519 AAAA... ops@felhom"
|
||
```
|
||
|
||
`FELHOM_DISK_SETUP` is any valid Proxmox `[disk-setup]` block — a `disk-list` OR a udev `filter.*`
|
||
glob. **A filter matching nothing, or a non-existent disk, fails-safe**: the installer aborts (exit
|
||
1, no disk touched) rather than installing on the wrong disk (spike S5c). Never commit a real key in
|
||
`FELHOM_ROOT_SSH_KEY` unless it is meant to ship in every ISO built from that profile.
|
||
|
||
A profile may also set `FELHOM_LOADER="shim"|"mkimage"` (see below); `--loader` on the command line
|
||
overrides it.
|
||
|
||
## Loader modes (`--loader shim|mkimage`, slice B)
|
||
|
||
The UEFI boot loader is selectable. Default **shim**; a profile can pin `FELHOM_LOADER`; `--loader`
|
||
wins over both.
|
||
|
||
- **`shim`** (default) — the stock output: the ISO's MS-signed shim → GRUB chain. **Secure Boot keeps
|
||
working** on compliant firmware (no MOK, no keypress — spike S2b). Use this everywhere it boots.
|
||
- **`mkimage`** — replaces the ISO's UEFI boot path with a **monolithic `grub-mkimage` loader** built
|
||
from the ISO's own GRUB modules. This is the fix for cheap AMI (`AN3PLUS 0.01`-class) mini-PC
|
||
firmware that **cannot UEFI-boot the ISO's stock GRUB from USB** (`relocation 0x0 is not implemented
|
||
yet`; SB-off and shim-bypass don't help) — the F1 finding of `tests/VALIDATION-n100-baremetal-2026-07-16.md`,
|
||
worked around live during that run and now first-class. The recipe (module set from the ISO's
|
||
`grub.cfg`, embedded `search --fs-uuid` → `configfile` the real menu) is reproduced from the run
|
||
evidence, not re-derived.
|
||
|
||
> **⚠ mkimage is UNSIGNED → the target board MUST have Secure Boot OFF.** With Secure Boot
|
||
> enforcing the firmware refuses the loader (`Access Denied`, no boot — proven, RUNBOOK-B leg 3).
|
||
> mkimage ISOs are named with a **`-mkimage`** suffix and the manifest's `loader:` line says so;
|
||
> never confuse them with a shim (SB-capable) build. shim mode's output is byte-for-byte the same as
|
||
> before this flag existed (the surgery is skipped entirely).
|
||
|
||
The surgery runs **after** `prepare-iso` and touches only the EFI boot path — the answer file and
|
||
the on-first-boot payload are provably byte-identical to the shim variant's (RUNBOOK-B leg 4).
|
||
|
||
## bootstrap-env (SECRET-BEARING)
|
||
|
||
The `--bootstrap-env` file becomes the in-ISO `/etc/felhom/bootstrap.env`:
|
||
|
||
```sh
|
||
FELHOM_CUSTOMER_ID=<hub customer id> # required
|
||
FELHOM_MODE=appliance # required (appliance|byo)
|
||
FELHOM_RETRIEVAL_PASSPHRASE=<customer passphrase> # required — SECRET
|
||
# optional:
|
||
# FELHOM_HUB_URL=https://hub.felhom.eu
|
||
# FELHOM_INSTALL_URL=https://felhom.eu/scripts/felhom-host-install.sh
|
||
# FELHOM_EXTRA_ARGS="--cores 4 --memory 8192" # profile-only flags, never secrets
|
||
```
|
||
|
||
**Why the ISO is secret-bearing (§4.4 finding):** the hub install-command's *distribution channel* is
|
||
the public `https://felhom.eu/scripts/felhom-host-install.sh` — no token, no secret in the command.
|
||
But `felhom-host-install.sh` requires the customer **retrieval passphrase** (normally a no-echo
|
||
prompt) to fetch the config and enroll the host. An *unattended* install must supply it via
|
||
`--passphrase-file`, so the ISO embeds it. **Rules for a secret-bearing ISO:**
|
||
|
||
- supervised / single-use only; **never distributed**; **delete after the run**.
|
||
- issue the passphrase with the shortest usable lifetime; rotate/retire after the install.
|
||
- the bootstrap `shred`s `bootstrap.env` on the box once host-install succeeds (reduces secret-at-rest).
|
||
- **slice C** removes this: the claim/pairing flow delivers the passphrase (or config) to the box
|
||
*after* the customer claims it, so the *distributed* ISO carries no secret.
|
||
|
||
## The first-boot chain
|
||
|
||
`stub-first-boot.sh` (exactly-once, `[first-boot]` from-iso/fully-up) is **dumb**: it lays down
|
||
`/usr/local/sbin/felhom-bootstrap.sh`, the systemd unit, and `/etc/felhom/bootstrap.env` (0600), then
|
||
enables + starts the unit. All fallible/network work lives in `felhom-bootstrap.service`, which
|
||
retries forever until `felhom-host-install.sh` exits 0, then writes `/etc/felhom/.bootstrap-done`,
|
||
disables itself, and scrubs the env. Retry-vs-resume: the first attempt is plain; any later attempt
|
||
that finds `/var/lib/felhom-install/state.json` adds `--resume` (safe per host-install v1.11.3 — its
|
||
producer steps re-run each pass).
|
||
|
||
## HP mini profile (second pilot board, 2026-07-21)
|
||
|
||
`profiles/hp.profile` — same shape as the N100 template with ONE deliberate difference: **the loader
|
||
default stays at the stock signed `shim`**, so Secure Boot keeps working. `mkimage` exists only to
|
||
work around the N100's AMI firmware GRUB defect and is unsigned.
|
||
|
||
**PROVEN on this board (2026-07-21):** the safety boot loaded shim, fetched and parsed the answer file,
|
||
and the match-nothing filter refused with **zero disk writes**; the armed ISO then installed end to end.
|
||
**Secure Boot stayed ENABLED throughout** (`mokutil --sb-state` on the installed box → `SecureBoot
|
||
enabled`) — so SB-off is an N100-firmware workaround, not a Felhom requirement.
|
||
so the loader, Secure Boot setting, and network/answer path are all confirmed before anything
|
||
destructive existed on a stick. `enp1s0f0` auto-detect verified on-board.
|
||
|
||
**Arming it:** never edit the committed profile. Copy it to an UNCOMMITTED working profile (the
|
||
`n100-demo.profile` precedent — a real serial never goes in git), replace the placeholder serial, and
|
||
rebuild. Verify the armed filter by extracting `answer.toml` **from inside the built ISO**, and again
|
||
from the written stick — the profile is what you intended, the ISO is what you shipped:
|
||
|
||
```
|
||
osirrox -indev <iso> -extract /answer.toml /tmp/a.toml && sed -n '/^\[disk-setup\]/,$p' /tmp/a.toml
|
||
```
|
||
|
||
**t740 NIC TRAP — the 4-port card gets no DHCP; use the onboard port.** This board presents FIVE
|
||
wired NICs and the install picks wrong:
|
||
|
||
| interface | driver | what it is | at this site |
|
||
|---|---|---|---|
|
||
| `enp1s0f0`–`f3` | `igb` | the 4-port expansion card | **no DHCP lease** (no link) |
|
||
| `enp2s0f0` | `r8169` | **the onboard port** | leases fine, 1000 Mb |
|
||
| `wlo1` | `iwlwifi` | wifi | unused |
|
||
|
||
Plug the cable into the **onboard** port. If the install already happened on the wrong port, the
|
||
symptom used to be nasty: the installer does not abort, it bakes its **192.168.100.2 fallback as a
|
||
STATIC `vmbr0` config** (address AND a dead `gateway 192.168.100.1`) and completes, so the box
|
||
looks installed and can never reach the hub (R-59). **Since v1.24.0 the trap has a gate and a
|
||
self-heal (R-59+R-60):** on first boot, hub-unreachable → the bootstrap sweeps every physical NIC
|
||
(re-point `bridge-ports` → bounded DHCP → hub probe) and keeps the first that reaches the hub
|
||
(persisted as `iface vmbr0 inet dhcp`, original in `interfaces.felhom-bak`); if nothing works it
|
||
paints a legible Hungarian console screen (NIC table: név/MAC/kábel/sebesség + remedy) and retries
|
||
every minute — so *moving the cable to any port is enough, no console repair needed*. The sweep
|
||
runs only before the install attempt begins (`state.json` gate); a box already past that point
|
||
gets the screen but its interfaces are never touched. Manual repair (pre-v1.24.0 boxes): point
|
||
`bridge-ports` at `enp2s0f0` in `/etc/network/interfaces`, set the correct address (or `dhcp`),
|
||
`ifreload -a` — and mind the baked stale `gateway` line. Proven end-to-end on the nested drill
|
||
(`documentation/audits/SPIKE-firstboot-nic-sweep-2026-07-22.md`); metal proof rides the next real
|
||
install.
|
||
|
||
**Confirm the serial is the system disk and not a data drive.** On this board the SanDisk X600 128GB
|
||
(`sda`) is the system disk; the 1TB NVMe is the future data drive and must stay OUTSIDE the filter —
|
||
it joins later through the normal Tárhely flow, not the installer.
|
||
|
||
> **The one known failure mode on a second-hand board — read step 3 below.** The auto-installer does
|
||
> NOT force-clear a pre-existing LVM PV signature. A disk carrying a previous Linux/Proxmox install
|
||
> aborts at *"cannot init physical volume"* (spike S2b). A previous Windows/NTFS install is fine. If
|
||
> the install aborts that way, wipe the target first (`blkdiscard -f /dev/<disk>`, or `wipefs -a` plus
|
||
> zeroing the PV region) from any live environment, then re-boot the same stick — the ISO does not
|
||
> need rebuilding.
|
||
|
||
## N100 profile (for RUNBOOK B)
|
||
|
||
Use the committed `profiles/n100.profile` template (cheap-board, `FELHOM_LOADER="mkimage"`). **A real
|
||
single-disk mini-PC uses a unique-serial udev filter, not `disk-list`:**
|
||
|
||
1. Boot once (or use a live env), harvest the target disk's udev id:
|
||
`udevadm info --query=property --name=/dev/sda | grep -E 'ID_SERIAL_SHORT|ID_MODEL|ID_WWN'`.
|
||
2. In the profile, set `filter.ID_SERIAL_SHORT = "<harvested>"` + `filter-match = "all"`. A
|
||
match-nothing filter fails-safe (won't wipe the wrong disk).
|
||
3. **Prior-LVM prerequisite:** the auto-installer does **not** force-clear a pre-existing LVM PV
|
||
signature — a disk with a previous Linux/Proxmox install aborts at "cannot init physical volume"
|
||
(spike S2b). Wipe first: `blkdiscard -f /dev/<disk>` (or `wipefs -a` + zero the PV region).
|
||
4. **Boot loader + Secure Boot (F1):** cheap AMI (`AN3PLUS`-class) firmware can't boot the stock GRUB
|
||
from USB, so `profiles/n100.profile` uses `--loader mkimage`. **That loader is unsigned → set
|
||
Secure Boot to OFF/Disabled in the BIOS** before booting the stick (with SB enforcing the board
|
||
refuses it: `Access Denied`, no boot). Firmware that boots the stock ISO fine needs no mkimage and
|
||
keeps Secure Boot working (default `shim`, spike S2b). *Optional pre-flight (operator, no risk):* a
|
||
`n100-safety` ISO — `--loader mkimage` + a **match-nothing** filter — boots the board, loads GRUB,
|
||
and the installer aborts touching nothing (S5c). It confirms the loader + SB-off setting before the
|
||
real install; its sha is recorded at build. Never boot the real install-armed ISO except in the
|
||
supervised rehearsal.
|
||
5. **BIOS "State After G3" → "Power On" (or "Last State")** before final sign-off — so the mini-PC
|
||
auto-powers-on after a mains loss (a managed appliance must come back without a physical press).
|
||
Set it in the BIOS power/ACPI menu; it is board-specific and cannot be scripted (operator step).
|
||
|
||
## Validation status (nested VM 310, 2026-07-16)
|
||
|
||
PASS: build gate + red-proof (C); disk-filter fail-safe through the pipeline (D); stub→unit→**real
|
||
public-channel fetch**→correct host-install invocation→retry; 2.3 resume decision; exactly-once stub;
|
||
no-network retry + auto-recovery (B). **Operator-gated (not live-proven here):** host-install rc-0
|
||
terminal success → hub-enrolled + guest-provisioned + done-flag + unit-disable — needs a properly
|
||
created drill customer (the hub create-UI is password-gated and a live-DB insert is unsafe). Also
|
||
out of scope: the physical N100 run (RUNBOOK B).
|