admin
bcdb04222a
Waiting to be bound is the NORMAL state of a freshly installed box, and it must
not be reported as failure. The PAIRING poll loop used to BE systemd's
Restart=on-failure/RestartSec=30 — one poll per invocation, exiting non-zero
until the bind landed — so every 30s systemd printed "Failed to start Felhom
host bootstrap" on the physical console the CUSTOMER is watching. The
2026-07-18 N100 rehearsal measured 52 FAILED lines in ~11 minutes while nothing
was wrong (VALIDATION-n100-rehearsal-2026-07-18.md F6).
felhom-bootstrap.sh: run_pairing() is now a while-loop that sleeps
POLL_INTERVAL (30s — the hub-side rate is unchanged) between polls, so the unit
sits in `activating`. Registration split into register_appliance(), which
returns non-zero for a transient problem (no network yet, no identity, no
token) and is retried by the loop instead of taking the unit down. Cadence
constants: POLL_INTERVAL=30, BANNER_EVERY=10 (5 min), HEARTBEAT_EVERY=20
(10 min).
Quiet without going dark: a 204 is logged once on entry (worded so nobody reads
it as an error) and then only on the 10-minute heartbeat with elapsed minutes;
404 and unexpected codes degrade the same way. 410 STILL exits non-zero on
purpose — delivery consumed but no local env is a real crash window, and a
clean systemd restart is the right response.
Console banner: every 5 min instead of every cycle, single accented spelling
instead of the parositasra/párosításra double, and the reassurance the
rehearsal showed was missing ("Ez a képernyő magától frissül — nincs teendő a
doboznál").
felhom-bootstrap.service: TimeoutStartSec=infinity. This is load-bearing, not
cosmetic — a Type=oneshot ExecStart is killed at DefaultTimeoutStartSec (90s),
so without it systemd would kill the new in-script wait after 90 seconds and
Restart=on-failure would silently reinstate the exact spam this removes, after
appearing to work for the first three polls. Restart=/RestartSec= are kept
deliberately: they still cover the DIRECT path, a failed host-install, and 410.
Verified behaviourally, not assumed: driven in a throwaway Debian container
against a stub hub answering 204 five times then delivering — logged the wait
once plus one heartbeat, never exited between polls, then consumed the
delivery, wrote the 0600 env, fell through to the direct install in the same
invocation and exited 0. The old design produced five unit invocations and five
"Failed to start" console lines for that same sequence.
Hub endpoints, payloads, polling rate and one-shot delivery semantics are all
unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
171 lines
11 KiB
Markdown
171 lines
11 KiB
Markdown
# Felhom bare-metal ISO pipeline (R-21 slice A + B)
|
||
|
||
Turns the official Proxmox VE ISO into a **Felhom auto-install ISO** whose first-boot chain,
|
||
unattended, fetches and runs `felhom-host-install.sh` until the host is enrolled and a guest is
|
||
provisioned. Boot the ISO on blank hardware → walk away → managed Felhom host.
|
||
|
||
Grounding: every install/first-boot/webhook mechanism used here is proven in
|
||
`documentation/audits/SPIKE-baremetal-iso-2026-07-16.md` (S1–S8). This pipeline **composes** them;
|
||
it does **not** modify `felhom-host-install.sh` (which it only invokes over the public distribution
|
||
channel).
|
||
|
||
> **Scope:** slice A = the pipeline + the in-ISO bootstrap, validated on nested VM 310. The physical
|
||
> N100 run is **RUNBOOK B** (separate, supervised). The customer-facing claim/pairing that removes the
|
||
> secret from the ISO is **slice C** (not built here).
|
||
|
||
## Files
|
||
|
||
| File | Role |
|
||
|---|---|
|
||
| `Dockerfile.assistant` | build host: Debian trixie + `proxmox-auto-install-assistant` (paired to the ISO's PVE generation by Debian codename) + `xorriso` + `shellcheck` |
|
||
| `build-felhom-iso.sh` | the pipeline: render answer, mint throwaway root hash, **validate-answer output-parse gate**, render stub, `prepare-iso`, **optional `--loader mkimage` surgery**, emit ISO + sha256 + manifest |
|
||
| `mkimage-surgery.sh` | slice-B loader surgery (runs in the container, post-`prepare-iso`): builds a monolithic `grub-mkimage` UEFI loader and swaps it into the ISO's EFI boot path (F1 firmware fix) |
|
||
| `answer.toml.tmpl` | Proxmox answer template (`__FQDN__` / `__ROOT_HASH__` / `__ROOT_SSH_KEYS__` / `__DISK_SETUP__` placeholders; `[first-boot]` from-iso/fully-up) |
|
||
| `profiles/*.profile` | disk-selection + fqdn fragments (sourceable, no secret) |
|
||
| `stub-first-boot.sh` | the ONE first-boot executable (skeleton; build injects the bootstrap script/unit/env as base64) |
|
||
| `felhom-bootstrap.sh` | per-attempt: fetch host-install from the public channel → run it with the retrieval passphrase → on rc 0 write done-flag + disable |
|
||
| `felhom-bootstrap.service` | retry-forever unit (`Type=oneshot`, `Restart=on-failure`, `RestartSec=30`, `StartLimitIntervalSec=0`, **`TimeoutStartSec=infinity`** — v1.21.0/R-33: the PAIRING wait loops inside the script, so the unit sits in `activating` instead of failing every 30s; without the lifted timeout systemd would kill it at 90s and restore the console spam) |
|
||
|
||
## Build
|
||
|
||
```bash
|
||
# 1. build the assistant image once (rebuild when the target PVE generation changes)
|
||
docker build -f scripts/iso/Dockerfile.assistant -t felhom-iso-assistant:trixie scripts/iso
|
||
|
||
# 2. build an ISO (on the build host; PVE ISO pre-downloaded + its official sha256)
|
||
scripts/iso/build-felhom-iso.sh \
|
||
--pve-iso /path/proxmox-ve_9.2-1.iso \
|
||
--iso-sha256 4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c \
|
||
--profile scripts/iso/profiles/nested-vm.profile \
|
||
--bootstrap-env /secure/bootstrap.env \
|
||
--out /mnt/5_hdd/felhom.eu/felhom-iso/out
|
||
```
|
||
> On the DooPlex build server (180) the felhom working dirs were moved off the SSD to
|
||
> `/mnt/5_hdd/felhom.eu/` (2026-07-18): the input PVE ISO lives at
|
||
> `/mnt/5_hdd/felhom.eu/drill/proxmox-ve_9.2-1.iso`. The script's `--out` **default is now
|
||
> `/mnt/5_hdd/felhom.eu/felhom-iso/out`** (override with `$FELHOM_ISO_OUT` on other hosts), so on 180 you
|
||
> can omit `--out` entirely and output still lands on the HDD, never the SSD.
|
||
|
||
Output: `felhom-pve-<pvever>-v<isover>-<profile>.iso` + `.sha256` + `.manifest.txt`.
|
||
|
||
The build **gates the answer on validate-answer's OUTPUT TEXT, never `$?`** — `validate-answer`
|
||
returns exit 0 even on a broken file (spike S1 trap; also `prepare-iso` was observed to exit 0 on a
|
||
bad answer). A broken answer aborts the build with the validator's message and produces **no ISO**.
|
||
Note `validate-answer` is *syntactic only*: disk existence is a runtime property (a non-existent disk
|
||
passes the build and fails-safe at install — spike S5c/S8b).
|
||
|
||
## Profile format
|
||
|
||
A sourceable shell fragment:
|
||
|
||
```sh
|
||
FELHOM_FQDN="felhom-host.local"
|
||
FELHOM_DISK_SETUP='[disk-setup]
|
||
filesystem = "ext4"
|
||
disk-list = ["sda"]'
|
||
# optional emergency/validation key baked into root's authorized_keys (blank -> not baked):
|
||
# FELHOM_ROOT_SSH_KEY="ssh-ed25519 AAAA... ops@felhom"
|
||
```
|
||
|
||
`FELHOM_DISK_SETUP` is any valid Proxmox `[disk-setup]` block — a `disk-list` OR a udev `filter.*`
|
||
glob. **A filter matching nothing, or a non-existent disk, fails-safe**: the installer aborts (exit
|
||
1, no disk touched) rather than installing on the wrong disk (spike S5c). Never commit a real key in
|
||
`FELHOM_ROOT_SSH_KEY` unless it is meant to ship in every ISO built from that profile.
|
||
|
||
A profile may also set `FELHOM_LOADER="shim"|"mkimage"` (see below); `--loader` on the command line
|
||
overrides it.
|
||
|
||
## Loader modes (`--loader shim|mkimage`, slice B)
|
||
|
||
The UEFI boot loader is selectable. Default **shim**; a profile can pin `FELHOM_LOADER`; `--loader`
|
||
wins over both.
|
||
|
||
- **`shim`** (default) — the stock output: the ISO's MS-signed shim → GRUB chain. **Secure Boot keeps
|
||
working** on compliant firmware (no MOK, no keypress — spike S2b). Use this everywhere it boots.
|
||
- **`mkimage`** — replaces the ISO's UEFI boot path with a **monolithic `grub-mkimage` loader** built
|
||
from the ISO's own GRUB modules. This is the fix for cheap AMI (`AN3PLUS 0.01`-class) mini-PC
|
||
firmware that **cannot UEFI-boot the ISO's stock GRUB from USB** (`relocation 0x0 is not implemented
|
||
yet`; SB-off and shim-bypass don't help) — the F1 finding of `tests/VALIDATION-n100-baremetal-2026-07-16.md`,
|
||
worked around live during that run and now first-class. The recipe (module set from the ISO's
|
||
`grub.cfg`, embedded `search --fs-uuid` → `configfile` the real menu) is reproduced from the run
|
||
evidence, not re-derived.
|
||
|
||
> **⚠ mkimage is UNSIGNED → the target board MUST have Secure Boot OFF.** With Secure Boot
|
||
> enforcing the firmware refuses the loader (`Access Denied`, no boot — proven, RUNBOOK-B leg 3).
|
||
> mkimage ISOs are named with a **`-mkimage`** suffix and the manifest's `loader:` line says so;
|
||
> never confuse them with a shim (SB-capable) build. shim mode's output is byte-for-byte the same as
|
||
> before this flag existed (the surgery is skipped entirely).
|
||
|
||
The surgery runs **after** `prepare-iso` and touches only the EFI boot path — the answer file and
|
||
the on-first-boot payload are provably byte-identical to the shim variant's (RUNBOOK-B leg 4).
|
||
|
||
## bootstrap-env (SECRET-BEARING)
|
||
|
||
The `--bootstrap-env` file becomes the in-ISO `/etc/felhom/bootstrap.env`:
|
||
|
||
```sh
|
||
FELHOM_CUSTOMER_ID=<hub customer id> # required
|
||
FELHOM_MODE=appliance # required (appliance|byo)
|
||
FELHOM_RETRIEVAL_PASSPHRASE=<customer passphrase> # required — SECRET
|
||
# optional:
|
||
# FELHOM_HUB_URL=https://hub.felhom.eu
|
||
# FELHOM_INSTALL_URL=https://felhom.eu/scripts/felhom-host-install.sh
|
||
# FELHOM_EXTRA_ARGS="--cores 4 --memory 8192" # profile-only flags, never secrets
|
||
```
|
||
|
||
**Why the ISO is secret-bearing (§4.4 finding):** the hub install-command's *distribution channel* is
|
||
the public `https://felhom.eu/scripts/felhom-host-install.sh` — no token, no secret in the command.
|
||
But `felhom-host-install.sh` requires the customer **retrieval passphrase** (normally a no-echo
|
||
prompt) to fetch the config and enroll the host. An *unattended* install must supply it via
|
||
`--passphrase-file`, so the ISO embeds it. **Rules for a secret-bearing ISO:**
|
||
|
||
- supervised / single-use only; **never distributed**; **delete after the run**.
|
||
- issue the passphrase with the shortest usable lifetime; rotate/retire after the install.
|
||
- the bootstrap `shred`s `bootstrap.env` on the box once host-install succeeds (reduces secret-at-rest).
|
||
- **slice C** removes this: the claim/pairing flow delivers the passphrase (or config) to the box
|
||
*after* the customer claims it, so the *distributed* ISO carries no secret.
|
||
|
||
## The first-boot chain
|
||
|
||
`stub-first-boot.sh` (exactly-once, `[first-boot]` from-iso/fully-up) is **dumb**: it lays down
|
||
`/usr/local/sbin/felhom-bootstrap.sh`, the systemd unit, and `/etc/felhom/bootstrap.env` (0600), then
|
||
enables + starts the unit. All fallible/network work lives in `felhom-bootstrap.service`, which
|
||
retries forever until `felhom-host-install.sh` exits 0, then writes `/etc/felhom/.bootstrap-done`,
|
||
disables itself, and scrubs the env. Retry-vs-resume: the first attempt is plain; any later attempt
|
||
that finds `/var/lib/felhom-install/state.json` adds `--resume` (safe per host-install v1.11.3 — its
|
||
producer steps re-run each pass).
|
||
|
||
## N100 profile (for RUNBOOK B)
|
||
|
||
Use the committed `profiles/n100.profile` template (cheap-board, `FELHOM_LOADER="mkimage"`). **A real
|
||
single-disk mini-PC uses a unique-serial udev filter, not `disk-list`:**
|
||
|
||
1. Boot once (or use a live env), harvest the target disk's udev id:
|
||
`udevadm info --query=property --name=/dev/sda | grep -E 'ID_SERIAL_SHORT|ID_MODEL|ID_WWN'`.
|
||
2. In the profile, set `filter.ID_SERIAL_SHORT = "<harvested>"` + `filter-match = "all"`. A
|
||
match-nothing filter fails-safe (won't wipe the wrong disk).
|
||
3. **Prior-LVM prerequisite:** the auto-installer does **not** force-clear a pre-existing LVM PV
|
||
signature — a disk with a previous Linux/Proxmox install aborts at "cannot init physical volume"
|
||
(spike S2b). Wipe first: `blkdiscard -f /dev/<disk>` (or `wipefs -a` + zero the PV region).
|
||
4. **Boot loader + Secure Boot (F1):** cheap AMI (`AN3PLUS`-class) firmware can't boot the stock GRUB
|
||
from USB, so `profiles/n100.profile` uses `--loader mkimage`. **That loader is unsigned → set
|
||
Secure Boot to OFF/Disabled in the BIOS** before booting the stick (with SB enforcing the board
|
||
refuses it: `Access Denied`, no boot). Firmware that boots the stock ISO fine needs no mkimage and
|
||
keeps Secure Boot working (default `shim`, spike S2b). *Optional pre-flight (operator, no risk):* a
|
||
`n100-safety` ISO — `--loader mkimage` + a **match-nothing** filter — boots the board, loads GRUB,
|
||
and the installer aborts touching nothing (S5c). It confirms the loader + SB-off setting before the
|
||
real install; its sha is recorded at build. Never boot the real install-armed ISO except in the
|
||
supervised rehearsal.
|
||
5. **BIOS "State After G3" → "Power On" (or "Last State")** before final sign-off — so the mini-PC
|
||
auto-powers-on after a mains loss (a managed appliance must come back without a physical press).
|
||
Set it in the BIOS power/ACPI menu; it is board-specific and cannot be scripted (operator step).
|
||
|
||
## Validation status (nested VM 310, 2026-07-16)
|
||
|
||
PASS: build gate + red-proof (C); disk-filter fail-safe through the pipeline (D); stub→unit→**real
|
||
public-channel fetch**→correct host-install invocation→retry; 2.3 resume decision; exactly-once stub;
|
||
no-network retry + auto-recovery (B). **Operator-gated (not live-proven here):** host-install rc-0
|
||
terminal success → hub-enrolled + guest-provisioned + done-flag + unit-disable — needs a properly
|
||
created drill customer (the hub create-UI is password-gated and a live-DB insert is unsafe). Also
|
||
out of scope: the physical N100 run (RUNBOOK B).
|