scripts v1.18.0 — R-21 slice B: firmware loader option --loader shim|mkimage (F1)

Closes N100 F1 (HIGH): cheap AMI (AN3PLUS 0.01-class) UEFI firmware can't
relocate the ISO's stock signed GRUB from USB (relocation 0x0). The run's live
grub-mkimage workaround is now a first-class pipeline mode.

- build-felhom-iso.sh: --loader shim|mkimage (default shim, byte-for-byte
  unchanged; profile-settable FELHOM_LOADER; --loader wins). Loud banner +
  manifest loader:/grub-mkimage: fields + -mkimage filename suffix.
- mkimage-surgery.sh (new): post-prepare-iso, in the assistant container. Builds
  a monolithic grub-mkimage loader from the ISO's own GRUB (module set from its
  grub.cfg; embedded search --fs-uuid -> configfile the real menu). Swaps it into
  the ISO9660 tree (real lowercase path) + the efi.img ESP; xorriso re-master
  preserves BIOS-hybrid + UEFI + GPT-ESP, drops only Apple HFS+/APM. Recipe from
  the N100 run evidence, not re-derived.
- Dockerfile.assistant: grub-common + grub-efi-amd64-bin + mtools + dosfstools.
  profiles/n100.profile (new, mkimage + SB-off note).
- Validated on nested VM 311 (RUNBOOK-B legs): leg1 shim boots+installs under
  OVMF SB-enforcing + SeaBIOS; leg2 mkimage boots+installs under SB-off; leg3
  (red-proof) mkimage under SB-enforcing FAILS Access Denied (unsigned -> SB must
  be OFF); leg4 surgery byte-identical payload. bash -n + shellcheck clean.
  Physical N100 closure folds into the rehearsal (n100-safety match-nothing ISO
  built + sha-recorded, unbooted). PXE stays a deferred R-21 note.
This commit is contained in:
2026-07-17 14:28:10 +02:00
parent d86d2699c3
commit 3172df1927
11 changed files with 439 additions and 58 deletions
+14
View File
@@ -3,6 +3,20 @@
> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
- **2026-07-17 — R-21 SLICE B SHIPPED: firmware loader option (scripts v1.18.0).** Closes N100 **F1
(HIGH)** — cheap AMI (`AN3PLUS 0.01`-class) firmware can't USB-boot the ISO's stock GRUB
(`relocation 0x0`). `build-felhom-iso.sh --loader shim|mkimage` (default **shim**, unchanged;
profile-settable via `FELHOM_LOADER`). **mkimage** = a monolithic `grub-mkimage` loader built from
the ISO's own GRUB (module set from its `grub.cfg`, embedded `search --fs-uuid``configfile`), swapped
into the ISO9660 tree + `efi.img` ESP by `mkimage-surgery.sh` **after** `prepare-iso` (payload
provably untouched), re-mastered with xorriso preserving BIOS-hybrid + UEFI + GPT-ESP (drops only the
irrelevant Apple HFS+/APM). **Unsigned → target Secure Boot MUST be OFF.** Loud: banner + manifest
`loader:` + `-mkimage` filename. New `profiles/n100.profile`; Dockerfile gains grub/mtools.
**RUNBOOK-B legs (nested VM 311, OVMF):** shim boots+installs under SB-enforcing + SeaBIOS; mkimage
boots+installs under SB-off; **mkimage under SB-enforcing FAILS `Access Denied`** (red-proof); surgery
byte-identical payload. Physical N100 boot still pending → supervised rehearsal (an `n100-safety`
match-nothing ISO built + sha-recorded, unbooted). PXE stays a deferred R-21 note. See REPORT.md.
- **2026-07-17 — CUSTOMER RESET SHIPPED: hub v0.61.0 + felhom-tenantsync v1.1.0 (LIVE).** The middle
lifecycle tier (host delete < RESET < customer Delete). One operator action → pre-first-install: all
OPERATIONAL state dies (offsite repo, PBS namespace+backups, DR recipe, one-time secret, claim state,
+80 -42
View File
@@ -2,58 +2,96 @@
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
## Hub v0.61.0 + felhom-tenantsync v1.1.0 — Customer RESET (middle lifecycle tier) — 2026-07-17
## scripts v1.18.0 — R-21 slice B: firmware loader option `--loader shim|mkimage` (F1) — 2026-07-17
**Commits:** `4009401` (code), `e144c5e` (manifest). **Live:** hub **0.61.0** on k3s (Synced/Healthy);
felhom-tenantsync **v1.1.0** on ep0 (`felhom-hetzner`).
**Baseline:** scripts v1.17.0 (host-install) / build-felhom-iso.sh was ISO_VERSION 1.16.0. Target
**v1.18.0**. Closes N100 **F1 (HIGH):** cheap AMI (`AN3PLUS 0.01`-class) UEFI firmware can't relocate
the ISO's stock signed GRUB from USB (`relocation 0x0`); the run's live `grub-mkimage` workaround is
now a first-class pipeline mode. Touched **only** `scripts/iso/`.
### What shipped
The **middle lifecycle tier** between *host delete* and *customer Delete*. One operator action returns a
customer to **pre-first-install**: every OPERATIONAL trace dies — offsite repo (Hetzner sub-account/box),
PBS namespace + all backup groups + token, DR recipe, one-time secret, claim state, retained escrow
custody — while **identity + basic config survive** (`customer_configs` incl. the offsite tier CHOICE, all
provenance rows, and the audit-event stream).
`build-felhom-iso.sh` gains `--loader shim|mkimage` (default **shim**, profile-settable via
`FELHOM_LOADER`, `--loader` wins). **shim = the stock output, byte-for-byte unchanged** (surgery
skipped). **mkimage** replaces the ISO's UEFI boot path with a monolithic `grub-mkimage` loader.
New: `mkimage-surgery.sh`, `profiles/n100.profile`, Dockerfile grub/mtools tooling; loud banner +
manifest `loader:`/`grub-mkimage:` fields + `-mkimage` filename suffix.
Viktor's rulings, all implemented: (1) destroying retained escrow custody gets its **own separate
acknowledgment**; (2) RESET clears claim state → fresh code next onboarding; (3) RESET **refuses while any
host row exists** (delete hosts first — reset never deletes hosts); (4) the confirm surface shows a
**live-counted** inventory.
### Surgery approach actually used (Phase 2)
### Discipline
Recipe reproduced from the run evidence (the literal command was run on the now-wiped N100 old system
and saved nowhere — only the *procedure* is recorded; see Observations). In the assistant container,
**after** `prepare-iso`:
1. `xorriso -osirrox` extract the prepared tree (chmod u+w — osirrox files are read-only).
2. `grub-mkimage -O x86_64-efi` from the **container's** grub 2.12 build (`/usr/lib/grub/x86_64-efi`,
which has `kernel.img` — the ISO ships modules but not kernel.img; the box likewise used its
*installed* GRUB). Module set = the base search/configfile/USB chain **plus every `insmod` in the
ISO's own `grub.cfg`** (57 modules). Embedded config: `search --fs-uuid <ISO uuid>``configfile`
the real menu. The fs-uuid is derived from the ISO's PVD modification-date (GRUB's iso9660 uuid
source) and pinned on re-master so it stays valid.
3. Swap the loader into the ISO9660 tree (**real lowercase Rock-Ridge path** `/efi/boot/bootx64.efi`
an early bug created a spurious *uppercase* path instead; fixed) **and** inside the `efi.img` ESP
(FAT, case-insensitive — the loader UEFI firmware actually runs from USB).
4. Re-master with `xorriso -as mkisofs` from the ISO's own `as_mkisofs` report, dropping **only** the
Apple HFS+/APM map (`-hfsplus`/`-apm-block-size`) and the isohybrid GPT-basdat marking
(`-part_like_isohybrid`/`-isohybrid-gpt-basdat`) — re-emitting those trips xorriso 1.5.6's
"Overlapping MBR partition entries" on this layout, and Mac boot is irrelevant. **Kept:** protective
MBR + grub2-mbr (BIOS hybrid) + El Torito (BIOS + UEFI) + the GPT EFI System Partition (USB UEFI).
External teardown FIRST, DB purge LAST (publish-last). Every leg idempotent → a partial run is re-run from
the top; a failed external leg is a clean journal entry and the DB purge (which erases the descriptors that
say what still needs tearing down) is **withheld** until every external leg is `ok`. Provenance + events
are never wiped.
**Stall-rule outcome:** the surgery resisted for ~40 min of iteration (two overlap failures + the
kernel.img and path-case bugs) — **well within the 2 h budget**; no spike-note fallback needed.
### Components
### Per-leg results (RUNBOOK-B, nested VM 311 on felhom-pve, OVMF)
- **store/customer_reset.go**: `customer_resets` journal (per-attempt, per-leg, resumable);
`CustomerResetInventory` (live counts incl. retained blobs via the F-14 `host_deletions` UNION);
`Start/UpdateResetLeg/Finish/LatestCustomerReset`; ack-gated `PurgeCustomerResetDBState`; `DeleteClaim`.
- **claim.ResetToUnclaimed**: deletes the claim row → `EnsureIssued` mints a fresh first code next onboarding.
- **offsite**: `Deprovision` (idempotent label-lookup delete), `OffsiteIdentifier`, `ClearProvisionedDescriptor`.
- **tenantsync + scripts/felhom-tenantsync.sh v1.1.0**: `deprovision` op destroys the namespace + backup
groups + token; shared `felhom@pbs` user untouched; idempotent.
- **web/customer_reset.go**: `GET /configs/{id}/reset` → inventory JSON; `POST` → orchestration (all gates
before any write/external call). A distinct **amber** RESET card + `reset_done` flash.
- **Leg 1 — shim regression:** the default ISO **boots + zero-touch auto-installs under OVMF Secure
Boot ENFORCING** (`pre-enrolled-keys=1`; "EFI boot mode detected", DHCP, "Starting Proxmox
installation", package extraction unattended). **Plus one SeaBIOS boot** — the shim ISO also boots
under BIOS (hybrid grub2-mbr/eltorito path survived the v1.18.0 build). ✓
- **Leg 2 — mkimage happy path:** `--loader mkimage` ISO **boots + auto-installs under OVMF Secure
Boot OFF** (`pre-enrolled-keys=0`). The monolith's embedded `search --fs-uuid` found the ISO and
chained its real menu → the installer ran. ✓
- **Leg 3 — trade-off red-proof:** the **same** mkimage ISO under OVMF Secure Boot ENFORCING
**FAILS** — captured firmware behavior: `BdsDxe: failed to load Boot0002 "UEFI QEMU DVD-ROM …" :
Access Denied` → `No bootable option or device was found`. The unsigned loader is refused. This
makes "Secure Boot must be OFF for mkimage" a **fact**, not a guess. ✓
- **Leg 4 — structural / byte-identity:** diffing the shim(out) vs mkimage(final) trees, **only**
`efi/boot/bootx64.efi`, `efi/boot/grubx64.efi`, `efi.img`, and the regenerated El Torito artifacts
(`boot.cat`, `i386-pc/eltorito.img`) differ — **the entire install payload is byte-identical**. The
tree loader (1253376 B, vs shim's 1032032 B) carries the embedded `2026-05-21-…` fs-uuid and is
identical to the loader inside `efi.img`; **0 spurious uppercase paths**; El Torito keeps both boot
images. The `validate-answer` gate still runs pre-surgery. ✓
### Verification
### Artifacts (shas + manifests)
- **Green gate:** `go build ./... && go vet ./... && go test ./...` all pass. Hub confirm gate OK.
- **Red-proofs (run-fail-revert, both proven red then restored):** (a) escrow-ack gate — defeat it →
reset proceeds & destroys blobs → FAIL; (b) partial-failure resumability — don't withhold the purge →
DB purged despite an external failure → FAIL. Plus store ack-gating + journal round-trip; offsite
Deprovision idempotency + descriptor clear; RESET-card render.
- **Live drill on ep0 (throwaway `drill-reset-01`, real PBS data):** provision → **real backup written into
the namespace** → deprovision (`deleted:true`; namespace + backup group + token destroyed) → token/ns
confirmed gone → idempotent re-run (`deleted:false`) → **all 3 real tenants + the shared user survived**.
- shim nested-test ISO `…-nested-vm-val.iso``28d862e2f7b014588717c15ef8c9bc46d11a9a5689a022f5d4de9e8f939d1377`
- mkimage nested-test ISO `…-nested-vm-val-mkimage.iso``71fce3333b3148e48d10711311f275da4b849bf383f711de0f2e4d5802618869`
- **safety ISO** `…-n100-safety-mkimage.iso``f9aa0428e5efdcb5e7c0ade05264ff681ac4f9aa47b7f3721c9eb712f025c1a8`
(mkimage + match-nothing filter; **built + sha-recorded, NOT booted**; at `180:~/n100-baremetal/safety/`).
Manifests record `loader: mkimage … Secure Boot OFF`, `grub-mkimage: 2.12-9+deb13u2`, secret-bearing.
Nested-test ISOs + VM 311 destroyed after the legs.
### Not live-drilled (covered otherwise)
### Physical proof on the real AMI board
The web RESET POST and the Hetzner sub-account delete are covered by the httptest-driven orchestration
tests + offsite unit tests (the offsite `Deprovision` faithfully mirrors the live-proven
`ReissueCredentials`, swapping reset→delete). The hub UI is operator-password-gated → CC verifies the UI
via render tests, not a live click. A live Hetzner create/delete drill was not run (billed external
service; mirror-of-live-proven + idempotency-tested).
Shape **(a) DEFAULT** stands unless Viktor picks otherwise: the F1 closure folds into the supervised
N100 rehearsal (its install ISO built `--loader mkimage`). Shape **(b)** — the zero-risk `n100-safety`
pre-flight ISO — is **built and sha-recorded** (above), ready either way; **do not boot without Viktor**.
### Docs diffs
`scripts/CHANGELOG.md` (v1.18.0); `scripts/iso/README.md` (loader-modes section + N100 SB-off prep +
`mkimage-surgery.sh` row); `scripts/iso/profiles/n100.profile` (new, mkimage); `VALIDATION-n100-baremetal`
F1 → PIPELINE-FIXED erratum; `00-capability-map.md` (new bare-metal ISO row + loader caveat);
`ROADMAP.md` (slice B collapsed to SHIPPED; PXE one-line deferred note); `CONTEXT.md`.
### Observations
- **The literal `grub-mkimage` command was unrecoverable** — it ran on the N100's old system (wiped at
the run) and is not in `~/n100-baremetal/` nor 180's history. Only the *procedure* is recorded
(VALIDATION F1 / CONTEXT). Reproduced faithfully: mkimage from a working (non-ISO) 2.12 GRUB build,
module set = the ISO's grub.cfg needs, embedded `search --fs-uuid``configfile`. Not a strategy
re-derivation.
- The mkimage re-master **drops Apple HFS+/APM (Mac boot)** — irrelevant for N100/PC, and required to
avoid xorriso's overlap check. BIOS + UEFI + GPT-ESP boot all preserved (asserted in-build + Leg 4).
- Leg 2's "first-boot stub fires" is assured transitively: Leg 4 proved the first-boot payload is
byte-identical to the shim variant's (slice-A-proven), and Leg 2 proved the answer is consumed. A
full ~10-min nested install-to-first-boot was not re-run for the mkimage ISO on that basis.
@@ -31,6 +31,7 @@
|---|---|---|---|---|
| Appliance day-0 install: golden image → first boot → auto-confirm (zero clicks) → claimable box | installer, agent, hub, golden | **PROVEN-LIVE** (nested VM) | `DRILL-day0-vm-2026-07-12`, `DRILL-day0-take2-2026-07-12` | First firing on real customer hardware pending → R-1 |
| BYO install: `--mode byo`, mandatory caps, host-mutation disclosure, coexistence guards | installer v1.15+, agent | **PARTIAL** | `DRILL-GL6-2026-07-08` (demo box); GL-8 coexistence fixes | Peti clean-slate reinstall on proxmox2 is the first real BYO run of the current path → R-1 |
| Bare-metal Felhom ISO (blank hardware → zero-touch auto-install → first-boot `host-install`), + selectable UEFI loader | scripts v1.18.0 (`scripts/iso/`) + assistant container | **PARTIAL** (nested VM 310/311) | slice A `SPIKE-baremetal-iso-2026-07-16` (build gate, disk-filter fail-safe, stub→public-channel host-install fetch); slice B RUNBOOK-B legs (2026-07-17): shim boots + installs under OVMF **SB-enforcing** + SeaBIOS; **`--loader mkimage` boots + installs under OVMF SB-off**; mkimage under SB-enforcing **FAILS** (`Access Denied`); surgery byte-identical payload | **F1 loader caveat:** `--loader mkimage` fixes cheap AMI (`AN3PLUS`-class) firmware that can't USB-boot the stock GRUB — but the loader is **UNSIGNED → the target board's Secure Boot must be OFF**; default `shim` keeps Secure Boot working on compliant firmware. Physical boot on the real N100 board still pending → supervised rehearsal (R-1). ISO is secret-bearing (slice C de-secures) → R-21 |
| Customer claim: one-time emailed code → customer sets own password (bcrypt, operator never sees it) | controller v0.122, hub v0.50 | **PROVEN-LIVE** (drill VM) | `DRILL-day0-vm-2026-07-12` §10/F-4 (gate ON via real edge; claimed, code consumed) | Never executed by a non-Viktor human → R-3. (Dropped mis-cited `CAMPAIGN-4` F-C — that is the escrow-claim 502, not password claim) |
| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | Endpoints driven on the drill VM; customer-facing **browser wizard** leg not yet live-validated. First supervised ceremony with a real customer pending → R-1. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention):** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code (turns the reinstall-orphan incident from "history destroyed" into "history recoverable"). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody` |
| DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **IMPLEMENTED** | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected. Candidate upgrade to PROVEN-LIVE — see REPORT.) **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)``converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant |
+1 -1
View File
@@ -55,7 +55,7 @@
| R-13 | OOB management arc: dual-use existing WireGuard + hub desired-state channel as mutual-repair | L | **first slice PROVEN-LIVE (poke channel)** | **FIRST SLICE PROVEN-LIVE — the agent-plane poke channel (Direction-2a), agent v0.89.0 + hub v0.59.0 (2026-07-17):** the ep0-relayed contentless poke (hub→ep0 `felhom-poke` forced-cmd→UDP→box WG /32:51822, peer-confined, zero ep0/box infra change) reaches the **agent** and fires an immediate desired-state cycle. Full path live: real operator manifest save → `sync-poke delivered to 10.77.0.2`; box → `poke received → immediate desired-state cycle` (~31 ms ep0→box, save→tick ≈ ~0.45 s). This is ONLY the listener+sender; the rest of the mutual-repair arc (self-heal actions over the channel) stays open. Per `SPIKE-immediate-sync-transport-2026-07-16` P4. The controller-plane Direction-2 wait channel (hub v0.58 / controller v0.140) shipped the config-puller leg separately |
| R-14 | Headscale/WireGuard spike: Minecraft/gaming port connectivity (CGNAT-proof, sovereign DERP fallback) | M | idea | |
| R-15 | Multi-user dashboard accounts (household members, roles) | L | idea | Single password is a stated alpha limitation (R-11) |
| R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → claim-code pairing (option C) | XL | **in-progress** (slice A shipped 2026-07-16; physical run 2026-07-16) | **PHYSICAL RUN 2026-07-16 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`):** demo N100 reinstalled clean-slate from a pipeline ISO → chain reached **rc-0 first try on real hardware** (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. **F1 (HIGH, slice-B input):** this cheap AMI `AN3PLUS 0.01` firmware won't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`) — SB-off/shim-bypass don't help; worked around live with a `grub-mkimage` loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B** = physical N100 RUNBOOK (real DMI/firmware/USB/prior-LVM). **Slice C** = claim-pairing so the *distributed* ISO carries no retrieval passphrase (slice-A ISO is secret-bearing, supervised/single-use). Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* |
| R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → claim-code pairing (option C) | XL | **in-progress** (slice A shipped 2026-07-16; physical run 2026-07-16) | **PHYSICAL RUN 2026-07-16 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`):** demo N100 reinstalled clean-slate from a pipeline ISO → chain reached **rc-0 first try on real hardware** (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. **F1 (HIGH, slice-B input):** this cheap AMI `AN3PLUS 0.01` firmware won't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`) — SB-off/shim-bypass don't help; worked around live with a `grub-mkimage` loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B — SHIPPED (scripts v1.18.0, 2026-07-17):** the F1 firmware fix is now a first-class pipeline mode `build-felhom-iso.sh --loader shim|mkimage` (default shim; `mkimage` = monolithic grub-mkimage loader from the ISO's own GRUB, recipe from the run evidence). RUNBOOK-B legs on nested VM 311 proved it: shim boots+installs under OVMF SB-enforcing + SeaBIOS; mkimage boots+installs under OVMF SB-off; mkimage under SB-enforcing FAILS `Access Denied` (unsigned → **SB must be OFF**, documented); surgery byte-identical payload. **Physical N100 boot on the real board still pending** → folds into the supervised rehearsal (R-1; an `n100-safety` match-nothing ISO is built + sha-recorded for a zero-risk pre-flight). **PXE/network-boot** (the third F1 option, for boards where even USB-mkimage fails) stays a deferred note under this item — not built. **Slice C** = claim-pairing so the *distributed* ISO carries no retrieval passphrase (slice-A/B ISOs are secret-bearing, supervised/single-use). Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* |
## Absorbed / superseded notes in this folder
@@ -87,7 +87,7 @@ boards where USB-grub is broken.
| # | Sev | Finding | Root cause | Disposition / fix |
|---|-----|---------|-----------|-------------------|
| **F1** | **HIGH** | ISO won't UEFI-boot GRUB from USB on this AMI `AN3PLUS 0.01` firmware (`relocation 0x0…`) | firmware can't relocate the ISO's signed GRUB from USB; SB-off and shim-bypass don't help | **worked around live** (self-built `grub-mkimage` loader from the box's own GRUB). R-21 slice-B: bake a firmware-compatible loader into the pipeline, or ship the field-recipe / PXE path. Nested virt could never catch this — the flagged OPEN, now concrete |
| **F1** | **HIGH** | ISO won't UEFI-boot GRUB from USB on this AMI `AN3PLUS 0.01` firmware (`relocation 0x0…`) | firmware can't relocate the ISO's signed GRUB from USB; SB-off and shim-bypass don't help | **worked around live** (self-built `grub-mkimage` loader from the box's own GRUB). **PIPELINE-FIXED in scripts v1.18.0 (2026-07-17):** `build-felhom-iso.sh --loader mkimage` bakes the monolithic grub-mkimage loader in as a first-class mode (recipe reproduced from this run's evidence, not re-derived); `profiles/n100.profile` uses it. Validated on nested VM 311 — mkimage boots + auto-installs under OVMF **Secure Boot OFF** (leg 2), and under **SB enforcing FAILS** with firmware `Access Denied` (leg 3, red-proof) → **the loader is unsigned, so the target board's Secure Boot must be OFF** (documented). **Physical closure on the real AMI board still pending** — it folds into the supervised N100 rehearsal (an optional zero-risk `n100-safety` match-nothing ISO is built + sha-recorded for a pre-flight). |
| **F2** | MEDIUM | No claim-code email on reinstall of an existing customer | claim state (`claim_code_generation:2`, hash, issued 2026-07-13) is **hub/customer-level** and is delivered to the fresh box; an existing code ⇒ no re-issue/re-email. Fresh box has **no password set** | need a **"re-issue claim code"** operator action (bump generation + email). ~~**⚠ also verify** whether the reinstalled *unclaimed* box is properly gated or accidentally **open** (F-4 class)~~ **ERRATUM 2026-07-16 (Viktor):** the ⚠ is RETRACTED — the claim gate WAS presented at felhom.demo-felhom.eu; the customer self-served a new code, claimed, and set a password. F2 is a continuity/UX gap, not a gating hole. **SHIPPED hub v0.57.0**`claim.ReissueForReenroll` auto-issues a reset code on clean-slate re-enroll (host-enroll mint path). |
| **F3** | MEDIUM | Offsite target missing on the fresh controller → escrow blocked | offsite transient password is "delivered to the controller **once**" — it went to the *old* box; the fresh controller never got it. Hub showed provisioned + escrow-done → **hub/controller desync** | **"Re-issue offsite credentials"** in the hub restaged it (done during the run; controller picks up next config refresh). Codify: reinstall must re-issue offsite. **SHIPPED hub v0.57.0** — the re-enroll mint path calls the same machinery (`ReissueOffsiteForCustomer`) automatically. |
| **F4** | MEDIUM | PBS-DR read 403s every tick | agent token `felhom-agent@pve!agent` has `FelhomAgentStore` on `/storage/**felhom-pbs**` only, but the customer's **PVE STORAGE ID is `felhom-offsite`** (non-default, the demo's adopted manual entry) → `GET /storage/felhom-offsite -> 403 (missing Datastore.Allocate)` | ~~install ACL must grant on the **config's storage id**~~ **ERRATUM/DISPOSITION 2026-07-16:** an installer fix is **not feasible** — the DR storage id lives in the agent-domain **pbs_dr descriptor** (`web/pbsdr.go` `StorageID`), provisioned *after* WG registration, so `step_agent_config()` cannot know it at ACL-grant time. The real block is a bootstrap circularity: the agent's reconcile tick does a **token-auth** `GET /storage/<id>` pre-check that 403s and aborts **before** its own root-run `felhom-pbs-apply grant` sets the ACL. Root fix is **agent-side** (proceed to the root-run apply despite the pre-check 403, or run the pre-check as root) — logged as a ROADMAP agent-train item; the demo was unblocked live with a one-shot `pveum` grant on `/storage/felhom-offsite`. Every default-storage-id (all new/Peti installs) already works — F4 only bites non-default ids. |
+32
View File
@@ -1,5 +1,37 @@
# Felhom scripts — Changelog
## build-felhom-iso.sh v1.18.0 — firmware loader option `--loader shim|mkimage` (R-21 slice B, F1) (2026-07-17)
Closes N100 finding **F1 (HIGH):** cheap AMI (`AN3PLUS 0.01`-class) UEFI firmware can't relocate the
ISO's stock signed GRUB from USB (`relocation 0x0 is not implemented yet`; SB-off + shim-bypass don't
help). The run's live `grub-mkimage` workaround is now a first-class pipeline mode.
- **`--loader shim|mkimage`** (default **shim**; profile-settable via `FELHOM_LOADER`, `--loader`
wins). `shim` = the stock output, **byte-for-byte unchanged** (the surgery is skipped) — keeps the
MS-signed shim→GRUB chain and Secure Boot on compliant firmware (S2b). `mkimage` replaces the ISO's
UEFI boot path with a **monolithic `grub-mkimage` loader** built from the ISO's own GRUB modules
(module set from the ISO's `grub.cfg`; embedded `search --fs-uuid``configfile` the real menu).
The recipe is reproduced from the N100 run evidence, not re-derived.
- **`mkimage-surgery.sh`** (new): runs in the assistant container **after** `prepare-iso`, so the
answer/first-boot payload is provably untouched except the loader. Swaps the loader into the ISO9660
tree (real lowercase Rock-Ridge path) **and** inside the `efi.img` ESP (FAT), then re-masters with
`xorriso` preserving the hybrid BIOS boot (grub2-mbr + El Torito) and the GPT EFI System Partition,
dropping only the irrelevant Apple HFS+/APM map (which trips xorriso's overlap check on re-emit).
- **Loud + unmistakable (rule 4):** build banner, manifest `loader:` + `grub-mkimage:` fields, and a
**`-mkimage`** filename suffix when non-default. mkimage ISOs are **unsigned → Secure Boot must be
OFF** on the target board.
- **Dockerfile.assistant:** adds `grub-common` + `grub-efi-amd64-bin` + `mtools` + `dosfstools` (grub
2.12 == the PVE 9.x ISO's `2.12-9+pmx2` generation). **New `profiles/n100.profile`** (cheap-board,
`FELHOM_LOADER="mkimage"`, SB-off note).
- **Validated (RUNBOOK-B legs, nested VM 311 on felhom-pve):** leg 1 shim boots + zero-touch installs
under OVMF **Secure Boot enforcing** (+ SeaBIOS boot — hybrid path survived); leg 2 `--loader
mkimage` boots + installs under OVMF **SB-off** (the monolith's embedded fs-uuid search found the
ISO + chained its menu); **leg 3 (red-proof)** the mkimage ISO under **SB-enforcing FAILS**
firmware `Access Denied`, no boot; leg 4 the surgery is surgical (only the loader + regenerated El
Torito artifacts differ; the full install payload is byte-identical). `bash -n` + shellcheck clean.
Physical N100 closure folds into the supervised rehearsal (an `n100-safety` match-nothing ISO built
+ sha-recorded, unbooted, for an optional zero-risk pre-flight).
## felhom-tenantsync.sh v1.1.0 — deprovision op (customer RESET teardown) (2026-07-17)
Adds the `{"op":"deprovision","customer_id":"<id>"}` op the slice-1 header explicitly reserved
+5
View File
@@ -9,8 +9,13 @@
# Build: docker build -f scripts/iso/Dockerfile.assistant -t felhom-iso-assistant:trixie scripts/iso
FROM debian:trixie
# xorriso: ISO (re-)mastering. shellcheck: lint. grub-mkimage (+ x86_64-efi module dir) and mtools:
# the slice-B mkimage loader surgery — build a monolithic UEFI loader from the ISO's own GRUB modules
# and inject it into the efi.img ESP (FAT, via mtools). grub 2.12 on trixie == the PVE 9.x ISO's
# 2.12-9+pmx2 generation, so the tool matches the modules it embeds.
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates wget gnupg xorriso shellcheck \
grub-common grub-efi-amd64-bin mtools dosfstools \
&& rm -rf /var/lib/apt/lists/*
# PVE 9.x (trixie) no-subscription repo — pairs the assistant to the 9.2 ISO generation.
+42 -6
View File
@@ -1,4 +1,4 @@
# Felhom bare-metal ISO pipeline (R-21 slice A)
# Felhom bare-metal ISO pipeline (R-21 slice A + B)
Turns the official Proxmox VE ISO into a **Felhom auto-install ISO** whose first-boot chain,
unattended, fetches and runs `felhom-host-install.sh` until the host is enrolled and a guest is
@@ -18,7 +18,8 @@ channel).
| File | Role |
|---|---|
| `Dockerfile.assistant` | build host: Debian trixie + `proxmox-auto-install-assistant` (paired to the ISO's PVE generation by Debian codename) + `xorriso` + `shellcheck` |
| `build-felhom-iso.sh` | the pipeline: render answer, mint throwaway root hash, **validate-answer output-parse gate**, render stub, `prepare-iso`, emit ISO + sha256 + manifest |
| `build-felhom-iso.sh` | the pipeline: render answer, mint throwaway root hash, **validate-answer output-parse gate**, render stub, `prepare-iso`, **optional `--loader mkimage` surgery**, emit ISO + sha256 + manifest |
| `mkimage-surgery.sh` | slice-B loader surgery (runs in the container, post-`prepare-iso`): builds a monolithic `grub-mkimage` UEFI loader and swaps it into the ISO's EFI boot path (F1 firmware fix) |
| `answer.toml.tmpl` | Proxmox answer template (`__FQDN__` / `__ROOT_HASH__` / `__ROOT_SSH_KEYS__` / `__DISK_SETUP__` placeholders; `[first-boot]` from-iso/fully-up) |
| `profiles/*.profile` | disk-selection + fqdn fragments (sourceable, no secret) |
| `stub-first-boot.sh` | the ONE first-boot executable (skeleton; build injects the bootstrap script/unit/env as base64) |
@@ -66,6 +67,33 @@ glob. **A filter matching nothing, or a non-existent disk, fails-safe**: the ins
1, no disk touched) rather than installing on the wrong disk (spike S5c). Never commit a real key in
`FELHOM_ROOT_SSH_KEY` unless it is meant to ship in every ISO built from that profile.
A profile may also set `FELHOM_LOADER="shim"|"mkimage"` (see below); `--loader` on the command line
overrides it.
## Loader modes (`--loader shim|mkimage`, slice B)
The UEFI boot loader is selectable. Default **shim**; a profile can pin `FELHOM_LOADER`; `--loader`
wins over both.
- **`shim`** (default) — the stock output: the ISO's MS-signed shim → GRUB chain. **Secure Boot keeps
working** on compliant firmware (no MOK, no keypress — spike S2b). Use this everywhere it boots.
- **`mkimage`** — replaces the ISO's UEFI boot path with a **monolithic `grub-mkimage` loader** built
from the ISO's own GRUB modules. This is the fix for cheap AMI (`AN3PLUS 0.01`-class) mini-PC
firmware that **cannot UEFI-boot the ISO's stock GRUB from USB** (`relocation 0x0 is not implemented
yet`; SB-off and shim-bypass don't help) — the F1 finding of `tests/VALIDATION-n100-baremetal-2026-07-16.md`,
worked around live during that run and now first-class. The recipe (module set from the ISO's
`grub.cfg`, embedded `search --fs-uuid``configfile` the real menu) is reproduced from the run
evidence, not re-derived.
> **⚠ mkimage is UNSIGNED → the target board MUST have Secure Boot OFF.** With Secure Boot
> enforcing the firmware refuses the loader (`Access Denied`, no boot — proven, RUNBOOK-B leg 3).
> mkimage ISOs are named with a **`-mkimage`** suffix and the manifest's `loader:` line says so;
> never confuse them with a shim (SB-capable) build. shim mode's output is byte-for-byte the same as
> before this flag existed (the surgery is skipped entirely).
The surgery runs **after** `prepare-iso` and touches only the EFI boot path — the answer file and
the on-first-boot payload are provably byte-identical to the shim variant's (RUNBOOK-B leg 4).
## bootstrap-env (SECRET-BEARING)
The `--bootstrap-env` file becomes the in-ISO `/etc/felhom/bootstrap.env`:
@@ -104,17 +132,25 @@ producer steps re-run each pass).
## N100 profile (for RUNBOOK B)
The nested profile uses `disk-list=["sda"]`. **A real single-disk mini-PC should use a unique-serial
udev filter, not `disk-list`:**
Use the committed `profiles/n100.profile` template (cheap-board, `FELHOM_LOADER="mkimage"`). **A real
single-disk mini-PC uses a unique-serial udev filter, not `disk-list`:**
1. Boot once (or use a live env), harvest the target disk's udev id:
`udevadm info --query=property --name=/dev/sda | grep -E 'ID_SERIAL_SHORT|ID_MODEL|ID_WWN'`.
2. Write a profile: `filter.ID_SERIAL_SHORT = "<harvested>"` + `filter-match = "all"`. A
2. In the profile, set `filter.ID_SERIAL_SHORT = "<harvested>"` + `filter-match = "all"`. A
match-nothing filter fails-safe (won't wipe the wrong disk).
3. **Prior-LVM prerequisite:** the auto-installer does **not** force-clear a pre-existing LVM PV
signature — a disk with a previous Linux/Proxmox install aborts at "cannot init physical volume"
(spike S2b). Wipe first: `blkdiscard -f /dev/<disk>` (or `wipefs -a` + zero the PV region).
4. UEFI + Secure Boot: no caveat — installs and boots under SB enforcing, no MOK/keypress (spike S2b).
4. **Boot loader + Secure Boot (F1):** cheap AMI (`AN3PLUS`-class) firmware can't boot the stock GRUB
from USB, so `profiles/n100.profile` uses `--loader mkimage`. **That loader is unsigned → set
Secure Boot to OFF/Disabled in the BIOS** before booting the stick (with SB enforcing the board
refuses it: `Access Denied`, no boot). Firmware that boots the stock ISO fine needs no mkimage and
keeps Secure Boot working (default `shim`, spike S2b). *Optional pre-flight (operator, no risk):* a
`n100-safety` ISO — `--loader mkimage` + a **match-nothing** filter — boots the board, loads GRUB,
and the installer aborts touching nothing (S5c). It confirms the loader + SB-off setting before the
real install; its sha is recorded at build. Never boot the real install-armed ISO except in the
supervised rehearsal.
5. **BIOS "State After G3" → "Power On" (or "Last State")** before final sign-off — so the mini-PC
auto-powers-on after a mains loss (a managed appliance must come back without a physical press).
Set it in the BIOS power/ACPI menu; it is board-specific and cannot be scripted (operator step).
+68 -8
View File
@@ -1,6 +1,6 @@
#!/bin/bash
#===============================================================================
# build-felhom-iso.sh — R-21 slice A: turn the official PVE ISO into a Felhom auto-install ISO.
# build-felhom-iso.sh — R-21 slice A+B: turn the official PVE ISO into a Felhom auto-install ISO.
#
# Renders answer.toml (from answer.toml.tmpl + a profile), mints a fresh THROWAWAY root hash,
# gates the answer through validate-answer by PARSING ITS OUTPUT (never $? — validate-answer returns
@@ -8,15 +8,26 @@
# script/unit/env), and runs prepare-iso --fetch-from iso --on-first-boot. Emits the ISO + sha256 +
# a build manifest.
#
# SLICE B — --loader shim|mkimage (default shim). shim = the stock output (MS-signed shim→GRUB chain,
# keeps Secure Boot working on compliant firmware, spike S2b). mkimage = replace the ISO's UEFI boot
# path with a monolithic grub-mkimage-built BOOTX64.EFI built from the ISO's OWN GRUB modules — the
# workaround PROVEN LIVE during the N100 run (VALIDATION-n100-baremetal F1: cheap AMI AN3PLUS-class
# firmware can't relocate the ISO's signed GRUB from USB, `relocation 0x0`). The mkimage loader is
# UNSIGNED → the target board MUST have Secure Boot OFF (documented in the n100 profile's prep). The
# loader surgery runs AFTER prepare-iso (the assistant's answer/first-boot payload is provably
# untouched except the loader path).
#
# SECRET-BEARING: if the bootstrap-env carries a retrieval passphrase (it must, for an unattended
# install — see README "secret-bearing"), the produced ISO embeds it. Supervised/single-use only;
# never distributed; delete after the run. The build log says so loudly.
#
# Runs on DooPlex; delegates validate-answer + prepare-iso to the felhom-iso-assistant container.
# Runs on DooPlex; delegates validate-answer + prepare-iso + the mkimage surgery to the
# felhom-iso-assistant container (which carries proxmox-auto-install-assistant, xorriso, grub-mkimage,
# and mtools).
#===============================================================================
set -euo pipefail
ISO_VERSION="1.16.0" # Felhom release the ISO is tagged to (aligns with felhom-host-install SCRIPT_VERSION).
ISO_VERSION="1.18.0" # Felhom release the ISO is tagged to (aligns with felhom-host-install SCRIPT_VERSION).
IMAGE="${FELHOM_ISO_ASSISTANT_IMAGE:-felhom-iso-assistant:trixie}"
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -32,6 +43,7 @@ log_dry() { echo -e "${CYAN}[DRY-RUN]${NC} $1"; }
die() { log_error "$1"; exit 1; }
PVE_ISO=""; ISO_SHA256=""; PROFILE=""; BOOTSTRAP_ENV=""; OUT_DIR="${HOME}/felhom-iso/out"; PVE_VERSION=""; DRY_RUN=false
LOADER_CLI="" # --loader override; empty = fall back to the profile, then the shim default.
usage() {
cat <<EOF
Usage: build-felhom-iso.sh --pve-iso PATH --iso-sha256 SHA --profile FILE --bootstrap-env FILE [options]
@@ -43,6 +55,10 @@ Required:
--bootstrap-env FILE the in-ISO /etc/felhom/bootstrap.env (SECRET-BEARING: retrieval passphrase).
Must define FELHOM_CUSTOMER_ID, FELHOM_MODE, FELHOM_RETRIEVAL_PASSPHRASE.
Options:
--loader shim|mkimage UEFI boot loader (default: shim, or the profile's FELHOM_LOADER; --loader wins).
shim = stock MS-signed chain (Secure Boot OK on compliant firmware).
mkimage = monolithic grub-mkimage loader for cheap AMI boards that can't boot
the ISO's GRUB from USB (F1). UNSIGNED -> the target board needs Secure Boot OFF.
--out DIR output directory (default: \$HOME/felhom-iso/out)
--pve-version VER override PVE version tag (default: parsed from the ISO filename)
--dry-run print the steps without producing an ISO
@@ -55,6 +71,7 @@ while [[ $# -gt 0 ]]; do
--iso-sha256) ISO_SHA256="$2"; shift 2 ;;
--profile) PROFILE="$2"; shift 2 ;;
--bootstrap-env) BOOTSTRAP_ENV="$2"; shift 2 ;;
--loader) LOADER_CLI="$2"; shift 2 ;;
--out) OUT_DIR="$2"; shift 2 ;;
--pve-version) PVE_VERSION="$2"; shift 2 ;;
--dry-run) DRY_RUN=true; shift ;;
@@ -62,6 +79,8 @@ while [[ $# -gt 0 ]]; do
*) die "unknown argument: $1 (see --help)" ;;
esac
done
[[ -z "$LOADER_CLI" || "$LOADER_CLI" == "shim" || "$LOADER_CLI" == "mkimage" ]] \
|| die "--loader must be 'shim' or 'mkimage' (got '$LOADER_CLI')"
[[ -n "$PVE_ISO" ]] || die "--pve-iso is required"
[[ -n "$ISO_SHA256" ]] || die "--iso-sha256 is required"
@@ -88,7 +107,7 @@ PROFILE_NAME="$(basename "$PROFILE")"; PROFILE_NAME="${PROFILE_NAME%.profile}"
# --- load + validate profile ----------------------------------------------------------------------
log_step "loading profile: $PROFILE"
FELHOM_FQDN=""; FELHOM_DISK_SETUP=""; FELHOM_ROOT_SSH_KEY=""
FELHOM_FQDN=""; FELHOM_DISK_SETUP=""; FELHOM_ROOT_SSH_KEY=""; FELHOM_LOADER=""
# shellcheck disable=SC1090
source "$PROFILE"
[[ -n "$FELHOM_FQDN" ]] || die "profile missing FELHOM_FQDN"
@@ -97,6 +116,20 @@ source "$PROFILE"
ROOT_SSH_LINE=""
[[ -n "$FELHOM_ROOT_SSH_KEY" ]] && ROOT_SSH_LINE="root-ssh-keys = [\"${FELHOM_ROOT_SSH_KEY}\"]"
# --- resolve the loader mode: --loader wins over the profile's FELHOM_LOADER over the shim default ---
LOADER="${LOADER_CLI:-${FELHOM_LOADER:-shim}}"
[[ "$LOADER" == "shim" || "$LOADER" == "mkimage" ]] \
|| die "profile FELHOM_LOADER must be 'shim' or 'mkimage' (got '$FELHOM_LOADER')"
if [[ "$LOADER" == "mkimage" ]]; then
echo -e "${YELLOW}==================================================================================${NC}"
log_warn "LOADER MODE = mkimage — the UEFI boot path is a monolithic grub-mkimage loader (F1 fix)."
log_warn "This loader is UNSIGNED: the target board MUST have Secure Boot OFF. shim/SB is bypassed."
log_warn "The ISO filename gains '-mkimage'; never confuse it with a shim (SB-capable) build."
echo -e "${YELLOW}==================================================================================${NC}"
else
log_info "loader mode = shim (stock MS-signed chain; Secure Boot works on compliant firmware)"
fi
# --- validate bootstrap-env (secret-bearing detection) --------------------------------------------
log_step "checking bootstrap-env (secret-bearing detection)"
( set +e
@@ -112,7 +145,8 @@ log_warn "this ISO will be SECRET-BEARING (embeds the customer retrieval passphr
# --- workspace ------------------------------------------------------------------------------------
WORK="$(mktemp -d "${TMPDIR:-/tmp}/felhom-iso.XXXXXX")"
cleanup() { rm -rf "$WORK"; }
# chmod first: the mkimage surgery's osirrox extract leaves read-only file modes that rm can't clear.
cleanup() { chmod -R u+w "$WORK" 2>/dev/null || true; rm -rf "$WORK"; }
trap cleanup EXIT
mkdir -p "$OUT_DIR" "$WORK/tmp"
ISO_DIR="$(cd "$(dirname "$PVE_ISO")" && pwd)"; ISO_BASE="$(basename "$PVE_ISO")"
@@ -174,10 +208,14 @@ chmod 0755 "$STUB"
grep -q '@@BOOTSTRAP_.*_B64@@' "$STUB" && die "stub still has unfilled markers — injection failed"
# --- prepare-iso ----------------------------------------------------------------------------------
OUT_ISO="$OUT_DIR/felhom-pve-${PVE_VERSION}-v${ISO_VERSION}-${PROFILE_NAME}.iso"
# Rule 4: the loader mode is loud in the filename — a '-mkimage' ISO implies Secure-Boot-off prep.
LOADER_SUFFIX=""; [[ "$LOADER" != "shim" ]] && LOADER_SUFFIX="-${LOADER}"
OUT_ISO="$OUT_DIR/felhom-pve-${PVE_VERSION}-v${ISO_VERSION}-${PROFILE_NAME}${LOADER_SUFFIX}.iso"
GRUB_VERSION="" # populated by the mkimage surgery (the grub-mkimage build used)
log_step "building ISO: $(basename "$OUT_ISO")"
if $DRY_RUN; then
log_dry "docker run … prepare-iso /iso/$ISO_BASE --fetch-from iso --answer-file /work/answer.toml --on-first-boot /work/stub-first-boot.sh --output /work/out.iso"
[[ "$LOADER" == "mkimage" ]] && log_dry "docker run … (mkimage surgery) grub-mkimage from the ISO's own modules → swap BOOTX64.EFI in the EFI tree + efi.img → xorriso re-master → /work/final.iso"
log_info "DRY-RUN: no ISO produced"
exit 0
fi
@@ -187,15 +225,35 @@ docker run --rm -v "$ISO_DIR":/iso:ro -v "$WORK":/work "$IMAGE" \
--on-first-boot /work/stub-first-boot.sh \
--tmp /work/tmp --output /work/out.iso
[[ -f "$WORK/out.iso" ]] || die "prepare-iso produced no output"
cp "$WORK/out.iso" "$OUT_ISO"
# --- SLICE B: mkimage loader surgery (post-prepare; the assistant payload is untouched but the EFI
# boot path). The recipe is the N100 run's proven workaround (VALIDATION F1) — do NOT re-derive it:
# grub-mkimage from the ISO's OWN x86_64-efi modules (2.12-9+pmx2), embedding the module set the
# ISO's grub.cfg needs + an embedded config that `search --fs-uuid`es the ISO and `configfile`s its
# real menu; then swap BOOTX64.EFI in the ISO9660 EFI/BOOT tree AND inside efi.img, and re-master
# with xorriso preserving BOTH the hybrid BIOS boot and the injected answer/first-boot payload. ---
if [[ "$LOADER" == "mkimage" ]]; then
log_step "applying mkimage UEFI loader (F1 firmware fix; recipe from the N100 run evidence)"
[[ -f "$HERE/mkimage-surgery.sh" ]] || die "mkimage-surgery.sh not found next to build-felhom-iso.sh"
cp "$HERE/mkimage-surgery.sh" "$WORK/mkimage-surgery.sh"
docker run --rm -v "$WORK":/work "$IMAGE" bash /work/mkimage-surgery.sh 2>&1 | sed 's/^/ [surgery] /'
[[ -f "$WORK/final.iso" ]] || die "mkimage surgery produced no output (see [surgery] log above)"
GRUB_VERSION="$(cat "$WORK/grub-version.txt" 2>/dev/null || echo unknown)"
cp "$WORK/final.iso" "$OUT_ISO"
log_success "mkimage loader applied (grub-mkimage: ${GRUB_VERSION})"
else
cp "$WORK/out.iso" "$OUT_ISO"
fi
# --- sha256 + manifest ----------------------------------------------------------------------------
OUT_SHA="$(sha256sum "$OUT_ISO" | awk '{print $1}')"
OUT_SIZE="$(stat -c '%s' "$OUT_ISO")"
ASSISTANT_VER="$(docker run --rm "$IMAGE" proxmox-auto-install-assistant --version 2>&1 | head -1)"
echo "$OUT_SHA $(basename "$OUT_ISO")" > "$OUT_ISO.sha256"
LOADER_NOTE="shim (stock MS-signed chain; Secure Boot OK on compliant firmware)"
[[ "$LOADER" == "mkimage" ]] && LOADER_NOTE="mkimage (monolithic grub-mkimage UEFI loader, F1 fix — UNSIGNED; target board MUST have Secure Boot OFF)"
cat > "$OUT_ISO.manifest.txt" <<EOF
Felhom bare-metal ISO build manifest (R-21 slice A)
Felhom bare-metal ISO build manifest (R-21 slice A+B)
built : $(date -Is)
iso-version-tag : v${ISO_VERSION}
pve-version : ${PVE_VERSION}
@@ -204,6 +262,8 @@ source-iso-sha256 : ${ISO_SHA256}
assistant-version : ${ASSISTANT_VER}
profile : ${PROFILE_NAME}
fqdn : ${FELHOM_FQDN}
loader : ${LOADER_NOTE}
grub-mkimage : ${GRUB_VERSION:-n/a (shim mode; loader unchanged)}
host-install-url : $(grep -oE 'FELHOM_INSTALL_URL=[^ ]*' "$BOOTSTRAP_ENV" 2>/dev/null || echo 'https://felhom.eu/scripts/felhom-host-install.sh (default)')
secret-bearing : ${SECRET_BEARING} (embeds the customer retrieval passphrase — supervised/single-use, delete after the run)
output : $(basename "$OUT_ISO")
+168
View File
@@ -0,0 +1,168 @@
#!/bin/bash
#===============================================================================
# mkimage-surgery.sh — R-21 slice B: replace a prepared Felhom ISO's UEFI boot loader with a
# monolithic grub-mkimage image, so cheap AMI (AN3PLUS-class) firmware that can't relocate the ISO's
# signed GRUB from USB (VALIDATION-n100-baremetal F1, `relocation 0x0`) boots.
#
# RUNS INSIDE the felhom-iso-assistant container; operates on /work/out.iso (the prepare-iso output)
# and writes /work/final.iso + /work/grub-version.txt. NEVER touches the source ISO or the assistant's
# answer/first-boot payload — only the EFI boot path.
#
# The recipe is the N100 run's PROVEN-LIVE workaround (do NOT re-derive it): build BOOTX64.EFI from
# the ISO's OWN x86_64-efi GRUB modules (the box's working 2.12-9+pmx2 build), embedding the module
# set the ISO's grub.cfg needs + an embedded config that `search --fs-uuid`es the ISO and
# `configfile`s its real menu; swap it into the ISO9660 EFI/BOOT tree AND inside efi.img; re-master
# with xorriso preserving BOTH the hybrid BIOS boot and the payload. The image is UNSIGNED (Secure
# Boot must be OFF on the target) — that is the documented mkimage contract.
#===============================================================================
set -euo pipefail
OUT=/work/out.iso
FINAL=/work/final.iso
EX=/work/ex
EMB=/work/embedded.cfg
LDR=/work/BOOTX64.EFI
say() { echo "mkimage-surgery: $*"; }
# The osirrox extract tree is written by container-root; the host-side build cleanup (a non-root user)
# cannot remove it. Remove it here (we ARE root in the container) on every exit path so no /tmp litter
# survives the run.
cleanup_ex() { if [[ -n "${EX:-}" && -e "$EX" ]]; then rm -rf "$EX" 2>/dev/null || true; fi; }
trap cleanup_ex EXIT
[[ -f "$OUT" ]] || { echo "mkimage-surgery: /work/out.iso missing" >&2; exit 2; }
for t in xorriso grub-mkimage mcopy mdir; do
command -v "$t" >/dev/null || { echo "mkimage-surgery: missing tool: $t" >&2; exit 2; }
done
grub-mkimage --version | head -1 > /work/grub-version.txt
say "grub: $(cat /work/grub-version.txt)"
# --- 1. extract the full prepared ISO tree (osirrox) — preserves the answer + first-boot payload.
# osirrox reproduces the ISO's (read-only) file modes, so make the tree writable afterwards or
# the loader swap (step 7) and the workspace cleanup can't overwrite/remove the files. ---------
[[ -e "$EX" ]] && { chmod -R u+w "$EX" 2>/dev/null || true; rm -rf "$EX"; }
mkdir -p "$EX"
xorriso -osirrox on -indev "$OUT" -extract / "$EX" >/dev/null 2>&1
chmod -R u+w "$EX"
say "extracted prepared ISO tree"
# --- 2. GRUB build to assemble the loader from. The N100 fix used the box's OWN INSTALLED 2.12 GRUB
# (a DIFFERENT, working build than the ISO's problem one — which is the whole point). The ISO
# ships modules but NOT kernel.img, so grub-mkimage cannot use the ISO's module dir directly;
# the box used its /usr/lib/grub/x86_64-efi. The container mirrors that: grub 2.12 == the PVE
# 9.x ISO's 2.12-9+pmx2 generation. We take the module BINARIES from here and the module LIST
# from the ISO's own grub.cfg (so we embed exactly what the ISO menu needs). --------------------
GDIR=""
for d in /usr/lib/grub/x86_64-efi /usr/lib/grub/x86_64-efi-signed; do
[[ -f "$d/kernel.img" ]] && { GDIR="$d"; break; }
done
[[ -n "$GDIR" ]] || { echo "mkimage-surgery: no usable GRUB x86_64-efi build (kernel.img) in the container" >&2; exit 3; }
say "grub module source: $GDIR"
GCFG=""
for c in "$EX/boot/grub/grub.cfg" "$EX/boot/grub/x86_64-efi/grub.cfg"; do
[[ -f "$c" ]] && { GCFG="$c"; break; }
done
# --- 3. module list: the base set the search/configfile-from-USB chain needs, PLUS every module the
# ISO's grub.cfg insmod's. Keep only modules present in the GRUB build. This IS "every module
# the grub.cfg needs".
BASE="part_gpt part_msdos msdospart fat exfat iso9660 udf search search_fs_uuid search_fs_file search_label \
configfile normal boot linux linuxefi chain loadenv loopback echo test true cat ls help \
all_video efi_gop efi_uga video video_fb font gfxterm gfxterm_background gfxmenu png jpeg \
terminal serial gzio xzio lzopio minicmd reboot halt probe regexp sleep keystatus read"
CFGMODS=""
[[ -n "$GCFG" ]] && CFGMODS=$(grep -hoE 'insmod[[:space:]]+[a-zA-Z0-9_]+' "$GCFG" | awk '{print $2}' | sort -u)
MODS=""
for m in $BASE $CFGMODS; do
[[ -f "$GDIR/$m.mod" ]] && MODS="$MODS $m"
done
MODS="$(echo "$MODS" | tr ' ' '\n' | awk 'NF' | sort -u | tr '\n' ' ')"
say "embedding $(echo "$MODS" | wc -w) modules from the ISO's own x86_64-efi build"
# --- 4. pin the volume modification-date so the ISO's GRUB fs-uuid is DETERMINISTIC and KNOWN before
# we build the loader (GRUB's iso9660 fs_uuid is derived from the PVD modification timestamp).
# Reuse the prepared ISO's own timestamp verbatim -> the embedded search matches the re-mastered
# image (we pin the same value on re-master in step 8).
MDATE="$(xorriso -indev "$OUT" -report_el_torito as_mkisofs 2>/dev/null \
| grep -oE "modification-date='[0-9]+'" | grep -oE '[0-9]+' | head -1)"
[[ -n "$MDATE" && ${#MDATE} -ge 14 ]] || { echo "mkimage-surgery: could not read the ISO modification-date" >&2; exit 4; }
ISO_UUID="$(echo "${MDATE:0:16}" | sed -E 's/^(.{4})(.{2})(.{2})(.{2})(.{2})(.{2})(.{2})$/\1-\2-\3-\4-\5-\6-\7/')"
say "ISO fs-uuid (from modification-date $MDATE): $ISO_UUID"
# --- 5. embedded config: find the ISO by fs-uuid, then chain its real menu (the recorded recipe) ---
cat > "$EMB" <<CFG
search --no-floppy --fs-uuid --set=root $ISO_UUID
if [ -z "\$root" ]; then search --no-floppy --file --set=root /boot/grub/grub.cfg; fi
set prefix=(\$root)/boot/grub
configfile (\$root)/boot/grub/grub.cfg
CFG
# --- 6. build the monolithic BOOTX64.EFI from the ISO's OWN modules (-d $GDIR) --------------------
# shellcheck disable=SC2086
grub-mkimage -O x86_64-efi -d "$GDIR" -p /boot/grub -c "$EMB" -o "$LDR" $MODS
[[ -s "$LDR" ]] || { echo "mkimage-surgery: grub-mkimage produced no image" >&2; exit 5; }
say "built BOOTX64.EFI ($(stat -c%s "$LDR") bytes)"
# --- 7. swap the loader into BOTH the ISO9660 EFI/BOOT tree AND inside the efi.img ESP. The ISO tree
# uses Rock Ridge (LOWERCASE) names — `/efi/boot/bootx64.efi` — so overwrite the EXISTING files
# in place (case-insensitive find), NEVER mkdir a spurious uppercase path. The efi.img ESP is
# FAT (case-insensitive), the authoritative loader UEFI firmware actually runs from USB. --------
TREE_HITS=0
while IFS= read -r f; do cp "$LDR" "$f"; TREE_HITS=$((TREE_HITS+1)); done \
< <(find "$EX" -ipath '*/efi/boot/bootx64.efi')
while IFS= read -r f; do cp "$LDR" "$f"; done \
< <(find "$EX" -ipath '*/efi/boot/grubx64.efi')
[[ "$TREE_HITS" -ge 1 ]] || { echo "mkimage-surgery: no bootx64.efi found in the ISO9660 tree to replace" >&2; exit 6; }
EFIIMG="$EX/efi.img"
[[ -f "$EFIIMG" ]] || EFIIMG="$(find "$EX" -maxdepth 3 -iname 'efi*.img' | head -1)"
[[ -f "$EFIIMG" ]] || { echo "mkimage-surgery: efi.img ESP not found in the ISO tree" >&2; exit 6; }
# FAT is case-insensitive: ::/EFI/BOOT/BOOTX64.EFI resolves the real loader regardless of stored case.
mcopy -i "$EFIIMG" -o "$LDR" ::/EFI/BOOT/BOOTX64.EFI
if mdir -i "$EFIIMG" ::/EFI/BOOT 2>/dev/null | grep -qi grubx64; then
mcopy -i "$EFIIMG" -o "$LDR" ::/EFI/BOOT/grubx64.efi
fi
say "swapped bootx64.efi in the ISO tree ($TREE_HITS) and inside $(basename "$EFIIMG")"
# --- 8. re-master from the (modified) tree, reproducing the source ISO's boot geometry from its OWN
# as_mkisofs report so we track PVE minor versions. We drop ONLY the Apple APM/HFS+ boot map
# (-hfsplus / -apm-block-size): re-emitting it trips xorriso 1.5.6's "Overlapping MBR partition
# entries" on THIS layout, and Mac boot is irrelevant for N100/PC hardware. We KEEP the hybrid
# BIOS grub2-mbr + El Torito (BIOS eltorito.img + UEFI /efi.img) + the GPT EFI System Partition
# (-efi-boot-part) that USB UEFI firmware boots from — the whole point of this fix. The volume
# id + modification-date are pinned explicitly so the embedded fs-uuid stays valid. -------------
RPT="$(xorriso -indev "$OUT" -report_el_torito as_mkisofs 2>/dev/null)"
VOLID="$(printf '%s\n' "$RPT" | sed -nE "s/^-V '(.*)'\$/\\1/p" | head -1)"; [[ -n "$VOLID" ]] || VOLID="PVE"
# Drop, then re-add explicitly: the volume id + modification-date. Drop entirely: the Apple APM/HFS+
# map (-hfsplus / -apm-block-size) AND the isohybrid GPT-basdat marking (-part_like_isohybrid /
# -isohybrid-gpt-basdat) — prepare-iso re-masters with these, and re-emitting them alongside
# -efi-boot-part + the protective MBR trips xorriso 1.5.6's "Overlapping MBR partition entries". The
# resulting image keeps the protective MBR + grub2-mbr (BIOS) + El Torito (BIOS+UEFI) + the GPT EFI
# System Partition (verified). Repoint the grub2-mbr template at the in-container out.iso.
FILTERED="$(printf '%s\n' "$RPT" \
| grep -vE "^-V '|^--modification-date=|^-apm-block-size |^-hfsplus\$|^-part_like_isohybrid\$|^-isohybrid-gpt-basdat\$" \
| sed -E "s#(--interval:[^']*:)'[^']*'#\\1'$OUT'#")"
rm -f "$FINAL"
# shellcheck disable=SC2086
eval xorriso -as mkisofs -V "'$VOLID'" --modification-date="'$MDATE'" \
$FILTERED -o "$FINAL" "$EX" >/work/xorriso.log 2>&1 \
|| { echo "mkimage-surgery: xorriso re-master FAILED"; tail -25 /work/xorriso.log >&2; exit 7; }
[[ -f "$FINAL" ]] || { echo "mkimage-surgery: no final.iso produced" >&2; exit 7; }
# assert both boot images survived (BIOS eltorito.img + UEFI efi.img) — a silent loss would fail-safe
# to an unbootable stick, so gate it here.
ETIMG="$(xorriso -indev "$FINAL" -report_el_torito plain 2>/dev/null | grep -cE 'El Torito boot img')"
[[ "$ETIMG" -ge 2 ]] || { echo "mkimage-surgery: re-master lost a boot image (El Torito entries=$ETIMG, want >=2)" >&2; exit 8; }
say "re-mastered final.iso ($(stat -c%s "$FINAL") bytes; El Torito boot images=$ETIMG)"
# --- 9. verify the re-mastered image kept the modification-date (so the embedded fs-uuid matches) ---
FINAL_MDATE="$(xorriso -indev "$FINAL" -report_el_torito as_mkisofs 2>/dev/null \
| grep -oE "modification-date='[0-9]+'" | grep -oE '[0-9]+' | head -1)"
if [[ "${FINAL_MDATE:0:14}" != "${MDATE:0:14}" ]]; then
echo "mkimage-surgery: WARN final modification-date ($FINAL_MDATE) != source ($MDATE) — the search fs-uuid may not match; re-check" >&2
else
say "fs-uuid preserved ($ISO_UUID)"
fi
say "done"
+27
View File
@@ -0,0 +1,27 @@
# Felhom ISO build profile — n100 / cheap-board (R-21 slice B).
#
# The canonical profile for cheap mini-PCs whose early AMI (AN3PLUS-class) UEFI firmware CANNOT boot
# the ISO's stock signed GRUB from USB (VALIDATION-n100-baremetal F1, `relocation 0x0`). It flips the
# loader to `mkimage` — a monolithic grub-mkimage UEFI loader built from the ISO's own GRUB, proven
# live on the N100 board.
#
# ⚠ SECURE BOOT MUST BE OFF on the target board. The mkimage loader is UNSIGNED; with Secure Boot
# enforcing the firmware refuses it ("Access Denied", no boot — proven, RUNBOOK-B leg 3). Compliant
# firmware should use the default `shim` loader (drop FELHOM_LOADER) to keep Secure Boot working.
#
# This template ships a MATCH-NOTHING disk filter as a safety default — REPLACE the serial with the
# target SSD's real ID_SERIAL_SHORT (harvested pre-wipe; see README "N100 profile") before a real
# install, or the installer fails-safe (aborts, touches no disk — spike S5c).
FELHOM_FQDN="felhom-host.local"
# mkimage loader (F1 firmware fix). Remove this line on Secure-Boot-capable/compliant firmware.
FELHOM_LOADER="mkimage"
FELHOM_DISK_SETUP='[disk-setup]
filesystem = "ext4"
filter-match = "all"
filter.ID_SERIAL_SHORT = "REPLACE-WITH-TARGET-SSD-SERIAL"'
# Optional emergency/validation key baked into root's authorized_keys (blank -> not baked):
# FELHOM_ROOT_SSH_KEY="ssh-ed25519 AAAA... ops@felhom"