diff --git a/CONTEXT.md b/CONTEXT.md index 98023ab..6323dd8 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,6 +3,20 @@ > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. +- **2026-07-17 — R-21 SLICE B SHIPPED: firmware loader option (scripts v1.18.0).** Closes N100 **F1 + (HIGH)** — cheap AMI (`AN3PLUS 0.01`-class) firmware can't USB-boot the ISO's stock GRUB + (`relocation 0x0`). `build-felhom-iso.sh --loader shim|mkimage` (default **shim**, unchanged; + profile-settable via `FELHOM_LOADER`). **mkimage** = a monolithic `grub-mkimage` loader built from + the ISO's own GRUB (module set from its `grub.cfg`, embedded `search --fs-uuid`→`configfile`), swapped + into the ISO9660 tree + `efi.img` ESP by `mkimage-surgery.sh` **after** `prepare-iso` (payload + provably untouched), re-mastered with xorriso preserving BIOS-hybrid + UEFI + GPT-ESP (drops only the + irrelevant Apple HFS+/APM). **Unsigned → target Secure Boot MUST be OFF.** Loud: banner + manifest + `loader:` + `-mkimage` filename. New `profiles/n100.profile`; Dockerfile gains grub/mtools. + **RUNBOOK-B legs (nested VM 311, OVMF):** shim boots+installs under SB-enforcing + SeaBIOS; mkimage + boots+installs under SB-off; **mkimage under SB-enforcing FAILS `Access Denied`** (red-proof); surgery + byte-identical payload. Physical N100 boot still pending → supervised rehearsal (an `n100-safety` + match-nothing ISO built + sha-recorded, unbooted). PXE stays a deferred R-21 note. See REPORT.md. + - **2026-07-17 — CUSTOMER RESET SHIPPED: hub v0.61.0 + felhom-tenantsync v1.1.0 (LIVE).** The middle lifecycle tier (host delete < RESET < customer Delete). One operator action → pre-first-install: all OPERATIONAL state dies (offsite repo, PBS namespace+backups, DR recipe, one-time secret, claim state, diff --git a/REPORT.md b/REPORT.md index ba0609c..ac01d56 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,58 +2,96 @@ > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md). -## Hub v0.61.0 + felhom-tenantsync v1.1.0 — Customer RESET (middle lifecycle tier) — 2026-07-17 +## scripts v1.18.0 — R-21 slice B: firmware loader option `--loader shim|mkimage` (F1) — 2026-07-17 -**Commits:** `4009401` (code), `e144c5e` (manifest). **Live:** hub **0.61.0** on k3s (Synced/Healthy); -felhom-tenantsync **v1.1.0** on ep0 (`felhom-hetzner`). +**Baseline:** scripts v1.17.0 (host-install) / build-felhom-iso.sh was ISO_VERSION 1.16.0. Target +**v1.18.0**. Closes N100 **F1 (HIGH):** cheap AMI (`AN3PLUS 0.01`-class) UEFI firmware can't relocate +the ISO's stock signed GRUB from USB (`relocation 0x0`); the run's live `grub-mkimage` workaround is +now a first-class pipeline mode. Touched **only** `scripts/iso/`. ### What shipped -The **middle lifecycle tier** between *host delete* and *customer Delete*. One operator action returns a -customer to **pre-first-install**: every OPERATIONAL trace dies — offsite repo (Hetzner sub-account/box), -PBS namespace + all backup groups + token, DR recipe, one-time secret, claim state, retained escrow -custody — while **identity + basic config survive** (`customer_configs` incl. the offsite tier CHOICE, all -provenance rows, and the audit-event stream). +`build-felhom-iso.sh` gains `--loader shim|mkimage` (default **shim**, profile-settable via +`FELHOM_LOADER`, `--loader` wins). **shim = the stock output, byte-for-byte unchanged** (surgery +skipped). **mkimage** replaces the ISO's UEFI boot path with a monolithic `grub-mkimage` loader. +New: `mkimage-surgery.sh`, `profiles/n100.profile`, Dockerfile grub/mtools tooling; loud banner + +manifest `loader:`/`grub-mkimage:` fields + `-mkimage` filename suffix. -Viktor's rulings, all implemented: (1) destroying retained escrow custody gets its **own separate -acknowledgment**; (2) RESET clears claim state → fresh code next onboarding; (3) RESET **refuses while any -host row exists** (delete hosts first — reset never deletes hosts); (4) the confirm surface shows a -**live-counted** inventory. +### Surgery approach actually used (Phase 2) -### Discipline +Recipe reproduced from the run evidence (the literal command was run on the now-wiped N100 old system +and saved nowhere — only the *procedure* is recorded; see Observations). In the assistant container, +**after** `prepare-iso`: +1. `xorriso -osirrox` extract the prepared tree (chmod u+w — osirrox files are read-only). +2. `grub-mkimage -O x86_64-efi` from the **container's** grub 2.12 build (`/usr/lib/grub/x86_64-efi`, + which has `kernel.img` — the ISO ships modules but not kernel.img; the box likewise used its + *installed* GRUB). Module set = the base search/configfile/USB chain **plus every `insmod` in the + ISO's own `grub.cfg`** (57 modules). Embedded config: `search --fs-uuid ` → `configfile` + the real menu. The fs-uuid is derived from the ISO's PVD modification-date (GRUB's iso9660 uuid + source) and pinned on re-master so it stays valid. +3. Swap the loader into the ISO9660 tree (**real lowercase Rock-Ridge path** `/efi/boot/bootx64.efi` — + an early bug created a spurious *uppercase* path instead; fixed) **and** inside the `efi.img` ESP + (FAT, case-insensitive — the loader UEFI firmware actually runs from USB). +4. Re-master with `xorriso -as mkisofs` from the ISO's own `as_mkisofs` report, dropping **only** the + Apple HFS+/APM map (`-hfsplus`/`-apm-block-size`) and the isohybrid GPT-basdat marking + (`-part_like_isohybrid`/`-isohybrid-gpt-basdat`) — re-emitting those trips xorriso 1.5.6's + "Overlapping MBR partition entries" on this layout, and Mac boot is irrelevant. **Kept:** protective + MBR + grub2-mbr (BIOS hybrid) + El Torito (BIOS + UEFI) + the GPT EFI System Partition (USB UEFI). -External teardown FIRST, DB purge LAST (publish-last). Every leg idempotent → a partial run is re-run from -the top; a failed external leg is a clean journal entry and the DB purge (which erases the descriptors that -say what still needs tearing down) is **withheld** until every external leg is `ok`. Provenance + events -are never wiped. +**Stall-rule outcome:** the surgery resisted for ~40 min of iteration (two overlap failures + the +kernel.img and path-case bugs) — **well within the 2 h budget**; no spike-note fallback needed. -### Components +### Per-leg results (RUNBOOK-B, nested VM 311 on felhom-pve, OVMF) -- **store/customer_reset.go**: `customer_resets` journal (per-attempt, per-leg, resumable); - `CustomerResetInventory` (live counts incl. retained blobs via the F-14 `host_deletions` UNION); - `Start/UpdateResetLeg/Finish/LatestCustomerReset`; ack-gated `PurgeCustomerResetDBState`; `DeleteClaim`. -- **claim.ResetToUnclaimed**: deletes the claim row → `EnsureIssued` mints a fresh first code next onboarding. -- **offsite**: `Deprovision` (idempotent label-lookup delete), `OffsiteIdentifier`, `ClearProvisionedDescriptor`. -- **tenantsync + scripts/felhom-tenantsync.sh v1.1.0**: `deprovision` op destroys the namespace + backup - groups + token; shared `felhom@pbs` user untouched; idempotent. -- **web/customer_reset.go**: `GET /configs/{id}/reset` → inventory JSON; `POST` → orchestration (all gates - before any write/external call). A distinct **amber** RESET card + `reset_done` flash. +- **Leg 1 — shim regression:** the default ISO **boots + zero-touch auto-installs under OVMF Secure + Boot ENFORCING** (`pre-enrolled-keys=1`; "EFI boot mode detected", DHCP, "Starting Proxmox + installation", package extraction unattended). **Plus one SeaBIOS boot** — the shim ISO also boots + under BIOS (hybrid grub2-mbr/eltorito path survived the v1.18.0 build). ✓ +- **Leg 2 — mkimage happy path:** `--loader mkimage` ISO **boots + auto-installs under OVMF Secure + Boot OFF** (`pre-enrolled-keys=0`). The monolith's embedded `search --fs-uuid` found the ISO and + chained its real menu → the installer ran. ✓ +- **Leg 3 — trade-off red-proof:** the **same** mkimage ISO under OVMF Secure Boot ENFORCING + **FAILS** — captured firmware behavior: `BdsDxe: failed to load Boot0002 "UEFI QEMU DVD-ROM …" : + Access Denied` → `No bootable option or device was found`. The unsigned loader is refused. This + makes "Secure Boot must be OFF for mkimage" a **fact**, not a guess. ✓ +- **Leg 4 — structural / byte-identity:** diffing the shim(out) vs mkimage(final) trees, **only** + `efi/boot/bootx64.efi`, `efi/boot/grubx64.efi`, `efi.img`, and the regenerated El Torito artifacts + (`boot.cat`, `i386-pc/eltorito.img`) differ — **the entire install payload is byte-identical**. The + tree loader (1253376 B, vs shim's 1032032 B) carries the embedded `2026-05-21-…` fs-uuid and is + identical to the loader inside `efi.img`; **0 spurious uppercase paths**; El Torito keeps both boot + images. The `validate-answer` gate still runs pre-surgery. ✓ -### Verification +### Artifacts (shas + manifests) -- **Green gate:** `go build ./... && go vet ./... && go test ./...` all pass. Hub confirm gate OK. -- **Red-proofs (run-fail-revert, both proven red then restored):** (a) escrow-ack gate — defeat it → - reset proceeds & destroys blobs → FAIL; (b) partial-failure resumability — don't withhold the purge → - DB purged despite an external failure → FAIL. Plus store ack-gating + journal round-trip; offsite - Deprovision idempotency + descriptor clear; RESET-card render. -- **Live drill on ep0 (throwaway `drill-reset-01`, real PBS data):** provision → **real backup written into - the namespace** → deprovision (`deleted:true`; namespace + backup group + token destroyed) → token/ns - confirmed gone → idempotent re-run (`deleted:false`) → **all 3 real tenants + the shared user survived**. +- shim nested-test ISO `…-nested-vm-val.iso` — `28d862e2f7b014588717c15ef8c9bc46d11a9a5689a022f5d4de9e8f939d1377` +- mkimage nested-test ISO `…-nested-vm-val-mkimage.iso` — `71fce3333b3148e48d10711311f275da4b849bf383f711de0f2e4d5802618869` +- **safety ISO** `…-n100-safety-mkimage.iso` — `f9aa0428e5efdcb5e7c0ade05264ff681ac4f9aa47b7f3721c9eb712f025c1a8` + (mkimage + match-nothing filter; **built + sha-recorded, NOT booted**; at `180:~/n100-baremetal/safety/`). + Manifests record `loader: mkimage … Secure Boot OFF`, `grub-mkimage: 2.12-9+deb13u2`, secret-bearing. + Nested-test ISOs + VM 311 destroyed after the legs. -### Not live-drilled (covered otherwise) +### Physical proof on the real AMI board -The web RESET POST and the Hetzner sub-account delete are covered by the httptest-driven orchestration -tests + offsite unit tests (the offsite `Deprovision` faithfully mirrors the live-proven -`ReissueCredentials`, swapping reset→delete). The hub UI is operator-password-gated → CC verifies the UI -via render tests, not a live click. A live Hetzner create/delete drill was not run (billed external -service; mirror-of-live-proven + idempotency-tested). +Shape **(a) DEFAULT** stands unless Viktor picks otherwise: the F1 closure folds into the supervised +N100 rehearsal (its install ISO built `--loader mkimage`). Shape **(b)** — the zero-risk `n100-safety` +pre-flight ISO — is **built and sha-recorded** (above), ready either way; **do not boot without Viktor**. + +### Docs diffs + +`scripts/CHANGELOG.md` (v1.18.0); `scripts/iso/README.md` (loader-modes section + N100 SB-off prep + +`mkimage-surgery.sh` row); `scripts/iso/profiles/n100.profile` (new, mkimage); `VALIDATION-n100-baremetal` +F1 → PIPELINE-FIXED erratum; `00-capability-map.md` (new bare-metal ISO row + loader caveat); +`ROADMAP.md` (slice B collapsed to SHIPPED; PXE one-line deferred note); `CONTEXT.md`. + +### Observations + +- **The literal `grub-mkimage` command was unrecoverable** — it ran on the N100's old system (wiped at + the run) and is not in `~/n100-baremetal/` nor 180's history. Only the *procedure* is recorded + (VALIDATION F1 / CONTEXT). Reproduced faithfully: mkimage from a working (non-ISO) 2.12 GRUB build, + module set = the ISO's grub.cfg needs, embedded `search --fs-uuid` → `configfile`. Not a strategy + re-derivation. +- The mkimage re-master **drops Apple HFS+/APM (Mac boot)** — irrelevant for N100/PC, and required to + avoid xorriso's overlap check. BIOS + UEFI + GPT-ESP boot all preserved (asserted in-build + Leg 4). +- Leg 2's "first-boot stub fires" is assured transitively: Leg 4 proved the first-boot payload is + byte-identical to the shim variant's (slice-A-proven), and Leg 2 proved the answer is consumed. A + full ~10-min nested install-to-first-boot was not re-run for the mkimage ISO on that basis. diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index 6a60dc0..dc6b000 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -31,6 +31,7 @@ |---|---|---|---|---| | Appliance day-0 install: golden image → first boot → auto-confirm (zero clicks) → claimable box | installer, agent, hub, golden | **PROVEN-LIVE** (nested VM) | `DRILL-day0-vm-2026-07-12`, `DRILL-day0-take2-2026-07-12` | First firing on real customer hardware pending → R-1 | | BYO install: `--mode byo`, mandatory caps, host-mutation disclosure, coexistence guards | installer v1.15+, agent | **PARTIAL** | `DRILL-GL6-2026-07-08` (demo box); GL-8 coexistence fixes | Peti clean-slate reinstall on proxmox2 is the first real BYO run of the current path → R-1 | +| Bare-metal Felhom ISO (blank hardware → zero-touch auto-install → first-boot `host-install`), + selectable UEFI loader | scripts v1.18.0 (`scripts/iso/`) + assistant container | **PARTIAL** (nested VM 310/311) | slice A `SPIKE-baremetal-iso-2026-07-16` (build gate, disk-filter fail-safe, stub→public-channel host-install fetch); slice B RUNBOOK-B legs (2026-07-17): shim boots + installs under OVMF **SB-enforcing** + SeaBIOS; **`--loader mkimage` boots + installs under OVMF SB-off**; mkimage under SB-enforcing **FAILS** (`Access Denied`); surgery byte-identical payload | **F1 loader caveat:** `--loader mkimage` fixes cheap AMI (`AN3PLUS`-class) firmware that can't USB-boot the stock GRUB — but the loader is **UNSIGNED → the target board's Secure Boot must be OFF**; default `shim` keeps Secure Boot working on compliant firmware. Physical boot on the real N100 board still pending → supervised rehearsal (R-1). ISO is secret-bearing (slice C de-secures) → R-21 | | Customer claim: one-time emailed code → customer sets own password (bcrypt, operator never sees it) | controller v0.122, hub v0.50 | **PROVEN-LIVE** (drill VM) | `DRILL-day0-vm-2026-07-12` §10/F-4 (gate ON via real edge; claimed, code consumed) | Never executed by a non-Viktor human → R-3. (Dropped mis-cited `CAMPAIGN-4` F-C — that is the escrow-claim 502, not password claim) | | Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | Endpoints driven on the drill VM; customer-facing **browser wizard** leg not yet live-validated. First supervised ceremony with a real customer pending → R-1. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention):** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code (turns the reinstall-orphan incident from "history destroyed" into "history recoverable"). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody` | | DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **IMPLEMENTED** | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected. Candidate upgrade to PROVEN-LIVE — see REPORT.) **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant | diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 6f31d40..723e6a7 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -55,7 +55,7 @@ | R-13 | OOB management arc: dual-use existing WireGuard + hub desired-state channel as mutual-repair | L | **first slice PROVEN-LIVE (poke channel)** | **FIRST SLICE PROVEN-LIVE — the agent-plane poke channel (Direction-2a), agent v0.89.0 + hub v0.59.0 (2026-07-17):** the ep0-relayed contentless poke (hub→ep0 `felhom-poke` forced-cmd→UDP→box WG /32:51822, peer-confined, zero ep0/box infra change) reaches the **agent** and fires an immediate desired-state cycle. Full path live: real operator manifest save → `sync-poke delivered to 10.77.0.2`; box → `poke received → immediate desired-state cycle` (~31 ms ep0→box, save→tick ≈ ~0.45 s). This is ONLY the listener+sender; the rest of the mutual-repair arc (self-heal actions over the channel) stays open. Per `SPIKE-immediate-sync-transport-2026-07-16` P4. The controller-plane Direction-2 wait channel (hub v0.58 / controller v0.140) shipped the config-puller leg separately | | R-14 | Headscale/WireGuard spike: Minecraft/gaming port connectivity (CGNAT-proof, sovereign DERP fallback) | M | idea | | | R-15 | Multi-user dashboard accounts (household members, roles) | L | idea | Single password is a stated alpha limitation (R-11) | -| R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → claim-code pairing (option C) | XL | **in-progress** (slice A shipped 2026-07-16; physical run 2026-07-16) | **PHYSICAL RUN 2026-07-16 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`):** demo N100 reinstalled clean-slate from a pipeline ISO → chain reached **rc-0 first try on real hardware** (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. **F1 (HIGH, slice-B input):** this cheap AMI `AN3PLUS 0.01` firmware won't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`) — SB-off/shim-bypass don't help; worked around live with a `grub-mkimage` loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B** = physical N100 RUNBOOK (real DMI/firmware/USB/prior-LVM). **Slice C** = claim-pairing so the *distributed* ISO carries no retrieval passphrase (slice-A ISO is secret-bearing, supervised/single-use). Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* | +| R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → claim-code pairing (option C) | XL | **in-progress** (slice A shipped 2026-07-16; physical run 2026-07-16) | **PHYSICAL RUN 2026-07-16 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`):** demo N100 reinstalled clean-slate from a pipeline ISO → chain reached **rc-0 first try on real hardware** (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. **F1 (HIGH, slice-B input):** this cheap AMI `AN3PLUS 0.01` firmware won't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`) — SB-off/shim-bypass don't help; worked around live with a `grub-mkimage` loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B — SHIPPED (scripts v1.18.0, 2026-07-17):** the F1 firmware fix is now a first-class pipeline mode `build-felhom-iso.sh --loader shim|mkimage` (default shim; `mkimage` = monolithic grub-mkimage loader from the ISO's own GRUB, recipe from the run evidence). RUNBOOK-B legs on nested VM 311 proved it: shim boots+installs under OVMF SB-enforcing + SeaBIOS; mkimage boots+installs under OVMF SB-off; mkimage under SB-enforcing FAILS `Access Denied` (unsigned → **SB must be OFF**, documented); surgery byte-identical payload. **Physical N100 boot on the real board still pending** → folds into the supervised rehearsal (R-1; an `n100-safety` match-nothing ISO is built + sha-recorded for a zero-risk pre-flight). **PXE/network-boot** (the third F1 option, for boards where even USB-mkimage fails) stays a deferred note under this item — not built. **Slice C** = claim-pairing so the *distributed* ISO carries no retrieval passphrase (slice-A/B ISOs are secret-bearing, supervised/single-use). Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* | ## Absorbed / superseded notes in this folder diff --git a/documentation/tests/VALIDATION-n100-baremetal-2026-07-16.md b/documentation/tests/VALIDATION-n100-baremetal-2026-07-16.md index a562af0..aa9c2e5 100644 --- a/documentation/tests/VALIDATION-n100-baremetal-2026-07-16.md +++ b/documentation/tests/VALIDATION-n100-baremetal-2026-07-16.md @@ -87,7 +87,7 @@ boards where USB-grub is broken. | # | Sev | Finding | Root cause | Disposition / fix | |---|-----|---------|-----------|-------------------| -| **F1** | **HIGH** | ISO won't UEFI-boot GRUB from USB on this AMI `AN3PLUS 0.01` firmware (`relocation 0x0…`) | firmware can't relocate the ISO's signed GRUB from USB; SB-off and shim-bypass don't help | **worked around live** (self-built `grub-mkimage` loader from the box's own GRUB). R-21 slice-B: bake a firmware-compatible loader into the pipeline, or ship the field-recipe / PXE path. Nested virt could never catch this — the flagged OPEN, now concrete | +| **F1** | **HIGH** | ISO won't UEFI-boot GRUB from USB on this AMI `AN3PLUS 0.01` firmware (`relocation 0x0…`) | firmware can't relocate the ISO's signed GRUB from USB; SB-off and shim-bypass don't help | **worked around live** (self-built `grub-mkimage` loader from the box's own GRUB). **PIPELINE-FIXED in scripts v1.18.0 (2026-07-17):** `build-felhom-iso.sh --loader mkimage` bakes the monolithic grub-mkimage loader in as a first-class mode (recipe reproduced from this run's evidence, not re-derived); `profiles/n100.profile` uses it. Validated on nested VM 311 — mkimage boots + auto-installs under OVMF **Secure Boot OFF** (leg 2), and under **SB enforcing FAILS** with firmware `Access Denied` (leg 3, red-proof) → **the loader is unsigned, so the target board's Secure Boot must be OFF** (documented). **Physical closure on the real AMI board still pending** — it folds into the supervised N100 rehearsal (an optional zero-risk `n100-safety` match-nothing ISO is built + sha-recorded for a pre-flight). | | **F2** | MEDIUM | No claim-code email on reinstall of an existing customer | claim state (`claim_code_generation:2`, hash, issued 2026-07-13) is **hub/customer-level** and is delivered to the fresh box; an existing code ⇒ no re-issue/re-email. Fresh box has **no password set** | need a **"re-issue claim code"** operator action (bump generation + email). ~~**⚠ also verify** whether the reinstalled *unclaimed* box is properly gated or accidentally **open** (F-4 class)~~ **ERRATUM 2026-07-16 (Viktor):** the ⚠ is RETRACTED — the claim gate WAS presented at felhom.demo-felhom.eu; the customer self-served a new code, claimed, and set a password. F2 is a continuity/UX gap, not a gating hole. **SHIPPED hub v0.57.0** — `claim.ReissueForReenroll` auto-issues a reset code on clean-slate re-enroll (host-enroll mint path). | | **F3** | MEDIUM | Offsite target missing on the fresh controller → escrow blocked | offsite transient password is "delivered to the controller **once**" — it went to the *old* box; the fresh controller never got it. Hub showed provisioned + escrow-done → **hub/controller desync** | **"Re-issue offsite credentials"** in the hub restaged it (done during the run; controller picks up next config refresh). Codify: reinstall must re-issue offsite. **SHIPPED hub v0.57.0** — the re-enroll mint path calls the same machinery (`ReissueOffsiteForCustomer`) automatically. | | **F4** | MEDIUM | PBS-DR read 403s every tick | agent token `felhom-agent@pve!agent` has `FelhomAgentStore` on `/storage/**felhom-pbs**` only, but the customer's **PVE STORAGE ID is `felhom-offsite`** (non-default, the demo's adopted manual entry) → `GET /storage/felhom-offsite -> 403 (missing Datastore.Allocate)` | ~~install ACL must grant on the **config's storage id**~~ **ERRATUM/DISPOSITION 2026-07-16:** an installer fix is **not feasible** — the DR storage id lives in the agent-domain **pbs_dr descriptor** (`web/pbsdr.go` `StorageID`), provisioned *after* WG registration, so `step_agent_config()` cannot know it at ACL-grant time. The real block is a bootstrap circularity: the agent's reconcile tick does a **token-auth** `GET /storage/` pre-check that 403s and aborts **before** its own root-run `felhom-pbs-apply grant` sets the ACL. Root fix is **agent-side** (proceed to the root-run apply despite the pre-check 403, or run the pre-check as root) — logged as a ROADMAP agent-train item; the demo was unblocked live with a one-shot `pveum` grant on `/storage/felhom-offsite`. Every default-storage-id (all new/Peti installs) already works — F4 only bites non-default ids. | diff --git a/scripts/CHANGELOG.md b/scripts/CHANGELOG.md index 619394f..0d008ab 100644 --- a/scripts/CHANGELOG.md +++ b/scripts/CHANGELOG.md @@ -1,5 +1,37 @@ # Felhom scripts — Changelog +## build-felhom-iso.sh v1.18.0 — firmware loader option `--loader shim|mkimage` (R-21 slice B, F1) (2026-07-17) + +Closes N100 finding **F1 (HIGH):** cheap AMI (`AN3PLUS 0.01`-class) UEFI firmware can't relocate the +ISO's stock signed GRUB from USB (`relocation 0x0 is not implemented yet`; SB-off + shim-bypass don't +help). The run's live `grub-mkimage` workaround is now a first-class pipeline mode. + +- **`--loader shim|mkimage`** (default **shim**; profile-settable via `FELHOM_LOADER`, `--loader` + wins). `shim` = the stock output, **byte-for-byte unchanged** (the surgery is skipped) — keeps the + MS-signed shim→GRUB chain and Secure Boot on compliant firmware (S2b). `mkimage` replaces the ISO's + UEFI boot path with a **monolithic `grub-mkimage` loader** built from the ISO's own GRUB modules + (module set from the ISO's `grub.cfg`; embedded `search --fs-uuid` → `configfile` the real menu). + The recipe is reproduced from the N100 run evidence, not re-derived. +- **`mkimage-surgery.sh`** (new): runs in the assistant container **after** `prepare-iso`, so the + answer/first-boot payload is provably untouched except the loader. Swaps the loader into the ISO9660 + tree (real lowercase Rock-Ridge path) **and** inside the `efi.img` ESP (FAT), then re-masters with + `xorriso` preserving the hybrid BIOS boot (grub2-mbr + El Torito) and the GPT EFI System Partition, + dropping only the irrelevant Apple HFS+/APM map (which trips xorriso's overlap check on re-emit). +- **Loud + unmistakable (rule 4):** build banner, manifest `loader:` + `grub-mkimage:` fields, and a + **`-mkimage`** filename suffix when non-default. mkimage ISOs are **unsigned → Secure Boot must be + OFF** on the target board. +- **Dockerfile.assistant:** adds `grub-common` + `grub-efi-amd64-bin` + `mtools` + `dosfstools` (grub + 2.12 == the PVE 9.x ISO's `2.12-9+pmx2` generation). **New `profiles/n100.profile`** (cheap-board, + `FELHOM_LOADER="mkimage"`, SB-off note). +- **Validated (RUNBOOK-B legs, nested VM 311 on felhom-pve):** leg 1 shim boots + zero-touch installs + under OVMF **Secure Boot enforcing** (+ SeaBIOS boot — hybrid path survived); leg 2 `--loader + mkimage` boots + installs under OVMF **SB-off** (the monolith's embedded fs-uuid search found the + ISO + chained its menu); **leg 3 (red-proof)** the mkimage ISO under **SB-enforcing FAILS** — + firmware `Access Denied`, no boot; leg 4 the surgery is surgical (only the loader + regenerated El + Torito artifacts differ; the full install payload is byte-identical). `bash -n` + shellcheck clean. + Physical N100 closure folds into the supervised rehearsal (an `n100-safety` match-nothing ISO built + + sha-recorded, unbooted, for an optional zero-risk pre-flight). + ## felhom-tenantsync.sh v1.1.0 — deprovision op (customer RESET teardown) (2026-07-17) Adds the `{"op":"deprovision","customer_id":""}` op the slice-1 header explicitly reserved diff --git a/scripts/iso/Dockerfile.assistant b/scripts/iso/Dockerfile.assistant index 9678347..656b49e 100644 --- a/scripts/iso/Dockerfile.assistant +++ b/scripts/iso/Dockerfile.assistant @@ -9,8 +9,13 @@ # Build: docker build -f scripts/iso/Dockerfile.assistant -t felhom-iso-assistant:trixie scripts/iso FROM debian:trixie +# xorriso: ISO (re-)mastering. shellcheck: lint. grub-mkimage (+ x86_64-efi module dir) and mtools: +# the slice-B mkimage loader surgery — build a monolithic UEFI loader from the ISO's own GRUB modules +# and inject it into the efi.img ESP (FAT, via mtools). grub 2.12 on trixie == the PVE 9.x ISO's +# 2.12-9+pmx2 generation, so the tool matches the modules it embeds. RUN apt-get update && apt-get install -y --no-install-recommends \ ca-certificates wget gnupg xorriso shellcheck \ + grub-common grub-efi-amd64-bin mtools dosfstools \ && rm -rf /var/lib/apt/lists/* # PVE 9.x (trixie) no-subscription repo — pairs the assistant to the 9.2 ISO generation. diff --git a/scripts/iso/README.md b/scripts/iso/README.md index a88a55c..e4b64cd 100644 --- a/scripts/iso/README.md +++ b/scripts/iso/README.md @@ -1,4 +1,4 @@ -# Felhom bare-metal ISO pipeline (R-21 slice A) +# Felhom bare-metal ISO pipeline (R-21 slice A + B) Turns the official Proxmox VE ISO into a **Felhom auto-install ISO** whose first-boot chain, unattended, fetches and runs `felhom-host-install.sh` until the host is enrolled and a guest is @@ -18,7 +18,8 @@ channel). | File | Role | |---|---| | `Dockerfile.assistant` | build host: Debian trixie + `proxmox-auto-install-assistant` (paired to the ISO's PVE generation by Debian codename) + `xorriso` + `shellcheck` | -| `build-felhom-iso.sh` | the pipeline: render answer, mint throwaway root hash, **validate-answer output-parse gate**, render stub, `prepare-iso`, emit ISO + sha256 + manifest | +| `build-felhom-iso.sh` | the pipeline: render answer, mint throwaway root hash, **validate-answer output-parse gate**, render stub, `prepare-iso`, **optional `--loader mkimage` surgery**, emit ISO + sha256 + manifest | +| `mkimage-surgery.sh` | slice-B loader surgery (runs in the container, post-`prepare-iso`): builds a monolithic `grub-mkimage` UEFI loader and swaps it into the ISO's EFI boot path (F1 firmware fix) | | `answer.toml.tmpl` | Proxmox answer template (`__FQDN__` / `__ROOT_HASH__` / `__ROOT_SSH_KEYS__` / `__DISK_SETUP__` placeholders; `[first-boot]` from-iso/fully-up) | | `profiles/*.profile` | disk-selection + fqdn fragments (sourceable, no secret) | | `stub-first-boot.sh` | the ONE first-boot executable (skeleton; build injects the bootstrap script/unit/env as base64) | @@ -66,6 +67,33 @@ glob. **A filter matching nothing, or a non-existent disk, fails-safe**: the ins 1, no disk touched) rather than installing on the wrong disk (spike S5c). Never commit a real key in `FELHOM_ROOT_SSH_KEY` unless it is meant to ship in every ISO built from that profile. +A profile may also set `FELHOM_LOADER="shim"|"mkimage"` (see below); `--loader` on the command line +overrides it. + +## Loader modes (`--loader shim|mkimage`, slice B) + +The UEFI boot loader is selectable. Default **shim**; a profile can pin `FELHOM_LOADER`; `--loader` +wins over both. + +- **`shim`** (default) — the stock output: the ISO's MS-signed shim → GRUB chain. **Secure Boot keeps + working** on compliant firmware (no MOK, no keypress — spike S2b). Use this everywhere it boots. +- **`mkimage`** — replaces the ISO's UEFI boot path with a **monolithic `grub-mkimage` loader** built + from the ISO's own GRUB modules. This is the fix for cheap AMI (`AN3PLUS 0.01`-class) mini-PC + firmware that **cannot UEFI-boot the ISO's stock GRUB from USB** (`relocation 0x0 is not implemented + yet`; SB-off and shim-bypass don't help) — the F1 finding of `tests/VALIDATION-n100-baremetal-2026-07-16.md`, + worked around live during that run and now first-class. The recipe (module set from the ISO's + `grub.cfg`, embedded `search --fs-uuid` → `configfile` the real menu) is reproduced from the run + evidence, not re-derived. + + > **⚠ mkimage is UNSIGNED → the target board MUST have Secure Boot OFF.** With Secure Boot + > enforcing the firmware refuses the loader (`Access Denied`, no boot — proven, RUNBOOK-B leg 3). + > mkimage ISOs are named with a **`-mkimage`** suffix and the manifest's `loader:` line says so; + > never confuse them with a shim (SB-capable) build. shim mode's output is byte-for-byte the same as + > before this flag existed (the surgery is skipped entirely). + + The surgery runs **after** `prepare-iso` and touches only the EFI boot path — the answer file and + the on-first-boot payload are provably byte-identical to the shim variant's (RUNBOOK-B leg 4). + ## bootstrap-env (SECRET-BEARING) The `--bootstrap-env` file becomes the in-ISO `/etc/felhom/bootstrap.env`: @@ -104,17 +132,25 @@ producer steps re-run each pass). ## N100 profile (for RUNBOOK B) -The nested profile uses `disk-list=["sda"]`. **A real single-disk mini-PC should use a unique-serial -udev filter, not `disk-list`:** +Use the committed `profiles/n100.profile` template (cheap-board, `FELHOM_LOADER="mkimage"`). **A real +single-disk mini-PC uses a unique-serial udev filter, not `disk-list`:** 1. Boot once (or use a live env), harvest the target disk's udev id: `udevadm info --query=property --name=/dev/sda | grep -E 'ID_SERIAL_SHORT|ID_MODEL|ID_WWN'`. -2. Write a profile: `filter.ID_SERIAL_SHORT = ""` + `filter-match = "all"`. A +2. In the profile, set `filter.ID_SERIAL_SHORT = ""` + `filter-match = "all"`. A match-nothing filter fails-safe (won't wipe the wrong disk). 3. **Prior-LVM prerequisite:** the auto-installer does **not** force-clear a pre-existing LVM PV signature — a disk with a previous Linux/Proxmox install aborts at "cannot init physical volume" (spike S2b). Wipe first: `blkdiscard -f /dev/` (or `wipefs -a` + zero the PV region). -4. UEFI + Secure Boot: no caveat — installs and boots under SB enforcing, no MOK/keypress (spike S2b). +4. **Boot loader + Secure Boot (F1):** cheap AMI (`AN3PLUS`-class) firmware can't boot the stock GRUB + from USB, so `profiles/n100.profile` uses `--loader mkimage`. **That loader is unsigned → set + Secure Boot to OFF/Disabled in the BIOS** before booting the stick (with SB enforcing the board + refuses it: `Access Denied`, no boot). Firmware that boots the stock ISO fine needs no mkimage and + keeps Secure Boot working (default `shim`, spike S2b). *Optional pre-flight (operator, no risk):* a + `n100-safety` ISO — `--loader mkimage` + a **match-nothing** filter — boots the board, loads GRUB, + and the installer aborts touching nothing (S5c). It confirms the loader + SB-off setting before the + real install; its sha is recorded at build. Never boot the real install-armed ISO except in the + supervised rehearsal. 5. **BIOS "State After G3" → "Power On" (or "Last State")** before final sign-off — so the mini-PC auto-powers-on after a mains loss (a managed appliance must come back without a physical press). Set it in the BIOS power/ACPI menu; it is board-specific and cannot be scripted (operator step). diff --git a/scripts/iso/build-felhom-iso.sh b/scripts/iso/build-felhom-iso.sh index 4032762..4e18674 100644 --- a/scripts/iso/build-felhom-iso.sh +++ b/scripts/iso/build-felhom-iso.sh @@ -1,6 +1,6 @@ #!/bin/bash #=============================================================================== -# build-felhom-iso.sh — R-21 slice A: turn the official PVE ISO into a Felhom auto-install ISO. +# build-felhom-iso.sh — R-21 slice A+B: turn the official PVE ISO into a Felhom auto-install ISO. # # Renders answer.toml (from answer.toml.tmpl + a profile), mints a fresh THROWAWAY root hash, # gates the answer through validate-answer by PARSING ITS OUTPUT (never $? — validate-answer returns @@ -8,15 +8,26 @@ # script/unit/env), and runs prepare-iso --fetch-from iso --on-first-boot. Emits the ISO + sha256 + # a build manifest. # +# SLICE B — --loader shim|mkimage (default shim). shim = the stock output (MS-signed shim→GRUB chain, +# keeps Secure Boot working on compliant firmware, spike S2b). mkimage = replace the ISO's UEFI boot +# path with a monolithic grub-mkimage-built BOOTX64.EFI built from the ISO's OWN GRUB modules — the +# workaround PROVEN LIVE during the N100 run (VALIDATION-n100-baremetal F1: cheap AMI AN3PLUS-class +# firmware can't relocate the ISO's signed GRUB from USB, `relocation 0x0`). The mkimage loader is +# UNSIGNED → the target board MUST have Secure Boot OFF (documented in the n100 profile's prep). The +# loader surgery runs AFTER prepare-iso (the assistant's answer/first-boot payload is provably +# untouched except the loader path). +# # SECRET-BEARING: if the bootstrap-env carries a retrieval passphrase (it must, for an unattended # install — see README "secret-bearing"), the produced ISO embeds it. Supervised/single-use only; # never distributed; delete after the run. The build log says so loudly. # -# Runs on DooPlex; delegates validate-answer + prepare-iso to the felhom-iso-assistant container. +# Runs on DooPlex; delegates validate-answer + prepare-iso + the mkimage surgery to the +# felhom-iso-assistant container (which carries proxmox-auto-install-assistant, xorriso, grub-mkimage, +# and mtools). #=============================================================================== set -euo pipefail -ISO_VERSION="1.16.0" # Felhom release the ISO is tagged to (aligns with felhom-host-install SCRIPT_VERSION). +ISO_VERSION="1.18.0" # Felhom release the ISO is tagged to (aligns with felhom-host-install SCRIPT_VERSION). IMAGE="${FELHOM_ISO_ASSISTANT_IMAGE:-felhom-iso-assistant:trixie}" HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -32,6 +43,7 @@ log_dry() { echo -e "${CYAN}[DRY-RUN]${NC} $1"; } die() { log_error "$1"; exit 1; } PVE_ISO=""; ISO_SHA256=""; PROFILE=""; BOOTSTRAP_ENV=""; OUT_DIR="${HOME}/felhom-iso/out"; PVE_VERSION=""; DRY_RUN=false +LOADER_CLI="" # --loader override; empty = fall back to the profile, then the shim default. usage() { cat < the target board needs Secure Boot OFF. --out DIR output directory (default: \$HOME/felhom-iso/out) --pve-version VER override PVE version tag (default: parsed from the ISO filename) --dry-run print the steps without producing an ISO @@ -55,6 +71,7 @@ while [[ $# -gt 0 ]]; do --iso-sha256) ISO_SHA256="$2"; shift 2 ;; --profile) PROFILE="$2"; shift 2 ;; --bootstrap-env) BOOTSTRAP_ENV="$2"; shift 2 ;; + --loader) LOADER_CLI="$2"; shift 2 ;; --out) OUT_DIR="$2"; shift 2 ;; --pve-version) PVE_VERSION="$2"; shift 2 ;; --dry-run) DRY_RUN=true; shift ;; @@ -62,6 +79,8 @@ while [[ $# -gt 0 ]]; do *) die "unknown argument: $1 (see --help)" ;; esac done +[[ -z "$LOADER_CLI" || "$LOADER_CLI" == "shim" || "$LOADER_CLI" == "mkimage" ]] \ + || die "--loader must be 'shim' or 'mkimage' (got '$LOADER_CLI')" [[ -n "$PVE_ISO" ]] || die "--pve-iso is required" [[ -n "$ISO_SHA256" ]] || die "--iso-sha256 is required" @@ -88,7 +107,7 @@ PROFILE_NAME="$(basename "$PROFILE")"; PROFILE_NAME="${PROFILE_NAME%.profile}" # --- load + validate profile ---------------------------------------------------------------------- log_step "loading profile: $PROFILE" -FELHOM_FQDN=""; FELHOM_DISK_SETUP=""; FELHOM_ROOT_SSH_KEY="" +FELHOM_FQDN=""; FELHOM_DISK_SETUP=""; FELHOM_ROOT_SSH_KEY=""; FELHOM_LOADER="" # shellcheck disable=SC1090 source "$PROFILE" [[ -n "$FELHOM_FQDN" ]] || die "profile missing FELHOM_FQDN" @@ -97,6 +116,20 @@ source "$PROFILE" ROOT_SSH_LINE="" [[ -n "$FELHOM_ROOT_SSH_KEY" ]] && ROOT_SSH_LINE="root-ssh-keys = [\"${FELHOM_ROOT_SSH_KEY}\"]" +# --- resolve the loader mode: --loader wins over the profile's FELHOM_LOADER over the shim default --- +LOADER="${LOADER_CLI:-${FELHOM_LOADER:-shim}}" +[[ "$LOADER" == "shim" || "$LOADER" == "mkimage" ]] \ + || die "profile FELHOM_LOADER must be 'shim' or 'mkimage' (got '$FELHOM_LOADER')" +if [[ "$LOADER" == "mkimage" ]]; then + echo -e "${YELLOW}==================================================================================${NC}" + log_warn "LOADER MODE = mkimage — the UEFI boot path is a monolithic grub-mkimage loader (F1 fix)." + log_warn "This loader is UNSIGNED: the target board MUST have Secure Boot OFF. shim/SB is bypassed." + log_warn "The ISO filename gains '-mkimage'; never confuse it with a shim (SB-capable) build." + echo -e "${YELLOW}==================================================================================${NC}" +else + log_info "loader mode = shim (stock MS-signed chain; Secure Boot works on compliant firmware)" +fi + # --- validate bootstrap-env (secret-bearing detection) -------------------------------------------- log_step "checking bootstrap-env (secret-bearing detection)" ( set +e @@ -112,7 +145,8 @@ log_warn "this ISO will be SECRET-BEARING (embeds the customer retrieval passphr # --- workspace ------------------------------------------------------------------------------------ WORK="$(mktemp -d "${TMPDIR:-/tmp}/felhom-iso.XXXXXX")" -cleanup() { rm -rf "$WORK"; } +# chmod first: the mkimage surgery's osirrox extract leaves read-only file modes that rm can't clear. +cleanup() { chmod -R u+w "$WORK" 2>/dev/null || true; rm -rf "$WORK"; } trap cleanup EXIT mkdir -p "$OUT_DIR" "$WORK/tmp" ISO_DIR="$(cd "$(dirname "$PVE_ISO")" && pwd)"; ISO_BASE="$(basename "$PVE_ISO")" @@ -174,10 +208,14 @@ chmod 0755 "$STUB" grep -q '@@BOOTSTRAP_.*_B64@@' "$STUB" && die "stub still has unfilled markers — injection failed" # --- prepare-iso ---------------------------------------------------------------------------------- -OUT_ISO="$OUT_DIR/felhom-pve-${PVE_VERSION}-v${ISO_VERSION}-${PROFILE_NAME}.iso" +# Rule 4: the loader mode is loud in the filename — a '-mkimage' ISO implies Secure-Boot-off prep. +LOADER_SUFFIX=""; [[ "$LOADER" != "shim" ]] && LOADER_SUFFIX="-${LOADER}" +OUT_ISO="$OUT_DIR/felhom-pve-${PVE_VERSION}-v${ISO_VERSION}-${PROFILE_NAME}${LOADER_SUFFIX}.iso" +GRUB_VERSION="" # populated by the mkimage surgery (the grub-mkimage build used) log_step "building ISO: $(basename "$OUT_ISO")" if $DRY_RUN; then log_dry "docker run … prepare-iso /iso/$ISO_BASE --fetch-from iso --answer-file /work/answer.toml --on-first-boot /work/stub-first-boot.sh --output /work/out.iso" + [[ "$LOADER" == "mkimage" ]] && log_dry "docker run … (mkimage surgery) grub-mkimage from the ISO's own modules → swap BOOTX64.EFI in the EFI tree + efi.img → xorriso re-master → /work/final.iso" log_info "DRY-RUN: no ISO produced" exit 0 fi @@ -187,15 +225,35 @@ docker run --rm -v "$ISO_DIR":/iso:ro -v "$WORK":/work "$IMAGE" \ --on-first-boot /work/stub-first-boot.sh \ --tmp /work/tmp --output /work/out.iso [[ -f "$WORK/out.iso" ]] || die "prepare-iso produced no output" -cp "$WORK/out.iso" "$OUT_ISO" + +# --- SLICE B: mkimage loader surgery (post-prepare; the assistant payload is untouched but the EFI +# boot path). The recipe is the N100 run's proven workaround (VALIDATION F1) — do NOT re-derive it: +# grub-mkimage from the ISO's OWN x86_64-efi modules (2.12-9+pmx2), embedding the module set the +# ISO's grub.cfg needs + an embedded config that `search --fs-uuid`es the ISO and `configfile`s its +# real menu; then swap BOOTX64.EFI in the ISO9660 EFI/BOOT tree AND inside efi.img, and re-master +# with xorriso preserving BOTH the hybrid BIOS boot and the injected answer/first-boot payload. --- +if [[ "$LOADER" == "mkimage" ]]; then + log_step "applying mkimage UEFI loader (F1 firmware fix; recipe from the N100 run evidence)" + [[ -f "$HERE/mkimage-surgery.sh" ]] || die "mkimage-surgery.sh not found next to build-felhom-iso.sh" + cp "$HERE/mkimage-surgery.sh" "$WORK/mkimage-surgery.sh" + docker run --rm -v "$WORK":/work "$IMAGE" bash /work/mkimage-surgery.sh 2>&1 | sed 's/^/ [surgery] /' + [[ -f "$WORK/final.iso" ]] || die "mkimage surgery produced no output (see [surgery] log above)" + GRUB_VERSION="$(cat "$WORK/grub-version.txt" 2>/dev/null || echo unknown)" + cp "$WORK/final.iso" "$OUT_ISO" + log_success "mkimage loader applied (grub-mkimage: ${GRUB_VERSION})" +else + cp "$WORK/out.iso" "$OUT_ISO" +fi # --- sha256 + manifest ---------------------------------------------------------------------------- OUT_SHA="$(sha256sum "$OUT_ISO" | awk '{print $1}')" OUT_SIZE="$(stat -c '%s' "$OUT_ISO")" ASSISTANT_VER="$(docker run --rm "$IMAGE" proxmox-auto-install-assistant --version 2>&1 | head -1)" echo "$OUT_SHA $(basename "$OUT_ISO")" > "$OUT_ISO.sha256" +LOADER_NOTE="shim (stock MS-signed chain; Secure Boot OK on compliant firmware)" +[[ "$LOADER" == "mkimage" ]] && LOADER_NOTE="mkimage (monolithic grub-mkimage UEFI loader, F1 fix — UNSIGNED; target board MUST have Secure Boot OFF)" cat > "$OUT_ISO.manifest.txt" </dev/null || echo 'https://felhom.eu/scripts/felhom-host-install.sh (default)') secret-bearing : ${SECRET_BEARING} (embeds the customer retrieval passphrase — supervised/single-use, delete after the run) output : $(basename "$OUT_ISO") diff --git a/scripts/iso/mkimage-surgery.sh b/scripts/iso/mkimage-surgery.sh new file mode 100644 index 0000000..f9e19f5 --- /dev/null +++ b/scripts/iso/mkimage-surgery.sh @@ -0,0 +1,168 @@ +#!/bin/bash +#=============================================================================== +# mkimage-surgery.sh — R-21 slice B: replace a prepared Felhom ISO's UEFI boot loader with a +# monolithic grub-mkimage image, so cheap AMI (AN3PLUS-class) firmware that can't relocate the ISO's +# signed GRUB from USB (VALIDATION-n100-baremetal F1, `relocation 0x0`) boots. +# +# RUNS INSIDE the felhom-iso-assistant container; operates on /work/out.iso (the prepare-iso output) +# and writes /work/final.iso + /work/grub-version.txt. NEVER touches the source ISO or the assistant's +# answer/first-boot payload — only the EFI boot path. +# +# The recipe is the N100 run's PROVEN-LIVE workaround (do NOT re-derive it): build BOOTX64.EFI from +# the ISO's OWN x86_64-efi GRUB modules (the box's working 2.12-9+pmx2 build), embedding the module +# set the ISO's grub.cfg needs + an embedded config that `search --fs-uuid`es the ISO and +# `configfile`s its real menu; swap it into the ISO9660 EFI/BOOT tree AND inside efi.img; re-master +# with xorriso preserving BOTH the hybrid BIOS boot and the payload. The image is UNSIGNED (Secure +# Boot must be OFF on the target) — that is the documented mkimage contract. +#=============================================================================== +set -euo pipefail + +OUT=/work/out.iso +FINAL=/work/final.iso +EX=/work/ex +EMB=/work/embedded.cfg +LDR=/work/BOOTX64.EFI + +say() { echo "mkimage-surgery: $*"; } + +# The osirrox extract tree is written by container-root; the host-side build cleanup (a non-root user) +# cannot remove it. Remove it here (we ARE root in the container) on every exit path so no /tmp litter +# survives the run. +cleanup_ex() { if [[ -n "${EX:-}" && -e "$EX" ]]; then rm -rf "$EX" 2>/dev/null || true; fi; } +trap cleanup_ex EXIT + +[[ -f "$OUT" ]] || { echo "mkimage-surgery: /work/out.iso missing" >&2; exit 2; } +for t in xorriso grub-mkimage mcopy mdir; do + command -v "$t" >/dev/null || { echo "mkimage-surgery: missing tool: $t" >&2; exit 2; } +done + +grub-mkimage --version | head -1 > /work/grub-version.txt +say "grub: $(cat /work/grub-version.txt)" + +# --- 1. extract the full prepared ISO tree (osirrox) — preserves the answer + first-boot payload. +# osirrox reproduces the ISO's (read-only) file modes, so make the tree writable afterwards or +# the loader swap (step 7) and the workspace cleanup can't overwrite/remove the files. --------- +[[ -e "$EX" ]] && { chmod -R u+w "$EX" 2>/dev/null || true; rm -rf "$EX"; } +mkdir -p "$EX" +xorriso -osirrox on -indev "$OUT" -extract / "$EX" >/dev/null 2>&1 +chmod -R u+w "$EX" +say "extracted prepared ISO tree" + +# --- 2. GRUB build to assemble the loader from. The N100 fix used the box's OWN INSTALLED 2.12 GRUB +# (a DIFFERENT, working build than the ISO's problem one — which is the whole point). The ISO +# ships modules but NOT kernel.img, so grub-mkimage cannot use the ISO's module dir directly; +# the box used its /usr/lib/grub/x86_64-efi. The container mirrors that: grub 2.12 == the PVE +# 9.x ISO's 2.12-9+pmx2 generation. We take the module BINARIES from here and the module LIST +# from the ISO's own grub.cfg (so we embed exactly what the ISO menu needs). -------------------- +GDIR="" +for d in /usr/lib/grub/x86_64-efi /usr/lib/grub/x86_64-efi-signed; do + [[ -f "$d/kernel.img" ]] && { GDIR="$d"; break; } +done +[[ -n "$GDIR" ]] || { echo "mkimage-surgery: no usable GRUB x86_64-efi build (kernel.img) in the container" >&2; exit 3; } +say "grub module source: $GDIR" +GCFG="" +for c in "$EX/boot/grub/grub.cfg" "$EX/boot/grub/x86_64-efi/grub.cfg"; do + [[ -f "$c" ]] && { GCFG="$c"; break; } +done + +# --- 3. module list: the base set the search/configfile-from-USB chain needs, PLUS every module the +# ISO's grub.cfg insmod's. Keep only modules present in the GRUB build. This IS "every module +# the grub.cfg needs". +BASE="part_gpt part_msdos msdospart fat exfat iso9660 udf search search_fs_uuid search_fs_file search_label \ +configfile normal boot linux linuxefi chain loadenv loopback echo test true cat ls help \ +all_video efi_gop efi_uga video video_fb font gfxterm gfxterm_background gfxmenu png jpeg \ +terminal serial gzio xzio lzopio minicmd reboot halt probe regexp sleep keystatus read" +CFGMODS="" +[[ -n "$GCFG" ]] && CFGMODS=$(grep -hoE 'insmod[[:space:]]+[a-zA-Z0-9_]+' "$GCFG" | awk '{print $2}' | sort -u) +MODS="" +for m in $BASE $CFGMODS; do + [[ -f "$GDIR/$m.mod" ]] && MODS="$MODS $m" +done +MODS="$(echo "$MODS" | tr ' ' '\n' | awk 'NF' | sort -u | tr '\n' ' ')" +say "embedding $(echo "$MODS" | wc -w) modules from the ISO's own x86_64-efi build" + +# --- 4. pin the volume modification-date so the ISO's GRUB fs-uuid is DETERMINISTIC and KNOWN before +# we build the loader (GRUB's iso9660 fs_uuid is derived from the PVD modification timestamp). +# Reuse the prepared ISO's own timestamp verbatim -> the embedded search matches the re-mastered +# image (we pin the same value on re-master in step 8). +MDATE="$(xorriso -indev "$OUT" -report_el_torito as_mkisofs 2>/dev/null \ + | grep -oE "modification-date='[0-9]+'" | grep -oE '[0-9]+' | head -1)" +[[ -n "$MDATE" && ${#MDATE} -ge 14 ]] || { echo "mkimage-surgery: could not read the ISO modification-date" >&2; exit 4; } +ISO_UUID="$(echo "${MDATE:0:16}" | sed -E 's/^(.{4})(.{2})(.{2})(.{2})(.{2})(.{2})(.{2})$/\1-\2-\3-\4-\5-\6-\7/')" +say "ISO fs-uuid (from modification-date $MDATE): $ISO_UUID" + +# --- 5. embedded config: find the ISO by fs-uuid, then chain its real menu (the recorded recipe) --- +cat > "$EMB" <&2; exit 5; } +say "built BOOTX64.EFI ($(stat -c%s "$LDR") bytes)" + +# --- 7. swap the loader into BOTH the ISO9660 EFI/BOOT tree AND inside the efi.img ESP. The ISO tree +# uses Rock Ridge (LOWERCASE) names — `/efi/boot/bootx64.efi` — so overwrite the EXISTING files +# in place (case-insensitive find), NEVER mkdir a spurious uppercase path. The efi.img ESP is +# FAT (case-insensitive), the authoritative loader UEFI firmware actually runs from USB. -------- +TREE_HITS=0 +while IFS= read -r f; do cp "$LDR" "$f"; TREE_HITS=$((TREE_HITS+1)); done \ + < <(find "$EX" -ipath '*/efi/boot/bootx64.efi') +while IFS= read -r f; do cp "$LDR" "$f"; done \ + < <(find "$EX" -ipath '*/efi/boot/grubx64.efi') +[[ "$TREE_HITS" -ge 1 ]] || { echo "mkimage-surgery: no bootx64.efi found in the ISO9660 tree to replace" >&2; exit 6; } + +EFIIMG="$EX/efi.img" +[[ -f "$EFIIMG" ]] || EFIIMG="$(find "$EX" -maxdepth 3 -iname 'efi*.img' | head -1)" +[[ -f "$EFIIMG" ]] || { echo "mkimage-surgery: efi.img ESP not found in the ISO tree" >&2; exit 6; } +# FAT is case-insensitive: ::/EFI/BOOT/BOOTX64.EFI resolves the real loader regardless of stored case. +mcopy -i "$EFIIMG" -o "$LDR" ::/EFI/BOOT/BOOTX64.EFI +if mdir -i "$EFIIMG" ::/EFI/BOOT 2>/dev/null | grep -qi grubx64; then + mcopy -i "$EFIIMG" -o "$LDR" ::/EFI/BOOT/grubx64.efi +fi +say "swapped bootx64.efi in the ISO tree ($TREE_HITS) and inside $(basename "$EFIIMG")" + +# --- 8. re-master from the (modified) tree, reproducing the source ISO's boot geometry from its OWN +# as_mkisofs report so we track PVE minor versions. We drop ONLY the Apple APM/HFS+ boot map +# (-hfsplus / -apm-block-size): re-emitting it trips xorriso 1.5.6's "Overlapping MBR partition +# entries" on THIS layout, and Mac boot is irrelevant for N100/PC hardware. We KEEP the hybrid +# BIOS grub2-mbr + El Torito (BIOS eltorito.img + UEFI /efi.img) + the GPT EFI System Partition +# (-efi-boot-part) that USB UEFI firmware boots from — the whole point of this fix. The volume +# id + modification-date are pinned explicitly so the embedded fs-uuid stays valid. ------------- +RPT="$(xorriso -indev "$OUT" -report_el_torito as_mkisofs 2>/dev/null)" +VOLID="$(printf '%s\n' "$RPT" | sed -nE "s/^-V '(.*)'\$/\\1/p" | head -1)"; [[ -n "$VOLID" ]] || VOLID="PVE" +# Drop, then re-add explicitly: the volume id + modification-date. Drop entirely: the Apple APM/HFS+ +# map (-hfsplus / -apm-block-size) AND the isohybrid GPT-basdat marking (-part_like_isohybrid / +# -isohybrid-gpt-basdat) — prepare-iso re-masters with these, and re-emitting them alongside +# -efi-boot-part + the protective MBR trips xorriso 1.5.6's "Overlapping MBR partition entries". The +# resulting image keeps the protective MBR + grub2-mbr (BIOS) + El Torito (BIOS+UEFI) + the GPT EFI +# System Partition (verified). Repoint the grub2-mbr template at the in-container out.iso. +FILTERED="$(printf '%s\n' "$RPT" \ + | grep -vE "^-V '|^--modification-date=|^-apm-block-size |^-hfsplus\$|^-part_like_isohybrid\$|^-isohybrid-gpt-basdat\$" \ + | sed -E "s#(--interval:[^']*:)'[^']*'#\\1'$OUT'#")" +rm -f "$FINAL" +# shellcheck disable=SC2086 +eval xorriso -as mkisofs -V "'$VOLID'" --modification-date="'$MDATE'" \ + $FILTERED -o "$FINAL" "$EX" >/work/xorriso.log 2>&1 \ + || { echo "mkimage-surgery: xorriso re-master FAILED"; tail -25 /work/xorriso.log >&2; exit 7; } + +[[ -f "$FINAL" ]] || { echo "mkimage-surgery: no final.iso produced" >&2; exit 7; } +# assert both boot images survived (BIOS eltorito.img + UEFI efi.img) — a silent loss would fail-safe +# to an unbootable stick, so gate it here. +ETIMG="$(xorriso -indev "$FINAL" -report_el_torito plain 2>/dev/null | grep -cE 'El Torito boot img')" +[[ "$ETIMG" -ge 2 ]] || { echo "mkimage-surgery: re-master lost a boot image (El Torito entries=$ETIMG, want >=2)" >&2; exit 8; } +say "re-mastered final.iso ($(stat -c%s "$FINAL") bytes; El Torito boot images=$ETIMG)" + +# --- 9. verify the re-mastered image kept the modification-date (so the embedded fs-uuid matches) --- +FINAL_MDATE="$(xorriso -indev "$FINAL" -report_el_torito as_mkisofs 2>/dev/null \ + | grep -oE "modification-date='[0-9]+'" | grep -oE '[0-9]+' | head -1)" +if [[ "${FINAL_MDATE:0:14}" != "${MDATE:0:14}" ]]; then + echo "mkimage-surgery: WARN final modification-date ($FINAL_MDATE) != source ($MDATE) — the search fs-uuid may not match; re-check" >&2 +else + say "fs-uuid preserved ($ISO_UUID)" +fi +say "done" diff --git a/scripts/iso/profiles/n100.profile b/scripts/iso/profiles/n100.profile new file mode 100644 index 0000000..2adb180 --- /dev/null +++ b/scripts/iso/profiles/n100.profile @@ -0,0 +1,27 @@ +# Felhom ISO build profile — n100 / cheap-board (R-21 slice B). +# +# The canonical profile for cheap mini-PCs whose early AMI (AN3PLUS-class) UEFI firmware CANNOT boot +# the ISO's stock signed GRUB from USB (VALIDATION-n100-baremetal F1, `relocation 0x0`). It flips the +# loader to `mkimage` — a monolithic grub-mkimage UEFI loader built from the ISO's own GRUB, proven +# live on the N100 board. +# +# ⚠ SECURE BOOT MUST BE OFF on the target board. The mkimage loader is UNSIGNED; with Secure Boot +# enforcing the firmware refuses it ("Access Denied", no boot — proven, RUNBOOK-B leg 3). Compliant +# firmware should use the default `shim` loader (drop FELHOM_LOADER) to keep Secure Boot working. +# +# This template ships a MATCH-NOTHING disk filter as a safety default — REPLACE the serial with the +# target SSD's real ID_SERIAL_SHORT (harvested pre-wipe; see README "N100 profile") before a real +# install, or the installer fails-safe (aborts, touches no disk — spike S5c). + +FELHOM_FQDN="felhom-host.local" + +# mkimage loader (F1 firmware fix). Remove this line on Secure-Boot-capable/compliant firmware. +FELHOM_LOADER="mkimage" + +FELHOM_DISK_SETUP='[disk-setup] +filesystem = "ext4" +filter-match = "all" +filter.ID_SERIAL_SHORT = "REPLACE-WITH-TARGET-SSD-SERIAL"' + +# Optional emergency/validation key baked into root's authorized_keys (blank -> not baked): +# FELHOM_ROOT_SSH_KEY="ssh-ed25519 AAAA... ops@felhom"