docs(tests): N100 bare-metal physical validation run (R-21) — findings + current state
Supervised clean-slate reinstall of the demo N100 from a pipeline-built Felhom ISO + onboarding rehearsal. Core objectives GREEN: first-boot chain reached rc-0 on the first attempt on real hardware (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict -> key on MAC+UUID. Hard obstacle was firmware not the pipeline: this cheap AMI AN3PLUS 0.01 board won't UEFI-boot the ISO's GRUB from USB (relocation 0x0; SB-off + shim-bypass don't help), worked around live with a grub-mkimage loader from the box's own GRUB (slice-B input). 7 findings logged (F1 firmware loader HIGH; F2/F3/F4 reused-customer edges -> R-1/Peti; F5 guest-RAM, F6 drive-init mount, F7 back-route). Full doc in documentation/tests/; ROADMAP R-21 + REPORT + CONTEXT updated. No production code changed.
This commit is contained in:
+11
@@ -3,6 +3,17 @@
|
||||
> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
|
||||
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
|
||||
|
||||
- **2026-07-16 — R-21 PHYSICAL RUN on the demo N100 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`).**
|
||||
demo-felhom host reinstalled clean-slate from a pipeline-built ISO; first-boot chain reached
|
||||
**rc-0 on the first attempt on real hardware** (closes slice A's operator-gated boundary),
|
||||
serial-filter safety proven on metal (external-HDD canary byte-identical), PBS-DR reconciler
|
||||
self-healed on the reused peer. **The obstacle was firmware, not us:** the cheap AMI `AN3PLUS 0.01`
|
||||
board can't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`; SB-off + shim-bypass don't help) —
|
||||
worked around live by rebuilding the stick's loader with `grub-mkimage` from the box's own working
|
||||
GRUB. Findings (to tackle): F1 firmware boot loader (slice-B), F2 claim-re-issue on reinstall,
|
||||
F3 offsite-re-issue, F4 PBS-DR ACL on non-default storage id, F5 guest-RAM configurability,
|
||||
F6 drive-init doesn't mount+attach, F7 back-route. Box healthy, BookStack deployed, demo live.
|
||||
|
||||
- **2026-07-16 — R-21 SLICE A SHIPPED: bare-metal Felhom ISO pipeline (`scripts/iso/`, scripts
|
||||
v1.16.0).** A DooPlex pipeline (`build-felhom-iso.sh` + `Dockerfile.assistant`) turns the official
|
||||
PVE ISO into a Felhom auto-install ISO whose first-boot stub installs a retry-forever
|
||||
|
||||
@@ -2,125 +2,36 @@
|
||||
|
||||
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
|
||||
|
||||
## R-21 slice A: bare-metal Felhom ISO (build pipeline + first-boot bootstrap) — 2026-07-16
|
||||
## R-21 physical run — N100 bare-metal reinstall + onboarding rehearsal — 2026-07-16
|
||||
|
||||
**Repo:** felhom.eu · **Baseline:** `main` @ `914d51a` → scripts **v1.16.0** (ISO pipeline tag;
|
||||
`felhom-host-install` was *already* v1.16.0 from 2026-07-13 — see Versioning note).
|
||||
**Validation:** nested VM **310** on felhom-pve; ISO builds on DooPlex. `felhom-host-install.sh`
|
||||
**not modified** (read-only, invoked only). Hub code untouched.
|
||||
Supervised run (Viktor at the box, CC on Phase-0 prep + the SSH-side legs). Full detail:
|
||||
**`documentation/tests/VALIDATION-n100-baremetal-2026-07-16.md`**. No production code changed.
|
||||
|
||||
### Files created (`scripts/iso/`)
|
||||
**Core objectives — GREEN.** The demo N100 (felhom-pve, serving demo-felhom.eu) was reinstalled
|
||||
clean-slate from a pipeline-built ISO (`felhom-pve-9.2-1-v1.16.0-n100-demo.iso`, profile filter
|
||||
`ID_SERIAL_SHORT=QDF922W009654S30EX`, appliance). The first-boot chain reached **rc-0 on the first
|
||||
attempt (`NRestarts=0`)** on real hardware — enrolled `demo-felhom`, provisioned guest 9201, agent
|
||||
0.88.0, WG up, `bootstrap.env` shredded. This **closes slice A's operator-gated rc-0 boundary on
|
||||
metal.** Serial-filter safety proven on real hardware (external-HDD canary byte-identical). PBS-DR
|
||||
reconciler self-healed on the reused peer (all four steps done incl. escrow). DMI verdict on real
|
||||
cheap hardware: all "Default string" → slice-C keys on **MAC + UUID**.
|
||||
|
||||
`Dockerfile.assistant`, `build-felhom-iso.sh`, `answer.toml.tmpl`, `stub-first-boot.sh`,
|
||||
`felhom-bootstrap.sh`, `felhom-bootstrap.service`, `profiles/nested-vm.profile`,
|
||||
`profiles/nested-canary.profile`, `README.md`. Plus `scripts/CHANGELOG.md`,
|
||||
`documentation/backlog/ROADMAP.md` (R-21 → in-progress), this `REPORT.md`, `CONTEXT.md`.
|
||||
**The obstacle was firmware, not the pipeline.** This board's early AMI `AN3PLUS 0.01` UEFI cannot
|
||||
boot the ISO's GRUB **from USB** (`relocation 0x0 is not implemented yet`); Secure-Boot-off and
|
||||
shim-bypass did **not** help (it's GRUB, not shim). The identical ISO boots in reference UEFI, and the
|
||||
box's installed GRUB boots fine from the SSD — so it's the firmware relocating the ISO's USB-loaded
|
||||
GRUB. **Worked around live** by rebuilding the stick's `BOOTX64.EFI` with `grub-mkimage` from the box's
|
||||
own working `2.12-9+pmx2` GRUB (all needed modules embedded, config that searches+boots the ISO). This
|
||||
is the top **slice-B input**: the pipeline should ship a firmware-compatible loader and/or a PXE path.
|
||||
|
||||
### Green gates
|
||||
**Findings to tackle (7):** F1 firmware boot loader (HIGH, slice-B); F2 no claim-code re-issue on
|
||||
reinstall (+ verify open-vs-gated); F3 offsite creds need re-issue on reprovision; F4 PBS-DR 403 —
|
||||
install ACL grants only default `felhom-pbs`, not the config's `felhom-offsite` storage id; F5 guest
|
||||
RAM 2 GB not configurable; F6 drive-initialize formats but doesn't mount+attach (client-disconnect
|
||||
aborts post-mkfs, no UI feedback); F7 storage Back-route → /settings. F2/F3/F4 are reused-customer
|
||||
clean-slate edges feeding R-1/Peti. BookStack deployed + works.
|
||||
|
||||
`bash -n` clean on all three scripts; `shellcheck -x` **exit 0, no findings** (run in the assistant
|
||||
container). Retry-unit behavior de-risked on systemd 257 before wiring (transient unit: `Type=oneshot`
|
||||
+ `Restart=on-failure` **does** retry — NRestarts climbed).
|
||||
|
||||
### §4.4 finding — does the distribution channel embed a secret? (decides slice C)
|
||||
|
||||
**The channel: no. The unattended install: yes (the passphrase).** The hub install-command generator
|
||||
(`hub/internal/web/templates/customer_unified.html`, Option 1) emits
|
||||
`curl -fsSL https://felhom.eu/scripts/felhom-host-install.sh -o … && sudo bash … --customer-id <id>
|
||||
--mode <appliance|byo>` — a **public URL, no token/secret in the command**. But
|
||||
`felhom-host-install.sh` **requires the customer retrieval passphrase** (`read_passphrase` →
|
||||
`die "empty passphrase"`; used for config fetch + `POST /host-enroll`). Unattended ⇒ it must be
|
||||
supplied via `--passphrase-file`, so **the slice-A ISO is secret-bearing** (embeds the passphrase).
|
||||
**Slice-C implication:** the *distributed* ISO must NOT bake the passphrase — the claim/pairing model
|
||||
must deliver it (or the config) to the box *after* the customer claims it. Slice-A ISOs are
|
||||
supervised/single-use; the bootstrap `shred`s `bootstrap.env` on success.
|
||||
|
||||
### 2.3 retry-vs-resume ruling (source-verified, encoded once)
|
||||
|
||||
**Plain first attempt; add `--resume` on any later attempt that finds
|
||||
`/var/lib/felhom-install/state.json`.** Source landmark: `felhom-host-install.sh` header **v1.11.3**
|
||||
("--resume correctness — the producer steps (token/enroll/grows) run every pass … a resumed install no
|
||||
longer writes a config missing hub.host_id/proxmox.token") — so `--resume` is safe, whereas a plain
|
||||
re-invoke over an existing install state would re-hit the populated-host leaf guard / existing-vmid
|
||||
refusal. `--mode` is required in both forms. Encoded in `felhom-bootstrap.sh` with the landmark cited
|
||||
inline; **empirically confirmed** (below).
|
||||
|
||||
### Per-scenario results (evidence under `180:~/felhom-iso/evidence/`)
|
||||
|
||||
- **C — build gate blocks a broken answer (+ red-proof) — PASS.** Answer with both `root-password`
|
||||
and `root-password-hashed`: the output-parse gate surfaced `` `global.root-password` and
|
||||
`global.root-password-hashed` cannot be set at the same time / Error: Found issues `` and aborted
|
||||
(`BUILD-EXIT=1`, **no ISO** — count unchanged), *even though `validate-answer` exits 0*.
|
||||
**Red-proof:** gutting the gate to trust `$?` made it wrongly report `[OK] answer validated`; the
|
||||
broken answer was then stopped only by `prepare-iso`'s own error **plus the build's `[[ -f out.iso ]]`
|
||||
guard** — notably `prepare-iso` *also* exited 0 on the error, so the file-existence check was the
|
||||
real backstop. Reverted (mutation done on the DooPlex copy; committed tree never touched).
|
||||
- **D — disk-filter fail-safe through the pipeline — PASS.** Pipeline-built canary ISO
|
||||
(`filter.ID_SERIAL_SHORT="no-such-disk-zzz"`) → `ERROR: Installation failed: filter did not match any
|
||||
device`, abort exit 1, no interactive fallback. 8 GB canary disk **byte-identical** after
|
||||
(`sha256 -c: OK`, host-side; `scenarioD-filter-abort.png`).
|
||||
- **A — full unattended chain — MECHANISM PASS; terminal success operator-gated.** Zero keypress;
|
||||
stub ran once (uid 0) and enabled+started `felhom-bootstrap`; the unit **fetched host-install from the
|
||||
REAL channel** `https://felhom.eu/scripts/felhom-host-install.sh` and invoked it
|
||||
`--customer-id … --mode appliance --hub-url …`; host-install ran unattended through preflight
|
||||
(hub reachable) — with the drill customer absent it correctly failed at config preflight (`404`) and
|
||||
the unit **retried** (`bootstrap-journal.log`; NRestarts ≥3; 12 host-install invocations).
|
||||
**Not live-proven:** host-install **rc-0** → hub-enrolled + guest-provisioned → done-flag +
|
||||
unit-disable + no-re-run. Reason: that needs a properly created drill customer, and CC cannot use the
|
||||
password-gated create-UI while a direct write to the *live, locked* hub `hub.db` (verified open by the
|
||||
running pod) risks corruption. The done-flag/disable/env-scrub success path is code-reviewed +
|
||||
shellcheck-clean; empirical proof requires an operator-created drill customer (supervised).
|
||||
- **B — no network at first boot — PASS.** During a host-side `link_down` window
|
||||
(13:37:56→13:39:37) the chain failed repeatedly — `host-install FAILED rc=6` (host-install's own hub
|
||||
fetch, no net) and `ERROR: host-install fetch failed (no network yet?)` — and **the unit kept
|
||||
retrying (never died)**; on link-up it **resumed automatically** (`13:40:04` fetch+run). No
|
||||
"single failure → dead unit". The stub-runs-without-network property is spike-proven (S8a);
|
||||
`bootstrap-journal.log`.
|
||||
- **2.3 resume decision — PASS.** After creating `/var/lib/felhom-install/state.json`, the next attempt
|
||||
logged `prior install state present (…) -> adding --resume`; earlier attempts ran plain.
|
||||
- **Exactly-once stub — PASS.** Host-side reboot → `/var/log/felhom-first-boot.log` hash **unchanged**
|
||||
(`13cf3fe4…`), pending-flag consumed; the retry unit correctly persisted (still `activating`, not done).
|
||||
|
||||
### ISO artifacts (build ≈ 12 s each, source already local)
|
||||
|
||||
| ISO | sha256 | size |
|
||||
|---|---|---|
|
||||
| `felhom-pve-9.2-1-v1.16.0-nested-vm.iso` | `8cbe5eaeaa0d…82bb` | 1,707,737,088 B |
|
||||
| `felhom-pve-9.2-1-v1.16.0-nested-canary.iso` | `6b4568eaeac0…0301` | 1,707,737,088 B |
|
||||
| (validation build, key baked) `…-nested-vm-val.iso` | `0042acfdb70a…98e7` | — |
|
||||
|
||||
Source ISO `proxmox-ve_9.2-1.iso` sha256 `4e88fe4…f2c6c` (verified vs official SHA256SUMS). Assistant
|
||||
`proxmox-installer-common 9.2.7` (trixie no-sub). Artifacts kept under `180:~/felhom-iso/out/`, never
|
||||
committed (secret-bearing).
|
||||
|
||||
### VM 310 final state
|
||||
|
||||
Stopped, kept; SeaBIOS, scsi0 32 GB + scsi1 8 GB (canary intact), ISO detached. Boundaries honored:
|
||||
VM 300 (drill) running untouched, 9000 stopped, LXC 9201 (demo) running untouched, hub untouched.
|
||||
Secret-bearing pipeline ISOs removed from felhom-pve (their embedded passphrase was a placeholder, not a
|
||||
real credential).
|
||||
|
||||
### NOT-live-validated
|
||||
|
||||
- **Physical N100 run (RUNBOOK B):** real DMI/firmware, USB-boot media, vendor Secure Boot key set,
|
||||
real-NIC naming, prior-LVM `blkdiscard` on real hardware. Not started (out of scope by design).
|
||||
- **host-install rc-0 terminal success** (hub-enrolled + guest-provisioned + done-flag + disable) and
|
||||
the **2.3 `--resume` executed against a real partial install** — both need an operator-created drill
|
||||
customer (password-gated create-UI; live-DB insert unsafe). The *decisions/plumbing* are proven; only
|
||||
the terminal success is deferred.
|
||||
|
||||
### Versioning note
|
||||
|
||||
`felhom-host-install` reached v1.16.0 on 2026-07-13 (before this task's stated 1.15.x baseline). The
|
||||
new ISO pipeline is tagged **v1.16.0** to stay coherent with that release (the ISO's baked invocation
|
||||
targets host-install's flag surface). The `scripts/CHANGELOG.md` entry is a new component under the
|
||||
same release tag, not a bump of host-install.
|
||||
|
||||
### Observations (recorded, not acted on)
|
||||
|
||||
- `prepare-iso` (not just `validate-answer`) was observed to **exit 0 on a bad answer** — the build's
|
||||
`[[ -f out.iso ]]` output-existence guard is therefore load-bearing, not just belt-and-suspenders.
|
||||
- host-install prints `unexpected hub status 404404` (doubled) for an absent customer — cosmetic
|
||||
host-install quirk; left untouched (read-only this task).
|
||||
- host-install exits **rc=6** when it cannot reach the hub (no net) — the bootstrap correctly treats
|
||||
any non-zero rc as retry.
|
||||
**Current state:** box healthy, demo-felhom.eu live, guest 9201 running (controller/traefik/cloudflared/
|
||||
filebrowser/bookstack), external HDD reformatted + attached as `hdd_1`. Onboarding not finished (claim
|
||||
+ escrow pending on F2/F3/F4). Boot stick (secret-bearing, with the GRUB workaround) to be wiped at
|
||||
teardown. Findings will be worked next per Viktor.
|
||||
|
||||
@@ -49,7 +49,7 @@
|
||||
| R-13 | OOB management arc: dual-use existing WireGuard + hub desired-state channel as mutual-repair | L | idea | |
|
||||
| R-14 | Headscale/WireGuard spike: Minecraft/gaming port connectivity (CGNAT-proof, sovereign DERP fallback) | M | idea | |
|
||||
| R-15 | Multi-user dashboard accounts (household members, roles) | L | idea | Single password is a stated alpha limitation (R-11) |
|
||||
| R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → claim-code pairing (option C) | XL | **in-progress** (slice A shipped 2026-07-16 — `scripts/iso/`, scripts v1.16.0) | **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B** = physical N100 RUNBOOK (real DMI/firmware/USB/prior-LVM). **Slice C** = claim-pairing so the *distributed* ISO carries no retrieval passphrase (slice-A ISO is secret-bearing, supervised/single-use). Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* |
|
||||
| R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → claim-code pairing (option C) | XL | **in-progress** (slice A shipped 2026-07-16; physical run 2026-07-16) | **PHYSICAL RUN 2026-07-16 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`):** demo N100 reinstalled clean-slate from a pipeline ISO → chain reached **rc-0 first try on real hardware** (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. **F1 (HIGH, slice-B input):** this cheap AMI `AN3PLUS 0.01` firmware won't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`) — SB-off/shim-bypass don't help; worked around live with a `grub-mkimage` loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B** = physical N100 RUNBOOK (real DMI/firmware/USB/prior-LVM). **Slice C** = claim-pairing so the *distributed* ISO carries no retrieval passphrase (slice-A ISO is secret-bearing, supervised/single-use). Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* |
|
||||
|
||||
## Absorbed / superseded notes in this folder
|
||||
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
# VALIDATION — N100 bare-metal reinstall via Felhom ISO (R-21 physical run + onboarding rehearsal), 2026-07-16
|
||||
|
||||
> Supervised run: Viktor at the box + keyboard, CC driving Phase-0 prep and the SSH-side legs. The
|
||||
> demo host (felhom-pve / the N100 serving demo-felhom.eu) was reinstalled clean-slate from a
|
||||
> pipeline-built Felhom ISO, then walked through the friend-alpha onboarding path with Viktor as
|
||||
> customer zero. **No production code changed** — this document + the ISO build artifacts (kept on
|
||||
> DooPlex `~/felhom-iso/`, `~/n100-baremetal/`) are the outputs. Findings will be tackled separately.
|
||||
>
|
||||
> **Headline — read this first.** The R-21 pipeline itself was **flawless**: the ISO was correct, and
|
||||
> the full unattended chain (install → enroll → provision → escrow) reached **rc-0 on the first
|
||||
> attempt (zero retries)** — the exact terminal success that was operator-gated in slice A, now
|
||||
> **proven on real, customer-shaped hardware**. The *one hard obstacle* was **not** our software: this
|
||||
> cheap N100's early AMI firmware **cannot UEFI-boot GRUB from USB** (`relocation 0x0 is not
|
||||
> implemented yet`) — the single class of thing nested virt could never surface. We worked around it
|
||||
> live by rebuilding the stick's loader from the box's own known-good GRUB. Everything downstream
|
||||
> worked; the onboarding surfaced a crop of **reused-customer / clean-slate edges** (the Peti-pattern
|
||||
> data the run was designed to produce) plus a couple of UX bugs. Net: **core objectives green,
|
||||
> ~7 findings to work.**
|
||||
|
||||
---
|
||||
|
||||
## Run header
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Hardware | Intel **N100** (AlderLake-N), 16 GB; AMI **Aptio** BIOS `AN3PLUS 0.01` (build 2024-06-11), UEFI 2.8, no CSM/Legacy. **DMI all "Default string"** (mfr/product/serial/baseboard/chassis); system UUID `001E91E9-…` real |
|
||||
| Disks | internal SATA M.2 SSD **AirDisk 512 GB** serial `QDF922W009654S30EX` (target); external USB HDD **Toshiba MQ04ABF100** (ADATA HD710 PRO enclosure) serial `65NOP3HDT`, 1 TB — held the Felhom **secondary backups**; + a 64 GB SD card + the 119 GB Samsung boot stick |
|
||||
| Customer | `demo-felhom` (Demo Ügyfél), domain **demo-felhom.eu**, email admin@felhom.eu, config MANAGED, created 145 d ago (previously onboarded) |
|
||||
| ISO | `felhom-pve-9.2-1-v1.16.0-n100-demo.iso` (pipeline `scripts/iso/`, secret-bearing), profile: `filter.ID_SERIAL_SHORT="QDF922W009654S30EX"`, appliance mode, my ed25519 key baked (durable post-wipe access) |
|
||||
| Outcome | **SUCCESS** — clean-slate reinstall, host enrolled, guest 9201 provisioned, BookStack deployed, demo-felhom.eu live. Install wall-clock ≈ 3 min; first-boot → rc-0 ≈ a few min |
|
||||
|
||||
## What succeeded (the core objectives)
|
||||
|
||||
- **Pipeline correctness on metal.** The ISO built by `build-felhom-iso.sh` installed and ran the
|
||||
first-boot chain exactly as designed. `felhom-bootstrap` fetched `felhom-host-install.sh` from the
|
||||
public `felhom.eu/scripts/` channel and ran it to **rc-0 on the first try (`NRestarts=0`)**: done-flag
|
||||
written, unit disabled, `/etc/felhom/bootstrap.env` **shredded**. This **closes slice A's
|
||||
operator-gated boundary on real hardware.**
|
||||
- **Serial-filter safety — proven on real hardware.** The install targeted only the SSD; the external
|
||||
USB HDD's canary file was **byte-identical** before/after (`sha256 de6e00cb…`, host-verified). The
|
||||
HDD even moved device node (`sdd`→`sdb`) across the reinstall — irrelevant, because we filter by
|
||||
**serial**, which is exactly why it was safe. (Match-nothing / wrong-disk fail-safe was also proven
|
||||
through the pipeline earlier in slice A, Scenario D.)
|
||||
- **Auto-installer over a complete prior install.** It did **not** abort on the pre-existing LVM — it
|
||||
wiped and installed. The spike's "prior-LVM abort" (SPIKE-baremetal-iso §S2b) was an artifact of a
|
||||
*half*-wiped disk; a **clean, complete prior PVE install is simply overwritten.** (So the runbook's
|
||||
manual `blkdiscard`/STOP-2 step was not needed here.)
|
||||
- **PBS-DR reconciler self-healed on the reused peer.** Hub showed all four steps `done`: host
|
||||
enrolled (`demo-felhom-01`), WG tunnel peer registered, descriptor provisioned (namespace
|
||||
`demo-felhom`, token `felhom@pbs!demo-felhom`), **key escrow present / ceremony done** — the S8
|
||||
headline, first real firing on physical customer-shaped hardware.
|
||||
- **UEFI + Secure Boot** was disabled during troubleshooting; the installed system boots fine from the
|
||||
SSD. (The SB-enforcing path was proven in the spike on reference firmware.)
|
||||
- **DMI verdict for slice C — settled on real cheap hardware:** every DMI identity string is
|
||||
**"Default string"**; only the **system UUID** and **NIC MAC** are usable. Slice-C keying must be
|
||||
**MAC + UUID**, never DMI serials.
|
||||
|
||||
## The one hard obstacle — GRUB won't UEFI-boot from USB (firmware)
|
||||
|
||||
Booting the stick showed only:
|
||||
```
|
||||
relocation 0x0 is not implemented yet
|
||||
Aborted. Press any key to exit.
|
||||
```
|
||||
and fell through to the internal Proxmox. Diagnosis ladder (all still safe — pre-wipe):
|
||||
1. **Secure Boot disabled** → same error (so it is *not* an SB/shim-verification issue). BIOS confirmed
|
||||
SB `Disabled / Not Active`.
|
||||
2. **Bypassed shim** (swapped the stick's `\EFI\BOOT\BOOTX64.EFI` from shim → the ISO's `grubx64.efi`)
|
||||
→ **same error** (so it is *not* shim; it is **GRUB** failing — the message is GRUB's).
|
||||
3. Our ISO is **correct** — the identical prepared image boots in reference UEFI (OVMF), and the box's
|
||||
*installed* GRUB `2.12-9+pmx2` boots fine **from the SSD**. So: this early AMI firmware cannot
|
||||
relocate the **ISO's (signed/shim-oriented) GRUB build when loaded from USB**.
|
||||
|
||||
**Workaround that fixed it (done live over SSH from the still-bootable old system):** rebuilt the
|
||||
stick's `BOOTX64.EFI` with `grub-mkimage` **from the box's own working `2.12-9+pmx2` GRUB**, embedding
|
||||
every module the ISO's `grub.cfg` needs (so nothing loads from the ISO's problem build) and an embedded
|
||||
config that `search --fs-uuid`es the ISO and `configfile`s its real menu. **That booted straight into
|
||||
the installer.** The installed system's SSD GRUB then booted normally (SSD path always worked here).
|
||||
|
||||
This is the highest-value **slice-B input**: some cheap boards need a firmware-compatible boot loader.
|
||||
Options to fold into the pipeline — (a) build `BOOTX64.EFI` with `grub-mkimage` at ISO-prep time
|
||||
instead of shipping the shim-oriented grub, or (b) ship a documented field-fix (the exact `grub-mkimage`
|
||||
recipe used here), or (c) offer a **PXE/network-boot** path (the assistant's `--pxe` artifacts) for
|
||||
boards where USB-grub is broken.
|
||||
|
||||
## Findings
|
||||
|
||||
| # | Sev | Finding | Root cause | Disposition / fix |
|
||||
|---|-----|---------|-----------|-------------------|
|
||||
| **F1** | **HIGH** | ISO won't UEFI-boot GRUB from USB on this AMI `AN3PLUS 0.01` firmware (`relocation 0x0…`) | firmware can't relocate the ISO's signed GRUB from USB; SB-off and shim-bypass don't help | **worked around live** (self-built `grub-mkimage` loader from the box's own GRUB). R-21 slice-B: bake a firmware-compatible loader into the pipeline, or ship the field-recipe / PXE path. Nested virt could never catch this — the flagged OPEN, now concrete |
|
||||
| **F2** | MEDIUM | No claim-code email on reinstall of an existing customer | claim state (`claim_code_generation:2`, hash, issued 2026-07-13) is **hub/customer-level** and is delivered to the fresh box; an existing code ⇒ no re-issue/re-email. Fresh box has **no password set** | need a **"re-issue claim code"** operator action (bump generation + email). **⚠ also verify** whether the reinstalled *unclaimed* box is properly gated or accidentally **open** (F-4 class) — dashboard routes were reachable directly |
|
||||
| **F3** | MEDIUM | Offsite target missing on the fresh controller → escrow blocked | offsite transient password is "delivered to the controller **once**" — it went to the *old* box; the fresh controller never got it. Hub showed provisioned + escrow-done → **hub/controller desync** | **"Re-issue offsite credentials"** in the hub restaged it (done during the run; controller picks up next config refresh). Codify: reinstall must re-issue offsite |
|
||||
| **F4** | MEDIUM | PBS-DR read 403s every tick | agent token `felhom-agent@pve!agent` has `FelhomAgentStore` on `/storage/**felhom-pbs**` only, but the customer's **PVE STORAGE ID is `felhom-offsite`** (non-default, the demo's adopted manual entry) → `GET /storage/felhom-offsite -> 403 (missing Datastore.Allocate)` | install ACL must grant on the **config's storage id**, not just default `felhom-pbs`. Fix now: `pveum acl modify /storage/felhom-offsite --tokens 'felhom-agent@pve!agent' --roles FelhomAgentStore` (+ user); or set the config's storage id to `felhom-pbs` |
|
||||
| **F5** | MEDIUM | Guest RAM = 2 GB (too low on a 16 GB host) | **golden default**; no `--memory` passed (appliance), no per-customer/host-aware sizing | make guest RAM/cores configurable (hub config or auto-size at provision). Currently only the install `--memory` flag exists and isn't surfaced |
|
||||
| **F6** | MEDIUM | Drive "initialize" formats but doesn't mount/attach; UI shows nothing | on confirm, the **format client disconnects** ("mkfs continues detached; poll GET /disks/format/status") — mkfs completes (`storage: formatted device /dev/sdb ext4`) but the **post-mkfs mount+register is aborted** and the UI never polls the status | users reasonably expect *initialize* to also mount+register. Fix the confirm→poll flow so the wizard finishes the mount+attach and reports progress/completion. (Viktor recovered by re-running the *attach* flow manually → `hdd_1` at `/mnt/hdd_1`) |
|
||||
| **F7** | LOW | "Vissza" (Back) on `/storage/init` and `/storage/attach` routes to `/settings` | frontend route bug | point Back → `/storage` |
|
||||
|
||||
**Non-findings / notes:** the HDD initially not showing under "attach" was **CC's leftover read-only
|
||||
canary mount** (fixed, not a product bug). BIOS **State After G3 = S5** — the box stays *off* after a
|
||||
power cut; for a headless server this should be "Power On / Last State" (set before final sign-off).
|
||||
Hub "Containers 0/0 → 2/2" was report lag, not a bug.
|
||||
|
||||
## Current state (end of run)
|
||||
|
||||
- felhom-pve = **PVE 9.2.2** on the SSD; **agent 0.88.0**; up ~30 min; boots normally from the SSD.
|
||||
- Guest **9201 running** (demo-felhom): `felhom-controller` (0.136.0), `traefik`, `cloudflared`,
|
||||
`filebrowser`, **`bookstack` + `bookstack-db`** — demo-felhom.eu live.
|
||||
- Storage: `local` + `local-lvm` (system, protected); **`hdd_1`** = the external HDD, reformatted ext4
|
||||
at `/mnt/hdd_1` (old secondary backups **gone**, per the reformat choice). SD card + Samsung stick
|
||||
still attached.
|
||||
- Offsite: re-issued (staged, picking up). **DR tier blocked on F4 (403)** until the ACL is granted.
|
||||
- Onboarding not fully completed: **claim/password not set** (F2), **escrow ceremony not run** (F3/F4).
|
||||
- The Samsung boot stick still carries the ISO with the CC-built GRUB workaround (secret-bearing —
|
||||
wipe at teardown per the runbook).
|
||||
|
||||
## Evidence
|
||||
|
||||
DooPlex `~/n100-baremetal/pre/` (G0.2 harvest: dmidecode, udev per-disk, efibootmgr, canary-before,
|
||||
summary with the rulings + identity), `~/n100-baremetal/evidence/`, `~/felhom-iso/out/` (the ISO +
|
||||
sha/manifest, secret-bearing, 0600). BIOS/console photos + hub screenshots captured in the session.
|
||||
Console error, GRUB-fix steps, and the rc-0 chain journal are quoted inline above.
|
||||
|
||||
## Recommendations (to tackle next)
|
||||
|
||||
1. **F1 (slice B):** decide the boot-loader strategy — pipeline-built `grub-mkimage` `BOOTX64.EFI`,
|
||||
documented field recipe, and/or a PXE path — so R-21 works on cheap boards, not just reference UEFI.
|
||||
2. **F4:** grant the DR ACL on the config's storage id (quick unblock) and fix the installer to derive
|
||||
the ACL storage set from the customer's PVE STORAGE ID.
|
||||
3. **F2 + F3 (Peti-pattern, R-1):** the **reinstall-of-an-existing-customer** path needs first-class
|
||||
support — re-issue claim code (+ verify gating), and re-issue offsite/PBS on reprovision. This is
|
||||
directly on Peti's convergence path.
|
||||
4. **F6:** finish the drive-initialize wizard (mount+attach + status polling).
|
||||
5. **F5 / F7 / G3-power-on:** guest RAM configurability, the Back-route fix, and set the BIOS power-on
|
||||
policy.
|
||||
Reference in New Issue
Block a user