docs(tests): N100 bare-metal physical validation run (R-21) — findings + current state

Supervised clean-slate reinstall of the demo N100 from a pipeline-built Felhom ISO +
onboarding rehearsal. Core objectives GREEN: first-boot chain reached rc-0 on the first
attempt on real hardware (closes slice A's operator-gated boundary), serial-filter safety
proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict -> key on
MAC+UUID. Hard obstacle was firmware not the pipeline: this cheap AMI AN3PLUS 0.01 board
won't UEFI-boot the ISO's GRUB from USB (relocation 0x0; SB-off + shim-bypass don't help),
worked around live with a grub-mkimage loader from the box's own GRUB (slice-B input).
7 findings logged (F1 firmware loader HIGH; F2/F3/F4 reused-customer edges -> R-1/Peti;
F5 guest-RAM, F6 drive-init mount, F7 back-route). Full doc in documentation/tests/;
ROADMAP R-21 + REPORT + CONTEXT updated. No production code changed.

REPORT.md

@@ -2,125 +2,36 @@
 
 > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
 
-## R-21 slice A: bare-metal Felhom ISO (build pipeline + first-boot bootstrap) — 2026-07-16
+## R-21 physical run — N100 bare-metal reinstall + onboarding rehearsal — 2026-07-16
 
-**Repo:** felhom.eu · **Baseline:** `main` @ `914d51a` → scripts **v1.16.0** (ISO pipeline tag;
-`felhom-host-install` was *already* v1.16.0 from 2026-07-13 — see Versioning note).
-**Validation:** nested VM **310** on felhom-pve; ISO builds on DooPlex. `felhom-host-install.sh`
-**not modified** (read-only, invoked only). Hub code untouched.
+Supervised run (Viktor at the box, CC on Phase-0 prep + the SSH-side legs). Full detail:
+**`documentation/tests/VALIDATION-n100-baremetal-2026-07-16.md`**. No production code changed.
 
-### Files created (`scripts/iso/`)
+**Core objectives — GREEN.** The demo N100 (felhom-pve, serving demo-felhom.eu) was reinstalled
+clean-slate from a pipeline-built ISO (`felhom-pve-9.2-1-v1.16.0-n100-demo.iso`, profile filter
+`ID_SERIAL_SHORT=QDF922W009654S30EX`, appliance). The first-boot chain reached **rc-0 on the first
+attempt (`NRestarts=0`)** on real hardware — enrolled `demo-felhom`, provisioned guest 9201, agent
+0.88.0, WG up, `bootstrap.env` shredded. This **closes slice A's operator-gated rc-0 boundary on
+metal.** Serial-filter safety proven on real hardware (external-HDD canary byte-identical). PBS-DR
+reconciler self-healed on the reused peer (all four steps done incl. escrow). DMI verdict on real
+cheap hardware: all "Default string" → slice-C keys on **MAC + UUID**.
 
-`Dockerfile.assistant`, `build-felhom-iso.sh`, `answer.toml.tmpl`, `stub-first-boot.sh`,
-`felhom-bootstrap.sh`, `felhom-bootstrap.service`, `profiles/nested-vm.profile`,
-`profiles/nested-canary.profile`, `README.md`. Plus `scripts/CHANGELOG.md`,
-`documentation/backlog/ROADMAP.md` (R-21 → in-progress), this `REPORT.md`, `CONTEXT.md`.
+**The obstacle was firmware, not the pipeline.** This board's early AMI `AN3PLUS 0.01` UEFI cannot
+boot the ISO's GRUB **from USB** (`relocation 0x0 is not implemented yet`); Secure-Boot-off and
+shim-bypass did **not** help (it's GRUB, not shim). The identical ISO boots in reference UEFI, and the
+box's installed GRUB boots fine from the SSD — so it's the firmware relocating the ISO's USB-loaded
+GRUB. **Worked around live** by rebuilding the stick's `BOOTX64.EFI` with `grub-mkimage` from the box's
+own working `2.12-9+pmx2` GRUB (all needed modules embedded, config that searches+boots the ISO). This
+is the top **slice-B input**: the pipeline should ship a firmware-compatible loader and/or a PXE path.
 
-### Green gates
+**Findings to tackle (7):** F1 firmware boot loader (HIGH, slice-B); F2 no claim-code re-issue on
+reinstall (+ verify open-vs-gated); F3 offsite creds need re-issue on reprovision; F4 PBS-DR 403 —
+install ACL grants only default `felhom-pbs`, not the config's `felhom-offsite` storage id; F5 guest
+RAM 2 GB not configurable; F6 drive-initialize formats but doesn't mount+attach (client-disconnect
+aborts post-mkfs, no UI feedback); F7 storage Back-route → /settings. F2/F3/F4 are reused-customer
+clean-slate edges feeding R-1/Peti. BookStack deployed + works.
 
-`bash -n` clean on all three scripts; `shellcheck -x` **exit 0, no findings** (run in the assistant
-container). Retry-unit behavior de-risked on systemd 257 before wiring (transient unit: `Type=oneshot`
-+ `Restart=on-failure` **does** retry — NRestarts climbed).
-
-### §4.4 finding — does the distribution channel embed a secret? (decides slice C)
-
-**The channel: no. The unattended install: yes (the passphrase).** The hub install-command generator
-(`hub/internal/web/templates/customer_unified.html`, Option 1) emits
-`curl -fsSL https://felhom.eu/scripts/felhom-host-install.sh -o … && sudo bash … --customer-id <id>
---mode <appliance|byo>` — a **public URL, no token/secret in the command**. But
-`felhom-host-install.sh` **requires the customer retrieval passphrase** (`read_passphrase` →
-`die "empty passphrase"`; used for config fetch + `POST /host-enroll`). Unattended ⇒ it must be
-supplied via `--passphrase-file`, so **the slice-A ISO is secret-bearing** (embeds the passphrase).
-**Slice-C implication:** the *distributed* ISO must NOT bake the passphrase — the claim/pairing model
-must deliver it (or the config) to the box *after* the customer claims it. Slice-A ISOs are
-supervised/single-use; the bootstrap `shred`s `bootstrap.env` on success.
-
-### 2.3 retry-vs-resume ruling (source-verified, encoded once)
-
-**Plain first attempt; add `--resume` on any later attempt that finds
-`/var/lib/felhom-install/state.json`.** Source landmark: `felhom-host-install.sh` header **v1.11.3**
-("--resume correctness — the producer steps (token/enroll/grows) run every pass … a resumed install no
-longer writes a config missing hub.host_id/proxmox.token") — so `--resume` is safe, whereas a plain
-re-invoke over an existing install state would re-hit the populated-host leaf guard / existing-vmid
-refusal. `--mode` is required in both forms. Encoded in `felhom-bootstrap.sh` with the landmark cited
-inline; **empirically confirmed** (below).
-
-### Per-scenario results (evidence under `180:~/felhom-iso/evidence/`)
-
-- **C — build gate blocks a broken answer (+ red-proof) — PASS.** Answer with both `root-password`
-  and `root-password-hashed`: the output-parse gate surfaced `` `global.root-password` and
-  `global.root-password-hashed` cannot be set at the same time / Error: Found issues `` and aborted
-  (`BUILD-EXIT=1`, **no ISO** — count unchanged), *even though `validate-answer` exits 0*.
-  **Red-proof:** gutting the gate to trust `$?` made it wrongly report `[OK] answer validated`; the
-  broken answer was then stopped only by `prepare-iso`'s own error **plus the build's `[[ -f out.iso ]]`
-  guard** — notably `prepare-iso` *also* exited 0 on the error, so the file-existence check was the
-  real backstop. Reverted (mutation done on the DooPlex copy; committed tree never touched).
-- **D — disk-filter fail-safe through the pipeline — PASS.** Pipeline-built canary ISO
-  (`filter.ID_SERIAL_SHORT="no-such-disk-zzz"`) → `ERROR: Installation failed: filter did not match any
-  device`, abort exit 1, no interactive fallback. 8 GB canary disk **byte-identical** after
-  (`sha256 -c: OK`, host-side; `scenarioD-filter-abort.png`).
-- **A — full unattended chain — MECHANISM PASS; terminal success operator-gated.** Zero keypress;
-  stub ran once (uid 0) and enabled+started `felhom-bootstrap`; the unit **fetched host-install from the
-  REAL channel** `https://felhom.eu/scripts/felhom-host-install.sh` and invoked it
-  `--customer-id … --mode appliance --hub-url …`; host-install ran unattended through preflight
-  (hub reachable) — with the drill customer absent it correctly failed at config preflight (`404`) and
-  the unit **retried** (`bootstrap-journal.log`; NRestarts ≥3; 12 host-install invocations).
-  **Not live-proven:** host-install **rc-0** → hub-enrolled + guest-provisioned → done-flag +
-  unit-disable + no-re-run. Reason: that needs a properly created drill customer, and CC cannot use the
-  password-gated create-UI while a direct write to the *live, locked* hub `hub.db` (verified open by the
-  running pod) risks corruption. The done-flag/disable/env-scrub success path is code-reviewed +
-  shellcheck-clean; empirical proof requires an operator-created drill customer (supervised).
-- **B — no network at first boot — PASS.** During a host-side `link_down` window
-  (13:37:56→13:39:37) the chain failed repeatedly — `host-install FAILED rc=6` (host-install's own hub
-  fetch, no net) and `ERROR: host-install fetch failed (no network yet?)` — and **the unit kept
-  retrying (never died)**; on link-up it **resumed automatically** (`13:40:04` fetch+run). No
-  "single failure → dead unit". The stub-runs-without-network property is spike-proven (S8a);
-  `bootstrap-journal.log`.
-- **2.3 resume decision — PASS.** After creating `/var/lib/felhom-install/state.json`, the next attempt
-  logged `prior install state present (…) -> adding --resume`; earlier attempts ran plain.
-- **Exactly-once stub — PASS.** Host-side reboot → `/var/log/felhom-first-boot.log` hash **unchanged**
-  (`13cf3fe4…`), pending-flag consumed; the retry unit correctly persisted (still `activating`, not done).
-
-### ISO artifacts (build ≈ 12 s each, source already local)
-
-| ISO | sha256 | size |
-|---|---|---|
-| `felhom-pve-9.2-1-v1.16.0-nested-vm.iso` | `8cbe5eaeaa0d…82bb` | 1,707,737,088 B |
-| `felhom-pve-9.2-1-v1.16.0-nested-canary.iso` | `6b4568eaeac0…0301` | 1,707,737,088 B |
-| (validation build, key baked) `…-nested-vm-val.iso` | `0042acfdb70a…98e7` | — |
-
-Source ISO `proxmox-ve_9.2-1.iso` sha256 `4e88fe4…f2c6c` (verified vs official SHA256SUMS). Assistant
-`proxmox-installer-common 9.2.7` (trixie no-sub). Artifacts kept under `180:~/felhom-iso/out/`, never
-committed (secret-bearing).
-
-### VM 310 final state
-
-Stopped, kept; SeaBIOS, scsi0 32 GB + scsi1 8 GB (canary intact), ISO detached. Boundaries honored:
-VM 300 (drill) running untouched, 9000 stopped, LXC 9201 (demo) running untouched, hub untouched.
-Secret-bearing pipeline ISOs removed from felhom-pve (their embedded passphrase was a placeholder, not a
-real credential).
-
-### NOT-live-validated
-
-- **Physical N100 run (RUNBOOK B):** real DMI/firmware, USB-boot media, vendor Secure Boot key set,
-  real-NIC naming, prior-LVM `blkdiscard` on real hardware. Not started (out of scope by design).
-- **host-install rc-0 terminal success** (hub-enrolled + guest-provisioned + done-flag + disable) and
-  the **2.3 `--resume` executed against a real partial install** — both need an operator-created drill
-  customer (password-gated create-UI; live-DB insert unsafe). The *decisions/plumbing* are proven; only
-  the terminal success is deferred.
-
-### Versioning note
-
-`felhom-host-install` reached v1.16.0 on 2026-07-13 (before this task's stated 1.15.x baseline). The
-new ISO pipeline is tagged **v1.16.0** to stay coherent with that release (the ISO's baked invocation
-targets host-install's flag surface). The `scripts/CHANGELOG.md` entry is a new component under the
-same release tag, not a bump of host-install.
-
-### Observations (recorded, not acted on)
-
-- `prepare-iso` (not just `validate-answer`) was observed to **exit 0 on a bad answer** — the build's
-  `[[ -f out.iso ]]` output-existence guard is therefore load-bearing, not just belt-and-suspenders.
-- host-install prints `unexpected hub status 404404` (doubled) for an absent customer — cosmetic
-  host-install quirk; left untouched (read-only this task).
-- host-install exits **rc=6** when it cannot reach the hub (no net) — the bootstrap correctly treats
-  any non-zero rc as retry.
+**Current state:** box healthy, demo-felhom.eu live, guest 9201 running (controller/traefik/cloudflared/
+filebrowser/bookstack), external HDD reformatted + attached as `hdd_1`. Onboarding not finished (claim
++ escrow pending on F2/F3/F4). Boot stick (secret-bearing, with the GRUB workaround) to be wiped at
+teardown. Findings will be worked next per Viktor.