felhom.eu/documentation/audits/SPIKE-universal-iso-2026-07-31.md

# SPIKE — what does a genuinely universal ISO require? (2026-07-31)

> **Class: Spike.** Output is findings, not shipped code. No production file was changed, no release
> ISO was built, nothing was published. Probe artefacts are quoted here and were removed at teardown.
>
> **Origin.** The ISO build/publish task (2026-07-31) stopped before its Part 2 when its Part 0 found
> two hazards that make a public download a product decision rather than a build step. This spike
> establishes what a genuinely universal image would require.

---

## 1. Baselines

Read fresh at the start of this session, not carried from the previous one.

| Baseline | Value | Source |
|---|---|---|
| `ISO_VERSION` | `1.25.0` | `scripts/iso/build-felhom-iso.sh:44` |
| `SCRIPT_VERSION` | `1.22.0` | `scripts/felhom-host-install.sh:187` |
| `felhom-bootstrap.sh` @ HEAD | `21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7` | `sha256sum scripts/iso/felhom-bootstrap.sh` |
| PVE base ISO | `proxmox-ve_9.2-1.iso`, `4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c` | `sha256sum /mnt/5_hdd/felhom.eu/drill/proxmox-ve_9.2-1.iso` — matches every ISO manifest's `source-iso-sha256` |
| `felhom.eu` HEAD | `5825ceeabfc50227bee2ccf9cf35c412c05c0f2a`, clean, `== origin/main` | `git status --porcelain` empty |
| assistant tooling | `proxmox-installer-common v9.2.7` | `proxmox-auto-install-assistant --version` |

**Fixture provenance.** Nested VMs ran under plain QEMU/KVM on **demo-hp** (t740, Tier 0, the
designated drill + build VM host per the 2026-07-25 standing ruling), with all scratch files under
`/mnt/nvme-1tb/spike-scratch/`. **Nothing was placed on demo-hp's `local-lvm`** and no PVE storage was
registered. `drill-r50` (VM 300) and guest 9201 were not touched. The probe ISO was built on DooPlex
from the baseline PVE ISO with `prepare-iso --fetch-from partition` and transferred once
(`sha256 337985c5c88b8d76c7a0a6e1678b48063f3b7d077222c91b9a1a38cfb6ca8c75`, verified identical at both
ends), so every scenario reused one image and varied only a 64 MB answer disk.

**Real-hardware work was read-only.** `device-info` was executed on `demo-felhom` and `demo-hp` via a
temporary copy of the assistant binary in `/tmp` (removed at teardown). No install, no partitioning,
no attach/detach.

---

## 2. Reconfirmation of the two established findings

Both re-verified at `file:line` rather than carried forward.

**(a) Every ISO bakes a root password hash.** `build-felhom-iso.sh:270-283` mints `ROOT_PLAIN`
unconditionally, hashes it with `openssl passwd -6`, and the renderer substitutes it into
`__ROOT_HASH__` (`:295-296`) from `answer.toml.tmpl:24`. There is no flag to suppress it.

Verified by **extraction, not inference**: `osirrox` pulled `/answer.toml` out of the shipped
`felhom-pve-9.2-1-v1.25.0-nested-vm-generic-mkimage.iso` and it carries a live `root-password-hashed`
field. The sibling `.rootpw.txt` plaintext was confirmed to hash to exactly that value. Any downloader
can perform the same extraction with no privilege.

**(b) `generic.profile` selects by name, not identity.** `profiles/generic.profile:16-18` sets
`disk-list = ["sda"]`. Confirmed present verbatim in the extracted `/answer.toml` of the v1.25.0 build.
`generic.profile:12-14` concedes the limit in its own comment. The serial pin that closed this hazard
on the N100 rehearsal (`hp.profile:36`) does not generalise to strangers' hardware.

---

## 3. Q1 — what the answer file can express about disk selection

### 3.1 The selection mechanisms (read from schema)

Extracted from the assistant binary's own serde field strings and confirmed against
`validate-answer` behaviour. `[disk-setup]` is `struct DiskSetup with 7 elements`:

| Key | Purpose |
|---|---|
| `filesystem` | `ext4` / `xfs` / `zfs` / `btrfs` |
| `disk-list` | explicit device names |
| `filter` | map of udev-property globs |
| `filter-match` | `any` \| `all` |
| `lvm` / `zfs` / `btrfs` | per-filesystem option sub-tables |

The binary's validation strings pin the rules:

```
Need either 'disk-list' or 'filter' set
Cannot use both, 'disk-list' and 'filter'
make sure to define only one disk for ext4 and xfs
filter did not match any devices
no filter defined
invalid glob in disk selection
```

Glob syntax is `?`, `*`, `[a]`, `[0-9]`, `[!a]` (`device-match --help`). **`[!…]` negates a single
character, not a match** — there is no "does not contain" and no "property is absent" operator.

**There is no exclusivity keyword.** Nothing in the 7 elements expresses "abort unless exactly one
candidate matches". The nearest thing is the `only one disk for ext4 and xfs` rule — a consequence of
the filesystem choice, not a selection control (measured behaviour in §3.3).

### 3.2 What distinguishes an internal system disk from external media — **measured**

`device-info -t disk`, captured on both Tier 0 boxes.

**`demo-felhom` — the hazard configuration** (internal SATA SSD + the 1 TB external USB drive):

| | `sda` (internal) | `sdb` (external USB) |
|---|---|---|
| `ID_BUS` | `ata` | **`ata`** |
| `ID_PATH` | `pci-0000:00:17.0-ata-2.0` | `pci-0000:00:14.0-usb-0:1:1.0-scsi-0:0:0:0` |
| `ID_MODEL` | `AirDisk_512GB_SSD` | `TOSHIBA_MQ04ABF100` |
| `ID_USB_DRIVER` | *absent* | `usb-storage` |
| `ID_USB_VENDOR` | *absent* | `ADATA` |
| `lsblk RM` (removable) | `0` | **`0`** |
| `lsblk HOTPLUG` | `0` | `1` |

Three findings, and the first two are the important ones:

1. **`ID_BUS` does not distinguish them.** Both report `ata`. The external unit is a SATA drive in a
   USB caddy (an ADATA HD710 PRO enclosure around a Toshiba MQ04ABF100), so the bus property describes
   the drive, not the attachment. A filter written as "internal means `ID_BUS=ata`" matches the
   customer's backup drive.
2. **There is no removability property at all** in `device-info` output — no `RM`, no `ID_REMOVABLE`.
   And `lsblk` reports `RM=0` for the external drive anyway, so even reaching outside `device-info`
   does not supply the missing concept. **Property filtering cannot express "not removable."**
3. What *does* distinguish them here is `ID_PATH` (`-ata-` vs `-usb-`) and the presence of the
   `ID_USB_*` key family — but only the former is expressible, because the filter language has no
   "key is absent" operator.

**`demo-hp`** (internal SATA system disk + a 1 TB NVMe data disk):

| | `sda` (SATA, the PVE system disk) | `nvme0n1` (NVMe, enrolled user data) |
|---|---|---|
| property count | 45 | 28 |
| `ID_BUS` | `ata` | **absent** |
| `ID_TYPE` | `disk` | **absent** |
| `ID_PATH` | `pci-0000:06:00.0-ata-1.0` | `pci-0000:04:00.0-nvme-1` |

Absence verified explicitly by key lookup, not by a grep that happened not to print.

4. **NVMe devices carry a different property set entirely** — no `ID_BUS`, no `ID_TYPE`. Any filter
   built on `ID_BUS` silently excludes every NVMe-only machine.
5. **Decisively: no property expresses *role*.** Every captured property describes hardware — bus,
   path, model, serial, size. None says "this is the disk the operating system belongs on." On
   `demo-hp` the system disk is the *small* SATA one and the big NVMe is user data; a heuristic like
   "largest disk" or "the NVMe one" targets the customer's data drive on exactly the configuration
   Felhom sells.

`device-match` confirmed the expressible cases on `demo-felhom` (read-only):

```
ID_BUS=ata          -> ["sda","sdb"]     both — the naive filter hits the backup drive
ID_PATH=*usb*       -> ["sdb"]           isolates the external
ID_PATH=pci-*-ata-* -> ["sda"]           isolates the internal, on this box
```

Both isolating filters are box-specific: the `-ata-` pattern excludes every NVMe system disk, and the
`*usb*` pattern is an exclusion that the filter language cannot invert.

### 3.3 Multi-match behaviour — **observed. It is the coin-flip.**

This is the question the spike existed to answer, and it was measured, not inferred.

**Method.** A nested VM on demo-hp with two identical 16 GB virtio-SCSI disks carrying serials
`SPIKE0` and `SPIKE1`, plus a separate answer disk (serial `ANSWERDISK`, label `proxmox-ais`). Answer:

```toml
[disk-setup]
filesystem = "ext4"
filter-match = "all"
filter.ID_SERIAL_SHORT = "SPIKE*"      # matches SPIKE0 and SPIKE1 — two devices
```

The answer **validated cleanly** (`The answer file was parsed successfully, no errors found!`) — so
`validate-answer` cannot catch this; it does not know what the filter will resolve to on the target.

The measurement is deliberately not a screen reading: both disk images were zeroed and hashed before
the run and re-hashed after, so "which disk was written" is objective.

| Disk | sha256 (first 32) BEFORE | AFTER | Verdict |
|---|---|---|---|
| `disk0.qcow2` (`SPIKE0`) | `de001102cbc8ff7aa8eb6c69d750cf22` | `382b55cad8027c60dfcbed254e723123` | **WIPED AND INSTALLED ONTO** |
| `disk1.qcow2` (`SPIKE1`) | `de001102cbc8ff7aa8eb6c69d750cf22` | `de001102cbc8ff7aa8eb6c69d750cf22` | untouched |

Observed mid-run, confirming it was a real install and not a partial write: `disk0.qcow2` had grown to
5,441,388,544 bytes while `disk1.qcow2` sat at 196,864 bytes (bare qcow2 metadata).

**The installer did not abort. It silently selected one of the two matching disks and wiped it.**

Two consequences:

1. **`filter-match` has no fail-safe on ambiguity.** The `filter did not match any devices` guard —
   the spike-S5c behaviour the `nested-canary` profile pins, and the safety default the `hp`/`n100`
   profiles rely on — protects only the **zero**-match case. The **many**-match case is unprotected and
   silent.
2. The binary's `make sure to define only one disk for ext4 and xfs` string does **not** cover this.
   It rejects an explicit multi-entry `disk-list`; a `filter` that *resolves* to several devices sails
   past both `validate-answer` and the installer.

**Why this matters more than the `disk-list=["sda"]` hazard.** The previously known hazard is that
`sda` may name the wrong disk. This one is worse in kind: a filter written specifically to be *safe*
— "match the internal SSD" — degrades to an arbitrary pick the moment a second device matches, with no
diagnostic. Given §3.2 (no removability property, `ID_BUS` identical on the hazard configuration),
writing a filter that provably matches exactly one device on unseen hardware is not achievable.

**Not established:** whether the selection is deterministic by enumeration order or genuinely
arbitrary. Only one ordering was tested. See §8.

### 3.4 Is the boot medium a candidate?

Partially measured. On `demo-felhom` the USB-attached drive **is** enumerated by `device-info` as an
ordinary candidate disk with no marking distinguishing it from the internal one — a USB installer
stick is the same class of block device, so nothing in the captured data suggests the enumeration
would exclude it.

**Not fully established.** Whether the installer *itself* excludes the live medium it booted from is a
separate question from whether `device-info` enumerates it, and this spike did not boot from USB on
real hardware. See §8.

---

## 4. Q2 — can the root password be generated on the box instead of baked?

### 4.1 Is a password required? — **measured**

The binary carries both rules verbatim:

```
One of `global.root-password` or `global.root-password-hashed` must be set
`global.root-password` and `global.root-password-hashed` cannot be set at the same time
```

Confirmed empirically against `validate-answer`:

| Answer | Result |
|---|---|
| root password omitted entirely | **rejected** — `One of global.root-password or global.root-password-hashed must be set` |
| `root-password-hashed = "*"` | **accepted** |
| `root-password-hashed = "!"` | **accepted** |
| `root-password-hashed = ""` | **accepted** |
| both plaintext and hashed set | **rejected** |

**A root password field cannot be omitted — but the classic locked-account crypt placeholders `*` and
`!` validate.** That is the opening: the field is mandatory, a *usable credential* is not.

That an empty string also validates is worth a fence of its own — `""` is not a locked account, and
whether it yields a passwordless root depends on how it reaches `/etc/shadow`.

**Validation is not installation, so it was installed** — and the first attempt to confirm that is
worth recording, because it nearly produced a false finding.

*S2, first attempt.* A nested VM ran an unattended install whose only credential was
`root-password-hashed = "*"`. `disk0.qcow2` went from `de001102cbc8ff7aa8eb6c69d750cf22` to
`f4ae86b8317cc36af4628a6cd9c09da2` and grew to several GB, which reads like success. Booting the
result showed otherwise: SeaBIOS printed `Booting from Hard Disk...` and hung. The install had been cut
off mid-flight by the probe's fixed 300 s window; a written disk is evidence of *writing*, not of
*completion*. **A changed hash is a `LastRun`-class timestamp — it records that something was
attempted.**

*S2b, with an actual completion signal.* `reboot-mode = "power-off"` is a valid enum value
(`"poweroff"` is rejected, so this is a real enum), and it makes the guest power itself off when the
install finishes — i.e. the qemu process exiting **is** the completion signal, and a timeout is
unambiguously a failure rather than an unknown.

| Disk | sha256 (first 32) BEFORE | AFTER | Completion signal |
|---|---|---|---|
| `disk0.qcow2` | `de001102cbc8ff7aa8eb6c69d750cf22` | `dbc12e129ed84e834f6d54754cb0a2c8` | **guest powered itself off after 200 s** |

**Established: an unattended install whose only credential is `root-password-hashed = "*"` runs to
completion.** The mandatory-field constraint does not force a *usable* baked credential — it forces a
*field*, and a locked-account placeholder satisfies it. This is the mechanism a universal image needs.

**Not established: that the resulting root account is actually locked.** The install completing is not
the same claim. Confirming it needs either `/etc/shadow` from the installed image or a refused login,
and neither was obtained — see §4.3 and §8.

### 4.2 Can `felhom-bootstrap` rotate root at first boot?

Mechanically yes — it already runs as root at first boot, and `felhom-host-install.sh:1831-1871`
(step 4b, G1) already implements exactly this: generate 24 url-safe bytes, set via `chpasswd` on
stdin, vault to the hub, scrub the variable. The code to rotate exists and is proven.

**The problem is not capability, it is ordering** — see §4.3.

### 4.3 The exposure window — **structural finding, and it is not zero**

`felhom-bootstrap.service` is ordered:

```
After=network-online.target pve-cluster.service pveproxy.service
Wants=network-online.target
```

So bootstrap **starts after the network is up and after pveproxy is listening**. Whatever it does to
the root password happens strictly later than the box becoming reachable. The window is open by
construction.

Worse, in pairing mode it does not close on its own. `felhom-bootstrap.sh:9-22,52-54,363` registers the
box as an unclaimed appliance and then **polls indefinitely** (`POLL_INTERVAL=30`) for the operator's
bind, deliberately staying in `activating` because "waiting is not failing" (v1.21.0, R-33). Step 4b
only runs after enroll, and enroll only happens after the bind. **A box that is downloaded, installed,
and never bound sits on its owner's LAN with the ISO-baked password indefinitely.**

**The measurement the task asked for was attempted twice and BOTH ATTEMPTS FAILED. No number is
reported, because both numbers were artefacts of the fixture rather than facts about the system.**

*Attempt 1 — invalid, and it produced a plausible false zero.* The probe booted the installed disk
under QEMU with `hostfwd` port forwards and polled with a TCP connect. Both ports "accepted" at
**+0 s**, which reads like "reachable instantly". It is not: QEMU's user-mode networking binds the
forwarding socket on the *host* when QEMU starts and accepts the connection before it ever tries the
guest. The same run then failed to read an SSH banner and reported `Connection timed out during banner
exchange` — i.e. nothing was listening in the guest at all. **A TCP connect against a `hostfwd` port
proves nothing.** Had the banner check not been in the same script, `0 s` would have been recorded as
a measured finding, and it would have been the exact opposite of the truth.

*Attempt 2 — invalid for a different reason.* The detector was rewritten to require a real `SSH-`
version banner and a real HTTP response line. It then reported `never` for both, over 6 minutes. That
is also not a fact about PVE: the S2b install VM was created **with no network device at all**, so the
installed system's interface configuration cannot match the NIC that only exists in the boot VM. The
fixture, not the system, produced the result.

**What stands instead** is the structural finding above, which does not depend on either probe: the
unit's own `After=network-online.target pveproxy.service` places every rotation opportunity strictly
after reachability, and the pairing loop makes the interval unbounded. That is a stronger claim than a
single timing number would have been — it holds for every box rather than for one VM.

**What it would take to measure it properly:** install and boot with an identical NIC present in both
phases, and probe at protocol level from the host. ~30 min. It is worth doing before a spec is
written, because the number is what tells the operator whether the window is seconds or minutes when
pairing *does* proceed promptly.

**The mechanism that would close it:** `[first-boot].ordering` accepts **`before-network`** as well as
`fully-up` — verified against `validate-answer` (`network-pre` and a bogus value are both rejected, so
this is a real enum, not a permissive parser). A hook running `before-network` executes before the box
is reachable at all, which is what a zero-window rotation requires.

Felhom currently uses `ordering = "fully-up"` (`answer.toml.tmpl:39`), deliberately, so that `pvesh`
and `pct` work (spike S3/S8a). Note the current stub (`stub-first-boot.sh`) only writes three files and
runs `systemctl enable` + `start --no-block` — none of which obviously needs the network, and the unit
it starts carries its own `After=network-online.target`. **Whether the stub survives being moved to
`before-network` is untested and is the single highest-value follow-up probe.** `[first-boot]` accepts
only one hook (`struct FirstBootHookInfo with 4 elements`), so this is a move, not an addition.

### 4.4 Console recovery if pairing fails

**Not tested — stated plainly rather than assumed.** The spike ran out of room before booting the S2b
image into a GRUB single-user / `init=/bin/sh` shell.

This is not a detail. It is the input that decides §4.5: if the standard GRUB recovery path yields a
root shell on a locked-root install, then discarding the password costs little, because a physically
present person can always get back in. If it does not — or if PVE's installed GRUB is configured
without an editable menu — then discarding turns a failed pairing into a reinstall, and the escrow
argument is much stronger. **The ruling in §4.5 should not be made before this is measured** (§8, #4).

### 4.5 Escrow or discard — the tradeoff, for the operator to rule

Not a recommendation; the inputs.

**For discarding** (no per-box root password is ever recoverable):
- The operator already holds agent-level access on every *bound* box, so console root is redundant for
  the normal case.
- Under the D6 trust model the hub alone must not be sufficient. A vault of console root passwords for
  the whole fleet makes the hub exactly that, for every box at once — a strictly worse blast radius
  than the shared-ISO-password problem this is meant to fix.
- It removes a class of credential that has to be rotated, audited, and leak-checked forever.

**For escrowing** (the current G1 behaviour, extended to pre-bind):
- An *unbound* box has no agent access — precisely the case where console access is the only route in,
  and precisely when things go wrong.
- G1 exists because the sshd path and the auto-heal both failing is a real, observed failure mode; the
  PVE web console at `:8006` is a distinct failure domain from sshd.
- Discarding turns "pairing failed" into a reinstall for a non-technical customer.

**The middle option worth costing:** rotate to a per-box random value at `before-network` and escrow
*nothing* until the box binds, at which point step 4b's existing vaulting takes over. This closes the
shared-credential window without adding a pre-bind fleet-wide vault — at the price of making an
unbound, broken box unrecoverable except by reinstall. Whether that price is acceptable is the
operator's call, and it is the one decision this spike asks for.

### 4.6 R-129 — SSH key provenance, **resolved**

The premise is confirmed: `ssh -o BatchMode=yes demo-hp` authenticates by key, while the documentation
says no baked key exists. The two are not in conflict — the key is not baked.

| Box | `/root/.ssh/authorized_keys` → `/etc/pve/priv/authorized_keys` | Provenance |
|---|---|---|
| `demo-hp` | `SHA256:pgQh228R…` `nagyfenyvesi.viktor@gmail.com` | the operator's own DooPlex key, added post-install (file mtime 2026-07-27, install 2026-07-21). **This is what authenticates CC.** |
| | `SHA256:fnasIK1Y…` `root@felhom-host` (RSA 4096) | PVE's own self-generated root key |
| `demo-felhom` | `SHA256:a8zH+ecm…` `kisfenyo@windows` | **baked by the ISO** — exact fingerprint match to `FELHOM_ROOT_SSH_KEY` in the uncommitted `n100-demo.profile` |
| | `SHA256:BnLVXqTc…` `root@demo-felhom` (RSA) | PVE self-key |
| | `SHA256:pgQh228R…` | the operator's DooPlex key |

Findings:

1. **demo-hp's key is not baked** — it is the operator's own, added by hand. The documentation was
   right about the ISO and wrong about the box.
2. **demo-felhom's key *is* baked**, via an uncommitted profile. So the "no baked key" claim was
   already false for one of the two demo boxes, and nothing recorded it.
3. **No committed profile bakes a key** — all five have `FELHOM_ROOT_SSH_KEY` commented out — and the
   shipped v1.25.0 ISO contains **zero** occurrences of `root-ssh-keys` in its `/answer.toml`.
4. **But the mechanism is one uncommented line from shipping a shared key to every downloader**
   (`build-felhom-iso.sh:175-176` → `answer.toml.tmpl:26`). For a published image that is the same
   shared-credential class as the baked password, and it needs a build-time assertion, not a
   convention.
5. `kisfenyo@windows` is the retired Windows workstation's key and is still authorized on
   `demo-felhom`. Stale credential; see R-rows.

---

## 5. Q3 — can the image offer both automated and interactive install?

**Yes, and it requires no PVE-side work at all.** Established by comparing three GRUB configs.

`prepare-iso` **does not touch `grub.cfg`**: the file extracted from the freshly prepared probe ISO is
byte-identical to the one in the stock PVE ISO (`diff` clean). The stock config already carries the
automated entry, gated on a file that `prepare-iso` creates:

```
if [ -f auto-installer-mode.toml ]; then
    set timeout-style=menu
    set timeout=10
    menuentry 'Install Proxmox VE (Automated)' { … proxmox-start-auto-installer }
fi
menuentry 'Install Proxmox VE (Graphical)'    { … }   # outside the conditional
menuentry 'Install Proxmox VE (Terminal UI)'  { … }   # outside the conditional
```

| ISO | menu entries |
|---|---|
| stock `proxmox-ve_9.2-1.iso` | 15 |
| **prepared** (probe, `prepare-iso` only) | **15** — automated **and** graphical **and** TUI |
| Felhom `v1.25.0` remastered | **1** (`Felhom telepítés`) |

So the answers are:

1. The current remastering **replaces** the menu. `iso-repack.sh` rewrites `grub.cfg` from
   `grub/grub.cfg.tmpl` to exactly one entry — by design, and documented as such
   (`build-felhom-iso.sh:16-23`: the stock entries "are not hidden, they are not emitted"). This is
   also the answer to the previous session's open question: the `mkimage`/non-`mkimage` split is
   **not** this distinction — it is a UEFI loader swap, orthogonal to the menu, and both variants get
   the same single-entry menu.
2. **Preparation does not consume the interactive path.** A multi-entry menu including the automated
   entry is the *native* shape of a prepared auto-install ISO. Felhom collapses it deliberately, for a
   reason that remains sound for a customer appliance (`build-felhom-iso.sh:18-21`: every stock entry
   is a route into the manual installer whose first question is which disk to wipe).
3. Therefore a two-entry menu is purely a change to Felhom's own `grub/grub.cfg.tmpl` — keeping
   `Felhom telepítés` as `default=0` and adding one guarded interactive entry. No PVE-side obstacle
   exists.

**Not demonstrated by boot.** The evidence here is configuration-level, which is weaker than a
screenshot of the menu. See §8.

---

## 6. Q4 — what else is baked that a stranger should not receive

Answered by **positive enumeration against the stock ISO**, not by grepping for patterns I guessed —
a pattern scan can only find what it was told to look for.

**Method.** Full recursive file listing of the shipped `v1.25.0` ISO and the baseline PVE ISO
(`osirrox -find`, 1576 vs 1573 paths), diffed both directions; content hashes compared for the OS
payloads; all three first-boot payloads decoded and hashed against repo HEAD; the answer file's
credential fields enumerated by name.

**Added by the Felhom pipeline — exactly six paths:**

| Path | Content | Secret-bearing? |
|---|---|---|
| `/answer.toml` | the rendered answer | **YES — `root-password-hashed`** |
| `/auto-installer-mode.toml` | `mode = "iso"` | no |
| `/proxmox-first-boot` | the rendered stub + 3 embedded payloads | no (below) |
| `/boot/grub/felhomtheme/` + `background.png` + `theme.txt` | branding | no |

**Removed:** the three stock `/boot/grub/pvetheme/` paths.

**The OS payloads are untouched** — `pve-base.squashfs` and `pve-installer.squashfs` are byte-identical
to the stock ISO. Nothing is hidden in the installed system image.

**The three first-boot payloads**, decoded from the stub:

| Payload | Result |
|---|---|
| `felhom-bootstrap.sh` | sha256 **identical** to repo HEAD |
| `felhom-bootstrap.service` | sha256 **identical** to repo HEAD |
| pairing env | only `FELHOM_HUB_URL` + `FELHOM_INSTALL_URL` — no customer id, no passphrase |

**Also scanned for** (across all four content-bearing files): PEM private-key headers, `ssh-rsa` /
`ssh-ed25519` key material, `Bearer` tokens, `api_key=`, `passphrase=`, `customer_id=`, `password=`,
and `token=` with ≥12-character values. The only hits were **empty variable initialisations** in
`felhom-bootstrap.sh:89` (`FELHOM_CUSTOMER_ID=""; FELHOM_MODE=""; FELHOM_RETRIEVAL_PASSPHRASE=""`) and
a substring test at `:480` — declarations, not values.

**Conclusion for the recommended shape.** The earlier recon's "three payloads, secret-free" finding
holds *for the payloads it examined* and was scanning the wrong file for the credential question:
`/answer.toml` is a fourth, separate artefact on the ISO root and is the only secret-bearing content in
a pairing-mode build. In a universal image the entire question reduces to two fields in that one file
— `root-password-hashed` and `root-ssh-keys` — both of which need a build-time assertion rather than a
convention.

---

## 7. The recommended shape of a universal ISO

Stated as constraints on an eventual build spec. **Not an implementation, and nothing here was built.**

1. **Disk selection cannot be solved by a filter, and must not be attempted.** §3.2 measured that no
   property expresses removability or role, that `ID_BUS` fails on the exact hazard configuration, and
   that NVMe carries a different property set. Any filter shipped to unknown hardware is a guess.
   → **Constraint: a universal image must not choose a disk by property on hardware it has not seen.**

2. **Ambiguity must fail closed, and the mechanism must be the one measured in §3.3** — not a filter
   the schema has no keyword for.

3. **The interactive installer must be reachable from the boot menu** (§5). This is the constraint
   that makes the image universal in the strong sense: unattended where it is provably safe, and never
   a dead end where it is not — including on hardware nobody anticipated, without a code change. It
   costs one entry in `grub/grub.cfg.tmpl`.

4. **No usable credential may be baked.** `root-password-hashed = "*"` is schema-valid (§4.1); the
   field cannot be omitted but a usable password can be denied. Bake a locked account.

5. **Rotation must precede reachability.** `ordering = "before-network"` is the only ordering that
   closes the window structurally (§4.3). Anything after `network-online.target` — including
   everything the current bootstrap does — is open by construction, and in pairing mode open
   indefinitely.

6. **Two build-time assertions, because both failures are silent and checkable:** the built image
   carries no usable `root-password-hashed`, and carries no `root-ssh-keys` at all (§4.6.4). A comment
   is not a guarantee; these are one-line greps over the rendered answer.

7. **The published artefact must be reproducible from committed inputs.** `nested-probe`'s profile no
   longer exists anywhere, so that ISO cannot be audited or rebuilt — a state a published image must
   never reach.

---

## 8. What is still unknown, and what it would take

Stated plainly rather than inferred. Several of these were reachable and were not reached; that is a
budget outcome, not a judgement that they do not matter.

| # | Unknown | What it would take |
|---|---|---|
| 1 | **Whether the multi-match pick is deterministic by enumeration order or arbitrary.** §3.3 tested one disk ordering and `SPIKE0` was chosen. "Always the first-enumerated" and "arbitrary" have the same evidence so far. | Re-run S1 with the two disks attached in reverse order, and a third run with differing sizes. ~15 min. Does not change the recommendation (either way it is unsafe), but it changes how the hazard is described to a tester. |
| 2 | **Whether the installer excludes the live medium it booted from.** §3.4 established only that USB block devices *are* enumerated as candidates. | Boot the installer from a USB stick on a real Tier 0 box with a filter that would match the stick. Needs physical presence — HUMAN. |
| 3 | **Whether the Felhom first-boot stub survives `ordering = "before-network"`.** This is the highest-value follow-up: it is the mechanism that would make the exposure window structurally zero (§4.3). The stub *appears* not to need the network, but "appears" is exactly the word this project has been burned by. | Build one probe ISO with `ordering = "before-network"` and the real stub, boot a nested VM, confirm the bootstrap unit still starts and pairing still completes. ~30 min. **Do this before any spec is written.** |
| 4 | **Whether GRUB single-user recovery works on a locked-root install** (Q2.4). Not tested. This is load-bearing for the discard-vs-escrow ruling: if `init=/bin/sh` from the GRUB menu gives a root shell, discarding is far cheaper than if it does not. | Boot the S2 disk image, edit the kernel line at the GRUB prompt, observe. Automatable via QMP `sendkey`. ~20 min. |
| 5 | **The two-entry menu is not demonstrated by boot** (§5). The evidence is configuration-level: `prepare-iso` provably does not touch `grub.cfg`, and the stock config provably contains both entries. A booted screenshot would be stronger. | Boot the probe ISO in a nested VM and screendump the menu. ~10 min. |
| 6 | **Whether `root-password-hashed = ""` produces a passwordless root or a locked one.** It validates (§4.1); what it writes to `/etc/shadow` was not observed. | One install + offline read. Low priority — nothing should ever set it — but it is a trap sitting in the schema. |

---

## 9. R-rows opened

Numbering continues from the highest in use (R-138).

| R | Severity | Finding |
|---|---|---|
| **R-139** | **HIGH** | **A disk filter that matches more than one device does not fail safe — the installer silently picks one and wipes it** (§3.3, observed). The `filter did not match any devices` guard covers the zero-match case only. `validate-answer` cannot catch it. This invalidates "write a careful filter" as a strategy for unknown hardware, and it is the reason a universal ISO cannot select a disk unattended. |
| **R-140** | MEDIUM | `proxmox-auto-install-assistant device-match` **prints `Error: … filter did not match any devices` and exits 0.** Joins `validate-answer` (spike S1) in the exit-codes-that-lie class. Any tooling built on `device-match` must parse output, never `$?`. |
| **R-141** | **HIGH** | **The answer schema makes a root credential mandatory** (§4.1), so every ISO ships one. On a *published* image that is a fleet-wide shared credential, and in pairing mode the window does not close on its own: bootstrap is ordered after `network-online.target`, and the pairing poll waits indefinitely (§4.3). A box that is never bound keeps the download's password forever. |
| **R-142** | MEDIUM | **`FELHOM_ROOT_SSH_KEY` will bake a shared SSH key into every copy of a published image, and nothing prevents it.** No committed profile sets it and the v1.25.0 build contains none — but an uncommitted profile already did exactly this on `demo-felhom` (§4.6). Same shared-credential class as R-141; needs a build-time assertion over the rendered answer, not a convention. |
| **R-143** | LOW | **A retired credential is still live:** the `kisfenyo@windows` key (the decommissioned Windows workstation, §4.6) remains in `demo-felhom`'s `authorized_keys`. |
| **R-144** | MEDIUM | **`felhom-pve-9.2-1-v1.24.0-nested-probe-generic.iso` cannot be reproduced or audited** — no `nested-probe` profile exists anywhere in the repo or the build tree. A published artefact must never reach this state. |
| **R-145** | LOW | **The ISO variant naming convention is undocumented** (carried from the 2026-07-31 Part 0). `out/` holds 17 images whose labels mix hardware targets, lab purposes, a pairing-mode suffix and a loader suffix, with no written key. §3 and §5 of this document supply the meanings; they need a home in the ISO README. |
| **R-146** | LOW | **`root-password-hashed = ""` passes validation** (§4.1). An empty hash is not a locked account. Worth a fence wherever answers are generated. |
| **R-147** | LOW | The answer renderer substitutes `__ROOT_HASH__`/`__FQDN__` **inside the template's own comment lines** (`build-felhom-iso.sh:295-296` is line-oriented and unconditional), so the shipped `/answer.toml` repeats the real hash three times, twice in prose that was meant to be documentation. Cosmetic while the hash is on the ISO anyway; it stops being cosmetic if the credential ever becomes conditional. |
| **R-128** | *(open, restated)* | `build-felhom-iso.sh:44`'s claim that `ISO_VERSION` "aligns with felhom-host-install `SCRIPT_VERSION`" **is a comment, not an assertion.** `SCRIPT_VERSION` appears nowhere in `scripts/iso/` except inside that comment's own text, so nothing evaluates it; the values have drifted (1.25.0 vs 1.22.0). Instance #7 of the "a comment asserting an invariant needs a test pinning it" class. |

---

## 10. Teardown

All three layers, per `PROMPT-TEMPLATE.md` §13, each verified by a positive check rather than asserted.

**Layer 1 — demo-hp (nested-VM host).**

| Check | Result |
|---|---|
| QEMU processes | `0` remaining |
| `/mnt/nvme-1tb/spike-scratch/` (ISO, disk images, answer disk, probe scripts) | removed — `No such file or directory` |
| `/tmp/paia` (temporary assistant binary) | removed |
| loop devices | `0` |
| `/mnt/nvme-1tb/` contents | back to `dump e2d-images felhom-data images lost+found` — the pre-spike set |
| `drill-r50` (VM 300) | `stopped` — untouched, not destroyed (R-93 fixture intact) |
| guest 9201 | `running` — untouched |
| `/etc/pve/storage.cfg` | 4 storages, unchanged; **no scratch storage was ever registered** |
| `local-lvm` | never written to |

**Layer 2 — demo-felhom (read-only probe host).**

| Check | Result |
|---|---|
| `/tmp/paia` | removed |
| guest 9201 | `running` — untouched |
| external USB drive `sdb` | `931.5G usb /mnt/hdd_1` — still attached and mounted exactly as found; never attached, detached, partitioned or written |

**Layer 3 — DooPlex (build host).**

| Check | Result |
|---|---|
| scratchpad | 3.2 GB → 28 K (probe ISO, extracted ISO trees, PPM/PNG screendumps, file lists removed) |
| throwaway root password file | `shred -u`'d |
| `felhom-iso/out/` | **17 ISOs, nothing deleted** (fence honoured) |
| repo working tree | clean apart from this document |
| production files | none modified — no `.go`, no `build-felhom-iso.sh`, no profile |

**Hub-side disposition — nothing to dispose.** This spike created no customer, no host, no appliance
registration: the nested VMs ran under plain QEMU with no network path to the hub, and the only hub
interaction was two authenticated **reads** of `/configuration` and `/configs`. Verified positively
rather than asserted — the customer list was fetched and searched for `spike|probe|scratch`-named
entries (none) and carries the same single pre-existing customer. **R-131 does not gain a row here.**

**Published artefacts: none.** Nothing was uploaded to R2 or anywhere else. The R2 credential was
never created, used, or handled in this session.