felhom.eu/documentation/audits/SPIKE-universal-iso-2026-07-31.md

SPIKE — what does a genuinely universal ISO require? (2026-07-31)

Class: Spike. Output is findings, not shipped code. No production file was changed, no release ISO was built, nothing was published. Probe artefacts are quoted here and were removed at teardown.

Origin. The ISO build/publish task (2026-07-31) stopped before its Part 2 when its Part 0 found two hazards that make a public download a product decision rather than a build step. This spike establishes what a genuinely universal image would require.

---

1. Baselines

Read fresh at the start of this session, not carried from the previous one.

Baseline	Value	Source
ISO_VERSION	1.25.0	scripts/iso/build-felhom-iso.sh:44
SCRIPT_VERSION	1.22.0	scripts/felhom-host-install.sh:187
felhom-bootstrap.sh @ HEAD	21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7	sha256sum scripts/iso/felhom-bootstrap.sh
PVE base ISO	proxmox-ve_9.2-1.iso, 4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c	sha256sum /mnt/5_hdd/felhom.eu/drill/proxmox-ve_9.2-1.iso — matches every ISO manifest's source-iso-sha256
felhom.eu HEAD	5825ceeabfc50227bee2ccf9cf35c412c05c0f2a, clean, == origin/main	git status --porcelain empty
assistant tooling	proxmox-installer-common v9.2.7	proxmox-auto-install-assistant --version

Fixture provenance. Nested VMs ran under plain QEMU/KVM on demo-hp (t740, Tier 0, the designated drill + build VM host per the 2026-07-25 standing ruling), with all scratch files under /mnt/nvme-1tb/spike-scratch/. Nothing was placed on demo-hp's local-lvm and no PVE storage was registered. drill-r50 (VM 300) and guest 9201 were not touched. The probe ISO was built on DooPlex from the baseline PVE ISO with prepare-iso --fetch-from partition and transferred once (sha256 337985c5c88b8d76c7a0a6e1678b48063f3b7d077222c91b9a1a38cfb6ca8c75, verified identical at both ends), so every scenario reused one image and varied only a 64 MB answer disk.

Real-hardware work was read-only. device-info was executed on demo-felhom and demo-hp via a temporary copy of the assistant binary in /tmp (removed at teardown). No install, no partitioning, no attach/detach.

---

2. Reconfirmation of the two established findings

Both re-verified at file:line rather than carried forward.

(a) Every ISO bakes a root password hash. build-felhom-iso.sh:270-283 mints ROOT_PLAIN unconditionally, hashes it with openssl passwd -6, and the renderer substitutes it into __ROOT_HASH__ (:295-296) from answer.toml.tmpl:24. There is no flag to suppress it.

Verified by extraction, not inference: osirrox pulled /answer.toml out of the shipped felhom-pve-9.2-1-v1.25.0-nested-vm-generic-mkimage.iso and it carries a live root-password-hashed field. The sibling .rootpw.txt plaintext was confirmed to hash to exactly that value. Any downloader can perform the same extraction with no privilege.

(b) generic.profile selects by name, not identity. profiles/generic.profile:16-18 sets disk-list = ["sda"]. Confirmed present verbatim in the extracted /answer.toml of the v1.25.0 build. generic.profile:12-14 concedes the limit in its own comment. The serial pin that closed this hazard on the N100 rehearsal (hp.profile:36) does not generalise to strangers' hardware.

---

3. Q1 — what the answer file can express about disk selection

3.1 The selection mechanisms (read from schema)

Extracted from the assistant binary's own serde field strings and confirmed against validate-answer behaviour. [disk-setup] is struct DiskSetup with 7 elements:

Key	Purpose
filesystem	ext4 / xfs / zfs / btrfs
disk-list	explicit device names
filter	map of udev-property globs
filter-match	any | all
lvm / zfs / btrfs	per-filesystem option sub-tables

The binary's validation strings pin the rules:

Need either 'disk-list' or 'filter' set
Cannot use both, 'disk-list' and 'filter'
make sure to define only one disk for ext4 and xfs
filter did not match any devices
no filter defined
invalid glob in disk selection

Glob syntax is ?, *, [a], [0-9], [!a] (device-match --help). [!…] negates a single character, not a match — there is no "does not contain" and no "property is absent" operator.

There is no exclusivity keyword. Nothing in the 7 elements expresses "abort unless exactly one candidate matches". The nearest thing is the only one disk for ext4 and xfs rule — a consequence of the filesystem choice, not a selection control (measured behaviour in §3.3).

3.2 What distinguishes an internal system disk from external media — measured

device-info -t disk, captured on both Tier 0 boxes.

demo-felhom — the hazard configuration (internal SATA SSD + the 1 TB external USB drive):

	sda (internal)	sdb (external USB)
ID_BUS	ata	ata
ID_PATH	pci-0000:00:17.0-ata-2.0	pci-0000:00:14.0-usb-0:1:1.0-scsi-0:0:0:0
ID_MODEL	AirDisk_512GB_SSD	TOSHIBA_MQ04ABF100
ID_USB_DRIVER	absent	usb-storage
ID_USB_VENDOR	absent	ADATA
lsblk RM (removable)	0	0
lsblk HOTPLUG	0	1

Three findings, and the first two are the important ones:

1. ID_BUS does not distinguish them. Both report ata. The external unit is a SATA drive in a USB caddy (an ADATA HD710 PRO enclosure around a Toshiba MQ04ABF100), so the bus property describes the drive, not the attachment. A filter written as "internal means ID_BUS=ata" matches the customer's backup drive.
2. There is no removability property at all in device-info output — no RM, no ID_REMOVABLE. And lsblk reports RM=0 for the external drive anyway, so even reaching outside device-info does not supply the missing concept. Property filtering cannot express "not removable."
3. What does distinguish them here is ID_PATH (-ata- vs -usb-) and the presence of the ID_USB_* key family — but only the former is expressible, because the filter language has no "key is absent" operator.

demo-hp (internal SATA system disk + a 1 TB NVMe data disk):

	sda (SATA, the PVE system disk)	nvme0n1 (NVMe, enrolled user data)
property count	45	28
ID_BUS	ata	absent
ID_TYPE	disk	absent
ID_PATH	pci-0000:06:00.0-ata-1.0	pci-0000:04:00.0-nvme-1

Absence verified explicitly by key lookup, not by a grep that happened not to print.

4. NVMe devices carry a different property set entirely — no ID_BUS, no ID_TYPE. Any filter built on ID_BUS silently excludes every NVMe-only machine.
5. Decisively: no property expresses role. Every captured property describes hardware — bus, path, model, serial, size. None says "this is the disk the operating system belongs on." On demo-hp the system disk is the small SATA one and the big NVMe is user data; a heuristic like "largest disk" or "the NVMe one" targets the customer's data drive on exactly the configuration Felhom sells.

device-match confirmed the expressible cases on demo-felhom (read-only):

ID_BUS=ata          -> ["sda","sdb"]     both — the naive filter hits the backup drive
ID_PATH=*usb*       -> ["sdb"]           isolates the external
ID_PATH=pci-*-ata-* -> ["sda"]           isolates the internal, on this box

Both isolating filters are box-specific: the -ata- pattern excludes every NVMe system disk, and the *usb* pattern is an exclusion that the filter language cannot invert.

3.3 Multi-match behaviour — observed. It is the coin-flip.

This is the question the spike existed to answer, and it was measured, not inferred.

Method. A nested VM on demo-hp with two identical 16 GB virtio-SCSI disks carrying serials SPIKE0 and SPIKE1, plus a separate answer disk (serial ANSWERDISK, label proxmox-ais). Answer:

[disk-setup]
filesystem = "ext4"
filter-match = "all"
filter.ID_SERIAL_SHORT = "SPIKE*"      # matches SPIKE0 and SPIKE1 — two devices

The answer validated cleanly (The answer file was parsed successfully, no errors found!) — so validate-answer cannot catch this; it does not know what the filter will resolve to on the target.

The measurement is deliberately not a screen reading: both disk images were zeroed and hashed before the run and re-hashed after, so "which disk was written" is objective.

Disk	sha256 (first 32) BEFORE	AFTER	Verdict
disk0.qcow2 (SPIKE0)	de001102cbc8ff7aa8eb6c69d750cf22	382b55cad8027c60dfcbed254e723123	WIPED AND INSTALLED ONTO
disk1.qcow2 (SPIKE1)	de001102cbc8ff7aa8eb6c69d750cf22	de001102cbc8ff7aa8eb6c69d750cf22	untouched

Observed mid-run, confirming it was a real install and not a partial write: disk0.qcow2 had grown to 5,441,388,544 bytes while disk1.qcow2 sat at 196,864 bytes (bare qcow2 metadata).

The installer did not abort. It silently selected one of the two matching disks and wiped it.

Two consequences:

1. filter-match has no fail-safe on ambiguity. The filter did not match any devices guard — the spike-S5c behaviour the nested-canary profile pins, and the safety default the hp/n100 profiles rely on — protects only the zero-match case. The many-match case is unprotected and silent.
2. The binary's make sure to define only one disk for ext4 and xfs string does not cover this. It rejects an explicit multi-entry disk-list; a filter that resolves to several devices sails past both validate-answer and the installer.

Why this matters more than the disk-list=["sda"] hazard. The previously known hazard is that sda may name the wrong disk. This one is worse in kind: a filter written specifically to be safe — "match the internal SSD" — degrades to an arbitrary pick the moment a second device matches, with no diagnostic. Given §3.2 (no removability property, ID_BUS identical on the hazard configuration), writing a filter that provably matches exactly one device on unseen hardware is not achievable.

Not established: whether the selection is deterministic by enumeration order or genuinely arbitrary. Only one ordering was tested. See §8.

3.4 Is the boot medium a candidate?

Partially measured. On demo-felhom the USB-attached drive is enumerated by device-info as an ordinary candidate disk with no marking distinguishing it from the internal one — a USB installer stick is the same class of block device, so nothing in the captured data suggests the enumeration would exclude it.

Not fully established. Whether the installer itself excludes the live medium it booted from is a separate question from whether device-info enumerates it, and this spike did not boot from USB on real hardware. See §8.

---

4. Q2 — can the root password be generated on the box instead of baked?

4.1 Is a password required? — measured

The binary carries both rules verbatim:

One of `global.root-password` or `global.root-password-hashed` must be set
`global.root-password` and `global.root-password-hashed` cannot be set at the same time

Confirmed empirically against validate-answer:

Answer	Result
root password omitted entirely	rejected — One of global.root-password or global.root-password-hashed must be set
root-password-hashed = "*"	accepted
root-password-hashed = "!"	accepted
root-password-hashed = ""	accepted
both plaintext and hashed set	rejected

A root password field cannot be omitted — but the classic locked-account crypt placeholders * and ! validate. That is the opening: the field is mandatory, a usable credential is not.

That an empty string also validates is worth a fence of its own — "" is not a locked account, and whether it yields a passwordless root depends on how it reaches /etc/shadow.

Validation is not installation, so it was installed — and the first attempt to confirm that is worth recording, because it nearly produced a false finding.

S2, first attempt. A nested VM ran an unattended install whose only credential was root-password-hashed = "*". disk0.qcow2 went from de001102cbc8ff7aa8eb6c69d750cf22 to f4ae86b8317cc36af4628a6cd9c09da2 and grew to several GB, which reads like success. Booting the result showed otherwise: SeaBIOS printed Booting from Hard Disk... and hung. The install had been cut off mid-flight by the probe's fixed 300 s window; a written disk is evidence of writing, not of completion. A changed hash is a LastRun-class timestamp — it records that something was attempted.

S2b, with an actual completion signal. reboot-mode = "power-off" is a valid enum value ("poweroff" is rejected, so this is a real enum), and it makes the guest power itself off when the install finishes — i.e. the qemu process exiting is the completion signal, and a timeout is unambiguously a failure rather than an unknown.

Disk	sha256 (first 32) BEFORE	AFTER	Completion signal
disk0.qcow2	de001102cbc8ff7aa8eb6c69d750cf22	dbc12e129ed84e834f6d54754cb0a2c8	guest powered itself off after 200 s

Established: an unattended install whose only credential is root-password-hashed = "*" runs to completion. The mandatory-field constraint does not force a usable baked credential — it forces a field, and a locked-account placeholder satisfies it. This is the mechanism a universal image needs.

Not established: that the resulting root account is actually locked. The install completing is not the same claim. Confirming it needs either /etc/shadow from the installed image or a refused login, and neither was obtained — see §4.3 and §8.

4.2 Can felhom-bootstrap rotate root at first boot?

Mechanically yes — it already runs as root at first boot, and felhom-host-install.sh:1831-1871 (step 4b, G1) already implements exactly this: generate 24 url-safe bytes, set via chpasswd on stdin, vault to the hub, scrub the variable. The code to rotate exists and is proven.

The problem is not capability, it is ordering — see §4.3.

4.3 The exposure window — structural finding, and it is not zero

felhom-bootstrap.service is ordered:

After=network-online.target pve-cluster.service pveproxy.service
Wants=network-online.target

So bootstrap starts after the network is up and after pveproxy is listening. Whatever it does to the root password happens strictly later than the box becoming reachable. The window is open by construction.

Worse, in pairing mode it does not close on its own. felhom-bootstrap.sh:9-22,52-54,363 registers the box as an unclaimed appliance and then polls indefinitely (POLL_INTERVAL=30) for the operator's bind, deliberately staying in activating because "waiting is not failing" (v1.21.0, R-33). Step 4b only runs after enroll, and enroll only happens after the bind. A box that is downloaded, installed, and never bound sits on its owner's LAN with the ISO-baked password indefinitely.

The measurement the task asked for was attempted twice and BOTH ATTEMPTS FAILED. No number is reported, because both numbers were artefacts of the fixture rather than facts about the system.

Attempt 1 — invalid, and it produced a plausible false zero. The probe booted the installed disk under QEMU with hostfwd port forwards and polled with a TCP connect. Both ports "accepted" at +0 s, which reads like "reachable instantly". It is not: QEMU's user-mode networking binds the forwarding socket on the host when QEMU starts and accepts the connection before it ever tries the guest. The same run then failed to read an SSH banner and reported Connection timed out during banner exchange — i.e. nothing was listening in the guest at all. A TCP connect against a hostfwd port proves nothing. Had the banner check not been in the same script, 0 s would have been recorded as a measured finding, and it would have been the exact opposite of the truth.

Attempt 2 — invalid for a different reason. The detector was rewritten to require a real SSH- version banner and a real HTTP response line. It then reported never for both, over 6 minutes. That is also not a fact about PVE: the S2b install VM was created with no network device at all, so the installed system's interface configuration cannot match the NIC that only exists in the boot VM. The fixture, not the system, produced the result.

What stands instead is the structural finding above, which does not depend on either probe: the unit's own After=network-online.target pveproxy.service places every rotation opportunity strictly after reachability, and the pairing loop makes the interval unbounded. That is a stronger claim than a single timing number would have been — it holds for every box rather than for one VM.

What it would take to measure it properly: install and boot with an identical NIC present in both phases, and probe at protocol level from the host. ~30 min. It is worth doing before a spec is written, because the number is what tells the operator whether the window is seconds or minutes when pairing does proceed promptly.

The mechanism that would close it: [first-boot].ordering accepts before-network as well as fully-up — verified against validate-answer (network-pre and a bogus value are both rejected, so this is a real enum, not a permissive parser). A hook running before-network executes before the box is reachable at all, which is what a zero-window rotation requires.

Felhom currently uses ordering = "fully-up" (answer.toml.tmpl:39), deliberately, so that pvesh and pct work (spike S3/S8a). Note the current stub (stub-first-boot.sh) only writes three files and runs systemctl enable + start --no-block — none of which obviously needs the network, and the unit it starts carries its own After=network-online.target. Whether the stub survives being moved to before-network is untested and is the single highest-value follow-up probe. [first-boot] accepts only one hook (struct FirstBootHookInfo with 4 elements), so this is a move, not an addition.

4.4 Console recovery if pairing fails

Not tested — stated plainly rather than assumed. The spike ran out of room before booting the S2b image into a GRUB single-user / init=/bin/sh shell.

This is not a detail. It is the input that decides §4.5: if the standard GRUB recovery path yields a root shell on a locked-root install, then discarding the password costs little, because a physically present person can always get back in. If it does not — or if PVE's installed GRUB is configured without an editable menu — then discarding turns a failed pairing into a reinstall, and the escrow argument is much stronger. The ruling in §4.5 should not be made before this is measured (§8, #4).

4.5 Escrow or discard — the tradeoff, for the operator to rule

Not a recommendation; the inputs.

For discarding (no per-box root password is ever recoverable):

• The operator already holds agent-level access on every bound box, so console root is redundant for the normal case.
• Under the D6 trust model the hub alone must not be sufficient. A vault of console root passwords for the whole fleet makes the hub exactly that, for every box at once — a strictly worse blast radius than the shared-ISO-password problem this is meant to fix.
• It removes a class of credential that has to be rotated, audited, and leak-checked forever.

For escrowing (the current G1 behaviour, extended to pre-bind):

• An unbound box has no agent access — precisely the case where console access is the only route in, and precisely when things go wrong.
• G1 exists because the sshd path and the auto-heal both failing is a real, observed failure mode; the PVE web console at :8006 is a distinct failure domain from sshd.
• Discarding turns "pairing failed" into a reinstall for a non-technical customer.

The middle option worth costing: rotate to a per-box random value at before-network and escrow nothing until the box binds, at which point step 4b's existing vaulting takes over. This closes the shared-credential window without adding a pre-bind fleet-wide vault — at the price of making an unbound, broken box unrecoverable except by reinstall. Whether that price is acceptable is the operator's call, and it is the one decision this spike asks for.

4.6 R-129 — SSH key provenance, resolved

The premise is confirmed: ssh -o BatchMode=yes demo-hp authenticates by key, while the documentation says no baked key exists. The two are not in conflict — the key is not baked.

Box	/root/.ssh/authorized_keys → /etc/pve/priv/authorized_keys	Provenance
demo-hp	SHA256:pgQh228R… nagyfenyvesi.viktor@gmail.com	the operator's own DooPlex key, added post-install (file mtime 2026-07-27, install 2026-07-21). This is what authenticates CC.
	SHA256:fnasIK1Y… root@felhom-host (RSA 4096)	PVE's own self-generated root key
demo-felhom	SHA256:a8zH+ecm… kisfenyo@windows	baked by the ISO — exact fingerprint match to FELHOM_ROOT_SSH_KEY in the uncommitted n100-demo.profile
	SHA256:BnLVXqTc… root@demo-felhom (RSA)	PVE self-key
	SHA256:pgQh228R…	the operator's DooPlex key

Findings:

1. demo-hp's key is not baked — it is the operator's own, added by hand. The documentation was right about the ISO and wrong about the box.
2. demo-felhom's key is baked, via an uncommitted profile. So the "no baked key" claim was already false for one of the two demo boxes, and nothing recorded it.
3. No committed profile bakes a key — all five have FELHOM_ROOT_SSH_KEY commented out — and the shipped v1.25.0 ISO contains zero occurrences of root-ssh-keys in its /answer.toml.
4. But the mechanism is one uncommented line from shipping a shared key to every downloader (build-felhom-iso.sh:175-176 → answer.toml.tmpl:26). For a published image that is the same shared-credential class as the baked password, and it needs a build-time assertion, not a convention.
5. kisfenyo@windows is the retired Windows workstation's key and is still authorized on demo-felhom. Stale credential; see R-rows.

---

5. Q3 — can the image offer both automated and interactive install?

Yes, and it requires no PVE-side work at all. Established by comparing three GRUB configs.

prepare-iso does not touch grub.cfg: the file extracted from the freshly prepared probe ISO is byte-identical to the one in the stock PVE ISO (diff clean). The stock config already carries the automated entry, gated on a file that prepare-iso creates:

if [ -f auto-installer-mode.toml ]; then
    set timeout-style=menu
    set timeout=10
    menuentry 'Install Proxmox VE (Automated)' { … proxmox-start-auto-installer }
fi
menuentry 'Install Proxmox VE (Graphical)'    { … }   # outside the conditional
menuentry 'Install Proxmox VE (Terminal UI)'  { … }   # outside the conditional

ISO	menu entries
stock proxmox-ve_9.2-1.iso	15
prepared (probe, prepare-iso only)	15 — automated and graphical and TUI
Felhom v1.25.0 remastered	1 (Felhom telepítés)

So the answers are:

1. The current remastering replaces the menu. iso-repack.sh rewrites grub.cfg from grub/grub.cfg.tmpl to exactly one entry — by design, and documented as such (build-felhom-iso.sh:16-23: the stock entries "are not hidden, they are not emitted"). This is also the answer to the previous session's open question: the mkimage/non-mkimage split is not this distinction — it is a UEFI loader swap, orthogonal to the menu, and both variants get the same single-entry menu.
2. Preparation does not consume the interactive path. A multi-entry menu including the automated entry is the native shape of a prepared auto-install ISO. Felhom collapses it deliberately, for a reason that remains sound for a customer appliance (build-felhom-iso.sh:18-21: every stock entry is a route into the manual installer whose first question is which disk to wipe).
3. Therefore a two-entry menu is purely a change to Felhom's own grub/grub.cfg.tmpl — keeping Felhom telepítés as default=0 and adding one guarded interactive entry. No PVE-side obstacle exists.

Not demonstrated by boot. The evidence here is configuration-level, which is weaker than a screenshot of the menu. See §8.

---

6. Q4 — what else is baked that a stranger should not receive

Answered by positive enumeration against the stock ISO, not by grepping for patterns I guessed — a pattern scan can only find what it was told to look for.

Method. Full recursive file listing of the shipped v1.25.0 ISO and the baseline PVE ISO (osirrox -find, 1576 vs 1573 paths), diffed both directions; content hashes compared for the OS payloads; all three first-boot payloads decoded and hashed against repo HEAD; the answer file's credential fields enumerated by name.

Added by the Felhom pipeline — exactly six paths:

Path	Content	Secret-bearing?
/answer.toml	the rendered answer	YES — root-password-hashed
/auto-installer-mode.toml	mode = "iso"	no
/proxmox-first-boot	the rendered stub + 3 embedded payloads	no (below)
/boot/grub/felhomtheme/ + background.png + theme.txt	branding	no

Removed: the three stock /boot/grub/pvetheme/ paths.

The OS payloads are untouched — pve-base.squashfs and pve-installer.squashfs are byte-identical to the stock ISO. Nothing is hidden in the installed system image.

The three first-boot payloads, decoded from the stub:

Payload	Result
felhom-bootstrap.sh	sha256 identical to repo HEAD
felhom-bootstrap.service	sha256 identical to repo HEAD
pairing env	only FELHOM_HUB_URL + FELHOM_INSTALL_URL — no customer id, no passphrase

Also scanned for (across all four content-bearing files): PEM private-key headers, ssh-rsa / ssh-ed25519 key material, Bearer tokens, api_key=, passphrase=, customer_id=, password=, and token= with ≥12-character values. The only hits were empty variable initialisations in felhom-bootstrap.sh:89 (FELHOM_CUSTOMER_ID=""; FELHOM_MODE=""; FELHOM_RETRIEVAL_PASSPHRASE="") and a substring test at :480 — declarations, not values.

Conclusion for the recommended shape. The earlier recon's "three payloads, secret-free" finding holds for the payloads it examined and was scanning the wrong file for the credential question: /answer.toml is a fourth, separate artefact on the ISO root and is the only secret-bearing content in a pairing-mode build. In a universal image the entire question reduces to two fields in that one file — root-password-hashed and root-ssh-keys — both of which need a build-time assertion rather than a convention.

---

7. The recommended shape of a universal ISO

Stated as constraints on an eventual build spec. Not an implementation, and nothing here was built.

1. Disk selection cannot be solved by a filter, and must not be attempted. §3.2 measured that no property expresses removability or role, that ID_BUS fails on the exact hazard configuration, and that NVMe carries a different property set. Any filter shipped to unknown hardware is a guess. → Constraint: a universal image must not choose a disk by property on hardware it has not seen.

2. Ambiguity must fail closed, and the mechanism must be the one measured in §3.3 — not a filter the schema has no keyword for.

3. The interactive installer must be reachable from the boot menu (§5). This is the constraint that makes the image universal in the strong sense: unattended where it is provably safe, and never a dead end where it is not — including on hardware nobody anticipated, without a code change. It costs one entry in grub/grub.cfg.tmpl.

4. No usable credential may be baked. root-password-hashed = "*" is schema-valid (§4.1); the field cannot be omitted but a usable password can be denied. Bake a locked account.

5. Rotation must precede reachability. ordering = "before-network" is the only ordering that closes the window structurally (§4.3). Anything after network-online.target — including everything the current bootstrap does — is open by construction, and in pairing mode open indefinitely.

6. Two build-time assertions, because both failures are silent and checkable: the built image carries no usable root-password-hashed, and carries no root-ssh-keys at all (§4.6.4). A comment is not a guarantee; these are one-line greps over the rendered answer.

7. The published artefact must be reproducible from committed inputs. nested-probe's profile no longer exists anywhere, so that ISO cannot be audited or rebuilt — a state a published image must never reach.

---

8. What is still unknown, and what it would take

Stated plainly rather than inferred. Several of these were reachable and were not reached; that is a budget outcome, not a judgement that they do not matter.

#	Unknown	What it would take
1	Whether the multi-match pick is deterministic by enumeration order or arbitrary. §3.3 tested one disk ordering and SPIKE0 was chosen. "Always the first-enumerated" and "arbitrary" have the same evidence so far.	Re-run S1 with the two disks attached in reverse order, and a third run with differing sizes. ~15 min. Does not change the recommendation (either way it is unsafe), but it changes how the hazard is described to a tester.
2	Whether the installer excludes the live medium it booted from. §3.4 established only that USB block devices are enumerated as candidates.	Boot the installer from a USB stick on a real Tier 0 box with a filter that would match the stick. Needs physical presence — HUMAN.
3	Whether the Felhom first-boot stub survives ordering = "before-network". This is the highest-value follow-up: it is the mechanism that would make the exposure window structurally zero (§4.3). The stub appears not to need the network, but "appears" is exactly the word this project has been burned by.	Build one probe ISO with ordering = "before-network" and the real stub, boot a nested VM, confirm the bootstrap unit still starts and pairing still completes. ~30 min. Do this before any spec is written.
4	Whether GRUB single-user recovery works on a locked-root install (Q2.4). Not tested. This is load-bearing for the discard-vs-escrow ruling: if init=/bin/sh from the GRUB menu gives a root shell, discarding is far cheaper than if it does not.	Boot the S2 disk image, edit the kernel line at the GRUB prompt, observe. Automatable via QMP sendkey. ~20 min.
5	The two-entry menu is not demonstrated by boot (§5). The evidence is configuration-level: prepare-iso provably does not touch grub.cfg, and the stock config provably contains both entries. A booted screenshot would be stronger.	Boot the probe ISO in a nested VM and screendump the menu. ~10 min.
6	Whether root-password-hashed = "" produces a passwordless root or a locked one. It validates (§4.1); what it writes to /etc/shadow was not observed.	One install + offline read. Low priority — nothing should ever set it — but it is a trap sitting in the schema.

---

9. R-rows opened

Numbering continues from the highest in use (R-138).

R	Severity	Finding
R-139	HIGH	A disk filter that matches more than one device does not fail safe — the installer silently picks one and wipes it (§3.3, observed). The filter did not match any devices guard covers the zero-match case only. validate-answer cannot catch it. This invalidates "write a careful filter" as a strategy for unknown hardware, and it is the reason a universal ISO cannot select a disk unattended.
R-140	MEDIUM	proxmox-auto-install-assistant device-match prints Error: … filter did not match any devices and exits 0. Joins validate-answer (spike S1) in the exit-codes-that-lie class. Any tooling built on device-match must parse output, never $?.
R-141	HIGH	The answer schema makes a root credential mandatory (§4.1), so every ISO ships one. On a published image that is a fleet-wide shared credential, and in pairing mode the window does not close on its own: bootstrap is ordered after network-online.target, and the pairing poll waits indefinitely (§4.3). A box that is never bound keeps the download's password forever.
R-142	MEDIUM	FELHOM_ROOT_SSH_KEY will bake a shared SSH key into every copy of a published image, and nothing prevents it. No committed profile sets it and the v1.25.0 build contains none — but an uncommitted profile already did exactly this on demo-felhom (§4.6). Same shared-credential class as R-141; needs a build-time assertion over the rendered answer, not a convention.
R-143	LOW	A retired credential is still live: the kisfenyo@windows key (the decommissioned Windows workstation, §4.6) remains in demo-felhom's authorized_keys.
R-144	MEDIUM	felhom-pve-9.2-1-v1.24.0-nested-probe-generic.iso cannot be reproduced or audited — no nested-probe profile exists anywhere in the repo or the build tree. A published artefact must never reach this state.
R-145	LOW	The ISO variant naming convention is undocumented (carried from the 2026-07-31 Part 0). out/ holds 17 images whose labels mix hardware targets, lab purposes, a pairing-mode suffix and a loader suffix, with no written key. §3 and §5 of this document supply the meanings; they need a home in the ISO README.
R-146	LOW	root-password-hashed = "" passes validation (§4.1). An empty hash is not a locked account. Worth a fence wherever answers are generated.
R-147	LOW	The answer renderer substitutes __ROOT_HASH__/__FQDN__ inside the template's own comment lines (build-felhom-iso.sh:295-296 is line-oriented and unconditional), so the shipped /answer.toml repeats the real hash three times, twice in prose that was meant to be documentation. Cosmetic while the hash is on the ISO anyway; it stops being cosmetic if the credential ever becomes conditional.
R-128	(open, restated)	build-felhom-iso.sh:44's claim that ISO_VERSION "aligns with felhom-host-install SCRIPT_VERSION" is a comment, not an assertion. SCRIPT_VERSION appears nowhere in scripts/iso/ except inside that comment's own text, so nothing evaluates it; the values have drifted (1.25.0 vs 1.22.0). Instance #7 of the "a comment asserting an invariant needs a test pinning it" class.

---

10. Teardown

All three layers, per PROMPT-TEMPLATE.md §13, each verified by a positive check rather than asserted.

Layer 1 — demo-hp (nested-VM host).

Check	Result
QEMU processes	0 remaining
/mnt/nvme-1tb/spike-scratch/ (ISO, disk images, answer disk, probe scripts)	removed — No such file or directory
/tmp/paia (temporary assistant binary)	removed
loop devices	0
/mnt/nvme-1tb/ contents	back to dump e2d-images felhom-data images lost+found — the pre-spike set
drill-r50 (VM 300)	stopped — untouched, not destroyed (R-93 fixture intact)
guest 9201	running — untouched
/etc/pve/storage.cfg	4 storages, unchanged; no scratch storage was ever registered
local-lvm	never written to

Layer 2 — demo-felhom (read-only probe host).

Check	Result
/tmp/paia	removed
guest 9201	running — untouched
external USB drive sdb	931.5G usb /mnt/hdd_1 — still attached and mounted exactly as found; never attached, detached, partitioned or written

Layer 3 — DooPlex (build host).

Check	Result
scratchpad	3.2 GB → 28 K (probe ISO, extracted ISO trees, PPM/PNG screendumps, file lists removed)
throwaway root password file	shred -u'd
felhom-iso/out/	17 ISOs, nothing deleted (fence honoured)
repo working tree	clean apart from this document
production files	none modified — no .go, no build-felhom-iso.sh, no profile

Hub-side disposition — nothing to dispose. This spike created no customer, no host, no appliance registration: the nested VMs ran under plain QEMU with no network path to the hub, and the only hub interaction was two authenticated reads of /configuration and /configs. Verified positively rather than asserted — the customer list was fetched and searched for spike|probe|scratch-named entries (none) and carries the same single pre-existing customer. R-131 does not gain a row here.

Published artefacts: none. Nothing was uploaded to R2 or anywhere else. The R2 credential was never created, used, or handled in this session.