felhom.eu/documentation/audits/SPIKE-universal-iso-2-2026-07-31.md

SPIKE 2 — closing the last mechanisms, and the universal ISO's shape (2026-07-31)

Class: Spike. Findings only. No production file changed, no ISO built, nothing published.

Continues SPIKE-universal-iso-2026-07-31.md; its results are premises here and are not re-derived.

All four questions are answered, and every answer that mattered is measured on a real PVE install. The session lost its venue mid-run (§0) and resumed when connectivity was restored.

---

0. Venue loss and recovery — what was tried

At 12:28 CEST both Tier 0 boxes went offline simultaneously. Routes attempted, in order:

#	Route	Result
1	ping 100.76.96.79 (tailnet direct)	100 % loss
2	ssh demo-hp	Connection timed out
3	ssh demo-hp-lan (LAN, ProxyJump felhom-pve)	Connection timed out during banner exchange
4	ssh felhom-pve (the jump host)	Connection timed out
5	DooPlex tailscale pod	Running 6d10h, healthy
6	tailscale status from the pod	both peers offline, last seen rising

Far end, not ours. No fallback venue was used — DooPlex is Tier 2, the 2026-07-25 ruling moved drill VMs off it, and this task did not authorise it. An absent fence is not permission.

Connectivity was restored (operator: a provider cable fault nearby) and the boxes returned with uptime 7 days — a pure network outage, no reboot. All probes then ran.

One thing the outage revealed. /mnt/nvme-1tb/spike2/ did not exist when demo-hp came back, and disk usage was unchanged at 6.6 GB: the staging mkdir had been inside the SSH call that timed out as the link degraded, so the fixture never landed and Layer 1 had nothing to clean. The interim report's "Layer 1 teardown INCOMPLETE" was therefore over-cautious rather than wrong — corrected here, and the fixture was rebuilt from scratch.

---

1. Baselines

Baseline	Value
ISO_VERSION	1.25.0 — scripts/iso/build-felhom-iso.sh:44
SCRIPT_VERSION	1.22.0 — scripts/felhom-host-install.sh:187
felhom-bootstrap.sh @ HEAD	21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7
PVE base	proxmox-ve_9.2-1.iso, 4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c
felhom.eu HEAD	ea00976, clean, == origin/main
proxmox-first-boot pkg	9.2.5 (from the ISO's /proxmox/packages/)
installed guest kernel	7.0.2-6-pve, GRUB 2.12-9+pmx2

Fixture provenance. Probe ISO built on DooPlex from the baseline PVE ISO (prepare-iso --fetch-from partition --partition-label proxmox-ais --on-first-boot diag.sh), sha256 dd97dba5518c70baf4300b119e1042eabf1012d5a046082acf31e60983efaeb9, verified byte-identical at both ends after transfer. Nested VMs ran under plain QEMU/KVM on demo-hp (Tier 0, the designated drill host) under /mnt/nvme-1tb/spike2/ — never local-lvm, no PVE storage registered, drill-r50 and guest 9201 untouched.

Both of Spike 1's fixture defects are corrected in this one: a NIC is present at install time (its absence made the installed system's interface config unmatchable at boot), and completion is signalled by reboot-mode = "power-off" — the guest powering itself off — never by a changed disk hash.

Three scenarios, all installed to completion:

Scenario	root credential	[first-boot].ordering	Install
A	root-password-hashed = "*" (locked)	fully-up	completed, powered off after 191 s
B	root-password-hashed = "*" (locked)	before-network	completed, powered off after 190 s
C	a known password (control)	fully-up	completed, powered off after 191 s

Evidence classes: [MEASURED-PVE] on a real PVE install · [MEASURED-MECH] measured in an isolated container · [STRUCTURAL] read from shipped source or unit files.

---

2. Q1 — is a locked root actually locked, and what else does it lock out?

2.1 The account's real state — [MEASURED-PVE]

From the first-boot diagnostic on both locked installs (A and B), emitted to ttyS0:

passwd_S_root   : root L 2026-07-31 0 99999 7 -1
shadow_shape    : literal-asterisk
sshd_PermitRoot : yes
pam_common_auth : 1
pve_pam_realm   : 0

The account is genuinely locked — passwd -S reports L, and the * reached /etc/shadow verbatim. Note sshd_PermitRoot: yes: PVE ships PermitRootLogin yes, so the lock is doing the work, not sshd's configuration.

pve_pam_realm: 0 — there is no PVE-specific file in /etc/pam.d/ matching proxmox|pve, and pam_unix is present in common-auth. PVE uses the stock PAM stack, which is why the §2.2 container measurement transfers to it rather than merely resembling it.

2.2 What can authenticate — [MEASURED-MECH], first attempt invalid

The first probe produced nothing and is recorded as failed: it ran everything as uid 0 so su never invoked pam_unix (rc=0 even with a deliberately wrong password), sulogin died on tcgetattr failed with no TTY, and a sed mangled the control hash. Four cases, zero observations.

Corrected (debian:trixie, util-linux 2.41, non-root caller, real pty via script, chpasswd):

Shadow field	passwd -S root	su root as a non-root user	sulogin (real pty)
real hash (control)	P	correct pw → SU_OK; wrong pw → Authentication failure	—
*	L	Authentication failure (any password)	Cannot open access to console, the root account is locked.
!	L	Authentication failure (any password)	Cannot open access to console, the root account is locked.

The control makes it readable: the same harness succeeds with the right password and fails with the wrong one, so the locked rows are the module refusing rather than the harness misfiring. * and ! are equivalent.

On PVE, root SSH was refused on every scenario — but that row is uninformative and is not evidence: the probe supplied a deliberately wrong password in the control scenario too, so it cannot distinguish "locked" from "wrong password". Recorded as a gap, not a result.

2.3 Does the Proxmox web interface still work? — [MEASURED-PVE]. No. It is closed.

The product-level answer, measured against the exact endpoint the web UI logs in through (POST /api2/json/access/ticket, realm root@pam), with a working control:

Scenario	Password offered	Result
C (control, known password)	correct	HTTP 200, TICKET ISSUED — login would SUCCEED
C (control)	wrong	HTTP 401, no ticket
C (control)	empty	HTTP 401, no ticket
A (locked root)	wrong	HTTP 401, no ticket
A (locked root)	empty	HTTP 401, no ticket

The control is what makes this a finding rather than a broken probe: the same endpoint on the same image demonstrably issues a ticket when the password is right. On the locked install no password can produce one, because no password exists.

Therefore a locked root closes the PVE web interface completely — to the customer and to the operator alike. This is a product decision, not a detail: it removes the browser-based route into the box entirely, on top of the console route (§3.2).

Reachability, incidentally measured on all three: SSH banner at +20 s, pveproxy TLS at +26–29 s after power-on.

2.4 Does anything else in PVE break? — not investigated

Cron, backup jobs and other consumers of a root credential were not examined. pveproxy itself runs and serves TLS (it answered /api2/json/version), so the daemon is unaffected — only authentication through it is.

---

3. Q2 — is there a way back into a box whose setup failed?

3.1 Does the installed GRUB expose a recovery entry? — [MEASURED-PVE]. Yes.

The installed system's GRUB is PVE's own, unaffected by Felhom's ISO menu collapse (§4). Screendumped from the locked install:

*Proxmox VE GNU/Linux
 Advanced options for Proxmox VE GNU/Linux
 Memory test (memtest86+x64.bin)
 Memory test (memtest86+x64.bin, serial console)

and inside Advanced options:

*Proxmox VE GNU/Linux, with Linux 7.0.2-6-pve
 Proxmox VE GNU/Linux, with Linux 7.0.2-6-pve (recovery mode)

A recovery entry exists, the menu is editable (e), a GRUB command line is available (c), and there is no GRUB password. Its timeout is 1 second — short enough that a first probe missed it entirely (§3.5).

3.2 sulogin with a locked root — [MEASURED-PVE]. It refuses.

Booting (recovery mode) on the locked install reaches rescue mode and then stops:

Started rescue.service - Rescue Shell.
Reached target rescue.target - Rescue Mode.
You are in rescue mode. After logging in, type "journalctl -xb" …

Cannot open access to console, the root account is locked.
See sulogin(8) man page for more details.

Press Enter to continue.

The advertised recovery path is closed on a locked-root box — verbatim the container result, now confirmed on real PVE. systemd invokes sulogin for both rescue.target and emergency.target, so both are closed by the same refusal.

3.3 What is left — [MEASURED-PVE]. init=/bin/bash works.

Appending init=/bin/bash to the kernel line in the GRUB editor and booting produced:

Found volume group "pve" using metadata type lvm2
3 logical volume(s) in volume group "pve" now active
/dev/mapper/pve-root: clean, 60332/561936 files, 1141450/2243584 blocks
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
root@(none):/#

An unauthenticated root shell on a locked-root system. No password is involved because PID 1 is the shell — sulogin is never reached.

Path	Works?	Realistic for whom
PVE web console (:8006)	No (§2.3)	—
GRUB (recovery mode) → sulogin	No (§3.2)	—
GRUB editor → init=/bin/bash	Yes (measured)	Operator only. Requires catching a 1-second menu, editing a kernel line, and knowing the incantation. Not a non-technical customer action.
Live USB / rescue media	not tested	Operator only
SSH by key	n/a	Only if a key were baked — which R-142 says must not happen in a public image

So a locked-root box is recoverable, but only by someone competent at the physical console. That is a materially weaker objection than "no path back at all", which is what the structural reading suggested before it was measured.

A security corollary that is worth stating plainly: the installed GRUB has no password, so locking root buys nothing against physical access. Anyone at the keyboard can take a root shell in under a minute. Locked root is a defence against remote credential reuse — the shared-ISO-password problem — and not a physical-security measure. R-152.

3.4 Does locked differ from rotated? — [MEASURED]. Yes, at exactly one point.

• Locked (*/!) — sulogin refuses (§3.2); the web UI refuses (§2.3); init=/bin/bash works.
• Rotated to a per-box value — pam_unix accepts the correct password (the §2.2 control proves this) and the web UI issues a ticket for it (the §2.3 control proves this), so both the console and the browser open to whoever holds it; init=/bin/bash works as well.

And this is what settles the escrow question. A rotated password that is discarded is operationally identical to a locked account: nobody holds it, so sulogin and the web UI refuse either way. The only thing escrow buys is that the operator keeps the web-UI and sulogin routes. It does not buy recoverability as such, because init=/bin/bash is available in every case.

3.5 A probe that failed, recorded as failed

The first GRUB attempt waited 3 seconds before acting; the menu's timeout is 1 second, so the system had already booted and the keystrokes went into the login prompt — the arrow-key escape sequences (^[[B) are visible there in the screendump. It tested nothing. A later attempt typed init=/bin/bash onto the echo line one below linux, which the pre-typing verification screendump caught before it was booted. Both were rerun; the numbers above are from the corrected runs.

---

4. Q3 — does the Felhom stub survive before-network?

4.1 Does it execute? — [MEASURED-PVE]. Yes.

The diagnostic emits @@SPIKE2_HOOK_RAN@@ BEGIN/END to ttyS0. On scenario B (ordering = "before-network") both markers are present (count 2). This is a positive observable the hook produced, which §4.4 explains is the only acceptable form of proof here.

4.2 Does it have what it needs that early? — [MEASURED-PVE]. Yes, everything.

4.3 Does it complete before anything is network-reachable? — [MEASURED-PVE + STRUCTURAL]. Yes.

The A/B comparison, same image, same diagnostic, only ordering differs:

Diagnostic	A — fully-up	B — before-network
hook markers	2 (ran)	2 (ran)
uptime_s at hook	18.32	6.58
entropy_avail	256	256
etc_writable	YES	YES
openssl/chpasswd/systemctl/passwd/curl/ip	all present	all present
openssl_rand_len	32	32
ip_global	vmbr0 10.0.2.15/24	(empty)
network.target	active	inactive
network-online.target	active	inactive
pveproxy	active	inactive
sshd	active	inactive
listen_22_8006	3	0

At before-network the hook runs with entropy, a writable /etc, every binary it would need, and a demonstrated ability to generate a 32-character random string — while the box has no IP address and nothing is listening on :22 or :8006. A credential rotation placed there has a zero-length exposure window, measured.

Equally, the fully-up column is the measured proof of the converse: today's ordering runs the hook when sshd and pveproxy are already listening. The window Spike 1 argued for structurally is now observed.

The structural half agrees and generalises it. The shipped unit (proxmox-first-boot 9.2.5, /usr/lib/systemd/system/proxmox-first-boot-network-pre.service):

After=systemd-remount-fs.service
Before=network-pre.target
Wants=network-pre.target
ConditionPathExists=/var/lib/proxmox-first-boot/pending-first-boot-setup
ConditionPathIsReadWrite=/var/lib
Type=oneshot

Before=network-pre.target + Type=oneshot makes it a hard barrier ahead of all network configuration — so this holds on every machine, not just this VM. Both installs then booted to full reachability (SSH +20 s, pveproxy +26–29 s), so the early hook does not break the boot.

A third ordering value exists that Spike 1 did not record. validate-answer accepts before-network, network-online, fully-up; it rejects network-pre, multi-user and a bogus value, so this is a real enum. The shipped package contains a matching unit for each. network-online is the unevaluated middle option. R-149.

4.4 Does the existing pairing flow still work? — [STRUCTURAL], and it is the one gap left

Not measured. The scenarios installed the spike's own diagnostic as the first-boot hook, not Felhom's stub-first-boot.sh, so stub-plus-pairing coexistence is argued, not observed.

The structural case is strong and has a correction in it:

The stub does not need fully-up. stub-first-boot.sh writes three files, then systemctl daemon-reload, enable, and start --no-block felhom-bootstrap.service. A grep for pvesh, pct, pveum and qm across it returns no match (rc=1). The unit it starts carries its own After=network-online.target pve-cluster.service pveproxy.service, so --no-block queues the job and systemd holds it until those are satisfied.

So answer.toml.tmpl:27 justifies the current ordering with a dependency the stub does not have — it reads "fully-up = the stub runs after pveproxy so pvesh/pct work", but that usage lives in felhom-bootstrap.sh, which never runs under the stub's ordering. A comment asserting a dependency that does not exist, and it has constrained the design ever since. R-148.

Why §4.1's positive marker is load-bearing. All four proxmox-first-boot-* units are guarded by ConditionPathExists= / ConditionPathIsReadWrite=, and in systemd a failed condition is a skip, not a failure — no error, successful boot. A hook that never ran and one that ran perfectly are indistinguishable from the absence of errors. R-150.

---

5. Q4 — the boot menu, and which entry is the default

5.1 Where the collapse happens — [STRUCTURAL], and it is not a simple removal

iso-repack.sh replaces the stock grub.cfg wholesale, rendering grub/grub.cfg.tmpl over it (:144-148) after lifting the linux/initrd lines verbatim from the stock Install Proxmox VE (Automated) entry (:108-131) so a PVE bump tracks automatically.

Four independent gates enforce the single entry:

Gate	Location	Effect
exactly one menuentry	:153-155	exit 14
zero submenu	:156	exit 14
no live proxtui, proxdebug, nomodeset, Rescue Boot, memtest, fwsetup	:160-164	exit 14
re-extracted from final.iso: one entry, theme present, entry named Felhom telepítés	:314-319	exit 15

The fourth proves the menu in the shipped image rather than the build tree — the seam-wiring lesson applied correctly.

What a second entry would cost: the two count gates move from == 1 to == 2, or to a named allowlist. The banned-token gate passes unchanged for a stock Install Proxmox VE (Graphical) entry — it uses none of those six tokens. The safety property the gates exist for (no route to a debug/rescue/manual-disk-picker path) survives a second graphical-installer entry intact.

5.2 Are default and timeout settable? — [STRUCTURAL]. Already are.

grub/grub.cfg.tmpl sets set timeout_style=menu, set timeout=5, set default=0. The template also records a real defect it fixed: the stock PVE config writes timeout-style (hyphen), which GRUB does not recognise as a variable name, so Felhom's underscore spelling is the one that works.

5.3 The design question — framed with evidence, deliberately not ruled

For interactive-as-default. Spike 1 proved the automated path cannot be made safe on unseen hardware: no property distinguishes a system disk from a customer's backup drive, and a filter matching two disks silently wipes one. A tester who boots and walks away on multi-disk hardware is walking away from an unattended destructive operation whose target cannot be predicted.

This session strengthens that case independently. If the public image also ships a locked root, then a box whose install succeeds but whose pairing fails has no web UI (§2.3) and no working recovery-mode shell (§3.2) — only an operator-grade init=/bin/bash (§3.3). An unattended default maximises the number of testers who reach that state without having read anything.

Against. Unattended install is a selling point and most first boxes are single-disk, where the automated path is fine.

Operator-built and public need not match, and two profiles from one build is cheap. An operator-built image targets known disks, so automated-default is safe there; the public image is the one facing unseen hardware. That split is available however the default is ruled.

---

6. Q5 — the resulting shape, as constraints

Constraints on a build spec, each with a release-gate criterion. Not an implementation.

#	Constraint	Release gate
1	No disk selection by property on unseen hardware (Spike 1 §3.2/§3.3).	Rendered answer contains no filter.* key, or the image is operator-built for a named machine.
2	The interactive installer must be reachable from the boot menu (§5.1).	final.iso's grub.cfg has ≥1 interactive entry and still no live proxtui/proxdebug/nomodeset/Rescue Boot/memtest/fwsetup — keep gate 3, relax gates 1 and 4 to an allowlist.
3	No usable baked credential — root-password-hashed = "*" installs cleanly (§1, three scenarios).	Rendered root-password-hashed is exactly *, and no .rootpw.txt is emitted. Two one-line assertions.
4	Any per-box password is set at before-network — the only ordering with a measured zero window (§4.3).	Rendered [first-boot].ordering is before-network, and the hook's positive marker is observed in a boot test (§4.4, R-150).
5	No root-ssh-keys, ever, in a public image (R-142).	grep -c root-ssh-keys on the rendered answer is 0.
6	The profile must be committed — demo-felhom's baked key comes from an uncommitted profile, so a reference box is not reproducible from main.	--profile resolves inside the repo tree and git status --porcelain is empty for it. R-151.
7	The artefact must be reproducible (R-144: nested-probe's profile is gone).	Manifest records the committed profile path + repo commit; both resolve.
8	The recovery path must be documented, and named operator-only if it is. §2.3 + §3.2 + §3.3: with a locked root the only way in is init=/bin/bash at the console.	Release notes state it, in those terms.

What the image must not contain: a usable root password hash, any root-ssh-keys, any customer id, claim code, retrieval passphrase or hub token, and any filter.* selector in the public variant. Spike 1 §6 established by enumeration that /answer.toml is the only secret-bearing file, so gates 3 and 5 over that one rendered file cover the credential surface.

---

7. The operator's decisions

Decision 1 — escrow or discard a per-box root password. §3.4 reshapes this twice over:

• Discard and lock are the same thing for recovery — if nobody holds the password, both sulogin and the web UI refuse.
• Neither makes a box unrecoverable. init=/bin/bash works in every case (§3.3), so an operator at the console can always get in. The earlier fear that locking root leaves "no path back at all" is measured to be false.

So what escrow actually buys is narrower than it looked: the operator keeps the web-UI route (§2.3) and the sulogin route (§3.2) without needing physical presence. Against that stands D6 — a hub holding console root for the whole fleet is exactly what the hub alone must not be.

Recommendation, offered because the measurements now support one: discard/lock for the public image, and accept operator-only physical recovery, because the credential that would be escrowed is the same one whose fleet-wide sharing started this whole line of work. Escrow remains defensible for operator-built boxes, where the operator installed the machine anyway. The decision is still the operator's; it is no longer blocked on a measurement.

Decision 2 — the default boot entry for a public image. Framed in §5.3, unruled. The disk evidence and the recovery evidence now point the same way (interactive), and the cost is the unattended experience. The operator-built-versus-public split is cheap and available either way.

---

8. What is still unknown, and what it would take

#	Unknown	What it takes
1	Felhom's real stub at before-network (§4.4) — coexistence with pairing is structural, not observed. The single remaining gap in Q3.	Build a probe ISO with stub-first-boot.sh as the hook and ordering = "before-network", boot, confirm felhom-bootstrap.service still starts and pairing completes. ~40 min. Do this before the spec.
2	Whether root SSH is refused because of the lock (§2.2) — the probe used a wrong password in the control too, so it discriminates nothing.	Re-run the SSH leg with the control's correct password. ~10 min.
3	What else in PVE assumes a usable root credential (§2.4) — cron, backup jobs.	Inspect a locked install over a longer uptime.
4	What network-online ordering buys (§4.3) — a real third option, never evaluated.	One extra scenario.
5	Whether the multi-match pick is deterministic by enumeration order (Spike 1, still open).	Re-run Spike 1's S1 with reversed disk order. ~15 min.
6	Whether the installer excludes its own boot medium (Spike 1, still open).	USB boot on real hardware — HUMAN.
7	Whether a relaxed-gate two-entry menu actually boots the stock installer (§5.1).	Not attempted — it would mean changing repo files, which §4 forbids.

---

9. R-rows opened

Continuing from R-147 (Spike 1).

R	Severity	Finding
R-148	MEDIUM	answer.toml.tmpl:27 justifies ordering = "fully-up" with a dependency the stub does not have. stub-first-boot.sh contains no pvesh/pct/pveum/qm call (grep rc=1); that usage is in felhom-bootstrap.sh, under its own After=network-online.target … pveproxy.service unit. A comment asserting a dependency rather than an invariant — and it blocked the ordering that §4.3 now measures as giving a zero credential window.
R-149	LOW	[first-boot].ordering has three values, not two — before-network, network-online, fully-up validate; network-pre and multi-user are rejected. The middle option has never been evaluated.
R-150	MEDIUM	A first-boot hook that never runs is indistinguishable from one that succeeded. All four proxmox-first-boot-*.service units are Condition*-guarded, and a failed condition is a systemd skip, not a failure — no error, successful boot. Any credential-rotation-in-first-boot design must be verified by a positive marker the hook emitted.
R-151	MEDIUM	A reference box is not reproducible from the repository. demo-felhom was installed from an ISO built with the uncommitted n100-demo.profile, which is where its baked SSH key comes from (Spike 1 §4.6). Gate 6 in §6 exists to stop a public image inheriting that shape.
R-152	LOW	A locked root is not a physical-security measure, and should never be described as one. The installed PVE GRUB has no password (§3.1), so init=/bin/bash yields an unauthenticated root shell in under a minute (§3.3, measured). Locking root defends against remote reuse of a shared ISO credential and nothing else. If a physical-access threat model is ever wanted, it needs a GRUB password, which is a separate decision with its own lockout risk.

---

10. Teardown

All three layers, each verified positively.

Layer 1 — demo-hp.

Check	Result
QEMU processes	0
/mnt/nvme-1tb/spike2/	removed — No such file or directory
/tmp/paia	removed
loop devices / /mnt/s2ans	0 / removed
/mnt/nvme-1tb/ contents	dump e2d-images felhom-data images lost+found — the pre-spike set
disk usage	6.6 G — identical to pre-spike
drill-r50 (VM 300)	stopped, untouched (R-93 fixture intact)
guest 9201	running, untouched
/etc/pve/storage.cfg	4 storages, unchanged; no scratch storage ever registered
local-lvm	never written

Layer 2 — demo-felhom. /tmp/paia removed; guest 9201 running; the external USB drive 931.5G usb /mnt/hdd_1 still attached and mounted exactly as found — never attached, detached, partitioned or written. Only device-info reads were ever performed there.

Layer 3 — DooPlex.

Check	Result
scratchpad	1.7 GB → 60 K (probe ISO, screendumps, squashfs/.deb extracts, schema test dirs)
control password (s2/.ctlpw) and scen/C.toml	shred -u'd / removed; verified absent. Never printed, never committed
mechanism-probe container	docker run --rm; no global Docker cleanup run (fence)
felhom-iso/out/	untouched, 17 ISOs
production files	none modified — no build-felhom-iso.sh, no iso-repack.sh, no profile, no felhom-bootstrap.sh

Hub-side disposition — nothing to dispose. No customer, host or appliance was created: the nested VMs ran under plain QEMU with user-mode networking and no route to the hub, and no install ever contacted it. R-131 gains no row.

Published artefacts: none. Nothing went to R2.