felhom.eu/documentation/audits/SPIKE-universal-iso-2-2026-07-31.md

SPIKE 2 — closing the last mechanisms, and the universal ISO's shape (2026-07-31)

Class: Spike. Findings only. No production file changed, no ISO built, nothing published.

Continues SPIKE-universal-iso-2026-07-31.md; its results are premises here and are not re-derived.

Read §0 first: the venue went away mid-session. Both Tier 0 hosts dropped off the tailnet at ~12:28 CEST and did not return. Q1, Q2 and Q3 each have a part that needs a nested VM, and those parts are BLOCKED, not answered. What could be established without them — from the shipped PVE source, the answer schema, and an isolated mechanism probe — is below and is labelled by how it was obtained.

---

0. Venue loss — what was tried

At 12:28 CEST, mid-run, both Tier 0 boxes went offline simultaneously:

100.76.96.79   demo-hp     active; relay "fra"; offline, last seen 5m ago, tx 1843821564 rx 63739848
100.70.170.35  felhom-pve  active; relay "waw"; offline, last seen 5m ago, tx 1560 rx 0

Attempts, in order (per the standing rule that a "no access" claim names what it tried):

#	Route	Result
1	ping 100.76.96.79 (tailnet direct)	100 % packet loss
2	ssh demo-hp (tailnet, ConnectTimeout=10)	connect to host 100.76.96.79 port 22: Connection timed out
3	ssh demo-hp-lan (LAN 192.168.0.87, ProxyJump felhom-pve)	Connection timed out during banner exchange
4	ssh felhom-pve (the jump host itself)	connect to host 100.70.170.35 port 22: Connection timed out
5	DooPlex tailscale pod health	pod tailscale-c84df7f84-tw2wz Running 6d10h, healthy; log shows open-conn-track: timeout … online=yes then peers dropping
6	tailscale status from the pod	both peers offline, last seen 5m and rising

Diagnosis: the far end, not ours. Our tailscale pod is healthy and every other peer route behaves; both demo boxes are at the same remote site (REMOTE until ~08-02) and went offline together, which is that site's power or uplink. Not fixable from here.

No fallback venue was used. DooPlex is Tier 2 and the 2026-07-25 standing ruling moved drill and build VMs off it; target-selection.md says a Tier 2 machine is used only when a task says so explicitly, and this one does not. An absent fence is not permission, so the probes simply did not run. A watch is armed for the boxes' return.

The probe fixture is built and staged and will run unchanged when they come back: probe ISO (prepare-iso --fetch-from partition, serial-capture first-boot diagnostic), three validated answer files (locked/fully-up, locked/before-network, known-password control), an install runner with reboot-mode = "power-off" as the completion signal, and a boot runner with protocol-level detectors and a NIC present at install time — the two fixture defects that invalidated Spike 1's measurements are already corrected.

---

1. Baselines

Read fresh this session.

Baseline	Value
ISO_VERSION	1.25.0 — scripts/iso/build-felhom-iso.sh:44
SCRIPT_VERSION	1.22.0 — scripts/felhom-host-install.sh:187
felhom-bootstrap.sh @ HEAD	21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7
PVE base	proxmox-ve_9.2-1.iso, 4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c
felhom.eu HEAD	ea00976, clean, == origin/main (the expected Spike 1 commit)
proxmox-first-boot pkg	9.2.5 (from the ISO's /proxmox/packages/)

Fixture provenance. Probe ISO built on DooPlex from the baseline PVE ISO. Nested VMs were to run under plain QEMU/KVM on demo-hp under /mnt/nvme-1tb/spike2/ — never local-lvm, no PVE storage registered. They did not run (§0). The mechanism probe in §2.2 ran in a throwaway debian:trixie Docker container on DooPlex, which is not an install and touches no box.

Evidence classes used throughout: [MEASURED-PVE] against the real PVE tooling or a PVE install · [MEASURED-MECH] measured, but in an isolated container rather than on PVE · [STRUCTURAL] read from shipped source or unit files · [BLOCKED] needs the nested VM.

---

2. Q1 — is a locked root actually locked, and what else does it lock out?

2.1 The account's state on a real PVE install — [BLOCKED]

Needs the nested VM. The fixture is staged. Spike 1 established only that root-password-hashed = "*" installs to completion; the resulting account state was explicitly left open there and remains open.

2.2 What can authenticate — [MEASURED-MECH], and the first attempt was invalid

The first probe produced nothing and is recorded as failed. It ran everything as uid 0, so su never invoked pam_unix at all (rc=0 even with a deliberately wrong password); sulogin died on tcgetattr failed: Invalid argument with no TTY; and a sed mangled the control hash so the control case had no password set. Four cases, zero valid observations. Rerun with a non-root caller, a real pty via script, and chpasswd instead of sed.

Corrected probe (debian:trixie, util-linux 2.41, pam_unix) — with a working control:

Shadow field	passwd -S root	su root as non-root user	sulogin (real pty)
real hash (control)	P	correct pw → SU_OK; wrong pw → Authentication failure	—
*	L	su: Authentication failure (any password)	Cannot open access to console, the root account is locked.
!	L	su: Authentication failure (any password)	Cannot open access to console, the root account is locked.

The control is what makes this readable: the same harness authenticates successfully with the right password and fails with the wrong one, so the two locked rows are the module refusing, not the harness misfiring.

So, at mechanism level: * and ! are equivalent, both report L, and no password authenticates.

Caveat, and it is not small. This is Debian's pam_unix and util-linux's sulogin — the same components PVE ships — but it is not PVE. PVE could ship PAM configuration, a sulogin override, or a SYSTEMD_SULOGIN_FORCE setting that changes the outcome. Confirming on a real PVE install is part of the blocked work.

2.3 Does the Proxmox web interface still work? — [BLOCKED], and this is the product question

No measurement was obtained, so no answer is given. Stating the reasoning without the measurement would be exactly the failure mode this spike exists to avoid.

What is established: PVE's root@pam realm authenticates the Linux root account through PAM, and §2.2 measured that pam_unix refuses a locked account for a non-root caller. If PVE's proxy authenticates the same way — which was not verified — then a locked root means nobody can log into the PVE web interface at all, neither customer nor operator, and that is a product decision rather than a detail.

The staged control scenario exists precisely to settle this: install C carries a known password, so the PVE API ticket endpoint can be exercised positively (correct password → ticket) before install A (locked) is tested against the same endpoint. Without that control a failure is indistinguishable from a broken probe — Spike 1's lesson.

2.4 Does anything else in PVE break? — [BLOCKED]

Not investigated. Cron, pveproxy, backup jobs and anything else assuming a usable root credential remain unexamined.

---

3. Q2 — is there a way back into a box whose setup failed?

3.1 Does the installed GRUB expose a recovery entry? — [BLOCKED]

Needs a boot. Note the installed system's GRUB is PVE's own, written by the installer — not the ISO's Felhom menu — so §5's single-entry collapse does not apply to it. What it offers is unverified.

3.2 sulogin with a locked root — [MEASURED-MECH]. It refuses.

The single observation the escrow-versus-discard ruling was said to rest on:

Cannot open access to console, the root account is locked.
See sulogin(8) man page for more details.

Both for * and for !, with a real pty. The standard Debian/systemd emergency and single-user path does not grant a shell on a locked-root system. systemd invokes sulogin for emergency.target and rescue.target, so both are closed by the same refusal.

Subject to the §2.2 caveat: measured in a container, not on PVE.

3.3 What is left if sulogin refuses — [STRUCTURAL], not measured

Path	Realistic for whom
init=/bin/bash on the kernel line (GRUB e)	Operator only. It bypasses sulogin because PID 1 becomes the shell — no authentication is involved. Requires editing a boot entry at the console. Not a non-technical customer action.
systemd.unit=emergency.target	Closed — it lands in sulogin, which refuses (§3.2).
Live USB / rescue media	Operator only; requires physical media and knowing to do it.
PVE web console (:8006)	Unknown, possibly closed — §2.3.
SSH by key	Only if a key was baked, which for a public image is the thing R-142 says must not happen.

The honest summary: on a locked-root box, every remaining route is operator-grade and physical. For a customer whose pairing failed, there is no self-service path back — which is precisely the objection raised when locked-root was first proposed, and §3.2 is the first evidence that the objection is well founded rather than speculative.

3.4 Does locked differ from rotated? — [PARTIALLY ESTABLISHED]

At mechanism level, yes, and materially:

• Locked (*/!) — sulogin refuses; console recovery is closed to everyone; the credential does not exist so it cannot leak, be escrowed, or be rotated.
• Rotated to a per-box random value — sulogin prompts and accepts the correct password (the control row in §2.2 shows pam_unix authenticating a real hash). So console recovery is open to whoever holds that password — which is the operator if it is escrowed, and nobody if it is discarded.

This makes the tradeoff concrete: "discard" and "lock" are the same outcome for recovery. A rotated-then-discarded password is operationally identical to a locked account, because nobody holds it. The real choice is therefore binary — escrow, or accept that console recovery is operator-only and physical. Framed for the ruling in §7.

---

4. Q3 — does the Felhom stub survive before-network?

4.1 Does it execute? — [BLOCKED]

Needs the VM. The staged diagnostic emits @@SPIKE2_FIRSTBOOT_RAN@@ marker=BEGIN/END to ttyS0 (captured by qemu -serial file:) precisely so the answer is a positive observable rather than the absence of an error — and §4.4 below is why that matters more than usual here.

4.2 Does it have what it needs that early? — [STRUCTURAL], partly answered

From the shipped unit (proxmox-first-boot 9.2.5, /usr/lib/systemd/system/proxmox-first-boot-network-pre.service):

After=systemd-remount-fs.service
Before=network-pre.target
Wants=network-pre.target
ConditionPathExists=/var/lib/proxmox-first-boot/pending-first-boot-setup
ConditionPathIsReadWrite=/var/lib
Type=oneshot

After=systemd-remount-fs.service plus ConditionPathIsReadWrite=/var/lib mean a writable filesystem is structurally guaranteed — or the unit does not run at all (§4.4). Entropy availability and binary presence at that point are not established structurally and are part of the staged diagnostic (entropy_avail, openssl rand length, command -v for openssl/chpasswd/systemctl).

4.3 Does it complete before anything is network-reachable? — [STRUCTURAL]. Yes, by construction.

This is the result the task asked to establish structurally rather than by timing, and the unit file settles it:

• Before=network-pre.target — network-pre.target is systemd's ordering barrier that precedes all network configuration; network managers order themselves after it.
• Type=oneshot — systemd treats the unit as complete only when the process exits, so Before= is a hard barrier rather than a hint.

Together: the hook runs to completion before any interface is configured, therefore before sshd can bind a routable address and before pveproxy serves. A rotation performed there has a zero-length exposure window by construction, not by being fast. This holds on every machine and does not vary with hardware — which is why it is stronger than the timing number Spike 1 failed twice to obtain.

A third ordering value exists that Spike 1 did not record. validate-answer accepts before-network, network-online, and fully-up — and rejects network-pre, multi-user and a bogus value, so this is a real enum and not a permissive parser. The shipped package contains a matching unit for each. Spike 1 reported two. network-online is the middle option — after the network, before pveproxy — and is worth considering if before-network proves too early for something.

4.4 Does the existing pairing flow still work? — [STRUCTURAL] yes; [BLOCKED] empirically

Two structural findings, and the second one corrects a premise this design has been carrying.

The stub does not need the network, and does not need pveproxy. stub-first-boot.sh writes three files, then systemctl daemon-reload, enable, and start --no-block felhom-bootstrap.service. A grep for pvesh, pct, pveum and qm across the stub returns no match (rc=1). The unit it starts carries its own After=network-online.target pve-cluster.service pveproxy.service, so --no-block merely queues the job and systemd holds it until those are satisfied. Structurally, stub and pairing coexist: the stub runs early, the bootstrap runs when its own dependencies are met.

So answer.toml.tmpl:27 justifies the fully-up choice with a dependency the stub does not have. The comment reads "fully-up = the stub runs after pveproxy so pvesh/pct work" — but the pvesh/pct usage is in felhom-bootstrap.sh, which never runs under the stub's ordering. This is the CLAUDE.md "a comment asserting an invariant" class in a new shape: a comment asserting a dependency that does not exist, which has constrained the design ever since. R-148.

The trap that makes §4.1's positive observable load-bearing. All four proxmox-first-boot-* units are guarded by ConditionPathExists= and ConditionPathIsReadWrite=. In systemd a failed condition is a skip, not a failure — the unit reports inactive/condition failed, the boot succeeds, and nothing is logged as wrong. A first-boot hook that never ran and one that ran perfectly are indistinguishable from the absence of errors. R-150. This is why §4.1 must be confirmed by a marker the hook emitted, never by "no errors appeared".

---

5. Q4 — the boot menu, and which entry is the default

5.1 Where the collapse happens — [STRUCTURAL], and it is not a simple removal

iso-repack.sh replaces the stock grub.cfg wholesale: it renders grub/grub.cfg.tmpl over it (iso-repack.sh:144-148), having first lifted the linux/initrd lines verbatim from the stock Install Proxmox VE (Automated) entry (:108-131) so a PVE bump tracks automatically.

Four independent gates then enforce the single entry, so restoring the stock menu is a change to the gates, not just to the template:

Gate	Location	Effect
exactly one menuentry	iso-repack.sh:153-155	exit 14 otherwise
zero submenu	iso-repack.sh:156	exit 14 otherwise
no live reference to proxtui, proxdebug, nomodeset, Rescue Boot, memtest, fwsetup	iso-repack.sh:160-164	exit 14 otherwise
re-extracted from final.iso: exactly one entry, theme present, entry named Felhom telepítés	iso-repack.sh:314-319	exit 15 otherwise

The fourth is the strongest of them — it proves the menu in the shipped image rather than in the build tree, which is the seam-wiring lesson applied correctly.

What adding one interactive entry would actually cost: the two count gates (rows 1 and 4) move from == 1 to == 2, or better to a named allowlist. The banned-token gate would still pass unchanged for a stock Install Proxmox VE (Graphical) entry — it uses none of proxtui, proxdebug, nomodeset, Rescue Boot, memtest, fwsetup. So the safety property the gates exist to protect (no route to a debug/rescue/manual-disk-picker path) survives a second graphical installer entry intact. That is a narrower change than it first appears.

5.2 Are default and timeout settable? — [STRUCTURAL]. Yes, already.

grub/grub.cfg.tmpl sets them explicitly:

set timeout_style=menu
set timeout=5
set default=0

Both are plain template values. The template also records a real defect it fixed: the stock PVE config writes timeout-style (hyphen), which GRUB does not recognise as a variable name, so Felhom's timeout_style (underscore) is the one that actually works. A two-entry menu would therefore have a functioning timeout and a settable default — which is exactly what the §5.3 question needs.

5.3 The design question — framed, deliberately not ruled

Which entry should be default=0 in a public image?

What the measurements say, and they only point one way: Spike 1 established that the automated path cannot be made safe on unseen hardware — no property distinguishes a system disk from a customer's backup drive, and a filter matching two disks silently wipes one. A tester who boots the image and walks away is, on multi-disk hardware, walking away from an unattended destructive operation whose target cannot be predicted.

What that costs: unattended install stops being the default for people who do not read instructions — and unattended install is a selling point. Most first boxes are single-disk, where the automated path is fine.

The measurement that would change the balance is §2.3 and §3: if a locked root also closes the PVE web interface and the console, then a tester whose unattended install completes but whose pairing fails has a machine nobody can reach — which argues for interactive-default independently of the disk question. Those measurements are blocked, so the balance cannot be struck yet.

Operator-built versus public need not match, and two profiles from one build is cheap. An operator-built image targets a machine whose disks are known, so automated-default is safe there. The public image is the one facing unseen hardware. Recommending the split is not the same as ruling the default, and the split is available regardless of how the ruling goes.

---

6. Q5 — the resulting shape, as constraints

Constraints on a build spec, each with a release-gate criterion. Not an implementation, and nothing was built. Constraints marked ⚠ depend on blocked measurements and are provisional.

#	Constraint	Release gate
1	The image must not select a disk by property on unseen hardware. Spike 1 §3.2/§3.3.	The rendered answer contains no filter.* key, or the image is operator-built for a named machine. Grep the rendered answer.toml.
2	The interactive installer must be reachable from the boot menu. §5.1 shows the cost is two count gates.	final.iso's grub.cfg contains ≥1 interactive entry and still contains no live proxtui/proxdebug/nomodeset/Rescue Boot/memtest/fwsetup — i.e. keep gate 3, relax gates 1 and 4 to an allowlist.
3	⚠ No usable baked credential. root-password-hashed = "*" is schema-valid and installs (Spike 1).	The rendered answer's root-password-hashed is exactly *, and no .rootpw.txt is emitted. Both are one-line assertions in the build.
4	⚠ If a per-box password is set, it is set at before-network. §4.3 — the only ordering with a zero window by construction.	The rendered answer's [first-boot].ordering is before-network; and the hook's positive marker is observed in a boot test, never inferred from absent errors (§4.4, R-150).
5	No root-ssh-keys, ever, in a public image. Spike 1 R-142.	grep -c root-ssh-keys on the rendered answer is 0.
6	The profile must be committed. Spike 1 found demo-felhom's baked key comes from an uncommitted profile, so a reference box is not reproducible from the repo.	The --profile path resolves inside the repo working tree and git status --porcelain is empty for it. R-151.
7	The artefact must be reproducible. Spike 1 R-144 (nested-probe's profile no longer exists).	Manifest records the profile's committed path and the repo commit; both resolve.
8	A recovery path must be documented and, if operator-only, said so plainly. §3.3.	The release notes name the recovery route for a failed pairing. If that route is init=/bin/bash, the notes say it is operator-only.

What the image must not contain: a usable root password hash, any root-ssh-keys, any customer id, claim code, retrieval passphrase or hub token, and any filter.* disk selector in the public variant. Spike 1 §6 established by enumeration that the only secret-bearing file is /answer.toml, so gates 3 and 5 over that one rendered file cover the credential surface.

---

7. The operator's decisions

Two, each with the measurement that informs it and its current state.

Decision 1 — escrow or discard a per-box root password.

§3.4 changed the shape of this question. "Discard" and "lock" are operationally identical — if nobody holds the password, sulogin refuses either way, and console recovery is closed to everyone. So the choice is binary:

• Escrow — the operator can always reach a box's console; the cost is a hub-held credential for every box, which is the thing the D6 trust model says the hub alone must not be.
• Discard / lock — no credential exists anywhere; the cost is that a failed pairing leaves a machine recoverable only by an operator at the keyboard (init=/bin/bash) or a reinstall.

Blocking measurement: §2.3 — whether a locked root also closes the PVE web interface. If it does, "discard" removes the last remote route as well as the local one, and the balance shifts sharply toward escrow. Do not rule until §2.3 is measured.

Decision 2 — the default boot entry for a public image.

Framed in §5.3. The disk evidence points at interactive-default; the product cost is the unattended experience. Partially blocked on the same §2.3 measurement, because an unreachable-after-failure box strengthens the interactive case independently. The operator-built-versus-public split (§5.3) is available regardless and is cheap.

---

8. What is still unknown, and what it would take

#	Unknown	What it takes
1	Everything in §2.1, §2.3, §2.4, §3.1, §4.1, §4.2 (entropy/binaries).	The staged fixture, unchanged, once a Tier 0 box returns. ~90 min for all three scenarios.
2	Whether PVE alters the §2.2/§3.2 mechanism results — PAM config, a sulogin override, SYSTEMD_SULOGIN_FORCE.	Same fixture; check /etc/pam.d/, the sulogin invocation, and systemd's emergency unit on the installed image.
3	Whether the multi-match pick is deterministic by enumeration order (Spike 1 §8 #1, still open).	Re-run Spike 1's S1 with reversed disk order. ~15 min.
4	Whether the installer excludes its own boot medium (Spike 1 §8 #2, still open).	USB boot on real hardware — HUMAN.
5	What network-online ordering buys (§4.3) — it is a real third option nobody has evaluated.	One extra scenario alongside the staged two.
6	Whether a second GRUB entry actually boots the stock installer after the gates are relaxed.	Build a probe ISO with a two-entry template and boot it. Not attempted — it would have meant changing repo files, which §4 forbids.

---

9. R-rows opened

Numbering continues from R-147 (Spike 1).

R	Severity	Finding
R-148	MEDIUM	answer.toml.tmpl:27 justifies ordering = "fully-up" with a dependency the stub does not have. The comment says the stub must run after pveproxy "so pvesh/pct work", but stub-first-boot.sh contains no pvesh/pct/pveum/qm call (grep rc=1); that usage lives in felhom-bootstrap.sh, which runs under its own After=network-online.target … pveproxy.service unit. A comment asserting a dependency rather than an invariant — and it has constrained the design, because before-network is the ordering that would make the credential window zero (§4.3).
R-149	LOW	The [first-boot].ordering enum has three values, not two. before-network, network-online, fully-up all validate; network-pre and multi-user are rejected. Spike 1 recorded two and the middle option has never been evaluated.
R-150	MEDIUM	A first-boot hook that never runs is indistinguishable from one that succeeded. All four proxmox-first-boot-*.service units are guarded by ConditionPathExists= / ConditionPathIsReadWrite=, and in systemd a failed condition is a skip, not a failure — no error, successful boot. Any design that puts credential rotation in a first-boot hook must verify it by a positive marker the hook emitted. Directly the "an absent log line is not evidence" class.
R-151	MEDIUM	A reference box is not reproducible from the repository. demo-felhom was installed from an ISO built with the uncommitted n100-demo.profile, which is where its baked SSH key comes from (Spike 1 §4.6). One of the two Tier 0 boxes therefore cannot be rebuilt from main. Related to R-144 (nested-probe's profile is gone entirely); this row is the reference box, which matters more. Gate 6 in §6 exists to stop a public image inheriting the shape.

---

10. Teardown

All three layers, verified positively.

Layer 1 — demo-hp. INCOMPLETE, and named as such. The box went offline (§0) while /mnt/nvme-1tb/spike2/ held the probe ISO (~1.7 GB) and the staged answer files and runners. Nothing was running on it: the last successful command before the outage confirmed 0 QEMU processes, and no install had been started this session. Nothing was registered as a PVE storage, nothing was written to local-lvm, and drill-r50 / guest 9201 were never touched this session.

Outstanding cleanup, to run when the box returns — recorded here because R-131 exists precisely because recorded commands never get run:

rm -rf /mnt/nvme-1tb/spike2 ; rm -f /tmp/paia ; pkill -9 qemu-system-x86

This is deliberately not claimed as done. Spike 1's /mnt/nvme-1tb/spike-scratch and both /tmp/paia copies were removed and verified at that session's teardown, so the outstanding item is this session's spike2/ directory alone.

Layer 2 — demo-felhom. Nothing placed this session — it was never reached before the outage. Spike 1's /tmp/paia was removed and verified then. No action outstanding.

Layer 3 — DooPlex.

Check	Result
scratchpad	4.4 GB → 48 K (probe ISO, squashfs extracts, .deb extract, schema test dirs removed)
control-password file (s2/.ctlpw)	shred -u'd — never printed, never committed; verified absent
mechanism-probe container	docker run --rm — no image or container retained; no global Docker cleanup run (fence)
felhom-iso/out/	untouched, 17 ISOs
production files	none modified — no build-felhom-iso.sh, no repack, no profile, no felhom-bootstrap.sh

One teardown error, recorded rather than quietly fixed. Shredding the control password (s2/.ctlpw) destroyed the plaintext that scenario C existed to use — a $6$ hash whose plaintext is gone cannot demonstrate "correct password → PVE ticket", which was C's entire purpose as the control for §2.3. s2/scen/C.toml was therefore deleted rather than left in place looking usable. Scenario C must be regenerated (one openssl passwd -6 against a fresh random plaintext, kept 0600) before the §2.3 measurement is attempted. Scenarios A and B are unaffected and remain staged.

Hub-side disposition — nothing to dispose. No customer, host or appliance was created: no install ran, and the only hub contact this session was none at all. Verified rather than asserted below.

Published artefacts: none. Nothing went to R2.