felhom.eu/documentation/audits/SPIKE-universal-iso-3-2026-07-31.md

SPIKE 3 — on an interactive install, does the ISO deliver anything at all? (2026-07-31)

Class: Spike. Findings only. No production file changed, no ISO built for release, nothing published.

Continues SPIKE-universal-iso-2026-07-31.md and SPIKE-universal-iso-2-2026-07-31.md; their results are premises and are not re-derived.

Q1 is answered, measured, with a control, and the answer is NO — decisively so. The first-boot hook does not merely fail to run on an interactive install: the mechanism is absent from the installed system entirely, down to the package. Everything else in this document is contingent on that, and §3 is where the contingency lands: a delivery mechanism does exist, and it is the one PVE uses itself.

---

1. Baselines and fixture

Baseline	Value
ISO_VERSION	1.25.0 — scripts/iso/build-felhom-iso.sh:44
SCRIPT_VERSION	1.22.0 — scripts/felhom-host-install.sh:187
felhom-bootstrap.sh @ HEAD	21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7
PVE base	proxmox-ve_9.2-1.iso, 4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c
felhom.eu HEAD	19c932a, clean, == origin/main
proxmox-first-boot pkg	9.2.5 (from the ISO's /proxmox/packages/)

Fixture provenance. One probe ISO, built on DooPlex from the baseline PVE ISO with prepare-iso --fetch-from partition --partition-label proxmox-ais --on-first-boot hook.sh, sha256 62d8a4cc948b31850a8153c129b8763353ca7dd01c19c06c3816ebed89b7c028, verified byte-identical at both ends after transfer. It carries 15 GRUB entries — Automated, Graphical, Terminal UI and the rest — so the control and the test come from the same build, which is what makes the negative meaningful.

Nested VMs ran under plain QEMU/KVM on demo-hp (Tier 0, the designated drill host) under /mnt/nvme-1tb/spike3/. Never local-lvm; no PVE storage registered; drill-r50 and guest 9201 untouched. The automated install used reboot-mode = "power-off" as its completion signal; the interactive install's completion was observed directly on the console.

The hook leaves three independent positive observables — a ttyS0 line, a persistent marker file, and its own log — because R-150 established that a first-boot hook which never runs is indistinguishable from one that succeeded if you only look for errors.

Evidence classes: [MEASURED] on a real install · [STRUCTURAL] read from shipped installer source · [NOT REACHED].

---

2. Q1 — does [first-boot] fire on an interactive install? [MEASURED] No.

2.1 The control — same image, automated entry

Observable	Result
ttyS0 marker	present — @@SPIKE3_HOOK_FIRED@@ … uptime=19.31
/var/log/SPIKE3-HOOK-FIRED	present, written at uptime 19.30
/var/lib/proxmox-first-boot/proxmox-first-boot	present, 572 bytes, mode 0700
/etc/systemd/system/proxmox-first-boot.service	present → /lib/systemd/system/proxmox-first-boot-multi-user.service
unit state	alias / active
pending-first-boot-setup flag	absent — removed by the unit's ExecStartPost, i.e. it ran to completion

So the hook demonstrably fires from this build.

2.2 The test — same image, Terminal UI entry, a normal manual install

Driven screen by screen with a screendump before every action. The installer behaved exactly as the operator's ruling assumes: it showed the target disk (/dev/sda (QEMU HARDDISK) (20.00 GiB)) and presented a summary listing Bootdisk(s) /dev/sda before erasing anything. The install completed and auto-rebooted. Installed hostname spikethree.felhom.eu, which identifies the box below.

Observable	Result
ttyS0 marker	absent (grep -c = 0)
/var/log/SPIKE3-HOOK-FIRED	absent — No such file or directory
/var/lib/proxmox-first-boot/	does not exist
/etc/systemd/system/proxmox-first-boot.service	absent
any proxmox-first-boot* unit known to systemd	none — only the stock first-boot-complete.target
the proxmox-first-boot package	not installed — dpkg-query: no packages found matching proxmox-first-boot
grep -rl SPIKE3_HOOK_FIRED /var /etc /root /usr/local	no hits

That is the search behind the negative, stated rather than asserted. The hook did not fail to run — nothing that could have run it was ever placed on the system.

2.3 The mechanism — [STRUCTURAL], cited

Three lines in the shipped installer explain it exactly.

1. Proxmox/Install/Config.pm:118-122 — the default is

   first_boot => { enabled => 0, ordering_target => 'multi-user' },
   

   and set_first_boot_opt (:300) is never called anywhere in the Perl tree. The flag can only become true through Config::merge, i.e. from the low-level config the auto-installer supplies.

2. Proxmox/Install.pm:746 — setup_proxmox_first_boot_service opens with

   return if !Proxmox::Install::Config::get_first_boot_opt('enabled');
   

   so on any path that does not set the flag, nothing is planted.

3. Proxmox/Install.pm:1360 — the package loop skips the package itself:

   next if $deb =~ /^proxmox-first-boot/ && !needs_first_boot_package();
   

   which is why §2.2 found the package missing, not merely unconfigured.

And the front-ends agree with the measurement: proxinstall — the graphical interactive installer — contains zero occurrences of first-boot; proxmox-low-level-installer likewise zero; proxmox-tui-installer has exactly one, the serde field name in the shared low-level config struct, with no way to set it. proxmox-auto-installer has eight.

Conclusion: [first-boot] is an automated-installer feature. It is unavailable on every interactive path, by construction, and this is not a Felhom defect but a property of the PVE installer.

---

3. Q2 — can the ISO plant anything into the installed system? [STRUCTURAL] Yes — one mechanism exists.

The search, stated: the installer's file-copy and package paths (Proxmox/Install.pm), the [first-boot] plumbing (§2.3), the answer schema's post-installation-webhook (an answer.toml key, so automated-only), and the proxmox-post-hook binary (part of the auto-install flow). Of these, one is mode-independent.

Install.pm:1343-1372 installs every .deb in the ISO's /proxmox/packages/ into the target, on every install path:

while (<${proxmox_pkgdir}/*.deb>) {
    ...
    next if $deb =~ /grub-pc_/            && (...);      # boot type / arch
    next if $deb =~ /^proxmox-secure-boot-support_/ && !$run_env->{secure_boot};
    next if $deb =~ /^proxmox-first-boot/ && !needs_first_boot_package();
    next if $deb =~ /^(amd64|intel)-microcode_/ && (...); # cpu vendor
    syscmd("chroot $targetdir dpkg ... --unpack /tmp/pkg/$deb") == 0 || die ...;
}

followed by chroot $targetdir dpkg --force-confold --configure -a (:1378) — which runs postinst scripts.

The skip-list is a fixed set of known package-name patterns. A package whose name matches none of them is unpacked and configured unconditionally. So a .deb added to /proxmox/packages/ is a working, existing, mode-independent way to plant a systemd unit and a script into the installed system — and it is precisely the mechanism PVE uses to deliver proxmox-first-boot itself.

Not tested. This is read from source, not measured, and it is the single most important thing to prove before any spec is written on it (§7). Two things in particular are unverified: that prepare-iso preserves an added /proxmox/packages/*.deb through its repack, and that a postinst running under dpkg --configure -a in the installer chroot can do what a first-boot hook does (it runs inside the installer's chroot, not on a booted system, so it can install a unit but cannot expect network, pveproxy, or a running systemd).

---

4. Q3 — the real stub at before-network — [NOT REACHED]

Not run. The interactive-install driving in §2.2 consumed the session's remaining room, and Q1 was the question everything else was contingent on.

What it needs (unchanged from the Spike 2 estimate, ~40 min): build a probe ISO with stub-first-boot.sh as the --on-first-boot payload and an answer with ordering = "before-network", install, boot, and confirm the stub's own log (/var/log/felhom-first-boot.log) exists, that felhom-bootstrap.service is enabled and started, and that pairing proceeds.

Spike 2's structural case is unchanged and unrefuted: the stub makes no pvesh/pct/pveum/qm call (grep rc=1), and the unit it starts carries its own After=network-online.target … pveproxy.service. It remains argued, not observed. Note §2 narrows where it matters: on the interactive path there is no first-boot hook at all, so this question now governs operator-built images only.

---

5. Q4 — how good can the manual path be? [MEASURED/STRUCTURAL] Two of three already exist.

1. The hub already renders the command with the customer-id filled in. hub/internal/web/templates/customer_unified.html:563-564:

   curl -fsSL https://felhom.eu/scripts/felhom-host-install.sh -o felhom-host-install.sh \
     && sudo bash felhom-host-install.sh --customer-id {{.CustomerID}} --mode <appliance|byo>
   

   with a JS generator at :1258-1267 that rebuilds it per selected role, including the right flags. Nothing needs building here.

2. The URL is already short, public and always current. https://felhom.eu/scripts/felhom-host-install.sh is the website's git-sync working tree tracking main on a 30-second period (documentation/.../day0-install.md:150-152) — no release tag, no staging copy, no version selector. Pushing the script publishes it.

3. Console display does not exist. felhom-host-install.sh does not write /etc/issue, /etc/issue.net or any MOTD (grep: no match). PVE's own /etc/issue banner is what the console shows after install — observed in this session's screendumps as "Welcome to the Proxmox Virtual Environment … connect to: https://10.0.2.15:8006/". That file is the obvious carrier for a Felhom line, and today nothing writes one. Reported, not built.

An incidental finding while reading (2): the hub's version-sync gate is currently red. hub/internal/web/configs.go:28 sets hostInstallVersion = "1.19.0" while SCRIPT_VERSION is 1.22.0, and the comment at :26 states "Drift is now gated: scripts/hostinstall_gates.py asserts this const == SCRIPT_VERSION (drill F-1)." Running that gate:

FAIL: hub Setup-tab hostInstallVersion=1.19.0 != SCRIPT_VERSION=1.22.0 (F-1: bump both together)
hostinstall gates: 1 FAILURE(S)

with a true exit code of 1 (verified without a pipe — an initial rc=0 reading was tail's exit status, not the script's, and is corrected here). So the gate is correct, works, and detects the drift; nothing runs it. The comment's "is now gated" is true of the gate's existence and false of its enforcement. R-153.

---

6. Q5 — what the public image contains, as constraints

Given the operator's ruling (no fully automatic install; disk selection always interactive) and Q1's negative, stated as constraints on a build spec — not an implementation, and nothing was built.

#	Constraint	Release gate (checkable against the built artifact)
1	The public image carries NO answer.toml. Q1 makes it pointless on the interactive path, and the ruling removes the automated path. Dropping it removes the baked root hash, the disk profile, and every question Spikes 1–2 were wrestling with, at once.	osirrox -indev <iso> -find / -maxdepth 1 contains no /answer.toml. One line, trivially assertable.
2	…and therefore no auto-installer-mode.toml either. Without it the stock grub.cfg does not even emit the Automated entry (Spike 1 §5: that entry is inside if [ -f auto-installer-mode.toml ]). The menu becomes interactive-only by construction, with no gate to maintain.	/auto-installer-mode.toml absent; grub.cfg contains no proxmox-start-auto-installer.
3	No baked credential of any kind. Falls out of 1: no root-password-hashed, no root-ssh-keys, and no .rootpw.txt companion emitted. The installer prompts the human, who sets their own.	grep -c root-password-hashed and grep -c root-ssh-keys over the ISO both 0; no *.rootpw.txt beside the output.
4	The boot menu is restored from the collapse, with the interactive entry as default and a human-length timeout. Spike 2 lost a probe to the installed system's 1-second menu; a person at an unfamiliar screen needs far longer. 10–30 s is the range the stock ISO itself uses (10).	final.iso's grub.cfg: set default= points at an interactive entry, set timeout ≥ 10, and timeout_style uses the underscore spelling (the stock hyphenated timeout-style is not a GRUB variable at all — Spike 2 §5.2).
5	Restoring the menu means relaxing gates, not deleting them. Four gates in iso-repack.sh enforce the single entry (:153-155, :156, :160-164, :314-319). Keep the banned-token gate exactly as it is — it is the one that prevents a route to a debug/rescue/manual-disk-picker path, and a stock Graphical entry passes it unchanged.	grub.cfg still has no live proxtui/proxdebug/nomodeset/Rescue Boot/memtest/fwsetup; entry count assertion becomes an allowlist rather than == 1.
6	No first-boot hook on this image. Q1: it cannot fire. Shipping one would be a payload that provably never executes — the inert-seam failure this project has had four times.	/proxmox-first-boot absent from the ISO.
7	Day-0 is the documented manual command, and the release notes say so plainly. §5 shows the hub already emits it per customer and the URL is already public and current.	Release notes contain the felhom.eu/scripts/felhom-host-install.sh command; the hub customer page renders it (already true).
8	The profile the image uses must be COMMITTED. Spike 1 found demo-felhom's baked SSH key comes from an uncommitted profile, so a Tier 0 reference box is not reproducible from main.	--profile path resolves inside the repo tree and git status --porcelain is empty for it.
9	The artifact must be reproducible. Spike 1 R-144: nested-probe's profile no longer exists anywhere.	Manifest records the committed profile path + repo commit; both resolve.

Is "no answer.toml" achievable? Yes, and it is cheaper than the alternative. prepare-iso is what creates /answer.toml and /auto-installer-mode.toml; an image that simply is not passed through prepare-iso has neither. The public image then needs only the Felhom branding half of iso-repack.sh. What it costs: the --fetch-from/--on-first-boot machinery becomes unusable on this image, so any future automated delivery would need Q2's .deb route (§3) — which is untested.

---

7. What is still unknown, and what it would take

#	Unknown	What it takes
1	Q2's .deb mechanism is unproven (§3) — and it is now the only candidate delivery path for a public image. Two sub-questions: does prepare-iso/iso-repack preserve an added /proxmox/packages/*.deb, and can a postinst running in the installer chroot usefully install a unit?	Build a trivial .deb with a marker postinst, add it to the ISO's package dir, run an interactive install, check for the marker. ~45 min. Do this before any spec relies on §3.
2	Q3: the real stub at before-network (§4) — still argued, not observed. Now scoped to operator-built images only.	~40 min, as described in §4.
3	Whether the graphical installer behaves identically to the TUI. §2.2 measured the Terminal UI path. proxinstall has zero first-boot references (strictly less capability than the TUI's one), so the negative is a fortiori — but it is inference, not measurement.	Repeat §2.2 via the Graphical entry. ~30 min.
4	Whether an image never passed through prepare-iso still boots and brands correctly (§6, constraint 1). iso-repack.sh:100-106 currently refuses an ISO without auto-installer-mode.toml (exit 10), so this is a real code change, not just an omission.	Build one and boot it. Note this is the one constraint in §6 that today's tooling actively blocks.
5	Spike 1's two open items — multi-match determinism, and whether the installer excludes its own boot medium.	Unchanged.

---

8. R-rows opened

Continuing from R-152 (Spike 2).

R	Severity	Finding
R-153	MEDIUM	A working gate is red and nobody runs it. hub/internal/web/configs.go:28 has hostInstallVersion = "1.19.0" against SCRIPT_VERSION = "1.22.0", and scripts/hostinstall_gates.py correctly detects it and exits 1. The comment at configs.go:26 says drift "is now gated" — true of the gate's existence, false of its enforcement, because no pipeline invokes it. The customer page's install-command generator therefore targets a flag surface three minor versions stale.
R-154	MEDIUM	[first-boot] is automated-install-only, and nothing in the Felhom tree says so. Measured (§2) and cited (§2.3). answer.toml.tmpl:26-28 documents the hook as the day-0 delivery mechanism without recording that it is unavailable the moment anyone boots the Graphical or Terminal UI entry — which the same ISO offers. Any image that ships both paths and relies on [first-boot] silently delivers nothing on one of them.
R-155	LOW	iso-repack.sh:100-106 blocks the cleanest public-image posture. It refuses (exit 10) any ISO lacking auto-installer-mode.toml, on the sound reasoning that a single "Felhom telepítés" entry must not boot a manual installer. But §6's constraint 1 — ship no answer.toml at all — produces exactly such an ISO deliberately. The guard needs to distinguish "prepared, single-entry" from "unprepared, interactive-by-design" rather than treating the second as the first's failure mode.

---

9. Teardown

All three layers, verified positively.

Layer 1 — demo-hp.

Check	Result
QEMU processes	0
/mnt/nvme-1tb/spike3/	removed
/tmp/paia, /mnt/s3a, loop devices	absent / removed / 0
/mnt/nvme-1tb/ contents	dump e2d-images felhom-data images lost+found — the pre-spike set
disk usage	6.6 G — identical to pre-spike
drill-r50 (VM 300)	stopped, untouched
guest 9201	running, untouched
/etc/pve/storage.cfg	4 storages, unchanged; none registered
local-lvm	never written

Layer 2 — demo-felhom. Nothing was placed there this session; /tmp/paia confirmed absent, guest 9201 running, external USB drive 931.5G usb /mnt/hdd_1 still attached and mounted as found.

Layer 3 — DooPlex. Scratchpad 4.4 GB → 80 K; the probe ISO, squashfs extract, all screendumps and both throwaway passwords removed (shred -u on the answer-file password, verified absent; the interactive install's console password was written only to a 0600 file on demo-hp for sshpass and deleted in the same command). felhom-iso/out/ untouched at 17 ISOs. No production file modified — no build-felhom-iso.sh, no iso-repack.sh, no profile, no answer.toml.tmpl, no felhom-bootstrap.sh.

Hub-side disposition — nothing to dispose. No customer, host or appliance was created: every VM ran under plain QEMU user-mode networking with no route to the hub. Verified by fetching the customer list and searching it — 0 spike-named objects, the same single pre-existing customer. R-131 gains no row.

Published artifacts: none.