SPIKE 3: [first-boot] does NOT fire on an interactive install

Findings only — no script, profile or build file changed; no release ISO built, nothing published.
documentation/audits/SPIKE-universal-iso-3-2026-07-31.md

MEASURED with a control from the SAME image (one ISO, 15 GRUB entries):
- Automated entry  -> hook fires: ttyS0 marker, marker file,
  /var/lib/proxmox-first-boot/proxmox-first-boot (0700), activation symlink, unit active.
- Terminal-UI entry, normal manual install -> ALL absent, and the proxmox-first-boot PACKAGE is
  not installed at all. A whole-filesystem grep for the marker returns nothing.

Mechanism cited: Config.pm:118 defaults first_boot.enabled=0 and set_first_boot_opt is never
called in the Perl tree; Install.pm:746 returns early without it; Install.pm:1360 skips the
package. proxinstall (graphical) has ZERO occurrences of first-boot. [first-boot] is an
automated-installer feature, unavailable on every interactive path by construction. R-154.

A delivery mechanism DOES exist and is UNTESTED: Install.pm:1343-1372 unpacks every .deb in the
ISO's /proxmox/packages/ into the target on every path (fixed skip-list), then dpkg --configure -a
runs postinsts (:1378) — how PVE ships first-boot itself. Read from source, not measured.

Q5: the public image should carry NO answer.toml at all — that removes the baked root hash, the
disk profile and the whole Spike 1-2 problem space, and makes it a one-line release gate. But
iso-repack.sh:100-106 refuses an ISO without auto-installer-mode.toml. R-155.

Incidental R-153: hub hostInstallVersion=1.19.0 vs SCRIPT_VERSION=1.22.0; hostinstall_gates.py
detects it and exits 1 — the gate works, nothing runs it.

Q3 (real stub at before-network) was NOT reached and is recorded as not reached.
This commit is contained in:
2026-07-31 14:47:18 +02:00
parent 19c932a693
commit bb29186d62
3 changed files with 387 additions and 65 deletions
@@ -0,0 +1,290 @@
# SPIKE 3 — on an interactive install, does the ISO deliver anything at all? (2026-07-31)
> **Class: Spike.** Findings only. No production file changed, no ISO built for release, nothing
> published.
>
> Continues `SPIKE-universal-iso-2026-07-31.md` and `SPIKE-universal-iso-2-2026-07-31.md`; their
> results are premises and are not re-derived.
**Q1 is answered, measured, with a control, and the answer is NO — decisively so.** The first-boot
hook does not merely fail to run on an interactive install: the mechanism is absent from the installed
system entirely, down to the package. Everything else in this document is contingent on that, and §3
is where the contingency lands: **a delivery mechanism does exist**, and it is the one PVE uses itself.
---
## 1. Baselines and fixture
| Baseline | Value |
|---|---|
| `ISO_VERSION` | `1.25.0``scripts/iso/build-felhom-iso.sh:44` |
| `SCRIPT_VERSION` | `1.22.0``scripts/felhom-host-install.sh:187` |
| `felhom-bootstrap.sh` @ HEAD | `21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7` |
| PVE base | `proxmox-ve_9.2-1.iso`, `4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c` |
| `felhom.eu` HEAD | **`19c932a`**, clean, `== origin/main` |
| `proxmox-first-boot` pkg | `9.2.5` (from the ISO's `/proxmox/packages/`) |
**Fixture provenance.** One probe ISO, built on DooPlex from the baseline PVE ISO with
`prepare-iso --fetch-from partition --partition-label proxmox-ais --on-first-boot hook.sh`, sha256
`62d8a4cc948b31850a8153c129b8763353ca7dd01c19c06c3816ebed89b7c028`, **verified byte-identical at both
ends** after transfer. It carries **15 GRUB entries** — Automated, Graphical, Terminal UI and the rest
— so the control and the test come from *the same build*, which is what makes the negative meaningful.
Nested VMs ran under plain QEMU/KVM on **demo-hp** (Tier 0, the designated drill host) under
`/mnt/nvme-1tb/spike3/`. Never `local-lvm`; no PVE storage registered; `drill-r50` and guest 9201
untouched. The automated install used `reboot-mode = "power-off"` as its completion signal; the
interactive install's completion was observed directly on the console.
**The hook leaves three independent positive observables** — a `ttyS0` line, a persistent marker file,
and its own log — because R-150 established that a first-boot hook which never runs is
indistinguishable from one that succeeded if you only look for errors.
**Evidence classes:** **[MEASURED]** on a real install · **[STRUCTURAL]** read from shipped installer
source · **[NOT REACHED]**.
---
## 2. Q1 — does `[first-boot]` fire on an interactive install? **[MEASURED] No.**
### 2.1 The control — same image, automated entry
| Observable | Result |
|---|---|
| `ttyS0` marker | **present**`@@SPIKE3_HOOK_FIRED@@ … uptime=19.31` |
| `/var/log/SPIKE3-HOOK-FIRED` | **present**, written at uptime 19.30 |
| `/var/lib/proxmox-first-boot/proxmox-first-boot` | **present**, 572 bytes, mode `0700` |
| `/etc/systemd/system/proxmox-first-boot.service` | **present**`/lib/systemd/system/proxmox-first-boot-multi-user.service` |
| unit state | `alias` / `active` |
| `pending-first-boot-setup` flag | **absent** — removed by the unit's `ExecStartPost`, i.e. it ran to completion |
So the hook demonstrably fires from this build.
### 2.2 The test — same image, Terminal UI entry, a normal manual install
Driven screen by screen with a screendump before every action. The installer behaved exactly as the
operator's ruling assumes: it **showed the target disk** (`/dev/sda (QEMU HARDDISK) (20.00 GiB)`) and
presented a summary listing `Bootdisk(s) /dev/sda` before erasing anything. The install completed and
auto-rebooted. Installed hostname `spikethree.felhom.eu`, which identifies the box below.
| Observable | Result |
|---|---|
| `ttyS0` marker | **absent** (`grep -c` = 0) |
| `/var/log/SPIKE3-HOOK-FIRED` | **absent**`No such file or directory` |
| `/var/lib/proxmox-first-boot/` | **does not exist** |
| `/etc/systemd/system/proxmox-first-boot.service` | **absent** |
| any `proxmox-first-boot*` unit known to systemd | **none** — only the stock `first-boot-complete.target` |
| **the `proxmox-first-boot` package** | **not installed**`dpkg-query: no packages found matching proxmox-first-boot` |
| `grep -rl SPIKE3_HOOK_FIRED /var /etc /root /usr/local` | **no hits** |
That is the search behind the negative, stated rather than asserted. **The hook did not fail to run —
nothing that could have run it was ever placed on the system.**
### 2.3 The mechanism — **[STRUCTURAL]**, cited
Three lines in the shipped installer explain it exactly.
1. `Proxmox/Install/Config.pm:118-122` — the default is
```perl
first_boot => { enabled => 0, ordering_target => 'multi-user' },
```
and `set_first_boot_opt` (`:300`) is **never called anywhere in the Perl tree**. The flag can only
become true through `Config::merge`, i.e. from the low-level config the auto-installer supplies.
2. `Proxmox/Install.pm:746` — `setup_proxmox_first_boot_service` opens with
```perl
return if !Proxmox::Install::Config::get_first_boot_opt('enabled');
```
so on any path that does not set the flag, nothing is planted.
3. `Proxmox/Install.pm:1360` — the package loop skips the package itself:
```perl
next if $deb =~ /^proxmox-first-boot/ && !needs_first_boot_package();
```
which is why §2.2 found the package missing, not merely unconfigured.
And the front-ends agree with the measurement: **`proxinstall` — the graphical interactive installer —
contains zero occurrences of `first-boot`**; `proxmox-low-level-installer` likewise zero;
`proxmox-tui-installer` has exactly one, the serde field name in the shared low-level config struct,
with no way to set it. `proxmox-auto-installer` has eight.
**Conclusion: `[first-boot]` is an automated-installer feature. It is unavailable on every interactive
path, by construction, and this is not a Felhom defect but a property of the PVE installer.**
---
## 3. Q2 — can the ISO plant anything into the installed system? **[STRUCTURAL] Yes — one mechanism exists.**
The search, stated: the installer's file-copy and package paths (`Proxmox/Install.pm`), the
`[first-boot]` plumbing (§2.3), the answer schema's `post-installation-webhook` (an `answer.toml` key,
so automated-only), and the `proxmox-post-hook` binary (part of the auto-install flow). Of these, one
is mode-independent.
**`Install.pm:1343-1372` installs every `.deb` in the ISO's `/proxmox/packages/` into the target**, on
every install path:
```perl
while (<${proxmox_pkgdir}/*.deb>) {
...
next if $deb =~ /grub-pc_/ && (...); # boot type / arch
next if $deb =~ /^proxmox-secure-boot-support_/ && !$run_env->{secure_boot};
next if $deb =~ /^proxmox-first-boot/ && !needs_first_boot_package();
next if $deb =~ /^(amd64|intel)-microcode_/ && (...); # cpu vendor
syscmd("chroot $targetdir dpkg ... --unpack /tmp/pkg/$deb") == 0 || die ...;
}
```
followed by `chroot $targetdir dpkg --force-confold --configure -a` (`:1378`) — **which runs postinst
scripts.**
The skip-list is a fixed set of known package-name patterns. A package whose name matches none of them
is unpacked and configured unconditionally. **So a `.deb` added to `/proxmox/packages/` is a working,
existing, mode-independent way to plant a systemd unit and a script into the installed system — and it
is precisely the mechanism PVE uses to deliver `proxmox-first-boot` itself.**
**Not tested. This is read from source, not measured**, and it is the single most important thing to
prove before any spec is written on it (§7). Two things in particular are unverified: that
`prepare-iso` preserves an added `/proxmox/packages/*.deb` through its repack, and that a postinst
running under `dpkg --configure -a` in the installer chroot can do what a first-boot hook does (it runs
inside the installer's chroot, **not** on a booted system, so it can install a unit but cannot expect
network, `pveproxy`, or a running systemd).
---
## 4. Q3 — the real stub at `before-network` — **[NOT REACHED]**
Not run. The interactive-install driving in §2.2 consumed the session's remaining room, and Q1 was the
question everything else was contingent on.
**What it needs** (unchanged from the Spike 2 estimate, ~40 min): build a probe ISO with
`stub-first-boot.sh` as the `--on-first-boot` payload and an answer with
`ordering = "before-network"`, install, boot, and confirm the stub's own log
(`/var/log/felhom-first-boot.log`) exists, that `felhom-bootstrap.service` is enabled and started, and
that pairing proceeds.
Spike 2's structural case is unchanged and unrefuted: the stub makes no `pvesh`/`pct`/`pveum`/`qm` call
(grep `rc=1`), and the unit it starts carries its own `After=network-online.target … pveproxy.service`.
**It remains argued, not observed.** Note §2 narrows where it matters: on the interactive path there is
no first-boot hook at all, so this question now governs **operator-built images only**.
---
## 5. Q4 — how good can the manual path be? **[MEASURED/STRUCTURAL] Two of three already exist.**
1. **The hub already renders the command with the customer-id filled in.**
`hub/internal/web/templates/customer_unified.html:563-564`:
```
curl -fsSL https://felhom.eu/scripts/felhom-host-install.sh -o felhom-host-install.sh \
&& sudo bash felhom-host-install.sh --customer-id {{.CustomerID}} --mode <appliance|byo>
```
with a JS generator at `:1258-1267` that rebuilds it per selected role, including the right flags.
**Nothing needs building here.**
2. **The URL is already short, public and always current.**
`https://felhom.eu/scripts/felhom-host-install.sh` is the website's git-sync working tree tracking
`main` on a 30-second period (`documentation/.../day0-install.md:150-152`) — no release tag, no
staging copy, no version selector. Pushing the script publishes it.
3. **Console display does not exist.** `felhom-host-install.sh` does not write `/etc/issue`,
`/etc/issue.net` or any MOTD (grep: no match). PVE's own `/etc/issue` banner is what the console
shows after install — observed in this session's screendumps as
*"Welcome to the Proxmox Virtual Environment … connect to: https://10.0.2.15:8006/"*. That file is
the obvious carrier for a Felhom line, and today nothing writes one. **Reported, not built.**
**An incidental finding while reading (2): the hub's version-sync gate is currently red.**
`hub/internal/web/configs.go:28` sets `hostInstallVersion = "1.19.0"` while `SCRIPT_VERSION` is
`1.22.0`, and the comment at `:26` states *"Drift is now gated: `scripts/hostinstall_gates.py` asserts
this const == SCRIPT_VERSION (drill F-1)."* Running that gate:
```
FAIL: hub Setup-tab hostInstallVersion=1.19.0 != SCRIPT_VERSION=1.22.0 (F-1: bump both together)
hostinstall gates: 1 FAILURE(S)
```
with a **true exit code of 1** (verified without a pipe — an initial `rc=0` reading was `tail`'s exit
status, not the script's, and is corrected here). So the gate is correct, works, and detects the drift;
**nothing runs it.** The comment's "is now gated" is true of the gate's existence and false of its
enforcement. **R-153.**
---
## 6. Q5 — what the public image contains, as constraints
Given the operator's ruling (no fully automatic install; disk selection always interactive) and Q1's
negative, stated as constraints on a build spec — **not an implementation, and nothing was built.**
| # | Constraint | Release gate (checkable against the built artifact) |
|---|---|---|
| 1 | **The public image carries NO `answer.toml`.** Q1 makes it pointless on the interactive path, and the ruling removes the automated path. Dropping it removes the baked root hash, the disk profile, and every question Spikes 12 were wrestling with, at once. | `osirrox -indev <iso> -find / -maxdepth 1` contains **no `/answer.toml`**. One line, trivially assertable. |
| 2 | **…and therefore no `auto-installer-mode.toml` either.** Without it the stock `grub.cfg` does not even emit the *Automated* entry (Spike 1 §5: that entry is inside `if [ -f auto-installer-mode.toml ]`). The menu becomes interactive-only *by construction*, with no gate to maintain. | `/auto-installer-mode.toml` absent; `grub.cfg` contains no `proxmox-start-auto-installer`. |
| 3 | **No baked credential of any kind.** Falls out of 1: no `root-password-hashed`, no `root-ssh-keys`, and **no `.rootpw.txt` companion emitted**. The installer prompts the human, who sets their own. | `grep -c root-password-hashed` and `grep -c root-ssh-keys` over the ISO both `0`; no `*.rootpw.txt` beside the output. |
| 4 | **The boot menu is restored from the collapse**, with the interactive entry as `default` and a **human-length timeout**. Spike 2 lost a probe to the installed system's **1-second** menu; a person at an unfamiliar screen needs far longer. 1030 s is the range the stock ISO itself uses (10). | `final.iso`'s `grub.cfg`: `set default=` points at an interactive entry, `set timeout` ≥ 10, and `timeout_style` uses the **underscore** spelling (the stock hyphenated `timeout-style` is not a GRUB variable at all — Spike 2 §5.2). |
| 5 | **Restoring the menu means relaxing gates, not deleting them.** Four gates in `iso-repack.sh` enforce the single entry (`:153-155`, `:156`, `:160-164`, `:314-319`). Keep the banned-token gate exactly as it is — it is the one that prevents a route to a debug/rescue/manual-disk-picker path, and a stock *Graphical* entry passes it unchanged. | `grub.cfg` still has no live `proxtui`/`proxdebug`/`nomodeset`/`Rescue Boot`/`memtest`/`fwsetup`; entry count assertion becomes an allowlist rather than `== 1`. |
| 6 | **No first-boot hook on this image.** Q1: it cannot fire. Shipping one would be a payload that provably never executes — the inert-seam failure this project has had four times. | `/proxmox-first-boot` absent from the ISO. |
| 7 | **Day-0 is the documented manual command**, and the release notes say so plainly. §5 shows the hub already emits it per customer and the URL is already public and current. | Release notes contain the `felhom.eu/scripts/felhom-host-install.sh` command; the hub customer page renders it (already true). |
| 8 | **The profile the image uses must be COMMITTED.** Spike 1 found demo-felhom's baked SSH key comes from an uncommitted profile, so a Tier 0 reference box is not reproducible from `main`. | `--profile` path resolves inside the repo tree **and** `git status --porcelain` is empty for it. |
| 9 | **The artifact must be reproducible.** Spike 1 R-144: `nested-probe`'s profile no longer exists anywhere. | Manifest records the committed profile path + repo commit; both resolve. |
**Is "no `answer.toml`" achievable?** Yes, and it is cheaper than the alternative. `prepare-iso` is what
creates `/answer.toml` and `/auto-installer-mode.toml`; **an image that simply is not passed through
`prepare-iso` has neither.** The public image then needs only the Felhom branding half of
`iso-repack.sh`. What it costs: the `--fetch-from`/`--on-first-boot` machinery becomes unusable on this
image, so any future automated delivery would need Q2's `.deb` route (§3) — which is untested.
---
## 7. What is still unknown, and what it would take
| # | Unknown | What it takes |
|---|---|---|
| 1 | **Q2's `.deb` mechanism is unproven** (§3) — and it is now the *only* candidate delivery path for a public image. Two sub-questions: does `prepare-iso`/`iso-repack` preserve an added `/proxmox/packages/*.deb`, and can a postinst running in the installer chroot usefully install a unit? | Build a trivial `.deb` with a marker postinst, add it to the ISO's package dir, run an interactive install, check for the marker. ~45 min. **Do this before any spec relies on §3.** |
| 2 | **Q3: the real stub at `before-network`** (§4) — still argued, not observed. Now scoped to operator-built images only. | ~40 min, as described in §4. |
| 3 | **Whether the graphical installer behaves identically to the TUI.** §2.2 measured the **Terminal UI** path. `proxinstall` has *zero* first-boot references (strictly less capability than the TUI's one), so the negative is a fortiori — but it is inference, not measurement. | Repeat §2.2 via the Graphical entry. ~30 min. |
| 4 | **Whether an image never passed through `prepare-iso` still boots and brands correctly** (§6, constraint 1). `iso-repack.sh:100-106` currently **refuses** an ISO without `auto-installer-mode.toml` (`exit 10`), so this is a real code change, not just an omission. | Build one and boot it. Note this is the one constraint in §6 that today's tooling actively blocks. |
| 5 | Spike 1's two open items — multi-match determinism, and whether the installer excludes its own boot medium. | Unchanged. |
---
## 8. R-rows opened
Continuing from R-152 (Spike 2).
| R | Severity | Finding |
|---|---|---|
| **R-153** | MEDIUM | **A working gate is red and nobody runs it.** `hub/internal/web/configs.go:28` has `hostInstallVersion = "1.19.0"` against `SCRIPT_VERSION = "1.22.0"`, and `scripts/hostinstall_gates.py` correctly detects it and exits 1. The comment at `configs.go:26` says drift "is now gated" — true of the gate's existence, false of its enforcement, because no pipeline invokes it. The customer page's install-command generator therefore targets a flag surface three minor versions stale. |
| **R-154** | MEDIUM | **`[first-boot]` is automated-install-only, and nothing in the Felhom tree says so.** Measured (§2) and cited (§2.3). `answer.toml.tmpl:26-28` documents the hook as the day-0 delivery mechanism without recording that it is unavailable the moment anyone boots the Graphical or Terminal UI entry — which the same ISO offers. Any image that ships both paths and relies on `[first-boot]` silently delivers nothing on one of them. |
| **R-155** | LOW | **`iso-repack.sh:100-106` blocks the cleanest public-image posture.** It refuses (`exit 10`) any ISO lacking `auto-installer-mode.toml`, on the sound reasoning that a single "Felhom telepítés" entry must not boot a manual installer. But §6's constraint 1 — ship no `answer.toml` at all — produces exactly such an ISO deliberately. The guard needs to distinguish "prepared, single-entry" from "unprepared, interactive-by-design" rather than treating the second as the first's failure mode. |
---
## 9. Teardown
All three layers, verified positively.
**Layer 1 — demo-hp.**
| Check | Result |
|---|---|
| QEMU processes | `0` |
| `/mnt/nvme-1tb/spike3/` | removed |
| `/tmp/paia`, `/mnt/s3a`, loop devices | absent / removed / `0` |
| `/mnt/nvme-1tb/` contents | `dump e2d-images felhom-data images lost+found` — the pre-spike set |
| disk usage | **6.6 G — identical to pre-spike** |
| `drill-r50` (VM 300) | `stopped`, untouched |
| guest 9201 | `running`, untouched |
| `/etc/pve/storage.cfg` | **4 storages, unchanged**; none registered |
| `local-lvm` | never written |
**Layer 2 — demo-felhom.** Nothing was placed there this session; `/tmp/paia` confirmed absent, guest
9201 `running`, external USB drive `931.5G usb /mnt/hdd_1` still attached and mounted as found.
**Layer 3 — DooPlex.** Scratchpad **4.4 GB → 80 K**; the probe ISO, squashfs extract, all screendumps
and both throwaway passwords removed (`shred -u` on the answer-file password, verified absent; the
interactive install's console password was written only to a `0600` file on demo-hp for `sshpass` and
deleted in the same command). `felhom-iso/out/` untouched at **17 ISOs**. No production file modified —
no `build-felhom-iso.sh`, no `iso-repack.sh`, no profile, no `answer.toml.tmpl`, no
`felhom-bootstrap.sh`.
**Hub-side disposition — nothing to dispose.** No customer, host or appliance was created: every VM ran
under plain QEMU user-mode networking with no route to the hub. Verified by fetching the customer list
and searching it — **0** spike-named objects, the same single pre-existing customer. **R-131 gains no
row.**
**Published artifacts: none.**