Files
felhom.eu/documentation/audits/SPIKE-universal-iso-3-2026-07-31.md
T
admin 4d6ec7c7bb
gates / gates (push) Successful in 14s
hub v0.104.0: the guest network gets a reader (R-319), and the hub half of the naming (R-295)
Four paper debts and one fact given a reader. Hub-only — nothing to bake.

A4 — the entry about "the tester's machine" named a risk correctly and labelled it
in a way that invited deleting it. Established from the hub's own store: `peti-felhom`
is a REAL machine (482 reports, 2026-02-27 → 2026-07-15, a named person's own box) and
the 3.6 GB with no key and no backup is real. `david` → `tester-1` is a DIFFERENT record
with no host, no escrow and no report, ever — deleted 07:55:49 and re-created 07:56:47
this morning. The prompt's premise conflated the two; the register now says which is which.

A1 — R-312/R-313/R-303 recorded as DECIDED with their re-open triggers, and moved out
of STATUS's "Waiting on you", which is now empty.

A3 — day0-install §C.1 said pushing the installer publishes it. It has not since
R-110. Corrected, with the two manifest pins named and an outside-verification command;
the one copy that repeated it (a dated audit, true when written) carries a superseded note.

A5 — standing rule 5: evidence comes off the machine at the end of the phase that
produced it, before any revert. Earned twice in three days on the same box at the same
point (R-320). Four homes, plus what to do when it is already gone.

R-295 hub half — „Beállító kód" everywhere; „Visszaállító kód" retired. New `reenroll`
mail kind so the mail names the page a REBUILT box actually shows („A szerver
beállítása"), not the „Elfelejtett jelszó" page it has no login screen to reach.
Naming only; the acceptance pin proves the secret is untouched.

R-319 — the hub models `guest_net` after 23 days of receiving and discarding it. The
signal is `heals_last_hour`, not `state`: a guest the watchdog keeps repairing reads
healthy between repairs. `heal_succeeded` decoded too (R-260's lesson). Unknown is never
drawn as healthy — three absences, three sentences. No alarm, deliberately.
Three red-proofs, mutations asserted applied. Wire-gate checked tags 182 → 190.

B1 — the operator's 2026-08-12 dispositions were NOT in the register; they are now.
Third allowlist kind for the five ruled "no reader wanted"; `reporting_disabled`
reclassified redundant. 8 read · 5 deliberately unread · 1 redundant · 6 still owed.

Also filed: R-321 (a deliberately-silent box still alarms stale/down — the checker is
age-only, and decoding the flag would not have fixed it), R-322 (the claim guard has
never scanned the hub; a hand scan returns zero, so it is a scope gap, not a defect).
2026-08-13 10:50:12 +02:00

299 lines
20 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# SPIKE 3 — on an interactive install, does the ISO deliver anything at all? (2026-07-31)
> **Class: Spike.** Findings only. No production file changed, no ISO built for release, nothing
> published.
>
> Continues `SPIKE-universal-iso-2026-07-31.md` and `SPIKE-universal-iso-2-2026-07-31.md`; their
> results are premises and are not re-derived.
**Q1 is answered, measured, with a control, and the answer is NO — decisively so.** The first-boot
hook does not merely fail to run on an interactive install: the mechanism is absent from the installed
system entirely, down to the package. Everything else in this document is contingent on that, and §3
is where the contingency lands: **a delivery mechanism does exist**, and it is the one PVE uses itself.
---
## 1. Baselines and fixture
| Baseline | Value |
|---|---|
| `ISO_VERSION` | `1.25.0``scripts/iso/build-felhom-iso.sh:44` |
| `SCRIPT_VERSION` | `1.22.0``scripts/felhom-host-install.sh:187` |
| `felhom-bootstrap.sh` @ HEAD | `21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7` |
| PVE base | `proxmox-ve_9.2-1.iso`, `4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c` |
| `felhom.eu` HEAD | **`19c932a`**, clean, `== origin/main` |
| `proxmox-first-boot` pkg | `9.2.5` (from the ISO's `/proxmox/packages/`) |
**Fixture provenance.** One probe ISO, built on DooPlex from the baseline PVE ISO with
`prepare-iso --fetch-from partition --partition-label proxmox-ais --on-first-boot hook.sh`, sha256
`62d8a4cc948b31850a8153c129b8763353ca7dd01c19c06c3816ebed89b7c028`, **verified byte-identical at both
ends** after transfer. It carries **15 GRUB entries** — Automated, Graphical, Terminal UI and the rest
— so the control and the test come from *the same build*, which is what makes the negative meaningful.
Nested VMs ran under plain QEMU/KVM on **demo-hp** (Tier 0, the designated drill host) under
`/mnt/nvme-1tb/spike3/`. Never `local-lvm`; no PVE storage registered; `drill-r50` and guest 9201
untouched. The automated install used `reboot-mode = "power-off"` as its completion signal; the
interactive install's completion was observed directly on the console.
**The hook leaves three independent positive observables** — a `ttyS0` line, a persistent marker file,
and its own log — because R-150 established that a first-boot hook which never runs is
indistinguishable from one that succeeded if you only look for errors.
**Evidence classes:** **[MEASURED]** on a real install · **[STRUCTURAL]** read from shipped installer
source · **[NOT REACHED]**.
---
## 2. Q1 — does `[first-boot]` fire on an interactive install? **[MEASURED] No.**
### 2.1 The control — same image, automated entry
| Observable | Result |
|---|---|
| `ttyS0` marker | **present**`@@SPIKE3_HOOK_FIRED@@ … uptime=19.31` |
| `/var/log/SPIKE3-HOOK-FIRED` | **present**, written at uptime 19.30 |
| `/var/lib/proxmox-first-boot/proxmox-first-boot` | **present**, 572 bytes, mode `0700` |
| `/etc/systemd/system/proxmox-first-boot.service` | **present**`/lib/systemd/system/proxmox-first-boot-multi-user.service` |
| unit state | `alias` / `active` |
| `pending-first-boot-setup` flag | **absent** — removed by the unit's `ExecStartPost`, i.e. it ran to completion |
So the hook demonstrably fires from this build.
### 2.2 The test — same image, Terminal UI entry, a normal manual install
Driven screen by screen with a screendump before every action. The installer behaved exactly as the
operator's ruling assumes: it **showed the target disk** (`/dev/sda (QEMU HARDDISK) (20.00 GiB)`) and
presented a summary listing `Bootdisk(s) /dev/sda` before erasing anything. The install completed and
auto-rebooted. Installed hostname `spikethree.felhom.eu`, which identifies the box below.
| Observable | Result |
|---|---|
| `ttyS0` marker | **absent** (`grep -c` = 0) |
| `/var/log/SPIKE3-HOOK-FIRED` | **absent**`No such file or directory` |
| `/var/lib/proxmox-first-boot/` | **does not exist** |
| `/etc/systemd/system/proxmox-first-boot.service` | **absent** |
| any `proxmox-first-boot*` unit known to systemd | **none** — only the stock `first-boot-complete.target` |
| **the `proxmox-first-boot` package** | **not installed**`dpkg-query: no packages found matching proxmox-first-boot` |
| `grep -rl SPIKE3_HOOK_FIRED /var /etc /root /usr/local` | **no hits** |
That is the search behind the negative, stated rather than asserted. **The hook did not fail to run —
nothing that could have run it was ever placed on the system.**
### 2.3 The mechanism — **[STRUCTURAL]**, cited
Three lines in the shipped installer explain it exactly.
1. `Proxmox/Install/Config.pm:118-122` — the default is
```perl
first_boot => { enabled => 0, ordering_target => 'multi-user' },
```
and `set_first_boot_opt` (`:300`) is **never called anywhere in the Perl tree**. The flag can only
become true through `Config::merge`, i.e. from the low-level config the auto-installer supplies.
2. `Proxmox/Install.pm:746` — `setup_proxmox_first_boot_service` opens with
```perl
return if !Proxmox::Install::Config::get_first_boot_opt('enabled');
```
so on any path that does not set the flag, nothing is planted.
3. `Proxmox/Install.pm:1360` — the package loop skips the package itself:
```perl
next if $deb =~ /^proxmox-first-boot/ && !needs_first_boot_package();
```
which is why §2.2 found the package missing, not merely unconfigured.
And the front-ends agree with the measurement: **`proxinstall` — the graphical interactive installer —
contains zero occurrences of `first-boot`**; `proxmox-low-level-installer` likewise zero;
`proxmox-tui-installer` has exactly one, the serde field name in the shared low-level config struct,
with no way to set it. `proxmox-auto-installer` has eight.
**Conclusion: `[first-boot]` is an automated-installer feature. It is unavailable on every interactive
path, by construction, and this is not a Felhom defect but a property of the PVE installer.**
---
## 3. Q2 — can the ISO plant anything into the installed system? **[STRUCTURAL] Yes — one mechanism exists.**
The search, stated: the installer's file-copy and package paths (`Proxmox/Install.pm`), the
`[first-boot]` plumbing (§2.3), the answer schema's `post-installation-webhook` (an `answer.toml` key,
so automated-only), and the `proxmox-post-hook` binary (part of the auto-install flow). Of these, one
is mode-independent.
**`Install.pm:1343-1372` installs every `.deb` in the ISO's `/proxmox/packages/` into the target**, on
every install path:
```perl
while (<${proxmox_pkgdir}/*.deb>) {
...
next if $deb =~ /grub-pc_/ && (...); # boot type / arch
next if $deb =~ /^proxmox-secure-boot-support_/ && !$run_env->{secure_boot};
next if $deb =~ /^proxmox-first-boot/ && !needs_first_boot_package();
next if $deb =~ /^(amd64|intel)-microcode_/ && (...); # cpu vendor
syscmd("chroot $targetdir dpkg ... --unpack /tmp/pkg/$deb") == 0 || die ...;
}
```
followed by `chroot $targetdir dpkg --force-confold --configure -a` (`:1378`) — **which runs postinst
scripts.**
The skip-list is a fixed set of known package-name patterns. A package whose name matches none of them
is unpacked and configured unconditionally. **So a `.deb` added to `/proxmox/packages/` is a working,
existing, mode-independent way to plant a systemd unit and a script into the installed system — and it
is precisely the mechanism PVE uses to deliver `proxmox-first-boot` itself.**
**Not tested. This is read from source, not measured**, and it is the single most important thing to
prove before any spec is written on it (§7). Two things in particular are unverified: that
`prepare-iso` preserves an added `/proxmox/packages/*.deb` through its repack, and that a postinst
running under `dpkg --configure -a` in the installer chroot can do what a first-boot hook does (it runs
inside the installer's chroot, **not** on a booted system, so it can install a unit but cannot expect
network, `pveproxy`, or a running systemd).
---
## 4. Q3 — the real stub at `before-network` — **[NOT REACHED]**
Not run. The interactive-install driving in §2.2 consumed the session's remaining room, and Q1 was the
question everything else was contingent on.
**What it needs** (unchanged from the Spike 2 estimate, ~40 min): build a probe ISO with
`stub-first-boot.sh` as the `--on-first-boot` payload and an answer with
`ordering = "before-network"`, install, boot, and confirm the stub's own log
(`/var/log/felhom-first-boot.log`) exists, that `felhom-bootstrap.service` is enabled and started, and
that pairing proceeds.
Spike 2's structural case is unchanged and unrefuted: the stub makes no `pvesh`/`pct`/`pveum`/`qm` call
(grep `rc=1`), and the unit it starts carries its own `After=network-online.target … pveproxy.service`.
**It remains argued, not observed.** Note §2 narrows where it matters: on the interactive path there is
no first-boot hook at all, so this question now governs **operator-built images only**.
---
## 5. Q4 — how good can the manual path be? **[MEASURED/STRUCTURAL] Two of three already exist.**
1. **The hub already renders the command with the customer-id filled in.**
`hub/internal/web/templates/customer_unified.html:563-564`:
```
curl -fsSL https://felhom.eu/scripts/felhom-host-install.sh -o felhom-host-install.sh \
&& sudo bash felhom-host-install.sh --customer-id {{.CustomerID}} --mode <appliance|byo>
```
with a JS generator at `:1258-1267` that rebuilds it per selected role, including the right flags.
**Nothing needs building here.**
2. **The URL is already short, public and always current.**
`https://felhom.eu/scripts/felhom-host-install.sh` is the website's git-sync working tree tracking
`main` on a 30-second period (`documentation/.../day0-install.md:150-152`) — no release tag, no
staging copy, no version selector. Pushing the script publishes it.
> **⚠ SUPERSEDED 2026-08-03, annotated 2026-08-13 (R-110, R-309). True on the day it was written;
> false ever since.** `/scripts/` is now served from the tag `installer-v<SCRIPT_VERSION>` and
> **pushing publishes nothing** — publication is a tag plus **two** `--ref` pins in
> `manifests/webpage.yaml`. The dated finding is kept as written rather than rewritten, because it
> is a record of what was true then; **the note is here because this paragraph cited
> `day0-install.md` as its source, which is how the wrong sentence spread.** Current procedure and
> the outside-verification command: `runbooks/day0-install.md` §C.1.
3. **Console display does not exist.** `felhom-host-install.sh` does not write `/etc/issue`,
`/etc/issue.net` or any MOTD (grep: no match). PVE's own `/etc/issue` banner is what the console
shows after install — observed in this session's screendumps as
*"Welcome to the Proxmox Virtual Environment … connect to: https://10.0.2.15:8006/"*. That file is
the obvious carrier for a Felhom line, and today nothing writes one. **Reported, not built.**
**An incidental finding while reading (2): the hub's version-sync gate is currently red.**
`hub/internal/web/configs.go:28` sets `hostInstallVersion = "1.19.0"` while `SCRIPT_VERSION` is
`1.22.0`, and the comment at `:26` states *"Drift is now gated: `scripts/hostinstall_gates.py` asserts
this const == SCRIPT_VERSION (drill F-1)."* Running that gate:
```
FAIL: hub Setup-tab hostInstallVersion=1.19.0 != SCRIPT_VERSION=1.22.0 (F-1: bump both together)
hostinstall gates: 1 FAILURE(S)
```
with a **true exit code of 1** (verified without a pipe — an initial `rc=0` reading was `tail`'s exit
status, not the script's, and is corrected here). So the gate is correct, works, and detects the drift;
**nothing runs it.** The comment's "is now gated" is true of the gate's existence and false of its
enforcement. **R-153.**
---
## 6. Q5 — what the public image contains, as constraints
Given the operator's ruling (no fully automatic install; disk selection always interactive) and Q1's
negative, stated as constraints on a build spec — **not an implementation, and nothing was built.**
| # | Constraint | Release gate (checkable against the built artifact) |
|---|---|---|
| 1 | **The public image carries NO `answer.toml`.** Q1 makes it pointless on the interactive path, and the ruling removes the automated path. Dropping it removes the baked root hash, the disk profile, and every question Spikes 12 were wrestling with, at once. | `osirrox -indev <iso> -find / -maxdepth 1` contains **no `/answer.toml`**. One line, trivially assertable. |
| 2 | **…and therefore no `auto-installer-mode.toml` either.** Without it the stock `grub.cfg` does not even emit the *Automated* entry (Spike 1 §5: that entry is inside `if [ -f auto-installer-mode.toml ]`). The menu becomes interactive-only *by construction*, with no gate to maintain. | `/auto-installer-mode.toml` absent; `grub.cfg` contains no `proxmox-start-auto-installer`. |
| 3 | **No baked credential of any kind.** Falls out of 1: no `root-password-hashed`, no `root-ssh-keys`, and **no `.rootpw.txt` companion emitted**. The installer prompts the human, who sets their own. | `grep -c root-password-hashed` and `grep -c root-ssh-keys` over the ISO both `0`; no `*.rootpw.txt` beside the output. |
| 4 | **The boot menu is restored from the collapse**, with the interactive entry as `default` and a **human-length timeout**. Spike 2 lost a probe to the installed system's **1-second** menu; a person at an unfamiliar screen needs far longer. 1030 s is the range the stock ISO itself uses (10). | `final.iso`'s `grub.cfg`: `set default=` points at an interactive entry, `set timeout` ≥ 10, and `timeout_style` uses the **underscore** spelling (the stock hyphenated `timeout-style` is not a GRUB variable at all — Spike 2 §5.2). |
| 5 | **Restoring the menu means relaxing gates, not deleting them.** Four gates in `iso-repack.sh` enforce the single entry (`:153-155`, `:156`, `:160-164`, `:314-319`). Keep the banned-token gate exactly as it is — it is the one that prevents a route to a debug/rescue/manual-disk-picker path, and a stock *Graphical* entry passes it unchanged. | `grub.cfg` still has no live `proxtui`/`proxdebug`/`nomodeset`/`Rescue Boot`/`memtest`/`fwsetup`; entry count assertion becomes an allowlist rather than `== 1`. |
| 6 | **No first-boot hook on this image.** Q1: it cannot fire. Shipping one would be a payload that provably never executes — the inert-seam failure this project has had four times. | `/proxmox-first-boot` absent from the ISO. |
| 7 | **Day-0 is the documented manual command**, and the release notes say so plainly. §5 shows the hub already emits it per customer and the URL is already public and current. | Release notes contain the `felhom.eu/scripts/felhom-host-install.sh` command; the hub customer page renders it (already true). |
| 8 | **The profile the image uses must be COMMITTED.** Spike 1 found demo-felhom's baked SSH key comes from an uncommitted profile, so a Tier 0 reference box is not reproducible from `main`. | `--profile` path resolves inside the repo tree **and** `git status --porcelain` is empty for it. |
| 9 | **The artifact must be reproducible.** Spike 1 R-144: `nested-probe`'s profile no longer exists anywhere. | Manifest records the committed profile path + repo commit; both resolve. |
**Is "no `answer.toml`" achievable?** Yes, and it is cheaper than the alternative. `prepare-iso` is what
creates `/answer.toml` and `/auto-installer-mode.toml`; **an image that simply is not passed through
`prepare-iso` has neither.** The public image then needs only the Felhom branding half of
`iso-repack.sh`. What it costs: the `--fetch-from`/`--on-first-boot` machinery becomes unusable on this
image, so any future automated delivery would need Q2's `.deb` route (§3) — which is untested.
---
## 7. What is still unknown, and what it would take
| # | Unknown | What it takes |
|---|---|---|
| 1 | **Q2's `.deb` mechanism is unproven** (§3) — and it is now the *only* candidate delivery path for a public image. Two sub-questions: does `prepare-iso`/`iso-repack` preserve an added `/proxmox/packages/*.deb`, and can a postinst running in the installer chroot usefully install a unit? | Build a trivial `.deb` with a marker postinst, add it to the ISO's package dir, run an interactive install, check for the marker. ~45 min. **Do this before any spec relies on §3.** |
| 2 | **Q3: the real stub at `before-network`** (§4) — still argued, not observed. Now scoped to operator-built images only. | ~40 min, as described in §4. |
| 3 | **Whether the graphical installer behaves identically to the TUI.** §2.2 measured the **Terminal UI** path. `proxinstall` has *zero* first-boot references (strictly less capability than the TUI's one), so the negative is a fortiori — but it is inference, not measurement. | Repeat §2.2 via the Graphical entry. ~30 min. |
| 4 | **Whether an image never passed through `prepare-iso` still boots and brands correctly** (§6, constraint 1). `iso-repack.sh:100-106` currently **refuses** an ISO without `auto-installer-mode.toml` (`exit 10`), so this is a real code change, not just an omission. | Build one and boot it. Note this is the one constraint in §6 that today's tooling actively blocks. |
| 5 | Spike 1's two open items — multi-match determinism, and whether the installer excludes its own boot medium. | Unchanged. |
---
## 8. R-rows opened
Continuing from R-152 (Spike 2).
| R | Severity | Finding |
|---|---|---|
| **R-153** | MEDIUM | **A working gate is red and nobody runs it.** `hub/internal/web/configs.go:28` has `hostInstallVersion = "1.19.0"` against `SCRIPT_VERSION = "1.22.0"`, and `scripts/hostinstall_gates.py` correctly detects it and exits 1. The comment at `configs.go:26` says drift "is now gated" — true of the gate's existence, false of its enforcement, because no pipeline invokes it. The customer page's install-command generator therefore targets a flag surface three minor versions stale. |
| **R-154** | MEDIUM | **`[first-boot]` is automated-install-only, and nothing in the Felhom tree says so.** Measured (§2) and cited (§2.3). `answer.toml.tmpl:26-28` documents the hook as the day-0 delivery mechanism without recording that it is unavailable the moment anyone boots the Graphical or Terminal UI entry — which the same ISO offers. Any image that ships both paths and relies on `[first-boot]` silently delivers nothing on one of them. |
| **R-155** | LOW | **`iso-repack.sh:100-106` blocks the cleanest public-image posture.** It refuses (`exit 10`) any ISO lacking `auto-installer-mode.toml`, on the sound reasoning that a single "Felhom telepítés" entry must not boot a manual installer. But §6's constraint 1 — ship no `answer.toml` at all — produces exactly such an ISO deliberately. The guard needs to distinguish "prepared, single-entry" from "unprepared, interactive-by-design" rather than treating the second as the first's failure mode. |
---
## 9. Teardown
All three layers, verified positively.
**Layer 1 — demo-hp.**
| Check | Result |
|---|---|
| QEMU processes | `0` |
| `/mnt/nvme-1tb/spike3/` | removed |
| `/tmp/paia`, `/mnt/s3a`, loop devices | absent / removed / `0` |
| `/mnt/nvme-1tb/` contents | `dump e2d-images felhom-data images lost+found` — the pre-spike set |
| disk usage | **6.6 G — identical to pre-spike** |
| `drill-r50` (VM 300) | `stopped`, untouched |
| guest 9201 | `running`, untouched |
| `/etc/pve/storage.cfg` | **4 storages, unchanged**; none registered |
| `local-lvm` | never written |
**Layer 2 — demo-felhom.** Nothing was placed there this session; `/tmp/paia` confirmed absent, guest
9201 `running`, external USB drive `931.5G usb /mnt/hdd_1` still attached and mounted as found.
**Layer 3 — DooPlex.** Scratchpad **4.4 GB → 80 K**; the probe ISO, squashfs extract, all screendumps
and both throwaway passwords removed (`shred -u` on the answer-file password, verified absent; the
interactive install's console password was written only to a `0600` file on demo-hp for `sshpass` and
deleted in the same command). `felhom-iso/out/` untouched at **17 ISOs**. No production file modified —
no `build-felhom-iso.sh`, no `iso-repack.sh`, no profile, no `answer.toml.tmpl`, no
`felhom-bootstrap.sh`.
**Hub-side disposition — nothing to dispose.** No customer, host or appliance was created: every VM ran
under plain QEMU user-mode networking with no route to the hub. Verified by fetching the customer list
and searching it — **0** spike-named objects, the same single pre-existing customer. **R-131 gains no
row.**
**Published artifacts: none.**