admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

SPIKE 4: a .deb in the ISO DOES deliver on an interactive install

Browse Source 
Findings only — no script, profile or build file changed; no release ISO built, nothing published.
documentation/audits/SPIKE-universal-iso-4-2026-07-31.md

MEASURED, with a control, and the negative control is in the SAME box. One ISO (15 GRUB entries),
a trivial probe .deb injected into /proxmox/packages/, two qm-created VMs on demo-hp (400
interactive / 401 automated control) on a scratch dir storage at the /mnt/nvme-1tb mount ROOT.

Interactive (Terminal UI) install:
- package installed (ii felhom-spike4-probe 0.0.1)
- postinst RAN (marker + content intact)
- it enabled a systemd unit, and that unit FIRED ON FIRST BOOT (uptime 7.98s, pid1=systemd)
- while on the same machine proxmox-first-boot is NOT installed and /var/lib/proxmox-first-boot
  does not exist — Spike 3's negative reproduced, not assumed.

Postinst environment (identical both paths): pid1=unconfigured.sh, NO running systemd, but
'systemctl enable' SUCCEEDS; /proc+/sys mounted; network+DNS happened to be up (inherited from the
installer's DHCP — must NOT be relied on). Constraints: never systemctl start/daemon-reload, never
require network, never fail, do the real work in the unit at first boot.

Repack preserves it, but a naive 'xorriso -boot_image any replay' fails with 'Overlapping MBR
partition entries' — iso-repack.sh:270-292 already documents that exact failure and its fix.

R-153 RETRACTED into R-94 leg (b): OPEN-ITEMS.md:15 carries it verbatim at READY (XS), and R-29
says explicitly 'do not mint a new ID for a new instance'. Spike 3's further claim that the drift
leaves the generator 'three minor versions stale' was FALSE and is corrected — R-94 retracts that
exact reading; the served script is always main, so 1.22.0 is what every install already gets.

No new R-rows opened.
This commit is contained in:
authentik Default Admin
2026-07-31 15:35:07 +02:00
parent bb29186d62
commit 61e9b55737
3 changed files with 411 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/audits/SPIKE-universal-iso-4-2026-07-31.md
+304
View File
@@ -0,0 +1,304 @@
# SPIKE 4 — can a `.deb` in the ISO deliver the stub on an interactive install? (2026-07-31)
> **Class: Spike.** Findings only. No production file changed, no release ISO built, nothing published.
>
> Continues `SPIKE-universal-iso-{1,2,3}-2026-07-31.md`. Spike 3's Q1 result and its `Install.pm`
> reading are premises and are not re-derived.
**Q1 is YES, measured, with a control — and the negative control sits in the same box.** On a normal
interactive install the Felhom `.deb` is installed, its postinst runs, it enables a systemd unit, and
**that unit fires on first boot** — while on that same machine the PVE first-boot hook is absent
exactly as Spike 3 measured. The two mechanisms are independent, and the one that survives the
interactive path is the `.deb`.
---
## 1. Venue — stated first, while it exists
| | |
|---|---|
| **Host** | `demo-hp` (t740), Tier 0, the designated drill + build VM host |
| **VMs** | **400 `spike4-interactive`** (the test) · **401 `spike4-auto-control`** (the control) |
| **Storage** | **`spike4`**, a `dir` storage created for this run at **`/mnt/nvme-1tb` — the mount ROOT, not a subdirectory**, `content=images` |
| **Why the root** | the agent's `exactMount` check reads a storage at a *subdirectory* of a mountpoint as `disconnected` forever. Placing it at the root avoids that. It coexists with the pre-existing `felhom-backup` dir storage on the same path (`content=backup`), which was **not modified** — the demo box's configuration is fenced |
| **Disks** | `/mnt/nvme-1tb/images/{400,401}/vm-*-disk-*.raw`. **Nothing on `local-lvm`** |
| **Console** | web console → VM → Console, or `qm terminal <vmid>` / `qm monitor <vmid>` on demo-hp. Both VMs were created with `qm`, so they appear in `qm list` and the web UI — unlike Spikes 1–3, which used raw QEMU processes PVE could not see |
| **Teardown** | VMs destroyed and the `spike4` storage removed — §7 |
### Baselines
| Baseline | Value |
|---|---|
| `ISO_VERSION` | `1.25.0` — `scripts/iso/build-felhom-iso.sh:44` |
| `SCRIPT_VERSION` | `1.22.0` — `scripts/felhom-host-install.sh:187` |
| `felhom-bootstrap.sh` @ HEAD | `21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7` |
| PVE base | `proxmox-ve_9.2-1.iso`, `4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c` |
| `felhom.eu` HEAD | **`bb29186`**, clean, `== origin/main` |
### Fixture provenance
One ISO for both paths. Built on DooPlex: baseline PVE ISO → `prepare-iso --fetch-from partition
--partition-label proxmox-ais --on-first-boot hook.sh` → **the probe `.deb` injected into
`/proxmox/packages/`** → re-mastered. Final sha256
`6f723910239bb7f333c952c1d1933ae3951f03c5b6cc536f15e33a8da37eba5f`, **verified identical after
transfer** to demo-hp. **15 GRUB menu entries** (Automated + Graphical + Terminal UI + the rest), so
control and test come from the same build.
The probe `.deb` (`felhom-spike4-probe_0.0.1_all.deb`, 2016 bytes) carries **no Felhom logic** — a
postinst that writes a marker and reports its environment, plus a trivial systemd unit and script. It
tests delivery, not payload.
**Completion signals:** the automated control used `reboot-mode = "power-off"` (guest powers itself
off); the interactive install's completion was observed on the console and then by an SSH banner from
the installed system. **No install was judged complete from a changed artifact** (Spike 1's `LastRun`
lesson) — and that mattered here: an ARP reply from `192.168.0.100` initially looked like the installed
system and was in fact the **installer environment**, caught by screendumping the console, which showed
the install still at 67 %.
---
## 2. Q1 — does the `.deb` reach the installed system? **YES, both observables, on both paths.**
The two observables are kept separate because "package present, postinst did not run" is a distinct
and important outcome.
| Observable | **Control — Automated entry** (VM 401, `spike4auto.local`) | **Test — Terminal UI entry** (VM 400, `spikefour.felhom.eu`) |
|---|---|---|
| **A. package installed** (`dpkg -l`) | `ii felhom-spike4-probe 0.0.1` | **`ii felhom-spike4-probe 0.0.1`** |
| **B. postinst ran** (marker + content) | `/var/log/SPIKE4-POSTINST`, 543 B, intact | **`/var/log/SPIKE4-POSTINST`, 542 B, intact** |
| unit enabled | `enabled` | **`enabled`** |
| **unit fired on FIRST BOOT** | `@@SPIKE4_FIRSTBOOT_RAN@@`, uptime 7.92 s | **`@@SPIKE4_FIRSTBOOT_RAN@@`, uptime 7.98 s** |
| PVE first-boot hook (`SPIKE4-HOOK`) | present — `@@SPIKE4_HOOK_FIRED@@` | **absent** |
| `proxmox-first-boot` package | installed | **`no packages found`** |
| `/var/lib/proxmox-first-boot/` | present | **absent** |
**The last three rows are the point.** On one machine, from one image, the `.deb` arrived and the PVE
first-boot hook did not. Spike 3's negative is reproduced here rather than assumed, and the `.deb`
mechanism is proven independent of it.
**Did the repack preserve the `.deb`?** Yes, and the answer is more useful than a simple yes.
- A naive in-place `xorriso … -boot_image any replay -map …` **fails**:
`libisofs: FAILURE : Overlapping MBR partition entries requested`.
- `iso-repack.sh:270-292` **documents this exact failure and its fix**: re-master from the ISO's own
`-report_el_torito as_mkisofs` report with `-V`, `--modification-date=`, `-apm-block-size`,
`-hfsplus`, `-part_like_isohybrid` and `-isohybrid-gpt-basdat` filtered out, pinning the volume id
and modification-date so the embedded fs-uuid stays valid.
- Mirroring that in a throwaway script produced a working image: **19 El Torito boot entries**
(`iso-repack.sh:298` requires ≥2), 543 `.deb`s, and the probe `.deb` extracted back out of the
finished ISO is **byte-identical** to the input (`sha256` first 32: `c624acec93aa9450ac168d39e877807f`
both sides).
So the delivery path is: **add the `.deb` to `/proxmox/packages/` in the extract tree, and re-master
with the existing recipe.** Felhom's own repack already performs that extract→modify→re-master cycle,
so this is an insertion into a step that runs anyway — not a new build stage.
**Mechanism, from Spike 3 and confirmed visually here:** `Install.pm:1343-1372` unpacks every `.deb`
in `/proxmox/packages/` with a fixed skip-list of known package-name patterns, then
`dpkg --force-confold --configure -a` at `:1378` runs the postinsts. The console screendump caught the
loop mid-run (`67 %`, `extracting mesa-libgallium_25.0.7-2_amd64.deb`).
---
## 3. Q2 — what a postinst can do from the installer chroot
The postinst reported its own environment. **Identical on both paths**, which is itself worth noting —
the chroot does not differ between automated and interactive installs:
| Property | Value | Consequence for a real postinst |
|---|---|---|
| `pid1_comm` | **`unconfigured.sh`** | it is the installer chroot, not a booted system |
| `systemd_running` (`/run/systemd/system`) | **`no`** | **`systemctl start` / `daemon-reload` are meaningless — never call them** |
| `systemctl_path` | `/bin/systemctl` | present, and usable in *offline* mode |
| **`systemctl enable`** | **SUCCEEDED** — `Created symlink '/etc/systemd/system/multi-user.target.wants/spike4-firstboot.service' → …` | **enabling a unit works; no need to hand-write the symlink** |
| `/proc`, `/sys` | both mounted | ordinary file/proc operations are fine |
| `default_route` | `default via 192.168.0.1 dev ens18` | network *happened* to be up — see the caveat below |
| `dns_resolvable` | `yes` | ditto |
| postinst arg | `configure` | standard `dpkg` first-install invocation |
**Answering Q2.2 directly: yes.** The realistic job — drop a unit, enable it, exit — works, and the
unit **fired on first boot** on both installs (uptime ≈ 8 s, `pid1: systemd`, network up).
**The constraints a real postinst must respect**, so they are written against rather than discovered:
1. **Never `systemctl start`, `daemon-reload`, or anything needing a running systemd.** There isn't
one. `enable` is the only systemd verb available.
2. **Never require the network, even though it was present.** It was present only because the
*installer* had DHCP and the chroot inherits its network namespace. A box installed with the cable
out, or on a LAN with no DHCP, gives a postinst with no route — and a postinst that fails takes the
whole install down. Fetch nothing; ship what is needed inside the package.
3. **Never fail.** `dpkg --configure -a` runs it; a non-zero exit surfaces as an install error to a
customer mid-install. Guard everything and `exit 0`.
4. **Do the real work in the unit at first boot, not in the postinst.** The postinst's only job is to
place files and enable the unit. That is also where network, systemd and a booted kernel actually
exist.
5. **Ordering within `dpkg --configure -a` was not tested.** The probe depended on nothing. A package
that needs another configured first is unproven here — see §6.
---
## 4. Q3 — can the image ship with no `answer.toml`?
**Where it refuses — cited.** `scripts/iso/iso-repack.sh:100-106`:
```bash
AIM="$(find "$EX" -maxdepth 2 -iname 'auto-installer-mode.toml' | head -1)"
[[ -n "$AIM" ]] || {
echo "iso-repack: auto-installer-mode.toml not found in the ISO — this is not a prepared" >&2
echo " auto-install ISO, so the single Felhom entry would boot the MANUAL installer. Refusing." >&2
exit 10
}
```
**It is a guard, not a structural requirement**, and its reasoning is sound *for the shape it was
written for*: the Felhom menu emits exactly one entry which boots the automated installer, so an ISO
without the mode file would put a "Felhom telepítés" label on a manual disk-picker. That is precisely
the outcome the single-entry menu exists to prevent.
Under the operator's ruling the premise is inverted — the menu should offer the interactive installer
deliberately — so the guard now blocks the desired shape. It needs to distinguish *"prepared,
single-entry"* from *"unprepared, interactive-by-design"* rather than treating the second as the
first's failure. Already filed as **R-155**; this session cites the exact refusal.
**Does the automated entry degrade gracefully with no answer file?** Two measurements bear on it:
- **With no `auto-installer-mode.toml` at all**, the stock `grub.cfg` does not emit the Automated entry
— it sits inside `if [ -f auto-installer-mode.toml ]` (Spike 1 §5). So the entry is *absent*, not
broken. Nothing to hang.
- **With the mode file but no reachable answer**, observed incidentally this session (Spike 3's VM
rebooting into the CD): the automated entry **aborts safely and loudly** —
`INFO: Testing partition search path "/dev/disk/by-label/proxmox-ais"` … `ERROR: Aborting: Could not
find any answer file!` … `Installation aborted - unable to continue (type exit or CTRL-D to reboot)`,
leaving a shell and touching no disk.
**Not measured: whether an ISO that never went through `prepare-iso` boots and installs correctly after
Felhom branding.** That needs the guard relaxed, which is a production-code change this spike is fenced
from making. See §6.
**Release gate, if the posture is adopted:** `osirrox -indev <iso> -find / -maxdepth 1` contains
**neither `/answer.toml` nor `/auto-installer-mode.toml`**. One line, checkable against the artifact.
---
## 5. Q4 — the resulting product shape, as constraints
Given the ruling (no fully automatic install) and Q1/Q2 positive. **Constraints on a build spec — not
an implementation, and nothing was built.**
| # | Constraint | Release gate (checkable against the artifact) |
|---|---|---|
| 1 | **No `answer.toml`, and no `auto-installer-mode.toml`.** Q1 makes the first-boot hook pointless on the shipped path; the ruling removes the automated path. Dropping both deletes the baked root hash, the disk profile and the whole Spike 1–2 problem space at once. | `osirrox -find / -maxdepth 1` contains neither file. |
| 2 | **A Felhom `.deb` in `/proxmox/packages/` is the delivery mechanism.** Measured (§2): it arrives and its postinst runs on the interactive path. | the ISO contains exactly one `felhom-*.deb`; extracting it back yields a sha256 matching the built package. |
| 3 | **The postinst places files and enables a unit. Nothing else.** §3's five constraints — no running systemd, no network dependency, never fail, real work in the unit. | the packaged postinst contains no `systemctl start`, no `daemon-reload`, and no network fetch (`curl`/`wget`/`apt`); it ends `exit 0`. |
| 4 | **No baked credential of any kind** — falls out of 1. The human sets their own password in the installer. | `grep -c root-password-hashed` and `grep -c root-ssh-keys` over the ISO both `0`; no `.rootpw.txt` emitted beside the output. |
| 5 | **The boot menu is restored, with an interactive entry as default and a human-length timeout.** Spike 2 lost a probe to the installed system's **1-second** menu; a person choosing between install paths needs far longer. The stock ISO uses 10 s. | `grub.cfg`: `set default=` points at an interactive entry; `set timeout` ≥ 10; `timeout_style` uses the **underscore** spelling (the stock hyphenated `timeout-style` is not a GRUB variable at all — Spike 2 §5.2). |
| 6 | **Relaxing the menu gates means relaxing the count gates only.** Four gates enforce the single entry (`iso-repack.sh:153-155`, `:156`, `:160-164`, `:314-319`). **Keep the banned-token gate exactly as it is** — it is what prevents a route to a debug/rescue/manual-disk-picker path, and a stock *Graphical* entry passes it unchanged. | `grub.cfg` still has no live `proxtui`/`proxdebug`/`nomodeset`/`Rescue Boot`/`memtest`/`fwsetup`; the entry-count assertion becomes an allowlist rather than `== 1`. |
| 7 | **`iso-repack.sh:100-106` must learn the new shape** (R-155) before constraint 1 is buildable. | the repack accepts an unprepared ISO when branding an interactive-by-design image, and still refuses one when emitting a single automated entry. |
| 8 | **The profile the public image uses must be COMMITTED.** Spike 1 found demo-felhom's baked SSH key comes from an uncommitted profile, so a Tier 0 reference box is not reproducible from `main`. | `--profile` resolves inside the repo tree **and** `git status --porcelain` is empty for it. |
| 9 | **The artifact must be reproducible** (R-144: `nested-probe`'s profile no longer exists). | the manifest records the committed profile path + repo commit; both resolve. |
**The product this describes:** insert the stick, install Proxmox normally choosing your own disk and
password, and the box sets itself up and waits for a claim code. **That is now measured, not hoped
for** — with the caveat that what was measured is a trivial probe package, not Felhom's real stub
(§6).
---
## 6. R-153 / R-94 — reconciled. **R-153 is retracted.**
The register grep the fence requires was run, and it shows Spike 3 should not have minted a row.
- **`OPEN-ITEMS.md:15` — R-94**, status **`READY (XS)`**, states the finding verbatim:
*"A hand-synced version constant drifts, and the gate that would catch it is never run —
`hub/internal/web/configs.go:28` pins `hostInstallVersion = "1.19.0"` while
`scripts/felhom-host-install.sh:187` is `SCRIPT_VERSION="1.22.0"`"*, with **leg (b)** being exactly
*"`scripts/hostinstall_gates.py` fails today and is invoked by no Makefile, hook or `CLAUDE.md` —
wire it or delete it."*
- **`OPEN-ITEMS.md:27` — R-29** is the **class**, and says in terms:
*"do not mint a new ID for a new instance"* — it already absorbed a third independent re-raise on
2026-07-29 without minting one, and **already names `scripts/hostinstall_gates.py` as R-94 leg (b)**.
**So R-153 duplicates R-94 leg (b) and violates R-29's explicit instruction. It is retracted here and
should be struck from Spike 3's row list.** Nothing is lost: R-94 covers it in more detail.
**A second correction, and this one matters more.** Spike 3's report claimed the drift meant *"the
customer page's install-command generator targets a flag surface three minor versions stale."*
**That is false, and R-94 explicitly retracts exactly that reading.** The constant selects no script:
its only consumers are `configs.go:487` and `render_test.go:219`, and it renders as a **text label** at
`customer_unified.html:494`. The command beneath it fetches
`https://felhom.eu/scripts/felhom-host-install.sh`, which the website git-syncs from `main` on a 30 s
period — **so 1.22.0 is what every install already gets**, and every flag the generator emits is parsed
by it. The defect is a wrong number on the operator's screen, nothing more. Spike 3's sentence
overstated it and is corrected here.
**The lesson for this arc:** the fence that caught this — *grep the register before minting* — worked,
and it worked one session too late. R-123 names this exact class.
---
## 7. What is still unknown
| # | Unknown | What it takes |
|---|---|---|
| 1 | **The real Felhom stub has not been packaged.** §2 proves a *trivial* `.deb` is delivered and its postinst can enable a unit that fires. It does not prove `felhom-bootstrap.sh` + its unit + the pairing env survive that route, nor that pairing completes end-to-end from an interactively-installed box. | Package the real stub as a `.deb` per §3's constraints, repeat the §2 run, and confirm pairing reaches the hub. ~60 min. **This is the last thing between here and a spec.** |
| 2 | **`dpkg --configure -a` ordering** (§3, constraint 5) — the probe depended on nothing. | A package with a dependency, or one that must run after `pve-manager`. Only matters if the real stub needs it. |
| 3 | **An ISO that never went through `prepare-iso`** (§4) — unmeasured, and blocked by R-155 which this spike is fenced from changing. | Relax the guard in a branch, build, boot. ~30 min. |
| 4 | **Spike 3's Q3 — the real stub at `before-network`** — **this session did not touch it.** It now governs operator-built images only, and note item 1 partly supersedes it: on the `.deb` route the unit's ordering is set by the unit file, not by `[first-boot].ordering`. | ~40 min if still wanted. |
| 5 | **The graphical installer was not tested** — §2 used the Terminal UI. `proxinstall` has strictly less first-boot capability, and the `.deb` path does not involve the front-end at all (it is `Install.pm`, shared), so the result should hold a fortiori — but that is inference. | Repeat §2 via the Graphical entry. ~30 min. |
| 6 | Spike 1's two open items — multi-match determinism, boot-medium exclusion. | Unchanged. |
---
## 8. R-rows opened
**None.** Each candidate was grepped against the register first, per the fence:
- The `.deb` delivery result is a **positive** finding, not a defect.
- The postinst constraints (§3) belong in the eventual build spec, not the defect register.
- The `iso-repack.sh` guard is **already R-155** (opened by Spike 3) — extended here with the exact
citation rather than duplicated.
- **R-153 is retracted** into R-94 leg (b) / R-29 (§6).
---
## 9. Teardown
**Layer 1 — demo-hp**, including the scratch storage.
| Check | Result |
|---|---|
| VMs 400, 401 | `qm destroy --purge` — *"purging VM 400/401 from related configurations"*; `qm list` now shows only `300 drill-r50 stopped` |
| **scratch storage `spike4`** | `pvesm remove spike4` — **removed**; `/etc/pve/storage.cfg` back to **4 storages**, `grep -c spike4` = **0** |
| `/mnt/nvme-1tb/images/` | **empty** — both VM disk directories gone with the purge |
| `/mnt/nvme-1tb/` contents | `dump e2d-images felhom-data images lost+found` — the pre-spike set |
| disk usage | **6.6 G — identical to pre-spike** |
| `probe4.iso` in the ISO store | removed |
| driver, answer file, password files, screendumps (`/root/…`) | all removed |
| loop devices / `/mnt/s4ans` | `0` / removed |
| `drill-r50` (VM 300) | `stopped`, untouched — never destroyed or unblocked (R-93 fixture intact) |
| guest 9201 | `running`, untouched |
| `felhom-backup` storage | **unmodified** — the demo box's own configuration was not changed |
| `local-lvm` | never written |
**Layer 2 — demo-felhom.** Nothing was placed there this session; it was not contacted.
**Layer 3 — DooPlex.**
| Check | Result |
|---|---|
| workspace scratch `/mnt/5_hdd/felhom.eu/spike4-scratch` (ISO extract tree + two 1.7 GB ISOs) | **4.8 GB → removed**; `/mnt/5_hdd` at 24 % |
| session scratchpad | **3.3 GB → 88 K** |
| throwaway passwords | the answer-file password `shred -u`'d and verified absent; the interactive install's console password existed only as a `0600` file on demo-hp, deleted in the same command |
| probe `.deb` build tree | removed |
| `felhom-iso/out/` | untouched, **17 ISOs** |
| production files | none modified — no `build-felhom-iso.sh`, no `iso-repack.sh`, no profile, no `answer.toml.tmpl`, no `felhom-bootstrap.sh` |
Note: the ISO extract used **disk-backed workspace scratch, not `/tmp`** — `/tmp` on DooPlex is a
32 GB **tmpfs**, and a 3.4 GB extract there would have taken RAM from a live k3s node.
**Hub-side disposition.** No customer, host or appliance was created. The VMs were bridged to the
demo-site LAN and took DHCP leases (`192.168.0.47`, `192.168.0.100`), but neither ran
`felhom-host-install.sh` nor contacted the hub — the probe package carries no Felhom logic. Verified
by fetching the customer list and searching it, not asserted.
**Published artifacts: none.**