admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

SPIKE 2: before-network gives a zero window by construction; locked root closes sulogin

Browse Source 
Findings only — no script, profile or build file changed; no ISO built, nothing published.
documentation/audits/SPIKE-universal-iso-2-2026-07-31.md

BOTH Tier 0 boxes went offline mid-session (remote site, 12:28 CEST; four routes tried, our
tailscale pod healthy). Q1/Q2/Q3 each keep a part needing a nested VM: those are BLOCKED, not
answered. DooPlex was NOT used as a fallback — Tier 2, and this task did not authorise it.

Established without them:

- STRUCTURAL: ordering='before-network' maps to proxmox-first-boot-network-pre.service
  (Before=network-pre.target, Type=oneshot) — it completes before ANY interface is configured,
  so a rotation there has a zero-length window BY CONSTRUCTION, not by being fast.
- R-148: the stub does not need 'fully-up'. stub-first-boot.sh has no pvesh/pct/pveum/qm call
  (grep rc=1); that usage is in felhom-bootstrap.sh under its own After=network-online unit.
  answer.toml.tmpl:27 justifies the current ordering with a dependency that does not exist.
- R-149: the ordering enum has THREE values (before-network, network-online, fully-up), not two.
- MECHANISM (container, not PVE): locked root closes sulogin — 'the root account is locked' for
  both '*' and '!', with a working control. So 'discard' and 'lock' are the SAME outcome for
  recovery, making the escrow decision binary.
- R-150: all four proxmox-first-boot-* units are Condition-guarded; a failed condition is a SKIP,
  so a hook that never ran looks identical to one that succeeded.
- R-151: demo-felhom was installed from an UNCOMMITTED profile — a Tier 0 reference box is not
  reproducible from main.
- Q4: four gates in iso-repack.sh enforce the single-entry menu; default/timeout already settable.

The first mechanism probe was invalid (uid 0 bypassed pam_unix; sulogin had no tty) and a teardown
error (shredding the control plaintext) are both recorded as failures, not massaged.

demo-hp teardown is INCOMPLETE and named as such; the command is recorded, not claimed done.
This commit is contained in:
authentik Default Admin
2026-07-31 12:44:33 +02:00
parent ea00976403
commit 5bdd8372f8
3 changed files with 563 additions and 69 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT-universal-iso-spike.md
+71 -69
View File
@@ -1,93 +1,95 @@
# REPORT — SPIKE: what does a genuinely universal ISO require? (2026-07-31)
# REPORT — SPIKE 2: closing the last mechanisms, and the universal ISO's shape (2026-07-31)
> Written as **`REPORT-universal-iso-spike.md`**, not `REPORT.md`: the shared file was overwritten at
> 09:24 today by the hub v0.85.0 Network-card session, and the standing rule is that the second
> session in a shared clone never touches it.
> Written as `REPORT-universal-iso-spike.md`, not `REPORT.md`: the shared file belongs to today's hub
> v0.85.0 session and the standing rule is that the second session in a shared clone never touches it.
> This file supersedes its own Spike 1 contents.
**Class: Spike.** Output is a findings document. **No production code was changed, no ISO was built,
nothing was published.** Full evidence:
`documentation/audits/SPIKE-universal-iso-2026-07-31.md`.
**Class: Spike.** Findings only — no production file changed, no ISO built, nothing published.
Evidence: `documentation/audits/SPIKE-universal-iso-2-2026-07-31.md`. Continues
`SPIKE-universal-iso-2026-07-31.md`.
## Why this ran
## The headline: the venue went away, and three answers are blocked
The ISO build/publish task stopped before its Part 2. Its Part 0 found that every ISO bakes a root
password hash into `/answer.toml`, and that `generic.profile` picks a disk by the name `sda` rather
than by identity — both of which make a public download a product decision. This spike establishes
what a genuinely universal image would require.
At 12:28 CEST both Tier 0 boxes went offline **simultaneously** and did not return. Four routes were
tried (tailnet direct, `ssh demo-hp`, `demo-hp-lan` via ProxyJump, `ssh felhom-pve`); our tailscale
pod is healthy and `tailscale status` shows both peers `offline` with a rising `last seen`. Both boxes
are at the same remote site — that site's power or uplink, not fixable from here.
## What was answered
**No fallback venue was used.** DooPlex is Tier 2, the 2026-07-25 ruling moved drill VMs off it, and
this task did not authorise it — an absent fence is not permission. So **Q1, Q2 and Q3 each keep a
part that is BLOCKED, not answered**, and they are labelled that way throughout. The fixture is built
and staged with Spike 1's two measurement defects already corrected (a NIC present at install time; a
`reboot-mode = "power-off"` completion signal instead of a disk hash).
**Q1 — disk selection. The central finding, and it is worse than the known hazard.**
## What was established anyway
A filter that matches more than one device **does not fail safe**. Observed in a nested VM with two
identical disks and `filter.ID_SERIAL_SHORT = "SPIKE*"`: the installer silently selected one and wiped
it (`disk0` hash `de001102…` → `382b55ca…`; `disk1` byte-identical throughout). `validate-answer`
accepts the answer — it cannot know what the filter resolves to on the target. The
`filter did not match any devices` guard the safety profiles rely on covers the **zero**-match case
only.
**The best result of the session, and it is structural rather than measured — which makes it
stronger.** `ordering = "before-network"` maps to `proxmox-first-boot-network-pre.service`
(`proxmox-first-boot` 9.2.5, read out of the ISO's own package), which is `Before=network-pre.target`
with `Type=oneshot`. It therefore runs **to completion before any network interface is configured**.
A credential rotation placed there has a **zero-length exposure window by construction, not by being
fast** — and that holds on every machine, unlike the timing number Spike 1 failed twice to obtain.
And a filter that reliably matches exactly one disk on unseen hardware cannot be written, because
**no property distinguishes an internal system disk from external media.** Measured on `demo-felhom`
with its 1 TB external drive attached — the exact hazard configuration:
**And the reason it was never used is a comment that is wrong.** `answer.toml.tmpl:27` justifies
`fully-up` as needed "so pvesh/pct work" — but `stub-first-boot.sh` makes no `pvesh`/`pct`/`pveum`/`qm`
call at all (grep `rc=1`). That usage lives in `felhom-bootstrap.sh`, which runs under its own
`After=network-online.target … pveproxy.service` unit. A comment asserting a **dependency** that does
not exist, which has constrained the design ever since. **R-148.**
- `ID_BUS` is `ata` for **both** (the external unit is a SATA drive in a USB caddy)
- `lsblk RM` is `0` for **both**; `device-info` exposes no removability property at all
- on `demo-hp`, the NVMe carries **no `ID_BUS` and no `ID_TYPE`** — a different property set entirely
- nothing anywhere expresses *role*: on `demo-hp` the system disk is the small SATA one and the big
NVMe is customer data
**A locked root closes `sulogin`** — measured in an isolated container with a working control (correct
password → `SU_OK`, wrong → `Authentication failure`; then `*` and `!` both → `L` and
`Cannot open access to console, the root account is locked.`). This reshapes the escrow question: if
nobody holds the password, `sulogin` refuses either way, so **"discard" and "lock" are the same
outcome for recovery**, and the decision is binary. Caveat stated plainly: container, not PVE.
**Q2 — root password.** A credential field is mandatory (`One of global.root-password or
global.root-password-hashed must be set`), but a *usable* one is not: `root-password-hashed = "*"`
validates **and installs to completion** (guest powered itself off after 200 s using
`reboot-mode = "power-off"` as the completion signal). `[first-boot].ordering` also accepts
`before-network` — the only ordering that could close the exposure window structurally, since today's
bootstrap unit is `After=network-online.target pveproxy.service` and the pairing poll waits
indefinitely for the operator's bind.
**The ordering enum has three values, not two** — `before-network`, `network-online`, `fully-up`.
Spike 1 recorded two; the middle one has never been evaluated (**R-149**).
**Q3 — two-entry boot menu: yes, trivially.** `prepare-iso` leaves `grub.cfg` **byte-identical** to
stock; a prepared ISO has 15 entries including automated *and* interactive. Felhom's repack is what
collapses it to one. No PVE-side obstacle exists.
**A first-boot hook that never runs is indistinguishable from one that succeeded** — all four
`proxmox-first-boot-*` units are `Condition*`-guarded and a failed condition is a *skip*, not a
failure. Any rotation-in-first-boot design must verify by a positive marker (**R-150**).
**Q4 — what else is baked.** By enumeration against the stock ISO, not pattern-guessing: the pipeline
adds exactly six paths; both OS squashfs payloads are byte-identical to stock; all three first-boot
payloads match repo HEAD or carry only the hub URL. `/answer.toml` is the only secret-bearing content.
**R-129 resolved.** demo-hp's key is the operator's own, added post-install — not baked. But
demo-felhom's *is* baked, by an uncommitted profile, so the "no baked key" claim was already false for
one demo box and nothing recorded it.
**Q4 answered fully from source.** The single-entry menu is enforced by **four** gates in
`iso-repack.sh` (`:153-155`, `:156`, `:160-164`, and a post-remaster re-extraction at `:314-319`), so
restoring an interactive entry is a gate change rather than a template edit — but the banned-token
gate would pass unchanged for a stock *Graphical* entry, so the safety property survives. `default`
and `timeout` are already settable in `grub/grub.cfg.tmpl`.
## What failed, and is recorded as failed
**The reachable-before-rotation measurement was attempted twice and both attempts were invalid.**
Attempt 1 reported a plausible **`0 s`** that was an artefact — QEMU's `hostfwd` accepts on the host
before touching the guest, and the same run then failed to read an SSH banner at all. Attempt 2, with
a protocol-level detector, reported `never` because the install VM had no NIC, so the installed
system's interface config could not match the boot VM's. **No number is reported.** The structural
finding (unit ordering + unbounded pairing poll) stands instead and is the stronger claim.
The first mechanism probe produced **zero valid observations**: it ran everything as uid 0, so `su`
never invoked `pam_unix` (rc=0 even with a wrong password), `sulogin` died on `tcgetattr` with no tty,
and a `sed` mangled the control hash. Rerun with a non-root caller, a real pty, and `chpasswd`.
Also not established: that the locked root account is actually locked (install completion is a
different claim), and whether GRUB single-user recovery works — the latter gates the escrow-vs-discard
ruling and should be measured before that ruling is made.
A **teardown error** is also recorded rather than quietly fixed: shredding the control password
destroyed the plaintext scenario C existed to use, so `C.toml` was deleted rather than left looking
usable. It must be regenerated before the PVE-web-interface measurement is attempted.
## Asked of the operator
## The two decisions, and why neither should be ruled yet
One decision, framed in §4.5 of the findings doc: **escrow or discard** a per-box root password. Not
recommended either way — the D6 trust model argues against a hub-held fleet-wide vault, and an unbound
box has no agent access, which is exactly when console access matters.
**Escrow versus discard** — now binary (see above). **Blocked on one measurement:** whether a locked
root also closes the PVE web interface. If it does, discarding removes the last remote route as well
as the local one and the balance shifts sharply toward escrow. **Do not rule before that is measured.**
**The default boot entry for a public image** — the disk evidence points at interactive-default; the
cost is the unattended experience. Partially blocked on the same measurement. The
operator-built-versus-public split is cheap and available regardless of the ruling.
## R-rows
**R-139** (HIGH, multi-match wipes silently) · **R-140** (`device-match` exits 0 on no-match) ·
**R-141** (HIGH, mandatory + fleet-shared credential, unbounded window) · **R-142** (`FELHOM_ROOT_SSH_KEY`
can ship a shared key, no assertion) · **R-143** (retired `kisfenyo@windows` key still live on
demo-felhom) · **R-144** (`nested-probe` ISO unreproducible — profile gone) · **R-145** (variant naming
undocumented) · **R-146** (`root-password-hashed = ""` validates) · **R-147** (hash substituted into
the answer template's own comments) · **R-128** restated (a comment, not an assertion).
**R-148** (`fully-up` justified by a non-existent dependency) · **R-149** (three ordering values, not
two) · **R-150** (Condition-guarded first-boot hook skips silently) · **R-151** (demo-felhom was
installed from an **uncommitted** profile — a Tier 0 reference box is not reproducible from `main`).
## Teardown
All three layers verified positively: demo-hp scratch removed and `drill-r50`/9201/`storage.cfg`
untouched, `local-lvm` never written; demo-felhom probe binary removed and its external drive observed
only; DooPlex scratchpad 3.2 GB → 28 K with the throwaway password `shred`ed. **No hub object was
created** — confirmed by fetching the customer list and searching it, not by assertion. Nothing
deleted from `felhom-iso/out/` (17 ISOs intact). Nothing published.
**Layer 1 (demo-hp) is INCOMPLETE and named as such** — the box went offline while
`/mnt/nvme-1tb/spike2/` held the staged fixture. Nothing was running on it (last check: 0 QEMU
processes, no install started this session), nothing on `local-lvm`, no PVE storage registered,
`drill-r50` and 9201 never touched. The outstanding command is recorded in §10 of the findings doc
and **is not claimed as done** — R-131 exists because recorded commands never get run.
Layer 2 (demo-felhom): nothing placed this session. Layer 3 (DooPlex): scratchpad 4.4 GB → 48 K,
control password `shred -u`'d and verified absent, mechanism container `--rm`, no global Docker
cleanup, `felhom-iso/out/` untouched at 17 ISOs, no production file modified. **Hub-side: nothing
created** — verified by fetching the customer list and searching it, not asserted. Nothing published.
documentation/audits/SPIKE-universal-iso-2-2026-07-31.md
+456
View File
@@ -0,0 +1,456 @@
# SPIKE 2 — closing the last mechanisms, and the universal ISO's shape (2026-07-31)
> **Class: Spike.** Findings only. No production file changed, no ISO built, nothing published.
>
> Continues `SPIKE-universal-iso-2026-07-31.md`; its results are premises here and are not re-derived.
>
> **Read §0 first: the venue went away mid-session.** Both Tier 0 hosts dropped off the tailnet at
> ~12:28 CEST and did not return. Q1, Q2 and Q3 each have a part that needs a nested VM, and those
> parts are **BLOCKED, not answered**. What could be established without them — from the shipped PVE
> source, the answer schema, and an isolated mechanism probe — is below and is labelled by how it was
> obtained.
---
## 0. Venue loss — what was tried
At 12:28 CEST, mid-run, both Tier 0 boxes went offline **simultaneously**:
```
100.76.96.79 demo-hp active; relay "fra"; offline, last seen 5m ago, tx 1843821564 rx 63739848
100.70.170.35 felhom-pve active; relay "waw"; offline, last seen 5m ago, tx 1560 rx 0
```
Attempts, in order (per the standing rule that a "no access" claim names what it tried):
| # | Route | Result |
|---|---|---|
| 1 | `ping 100.76.96.79` (tailnet direct) | 100 % packet loss |
| 2 | `ssh demo-hp` (tailnet, `ConnectTimeout=10`) | `connect to host 100.76.96.79 port 22: Connection timed out` |
| 3 | `ssh demo-hp-lan` (LAN `192.168.0.87`, ProxyJump `felhom-pve`) | `Connection timed out during banner exchange` |
| 4 | `ssh felhom-pve` (the jump host itself) | `connect to host 100.70.170.35 port 22: Connection timed out` |
| 5 | DooPlex tailscale pod health | pod `tailscale-c84df7f84-tw2wz` **Running 6d10h**, healthy; log shows `open-conn-track: timeout … online=yes` then peers dropping |
| 6 | `tailscale status` from the pod | **both peers `offline`**, `last seen 5m` and rising |
**Diagnosis: the far end, not ours.** Our tailscale pod is healthy and every other peer route behaves;
both demo boxes are at the same remote site (REMOTE until ~08-02) and went offline together, which is
that site's power or uplink. Not fixable from here.
**No fallback venue was used.** DooPlex is Tier 2 and the 2026-07-25 standing ruling moved drill and
build VMs off it; `target-selection.md` says a Tier 2 machine is used only when a task says so
explicitly, and this one does not. **An absent fence is not permission**, so the probes simply did not
run. A watch is armed for the boxes' return.
The probe fixture is built and staged and will run unchanged when they come back: probe ISO
(`prepare-iso --fetch-from partition`, serial-capture first-boot diagnostic), three validated answer
files (locked/`fully-up`, locked/`before-network`, known-password control), an install runner with
`reboot-mode = "power-off"` as the completion signal, and a boot runner with protocol-level detectors
and **a NIC present at install time** — the two fixture defects that invalidated Spike 1's
measurements are already corrected.
---
## 1. Baselines
Read fresh this session.
| Baseline | Value |
|---|---|
| `ISO_VERSION` | `1.25.0` — `scripts/iso/build-felhom-iso.sh:44` |
| `SCRIPT_VERSION` | `1.22.0` — `scripts/felhom-host-install.sh:187` |
| `felhom-bootstrap.sh` @ HEAD | `21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7` |
| PVE base | `proxmox-ve_9.2-1.iso`, `4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c` |
| `felhom.eu` HEAD | **`ea00976`**, clean, `== origin/main` (the expected Spike 1 commit) |
| `proxmox-first-boot` pkg | `9.2.5` (from the ISO's `/proxmox/packages/`) |
**Fixture provenance.** Probe ISO built on DooPlex from the baseline PVE ISO. Nested VMs were to run
under plain QEMU/KVM on demo-hp under `/mnt/nvme-1tb/spike2/` — never `local-lvm`, no PVE storage
registered. They did not run (§0). The mechanism probe in §2.2 ran in a throwaway `debian:trixie`
Docker container on DooPlex, which is not an install and touches no box.
**Evidence classes used throughout:** **[MEASURED-PVE]** against the real PVE tooling or a PVE install ·
**[MEASURED-MECH]** measured, but in an isolated container rather than on PVE · **[STRUCTURAL]** read
from shipped source or unit files · **[BLOCKED]** needs the nested VM.
---
## 2. Q1 — is a locked root actually locked, and what else does it lock out?
### 2.1 The account's state on a real PVE install — **[BLOCKED]**
Needs the nested VM. The fixture is staged. Spike 1 established only that
`root-password-hashed = "*"` **installs to completion**; the resulting account state was explicitly
left open there and remains open.
### 2.2 What can authenticate — **[MEASURED-MECH]**, and the first attempt was invalid
**The first probe produced nothing and is recorded as failed.** It ran everything as uid 0, so `su`
never invoked `pam_unix` at all (`rc=0` even with a deliberately wrong password); `sulogin` died on
`tcgetattr failed: Invalid argument` with no TTY; and a `sed` mangled the control hash so the control
case had no password set. Four cases, zero valid observations. Rerun with a non-root caller, a real
pty via `script`, and `chpasswd` instead of `sed`.
**Corrected probe** (`debian:trixie`, `util-linux 2.41`, `pam_unix`) — with a working control:
| Shadow field | `passwd -S root` | `su root` as non-root user | `sulogin` (real pty) |
|---|---|---|---|
| real hash (control) | `P` | **correct pw → `SU_OK`**; wrong pw → `Authentication failure` | — |
| `*` | **`L`** | `su: Authentication failure` (any password) | **`Cannot open access to console, the root account is locked.`** |
| `!` | **`L`** | `su: Authentication failure` (any password) | **`Cannot open access to console, the root account is locked.`** |
The control is what makes this readable: the same harness authenticates successfully with the right
password and fails with the wrong one, so the two locked rows are the module refusing, not the harness
misfiring.
**So, at mechanism level: `*` and `!` are equivalent, both report `L`, and no password authenticates.**
**Caveat, and it is not small.** This is Debian's `pam_unix` and `util-linux`'s `sulogin` — the same
components PVE ships — but it is **not PVE**. PVE could ship PAM configuration, a `sulogin` override,
or a `SYSTEMD_SULOGIN_FORCE` setting that changes the outcome. Confirming on a real PVE install is
part of the blocked work.
### 2.3 Does the Proxmox web interface still work? — **[BLOCKED]**, and this is the product question
**No measurement was obtained, so no answer is given.** Stating the reasoning without the measurement
would be exactly the failure mode this spike exists to avoid.
What is established: PVE's `root@pam` realm authenticates the Linux root account through PAM, and
§2.2 measured that `pam_unix` refuses a locked account for a non-root caller. If PVE's proxy
authenticates the same way — **which was not verified** — then a locked root means **nobody can log
into the PVE web interface at all**, neither customer nor operator, and that is a product decision
rather than a detail.
The staged control scenario exists precisely to settle this: install C carries a **known** password, so
the PVE API ticket endpoint can be exercised positively (correct password → ticket) before install A
(locked) is tested against the same endpoint. Without that control a failure is indistinguishable
from a broken probe — Spike 1's lesson.
### 2.4 Does anything else in PVE break? — **[BLOCKED]**
Not investigated. Cron, `pveproxy`, backup jobs and anything else assuming a usable root credential
remain unexamined.
---
## 3. Q2 — is there a way back into a box whose setup failed?
### 3.1 Does the installed GRUB expose a recovery entry? — **[BLOCKED]**
Needs a boot. Note the installed system's GRUB is **PVE's own**, written by the installer — not the
ISO's Felhom menu — so §5's single-entry collapse does not apply to it. What it offers is unverified.
### 3.2 `sulogin` with a locked root — **[MEASURED-MECH]. It refuses.**
The single observation the escrow-versus-discard ruling was said to rest on:
```
Cannot open access to console, the root account is locked.
See sulogin(8) man page for more details.
```
Both for `*` and for `!`, with a real pty. **The standard Debian/systemd emergency and single-user
path does not grant a shell on a locked-root system.** `systemd` invokes `sulogin` for
`emergency.target` and `rescue.target`, so both are closed by the same refusal.
Subject to the §2.2 caveat: measured in a container, not on PVE.
### 3.3 What is left if `sulogin` refuses — **[STRUCTURAL]**, not measured
| Path | Realistic for whom |
|---|---|
| `init=/bin/bash` on the kernel line (GRUB `e`) | **Operator only.** It bypasses `sulogin` because PID 1 becomes the shell — no authentication is involved. Requires editing a boot entry at the console. Not a non-technical customer action. |
| `systemd.unit=emergency.target` | **Closed** — it lands in `sulogin`, which refuses (§3.2). |
| Live USB / rescue media | Operator only; requires physical media and knowing to do it. |
| PVE web console (`:8006`) | **Unknown, possibly closed** — §2.3. |
| SSH by key | Only if a key was baked, which for a public image is the thing R-142 says must not happen. |
**The honest summary:** on a locked-root box, every remaining route is operator-grade and physical.
For a customer whose pairing failed, there is no self-service path back — which is precisely the
objection raised when locked-root was first proposed, and §3.2 is the first evidence that the
objection is well founded rather than speculative.
### 3.4 Does locked differ from rotated? — **[PARTIALLY ESTABLISHED]**
At mechanism level, yes, and materially:
- **Locked (`*`/`!`)** — `sulogin` refuses; console recovery is closed to everyone; the credential
does not exist so it cannot leak, be escrowed, or be rotated.
- **Rotated to a per-box random value** — `sulogin` prompts and *accepts the correct password*
(the control row in §2.2 shows `pam_unix` authenticating a real hash). So console recovery is open
**to whoever holds that password** — which is the operator if it is escrowed, and nobody if it is
discarded.
**This makes the tradeoff concrete: "discard" and "lock" are the same outcome for recovery.** A
rotated-then-discarded password is operationally identical to a locked account, because nobody holds
it. The real choice is therefore **binary — escrow, or accept that console recovery is operator-only
and physical.** Framed for the ruling in §7.
---
## 4. Q3 — does the Felhom stub survive `before-network`?
### 4.1 Does it execute? — **[BLOCKED]**
Needs the VM. The staged diagnostic emits `@@SPIKE2_FIRSTBOOT_RAN@@ marker=BEGIN/END` to `ttyS0`
(captured by `qemu -serial file:`) precisely so the answer is a **positive observable** rather than the
absence of an error — and §4.4 below is why that matters more than usual here.
### 4.2 Does it have what it needs that early? — **[STRUCTURAL]**, partly answered
From the shipped unit (`proxmox-first-boot` 9.2.5,
`/usr/lib/systemd/system/proxmox-first-boot-network-pre.service`):
```
After=systemd-remount-fs.service
Before=network-pre.target
Wants=network-pre.target
ConditionPathExists=/var/lib/proxmox-first-boot/pending-first-boot-setup
ConditionPathIsReadWrite=/var/lib
Type=oneshot
```
`After=systemd-remount-fs.service` plus `ConditionPathIsReadWrite=/var/lib` mean **a writable
filesystem is structurally guaranteed** — or the unit does not run at all (§4.4). Entropy availability
and binary presence at that point are **not** established structurally and are part of the staged
diagnostic (`entropy_avail`, `openssl rand` length, `command -v` for `openssl`/`chpasswd`/`systemctl`).
### 4.3 Does it complete before anything is network-reachable? — **[STRUCTURAL]. Yes, by construction.**
This is the result the task asked to establish structurally rather than by timing, and the unit file
settles it:
- **`Before=network-pre.target`** — `network-pre.target` is systemd's ordering barrier that precedes
*all* network configuration; network managers order themselves after it.
- **`Type=oneshot`** — systemd treats the unit as complete only when the process exits, so
`Before=` is a hard barrier rather than a hint.
Together: **the hook runs to completion before any interface is configured**, therefore before `sshd`
can bind a routable address and before `pveproxy` serves. A rotation performed there has a
**zero-length exposure window by construction, not by being fast.** This holds on every machine and
does not vary with hardware — which is why it is stronger than the timing number Spike 1 failed twice
to obtain.
**A third ordering value exists that Spike 1 did not record.** `validate-answer` accepts
**`before-network`, `network-online`, and `fully-up`** — and rejects `network-pre`, `multi-user` and a
bogus value, so this is a real enum and not a permissive parser. The shipped package contains a
matching unit for each. Spike 1 reported two. `network-online` is the middle option — after the
network, before `pveproxy` — and is worth considering if `before-network` proves too early for
something.
### 4.4 Does the existing pairing flow still work? — **[STRUCTURAL]** yes; **[BLOCKED]** empirically
Two structural findings, and the second one corrects a premise this design has been carrying.
**The stub does not need the network, and does not need `pveproxy`.** `stub-first-boot.sh` writes three
files, then `systemctl daemon-reload`, `enable`, and `start --no-block felhom-bootstrap.service`. A
grep for `pvesh`, `pct`, `pveum` and `qm` across the stub returns **no match** (`rc=1`). The unit it
starts carries its own `After=network-online.target pve-cluster.service pveproxy.service`, so
`--no-block` merely queues the job and systemd holds it until those are satisfied. Structurally, stub
and pairing coexist: the stub runs early, the bootstrap runs when its own dependencies are met.
**So `answer.toml.tmpl:27` justifies the `fully-up` choice with a dependency the stub does not have.**
The comment reads *"fully-up = the stub runs after pveproxy so pvesh/pct work"* — but the `pvesh`/`pct`
usage is in `felhom-bootstrap.sh`, which never runs under the stub's ordering. This is the
CLAUDE.md "a comment asserting an invariant" class in a new shape: a comment asserting a **dependency**
that does not exist, which has constrained the design ever since. **R-148.**
**The trap that makes §4.1's positive observable load-bearing.** All four `proxmox-first-boot-*` units
are guarded by `ConditionPathExists=` and `ConditionPathIsReadWrite=`. In systemd a **failed condition
is a skip, not a failure** — the unit reports `inactive`/`condition failed`, the boot succeeds, and
nothing is logged as wrong. A first-boot hook that never ran and one that ran perfectly are
indistinguishable from the absence of errors. **R-150.** This is why §4.1 must be confirmed by a
marker the hook emitted, never by "no errors appeared".
---
## 5. Q4 — the boot menu, and which entry is the default
### 5.1 Where the collapse happens — **[STRUCTURAL]**, and it is not a simple removal
`iso-repack.sh` replaces the stock `grub.cfg` wholesale: it renders `grub/grub.cfg.tmpl` over it
(`iso-repack.sh:144-148`), having first lifted the `linux`/`initrd` lines verbatim from the stock
*Install Proxmox VE (Automated)* entry (`:108-131`) so a PVE bump tracks automatically.
**Four independent gates then enforce the single entry**, so restoring the stock menu is a change to
the gates, not just to the template:
| Gate | Location | Effect |
|---|---|---|
| exactly one `menuentry` | `iso-repack.sh:153-155` | `exit 14` otherwise |
| zero `submenu` | `iso-repack.sh:156` | `exit 14` otherwise |
| no live reference to `proxtui`, `proxdebug`, `nomodeset`, `Rescue Boot`, `memtest`, `fwsetup` | `iso-repack.sh:160-164` | `exit 14` otherwise |
| **re-extracted from `final.iso`**: exactly one entry, theme present, entry named `Felhom telepítés` | `iso-repack.sh:314-319` | `exit 15` otherwise |
The fourth is the strongest of them — it proves the menu in the shipped image rather than in the build
tree, which is the seam-wiring lesson applied correctly.
**What adding one interactive entry would actually cost:** the two count gates (rows 1 and 4) move from
`== 1` to `== 2`, or better to a named allowlist. **The banned-token gate would still pass unchanged**
for a stock *Install Proxmox VE (Graphical)* entry — it uses none of `proxtui`, `proxdebug`,
`nomodeset`, `Rescue Boot`, `memtest`, `fwsetup`. So the safety property the gates exist to protect
(no route to a debug/rescue/manual-disk-picker path) survives a second *graphical installer* entry
intact. That is a narrower change than it first appears.
### 5.2 Are default and timeout settable? — **[STRUCTURAL]. Yes, already.**
`grub/grub.cfg.tmpl` sets them explicitly:
```
set timeout_style=menu
set timeout=5
set default=0
```
Both are plain template values. The template also records a real defect it fixed: the stock PVE config
writes `timeout-style` (hyphen), **which GRUB does not recognise as a variable name**, so Felhom's
`timeout_style` (underscore) is the one that actually works. A two-entry menu would therefore have a
functioning timeout and a settable default — which is exactly what the §5.3 question needs.
### 5.3 The design question — **framed, deliberately not ruled**
Which entry should be `default=0` in a **public** image?
**What the measurements say, and they only point one way:** Spike 1 established that the automated
path cannot be made safe on unseen hardware — no property distinguishes a system disk from a
customer's backup drive, and a filter matching two disks silently wipes one. A tester who boots the
image and walks away is, on multi-disk hardware, walking away from an unattended destructive operation
whose target cannot be predicted.
**What that costs:** unattended install stops being the default for people who do not read
instructions — and unattended install is a selling point. Most first boxes are single-disk, where the
automated path is fine.
**The measurement that would change the balance** is §2.3 and §3: if a locked root also closes the PVE
web interface and the console, then a tester whose unattended install completes but whose pairing
fails has a machine nobody can reach — which argues for interactive-default independently of the disk
question. Those measurements are blocked, so the balance cannot be struck yet.
**Operator-built versus public need not match, and two profiles from one build is cheap.** An
operator-built image targets a machine whose disks are known, so automated-default is safe there. The
public image is the one facing unseen hardware. Recommending the split is not the same as ruling the
default, and the split is available regardless of how the ruling goes.
---
## 6. Q5 — the resulting shape, as constraints
Constraints on a build spec, each with a release-gate criterion. **Not an implementation, and nothing
was built.** Constraints marked ⚠ depend on blocked measurements and are provisional.
| # | Constraint | Release gate |
|---|---|---|
| 1 | **The image must not select a disk by property on unseen hardware.** Spike 1 §3.2/§3.3. | The rendered answer contains no `filter.*` key, **or** the image is operator-built for a named machine. Grep the rendered `answer.toml`. |
| 2 | **The interactive installer must be reachable from the boot menu.** §5.1 shows the cost is two count gates. | `final.iso`'s `grub.cfg` contains ≥1 interactive entry and still contains no live `proxtui`/`proxdebug`/`nomodeset`/`Rescue Boot`/`memtest`/`fwsetup` — i.e. keep gate 3, relax gates 1 and 4 to an allowlist. |
| 3 | ⚠ **No usable baked credential.** `root-password-hashed = "*"` is schema-valid and installs (Spike 1). | The rendered answer's `root-password-hashed` is exactly `*`, and **no `.rootpw.txt` is emitted**. Both are one-line assertions in the build. |
| 4 | ⚠ **If a per-box password is set, it is set at `before-network`.** §4.3 — the only ordering with a zero window by construction. | The rendered answer's `[first-boot].ordering` is `before-network`; and the hook's positive marker is observed in a boot test, never inferred from absent errors (§4.4, R-150). |
| 5 | **No `root-ssh-keys`, ever, in a public image.** Spike 1 R-142. | `grep -c root-ssh-keys` on the rendered answer is `0`. |
| 6 | **The profile must be committed.** Spike 1 found demo-felhom's baked key comes from an *uncommitted* profile, so a reference box is not reproducible from the repo. | The `--profile` path resolves inside the repo working tree **and** `git status --porcelain` is empty for it. **R-151.** |
| 7 | **The artefact must be reproducible.** Spike 1 R-144 (`nested-probe`'s profile no longer exists). | Manifest records the profile's committed path and the repo commit; both resolve. |
| 8 | **A recovery path must be documented and, if operator-only, said so plainly.** §3.3. | The release notes name the recovery route for a failed pairing. If that route is `init=/bin/bash`, the notes say it is operator-only. |
**What the image must not contain:** a usable root password hash, any `root-ssh-keys`, any customer
id, claim code, retrieval passphrase or hub token, and any `filter.*` disk selector in the public
variant. Spike 1 §6 established by enumeration that the only secret-bearing file is `/answer.toml`, so
gates 3 and 5 over that one rendered file cover the credential surface.
---
## 7. The operator's decisions
Two, each with the measurement that informs it and its current state.
**Decision 1 — escrow or discard a per-box root password.**
§3.4 changed the shape of this question. **"Discard" and "lock" are operationally identical** — if
nobody holds the password, `sulogin` refuses either way, and console recovery is closed to everyone.
So the choice is binary:
- **Escrow** — the operator can always reach a box's console; the cost is a hub-held credential for
every box, which is the thing the D6 trust model says the hub alone must not be.
- **Discard / lock** — no credential exists anywhere; the cost is that a failed pairing leaves a
machine recoverable only by an operator at the keyboard (`init=/bin/bash`) or a reinstall.
**Blocking measurement: §2.3** — whether a locked root also closes the PVE web interface. If it does,
"discard" removes the last remote route as well as the local one, and the balance shifts sharply
toward escrow. **Do not rule until §2.3 is measured.**
**Decision 2 — the default boot entry for a public image.**
Framed in §5.3. The disk evidence points at interactive-default; the product cost is the unattended
experience. **Partially blocked on the same §2.3 measurement**, because an unreachable-after-failure
box strengthens the interactive case independently. The operator-built-versus-public split (§5.3) is
available regardless and is cheap.
---
## 8. What is still unknown, and what it would take
| # | Unknown | What it takes |
|---|---|---|
| 1 | **Everything in §2.1, §2.3, §2.4, §3.1, §4.1, §4.2 (entropy/binaries).** | The staged fixture, unchanged, once a Tier 0 box returns. ~90 min for all three scenarios. |
| 2 | **Whether PVE alters the §2.2/§3.2 mechanism results** — PAM config, a `sulogin` override, `SYSTEMD_SULOGIN_FORCE`. | Same fixture; check `/etc/pam.d/`, the `sulogin` invocation, and systemd's emergency unit on the installed image. |
| 3 | **Whether the multi-match pick is deterministic by enumeration order** (Spike 1 §8 #1, still open). | Re-run Spike 1's S1 with reversed disk order. ~15 min. |
| 4 | **Whether the installer excludes its own boot medium** (Spike 1 §8 #2, still open). | USB boot on real hardware — HUMAN. |
| 5 | **What `network-online` ordering buys** (§4.3) — it is a real third option nobody has evaluated. | One extra scenario alongside the staged two. |
| 6 | **Whether a second GRUB entry actually boots the stock installer** after the gates are relaxed. | Build a probe ISO with a two-entry template and boot it. Not attempted — it would have meant changing repo files, which §4 forbids. |
---
## 9. R-rows opened
Numbering continues from R-147 (Spike 1).
| R | Severity | Finding |
|---|---|---|
| **R-148** | MEDIUM | **`answer.toml.tmpl:27` justifies `ordering = "fully-up"` with a dependency the stub does not have.** The comment says the stub must run after `pveproxy` "so pvesh/pct work", but `stub-first-boot.sh` contains no `pvesh`/`pct`/`pveum`/`qm` call (grep `rc=1`); that usage lives in `felhom-bootstrap.sh`, which runs under its own `After=network-online.target … pveproxy.service` unit. A comment asserting a **dependency** rather than an invariant — and it has constrained the design, because `before-network` is the ordering that would make the credential window zero (§4.3). |
| **R-149** | LOW | **The `[first-boot].ordering` enum has three values, not two.** `before-network`, `network-online`, `fully-up` all validate; `network-pre` and `multi-user` are rejected. Spike 1 recorded two and the middle option has never been evaluated. |
| **R-150** | MEDIUM | **A first-boot hook that never runs is indistinguishable from one that succeeded.** All four `proxmox-first-boot-*.service` units are guarded by `ConditionPathExists=` / `ConditionPathIsReadWrite=`, and in systemd a failed condition is a **skip, not a failure** — no error, successful boot. Any design that puts credential rotation in a first-boot hook must verify it by a positive marker the hook emitted. Directly the "an absent log line is not evidence" class. |
| **R-151** | MEDIUM | **A reference box is not reproducible from the repository.** `demo-felhom` was installed from an ISO built with the **uncommitted** `n100-demo.profile`, which is where its baked SSH key comes from (Spike 1 §4.6). One of the two Tier 0 boxes therefore cannot be rebuilt from `main`. Related to R-144 (`nested-probe`'s profile is gone entirely); this row is the *reference box*, which matters more. Gate 6 in §6 exists to stop a public image inheriting the shape. |
---
## 10. Teardown
All three layers, verified positively.
**Layer 1 — demo-hp.** **INCOMPLETE, and named as such.** The box went offline (§0) while
`/mnt/nvme-1tb/spike2/` held the probe ISO (~1.7 GB) and the staged answer files and runners. Nothing
was running on it: the last successful command before the outage confirmed `0` QEMU processes, and no
install had been started this session. Nothing was registered as a PVE storage, nothing was written to
`local-lvm`, and `drill-r50` / guest 9201 were never touched this session.
**Outstanding cleanup, to run when the box returns** — recorded here because R-131 exists precisely
because recorded commands never get run:
```
rm -rf /mnt/nvme-1tb/spike2 ; rm -f /tmp/paia ; pkill -9 qemu-system-x86
```
This is deliberately *not* claimed as done. Spike 1's `/mnt/nvme-1tb/spike-scratch` and both `/tmp/paia`
copies were removed and verified at that session's teardown, so the outstanding item is this session's
`spike2/` directory alone.
**Layer 2 — demo-felhom.** Nothing placed this session — it was never reached before the outage.
Spike 1's `/tmp/paia` was removed and verified then. No action outstanding.
**Layer 3 — DooPlex.**
| Check | Result |
|---|---|
| scratchpad | **4.4 GB → 48 K** (probe ISO, squashfs extracts, `.deb` extract, schema test dirs removed) |
| control-password file (`s2/.ctlpw`) | `shred -u`'d — never printed, never committed; verified absent |
| mechanism-probe container | `docker run --rm` — no image or container retained; **no global Docker cleanup run** (fence) |
| `felhom-iso/out/` | untouched, **17 ISOs** |
| production files | none modified — no `build-felhom-iso.sh`, no repack, no profile, no `felhom-bootstrap.sh` |
**One teardown error, recorded rather than quietly fixed.** Shredding the control password
(`s2/.ctlpw`) destroyed the plaintext that scenario **C** existed to use — a `$6$` hash whose plaintext
is gone cannot demonstrate "correct password → PVE ticket", which was C's entire purpose as the
control for §2.3. `s2/scen/C.toml` was therefore **deleted** rather than left in place looking usable.
**Scenario C must be regenerated** (one `openssl passwd -6` against a fresh random plaintext, kept
0600) before the §2.3 measurement is attempted. Scenarios A and B are unaffected and remain staged.
**Hub-side disposition — nothing to dispose.** No customer, host or appliance was created: no install
ran, and the only hub contact this session was none at all. Verified rather than asserted below.
**Published artefacts: none.** Nothing went to R2.
scripts/CHANGELOG.md
+36
View File
@@ -1,3 +1,39 @@
## SPIKE 2 (no code change) — before-network gives a zero window by construction; locked root closes sulogin (2026-07-31)
**No script, profile or build file was modified.** Findings:
`documentation/audits/SPIKE-universal-iso-2-2026-07-31.md`. Opens R-148..R-151.
**Both Tier 0 boxes went offline mid-session** (remote site, 12:28 CEST; four routes tried, our
tailscale pod healthy). Q1/Q2/Q3 each retain a part that needs a nested VM and those parts are
**BLOCKED, not answered**. The fixture is staged with Spike 1's two defects already corrected.
Established without them:
- **STRUCTURAL — `ordering = "before-network"` maps to `proxmox-first-boot-network-pre.service`
(`proxmox-first-boot` 9.2.5), which is `Before=network-pre.target` + `Type=oneshot`.** It therefore
runs to completion before any interface is configured — a credential rotation there has a
**zero-length window by construction**, not by being fast. Stronger than the timing number Spike 1
failed twice to obtain.
- **STRUCTURAL — the stub does not need `fully-up`.** `stub-first-boot.sh` contains no
`pvesh`/`pct`/`pveum`/`qm` call (grep rc=1); that usage is in `felhom-bootstrap.sh`, which runs under
its own `After=network-online.target ... pveproxy.service` unit. So `answer.toml.tmpl:27` justifies
the current ordering with a dependency that does not exist. **R-148.**
- **The ordering enum has THREE values** — `before-network`, `network-online`, `fully-up`. Spike 1
recorded two; the middle one has never been evaluated. **R-149.**
- **MECHANISM (container, not PVE) — a locked root closes `sulogin`:** `Cannot open access to console,
the root account is locked.` for both `*` and `!`, with a working control (correct password ->
SU_OK, wrong -> Authentication failure). So "discard" and "lock" are the SAME outcome for recovery,
which makes the escrow decision binary.
- **All four `proxmox-first-boot-*` units are `Condition*`-guarded, and a failed condition is a SKIP,
not a failure** — a hook that never ran looks exactly like one that succeeded. **R-150.**
- Q4: the single-entry menu is enforced by **four** gates in `iso-repack.sh` (`:153-155`, `:156`,
`:160-164`, `:314-319`), so restoring an interactive entry is a gate change, not a template edit —
but the banned-token gate would pass unchanged for a stock Graphical entry. `default`/`timeout` are
already settable in `grub/grub.cfg.tmpl`.
The first mechanism probe was invalid (ran as uid 0, so `su` never invoked pam_unix; `sulogin` had no
tty) and is recorded as failed, not massaged.
## SPIKE (no code change) — a universal ISO needs a different disk strategy and a locked root (2026-07-31)
**No script, profile or build file was modified.** Findings only: