admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

SPIKE 2 complete: locked root closes the PVE web UI; before-network gives a measured zero window

Browse Source 
Findings only — no script, profile or build file changed; no ISO built, nothing published.
documentation/audits/SPIKE-universal-iso-2-2026-07-31.md

Both Tier 0 boxes went offline mid-session (provider cable fault; four routes tried, no Tier 2
fallback used) and returned. All three scenarios then ran to completion on real PVE, each signalled
by reboot-mode='power-off' rather than a disk hash.

- A LOCKED ROOT CLOSES THE PVE WEB INTERFACE. Measured at the exact endpoint the UI uses
  (POST /api2/json/access/ticket, root@pam) WITH A WORKING CONTROL: known-password install returns
  HTTP 200 + ticket; locked install returns 401 for every password and none can exist.
  passwd -S root = L, shadow = literal-asterisk, PVE uses the stock PAM stack.
- GRUB recovery mode is also closed ('the root account is locked') — but init=/bin/bash still gives
  an unauthenticated root@(none):/#. A locked box is recoverable, operator-only, at the console.
  The installed GRUB has NO password, so locking root is not a physical-security measure. R-152.
- before-network MEASURED (A/B, same image): the hook RUNS (marker, uptime 6.58s) with entropy 256,
  writable /etc, all binaries and openssl_rand_len=32, while ip_global is EMPTY and
  listen_22_8006 = 0. fully-up is the converse: sshd+pveproxy active, 3 listening. Zero window.
- R-148: answer.toml.tmpl:27 justifies fully-up with a pvesh/pct dependency the stub does not have
  (grep rc=1) — it blocked the ordering now measured as the fix.
- R-149 three ordering values; R-150 Condition-guarded hooks skip silently; R-151 demo-felhom built
  from an uncommitted profile.

Three probes failed and are recorded as failed: a container probe that ran as uid 0, a GRUB probe
that missed the 1-second menu timeout, and a kernel-line edit one line off (caught by a pre-typing
verification screendump). The interim 'Layer 1 teardown INCOMPLETE' is corrected — the fixture had
never landed, because the staging mkdir was in the SSH call that timed out.
This commit is contained in:
authentik Default Admin
2026-07-31 13:57:21 +02:00
parent 5bdd8372f8
commit 19c932a693
3 changed files with 424 additions and 388 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT-universal-iso-spike.md
+63 -69
View File
@@ -1,95 +1,89 @@
# REPORT — SPIKE 2: closing the last mechanisms, and the universal ISO's shape (2026-07-31)
> Written as `REPORT-universal-iso-spike.md`, not `REPORT.md`: the shared file belongs to today's hub
> v0.85.0 session and the standing rule is that the second session in a shared clone never touches it.
> This file supersedes its own Spike 1 contents.
> v0.85.0 session and the second session in a shared clone never touches it.
**Class: Spike.** Findings only — no production file changed, no ISO built, nothing published.
Evidence: `documentation/audits/SPIKE-universal-iso-2-2026-07-31.md`. Continues
`SPIKE-universal-iso-2026-07-31.md`.
Evidence: `documentation/audits/SPIKE-universal-iso-2-2026-07-31.md`.
## The headline: the venue went away, and three answers are blocked
**All four questions are answered**, and every answer that matters is measured on a real PVE install.
The session lost both Tier 0 boxes mid-run (provider cable fault; four routes tried, no Tier 2
fallback used) and resumed when connectivity returned.
At 12:28 CEST both Tier 0 boxes went offline **simultaneously** and did not return. Four routes were
tried (tailnet direct, `ssh demo-hp`, `demo-hp-lan` via ProxyJump, `ssh felhom-pve`); our tailscale
pod is healthy and `tailscale status` shows both peers `offline` with a rising `last seen`. Both boxes
are at the same remote site — that site's power or uplink, not fixable from here.
## The two results that decide the product questions
**No fallback venue was used.** DooPlex is Tier 2, the 2026-07-25 ruling moved drill VMs off it, and
this task did not authorise it — an absent fence is not permission. So **Q1, Q2 and Q3 each keep a
part that is BLOCKED, not answered**, and they are labelled that way throughout. The fixture is built
and staged with Spike 1's two measurement defects already corrected (a NIC present at install time; a
`reboot-mode = "power-off"` completion signal instead of a disk hash).
**A locked root closes the PVE web interface.** Measured against the exact endpoint the UI logs in
through — `POST /api2/json/access/ticket`, realm `root@pam` — **with a working control**: the
known-password install returns **HTTP 200 and a ticket**; the locked install returns 401 for every
password offered, and no password exists that could work. The control is what makes this a finding
rather than a broken probe. `passwd -S root` reports `L`, the shadow field is `literal-asterisk`, and
PVE uses the stock PAM stack.
## What was established anyway
**GRUB's recovery mode is closed too** — `Cannot open access to console, the root account is locked.`
— **but `init=/bin/bash` still works**, giving an unauthenticated `root@(none):/#`. So a locked-root
box is *not* unrecoverable, as the structural reading had feared; it is recoverable **operator-only, at
the physical console**. The installed GRUB has no password, which also means locking root is not a
physical-security measure and must never be described as one (**R-152**).
**The best result of the session, and it is structural rather than measured — which makes it
stronger.** `ordering = "before-network"` maps to `proxmox-first-boot-network-pre.service`
(`proxmox-first-boot` 9.2.5, read out of the ISO's own package), which is `Before=network-pre.target`
with `Type=oneshot`. It therefore runs **to completion before any network interface is configured**.
A credential rotation placed there has a **zero-length exposure window by construction, not by being
fast** — and that holds on every machine, unlike the timing number Spike 1 failed twice to obtain.
## The mechanism that fixes the credential window
**`before-network` measured, A/B, same image, only `ordering` differing.** The hook **runs** (positive
marker, uptime 6.58 s) with entropy 256, a writable `/etc`, every binary it needs and a demonstrated
`openssl_rand_len=32` — while `ip_global` is **empty** and `listen_22_8006` is **0**. The `fully-up`
column is the converse proof: sshd and pveproxy already active, 3 sockets listening. **Zero credential
window, measured**, and the shipped unit (`Before=network-pre.target`, `Type=oneshot`) generalises it
to every machine.
**And the reason it was never used is a comment that is wrong.** `answer.toml.tmpl:27` justifies
`fully-up` as needed "so pvesh/pct work" — but `stub-first-boot.sh` makes no `pvesh`/`pct`/`pveum`/`qm`
call at all (grep `rc=1`). That usage lives in `felhom-bootstrap.sh`, which runs under its own
`After=network-online.target … pveproxy.service` unit. A comment asserting a **dependency** that does
not exist, which has constrained the design ever since. **R-148.**
`fully-up` as needed "so pvesh/pct work", but `stub-first-boot.sh` makes no `pvesh`/`pct`/`pveum`/`qm`
call at all (grep `rc=1`); that usage lives in `felhom-bootstrap.sh`, under its own properly-ordered
unit. **R-148** — a comment asserting a *dependency* that does not exist, which blocked the fix.
**A locked root closes `sulogin`** — measured in an isolated container with a working control (correct
password → `SU_OK`, wrong → `Authentication failure`; then `*` and `!` both → `L` and
`Cannot open access to console, the root account is locked.`). This reshapes the escrow question: if
nobody holds the password, `sulogin` refuses either way, so **"discard" and "lock" are the same
outcome for recovery**, and the decision is binary. Caveat stated plainly: container, not PVE.
## Probes that failed, recorded as failed
**The ordering enum has three values, not two** — `before-network`, `network-online`, `fully-up`.
Spike 1 recorded two; the middle one has never been evaluated (**R-149**).
Three, none massaged: a container probe that ran as uid 0 so `su` never invoked `pam_unix` (rc=0 even
with a wrong password); a GRUB probe that missed the **1-second** menu timeout and typed into the login
prompt; and a kernel-line edit that landed one line off — caught by a pre-typing verification
screendump before it was booted. All were rerun.
**A first-boot hook that never runs is indistinguishable from one that succeeded** — all four
`proxmox-first-boot-*` units are `Condition*`-guarded and a failed condition is a *skip*, not a
failure. Any rotation-in-first-boot design must verify by a positive marker (**R-150**).
Also corrected: the interim report's "Layer 1 teardown INCOMPLETE" was over-cautious. When demo-hp
returned, `/mnt/nvme-1tb/spike2/` did not exist and disk usage was unchanged — the staging `mkdir` had
been inside the SSH call that timed out as the link degraded, so **the fixture never landed**.
**Q4 answered fully from source.** The single-entry menu is enforced by **four** gates in
`iso-repack.sh` (`:153-155`, `:156`, `:160-164`, and a post-remaster re-extraction at `:314-319`), so
restoring an interactive entry is a gate change rather than a template edit — but the banned-token
gate would pass unchanged for a stock *Graphical* entry, so the safety property survives. `default`
and `timeout` are already settable in `grub/grub.cfg.tmpl`.
## The operator's decisions
## What failed, and is recorded as failed
**Escrow versus discard — no longer blocked, and reshaped twice.** "Discard" and "lock" are the same
outcome (nobody holds the password, so both `sulogin` and the web UI refuse), and **neither makes a box
unrecoverable** because `init=/bin/bash` always works. So escrow buys only this: the operator keeps the
web-UI and `sulogin` routes without physical presence. Against that stands D6 — a hub holding console
root for the whole fleet is what the hub alone must not be. **I recommend discard/lock for the public
image**, escrow remaining defensible for operator-built boxes. Your call; it is no longer waiting on a
measurement.
The first mechanism probe produced **zero valid observations**: it ran everything as uid 0, so `su`
never invoked `pam_unix` (rc=0 even with a wrong password), `sulogin` died on `tcgetattr` with no tty,
and a `sed` mangled the control hash. Rerun with a non-root caller, a real pty, and `chpasswd`.
**The default boot entry — framed, unruled.** The disk evidence (Spike 1) and the recovery evidence
(this session) now point the same way: a tester whose unattended install succeeds but whose pairing
fails has no web UI and no recovery shell. The cost is the unattended experience. Two profiles from one
build is cheap, so operator-built and public need not match.
A **teardown error** is also recorded rather than quietly fixed: shredding the control password
destroyed the plaintext scenario C existed to use, so `C.toml` was deleted rather than left looking
usable. It must be regenerated before the PVE-web-interface measurement is attempted.
## Still open
## The two decisions, and why neither should be ruled yet
**Escrow versus discard** — now binary (see above). **Blocked on one measurement:** whether a locked
root also closes the PVE web interface. If it does, discarding removes the last remote route as well
as the local one and the balance shifts sharply toward escrow. **Do not rule before that is measured.**
**The default boot entry for a public image** — the disk evidence points at interactive-default; the
cost is the unattended experience. Partially blocked on the same measurement. The
operator-built-versus-public split is cheap and available regardless of the ruling.
One gap in Q3: **Felhom's real stub at `before-network`** is argued structurally, not observed — the
scenarios used the spike's own diagnostic as the hook. That is the one thing to measure before a spec
is written (~40 min). Also: the SSH-as-root leg used a wrong password in the control too, so it
discriminates nothing and is recorded as a gap rather than a result.
## R-rows
**R-148** (`fully-up` justified by a non-existent dependency) · **R-149** (three ordering values, not
two) · **R-150** (Condition-guarded first-boot hook skips silently) · **R-151** (demo-felhom was
installed from an **uncommitted** profile — a Tier 0 reference box is not reproducible from `main`).
**R-148** (`fully-up` justified by a non-existent dependency) · **R-149** (three ordering values) ·
**R-150** (Condition-guarded hooks skip silently) · **R-151** (demo-felhom built from an uncommitted
profile) · **R-152** (a locked root is not a physical-security measure).
## Teardown
**Layer 1 (demo-hp) is INCOMPLETE and named as such** — the box went offline while
`/mnt/nvme-1tb/spike2/` held the staged fixture. Nothing was running on it (last check: 0 QEMU
processes, no install started this session), nothing on `local-lvm`, no PVE storage registered,
`drill-r50` and 9201 never touched. The outstanding command is recorded in §10 of the findings doc
and **is not claimed as done** — R-131 exists because recorded commands never get run.
Layer 2 (demo-felhom): nothing placed this session. Layer 3 (DooPlex): scratchpad 4.4 GB → 48 K,
control password `shred -u`'d and verified absent, mechanism container `--rm`, no global Docker
cleanup, `felhom-iso/out/` untouched at 17 ISOs, no production file modified. **Hub-side: nothing
created** — verified by fetching the customer list and searching it, not asserted. Nothing published.
All three layers verified positively. demo-hp is back to its exact pre-spike state: `spike2/` and
`/tmp/paia` removed, 0 QEMU processes, 0 loop devices, **disk usage 6.6 G — identical to pre-spike**,
4 storages unchanged, nothing on `local-lvm`, `drill-r50` stopped and 9201 running, both untouched.
demo-felhom: probe binary removed, external USB drive observed only, still mounted as found. DooPlex:
scratchpad 1.7 GB → 60 K, control password `shred -u`'d and verified absent, `felhom-iso/out/`
untouched at 17 ISOs, no production file modified. **Hub-side: nothing created** — the VMs had no route
to it. Nothing published.
documentation/audits/SPIKE-universal-iso-2-2026-07-31.md
+334 -290
View File
@@ -4,201 +4,290 @@
>
> Continues `SPIKE-universal-iso-2026-07-31.md`; its results are premises here and are not re-derived.
>
> **Read §0 first: the venue went away mid-session.** Both Tier 0 hosts dropped off the tailnet at
> ~12:28 CEST and did not return. Q1, Q2 and Q3 each have a part that needs a nested VM, and those
> parts are **BLOCKED, not answered**. What could be established without them — from the shipped PVE
> source, the answer schema, and an isolated mechanism probe — is below and is labelled by how it was
> obtained.
> **All four questions are answered, and every answer that mattered is measured on a real PVE install.**
> The session lost its venue mid-run (§0) and resumed when connectivity was restored.
---
## 0. Venue loss — what was tried
## 0. Venue loss and recovery — what was tried
At 12:28 CEST, mid-run, both Tier 0 boxes went offline **simultaneously**:
```
100.76.96.79 demo-hp active; relay "fra"; offline, last seen 5m ago, tx 1843821564 rx 63739848
100.70.170.35 felhom-pve active; relay "waw"; offline, last seen 5m ago, tx 1560 rx 0
```
Attempts, in order (per the standing rule that a "no access" claim names what it tried):
At 12:28 CEST both Tier 0 boxes went offline **simultaneously**. Routes attempted, in order:
| # | Route | Result |
|---|---|---|
| 1 | `ping 100.76.96.79` (tailnet direct) | 100 % packet loss |
| 2 | `ssh demo-hp` (tailnet, `ConnectTimeout=10`) | `connect to host 100.76.96.79 port 22: Connection timed out` |
| 3 | `ssh demo-hp-lan` (LAN `192.168.0.87`, ProxyJump `felhom-pve`) | `Connection timed out during banner exchange` |
| 4 | `ssh felhom-pve` (the jump host itself) | `connect to host 100.70.170.35 port 22: Connection timed out` |
| 5 | DooPlex tailscale pod health | pod `tailscale-c84df7f84-tw2wz` **Running 6d10h**, healthy; log shows `open-conn-track: timeout … online=yes` then peers dropping |
| 6 | `tailscale status` from the pod | **both peers `offline`**, `last seen 5m` and rising |
| 1 | `ping 100.76.96.79` (tailnet direct) | 100 % loss |
| 2 | `ssh demo-hp` | `Connection timed out` |
| 3 | `ssh demo-hp-lan` (LAN, ProxyJump `felhom-pve`) | `Connection timed out during banner exchange` |
| 4 | `ssh felhom-pve` (the jump host) | `Connection timed out` |
| 5 | DooPlex tailscale pod | `Running 6d10h`, healthy |
| 6 | `tailscale status` from the pod | **both peers `offline`**, `last seen` rising |
**Diagnosis: the far end, not ours.** Our tailscale pod is healthy and every other peer route behaves;
both demo boxes are at the same remote site (REMOTE until ~08-02) and went offline together, which is
that site's power or uplink. Not fixable from here.
Far end, not ours. **No fallback venue was used** — DooPlex is Tier 2, the 2026-07-25 ruling moved
drill VMs off it, and this task did not authorise it. An absent fence is not permission.
**No fallback venue was used.** DooPlex is Tier 2 and the 2026-07-25 standing ruling moved drill and
build VMs off it; `target-selection.md` says a Tier 2 machine is used only when a task says so
explicitly, and this one does not. **An absent fence is not permission**, so the probes simply did not
run. A watch is armed for the boxes' return.
Connectivity was restored (operator: a provider cable fault nearby) and the boxes returned with
**uptime 7 days** — a pure network outage, no reboot. All probes then ran.
The probe fixture is built and staged and will run unchanged when they come back: probe ISO
(`prepare-iso --fetch-from partition`, serial-capture first-boot diagnostic), three validated answer
files (locked/`fully-up`, locked/`before-network`, known-password control), an install runner with
`reboot-mode = "power-off"` as the completion signal, and a boot runner with protocol-level detectors
and **a NIC present at install time** — the two fixture defects that invalidated Spike 1's
measurements are already corrected.
**One thing the outage revealed.** `/mnt/nvme-1tb/spike2/` did not exist when demo-hp came back, and
disk usage was unchanged at 6.6 GB: the staging `mkdir` had been inside the SSH call that timed out as
the link degraded, so **the fixture never landed and Layer 1 had nothing to clean**. The interim
report's "Layer 1 teardown INCOMPLETE" was therefore over-cautious rather than wrong — corrected here,
and the fixture was rebuilt from scratch.
---
## 1. Baselines
Read fresh this session.
| Baseline | Value |
|---|---|
| `ISO_VERSION` | `1.25.0` — `scripts/iso/build-felhom-iso.sh:44` |
| `SCRIPT_VERSION` | `1.22.0` — `scripts/felhom-host-install.sh:187` |
| `felhom-bootstrap.sh` @ HEAD | `21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7` |
| PVE base | `proxmox-ve_9.2-1.iso`, `4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c` |
| `felhom.eu` HEAD | **`ea00976`**, clean, `== origin/main` (the expected Spike 1 commit) |
| `felhom.eu` HEAD | **`ea00976`**, clean, `== origin/main` |
| `proxmox-first-boot` pkg | `9.2.5` (from the ISO's `/proxmox/packages/`) |
| installed guest kernel | `7.0.2-6-pve`, GRUB `2.12-9+pmx2` |
**Fixture provenance.** Probe ISO built on DooPlex from the baseline PVE ISO. Nested VMs were to run
under plain QEMU/KVM on demo-hp under `/mnt/nvme-1tb/spike2/` — never `local-lvm`, no PVE storage
registered. They did not run (§0). The mechanism probe in §2.2 ran in a throwaway `debian:trixie`
Docker container on DooPlex, which is not an install and touches no box.
**Fixture provenance.** Probe ISO built on DooPlex from the baseline PVE ISO
(`prepare-iso --fetch-from partition --partition-label proxmox-ais --on-first-boot diag.sh`),
sha256 `dd97dba5518c70baf4300b119e1042eabf1012d5a046082acf31e60983efaeb9`, **verified byte-identical at
both ends** after transfer. Nested VMs ran under plain QEMU/KVM on **demo-hp** (Tier 0, the designated
drill host) under `/mnt/nvme-1tb/spike2/` — never `local-lvm`, no PVE storage registered, `drill-r50`
and guest 9201 untouched.
**Evidence classes used throughout:** **[MEASURED-PVE]** against the real PVE tooling or a PVE install ·
**[MEASURED-MECH]** measured, but in an isolated container rather than on PVE · **[STRUCTURAL]** read
from shipped source or unit files · **[BLOCKED]** needs the nested VM.
**Both of Spike 1's fixture defects are corrected in this one:** a NIC is present **at install time**
(its absence made the installed system's interface config unmatchable at boot), and completion is
signalled by `reboot-mode = "power-off"` — the guest powering itself off — never by a changed disk hash.
Three scenarios, all installed to completion:
| Scenario | root credential | `[first-boot].ordering` | Install |
|---|---|---|---|
| **A** | `root-password-hashed = "*"` (locked) | `fully-up` | completed, powered off after **191 s** |
| **B** | `root-password-hashed = "*"` (locked) | `before-network` | completed, powered off after **190 s** |
| **C** | a **known** password (control) | `fully-up` | completed, powered off after **191 s** |
**Evidence classes:** **[MEASURED-PVE]** on a real PVE install · **[MEASURED-MECH]** measured in an
isolated container · **[STRUCTURAL]** read from shipped source or unit files.
---
## 2. Q1 — is a locked root actually locked, and what else does it lock out?
### 2.1 The account's state on a real PVE install — **[BLOCKED]**
### 2.1 The account's real state — **[MEASURED-PVE]**
Needs the nested VM. The fixture is staged. Spike 1 established only that
`root-password-hashed = "*"` **installs to completion**; the resulting account state was explicitly
left open there and remains open.
From the first-boot diagnostic on both locked installs (A and B), emitted to `ttyS0`:
### 2.2 What can authenticate — **[MEASURED-MECH]**, and the first attempt was invalid
```
passwd_S_root : root L 2026-07-31 0 99999 7 -1
shadow_shape : literal-asterisk
sshd_PermitRoot : yes
pam_common_auth : 1
pve_pam_realm : 0
```
**The first probe produced nothing and is recorded as failed.** It ran everything as uid 0, so `su`
never invoked `pam_unix` at all (`rc=0` even with a deliberately wrong password); `sulogin` died on
`tcgetattr failed: Invalid argument` with no TTY; and a `sed` mangled the control hash so the control
case had no password set. Four cases, zero valid observations. Rerun with a non-root caller, a real
pty via `script`, and `chpasswd` instead of `sed`.
**The account is genuinely locked** — `passwd -S` reports `L`, and the `*` reached `/etc/shadow`
verbatim. Note `sshd_PermitRoot: yes`: PVE ships `PermitRootLogin yes`, so the lock is doing the work,
not sshd's configuration.
**Corrected probe** (`debian:trixie`, `util-linux 2.41`, `pam_unix`) — with a working control:
`pve_pam_realm: 0` — there is no PVE-specific file in `/etc/pam.d/` matching `proxmox|pve`, and
`pam_unix` is present in `common-auth`. **PVE uses the stock PAM stack**, which is why the §2.2
container measurement transfers to it rather than merely resembling it.
| Shadow field | `passwd -S root` | `su root` as non-root user | `sulogin` (real pty) |
### 2.2 What can authenticate — **[MEASURED-MECH]**, first attempt invalid
**The first probe produced nothing and is recorded as failed:** it ran everything as uid 0 so `su`
never invoked `pam_unix` (`rc=0` even with a deliberately wrong password), `sulogin` died on
`tcgetattr failed` with no TTY, and a `sed` mangled the control hash. Four cases, zero observations.
Corrected (`debian:trixie`, `util-linux 2.41`, non-root caller, real pty via `script`, `chpasswd`):
| Shadow field | `passwd -S root` | `su root` as a non-root user | `sulogin` (real pty) |
|---|---|---|---|
| real hash (control) | `P` | **correct pw → `SU_OK`**; wrong pw → `Authentication failure` | — |
| `*` | **`L`** | `su: Authentication failure` (any password) | **`Cannot open access to console, the root account is locked.`** |
| `!` | **`L`** | `su: Authentication failure` (any password) | **`Cannot open access to console, the root account is locked.`** |
| real hash (**control**) | `P` | **correct pw → `SU_OK`**; wrong pw → `Authentication failure` | — |
| `*` | `L` | `Authentication failure` (any password) | `Cannot open access to console, the root account is locked.` |
| `!` | `L` | `Authentication failure` (any password) | `Cannot open access to console, the root account is locked.` |
The control is what makes this readable: the same harness authenticates successfully with the right
password and fails with the wrong one, so the two locked rows are the module refusing, not the harness
misfiring.
The control makes it readable: the same harness succeeds with the right password and fails with the
wrong one, so the locked rows are the module refusing rather than the harness misfiring. **`*` and `!`
are equivalent.**
**So, at mechanism level: `*` and `!` are equivalent, both report `L`, and no password authenticates.**
**On PVE, root SSH was refused on every scenario** — but that row is **uninformative and is not
evidence**: the probe supplied a deliberately wrong password in the control scenario too, so it cannot
distinguish "locked" from "wrong password". Recorded as a gap, not a result.
**Caveat, and it is not small.** This is Debian's `pam_unix` and `util-linux`'s `sulogin` — the same
components PVE ships — but it is **not PVE**. PVE could ship PAM configuration, a `sulogin` override,
or a `SYSTEMD_SULOGIN_FORCE` setting that changes the outcome. Confirming on a real PVE install is
part of the blocked work.
### 2.3 Does the Proxmox web interface still work? — **[MEASURED-PVE]. No. It is closed.**
### 2.3 Does the Proxmox web interface still work? — **[BLOCKED]**, and this is the product question
The product-level answer, measured against the **exact endpoint the web UI logs in through**
(`POST /api2/json/access/ticket`, realm `root@pam`), with a working control:
**No measurement was obtained, so no answer is given.** Stating the reasoning without the measurement
would be exactly the failure mode this spike exists to avoid.
| Scenario | Password offered | Result |
|---|---|---|
| **C (control, known password)** | **correct** | **HTTP 200, TICKET ISSUED — login would SUCCEED** |
| C (control) | wrong | HTTP 401, no ticket |
| C (control) | empty | HTTP 401, no ticket |
| **A (locked root)** | wrong | HTTP 401, no ticket |
| **A (locked root)** | empty | HTTP 401, no ticket |
What is established: PVE's `root@pam` realm authenticates the Linux root account through PAM, and
§2.2 measured that `pam_unix` refuses a locked account for a non-root caller. If PVE's proxy
authenticates the same way — **which was not verified** — then a locked root means **nobody can log
into the PVE web interface at all**, neither customer nor operator, and that is a product decision
rather than a detail.
The control is what makes this a finding rather than a broken probe: **the same endpoint on the same
image demonstrably issues a ticket when the password is right.** On the locked install no password can
produce one, because no password exists.
The staged control scenario exists precisely to settle this: install C carries a **known** password, so
the PVE API ticket endpoint can be exercised positively (correct password → ticket) before install A
(locked) is tested against the same endpoint. Without that control a failure is indistinguishable
from a broken probe — Spike 1's lesson.
**Therefore a locked root closes the PVE web interface completely — to the customer and to the
operator alike.** This is a product decision, not a detail: it removes the browser-based route into
the box entirely, on top of the console route (§3.2).
### 2.4 Does anything else in PVE break? — **[BLOCKED]**
Reachability, incidentally measured on all three: **SSH banner at +20 s, pveproxy TLS at +26–29 s**
after power-on.
Not investigated. Cron, `pveproxy`, backup jobs and anything else assuming a usable root credential
remain unexamined.
### 2.4 Does anything else in PVE break? — **not investigated**
Cron, backup jobs and other consumers of a root credential were not examined. `pveproxy` itself runs
and serves TLS (it answered `/api2/json/version`), so the daemon is unaffected — only authentication
through it is.
---
## 3. Q2 — is there a way back into a box whose setup failed?
### 3.1 Does the installed GRUB expose a recovery entry? — **[BLOCKED]**
### 3.1 Does the installed GRUB expose a recovery entry? — **[MEASURED-PVE]. Yes.**
Needs a boot. Note the installed system's GRUB is **PVE's own**, written by the installer — not the
ISO's Felhom menu — so §5's single-entry collapse does not apply to it. What it offers is unverified.
### 3.2 `sulogin` with a locked root — **[MEASURED-MECH]. It refuses.**
The single observation the escrow-versus-discard ruling was said to rest on:
The installed system's GRUB is PVE's own, unaffected by Felhom's ISO menu collapse (§4). Screendumped
from the locked install:
```
*Proxmox VE GNU/Linux
Advanced options for Proxmox VE GNU/Linux
Memory test (memtest86+x64.bin)
Memory test (memtest86+x64.bin, serial console)
```
and inside *Advanced options*:
```
*Proxmox VE GNU/Linux, with Linux 7.0.2-6-pve
Proxmox VE GNU/Linux, with Linux 7.0.2-6-pve (recovery mode)
```
**A recovery entry exists, the menu is editable (`e`), a GRUB command line is available (`c`), and
there is no GRUB password.** Its timeout is **1 second** — short enough that a first probe missed it
entirely (§3.5).
### 3.2 `sulogin` with a locked root — **[MEASURED-PVE]. It refuses.**
Booting *(recovery mode)* on the locked install reaches rescue mode and then stops:
```
Started rescue.service - Rescue Shell.
Reached target rescue.target - Rescue Mode.
You are in rescue mode. After logging in, type "journalctl -xb" …
Cannot open access to console, the root account is locked.
See sulogin(8) man page for more details.
Press Enter to continue.
```
Both for `*` and for `!`, with a real pty. **The standard Debian/systemd emergency and single-user
path does not grant a shell on a locked-root system.** `systemd` invokes `sulogin` for
`emergency.target` and `rescue.target`, so both are closed by the same refusal.
**The advertised recovery path is closed on a locked-root box** — verbatim the container result,
now confirmed on real PVE. `systemd` invokes `sulogin` for both `rescue.target` and
`emergency.target`, so both are closed by the same refusal.
Subject to the §2.2 caveat: measured in a container, not on PVE.
### 3.3 What is left — **[MEASURED-PVE]. `init=/bin/bash` works.**
### 3.3 What is left if `sulogin` refuses — **[STRUCTURAL]**, not measured
Appending `init=/bin/bash` to the kernel line in the GRUB editor and booting produced:
| Path | Realistic for whom |
|---|---|
| `init=/bin/bash` on the kernel line (GRUB `e`) | **Operator only.** It bypasses `sulogin` because PID 1 becomes the shell — no authentication is involved. Requires editing a boot entry at the console. Not a non-technical customer action. |
| `systemd.unit=emergency.target` | **Closed** — it lands in `sulogin`, which refuses (§3.2). |
| Live USB / rescue media | Operator only; requires physical media and knowing to do it. |
| PVE web console (`:8006`) | **Unknown, possibly closed** — §2.3. |
| SSH by key | Only if a key was baked, which for a public image is the thing R-142 says must not happen. |
```
Found volume group "pve" using metadata type lvm2
3 logical volume(s) in volume group "pve" now active
/dev/mapper/pve-root: clean, 60332/561936 files, 1141450/2243584 blocks
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
root@(none):/#
```
**The honest summary:** on a locked-root box, every remaining route is operator-grade and physical.
For a customer whose pairing failed, there is no self-service path back — which is precisely the
objection raised when locked-root was first proposed, and §3.2 is the first evidence that the
objection is well founded rather than speculative.
**An unauthenticated root shell on a locked-root system.** No password is involved because PID 1 *is*
the shell — `sulogin` is never reached.
### 3.4 Does locked differ from rotated? — **[PARTIALLY ESTABLISHED]**
| Path | Works? | Realistic for whom |
|---|---|---|
| PVE web console (`:8006`) | **No** (§2.3) | — |
| GRUB *(recovery mode)* → `sulogin` | **No** (§3.2) | — |
| GRUB editor → `init=/bin/bash` | **Yes** (measured) | **Operator only.** Requires catching a 1-second menu, editing a kernel line, and knowing the incantation. Not a non-technical customer action. |
| Live USB / rescue media | not tested | Operator only |
| SSH by key | n/a | Only if a key were baked — which R-142 says must not happen in a public image |
At mechanism level, yes, and materially:
**So a locked-root box is recoverable, but only by someone competent at the physical console.** That is
a materially weaker objection than "no path back at all", which is what the structural reading
suggested before it was measured.
- **Locked (`*`/`!`)** — `sulogin` refuses; console recovery is closed to everyone; the credential
does not exist so it cannot leak, be escrowed, or be rotated.
- **Rotated to a per-box random value** — `sulogin` prompts and *accepts the correct password*
(the control row in §2.2 shows `pam_unix` authenticating a real hash). So console recovery is open
**to whoever holds that password** — which is the operator if it is escrowed, and nobody if it is
discarded.
**A security corollary that is worth stating plainly: the installed GRUB has no password, so locking
root buys nothing against physical access.** Anyone at the keyboard can take a root shell in under a
minute. Locked root is a defence against *remote* credential reuse — the shared-ISO-password problem —
and not a physical-security measure. **R-152.**
**This makes the tradeoff concrete: "discard" and "lock" are the same outcome for recovery.** A
rotated-then-discarded password is operationally identical to a locked account, because nobody holds
it. The real choice is therefore **binary — escrow, or accept that console recovery is operator-only
and physical.** Framed for the ruling in §7.
### 3.4 Does locked differ from rotated? — **[MEASURED]. Yes, at exactly one point.**
- **Locked (`*`/`!`)** — `sulogin` refuses (§3.2); the web UI refuses (§2.3); `init=/bin/bash` works.
- **Rotated to a per-box value** — `pam_unix` accepts the correct password (the §2.2 control proves
this) and the web UI issues a ticket for it (the §2.3 control proves this), so **both** the console
and the browser open **to whoever holds it**; `init=/bin/bash` works as well.
**And this is what settles the escrow question.** A rotated password that is *discarded* is
operationally identical to a locked account: nobody holds it, so `sulogin` and the web UI refuse
either way. The only thing escrow buys is that the **operator** keeps the web-UI and `sulogin` routes.
It does not buy recoverability as such, because `init=/bin/bash` is available in every case.
### 3.5 A probe that failed, recorded as failed
The first GRUB attempt waited 3 seconds before acting; the menu's timeout is **1 second**, so the
system had already booted and the keystrokes went into the login prompt — the arrow-key escape
sequences (`^[[B`) are visible there in the screendump. It tested nothing. A later attempt typed
`init=/bin/bash` onto the `echo` line one below `linux`, which the **pre-typing verification
screendump caught** before it was booted. Both were rerun; the numbers above are from the corrected
runs.
---
## 4. Q3 — does the Felhom stub survive `before-network`?
### 4.1 Does it execute? — **[BLOCKED]**
### 4.1 Does it execute? — **[MEASURED-PVE]. Yes.**
Needs the VM. The staged diagnostic emits `@@SPIKE2_FIRSTBOOT_RAN@@ marker=BEGIN/END` to `ttyS0`
(captured by `qemu -serial file:`) precisely so the answer is a **positive observable** rather than the
absence of an error — and §4.4 below is why that matters more than usual here.
The diagnostic emits `@@SPIKE2_HOOK_RAN@@ BEGIN`/`END` to `ttyS0`. On scenario B
(`ordering = "before-network"`) **both markers are present** (count `2`). This is a positive
observable the hook produced, which §4.4 explains is the only acceptable form of proof here.
### 4.2 Does it have what it needs that early? — **[STRUCTURAL]**, partly answered
### 4.2 Does it have what it needs that early? — **[MEASURED-PVE]. Yes, everything.**
From the shipped unit (`proxmox-first-boot` 9.2.5,
`/usr/lib/systemd/system/proxmox-first-boot-network-pre.service`):
### 4.3 Does it complete before anything is network-reachable? — **[MEASURED-PVE + STRUCTURAL]. Yes.**
The A/B comparison, same image, same diagnostic, only `ordering` differs:
| Diagnostic | **A — `fully-up`** | **B — `before-network`** |
|---|---|---|
| hook markers | 2 (ran) | **2 (ran)** |
| `uptime_s` at hook | 18.32 | **6.58** |
| `entropy_avail` | 256 | **256** |
| `etc_writable` | YES | **YES** |
| `openssl`/`chpasswd`/`systemctl`/`passwd`/`curl`/`ip` | all present | **all present** |
| `openssl_rand_len` | 32 | **32** |
| `ip_global` | `vmbr0 10.0.2.15/24` | **(empty)** |
| `network.target` | active | **inactive** |
| `network-online.target` | active | **inactive** |
| `pveproxy` | active | **inactive** |
| `sshd` | active | **inactive** |
| **`listen_22_8006`** | **3** | **0** |
**At `before-network` the hook runs with entropy, a writable `/etc`, every binary it would need, and a
demonstrated ability to generate a 32-character random string — while the box has no IP address and
nothing is listening on `:22` or `:8006`.** A credential rotation placed there has a **zero-length
exposure window, measured.**
Equally, the `fully-up` column is the measured proof of the converse: **today's ordering runs the hook
when sshd and pveproxy are already listening.** The window Spike 1 argued for structurally is now
observed.
The structural half agrees and generalises it. The shipped unit
(`proxmox-first-boot` 9.2.5, `/usr/lib/systemd/system/proxmox-first-boot-network-pre.service`):
```
After=systemd-remount-fs.service
@@ -209,57 +298,37 @@ ConditionPathIsReadWrite=/var/lib
Type=oneshot
```
`After=systemd-remount-fs.service` plus `ConditionPathIsReadWrite=/var/lib` mean **a writable
filesystem is structurally guaranteed** — or the unit does not run at all (§4.4). Entropy availability
and binary presence at that point are **not** established structurally and are part of the staged
diagnostic (`entropy_avail`, `openssl rand` length, `command -v` for `openssl`/`chpasswd`/`systemctl`).
### 4.3 Does it complete before anything is network-reachable? — **[STRUCTURAL]. Yes, by construction.**
This is the result the task asked to establish structurally rather than by timing, and the unit file
settles it:
- **`Before=network-pre.target`** — `network-pre.target` is systemd's ordering barrier that precedes
*all* network configuration; network managers order themselves after it.
- **`Type=oneshot`** — systemd treats the unit as complete only when the process exits, so
`Before=` is a hard barrier rather than a hint.
Together: **the hook runs to completion before any interface is configured**, therefore before `sshd`
can bind a routable address and before `pveproxy` serves. A rotation performed there has a
**zero-length exposure window by construction, not by being fast.** This holds on every machine and
does not vary with hardware — which is why it is stronger than the timing number Spike 1 failed twice
to obtain.
`Before=network-pre.target` + `Type=oneshot` makes it a hard barrier ahead of all network
configuration — so this holds on every machine, not just this VM. Both installs then booted to full
reachability (SSH +20 s, pveproxy +26–29 s), so the early hook does not break the boot.
**A third ordering value exists that Spike 1 did not record.** `validate-answer` accepts
**`before-network`, `network-online`, and `fully-up`** — and rejects `network-pre`, `multi-user` and a
bogus value, so this is a real enum and not a permissive parser. The shipped package contains a
matching unit for each. Spike 1 reported two. `network-online` is the middle option — after the
network, before `pveproxy` — and is worth considering if `before-network` proves too early for
something.
**`before-network`, `network-online`, `fully-up`**; it rejects `network-pre`, `multi-user` and a bogus
value, so this is a real enum. The shipped package contains a matching unit for each.
`network-online` is the unevaluated middle option. **R-149.**
### 4.4 Does the existing pairing flow still work? — **[STRUCTURAL]** yes; **[BLOCKED]** empirically
### 4.4 Does the existing pairing flow still work? — **[STRUCTURAL]**, and it is the one gap left
Two structural findings, and the second one corrects a premise this design has been carrying.
**Not measured.** The scenarios installed the spike's own diagnostic as the first-boot hook, not
Felhom's `stub-first-boot.sh`, so stub-plus-pairing coexistence is argued, not observed.
**The stub does not need the network, and does not need `pveproxy`.** `stub-first-boot.sh` writes three
files, then `systemctl daemon-reload`, `enable`, and `start --no-block felhom-bootstrap.service`. A
grep for `pvesh`, `pct`, `pveum` and `qm` across the stub returns **no match** (`rc=1`). The unit it
starts carries its own `After=network-online.target pve-cluster.service pveproxy.service`, so
`--no-block` merely queues the job and systemd holds it until those are satisfied. Structurally, stub
and pairing coexist: the stub runs early, the bootstrap runs when its own dependencies are met.
The structural case is strong and has a correction in it:
**So `answer.toml.tmpl:27` justifies the `fully-up` choice with a dependency the stub does not have.**
The comment reads *"fully-up = the stub runs after pveproxy so pvesh/pct work"* — but the `pvesh`/`pct`
usage is in `felhom-bootstrap.sh`, which never runs under the stub's ordering. This is the
CLAUDE.md "a comment asserting an invariant" class in a new shape: a comment asserting a **dependency**
that does not exist, which has constrained the design ever since. **R-148.**
**The stub does not need `fully-up`.** `stub-first-boot.sh` writes three files, then
`systemctl daemon-reload`, `enable`, and `start --no-block felhom-bootstrap.service`. A grep for
`pvesh`, `pct`, `pveum` and `qm` across it returns **no match** (`rc=1`). The unit it starts carries its
own `After=network-online.target pve-cluster.service pveproxy.service`, so `--no-block` queues the job
and systemd holds it until those are satisfied.
**The trap that makes §4.1's positive observable load-bearing.** All four `proxmox-first-boot-*` units
are guarded by `ConditionPathExists=` and `ConditionPathIsReadWrite=`. In systemd a **failed condition
is a skip, not a failure** — the unit reports `inactive`/`condition failed`, the boot succeeds, and
nothing is logged as wrong. A first-boot hook that never ran and one that ran perfectly are
indistinguishable from the absence of errors. **R-150.** This is why §4.1 must be confirmed by a
marker the hook emitted, never by "no errors appeared".
**So `answer.toml.tmpl:27` justifies the current ordering with a dependency the stub does not have** —
it reads *"fully-up = the stub runs after pveproxy so pvesh/pct work"*, but that usage lives in
`felhom-bootstrap.sh`, which never runs under the stub's ordering. A comment asserting a **dependency**
that does not exist, and it has constrained the design ever since. **R-148.**
**Why §4.1's positive marker is load-bearing.** All four `proxmox-first-boot-*` units are guarded by
`ConditionPathExists=` / `ConditionPathIsReadWrite=`, and in systemd **a failed condition is a skip,
not a failure** — no error, successful boot. A hook that never ran and one that ran perfectly are
indistinguishable from the absence of errors. **R-150.**
---
@@ -267,119 +336,99 @@ marker the hook emitted, never by "no errors appeared".
### 5.1 Where the collapse happens — **[STRUCTURAL]**, and it is not a simple removal
`iso-repack.sh` replaces the stock `grub.cfg` wholesale: it renders `grub/grub.cfg.tmpl` over it
(`iso-repack.sh:144-148`), having first lifted the `linux`/`initrd` lines verbatim from the stock
*Install Proxmox VE (Automated)* entry (`:108-131`) so a PVE bump tracks automatically.
`iso-repack.sh` replaces the stock `grub.cfg` wholesale, rendering `grub/grub.cfg.tmpl` over it
(`:144-148`) after lifting the `linux`/`initrd` lines verbatim from the stock *Install Proxmox VE
(Automated)* entry (`:108-131`) so a PVE bump tracks automatically.
**Four independent gates then enforce the single entry**, so restoring the stock menu is a change to
the gates, not just to the template:
**Four independent gates enforce the single entry:**
| Gate | Location | Effect |
|---|---|---|
| exactly one `menuentry` | `iso-repack.sh:153-155` | `exit 14` otherwise |
| zero `submenu` | `iso-repack.sh:156` | `exit 14` otherwise |
| no live reference to `proxtui`, `proxdebug`, `nomodeset`, `Rescue Boot`, `memtest`, `fwsetup` | `iso-repack.sh:160-164` | `exit 14` otherwise |
| **re-extracted from `final.iso`**: exactly one entry, theme present, entry named `Felhom telepítés` | `iso-repack.sh:314-319` | `exit 15` otherwise |
| exactly one `menuentry` | `:153-155` | `exit 14` |
| zero `submenu` | `:156` | `exit 14` |
| no live `proxtui`, `proxdebug`, `nomodeset`, `Rescue Boot`, `memtest`, `fwsetup` | `:160-164` | `exit 14` |
| **re-extracted from `final.iso`**: one entry, theme present, entry named `Felhom telepítés` | `:314-319` | `exit 15` |
The fourth is the strongest of them — it proves the menu in the shipped image rather than in the build
tree, which is the seam-wiring lesson applied correctly.
The fourth proves the menu in the shipped image rather than the build tree — the seam-wiring lesson
applied correctly.
**What adding one interactive entry would actually cost:** the two count gates (rows 1 and 4) move from
`== 1` to `== 2`, or better to a named allowlist. **The banned-token gate would still pass unchanged**
for a stock *Install Proxmox VE (Graphical)* entry — it uses none of `proxtui`, `proxdebug`,
`nomodeset`, `Rescue Boot`, `memtest`, `fwsetup`. So the safety property the gates exist to protect
(no route to a debug/rescue/manual-disk-picker path) survives a second *graphical installer* entry
intact. That is a narrower change than it first appears.
**What a second entry would cost:** the two count gates move from `== 1` to `== 2`, or to a named
allowlist. **The banned-token gate passes unchanged** for a stock *Install Proxmox VE (Graphical)*
entry — it uses none of those six tokens. The safety property the gates exist for (no route to a
debug/rescue/manual-disk-picker path) survives a second graphical-installer entry intact.
### 5.2 Are default and timeout settable? — **[STRUCTURAL]. Yes, already.**
### 5.2 Are default and timeout settable? — **[STRUCTURAL]. Already are.**
`grub/grub.cfg.tmpl` sets them explicitly:
`grub/grub.cfg.tmpl` sets `set timeout_style=menu`, `set timeout=5`, `set default=0`. The template also
records a real defect it fixed: the stock PVE config writes `timeout-style` (hyphen), **which GRUB does
not recognise as a variable name**, so Felhom's underscore spelling is the one that works.
```
set timeout_style=menu
set timeout=5
set default=0
```
### 5.3 The design question — **framed with evidence, deliberately not ruled**
Both are plain template values. The template also records a real defect it fixed: the stock PVE config
writes `timeout-style` (hyphen), **which GRUB does not recognise as a variable name**, so Felhom's
`timeout_style` (underscore) is the one that actually works. A two-entry menu would therefore have a
functioning timeout and a settable default — which is exactly what the §5.3 question needs.
**For interactive-as-default.** Spike 1 proved the automated path cannot be made safe on unseen
hardware: no property distinguishes a system disk from a customer's backup drive, and a filter matching
two disks silently wipes one. A tester who boots and walks away on multi-disk hardware is walking away
from an unattended destructive operation whose target cannot be predicted.
### 5.3 The design question — **framed, deliberately not ruled**
**This session strengthens that case independently.** If the public image also ships a locked root,
then a box whose install succeeds but whose pairing fails has **no web UI** (§2.3) and **no working
recovery-mode shell** (§3.2) — only an operator-grade `init=/bin/bash` (§3.3). An unattended default
maximises the number of testers who reach that state without having read anything.
Which entry should be `default=0` in a **public** image?
**What the measurements say, and they only point one way:** Spike 1 established that the automated
path cannot be made safe on unseen hardware — no property distinguishes a system disk from a
customer's backup drive, and a filter matching two disks silently wipes one. A tester who boots the
image and walks away is, on multi-disk hardware, walking away from an unattended destructive operation
whose target cannot be predicted.
**What that costs:** unattended install stops being the default for people who do not read
instructions — and unattended install is a selling point. Most first boxes are single-disk, where the
**Against.** Unattended install is a selling point and most first boxes are single-disk, where the
automated path is fine.
**The measurement that would change the balance** is §2.3 and §3: if a locked root also closes the PVE
web interface and the console, then a tester whose unattended install completes but whose pairing
fails has a machine nobody can reach — which argues for interactive-default independently of the disk
question. Those measurements are blocked, so the balance cannot be struck yet.
**Operator-built versus public need not match, and two profiles from one build is cheap.** An
operator-built image targets a machine whose disks are known, so automated-default is safe there. The
public image is the one facing unseen hardware. Recommending the split is not the same as ruling the
default, and the split is available regardless of how the ruling goes.
**Operator-built and public need not match, and two profiles from one build is cheap.** An
operator-built image targets known disks, so automated-default is safe there; the public image is the
one facing unseen hardware. That split is available however the default is ruled.
---
## 6. Q5 — the resulting shape, as constraints
Constraints on a build spec, each with a release-gate criterion. **Not an implementation, and nothing
was built.** Constraints marked ⚠ depend on blocked measurements and are provisional.
Constraints on a build spec, each with a release-gate criterion. **Not an implementation.**
| # | Constraint | Release gate |
|---|---|---|
| 1 | **The image must not select a disk by property on unseen hardware.** Spike 1 §3.2/§3.3. | The rendered answer contains no `filter.*` key, **or** the image is operator-built for a named machine. Grep the rendered `answer.toml`. |
| 2 | **The interactive installer must be reachable from the boot menu.** §5.1 shows the cost is two count gates. | `final.iso`'s `grub.cfg` contains ≥1 interactive entry and still contains no live `proxtui`/`proxdebug`/`nomodeset`/`Rescue Boot`/`memtest`/`fwsetup` — i.e. keep gate 3, relax gates 1 and 4 to an allowlist. |
| 3 | ⚠ **No usable baked credential.** `root-password-hashed = "*"` is schema-valid and installs (Spike 1). | The rendered answer's `root-password-hashed` is exactly `*`, and **no `.rootpw.txt` is emitted**. Both are one-line assertions in the build. |
| 4 | ⚠ **If a per-box password is set, it is set at `before-network`.** §4.3 — the only ordering with a zero window by construction. | The rendered answer's `[first-boot].ordering` is `before-network`; and the hook's positive marker is observed in a boot test, never inferred from absent errors (§4.4, R-150). |
| 5 | **No `root-ssh-keys`, ever, in a public image.** Spike 1 R-142. | `grep -c root-ssh-keys` on the rendered answer is `0`. |
| 6 | **The profile must be committed.** Spike 1 found demo-felhom's baked key comes from an *uncommitted* profile, so a reference box is not reproducible from the repo. | The `--profile` path resolves inside the repo working tree **and** `git status --porcelain` is empty for it. **R-151.** |
| 7 | **The artefact must be reproducible.** Spike 1 R-144 (`nested-probe`'s profile no longer exists). | Manifest records the profile's committed path and the repo commit; both resolve. |
| 8 | **A recovery path must be documented and, if operator-only, said so plainly.** §3.3. | The release notes name the recovery route for a failed pairing. If that route is `init=/bin/bash`, the notes say it is operator-only. |
| 1 | **No disk selection by property on unseen hardware** (Spike 1 §3.2/§3.3). | Rendered answer contains no `filter.*` key, **or** the image is operator-built for a named machine. |
| 2 | **The interactive installer must be reachable from the boot menu** (§5.1). | `final.iso`'s `grub.cfg` has ≥1 interactive entry **and** still no live `proxtui`/`proxdebug`/`nomodeset`/`Rescue Boot`/`memtest`/`fwsetup` — keep gate 3, relax gates 1 and 4 to an allowlist. |
| 3 | **No usable baked credential** — `root-password-hashed = "*"` installs cleanly (§1, three scenarios). | Rendered `root-password-hashed` is exactly `*`, **and no `.rootpw.txt` is emitted**. Two one-line assertions. |
| 4 | **Any per-box password is set at `before-network`** — the only ordering with a measured zero window (§4.3). | Rendered `[first-boot].ordering` is `before-network`, **and** the hook's positive marker is observed in a boot test (§4.4, R-150). |
| 5 | **No `root-ssh-keys`, ever, in a public image** (R-142). | `grep -c root-ssh-keys` on the rendered answer is `0`. |
| 6 | **The profile must be committed** — demo-felhom's baked key comes from an *uncommitted* profile, so a reference box is not reproducible from `main`. | `--profile` resolves inside the repo tree **and** `git status --porcelain` is empty for it. **R-151.** |
| 7 | **The artefact must be reproducible** (R-144: `nested-probe`'s profile is gone). | Manifest records the committed profile path + repo commit; both resolve. |
| 8 | **The recovery path must be documented, and named operator-only if it is.** §2.3 + §3.2 + §3.3: with a locked root the only way in is `init=/bin/bash` at the console. | Release notes state it, in those terms. |
**What the image must not contain:** a usable root password hash, any `root-ssh-keys`, any customer
id, claim code, retrieval passphrase or hub token, and any `filter.*` disk selector in the public
variant. Spike 1 §6 established by enumeration that the only secret-bearing file is `/answer.toml`, so
gates 3 and 5 over that one rendered file cover the credential surface.
**What the image must not contain:** a usable root password hash, any `root-ssh-keys`, any customer id,
claim code, retrieval passphrase or hub token, and any `filter.*` selector in the public variant.
Spike 1 §6 established by enumeration that `/answer.toml` is the only secret-bearing file, so gates 3
and 5 over that one rendered file cover the credential surface.
---
## 7. The operator's decisions
Two, each with the measurement that informs it and its current state.
**Decision 1 — escrow or discard a per-box root password.** §3.4 reshapes this twice over:
**Decision 1 — escrow or discard a per-box root password.**
- **Discard and lock are the same thing** for recovery — if nobody holds the password, both `sulogin`
and the web UI refuse.
- **Neither makes a box unrecoverable.** `init=/bin/bash` works in every case (§3.3), so an operator at
the console can always get in. The earlier fear that locking root leaves "no path back at all" is
**measured to be false**.
§3.4 changed the shape of this question. **"Discard" and "lock" are operationally identical** — if
nobody holds the password, `sulogin` refuses either way, and console recovery is closed to everyone.
So the choice is binary:
So what escrow actually buys is narrower than it looked: **the operator keeps the web-UI route (§2.3)
and the `sulogin` route (§3.2) without needing physical presence.** Against that stands D6 — a hub
holding console root for the whole fleet is exactly what the hub alone must not be.
- **Escrow** — the operator can always reach a box's console; the cost is a hub-held credential for
every box, which is the thing the D6 trust model says the hub alone must not be.
- **Discard / lock** — no credential exists anywhere; the cost is that a failed pairing leaves a
machine recoverable only by an operator at the keyboard (`init=/bin/bash`) or a reinstall.
**Recommendation, offered because the measurements now support one:** discard/lock for the **public**
image, and accept operator-only physical recovery, because the credential that would be escrowed is
the same one whose fleet-wide sharing started this whole line of work. Escrow remains defensible for
**operator-built** boxes, where the operator installed the machine anyway. The decision is still the
operator's; it is no longer blocked on a measurement.
**Blocking measurement: §2.3** — whether a locked root also closes the PVE web interface. If it does,
"discard" removes the last remote route as well as the local one, and the balance shifts sharply
toward escrow. **Do not rule until §2.3 is measured.**
**Decision 2 — the default boot entry for a public image.**
Framed in §5.3. The disk evidence points at interactive-default; the product cost is the unattended
experience. **Partially blocked on the same §2.3 measurement**, because an unreachable-after-failure
box strengthens the interactive case independently. The operator-built-versus-public split (§5.3) is
available regardless and is cheap.
**Decision 2 — the default boot entry for a public image.** Framed in §5.3, unruled. The disk evidence
and the recovery evidence now point the same way (interactive), and the cost is the unattended
experience. The operator-built-versus-public split is cheap and available either way.
---
@@ -387,70 +436,65 @@ available regardless and is cheap.
| # | Unknown | What it takes |
|---|---|---|
| 1 | **Everything in §2.1, §2.3, §2.4, §3.1, §4.1, §4.2 (entropy/binaries).** | The staged fixture, unchanged, once a Tier 0 box returns. ~90 min for all three scenarios. |
| 2 | **Whether PVE alters the §2.2/§3.2 mechanism results** — PAM config, a `sulogin` override, `SYSTEMD_SULOGIN_FORCE`. | Same fixture; check `/etc/pam.d/`, the `sulogin` invocation, and systemd's emergency unit on the installed image. |
| 3 | **Whether the multi-match pick is deterministic by enumeration order** (Spike 1 §8 #1, still open). | Re-run Spike 1's S1 with reversed disk order. ~15 min. |
| 4 | **Whether the installer excludes its own boot medium** (Spike 1 §8 #2, still open). | USB boot on real hardware — HUMAN. |
| 5 | **What `network-online` ordering buys** (§4.3) — it is a real third option nobody has evaluated. | One extra scenario alongside the staged two. |
| 6 | **Whether a second GRUB entry actually boots the stock installer** after the gates are relaxed. | Build a probe ISO with a two-entry template and boot it. Not attempted — it would have meant changing repo files, which §4 forbids. |
| 1 | **Felhom's real stub at `before-network`** (§4.4) — coexistence with pairing is structural, not observed. The single remaining gap in Q3. | Build a probe ISO with `stub-first-boot.sh` as the hook and `ordering = "before-network"`, boot, confirm `felhom-bootstrap.service` still starts and pairing completes. ~40 min. **Do this before the spec.** |
| 2 | **Whether root SSH is refused *because* of the lock** (§2.2) — the probe used a wrong password in the control too, so it discriminates nothing. | Re-run the SSH leg with the control's correct password. ~10 min. |
| 3 | **What else in PVE assumes a usable root credential** (§2.4) — cron, backup jobs. | Inspect a locked install over a longer uptime. |
| 4 | **What `network-online` ordering buys** (§4.3) — a real third option, never evaluated. | One extra scenario. |
| 5 | **Whether the multi-match pick is deterministic by enumeration order** (Spike 1, still open). | Re-run Spike 1's S1 with reversed disk order. ~15 min. |
| 6 | **Whether the installer excludes its own boot medium** (Spike 1, still open). | USB boot on real hardware — HUMAN. |
| 7 | **Whether a relaxed-gate two-entry menu actually boots the stock installer** (§5.1). | Not attempted — it would mean changing repo files, which §4 forbids. |
---
## 9. R-rows opened
Numbering continues from R-147 (Spike 1).
Continuing from R-147 (Spike 1).
| R | Severity | Finding |
|---|---|---|
| **R-148** | MEDIUM | **`answer.toml.tmpl:27` justifies `ordering = "fully-up"` with a dependency the stub does not have.** The comment says the stub must run after `pveproxy` "so pvesh/pct work", but `stub-first-boot.sh` contains no `pvesh`/`pct`/`pveum`/`qm` call (grep `rc=1`); that usage lives in `felhom-bootstrap.sh`, which runs under its own `After=network-online.target … pveproxy.service` unit. A comment asserting a **dependency** rather than an invariant — and it has constrained the design, because `before-network` is the ordering that would make the credential window zero (§4.3). |
| **R-149** | LOW | **The `[first-boot].ordering` enum has three values, not two.** `before-network`, `network-online`, `fully-up` all validate; `network-pre` and `multi-user` are rejected. Spike 1 recorded two and the middle option has never been evaluated. |
| **R-150** | MEDIUM | **A first-boot hook that never runs is indistinguishable from one that succeeded.** All four `proxmox-first-boot-*.service` units are guarded by `ConditionPathExists=` / `ConditionPathIsReadWrite=`, and in systemd a failed condition is a **skip, not a failure** — no error, successful boot. Any design that puts credential rotation in a first-boot hook must verify it by a positive marker the hook emitted. Directly the "an absent log line is not evidence" class. |
| **R-151** | MEDIUM | **A reference box is not reproducible from the repository.** `demo-felhom` was installed from an ISO built with the **uncommitted** `n100-demo.profile`, which is where its baked SSH key comes from (Spike 1 §4.6). One of the two Tier 0 boxes therefore cannot be rebuilt from `main`. Related to R-144 (`nested-probe`'s profile is gone entirely); this row is the *reference box*, which matters more. Gate 6 in §6 exists to stop a public image inheriting the shape. |
| **R-148** | MEDIUM | **`answer.toml.tmpl:27` justifies `ordering = "fully-up"` with a dependency the stub does not have.** `stub-first-boot.sh` contains no `pvesh`/`pct`/`pveum`/`qm` call (grep `rc=1`); that usage is in `felhom-bootstrap.sh`, under its own `After=network-online.target … pveproxy.service` unit. A comment asserting a **dependency** rather than an invariant — and it blocked the ordering that §4.3 now measures as giving a zero credential window. |
| **R-149** | LOW | **`[first-boot].ordering` has three values, not two** — `before-network`, `network-online`, `fully-up` validate; `network-pre` and `multi-user` are rejected. The middle option has never been evaluated. |
| **R-150** | MEDIUM | **A first-boot hook that never runs is indistinguishable from one that succeeded.** All four `proxmox-first-boot-*.service` units are `Condition*`-guarded, and a failed condition is a systemd **skip, not a failure** — no error, successful boot. Any credential-rotation-in-first-boot design must be verified by a positive marker the hook emitted. |
| **R-151** | MEDIUM | **A reference box is not reproducible from the repository.** `demo-felhom` was installed from an ISO built with the **uncommitted** `n100-demo.profile`, which is where its baked SSH key comes from (Spike 1 §4.6). Gate 6 in §6 exists to stop a public image inheriting that shape. |
| **R-152** | LOW | **A locked root is not a physical-security measure, and should never be described as one.** The installed PVE GRUB has **no password** (§3.1), so `init=/bin/bash` yields an unauthenticated root shell in under a minute (§3.3, measured). Locking root defends against *remote* reuse of a shared ISO credential and nothing else. If a physical-access threat model is ever wanted, it needs a GRUB password, which is a separate decision with its own lockout risk. |
---
## 10. Teardown
All three layers, verified positively.
All three layers, each verified positively.
**Layer 1 — demo-hp.** **INCOMPLETE, and named as such.** The box went offline (§0) while
`/mnt/nvme-1tb/spike2/` held the probe ISO (~1.7 GB) and the staged answer files and runners. Nothing
was running on it: the last successful command before the outage confirmed `0` QEMU processes, and no
install had been started this session. Nothing was registered as a PVE storage, nothing was written to
`local-lvm`, and `drill-r50` / guest 9201 were never touched this session.
**Layer 1 — demo-hp.**
**Outstanding cleanup, to run when the box returns** — recorded here because R-131 exists precisely
because recorded commands never get run:
| Check | Result |
|---|---|
| QEMU processes | `0` |
| `/mnt/nvme-1tb/spike2/` | removed — `No such file or directory` |
| `/tmp/paia` | removed |
| loop devices / `/mnt/s2ans` | `0` / removed |
| `/mnt/nvme-1tb/` contents | `dump e2d-images felhom-data images lost+found` — the pre-spike set |
| disk usage | **6.6 G — identical to pre-spike** |
| `drill-r50` (VM 300) | `stopped`, untouched (R-93 fixture intact) |
| guest 9201 | `running`, untouched |
| `/etc/pve/storage.cfg` | **4 storages, unchanged**; no scratch storage ever registered |
| `local-lvm` | never written |
```
rm -rf /mnt/nvme-1tb/spike2 ; rm -f /tmp/paia ; pkill -9 qemu-system-x86
```
This is deliberately *not* claimed as done. Spike 1's `/mnt/nvme-1tb/spike-scratch` and both `/tmp/paia`
copies were removed and verified at that session's teardown, so the outstanding item is this session's
`spike2/` directory alone.
**Layer 2 — demo-felhom.** Nothing placed this session — it was never reached before the outage.
Spike 1's `/tmp/paia` was removed and verified then. No action outstanding.
**Layer 2 — demo-felhom.** `/tmp/paia` removed; guest 9201 `running`; the external USB drive
`931.5G usb /mnt/hdd_1` still attached and mounted exactly as found — never attached, detached,
partitioned or written. Only `device-info` reads were ever performed there.
**Layer 3 — DooPlex.**
| Check | Result |
|---|---|
| scratchpad | **4.4 GB → 48 K** (probe ISO, squashfs extracts, `.deb` extract, schema test dirs removed) |
| control-password file (`s2/.ctlpw`) | `shred -u`'d — never printed, never committed; verified absent |
| mechanism-probe container | `docker run --rm` — no image or container retained; **no global Docker cleanup run** (fence) |
| scratchpad | **1.7 GB → 60 K** (probe ISO, screendumps, squashfs/`.deb` extracts, schema test dirs) |
| control password (`s2/.ctlpw`) and `scen/C.toml` | `shred -u`'d / removed; verified absent. Never printed, never committed |
| mechanism-probe container | `docker run --rm`; **no global Docker cleanup run** (fence) |
| `felhom-iso/out/` | untouched, **17 ISOs** |
| production files | none modified — no `build-felhom-iso.sh`, no repack, no profile, no `felhom-bootstrap.sh` |
| production files | none modified — no `build-felhom-iso.sh`, no `iso-repack.sh`, no profile, no `felhom-bootstrap.sh` |
**One teardown error, recorded rather than quietly fixed.** Shredding the control password
(`s2/.ctlpw`) destroyed the plaintext that scenario **C** existed to use — a `$6$` hash whose plaintext
is gone cannot demonstrate "correct password → PVE ticket", which was C's entire purpose as the
control for §2.3. `s2/scen/C.toml` was therefore **deleted** rather than left in place looking usable.
**Scenario C must be regenerated** (one `openssl passwd -6` against a fresh random plaintext, kept
0600) before the §2.3 measurement is attempted. Scenarios A and B are unaffected and remain staged.
**Hub-side disposition — nothing to dispose.** No customer, host or appliance was created: no install
ran, and the only hub contact this session was none at all. Verified rather than asserted below.
**Hub-side disposition — nothing to dispose.** No customer, host or appliance was created: the nested
VMs ran under plain QEMU with user-mode networking and no route to the hub, and no install ever
contacted it. **R-131 gains no row.**
**Published artefacts: none.** Nothing went to R2.
scripts/CHANGELOG.md
+27 -29
View File
@@ -1,38 +1,36 @@
## SPIKE 2 (no code change) — before-network gives a zero window by construction; locked root closes sulogin (2026-07-31)
## SPIKE 2 (no code change) — locked root closes the PVE web UI; before-network gives a measured zero window (2026-07-31)
**No script, profile or build file was modified.** Findings:
`documentation/audits/SPIKE-universal-iso-2-2026-07-31.md`. Opens R-148..R-151.
`documentation/audits/SPIKE-universal-iso-2-2026-07-31.md`. Opens R-148..R-152.
**Both Tier 0 boxes went offline mid-session** (remote site, 12:28 CEST; four routes tried, our
tailscale pod healthy). Q1/Q2/Q3 each retain a part that needs a nested VM and those parts are
**BLOCKED, not answered**. The fixture is staged with Spike 1's two defects already corrected.
Both Tier 0 boxes went offline mid-session (provider cable fault; four routes tried) and returned;
all three scenarios then ran to completion on real PVE, each signalled by `reboot-mode="power-off"`.
Established without them:
- **STRUCTURAL — `ordering = "before-network"` maps to `proxmox-first-boot-network-pre.service`
(`proxmox-first-boot` 9.2.5), which is `Before=network-pre.target` + `Type=oneshot`.** It therefore
runs to completion before any interface is configured — a credential rotation there has a
**zero-length window by construction**, not by being fast. Stronger than the timing number Spike 1
failed twice to obtain.
- **STRUCTURAL — the stub does not need `fully-up`.** `stub-first-boot.sh` contains no
`pvesh`/`pct`/`pveum`/`qm` call (grep rc=1); that usage is in `felhom-bootstrap.sh`, which runs under
its own `After=network-online.target ... pveproxy.service` unit. So `answer.toml.tmpl:27` justifies
the current ordering with a dependency that does not exist. **R-148.**
- **The ordering enum has THREE values** — `before-network`, `network-online`, `fully-up`. Spike 1
recorded two; the middle one has never been evaluated. **R-149.**
- **MECHANISM (container, not PVE) — a locked root closes `sulogin`:** `Cannot open access to console,
the root account is locked.` for both `*` and `!`, with a working control (correct password ->
SU_OK, wrong -> Authentication failure). So "discard" and "lock" are the SAME outcome for recovery,
which makes the escrow decision binary.
- **All four `proxmox-first-boot-*` units are `Condition*`-guarded, and a failed condition is a SKIP,
not a failure** — a hook that never ran looks exactly like one that succeeded. **R-150.**
- Q4: the single-entry menu is enforced by **four** gates in `iso-repack.sh` (`:153-155`, `:156`,
`:160-164`, `:314-319`), so restoring an interactive entry is a gate change, not a template edit —
but the banned-token gate would pass unchanged for a stock Graphical entry. `default`/`timeout` are
- **A locked root CLOSES the PVE web interface.** Measured against the exact endpoint the UI logs in
through (`POST /api2/json/access/ticket`, `root@pam`) **with a working control**: the known-password
install returns **HTTP 200 + ticket**; the locked install returns 401 for every password, and no
password exists. `passwd -S root` = `L`, shadow = `literal-asterisk`. PVE uses the stock PAM stack
(`pve_pam_realm: 0`).
- **GRUB recovery mode is ALSO closed**: `Cannot open access to console, the root account is locked.`
— the container mechanism result confirmed verbatim on real PVE.
- **But `init=/bin/bash` still works** — an unauthenticated `root@(none):/#`. So a locked-root box is
recoverable, operator-only, at the physical console. The installed GRUB has **no password**, so
locking root is NOT a physical-security measure. **R-152.**
- **`before-network` measured, A/B, same image:** the hook runs (positive marker, uptime 6.58s) with
entropy 256, writable /etc, all binaries and `openssl_rand_len=32`, while `ip_global` is EMPTY and
`listen_22_8006` is **0**. The `fully-up` column is the converse proof: sshd + pveproxy already
active, 3 listening. Zero credential window, measured.
- **R-148:** `answer.toml.tmpl:27` justifies `fully-up` with a `pvesh`/`pct` dependency the stub does
not have (grep rc=1) — it blocked the ordering that is now measured as the fix.
- **R-149** three ordering values, not two · **R-150** Condition-guarded hooks skip silently ·
**R-151** demo-felhom built from an uncommitted profile.
- Q4 from source: four gates in `iso-repack.sh` enforce the single-entry menu; `default`/`timeout`
already settable in `grub/grub.cfg.tmpl`.
The first mechanism probe was invalid (ran as uid 0, so `su` never invoked pam_unix; `sulogin` had no
tty) and is recorded as failed, not massaged.
Three probes failed and are recorded as failed, not massaged: a container probe that ran as uid 0
(so `su` never invoked pam_unix), a GRUB probe that missed the **1-second** menu timeout and typed into
the login prompt, and a kernel-line edit that landed one line off — caught by a pre-typing
verification screendump.
## SPIKE (no code change) — a universal ISO needs a different disk strategy and a locked root (2026-07-31)