admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

SPIKE 2 complete: locked root closes the PVE web UI; before-network gives a measured zero window

Browse Source 
Findings only — no script, profile or build file changed; no ISO built, nothing published.
documentation/audits/SPIKE-universal-iso-2-2026-07-31.md

Both Tier 0 boxes went offline mid-session (provider cable fault; four routes tried, no Tier 2
fallback used) and returned. All three scenarios then ran to completion on real PVE, each signalled
by reboot-mode='power-off' rather than a disk hash.

- A LOCKED ROOT CLOSES THE PVE WEB INTERFACE. Measured at the exact endpoint the UI uses
  (POST /api2/json/access/ticket, root@pam) WITH A WORKING CONTROL: known-password install returns
  HTTP 200 + ticket; locked install returns 401 for every password and none can exist.
  passwd -S root = L, shadow = literal-asterisk, PVE uses the stock PAM stack.
- GRUB recovery mode is also closed ('the root account is locked') — but init=/bin/bash still gives
  an unauthenticated root@(none):/#. A locked box is recoverable, operator-only, at the console.
  The installed GRUB has NO password, so locking root is not a physical-security measure. R-152.
- before-network MEASURED (A/B, same image): the hook RUNS (marker, uptime 6.58s) with entropy 256,
  writable /etc, all binaries and openssl_rand_len=32, while ip_global is EMPTY and
  listen_22_8006 = 0. fully-up is the converse: sshd+pveproxy active, 3 listening. Zero window.
- R-148: answer.toml.tmpl:27 justifies fully-up with a pvesh/pct dependency the stub does not have
  (grep rc=1) — it blocked the ordering now measured as the fix.
- R-149 three ordering values; R-150 Condition-guarded hooks skip silently; R-151 demo-felhom built
  from an uncommitted profile.

Three probes failed and are recorded as failed: a container probe that ran as uid 0, a GRUB probe
that missed the 1-second menu timeout, and a kernel-line edit one line off (caught by a pre-typing
verification screendump). The interim 'Layer 1 teardown INCOMPLETE' is corrected — the fixture had
never landed, because the staging mkdir was in the SSH call that timed out.
This commit is contained in:
authentik Default Admin
2026-07-31 13:57:21 +02:00
parent 5bdd8372f8
commit 19c932a693
3 changed files with 424 additions and 388 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT-universal-iso-spike.md
+63 -69
View File
@@ -1,95 +1,89 @@
# REPORT — SPIKE 2: closing the last mechanisms, and the universal ISO's shape (2026-07-31) # REPORT — SPIKE 2: closing the last mechanisms, and the universal ISO's shape (2026-07-31)
> Written as `REPORT-universal-iso-spike.md`, not `REPORT.md`: the shared file belongs to today's hub > Written as `REPORT-universal-iso-spike.md`, not `REPORT.md`: the shared file belongs to today's hub
> v0.85.0 session and the standing rule is that the second session in a shared clone never touches it. > v0.85.0 session and the second session in a shared clone never touches it.
> This file supersedes its own Spike 1 contents.
**Class: Spike.** Findings only — no production file changed, no ISO built, nothing published. **Class: Spike.** Findings only — no production file changed, no ISO built, nothing published.
Evidence: `documentation/audits/SPIKE-universal-iso-2-2026-07-31.md`. Continues Evidence: `documentation/audits/SPIKE-universal-iso-2-2026-07-31.md`.
`SPIKE-universal-iso-2026-07-31.md`.
## The headline: the venue went away, and three answers are blocked **All four questions are answered**, and every answer that matters is measured on a real PVE install.
The session lost both Tier 0 boxes mid-run (provider cable fault; four routes tried, no Tier 2
fallback used) and resumed when connectivity returned.
At 12:28 CEST both Tier 0 boxes went offline **simultaneously** and did not return. Four routes were ## The two results that decide the product questions
tried (tailnet direct, `ssh demo-hp`, `demo-hp-lan` via ProxyJump, `ssh felhom-pve`); our tailscale
pod is healthy and `tailscale status` shows both peers `offline` with a rising `last seen`. Both boxes
are at the same remote site — that site's power or uplink, not fixable from here.
**No fallback venue was used.** DooPlex is Tier 2, the 2026-07-25 ruling moved drill VMs off it, and **A locked root closes the PVE web interface.** Measured against the exact endpoint the UI logs in
this task did not authorise it — an absent fence is not permission. So **Q1, Q2 and Q3 each keep a through — `POST /api2/json/access/ticket`, realm `root@pam` — **with a working control**: the
part that is BLOCKED, not answered**, and they are labelled that way throughout. The fixture is built known-password install returns **HTTP 200 and a ticket**; the locked install returns 401 for every
and staged with Spike 1's two measurement defects already corrected (a NIC present at install time; a password offered, and no password exists that could work. The control is what makes this a finding
`reboot-mode = "power-off"` completion signal instead of a disk hash). rather than a broken probe. `passwd -S root` reports `L`, the shadow field is `literal-asterisk`, and
PVE uses the stock PAM stack.
## What was established anyway **GRUB's recovery mode is closed too** — `Cannot open access to console, the root account is locked.`
— **but `init=/bin/bash` still works**, giving an unauthenticated `root@(none):/#`. So a locked-root
box is *not* unrecoverable, as the structural reading had feared; it is recoverable **operator-only, at
the physical console**. The installed GRUB has no password, which also means locking root is not a
physical-security measure and must never be described as one (**R-152**).
**The best result of the session, and it is structural rather than measured — which makes it ## The mechanism that fixes the credential window
stronger.** `ordering = "before-network"` maps to `proxmox-first-boot-network-pre.service`
(`proxmox-first-boot` 9.2.5, read out of the ISO's own package), which is `Before=network-pre.target` **`before-network` measured, A/B, same image, only `ordering` differing.** The hook **runs** (positive
with `Type=oneshot`. It therefore runs **to completion before any network interface is configured**. marker, uptime 6.58 s) with entropy 256, a writable `/etc`, every binary it needs and a demonstrated
A credential rotation placed there has a **zero-length exposure window by construction, not by being `openssl_rand_len=32` — while `ip_global` is **empty** and `listen_22_8006` is **0**. The `fully-up`
fast** — and that holds on every machine, unlike the timing number Spike 1 failed twice to obtain. column is the converse proof: sshd and pveproxy already active, 3 sockets listening. **Zero credential
window, measured**, and the shipped unit (`Before=network-pre.target`, `Type=oneshot`) generalises it
to every machine.
**And the reason it was never used is a comment that is wrong.** `answer.toml.tmpl:27` justifies **And the reason it was never used is a comment that is wrong.** `answer.toml.tmpl:27` justifies
`fully-up` as needed "so pvesh/pct work" — but `stub-first-boot.sh` makes no `pvesh`/`pct`/`pveum`/`qm` `fully-up` as needed "so pvesh/pct work", but `stub-first-boot.sh` makes no `pvesh`/`pct`/`pveum`/`qm`
call at all (grep `rc=1`). That usage lives in `felhom-bootstrap.sh`, which runs under its own call at all (grep `rc=1`); that usage lives in `felhom-bootstrap.sh`, under its own properly-ordered
`After=network-online.target … pveproxy.service` unit. A comment asserting a **dependency** that does unit. **R-148** — a comment asserting a *dependency* that does not exist, which blocked the fix.
not exist, which has constrained the design ever since. **R-148.**
**A locked root closes `sulogin`** — measured in an isolated container with a working control (correct ## Probes that failed, recorded as failed
password → `SU_OK`, wrong → `Authentication failure`; then `*` and `!` both → `L` and
`Cannot open access to console, the root account is locked.`). This reshapes the escrow question: if
nobody holds the password, `sulogin` refuses either way, so **"discard" and "lock" are the same
outcome for recovery**, and the decision is binary. Caveat stated plainly: container, not PVE.
**The ordering enum has three values, not two** — `before-network`, `network-online`, `fully-up`. Three, none massaged: a container probe that ran as uid 0 so `su` never invoked `pam_unix` (rc=0 even
Spike 1 recorded two; the middle one has never been evaluated (**R-149**). with a wrong password); a GRUB probe that missed the **1-second** menu timeout and typed into the login
prompt; and a kernel-line edit that landed one line off — caught by a pre-typing verification
screendump before it was booted. All were rerun.
**A first-boot hook that never runs is indistinguishable from one that succeeded** — all four Also corrected: the interim report's "Layer 1 teardown INCOMPLETE" was over-cautious. When demo-hp
`proxmox-first-boot-*` units are `Condition*`-guarded and a failed condition is a *skip*, not a returned, `/mnt/nvme-1tb/spike2/` did not exist and disk usage was unchanged — the staging `mkdir` had
failure. Any rotation-in-first-boot design must verify by a positive marker (**R-150**). been inside the SSH call that timed out as the link degraded, so **the fixture never landed**.
**Q4 answered fully from source.** The single-entry menu is enforced by **four** gates in ## The operator's decisions
`iso-repack.sh` (`:153-155`, `:156`, `:160-164`, and a post-remaster re-extraction at `:314-319`), so
restoring an interactive entry is a gate change rather than a template edit — but the banned-token
gate would pass unchanged for a stock *Graphical* entry, so the safety property survives. `default`
and `timeout` are already settable in `grub/grub.cfg.tmpl`.
## What failed, and is recorded as failed **Escrow versus discard — no longer blocked, and reshaped twice.** "Discard" and "lock" are the same
outcome (nobody holds the password, so both `sulogin` and the web UI refuse), and **neither makes a box
unrecoverable** because `init=/bin/bash` always works. So escrow buys only this: the operator keeps the
web-UI and `sulogin` routes without physical presence. Against that stands D6 — a hub holding console
root for the whole fleet is what the hub alone must not be. **I recommend discard/lock for the public
image**, escrow remaining defensible for operator-built boxes. Your call; it is no longer waiting on a
measurement.
The first mechanism probe produced **zero valid observations**: it ran everything as uid 0, so `su` **The default boot entry — framed, unruled.** The disk evidence (Spike 1) and the recovery evidence
never invoked `pam_unix` (rc=0 even with a wrong password), `sulogin` died on `tcgetattr` with no tty, (this session) now point the same way: a tester whose unattended install succeeds but whose pairing
and a `sed` mangled the control hash. Rerun with a non-root caller, a real pty, and `chpasswd`. fails has no web UI and no recovery shell. The cost is the unattended experience. Two profiles from one
build is cheap, so operator-built and public need not match.
A **teardown error** is also recorded rather than quietly fixed: shredding the control password ## Still open
destroyed the plaintext scenario C existed to use, so `C.toml` was deleted rather than left looking
usable. It must be regenerated before the PVE-web-interface measurement is attempted.
## The two decisions, and why neither should be ruled yet One gap in Q3: **Felhom's real stub at `before-network`** is argued structurally, not observed — the
scenarios used the spike's own diagnostic as the hook. That is the one thing to measure before a spec
**Escrow versus discard** — now binary (see above). **Blocked on one measurement:** whether a locked is written (~40 min). Also: the SSH-as-root leg used a wrong password in the control too, so it
root also closes the PVE web interface. If it does, discarding removes the last remote route as well discriminates nothing and is recorded as a gap rather than a result.
as the local one and the balance shifts sharply toward escrow. **Do not rule before that is measured.**
**The default boot entry for a public image** — the disk evidence points at interactive-default; the
cost is the unattended experience. Partially blocked on the same measurement. The
operator-built-versus-public split is cheap and available regardless of the ruling.
## R-rows ## R-rows
**R-148** (`fully-up` justified by a non-existent dependency) · **R-149** (three ordering values, not **R-148** (`fully-up` justified by a non-existent dependency) · **R-149** (three ordering values) ·
two) · **R-150** (Condition-guarded first-boot hook skips silently) · **R-151** (demo-felhom was **R-150** (Condition-guarded hooks skip silently) · **R-151** (demo-felhom built from an uncommitted
installed from an **uncommitted** profile — a Tier 0 reference box is not reproducible from `main`). profile) · **R-152** (a locked root is not a physical-security measure).
## Teardown ## Teardown
**Layer 1 (demo-hp) is INCOMPLETE and named as such** — the box went offline while All three layers verified positively. demo-hp is back to its exact pre-spike state: `spike2/` and
`/mnt/nvme-1tb/spike2/` held the staged fixture. Nothing was running on it (last check: 0 QEMU `/tmp/paia` removed, 0 QEMU processes, 0 loop devices, **disk usage 6.6 G — identical to pre-spike**,
processes, no install started this session), nothing on `local-lvm`, no PVE storage registered, 4 storages unchanged, nothing on `local-lvm`, `drill-r50` stopped and 9201 running, both untouched.
`drill-r50` and 9201 never touched. The outstanding command is recorded in §10 of the findings doc demo-felhom: probe binary removed, external USB drive observed only, still mounted as found. DooPlex:
and **is not claimed as done** — R-131 exists because recorded commands never get run. scratchpad 1.7 GB → 60 K, control password `shred -u`'d and verified absent, `felhom-iso/out/`
untouched at 17 ISOs, no production file modified. **Hub-side: nothing created** — the VMs had no route
Layer 2 (demo-felhom): nothing placed this session. Layer 3 (DooPlex): scratchpad 4.4 GB → 48 K, to it. Nothing published.
control password `shred -u`'d and verified absent, mechanism container `--rm`, no global Docker
cleanup, `felhom-iso/out/` untouched at 17 ISOs, no production file modified. **Hub-side: nothing
created** — verified by fetching the customer list and searching it, not asserted. Nothing published.
documentation/audits/SPIKE-universal-iso-2-2026-07-31.md
+334 -290
View File
@@ -4,201 +4,290 @@
> >
> Continues `SPIKE-universal-iso-2026-07-31.md`; its results are premises here and are not re-derived. > Continues `SPIKE-universal-iso-2026-07-31.md`; its results are premises here and are not re-derived.
> >
> **Read §0 first: the venue went away mid-session.** Both Tier 0 hosts dropped off the tailnet at > **All four questions are answered, and every answer that mattered is measured on a real PVE install.**
> ~12:28 CEST and did not return. Q1, Q2 and Q3 each have a part that needs a nested VM, and those > The session lost its venue mid-run (§0) and resumed when connectivity was restored.
> parts are **BLOCKED, not answered**. What could be established without them — from the shipped PVE
> source, the answer schema, and an isolated mechanism probe — is below and is labelled by how it was
> obtained.
--- ---
## 0. Venue loss — what was tried ## 0. Venue loss and recovery — what was tried
At 12:28 CEST, mid-run, both Tier 0 boxes went offline **simultaneously**: At 12:28 CEST both Tier 0 boxes went offline **simultaneously**. Routes attempted, in order:
```
100.76.96.79 demo-hp active; relay "fra"; offline, last seen 5m ago, tx 1843821564 rx 63739848
100.70.170.35 felhom-pve active; relay "waw"; offline, last seen 5m ago, tx 1560 rx 0
```
Attempts, in order (per the standing rule that a "no access" claim names what it tried):
| # | Route | Result | | # | Route | Result |
|---|---|---| |---|---|---|
| 1 | `ping 100.76.96.79` (tailnet direct) | 100 % packet loss | | 1 | `ping 100.76.96.79` (tailnet direct) | 100 % loss |
| 2 | `ssh demo-hp` (tailnet, `ConnectTimeout=10`) | `connect to host 100.76.96.79 port 22: Connection timed out` | | 2 | `ssh demo-hp` | `Connection timed out` |
| 3 | `ssh demo-hp-lan` (LAN `192.168.0.87`, ProxyJump `felhom-pve`) | `Connection timed out during banner exchange` | | 3 | `ssh demo-hp-lan` (LAN, ProxyJump `felhom-pve`) | `Connection timed out during banner exchange` |
| 4 | `ssh felhom-pve` (the jump host itself) | `connect to host 100.70.170.35 port 22: Connection timed out` | | 4 | `ssh felhom-pve` (the jump host) | `Connection timed out` |
| 5 | DooPlex tailscale pod health | pod `tailscale-c84df7f84-tw2wz` **Running 6d10h**, healthy; log shows `open-conn-track: timeout … online=yes` then peers dropping | | 5 | DooPlex tailscale pod | `Running 6d10h`, healthy |
| 6 | `tailscale status` from the pod | **both peers `offline`**, `last seen 5m` and rising | | 6 | `tailscale status` from the pod | **both peers `offline`**, `last seen` rising |
**Diagnosis: the far end, not ours.** Our tailscale pod is healthy and every other peer route behaves; Far end, not ours. **No fallback venue was used** — DooPlex is Tier 2, the 2026-07-25 ruling moved
both demo boxes are at the same remote site (REMOTE until ~08-02) and went offline together, which is drill VMs off it, and this task did not authorise it. An absent fence is not permission.
that site's power or uplink. Not fixable from here.
**No fallback venue was used.** DooPlex is Tier 2 and the 2026-07-25 standing ruling moved drill and Connectivity was restored (operator: a provider cable fault nearby) and the boxes returned with
build VMs off it; `target-selection.md` says a Tier 2 machine is used only when a task says so **uptime 7 days** — a pure network outage, no reboot. All probes then ran.
explicitly, and this one does not. **An absent fence is not permission**, so the probes simply did not
run. A watch is armed for the boxes' return.
The probe fixture is built and staged and will run unchanged when they come back: probe ISO **One thing the outage revealed.** `/mnt/nvme-1tb/spike2/` did not exist when demo-hp came back, and
(`prepare-iso --fetch-from partition`, serial-capture first-boot diagnostic), three validated answer disk usage was unchanged at 6.6 GB: the staging `mkdir` had been inside the SSH call that timed out as
files (locked/`fully-up`, locked/`before-network`, known-password control), an install runner with the link degraded, so **the fixture never landed and Layer 1 had nothing to clean**. The interim
`reboot-mode = "power-off"` as the completion signal, and a boot runner with protocol-level detectors report's "Layer 1 teardown INCOMPLETE" was therefore over-cautious rather than wrong — corrected here,
and **a NIC present at install time** — the two fixture defects that invalidated Spike 1's and the fixture was rebuilt from scratch.
measurements are already corrected.
--- ---
## 1. Baselines ## 1. Baselines
Read fresh this session.
| Baseline | Value | | Baseline | Value |
|---|---| |---|---|
| `ISO_VERSION` | `1.25.0` — `scripts/iso/build-felhom-iso.sh:44` | | `ISO_VERSION` | `1.25.0` — `scripts/iso/build-felhom-iso.sh:44` |
| `SCRIPT_VERSION` | `1.22.0` — `scripts/felhom-host-install.sh:187` | | `SCRIPT_VERSION` | `1.22.0` — `scripts/felhom-host-install.sh:187` |
| `felhom-bootstrap.sh` @ HEAD | `21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7` | | `felhom-bootstrap.sh` @ HEAD | `21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7` |
| PVE base | `proxmox-ve_9.2-1.iso`, `4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c` | | PVE base | `proxmox-ve_9.2-1.iso`, `4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c` |
| `felhom.eu` HEAD | **`ea00976`**, clean, `== origin/main` (the expected Spike 1 commit) | | `felhom.eu` HEAD | **`ea00976`**, clean, `== origin/main` |
| `proxmox-first-boot` pkg | `9.2.5` (from the ISO's `/proxmox/packages/`) | | `proxmox-first-boot` pkg | `9.2.5` (from the ISO's `/proxmox/packages/`) |
| installed guest kernel | `7.0.2-6-pve`, GRUB `2.12-9+pmx2` |
**Fixture provenance.** Probe ISO built on DooPlex from the baseline PVE ISO. Nested VMs were to run **Fixture provenance.** Probe ISO built on DooPlex from the baseline PVE ISO
under plain QEMU/KVM on demo-hp under `/mnt/nvme-1tb/spike2/` — never `local-lvm`, no PVE storage (`prepare-iso --fetch-from partition --partition-label proxmox-ais --on-first-boot diag.sh`),
registered. They did not run (§0). The mechanism probe in §2.2 ran in a throwaway `debian:trixie` sha256 `dd97dba5518c70baf4300b119e1042eabf1012d5a046082acf31e60983efaeb9`, **verified byte-identical at
Docker container on DooPlex, which is not an install and touches no box. both ends** after transfer. Nested VMs ran under plain QEMU/KVM on **demo-hp** (Tier 0, the designated
drill host) under `/mnt/nvme-1tb/spike2/` — never `local-lvm`, no PVE storage registered, `drill-r50`
and guest 9201 untouched.
**Evidence classes used throughout:** **[MEASURED-PVE]** against the real PVE tooling or a PVE install · **Both of Spike 1's fixture defects are corrected in this one:** a NIC is present **at install time**
**[MEASURED-MECH]** measured, but in an isolated container rather than on PVE · **[STRUCTURAL]** read (its absence made the installed system's interface config unmatchable at boot), and completion is
from shipped source or unit files · **[BLOCKED]** needs the nested VM. signalled by `reboot-mode = "power-off"` — the guest powering itself off — never by a changed disk hash.
Three scenarios, all installed to completion:
| Scenario | root credential | `[first-boot].ordering` | Install |
|---|---|---|---|
| **A** | `root-password-hashed = "*"` (locked) | `fully-up` | completed, powered off after **191 s** |
| **B** | `root-password-hashed = "*"` (locked) | `before-network` | completed, powered off after **190 s** |
| **C** | a **known** password (control) | `fully-up` | completed, powered off after **191 s** |
**Evidence classes:** **[MEASURED-PVE]** on a real PVE install · **[MEASURED-MECH]** measured in an
isolated container · **[STRUCTURAL]** read from shipped source or unit files.
--- ---
## 2. Q1 — is a locked root actually locked, and what else does it lock out? ## 2. Q1 — is a locked root actually locked, and what else does it lock out?
### 2.1 The account's state on a real PVE install — **[BLOCKED]** ### 2.1 The account's real state — **[MEASURED-PVE]**
Needs the nested VM. The fixture is staged. Spike 1 established only that From the first-boot diagnostic on both locked installs (A and B), emitted to `ttyS0`:
`root-password-hashed = "*"` **installs to completion**; the resulting account state was explicitly
left open there and remains open.
### 2.2 What can authenticate — **[MEASURED-MECH]**, and the first attempt was invalid ```
passwd_S_root : root L 2026-07-31 0 99999 7 -1
shadow_shape : literal-asterisk
sshd_PermitRoot : yes
pam_common_auth : 1
pve_pam_realm : 0
```
**The first probe produced nothing and is recorded as failed.** It ran everything as uid 0, so `su` **The account is genuinely locked** — `passwd -S` reports `L`, and the `*` reached `/etc/shadow`
never invoked `pam_unix` at all (`rc=0` even with a deliberately wrong password); `sulogin` died on verbatim. Note `sshd_PermitRoot: yes`: PVE ships `PermitRootLogin yes`, so the lock is doing the work,
`tcgetattr failed: Invalid argument` with no TTY; and a `sed` mangled the control hash so the control not sshd's configuration.
case had no password set. Four cases, zero valid observations. Rerun with a non-root caller, a real
pty via `script`, and `chpasswd` instead of `sed`.
**Corrected probe** (`debian:trixie`, `util-linux 2.41`, `pam_unix`) — with a working control: `pve_pam_realm: 0` — there is no PVE-specific file in `/etc/pam.d/` matching `proxmox|pve`, and
`pam_unix` is present in `common-auth`. **PVE uses the stock PAM stack**, which is why the §2.2
container measurement transfers to it rather than merely resembling it.
| Shadow field | `passwd -S root` | `su root` as non-root user | `sulogin` (real pty) | ### 2.2 What can authenticate — **[MEASURED-MECH]**, first attempt invalid
**The first probe produced nothing and is recorded as failed:** it ran everything as uid 0 so `su`
never invoked `pam_unix` (`rc=0` even with a deliberately wrong password), `sulogin` died on
`tcgetattr failed` with no TTY, and a `sed` mangled the control hash. Four cases, zero observations.
Corrected (`debian:trixie`, `util-linux 2.41`, non-root caller, real pty via `script`, `chpasswd`):
| Shadow field | `passwd -S root` | `su root` as a non-root user | `sulogin` (real pty) |
|---|---|---|---| |---|---|---|---|
| real hash (control) | `P` | **correct pw → `SU_OK`**; wrong pw → `Authentication failure` | — | | real hash (**control**) | `P` | **correct pw → `SU_OK`**; wrong pw → `Authentication failure` | — |
| `*` | **`L`** | `su: Authentication failure` (any password) | **`Cannot open access to console, the root account is locked.`** | | `*` | `L` | `Authentication failure` (any password) | `Cannot open access to console, the root account is locked.` |
| `!` | **`L`** | `su: Authentication failure` (any password) | **`Cannot open access to console, the root account is locked.`** | | `!` | `L` | `Authentication failure` (any password) | `Cannot open access to console, the root account is locked.` |
The control is what makes this readable: the same harness authenticates successfully with the right The control makes it readable: the same harness succeeds with the right password and fails with the
password and fails with the wrong one, so the two locked rows are the module refusing, not the harness wrong one, so the locked rows are the module refusing rather than the harness misfiring. **`*` and `!`
misfiring. are equivalent.**
**So, at mechanism level: `*` and `!` are equivalent, both report `L`, and no password authenticates.** **On PVE, root SSH was refused on every scenario** — but that row is **uninformative and is not
evidence**: the probe supplied a deliberately wrong password in the control scenario too, so it cannot
distinguish "locked" from "wrong password". Recorded as a gap, not a result.
**Caveat, and it is not small.** This is Debian's `pam_unix` and `util-linux`'s `sulogin` — the same ### 2.3 Does the Proxmox web interface still work? — **[MEASURED-PVE]. No. It is closed.**
components PVE ships — but it is **not PVE**. PVE could ship PAM configuration, a `sulogin` override,
or a `SYSTEMD_SULOGIN_FORCE` setting that changes the outcome. Confirming on a real PVE install is
part of the blocked work.
### 2.3 Does the Proxmox web interface still work? — **[BLOCKED]**, and this is the product question The product-level answer, measured against the **exact endpoint the web UI logs in through**
(`POST /api2/json/access/ticket`, realm `root@pam`), with a working control:
**No measurement was obtained, so no answer is given.** Stating the reasoning without the measurement | Scenario | Password offered | Result |
would be exactly the failure mode this spike exists to avoid. |---|---|---|
| **C (control, known password)** | **correct** | **HTTP 200, TICKET ISSUED — login would SUCCEED** |
| C (control) | wrong | HTTP 401, no ticket |
| C (control) | empty | HTTP 401, no ticket |
| **A (locked root)** | wrong | HTTP 401, no ticket |
| **A (locked root)** | empty | HTTP 401, no ticket |
What is established: PVE's `root@pam` realm authenticates the Linux root account through PAM, and The control is what makes this a finding rather than a broken probe: **the same endpoint on the same
§2.2 measured that `pam_unix` refuses a locked account for a non-root caller. If PVE's proxy image demonstrably issues a ticket when the password is right.** On the locked install no password can
authenticates the same way — **which was not verified** — then a locked root means **nobody can log produce one, because no password exists.
into the PVE web interface at all**, neither customer nor operator, and that is a product decision
rather than a detail.
The staged control scenario exists precisely to settle this: install C carries a **known** password, so **Therefore a locked root closes the PVE web interface completely — to the customer and to the
the PVE API ticket endpoint can be exercised positively (correct password → ticket) before install A operator alike.** This is a product decision, not a detail: it removes the browser-based route into
(locked) is tested against the same endpoint. Without that control a failure is indistinguishable the box entirely, on top of the console route (§3.2).
from a broken probe — Spike 1's lesson.
### 2.4 Does anything else in PVE break? — **[BLOCKED]** Reachability, incidentally measured on all three: **SSH banner at +20 s, pveproxy TLS at +26–29 s**
after power-on.
Not investigated. Cron, `pveproxy`, backup jobs and anything else assuming a usable root credential ### 2.4 Does anything else in PVE break? — **not investigated**
remain unexamined.
Cron, backup jobs and other consumers of a root credential were not examined. `pveproxy` itself runs
and serves TLS (it answered `/api2/json/version`), so the daemon is unaffected — only authentication
through it is.
--- ---
## 3. Q2 — is there a way back into a box whose setup failed? ## 3. Q2 — is there a way back into a box whose setup failed?
### 3.1 Does the installed GRUB expose a recovery entry? — **[BLOCKED]** ### 3.1 Does the installed GRUB expose a recovery entry? — **[MEASURED-PVE]. Yes.**
Needs a boot. Note the installed system's GRUB is **PVE's own**, written by the installer — not the The installed system's GRUB is PVE's own, unaffected by Felhom's ISO menu collapse (§4). Screendumped
ISO's Felhom menu — so §5's single-entry collapse does not apply to it. What it offers is unverified. from the locked install:
### 3.2 `sulogin` with a locked root — **[MEASURED-MECH]. It refuses.**
The single observation the escrow-versus-discard ruling was said to rest on:
``` ```
*Proxmox VE GNU/Linux
Advanced options for Proxmox VE GNU/Linux
Memory test (memtest86+x64.bin)
Memory test (memtest86+x64.bin, serial console)
```
and inside *Advanced options*:
```
*Proxmox VE GNU/Linux, with Linux 7.0.2-6-pve
Proxmox VE GNU/Linux, with Linux 7.0.2-6-pve (recovery mode)
```
**A recovery entry exists, the menu is editable (`e`), a GRUB command line is available (`c`), and
there is no GRUB password.** Its timeout is **1 second** — short enough that a first probe missed it
entirely (§3.5).
### 3.2 `sulogin` with a locked root — **[MEASURED-PVE]. It refuses.**
Booting *(recovery mode)* on the locked install reaches rescue mode and then stops:
```
Started rescue.service - Rescue Shell.
Reached target rescue.target - Rescue Mode.
You are in rescue mode. After logging in, type "journalctl -xb" …
Cannot open access to console, the root account is locked. Cannot open access to console, the root account is locked.
See sulogin(8) man page for more details. See sulogin(8) man page for more details.
Press Enter to continue.
``` ```
Both for `*` and for `!`, with a real pty. **The standard Debian/systemd emergency and single-user **The advertised recovery path is closed on a locked-root box** — verbatim the container result,
path does not grant a shell on a locked-root system.** `systemd` invokes `sulogin` for now confirmed on real PVE. `systemd` invokes `sulogin` for both `rescue.target` and
`emergency.target` and `rescue.target`, so both are closed by the same refusal. `emergency.target`, so both are closed by the same refusal.
Subject to the §2.2 caveat: measured in a container, not on PVE. ### 3.3 What is left — **[MEASURED-PVE]. `init=/bin/bash` works.**
### 3.3 What is left if `sulogin` refuses — **[STRUCTURAL]**, not measured Appending `init=/bin/bash` to the kernel line in the GRUB editor and booting produced:
| Path | Realistic for whom | ```
|---|---| Found volume group "pve" using metadata type lvm2
| `init=/bin/bash` on the kernel line (GRUB `e`) | **Operator only.** It bypasses `sulogin` because PID 1 becomes the shell — no authentication is involved. Requires editing a boot entry at the console. Not a non-technical customer action. | 3 logical volume(s) in volume group "pve" now active
| `systemd.unit=emergency.target` | **Closed** — it lands in `sulogin`, which refuses (§3.2). | /dev/mapper/pve-root: clean, 60332/561936 files, 1141450/2243584 blocks
| Live USB / rescue media | Operator only; requires physical media and knowing to do it. | bash: cannot set terminal process group (-1): Inappropriate ioctl for device
| PVE web console (`:8006`) | **Unknown, possibly closed** — §2.3. | bash: no job control in this shell
| SSH by key | Only if a key was baked, which for a public image is the thing R-142 says must not happen. | root@(none):/#
```
**The honest summary:** on a locked-root box, every remaining route is operator-grade and physical. **An unauthenticated root shell on a locked-root system.** No password is involved because PID 1 *is*
For a customer whose pairing failed, there is no self-service path back — which is precisely the the shell — `sulogin` is never reached.
objection raised when locked-root was first proposed, and §3.2 is the first evidence that the
objection is well founded rather than speculative.
### 3.4 Does locked differ from rotated? — **[PARTIALLY ESTABLISHED]** | Path | Works? | Realistic for whom |
|---|---|---|
| PVE web console (`:8006`) | **No** (§2.3) | — |
| GRUB *(recovery mode)* → `sulogin` | **No** (§3.2) | — |
| GRUB editor → `init=/bin/bash` | **Yes** (measured) | **Operator only.** Requires catching a 1-second menu, editing a kernel line, and knowing the incantation. Not a non-technical customer action. |
| Live USB / rescue media | not tested | Operator only |
| SSH by key | n/a | Only if a key were baked — which R-142 says must not happen in a public image |
At mechanism level, yes, and materially: **So a locked-root box is recoverable, but only by someone competent at the physical console.** That is
a materially weaker objection than "no path back at all", which is what the structural reading
suggested before it was measured.
- **Locked (`*`/`!`)** — `sulogin` refuses; console recovery is closed to everyone; the credential **A security corollary that is worth stating plainly: the installed GRUB has no password, so locking
does not exist so it cannot leak, be escrowed, or be rotated. root buys nothing against physical access.** Anyone at the keyboard can take a root shell in under a
- **Rotated to a per-box random value** — `sulogin` prompts and *accepts the correct password* minute. Locked root is a defence against *remote* credential reuse — the shared-ISO-password problem —
(the control row in §2.2 shows `pam_unix` authenticating a real hash). So console recovery is open and not a physical-security measure. **R-152.**
**to whoever holds that password** — which is the operator if it is escrowed, and nobody if it is
discarded.
**This makes the tradeoff concrete: "discard" and "lock" are the same outcome for recovery.** A ### 3.4 Does locked differ from rotated? — **[MEASURED]. Yes, at exactly one point.**
rotated-then-discarded password is operationally identical to a locked account, because nobody holds
it. The real choice is therefore **binary — escrow, or accept that console recovery is operator-only - **Locked (`*`/`!`)** — `sulogin` refuses (§3.2); the web UI refuses (§2.3); `init=/bin/bash` works.
and physical.** Framed for the ruling in §7. - **Rotated to a per-box value** — `pam_unix` accepts the correct password (the §2.2 control proves
this) and the web UI issues a ticket for it (the §2.3 control proves this), so **both** the console
and the browser open **to whoever holds it**; `init=/bin/bash` works as well.
**And this is what settles the escrow question.** A rotated password that is *discarded* is
operationally identical to a locked account: nobody holds it, so `sulogin` and the web UI refuse
either way. The only thing escrow buys is that the **operator** keeps the web-UI and `sulogin` routes.
It does not buy recoverability as such, because `init=/bin/bash` is available in every case.
### 3.5 A probe that failed, recorded as failed
The first GRUB attempt waited 3 seconds before acting; the menu's timeout is **1 second**, so the
system had already booted and the keystrokes went into the login prompt — the arrow-key escape
sequences (`^[[B`) are visible there in the screendump. It tested nothing. A later attempt typed
`init=/bin/bash` onto the `echo` line one below `linux`, which the **pre-typing verification
screendump caught** before it was booted. Both were rerun; the numbers above are from the corrected
runs.
--- ---
## 4. Q3 — does the Felhom stub survive `before-network`? ## 4. Q3 — does the Felhom stub survive `before-network`?
### 4.1 Does it execute? — **[BLOCKED]** ### 4.1 Does it execute? — **[MEASURED-PVE]. Yes.**
Needs the VM. The staged diagnostic emits `@@SPIKE2_FIRSTBOOT_RAN@@ marker=BEGIN/END` to `ttyS0` The diagnostic emits `@@SPIKE2_HOOK_RAN@@ BEGIN`/`END` to `ttyS0`. On scenario B
(captured by `qemu -serial file:`) precisely so the answer is a **positive observable** rather than the (`ordering = "before-network"`) **both markers are present** (count `2`). This is a positive
absence of an error — and §4.4 below is why that matters more than usual here. observable the hook produced, which §4.4 explains is the only acceptable form of proof here.
### 4.2 Does it have what it needs that early? — **[STRUCTURAL]**, partly answered ### 4.2 Does it have what it needs that early? — **[MEASURED-PVE]. Yes, everything.**
From the shipped unit (`proxmox-first-boot` 9.2.5, ### 4.3 Does it complete before anything is network-reachable? — **[MEASURED-PVE + STRUCTURAL]. Yes.**
`/usr/lib/systemd/system/proxmox-first-boot-network-pre.service`):
The A/B comparison, same image, same diagnostic, only `ordering` differs:
| Diagnostic | **A — `fully-up`** | **B — `before-network`** |
|---|---|---|
| hook markers | 2 (ran) | **2 (ran)** |
| `uptime_s` at hook | 18.32 | **6.58** |
| `entropy_avail` | 256 | **256** |
| `etc_writable` | YES | **YES** |
| `openssl`/`chpasswd`/`systemctl`/`passwd`/`curl`/`ip` | all present | **all present** |
| `openssl_rand_len` | 32 | **32** |
| `ip_global` | `vmbr0 10.0.2.15/24` | **(empty)** |
| `network.target` | active | **inactive** |
| `network-online.target` | active | **inactive** |
| `pveproxy` | active | **inactive** |
| `sshd` | active | **inactive** |
| **`listen_22_8006`** | **3** | **0** |
**At `before-network` the hook runs with entropy, a writable `/etc`, every binary it would need, and a
demonstrated ability to generate a 32-character random string — while the box has no IP address and
nothing is listening on `:22` or `:8006`.** A credential rotation placed there has a **zero-length
exposure window, measured.**
Equally, the `fully-up` column is the measured proof of the converse: **today's ordering runs the hook
when sshd and pveproxy are already listening.** The window Spike 1 argued for structurally is now
observed.
The structural half agrees and generalises it. The shipped unit
(`proxmox-first-boot` 9.2.5, `/usr/lib/systemd/system/proxmox-first-boot-network-pre.service`):
``` ```
After=systemd-remount-fs.service After=systemd-remount-fs.service
@@ -209,57 +298,37 @@ ConditionPathIsReadWrite=/var/lib
Type=oneshot Type=oneshot
``` ```
`After=systemd-remount-fs.service` plus `ConditionPathIsReadWrite=/var/lib` mean **a writable `Before=network-pre.target` + `Type=oneshot` makes it a hard barrier ahead of all network
filesystem is structurally guaranteed** — or the unit does not run at all (§4.4). Entropy availability configuration — so this holds on every machine, not just this VM. Both installs then booted to full
and binary presence at that point are **not** established structurally and are part of the staged reachability (SSH +20 s, pveproxy +26–29 s), so the early hook does not break the boot.
diagnostic (`entropy_avail`, `openssl rand` length, `command -v` for `openssl`/`chpasswd`/`systemctl`).
### 4.3 Does it complete before anything is network-reachable? — **[STRUCTURAL]. Yes, by construction.**
This is the result the task asked to establish structurally rather than by timing, and the unit file
settles it:
- **`Before=network-pre.target`** — `network-pre.target` is systemd's ordering barrier that precedes
*all* network configuration; network managers order themselves after it.
- **`Type=oneshot`** — systemd treats the unit as complete only when the process exits, so
`Before=` is a hard barrier rather than a hint.
Together: **the hook runs to completion before any interface is configured**, therefore before `sshd`
can bind a routable address and before `pveproxy` serves. A rotation performed there has a
**zero-length exposure window by construction, not by being fast.** This holds on every machine and
does not vary with hardware — which is why it is stronger than the timing number Spike 1 failed twice
to obtain.
**A third ordering value exists that Spike 1 did not record.** `validate-answer` accepts **A third ordering value exists that Spike 1 did not record.** `validate-answer` accepts
**`before-network`, `network-online`, and `fully-up`** — and rejects `network-pre`, `multi-user` and a **`before-network`, `network-online`, `fully-up`**; it rejects `network-pre`, `multi-user` and a bogus
bogus value, so this is a real enum and not a permissive parser. The shipped package contains a value, so this is a real enum. The shipped package contains a matching unit for each.
matching unit for each. Spike 1 reported two. `network-online` is the middle option — after the `network-online` is the unevaluated middle option. **R-149.**
network, before `pveproxy` — and is worth considering if `before-network` proves too early for
something.
### 4.4 Does the existing pairing flow still work? — **[STRUCTURAL]** yes; **[BLOCKED]** empirically ### 4.4 Does the existing pairing flow still work? — **[STRUCTURAL]**, and it is the one gap left
Two structural findings, and the second one corrects a premise this design has been carrying. **Not measured.** The scenarios installed the spike's own diagnostic as the first-boot hook, not
Felhom's `stub-first-boot.sh`, so stub-plus-pairing coexistence is argued, not observed.
**The stub does not need the network, and does not need `pveproxy`.** `stub-first-boot.sh` writes three The structural case is strong and has a correction in it:
files, then `systemctl daemon-reload`, `enable`, and `start --no-block felhom-bootstrap.service`. A
grep for `pvesh`, `pct`, `pveum` and `qm` across the stub returns **no match** (`rc=1`). The unit it
starts carries its own `After=network-online.target pve-cluster.service pveproxy.service`, so
`--no-block` merely queues the job and systemd holds it until those are satisfied. Structurally, stub
and pairing coexist: the stub runs early, the bootstrap runs when its own dependencies are met.
**So `answer.toml.tmpl:27` justifies the `fully-up` choice with a dependency the stub does not have.** **The stub does not need `fully-up`.** `stub-first-boot.sh` writes three files, then
The comment reads *"fully-up = the stub runs after pveproxy so pvesh/pct work"* — but the `pvesh`/`pct` `systemctl daemon-reload`, `enable`, and `start --no-block felhom-bootstrap.service`. A grep for
usage is in `felhom-bootstrap.sh`, which never runs under the stub's ordering. This is the `pvesh`, `pct`, `pveum` and `qm` across it returns **no match** (`rc=1`). The unit it starts carries its
CLAUDE.md "a comment asserting an invariant" class in a new shape: a comment asserting a **dependency** own `After=network-online.target pve-cluster.service pveproxy.service`, so `--no-block` queues the job
that does not exist, which has constrained the design ever since. **R-148.** and systemd holds it until those are satisfied.
**The trap that makes §4.1's positive observable load-bearing.** All four `proxmox-first-boot-*` units **So `answer.toml.tmpl:27` justifies the current ordering with a dependency the stub does not have** —
are guarded by `ConditionPathExists=` and `ConditionPathIsReadWrite=`. In systemd a **failed condition it reads *"fully-up = the stub runs after pveproxy so pvesh/pct work"*, but that usage lives in
is a skip, not a failure** — the unit reports `inactive`/`condition failed`, the boot succeeds, and `felhom-bootstrap.sh`, which never runs under the stub's ordering. A comment asserting a **dependency**
nothing is logged as wrong. A first-boot hook that never ran and one that ran perfectly are that does not exist, and it has constrained the design ever since. **R-148.**
indistinguishable from the absence of errors. **R-150.** This is why §4.1 must be confirmed by a
marker the hook emitted, never by "no errors appeared". **Why §4.1's positive marker is load-bearing.** All four `proxmox-first-boot-*` units are guarded by
`ConditionPathExists=` / `ConditionPathIsReadWrite=`, and in systemd **a failed condition is a skip,
not a failure** — no error, successful boot. A hook that never ran and one that ran perfectly are
indistinguishable from the absence of errors. **R-150.**
--- ---
@@ -267,119 +336,99 @@ marker the hook emitted, never by "no errors appeared".
### 5.1 Where the collapse happens — **[STRUCTURAL]**, and it is not a simple removal ### 5.1 Where the collapse happens — **[STRUCTURAL]**, and it is not a simple removal
`iso-repack.sh` replaces the stock `grub.cfg` wholesale: it renders `grub/grub.cfg.tmpl` over it `iso-repack.sh` replaces the stock `grub.cfg` wholesale, rendering `grub/grub.cfg.tmpl` over it
(`iso-repack.sh:144-148`), having first lifted the `linux`/`initrd` lines verbatim from the stock (`:144-148`) after lifting the `linux`/`initrd` lines verbatim from the stock *Install Proxmox VE
*Install Proxmox VE (Automated)* entry (`:108-131`) so a PVE bump tracks automatically. (Automated)* entry (`:108-131`) so a PVE bump tracks automatically.
**Four independent gates then enforce the single entry**, so restoring the stock menu is a change to **Four independent gates enforce the single entry:**
the gates, not just to the template:
| Gate | Location | Effect | | Gate | Location | Effect |
|---|---|---| |---|---|---|
| exactly one `menuentry` | `iso-repack.sh:153-155` | `exit 14` otherwise | | exactly one `menuentry` | `:153-155` | `exit 14` |
| zero `submenu` | `iso-repack.sh:156` | `exit 14` otherwise | | zero `submenu` | `:156` | `exit 14` |
| no live reference to `proxtui`, `proxdebug`, `nomodeset`, `Rescue Boot`, `memtest`, `fwsetup` | `iso-repack.sh:160-164` | `exit 14` otherwise | | no live `proxtui`, `proxdebug`, `nomodeset`, `Rescue Boot`, `memtest`, `fwsetup` | `:160-164` | `exit 14` |
| **re-extracted from `final.iso`**: exactly one entry, theme present, entry named `Felhom telepítés` | `iso-repack.sh:314-319` | `exit 15` otherwise | | **re-extracted from `final.iso`**: one entry, theme present, entry named `Felhom telepítés` | `:314-319` | `exit 15` |
The fourth is the strongest of them — it proves the menu in the shipped image rather than in the build The fourth proves the menu in the shipped image rather than the build tree — the seam-wiring lesson
tree, which is the seam-wiring lesson applied correctly. applied correctly.
**What adding one interactive entry would actually cost:** the two count gates (rows 1 and 4) move from **What a second entry would cost:** the two count gates move from `== 1` to `== 2`, or to a named
`== 1` to `== 2`, or better to a named allowlist. **The banned-token gate would still pass unchanged** allowlist. **The banned-token gate passes unchanged** for a stock *Install Proxmox VE (Graphical)*
for a stock *Install Proxmox VE (Graphical)* entry — it uses none of `proxtui`, `proxdebug`, entry — it uses none of those six tokens. The safety property the gates exist for (no route to a
`nomodeset`, `Rescue Boot`, `memtest`, `fwsetup`. So the safety property the gates exist to protect debug/rescue/manual-disk-picker path) survives a second graphical-installer entry intact.
(no route to a debug/rescue/manual-disk-picker path) survives a second *graphical installer* entry
intact. That is a narrower change than it first appears.
### 5.2 Are default and timeout settable? — **[STRUCTURAL]. Yes, already.** ### 5.2 Are default and timeout settable? — **[STRUCTURAL]. Already are.**
`grub/grub.cfg.tmpl` sets them explicitly: `grub/grub.cfg.tmpl` sets `set timeout_style=menu`, `set timeout=5`, `set default=0`. The template also
records a real defect it fixed: the stock PVE config writes `timeout-style` (hyphen), **which GRUB does
not recognise as a variable name**, so Felhom's underscore spelling is the one that works.
``` ### 5.3 The design question — **framed with evidence, deliberately not ruled**
set timeout_style=menu
set timeout=5
set default=0
```
Both are plain template values. The template also records a real defect it fixed: the stock PVE config **For interactive-as-default.** Spike 1 proved the automated path cannot be made safe on unseen
writes `timeout-style` (hyphen), **which GRUB does not recognise as a variable name**, so Felhom's hardware: no property distinguishes a system disk from a customer's backup drive, and a filter matching
`timeout_style` (underscore) is the one that actually works. A two-entry menu would therefore have a two disks silently wipes one. A tester who boots and walks away on multi-disk hardware is walking away
functioning timeout and a settable default — which is exactly what the §5.3 question needs. from an unattended destructive operation whose target cannot be predicted.
### 5.3 The design question — **framed, deliberately not ruled** **This session strengthens that case independently.** If the public image also ships a locked root,
then a box whose install succeeds but whose pairing fails has **no web UI** (§2.3) and **no working
recovery-mode shell** (§3.2) — only an operator-grade `init=/bin/bash` (§3.3). An unattended default
maximises the number of testers who reach that state without having read anything.
Which entry should be `default=0` in a **public** image? **Against.** Unattended install is a selling point and most first boxes are single-disk, where the
**What the measurements say, and they only point one way:** Spike 1 established that the automated
path cannot be made safe on unseen hardware — no property distinguishes a system disk from a
customer's backup drive, and a filter matching two disks silently wipes one. A tester who boots the
image and walks away is, on multi-disk hardware, walking away from an unattended destructive operation
whose target cannot be predicted.
**What that costs:** unattended install stops being the default for people who do not read
instructions — and unattended install is a selling point. Most first boxes are single-disk, where the
automated path is fine. automated path is fine.
**The measurement that would change the balance** is §2.3 and §3: if a locked root also closes the PVE **Operator-built and public need not match, and two profiles from one build is cheap.** An
web interface and the console, then a tester whose unattended install completes but whose pairing operator-built image targets known disks, so automated-default is safe there; the public image is the
fails has a machine nobody can reach — which argues for interactive-default independently of the disk one facing unseen hardware. That split is available however the default is ruled.
question. Those measurements are blocked, so the balance cannot be struck yet.
**Operator-built versus public need not match, and two profiles from one build is cheap.** An
operator-built image targets a machine whose disks are known, so automated-default is safe there. The
public image is the one facing unseen hardware. Recommending the split is not the same as ruling the
default, and the split is available regardless of how the ruling goes.
--- ---
## 6. Q5 — the resulting shape, as constraints ## 6. Q5 — the resulting shape, as constraints
Constraints on a build spec, each with a release-gate criterion. **Not an implementation, and nothing Constraints on a build spec, each with a release-gate criterion. **Not an implementation.**
was built.** Constraints marked ⚠ depend on blocked measurements and are provisional.
| # | Constraint | Release gate | | # | Constraint | Release gate |
|---|---|---| |---|---|---|
| 1 | **The image must not select a disk by property on unseen hardware.** Spike 1 §3.2/§3.3. | The rendered answer contains no `filter.*` key, **or** the image is operator-built for a named machine. Grep the rendered `answer.toml`. | | 1 | **No disk selection by property on unseen hardware** (Spike 1 §3.2/§3.3). | Rendered answer contains no `filter.*` key, **or** the image is operator-built for a named machine. |
| 2 | **The interactive installer must be reachable from the boot menu.** §5.1 shows the cost is two count gates. | `final.iso`'s `grub.cfg` contains ≥1 interactive entry and still contains no live `proxtui`/`proxdebug`/`nomodeset`/`Rescue Boot`/`memtest`/`fwsetup` — i.e. keep gate 3, relax gates 1 and 4 to an allowlist. | | 2 | **The interactive installer must be reachable from the boot menu** (§5.1). | `final.iso`'s `grub.cfg` has ≥1 interactive entry **and** still no live `proxtui`/`proxdebug`/`nomodeset`/`Rescue Boot`/`memtest`/`fwsetup` — keep gate 3, relax gates 1 and 4 to an allowlist. |
| 3 | ⚠ **No usable baked credential.** `root-password-hashed = "*"` is schema-valid and installs (Spike 1). | The rendered answer's `root-password-hashed` is exactly `*`, and **no `.rootpw.txt` is emitted**. Both are one-line assertions in the build. | | 3 | **No usable baked credential** — `root-password-hashed = "*"` installs cleanly (§1, three scenarios). | Rendered `root-password-hashed` is exactly `*`, **and no `.rootpw.txt` is emitted**. Two one-line assertions. |
| 4 | ⚠ **If a per-box password is set, it is set at `before-network`.** §4.3 — the only ordering with a zero window by construction. | The rendered answer's `[first-boot].ordering` is `before-network`; and the hook's positive marker is observed in a boot test, never inferred from absent errors (§4.4, R-150). | | 4 | **Any per-box password is set at `before-network`** — the only ordering with a measured zero window (§4.3). | Rendered `[first-boot].ordering` is `before-network`, **and** the hook's positive marker is observed in a boot test (§4.4, R-150). |
| 5 | **No `root-ssh-keys`, ever, in a public image.** Spike 1 R-142. | `grep -c root-ssh-keys` on the rendered answer is `0`. | | 5 | **No `root-ssh-keys`, ever, in a public image** (R-142). | `grep -c root-ssh-keys` on the rendered answer is `0`. |
| 6 | **The profile must be committed.** Spike 1 found demo-felhom's baked key comes from an *uncommitted* profile, so a reference box is not reproducible from the repo. | The `--profile` path resolves inside the repo working tree **and** `git status --porcelain` is empty for it. **R-151.** | | 6 | **The profile must be committed** — demo-felhom's baked key comes from an *uncommitted* profile, so a reference box is not reproducible from `main`. | `--profile` resolves inside the repo tree **and** `git status --porcelain` is empty for it. **R-151.** |
| 7 | **The artefact must be reproducible.** Spike 1 R-144 (`nested-probe`'s profile no longer exists). | Manifest records the profile's committed path and the repo commit; both resolve. | | 7 | **The artefact must be reproducible** (R-144: `nested-probe`'s profile is gone). | Manifest records the committed profile path + repo commit; both resolve. |
| 8 | **A recovery path must be documented and, if operator-only, said so plainly.** §3.3. | The release notes name the recovery route for a failed pairing. If that route is `init=/bin/bash`, the notes say it is operator-only. | | 8 | **The recovery path must be documented, and named operator-only if it is.** §2.3 + §3.2 + §3.3: with a locked root the only way in is `init=/bin/bash` at the console. | Release notes state it, in those terms. |
**What the image must not contain:** a usable root password hash, any `root-ssh-keys`, any customer **What the image must not contain:** a usable root password hash, any `root-ssh-keys`, any customer id,
id, claim code, retrieval passphrase or hub token, and any `filter.*` disk selector in the public claim code, retrieval passphrase or hub token, and any `filter.*` selector in the public variant.
variant. Spike 1 §6 established by enumeration that the only secret-bearing file is `/answer.toml`, so Spike 1 §6 established by enumeration that `/answer.toml` is the only secret-bearing file, so gates 3
gates 3 and 5 over that one rendered file cover the credential surface. and 5 over that one rendered file cover the credential surface.
--- ---
## 7. The operator's decisions ## 7. The operator's decisions
Two, each with the measurement that informs it and its current state. **Decision 1 — escrow or discard a per-box root password.** §3.4 reshapes this twice over:
**Decision 1 — escrow or discard a per-box root password.** - **Discard and lock are the same thing** for recovery — if nobody holds the password, both `sulogin`
and the web UI refuse.
- **Neither makes a box unrecoverable.** `init=/bin/bash` works in every case (§3.3), so an operator at
the console can always get in. The earlier fear that locking root leaves "no path back at all" is
**measured to be false**.
§3.4 changed the shape of this question. **"Discard" and "lock" are operationally identical** — if So what escrow actually buys is narrower than it looked: **the operator keeps the web-UI route (§2.3)
nobody holds the password, `sulogin` refuses either way, and console recovery is closed to everyone. and the `sulogin` route (§3.2) without needing physical presence.** Against that stands D6 — a hub
So the choice is binary: holding console root for the whole fleet is exactly what the hub alone must not be.
- **Escrow** — the operator can always reach a box's console; the cost is a hub-held credential for **Recommendation, offered because the measurements now support one:** discard/lock for the **public**
every box, which is the thing the D6 trust model says the hub alone must not be. image, and accept operator-only physical recovery, because the credential that would be escrowed is
- **Discard / lock** — no credential exists anywhere; the cost is that a failed pairing leaves a the same one whose fleet-wide sharing started this whole line of work. Escrow remains defensible for
machine recoverable only by an operator at the keyboard (`init=/bin/bash`) or a reinstall. **operator-built** boxes, where the operator installed the machine anyway. The decision is still the
operator's; it is no longer blocked on a measurement.
**Blocking measurement: §2.3** — whether a locked root also closes the PVE web interface. If it does, **Decision 2 — the default boot entry for a public image.** Framed in §5.3, unruled. The disk evidence
"discard" removes the last remote route as well as the local one, and the balance shifts sharply and the recovery evidence now point the same way (interactive), and the cost is the unattended
toward escrow. **Do not rule until §2.3 is measured.** experience. The operator-built-versus-public split is cheap and available either way.
**Decision 2 — the default boot entry for a public image.**
Framed in §5.3. The disk evidence points at interactive-default; the product cost is the unattended
experience. **Partially blocked on the same §2.3 measurement**, because an unreachable-after-failure
box strengthens the interactive case independently. The operator-built-versus-public split (§5.3) is
available regardless and is cheap.
--- ---
@@ -387,70 +436,65 @@ available regardless and is cheap.
| # | Unknown | What it takes | | # | Unknown | What it takes |
|---|---|---| |---|---|---|
| 1 | **Everything in §2.1, §2.3, §2.4, §3.1, §4.1, §4.2 (entropy/binaries).** | The staged fixture, unchanged, once a Tier 0 box returns. ~90 min for all three scenarios. | | 1 | **Felhom's real stub at `before-network`** (§4.4) — coexistence with pairing is structural, not observed. The single remaining gap in Q3. | Build a probe ISO with `stub-first-boot.sh` as the hook and `ordering = "before-network"`, boot, confirm `felhom-bootstrap.service` still starts and pairing completes. ~40 min. **Do this before the spec.** |
| 2 | **Whether PVE alters the §2.2/§3.2 mechanism results** — PAM config, a `sulogin` override, `SYSTEMD_SULOGIN_FORCE`. | Same fixture; check `/etc/pam.d/`, the `sulogin` invocation, and systemd's emergency unit on the installed image. | | 2 | **Whether root SSH is refused *because* of the lock** (§2.2) — the probe used a wrong password in the control too, so it discriminates nothing. | Re-run the SSH leg with the control's correct password. ~10 min. |
| 3 | **Whether the multi-match pick is deterministic by enumeration order** (Spike 1 §8 #1, still open). | Re-run Spike 1's S1 with reversed disk order. ~15 min. | | 3 | **What else in PVE assumes a usable root credential** (§2.4) — cron, backup jobs. | Inspect a locked install over a longer uptime. |
| 4 | **Whether the installer excludes its own boot medium** (Spike 1 §8 #2, still open). | USB boot on real hardware — HUMAN. | | 4 | **What `network-online` ordering buys** (§4.3) — a real third option, never evaluated. | One extra scenario. |
| 5 | **What `network-online` ordering buys** (§4.3) — it is a real third option nobody has evaluated. | One extra scenario alongside the staged two. | | 5 | **Whether the multi-match pick is deterministic by enumeration order** (Spike 1, still open). | Re-run Spike 1's S1 with reversed disk order. ~15 min. |
| 6 | **Whether a second GRUB entry actually boots the stock installer** after the gates are relaxed. | Build a probe ISO with a two-entry template and boot it. Not attempted — it would have meant changing repo files, which §4 forbids. | | 6 | **Whether the installer excludes its own boot medium** (Spike 1, still open). | USB boot on real hardware — HUMAN. |
| 7 | **Whether a relaxed-gate two-entry menu actually boots the stock installer** (§5.1). | Not attempted — it would mean changing repo files, which §4 forbids. |
--- ---
## 9. R-rows opened ## 9. R-rows opened
Numbering continues from R-147 (Spike 1). Continuing from R-147 (Spike 1).
| R | Severity | Finding | | R | Severity | Finding |
|---|---|---| |---|---|---|
| **R-148** | MEDIUM | **`answer.toml.tmpl:27` justifies `ordering = "fully-up"` with a dependency the stub does not have.** The comment says the stub must run after `pveproxy` "so pvesh/pct work", but `stub-first-boot.sh` contains no `pvesh`/`pct`/`pveum`/`qm` call (grep `rc=1`); that usage lives in `felhom-bootstrap.sh`, which runs under its own `After=network-online.target … pveproxy.service` unit. A comment asserting a **dependency** rather than an invariant — and it has constrained the design, because `before-network` is the ordering that would make the credential window zero (§4.3). | | **R-148** | MEDIUM | **`answer.toml.tmpl:27` justifies `ordering = "fully-up"` with a dependency the stub does not have.** `stub-first-boot.sh` contains no `pvesh`/`pct`/`pveum`/`qm` call (grep `rc=1`); that usage is in `felhom-bootstrap.sh`, under its own `After=network-online.target … pveproxy.service` unit. A comment asserting a **dependency** rather than an invariant — and it blocked the ordering that §4.3 now measures as giving a zero credential window. |
| **R-149** | LOW | **The `[first-boot].ordering` enum has three values, not two.** `before-network`, `network-online`, `fully-up` all validate; `network-pre` and `multi-user` are rejected. Spike 1 recorded two and the middle option has never been evaluated. | | **R-149** | LOW | **`[first-boot].ordering` has three values, not two** — `before-network`, `network-online`, `fully-up` validate; `network-pre` and `multi-user` are rejected. The middle option has never been evaluated. |
| **R-150** | MEDIUM | **A first-boot hook that never runs is indistinguishable from one that succeeded.** All four `proxmox-first-boot-*.service` units are guarded by `ConditionPathExists=` / `ConditionPathIsReadWrite=`, and in systemd a failed condition is a **skip, not a failure** — no error, successful boot. Any design that puts credential rotation in a first-boot hook must verify it by a positive marker the hook emitted. Directly the "an absent log line is not evidence" class. | | **R-150** | MEDIUM | **A first-boot hook that never runs is indistinguishable from one that succeeded.** All four `proxmox-first-boot-*.service` units are `Condition*`-guarded, and a failed condition is a systemd **skip, not a failure** — no error, successful boot. Any credential-rotation-in-first-boot design must be verified by a positive marker the hook emitted. |
| **R-151** | MEDIUM | **A reference box is not reproducible from the repository.** `demo-felhom` was installed from an ISO built with the **uncommitted** `n100-demo.profile`, which is where its baked SSH key comes from (Spike 1 §4.6). One of the two Tier 0 boxes therefore cannot be rebuilt from `main`. Related to R-144 (`nested-probe`'s profile is gone entirely); this row is the *reference box*, which matters more. Gate 6 in §6 exists to stop a public image inheriting the shape. | | **R-151** | MEDIUM | **A reference box is not reproducible from the repository.** `demo-felhom` was installed from an ISO built with the **uncommitted** `n100-demo.profile`, which is where its baked SSH key comes from (Spike 1 §4.6). Gate 6 in §6 exists to stop a public image inheriting that shape. |
| **R-152** | LOW | **A locked root is not a physical-security measure, and should never be described as one.** The installed PVE GRUB has **no password** (§3.1), so `init=/bin/bash` yields an unauthenticated root shell in under a minute (§3.3, measured). Locking root defends against *remote* reuse of a shared ISO credential and nothing else. If a physical-access threat model is ever wanted, it needs a GRUB password, which is a separate decision with its own lockout risk. |
--- ---
## 10. Teardown ## 10. Teardown
All three layers, verified positively. All three layers, each verified positively.
**Layer 1 — demo-hp.** **INCOMPLETE, and named as such.** The box went offline (§0) while **Layer 1 — demo-hp.**
`/mnt/nvme-1tb/spike2/` held the probe ISO (~1.7 GB) and the staged answer files and runners. Nothing
was running on it: the last successful command before the outage confirmed `0` QEMU processes, and no
install had been started this session. Nothing was registered as a PVE storage, nothing was written to
`local-lvm`, and `drill-r50` / guest 9201 were never touched this session.
**Outstanding cleanup, to run when the box returns** — recorded here because R-131 exists precisely | Check | Result |
because recorded commands never get run: |---|---|
| QEMU processes | `0` |
| `/mnt/nvme-1tb/spike2/` | removed — `No such file or directory` |
| `/tmp/paia` | removed |
| loop devices / `/mnt/s2ans` | `0` / removed |
| `/mnt/nvme-1tb/` contents | `dump e2d-images felhom-data images lost+found` — the pre-spike set |
| disk usage | **6.6 G — identical to pre-spike** |
| `drill-r50` (VM 300) | `stopped`, untouched (R-93 fixture intact) |
| guest 9201 | `running`, untouched |
| `/etc/pve/storage.cfg` | **4 storages, unchanged**; no scratch storage ever registered |
| `local-lvm` | never written |
``` **Layer 2 — demo-felhom.** `/tmp/paia` removed; guest 9201 `running`; the external USB drive
rm -rf /mnt/nvme-1tb/spike2 ; rm -f /tmp/paia ; pkill -9 qemu-system-x86 `931.5G usb /mnt/hdd_1` still attached and mounted exactly as found — never attached, detached,
``` partitioned or written. Only `device-info` reads were ever performed there.
This is deliberately *not* claimed as done. Spike 1's `/mnt/nvme-1tb/spike-scratch` and both `/tmp/paia`
copies were removed and verified at that session's teardown, so the outstanding item is this session's
`spike2/` directory alone.
**Layer 2 — demo-felhom.** Nothing placed this session — it was never reached before the outage.
Spike 1's `/tmp/paia` was removed and verified then. No action outstanding.
**Layer 3 — DooPlex.** **Layer 3 — DooPlex.**
| Check | Result | | Check | Result |
|---|---| |---|---|
| scratchpad | **4.4 GB → 48 K** (probe ISO, squashfs extracts, `.deb` extract, schema test dirs removed) | | scratchpad | **1.7 GB → 60 K** (probe ISO, screendumps, squashfs/`.deb` extracts, schema test dirs) |
| control-password file (`s2/.ctlpw`) | `shred -u`'d — never printed, never committed; verified absent | | control password (`s2/.ctlpw`) and `scen/C.toml` | `shred -u`'d / removed; verified absent. Never printed, never committed |
| mechanism-probe container | `docker run --rm` — no image or container retained; **no global Docker cleanup run** (fence) | | mechanism-probe container | `docker run --rm`; **no global Docker cleanup run** (fence) |
| `felhom-iso/out/` | untouched, **17 ISOs** | | `felhom-iso/out/` | untouched, **17 ISOs** |
| production files | none modified — no `build-felhom-iso.sh`, no repack, no profile, no `felhom-bootstrap.sh` | | production files | none modified — no `build-felhom-iso.sh`, no `iso-repack.sh`, no profile, no `felhom-bootstrap.sh` |
**One teardown error, recorded rather than quietly fixed.** Shredding the control password **Hub-side disposition — nothing to dispose.** No customer, host or appliance was created: the nested
(`s2/.ctlpw`) destroyed the plaintext that scenario **C** existed to use — a `$6$` hash whose plaintext VMs ran under plain QEMU with user-mode networking and no route to the hub, and no install ever
is gone cannot demonstrate "correct password → PVE ticket", which was C's entire purpose as the contacted it. **R-131 gains no row.**
control for §2.3. `s2/scen/C.toml` was therefore **deleted** rather than left in place looking usable.
**Scenario C must be regenerated** (one `openssl passwd -6` against a fresh random plaintext, kept
0600) before the §2.3 measurement is attempted. Scenarios A and B are unaffected and remain staged.
**Hub-side disposition — nothing to dispose.** No customer, host or appliance was created: no install
ran, and the only hub contact this session was none at all. Verified rather than asserted below.
**Published artefacts: none.** Nothing went to R2. **Published artefacts: none.** Nothing went to R2.
scripts/CHANGELOG.md
+27 -29
View File
@@ -1,38 +1,36 @@
## SPIKE 2 (no code change) — before-network gives a zero window by construction; locked root closes sulogin (2026-07-31) ## SPIKE 2 (no code change) — locked root closes the PVE web UI; before-network gives a measured zero window (2026-07-31)
**No script, profile or build file was modified.** Findings: **No script, profile or build file was modified.** Findings:
`documentation/audits/SPIKE-universal-iso-2-2026-07-31.md`. Opens R-148..R-151. `documentation/audits/SPIKE-universal-iso-2-2026-07-31.md`. Opens R-148..R-152.
**Both Tier 0 boxes went offline mid-session** (remote site, 12:28 CEST; four routes tried, our Both Tier 0 boxes went offline mid-session (provider cable fault; four routes tried) and returned;
tailscale pod healthy). Q1/Q2/Q3 each retain a part that needs a nested VM and those parts are all three scenarios then ran to completion on real PVE, each signalled by `reboot-mode="power-off"`.
**BLOCKED, not answered**. The fixture is staged with Spike 1's two defects already corrected.
Established without them: - **A locked root CLOSES the PVE web interface.** Measured against the exact endpoint the UI logs in
through (`POST /api2/json/access/ticket`, `root@pam`) **with a working control**: the known-password
- **STRUCTURAL — `ordering = "before-network"` maps to `proxmox-first-boot-network-pre.service` install returns **HTTP 200 + ticket**; the locked install returns 401 for every password, and no
(`proxmox-first-boot` 9.2.5), which is `Before=network-pre.target` + `Type=oneshot`.** It therefore password exists. `passwd -S root` = `L`, shadow = `literal-asterisk`. PVE uses the stock PAM stack
runs to completion before any interface is configured — a credential rotation there has a (`pve_pam_realm: 0`).
**zero-length window by construction**, not by being fast. Stronger than the timing number Spike 1 - **GRUB recovery mode is ALSO closed**: `Cannot open access to console, the root account is locked.`
failed twice to obtain. — the container mechanism result confirmed verbatim on real PVE.
- **STRUCTURAL — the stub does not need `fully-up`.** `stub-first-boot.sh` contains no - **But `init=/bin/bash` still works** — an unauthenticated `root@(none):/#`. So a locked-root box is
`pvesh`/`pct`/`pveum`/`qm` call (grep rc=1); that usage is in `felhom-bootstrap.sh`, which runs under recoverable, operator-only, at the physical console. The installed GRUB has **no password**, so
its own `After=network-online.target ... pveproxy.service` unit. So `answer.toml.tmpl:27` justifies locking root is NOT a physical-security measure. **R-152.**
the current ordering with a dependency that does not exist. **R-148.** - **`before-network` measured, A/B, same image:** the hook runs (positive marker, uptime 6.58s) with
- **The ordering enum has THREE values** — `before-network`, `network-online`, `fully-up`. Spike 1 entropy 256, writable /etc, all binaries and `openssl_rand_len=32`, while `ip_global` is EMPTY and
recorded two; the middle one has never been evaluated. **R-149.** `listen_22_8006` is **0**. The `fully-up` column is the converse proof: sshd + pveproxy already
- **MECHANISM (container, not PVE) — a locked root closes `sulogin`:** `Cannot open access to console, active, 3 listening. Zero credential window, measured.
the root account is locked.` for both `*` and `!`, with a working control (correct password -> - **R-148:** `answer.toml.tmpl:27` justifies `fully-up` with a `pvesh`/`pct` dependency the stub does
SU_OK, wrong -> Authentication failure). So "discard" and "lock" are the SAME outcome for recovery, not have (grep rc=1) — it blocked the ordering that is now measured as the fix.
which makes the escrow decision binary. - **R-149** three ordering values, not two · **R-150** Condition-guarded hooks skip silently ·
- **All four `proxmox-first-boot-*` units are `Condition*`-guarded, and a failed condition is a SKIP, **R-151** demo-felhom built from an uncommitted profile.
not a failure** — a hook that never ran looks exactly like one that succeeded. **R-150.** - Q4 from source: four gates in `iso-repack.sh` enforce the single-entry menu; `default`/`timeout`
- Q4: the single-entry menu is enforced by **four** gates in `iso-repack.sh` (`:153-155`, `:156`,
`:160-164`, `:314-319`), so restoring an interactive entry is a gate change, not a template edit —
but the banned-token gate would pass unchanged for a stock Graphical entry. `default`/`timeout` are
already settable in `grub/grub.cfg.tmpl`. already settable in `grub/grub.cfg.tmpl`.
The first mechanism probe was invalid (ran as uid 0, so `su` never invoked pam_unix; `sulogin` had no Three probes failed and are recorded as failed, not massaged: a container probe that ran as uid 0
tty) and is recorded as failed, not massaged. (so `su` never invoked pam_unix), a GRUB probe that missed the **1-second** menu timeout and typed into
the login prompt, and a kernel-line edit that landed one line off — caught by a pre-typing
verification screendump.
## SPIKE (no code change) — a universal ISO needs a different disk strategy and a locked root (2026-07-31) ## SPIKE (no code change) — a universal ISO needs a different disk strategy and a locked root (2026-07-31)