admin
19c932a693
Findings only — no script, profile or build file changed; no ISO built, nothing published.
documentation/audits/SPIKE-universal-iso-2-2026-07-31.md
Both Tier 0 boxes went offline mid-session (provider cable fault; four routes tried, no Tier 2
fallback used) and returned. All three scenarios then ran to completion on real PVE, each signalled
by reboot-mode='power-off' rather than a disk hash.
- A LOCKED ROOT CLOSES THE PVE WEB INTERFACE. Measured at the exact endpoint the UI uses
(POST /api2/json/access/ticket, root@pam) WITH A WORKING CONTROL: known-password install returns
HTTP 200 + ticket; locked install returns 401 for every password and none can exist.
passwd -S root = L, shadow = literal-asterisk, PVE uses the stock PAM stack.
- GRUB recovery mode is also closed ('the root account is locked') — but init=/bin/bash still gives
an unauthenticated root@(none):/#. A locked box is recoverable, operator-only, at the console.
The installed GRUB has NO password, so locking root is not a physical-security measure. R-152.
- before-network MEASURED (A/B, same image): the hook RUNS (marker, uptime 6.58s) with entropy 256,
writable /etc, all binaries and openssl_rand_len=32, while ip_global is EMPTY and
listen_22_8006 = 0. fully-up is the converse: sshd+pveproxy active, 3 listening. Zero window.
- R-148: answer.toml.tmpl:27 justifies fully-up with a pvesh/pct dependency the stub does not have
(grep rc=1) — it blocked the ordering now measured as the fix.
- R-149 three ordering values; R-150 Condition-guarded hooks skip silently; R-151 demo-felhom built
from an uncommitted profile.
Three probes failed and are recorded as failed: a container probe that ran as uid 0, a GRUB probe
that missed the 1-second menu timeout, and a kernel-line edit one line off (caught by a pre-typing
verification screendump). The interim 'Layer 1 teardown INCOMPLETE' is corrected — the fixture had
never landed, because the staging mkdir was in the SSH call that timed out.
90 lines
5.7 KiB
Markdown
90 lines
5.7 KiB
Markdown
# REPORT — SPIKE 2: closing the last mechanisms, and the universal ISO's shape (2026-07-31)
|
|
|
|
> Written as `REPORT-universal-iso-spike.md`, not `REPORT.md`: the shared file belongs to today's hub
|
|
> v0.85.0 session and the second session in a shared clone never touches it.
|
|
|
|
**Class: Spike.** Findings only — no production file changed, no ISO built, nothing published.
|
|
Evidence: `documentation/audits/SPIKE-universal-iso-2-2026-07-31.md`.
|
|
|
|
**All four questions are answered**, and every answer that matters is measured on a real PVE install.
|
|
The session lost both Tier 0 boxes mid-run (provider cable fault; four routes tried, no Tier 2
|
|
fallback used) and resumed when connectivity returned.
|
|
|
|
## The two results that decide the product questions
|
|
|
|
**A locked root closes the PVE web interface.** Measured against the exact endpoint the UI logs in
|
|
through — `POST /api2/json/access/ticket`, realm `root@pam` — **with a working control**: the
|
|
known-password install returns **HTTP 200 and a ticket**; the locked install returns 401 for every
|
|
password offered, and no password exists that could work. The control is what makes this a finding
|
|
rather than a broken probe. `passwd -S root` reports `L`, the shadow field is `literal-asterisk`, and
|
|
PVE uses the stock PAM stack.
|
|
|
|
**GRUB's recovery mode is closed too** — `Cannot open access to console, the root account is locked.`
|
|
— **but `init=/bin/bash` still works**, giving an unauthenticated `root@(none):/#`. So a locked-root
|
|
box is *not* unrecoverable, as the structural reading had feared; it is recoverable **operator-only, at
|
|
the physical console**. The installed GRUB has no password, which also means locking root is not a
|
|
physical-security measure and must never be described as one (**R-152**).
|
|
|
|
## The mechanism that fixes the credential window
|
|
|
|
**`before-network` measured, A/B, same image, only `ordering` differing.** The hook **runs** (positive
|
|
marker, uptime 6.58 s) with entropy 256, a writable `/etc`, every binary it needs and a demonstrated
|
|
`openssl_rand_len=32` — while `ip_global` is **empty** and `listen_22_8006` is **0**. The `fully-up`
|
|
column is the converse proof: sshd and pveproxy already active, 3 sockets listening. **Zero credential
|
|
window, measured**, and the shipped unit (`Before=network-pre.target`, `Type=oneshot`) generalises it
|
|
to every machine.
|
|
|
|
**And the reason it was never used is a comment that is wrong.** `answer.toml.tmpl:27` justifies
|
|
`fully-up` as needed "so pvesh/pct work", but `stub-first-boot.sh` makes no `pvesh`/`pct`/`pveum`/`qm`
|
|
call at all (grep `rc=1`); that usage lives in `felhom-bootstrap.sh`, under its own properly-ordered
|
|
unit. **R-148** — a comment asserting a *dependency* that does not exist, which blocked the fix.
|
|
|
|
## Probes that failed, recorded as failed
|
|
|
|
Three, none massaged: a container probe that ran as uid 0 so `su` never invoked `pam_unix` (rc=0 even
|
|
with a wrong password); a GRUB probe that missed the **1-second** menu timeout and typed into the login
|
|
prompt; and a kernel-line edit that landed one line off — caught by a pre-typing verification
|
|
screendump before it was booted. All were rerun.
|
|
|
|
Also corrected: the interim report's "Layer 1 teardown INCOMPLETE" was over-cautious. When demo-hp
|
|
returned, `/mnt/nvme-1tb/spike2/` did not exist and disk usage was unchanged — the staging `mkdir` had
|
|
been inside the SSH call that timed out as the link degraded, so **the fixture never landed**.
|
|
|
|
## The operator's decisions
|
|
|
|
**Escrow versus discard — no longer blocked, and reshaped twice.** "Discard" and "lock" are the same
|
|
outcome (nobody holds the password, so both `sulogin` and the web UI refuse), and **neither makes a box
|
|
unrecoverable** because `init=/bin/bash` always works. So escrow buys only this: the operator keeps the
|
|
web-UI and `sulogin` routes without physical presence. Against that stands D6 — a hub holding console
|
|
root for the whole fleet is what the hub alone must not be. **I recommend discard/lock for the public
|
|
image**, escrow remaining defensible for operator-built boxes. Your call; it is no longer waiting on a
|
|
measurement.
|
|
|
|
**The default boot entry — framed, unruled.** The disk evidence (Spike 1) and the recovery evidence
|
|
(this session) now point the same way: a tester whose unattended install succeeds but whose pairing
|
|
fails has no web UI and no recovery shell. The cost is the unattended experience. Two profiles from one
|
|
build is cheap, so operator-built and public need not match.
|
|
|
|
## Still open
|
|
|
|
One gap in Q3: **Felhom's real stub at `before-network`** is argued structurally, not observed — the
|
|
scenarios used the spike's own diagnostic as the hook. That is the one thing to measure before a spec
|
|
is written (~40 min). Also: the SSH-as-root leg used a wrong password in the control too, so it
|
|
discriminates nothing and is recorded as a gap rather than a result.
|
|
|
|
## R-rows
|
|
|
|
**R-148** (`fully-up` justified by a non-existent dependency) · **R-149** (three ordering values) ·
|
|
**R-150** (Condition-guarded hooks skip silently) · **R-151** (demo-felhom built from an uncommitted
|
|
profile) · **R-152** (a locked root is not a physical-security measure).
|
|
|
|
## Teardown
|
|
|
|
All three layers verified positively. demo-hp is back to its exact pre-spike state: `spike2/` and
|
|
`/tmp/paia` removed, 0 QEMU processes, 0 loop devices, **disk usage 6.6 G — identical to pre-spike**,
|
|
4 storages unchanged, nothing on `local-lvm`, `drill-r50` stopped and 9201 running, both untouched.
|
|
demo-felhom: probe binary removed, external USB drive observed only, still mounted as found. DooPlex:
|
|
scratchpad 1.7 GB → 60 K, control password `shred -u`'d and verified absent, `felhom-iso/out/`
|
|
untouched at 17 ISOs, no production file modified. **Hub-side: nothing created** — the VMs had no route
|
|
to it. Nothing published.
|