admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

SPIKE 2 complete: locked root closes the PVE web UI; before-network gives a measured zero window

Browse Source 
Findings only — no script, profile or build file changed; no ISO built, nothing published.
documentation/audits/SPIKE-universal-iso-2-2026-07-31.md

Both Tier 0 boxes went offline mid-session (provider cable fault; four routes tried, no Tier 2
fallback used) and returned. All three scenarios then ran to completion on real PVE, each signalled
by reboot-mode='power-off' rather than a disk hash.

- A LOCKED ROOT CLOSES THE PVE WEB INTERFACE. Measured at the exact endpoint the UI uses
  (POST /api2/json/access/ticket, root@pam) WITH A WORKING CONTROL: known-password install returns
  HTTP 200 + ticket; locked install returns 401 for every password and none can exist.
  passwd -S root = L, shadow = literal-asterisk, PVE uses the stock PAM stack.
- GRUB recovery mode is also closed ('the root account is locked') — but init=/bin/bash still gives
  an unauthenticated root@(none):/#. A locked box is recoverable, operator-only, at the console.
  The installed GRUB has NO password, so locking root is not a physical-security measure. R-152.
- before-network MEASURED (A/B, same image): the hook RUNS (marker, uptime 6.58s) with entropy 256,
  writable /etc, all binaries and openssl_rand_len=32, while ip_global is EMPTY and
  listen_22_8006 = 0. fully-up is the converse: sshd+pveproxy active, 3 listening. Zero window.
- R-148: answer.toml.tmpl:27 justifies fully-up with a pvesh/pct dependency the stub does not have
  (grep rc=1) — it blocked the ordering now measured as the fix.
- R-149 three ordering values; R-150 Condition-guarded hooks skip silently; R-151 demo-felhom built
  from an uncommitted profile.

Three probes failed and are recorded as failed: a container probe that ran as uid 0, a GRUB probe
that missed the 1-second menu timeout, and a kernel-line edit one line off (caught by a pre-typing
verification screendump). The interim 'Layer 1 teardown INCOMPLETE' is corrected — the fixture had
never landed, because the staging mkdir was in the SSH call that timed out.
This commit is contained in:
authentik Default Admin
2026-07-31 13:57:21 +02:00
parent 5bdd8372f8
commit 19c932a693
3 changed files with 424 additions and 388 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT-universal-iso-spike.md
+63 -69
View File
@@ -1,95 +1,89 @@
# REPORT — SPIKE 2: closing the last mechanisms, and the universal ISO's shape (2026-07-31)
> Written as `REPORT-universal-iso-spike.md`, not `REPORT.md`: the shared file belongs to today's hub
> v0.85.0 session and the standing rule is that the second session in a shared clone never touches it.
> This file supersedes its own Spike 1 contents.
> v0.85.0 session and the second session in a shared clone never touches it.
**Class: Spike.** Findings only — no production file changed, no ISO built, nothing published.
Evidence: `documentation/audits/SPIKE-universal-iso-2-2026-07-31.md`. Continues
`SPIKE-universal-iso-2026-07-31.md`.
Evidence: `documentation/audits/SPIKE-universal-iso-2-2026-07-31.md`.
## The headline: the venue went away, and three answers are blocked
**All four questions are answered**, and every answer that matters is measured on a real PVE install.
The session lost both Tier 0 boxes mid-run (provider cable fault; four routes tried, no Tier 2
fallback used) and resumed when connectivity returned.
At 12:28 CEST both Tier 0 boxes went offline **simultaneously** and did not return. Four routes were
tried (tailnet direct, `ssh demo-hp`, `demo-hp-lan` via ProxyJump, `ssh felhom-pve`); our tailscale
pod is healthy and `tailscale status` shows both peers `offline` with a rising `last seen`. Both boxes
are at the same remote site — that site's power or uplink, not fixable from here.
## The two results that decide the product questions
**No fallback venue was used.** DooPlex is Tier 2, the 2026-07-25 ruling moved drill VMs off it, and
this task did not authorise it — an absent fence is not permission. So **Q1, Q2 and Q3 each keep a
part that is BLOCKED, not answered**, and they are labelled that way throughout. The fixture is built
and staged with Spike 1's two measurement defects already corrected (a NIC present at install time; a
`reboot-mode = "power-off"` completion signal instead of a disk hash).
**A locked root closes the PVE web interface.** Measured against the exact endpoint the UI logs in
through — `POST /api2/json/access/ticket`, realm `root@pam` — **with a working control**: the
known-password install returns **HTTP 200 and a ticket**; the locked install returns 401 for every
password offered, and no password exists that could work. The control is what makes this a finding
rather than a broken probe. `passwd -S root` reports `L`, the shadow field is `literal-asterisk`, and
PVE uses the stock PAM stack.
## What was established anyway
**GRUB's recovery mode is closed too** — `Cannot open access to console, the root account is locked.`
— **but `init=/bin/bash` still works**, giving an unauthenticated `root@(none):/#`. So a locked-root
box is *not* unrecoverable, as the structural reading had feared; it is recoverable **operator-only, at
the physical console**. The installed GRUB has no password, which also means locking root is not a
physical-security measure and must never be described as one (**R-152**).
**The best result of the session, and it is structural rather than measured — which makes it
stronger.** `ordering = "before-network"` maps to `proxmox-first-boot-network-pre.service`
(`proxmox-first-boot` 9.2.5, read out of the ISO's own package), which is `Before=network-pre.target`
with `Type=oneshot`. It therefore runs **to completion before any network interface is configured**.
A credential rotation placed there has a **zero-length exposure window by construction, not by being
fast** — and that holds on every machine, unlike the timing number Spike 1 failed twice to obtain.
## The mechanism that fixes the credential window
**`before-network` measured, A/B, same image, only `ordering` differing.** The hook **runs** (positive
marker, uptime 6.58 s) with entropy 256, a writable `/etc`, every binary it needs and a demonstrated
`openssl_rand_len=32` — while `ip_global` is **empty** and `listen_22_8006` is **0**. The `fully-up`
column is the converse proof: sshd and pveproxy already active, 3 sockets listening. **Zero credential
window, measured**, and the shipped unit (`Before=network-pre.target`, `Type=oneshot`) generalises it
to every machine.
**And the reason it was never used is a comment that is wrong.** `answer.toml.tmpl:27` justifies
`fully-up` as needed "so pvesh/pct work" — but `stub-first-boot.sh` makes no `pvesh`/`pct`/`pveum`/`qm`
call at all (grep `rc=1`). That usage lives in `felhom-bootstrap.sh`, which runs under its own
`After=network-online.target … pveproxy.service` unit. A comment asserting a **dependency** that does
not exist, which has constrained the design ever since. **R-148.**
`fully-up` as needed "so pvesh/pct work", but `stub-first-boot.sh` makes no `pvesh`/`pct`/`pveum`/`qm`
call at all (grep `rc=1`); that usage lives in `felhom-bootstrap.sh`, under its own properly-ordered
unit. **R-148** — a comment asserting a *dependency* that does not exist, which blocked the fix.
**A locked root closes `sulogin`** — measured in an isolated container with a working control (correct
password → `SU_OK`, wrong → `Authentication failure`; then `*` and `!` both → `L` and
`Cannot open access to console, the root account is locked.`). This reshapes the escrow question: if
nobody holds the password, `sulogin` refuses either way, so **"discard" and "lock" are the same
outcome for recovery**, and the decision is binary. Caveat stated plainly: container, not PVE.
## Probes that failed, recorded as failed
**The ordering enum has three values, not two** — `before-network`, `network-online`, `fully-up`.
Spike 1 recorded two; the middle one has never been evaluated (**R-149**).
Three, none massaged: a container probe that ran as uid 0 so `su` never invoked `pam_unix` (rc=0 even
with a wrong password); a GRUB probe that missed the **1-second** menu timeout and typed into the login
prompt; and a kernel-line edit that landed one line off — caught by a pre-typing verification
screendump before it was booted. All were rerun.
**A first-boot hook that never runs is indistinguishable from one that succeeded** — all four
`proxmox-first-boot-*` units are `Condition*`-guarded and a failed condition is a *skip*, not a
failure. Any rotation-in-first-boot design must verify by a positive marker (**R-150**).
Also corrected: the interim report's "Layer 1 teardown INCOMPLETE" was over-cautious. When demo-hp
returned, `/mnt/nvme-1tb/spike2/` did not exist and disk usage was unchanged — the staging `mkdir` had
been inside the SSH call that timed out as the link degraded, so **the fixture never landed**.
**Q4 answered fully from source.** The single-entry menu is enforced by **four** gates in
`iso-repack.sh` (`:153-155`, `:156`, `:160-164`, and a post-remaster re-extraction at `:314-319`), so
restoring an interactive entry is a gate change rather than a template edit — but the banned-token
gate would pass unchanged for a stock *Graphical* entry, so the safety property survives. `default`
and `timeout` are already settable in `grub/grub.cfg.tmpl`.
## The operator's decisions
## What failed, and is recorded as failed
**Escrow versus discard — no longer blocked, and reshaped twice.** "Discard" and "lock" are the same
outcome (nobody holds the password, so both `sulogin` and the web UI refuse), and **neither makes a box
unrecoverable** because `init=/bin/bash` always works. So escrow buys only this: the operator keeps the
web-UI and `sulogin` routes without physical presence. Against that stands D6 — a hub holding console
root for the whole fleet is what the hub alone must not be. **I recommend discard/lock for the public
image**, escrow remaining defensible for operator-built boxes. Your call; it is no longer waiting on a
measurement.
The first mechanism probe produced **zero valid observations**: it ran everything as uid 0, so `su`
never invoked `pam_unix` (rc=0 even with a wrong password), `sulogin` died on `tcgetattr` with no tty,
and a `sed` mangled the control hash. Rerun with a non-root caller, a real pty, and `chpasswd`.
**The default boot entry — framed, unruled.** The disk evidence (Spike 1) and the recovery evidence
(this session) now point the same way: a tester whose unattended install succeeds but whose pairing
fails has no web UI and no recovery shell. The cost is the unattended experience. Two profiles from one
build is cheap, so operator-built and public need not match.
A **teardown error** is also recorded rather than quietly fixed: shredding the control password
destroyed the plaintext scenario C existed to use, so `C.toml` was deleted rather than left looking
usable. It must be regenerated before the PVE-web-interface measurement is attempted.
## Still open
## The two decisions, and why neither should be ruled yet
**Escrow versus discard** — now binary (see above). **Blocked on one measurement:** whether a locked
root also closes the PVE web interface. If it does, discarding removes the last remote route as well
as the local one and the balance shifts sharply toward escrow. **Do not rule before that is measured.**
**The default boot entry for a public image** — the disk evidence points at interactive-default; the
cost is the unattended experience. Partially blocked on the same measurement. The
operator-built-versus-public split is cheap and available regardless of the ruling.
One gap in Q3: **Felhom's real stub at `before-network`** is argued structurally, not observed — the
scenarios used the spike's own diagnostic as the hook. That is the one thing to measure before a spec
is written (~40 min). Also: the SSH-as-root leg used a wrong password in the control too, so it
discriminates nothing and is recorded as a gap rather than a result.
## R-rows
**R-148** (`fully-up` justified by a non-existent dependency) · **R-149** (three ordering values, not
two) · **R-150** (Condition-guarded first-boot hook skips silently) · **R-151** (demo-felhom was
installed from an **uncommitted** profile — a Tier 0 reference box is not reproducible from `main`).
**R-148** (`fully-up` justified by a non-existent dependency) · **R-149** (three ordering values) ·
**R-150** (Condition-guarded hooks skip silently) · **R-151** (demo-felhom built from an uncommitted
profile) · **R-152** (a locked root is not a physical-security measure).
## Teardown
**Layer 1 (demo-hp) is INCOMPLETE and named as such** — the box went offline while
`/mnt/nvme-1tb/spike2/` held the staged fixture. Nothing was running on it (last check: 0 QEMU
processes, no install started this session), nothing on `local-lvm`, no PVE storage registered,
`drill-r50` and 9201 never touched. The outstanding command is recorded in §10 of the findings doc
and **is not claimed as done** — R-131 exists because recorded commands never get run.
Layer 2 (demo-felhom): nothing placed this session. Layer 3 (DooPlex): scratchpad 4.4 GB → 48 K,
control password `shred -u`'d and verified absent, mechanism container `--rm`, no global Docker
cleanup, `felhom-iso/out/` untouched at 17 ISOs, no production file modified. **Hub-side: nothing
created** — verified by fetching the customer list and searching it, not asserted. Nothing published.
All three layers verified positively. demo-hp is back to its exact pre-spike state: `spike2/` and
`/tmp/paia` removed, 0 QEMU processes, 0 loop devices, **disk usage 6.6 G — identical to pre-spike**,
4 storages unchanged, nothing on `local-lvm`, `drill-r50` stopped and 9201 running, both untouched.
demo-felhom: probe binary removed, external USB drive observed only, still mounted as found. DooPlex:
scratchpad 1.7 GB → 60 K, control password `shred -u`'d and verified absent, `felhom-iso/out/`
untouched at 17 ISOs, no production file modified. **Hub-side: nothing created** — the VMs had no route
to it. Nothing published.