diff --git a/REPORT-universal-iso-spike.md b/REPORT-universal-iso-spike.md index 532188e..2e76425 100644 --- a/REPORT-universal-iso-spike.md +++ b/REPORT-universal-iso-spike.md @@ -1,95 +1,89 @@ # REPORT — SPIKE 2: closing the last mechanisms, and the universal ISO's shape (2026-07-31) > Written as `REPORT-universal-iso-spike.md`, not `REPORT.md`: the shared file belongs to today's hub -> v0.85.0 session and the standing rule is that the second session in a shared clone never touches it. -> This file supersedes its own Spike 1 contents. +> v0.85.0 session and the second session in a shared clone never touches it. **Class: Spike.** Findings only — no production file changed, no ISO built, nothing published. -Evidence: `documentation/audits/SPIKE-universal-iso-2-2026-07-31.md`. Continues -`SPIKE-universal-iso-2026-07-31.md`. +Evidence: `documentation/audits/SPIKE-universal-iso-2-2026-07-31.md`. -## The headline: the venue went away, and three answers are blocked +**All four questions are answered**, and every answer that matters is measured on a real PVE install. +The session lost both Tier 0 boxes mid-run (provider cable fault; four routes tried, no Tier 2 +fallback used) and resumed when connectivity returned. -At 12:28 CEST both Tier 0 boxes went offline **simultaneously** and did not return. Four routes were -tried (tailnet direct, `ssh demo-hp`, `demo-hp-lan` via ProxyJump, `ssh felhom-pve`); our tailscale -pod is healthy and `tailscale status` shows both peers `offline` with a rising `last seen`. Both boxes -are at the same remote site — that site's power or uplink, not fixable from here. +## The two results that decide the product questions -**No fallback venue was used.** DooPlex is Tier 2, the 2026-07-25 ruling moved drill VMs off it, and -this task did not authorise it — an absent fence is not permission. So **Q1, Q2 and Q3 each keep a -part that is BLOCKED, not answered**, and they are labelled that way throughout. The fixture is built -and staged with Spike 1's two measurement defects already corrected (a NIC present at install time; a -`reboot-mode = "power-off"` completion signal instead of a disk hash). +**A locked root closes the PVE web interface.** Measured against the exact endpoint the UI logs in +through — `POST /api2/json/access/ticket`, realm `root@pam` — **with a working control**: the +known-password install returns **HTTP 200 and a ticket**; the locked install returns 401 for every +password offered, and no password exists that could work. The control is what makes this a finding +rather than a broken probe. `passwd -S root` reports `L`, the shadow field is `literal-asterisk`, and +PVE uses the stock PAM stack. -## What was established anyway +**GRUB's recovery mode is closed too** — `Cannot open access to console, the root account is locked.` +— **but `init=/bin/bash` still works**, giving an unauthenticated `root@(none):/#`. So a locked-root +box is *not* unrecoverable, as the structural reading had feared; it is recoverable **operator-only, at +the physical console**. The installed GRUB has no password, which also means locking root is not a +physical-security measure and must never be described as one (**R-152**). -**The best result of the session, and it is structural rather than measured — which makes it -stronger.** `ordering = "before-network"` maps to `proxmox-first-boot-network-pre.service` -(`proxmox-first-boot` 9.2.5, read out of the ISO's own package), which is `Before=network-pre.target` -with `Type=oneshot`. It therefore runs **to completion before any network interface is configured**. -A credential rotation placed there has a **zero-length exposure window by construction, not by being -fast** — and that holds on every machine, unlike the timing number Spike 1 failed twice to obtain. +## The mechanism that fixes the credential window + +**`before-network` measured, A/B, same image, only `ordering` differing.** The hook **runs** (positive +marker, uptime 6.58 s) with entropy 256, a writable `/etc`, every binary it needs and a demonstrated +`openssl_rand_len=32` — while `ip_global` is **empty** and `listen_22_8006` is **0**. The `fully-up` +column is the converse proof: sshd and pveproxy already active, 3 sockets listening. **Zero credential +window, measured**, and the shipped unit (`Before=network-pre.target`, `Type=oneshot`) generalises it +to every machine. **And the reason it was never used is a comment that is wrong.** `answer.toml.tmpl:27` justifies -`fully-up` as needed "so pvesh/pct work" — but `stub-first-boot.sh` makes no `pvesh`/`pct`/`pveum`/`qm` -call at all (grep `rc=1`). That usage lives in `felhom-bootstrap.sh`, which runs under its own -`After=network-online.target … pveproxy.service` unit. A comment asserting a **dependency** that does -not exist, which has constrained the design ever since. **R-148.** +`fully-up` as needed "so pvesh/pct work", but `stub-first-boot.sh` makes no `pvesh`/`pct`/`pveum`/`qm` +call at all (grep `rc=1`); that usage lives in `felhom-bootstrap.sh`, under its own properly-ordered +unit. **R-148** — a comment asserting a *dependency* that does not exist, which blocked the fix. -**A locked root closes `sulogin`** — measured in an isolated container with a working control (correct -password → `SU_OK`, wrong → `Authentication failure`; then `*` and `!` both → `L` and -`Cannot open access to console, the root account is locked.`). This reshapes the escrow question: if -nobody holds the password, `sulogin` refuses either way, so **"discard" and "lock" are the same -outcome for recovery**, and the decision is binary. Caveat stated plainly: container, not PVE. +## Probes that failed, recorded as failed -**The ordering enum has three values, not two** — `before-network`, `network-online`, `fully-up`. -Spike 1 recorded two; the middle one has never been evaluated (**R-149**). +Three, none massaged: a container probe that ran as uid 0 so `su` never invoked `pam_unix` (rc=0 even +with a wrong password); a GRUB probe that missed the **1-second** menu timeout and typed into the login +prompt; and a kernel-line edit that landed one line off — caught by a pre-typing verification +screendump before it was booted. All were rerun. -**A first-boot hook that never runs is indistinguishable from one that succeeded** — all four -`proxmox-first-boot-*` units are `Condition*`-guarded and a failed condition is a *skip*, not a -failure. Any rotation-in-first-boot design must verify by a positive marker (**R-150**). +Also corrected: the interim report's "Layer 1 teardown INCOMPLETE" was over-cautious. When demo-hp +returned, `/mnt/nvme-1tb/spike2/` did not exist and disk usage was unchanged — the staging `mkdir` had +been inside the SSH call that timed out as the link degraded, so **the fixture never landed**. -**Q4 answered fully from source.** The single-entry menu is enforced by **four** gates in -`iso-repack.sh` (`:153-155`, `:156`, `:160-164`, and a post-remaster re-extraction at `:314-319`), so -restoring an interactive entry is a gate change rather than a template edit — but the banned-token -gate would pass unchanged for a stock *Graphical* entry, so the safety property survives. `default` -and `timeout` are already settable in `grub/grub.cfg.tmpl`. +## The operator's decisions -## What failed, and is recorded as failed +**Escrow versus discard — no longer blocked, and reshaped twice.** "Discard" and "lock" are the same +outcome (nobody holds the password, so both `sulogin` and the web UI refuse), and **neither makes a box +unrecoverable** because `init=/bin/bash` always works. So escrow buys only this: the operator keeps the +web-UI and `sulogin` routes without physical presence. Against that stands D6 — a hub holding console +root for the whole fleet is what the hub alone must not be. **I recommend discard/lock for the public +image**, escrow remaining defensible for operator-built boxes. Your call; it is no longer waiting on a +measurement. -The first mechanism probe produced **zero valid observations**: it ran everything as uid 0, so `su` -never invoked `pam_unix` (rc=0 even with a wrong password), `sulogin` died on `tcgetattr` with no tty, -and a `sed` mangled the control hash. Rerun with a non-root caller, a real pty, and `chpasswd`. +**The default boot entry — framed, unruled.** The disk evidence (Spike 1) and the recovery evidence +(this session) now point the same way: a tester whose unattended install succeeds but whose pairing +fails has no web UI and no recovery shell. The cost is the unattended experience. Two profiles from one +build is cheap, so operator-built and public need not match. -A **teardown error** is also recorded rather than quietly fixed: shredding the control password -destroyed the plaintext scenario C existed to use, so `C.toml` was deleted rather than left looking -usable. It must be regenerated before the PVE-web-interface measurement is attempted. +## Still open -## The two decisions, and why neither should be ruled yet - -**Escrow versus discard** — now binary (see above). **Blocked on one measurement:** whether a locked -root also closes the PVE web interface. If it does, discarding removes the last remote route as well -as the local one and the balance shifts sharply toward escrow. **Do not rule before that is measured.** - -**The default boot entry for a public image** — the disk evidence points at interactive-default; the -cost is the unattended experience. Partially blocked on the same measurement. The -operator-built-versus-public split is cheap and available regardless of the ruling. +One gap in Q3: **Felhom's real stub at `before-network`** is argued structurally, not observed — the +scenarios used the spike's own diagnostic as the hook. That is the one thing to measure before a spec +is written (~40 min). Also: the SSH-as-root leg used a wrong password in the control too, so it +discriminates nothing and is recorded as a gap rather than a result. ## R-rows -**R-148** (`fully-up` justified by a non-existent dependency) · **R-149** (three ordering values, not -two) · **R-150** (Condition-guarded first-boot hook skips silently) · **R-151** (demo-felhom was -installed from an **uncommitted** profile — a Tier 0 reference box is not reproducible from `main`). +**R-148** (`fully-up` justified by a non-existent dependency) · **R-149** (three ordering values) · +**R-150** (Condition-guarded hooks skip silently) · **R-151** (demo-felhom built from an uncommitted +profile) · **R-152** (a locked root is not a physical-security measure). ## Teardown -**Layer 1 (demo-hp) is INCOMPLETE and named as such** — the box went offline while -`/mnt/nvme-1tb/spike2/` held the staged fixture. Nothing was running on it (last check: 0 QEMU -processes, no install started this session), nothing on `local-lvm`, no PVE storage registered, -`drill-r50` and 9201 never touched. The outstanding command is recorded in §10 of the findings doc -and **is not claimed as done** — R-131 exists because recorded commands never get run. - -Layer 2 (demo-felhom): nothing placed this session. Layer 3 (DooPlex): scratchpad 4.4 GB → 48 K, -control password `shred -u`'d and verified absent, mechanism container `--rm`, no global Docker -cleanup, `felhom-iso/out/` untouched at 17 ISOs, no production file modified. **Hub-side: nothing -created** — verified by fetching the customer list and searching it, not asserted. Nothing published. +All three layers verified positively. demo-hp is back to its exact pre-spike state: `spike2/` and +`/tmp/paia` removed, 0 QEMU processes, 0 loop devices, **disk usage 6.6 G — identical to pre-spike**, +4 storages unchanged, nothing on `local-lvm`, `drill-r50` stopped and 9201 running, both untouched. +demo-felhom: probe binary removed, external USB drive observed only, still mounted as found. DooPlex: +scratchpad 1.7 GB → 60 K, control password `shred -u`'d and verified absent, `felhom-iso/out/` +untouched at 17 ISOs, no production file modified. **Hub-side: nothing created** — the VMs had no route +to it. Nothing published. diff --git a/documentation/audits/SPIKE-universal-iso-2-2026-07-31.md b/documentation/audits/SPIKE-universal-iso-2-2026-07-31.md index d2b9fc1..40075b1 100644 --- a/documentation/audits/SPIKE-universal-iso-2-2026-07-31.md +++ b/documentation/audits/SPIKE-universal-iso-2-2026-07-31.md @@ -4,201 +4,290 @@ > > Continues `SPIKE-universal-iso-2026-07-31.md`; its results are premises here and are not re-derived. > -> **Read §0 first: the venue went away mid-session.** Both Tier 0 hosts dropped off the tailnet at -> ~12:28 CEST and did not return. Q1, Q2 and Q3 each have a part that needs a nested VM, and those -> parts are **BLOCKED, not answered**. What could be established without them — from the shipped PVE -> source, the answer schema, and an isolated mechanism probe — is below and is labelled by how it was -> obtained. +> **All four questions are answered, and every answer that mattered is measured on a real PVE install.** +> The session lost its venue mid-run (§0) and resumed when connectivity was restored. --- -## 0. Venue loss — what was tried +## 0. Venue loss and recovery — what was tried -At 12:28 CEST, mid-run, both Tier 0 boxes went offline **simultaneously**: - -``` -100.76.96.79 demo-hp active; relay "fra"; offline, last seen 5m ago, tx 1843821564 rx 63739848 -100.70.170.35 felhom-pve active; relay "waw"; offline, last seen 5m ago, tx 1560 rx 0 -``` - -Attempts, in order (per the standing rule that a "no access" claim names what it tried): +At 12:28 CEST both Tier 0 boxes went offline **simultaneously**. Routes attempted, in order: | # | Route | Result | |---|---|---| -| 1 | `ping 100.76.96.79` (tailnet direct) | 100 % packet loss | -| 2 | `ssh demo-hp` (tailnet, `ConnectTimeout=10`) | `connect to host 100.76.96.79 port 22: Connection timed out` | -| 3 | `ssh demo-hp-lan` (LAN `192.168.0.87`, ProxyJump `felhom-pve`) | `Connection timed out during banner exchange` | -| 4 | `ssh felhom-pve` (the jump host itself) | `connect to host 100.70.170.35 port 22: Connection timed out` | -| 5 | DooPlex tailscale pod health | pod `tailscale-c84df7f84-tw2wz` **Running 6d10h**, healthy; log shows `open-conn-track: timeout … online=yes` then peers dropping | -| 6 | `tailscale status` from the pod | **both peers `offline`**, `last seen 5m` and rising | +| 1 | `ping 100.76.96.79` (tailnet direct) | 100 % loss | +| 2 | `ssh demo-hp` | `Connection timed out` | +| 3 | `ssh demo-hp-lan` (LAN, ProxyJump `felhom-pve`) | `Connection timed out during banner exchange` | +| 4 | `ssh felhom-pve` (the jump host) | `Connection timed out` | +| 5 | DooPlex tailscale pod | `Running 6d10h`, healthy | +| 6 | `tailscale status` from the pod | **both peers `offline`**, `last seen` rising | -**Diagnosis: the far end, not ours.** Our tailscale pod is healthy and every other peer route behaves; -both demo boxes are at the same remote site (REMOTE until ~08-02) and went offline together, which is -that site's power or uplink. Not fixable from here. +Far end, not ours. **No fallback venue was used** — DooPlex is Tier 2, the 2026-07-25 ruling moved +drill VMs off it, and this task did not authorise it. An absent fence is not permission. -**No fallback venue was used.** DooPlex is Tier 2 and the 2026-07-25 standing ruling moved drill and -build VMs off it; `target-selection.md` says a Tier 2 machine is used only when a task says so -explicitly, and this one does not. **An absent fence is not permission**, so the probes simply did not -run. A watch is armed for the boxes' return. +Connectivity was restored (operator: a provider cable fault nearby) and the boxes returned with +**uptime 7 days** — a pure network outage, no reboot. All probes then ran. -The probe fixture is built and staged and will run unchanged when they come back: probe ISO -(`prepare-iso --fetch-from partition`, serial-capture first-boot diagnostic), three validated answer -files (locked/`fully-up`, locked/`before-network`, known-password control), an install runner with -`reboot-mode = "power-off"` as the completion signal, and a boot runner with protocol-level detectors -and **a NIC present at install time** — the two fixture defects that invalidated Spike 1's -measurements are already corrected. +**One thing the outage revealed.** `/mnt/nvme-1tb/spike2/` did not exist when demo-hp came back, and +disk usage was unchanged at 6.6 GB: the staging `mkdir` had been inside the SSH call that timed out as +the link degraded, so **the fixture never landed and Layer 1 had nothing to clean**. The interim +report's "Layer 1 teardown INCOMPLETE" was therefore over-cautious rather than wrong — corrected here, +and the fixture was rebuilt from scratch. --- ## 1. Baselines -Read fresh this session. - | Baseline | Value | |---|---| | `ISO_VERSION` | `1.25.0` — `scripts/iso/build-felhom-iso.sh:44` | | `SCRIPT_VERSION` | `1.22.0` — `scripts/felhom-host-install.sh:187` | | `felhom-bootstrap.sh` @ HEAD | `21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7` | | PVE base | `proxmox-ve_9.2-1.iso`, `4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c` | -| `felhom.eu` HEAD | **`ea00976`**, clean, `== origin/main` (the expected Spike 1 commit) | +| `felhom.eu` HEAD | **`ea00976`**, clean, `== origin/main` | | `proxmox-first-boot` pkg | `9.2.5` (from the ISO's `/proxmox/packages/`) | +| installed guest kernel | `7.0.2-6-pve`, GRUB `2.12-9+pmx2` | -**Fixture provenance.** Probe ISO built on DooPlex from the baseline PVE ISO. Nested VMs were to run -under plain QEMU/KVM on demo-hp under `/mnt/nvme-1tb/spike2/` — never `local-lvm`, no PVE storage -registered. They did not run (§0). The mechanism probe in §2.2 ran in a throwaway `debian:trixie` -Docker container on DooPlex, which is not an install and touches no box. +**Fixture provenance.** Probe ISO built on DooPlex from the baseline PVE ISO +(`prepare-iso --fetch-from partition --partition-label proxmox-ais --on-first-boot diag.sh`), +sha256 `dd97dba5518c70baf4300b119e1042eabf1012d5a046082acf31e60983efaeb9`, **verified byte-identical at +both ends** after transfer. Nested VMs ran under plain QEMU/KVM on **demo-hp** (Tier 0, the designated +drill host) under `/mnt/nvme-1tb/spike2/` — never `local-lvm`, no PVE storage registered, `drill-r50` +and guest 9201 untouched. -**Evidence classes used throughout:** **[MEASURED-PVE]** against the real PVE tooling or a PVE install · -**[MEASURED-MECH]** measured, but in an isolated container rather than on PVE · **[STRUCTURAL]** read -from shipped source or unit files · **[BLOCKED]** needs the nested VM. +**Both of Spike 1's fixture defects are corrected in this one:** a NIC is present **at install time** +(its absence made the installed system's interface config unmatchable at boot), and completion is +signalled by `reboot-mode = "power-off"` — the guest powering itself off — never by a changed disk hash. + +Three scenarios, all installed to completion: + +| Scenario | root credential | `[first-boot].ordering` | Install | +|---|---|---|---| +| **A** | `root-password-hashed = "*"` (locked) | `fully-up` | completed, powered off after **191 s** | +| **B** | `root-password-hashed = "*"` (locked) | `before-network` | completed, powered off after **190 s** | +| **C** | a **known** password (control) | `fully-up` | completed, powered off after **191 s** | + +**Evidence classes:** **[MEASURED-PVE]** on a real PVE install · **[MEASURED-MECH]** measured in an +isolated container · **[STRUCTURAL]** read from shipped source or unit files. --- ## 2. Q1 — is a locked root actually locked, and what else does it lock out? -### 2.1 The account's state on a real PVE install — **[BLOCKED]** +### 2.1 The account's real state — **[MEASURED-PVE]** -Needs the nested VM. The fixture is staged. Spike 1 established only that -`root-password-hashed = "*"` **installs to completion**; the resulting account state was explicitly -left open there and remains open. +From the first-boot diagnostic on both locked installs (A and B), emitted to `ttyS0`: -### 2.2 What can authenticate — **[MEASURED-MECH]**, and the first attempt was invalid +``` +passwd_S_root : root L 2026-07-31 0 99999 7 -1 +shadow_shape : literal-asterisk +sshd_PermitRoot : yes +pam_common_auth : 1 +pve_pam_realm : 0 +``` -**The first probe produced nothing and is recorded as failed.** It ran everything as uid 0, so `su` -never invoked `pam_unix` at all (`rc=0` even with a deliberately wrong password); `sulogin` died on -`tcgetattr failed: Invalid argument` with no TTY; and a `sed` mangled the control hash so the control -case had no password set. Four cases, zero valid observations. Rerun with a non-root caller, a real -pty via `script`, and `chpasswd` instead of `sed`. +**The account is genuinely locked** — `passwd -S` reports `L`, and the `*` reached `/etc/shadow` +verbatim. Note `sshd_PermitRoot: yes`: PVE ships `PermitRootLogin yes`, so the lock is doing the work, +not sshd's configuration. -**Corrected probe** (`debian:trixie`, `util-linux 2.41`, `pam_unix`) — with a working control: +`pve_pam_realm: 0` — there is no PVE-specific file in `/etc/pam.d/` matching `proxmox|pve`, and +`pam_unix` is present in `common-auth`. **PVE uses the stock PAM stack**, which is why the §2.2 +container measurement transfers to it rather than merely resembling it. -| Shadow field | `passwd -S root` | `su root` as non-root user | `sulogin` (real pty) | +### 2.2 What can authenticate — **[MEASURED-MECH]**, first attempt invalid + +**The first probe produced nothing and is recorded as failed:** it ran everything as uid 0 so `su` +never invoked `pam_unix` (`rc=0` even with a deliberately wrong password), `sulogin` died on +`tcgetattr failed` with no TTY, and a `sed` mangled the control hash. Four cases, zero observations. + +Corrected (`debian:trixie`, `util-linux 2.41`, non-root caller, real pty via `script`, `chpasswd`): + +| Shadow field | `passwd -S root` | `su root` as a non-root user | `sulogin` (real pty) | |---|---|---|---| -| real hash (control) | `P` | **correct pw → `SU_OK`**; wrong pw → `Authentication failure` | — | -| `*` | **`L`** | `su: Authentication failure` (any password) | **`Cannot open access to console, the root account is locked.`** | -| `!` | **`L`** | `su: Authentication failure` (any password) | **`Cannot open access to console, the root account is locked.`** | +| real hash (**control**) | `P` | **correct pw → `SU_OK`**; wrong pw → `Authentication failure` | — | +| `*` | `L` | `Authentication failure` (any password) | `Cannot open access to console, the root account is locked.` | +| `!` | `L` | `Authentication failure` (any password) | `Cannot open access to console, the root account is locked.` | -The control is what makes this readable: the same harness authenticates successfully with the right -password and fails with the wrong one, so the two locked rows are the module refusing, not the harness -misfiring. +The control makes it readable: the same harness succeeds with the right password and fails with the +wrong one, so the locked rows are the module refusing rather than the harness misfiring. **`*` and `!` +are equivalent.** -**So, at mechanism level: `*` and `!` are equivalent, both report `L`, and no password authenticates.** +**On PVE, root SSH was refused on every scenario** — but that row is **uninformative and is not +evidence**: the probe supplied a deliberately wrong password in the control scenario too, so it cannot +distinguish "locked" from "wrong password". Recorded as a gap, not a result. -**Caveat, and it is not small.** This is Debian's `pam_unix` and `util-linux`'s `sulogin` — the same -components PVE ships — but it is **not PVE**. PVE could ship PAM configuration, a `sulogin` override, -or a `SYSTEMD_SULOGIN_FORCE` setting that changes the outcome. Confirming on a real PVE install is -part of the blocked work. +### 2.3 Does the Proxmox web interface still work? — **[MEASURED-PVE]. No. It is closed.** -### 2.3 Does the Proxmox web interface still work? — **[BLOCKED]**, and this is the product question +The product-level answer, measured against the **exact endpoint the web UI logs in through** +(`POST /api2/json/access/ticket`, realm `root@pam`), with a working control: -**No measurement was obtained, so no answer is given.** Stating the reasoning without the measurement -would be exactly the failure mode this spike exists to avoid. +| Scenario | Password offered | Result | +|---|---|---| +| **C (control, known password)** | **correct** | **HTTP 200, TICKET ISSUED — login would SUCCEED** | +| C (control) | wrong | HTTP 401, no ticket | +| C (control) | empty | HTTP 401, no ticket | +| **A (locked root)** | wrong | HTTP 401, no ticket | +| **A (locked root)** | empty | HTTP 401, no ticket | -What is established: PVE's `root@pam` realm authenticates the Linux root account through PAM, and -§2.2 measured that `pam_unix` refuses a locked account for a non-root caller. If PVE's proxy -authenticates the same way — **which was not verified** — then a locked root means **nobody can log -into the PVE web interface at all**, neither customer nor operator, and that is a product decision -rather than a detail. +The control is what makes this a finding rather than a broken probe: **the same endpoint on the same +image demonstrably issues a ticket when the password is right.** On the locked install no password can +produce one, because no password exists. -The staged control scenario exists precisely to settle this: install C carries a **known** password, so -the PVE API ticket endpoint can be exercised positively (correct password → ticket) before install A -(locked) is tested against the same endpoint. Without that control a failure is indistinguishable -from a broken probe — Spike 1's lesson. +**Therefore a locked root closes the PVE web interface completely — to the customer and to the +operator alike.** This is a product decision, not a detail: it removes the browser-based route into +the box entirely, on top of the console route (§3.2). -### 2.4 Does anything else in PVE break? — **[BLOCKED]** +Reachability, incidentally measured on all three: **SSH banner at +20 s, pveproxy TLS at +26–29 s** +after power-on. -Not investigated. Cron, `pveproxy`, backup jobs and anything else assuming a usable root credential -remain unexamined. +### 2.4 Does anything else in PVE break? — **not investigated** + +Cron, backup jobs and other consumers of a root credential were not examined. `pveproxy` itself runs +and serves TLS (it answered `/api2/json/version`), so the daemon is unaffected — only authentication +through it is. --- ## 3. Q2 — is there a way back into a box whose setup failed? -### 3.1 Does the installed GRUB expose a recovery entry? — **[BLOCKED]** +### 3.1 Does the installed GRUB expose a recovery entry? — **[MEASURED-PVE]. Yes.** -Needs a boot. Note the installed system's GRUB is **PVE's own**, written by the installer — not the -ISO's Felhom menu — so §5's single-entry collapse does not apply to it. What it offers is unverified. - -### 3.2 `sulogin` with a locked root — **[MEASURED-MECH]. It refuses.** - -The single observation the escrow-versus-discard ruling was said to rest on: +The installed system's GRUB is PVE's own, unaffected by Felhom's ISO menu collapse (§4). Screendumped +from the locked install: ``` +*Proxmox VE GNU/Linux + Advanced options for Proxmox VE GNU/Linux + Memory test (memtest86+x64.bin) + Memory test (memtest86+x64.bin, serial console) +``` + +and inside *Advanced options*: + +``` +*Proxmox VE GNU/Linux, with Linux 7.0.2-6-pve + Proxmox VE GNU/Linux, with Linux 7.0.2-6-pve (recovery mode) +``` + +**A recovery entry exists, the menu is editable (`e`), a GRUB command line is available (`c`), and +there is no GRUB password.** Its timeout is **1 second** — short enough that a first probe missed it +entirely (§3.5). + +### 3.2 `sulogin` with a locked root — **[MEASURED-PVE]. It refuses.** + +Booting *(recovery mode)* on the locked install reaches rescue mode and then stops: + +``` +Started rescue.service - Rescue Shell. +Reached target rescue.target - Rescue Mode. +You are in rescue mode. After logging in, type "journalctl -xb" … + Cannot open access to console, the root account is locked. See sulogin(8) man page for more details. + +Press Enter to continue. ``` -Both for `*` and for `!`, with a real pty. **The standard Debian/systemd emergency and single-user -path does not grant a shell on a locked-root system.** `systemd` invokes `sulogin` for -`emergency.target` and `rescue.target`, so both are closed by the same refusal. +**The advertised recovery path is closed on a locked-root box** — verbatim the container result, +now confirmed on real PVE. `systemd` invokes `sulogin` for both `rescue.target` and +`emergency.target`, so both are closed by the same refusal. -Subject to the §2.2 caveat: measured in a container, not on PVE. +### 3.3 What is left — **[MEASURED-PVE]. `init=/bin/bash` works.** -### 3.3 What is left if `sulogin` refuses — **[STRUCTURAL]**, not measured +Appending `init=/bin/bash` to the kernel line in the GRUB editor and booting produced: -| Path | Realistic for whom | -|---|---| -| `init=/bin/bash` on the kernel line (GRUB `e`) | **Operator only.** It bypasses `sulogin` because PID 1 becomes the shell — no authentication is involved. Requires editing a boot entry at the console. Not a non-technical customer action. | -| `systemd.unit=emergency.target` | **Closed** — it lands in `sulogin`, which refuses (§3.2). | -| Live USB / rescue media | Operator only; requires physical media and knowing to do it. | -| PVE web console (`:8006`) | **Unknown, possibly closed** — §2.3. | -| SSH by key | Only if a key was baked, which for a public image is the thing R-142 says must not happen. | +``` +Found volume group "pve" using metadata type lvm2 +3 logical volume(s) in volume group "pve" now active +/dev/mapper/pve-root: clean, 60332/561936 files, 1141450/2243584 blocks +bash: cannot set terminal process group (-1): Inappropriate ioctl for device +bash: no job control in this shell +root@(none):/# +``` -**The honest summary:** on a locked-root box, every remaining route is operator-grade and physical. -For a customer whose pairing failed, there is no self-service path back — which is precisely the -objection raised when locked-root was first proposed, and §3.2 is the first evidence that the -objection is well founded rather than speculative. +**An unauthenticated root shell on a locked-root system.** No password is involved because PID 1 *is* +the shell — `sulogin` is never reached. -### 3.4 Does locked differ from rotated? — **[PARTIALLY ESTABLISHED]** +| Path | Works? | Realistic for whom | +|---|---|---| +| PVE web console (`:8006`) | **No** (§2.3) | — | +| GRUB *(recovery mode)* → `sulogin` | **No** (§3.2) | — | +| GRUB editor → `init=/bin/bash` | **Yes** (measured) | **Operator only.** Requires catching a 1-second menu, editing a kernel line, and knowing the incantation. Not a non-technical customer action. | +| Live USB / rescue media | not tested | Operator only | +| SSH by key | n/a | Only if a key were baked — which R-142 says must not happen in a public image | -At mechanism level, yes, and materially: +**So a locked-root box is recoverable, but only by someone competent at the physical console.** That is +a materially weaker objection than "no path back at all", which is what the structural reading +suggested before it was measured. -- **Locked (`*`/`!`)** — `sulogin` refuses; console recovery is closed to everyone; the credential - does not exist so it cannot leak, be escrowed, or be rotated. -- **Rotated to a per-box random value** — `sulogin` prompts and *accepts the correct password* - (the control row in §2.2 shows `pam_unix` authenticating a real hash). So console recovery is open - **to whoever holds that password** — which is the operator if it is escrowed, and nobody if it is - discarded. +**A security corollary that is worth stating plainly: the installed GRUB has no password, so locking +root buys nothing against physical access.** Anyone at the keyboard can take a root shell in under a +minute. Locked root is a defence against *remote* credential reuse — the shared-ISO-password problem — +and not a physical-security measure. **R-152.** -**This makes the tradeoff concrete: "discard" and "lock" are the same outcome for recovery.** A -rotated-then-discarded password is operationally identical to a locked account, because nobody holds -it. The real choice is therefore **binary — escrow, or accept that console recovery is operator-only -and physical.** Framed for the ruling in §7. +### 3.4 Does locked differ from rotated? — **[MEASURED]. Yes, at exactly one point.** + +- **Locked (`*`/`!`)** — `sulogin` refuses (§3.2); the web UI refuses (§2.3); `init=/bin/bash` works. +- **Rotated to a per-box value** — `pam_unix` accepts the correct password (the §2.2 control proves + this) and the web UI issues a ticket for it (the §2.3 control proves this), so **both** the console + and the browser open **to whoever holds it**; `init=/bin/bash` works as well. + +**And this is what settles the escrow question.** A rotated password that is *discarded* is +operationally identical to a locked account: nobody holds it, so `sulogin` and the web UI refuse +either way. The only thing escrow buys is that the **operator** keeps the web-UI and `sulogin` routes. +It does not buy recoverability as such, because `init=/bin/bash` is available in every case. + +### 3.5 A probe that failed, recorded as failed + +The first GRUB attempt waited 3 seconds before acting; the menu's timeout is **1 second**, so the +system had already booted and the keystrokes went into the login prompt — the arrow-key escape +sequences (`^[[B`) are visible there in the screendump. It tested nothing. A later attempt typed +`init=/bin/bash` onto the `echo` line one below `linux`, which the **pre-typing verification +screendump caught** before it was booted. Both were rerun; the numbers above are from the corrected +runs. --- ## 4. Q3 — does the Felhom stub survive `before-network`? -### 4.1 Does it execute? — **[BLOCKED]** +### 4.1 Does it execute? — **[MEASURED-PVE]. Yes.** -Needs the VM. The staged diagnostic emits `@@SPIKE2_FIRSTBOOT_RAN@@ marker=BEGIN/END` to `ttyS0` -(captured by `qemu -serial file:`) precisely so the answer is a **positive observable** rather than the -absence of an error — and §4.4 below is why that matters more than usual here. +The diagnostic emits `@@SPIKE2_HOOK_RAN@@ BEGIN`/`END` to `ttyS0`. On scenario B +(`ordering = "before-network"`) **both markers are present** (count `2`). This is a positive +observable the hook produced, which §4.4 explains is the only acceptable form of proof here. -### 4.2 Does it have what it needs that early? — **[STRUCTURAL]**, partly answered +### 4.2 Does it have what it needs that early? — **[MEASURED-PVE]. Yes, everything.** -From the shipped unit (`proxmox-first-boot` 9.2.5, -`/usr/lib/systemd/system/proxmox-first-boot-network-pre.service`): +### 4.3 Does it complete before anything is network-reachable? — **[MEASURED-PVE + STRUCTURAL]. Yes.** + +The A/B comparison, same image, same diagnostic, only `ordering` differs: + +| Diagnostic | **A — `fully-up`** | **B — `before-network`** | +|---|---|---| +| hook markers | 2 (ran) | **2 (ran)** | +| `uptime_s` at hook | 18.32 | **6.58** | +| `entropy_avail` | 256 | **256** | +| `etc_writable` | YES | **YES** | +| `openssl`/`chpasswd`/`systemctl`/`passwd`/`curl`/`ip` | all present | **all present** | +| `openssl_rand_len` | 32 | **32** | +| `ip_global` | `vmbr0 10.0.2.15/24` | **(empty)** | +| `network.target` | active | **inactive** | +| `network-online.target` | active | **inactive** | +| `pveproxy` | active | **inactive** | +| `sshd` | active | **inactive** | +| **`listen_22_8006`** | **3** | **0** | + +**At `before-network` the hook runs with entropy, a writable `/etc`, every binary it would need, and a +demonstrated ability to generate a 32-character random string — while the box has no IP address and +nothing is listening on `:22` or `:8006`.** A credential rotation placed there has a **zero-length +exposure window, measured.** + +Equally, the `fully-up` column is the measured proof of the converse: **today's ordering runs the hook +when sshd and pveproxy are already listening.** The window Spike 1 argued for structurally is now +observed. + +The structural half agrees and generalises it. The shipped unit +(`proxmox-first-boot` 9.2.5, `/usr/lib/systemd/system/proxmox-first-boot-network-pre.service`): ``` After=systemd-remount-fs.service @@ -209,57 +298,37 @@ ConditionPathIsReadWrite=/var/lib Type=oneshot ``` -`After=systemd-remount-fs.service` plus `ConditionPathIsReadWrite=/var/lib` mean **a writable -filesystem is structurally guaranteed** — or the unit does not run at all (§4.4). Entropy availability -and binary presence at that point are **not** established structurally and are part of the staged -diagnostic (`entropy_avail`, `openssl rand` length, `command -v` for `openssl`/`chpasswd`/`systemctl`). - -### 4.3 Does it complete before anything is network-reachable? — **[STRUCTURAL]. Yes, by construction.** - -This is the result the task asked to establish structurally rather than by timing, and the unit file -settles it: - -- **`Before=network-pre.target`** — `network-pre.target` is systemd's ordering barrier that precedes - *all* network configuration; network managers order themselves after it. -- **`Type=oneshot`** — systemd treats the unit as complete only when the process exits, so - `Before=` is a hard barrier rather than a hint. - -Together: **the hook runs to completion before any interface is configured**, therefore before `sshd` -can bind a routable address and before `pveproxy` serves. A rotation performed there has a -**zero-length exposure window by construction, not by being fast.** This holds on every machine and -does not vary with hardware — which is why it is stronger than the timing number Spike 1 failed twice -to obtain. +`Before=network-pre.target` + `Type=oneshot` makes it a hard barrier ahead of all network +configuration — so this holds on every machine, not just this VM. Both installs then booted to full +reachability (SSH +20 s, pveproxy +26–29 s), so the early hook does not break the boot. **A third ordering value exists that Spike 1 did not record.** `validate-answer` accepts -**`before-network`, `network-online`, and `fully-up`** — and rejects `network-pre`, `multi-user` and a -bogus value, so this is a real enum and not a permissive parser. The shipped package contains a -matching unit for each. Spike 1 reported two. `network-online` is the middle option — after the -network, before `pveproxy` — and is worth considering if `before-network` proves too early for -something. +**`before-network`, `network-online`, `fully-up`**; it rejects `network-pre`, `multi-user` and a bogus +value, so this is a real enum. The shipped package contains a matching unit for each. +`network-online` is the unevaluated middle option. **R-149.** -### 4.4 Does the existing pairing flow still work? — **[STRUCTURAL]** yes; **[BLOCKED]** empirically +### 4.4 Does the existing pairing flow still work? — **[STRUCTURAL]**, and it is the one gap left -Two structural findings, and the second one corrects a premise this design has been carrying. +**Not measured.** The scenarios installed the spike's own diagnostic as the first-boot hook, not +Felhom's `stub-first-boot.sh`, so stub-plus-pairing coexistence is argued, not observed. -**The stub does not need the network, and does not need `pveproxy`.** `stub-first-boot.sh` writes three -files, then `systemctl daemon-reload`, `enable`, and `start --no-block felhom-bootstrap.service`. A -grep for `pvesh`, `pct`, `pveum` and `qm` across the stub returns **no match** (`rc=1`). The unit it -starts carries its own `After=network-online.target pve-cluster.service pveproxy.service`, so -`--no-block` merely queues the job and systemd holds it until those are satisfied. Structurally, stub -and pairing coexist: the stub runs early, the bootstrap runs when its own dependencies are met. +The structural case is strong and has a correction in it: -**So `answer.toml.tmpl:27` justifies the `fully-up` choice with a dependency the stub does not have.** -The comment reads *"fully-up = the stub runs after pveproxy so pvesh/pct work"* — but the `pvesh`/`pct` -usage is in `felhom-bootstrap.sh`, which never runs under the stub's ordering. This is the -CLAUDE.md "a comment asserting an invariant" class in a new shape: a comment asserting a **dependency** -that does not exist, which has constrained the design ever since. **R-148.** +**The stub does not need `fully-up`.** `stub-first-boot.sh` writes three files, then +`systemctl daemon-reload`, `enable`, and `start --no-block felhom-bootstrap.service`. A grep for +`pvesh`, `pct`, `pveum` and `qm` across it returns **no match** (`rc=1`). The unit it starts carries its +own `After=network-online.target pve-cluster.service pveproxy.service`, so `--no-block` queues the job +and systemd holds it until those are satisfied. -**The trap that makes §4.1's positive observable load-bearing.** All four `proxmox-first-boot-*` units -are guarded by `ConditionPathExists=` and `ConditionPathIsReadWrite=`. In systemd a **failed condition -is a skip, not a failure** — the unit reports `inactive`/`condition failed`, the boot succeeds, and -nothing is logged as wrong. A first-boot hook that never ran and one that ran perfectly are -indistinguishable from the absence of errors. **R-150.** This is why §4.1 must be confirmed by a -marker the hook emitted, never by "no errors appeared". +**So `answer.toml.tmpl:27` justifies the current ordering with a dependency the stub does not have** — +it reads *"fully-up = the stub runs after pveproxy so pvesh/pct work"*, but that usage lives in +`felhom-bootstrap.sh`, which never runs under the stub's ordering. A comment asserting a **dependency** +that does not exist, and it has constrained the design ever since. **R-148.** + +**Why §4.1's positive marker is load-bearing.** All four `proxmox-first-boot-*` units are guarded by +`ConditionPathExists=` / `ConditionPathIsReadWrite=`, and in systemd **a failed condition is a skip, +not a failure** — no error, successful boot. A hook that never ran and one that ran perfectly are +indistinguishable from the absence of errors. **R-150.** --- @@ -267,119 +336,99 @@ marker the hook emitted, never by "no errors appeared". ### 5.1 Where the collapse happens — **[STRUCTURAL]**, and it is not a simple removal -`iso-repack.sh` replaces the stock `grub.cfg` wholesale: it renders `grub/grub.cfg.tmpl` over it -(`iso-repack.sh:144-148`), having first lifted the `linux`/`initrd` lines verbatim from the stock -*Install Proxmox VE (Automated)* entry (`:108-131`) so a PVE bump tracks automatically. +`iso-repack.sh` replaces the stock `grub.cfg` wholesale, rendering `grub/grub.cfg.tmpl` over it +(`:144-148`) after lifting the `linux`/`initrd` lines verbatim from the stock *Install Proxmox VE +(Automated)* entry (`:108-131`) so a PVE bump tracks automatically. -**Four independent gates then enforce the single entry**, so restoring the stock menu is a change to -the gates, not just to the template: +**Four independent gates enforce the single entry:** | Gate | Location | Effect | |---|---|---| -| exactly one `menuentry` | `iso-repack.sh:153-155` | `exit 14` otherwise | -| zero `submenu` | `iso-repack.sh:156` | `exit 14` otherwise | -| no live reference to `proxtui`, `proxdebug`, `nomodeset`, `Rescue Boot`, `memtest`, `fwsetup` | `iso-repack.sh:160-164` | `exit 14` otherwise | -| **re-extracted from `final.iso`**: exactly one entry, theme present, entry named `Felhom telepítés` | `iso-repack.sh:314-319` | `exit 15` otherwise | +| exactly one `menuentry` | `:153-155` | `exit 14` | +| zero `submenu` | `:156` | `exit 14` | +| no live `proxtui`, `proxdebug`, `nomodeset`, `Rescue Boot`, `memtest`, `fwsetup` | `:160-164` | `exit 14` | +| **re-extracted from `final.iso`**: one entry, theme present, entry named `Felhom telepítés` | `:314-319` | `exit 15` | -The fourth is the strongest of them — it proves the menu in the shipped image rather than in the build -tree, which is the seam-wiring lesson applied correctly. +The fourth proves the menu in the shipped image rather than the build tree — the seam-wiring lesson +applied correctly. -**What adding one interactive entry would actually cost:** the two count gates (rows 1 and 4) move from -`== 1` to `== 2`, or better to a named allowlist. **The banned-token gate would still pass unchanged** -for a stock *Install Proxmox VE (Graphical)* entry — it uses none of `proxtui`, `proxdebug`, -`nomodeset`, `Rescue Boot`, `memtest`, `fwsetup`. So the safety property the gates exist to protect -(no route to a debug/rescue/manual-disk-picker path) survives a second *graphical installer* entry -intact. That is a narrower change than it first appears. +**What a second entry would cost:** the two count gates move from `== 1` to `== 2`, or to a named +allowlist. **The banned-token gate passes unchanged** for a stock *Install Proxmox VE (Graphical)* +entry — it uses none of those six tokens. The safety property the gates exist for (no route to a +debug/rescue/manual-disk-picker path) survives a second graphical-installer entry intact. -### 5.2 Are default and timeout settable? — **[STRUCTURAL]. Yes, already.** +### 5.2 Are default and timeout settable? — **[STRUCTURAL]. Already are.** -`grub/grub.cfg.tmpl` sets them explicitly: +`grub/grub.cfg.tmpl` sets `set timeout_style=menu`, `set timeout=5`, `set default=0`. The template also +records a real defect it fixed: the stock PVE config writes `timeout-style` (hyphen), **which GRUB does +not recognise as a variable name**, so Felhom's underscore spelling is the one that works. -``` -set timeout_style=menu -set timeout=5 -set default=0 -``` +### 5.3 The design question — **framed with evidence, deliberately not ruled** -Both are plain template values. The template also records a real defect it fixed: the stock PVE config -writes `timeout-style` (hyphen), **which GRUB does not recognise as a variable name**, so Felhom's -`timeout_style` (underscore) is the one that actually works. A two-entry menu would therefore have a -functioning timeout and a settable default — which is exactly what the §5.3 question needs. +**For interactive-as-default.** Spike 1 proved the automated path cannot be made safe on unseen +hardware: no property distinguishes a system disk from a customer's backup drive, and a filter matching +two disks silently wipes one. A tester who boots and walks away on multi-disk hardware is walking away +from an unattended destructive operation whose target cannot be predicted. -### 5.3 The design question — **framed, deliberately not ruled** +**This session strengthens that case independently.** If the public image also ships a locked root, +then a box whose install succeeds but whose pairing fails has **no web UI** (§2.3) and **no working +recovery-mode shell** (§3.2) — only an operator-grade `init=/bin/bash` (§3.3). An unattended default +maximises the number of testers who reach that state without having read anything. -Which entry should be `default=0` in a **public** image? - -**What the measurements say, and they only point one way:** Spike 1 established that the automated -path cannot be made safe on unseen hardware — no property distinguishes a system disk from a -customer's backup drive, and a filter matching two disks silently wipes one. A tester who boots the -image and walks away is, on multi-disk hardware, walking away from an unattended destructive operation -whose target cannot be predicted. - -**What that costs:** unattended install stops being the default for people who do not read -instructions — and unattended install is a selling point. Most first boxes are single-disk, where the +**Against.** Unattended install is a selling point and most first boxes are single-disk, where the automated path is fine. -**The measurement that would change the balance** is §2.3 and §3: if a locked root also closes the PVE -web interface and the console, then a tester whose unattended install completes but whose pairing -fails has a machine nobody can reach — which argues for interactive-default independently of the disk -question. Those measurements are blocked, so the balance cannot be struck yet. - -**Operator-built versus public need not match, and two profiles from one build is cheap.** An -operator-built image targets a machine whose disks are known, so automated-default is safe there. The -public image is the one facing unseen hardware. Recommending the split is not the same as ruling the -default, and the split is available regardless of how the ruling goes. +**Operator-built and public need not match, and two profiles from one build is cheap.** An +operator-built image targets known disks, so automated-default is safe there; the public image is the +one facing unseen hardware. That split is available however the default is ruled. --- ## 6. Q5 — the resulting shape, as constraints -Constraints on a build spec, each with a release-gate criterion. **Not an implementation, and nothing -was built.** Constraints marked ⚠ depend on blocked measurements and are provisional. +Constraints on a build spec, each with a release-gate criterion. **Not an implementation.** | # | Constraint | Release gate | |---|---|---| -| 1 | **The image must not select a disk by property on unseen hardware.** Spike 1 §3.2/§3.3. | The rendered answer contains no `filter.*` key, **or** the image is operator-built for a named machine. Grep the rendered `answer.toml`. | -| 2 | **The interactive installer must be reachable from the boot menu.** §5.1 shows the cost is two count gates. | `final.iso`'s `grub.cfg` contains ≥1 interactive entry and still contains no live `proxtui`/`proxdebug`/`nomodeset`/`Rescue Boot`/`memtest`/`fwsetup` — i.e. keep gate 3, relax gates 1 and 4 to an allowlist. | -| 3 | ⚠ **No usable baked credential.** `root-password-hashed = "*"` is schema-valid and installs (Spike 1). | The rendered answer's `root-password-hashed` is exactly `*`, and **no `.rootpw.txt` is emitted**. Both are one-line assertions in the build. | -| 4 | ⚠ **If a per-box password is set, it is set at `before-network`.** §4.3 — the only ordering with a zero window by construction. | The rendered answer's `[first-boot].ordering` is `before-network`; and the hook's positive marker is observed in a boot test, never inferred from absent errors (§4.4, R-150). | -| 5 | **No `root-ssh-keys`, ever, in a public image.** Spike 1 R-142. | `grep -c root-ssh-keys` on the rendered answer is `0`. | -| 6 | **The profile must be committed.** Spike 1 found demo-felhom's baked key comes from an *uncommitted* profile, so a reference box is not reproducible from the repo. | The `--profile` path resolves inside the repo working tree **and** `git status --porcelain` is empty for it. **R-151.** | -| 7 | **The artefact must be reproducible.** Spike 1 R-144 (`nested-probe`'s profile no longer exists). | Manifest records the profile's committed path and the repo commit; both resolve. | -| 8 | **A recovery path must be documented and, if operator-only, said so plainly.** §3.3. | The release notes name the recovery route for a failed pairing. If that route is `init=/bin/bash`, the notes say it is operator-only. | +| 1 | **No disk selection by property on unseen hardware** (Spike 1 §3.2/§3.3). | Rendered answer contains no `filter.*` key, **or** the image is operator-built for a named machine. | +| 2 | **The interactive installer must be reachable from the boot menu** (§5.1). | `final.iso`'s `grub.cfg` has ≥1 interactive entry **and** still no live `proxtui`/`proxdebug`/`nomodeset`/`Rescue Boot`/`memtest`/`fwsetup` — keep gate 3, relax gates 1 and 4 to an allowlist. | +| 3 | **No usable baked credential** — `root-password-hashed = "*"` installs cleanly (§1, three scenarios). | Rendered `root-password-hashed` is exactly `*`, **and no `.rootpw.txt` is emitted**. Two one-line assertions. | +| 4 | **Any per-box password is set at `before-network`** — the only ordering with a measured zero window (§4.3). | Rendered `[first-boot].ordering` is `before-network`, **and** the hook's positive marker is observed in a boot test (§4.4, R-150). | +| 5 | **No `root-ssh-keys`, ever, in a public image** (R-142). | `grep -c root-ssh-keys` on the rendered answer is `0`. | +| 6 | **The profile must be committed** — demo-felhom's baked key comes from an *uncommitted* profile, so a reference box is not reproducible from `main`. | `--profile` resolves inside the repo tree **and** `git status --porcelain` is empty for it. **R-151.** | +| 7 | **The artefact must be reproducible** (R-144: `nested-probe`'s profile is gone). | Manifest records the committed profile path + repo commit; both resolve. | +| 8 | **The recovery path must be documented, and named operator-only if it is.** §2.3 + §3.2 + §3.3: with a locked root the only way in is `init=/bin/bash` at the console. | Release notes state it, in those terms. | -**What the image must not contain:** a usable root password hash, any `root-ssh-keys`, any customer -id, claim code, retrieval passphrase or hub token, and any `filter.*` disk selector in the public -variant. Spike 1 §6 established by enumeration that the only secret-bearing file is `/answer.toml`, so -gates 3 and 5 over that one rendered file cover the credential surface. +**What the image must not contain:** a usable root password hash, any `root-ssh-keys`, any customer id, +claim code, retrieval passphrase or hub token, and any `filter.*` selector in the public variant. +Spike 1 §6 established by enumeration that `/answer.toml` is the only secret-bearing file, so gates 3 +and 5 over that one rendered file cover the credential surface. --- ## 7. The operator's decisions -Two, each with the measurement that informs it and its current state. +**Decision 1 — escrow or discard a per-box root password.** §3.4 reshapes this twice over: -**Decision 1 — escrow or discard a per-box root password.** +- **Discard and lock are the same thing** for recovery — if nobody holds the password, both `sulogin` + and the web UI refuse. +- **Neither makes a box unrecoverable.** `init=/bin/bash` works in every case (§3.3), so an operator at + the console can always get in. The earlier fear that locking root leaves "no path back at all" is + **measured to be false**. -§3.4 changed the shape of this question. **"Discard" and "lock" are operationally identical** — if -nobody holds the password, `sulogin` refuses either way, and console recovery is closed to everyone. -So the choice is binary: +So what escrow actually buys is narrower than it looked: **the operator keeps the web-UI route (§2.3) +and the `sulogin` route (§3.2) without needing physical presence.** Against that stands D6 — a hub +holding console root for the whole fleet is exactly what the hub alone must not be. -- **Escrow** — the operator can always reach a box's console; the cost is a hub-held credential for - every box, which is the thing the D6 trust model says the hub alone must not be. -- **Discard / lock** — no credential exists anywhere; the cost is that a failed pairing leaves a - machine recoverable only by an operator at the keyboard (`init=/bin/bash`) or a reinstall. +**Recommendation, offered because the measurements now support one:** discard/lock for the **public** +image, and accept operator-only physical recovery, because the credential that would be escrowed is +the same one whose fleet-wide sharing started this whole line of work. Escrow remains defensible for +**operator-built** boxes, where the operator installed the machine anyway. The decision is still the +operator's; it is no longer blocked on a measurement. -**Blocking measurement: §2.3** — whether a locked root also closes the PVE web interface. If it does, -"discard" removes the last remote route as well as the local one, and the balance shifts sharply -toward escrow. **Do not rule until §2.3 is measured.** - -**Decision 2 — the default boot entry for a public image.** - -Framed in §5.3. The disk evidence points at interactive-default; the product cost is the unattended -experience. **Partially blocked on the same §2.3 measurement**, because an unreachable-after-failure -box strengthens the interactive case independently. The operator-built-versus-public split (§5.3) is -available regardless and is cheap. +**Decision 2 — the default boot entry for a public image.** Framed in §5.3, unruled. The disk evidence +and the recovery evidence now point the same way (interactive), and the cost is the unattended +experience. The operator-built-versus-public split is cheap and available either way. --- @@ -387,70 +436,65 @@ available regardless and is cheap. | # | Unknown | What it takes | |---|---|---| -| 1 | **Everything in §2.1, §2.3, §2.4, §3.1, §4.1, §4.2 (entropy/binaries).** | The staged fixture, unchanged, once a Tier 0 box returns. ~90 min for all three scenarios. | -| 2 | **Whether PVE alters the §2.2/§3.2 mechanism results** — PAM config, a `sulogin` override, `SYSTEMD_SULOGIN_FORCE`. | Same fixture; check `/etc/pam.d/`, the `sulogin` invocation, and systemd's emergency unit on the installed image. | -| 3 | **Whether the multi-match pick is deterministic by enumeration order** (Spike 1 §8 #1, still open). | Re-run Spike 1's S1 with reversed disk order. ~15 min. | -| 4 | **Whether the installer excludes its own boot medium** (Spike 1 §8 #2, still open). | USB boot on real hardware — HUMAN. | -| 5 | **What `network-online` ordering buys** (§4.3) — it is a real third option nobody has evaluated. | One extra scenario alongside the staged two. | -| 6 | **Whether a second GRUB entry actually boots the stock installer** after the gates are relaxed. | Build a probe ISO with a two-entry template and boot it. Not attempted — it would have meant changing repo files, which §4 forbids. | +| 1 | **Felhom's real stub at `before-network`** (§4.4) — coexistence with pairing is structural, not observed. The single remaining gap in Q3. | Build a probe ISO with `stub-first-boot.sh` as the hook and `ordering = "before-network"`, boot, confirm `felhom-bootstrap.service` still starts and pairing completes. ~40 min. **Do this before the spec.** | +| 2 | **Whether root SSH is refused *because* of the lock** (§2.2) — the probe used a wrong password in the control too, so it discriminates nothing. | Re-run the SSH leg with the control's correct password. ~10 min. | +| 3 | **What else in PVE assumes a usable root credential** (§2.4) — cron, backup jobs. | Inspect a locked install over a longer uptime. | +| 4 | **What `network-online` ordering buys** (§4.3) — a real third option, never evaluated. | One extra scenario. | +| 5 | **Whether the multi-match pick is deterministic by enumeration order** (Spike 1, still open). | Re-run Spike 1's S1 with reversed disk order. ~15 min. | +| 6 | **Whether the installer excludes its own boot medium** (Spike 1, still open). | USB boot on real hardware — HUMAN. | +| 7 | **Whether a relaxed-gate two-entry menu actually boots the stock installer** (§5.1). | Not attempted — it would mean changing repo files, which §4 forbids. | --- ## 9. R-rows opened -Numbering continues from R-147 (Spike 1). +Continuing from R-147 (Spike 1). | R | Severity | Finding | |---|---|---| -| **R-148** | MEDIUM | **`answer.toml.tmpl:27` justifies `ordering = "fully-up"` with a dependency the stub does not have.** The comment says the stub must run after `pveproxy` "so pvesh/pct work", but `stub-first-boot.sh` contains no `pvesh`/`pct`/`pveum`/`qm` call (grep `rc=1`); that usage lives in `felhom-bootstrap.sh`, which runs under its own `After=network-online.target … pveproxy.service` unit. A comment asserting a **dependency** rather than an invariant — and it has constrained the design, because `before-network` is the ordering that would make the credential window zero (§4.3). | -| **R-149** | LOW | **The `[first-boot].ordering` enum has three values, not two.** `before-network`, `network-online`, `fully-up` all validate; `network-pre` and `multi-user` are rejected. Spike 1 recorded two and the middle option has never been evaluated. | -| **R-150** | MEDIUM | **A first-boot hook that never runs is indistinguishable from one that succeeded.** All four `proxmox-first-boot-*.service` units are guarded by `ConditionPathExists=` / `ConditionPathIsReadWrite=`, and in systemd a failed condition is a **skip, not a failure** — no error, successful boot. Any design that puts credential rotation in a first-boot hook must verify it by a positive marker the hook emitted. Directly the "an absent log line is not evidence" class. | -| **R-151** | MEDIUM | **A reference box is not reproducible from the repository.** `demo-felhom` was installed from an ISO built with the **uncommitted** `n100-demo.profile`, which is where its baked SSH key comes from (Spike 1 §4.6). One of the two Tier 0 boxes therefore cannot be rebuilt from `main`. Related to R-144 (`nested-probe`'s profile is gone entirely); this row is the *reference box*, which matters more. Gate 6 in §6 exists to stop a public image inheriting the shape. | +| **R-148** | MEDIUM | **`answer.toml.tmpl:27` justifies `ordering = "fully-up"` with a dependency the stub does not have.** `stub-first-boot.sh` contains no `pvesh`/`pct`/`pveum`/`qm` call (grep `rc=1`); that usage is in `felhom-bootstrap.sh`, under its own `After=network-online.target … pveproxy.service` unit. A comment asserting a **dependency** rather than an invariant — and it blocked the ordering that §4.3 now measures as giving a zero credential window. | +| **R-149** | LOW | **`[first-boot].ordering` has three values, not two** — `before-network`, `network-online`, `fully-up` validate; `network-pre` and `multi-user` are rejected. The middle option has never been evaluated. | +| **R-150** | MEDIUM | **A first-boot hook that never runs is indistinguishable from one that succeeded.** All four `proxmox-first-boot-*.service` units are `Condition*`-guarded, and a failed condition is a systemd **skip, not a failure** — no error, successful boot. Any credential-rotation-in-first-boot design must be verified by a positive marker the hook emitted. | +| **R-151** | MEDIUM | **A reference box is not reproducible from the repository.** `demo-felhom` was installed from an ISO built with the **uncommitted** `n100-demo.profile`, which is where its baked SSH key comes from (Spike 1 §4.6). Gate 6 in §6 exists to stop a public image inheriting that shape. | +| **R-152** | LOW | **A locked root is not a physical-security measure, and should never be described as one.** The installed PVE GRUB has **no password** (§3.1), so `init=/bin/bash` yields an unauthenticated root shell in under a minute (§3.3, measured). Locking root defends against *remote* reuse of a shared ISO credential and nothing else. If a physical-access threat model is ever wanted, it needs a GRUB password, which is a separate decision with its own lockout risk. | --- ## 10. Teardown -All three layers, verified positively. +All three layers, each verified positively. -**Layer 1 — demo-hp.** **INCOMPLETE, and named as such.** The box went offline (§0) while -`/mnt/nvme-1tb/spike2/` held the probe ISO (~1.7 GB) and the staged answer files and runners. Nothing -was running on it: the last successful command before the outage confirmed `0` QEMU processes, and no -install had been started this session. Nothing was registered as a PVE storage, nothing was written to -`local-lvm`, and `drill-r50` / guest 9201 were never touched this session. +**Layer 1 — demo-hp.** -**Outstanding cleanup, to run when the box returns** — recorded here because R-131 exists precisely -because recorded commands never get run: +| Check | Result | +|---|---| +| QEMU processes | `0` | +| `/mnt/nvme-1tb/spike2/` | removed — `No such file or directory` | +| `/tmp/paia` | removed | +| loop devices / `/mnt/s2ans` | `0` / removed | +| `/mnt/nvme-1tb/` contents | `dump e2d-images felhom-data images lost+found` — the pre-spike set | +| disk usage | **6.6 G — identical to pre-spike** | +| `drill-r50` (VM 300) | `stopped`, untouched (R-93 fixture intact) | +| guest 9201 | `running`, untouched | +| `/etc/pve/storage.cfg` | **4 storages, unchanged**; no scratch storage ever registered | +| `local-lvm` | never written | -``` -rm -rf /mnt/nvme-1tb/spike2 ; rm -f /tmp/paia ; pkill -9 qemu-system-x86 -``` - -This is deliberately *not* claimed as done. Spike 1's `/mnt/nvme-1tb/spike-scratch` and both `/tmp/paia` -copies were removed and verified at that session's teardown, so the outstanding item is this session's -`spike2/` directory alone. - -**Layer 2 — demo-felhom.** Nothing placed this session — it was never reached before the outage. -Spike 1's `/tmp/paia` was removed and verified then. No action outstanding. +**Layer 2 — demo-felhom.** `/tmp/paia` removed; guest 9201 `running`; the external USB drive +`931.5G usb /mnt/hdd_1` still attached and mounted exactly as found — never attached, detached, +partitioned or written. Only `device-info` reads were ever performed there. **Layer 3 — DooPlex.** | Check | Result | |---|---| -| scratchpad | **4.4 GB → 48 K** (probe ISO, squashfs extracts, `.deb` extract, schema test dirs removed) | -| control-password file (`s2/.ctlpw`) | `shred -u`'d — never printed, never committed; verified absent | -| mechanism-probe container | `docker run --rm` — no image or container retained; **no global Docker cleanup run** (fence) | +| scratchpad | **1.7 GB → 60 K** (probe ISO, screendumps, squashfs/`.deb` extracts, schema test dirs) | +| control password (`s2/.ctlpw`) and `scen/C.toml` | `shred -u`'d / removed; verified absent. Never printed, never committed | +| mechanism-probe container | `docker run --rm`; **no global Docker cleanup run** (fence) | | `felhom-iso/out/` | untouched, **17 ISOs** | -| production files | none modified — no `build-felhom-iso.sh`, no repack, no profile, no `felhom-bootstrap.sh` | +| production files | none modified — no `build-felhom-iso.sh`, no `iso-repack.sh`, no profile, no `felhom-bootstrap.sh` | -**One teardown error, recorded rather than quietly fixed.** Shredding the control password -(`s2/.ctlpw`) destroyed the plaintext that scenario **C** existed to use — a `$6$` hash whose plaintext -is gone cannot demonstrate "correct password → PVE ticket", which was C's entire purpose as the -control for §2.3. `s2/scen/C.toml` was therefore **deleted** rather than left in place looking usable. -**Scenario C must be regenerated** (one `openssl passwd -6` against a fresh random plaintext, kept -0600) before the §2.3 measurement is attempted. Scenarios A and B are unaffected and remain staged. - -**Hub-side disposition — nothing to dispose.** No customer, host or appliance was created: no install -ran, and the only hub contact this session was none at all. Verified rather than asserted below. +**Hub-side disposition — nothing to dispose.** No customer, host or appliance was created: the nested +VMs ran under plain QEMU with user-mode networking and no route to the hub, and no install ever +contacted it. **R-131 gains no row.** **Published artefacts: none.** Nothing went to R2. diff --git a/scripts/CHANGELOG.md b/scripts/CHANGELOG.md index 1195ce7..e508104 100644 --- a/scripts/CHANGELOG.md +++ b/scripts/CHANGELOG.md @@ -1,38 +1,36 @@ -## SPIKE 2 (no code change) — before-network gives a zero window by construction; locked root closes sulogin (2026-07-31) +## SPIKE 2 (no code change) — locked root closes the PVE web UI; before-network gives a measured zero window (2026-07-31) **No script, profile or build file was modified.** Findings: -`documentation/audits/SPIKE-universal-iso-2-2026-07-31.md`. Opens R-148..R-151. +`documentation/audits/SPIKE-universal-iso-2-2026-07-31.md`. Opens R-148..R-152. -**Both Tier 0 boxes went offline mid-session** (remote site, 12:28 CEST; four routes tried, our -tailscale pod healthy). Q1/Q2/Q3 each retain a part that needs a nested VM and those parts are -**BLOCKED, not answered**. The fixture is staged with Spike 1's two defects already corrected. +Both Tier 0 boxes went offline mid-session (provider cable fault; four routes tried) and returned; +all three scenarios then ran to completion on real PVE, each signalled by `reboot-mode="power-off"`. -Established without them: - -- **STRUCTURAL — `ordering = "before-network"` maps to `proxmox-first-boot-network-pre.service` - (`proxmox-first-boot` 9.2.5), which is `Before=network-pre.target` + `Type=oneshot`.** It therefore - runs to completion before any interface is configured — a credential rotation there has a - **zero-length window by construction**, not by being fast. Stronger than the timing number Spike 1 - failed twice to obtain. -- **STRUCTURAL — the stub does not need `fully-up`.** `stub-first-boot.sh` contains no - `pvesh`/`pct`/`pveum`/`qm` call (grep rc=1); that usage is in `felhom-bootstrap.sh`, which runs under - its own `After=network-online.target ... pveproxy.service` unit. So `answer.toml.tmpl:27` justifies - the current ordering with a dependency that does not exist. **R-148.** -- **The ordering enum has THREE values** — `before-network`, `network-online`, `fully-up`. Spike 1 - recorded two; the middle one has never been evaluated. **R-149.** -- **MECHANISM (container, not PVE) — a locked root closes `sulogin`:** `Cannot open access to console, - the root account is locked.` for both `*` and `!`, with a working control (correct password -> - SU_OK, wrong -> Authentication failure). So "discard" and "lock" are the SAME outcome for recovery, - which makes the escrow decision binary. -- **All four `proxmox-first-boot-*` units are `Condition*`-guarded, and a failed condition is a SKIP, - not a failure** — a hook that never ran looks exactly like one that succeeded. **R-150.** -- Q4: the single-entry menu is enforced by **four** gates in `iso-repack.sh` (`:153-155`, `:156`, - `:160-164`, `:314-319`), so restoring an interactive entry is a gate change, not a template edit — - but the banned-token gate would pass unchanged for a stock Graphical entry. `default`/`timeout` are +- **A locked root CLOSES the PVE web interface.** Measured against the exact endpoint the UI logs in + through (`POST /api2/json/access/ticket`, `root@pam`) **with a working control**: the known-password + install returns **HTTP 200 + ticket**; the locked install returns 401 for every password, and no + password exists. `passwd -S root` = `L`, shadow = `literal-asterisk`. PVE uses the stock PAM stack + (`pve_pam_realm: 0`). +- **GRUB recovery mode is ALSO closed**: `Cannot open access to console, the root account is locked.` + — the container mechanism result confirmed verbatim on real PVE. +- **But `init=/bin/bash` still works** — an unauthenticated `root@(none):/#`. So a locked-root box is + recoverable, operator-only, at the physical console. The installed GRUB has **no password**, so + locking root is NOT a physical-security measure. **R-152.** +- **`before-network` measured, A/B, same image:** the hook runs (positive marker, uptime 6.58s) with + entropy 256, writable /etc, all binaries and `openssl_rand_len=32`, while `ip_global` is EMPTY and + `listen_22_8006` is **0**. The `fully-up` column is the converse proof: sshd + pveproxy already + active, 3 listening. Zero credential window, measured. +- **R-148:** `answer.toml.tmpl:27` justifies `fully-up` with a `pvesh`/`pct` dependency the stub does + not have (grep rc=1) — it blocked the ordering that is now measured as the fix. +- **R-149** three ordering values, not two · **R-150** Condition-guarded hooks skip silently · + **R-151** demo-felhom built from an uncommitted profile. +- Q4 from source: four gates in `iso-repack.sh` enforce the single-entry menu; `default`/`timeout` already settable in `grub/grub.cfg.tmpl`. -The first mechanism probe was invalid (ran as uid 0, so `su` never invoked pam_unix; `sulogin` had no -tty) and is recorded as failed, not massaged. +Three probes failed and are recorded as failed, not massaged: a container probe that ran as uid 0 +(so `su` never invoked pam_unix), a GRUB probe that missed the **1-second** menu timeout and typed into +the login prompt, and a kernel-line edit that landed one line off — caught by a pre-typing +verification screendump. ## SPIKE (no code change) — a universal ISO needs a different disk strategy and a locked root (2026-07-31)