diff --git a/REPORT-universal-iso-spike.md b/REPORT-universal-iso-spike.md index cb50dc5..532188e 100644 --- a/REPORT-universal-iso-spike.md +++ b/REPORT-universal-iso-spike.md @@ -1,93 +1,95 @@ -# REPORT — SPIKE: what does a genuinely universal ISO require? (2026-07-31) +# REPORT — SPIKE 2: closing the last mechanisms, and the universal ISO's shape (2026-07-31) -> Written as **`REPORT-universal-iso-spike.md`**, not `REPORT.md`: the shared file was overwritten at -> 09:24 today by the hub v0.85.0 Network-card session, and the standing rule is that the second -> session in a shared clone never touches it. +> Written as `REPORT-universal-iso-spike.md`, not `REPORT.md`: the shared file belongs to today's hub +> v0.85.0 session and the standing rule is that the second session in a shared clone never touches it. +> This file supersedes its own Spike 1 contents. -**Class: Spike.** Output is a findings document. **No production code was changed, no ISO was built, -nothing was published.** Full evidence: -`documentation/audits/SPIKE-universal-iso-2026-07-31.md`. +**Class: Spike.** Findings only — no production file changed, no ISO built, nothing published. +Evidence: `documentation/audits/SPIKE-universal-iso-2-2026-07-31.md`. Continues +`SPIKE-universal-iso-2026-07-31.md`. -## Why this ran +## The headline: the venue went away, and three answers are blocked -The ISO build/publish task stopped before its Part 2. Its Part 0 found that every ISO bakes a root -password hash into `/answer.toml`, and that `generic.profile` picks a disk by the name `sda` rather -than by identity — both of which make a public download a product decision. This spike establishes -what a genuinely universal image would require. +At 12:28 CEST both Tier 0 boxes went offline **simultaneously** and did not return. Four routes were +tried (tailnet direct, `ssh demo-hp`, `demo-hp-lan` via ProxyJump, `ssh felhom-pve`); our tailscale +pod is healthy and `tailscale status` shows both peers `offline` with a rising `last seen`. Both boxes +are at the same remote site — that site's power or uplink, not fixable from here. -## What was answered +**No fallback venue was used.** DooPlex is Tier 2, the 2026-07-25 ruling moved drill VMs off it, and +this task did not authorise it — an absent fence is not permission. So **Q1, Q2 and Q3 each keep a +part that is BLOCKED, not answered**, and they are labelled that way throughout. The fixture is built +and staged with Spike 1's two measurement defects already corrected (a NIC present at install time; a +`reboot-mode = "power-off"` completion signal instead of a disk hash). -**Q1 — disk selection. The central finding, and it is worse than the known hazard.** +## What was established anyway -A filter that matches more than one device **does not fail safe**. Observed in a nested VM with two -identical disks and `filter.ID_SERIAL_SHORT = "SPIKE*"`: the installer silently selected one and wiped -it (`disk0` hash `de001102…` → `382b55ca…`; `disk1` byte-identical throughout). `validate-answer` -accepts the answer — it cannot know what the filter resolves to on the target. The -`filter did not match any devices` guard the safety profiles rely on covers the **zero**-match case -only. +**The best result of the session, and it is structural rather than measured — which makes it +stronger.** `ordering = "before-network"` maps to `proxmox-first-boot-network-pre.service` +(`proxmox-first-boot` 9.2.5, read out of the ISO's own package), which is `Before=network-pre.target` +with `Type=oneshot`. It therefore runs **to completion before any network interface is configured**. +A credential rotation placed there has a **zero-length exposure window by construction, not by being +fast** — and that holds on every machine, unlike the timing number Spike 1 failed twice to obtain. -And a filter that reliably matches exactly one disk on unseen hardware cannot be written, because -**no property distinguishes an internal system disk from external media.** Measured on `demo-felhom` -with its 1 TB external drive attached — the exact hazard configuration: +**And the reason it was never used is a comment that is wrong.** `answer.toml.tmpl:27` justifies +`fully-up` as needed "so pvesh/pct work" — but `stub-first-boot.sh` makes no `pvesh`/`pct`/`pveum`/`qm` +call at all (grep `rc=1`). That usage lives in `felhom-bootstrap.sh`, which runs under its own +`After=network-online.target … pveproxy.service` unit. A comment asserting a **dependency** that does +not exist, which has constrained the design ever since. **R-148.** -- `ID_BUS` is `ata` for **both** (the external unit is a SATA drive in a USB caddy) -- `lsblk RM` is `0` for **both**; `device-info` exposes no removability property at all -- on `demo-hp`, the NVMe carries **no `ID_BUS` and no `ID_TYPE`** — a different property set entirely -- nothing anywhere expresses *role*: on `demo-hp` the system disk is the small SATA one and the big - NVMe is customer data +**A locked root closes `sulogin`** — measured in an isolated container with a working control (correct +password → `SU_OK`, wrong → `Authentication failure`; then `*` and `!` both → `L` and +`Cannot open access to console, the root account is locked.`). This reshapes the escrow question: if +nobody holds the password, `sulogin` refuses either way, so **"discard" and "lock" are the same +outcome for recovery**, and the decision is binary. Caveat stated plainly: container, not PVE. -**Q2 — root password.** A credential field is mandatory (`One of global.root-password or -global.root-password-hashed must be set`), but a *usable* one is not: `root-password-hashed = "*"` -validates **and installs to completion** (guest powered itself off after 200 s using -`reboot-mode = "power-off"` as the completion signal). `[first-boot].ordering` also accepts -`before-network` — the only ordering that could close the exposure window structurally, since today's -bootstrap unit is `After=network-online.target pveproxy.service` and the pairing poll waits -indefinitely for the operator's bind. +**The ordering enum has three values, not two** — `before-network`, `network-online`, `fully-up`. +Spike 1 recorded two; the middle one has never been evaluated (**R-149**). -**Q3 — two-entry boot menu: yes, trivially.** `prepare-iso` leaves `grub.cfg` **byte-identical** to -stock; a prepared ISO has 15 entries including automated *and* interactive. Felhom's repack is what -collapses it to one. No PVE-side obstacle exists. +**A first-boot hook that never runs is indistinguishable from one that succeeded** — all four +`proxmox-first-boot-*` units are `Condition*`-guarded and a failed condition is a *skip*, not a +failure. Any rotation-in-first-boot design must verify by a positive marker (**R-150**). -**Q4 — what else is baked.** By enumeration against the stock ISO, not pattern-guessing: the pipeline -adds exactly six paths; both OS squashfs payloads are byte-identical to stock; all three first-boot -payloads match repo HEAD or carry only the hub URL. `/answer.toml` is the only secret-bearing content. - -**R-129 resolved.** demo-hp's key is the operator's own, added post-install — not baked. But -demo-felhom's *is* baked, by an uncommitted profile, so the "no baked key" claim was already false for -one demo box and nothing recorded it. +**Q4 answered fully from source.** The single-entry menu is enforced by **four** gates in +`iso-repack.sh` (`:153-155`, `:156`, `:160-164`, and a post-remaster re-extraction at `:314-319`), so +restoring an interactive entry is a gate change rather than a template edit — but the banned-token +gate would pass unchanged for a stock *Graphical* entry, so the safety property survives. `default` +and `timeout` are already settable in `grub/grub.cfg.tmpl`. ## What failed, and is recorded as failed -**The reachable-before-rotation measurement was attempted twice and both attempts were invalid.** -Attempt 1 reported a plausible **`0 s`** that was an artefact — QEMU's `hostfwd` accepts on the host -before touching the guest, and the same run then failed to read an SSH banner at all. Attempt 2, with -a protocol-level detector, reported `never` because the install VM had no NIC, so the installed -system's interface config could not match the boot VM's. **No number is reported.** The structural -finding (unit ordering + unbounded pairing poll) stands instead and is the stronger claim. +The first mechanism probe produced **zero valid observations**: it ran everything as uid 0, so `su` +never invoked `pam_unix` (rc=0 even with a wrong password), `sulogin` died on `tcgetattr` with no tty, +and a `sed` mangled the control hash. Rerun with a non-root caller, a real pty, and `chpasswd`. -Also not established: that the locked root account is actually locked (install completion is a -different claim), and whether GRUB single-user recovery works — the latter gates the escrow-vs-discard -ruling and should be measured before that ruling is made. +A **teardown error** is also recorded rather than quietly fixed: shredding the control password +destroyed the plaintext scenario C existed to use, so `C.toml` was deleted rather than left looking +usable. It must be regenerated before the PVE-web-interface measurement is attempted. -## Asked of the operator +## The two decisions, and why neither should be ruled yet -One decision, framed in §4.5 of the findings doc: **escrow or discard** a per-box root password. Not -recommended either way — the D6 trust model argues against a hub-held fleet-wide vault, and an unbound -box has no agent access, which is exactly when console access matters. +**Escrow versus discard** — now binary (see above). **Blocked on one measurement:** whether a locked +root also closes the PVE web interface. If it does, discarding removes the last remote route as well +as the local one and the balance shifts sharply toward escrow. **Do not rule before that is measured.** + +**The default boot entry for a public image** — the disk evidence points at interactive-default; the +cost is the unattended experience. Partially blocked on the same measurement. The +operator-built-versus-public split is cheap and available regardless of the ruling. ## R-rows -**R-139** (HIGH, multi-match wipes silently) · **R-140** (`device-match` exits 0 on no-match) · -**R-141** (HIGH, mandatory + fleet-shared credential, unbounded window) · **R-142** (`FELHOM_ROOT_SSH_KEY` -can ship a shared key, no assertion) · **R-143** (retired `kisfenyo@windows` key still live on -demo-felhom) · **R-144** (`nested-probe` ISO unreproducible — profile gone) · **R-145** (variant naming -undocumented) · **R-146** (`root-password-hashed = ""` validates) · **R-147** (hash substituted into -the answer template's own comments) · **R-128** restated (a comment, not an assertion). +**R-148** (`fully-up` justified by a non-existent dependency) · **R-149** (three ordering values, not +two) · **R-150** (Condition-guarded first-boot hook skips silently) · **R-151** (demo-felhom was +installed from an **uncommitted** profile — a Tier 0 reference box is not reproducible from `main`). ## Teardown -All three layers verified positively: demo-hp scratch removed and `drill-r50`/9201/`storage.cfg` -untouched, `local-lvm` never written; demo-felhom probe binary removed and its external drive observed -only; DooPlex scratchpad 3.2 GB → 28 K with the throwaway password `shred`ed. **No hub object was -created** — confirmed by fetching the customer list and searching it, not by assertion. Nothing -deleted from `felhom-iso/out/` (17 ISOs intact). Nothing published. +**Layer 1 (demo-hp) is INCOMPLETE and named as such** — the box went offline while +`/mnt/nvme-1tb/spike2/` held the staged fixture. Nothing was running on it (last check: 0 QEMU +processes, no install started this session), nothing on `local-lvm`, no PVE storage registered, +`drill-r50` and 9201 never touched. The outstanding command is recorded in §10 of the findings doc +and **is not claimed as done** — R-131 exists because recorded commands never get run. + +Layer 2 (demo-felhom): nothing placed this session. Layer 3 (DooPlex): scratchpad 4.4 GB → 48 K, +control password `shred -u`'d and verified absent, mechanism container `--rm`, no global Docker +cleanup, `felhom-iso/out/` untouched at 17 ISOs, no production file modified. **Hub-side: nothing +created** — verified by fetching the customer list and searching it, not asserted. Nothing published. diff --git a/documentation/audits/SPIKE-universal-iso-2-2026-07-31.md b/documentation/audits/SPIKE-universal-iso-2-2026-07-31.md new file mode 100644 index 0000000..d2b9fc1 --- /dev/null +++ b/documentation/audits/SPIKE-universal-iso-2-2026-07-31.md @@ -0,0 +1,456 @@ +# SPIKE 2 — closing the last mechanisms, and the universal ISO's shape (2026-07-31) + +> **Class: Spike.** Findings only. No production file changed, no ISO built, nothing published. +> +> Continues `SPIKE-universal-iso-2026-07-31.md`; its results are premises here and are not re-derived. +> +> **Read §0 first: the venue went away mid-session.** Both Tier 0 hosts dropped off the tailnet at +> ~12:28 CEST and did not return. Q1, Q2 and Q3 each have a part that needs a nested VM, and those +> parts are **BLOCKED, not answered**. What could be established without them — from the shipped PVE +> source, the answer schema, and an isolated mechanism probe — is below and is labelled by how it was +> obtained. + +--- + +## 0. Venue loss — what was tried + +At 12:28 CEST, mid-run, both Tier 0 boxes went offline **simultaneously**: + +``` +100.76.96.79 demo-hp active; relay "fra"; offline, last seen 5m ago, tx 1843821564 rx 63739848 +100.70.170.35 felhom-pve active; relay "waw"; offline, last seen 5m ago, tx 1560 rx 0 +``` + +Attempts, in order (per the standing rule that a "no access" claim names what it tried): + +| # | Route | Result | +|---|---|---| +| 1 | `ping 100.76.96.79` (tailnet direct) | 100 % packet loss | +| 2 | `ssh demo-hp` (tailnet, `ConnectTimeout=10`) | `connect to host 100.76.96.79 port 22: Connection timed out` | +| 3 | `ssh demo-hp-lan` (LAN `192.168.0.87`, ProxyJump `felhom-pve`) | `Connection timed out during banner exchange` | +| 4 | `ssh felhom-pve` (the jump host itself) | `connect to host 100.70.170.35 port 22: Connection timed out` | +| 5 | DooPlex tailscale pod health | pod `tailscale-c84df7f84-tw2wz` **Running 6d10h**, healthy; log shows `open-conn-track: timeout … online=yes` then peers dropping | +| 6 | `tailscale status` from the pod | **both peers `offline`**, `last seen 5m` and rising | + +**Diagnosis: the far end, not ours.** Our tailscale pod is healthy and every other peer route behaves; +both demo boxes are at the same remote site (REMOTE until ~08-02) and went offline together, which is +that site's power or uplink. Not fixable from here. + +**No fallback venue was used.** DooPlex is Tier 2 and the 2026-07-25 standing ruling moved drill and +build VMs off it; `target-selection.md` says a Tier 2 machine is used only when a task says so +explicitly, and this one does not. **An absent fence is not permission**, so the probes simply did not +run. A watch is armed for the boxes' return. + +The probe fixture is built and staged and will run unchanged when they come back: probe ISO +(`prepare-iso --fetch-from partition`, serial-capture first-boot diagnostic), three validated answer +files (locked/`fully-up`, locked/`before-network`, known-password control), an install runner with +`reboot-mode = "power-off"` as the completion signal, and a boot runner with protocol-level detectors +and **a NIC present at install time** — the two fixture defects that invalidated Spike 1's +measurements are already corrected. + +--- + +## 1. Baselines + +Read fresh this session. + +| Baseline | Value | +|---|---| +| `ISO_VERSION` | `1.25.0` — `scripts/iso/build-felhom-iso.sh:44` | +| `SCRIPT_VERSION` | `1.22.0` — `scripts/felhom-host-install.sh:187` | +| `felhom-bootstrap.sh` @ HEAD | `21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7` | +| PVE base | `proxmox-ve_9.2-1.iso`, `4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c` | +| `felhom.eu` HEAD | **`ea00976`**, clean, `== origin/main` (the expected Spike 1 commit) | +| `proxmox-first-boot` pkg | `9.2.5` (from the ISO's `/proxmox/packages/`) | + +**Fixture provenance.** Probe ISO built on DooPlex from the baseline PVE ISO. Nested VMs were to run +under plain QEMU/KVM on demo-hp under `/mnt/nvme-1tb/spike2/` — never `local-lvm`, no PVE storage +registered. They did not run (§0). The mechanism probe in §2.2 ran in a throwaway `debian:trixie` +Docker container on DooPlex, which is not an install and touches no box. + +**Evidence classes used throughout:** **[MEASURED-PVE]** against the real PVE tooling or a PVE install · +**[MEASURED-MECH]** measured, but in an isolated container rather than on PVE · **[STRUCTURAL]** read +from shipped source or unit files · **[BLOCKED]** needs the nested VM. + +--- + +## 2. Q1 — is a locked root actually locked, and what else does it lock out? + +### 2.1 The account's state on a real PVE install — **[BLOCKED]** + +Needs the nested VM. The fixture is staged. Spike 1 established only that +`root-password-hashed = "*"` **installs to completion**; the resulting account state was explicitly +left open there and remains open. + +### 2.2 What can authenticate — **[MEASURED-MECH]**, and the first attempt was invalid + +**The first probe produced nothing and is recorded as failed.** It ran everything as uid 0, so `su` +never invoked `pam_unix` at all (`rc=0` even with a deliberately wrong password); `sulogin` died on +`tcgetattr failed: Invalid argument` with no TTY; and a `sed` mangled the control hash so the control +case had no password set. Four cases, zero valid observations. Rerun with a non-root caller, a real +pty via `script`, and `chpasswd` instead of `sed`. + +**Corrected probe** (`debian:trixie`, `util-linux 2.41`, `pam_unix`) — with a working control: + +| Shadow field | `passwd -S root` | `su root` as non-root user | `sulogin` (real pty) | +|---|---|---|---| +| real hash (control) | `P` | **correct pw → `SU_OK`**; wrong pw → `Authentication failure` | — | +| `*` | **`L`** | `su: Authentication failure` (any password) | **`Cannot open access to console, the root account is locked.`** | +| `!` | **`L`** | `su: Authentication failure` (any password) | **`Cannot open access to console, the root account is locked.`** | + +The control is what makes this readable: the same harness authenticates successfully with the right +password and fails with the wrong one, so the two locked rows are the module refusing, not the harness +misfiring. + +**So, at mechanism level: `*` and `!` are equivalent, both report `L`, and no password authenticates.** + +**Caveat, and it is not small.** This is Debian's `pam_unix` and `util-linux`'s `sulogin` — the same +components PVE ships — but it is **not PVE**. PVE could ship PAM configuration, a `sulogin` override, +or a `SYSTEMD_SULOGIN_FORCE` setting that changes the outcome. Confirming on a real PVE install is +part of the blocked work. + +### 2.3 Does the Proxmox web interface still work? — **[BLOCKED]**, and this is the product question + +**No measurement was obtained, so no answer is given.** Stating the reasoning without the measurement +would be exactly the failure mode this spike exists to avoid. + +What is established: PVE's `root@pam` realm authenticates the Linux root account through PAM, and +§2.2 measured that `pam_unix` refuses a locked account for a non-root caller. If PVE's proxy +authenticates the same way — **which was not verified** — then a locked root means **nobody can log +into the PVE web interface at all**, neither customer nor operator, and that is a product decision +rather than a detail. + +The staged control scenario exists precisely to settle this: install C carries a **known** password, so +the PVE API ticket endpoint can be exercised positively (correct password → ticket) before install A +(locked) is tested against the same endpoint. Without that control a failure is indistinguishable +from a broken probe — Spike 1's lesson. + +### 2.4 Does anything else in PVE break? — **[BLOCKED]** + +Not investigated. Cron, `pveproxy`, backup jobs and anything else assuming a usable root credential +remain unexamined. + +--- + +## 3. Q2 — is there a way back into a box whose setup failed? + +### 3.1 Does the installed GRUB expose a recovery entry? — **[BLOCKED]** + +Needs a boot. Note the installed system's GRUB is **PVE's own**, written by the installer — not the +ISO's Felhom menu — so §5's single-entry collapse does not apply to it. What it offers is unverified. + +### 3.2 `sulogin` with a locked root — **[MEASURED-MECH]. It refuses.** + +The single observation the escrow-versus-discard ruling was said to rest on: + +``` +Cannot open access to console, the root account is locked. +See sulogin(8) man page for more details. +``` + +Both for `*` and for `!`, with a real pty. **The standard Debian/systemd emergency and single-user +path does not grant a shell on a locked-root system.** `systemd` invokes `sulogin` for +`emergency.target` and `rescue.target`, so both are closed by the same refusal. + +Subject to the §2.2 caveat: measured in a container, not on PVE. + +### 3.3 What is left if `sulogin` refuses — **[STRUCTURAL]**, not measured + +| Path | Realistic for whom | +|---|---| +| `init=/bin/bash` on the kernel line (GRUB `e`) | **Operator only.** It bypasses `sulogin` because PID 1 becomes the shell — no authentication is involved. Requires editing a boot entry at the console. Not a non-technical customer action. | +| `systemd.unit=emergency.target` | **Closed** — it lands in `sulogin`, which refuses (§3.2). | +| Live USB / rescue media | Operator only; requires physical media and knowing to do it. | +| PVE web console (`:8006`) | **Unknown, possibly closed** — §2.3. | +| SSH by key | Only if a key was baked, which for a public image is the thing R-142 says must not happen. | + +**The honest summary:** on a locked-root box, every remaining route is operator-grade and physical. +For a customer whose pairing failed, there is no self-service path back — which is precisely the +objection raised when locked-root was first proposed, and §3.2 is the first evidence that the +objection is well founded rather than speculative. + +### 3.4 Does locked differ from rotated? — **[PARTIALLY ESTABLISHED]** + +At mechanism level, yes, and materially: + +- **Locked (`*`/`!`)** — `sulogin` refuses; console recovery is closed to everyone; the credential + does not exist so it cannot leak, be escrowed, or be rotated. +- **Rotated to a per-box random value** — `sulogin` prompts and *accepts the correct password* + (the control row in §2.2 shows `pam_unix` authenticating a real hash). So console recovery is open + **to whoever holds that password** — which is the operator if it is escrowed, and nobody if it is + discarded. + +**This makes the tradeoff concrete: "discard" and "lock" are the same outcome for recovery.** A +rotated-then-discarded password is operationally identical to a locked account, because nobody holds +it. The real choice is therefore **binary — escrow, or accept that console recovery is operator-only +and physical.** Framed for the ruling in §7. + +--- + +## 4. Q3 — does the Felhom stub survive `before-network`? + +### 4.1 Does it execute? — **[BLOCKED]** + +Needs the VM. The staged diagnostic emits `@@SPIKE2_FIRSTBOOT_RAN@@ marker=BEGIN/END` to `ttyS0` +(captured by `qemu -serial file:`) precisely so the answer is a **positive observable** rather than the +absence of an error — and §4.4 below is why that matters more than usual here. + +### 4.2 Does it have what it needs that early? — **[STRUCTURAL]**, partly answered + +From the shipped unit (`proxmox-first-boot` 9.2.5, +`/usr/lib/systemd/system/proxmox-first-boot-network-pre.service`): + +``` +After=systemd-remount-fs.service +Before=network-pre.target +Wants=network-pre.target +ConditionPathExists=/var/lib/proxmox-first-boot/pending-first-boot-setup +ConditionPathIsReadWrite=/var/lib +Type=oneshot +``` + +`After=systemd-remount-fs.service` plus `ConditionPathIsReadWrite=/var/lib` mean **a writable +filesystem is structurally guaranteed** — or the unit does not run at all (§4.4). Entropy availability +and binary presence at that point are **not** established structurally and are part of the staged +diagnostic (`entropy_avail`, `openssl rand` length, `command -v` for `openssl`/`chpasswd`/`systemctl`). + +### 4.3 Does it complete before anything is network-reachable? — **[STRUCTURAL]. Yes, by construction.** + +This is the result the task asked to establish structurally rather than by timing, and the unit file +settles it: + +- **`Before=network-pre.target`** — `network-pre.target` is systemd's ordering barrier that precedes + *all* network configuration; network managers order themselves after it. +- **`Type=oneshot`** — systemd treats the unit as complete only when the process exits, so + `Before=` is a hard barrier rather than a hint. + +Together: **the hook runs to completion before any interface is configured**, therefore before `sshd` +can bind a routable address and before `pveproxy` serves. A rotation performed there has a +**zero-length exposure window by construction, not by being fast.** This holds on every machine and +does not vary with hardware — which is why it is stronger than the timing number Spike 1 failed twice +to obtain. + +**A third ordering value exists that Spike 1 did not record.** `validate-answer` accepts +**`before-network`, `network-online`, and `fully-up`** — and rejects `network-pre`, `multi-user` and a +bogus value, so this is a real enum and not a permissive parser. The shipped package contains a +matching unit for each. Spike 1 reported two. `network-online` is the middle option — after the +network, before `pveproxy` — and is worth considering if `before-network` proves too early for +something. + +### 4.4 Does the existing pairing flow still work? — **[STRUCTURAL]** yes; **[BLOCKED]** empirically + +Two structural findings, and the second one corrects a premise this design has been carrying. + +**The stub does not need the network, and does not need `pveproxy`.** `stub-first-boot.sh` writes three +files, then `systemctl daemon-reload`, `enable`, and `start --no-block felhom-bootstrap.service`. A +grep for `pvesh`, `pct`, `pveum` and `qm` across the stub returns **no match** (`rc=1`). The unit it +starts carries its own `After=network-online.target pve-cluster.service pveproxy.service`, so +`--no-block` merely queues the job and systemd holds it until those are satisfied. Structurally, stub +and pairing coexist: the stub runs early, the bootstrap runs when its own dependencies are met. + +**So `answer.toml.tmpl:27` justifies the `fully-up` choice with a dependency the stub does not have.** +The comment reads *"fully-up = the stub runs after pveproxy so pvesh/pct work"* — but the `pvesh`/`pct` +usage is in `felhom-bootstrap.sh`, which never runs under the stub's ordering. This is the +CLAUDE.md "a comment asserting an invariant" class in a new shape: a comment asserting a **dependency** +that does not exist, which has constrained the design ever since. **R-148.** + +**The trap that makes §4.1's positive observable load-bearing.** All four `proxmox-first-boot-*` units +are guarded by `ConditionPathExists=` and `ConditionPathIsReadWrite=`. In systemd a **failed condition +is a skip, not a failure** — the unit reports `inactive`/`condition failed`, the boot succeeds, and +nothing is logged as wrong. A first-boot hook that never ran and one that ran perfectly are +indistinguishable from the absence of errors. **R-150.** This is why §4.1 must be confirmed by a +marker the hook emitted, never by "no errors appeared". + +--- + +## 5. Q4 — the boot menu, and which entry is the default + +### 5.1 Where the collapse happens — **[STRUCTURAL]**, and it is not a simple removal + +`iso-repack.sh` replaces the stock `grub.cfg` wholesale: it renders `grub/grub.cfg.tmpl` over it +(`iso-repack.sh:144-148`), having first lifted the `linux`/`initrd` lines verbatim from the stock +*Install Proxmox VE (Automated)* entry (`:108-131`) so a PVE bump tracks automatically. + +**Four independent gates then enforce the single entry**, so restoring the stock menu is a change to +the gates, not just to the template: + +| Gate | Location | Effect | +|---|---|---| +| exactly one `menuentry` | `iso-repack.sh:153-155` | `exit 14` otherwise | +| zero `submenu` | `iso-repack.sh:156` | `exit 14` otherwise | +| no live reference to `proxtui`, `proxdebug`, `nomodeset`, `Rescue Boot`, `memtest`, `fwsetup` | `iso-repack.sh:160-164` | `exit 14` otherwise | +| **re-extracted from `final.iso`**: exactly one entry, theme present, entry named `Felhom telepítés` | `iso-repack.sh:314-319` | `exit 15` otherwise | + +The fourth is the strongest of them — it proves the menu in the shipped image rather than in the build +tree, which is the seam-wiring lesson applied correctly. + +**What adding one interactive entry would actually cost:** the two count gates (rows 1 and 4) move from +`== 1` to `== 2`, or better to a named allowlist. **The banned-token gate would still pass unchanged** +for a stock *Install Proxmox VE (Graphical)* entry — it uses none of `proxtui`, `proxdebug`, +`nomodeset`, `Rescue Boot`, `memtest`, `fwsetup`. So the safety property the gates exist to protect +(no route to a debug/rescue/manual-disk-picker path) survives a second *graphical installer* entry +intact. That is a narrower change than it first appears. + +### 5.2 Are default and timeout settable? — **[STRUCTURAL]. Yes, already.** + +`grub/grub.cfg.tmpl` sets them explicitly: + +``` +set timeout_style=menu +set timeout=5 +set default=0 +``` + +Both are plain template values. The template also records a real defect it fixed: the stock PVE config +writes `timeout-style` (hyphen), **which GRUB does not recognise as a variable name**, so Felhom's +`timeout_style` (underscore) is the one that actually works. A two-entry menu would therefore have a +functioning timeout and a settable default — which is exactly what the §5.3 question needs. + +### 5.3 The design question — **framed, deliberately not ruled** + +Which entry should be `default=0` in a **public** image? + +**What the measurements say, and they only point one way:** Spike 1 established that the automated +path cannot be made safe on unseen hardware — no property distinguishes a system disk from a +customer's backup drive, and a filter matching two disks silently wipes one. A tester who boots the +image and walks away is, on multi-disk hardware, walking away from an unattended destructive operation +whose target cannot be predicted. + +**What that costs:** unattended install stops being the default for people who do not read +instructions — and unattended install is a selling point. Most first boxes are single-disk, where the +automated path is fine. + +**The measurement that would change the balance** is §2.3 and §3: if a locked root also closes the PVE +web interface and the console, then a tester whose unattended install completes but whose pairing +fails has a machine nobody can reach — which argues for interactive-default independently of the disk +question. Those measurements are blocked, so the balance cannot be struck yet. + +**Operator-built versus public need not match, and two profiles from one build is cheap.** An +operator-built image targets a machine whose disks are known, so automated-default is safe there. The +public image is the one facing unseen hardware. Recommending the split is not the same as ruling the +default, and the split is available regardless of how the ruling goes. + +--- + +## 6. Q5 — the resulting shape, as constraints + +Constraints on a build spec, each with a release-gate criterion. **Not an implementation, and nothing +was built.** Constraints marked ⚠ depend on blocked measurements and are provisional. + +| # | Constraint | Release gate | +|---|---|---| +| 1 | **The image must not select a disk by property on unseen hardware.** Spike 1 §3.2/§3.3. | The rendered answer contains no `filter.*` key, **or** the image is operator-built for a named machine. Grep the rendered `answer.toml`. | +| 2 | **The interactive installer must be reachable from the boot menu.** §5.1 shows the cost is two count gates. | `final.iso`'s `grub.cfg` contains ≥1 interactive entry and still contains no live `proxtui`/`proxdebug`/`nomodeset`/`Rescue Boot`/`memtest`/`fwsetup` — i.e. keep gate 3, relax gates 1 and 4 to an allowlist. | +| 3 | ⚠ **No usable baked credential.** `root-password-hashed = "*"` is schema-valid and installs (Spike 1). | The rendered answer's `root-password-hashed` is exactly `*`, and **no `.rootpw.txt` is emitted**. Both are one-line assertions in the build. | +| 4 | ⚠ **If a per-box password is set, it is set at `before-network`.** §4.3 — the only ordering with a zero window by construction. | The rendered answer's `[first-boot].ordering` is `before-network`; and the hook's positive marker is observed in a boot test, never inferred from absent errors (§4.4, R-150). | +| 5 | **No `root-ssh-keys`, ever, in a public image.** Spike 1 R-142. | `grep -c root-ssh-keys` on the rendered answer is `0`. | +| 6 | **The profile must be committed.** Spike 1 found demo-felhom's baked key comes from an *uncommitted* profile, so a reference box is not reproducible from the repo. | The `--profile` path resolves inside the repo working tree **and** `git status --porcelain` is empty for it. **R-151.** | +| 7 | **The artefact must be reproducible.** Spike 1 R-144 (`nested-probe`'s profile no longer exists). | Manifest records the profile's committed path and the repo commit; both resolve. | +| 8 | **A recovery path must be documented and, if operator-only, said so plainly.** §3.3. | The release notes name the recovery route for a failed pairing. If that route is `init=/bin/bash`, the notes say it is operator-only. | + +**What the image must not contain:** a usable root password hash, any `root-ssh-keys`, any customer +id, claim code, retrieval passphrase or hub token, and any `filter.*` disk selector in the public +variant. Spike 1 §6 established by enumeration that the only secret-bearing file is `/answer.toml`, so +gates 3 and 5 over that one rendered file cover the credential surface. + +--- + +## 7. The operator's decisions + +Two, each with the measurement that informs it and its current state. + +**Decision 1 — escrow or discard a per-box root password.** + +§3.4 changed the shape of this question. **"Discard" and "lock" are operationally identical** — if +nobody holds the password, `sulogin` refuses either way, and console recovery is closed to everyone. +So the choice is binary: + +- **Escrow** — the operator can always reach a box's console; the cost is a hub-held credential for + every box, which is the thing the D6 trust model says the hub alone must not be. +- **Discard / lock** — no credential exists anywhere; the cost is that a failed pairing leaves a + machine recoverable only by an operator at the keyboard (`init=/bin/bash`) or a reinstall. + +**Blocking measurement: §2.3** — whether a locked root also closes the PVE web interface. If it does, +"discard" removes the last remote route as well as the local one, and the balance shifts sharply +toward escrow. **Do not rule until §2.3 is measured.** + +**Decision 2 — the default boot entry for a public image.** + +Framed in §5.3. The disk evidence points at interactive-default; the product cost is the unattended +experience. **Partially blocked on the same §2.3 measurement**, because an unreachable-after-failure +box strengthens the interactive case independently. The operator-built-versus-public split (§5.3) is +available regardless and is cheap. + +--- + +## 8. What is still unknown, and what it would take + +| # | Unknown | What it takes | +|---|---|---| +| 1 | **Everything in §2.1, §2.3, §2.4, §3.1, §4.1, §4.2 (entropy/binaries).** | The staged fixture, unchanged, once a Tier 0 box returns. ~90 min for all three scenarios. | +| 2 | **Whether PVE alters the §2.2/§3.2 mechanism results** — PAM config, a `sulogin` override, `SYSTEMD_SULOGIN_FORCE`. | Same fixture; check `/etc/pam.d/`, the `sulogin` invocation, and systemd's emergency unit on the installed image. | +| 3 | **Whether the multi-match pick is deterministic by enumeration order** (Spike 1 §8 #1, still open). | Re-run Spike 1's S1 with reversed disk order. ~15 min. | +| 4 | **Whether the installer excludes its own boot medium** (Spike 1 §8 #2, still open). | USB boot on real hardware — HUMAN. | +| 5 | **What `network-online` ordering buys** (§4.3) — it is a real third option nobody has evaluated. | One extra scenario alongside the staged two. | +| 6 | **Whether a second GRUB entry actually boots the stock installer** after the gates are relaxed. | Build a probe ISO with a two-entry template and boot it. Not attempted — it would have meant changing repo files, which §4 forbids. | + +--- + +## 9. R-rows opened + +Numbering continues from R-147 (Spike 1). + +| R | Severity | Finding | +|---|---|---| +| **R-148** | MEDIUM | **`answer.toml.tmpl:27` justifies `ordering = "fully-up"` with a dependency the stub does not have.** The comment says the stub must run after `pveproxy` "so pvesh/pct work", but `stub-first-boot.sh` contains no `pvesh`/`pct`/`pveum`/`qm` call (grep `rc=1`); that usage lives in `felhom-bootstrap.sh`, which runs under its own `After=network-online.target … pveproxy.service` unit. A comment asserting a **dependency** rather than an invariant — and it has constrained the design, because `before-network` is the ordering that would make the credential window zero (§4.3). | +| **R-149** | LOW | **The `[first-boot].ordering` enum has three values, not two.** `before-network`, `network-online`, `fully-up` all validate; `network-pre` and `multi-user` are rejected. Spike 1 recorded two and the middle option has never been evaluated. | +| **R-150** | MEDIUM | **A first-boot hook that never runs is indistinguishable from one that succeeded.** All four `proxmox-first-boot-*.service` units are guarded by `ConditionPathExists=` / `ConditionPathIsReadWrite=`, and in systemd a failed condition is a **skip, not a failure** — no error, successful boot. Any design that puts credential rotation in a first-boot hook must verify it by a positive marker the hook emitted. Directly the "an absent log line is not evidence" class. | +| **R-151** | MEDIUM | **A reference box is not reproducible from the repository.** `demo-felhom` was installed from an ISO built with the **uncommitted** `n100-demo.profile`, which is where its baked SSH key comes from (Spike 1 §4.6). One of the two Tier 0 boxes therefore cannot be rebuilt from `main`. Related to R-144 (`nested-probe`'s profile is gone entirely); this row is the *reference box*, which matters more. Gate 6 in §6 exists to stop a public image inheriting the shape. | + +--- + +## 10. Teardown + +All three layers, verified positively. + +**Layer 1 — demo-hp.** **INCOMPLETE, and named as such.** The box went offline (§0) while +`/mnt/nvme-1tb/spike2/` held the probe ISO (~1.7 GB) and the staged answer files and runners. Nothing +was running on it: the last successful command before the outage confirmed `0` QEMU processes, and no +install had been started this session. Nothing was registered as a PVE storage, nothing was written to +`local-lvm`, and `drill-r50` / guest 9201 were never touched this session. + +**Outstanding cleanup, to run when the box returns** — recorded here because R-131 exists precisely +because recorded commands never get run: + +``` +rm -rf /mnt/nvme-1tb/spike2 ; rm -f /tmp/paia ; pkill -9 qemu-system-x86 +``` + +This is deliberately *not* claimed as done. Spike 1's `/mnt/nvme-1tb/spike-scratch` and both `/tmp/paia` +copies were removed and verified at that session's teardown, so the outstanding item is this session's +`spike2/` directory alone. + +**Layer 2 — demo-felhom.** Nothing placed this session — it was never reached before the outage. +Spike 1's `/tmp/paia` was removed and verified then. No action outstanding. + +**Layer 3 — DooPlex.** + +| Check | Result | +|---|---| +| scratchpad | **4.4 GB → 48 K** (probe ISO, squashfs extracts, `.deb` extract, schema test dirs removed) | +| control-password file (`s2/.ctlpw`) | `shred -u`'d — never printed, never committed; verified absent | +| mechanism-probe container | `docker run --rm` — no image or container retained; **no global Docker cleanup run** (fence) | +| `felhom-iso/out/` | untouched, **17 ISOs** | +| production files | none modified — no `build-felhom-iso.sh`, no repack, no profile, no `felhom-bootstrap.sh` | + +**One teardown error, recorded rather than quietly fixed.** Shredding the control password +(`s2/.ctlpw`) destroyed the plaintext that scenario **C** existed to use — a `$6$` hash whose plaintext +is gone cannot demonstrate "correct password → PVE ticket", which was C's entire purpose as the +control for §2.3. `s2/scen/C.toml` was therefore **deleted** rather than left in place looking usable. +**Scenario C must be regenerated** (one `openssl passwd -6` against a fresh random plaintext, kept +0600) before the §2.3 measurement is attempted. Scenarios A and B are unaffected and remain staged. + +**Hub-side disposition — nothing to dispose.** No customer, host or appliance was created: no install +ran, and the only hub contact this session was none at all. Verified rather than asserted below. + +**Published artefacts: none.** Nothing went to R2. diff --git a/scripts/CHANGELOG.md b/scripts/CHANGELOG.md index 7ae48ff..1195ce7 100644 --- a/scripts/CHANGELOG.md +++ b/scripts/CHANGELOG.md @@ -1,3 +1,39 @@ +## SPIKE 2 (no code change) — before-network gives a zero window by construction; locked root closes sulogin (2026-07-31) + +**No script, profile or build file was modified.** Findings: +`documentation/audits/SPIKE-universal-iso-2-2026-07-31.md`. Opens R-148..R-151. + +**Both Tier 0 boxes went offline mid-session** (remote site, 12:28 CEST; four routes tried, our +tailscale pod healthy). Q1/Q2/Q3 each retain a part that needs a nested VM and those parts are +**BLOCKED, not answered**. The fixture is staged with Spike 1's two defects already corrected. + +Established without them: + +- **STRUCTURAL — `ordering = "before-network"` maps to `proxmox-first-boot-network-pre.service` + (`proxmox-first-boot` 9.2.5), which is `Before=network-pre.target` + `Type=oneshot`.** It therefore + runs to completion before any interface is configured — a credential rotation there has a + **zero-length window by construction**, not by being fast. Stronger than the timing number Spike 1 + failed twice to obtain. +- **STRUCTURAL — the stub does not need `fully-up`.** `stub-first-boot.sh` contains no + `pvesh`/`pct`/`pveum`/`qm` call (grep rc=1); that usage is in `felhom-bootstrap.sh`, which runs under + its own `After=network-online.target ... pveproxy.service` unit. So `answer.toml.tmpl:27` justifies + the current ordering with a dependency that does not exist. **R-148.** +- **The ordering enum has THREE values** — `before-network`, `network-online`, `fully-up`. Spike 1 + recorded two; the middle one has never been evaluated. **R-149.** +- **MECHANISM (container, not PVE) — a locked root closes `sulogin`:** `Cannot open access to console, + the root account is locked.` for both `*` and `!`, with a working control (correct password -> + SU_OK, wrong -> Authentication failure). So "discard" and "lock" are the SAME outcome for recovery, + which makes the escrow decision binary. +- **All four `proxmox-first-boot-*` units are `Condition*`-guarded, and a failed condition is a SKIP, + not a failure** — a hook that never ran looks exactly like one that succeeded. **R-150.** +- Q4: the single-entry menu is enforced by **four** gates in `iso-repack.sh` (`:153-155`, `:156`, + `:160-164`, `:314-319`), so restoring an interactive entry is a gate change, not a template edit — + but the banned-token gate would pass unchanged for a stock Graphical entry. `default`/`timeout` are + already settable in `grub/grub.cfg.tmpl`. + +The first mechanism probe was invalid (ran as uid 0, so `su` never invoked pam_unix; `sulogin` had no +tty) and is recorded as failed, not massaged. + ## SPIKE (no code change) — a universal ISO needs a different disk strategy and a locked root (2026-07-31) **No script, profile or build file was modified.** Findings only: