diff --git a/REPORT-universal-iso-spike.md b/REPORT-universal-iso-spike.md new file mode 100644 index 0000000..cb50dc5 --- /dev/null +++ b/REPORT-universal-iso-spike.md @@ -0,0 +1,93 @@ +# REPORT — SPIKE: what does a genuinely universal ISO require? (2026-07-31) + +> Written as **`REPORT-universal-iso-spike.md`**, not `REPORT.md`: the shared file was overwritten at +> 09:24 today by the hub v0.85.0 Network-card session, and the standing rule is that the second +> session in a shared clone never touches it. + +**Class: Spike.** Output is a findings document. **No production code was changed, no ISO was built, +nothing was published.** Full evidence: +`documentation/audits/SPIKE-universal-iso-2026-07-31.md`. + +## Why this ran + +The ISO build/publish task stopped before its Part 2. Its Part 0 found that every ISO bakes a root +password hash into `/answer.toml`, and that `generic.profile` picks a disk by the name `sda` rather +than by identity — both of which make a public download a product decision. This spike establishes +what a genuinely universal image would require. + +## What was answered + +**Q1 — disk selection. The central finding, and it is worse than the known hazard.** + +A filter that matches more than one device **does not fail safe**. Observed in a nested VM with two +identical disks and `filter.ID_SERIAL_SHORT = "SPIKE*"`: the installer silently selected one and wiped +it (`disk0` hash `de001102…` → `382b55ca…`; `disk1` byte-identical throughout). `validate-answer` +accepts the answer — it cannot know what the filter resolves to on the target. The +`filter did not match any devices` guard the safety profiles rely on covers the **zero**-match case +only. + +And a filter that reliably matches exactly one disk on unseen hardware cannot be written, because +**no property distinguishes an internal system disk from external media.** Measured on `demo-felhom` +with its 1 TB external drive attached — the exact hazard configuration: + +- `ID_BUS` is `ata` for **both** (the external unit is a SATA drive in a USB caddy) +- `lsblk RM` is `0` for **both**; `device-info` exposes no removability property at all +- on `demo-hp`, the NVMe carries **no `ID_BUS` and no `ID_TYPE`** — a different property set entirely +- nothing anywhere expresses *role*: on `demo-hp` the system disk is the small SATA one and the big + NVMe is customer data + +**Q2 — root password.** A credential field is mandatory (`One of global.root-password or +global.root-password-hashed must be set`), but a *usable* one is not: `root-password-hashed = "*"` +validates **and installs to completion** (guest powered itself off after 200 s using +`reboot-mode = "power-off"` as the completion signal). `[first-boot].ordering` also accepts +`before-network` — the only ordering that could close the exposure window structurally, since today's +bootstrap unit is `After=network-online.target pveproxy.service` and the pairing poll waits +indefinitely for the operator's bind. + +**Q3 — two-entry boot menu: yes, trivially.** `prepare-iso` leaves `grub.cfg` **byte-identical** to +stock; a prepared ISO has 15 entries including automated *and* interactive. Felhom's repack is what +collapses it to one. No PVE-side obstacle exists. + +**Q4 — what else is baked.** By enumeration against the stock ISO, not pattern-guessing: the pipeline +adds exactly six paths; both OS squashfs payloads are byte-identical to stock; all three first-boot +payloads match repo HEAD or carry only the hub URL. `/answer.toml` is the only secret-bearing content. + +**R-129 resolved.** demo-hp's key is the operator's own, added post-install — not baked. But +demo-felhom's *is* baked, by an uncommitted profile, so the "no baked key" claim was already false for +one demo box and nothing recorded it. + +## What failed, and is recorded as failed + +**The reachable-before-rotation measurement was attempted twice and both attempts were invalid.** +Attempt 1 reported a plausible **`0 s`** that was an artefact — QEMU's `hostfwd` accepts on the host +before touching the guest, and the same run then failed to read an SSH banner at all. Attempt 2, with +a protocol-level detector, reported `never` because the install VM had no NIC, so the installed +system's interface config could not match the boot VM's. **No number is reported.** The structural +finding (unit ordering + unbounded pairing poll) stands instead and is the stronger claim. + +Also not established: that the locked root account is actually locked (install completion is a +different claim), and whether GRUB single-user recovery works — the latter gates the escrow-vs-discard +ruling and should be measured before that ruling is made. + +## Asked of the operator + +One decision, framed in §4.5 of the findings doc: **escrow or discard** a per-box root password. Not +recommended either way — the D6 trust model argues against a hub-held fleet-wide vault, and an unbound +box has no agent access, which is exactly when console access matters. + +## R-rows + +**R-139** (HIGH, multi-match wipes silently) · **R-140** (`device-match` exits 0 on no-match) · +**R-141** (HIGH, mandatory + fleet-shared credential, unbounded window) · **R-142** (`FELHOM_ROOT_SSH_KEY` +can ship a shared key, no assertion) · **R-143** (retired `kisfenyo@windows` key still live on +demo-felhom) · **R-144** (`nested-probe` ISO unreproducible — profile gone) · **R-145** (variant naming +undocumented) · **R-146** (`root-password-hashed = ""` validates) · **R-147** (hash substituted into +the answer template's own comments) · **R-128** restated (a comment, not an assertion). + +## Teardown + +All three layers verified positively: demo-hp scratch removed and `drill-r50`/9201/`storage.cfg` +untouched, `local-lvm` never written; demo-felhom probe binary removed and its external drive observed +only; DooPlex scratchpad 3.2 GB → 28 K with the throwaway password `shred`ed. **No hub object was +created** — confirmed by fetching the customer list and searching it, not by assertion. Nothing +deleted from `felhom-iso/out/` (17 ISOs intact). Nothing published. diff --git a/documentation/audits/SPIKE-universal-iso-2026-07-31.md b/documentation/audits/SPIKE-universal-iso-2026-07-31.md new file mode 100644 index 0000000..db84d0c --- /dev/null +++ b/documentation/audits/SPIKE-universal-iso-2026-07-31.md @@ -0,0 +1,606 @@ +# SPIKE — what does a genuinely universal ISO require? (2026-07-31) + +> **Class: Spike.** Output is findings, not shipped code. No production file was changed, no release +> ISO was built, nothing was published. Probe artefacts are quoted here and were removed at teardown. +> +> **Origin.** The ISO build/publish task (2026-07-31) stopped before its Part 2 when its Part 0 found +> two hazards that make a public download a product decision rather than a build step. This spike +> establishes what a genuinely universal image would require. + +--- + +## 1. Baselines + +Read fresh at the start of this session, not carried from the previous one. + +| Baseline | Value | Source | +|---|---|---| +| `ISO_VERSION` | `1.25.0` | `scripts/iso/build-felhom-iso.sh:44` | +| `SCRIPT_VERSION` | `1.22.0` | `scripts/felhom-host-install.sh:187` | +| `felhom-bootstrap.sh` @ HEAD | `21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7` | `sha256sum scripts/iso/felhom-bootstrap.sh` | +| PVE base ISO | `proxmox-ve_9.2-1.iso`, `4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c` | `sha256sum /mnt/5_hdd/felhom.eu/drill/proxmox-ve_9.2-1.iso` — matches every ISO manifest's `source-iso-sha256` | +| `felhom.eu` HEAD | `5825ceeabfc50227bee2ccf9cf35c412c05c0f2a`, clean, `== origin/main` | `git status --porcelain` empty | +| assistant tooling | `proxmox-installer-common v9.2.7` | `proxmox-auto-install-assistant --version` | + +**Fixture provenance.** Nested VMs ran under plain QEMU/KVM on **demo-hp** (t740, Tier 0, the +designated drill + build VM host per the 2026-07-25 standing ruling), with all scratch files under +`/mnt/nvme-1tb/spike-scratch/`. **Nothing was placed on demo-hp's `local-lvm`** and no PVE storage was +registered. `drill-r50` (VM 300) and guest 9201 were not touched. The probe ISO was built on DooPlex +from the baseline PVE ISO with `prepare-iso --fetch-from partition` and transferred once +(`sha256 337985c5c88b8d76c7a0a6e1678b48063f3b7d077222c91b9a1a38cfb6ca8c75`, verified identical at both +ends), so every scenario reused one image and varied only a 64 MB answer disk. + +**Real-hardware work was read-only.** `device-info` was executed on `demo-felhom` and `demo-hp` via a +temporary copy of the assistant binary in `/tmp` (removed at teardown). No install, no partitioning, +no attach/detach. + +--- + +## 2. Reconfirmation of the two established findings + +Both re-verified at `file:line` rather than carried forward. + +**(a) Every ISO bakes a root password hash.** `build-felhom-iso.sh:270-283` mints `ROOT_PLAIN` +unconditionally, hashes it with `openssl passwd -6`, and the renderer substitutes it into +`__ROOT_HASH__` (`:295-296`) from `answer.toml.tmpl:24`. There is no flag to suppress it. + +Verified by **extraction, not inference**: `osirrox` pulled `/answer.toml` out of the shipped +`felhom-pve-9.2-1-v1.25.0-nested-vm-generic-mkimage.iso` and it carries a live `root-password-hashed` +field. The sibling `.rootpw.txt` plaintext was confirmed to hash to exactly that value. Any downloader +can perform the same extraction with no privilege. + +**(b) `generic.profile` selects by name, not identity.** `profiles/generic.profile:16-18` sets +`disk-list = ["sda"]`. Confirmed present verbatim in the extracted `/answer.toml` of the v1.25.0 build. +`generic.profile:12-14` concedes the limit in its own comment. The serial pin that closed this hazard +on the N100 rehearsal (`hp.profile:36`) does not generalise to strangers' hardware. + +--- + +## 3. Q1 — what the answer file can express about disk selection + +### 3.1 The selection mechanisms (read from schema) + +Extracted from the assistant binary's own serde field strings and confirmed against +`validate-answer` behaviour. `[disk-setup]` is `struct DiskSetup with 7 elements`: + +| Key | Purpose | +|---|---| +| `filesystem` | `ext4` / `xfs` / `zfs` / `btrfs` | +| `disk-list` | explicit device names | +| `filter` | map of udev-property globs | +| `filter-match` | `any` \| `all` | +| `lvm` / `zfs` / `btrfs` | per-filesystem option sub-tables | + +The binary's validation strings pin the rules: + +``` +Need either 'disk-list' or 'filter' set +Cannot use both, 'disk-list' and 'filter' +make sure to define only one disk for ext4 and xfs +filter did not match any devices +no filter defined +invalid glob in disk selection +``` + +Glob syntax is `?`, `*`, `[a]`, `[0-9]`, `[!a]` (`device-match --help`). **`[!…]` negates a single +character, not a match** — there is no "does not contain" and no "property is absent" operator. + +**There is no exclusivity keyword.** Nothing in the 7 elements expresses "abort unless exactly one +candidate matches". The nearest thing is the `only one disk for ext4 and xfs` rule — a consequence of +the filesystem choice, not a selection control (measured behaviour in §3.3). + +### 3.2 What distinguishes an internal system disk from external media — **measured** + +`device-info -t disk`, captured on both Tier 0 boxes. + +**`demo-felhom` — the hazard configuration** (internal SATA SSD + the 1 TB external USB drive): + +| | `sda` (internal) | `sdb` (external USB) | +|---|---|---| +| `ID_BUS` | `ata` | **`ata`** | +| `ID_PATH` | `pci-0000:00:17.0-ata-2.0` | `pci-0000:00:14.0-usb-0:1:1.0-scsi-0:0:0:0` | +| `ID_MODEL` | `AirDisk_512GB_SSD` | `TOSHIBA_MQ04ABF100` | +| `ID_USB_DRIVER` | *absent* | `usb-storage` | +| `ID_USB_VENDOR` | *absent* | `ADATA` | +| `lsblk RM` (removable) | `0` | **`0`** | +| `lsblk HOTPLUG` | `0` | `1` | + +Three findings, and the first two are the important ones: + +1. **`ID_BUS` does not distinguish them.** Both report `ata`. The external unit is a SATA drive in a + USB caddy (an ADATA HD710 PRO enclosure around a Toshiba MQ04ABF100), so the bus property describes + the drive, not the attachment. A filter written as "internal means `ID_BUS=ata`" matches the + customer's backup drive. +2. **There is no removability property at all** in `device-info` output — no `RM`, no `ID_REMOVABLE`. + And `lsblk` reports `RM=0` for the external drive anyway, so even reaching outside `device-info` + does not supply the missing concept. **Property filtering cannot express "not removable."** +3. What *does* distinguish them here is `ID_PATH` (`-ata-` vs `-usb-`) and the presence of the + `ID_USB_*` key family — but only the former is expressible, because the filter language has no + "key is absent" operator. + +**`demo-hp`** (internal SATA system disk + a 1 TB NVMe data disk): + +| | `sda` (SATA, the PVE system disk) | `nvme0n1` (NVMe, enrolled user data) | +|---|---|---| +| property count | 45 | 28 | +| `ID_BUS` | `ata` | **absent** | +| `ID_TYPE` | `disk` | **absent** | +| `ID_PATH` | `pci-0000:06:00.0-ata-1.0` | `pci-0000:04:00.0-nvme-1` | + +Absence verified explicitly by key lookup, not by a grep that happened not to print. + +4. **NVMe devices carry a different property set entirely** — no `ID_BUS`, no `ID_TYPE`. Any filter + built on `ID_BUS` silently excludes every NVMe-only machine. +5. **Decisively: no property expresses *role*.** Every captured property describes hardware — bus, + path, model, serial, size. None says "this is the disk the operating system belongs on." On + `demo-hp` the system disk is the *small* SATA one and the big NVMe is user data; a heuristic like + "largest disk" or "the NVMe one" targets the customer's data drive on exactly the configuration + Felhom sells. + +`device-match` confirmed the expressible cases on `demo-felhom` (read-only): + +``` +ID_BUS=ata -> ["sda","sdb"] both — the naive filter hits the backup drive +ID_PATH=*usb* -> ["sdb"] isolates the external +ID_PATH=pci-*-ata-* -> ["sda"] isolates the internal, on this box +``` + +Both isolating filters are box-specific: the `-ata-` pattern excludes every NVMe system disk, and the +`*usb*` pattern is an exclusion that the filter language cannot invert. + +### 3.3 Multi-match behaviour — **observed. It is the coin-flip.** + +This is the question the spike existed to answer, and it was measured, not inferred. + +**Method.** A nested VM on demo-hp with two identical 16 GB virtio-SCSI disks carrying serials +`SPIKE0` and `SPIKE1`, plus a separate answer disk (serial `ANSWERDISK`, label `proxmox-ais`). Answer: + +```toml +[disk-setup] +filesystem = "ext4" +filter-match = "all" +filter.ID_SERIAL_SHORT = "SPIKE*" # matches SPIKE0 and SPIKE1 — two devices +``` + +The answer **validated cleanly** (`The answer file was parsed successfully, no errors found!`) — so +`validate-answer` cannot catch this; it does not know what the filter will resolve to on the target. + +The measurement is deliberately not a screen reading: both disk images were zeroed and hashed before +the run and re-hashed after, so "which disk was written" is objective. + +| Disk | sha256 (first 32) BEFORE | AFTER | Verdict | +|---|---|---|---| +| `disk0.qcow2` (`SPIKE0`) | `de001102cbc8ff7aa8eb6c69d750cf22` | `382b55cad8027c60dfcbed254e723123` | **WIPED AND INSTALLED ONTO** | +| `disk1.qcow2` (`SPIKE1`) | `de001102cbc8ff7aa8eb6c69d750cf22` | `de001102cbc8ff7aa8eb6c69d750cf22` | untouched | + +Observed mid-run, confirming it was a real install and not a partial write: `disk0.qcow2` had grown to +5,441,388,544 bytes while `disk1.qcow2` sat at 196,864 bytes (bare qcow2 metadata). + +**The installer did not abort. It silently selected one of the two matching disks and wiped it.** + +Two consequences: + +1. **`filter-match` has no fail-safe on ambiguity.** The `filter did not match any devices` guard — + the spike-S5c behaviour the `nested-canary` profile pins, and the safety default the `hp`/`n100` + profiles rely on — protects only the **zero**-match case. The **many**-match case is unprotected and + silent. +2. The binary's `make sure to define only one disk for ext4 and xfs` string does **not** cover this. + It rejects an explicit multi-entry `disk-list`; a `filter` that *resolves* to several devices sails + past both `validate-answer` and the installer. + +**Why this matters more than the `disk-list=["sda"]` hazard.** The previously known hazard is that +`sda` may name the wrong disk. This one is worse in kind: a filter written specifically to be *safe* +— "match the internal SSD" — degrades to an arbitrary pick the moment a second device matches, with no +diagnostic. Given §3.2 (no removability property, `ID_BUS` identical on the hazard configuration), +writing a filter that provably matches exactly one device on unseen hardware is not achievable. + +**Not established:** whether the selection is deterministic by enumeration order or genuinely +arbitrary. Only one ordering was tested. See §8. + +### 3.4 Is the boot medium a candidate? + +Partially measured. On `demo-felhom` the USB-attached drive **is** enumerated by `device-info` as an +ordinary candidate disk with no marking distinguishing it from the internal one — a USB installer +stick is the same class of block device, so nothing in the captured data suggests the enumeration +would exclude it. + +**Not fully established.** Whether the installer *itself* excludes the live medium it booted from is a +separate question from whether `device-info` enumerates it, and this spike did not boot from USB on +real hardware. See §8. + +--- + +## 4. Q2 — can the root password be generated on the box instead of baked? + +### 4.1 Is a password required? — **measured** + +The binary carries both rules verbatim: + +``` +One of `global.root-password` or `global.root-password-hashed` must be set +`global.root-password` and `global.root-password-hashed` cannot be set at the same time +``` + +Confirmed empirically against `validate-answer`: + +| Answer | Result | +|---|---| +| root password omitted entirely | **rejected** — `One of global.root-password or global.root-password-hashed must be set` | +| `root-password-hashed = "*"` | **accepted** | +| `root-password-hashed = "!"` | **accepted** | +| `root-password-hashed = ""` | **accepted** | +| both plaintext and hashed set | **rejected** | + +**A root password field cannot be omitted — but the classic locked-account crypt placeholders `*` and +`!` validate.** That is the opening: the field is mandatory, a *usable credential* is not. + +That an empty string also validates is worth a fence of its own — `""` is not a locked account, and +whether it yields a passwordless root depends on how it reaches `/etc/shadow`. + +**Validation is not installation, so it was installed** — and the first attempt to confirm that is +worth recording, because it nearly produced a false finding. + +*S2, first attempt.* A nested VM ran an unattended install whose only credential was +`root-password-hashed = "*"`. `disk0.qcow2` went from `de001102cbc8ff7aa8eb6c69d750cf22` to +`f4ae86b8317cc36af4628a6cd9c09da2` and grew to several GB, which reads like success. Booting the +result showed otherwise: SeaBIOS printed `Booting from Hard Disk...` and hung. The install had been cut +off mid-flight by the probe's fixed 300 s window; a written disk is evidence of *writing*, not of +*completion*. **A changed hash is a `LastRun`-class timestamp — it records that something was +attempted.** + +*S2b, with an actual completion signal.* `reboot-mode = "power-off"` is a valid enum value +(`"poweroff"` is rejected, so this is a real enum), and it makes the guest power itself off when the +install finishes — i.e. the qemu process exiting **is** the completion signal, and a timeout is +unambiguously a failure rather than an unknown. + +| Disk | sha256 (first 32) BEFORE | AFTER | Completion signal | +|---|---|---|---| +| `disk0.qcow2` | `de001102cbc8ff7aa8eb6c69d750cf22` | `dbc12e129ed84e834f6d54754cb0a2c8` | **guest powered itself off after 200 s** | + +**Established: an unattended install whose only credential is `root-password-hashed = "*"` runs to +completion.** The mandatory-field constraint does not force a *usable* baked credential — it forces a +*field*, and a locked-account placeholder satisfies it. This is the mechanism a universal image needs. + +**Not established: that the resulting root account is actually locked.** The install completing is not +the same claim. Confirming it needs either `/etc/shadow` from the installed image or a refused login, +and neither was obtained — see §4.3 and §8. + +### 4.2 Can `felhom-bootstrap` rotate root at first boot? + +Mechanically yes — it already runs as root at first boot, and `felhom-host-install.sh:1831-1871` +(step 4b, G1) already implements exactly this: generate 24 url-safe bytes, set via `chpasswd` on +stdin, vault to the hub, scrub the variable. The code to rotate exists and is proven. + +**The problem is not capability, it is ordering** — see §4.3. + +### 4.3 The exposure window — **structural finding, and it is not zero** + +`felhom-bootstrap.service` is ordered: + +``` +After=network-online.target pve-cluster.service pveproxy.service +Wants=network-online.target +``` + +So bootstrap **starts after the network is up and after pveproxy is listening**. Whatever it does to +the root password happens strictly later than the box becoming reachable. The window is open by +construction. + +Worse, in pairing mode it does not close on its own. `felhom-bootstrap.sh:9-22,52-54,363` registers the +box as an unclaimed appliance and then **polls indefinitely** (`POLL_INTERVAL=30`) for the operator's +bind, deliberately staying in `activating` because "waiting is not failing" (v1.21.0, R-33). Step 4b +only runs after enroll, and enroll only happens after the bind. **A box that is downloaded, installed, +and never bound sits on its owner's LAN with the ISO-baked password indefinitely.** + +**The measurement the task asked for was attempted twice and BOTH ATTEMPTS FAILED. No number is +reported, because both numbers were artefacts of the fixture rather than facts about the system.** + +*Attempt 1 — invalid, and it produced a plausible false zero.* The probe booted the installed disk +under QEMU with `hostfwd` port forwards and polled with a TCP connect. Both ports "accepted" at +**+0 s**, which reads like "reachable instantly". It is not: QEMU's user-mode networking binds the +forwarding socket on the *host* when QEMU starts and accepts the connection before it ever tries the +guest. The same run then failed to read an SSH banner and reported `Connection timed out during banner +exchange` — i.e. nothing was listening in the guest at all. **A TCP connect against a `hostfwd` port +proves nothing.** Had the banner check not been in the same script, `0 s` would have been recorded as +a measured finding, and it would have been the exact opposite of the truth. + +*Attempt 2 — invalid for a different reason.* The detector was rewritten to require a real `SSH-` +version banner and a real HTTP response line. It then reported `never` for both, over 6 minutes. That +is also not a fact about PVE: the S2b install VM was created **with no network device at all**, so the +installed system's interface configuration cannot match the NIC that only exists in the boot VM. The +fixture, not the system, produced the result. + +**What stands instead** is the structural finding above, which does not depend on either probe: the +unit's own `After=network-online.target pveproxy.service` places every rotation opportunity strictly +after reachability, and the pairing loop makes the interval unbounded. That is a stronger claim than a +single timing number would have been — it holds for every box rather than for one VM. + +**What it would take to measure it properly:** install and boot with an identical NIC present in both +phases, and probe at protocol level from the host. ~30 min. It is worth doing before a spec is +written, because the number is what tells the operator whether the window is seconds or minutes when +pairing *does* proceed promptly. + +**The mechanism that would close it:** `[first-boot].ordering` accepts **`before-network`** as well as +`fully-up` — verified against `validate-answer` (`network-pre` and a bogus value are both rejected, so +this is a real enum, not a permissive parser). A hook running `before-network` executes before the box +is reachable at all, which is what a zero-window rotation requires. + +Felhom currently uses `ordering = "fully-up"` (`answer.toml.tmpl:39`), deliberately, so that `pvesh` +and `pct` work (spike S3/S8a). Note the current stub (`stub-first-boot.sh`) only writes three files and +runs `systemctl enable` + `start --no-block` — none of which obviously needs the network, and the unit +it starts carries its own `After=network-online.target`. **Whether the stub survives being moved to +`before-network` is untested and is the single highest-value follow-up probe.** `[first-boot]` accepts +only one hook (`struct FirstBootHookInfo with 4 elements`), so this is a move, not an addition. + +### 4.4 Console recovery if pairing fails + +**Not tested — stated plainly rather than assumed.** The spike ran out of room before booting the S2b +image into a GRUB single-user / `init=/bin/sh` shell. + +This is not a detail. It is the input that decides §4.5: if the standard GRUB recovery path yields a +root shell on a locked-root install, then discarding the password costs little, because a physically +present person can always get back in. If it does not — or if PVE's installed GRUB is configured +without an editable menu — then discarding turns a failed pairing into a reinstall, and the escrow +argument is much stronger. **The ruling in §4.5 should not be made before this is measured** (§8, #4). + +### 4.5 Escrow or discard — the tradeoff, for the operator to rule + +Not a recommendation; the inputs. + +**For discarding** (no per-box root password is ever recoverable): +- The operator already holds agent-level access on every *bound* box, so console root is redundant for + the normal case. +- Under the D6 trust model the hub alone must not be sufficient. A vault of console root passwords for + the whole fleet makes the hub exactly that, for every box at once — a strictly worse blast radius + than the shared-ISO-password problem this is meant to fix. +- It removes a class of credential that has to be rotated, audited, and leak-checked forever. + +**For escrowing** (the current G1 behaviour, extended to pre-bind): +- An *unbound* box has no agent access — precisely the case where console access is the only route in, + and precisely when things go wrong. +- G1 exists because the sshd path and the auto-heal both failing is a real, observed failure mode; the + PVE web console at `:8006` is a distinct failure domain from sshd. +- Discarding turns "pairing failed" into a reinstall for a non-technical customer. + +**The middle option worth costing:** rotate to a per-box random value at `before-network` and escrow +*nothing* until the box binds, at which point step 4b's existing vaulting takes over. This closes the +shared-credential window without adding a pre-bind fleet-wide vault — at the price of making an +unbound, broken box unrecoverable except by reinstall. Whether that price is acceptable is the +operator's call, and it is the one decision this spike asks for. + +### 4.6 R-129 — SSH key provenance, **resolved** + +The premise is confirmed: `ssh -o BatchMode=yes demo-hp` authenticates by key, while the documentation +says no baked key exists. The two are not in conflict — the key is not baked. + +| Box | `/root/.ssh/authorized_keys` → `/etc/pve/priv/authorized_keys` | Provenance | +|---|---|---| +| `demo-hp` | `SHA256:pgQh228R…` `nagyfenyvesi.viktor@gmail.com` | the operator's own DooPlex key, added post-install (file mtime 2026-07-27, install 2026-07-21). **This is what authenticates CC.** | +| | `SHA256:fnasIK1Y…` `root@felhom-host` (RSA 4096) | PVE's own self-generated root key | +| `demo-felhom` | `SHA256:a8zH+ecm…` `kisfenyo@windows` | **baked by the ISO** — exact fingerprint match to `FELHOM_ROOT_SSH_KEY` in the uncommitted `n100-demo.profile` | +| | `SHA256:BnLVXqTc…` `root@demo-felhom` (RSA) | PVE self-key | +| | `SHA256:pgQh228R…` | the operator's DooPlex key | + +Findings: + +1. **demo-hp's key is not baked** — it is the operator's own, added by hand. The documentation was + right about the ISO and wrong about the box. +2. **demo-felhom's key *is* baked**, via an uncommitted profile. So the "no baked key" claim was + already false for one of the two demo boxes, and nothing recorded it. +3. **No committed profile bakes a key** — all five have `FELHOM_ROOT_SSH_KEY` commented out — and the + shipped v1.25.0 ISO contains **zero** occurrences of `root-ssh-keys` in its `/answer.toml`. +4. **But the mechanism is one uncommented line from shipping a shared key to every downloader** + (`build-felhom-iso.sh:175-176` → `answer.toml.tmpl:26`). For a published image that is the same + shared-credential class as the baked password, and it needs a build-time assertion, not a + convention. +5. `kisfenyo@windows` is the retired Windows workstation's key and is still authorized on + `demo-felhom`. Stale credential; see R-rows. + +--- + +## 5. Q3 — can the image offer both automated and interactive install? + +**Yes, and it requires no PVE-side work at all.** Established by comparing three GRUB configs. + +`prepare-iso` **does not touch `grub.cfg`**: the file extracted from the freshly prepared probe ISO is +byte-identical to the one in the stock PVE ISO (`diff` clean). The stock config already carries the +automated entry, gated on a file that `prepare-iso` creates: + +``` +if [ -f auto-installer-mode.toml ]; then + set timeout-style=menu + set timeout=10 + menuentry 'Install Proxmox VE (Automated)' { … proxmox-start-auto-installer } +fi +menuentry 'Install Proxmox VE (Graphical)' { … } # outside the conditional +menuentry 'Install Proxmox VE (Terminal UI)' { … } # outside the conditional +``` + +| ISO | menu entries | +|---|---| +| stock `proxmox-ve_9.2-1.iso` | 15 | +| **prepared** (probe, `prepare-iso` only) | **15** — automated **and** graphical **and** TUI | +| Felhom `v1.25.0` remastered | **1** (`Felhom telepítés`) | + +So the answers are: + +1. The current remastering **replaces** the menu. `iso-repack.sh` rewrites `grub.cfg` from + `grub/grub.cfg.tmpl` to exactly one entry — by design, and documented as such + (`build-felhom-iso.sh:16-23`: the stock entries "are not hidden, they are not emitted"). This is + also the answer to the previous session's open question: the `mkimage`/non-`mkimage` split is + **not** this distinction — it is a UEFI loader swap, orthogonal to the menu, and both variants get + the same single-entry menu. +2. **Preparation does not consume the interactive path.** A multi-entry menu including the automated + entry is the *native* shape of a prepared auto-install ISO. Felhom collapses it deliberately, for a + reason that remains sound for a customer appliance (`build-felhom-iso.sh:18-21`: every stock entry + is a route into the manual installer whose first question is which disk to wipe). +3. Therefore a two-entry menu is purely a change to Felhom's own `grub/grub.cfg.tmpl` — keeping + `Felhom telepítés` as `default=0` and adding one guarded interactive entry. No PVE-side obstacle + exists. + +**Not demonstrated by boot.** The evidence here is configuration-level, which is weaker than a +screenshot of the menu. See §8. + +--- + +## 6. Q4 — what else is baked that a stranger should not receive + +Answered by **positive enumeration against the stock ISO**, not by grepping for patterns I guessed — +a pattern scan can only find what it was told to look for. + +**Method.** Full recursive file listing of the shipped `v1.25.0` ISO and the baseline PVE ISO +(`osirrox -find`, 1576 vs 1573 paths), diffed both directions; content hashes compared for the OS +payloads; all three first-boot payloads decoded and hashed against repo HEAD; the answer file's +credential fields enumerated by name. + +**Added by the Felhom pipeline — exactly six paths:** + +| Path | Content | Secret-bearing? | +|---|---|---| +| `/answer.toml` | the rendered answer | **YES — `root-password-hashed`** | +| `/auto-installer-mode.toml` | `mode = "iso"` | no | +| `/proxmox-first-boot` | the rendered stub + 3 embedded payloads | no (below) | +| `/boot/grub/felhomtheme/` + `background.png` + `theme.txt` | branding | no | + +**Removed:** the three stock `/boot/grub/pvetheme/` paths. + +**The OS payloads are untouched** — `pve-base.squashfs` and `pve-installer.squashfs` are byte-identical +to the stock ISO. Nothing is hidden in the installed system image. + +**The three first-boot payloads**, decoded from the stub: + +| Payload | Result | +|---|---| +| `felhom-bootstrap.sh` | sha256 **identical** to repo HEAD | +| `felhom-bootstrap.service` | sha256 **identical** to repo HEAD | +| pairing env | only `FELHOM_HUB_URL` + `FELHOM_INSTALL_URL` — no customer id, no passphrase | + +**Also scanned for** (across all four content-bearing files): PEM private-key headers, `ssh-rsa` / +`ssh-ed25519` key material, `Bearer` tokens, `api_key=`, `passphrase=`, `customer_id=`, `password=`, +and `token=` with ≥12-character values. The only hits were **empty variable initialisations** in +`felhom-bootstrap.sh:89` (`FELHOM_CUSTOMER_ID=""; FELHOM_MODE=""; FELHOM_RETRIEVAL_PASSPHRASE=""`) and +a substring test at `:480` — declarations, not values. + +**Conclusion for the recommended shape.** The earlier recon's "three payloads, secret-free" finding +holds *for the payloads it examined* and was scanning the wrong file for the credential question: +`/answer.toml` is a fourth, separate artefact on the ISO root and is the only secret-bearing content in +a pairing-mode build. In a universal image the entire question reduces to two fields in that one file +— `root-password-hashed` and `root-ssh-keys` — both of which need a build-time assertion rather than a +convention. + +--- + +## 7. The recommended shape of a universal ISO + +Stated as constraints on an eventual build spec. **Not an implementation, and nothing here was built.** + +1. **Disk selection cannot be solved by a filter, and must not be attempted.** §3.2 measured that no + property expresses removability or role, that `ID_BUS` fails on the exact hazard configuration, and + that NVMe carries a different property set. Any filter shipped to unknown hardware is a guess. + → **Constraint: a universal image must not choose a disk by property on hardware it has not seen.** + +2. **Ambiguity must fail closed, and the mechanism must be the one measured in §3.3** — not a filter + the schema has no keyword for. + +3. **The interactive installer must be reachable from the boot menu** (§5). This is the constraint + that makes the image universal in the strong sense: unattended where it is provably safe, and never + a dead end where it is not — including on hardware nobody anticipated, without a code change. It + costs one entry in `grub/grub.cfg.tmpl`. + +4. **No usable credential may be baked.** `root-password-hashed = "*"` is schema-valid (§4.1); the + field cannot be omitted but a usable password can be denied. Bake a locked account. + +5. **Rotation must precede reachability.** `ordering = "before-network"` is the only ordering that + closes the window structurally (§4.3). Anything after `network-online.target` — including + everything the current bootstrap does — is open by construction, and in pairing mode open + indefinitely. + +6. **Two build-time assertions, because both failures are silent and checkable:** the built image + carries no usable `root-password-hashed`, and carries no `root-ssh-keys` at all (§4.6.4). A comment + is not a guarantee; these are one-line greps over the rendered answer. + +7. **The published artefact must be reproducible from committed inputs.** `nested-probe`'s profile no + longer exists anywhere, so that ISO cannot be audited or rebuilt — a state a published image must + never reach. + +--- + +## 8. What is still unknown, and what it would take + +Stated plainly rather than inferred. Several of these were reachable and were not reached; that is a +budget outcome, not a judgement that they do not matter. + +| # | Unknown | What it would take | +|---|---|---| +| 1 | **Whether the multi-match pick is deterministic by enumeration order or arbitrary.** §3.3 tested one disk ordering and `SPIKE0` was chosen. "Always the first-enumerated" and "arbitrary" have the same evidence so far. | Re-run S1 with the two disks attached in reverse order, and a third run with differing sizes. ~15 min. Does not change the recommendation (either way it is unsafe), but it changes how the hazard is described to a tester. | +| 2 | **Whether the installer excludes the live medium it booted from.** §3.4 established only that USB block devices *are* enumerated as candidates. | Boot the installer from a USB stick on a real Tier 0 box with a filter that would match the stick. Needs physical presence — HUMAN. | +| 3 | **Whether the Felhom first-boot stub survives `ordering = "before-network"`.** This is the highest-value follow-up: it is the mechanism that would make the exposure window structurally zero (§4.3). The stub *appears* not to need the network, but "appears" is exactly the word this project has been burned by. | Build one probe ISO with `ordering = "before-network"` and the real stub, boot a nested VM, confirm the bootstrap unit still starts and pairing still completes. ~30 min. **Do this before any spec is written.** | +| 4 | **Whether GRUB single-user recovery works on a locked-root install** (Q2.4). Not tested. This is load-bearing for the discard-vs-escrow ruling: if `init=/bin/sh` from the GRUB menu gives a root shell, discarding is far cheaper than if it does not. | Boot the S2 disk image, edit the kernel line at the GRUB prompt, observe. Automatable via QMP `sendkey`. ~20 min. | +| 5 | **The two-entry menu is not demonstrated by boot** (§5). The evidence is configuration-level: `prepare-iso` provably does not touch `grub.cfg`, and the stock config provably contains both entries. A booted screenshot would be stronger. | Boot the probe ISO in a nested VM and screendump the menu. ~10 min. | +| 6 | **Whether `root-password-hashed = ""` produces a passwordless root or a locked one.** It validates (§4.1); what it writes to `/etc/shadow` was not observed. | One install + offline read. Low priority — nothing should ever set it — but it is a trap sitting in the schema. | + +--- + +## 9. R-rows opened + +Numbering continues from the highest in use (R-138). + +| R | Severity | Finding | +|---|---|---| +| **R-139** | **HIGH** | **A disk filter that matches more than one device does not fail safe — the installer silently picks one and wipes it** (§3.3, observed). The `filter did not match any devices` guard covers the zero-match case only. `validate-answer` cannot catch it. This invalidates "write a careful filter" as a strategy for unknown hardware, and it is the reason a universal ISO cannot select a disk unattended. | +| **R-140** | MEDIUM | `proxmox-auto-install-assistant device-match` **prints `Error: … filter did not match any devices` and exits 0.** Joins `validate-answer` (spike S1) in the exit-codes-that-lie class. Any tooling built on `device-match` must parse output, never `$?`. | +| **R-141** | **HIGH** | **The answer schema makes a root credential mandatory** (§4.1), so every ISO ships one. On a *published* image that is a fleet-wide shared credential, and in pairing mode the window does not close on its own: bootstrap is ordered after `network-online.target`, and the pairing poll waits indefinitely (§4.3). A box that is never bound keeps the download's password forever. | +| **R-142** | MEDIUM | **`FELHOM_ROOT_SSH_KEY` will bake a shared SSH key into every copy of a published image, and nothing prevents it.** No committed profile sets it and the v1.25.0 build contains none — but an uncommitted profile already did exactly this on `demo-felhom` (§4.6). Same shared-credential class as R-141; needs a build-time assertion over the rendered answer, not a convention. | +| **R-143** | LOW | **A retired credential is still live:** the `kisfenyo@windows` key (the decommissioned Windows workstation, §4.6) remains in `demo-felhom`'s `authorized_keys`. | +| **R-144** | MEDIUM | **`felhom-pve-9.2-1-v1.24.0-nested-probe-generic.iso` cannot be reproduced or audited** — no `nested-probe` profile exists anywhere in the repo or the build tree. A published artefact must never reach this state. | +| **R-145** | LOW | **The ISO variant naming convention is undocumented** (carried from the 2026-07-31 Part 0). `out/` holds 17 images whose labels mix hardware targets, lab purposes, a pairing-mode suffix and a loader suffix, with no written key. §3 and §5 of this document supply the meanings; they need a home in the ISO README. | +| **R-146** | LOW | **`root-password-hashed = ""` passes validation** (§4.1). An empty hash is not a locked account. Worth a fence wherever answers are generated. | +| **R-147** | LOW | The answer renderer substitutes `__ROOT_HASH__`/`__FQDN__` **inside the template's own comment lines** (`build-felhom-iso.sh:295-296` is line-oriented and unconditional), so the shipped `/answer.toml` repeats the real hash three times, twice in prose that was meant to be documentation. Cosmetic while the hash is on the ISO anyway; it stops being cosmetic if the credential ever becomes conditional. | +| **R-128** | *(open, restated)* | `build-felhom-iso.sh:44`'s claim that `ISO_VERSION` "aligns with felhom-host-install `SCRIPT_VERSION`" **is a comment, not an assertion.** `SCRIPT_VERSION` appears nowhere in `scripts/iso/` except inside that comment's own text, so nothing evaluates it; the values have drifted (1.25.0 vs 1.22.0). Instance #7 of the "a comment asserting an invariant needs a test pinning it" class. | + +--- + +## 10. Teardown + +All three layers, per `PROMPT-TEMPLATE.md` §13, each verified by a positive check rather than asserted. + +**Layer 1 — demo-hp (nested-VM host).** + +| Check | Result | +|---|---| +| QEMU processes | `0` remaining | +| `/mnt/nvme-1tb/spike-scratch/` (ISO, disk images, answer disk, probe scripts) | removed — `No such file or directory` | +| `/tmp/paia` (temporary assistant binary) | removed | +| loop devices | `0` | +| `/mnt/nvme-1tb/` contents | back to `dump e2d-images felhom-data images lost+found` — the pre-spike set | +| `drill-r50` (VM 300) | `stopped` — untouched, not destroyed (R-93 fixture intact) | +| guest 9201 | `running` — untouched | +| `/etc/pve/storage.cfg` | 4 storages, unchanged; **no scratch storage was ever registered** | +| `local-lvm` | never written to | + +**Layer 2 — demo-felhom (read-only probe host).** + +| Check | Result | +|---|---| +| `/tmp/paia` | removed | +| guest 9201 | `running` — untouched | +| external USB drive `sdb` | `931.5G usb /mnt/hdd_1` — still attached and mounted exactly as found; never attached, detached, partitioned or written | + +**Layer 3 — DooPlex (build host).** + +| Check | Result | +|---|---| +| scratchpad | 3.2 GB → 28 K (probe ISO, extracted ISO trees, PPM/PNG screendumps, file lists removed) | +| throwaway root password file | `shred -u`'d | +| `felhom-iso/out/` | **17 ISOs, nothing deleted** (fence honoured) | +| repo working tree | clean apart from this document | +| production files | none modified — no `.go`, no `build-felhom-iso.sh`, no profile | + +**Hub-side disposition — nothing to dispose.** This spike created no customer, no host, no appliance +registration: the nested VMs ran under plain QEMU with no network path to the hub, and the only hub +interaction was two authenticated **reads** of `/configuration` and `/configs`. Verified positively +rather than asserted — the customer list was fetched and searched for `spike|probe|scratch`-named +entries (none) and carries the same single pre-existing customer. **R-131 does not gain a row here.** + +**Published artefacts: none.** Nothing was uploaded to R2 or anywhere else. The R2 credential was +never created, used, or handled in this session. diff --git a/scripts/CHANGELOG.md b/scripts/CHANGELOG.md index 4ec3e53..7ae48ff 100644 --- a/scripts/CHANGELOG.md +++ b/scripts/CHANGELOG.md @@ -1,3 +1,30 @@ +## SPIKE (no code change) — a universal ISO needs a different disk strategy and a locked root (2026-07-31) + +**No script, profile or build file was modified.** Findings only: +`documentation/audits/SPIKE-universal-iso-2026-07-31.md`. Opened R-139..R-147; restated R-128. + +Three results that constrain any future universal-ISO spec: + +- **A disk filter matching more than one device does not fail safe.** Observed in a nested VM: with + `filter.ID_SERIAL_SHORT = "SPIKE*"` matching two disks and `filesystem = "ext4"`, the installer + silently picked one and wiped it (`disk0` hash changed, `disk1` byte-identical). `validate-answer` + accepts such an answer. The `filter did not match any devices` guard covers the ZERO-match case + only. **R-139.** +- **No udev property distinguishes an internal system disk from external media.** Measured on + `demo-felhom` with its 1 TB external drive attached: `ID_BUS` is `ata` for *both* (the external is a + SATA drive in a USB caddy), `lsblk RM` is `0` for both, and `device-info` exposes no removability + property at all. On `demo-hp` the NVMe carries no `ID_BUS` or `ID_TYPE` whatsoever. Nothing + expresses *role*. **R-141 context.** +- **A root credential is mandatory in the answer schema, but a usable one is not.** + `root-password-hashed = "*"` validates and installs to completion (guest powered off after 200 s + with `reboot-mode = "power-off"`). `[first-boot].ordering` also accepts `before-network`, which is + the only ordering that could close the exposure window structurally — today's bootstrap unit is + `After=network-online.target pveproxy.service` and the pairing poll waits indefinitely. **R-141.** + +Also: `prepare-iso` leaves `grub.cfg` byte-identical to stock (15 menu entries, automated **and** +interactive), so a two-entry boot menu is purely a change to Felhom's own `grub.cfg.tmpl` — the +single-entry menu is Felhom's repack, not a constraint of ISO preparation. + ## v1.22.0 — E-2 Part 2: new boxes get a real backup target, or are told they do not (2026-07-29) Every box installed before this got `local_backup_target: "local"` — the vzdump target on the SAME