SPIKE 2: before-network gives a zero window by construction; locked root closes sulogin
Findings only — no script, profile or build file changed; no ISO built, nothing published. documentation/audits/SPIKE-universal-iso-2-2026-07-31.md BOTH Tier 0 boxes went offline mid-session (remote site, 12:28 CEST; four routes tried, our tailscale pod healthy). Q1/Q2/Q3 each keep a part needing a nested VM: those are BLOCKED, not answered. DooPlex was NOT used as a fallback — Tier 2, and this task did not authorise it. Established without them: - STRUCTURAL: ordering='before-network' maps to proxmox-first-boot-network-pre.service (Before=network-pre.target, Type=oneshot) — it completes before ANY interface is configured, so a rotation there has a zero-length window BY CONSTRUCTION, not by being fast. - R-148: the stub does not need 'fully-up'. stub-first-boot.sh has no pvesh/pct/pveum/qm call (grep rc=1); that usage is in felhom-bootstrap.sh under its own After=network-online unit. answer.toml.tmpl:27 justifies the current ordering with a dependency that does not exist. - R-149: the ordering enum has THREE values (before-network, network-online, fully-up), not two. - MECHANISM (container, not PVE): locked root closes sulogin — 'the root account is locked' for both '*' and '!', with a working control. So 'discard' and 'lock' are the SAME outcome for recovery, making the escrow decision binary. - R-150: all four proxmox-first-boot-* units are Condition-guarded; a failed condition is a SKIP, so a hook that never ran looks identical to one that succeeded. - R-151: demo-felhom was installed from an UNCOMMITTED profile — a Tier 0 reference box is not reproducible from main. - Q4: four gates in iso-repack.sh enforce the single-entry menu; default/timeout already settable. The first mechanism probe was invalid (uid 0 bypassed pam_unix; sulogin had no tty) and a teardown error (shredding the control plaintext) are both recorded as failures, not massaged. demo-hp teardown is INCOMPLETE and named as such; the command is recorded, not claimed done.
This commit is contained in:
@@ -0,0 +1,456 @@
|
||||
# SPIKE 2 — closing the last mechanisms, and the universal ISO's shape (2026-07-31)
|
||||
|
||||
> **Class: Spike.** Findings only. No production file changed, no ISO built, nothing published.
|
||||
>
|
||||
> Continues `SPIKE-universal-iso-2026-07-31.md`; its results are premises here and are not re-derived.
|
||||
>
|
||||
> **Read §0 first: the venue went away mid-session.** Both Tier 0 hosts dropped off the tailnet at
|
||||
> ~12:28 CEST and did not return. Q1, Q2 and Q3 each have a part that needs a nested VM, and those
|
||||
> parts are **BLOCKED, not answered**. What could be established without them — from the shipped PVE
|
||||
> source, the answer schema, and an isolated mechanism probe — is below and is labelled by how it was
|
||||
> obtained.
|
||||
|
||||
---
|
||||
|
||||
## 0. Venue loss — what was tried
|
||||
|
||||
At 12:28 CEST, mid-run, both Tier 0 boxes went offline **simultaneously**:
|
||||
|
||||
```
|
||||
100.76.96.79 demo-hp active; relay "fra"; offline, last seen 5m ago, tx 1843821564 rx 63739848
|
||||
100.70.170.35 felhom-pve active; relay "waw"; offline, last seen 5m ago, tx 1560 rx 0
|
||||
```
|
||||
|
||||
Attempts, in order (per the standing rule that a "no access" claim names what it tried):
|
||||
|
||||
| # | Route | Result |
|
||||
|---|---|---|
|
||||
| 1 | `ping 100.76.96.79` (tailnet direct) | 100 % packet loss |
|
||||
| 2 | `ssh demo-hp` (tailnet, `ConnectTimeout=10`) | `connect to host 100.76.96.79 port 22: Connection timed out` |
|
||||
| 3 | `ssh demo-hp-lan` (LAN `192.168.0.87`, ProxyJump `felhom-pve`) | `Connection timed out during banner exchange` |
|
||||
| 4 | `ssh felhom-pve` (the jump host itself) | `connect to host 100.70.170.35 port 22: Connection timed out` |
|
||||
| 5 | DooPlex tailscale pod health | pod `tailscale-c84df7f84-tw2wz` **Running 6d10h**, healthy; log shows `open-conn-track: timeout … online=yes` then peers dropping |
|
||||
| 6 | `tailscale status` from the pod | **both peers `offline`**, `last seen 5m` and rising |
|
||||
|
||||
**Diagnosis: the far end, not ours.** Our tailscale pod is healthy and every other peer route behaves;
|
||||
both demo boxes are at the same remote site (REMOTE until ~08-02) and went offline together, which is
|
||||
that site's power or uplink. Not fixable from here.
|
||||
|
||||
**No fallback venue was used.** DooPlex is Tier 2 and the 2026-07-25 standing ruling moved drill and
|
||||
build VMs off it; `target-selection.md` says a Tier 2 machine is used only when a task says so
|
||||
explicitly, and this one does not. **An absent fence is not permission**, so the probes simply did not
|
||||
run. A watch is armed for the boxes' return.
|
||||
|
||||
The probe fixture is built and staged and will run unchanged when they come back: probe ISO
|
||||
(`prepare-iso --fetch-from partition`, serial-capture first-boot diagnostic), three validated answer
|
||||
files (locked/`fully-up`, locked/`before-network`, known-password control), an install runner with
|
||||
`reboot-mode = "power-off"` as the completion signal, and a boot runner with protocol-level detectors
|
||||
and **a NIC present at install time** — the two fixture defects that invalidated Spike 1's
|
||||
measurements are already corrected.
|
||||
|
||||
---
|
||||
|
||||
## 1. Baselines
|
||||
|
||||
Read fresh this session.
|
||||
|
||||
| Baseline | Value |
|
||||
|---|---|
|
||||
| `ISO_VERSION` | `1.25.0` — `scripts/iso/build-felhom-iso.sh:44` |
|
||||
| `SCRIPT_VERSION` | `1.22.0` — `scripts/felhom-host-install.sh:187` |
|
||||
| `felhom-bootstrap.sh` @ HEAD | `21bf6a6bde0cb13e3809e2f5c136a49929dcc82eb8d40bbdf6f290a886ee8ab7` |
|
||||
| PVE base | `proxmox-ve_9.2-1.iso`, `4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c` |
|
||||
| `felhom.eu` HEAD | **`ea00976`**, clean, `== origin/main` (the expected Spike 1 commit) |
|
||||
| `proxmox-first-boot` pkg | `9.2.5` (from the ISO's `/proxmox/packages/`) |
|
||||
|
||||
**Fixture provenance.** Probe ISO built on DooPlex from the baseline PVE ISO. Nested VMs were to run
|
||||
under plain QEMU/KVM on demo-hp under `/mnt/nvme-1tb/spike2/` — never `local-lvm`, no PVE storage
|
||||
registered. They did not run (§0). The mechanism probe in §2.2 ran in a throwaway `debian:trixie`
|
||||
Docker container on DooPlex, which is not an install and touches no box.
|
||||
|
||||
**Evidence classes used throughout:** **[MEASURED-PVE]** against the real PVE tooling or a PVE install ·
|
||||
**[MEASURED-MECH]** measured, but in an isolated container rather than on PVE · **[STRUCTURAL]** read
|
||||
from shipped source or unit files · **[BLOCKED]** needs the nested VM.
|
||||
|
||||
---
|
||||
|
||||
## 2. Q1 — is a locked root actually locked, and what else does it lock out?
|
||||
|
||||
### 2.1 The account's state on a real PVE install — **[BLOCKED]**
|
||||
|
||||
Needs the nested VM. The fixture is staged. Spike 1 established only that
|
||||
`root-password-hashed = "*"` **installs to completion**; the resulting account state was explicitly
|
||||
left open there and remains open.
|
||||
|
||||
### 2.2 What can authenticate — **[MEASURED-MECH]**, and the first attempt was invalid
|
||||
|
||||
**The first probe produced nothing and is recorded as failed.** It ran everything as uid 0, so `su`
|
||||
never invoked `pam_unix` at all (`rc=0` even with a deliberately wrong password); `sulogin` died on
|
||||
`tcgetattr failed: Invalid argument` with no TTY; and a `sed` mangled the control hash so the control
|
||||
case had no password set. Four cases, zero valid observations. Rerun with a non-root caller, a real
|
||||
pty via `script`, and `chpasswd` instead of `sed`.
|
||||
|
||||
**Corrected probe** (`debian:trixie`, `util-linux 2.41`, `pam_unix`) — with a working control:
|
||||
|
||||
| Shadow field | `passwd -S root` | `su root` as non-root user | `sulogin` (real pty) |
|
||||
|---|---|---|---|
|
||||
| real hash (control) | `P` | **correct pw → `SU_OK`**; wrong pw → `Authentication failure` | — |
|
||||
| `*` | **`L`** | `su: Authentication failure` (any password) | **`Cannot open access to console, the root account is locked.`** |
|
||||
| `!` | **`L`** | `su: Authentication failure` (any password) | **`Cannot open access to console, the root account is locked.`** |
|
||||
|
||||
The control is what makes this readable: the same harness authenticates successfully with the right
|
||||
password and fails with the wrong one, so the two locked rows are the module refusing, not the harness
|
||||
misfiring.
|
||||
|
||||
**So, at mechanism level: `*` and `!` are equivalent, both report `L`, and no password authenticates.**
|
||||
|
||||
**Caveat, and it is not small.** This is Debian's `pam_unix` and `util-linux`'s `sulogin` — the same
|
||||
components PVE ships — but it is **not PVE**. PVE could ship PAM configuration, a `sulogin` override,
|
||||
or a `SYSTEMD_SULOGIN_FORCE` setting that changes the outcome. Confirming on a real PVE install is
|
||||
part of the blocked work.
|
||||
|
||||
### 2.3 Does the Proxmox web interface still work? — **[BLOCKED]**, and this is the product question
|
||||
|
||||
**No measurement was obtained, so no answer is given.** Stating the reasoning without the measurement
|
||||
would be exactly the failure mode this spike exists to avoid.
|
||||
|
||||
What is established: PVE's `root@pam` realm authenticates the Linux root account through PAM, and
|
||||
§2.2 measured that `pam_unix` refuses a locked account for a non-root caller. If PVE's proxy
|
||||
authenticates the same way — **which was not verified** — then a locked root means **nobody can log
|
||||
into the PVE web interface at all**, neither customer nor operator, and that is a product decision
|
||||
rather than a detail.
|
||||
|
||||
The staged control scenario exists precisely to settle this: install C carries a **known** password, so
|
||||
the PVE API ticket endpoint can be exercised positively (correct password → ticket) before install A
|
||||
(locked) is tested against the same endpoint. Without that control a failure is indistinguishable
|
||||
from a broken probe — Spike 1's lesson.
|
||||
|
||||
### 2.4 Does anything else in PVE break? — **[BLOCKED]**
|
||||
|
||||
Not investigated. Cron, `pveproxy`, backup jobs and anything else assuming a usable root credential
|
||||
remain unexamined.
|
||||
|
||||
---
|
||||
|
||||
## 3. Q2 — is there a way back into a box whose setup failed?
|
||||
|
||||
### 3.1 Does the installed GRUB expose a recovery entry? — **[BLOCKED]**
|
||||
|
||||
Needs a boot. Note the installed system's GRUB is **PVE's own**, written by the installer — not the
|
||||
ISO's Felhom menu — so §5's single-entry collapse does not apply to it. What it offers is unverified.
|
||||
|
||||
### 3.2 `sulogin` with a locked root — **[MEASURED-MECH]. It refuses.**
|
||||
|
||||
The single observation the escrow-versus-discard ruling was said to rest on:
|
||||
|
||||
```
|
||||
Cannot open access to console, the root account is locked.
|
||||
See sulogin(8) man page for more details.
|
||||
```
|
||||
|
||||
Both for `*` and for `!`, with a real pty. **The standard Debian/systemd emergency and single-user
|
||||
path does not grant a shell on a locked-root system.** `systemd` invokes `sulogin` for
|
||||
`emergency.target` and `rescue.target`, so both are closed by the same refusal.
|
||||
|
||||
Subject to the §2.2 caveat: measured in a container, not on PVE.
|
||||
|
||||
### 3.3 What is left if `sulogin` refuses — **[STRUCTURAL]**, not measured
|
||||
|
||||
| Path | Realistic for whom |
|
||||
|---|---|
|
||||
| `init=/bin/bash` on the kernel line (GRUB `e`) | **Operator only.** It bypasses `sulogin` because PID 1 becomes the shell — no authentication is involved. Requires editing a boot entry at the console. Not a non-technical customer action. |
|
||||
| `systemd.unit=emergency.target` | **Closed** — it lands in `sulogin`, which refuses (§3.2). |
|
||||
| Live USB / rescue media | Operator only; requires physical media and knowing to do it. |
|
||||
| PVE web console (`:8006`) | **Unknown, possibly closed** — §2.3. |
|
||||
| SSH by key | Only if a key was baked, which for a public image is the thing R-142 says must not happen. |
|
||||
|
||||
**The honest summary:** on a locked-root box, every remaining route is operator-grade and physical.
|
||||
For a customer whose pairing failed, there is no self-service path back — which is precisely the
|
||||
objection raised when locked-root was first proposed, and §3.2 is the first evidence that the
|
||||
objection is well founded rather than speculative.
|
||||
|
||||
### 3.4 Does locked differ from rotated? — **[PARTIALLY ESTABLISHED]**
|
||||
|
||||
At mechanism level, yes, and materially:
|
||||
|
||||
- **Locked (`*`/`!`)** — `sulogin` refuses; console recovery is closed to everyone; the credential
|
||||
does not exist so it cannot leak, be escrowed, or be rotated.
|
||||
- **Rotated to a per-box random value** — `sulogin` prompts and *accepts the correct password*
|
||||
(the control row in §2.2 shows `pam_unix` authenticating a real hash). So console recovery is open
|
||||
**to whoever holds that password** — which is the operator if it is escrowed, and nobody if it is
|
||||
discarded.
|
||||
|
||||
**This makes the tradeoff concrete: "discard" and "lock" are the same outcome for recovery.** A
|
||||
rotated-then-discarded password is operationally identical to a locked account, because nobody holds
|
||||
it. The real choice is therefore **binary — escrow, or accept that console recovery is operator-only
|
||||
and physical.** Framed for the ruling in §7.
|
||||
|
||||
---
|
||||
|
||||
## 4. Q3 — does the Felhom stub survive `before-network`?
|
||||
|
||||
### 4.1 Does it execute? — **[BLOCKED]**
|
||||
|
||||
Needs the VM. The staged diagnostic emits `@@SPIKE2_FIRSTBOOT_RAN@@ marker=BEGIN/END` to `ttyS0`
|
||||
(captured by `qemu -serial file:`) precisely so the answer is a **positive observable** rather than the
|
||||
absence of an error — and §4.4 below is why that matters more than usual here.
|
||||
|
||||
### 4.2 Does it have what it needs that early? — **[STRUCTURAL]**, partly answered
|
||||
|
||||
From the shipped unit (`proxmox-first-boot` 9.2.5,
|
||||
`/usr/lib/systemd/system/proxmox-first-boot-network-pre.service`):
|
||||
|
||||
```
|
||||
After=systemd-remount-fs.service
|
||||
Before=network-pre.target
|
||||
Wants=network-pre.target
|
||||
ConditionPathExists=/var/lib/proxmox-first-boot/pending-first-boot-setup
|
||||
ConditionPathIsReadWrite=/var/lib
|
||||
Type=oneshot
|
||||
```
|
||||
|
||||
`After=systemd-remount-fs.service` plus `ConditionPathIsReadWrite=/var/lib` mean **a writable
|
||||
filesystem is structurally guaranteed** — or the unit does not run at all (§4.4). Entropy availability
|
||||
and binary presence at that point are **not** established structurally and are part of the staged
|
||||
diagnostic (`entropy_avail`, `openssl rand` length, `command -v` for `openssl`/`chpasswd`/`systemctl`).
|
||||
|
||||
### 4.3 Does it complete before anything is network-reachable? — **[STRUCTURAL]. Yes, by construction.**
|
||||
|
||||
This is the result the task asked to establish structurally rather than by timing, and the unit file
|
||||
settles it:
|
||||
|
||||
- **`Before=network-pre.target`** — `network-pre.target` is systemd's ordering barrier that precedes
|
||||
*all* network configuration; network managers order themselves after it.
|
||||
- **`Type=oneshot`** — systemd treats the unit as complete only when the process exits, so
|
||||
`Before=` is a hard barrier rather than a hint.
|
||||
|
||||
Together: **the hook runs to completion before any interface is configured**, therefore before `sshd`
|
||||
can bind a routable address and before `pveproxy` serves. A rotation performed there has a
|
||||
**zero-length exposure window by construction, not by being fast.** This holds on every machine and
|
||||
does not vary with hardware — which is why it is stronger than the timing number Spike 1 failed twice
|
||||
to obtain.
|
||||
|
||||
**A third ordering value exists that Spike 1 did not record.** `validate-answer` accepts
|
||||
**`before-network`, `network-online`, and `fully-up`** — and rejects `network-pre`, `multi-user` and a
|
||||
bogus value, so this is a real enum and not a permissive parser. The shipped package contains a
|
||||
matching unit for each. Spike 1 reported two. `network-online` is the middle option — after the
|
||||
network, before `pveproxy` — and is worth considering if `before-network` proves too early for
|
||||
something.
|
||||
|
||||
### 4.4 Does the existing pairing flow still work? — **[STRUCTURAL]** yes; **[BLOCKED]** empirically
|
||||
|
||||
Two structural findings, and the second one corrects a premise this design has been carrying.
|
||||
|
||||
**The stub does not need the network, and does not need `pveproxy`.** `stub-first-boot.sh` writes three
|
||||
files, then `systemctl daemon-reload`, `enable`, and `start --no-block felhom-bootstrap.service`. A
|
||||
grep for `pvesh`, `pct`, `pveum` and `qm` across the stub returns **no match** (`rc=1`). The unit it
|
||||
starts carries its own `After=network-online.target pve-cluster.service pveproxy.service`, so
|
||||
`--no-block` merely queues the job and systemd holds it until those are satisfied. Structurally, stub
|
||||
and pairing coexist: the stub runs early, the bootstrap runs when its own dependencies are met.
|
||||
|
||||
**So `answer.toml.tmpl:27` justifies the `fully-up` choice with a dependency the stub does not have.**
|
||||
The comment reads *"fully-up = the stub runs after pveproxy so pvesh/pct work"* — but the `pvesh`/`pct`
|
||||
usage is in `felhom-bootstrap.sh`, which never runs under the stub's ordering. This is the
|
||||
CLAUDE.md "a comment asserting an invariant" class in a new shape: a comment asserting a **dependency**
|
||||
that does not exist, which has constrained the design ever since. **R-148.**
|
||||
|
||||
**The trap that makes §4.1's positive observable load-bearing.** All four `proxmox-first-boot-*` units
|
||||
are guarded by `ConditionPathExists=` and `ConditionPathIsReadWrite=`. In systemd a **failed condition
|
||||
is a skip, not a failure** — the unit reports `inactive`/`condition failed`, the boot succeeds, and
|
||||
nothing is logged as wrong. A first-boot hook that never ran and one that ran perfectly are
|
||||
indistinguishable from the absence of errors. **R-150.** This is why §4.1 must be confirmed by a
|
||||
marker the hook emitted, never by "no errors appeared".
|
||||
|
||||
---
|
||||
|
||||
## 5. Q4 — the boot menu, and which entry is the default
|
||||
|
||||
### 5.1 Where the collapse happens — **[STRUCTURAL]**, and it is not a simple removal
|
||||
|
||||
`iso-repack.sh` replaces the stock `grub.cfg` wholesale: it renders `grub/grub.cfg.tmpl` over it
|
||||
(`iso-repack.sh:144-148`), having first lifted the `linux`/`initrd` lines verbatim from the stock
|
||||
*Install Proxmox VE (Automated)* entry (`:108-131`) so a PVE bump tracks automatically.
|
||||
|
||||
**Four independent gates then enforce the single entry**, so restoring the stock menu is a change to
|
||||
the gates, not just to the template:
|
||||
|
||||
| Gate | Location | Effect |
|
||||
|---|---|---|
|
||||
| exactly one `menuentry` | `iso-repack.sh:153-155` | `exit 14` otherwise |
|
||||
| zero `submenu` | `iso-repack.sh:156` | `exit 14` otherwise |
|
||||
| no live reference to `proxtui`, `proxdebug`, `nomodeset`, `Rescue Boot`, `memtest`, `fwsetup` | `iso-repack.sh:160-164` | `exit 14` otherwise |
|
||||
| **re-extracted from `final.iso`**: exactly one entry, theme present, entry named `Felhom telepítés` | `iso-repack.sh:314-319` | `exit 15` otherwise |
|
||||
|
||||
The fourth is the strongest of them — it proves the menu in the shipped image rather than in the build
|
||||
tree, which is the seam-wiring lesson applied correctly.
|
||||
|
||||
**What adding one interactive entry would actually cost:** the two count gates (rows 1 and 4) move from
|
||||
`== 1` to `== 2`, or better to a named allowlist. **The banned-token gate would still pass unchanged**
|
||||
for a stock *Install Proxmox VE (Graphical)* entry — it uses none of `proxtui`, `proxdebug`,
|
||||
`nomodeset`, `Rescue Boot`, `memtest`, `fwsetup`. So the safety property the gates exist to protect
|
||||
(no route to a debug/rescue/manual-disk-picker path) survives a second *graphical installer* entry
|
||||
intact. That is a narrower change than it first appears.
|
||||
|
||||
### 5.2 Are default and timeout settable? — **[STRUCTURAL]. Yes, already.**
|
||||
|
||||
`grub/grub.cfg.tmpl` sets them explicitly:
|
||||
|
||||
```
|
||||
set timeout_style=menu
|
||||
set timeout=5
|
||||
set default=0
|
||||
```
|
||||
|
||||
Both are plain template values. The template also records a real defect it fixed: the stock PVE config
|
||||
writes `timeout-style` (hyphen), **which GRUB does not recognise as a variable name**, so Felhom's
|
||||
`timeout_style` (underscore) is the one that actually works. A two-entry menu would therefore have a
|
||||
functioning timeout and a settable default — which is exactly what the §5.3 question needs.
|
||||
|
||||
### 5.3 The design question — **framed, deliberately not ruled**
|
||||
|
||||
Which entry should be `default=0` in a **public** image?
|
||||
|
||||
**What the measurements say, and they only point one way:** Spike 1 established that the automated
|
||||
path cannot be made safe on unseen hardware — no property distinguishes a system disk from a
|
||||
customer's backup drive, and a filter matching two disks silently wipes one. A tester who boots the
|
||||
image and walks away is, on multi-disk hardware, walking away from an unattended destructive operation
|
||||
whose target cannot be predicted.
|
||||
|
||||
**What that costs:** unattended install stops being the default for people who do not read
|
||||
instructions — and unattended install is a selling point. Most first boxes are single-disk, where the
|
||||
automated path is fine.
|
||||
|
||||
**The measurement that would change the balance** is §2.3 and §3: if a locked root also closes the PVE
|
||||
web interface and the console, then a tester whose unattended install completes but whose pairing
|
||||
fails has a machine nobody can reach — which argues for interactive-default independently of the disk
|
||||
question. Those measurements are blocked, so the balance cannot be struck yet.
|
||||
|
||||
**Operator-built versus public need not match, and two profiles from one build is cheap.** An
|
||||
operator-built image targets a machine whose disks are known, so automated-default is safe there. The
|
||||
public image is the one facing unseen hardware. Recommending the split is not the same as ruling the
|
||||
default, and the split is available regardless of how the ruling goes.
|
||||
|
||||
---
|
||||
|
||||
## 6. Q5 — the resulting shape, as constraints
|
||||
|
||||
Constraints on a build spec, each with a release-gate criterion. **Not an implementation, and nothing
|
||||
was built.** Constraints marked ⚠ depend on blocked measurements and are provisional.
|
||||
|
||||
| # | Constraint | Release gate |
|
||||
|---|---|---|
|
||||
| 1 | **The image must not select a disk by property on unseen hardware.** Spike 1 §3.2/§3.3. | The rendered answer contains no `filter.*` key, **or** the image is operator-built for a named machine. Grep the rendered `answer.toml`. |
|
||||
| 2 | **The interactive installer must be reachable from the boot menu.** §5.1 shows the cost is two count gates. | `final.iso`'s `grub.cfg` contains ≥1 interactive entry and still contains no live `proxtui`/`proxdebug`/`nomodeset`/`Rescue Boot`/`memtest`/`fwsetup` — i.e. keep gate 3, relax gates 1 and 4 to an allowlist. |
|
||||
| 3 | ⚠ **No usable baked credential.** `root-password-hashed = "*"` is schema-valid and installs (Spike 1). | The rendered answer's `root-password-hashed` is exactly `*`, and **no `.rootpw.txt` is emitted**. Both are one-line assertions in the build. |
|
||||
| 4 | ⚠ **If a per-box password is set, it is set at `before-network`.** §4.3 — the only ordering with a zero window by construction. | The rendered answer's `[first-boot].ordering` is `before-network`; and the hook's positive marker is observed in a boot test, never inferred from absent errors (§4.4, R-150). |
|
||||
| 5 | **No `root-ssh-keys`, ever, in a public image.** Spike 1 R-142. | `grep -c root-ssh-keys` on the rendered answer is `0`. |
|
||||
| 6 | **The profile must be committed.** Spike 1 found demo-felhom's baked key comes from an *uncommitted* profile, so a reference box is not reproducible from the repo. | The `--profile` path resolves inside the repo working tree **and** `git status --porcelain` is empty for it. **R-151.** |
|
||||
| 7 | **The artefact must be reproducible.** Spike 1 R-144 (`nested-probe`'s profile no longer exists). | Manifest records the profile's committed path and the repo commit; both resolve. |
|
||||
| 8 | **A recovery path must be documented and, if operator-only, said so plainly.** §3.3. | The release notes name the recovery route for a failed pairing. If that route is `init=/bin/bash`, the notes say it is operator-only. |
|
||||
|
||||
**What the image must not contain:** a usable root password hash, any `root-ssh-keys`, any customer
|
||||
id, claim code, retrieval passphrase or hub token, and any `filter.*` disk selector in the public
|
||||
variant. Spike 1 §6 established by enumeration that the only secret-bearing file is `/answer.toml`, so
|
||||
gates 3 and 5 over that one rendered file cover the credential surface.
|
||||
|
||||
---
|
||||
|
||||
## 7. The operator's decisions
|
||||
|
||||
Two, each with the measurement that informs it and its current state.
|
||||
|
||||
**Decision 1 — escrow or discard a per-box root password.**
|
||||
|
||||
§3.4 changed the shape of this question. **"Discard" and "lock" are operationally identical** — if
|
||||
nobody holds the password, `sulogin` refuses either way, and console recovery is closed to everyone.
|
||||
So the choice is binary:
|
||||
|
||||
- **Escrow** — the operator can always reach a box's console; the cost is a hub-held credential for
|
||||
every box, which is the thing the D6 trust model says the hub alone must not be.
|
||||
- **Discard / lock** — no credential exists anywhere; the cost is that a failed pairing leaves a
|
||||
machine recoverable only by an operator at the keyboard (`init=/bin/bash`) or a reinstall.
|
||||
|
||||
**Blocking measurement: §2.3** — whether a locked root also closes the PVE web interface. If it does,
|
||||
"discard" removes the last remote route as well as the local one, and the balance shifts sharply
|
||||
toward escrow. **Do not rule until §2.3 is measured.**
|
||||
|
||||
**Decision 2 — the default boot entry for a public image.**
|
||||
|
||||
Framed in §5.3. The disk evidence points at interactive-default; the product cost is the unattended
|
||||
experience. **Partially blocked on the same §2.3 measurement**, because an unreachable-after-failure
|
||||
box strengthens the interactive case independently. The operator-built-versus-public split (§5.3) is
|
||||
available regardless and is cheap.
|
||||
|
||||
---
|
||||
|
||||
## 8. What is still unknown, and what it would take
|
||||
|
||||
| # | Unknown | What it takes |
|
||||
|---|---|---|
|
||||
| 1 | **Everything in §2.1, §2.3, §2.4, §3.1, §4.1, §4.2 (entropy/binaries).** | The staged fixture, unchanged, once a Tier 0 box returns. ~90 min for all three scenarios. |
|
||||
| 2 | **Whether PVE alters the §2.2/§3.2 mechanism results** — PAM config, a `sulogin` override, `SYSTEMD_SULOGIN_FORCE`. | Same fixture; check `/etc/pam.d/`, the `sulogin` invocation, and systemd's emergency unit on the installed image. |
|
||||
| 3 | **Whether the multi-match pick is deterministic by enumeration order** (Spike 1 §8 #1, still open). | Re-run Spike 1's S1 with reversed disk order. ~15 min. |
|
||||
| 4 | **Whether the installer excludes its own boot medium** (Spike 1 §8 #2, still open). | USB boot on real hardware — HUMAN. |
|
||||
| 5 | **What `network-online` ordering buys** (§4.3) — it is a real third option nobody has evaluated. | One extra scenario alongside the staged two. |
|
||||
| 6 | **Whether a second GRUB entry actually boots the stock installer** after the gates are relaxed. | Build a probe ISO with a two-entry template and boot it. Not attempted — it would have meant changing repo files, which §4 forbids. |
|
||||
|
||||
---
|
||||
|
||||
## 9. R-rows opened
|
||||
|
||||
Numbering continues from R-147 (Spike 1).
|
||||
|
||||
| R | Severity | Finding |
|
||||
|---|---|---|
|
||||
| **R-148** | MEDIUM | **`answer.toml.tmpl:27` justifies `ordering = "fully-up"` with a dependency the stub does not have.** The comment says the stub must run after `pveproxy` "so pvesh/pct work", but `stub-first-boot.sh` contains no `pvesh`/`pct`/`pveum`/`qm` call (grep `rc=1`); that usage lives in `felhom-bootstrap.sh`, which runs under its own `After=network-online.target … pveproxy.service` unit. A comment asserting a **dependency** rather than an invariant — and it has constrained the design, because `before-network` is the ordering that would make the credential window zero (§4.3). |
|
||||
| **R-149** | LOW | **The `[first-boot].ordering` enum has three values, not two.** `before-network`, `network-online`, `fully-up` all validate; `network-pre` and `multi-user` are rejected. Spike 1 recorded two and the middle option has never been evaluated. |
|
||||
| **R-150** | MEDIUM | **A first-boot hook that never runs is indistinguishable from one that succeeded.** All four `proxmox-first-boot-*.service` units are guarded by `ConditionPathExists=` / `ConditionPathIsReadWrite=`, and in systemd a failed condition is a **skip, not a failure** — no error, successful boot. Any design that puts credential rotation in a first-boot hook must verify it by a positive marker the hook emitted. Directly the "an absent log line is not evidence" class. |
|
||||
| **R-151** | MEDIUM | **A reference box is not reproducible from the repository.** `demo-felhom` was installed from an ISO built with the **uncommitted** `n100-demo.profile`, which is where its baked SSH key comes from (Spike 1 §4.6). One of the two Tier 0 boxes therefore cannot be rebuilt from `main`. Related to R-144 (`nested-probe`'s profile is gone entirely); this row is the *reference box*, which matters more. Gate 6 in §6 exists to stop a public image inheriting the shape. |
|
||||
|
||||
---
|
||||
|
||||
## 10. Teardown
|
||||
|
||||
All three layers, verified positively.
|
||||
|
||||
**Layer 1 — demo-hp.** **INCOMPLETE, and named as such.** The box went offline (§0) while
|
||||
`/mnt/nvme-1tb/spike2/` held the probe ISO (~1.7 GB) and the staged answer files and runners. Nothing
|
||||
was running on it: the last successful command before the outage confirmed `0` QEMU processes, and no
|
||||
install had been started this session. Nothing was registered as a PVE storage, nothing was written to
|
||||
`local-lvm`, and `drill-r50` / guest 9201 were never touched this session.
|
||||
|
||||
**Outstanding cleanup, to run when the box returns** — recorded here because R-131 exists precisely
|
||||
because recorded commands never get run:
|
||||
|
||||
```
|
||||
rm -rf /mnt/nvme-1tb/spike2 ; rm -f /tmp/paia ; pkill -9 qemu-system-x86
|
||||
```
|
||||
|
||||
This is deliberately *not* claimed as done. Spike 1's `/mnt/nvme-1tb/spike-scratch` and both `/tmp/paia`
|
||||
copies were removed and verified at that session's teardown, so the outstanding item is this session's
|
||||
`spike2/` directory alone.
|
||||
|
||||
**Layer 2 — demo-felhom.** Nothing placed this session — it was never reached before the outage.
|
||||
Spike 1's `/tmp/paia` was removed and verified then. No action outstanding.
|
||||
|
||||
**Layer 3 — DooPlex.**
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| scratchpad | **4.4 GB → 48 K** (probe ISO, squashfs extracts, `.deb` extract, schema test dirs removed) |
|
||||
| control-password file (`s2/.ctlpw`) | `shred -u`'d — never printed, never committed; verified absent |
|
||||
| mechanism-probe container | `docker run --rm` — no image or container retained; **no global Docker cleanup run** (fence) |
|
||||
| `felhom-iso/out/` | untouched, **17 ISOs** |
|
||||
| production files | none modified — no `build-felhom-iso.sh`, no repack, no profile, no `felhom-bootstrap.sh` |
|
||||
|
||||
**One teardown error, recorded rather than quietly fixed.** Shredding the control password
|
||||
(`s2/.ctlpw`) destroyed the plaintext that scenario **C** existed to use — a `$6$` hash whose plaintext
|
||||
is gone cannot demonstrate "correct password → PVE ticket", which was C's entire purpose as the
|
||||
control for §2.3. `s2/scen/C.toml` was therefore **deleted** rather than left in place looking usable.
|
||||
**Scenario C must be regenerated** (one `openssl passwd -6` against a fresh random plaintext, kept
|
||||
0600) before the §2.3 measurement is attempted. Scenarios A and B are unaffected and remain staged.
|
||||
|
||||
**Hub-side disposition — nothing to dispose.** No customer, host or appliance was created: no install
|
||||
ran, and the only hub contact this session was none at all. Verified rather than asserted below.
|
||||
|
||||
**Published artefacts: none.** Nothing went to R2.
|
||||
Reference in New Issue
Block a user