felhom.eu/documentation/runbooks/iso-release-gate.md

# The public ISO release gate

> **Written before the first release image was built** (2026-07-31), deliberately: a standard defined
> in advance cannot be rationalised afterwards, and this is the artifact that most needs one — once a
> file is on `iso.felhom.eu` and someone has downloaded it, it cannot be recalled.
>
> Design inputs: `documentation/audits/SPIKE-universal-iso-{1,2,3,4}-2026-07-31.md`. Every criterion
> below exists because one of those spikes measured why.

## How to use it

**Run every check against the exact file that will be uploaded** — not against build inputs, not
against a sibling built the same way. A build is reproducible in theory and the uploaded bytes are
what a stranger receives.

**An absent finding is not a finding.** Each check records *what was scanned for*, so "no hits" means
"this pattern was searched and did not appear", not "nothing occurred to me".

**Any FAIL stops the publication.** Stopping is a good outcome.

---

## The criteria

Let `$ISO` be the file about to be uploaded.

### G1 — No `answer.toml`

```bash
osirrox -indev "$ISO" -find / -maxdepth 1 2>/dev/null | grep -cE "'/answer\.toml'|'/auto-installer-mode\.toml'"
```
**PASS = `0`.**

*Why:* the answer file is the only secret-bearing artefact a Felhom ISO has ever carried (Spike 1 §6,
by enumeration against the stock ISO). Removing it deletes the baked root hash, the disk profile, and
the entire Spike 1–2 problem space in one move, and turns the most dangerous property of the image
into a one-line assertion. Without `auto-installer-mode.toml` the stock `grub.cfg` does not emit the
Automated entry at all (Spike 1 §5), so the unsafe path is absent by construction rather than guarded.

### G2 — No root password or hash

```bash
osirrox -indev "$ISO" -extract /answer.toml - 2>/dev/null   # must not exist (G1)
# and, across every file the pipeline adds:
grep -c 'root-password\|root-password-hashed\|\$6\$\|\$y\$\|\$2[aby]\$'
```
**PASS = `0` occurrences, and no `*.rootpw.txt` emitted beside the output.**

*Why:* `build-felhom-iso.sh:270-283` mints a root hash unconditionally in the answer-file modes, and
Spike 1 proved by extraction that it ships inside `/answer.toml` where any downloader can read it. On
a published image that is one credential shared by every box installed from it.

### G3 — No SSH key baked

```bash
grep -c 'root-ssh-keys\|ssh-rsa\|ssh-ed25519\|ssh-dss'
```
**PASS = `0`** across the pipeline-added files.

*Why:* `FELHOM_ROOT_SSH_KEY` is one uncommented profile line away from shipping a shared key to every
downloader, and Spike 1 §4.6 found exactly that had already happened on `demo-felhom` — from an
**uncommitted** profile. Same shared-credential class as G2.

### G4 — No customer identity

```bash
grep -c 'FELHOM_CUSTOMER_ID=[^"]\|FELHOM_RETRIEVAL_PASSPHRASE=[^"]\|claim[_-]code\|api[_-]key\|Bearer '
```
**PASS = `0` with a value.** Empty variable *declarations* inside `felhom-bootstrap.sh` are expected
and are not hits (Spike 1 §6 established this distinction — `:89` initialises them empty).

### G5 — No credential of any kind, by enumeration

Diff the file list against the stock PVE ISO and inspect **every added path**:

```bash
osirrox -indev <stock-pve.iso> -find / | sort > /tmp/stock.txt
osirrox -indev "$ISO"          -find / | sort > /tmp/rel.txt
comm -13 /tmp/stock.txt /tmp/rel.txt      # everything the pipeline added
```
**PASS =** every added path is accounted for and none carries a secret. Scan the content-bearing ones
for: PEM private-key headers, `ssh-rsa`/`ssh-ed25519` material, `Bearer` tokens, `api_key=`,
`passphrase=`, `password=`, `token=` with ≥12-char values, and crypt hashes (`$6$`, `$y$`, `$2[aby]$`).

*Why enumeration and not a pattern sweep:* a grep only finds what it was told to look for. Spike 1 §6
found `/answer.toml` this way precisely because the earlier recon had grepped the wrong file.

### G6 — The boot menu is present, with both paths and a human timeout

```bash
osirrox -indev "$ISO" -extract /boot/grub/grub.cfg /tmp/g.cfg
grep -c '^[[:space:]]*menuentry ' /tmp/g.cfg      # >= 2
grep -E '^set (default|timeout|timeout_style)=' /tmp/g.cfg
```
**PASS =** at least one interactive entry present; `set default=` points at an **interactive** entry;
`set timeout` **≥ 10**; and `timeout_style` uses the **underscore** spelling.

*Why the timeout:* Spike 2 lost an entire probe to a **1-second** menu. A human choosing between two
install paths on unfamiliar hardware needs to read them first. *Why the spelling:* the stock PVE config
writes `timeout-style`, which GRUB does not recognise as a variable name at all (Spike 2 §5.2) —
Felhom's underscore form is the one that works.

**Also assert the safety property still holds:**

```bash
grep -v '^[[:space:]]*#' /tmp/g.cfg | grep -cE 'proxdebug|Rescue Boot|memtest|fwsetup'
```
**PASS = `0`.**

> **Amendment, 2026-07-31, before the first build — recorded rather than made quietly.**
> This criterion was first written as the six-token list `iso-repack.sh:160-164` enforces
> (`proxtui|proxdebug|nomodeset|Rescue Boot|memtest|fwsetup`), on the stated rationale *"no live route
> to a manual disk-picker"*. **That rationale is obsolete for a public image and was already obsolete
> when it was written.** The operator's ruling makes the manual installer *the product*: the person
> installing chooses their own disk in the stock installer, which shows the target and a `Bootdisk(s)`
> summary before erasing. A token list whose purpose is to keep users away from the manual installer
> cannot be applied to an image whose purpose is to offer it.
>
> Two tokens are therefore dropped **for the release image only**:
> - **`proxtui`** — it is the Terminal UI *installer*, one of the two entries the image deliberately
>   ships; and
> - **`nomodeset`** — a graphics fallback for the same installer, needed on hardware whose GPU the
>   default mode cannot drive.
>
> **Four are kept, and they are the ones that were ever the real hazard:** `proxdebug` (drops to a
> shell), `Rescue Boot` (boots an existing system rather than installing), `memtest` and `fwsetup`
> (not installers at all). The original six-token gate remains **unchanged** for the single-entry
> appliance mode, where its original rationale still holds exactly.

### G7 — The Felhom package is present, at a recorded version

```bash
osirrox -indev "$ISO" -find /proxmox/packages 2>/dev/null | grep -c 'felhom-'
osirrox -indev "$ISO" -extract /proxmox/packages/<pkg>.deb /tmp/p.deb
dpkg-deb -I /tmp/p.deb | grep -E '^ (Package|Version):'
```
**PASS =** exactly one `felhom-*.deb`, its version recorded in the manifest, and **no `Depends`**
(the payload is a script and a unit file; the binaries it calls run at first boot, not at postinst
time, so there is nothing for `dpkg --configure -a` to order against).

*Why:* Spike 4 measured that this is the only delivery mechanism that survives an interactive install
(`[first-boot]` is never placed on the system — `Config.pm:118`, `Install.pm:746`, `:1360`).

> **Amendment, 2026-07-31 — a sub-clause that could never pass is removed, and the reason recorded.**
> This criterion originally also required *"its sha256 matching the package built from committed
> source"*. **That is unsatisfiable by construction, not by accident:** `dpkg-deb` stamps the build
> time into every archive, so two builds of byte-identical source produce different `.deb` hashes.
> The check was written expecting reproducibility the tool does not offer, and it was already failing
> when the 1.26.1 release ran it.
>
> A criterion nobody can satisfy is worse than no criterion: it gets waived the first time and read
> as advisory ever after. This project already carries a shelf of gates that exist and are never run
> (**R-29**), and that is exactly how they got there.
>
> **The payload-integrity claim it was reaching for is carried by G9**, which compares the *contents*
> rather than the container — see G9's own amendment, which widens it to cover both payload files.
> If `dpkg-deb` is ever made reproducible here (`SOURCE_DATE_EPOCH`), this sub-clause can be restored
> and would then mean something.

### G8 — The postinst cannot fail, and cannot need what it hasn't got

```bash
dpkg-deb --ctrl-tarfile /tmp/p.deb | tar -xO ./postinst > /tmp/postinst
grep -cE 'systemctl (start|daemon-reload|restart)' /tmp/postinst    # 0
grep -cE '\b(curl|wget|apt-get|apt|nc|ping)\b'      /tmp/postinst    # 0
grep -c 'set -e'                                     /tmp/postinst    # 0
tail -1 /tmp/postinst                                                # exit 0
```
**PASS =** all four.

*Why, all four measured in Spike 4 §3:* `pid1` in the installer chroot is `unconfigured.sh` and **no
systemd is running**, so `start`/`daemon-reload` are meaningless; the network was up only because the
installer's DHCP happened to hold, so a box installed with the cable out has none; and a postinst that
exits non-zero surfaces as an error in the middle of a customer's install — far worse than the stub
not running. `set -e` is banned because it converts any unexpected non-zero into exactly that failure.

### G9 — **both** payload files match repo HEAD byte-for-byte

Extract each payload file from the package and compare:
```bash
dpkg-deb --fsys-tarfile /tmp/p.deb | tar -xO ./usr/local/sbin/felhom-bootstrap.sh      | sha256sum
sha256sum scripts/iso/felhom-bootstrap.sh
dpkg-deb --fsys-tarfile /tmp/p.deb | tar -xO ./lib/systemd/system/felhom-bootstrap.service | sha256sum
sha256sum scripts/iso/felhom-bootstrap.service
```
**PASS =** both identical.

*Why:* these are the genuinely frozen, drift-capable payloads — everything else the box uses is
fetched at run time from `main`. An ISO in a drawer for six months still runs exactly these.

> **Amendment, 2026-07-31 — widened from one file to two, because the unit was uncovered.**
> G9 originally checked only `felhom-bootstrap.sh`. The package ships **two** payload files
> (`scripts/iso/pkg/build-deb.sh:54-55`), and the second one — the systemd unit — was checked by no
> criterion at all: G7 covered the container (and its sha sub-clause was unsatisfiable anyway), G8
> covers the postinst *behaviourally*, G13 covers directory *presence*.
>
> The unit is not incidental. Its `After=network-online.target pve-cluster.service pveproxy.service`,
> its `ConditionPathExists=!/etc/felhom/.bootstrap-done` and its `Restart=on-failure` are what decide
> **when and whether day-0 runs at all**. A drifted or hand-edited unit would have shipped silently —
> the same shape as the `/etc/felhom/` miss that G13 exists to prevent, where a check proved the thing
> present and said nothing about what it depended on.
>
> The check passes today: `build-deb.sh` installs both files verbatim from the same source directory.
> **Note it self-asserts only the script** (`build-deb.sh:86-89`) — extending that assertion to the
> unit is a code change, filed as an observation rather than made here.

### G10 — The profile and every build input are committed

```bash
git status --porcelain scripts/iso/           # empty
git rev-parse HEAD; git rev-parse origin/main # equal
```
**PASS =** clean and pushed, and the manifest records the commit.

*Why:* Spike 1 found `demo-felhom` was installed from an **uncommitted** profile, so one of the two
reference boxes cannot be rebuilt from `main`. A published image must never reach that state, and
R-144 records a lab ISO that already has (`nested-probe`'s profile no longer exists).

### G11 — A published checksum, and a verified round trip

**PASS =** the `.sha256` and the manifest are uploaded beside the ISO, **and** the file downloaded from
`https://iso.felhom.eu/<name>` checksums to the same value. The round trip is the claim — a local
checksum proves only that the local file is what it is.

### G12 — The bucket stays private

**PASS =** the bucket's `Public Access` remains **Disabled**; the custom domain is the only public path.

### G13 — every directory the payload writes into is IN the package

```bash
dpkg-deb -c /tmp/p.deb | awk '{print $6}' | grep -x './etc/felhom/'
```
**PASS =** present, along with `./usr/local/sbin/` and `./lib/systemd/system/`.

*Why, and it is the most expensive lesson in this file:* the first release build passed G7, G8 **and
G9** and still produced a box that could never pair. `felhom-bootstrap.sh` writes its appliance token
(`:431`), its pairing code (`:435`) and `.bootstrap-done` into `/etc/felhom/`, and the package did not
ship that directory — the old `stub-first-boot.sh` had created it explicitly and the packaging dropped
it. The installed box registered at the hub, failed to persist the token, and then returned
`HTTP 401 — still retrying` forever, with no claim code ever shown.

**G9 proves the payload is the right payload. It says nothing about what the payload depends on.**
Any future criterion of the form "the correct file is present" should be paired with one of the form
"and everything it needs at run time is too". `build-deb.sh` asserts this itself and is red-proofed.

---

## Result recording

Record each criterion as PASS/FAIL **with the observed value and what was scanned for**, in the
release report. A criterion with no recorded observation is a criterion that was not run.