Files
felhom.eu/documentation/runbooks/iso-release-gate.md
T

18 KiB
Raw Blame History

The public ISO release gate

Written before the first release image was built (2026-07-31), deliberately: a standard defined in advance cannot be rationalised afterwards, and this is the artifact that most needs one — once a file is on iso.felhom.eu and someone has downloaded it, it cannot be recalled.

Design inputs: documentation/audits/SPIKE-universal-iso-{1,2,3,4}-2026-07-31.md. Every criterion below exists because one of those spikes measured why.

How to use it

Run every check against the exact file that will be uploaded — not against build inputs, not against a sibling built the same way. A build is reproducible in theory and the uploaded bytes are what a stranger receives.

An absent finding is not a finding. Each check records what was scanned for, so "no hits" means "this pattern was searched and did not appear", not "nothing occurred to me".

Any FAIL stops the publication. Stopping is a good outcome.


The criteria

Let $ISO be the file about to be uploaded.

G1 — No answer.toml

osirrox -indev "$ISO" -find / -maxdepth 1 2>/dev/null | grep -cE "'/answer\.toml'|'/auto-installer-mode\.toml'"

PASS = 0.

Why: the answer file is the only secret-bearing artefact a Felhom ISO has ever carried (Spike 1 §6, by enumeration against the stock ISO). Removing it deletes the baked root hash, the disk profile, and the entire Spike 1–2 problem space in one move, and turns the most dangerous property of the image into a one-line assertion. Without auto-installer-mode.toml the stock grub.cfg does not emit the Automated entry at all (Spike 1 §5), so the unsafe path is absent by construction rather than guarded.

G2 — No root password or hash

osirrox -indev "$ISO" -extract /answer.toml - 2>/dev/null   # must not exist (G1)
# and, across every file the pipeline adds:
grep -c 'root-password\|root-password-hashed\|\$6\$\|\$y\$\|\$2[aby]\$'

PASS = 0 occurrences, and no *.rootpw.txt emitted beside the output.

Why: build-felhom-iso.sh:270-283 mints a root hash unconditionally in the answer-file modes, and Spike 1 proved by extraction that it ships inside /answer.toml where any downloader can read it. On a published image that is one credential shared by every box installed from it.

G3 — No SSH key baked

grep -c 'root-ssh-keys\|ssh-rsa\|ssh-ed25519\|ssh-dss'

PASS = 0 across the pipeline-added files.

Why: FELHOM_ROOT_SSH_KEY is one uncommented profile line away from shipping a shared key to every downloader, and Spike 1 §4.6 found exactly that had already happened on demo-felhom — from an uncommitted profile. Same shared-credential class as G2.

G4 — No customer identity

grep -c 'FELHOM_CUSTOMER_ID=[^"]\|FELHOM_RETRIEVAL_PASSPHRASE=[^"]\|claim[_-]code\|api[_-]key\|Bearer '

PASS = 0 with a value. Empty variable declarations inside felhom-bootstrap.sh are expected and are not hits (Spike 1 §6 established this distinction — :89 initialises them empty).

G5 — No credential of any kind, by enumeration

Diff the file list against the stock PVE ISO and inspect every added path:

osirrox -indev <stock-pve.iso> -find / | sort > /tmp/stock.txt
osirrox -indev "$ISO"          -find / | sort > /tmp/rel.txt
comm -13 /tmp/stock.txt /tmp/rel.txt      # everything the pipeline added

PASS = every added path is accounted for and none carries a secret. Scan the content-bearing ones for: PEM private-key headers, ssh-rsa/ssh-ed25519 material, Bearer tokens, api_key=, passphrase=, password=, token= with ≥12-char values, and crypt hashes ($6$, $y$, $2[aby]$).

Why enumeration and not a pattern sweep: a grep only finds what it was told to look for. Spike 1 §6 found /answer.toml this way precisely because the earlier recon had grepped the wrong file.

G6 — The boot menu is present, with both paths and a human timeout

osirrox -indev "$ISO" -extract /boot/grub/grub.cfg /tmp/g.cfg
grep -c '^[[:space:]]*menuentry ' /tmp/g.cfg      # >= 2
grep -E '^set (default|timeout|timeout_style)=' /tmp/g.cfg

PASS = at least one interactive entry present; set default= points at an interactive entry; set timeout ≥ 10; and timeout_style uses the underscore spelling.

Why the timeout: Spike 2 lost an entire probe to a 1-second menu. A human choosing between two install paths on unfamiliar hardware needs to read them first. Why the spelling: the stock PVE config writes timeout-style, which GRUB does not recognise as a variable name at all (Spike 2 §5.2) — Felhom's underscore form is the one that works.

Also assert the safety property still holds:

grep -v '^[[:space:]]*#' /tmp/g.cfg | grep -cE 'proxdebug|Rescue Boot|memtest|fwsetup'

PASS = 0.

Amendment, 2026-07-31, before the first build — recorded rather than made quietly. This criterion was first written as the six-token list iso-repack.sh:160-164 enforces (proxtui|proxdebug|nomodeset|Rescue Boot|memtest|fwsetup), on the stated rationale "no live route to a manual disk-picker". That rationale is obsolete for a public image and was already obsolete when it was written. The operator's ruling makes the manual installer the product: the person installing chooses their own disk in the stock installer, which shows the target and a Bootdisk(s) summary before erasing. A token list whose purpose is to keep users away from the manual installer cannot be applied to an image whose purpose is to offer it.

Two tokens are therefore dropped for the release image only:

  • proxtui — it is the Terminal UI installer, one of the two entries the image deliberately ships; and
  • nomodeset — a graphics fallback for the same installer, needed on hardware whose GPU the default mode cannot drive.

Four are kept, and they are the ones that were ever the real hazard: proxdebug (drops to a shell), Rescue Boot (boots an existing system rather than installing), memtest and fwsetup (not installers at all). The original six-token gate remains unchanged for the single-entry appliance mode, where its original rationale still holds exactly.

G7 — The Felhom package is present, at a recorded version

osirrox -indev "$ISO" -find /proxmox/packages 2>/dev/null | grep -c 'felhom-'
osirrox -indev "$ISO" -extract /proxmox/packages/<pkg>.deb /tmp/p.deb
dpkg-deb -I /tmp/p.deb | grep -E '^ (Package|Version):'

PASS = exactly one felhom-*.deb, its version recorded in the manifest, and no Depends (the payload is a script and a unit file; the binaries it calls run at first boot, not at postinst time, so there is nothing for dpkg --configure -a to order against).

Why: Spike 4 measured that this is the only delivery mechanism that survives an interactive install ([first-boot] is never placed on the system — Config.pm:118, Install.pm:746, :1360).

Amendment, 2026-07-31 — a sub-clause that could never pass is removed, and the reason recorded. This criterion originally also required "its sha256 matching the package built from committed source". That is unsatisfiable by construction, not by accident: dpkg-deb stamps the build time into every archive, so two builds of byte-identical source produce different .deb hashes. The check was written expecting reproducibility the tool does not offer, and it was already failing when the 1.26.1 release ran it.

A criterion nobody can satisfy is worse than no criterion: it gets waived the first time and read as advisory ever after. This project already carries a shelf of gates that exist and are never run (R-29), and that is exactly how they got there.

The payload-integrity claim it was reaching for is carried by G9, which compares the contents rather than the container — see G9's own amendment, which widens it to cover both payload files. If dpkg-deb is ever made reproducible here (SOURCE_DATE_EPOCH), this sub-clause can be restored and would then mean something.

G8 — The postinst cannot fail, and cannot need what it hasn't got

dpkg-deb --ctrl-tarfile /tmp/p.deb | tar -xO ./postinst > /tmp/postinst
grep -cE 'systemctl (start|daemon-reload|restart)' /tmp/postinst    # 0
grep -cE '\b(curl|wget|apt-get|apt|nc|ping)\b'      /tmp/postinst    # 0
grep -c 'set -e'                                     /tmp/postinst    # 0
tail -1 /tmp/postinst                                                # exit 0

PASS = all four.

Why, all four measured in Spike 4 §3: pid1 in the installer chroot is unconfigured.sh and no systemd is running, so start/daemon-reload are meaningless; the network was up only because the installer's DHCP happened to hold, so a box installed with the cable out has none; and a postinst that exits non-zero surfaces as an error in the middle of a customer's install — far worse than the stub not running. set -e is banned because it converts any unexpected non-zero into exactly that failure.

G9 — both payload files match repo HEAD byte-for-byte

Extract each payload file from the package and compare:

dpkg-deb --fsys-tarfile /tmp/p.deb | tar -xO ./usr/local/sbin/felhom-bootstrap.sh      | sha256sum
sha256sum scripts/iso/felhom-bootstrap.sh
dpkg-deb --fsys-tarfile /tmp/p.deb | tar -xO ./lib/systemd/system/felhom-bootstrap.service | sha256sum
sha256sum scripts/iso/felhom-bootstrap.service

PASS = both identical.

Why: these are the genuinely frozen, drift-capable payloads — everything else the box uses is fetched at run time from main. An ISO in a drawer for six months still runs exactly these.

Amendment, 2026-07-31 — widened from one file to two, because the unit was uncovered. G9 originally checked only felhom-bootstrap.sh. The package ships two payload files (scripts/iso/pkg/build-deb.sh:54-55), and the second one — the systemd unit — was checked by no criterion at all: G7 covered the container (and its sha sub-clause was unsatisfiable anyway), G8 covers the postinst behaviourally, G13 covers directory presence.

The unit is not incidental. Its After=network-online.target pve-cluster.service pveproxy.service, its ConditionPathExists=!/etc/felhom/.bootstrap-done and its Restart=on-failure are what decide when and whether day-0 runs at all. A drifted or hand-edited unit would have shipped silently — the same shape as the /etc/felhom/ miss that G13 exists to prevent, where a check proved the thing present and said nothing about what it depended on.

The check passes today: build-deb.sh installs both files verbatim from the same source directory. Note it self-asserts only the script (build-deb.sh:86-89) — extending that assertion to the unit is a code change, filed as an observation rather than made here.

G10 — The profile and every build input are committed

git status --porcelain scripts/iso/           # empty
git rev-parse HEAD; git rev-parse origin/main # equal

PASS = clean and pushed, and the manifest records the commit.

Why: Spike 1 found demo-felhom was installed from an uncommitted profile, so one of the two reference boxes cannot be rebuilt from main. A published image must never reach that state, and R-144 records a lab ISO that already has (nested-probe's profile no longer exists).

G11 — A published checksum, and a verified round trip

PASS = the .sha256 and the manifest are uploaded beside the ISO, and the file downloaded from https://iso.felhom.eu/<name> checksums to the same value. The round trip is the claim — a local checksum proves only that the local file is what it is.

G12 — The bucket stays private

PASS = the bucket's Public Access remains Disabled; the custom domain is the only public path.

G13 — every directory the payload writes into is IN the package

dpkg-deb -c /tmp/p.deb | awk '{print $6}' | grep -x './etc/felhom/'

PASS = present, along with ./usr/local/sbin/ and ./lib/systemd/system/.

Why, and it is the most expensive lesson in this file: the first release build passed G7, G8 and G9 and still produced a box that could never pair. felhom-bootstrap.sh writes its appliance token (:431), its pairing code (:435) and .bootstrap-done into /etc/felhom/, and the package did not ship that directory — the old stub-first-boot.sh had created it explicitly and the packaging dropped it. The installed box registered at the hub, failed to persist the token, and then returned HTTP 401 — still retrying forever, with no claim code ever shown.

G9 proves the payload is the right payload. It says nothing about what the payload depends on. Any future criterion of the form "the correct file is present" should be paired with one of the form "and everything it needs at run time is too". build-deb.sh asserts this itself and is red-proofed.

G14 — a PERSON chooses the disk; the image never guesses (operator rulings 2026-07-31, re-affirmed 2026-09-14)

osirrox -indev "$ISO" -find / -maxdepth 1 2>/dev/null | grep -cE "'/answer\.toml'|'/auto-installer-mode\.toml'"   # 0 (= G1)
grep -c 'filter\.' <every pipeline-added text file>                                                     # 0

PASS = both 0, and the proof install shows the installer's own disk screen and Bootdisk(s) summary before erasing, and runbooks/VOLUNTEER-first-hour.md tells a volunteer, in Hungarian, how to recognise the system disk on that screen.

The rule, in one paragraph a volunteer could read: the installer never picks a disk for you. It lists every disk it sees, with its size and model; you choose the one the system goes on, and that disk is erased. If the machine has more than one disk, the guide tells you how to tell them apart before you press Next — and the external backup drive should not be plugged in during the install.

Why: SPIKE-universal-iso-1 §3.2–3.3 measured that no udev property separates an internal disk from a USB-caddy backup drive and that a filter matching two disks silently wipes one. On 2026-09-14 the operator was offered an install-time "use the only disk" rule and chose to keep the person as the safety mechanism. Changing this criterion reverses two rulings; the spike that would inform it is its own register row.

G15 — the console is Felhom's after first boot, with no admin URL

On the proof install, after the first boot and after one reboot:

grep -c 8006 /etc/issue                          # 0
grep -c 'Felhom otthoni szerver' /etc/issue      # 1
systemctl is-enabled pvebanner.service           # masked

PASS = all three, and a console screendump shows no https://…:8006/ line. Why: Proxmox's pvebanner.service rewrites /etc/issue on every boot; the 2026-09-14 drill (screen s29) found that text — the operator admin UI, in English — was the first thing a household read. The reboot is part of the criterion because an overwrite without the mask passes once and fails on the next boot.

G16 — every Felhom-authored string is Hungarian FIRST, and names each secret once

# 1. the Hungarian is unchanged, and it is a golden rather than a grep
docker run --rm -v <repo>/scripts/iso:/work felhom-iso-assistant:trixie bash /work/test/bootstrap-modes.sh
#    -> the four R-559 golden/English/width checks per banner, and the /etc/issue pair, all ok
# 2. the secret names, on the extracted script
grep -c 'jelszavad' <extracted felhom-bootstrap.sh>                  # 0
grep -c 'Tulajdonosi jelmondat' <extracted felhom-bootstrap.sh>      # >= 1
grep -c 'Owner passphrase' <extracted felhom-bootstrap.sh>           # >= 1  (its English twin)

PASS = the harness green on every R-559 check, jelszavad absent, and each secret named once per language. Search with ASCII fragments and keep one positive and one negative control in the record.

AMENDED 2026-09-18 (ISO 1.29.0, R-559), and the amendment is the point of this note.

This criterion used to read "every Felhom-authored string on the volunteer's path is Hungarian" with PASS = no English sentence in the review list. It encoded the 2026-07-31 scope, and it was right when it was written.

Operator ruling 1b of 2026-09-17 supersedes that scope: "The console banner and the download page ARE in scope" (documentation/architecture/10-localisation.md §11). Slice 4 makes the three console texts bilingual — the Hungarian block byte-for-byte as before, an English block appended inside the same frame. Under the old wording that work would have FAILED this gate and the publication would have stopped, which is the gate doing exactly its job on an instruction that had moved on without it.

The criterion was therefore rewritten, not waived. What it protects is unchanged and is now stronger: the Hungarian is pinned by a GOLDEN captured before any English existed, rather than by a reviewer reading a list. What it no longer forbids is an English sentence below the Hungarian one.

Scope, stated so it is not read wider: the Proxmox installer's own screens are English and stay so under the 2026-07-31 ruling; G14 requires the guide to answer each of them. This criterion covers what Felhom writes: the GRUB menu, the console banners and /etc/issue.


Result recording

Record each criterion as PASS/FAIL with the observed value and what was scanned for, in the release report. A criterion with no recorded observation is a criterion that was not run.

The ONE home for a release's record is documentation/tests/iso-release-<version>-<date>/ (R-588, 2026-10-05) — one directory per published ISO, so „was the gate run for this image?" is answered by looking, not by a full-text search for a checksum. A record made elsewhere (inside an audit) gets a directory here whose README.md points at it. Today: 1.27.0, 1.27.1, 1.28.0 (pointer), 1.29.0.