felhom.eu/REPORT.md

REPORT — scripts/ISO train v1.24.0: R-59 + R-60 + R-61 (2026-07-22)

Task: the three HP-t740 findings — the box that can never call home (R-59 refuse-loudly gate), the first-boot NIC sweep self-heal (R-60), and the knowable baked root password (R-61 slice 1) — implemented in scripts/iso/, spike-proven, and driven end-to-end on a nested wrong-NIC install. No hub/agent/controller/catalog changes; golden image and floors untouched.

Baselines

• felhom.eu main at start: 1613bc7 (scripts CHANGELOG head = build-hub.sh v1.23.0); clean-tree gate held before both ISO builds.
• Target: scripts train v1.24.0 (ISO_VERSION="1.24.0" in build-felhom-iso.sh).

Commits (this train, in order)

Hash	What
9e29295	R-61 slice 1: rootpw sibling file (0600) + manifest pointer; test/rootpw-emission.sh; OUT_ISO hoist; FELHOM_ISO_KEEP_WORK escape
ad1d26a	Harness catches up with the v1.21.0 in-script pairing wait (old scenarios would hang); assistant image + python3
c043ba9	Phase-0 spike doc — sweep mechanics GO, F-P1..F-P8 (STOP-0 discharged before gate code)
cf1862c	R-59+R-60: network_gate + sweep_nics + Hungarian console screen; 4 harness scenarios; 3 red-proofs run
699325b	R-60 drill fix (F-P9): flush vmbr0 before the bounded dhclient — the baked fallback default route otherwise poisons the hub probe
(this commit)	v1.24.0 CHANGELOG, README, ROADMAP (R-59/60/61 SHIPPED + new R-62), onboarding runbook, capability map, spike-doc drill section + screen capture, this REPORT

Harness results

• test/bootstrap-modes.sh (in felhom-iso-assistant:trixie): 40/40 green — D (direct, zero appliance calls, G1 zero-new-behavior invariant incl. zero sleeps + fixture untouched), P (v1.21.0 single-invocation pairing loop), 410 crash-window, G2 sweep-success (re-point + persist inet dhcp + .felhom-bak + proceed), G3 sweep-fail (screen content + byte-identical interfaces + no bak), G4 state.json-present (sweep never invoked by fake call-count, interfaces untouched).
• test/rootpw-emission.sh: 11/11 green — dry-run emission, 0600, plaintext↔answer-hash cross-check (openssl passwd -6 -salt), plaintext absent from build output, manifest-heredoc pointer/no-leak guard.
• shellcheck -S warning felhom-bootstrap.sh: clean.

Red-proofs (all four run, failed exactly on their guarding assertion, restored)

1. Manifest plaintext leak (R-61): root-password-plain : ${ROOT_PLAIN} injected into the manifest heredoc → manifest never references ROOT_PLAIN FAILED → restored.
2. Restore-on-failure dropped (R-60): sweep's byte-identical restore removed → G3 interfaces BYTE-IDENTICAL after failed sweep FAILED → restored.
3. State gate removed (R-60): [[ ! -e $STATE_FILE ]] → true → G4 zero-ifreload/zero- dhclient FAILED → restored.
4. Hub-reachable early return removed (R-59): all five G1 invariants FAILED → restored.

Phase-0 spike (STOP-0 honored)

documentation/audits/SPIKE-firstboot-nic-sweep-2026-07-22.md — written and committed BEFORE the gate code. Verdict GO; key inputs: success leg ≈ 4 s, dead candidate = exactly its DHCP bound; unused NICs are admin-DOWN (raise before carrier read); ANY HTTP status = reachable (hub answers 302); trigger on hub-unreachable, never the fallback signature (the drill found the second wrong-NIC variant: the installer picks the NIC that LEASES and bakes the lease as static).

Nested end-to-end drill (evidence in the spike doc)

• Round 1 caught a real defect the harness could not: the fallback bake includes a default route via 192.168.100.1, and dhclient-script never replaces an existing default route → the sweep's hub probe rode the dead gateway and the gate oscillated. Fixed (699325b, flush before dhclient), validated by hot-patch (heal in 33 s), then the ISO was rebuilt.
• Round 2, virgin final ISO, zero patching: dead-NIC install → fallback baked → R-59 screen on the console (capture committed: SPIKE-firstboot-nic-sweep-2026-07-22-screen.png) → failed-sweep restore proven safe (flushed static re-applied by ifreload) → cable move → self-heal + hub registration in 23 s unaided (persisted inet dhcp + winner port, original in interfaces.felhom-bak, box at a proper DHCP address, pairing banner + code on console).
• R-61 lockout closed, proven: the build's .rootpw.txt password recomputed against the installed box's /etc/shadow root hash → exact match (computed on the build host; the plaintext never left it). Value NOT included here by design.
• Validation method note: no browser on DooPlex — console proof is QEMU screendump; hub-side proof is the box's own registration/pairing-code delivery (endpoint-level).

Artifacts

• Final drill ISO: felhom-pve-9.2-1-v1.24.0-nested-probe-generic.iso, sha256 c25cdbbf35d2134b0bf278c4b069a3d5849c0960cf0a5e4500e52638489c8947, built from clean tree at 699325b with an uncommitted key-bearing probe profile (nested-vm + ops SSH key — never committed, per the armed-profile rule). Its .rootpw.txt exists beside it (0600) — verified, value stored out-of-band only.
• Shipping rebuilds for real profiles (hp/n100/generic) happen at next need; the committed pipeline builds them from main as usual.

Cleanup verification

Nested VMs 9310/9311/9312 destroyed; throwaway vmbr9 (+ its temporary host address) deleted; transferred ISO + screendumps removed from felhom-pve; brctl show back to exactly enp1s0 + veth9201i0 on vmbr0; /etc/network/interfaces on felhom-pve diffed BYTE-IDENTICAL against the pre-task copy (sha c7ace087…). Residue by design: three unclaimed-appliance registrations at the production hub (all three source machines destroyed) — operator may discard them from the Unclaimed list.

Observations / follow-ups

• F-P8 (out of scope, noted): the pairing loop's register-failure line prints every ~42 s on a no-network box; v1.21.0 quieted the 204 wait but not this path. The gate makes it mostly moot (it holds before register), but the line exists for a box that loses network AFTER install start (state.json present → screen + retry, register never reached).
• The console font renders Hungarian long vowels (ő/ű) approximately on the text console — same behavior as the existing pairing banner; acceptable.
• qm screendump (qm monitor → screendump) is a serviceable no-browser console-capture method for nested drills — worth remembering for future runbooks.
• R-62 (hub delete-dialog cosmetics) filed on the ROADMAP as agreed — docs-only here.