felhom.eu/REPORT.md

# REPORT — scripts/ISO train v1.24.0: R-59 + R-60 + R-61 (2026-07-22)

**Task:** the three HP-t740 findings — the box that can never call home (R-59 refuse-loudly gate),
the first-boot NIC sweep self-heal (R-60), and the knowable baked root password (R-61 slice 1) —
implemented in `scripts/iso/`, spike-proven, and driven end-to-end on a nested wrong-NIC install.
No hub/agent/controller/catalog changes; golden image and floors untouched.

## Baselines

- felhom.eu `main` at start: `1613bc7` (scripts CHANGELOG head = build-hub.sh v1.23.0); clean-tree
  gate held before both ISO builds.
- Target: scripts train **v1.24.0** (`ISO_VERSION="1.24.0"` in `build-felhom-iso.sh`).

## Commits (this train, in order)

| Hash | What |
|---|---|
| `9e29295` | R-61 slice 1: rootpw sibling file (0600) + manifest pointer; `test/rootpw-emission.sh`; OUT_ISO hoist; `FELHOM_ISO_KEEP_WORK` escape |
| `ad1d26a` | Harness catches up with the v1.21.0 in-script pairing wait (old scenarios would hang); assistant image + python3 |
| `c043ba9` | Phase-0 spike doc — sweep mechanics GO, F-P1..F-P8 (STOP-0 discharged before gate code) |
| `cf1862c` | R-59+R-60: `network_gate` + `sweep_nics` + Hungarian console screen; 4 harness scenarios; 3 red-proofs run |
| `699325b` | R-60 drill fix (F-P9): flush vmbr0 before the bounded dhclient — the baked fallback default route otherwise poisons the hub probe |
| (this commit) | v1.24.0 CHANGELOG, README, ROADMAP (R-59/60/61 SHIPPED + new R-62), onboarding runbook, capability map, spike-doc drill section + screen capture, this REPORT |

## Harness results

- `test/bootstrap-modes.sh` (in `felhom-iso-assistant:trixie`): **40/40 green** — D (direct,
  zero appliance calls, G1 zero-new-behavior invariant incl. zero sleeps + fixture untouched),
  P (v1.21.0 single-invocation pairing loop), 410 crash-window, G2 sweep-success
  (re-point + persist `inet dhcp` + `.felhom-bak` + proceed), G3 sweep-fail (screen content +
  **byte-identical** interfaces + no bak), G4 state.json-present (sweep **never** invoked by
  fake call-count, interfaces untouched).
- `test/rootpw-emission.sh`: **11/11 green** — dry-run emission, 0600, plaintext↔answer-hash
  cross-check (`openssl passwd -6 -salt`), plaintext absent from build output, manifest-heredoc
  pointer/no-leak guard.
- `shellcheck -S warning felhom-bootstrap.sh`: clean.

## Red-proofs (all four run, failed exactly on their guarding assertion, restored)

1. **Manifest plaintext leak** (R-61): `root-password-plain : ${ROOT_PLAIN}` injected into the
   manifest heredoc → `manifest never references ROOT_PLAIN` FAILED → restored.
2. **Restore-on-failure dropped** (R-60): sweep's byte-identical restore removed → G3
   `interfaces BYTE-IDENTICAL after failed sweep` FAILED → restored.
3. **State gate removed** (R-60): `[[ ! -e $STATE_FILE ]]` → `true` → G4 zero-ifreload/zero-
   dhclient FAILED → restored.
4. **Hub-reachable early return removed** (R-59): all five G1 invariants FAILED → restored.

## Phase-0 spike (STOP-0 honored)

`documentation/audits/SPIKE-firstboot-nic-sweep-2026-07-22.md` — written and committed BEFORE the
gate code. Verdict GO; key inputs: success leg ≈ 4 s, dead candidate = exactly its DHCP bound;
unused NICs are admin-DOWN (raise before carrier read); ANY HTTP status = reachable (hub answers
302); trigger on hub-unreachable, never the fallback signature (the drill found the second
wrong-NIC variant: the installer picks the NIC that LEASES and bakes the lease as static).

## Nested end-to-end drill (evidence in the spike doc)

- **Round 1** caught a real defect the harness could not: the fallback bake includes a **default
  route via 192.168.100.1**, and dhclient-script never replaces an existing default route → the
  sweep's hub probe rode the dead gateway and the gate oscillated. Fixed (`699325b`, flush before
  dhclient), validated by hot-patch (heal in 33 s), then the ISO was rebuilt.
- **Round 2, virgin final ISO, zero patching:** dead-NIC install → fallback baked → **R-59 screen
  on the console** (capture committed: `SPIKE-firstboot-nic-sweep-2026-07-22-screen.png`) →
  failed-sweep restore proven safe (flushed static re-applied by ifreload) → cable move →
  **self-heal + hub registration in 23 s unaided** (persisted `inet dhcp` + winner port, original
  in `interfaces.felhom-bak`, box at a proper DHCP address, pairing banner + code on console).
- **R-61 lockout closed, proven:** the build's `.rootpw.txt` password recomputed against the
  installed box's `/etc/shadow` root hash → exact match (computed on the build host; the
  plaintext never left it). Value NOT included here by design.
- Validation method note: no browser on DooPlex — console proof is QEMU screendump; hub-side
  proof is the box's own registration/pairing-code delivery (endpoint-level).

## Artifacts

- Final drill ISO: `felhom-pve-9.2-1-v1.24.0-nested-probe-generic.iso`, sha256
  `c25cdbbf35d2134b0bf278c4b069a3d5849c0960cf0a5e4500e52638489c8947`, built from clean tree at
  `699325b` with an **uncommitted** key-bearing probe profile (nested-vm + ops SSH key — never
  committed, per the armed-profile rule). Its `.rootpw.txt` exists beside it (0600) — verified,
  value stored out-of-band only.
- Shipping rebuilds for real profiles (hp/n100/generic) happen at next need; the committed
  pipeline builds them from `main` as usual.

## Cleanup verification

Nested VMs 9310/9311/9312 destroyed; throwaway `vmbr9` (+ its temporary host address) deleted;
transferred ISO + screendumps removed from felhom-pve; `brctl show` back to exactly `enp1s0` +
`veth9201i0` on vmbr0; **`/etc/network/interfaces` on felhom-pve diffed BYTE-IDENTICAL against
the pre-task copy** (sha `c7ace087…`). Residue by design: **three unclaimed-appliance
registrations at the production hub** (all three source machines destroyed) — operator may
discard them from the Unclaimed list.

## Observations / follow-ups

- **F-P8 (out of scope, noted):** the pairing loop's register-failure line prints every ~42 s on
  a no-network box; v1.21.0 quieted the 204 wait but not this path. The gate makes it mostly moot
  (it holds before register), but the line exists for a box that loses network AFTER install
  start (state.json present → screen + retry, register never reached).
- The console font renders Hungarian long vowels (ő/ű) approximately on the text console —
  same behavior as the existing pairing banner; acceptable.
- `qm` screendump (`qm monitor` → `screendump`) is a serviceable no-browser console-capture
  method for nested drills — worth remembering for future runbooks.
- R-62 (hub delete-dialog cosmetics) filed on the ROADMAP as agreed — docs-only here.