felhom.eu/documentation/audits/SPIKE-firstboot-nic-sweep-2026-07-22.md

SPIKE — first-boot NIC sweep mechanics (R-59/R-60 Phase-0), 2026-07-22

Question: on a Proxmox box that installed against the wrong NIC (the demo-hp shape: no DHCP on any cable-bearing port at install time → the installer bakes its 192.168.100.2 static fallback into vmbr0 and completes), does this sequence work, and is it bounded: re-point vmbr0's bridge-ports to a candidate NIC → reload networking → bounded DHCP on vmbr0 → hub HTTPS probe?

Verdict up front: GO. The whole success leg is ~4 s; a dead candidate costs exactly its DHCP bound (20 s in the probe). Details and the design inputs below.

Vehicle: PVE-in-a-VM on felhom-pve (nested VM 9310, throwaway), installed from the v1.24.0 pairing ISO (uncommitted nested-probe profile = nested-vm + the ops SSH key for dead-network access). Guest: PVE 9.2, ifupdown2 3.3.0-1+pmx12, isc-dhcp-client 4.4.3-P1-8. NICs: ens18 (net0) / ens19 (net1).

Access-path deviation (recorded): the probe needs a shell on a box whose network is by definition dead. The plan was to use the installer's fallback address as the way in (host carries 192.168.100.1/24 on the port-less throwaway vmbr9, ssh root@192.168.100.2 over the dead bridge, root key = the probe profile's baked ops key). The run then took a different door (below), but the vmbr9 address was mounted and stays part of the Part-4 drill plan.

Reproduction — and an unplanned second trap variant

The plan was both NICs dark from qm start. In practice the tap re-point to the dead bridge landed ~1–2 min after start, and the installer had already leased on ens19 (192.168.0.141) in that window — after which the flip cut it off. What that accident bought:

• The auto-installer picks the NIC that leases, not the first by name. bridge-ports ens19 (net1, the second NIC) was baked, with the lease written as a STATIC stanza (address 192.168.0.141/24, gateway 192.168.0.1). So the t740's "picked wrong" is not name-order bad luck: at that site the 4-port card was the only cabled thing, no NIC leased, and the installer fell back. Two distinct wrong-NIC install variants therefore exist:
  1. no lease anywhere → 192.168.100.2 fallback baked static (demo-hp, live-proven 2026-07-21; README t740 trap) — R-59's detection signature;
  2. a lease existed at install time but the truth changed after (cable moved / switch port died) → a plausible-looking static config that is nonetheless dead — no 192.168.100.2 to key on. The gate must therefore key on "hub unreachable", not on the fallback signature; the signature is a diagnosis detail for the screen/log, not the trigger.
• The silent surface is real and was watched live for 30 minutes: felhom-bootstrap in the pairing loop logged registration did not go through (no network yet?) — retrying every ~42 s (≈12 s curl DNS failure + 30 s sleep), unit activating, console showing only the login prompt. Nothing on the physical screen names the problem — exactly R-59's complaint.

Probe log — the sweep walked by hand (detached; /root/probe-sweep.log)

Shape at probe time (the "cable move"): vmbr0 configured for ens19 = dead (tap on vmbr9), ens18 = live (tap on vmbr0, site DHCP + internet available). Script scheduled via systemd-run --on-active=45s, both taps flipped, box left alone.

Step	Result	Wall time
Leg 1: timeout 20 dhclient -1 -v vmbr0 on the DEAD shape	rc=124 (killed by timeout; dhclient never gives up on its own)	20.00 s (the full bound, exactly)
Rewrite bridge-ports ens19 → ens18 (tmp + mv, original kept)	one-line diff	~10 ms
ifreload -a	rc=0; raised the admin-DOWN candidate itself, port forwarding immediately (STP off, fd 0)	0.36 s
timeout 30 dhclient -1 -v vmbr0	DISCOVER→OFFER→REQUEST→ACK, bound 192.168.0.245	3.32 s
curl -fsS --max-time 15 https://hub.felhom.eu/	HTTP 302, rc=0	0.16 s

Post state: box reachable again at its baked static address over the new port (the static stanza was deliberately not touched — the probe changed only bridge-ports); dhclient's lease sits as a second address on vmbr0; default route unchanged.

Findings (design inputs for Part 3)

• F-P1 — the sequence works and is tightly bounded. Success ≈ 4 s; failed candidate = its DHCP bound + ε. With a 20 s bound: worst case ≈ NICs × ~21 s + one hub-probe timeout each. GO.
• F-P2 — unused physical NICs are admin-DOWN and their carrier is unreadable while down (/sys/class/net/*/carrier reads empty). A carrier-based candidate filter must ip link set <nic> up + settle (~2 s) before reading. ifreload -a brings the configured bridge-port up on its own, so the sweep proper needs no manual link handling — only the diagnosis table (NIC/MAC/cable) needs the raise-first step.
• F-P3 — carrier is necessary but not sufficient (virtio on a dead bridge shows carrier=1; a cabled port on a router with a dead uplink would too). Carrier selects/orders candidates; only DHCP-lease + hub-probe decides. Fine — that is the R-60 design anyway.
• F-P4 — dhclient -1 must be externally bounded (timeout N dhclient -1); on a dead segment it retries past any useful window (rc=124 at whatever bound is set). After a killed attempt, pkill dhclient before the next candidate — the second run then starts clean (proven: leg 2's dhclient ran without pidfile/lease complaints after leg 1's kill + pkill).
• F-P5 — treat ANY HTTP response as "hub reachable". The hub answers / with a 302; -f only fails ≥ 400, and even a 4xx would prove TLS+HTTP reachability. The gate's probe should key on "got an HTTP status" (curl rc 0, or rc 22 with a status code), not on 200.
• F-P6 — the trigger is "hub unreachable", never the 192.168.100.2 signature (see variant 2 above). The signature is named in the log/screen when present, as the likely cause.
• F-P7 — persisting only bridge-ports is enough to restore reachability in variant 2 (the static address stays valid when the cable moved ports). In variant 1 (fallback bake) the address itself is garbage — after a candidate wins, the persisted config must also carry the working addressing. Simplest shape that covers both: persist bridge-ports <winner> + iface vmbr0 inet dhcp (the winner is by definition a leasing NIC — DHCP is what proved it). Note for the R-59 screen loop: a box that self-healed onto DHCP addressing keeps working across router reboots; the operator can still pin it later.
• F-P8 (observation, out of scope): the pairing loop's register-failure line prints every cycle (~42 s) — the v1.21.0 quieting covered the 204 wait but not the no-network register path. The Part-3 gate makes this moot on the no-network path (the gate holds before register), but it is worth knowing the line exists.

Part-4 drill — the shipped v1.24.0 ISO, end to end (same day)

Two drill rounds on the same rig (both NICs on the dead vmbr9 from qm create, so no lease exists at ANY point of the install — the strict demo-hp shape):

Round 1 (VM 9311, first v1.24.0 build) — reproduced the trap AND caught a real gate defect. The installer baked the fallback (address 192.168.100.2/24 + gateway 192.168.100.1 + first NIC) and completed; the gate diagnosed, swept, painted the R-59 screen and retried — but after the cable move the heal FAILED and the gate oscillated: the sweep's candidate leased fine, yet the hub probe rode the baked stale default route (dhclient-script never replaces an existing default route) and failed, so the sweep dutifully restored byte-identically every cycle. F-P9 (fix, shipped in felhom-bootstrap.sh before the final build): flush vmbr0's addresses+routes after the re-point, before the bounded dhclient — a candidate is judged on the lease's OWN addressing/routing. Fix validated by hot-patching 9311 live: heal + hub registration 33 s after unit restart (pairing banner with code on the console).

Round 2 (VM 9312, virgin final ISO c25cdbbf…, zero patching) — the full claim:

Leg	Evidence
(a) dead-NIC install reproduces	installer env on 192.168.100.2 at 11:32; installed box: static fallback + dead gateway + bridge-ports ens18 baked
R-59 screen renders	console screendump committed beside this doc (SPIKE-firstboot-nic-sweep-2026-07-22-screen.png): header, NIC table (név/MAC/kábel/sebesség), fallback-signature explanation naming 192.168.100.2, remedy line
failed-sweep restore is safe	after full sweep cycles (with F-P9 flushes) the runtime had 192.168.100.2/24 back on vmbr0 — ifreload re-applies the flushed pristine config; interfaces file byte-shape intact
(b) sweep to the moved cable, unaided	net1 → vmbr0 at 11:35:27; network self-heal: vmbr0 -> ens19 + hub registration at 11:35:50 — 23 s, no human action; persisted iface vmbr0 inet dhcp + bridge-ports ens19, original in interfaces.felhom-bak; runtime 192.168.0.88 with a correct default route
(c) unclaimed appliance at the hub	registered — appliance token stored (0600), pairing code delivered + banner on console; unit activating in the normal pairing wait
R-61 lockout closed	the build's .rootpw.txt password recomputed against the installed box's /etc/shadow root hash (openssl passwd -6 -salt <shadow-salt>) → exact match; comparison done on the build host, the plaintext never left it

Cleanup (verified at the end of the run)

Nested VMs 9310/9311/9312 destroyed (--purge); vmbr9 deleted (its temporary 192.168.100.1/24 went with it); the transferred ISO and screendump temp files removed from felhom-pve; brctl show shows vmbr0 with exactly its original ports (enp1s0, veth9201i0); felhom-pve /etc/network/interfaces diffed byte-identical against the pre-task copy (sha c7ace087183cb72e031ad0cd1974b9239d539747528dd19523d993f8e9c4ec7a) — the file was never touched (the throwaway bridge lived only as an imperative ip link object). Residue by design: three unclaimed-appliance registrations at the production hub (the probe box, the hot-patched drill box, the virgin drill box — all three machines destroyed); the operator can discard them from the Unclaimed list at leisure.