felhom.eu/documentation/audits/SPIKE-firstboot-nic-sweep-2026-07-22.md

SPIKE — first-boot NIC sweep mechanics (R-59/R-60 Phase-0), 2026-07-22

Question: on a Proxmox box that installed against the wrong NIC (the demo-hp shape: no DHCP on any cable-bearing port at install time → the installer bakes its 192.168.100.2 static fallback into vmbr0 and completes), does this sequence work, and is it bounded: re-point vmbr0's bridge-ports to a candidate NIC → reload networking → bounded DHCP on vmbr0 → hub HTTPS probe?

Verdict up front: GO. The whole success leg is ~4 s; a dead candidate costs exactly its DHCP bound (20 s in the probe). Details and the design inputs below.

Vehicle: PVE-in-a-VM on felhom-pve (nested VM 9310, throwaway), installed from the v1.24.0 pairing ISO (uncommitted nested-probe profile = nested-vm + the ops SSH key for dead-network access). Guest: PVE 9.2, ifupdown2 3.3.0-1+pmx12, isc-dhcp-client 4.4.3-P1-8. NICs: ens18 (net0) / ens19 (net1).

Access-path deviation (recorded): the probe needs a shell on a box whose network is by definition dead. The plan was to use the installer's fallback address as the way in (host carries 192.168.100.1/24 on the port-less throwaway vmbr9, ssh root@192.168.100.2 over the dead bridge, root key = the probe profile's baked ops key). The run then took a different door (below), but the vmbr9 address was mounted and stays part of the Part-4 drill plan.

Reproduction — and an unplanned second trap variant

The plan was both NICs dark from qm start. In practice the tap re-point to the dead bridge landed ~1–2 min after start, and the installer had already leased on ens19 (192.168.0.141) in that window — after which the flip cut it off. What that accident bought:

• The auto-installer picks the NIC that leases, not the first by name. bridge-ports ens19 (net1, the second NIC) was baked, with the lease written as a STATIC stanza (address 192.168.0.141/24, gateway 192.168.0.1). So the t740's "picked wrong" is not name-order bad luck: at that site the 4-port card was the only cabled thing, no NIC leased, and the installer fell back. Two distinct wrong-NIC install variants therefore exist:
  1. no lease anywhere → 192.168.100.2 fallback baked static (demo-hp, live-proven 2026-07-21; README t740 trap) — R-59's detection signature;
  2. a lease existed at install time but the truth changed after (cable moved / switch port died) → a plausible-looking static config that is nonetheless dead — no 192.168.100.2 to key on. The gate must therefore key on "hub unreachable", not on the fallback signature; the signature is a diagnosis detail for the screen/log, not the trigger.
• The silent surface is real and was watched live for 30 minutes: felhom-bootstrap in the pairing loop logged registration did not go through (no network yet?) — retrying every ~42 s (≈12 s curl DNS failure + 30 s sleep), unit activating, console showing only the login prompt. Nothing on the physical screen names the problem — exactly R-59's complaint.

Probe log — the sweep walked by hand (detached; /root/probe-sweep.log)

Shape at probe time (the "cable move"): vmbr0 configured for ens19 = dead (tap on vmbr9), ens18 = live (tap on vmbr0, site DHCP + internet available). Script scheduled via systemd-run --on-active=45s, both taps flipped, box left alone.

Step	Result	Wall time
Leg 1: timeout 20 dhclient -1 -v vmbr0 on the DEAD shape	rc=124 (killed by timeout; dhclient never gives up on its own)	20.00 s (the full bound, exactly)
Rewrite bridge-ports ens19 → ens18 (tmp + mv, original kept)	one-line diff	~10 ms
ifreload -a	rc=0; raised the admin-DOWN candidate itself, port forwarding immediately (STP off, fd 0)	0.36 s
timeout 30 dhclient -1 -v vmbr0	DISCOVER→OFFER→REQUEST→ACK, bound 192.168.0.245	3.32 s
curl -fsS --max-time 15 https://hub.felhom.eu/	HTTP 302, rc=0	0.16 s

Post state: box reachable again at its baked static address over the new port (the static stanza was deliberately not touched — the probe changed only bridge-ports); dhclient's lease sits as a second address on vmbr0; default route unchanged.

Findings (design inputs for Part 3)

• F-P1 — the sequence works and is tightly bounded. Success ≈ 4 s; failed candidate = its DHCP bound + ε. With a 20 s bound: worst case ≈ NICs × ~21 s + one hub-probe timeout each. GO.
• F-P2 — unused physical NICs are admin-DOWN and their carrier is unreadable while down (/sys/class/net/*/carrier reads empty). A carrier-based candidate filter must ip link set <nic> up + settle (~2 s) before reading. ifreload -a brings the configured bridge-port up on its own, so the sweep proper needs no manual link handling — only the diagnosis table (NIC/MAC/cable) needs the raise-first step.
• F-P3 — carrier is necessary but not sufficient (virtio on a dead bridge shows carrier=1; a cabled port on a router with a dead uplink would too). Carrier selects/orders candidates; only DHCP-lease + hub-probe decides. Fine — that is the R-60 design anyway.
• F-P4 — dhclient -1 must be externally bounded (timeout N dhclient -1); on a dead segment it retries past any useful window (rc=124 at whatever bound is set). After a killed attempt, pkill dhclient before the next candidate — the second run then starts clean (proven: leg 2's dhclient ran without pidfile/lease complaints after leg 1's kill + pkill).
• F-P5 — treat ANY HTTP response as "hub reachable". The hub answers / with a 302; -f only fails ≥ 400, and even a 4xx would prove TLS+HTTP reachability. The gate's probe should key on "got an HTTP status" (curl rc 0, or rc 22 with a status code), not on 200.
• F-P6 — the trigger is "hub unreachable", never the 192.168.100.2 signature (see variant 2 above). The signature is named in the log/screen when present, as the likely cause.
• F-P7 — persisting only bridge-ports is enough to restore reachability in variant 2 (the static address stays valid when the cable moved ports). In variant 1 (fallback bake) the address itself is garbage — after a candidate wins, the persisted config must also carry the working addressing. Simplest shape that covers both: persist bridge-ports <winner> + iface vmbr0 inet dhcp (the winner is by definition a leasing NIC — DHCP is what proved it). Note for the R-59 screen loop: a box that self-healed onto DHCP addressing keeps working across router reboots; the operator can still pin it later.
• F-P8 (observation, out of scope): the pairing loop's register-failure line prints every cycle (~42 s) — the v1.21.0 quieting covered the 204 wait but not the no-network register path. The Part-3 gate makes this moot on the no-network path (the gate holds before register), but it is worth knowing the line exists.

Cleanup state after the probe

Nested VM 9310 stopped (destroyed in Part 4 after the drill re-uses the rig); vmbr9 + host-side 192.168.100.1/24 still mounted for Part 4; felhom-pve /etc/network/interfaces untouched throughout (imperative ip link bridge only; pre-task copy sha c7ace087183cb72e031ad0cd1974b9239d539747528dd19523d993f8e9c4ec7a — final diff happens at Part-4 cleanup). The probe box self-registered at the production hub as an unclaimed appliance while its network was up (expected; the record can be discarded, or will simply be superseded by the Part-4 drill box's registration).