diff --git a/documentation/audits/SPIKE-firstboot-nic-sweep-2026-07-22.md b/documentation/audits/SPIKE-firstboot-nic-sweep-2026-07-22.md new file mode 100644 index 0000000..432a56e --- /dev/null +++ b/documentation/audits/SPIKE-firstboot-nic-sweep-2026-07-22.md @@ -0,0 +1,104 @@ +# SPIKE — first-boot NIC sweep mechanics (R-59/R-60 Phase-0), 2026-07-22 + +**Question:** on a Proxmox box that installed against the wrong NIC (the demo-hp shape: no DHCP on +any cable-bearing port at install time → the installer bakes its **192.168.100.2 static fallback** +into `vmbr0` and completes), does this sequence work, and is it bounded: +*re-point `vmbr0`'s `bridge-ports` to a candidate NIC → reload networking → bounded DHCP on vmbr0 → +hub HTTPS probe*? + +**Verdict up front: GO.** The whole success leg is ~4 s; a dead candidate costs exactly its DHCP +bound (20 s in the probe). Details and the design inputs below. + +**Vehicle:** PVE-in-a-VM on felhom-pve (nested VM 9310, throwaway), installed from the v1.24.0 +pairing ISO (uncommitted `nested-probe` profile = `nested-vm` + the ops SSH key for dead-network +access). Guest: PVE 9.2, `ifupdown2 3.3.0-1+pmx12`, `isc-dhcp-client 4.4.3-P1-8`. NICs: `ens18` +(net0) / `ens19` (net1). + +**Access-path deviation (recorded):** the probe needs a shell on a box whose network is by +definition dead. The plan was to use the installer's fallback address as the way in (host carries +`192.168.100.1/24` on the port-less throwaway `vmbr9`, `ssh root@192.168.100.2` over the dead +bridge, root key = the probe profile's baked ops key). The run then took a different door (below), +but the vmbr9 address was mounted and stays part of the Part-4 drill plan. + +## Reproduction — and an unplanned second trap variant + +The plan was both NICs dark from `qm start`. In practice the tap re-point to the dead bridge landed +~1–2 min after start, and **the installer had already leased on `ens19` (192.168.0.141) in that +window** — after which the flip cut it off. What that accident bought: + +- **The auto-installer picks the NIC that leases, not the first by name.** `bridge-ports ens19` + (net1, the second NIC) was baked, with the lease written as a **STATIC** stanza + (`address 192.168.0.141/24`, `gateway 192.168.0.1`). So the t740's "picked wrong" is not + name-order bad luck: at that site the 4-port card was the only cabled thing, no NIC leased, and + the installer fell back. Two distinct wrong-NIC install variants therefore exist: + 1. **no lease anywhere → 192.168.100.2 fallback baked static** (demo-hp, live-proven 2026-07-21; + README t740 trap) — R-59's detection signature; + 2. **a lease existed at install time but the truth changed after** (cable moved / switch port + died) → a plausible-looking static config that is nonetheless dead — no 192.168.100.2 to key + on. **The gate must therefore key on "hub unreachable", not on the fallback signature**; the + signature is a diagnosis detail for the screen/log, not the trigger. +- **The silent surface is real and was watched live for 30 minutes:** `felhom-bootstrap` in the + pairing loop logged `registration did not go through (no network yet?) — retrying` every ~42 s + (≈12 s curl DNS failure + 30 s sleep), unit `activating`, console showing only the login prompt. + Nothing on the physical screen names the problem — exactly R-59's complaint. + +## Probe log — the sweep walked by hand (detached; `/root/probe-sweep.log`) + +Shape at probe time (the "cable move"): `vmbr0` configured for `ens19` = dead (tap on vmbr9), +`ens18` = live (tap on vmbr0, site DHCP + internet available). Script scheduled via +`systemd-run --on-active=45s`, both taps flipped, box left alone. + +| Step | Result | Wall time | +|---|---|---| +| Leg 1: `timeout 20 dhclient -1 -v vmbr0` on the DEAD shape | rc=124 (killed by `timeout`; dhclient never gives up on its own) | **20.00 s** (the full bound, exactly) | +| Rewrite `bridge-ports ens19` → `ens18` (tmp + `mv`, original kept) | one-line diff | ~10 ms | +| `ifreload -a` | rc=0; **raised the admin-DOWN candidate itself**, port `forwarding` immediately (STP off, fd 0) | **0.36 s** | +| `timeout 30 dhclient -1 -v vmbr0` | DISCOVER→OFFER→REQUEST→ACK, bound 192.168.0.245 | **3.32 s** | +| `curl -fsS --max-time 15 https://hub.felhom.eu/` | HTTP **302**, rc=0 | **0.16 s** | + +Post state: box reachable again at its baked static address over the new port (the static stanza +was deliberately not touched — the probe changed only `bridge-ports`); dhclient's lease sits as a +second address on vmbr0; default route unchanged. + +## Findings (design inputs for Part 3) + +- **F-P1 — the sequence works and is tightly bounded.** Success ≈ 4 s; failed candidate = its DHCP + bound + ε. With a 20 s bound: worst case ≈ `NICs × ~21 s` + one hub-probe timeout each. GO. +- **F-P2 — unused physical NICs are admin-DOWN and their carrier is unreadable while down** + (`/sys/class/net/*/carrier` reads empty). A carrier-based candidate filter must + `ip link set up` + settle (~2 s) before reading. `ifreload -a` brings the *configured* + bridge-port up on its own, so the sweep proper needs no manual link handling — only the + *diagnosis table* (NIC/MAC/cable) needs the raise-first step. +- **F-P3 — carrier is necessary but not sufficient** (virtio on a dead bridge shows carrier=1; a + cabled port on a router with a dead uplink would too). Carrier selects/orders candidates; only + DHCP-lease + hub-probe decides. Fine — that is the R-60 design anyway. +- **F-P4 — `dhclient -1` must be externally bounded** (`timeout N dhclient -1`); on a dead segment + it retries past any useful window (rc=124 at whatever bound is set). After a killed attempt, + `pkill` dhclient before the next candidate — the second run then starts clean (proven: leg 2's + dhclient ran without pidfile/lease complaints after leg 1's kill + pkill). +- **F-P5 — treat ANY HTTP response as "hub reachable".** The hub answers `/` with a 302; `-f` + only fails ≥ 400, and even a 4xx would prove TLS+HTTP reachability. The gate's probe should key + on "got an HTTP status" (curl rc 0, or rc 22 with a status code), not on 200. +- **F-P6 — the trigger is "hub unreachable", never the 192.168.100.2 signature** (see variant 2 + above). The signature is named in the log/screen when present, as the likely cause. +- **F-P7 — persisting only `bridge-ports` is enough to restore reachability in variant 2** (the + static address stays valid when the cable moved ports). In variant 1 (fallback bake) the address + itself is garbage — after a candidate wins, the persisted config must also carry the working + addressing. Simplest shape that covers both: persist `bridge-ports ` + `iface vmbr0 inet + dhcp` (the winner is by definition a leasing NIC — DHCP is what proved it). Note for the R-59 + screen loop: a box that self-healed onto DHCP addressing keeps working across router reboots; + the operator can still pin it later. +- **F-P8 (observation, out of scope):** the pairing loop's register-failure line prints every + cycle (~42 s) — the v1.21.0 quieting covered the 204 wait but not the no-network register path. + The Part-3 gate makes this moot on the no-network path (the gate holds before register), but it + is worth knowing the line exists. + +## Cleanup state after the probe + +Nested VM 9310 **stopped** (destroyed in Part 4 after the drill re-uses the rig); `vmbr9` + +host-side `192.168.100.1/24` still mounted for Part 4; felhom-pve `/etc/network/interfaces` +untouched throughout (imperative `ip link` bridge only; pre-task copy sha +`c7ace087183cb72e031ad0cd1974b9239d539747528dd19523d993f8e9c4ec7a` — final diff happens at Part-4 +cleanup). The probe box self-registered at the production hub as an unclaimed appliance while its +network was up (expected; the record can be discarded, or will simply be superseded by the Part-4 +drill box's registration).