felhom.eu/documentation/audits/SPIKE-island-bridge-2026-07-25.md

SPIKE — R-50 island-bridge control plane, 2026-07-25

Question: the agent binds a LAN literal (local_api.listen_addr) and the guest dials that literal from bootstrap.json. A DHCP/site change that moves the host's LAN address makes the agent fail to start (bind: cannot assign requested address) — storage/PBS/quiesce/restore-test/DR all go down silently (the 2026-07-20 vacation incident, F1). Target: a host-internal bridge (no physical port) with fixed private addresses, so the control plane survives any router/lease/site move. This spike was to validate the whole chain end-to-end on the DRILL environment (qm300 / demo-vm-felhom-2f4b00) before any production spec exists.

VERDICT UP FRONT: GO — validated end-to-end on a live drill (2026-07-25, evening).

A nested-PVE drill appliance was provisioned on the t740 (demo-hp) through the REAL day-0 pipeline, and probes P1–P8 ALL PASS. The island-bridge control plane works end-to-end, survives the exact F1 failure that motivated the row, the pin is address-independent as predicted, the dnsmasq trap is confirmed live AND its fix proven, and the whole topology survives a cold host reboot untouched. Recommendation: GO — write the production implementation spec (Phase A/B/C skeleton at the bottom). See “## Empirical validation (2026-07-25 PM) — probes P1–P8” below. The two earlier BLOCKED attempts (no drill existed) are retained further down for provenance.

Method caveat (honest scope) — CLOSED 2026-07-25 by the implementation.

The probes drove the mechanism via manual, surgical config edits on the drill. That caveat is now resolved: R-50 shipped (agent v0.96.0 — buildBringUpConfig attaches the island net1 from local_api.island_bridge/island_guest_addr; host-install v1.19.0 creates vmbr9, writes the island bind + island_* + the lan_resolver.host_ip LAN pin). The provisioning path is now proven live, not just the runtime: (a) the migration runbook ran verbatim on the drill (RUNBOOK-island-migration.md, B1); and (b) a scratch guest provisioned by the agent from the golden came up with net1 on the island automatically, zero manual edits (A4). Fleet migration of the two demo boxes remains (Phase B, operator go/no-go). See felhom-agent/CHANGELOG.md v0.96.0 + scripts/CHANGELOG.md v1.19.0.

Empirical validation (2026-07-25 PM) — probes P1–P8, ALL PASS

Drill: nested-PVE appliance drill-r50 (QEMU VM 300 on demo-hp/t740), installed from the current release ISO felhom-pve-9.2-1-v1.25.0-nested-vm-generic-mkimage.iso through the real day-0 (self-register → operator bind → deliver → guest provision). Reached the exact R-50 starting condition: agent local_api.listen_addr = 192.168.0.176:8443 (the LAN literal) and the guest's bootstrap.json local_api.endpoint = 192.168.0.176:8443 — the F1 literal baked in both places. Nested guest 9201 runs the controller (agent 0.93.0, golden 0.161.0). Access to the drill-PVE: break-glass root via hub host_recovery/drill-r50-0a4f9a, reached through demo-hp. (Day-0 tail: claim left PENDING — customer-email-gated, no inbox on a scratch customer; escrow N/A — no DR/PBS tier on the minimal drill. Neither gates the control-plane probes.)

Probe	What it validates	Result
P1 snapshot	rollback point before mutation	PASS — qm snapshot 300 r50pre (retained)
P2 (3) vmbr9 create	portless host bridge, non-disruptive	PASS — vmbr9 169.254.253.1/30 up via ifreload -a; vmbr0 untouched (.176/24), LAN gw still reachable, guest kept running
P3 (4) guest NIC hot-add	live hotplug, LAN leg undisturbed	PASS — pct set 9201 -net1 …ip=169.254.253.2/30; eth1 up on the island, eth0/LAN undisturbed (192.168.0.15/24), host↔guest island ping both ways
P4 (7) dnsmasq trap + fix	the Finding-1 coupling, live	PASS (both) — with host_ip unset, moving listen_addr to the island rebound dnsmasq to 169.254.253.1:53 and LAN DNS 192.168.0.176:53 died; setting lan_resolver.host_ip=192.168.0.176 put dnsmasq back on 192.168.0.176:53 while the API bind stayed on the island
P5 (6) pin	pin survives the address move	PASS — served leaf SHA-256 over the island identical (4ef1d953fe…f219bab); authenticated GET https://169.254.253.1:8443/storage from the guest → HTTP 200. No cert re-issue. Finding 6 confirmed live
P6 (5) F1 replay	control plane survives a LAN move	PASS — the money shot. LAN moved 192.168.0.176 → .200, listen_addr left on the island → agent stays active, still bound 169.254.253.1:8443, control plane HTTP 200. Contrast (original bug reproduced): set listen_addr back to the now-absent .176 → level=ERROR "daemon: exited with error" err="localapi: bind 192.168.0.176:8443: listen…", systemd status=1/FAILURE, nothing bound — the 2026-07-20 incident, verbatim
P7 (8) survival matrix	topology is reboot-durable	PASS — agent restart: island bind + 200; guest reboot: eth1 island NIC + controller + 200 returned; host (qm300) COLD reboot: vmbr9, agent island bind, dnsmasq on the LAN IP, guest autostart (onboot=1) with eth1, controller healthy, control plane HTTP 200 — all with zero intervention

All source-grounded findings are now empirically confirmed: the F1 double-bake (agent bind + guest dial) moves atomically and works; the pin is DER-based and address-transparent (Finding 6); the dnsmasq trap is real and its lan_resolver.host_ip = LAN IP fix works (Finding 1); link-local 169.254.253.x/30 is a clean host-internal fabric that no LAN move perturbs. The drill was left in the working island configuration (not rolled back); snapshot r50pre preserves the clean LAN-literal day-0 if a re-run is wanted.

GO/NO-GO: GO. Proceed to the implementation spec (Phase A/B/C below). No blocker remains; the one

residual is that the provisioning path (host-install writing the island config + golden-bake carrying the new bootstrap template) is unbuilt — that IS the impl task, now de-risked.

---

Empirical attempt 2 (2026-07-25, PM) — the t740 was checked; there is no drill VM there. — SUPERSEDED by the validation above (a drill was then provisioned and probed).

The operator ruled (2026-07-25) that drill + build VMs are hosted on the HP t740 (demo-hp) from now on, so this half was re-attempted there. Discovery (read-only, no LAN scan): the t740 = demo-hp (Tailscale 100.76.96.79 / LAN 192.168.0.87), accessed via the hub-vaulted G1 break-glass root credential (host_recovery/demo-hp-bb76ea; secret redacted). Findings:

t740 (demo-hp, PVE 9.2.2): qm list → (empty); /etc/pve/qemu-server/ → empty; standalone (no cluster);
                           only LXC guest 9201 (its OWN live demo controller). NO nested drill PVE VM.
hub registry:              only demo-hp-bb76ea (t740) + demo-felhom-8363b5 (N100) — NO drill appliance.
DooPlex:                   drill.qcow2 EXISTS (/mnt/5_hdd/felhom.eu/drill/, 18G) but is POWERED OFF —
                           a GOLDEN-BAKE appliance (bake logs to 0.153.0, last Jul 18–20); its nested
                           guest (9100) is purged after each bake, so it has no island-bridge topology
                           (agent + nested guest + bootstrap.json) to probe. DooPlex is the production
                           k3s node and is loaded (~17G of 62G free).

So the operator's ruling is forward-looking and not yet realized — no probeable drill appliance exists on the t740, and the only artifact is a stale bake VM on the production node. Per the hard rule ("do not improvise on production"), probes P2–P7 were NOT run; felhom-pve, the t740 host networking, and guest 9201 were left UNTOUCHED. Operator decision (2026-07-25): they will provision a nested-PVE drill VM on the t740 (agent + nested guest); this task re-runs then. GO/NO-GO stays PENDING.

Empirical attempt 1 (2026-07-25, AM) — drill env was GONE on felhom-pve.

Probe 1 (drill env health check) — FAIL/ABSENT. On felhom-pve (2026-07-25):

# qm list        → (empty)  — no QEMU VMs at all; the nested drill PVE VM qm300 does not exist
# pct list       → VMID 9201 "demo-felhom" running  — the LIVE demo controller only
# grep -rl '2f4b00|drill' /etc/pve/{qemu-server,lxc}/  → (no matches)

The drill appliance (a throwaway nested PVE-in-a-VM) has been torn down since the last drill. Per the task's hard environment rule — "if the drill env is down or unreachable: STOP, report, done; do not improvise on production" — probes 3–8 (bridge create, guest NIC hot-add, island bind, the F1 replay, survival matrix) were NOT run. felhom-pve's bridges/NICs, the live agent's listen_addr, and guest 9201 were left UNTOUCHED. No GO/NO-GO can be issued on live evidence; a rebuilt drill VM is required to validate the chain.

What follows is the source-only analysis (read-only; no host mutation) — the address plan, the three named findings, and the provisioning inventory — so the eventual implementation spec (and the next drill run) start from a real base rather than zero.

---

Probes 3–8 — NOT RUN (drill env absent) — SUPERSEDED: all now PASS, see “Empirical validation (2026-07-25 PM)” above.

Probe	Would validate	Status
3 Bridge create (vmbr9, no ports, ifreload -a) w/ vmbr0 untouched	non-disruptive host bridge add	BLOCKED — needs drill host
4 Guest NIC hot-add (pct set -net1 …ip=169.254.253.2/30)	live hotplug, LAN leg undisturbed	BLOCKED
5 Agent island bind + F1 replay (renumber LAN → agent still starts/serves)	the money shot	BLOCKED
6 Pin confirmation (bootstrap→island, controller round-trip, unchanged fingerprint)	pin survives address move	BLOCKED (but see source finding below)
7 dnsmasq trap probe	LAN DNS on the LAN bridge, not the island	BLOCKED (source-confirmed below)
8 Survival matrix (agent restart / guest reboot / host reboot)	control plane returns on the island	BLOCKED

---

Source-confirmed findings (read-only; no probe needed)

F1 root cause — the LAN literal is baked in TWO places, together

• Agent bind: felhom-agent/internal/config/config.go:229 — ListenAddr string "bridge IP:port, e.g. 192.168.0.162:8443"; validated as host:port (:280–284). The install script sets it from the vmbr0 (LAN) IP: felhom.eu/scripts/felhom-host-install.sh:2162 (ip -4 addr show vmbr0) → :2226 (base['local_api']['listen_addr'] = BRIDGE_ADDR), --bridge-ip default = "vmbr0 IP : 8443" (:114).
• Guest dial: felhom-agent/internal/provision/backhalf.go:75/129 writes that same Endpoint (bridge IP:port) into bootstrap.json's local_api.endpoint (internal/provision/doc.go:44) — the address the in-guest controller dials. → Both must move to the island address atomically (agent bind + the guest's bootstrap endpoint), or the controller can't reach the agent.

Finding 1 — the dnsmasq trap (CONFIRMED in source; a MUST for the spec)

felhom-agent/internal/config/config.go:182–210: LANResolverConfig.HostIP is the dnsmasq listen-address, and WithDefaults(localAPIListen) derives it from the local-API bind addr when unset (:208–210, l.HostIP = host(localAPIListen)). So the instant listen_addr moves to the island (169.254.253.1), the split-horizon LAN resolver would bind its DNS to the island bridge and LAN DNS dies — a silent, nasty coupling. Spec requirement: set LANResolverConfig.HostIP explicitly to the LAN (vmbr0) bridge IP; never let it default off listen_addr once the two diverge. (The install script must write lan_resolver.host_ip alongside the new listen_addr.)

Finding 6 (pin) — moving the bind should NOT need a cert re-issue

The controller pins the agent's leaf-cert DER SHA-256, not its SAN/address: felhom-agent/internal/localapi/cert_test.go:37 ("the reported fingerprint must equal the SHA-256 of the served leaf DER"); the pinning verifier keys on the cert bytes, not the connect address. So an address move is transparent to the pin as long as the same leaf cert is served — no re-issue expected. (Unproven live — that was probe 6; asserted from source.) Caveat: if a future cert is minted with the island IP in its SAN and something ever DOES check SAN, that changes — but today's pin does not.

Address-plan recommendation

• Recommend link-local 169.254.253.1/30 (host) ↔ 169.254.253.2/30 (guest) on vmbr9. Rationale: RFC-3927 link-local (169.254.0.0/16) is, by standard, neither routed nor DHCP-assigned on a LAN — a home router won't hand it out or forward it, so a host-internal /30 cannot collide with any customer LAN, ever. A /30 is exactly two usable hosts (host + guest) — no waste, no ambiguity.
• Alternative (obscure RFC-1918 /30, e.g. 10.201.253.0/30): also works, but a customer LAN could legitimately use 10.0.0.0/8 and route it, so it carries a small non-zero collision risk that link-local structurally does not. Prefer link-local.

Provisioning inventory (what an implementation must touch — file:line)

Concern	Location
Create vmbr9 (portless) + host island IP	felhom.eu/scripts/felhom-host-install.sh (new: /etc/network/interfaces stanza + ifreload -a)
Guest net1 on vmbr9 (static island IP)	felhom-host-install.sh (new pct set … -net1 …) — R-54 interplay: guest eth0 stays DHCP (the guestnet healer keeps owning the LAN leg); the island NIC is static; the control plane simply stops depending on the LAN leg
listen_addr → island IP	felhom-host-install.sh:2226 (+ default logic :2162/:114) → agent config.go:229
lan_resolver.host_ip → LAN vmbr0 IP (Finding 1)	new write in felhom-host-install.sh; consumed by config.go:198 WithDefaults
bootstrap.json endpoint → island IP	felhom-agent/internal/provision/backhalf.go:129 (Endpoint) + the golden/bootstrap template
Leaf cert	no change expected (pin is DER-based; Finding 6)

Cluster-parity finding (Peti's two-node cluster)

A one-host demo needs vmbr9 on that host only. A cluster needs bridge parity on BOTH nodes — either per-node identical /etc/network/interfaces stanzas (simplest, drift-prone) or a Proxmox SDN zone/vnet defined cluster-wide (one definition, auto-applied per node; preferred at ≥2 nodes). The guest's island IP is per-guest and node-independent; the agent-follows-guest rule (each host's agent binds its own vmbr9 island IP) holds on both nodes. Migration order for the eventual rollout: drill-proven → demo (one host) → Peti (SDN on the 2-node cluster).

---

Implementation-spec skeleton (for the future task — NOT this one)

1. Phase A (agent + config): teach the install/provision path to create vmbr9, add guest net1, set listen_addr + bootstrap.json endpoint to the island IP, and write lan_resolver.host_ip = LAN IP (Finding 1). Repos: felhom.eu (host-install), felhom-agent (provision/bootstrap template). MinAgent/floor: additive to provisioning; no controller coupling → MinAgent unchanged; a golden re-bake carries the new bootstrap template.
2. Phase B (migration): per-box runbook (bridge add → net1 → agent+bootstrap renumber → restart → F1-replay check → rollback = revert listen_addr+endpoint to the LAN IP), drill-first.
3. Phase C (cluster): SDN vnet on Peti's cluster; parity check on both nodes.

STOP — the implementation is its own future task. This spike is BLOCKED on a rebuilt drill VM for the empirical (probes 3–8) half; the design half above is source-grounded and ready.