SPIKE — R-50 island-bridge control plane, 2026-07-25

Question: the agent binds a LAN literal (local_api.listen_addr) and the guest dials that literal from bootstrap.json. A DHCP/site change that moves the host's LAN address makes the agent fail to start (bind: cannot assign requested address) — storage/PBS/quiesce/restore-test/DR all go down silently (the 2026-07-20 vacation incident, F1). Target: a host-internal bridge (no physical port) with fixed private addresses, so the control plane survives any router/lease/site move. This spike was to validate the whole chain end-to-end on the DRILL environment (qm300 / demo-vm-felhom-2f4b00) before any production spec exists.

VERDICT UP FRONT: SPIKE BLOCKED (empirically) — drill environment is GONE.

Probe 1 (drill env health check) — FAIL/ABSENT. On felhom-pve (2026-07-25):

# qm list        → (empty)  — no QEMU VMs at all; the nested drill PVE VM qm300 does not exist
# pct list       → VMID 9201 "demo-felhom" running  — the LIVE demo controller only
# grep -rl '2f4b00|drill' /etc/pve/{qemu-server,lxc}/  → (no matches)

The drill appliance (a throwaway nested PVE-in-a-VM) has been torn down since the last drill. Per the task's hard environment rule — "if the drill env is down or unreachable: STOP, report, done; do not improvise on production" — probes 3–8 (bridge create, guest NIC hot-add, island bind, the F1 replay, survival matrix) were NOT run. felhom-pve's bridges/NICs, the live agent's listen_addr, and guest 9201 were left UNTOUCHED. No GO/NO-GO can be issued on live evidence; a rebuilt drill VM is required to validate the chain.

What follows is the source-only analysis (read-only; no host mutation) — the address plan, the three named findings, and the provisioning inventory — so the eventual implementation spec (and the next drill run) start from a real base rather than zero.

Probes 3–8 — NOT RUN (drill env absent)

Probe	Would validate	Status
3 Bridge create (`vmbr9`, no ports, `ifreload -a`) w/ vmbr0 untouched	non-disruptive host bridge add	BLOCKED — needs drill host
4 Guest NIC hot-add (`pct set -net1 …ip=169.254.253.2/30`)	live hotplug, LAN leg undisturbed	BLOCKED
5 Agent island bind + F1 replay (renumber LAN → agent still starts/serves)	the money shot	BLOCKED
6 Pin confirmation (bootstrap→island, controller round-trip, unchanged fingerprint)	pin survives address move	BLOCKED (but see source finding below)
7 dnsmasq trap probe	LAN DNS on the LAN bridge, not the island	BLOCKED (source-confirmed below)
8 Survival matrix (agent restart / guest reboot / host reboot)	control plane returns on the island	BLOCKED

Source-confirmed findings (read-only; no probe needed)

F1 root cause — the LAN literal is baked in TWO places, together

Agent bind: felhom-agent/internal/config/config.go:229 — ListenAddr string "bridge IP:port, e.g. 192.168.0.162:8443"; validated as host:port (:280–284). The install script sets it from the vmbr0 (LAN) IP: felhom.eu/scripts/felhom-host-install.sh:2162 (ip -4 addr show vmbr0) → :2226 (base['local_api']['listen_addr'] = BRIDGE_ADDR), --bridge-ip default = "vmbr0 IP : 8443" (:114).
Guest dial: felhom-agent/internal/provision/backhalf.go:75/129 writes that same Endpoint (bridge IP:port) into bootstrap.json's local_api.endpoint (internal/provision/doc.go:44) — the address the in-guest controller dials. → Both must move to the island address atomically (agent bind + the guest's bootstrap endpoint), or the controller can't reach the agent.

Finding 1 — the dnsmasq trap (CONFIRMED in source; a MUST for the spec)

felhom-agent/internal/config/config.go:182–210: LANResolverConfig.HostIP is the dnsmasq listen-address, and WithDefaults(localAPIListen) derives it from the local-API bind addr when unset (:208–210, l.HostIP = host(localAPIListen)). So the instant listen_addr moves to the island (169.254.253.1), the split-horizon LAN resolver would bind its DNS to the island bridge and LAN DNS dies — a silent, nasty coupling. Spec requirement: set LANResolverConfig.HostIP explicitly to the LAN (vmbr0) bridge IP; never let it default off listen_addr once the two diverge. (The install script must write lan_resolver.host_ip alongside the new listen_addr.)

Finding 6 (pin) — moving the bind should NOT need a cert re-issue

The controller pins the agent's leaf-cert DER SHA-256, not its SAN/address: felhom-agent/internal/localapi/cert_test.go:37 ("the reported fingerprint must equal the SHA-256 of the served leaf DER"); the pinning verifier keys on the cert bytes, not the connect address. So an address move is transparent to the pin as long as the same leaf cert is served — no re-issue expected. (Unproven live — that was probe 6; asserted from source.) Caveat: if a future cert is minted with the island IP in its SAN and something ever DOES check SAN, that changes — but today's pin does not.

Address-plan recommendation

Recommend link-local 169.254.253.1/30 (host) ↔ 169.254.253.2/30 (guest) on vmbr9. Rationale: RFC-3927 link-local (169.254.0.0/16) is, by standard, neither routed nor DHCP-assigned on a LAN — a home router won't hand it out or forward it, so a host-internal /30 cannot collide with any customer LAN, ever. A /30 is exactly two usable hosts (host + guest) — no waste, no ambiguity.
Alternative (obscure RFC-1918 /30, e.g. 10.201.253.0/30): also works, but a customer LAN could legitimately use 10.0.0.0/8 and route it, so it carries a small non-zero collision risk that link-local structurally does not. Prefer link-local.

Provisioning inventory (what an implementation must touch — file:line)

Concern	Location
Create `vmbr9` (portless) + host island IP	`felhom.eu/scripts/felhom-host-install.sh` (new: `/etc/network/interfaces` stanza + `ifreload -a`)
Guest `net1` on `vmbr9` (static island IP)	`felhom-host-install.sh` (new `pct set … -net1 …`) — R-54 interplay: guest eth0 stays DHCP (the guestnet healer keeps owning the LAN leg); the island NIC is static; the control plane simply stops depending on the LAN leg
`listen_addr` → island IP	`felhom-host-install.sh:2226` (+ default logic `:2162/:114`) → agent `config.go:229`
`lan_resolver.host_ip` → LAN vmbr0 IP (Finding 1)	new write in `felhom-host-install.sh`; consumed by `config.go:198 WithDefaults`
`bootstrap.json` endpoint → island IP	`felhom-agent/internal/provision/backhalf.go:129` (`Endpoint`) + the golden/bootstrap template
Leaf cert	no change expected (pin is DER-based; Finding 6)

Cluster-parity finding (Peti's two-node cluster)

A one-host demo needs vmbr9 on that host only. A cluster needs bridge parity on BOTH nodes — either per-node identical /etc/network/interfaces stanzas (simplest, drift-prone) or a Proxmox SDN zone/vnet defined cluster-wide (one definition, auto-applied per node; preferred at ≥2 nodes). The guest's island IP is per-guest and node-independent; the agent-follows-guest rule (each host's agent binds its own vmbr9 island IP) holds on both nodes. Migration order for the eventual rollout: drill-proven → demo (one host) → Peti (SDN on the 2-node cluster).

Implementation-spec skeleton (for the future task — NOT this one)

Phase A (agent + config): teach the install/provision path to create vmbr9, add guest net1, set listen_addr + bootstrap.json endpoint to the island IP, and write lan_resolver.host_ip = LAN IP (Finding 1). Repos: felhom.eu (host-install), felhom-agent (provision/bootstrap template). MinAgent/floor: additive to provisioning; no controller coupling → MinAgent unchanged; a golden re-bake carries the new bootstrap template.
Phase B (migration): per-box runbook (bridge add → net1 → agent+bootstrap renumber → restart → F1-replay check → rollback = revert listen_addr+endpoint to the LAN IP), drill-first.
Phase C (cluster): SDN vnet on Peti's cluster; parity check on both nodes.

STOP — the implementation is its own future task. This spike is BLOCKED on a rebuilt drill VM for the empirical (probes 3–8) half; the design half above is source-grounded and ready.

8.4 KiB Raw Blame History Unescape Escape