# SPIKE — R-50 island-bridge control plane, 2026-07-25

**Question:** the agent binds a LAN literal (`local_api.listen_addr`) and the guest dials that literal
from `bootstrap.json`. A DHCP/site change that moves the host's LAN address makes the agent **fail to
start** (`bind: cannot assign requested address`) — storage/PBS/quiesce/restore-test/DR all go down
silently (the 2026-07-20 vacation incident, F1). Target: a host-internal bridge (no physical port) with
fixed private addresses, so the control plane survives any router/lease/site move. This spike was to
**validate the whole chain end-to-end on the DRILL environment (qm300 / `demo-vm-felhom-2f4b00`)**
before any production spec exists.

## VERDICT UP FRONT: **GO** — validated end-to-end on a live drill (2026-07-25, evening).

**A nested-PVE drill appliance was provisioned on the t740 (`demo-hp`) through the REAL day-0 pipeline,
and probes P1–P8 ALL PASS.** The island-bridge control plane works end-to-end, survives the exact F1
failure that motivated the row, the pin is address-independent as predicted, the dnsmasq trap is
confirmed live AND its fix proven, and the whole topology survives a cold host reboot untouched.
**Recommendation: GO — write the production implementation spec (Phase A/B/C skeleton at the bottom).**
See **“## Empirical validation (2026-07-25 PM) — probes P1–P8”** below. The two earlier BLOCKED attempts
(no drill existed) are retained further down for provenance.

### Method caveat (honest scope) — **CLOSED 2026-07-25 by the implementation.**
The probes drove the mechanism via **manual, surgical config edits** on the drill. That caveat is now
**resolved**: R-50 shipped (agent **v0.96.0** — `buildBringUpConfig` attaches the island `net1` from
`local_api.island_bridge`/`island_guest_addr`; host-install **v1.19.0** creates `vmbr9`, writes the
island bind + `island_*` + the `lan_resolver.host_ip` LAN pin). The *provisioning* path is now proven
live, not just the runtime: (a) the migration runbook ran verbatim on the drill (`RUNBOOK-island-migration.md`,
B1); and (b) a scratch guest provisioned by the agent from the golden came up with `net1` on the island
**automatically, zero manual edits** (A4). Fleet migration of the two demo boxes remains (Phase B, operator
go/no-go). See `felhom-agent/CHANGELOG.md` v0.96.0 + `scripts/CHANGELOG.md` v1.19.0.

## Empirical validation (2026-07-25 PM) — probes P1–P8, ALL PASS

**Drill:** nested-PVE appliance `drill-r50` (QEMU VM **300** on `demo-hp`/t740), installed from the
current release ISO `felhom-pve-9.2-1-v1.25.0-nested-vm-generic-mkimage.iso` through the **real day-0**
(self-register → operator bind → deliver → guest provision). Reached the exact R-50 starting condition:
agent `local_api.listen_addr = 192.168.0.176:8443` (the LAN literal) **and** the guest's
`bootstrap.json local_api.endpoint = 192.168.0.176:8443` — the F1 literal baked in both places. Nested
guest **9201** runs the controller (agent 0.93.0, golden 0.161.0). Access to the drill-PVE:
break-glass root via hub `host_recovery/drill-r50-0a4f9a`, reached through `demo-hp`.
(Day-0 tail: **claim** left PENDING — customer-email-gated, no inbox on a scratch customer; **escrow**
N/A — no DR/PBS tier on the minimal drill. Neither gates the control-plane probes.)

| Probe | What it validates | Result |
|---|---|---|
| **P1** snapshot | rollback point before mutation | **PASS** — `qm snapshot 300 r50pre` (retained) |
| **P2 (3)** `vmbr9` create | portless host bridge, non-disruptive | **PASS** — `vmbr9 169.254.253.1/30` up via `ifreload -a`; **vmbr0 untouched** (`.176/24`), LAN gw still reachable, guest kept running |
| **P3 (4)** guest NIC hot-add | live hotplug, LAN leg undisturbed | **PASS** — `pct set 9201 -net1 …ip=169.254.253.2/30`; `eth1` up on the island, **`eth0`/LAN undisturbed** (`192.168.0.15/24`), host↔guest island ping both ways |
| **P4 (7)** dnsmasq trap **+ fix** | the Finding-1 coupling, live | **PASS (both)** — with `host_ip` **unset**, moving `listen_addr` to the island rebound dnsmasq to **`169.254.253.1:53`** and **LAN DNS `192.168.0.176:53` died**; setting `lan_resolver.host_ip=192.168.0.176` put dnsmasq **back on `192.168.0.176:53`** while the API bind stayed on the island |
| **P5 (6)** pin | pin survives the address move | **PASS** — served leaf SHA-256 over the island **identical** (`4ef1d953fe…f219bab`); authenticated `GET https://169.254.253.1:8443/storage` from the guest → **HTTP 200**. No cert re-issue. Finding 6 confirmed live |
| **P6 (5)** **F1 replay** | control plane survives a LAN move | **PASS — the money shot.** LAN moved `192.168.0.176 → .200`, `listen_addr` left on the island → agent **stays `active`**, still bound `169.254.253.1:8443`, control plane **HTTP 200**. **Contrast (original bug reproduced):** set `listen_addr` back to the now-absent `.176` → `level=ERROR "daemon: exited with error" err="localapi: bind 192.168.0.176:8443: listen…"`, systemd `status=1/FAILURE`, nothing bound — the 2026-07-20 incident, verbatim |
| **P7 (8)** survival matrix | topology is reboot-durable | **PASS** — **agent restart**: island bind + 200; **guest reboot**: `eth1` island NIC + controller + 200 returned; **host (qm300) COLD reboot**: `vmbr9`, agent island bind, dnsmasq on the LAN IP, guest autostart (`onboot=1`) with `eth1`, controller healthy, control plane **HTTP 200** — **all with zero intervention** |

**All source-grounded findings are now empirically confirmed:** the F1 double-bake (agent bind + guest
dial) moves atomically and works; the pin is DER-based and address-transparent (Finding 6); the dnsmasq
trap is real and its `lan_resolver.host_ip = LAN IP` fix works (Finding 1); link-local `169.254.253.x/30`
is a clean host-internal fabric that no LAN move perturbs. **The drill was left in the working island
configuration** (not rolled back); snapshot `r50pre` preserves the clean LAN-literal day-0 if a re-run
is wanted.

### GO/NO-GO: **GO.** Proceed to the implementation spec (Phase A/B/C below). No blocker remains; the one
residual is that the *provisioning* path (host-install writing the island config + golden-bake carrying
the new bootstrap template) is unbuilt — that IS the impl task, now de-risked.

---

### Empirical attempt 2 (2026-07-25, PM) — the t740 was checked; there is no drill VM there. — **SUPERSEDED by the validation above (a drill was then provisioned and probed).**
The operator ruled (2026-07-25) that **drill + build VMs are hosted on the HP t740 (`demo-hp`) from
now on**, so this half was re-attempted there. Discovery (read-only, no LAN scan): the t740 = `demo-hp`
(Tailscale `100.76.96.79` / LAN `192.168.0.87`), accessed via the hub-vaulted G1 break-glass root
credential (`host_recovery/demo-hp-bb76ea`; secret redacted). **Findings:**
```
t740 (demo-hp, PVE 9.2.2): qm list → (empty); /etc/pve/qemu-server/ → empty; standalone (no cluster);
                           only LXC guest 9201 (its OWN live demo controller). NO nested drill PVE VM.
hub registry:              only demo-hp-bb76ea (t740) + demo-felhom-8363b5 (N100) — NO drill appliance.
DooPlex:                   drill.qcow2 EXISTS (/mnt/5_hdd/felhom.eu/drill/, 18G) but is POWERED OFF —
                           a GOLDEN-BAKE appliance (bake logs to 0.153.0, last Jul 18–20); its nested
                           guest (9100) is purged after each bake, so it has no island-bridge topology
                           (agent + nested guest + bootstrap.json) to probe. DooPlex is the production
                           k3s node and is loaded (~17G of 62G free).
```
So the operator's ruling is **forward-looking and not yet realized** — no probeable drill appliance
exists on the t740, and the only artifact is a stale bake VM on the production node. **Per the hard rule
("do not improvise on production"), probes P2–P7 were NOT run; felhom-pve, the t740 host networking, and
guest 9201 were left UNTOUCHED.** **Operator decision (2026-07-25): they will provision a nested-PVE
drill VM on the t740 (agent + nested guest); this task re-runs then.** GO/NO-GO stays PENDING.

### Empirical attempt 1 (2026-07-25, AM) — drill env was GONE on felhom-pve.

**Probe 1 (drill env health check) — FAIL/ABSENT.** On felhom-pve (2026-07-25):

```
# qm list        → (empty)  — no QEMU VMs at all; the nested drill PVE VM qm300 does not exist
# pct list       → VMID 9201 "demo-felhom" running  — the LIVE demo controller only
# grep -rl '2f4b00|drill' /etc/pve/{qemu-server,lxc}/  → (no matches)
```

The drill appliance (a throwaway nested PVE-in-a-VM) has been torn down since the last drill. **Per the
task's hard environment rule — "if the drill env is down or unreachable: STOP, report, done; do not
improvise on production" — probes 3–8 (bridge create, guest NIC hot-add, island bind, the F1 replay,
survival matrix) were NOT run.** felhom-pve's bridges/NICs, the live agent's `listen_addr`, and guest
9201 were left **UNTOUCHED**. **No GO/NO-GO can be issued on live evidence; a rebuilt drill VM is
required to validate the chain.**

What follows is the **source-only analysis** (read-only; no host mutation) — the address plan, the
three named findings, and the provisioning inventory — so the eventual implementation spec (and the
next drill run) start from a real base rather than zero.

---

## Probes 3–8 — NOT RUN (drill env absent) — **SUPERSEDED: all now PASS, see “Empirical validation (2026-07-25 PM)” above.**

| Probe | Would validate | Status |
|---|---|---|
| 3 Bridge create (`vmbr9`, no ports, `ifreload -a`) w/ vmbr0 untouched | non-disruptive host bridge add | **BLOCKED** — needs drill host |
| 4 Guest NIC hot-add (`pct set -net1 …ip=169.254.253.2/30`) | live hotplug, LAN leg undisturbed | **BLOCKED** |
| 5 Agent island bind + **F1 replay** (renumber LAN → agent still starts/serves) | the money shot | **BLOCKED** |
| 6 Pin confirmation (bootstrap→island, controller round-trip, unchanged fingerprint) | pin survives address move | **BLOCKED** (but see source finding below) |
| 7 dnsmasq trap probe | LAN DNS on the LAN bridge, not the island | **BLOCKED** (source-confirmed below) |
| 8 Survival matrix (agent restart / guest reboot / host reboot) | control plane returns on the island | **BLOCKED** |

---

## Source-confirmed findings (read-only; no probe needed)

### F1 root cause — the LAN literal is baked in TWO places, together
- **Agent bind:** `felhom-agent/internal/config/config.go:229` — `ListenAddr string` "bridge IP:port,
  e.g. `192.168.0.162:8443`"; validated as `host:port` (`:280–284`). The install script sets it from
  the **vmbr0 (LAN) IP**: `felhom.eu/scripts/felhom-host-install.sh:2162` (`ip -4 addr show vmbr0`) →
  `:2226` (`base['local_api']['listen_addr'] = BRIDGE_ADDR`), `--bridge-ip` default = "vmbr0 IP : 8443"
  (`:114`).
- **Guest dial:** `felhom-agent/internal/provision/backhalf.go:75/129` writes that same `Endpoint`
  (bridge IP:port) into `bootstrap.json`'s `local_api.endpoint` (`internal/provision/doc.go:44`) — the
  address the in-guest controller dials.
→ Both must move to the island address **atomically** (agent bind + the guest's bootstrap endpoint), or
the controller can't reach the agent.

### Finding 1 — the dnsmasq trap (CONFIRMED in source; a MUST for the spec)
`felhom-agent/internal/config/config.go:182–210`: `LANResolverConfig.HostIP` is the **dnsmasq
listen-address**, and `WithDefaults(localAPIListen)` **derives it from the local-API bind addr** when
unset (`:208–210`, `l.HostIP = host(localAPIListen)`). So the instant `listen_addr` moves to the island
(`169.254.253.1`), the split-horizon LAN resolver would bind its DNS to the **island** bridge and **LAN
DNS dies** — a silent, nasty coupling. **Spec requirement:** set `LANResolverConfig.HostIP` **explicitly
to the LAN (vmbr0) bridge IP**; never let it default off `listen_addr` once the two diverge. (The
install script must write `lan_resolver.host_ip` alongside the new `listen_addr`.)

### Finding 6 (pin) — moving the bind should NOT need a cert re-issue
The controller pins the agent's **leaf-cert DER SHA-256**, not its SAN/address:
`felhom-agent/internal/localapi/cert_test.go:37` ("the reported fingerprint must equal the SHA-256 of
the served leaf DER"); the pinning verifier keys on the cert bytes, not the connect address. So an
address move is transparent to the pin **as long as the same leaf cert is served** — no re-issue
expected. *(Unproven live — that was probe 6; asserted from source.)* Caveat: if a future cert is minted
with the island IP in its SAN and something ever DOES check SAN, that changes — but today's pin does not.

### Address-plan recommendation
- **Recommend link-local `169.254.253.1/30` (host) ↔ `169.254.253.2/30` (guest) on `vmbr9`.**
  Rationale: RFC-3927 link-local (`169.254.0.0/16`) is, by standard, **neither routed nor DHCP-assigned
  on a LAN** — a home router won't hand it out or forward it, so a host-internal /30 cannot collide with
  any customer LAN, ever. A /30 is exactly two usable hosts (host + guest) — no waste, no ambiguity.
- Alternative (obscure RFC-1918 /30, e.g. `10.201.253.0/30`): also works, but a customer LAN *could*
  legitimately use `10.0.0.0/8` and route it, so it carries a small non-zero collision risk that
  link-local structurally does not. **Prefer link-local.**

### Provisioning inventory (what an implementation must touch — file:line)
| Concern | Location |
|---|---|
| Create `vmbr9` (portless) + host island IP | `felhom.eu/scripts/felhom-host-install.sh` (new: `/etc/network/interfaces` stanza + `ifreload -a`) |
| Guest `net1` on `vmbr9` (static island IP) | `felhom-host-install.sh` (new `pct set … -net1 …`) — **R-54 interplay:** guest **eth0 stays DHCP** (the guestnet healer keeps owning the LAN leg); the island NIC is static; the control plane simply stops depending on the LAN leg |
| `listen_addr` → island IP | `felhom-host-install.sh:2226` (+ default logic `:2162/:114`) → agent `config.go:229` |
| `lan_resolver.host_ip` → **LAN** vmbr0 IP (Finding 1) | new write in `felhom-host-install.sh`; consumed by `config.go:198 WithDefaults` |
| `bootstrap.json` endpoint → island IP | `felhom-agent/internal/provision/backhalf.go:129` (`Endpoint`) + the golden/bootstrap template |
| Leaf cert | no change expected (pin is DER-based; Finding 6) |

### Cluster-parity finding (Peti's two-node cluster)
A one-host demo needs `vmbr9` on that host only. A **cluster** needs **bridge parity on BOTH nodes** —
either per-node identical `/etc/network/interfaces` stanzas (simplest, drift-prone) or a Proxmox **SDN**
zone/vnet defined cluster-wide (one definition, auto-applied per node; preferred at ≥2 nodes). The
guest's island IP is per-guest and node-independent; the **agent-follows-guest** rule (each host's agent
binds its own `vmbr9` island IP) holds on both nodes. Migration order for the eventual rollout:
**drill-proven → demo (one host) → Peti (SDN on the 2-node cluster)**.

---

## Implementation-spec skeleton (for the future task — NOT this one)
1. **Phase A (agent + config):** teach the install/provision path to create `vmbr9`, add guest `net1`,
   set `listen_addr` + `bootstrap.json` endpoint to the island IP, and write `lan_resolver.host_ip = LAN
   IP` (Finding 1). Repos: `felhom.eu` (host-install), `felhom-agent` (provision/bootstrap template).
   **MinAgent/floor:** additive to provisioning; no controller coupling → **MinAgent unchanged**; a
   golden re-bake carries the new bootstrap template.
2. **Phase B (migration):** per-box runbook (bridge add → net1 → agent+bootstrap renumber → restart →
   F1-replay check → rollback = revert `listen_addr`+endpoint to the LAN IP), drill-first.
3. **Phase C (cluster):** SDN vnet on Peti's cluster; parity check on both nodes.

**STOP — the implementation is its own future task. This spike is BLOCKED on a rebuilt drill VM for the
empirical (probes 3–8) half; the design half above is source-grounded and ready.**