119 lines
8.4 KiB
Markdown
119 lines
8.4 KiB
Markdown
# SPIKE — R-50 island-bridge control plane, 2026-07-25
|
||
|
||
**Question:** the agent binds a LAN literal (`local_api.listen_addr`) and the guest dials that literal
|
||
from `bootstrap.json`. A DHCP/site change that moves the host's LAN address makes the agent **fail to
|
||
start** (`bind: cannot assign requested address`) — storage/PBS/quiesce/restore-test/DR all go down
|
||
silently (the 2026-07-20 vacation incident, F1). Target: a host-internal bridge (no physical port) with
|
||
fixed private addresses, so the control plane survives any router/lease/site move. This spike was to
|
||
**validate the whole chain end-to-end on the DRILL environment (qm300 / `demo-vm-felhom-2f4b00`)**
|
||
before any production spec exists.
|
||
|
||
## VERDICT UP FRONT: SPIKE BLOCKED (empirically) — drill environment is GONE.
|
||
|
||
**Probe 1 (drill env health check) — FAIL/ABSENT.** On felhom-pve (2026-07-25):
|
||
|
||
```
|
||
# qm list → (empty) — no QEMU VMs at all; the nested drill PVE VM qm300 does not exist
|
||
# pct list → VMID 9201 "demo-felhom" running — the LIVE demo controller only
|
||
# grep -rl '2f4b00|drill' /etc/pve/{qemu-server,lxc}/ → (no matches)
|
||
```
|
||
|
||
The drill appliance (a throwaway nested PVE-in-a-VM) has been torn down since the last drill. **Per the
|
||
task's hard environment rule — "if the drill env is down or unreachable: STOP, report, done; do not
|
||
improvise on production" — probes 3–8 (bridge create, guest NIC hot-add, island bind, the F1 replay,
|
||
survival matrix) were NOT run.** felhom-pve's bridges/NICs, the live agent's `listen_addr`, and guest
|
||
9201 were left **UNTOUCHED**. **No GO/NO-GO can be issued on live evidence; a rebuilt drill VM is
|
||
required to validate the chain.**
|
||
|
||
What follows is the **source-only analysis** (read-only; no host mutation) — the address plan, the
|
||
three named findings, and the provisioning inventory — so the eventual implementation spec (and the
|
||
next drill run) start from a real base rather than zero.
|
||
|
||
---
|
||
|
||
## Probes 3–8 — NOT RUN (drill env absent)
|
||
|
||
| Probe | Would validate | Status |
|
||
|---|---|---|
|
||
| 3 Bridge create (`vmbr9`, no ports, `ifreload -a`) w/ vmbr0 untouched | non-disruptive host bridge add | **BLOCKED** — needs drill host |
|
||
| 4 Guest NIC hot-add (`pct set -net1 …ip=169.254.253.2/30`) | live hotplug, LAN leg undisturbed | **BLOCKED** |
|
||
| 5 Agent island bind + **F1 replay** (renumber LAN → agent still starts/serves) | the money shot | **BLOCKED** |
|
||
| 6 Pin confirmation (bootstrap→island, controller round-trip, unchanged fingerprint) | pin survives address move | **BLOCKED** (but see source finding below) |
|
||
| 7 dnsmasq trap probe | LAN DNS on the LAN bridge, not the island | **BLOCKED** (source-confirmed below) |
|
||
| 8 Survival matrix (agent restart / guest reboot / host reboot) | control plane returns on the island | **BLOCKED** |
|
||
|
||
---
|
||
|
||
## Source-confirmed findings (read-only; no probe needed)
|
||
|
||
### F1 root cause — the LAN literal is baked in TWO places, together
|
||
- **Agent bind:** `felhom-agent/internal/config/config.go:229` — `ListenAddr string` "bridge IP:port,
|
||
e.g. `192.168.0.162:8443`"; validated as `host:port` (`:280–284`). The install script sets it from
|
||
the **vmbr0 (LAN) IP**: `felhom.eu/scripts/felhom-host-install.sh:2162` (`ip -4 addr show vmbr0`) →
|
||
`:2226` (`base['local_api']['listen_addr'] = BRIDGE_ADDR`), `--bridge-ip` default = "vmbr0 IP : 8443"
|
||
(`:114`).
|
||
- **Guest dial:** `felhom-agent/internal/provision/backhalf.go:75/129` writes that same `Endpoint`
|
||
(bridge IP:port) into `bootstrap.json`'s `local_api.endpoint` (`internal/provision/doc.go:44`) — the
|
||
address the in-guest controller dials.
|
||
→ Both must move to the island address **atomically** (agent bind + the guest's bootstrap endpoint), or
|
||
the controller can't reach the agent.
|
||
|
||
### Finding 1 — the dnsmasq trap (CONFIRMED in source; a MUST for the spec)
|
||
`felhom-agent/internal/config/config.go:182–210`: `LANResolverConfig.HostIP` is the **dnsmasq
|
||
listen-address**, and `WithDefaults(localAPIListen)` **derives it from the local-API bind addr** when
|
||
unset (`:208–210`, `l.HostIP = host(localAPIListen)`). So the instant `listen_addr` moves to the island
|
||
(`169.254.253.1`), the split-horizon LAN resolver would bind its DNS to the **island** bridge and **LAN
|
||
DNS dies** — a silent, nasty coupling. **Spec requirement:** set `LANResolverConfig.HostIP` **explicitly
|
||
to the LAN (vmbr0) bridge IP**; never let it default off `listen_addr` once the two diverge. (The
|
||
install script must write `lan_resolver.host_ip` alongside the new `listen_addr`.)
|
||
|
||
### Finding 6 (pin) — moving the bind should NOT need a cert re-issue
|
||
The controller pins the agent's **leaf-cert DER SHA-256**, not its SAN/address:
|
||
`felhom-agent/internal/localapi/cert_test.go:37` ("the reported fingerprint must equal the SHA-256 of
|
||
the served leaf DER"); the pinning verifier keys on the cert bytes, not the connect address. So an
|
||
address move is transparent to the pin **as long as the same leaf cert is served** — no re-issue
|
||
expected. *(Unproven live — that was probe 6; asserted from source.)* Caveat: if a future cert is minted
|
||
with the island IP in its SAN and something ever DOES check SAN, that changes — but today's pin does not.
|
||
|
||
### Address-plan recommendation
|
||
- **Recommend link-local `169.254.253.1/30` (host) ↔ `169.254.253.2/30` (guest) on `vmbr9`.**
|
||
Rationale: RFC-3927 link-local (`169.254.0.0/16`) is, by standard, **neither routed nor DHCP-assigned
|
||
on a LAN** — a home router won't hand it out or forward it, so a host-internal /30 cannot collide with
|
||
any customer LAN, ever. A /30 is exactly two usable hosts (host + guest) — no waste, no ambiguity.
|
||
- Alternative (obscure RFC-1918 /30, e.g. `10.201.253.0/30`): also works, but a customer LAN *could*
|
||
legitimately use `10.0.0.0/8` and route it, so it carries a small non-zero collision risk that
|
||
link-local structurally does not. **Prefer link-local.**
|
||
|
||
### Provisioning inventory (what an implementation must touch — file:line)
|
||
| Concern | Location |
|
||
|---|---|
|
||
| Create `vmbr9` (portless) + host island IP | `felhom.eu/scripts/felhom-host-install.sh` (new: `/etc/network/interfaces` stanza + `ifreload -a`) |
|
||
| Guest `net1` on `vmbr9` (static island IP) | `felhom-host-install.sh` (new `pct set … -net1 …`) — **R-54 interplay:** guest **eth0 stays DHCP** (the guestnet healer keeps owning the LAN leg); the island NIC is static; the control plane simply stops depending on the LAN leg |
|
||
| `listen_addr` → island IP | `felhom-host-install.sh:2226` (+ default logic `:2162/:114`) → agent `config.go:229` |
|
||
| `lan_resolver.host_ip` → **LAN** vmbr0 IP (Finding 1) | new write in `felhom-host-install.sh`; consumed by `config.go:198 WithDefaults` |
|
||
| `bootstrap.json` endpoint → island IP | `felhom-agent/internal/provision/backhalf.go:129` (`Endpoint`) + the golden/bootstrap template |
|
||
| Leaf cert | no change expected (pin is DER-based; Finding 6) |
|
||
|
||
### Cluster-parity finding (Peti's two-node cluster)
|
||
A one-host demo needs `vmbr9` on that host only. A **cluster** needs **bridge parity on BOTH nodes** —
|
||
either per-node identical `/etc/network/interfaces` stanzas (simplest, drift-prone) or a Proxmox **SDN**
|
||
zone/vnet defined cluster-wide (one definition, auto-applied per node; preferred at ≥2 nodes). The
|
||
guest's island IP is per-guest and node-independent; the **agent-follows-guest** rule (each host's agent
|
||
binds its own `vmbr9` island IP) holds on both nodes. Migration order for the eventual rollout:
|
||
**drill-proven → demo (one host) → Peti (SDN on the 2-node cluster)**.
|
||
|
||
---
|
||
|
||
## Implementation-spec skeleton (for the future task — NOT this one)
|
||
1. **Phase A (agent + config):** teach the install/provision path to create `vmbr9`, add guest `net1`,
|
||
set `listen_addr` + `bootstrap.json` endpoint to the island IP, and write `lan_resolver.host_ip = LAN
|
||
IP` (Finding 1). Repos: `felhom.eu` (host-install), `felhom-agent` (provision/bootstrap template).
|
||
**MinAgent/floor:** additive to provisioning; no controller coupling → **MinAgent unchanged**; a
|
||
golden re-bake carries the new bootstrap template.
|
||
2. **Phase B (migration):** per-box runbook (bridge add → net1 → agent+bootstrap renumber → restart →
|
||
F1-replay check → rollback = revert `listen_addr`+endpoint to the LAN IP), drill-first.
|
||
3. **Phase C (cluster):** SDN vnet on Peti's cluster; parity check on both nodes.
|
||
|
||
**STOP — the implementation is its own future task. This spike is BLOCKED on a rebuilt drill VM for the
|
||
empirical (probes 3–8) half; the design half above is source-grounded and ready.**
|