diff --git a/documentation/runbooks/RUNBOOK-island-migration.md b/documentation/runbooks/RUNBOOK-island-migration.md new file mode 100644 index 0000000..4831e69 --- /dev/null +++ b/documentation/runbooks/RUNBOOK-island-migration.md @@ -0,0 +1,179 @@ +# RUNBOOK — migrate a live box to the R-50 island control plane + +> **What this does.** Moves an already-deployed appliance from a LAN-literal local-API bind to the +> host-internal island bridge (`vmbr9`, `169.254.253.1/30`↔`.2/30`), so the controller→agent control +> plane survives any LAN/DHCP/site move (the F1 fix). **Idempotent, reversible, non-disruptive to apps.** +> Authority: `audits/SPIKE-island-bridge-2026-07-25.md` (GO). Artifacts: agent ≥ **0.96.0**, host-install +> **v1.19.0**. Params are LAW (spike-validated): `vmbr9` portless, host `169.254.253.1/30`, guest +> `169.254.253.2/30` on `net1`, `listen_addr=169.254.253.1:8443`, `lan_resolver.host_ip=`. +> +> **Scope:** ONE host with ONE customer guest (the current fleet shape). A cluster (Peti, 2 nodes) is +> **out** — it needs bridge parity on both nodes / an SDN vnet; that is Phase C, its own runbook. +> +> **The apps never stop.** Only the management channel moves; `cloudflared` + the app containers keep +> serving throughout. If any app-reachability check fails, **roll back and STOP** (§7). + +## 0. Preconditions (verify before touching anything) + +- Operator go for THIS box (demo-hp before demo-felhom; each is its own STOP point). +- Root shell on the target host (felhom-pve: `ssh felhom-pve`; demo-hp: break-glass, see `operations/nodes.md`). +- Agent binary **0.96.0** available (built + published + vouched). `felhom-agent --version` will read 0.96.0 after step 2. +- The box is healthy NOW: `systemctl is-active felhom-agent` = active; the guest is running; the app URL answers. +- **Not inside the box's backup window** (a vzdump lock blocks `pct set`). +- Record the guest VMID (`pct list`) and the LAN NIC of `local_api` (usually `vmbr0`). + +## 1. Fill the rollback table FIRST (record every value you will change) + +Capture the current state so any step can be reverted exactly. Example (fill with real values): + +| what | current (rollback to) | target | +|---|---|---| +| agent `local_api.listen_addr` | `192.168.0.87:8443` | `169.254.253.1:8443` | +| agent `lan_resolver.host_ip` | *(absent → derived)* | `` | +| agent `island_bridge` / `island_guest_addr` | *(absent)* | `vmbr9` / `169.254.253.2/30` | +| agent version | `0.95.0` | `0.96.0` | +| guest `net1` | *(absent)* | `name=eth1,bridge=vmbr9,ip=169.254.253.2/30` | +| guest `bootstrap.json` `local_api.endpoint` | `192.168.0.87:8443` | `169.254.253.1:8443` | +| `/etc/network/interfaces` | *(no vmbr9 stanza)* | `+ vmbr9 stanza` | +| host firewall (LAN 8443) | *(as-is)* | island-narrowed (LAST) | + +Grab them: +```bash +grep -oE '"listen_addr":[^,]*|"host_ip":[^,}]*|"island_[a-z_]*":[^,}]*' /etc/felhom-agent/agent.json +felhom-agent --version +VMID=9201 # <-- the customer guest +pct config "$VMID" | grep -E '^net[0-9]' +pct exec "$VMID" -- grep -oE '"endpoint":"[^"]*"' /etc/felhom-bootstrap/bootstrap.json 2>/dev/null \ + || cat /var/lib/felhom-agent/guests/"$VMID"/bootstrap/bootstrap.json # host-side writable source +LAN_IP=$(ip -4 -o addr show vmbr0 | awk '{print $4}' | cut -d/ -f1 | head -1); echo "LAN_IP=$LAN_IP" +``` + +## 2. Deploy agent 0.96.0 (if not already there) + +Standard sanctioned deploy (build on DooPlex → scp → backup `.bak-` → `install -m0755` → restart). +Verify `felhom-agent --version` = 0.96.0, `capabilities self-check … degraded=0`, guest still running. +*(Any agent ≥ 0.96.0 binds a configured island `listen_addr`; 0.96.0 also makes future re-provisions on +this box island-aware. `configs/` unchanged since 0.93.0 → binary-only.)* + +## 3. Create the island bridge (idempotent, non-disruptive — spike probe P2) + +```bash +grep -qE '^\s*iface\s+vmbr9\s' /etc/network/interfaces || { cp -a /etc/network/interfaces /etc/network/interfaces.pre-island.bak +cat >> /etc/network/interfaces <<'EOF' + +auto vmbr9 +iface vmbr9 inet static + address 169.254.253.1/30 + bridge-ports none + bridge-stp off + bridge-fd 0 +EOF +} +ifreload -a +ip -4 -o addr show vmbr9 | grep 169.254.253.1 # expect it up +ip -4 -o addr show vmbr0 # UNCHANGED +ping -c1 -W2 # LAN still reachable +``` +**Abort check:** vmbr0 changed or LAN gw unreachable → `ifreload` after removing the stanza; STOP. + +## 4. Hot-add the guest island NIC (spike probe P3) + +```bash +pct set "$VMID" -net1 name=eth1,bridge=vmbr9,ip=169.254.253.2/30 # eth0/LAN untouched +sleep 3 +pct exec "$VMID" -- ip -4 -o addr show eth1 | grep 169.254.253.2 # up +pct exec "$VMID" -- ip -4 -o addr show eth0 # LAN leg undisturbed +ping -c2 -W2 169.254.253.2 && pct exec "$VMID" -- ping -c2 -W2 169.254.253.1 # both ways +``` +*(If `CT is locked (backup)`: a vzdump is running — wait for it, do NOT `pct unlock`.)* + +## 5. Move the agent bind to the island + PIN LAN DNS (Finding-1) + +Edit `/etc/felhom-agent/agent.json` (JSON-safe; python shown): +```bash +python3 - <:53 (NOT the island) +``` +**Abort check:** agent not active, or dnsmasq bound to `169.254.253.1:53` instead of the LAN IP → the +`host_ip` pin was missed; fix it and restart, else roll back §7. + +## 6. Point the guest at the island + restart the controller + +```bash +python3 - <", urllib.request.urlopen(urllib.request.Request("https://"+ep+"/storage",headers={"Authorization":"Bearer "+tok}),context=ctx,timeout=8).status) +except urllib.error.HTTPError as e: print("island /storage ->", e.code) +PY +# b) LAN DNS answering on the LAN IP c) apps reachable (the public URL / a container health) +dig @"$LAN_IP" +time=2 +tries=1 example.com >/dev/null && echo "LAN DNS OK" +pct exec "$VMID" -- docker ps --format '{{.Names}} {{.Status}}' | grep -E 'controller|cloudflared' +# d) hub report lands (operator-side): the host's last report is current after a poke +``` +Expected: (a) **HTTP 200**, (b) LAN DNS OK, (c) controller + cloudflared `Up`/healthy, (d) hub current. + +## 8. Firewall — LAST, and only after §7 is green + +With the island bind, nothing listens on the LAN IP anymore (`ss -lnt 'sport = :8443'` shows only the +island) — the LAN:8443 surface is already CLOSED by the bind. Optionally apply the island narrowing from +`felhom-agent/configs/felhom-localapi-firewall.example` (belt-and-suspenders; changes nothing since vmbr9 +is portless). **Never do this before §7 passes.** + +## Rollback (any abort criterion, or apps degrade) + +Reverse in order, using the table from §1: +1. Guest bootstrap `endpoint` → the LAN value; `docker restart felhom-controller`. +2. Agent `listen_addr` → LAN value; remove `island_*`; `lan_resolver.host_ip` → original; restart agent. +3. `pct set "$VMID" -delete net1`. +4. Remove the `vmbr9` stanza (restore `/etc/network/interfaces.pre-island.bak`); `ifreload -a`. +5. (agent binary rollback if needed: `install` the `.bak-`; restart.) +Re-run §7's app checks — the box is back on the LAN bind, apps serving. + +## Abort criteria (hard) + +- Any loss of app reachability (public URL / container health) at any step → roll back, STOP, report. +- Agent will not start after a step → roll back that step, STOP. +- Hub stops receiving reports for > 1 poll after the move → investigate before proceeding; roll back if unresolved. + +## Per-box execution log (append one per box) + +Record: box, date, the filled rollback table, each step's result, §7 evidence (island 200, LAN DNS, +apps, hub), firewall state, and whether any rollback fired. + +### drill-r50 (VM 300 on demo-hp) — B1 validation run, 2026-07-25 — PASS +Rolled to `r50pre` (clean LAN-literal day-0, agent 0.93.0, `listen_addr=192.168.0.176:8443`, no vmbr9, +guest net0-only), then ran this runbook verbatim: +- §2 agent 0.93.0 → **0.96.0** (sha `af938601…`, caps ok, guest running). +- §3 `vmbr9` up `169.254.253.1/30`; **vmbr0 unchanged**; LAN reachable. +- §4 guest `eth1 169.254.253.2/30` hot-added; eth0/LAN undisturbed. +- §5 agent bind → `169.254.253.1:8443`; **dnsmasq on `127.0.0.1:53` + `192.168.0.176:53` (LAN), NOT the + island** — the `lan_resolver.host_ip` pin held (Finding-1). +- §6 bootstrap endpoint → island; controller restarted. +- §7 **island `/storage` → HTTP 200**; LAN DNS OK; `felhom-controller` healthy + `traefik` up; hub reports + agent 0.96.0 @ 12:29:52. No rollback fired. +Runbook validated on the disposable box before any live one.