# RUNBOOK — migrate a live box to the R-50 island control plane

> **What this does.** Moves an already-deployed appliance from a LAN-literal local-API bind to the
> host-internal island bridge (`vmbr9`, `169.254.253.1/30`↔`.2/30`), so the controller→agent control
> plane survives any LAN/DHCP/site move (the F1 fix). **Idempotent, reversible, non-disruptive to apps.**
> Authority: `audits/SPIKE-island-bridge-2026-07-25.md` (GO). Artifacts: agent ≥ **0.96.0**, host-install
> **v1.19.0**. Params are LAW (spike-validated): `vmbr9` portless, host `169.254.253.1/30`, guest
> `169.254.253.2/30` on `net1`, `listen_addr=169.254.253.1:8443`, `lan_resolver.host_ip=<LAN IP>`.
>
> **Scope:** ONE host with ONE customer guest (the current fleet shape). A cluster (Peti, 2 nodes) is
> **out** — it needs bridge parity on both nodes / an SDN vnet; that is Phase C, its own runbook.
>
> **The apps never stop.** Only the management channel moves; `cloudflared` + the app containers keep
> serving throughout. If any app-reachability check fails, **roll back and STOP** (§7).

## 0. Preconditions (verify before touching anything)

- Operator go for THIS box (demo-hp before demo-felhom; each is its own STOP point).
- Root shell on the target host (felhom-pve: `ssh felhom-pve`; demo-hp: break-glass, see `operations/nodes.md`).
- Agent binary **0.96.0** available (built + published + vouched). `felhom-agent --version` will read 0.96.0 after step 2.
- The box is healthy NOW: `systemctl is-active felhom-agent` = active; the guest is running; the app URL answers.
- **Not inside the box's backup window** (a vzdump lock blocks `pct set`).
- Record the guest VMID (`pct list`) and the LAN NIC of `local_api` (usually `vmbr0`).

## 1. Fill the rollback table FIRST (record every value you will change)

Capture the current state so any step can be reverted exactly. Example (fill with real values):

| what | current (rollback to) | target |
|---|---|---|
| agent `local_api.listen_addr` | `192.168.0.87:8443` | `169.254.253.1:8443` |
| agent `lan_resolver.host_ip` | *(absent → derived)* | `<LAN IP, e.g. 192.168.0.87>` |
| agent `island_bridge` / `island_guest_addr` | *(absent)* | `vmbr9` / `169.254.253.2/30` |
| agent version | `0.95.0` | `0.96.0` |
| guest `net1` | *(absent)* | `name=eth1,bridge=vmbr9,ip=169.254.253.2/30` |
| guest `bootstrap.json` `local_api.endpoint` | `192.168.0.87:8443` | `169.254.253.1:8443` |
| `/etc/network/interfaces` | *(no vmbr9 stanza)* | `+ vmbr9 stanza` |
| host firewall (LAN 8443) | *(as-is)* | island-narrowed (LAST) |

Grab them:
```bash
grep -oE '"listen_addr":[^,]*|"host_ip":[^,}]*|"island_[a-z_]*":[^,}]*' /etc/felhom-agent/agent.json
felhom-agent --version
VMID=9201   # <-- the customer guest
pct config "$VMID" | grep -E '^net[0-9]'
pct exec "$VMID" -- grep -oE '"endpoint":"[^"]*"' /etc/felhom-bootstrap/bootstrap.json 2>/dev/null \
  || cat /var/lib/felhom-agent/guests/"$VMID"/bootstrap/bootstrap.json   # host-side writable source
LAN_IP=$(ip -4 -o addr show vmbr0 | awk '{print $4}' | cut -d/ -f1 | head -1); echo "LAN_IP=$LAN_IP"
```

## 2. Deploy agent 0.96.0 (if not already there)

Standard sanctioned deploy (build on DooPlex → scp → backup `.bak-<old>` → `install -m0755` → restart).
Verify `felhom-agent --version` = 0.96.0, `capabilities self-check … degraded=0`, guest still running.
*(Any agent ≥ 0.96.0 binds a configured island `listen_addr`; 0.96.0 also makes future re-provisions on
this box island-aware. `configs/` unchanged since 0.93.0 → binary-only.)*

## 3. Create the island bridge (idempotent, non-disruptive — spike probe P2)

```bash
grep -qE '^\s*iface\s+vmbr9\s' /etc/network/interfaces || { cp -a /etc/network/interfaces /etc/network/interfaces.pre-island.bak
cat >> /etc/network/interfaces <<'EOF'

auto vmbr9
iface vmbr9 inet static
    address 169.254.253.1/30
    bridge-ports none
    bridge-stp off
    bridge-fd 0
EOF
}
ifreload -a
ip -4 -o addr show vmbr9 | grep 169.254.253.1     # expect it up
ip -4 -o addr show vmbr0                            # UNCHANGED
ping -c1 -W2 <default-gw>                           # LAN still reachable
```
**Abort check:** vmbr0 changed or LAN gw unreachable → `ifreload` after removing the stanza; STOP.

## 4. Hot-add the guest island NIC (spike probe P3)

```bash
pct set "$VMID" -net1 name=eth1,bridge=vmbr9,ip=169.254.253.2/30   # eth0/LAN untouched
sleep 3
pct exec "$VMID" -- ip -4 -o addr show eth1 | grep 169.254.253.2    # up
pct exec "$VMID" -- ip -4 -o addr show eth0                         # LAN leg undisturbed
ping -c2 -W2 169.254.253.2 && pct exec "$VMID" -- ping -c2 -W2 169.254.253.1   # both ways
```
*(If `CT is locked (backup)`: a vzdump is running — wait for it, do NOT `pct unlock`.)*

## 5. Move the agent bind to the island + PIN LAN DNS (Finding-1)

Edit `/etc/felhom-agent/agent.json` (JSON-safe; python shown):
```bash
python3 - <<PY
import json; p="/etc/felhom-agent/agent.json"; d=json.load(open(p))
d["local_api"]["listen_addr"]="169.254.253.1:8443"
d["local_api"]["island_bridge"]="vmbr9"
d["local_api"]["island_guest_addr"]="169.254.253.2/30"
d.setdefault("lan_resolver",{})["enable"]=True
d["lan_resolver"]["host_ip"]="$LAN_IP"     # MANDATORY — else DNS follows the bind onto the island and LAN DNS dies
json.dump(d,open(p,"w"),indent=2)
PY
systemctl restart felhom-agent; sleep 4
systemctl is-active felhom-agent
ss -lntH 'sport = :8443' | awk '{print $4}'                  # expect 169.254.253.1:8443
ss -lunH 'sport = :53'  | awk '{print $4}'                   # expect 127.0.0.1:53 AND <LAN_IP>:53 (NOT the island)
```
**Abort check:** agent not active, or dnsmasq bound to `169.254.253.1:53` instead of the LAN IP → the
`host_ip` pin was missed; fix it and restart, else roll back §7.

## 6. Point the guest at the island + restart the controller

```bash
python3 - <<PY
import json; p="/var/lib/felhom-agent/guests/$VMID/bootstrap/bootstrap.json"; d=json.load(open(p))
d["local_api"]["endpoint"]="169.254.253.1:8443"
json.dump(d,open(p,"w"))
PY
pct exec "$VMID" -- docker restart felhom-controller ; sleep 8
```

## 7. VERIFY (all must pass; any app failure → ROLL BACK, STOP)

```bash
# a) control plane over the island — authenticated round-trip
pct exec "$VMID" -- python3 - <<PY
import json,ssl,urllib.request,urllib.error
d=json.load(open("/etc/felhom-bootstrap/bootstrap.json")); tok=d["local_api"]["token"]; ep=d["local_api"]["endpoint"]
ctx=ssl.create_default_context(); ctx.check_hostname=False; ctx.verify_mode=ssl.CERT_NONE
try: print("island /storage ->", urllib.request.urlopen(urllib.request.Request("https://"+ep+"/storage",headers={"Authorization":"Bearer "+tok}),context=ctx,timeout=8).status)
except urllib.error.HTTPError as e: print("island /storage ->", e.code)
PY
# b) LAN DNS answering on the LAN IP     c) apps reachable (the public URL / a container health)
dig @"$LAN_IP" +time=2 +tries=1 example.com >/dev/null && echo "LAN DNS OK"
pct exec "$VMID" -- docker ps --format '{{.Names}} {{.Status}}' | grep -E 'controller|cloudflared'
# d) hub report lands (operator-side): the host's last report is current after a poke
```
Expected: (a) **HTTP 200**, (b) LAN DNS OK, (c) controller + cloudflared `Up`/healthy, (d) hub current.

## 8. Firewall — LAST, and only after §7 is green

With the island bind, nothing listens on the LAN IP anymore (`ss -lnt 'sport = :8443'` shows only the
island) — the LAN:8443 surface is already CLOSED by the bind. Optionally apply the island narrowing from
`felhom-agent/configs/felhom-localapi-firewall.example` (belt-and-suspenders; changes nothing since vmbr9
is portless). **Never do this before §7 passes.**

## Rollback (any abort criterion, or apps degrade)

Reverse in order, using the table from §1:
1. Guest bootstrap `endpoint` → the LAN value; `docker restart felhom-controller`.
2. Agent `listen_addr` → LAN value; remove `island_*`; `lan_resolver.host_ip` → original; restart agent.
3. `pct set "$VMID" -delete net1`.
4. Remove the `vmbr9` stanza (restore `/etc/network/interfaces.pre-island.bak`); `ifreload -a`.
5. (agent binary rollback if needed: `install` the `.bak-<old>`; restart.)
Re-run §7's app checks — the box is back on the LAN bind, apps serving.

## Abort criteria (hard)

- Any loss of app reachability (public URL / container health) at any step → roll back, STOP, report.
- Agent will not start after a step → roll back that step, STOP.
- Hub stops receiving reports for > 1 poll after the move → investigate before proceeding; roll back if unresolved.

## Per-box execution log (append one per box)

Record: box, date, the filled rollback table, each step's result, §7 evidence (island 200, LAN DNS,
apps, hub), firewall state, and whether any rollback fired.

### drill-r50 (VM 300 on demo-hp) — B1 validation run, 2026-07-25 — PASS
Rolled to `r50pre` (clean LAN-literal day-0, agent 0.93.0, `listen_addr=192.168.0.176:8443`, no vmbr9,
guest net0-only), then ran this runbook verbatim:
- §2 agent 0.93.0 → **0.96.0** (sha `af938601…`, caps ok, guest running).
- §3 `vmbr9` up `169.254.253.1/30`; **vmbr0 unchanged**; LAN reachable.
- §4 guest `eth1 169.254.253.2/30` hot-added; eth0/LAN undisturbed.
- §5 agent bind → `169.254.253.1:8443`; **dnsmasq on `127.0.0.1:53` + `192.168.0.176:53` (LAN), NOT the
  island** — the `lan_resolver.host_ip` pin held (Finding-1).
- §6 bootstrap endpoint → island; controller restarted.
- §7 **island `/storage` → HTTP 200**; LAN DNS OK; `felhom-controller` healthy + `traefik` up; hub reports
  agent 0.96.0 @ 12:29:52. No rollback fired.
Runbook validated on the disposable box before any live one.