felhom.eu/documentation/runbooks/RUNBOOK-island-migration.md

RUNBOOK — migrate a live box to the R-50 island control plane

What this does. Moves an already-deployed appliance from a LAN-literal local-API bind to the host-internal island bridge (vmbr9, 169.254.253.1/30↔.2/30), so the controller→agent control plane survives any LAN/DHCP/site move (the F1 fix). Idempotent, reversible, non-disruptive to apps. Authority: audits/SPIKE-island-bridge-2026-07-25.md (GO). Artifacts: agent ≥ 0.96.0, host-install v1.19.0. Params are LAW (spike-validated): vmbr9 portless, host 169.254.253.1/30, guest 169.254.253.2/30 on net1, listen_addr=169.254.253.1:8443, lan_resolver.host_ip=<LAN IP>.

Scope: ONE host with ONE customer guest (the current fleet shape). A cluster (Peti, 2 nodes) is out — it needs bridge parity on both nodes / an SDN vnet; that is Phase C, its own runbook.

The apps never stop. Only the management channel moves; cloudflared + the app containers keep serving throughout. If any app-reachability check fails, roll back and STOP (§7).

0. Preconditions (verify before touching anything)

• Operator go for THIS box (demo-hp before demo-felhom; each is its own STOP point).
• Root shell on the target host (felhom-pve: ssh felhom-pve; demo-hp: break-glass, see operations/nodes.md).
• Agent binary 0.96.0 available (built + published + vouched). felhom-agent --version will read 0.96.0 after step 2.
• The box is healthy NOW: systemctl is-active felhom-agent = active; the guest is running; the app URL answers.
• Not inside the box's backup window (a vzdump lock blocks pct set).
• Record the guest VMID (pct list) and the LAN NIC of local_api (usually vmbr0).

1. Fill the rollback table FIRST (record every value you will change)

Capture the current state so any step can be reverted exactly. Example (fill with real values):

what	current (rollback to)	target
agent local_api.listen_addr	192.168.0.87:8443	169.254.253.1:8443
agent lan_resolver.host_ip	(absent → derived)	<LAN IP, e.g. 192.168.0.87>
agent island_bridge / island_guest_addr	(absent)	vmbr9 / 169.254.253.2/30
agent version	0.95.0	0.96.0
guest net1	(absent)	name=eth1,bridge=vmbr9,ip=169.254.253.2/30
guest bootstrap.json local_api.endpoint	192.168.0.87:8443	169.254.253.1:8443
/etc/network/interfaces	(no vmbr9 stanza)	+ vmbr9 stanza
host firewall (LAN 8443)	(as-is)	island-narrowed (LAST)

Grab them:

grep -oE '"listen_addr":[^,]*|"host_ip":[^,}]*|"island_[a-z_]*":[^,}]*' /etc/felhom-agent/agent.json
felhom-agent --version
VMID=9201   # <-- the customer guest
pct config "$VMID" | grep -E '^net[0-9]'
pct exec "$VMID" -- grep -oE '"endpoint":"[^"]*"' /etc/felhom-bootstrap/bootstrap.json 2>/dev/null \
  || cat /var/lib/felhom-agent/guests/"$VMID"/bootstrap/bootstrap.json   # host-side writable source
LAN_IP=$(ip -4 -o addr show vmbr0 | awk '{print $4}' | cut -d/ -f1 | head -1); echo "LAN_IP=$LAN_IP"

2. Deploy agent 0.96.0 (if not already there)

Standard sanctioned deploy (build on DooPlex → scp → backup .bak-<old> → install -m0755 → restart). Verify felhom-agent --version = 0.96.0, capabilities self-check … degraded=0, guest still running. (Any agent ≥ 0.96.0 binds a configured island listen_addr; 0.96.0 also makes future re-provisions on this box island-aware. configs/ unchanged since 0.93.0 → binary-only.)

3. Create the island bridge (idempotent, non-disruptive — spike probe P2)

grep -qE '^\s*iface\s+vmbr9\s' /etc/network/interfaces || { cp -a /etc/network/interfaces /etc/network/interfaces.pre-island.bak
cat >> /etc/network/interfaces <<'EOF'

auto vmbr9
iface vmbr9 inet static
    address 169.254.253.1/30
    bridge-ports none
    bridge-stp off
    bridge-fd 0
EOF
}
ifreload -a
ip -4 -o addr show vmbr9 | grep 169.254.253.1     # expect it up
ip -4 -o addr show vmbr0                            # UNCHANGED
ping -c1 -W2 <default-gw>                           # LAN still reachable

Abort check: vmbr0 changed or LAN gw unreachable → ifreload after removing the stanza; STOP.

4. Hot-add the guest island NIC (spike probe P3)

pct set "$VMID" -net1 name=eth1,bridge=vmbr9,ip=169.254.253.2/30   # eth0/LAN untouched
sleep 3
pct exec "$VMID" -- ip -4 -o addr show eth1 | grep 169.254.253.2    # up
pct exec "$VMID" -- ip -4 -o addr show eth0                         # LAN leg undisturbed
ping -c2 -W2 169.254.253.2 && pct exec "$VMID" -- ping -c2 -W2 169.254.253.1   # both ways

(If CT is locked (backup): a vzdump is running — wait for it, do NOT pct unlock.)

5. Move the agent bind to the island + PIN LAN DNS (Finding-1)

Edit /etc/felhom-agent/agent.json (JSON-safe; python shown):

python3 - <<PY
import json; p="/etc/felhom-agent/agent.json"; d=json.load(open(p))
d["local_api"]["listen_addr"]="169.254.253.1:8443"
d["local_api"]["island_bridge"]="vmbr9"
d["local_api"]["island_guest_addr"]="169.254.253.2/30"
d.setdefault("lan_resolver",{})["enable"]=True
d["lan_resolver"]["host_ip"]="$LAN_IP"     # MANDATORY — else DNS follows the bind onto the island and LAN DNS dies
json.dump(d,open(p,"w"),indent=2)
PY
systemctl restart felhom-agent; sleep 4
systemctl is-active felhom-agent
ss -lntH 'sport = :8443' | awk '{print $4}'                  # expect 169.254.253.1:8443
ss -lunH 'sport = :53'  | awk '{print $4}'                   # expect 127.0.0.1:53 AND <LAN_IP>:53 (NOT the island)

Abort check: agent not active, or dnsmasq bound to 169.254.253.1:53 instead of the LAN IP → the host_ip pin was missed; fix it and restart, else roll back §7.

6. Point the guest at the island + restart the controller

python3 - <<PY
import json; p="/var/lib/felhom-agent/guests/$VMID/bootstrap/bootstrap.json"; d=json.load(open(p))
d["local_api"]["endpoint"]="169.254.253.1:8443"
json.dump(d,open(p,"w"))
PY
pct exec "$VMID" -- docker restart felhom-controller ; sleep 8

7. VERIFY (all must pass; any app failure → ROLL BACK, STOP)

# a) control plane over the island — authenticated round-trip
pct exec "$VMID" -- python3 - <<PY
import json,ssl,urllib.request,urllib.error
d=json.load(open("/etc/felhom-bootstrap/bootstrap.json")); tok=d["local_api"]["token"]; ep=d["local_api"]["endpoint"]
ctx=ssl.create_default_context(); ctx.check_hostname=False; ctx.verify_mode=ssl.CERT_NONE
try: print("island /storage ->", urllib.request.urlopen(urllib.request.Request("https://"+ep+"/storage",headers={"Authorization":"Bearer "+tok}),context=ctx,timeout=8).status)
except urllib.error.HTTPError as e: print("island /storage ->", e.code)
PY
# b) LAN DNS answering on the LAN IP     c) apps reachable (the public URL / a container health)
dig @"$LAN_IP" +time=2 +tries=1 example.com >/dev/null && echo "LAN DNS OK"
pct exec "$VMID" -- docker ps --format '{{.Names}} {{.Status}}' | grep -E 'controller|cloudflared'
# d) hub report lands (operator-side): the host's last report is current after a poke

Expected: (a) HTTP 200, (b) LAN DNS OK, (c) controller + cloudflared Up/healthy, (d) hub current.

8. Firewall — LAST, and only after §7 is green

With the island bind, nothing listens on the LAN IP anymore (ss -lnt 'sport = :8443' shows only the island) — the LAN:8443 surface is already CLOSED by the bind. Optionally apply the island narrowing from felhom-agent/configs/felhom-localapi-firewall.example (belt-and-suspenders; changes nothing since vmbr9 is portless). Never do this before §7 passes.

Rollback (any abort criterion, or apps degrade)

Reverse in order, using the table from §1:

1. Guest bootstrap endpoint → the LAN value; docker restart felhom-controller.
2. Agent listen_addr → LAN value; remove island_*; lan_resolver.host_ip → original; restart agent.
3. pct set "$VMID" -delete net1.
4. Remove the vmbr9 stanza (restore /etc/network/interfaces.pre-island.bak); ifreload -a.
5. (agent binary rollback if needed: install the .bak-<old>; restart.) Re-run §7's app checks — the box is back on the LAN bind, apps serving.

Abort criteria (hard)

• Any loss of app reachability (public URL / container health) at any step → roll back, STOP, report.
• Agent will not start after a step → roll back that step, STOP.
• Hub stops receiving reports for > 1 poll after the move → investigate before proceeding; roll back if unresolved.

Per-box execution log (append one per box)

Record: box, date, the filled rollback table, each step's result, §7 evidence (island 200, LAN DNS, apps, hub), firewall state, and whether any rollback fired.

drill-r50 (VM 300 on demo-hp) — B1 validation run, 2026-07-25 — PASS

Rolled to r50pre (clean LAN-literal day-0, agent 0.93.0, listen_addr=192.168.0.176:8443, no vmbr9, guest net0-only), then ran this runbook verbatim:

• §2 agent 0.93.0 → 0.96.0 (sha af938601…, caps ok, guest running).
• §3 vmbr9 up 169.254.253.1/30; vmbr0 unchanged; LAN reachable.
• §4 guest eth1 169.254.253.2/30 hot-added; eth0/LAN undisturbed.
• §5 agent bind → 169.254.253.1:8443; dnsmasq on 127.0.0.1:53 + 192.168.0.176:53 (LAN), NOT the island — the lan_resolver.host_ip pin held (Finding-1).
• §6 bootstrap endpoint → island; controller restarted.
• §7 island /storage → HTTP 200; LAN DNS OK; felhom-controller healthy + traefik up; hub reports agent 0.96.0 @ 12:29:52. No rollback fired. Runbook validated on the disposable box before any live one.