R-50 Phase B: island migration runbook (B0) + drill validation (B1 PASS)
Idempotent LAN->island migration procedure with rollback table + abort criteria (firewall LAST). Validated verbatim on drill VM 300: rolled to r50pre, migrated, island /storage 200, LAN DNS held on the LAN IP (Finding-1 pin), apps healthy, hub reports 0.96.0. No rollback fired.
This commit is contained in:
@@ -0,0 +1,179 @@
|
||||
# RUNBOOK — migrate a live box to the R-50 island control plane
|
||||
|
||||
> **What this does.** Moves an already-deployed appliance from a LAN-literal local-API bind to the
|
||||
> host-internal island bridge (`vmbr9`, `169.254.253.1/30`↔`.2/30`), so the controller→agent control
|
||||
> plane survives any LAN/DHCP/site move (the F1 fix). **Idempotent, reversible, non-disruptive to apps.**
|
||||
> Authority: `audits/SPIKE-island-bridge-2026-07-25.md` (GO). Artifacts: agent ≥ **0.96.0**, host-install
|
||||
> **v1.19.0**. Params are LAW (spike-validated): `vmbr9` portless, host `169.254.253.1/30`, guest
|
||||
> `169.254.253.2/30` on `net1`, `listen_addr=169.254.253.1:8443`, `lan_resolver.host_ip=<LAN IP>`.
|
||||
>
|
||||
> **Scope:** ONE host with ONE customer guest (the current fleet shape). A cluster (Peti, 2 nodes) is
|
||||
> **out** — it needs bridge parity on both nodes / an SDN vnet; that is Phase C, its own runbook.
|
||||
>
|
||||
> **The apps never stop.** Only the management channel moves; `cloudflared` + the app containers keep
|
||||
> serving throughout. If any app-reachability check fails, **roll back and STOP** (§7).
|
||||
|
||||
## 0. Preconditions (verify before touching anything)
|
||||
|
||||
- Operator go for THIS box (demo-hp before demo-felhom; each is its own STOP point).
|
||||
- Root shell on the target host (felhom-pve: `ssh felhom-pve`; demo-hp: break-glass, see `operations/nodes.md`).
|
||||
- Agent binary **0.96.0** available (built + published + vouched). `felhom-agent --version` will read 0.96.0 after step 2.
|
||||
- The box is healthy NOW: `systemctl is-active felhom-agent` = active; the guest is running; the app URL answers.
|
||||
- **Not inside the box's backup window** (a vzdump lock blocks `pct set`).
|
||||
- Record the guest VMID (`pct list`) and the LAN NIC of `local_api` (usually `vmbr0`).
|
||||
|
||||
## 1. Fill the rollback table FIRST (record every value you will change)
|
||||
|
||||
Capture the current state so any step can be reverted exactly. Example (fill with real values):
|
||||
|
||||
| what | current (rollback to) | target |
|
||||
|---|---|---|
|
||||
| agent `local_api.listen_addr` | `192.168.0.87:8443` | `169.254.253.1:8443` |
|
||||
| agent `lan_resolver.host_ip` | *(absent → derived)* | `<LAN IP, e.g. 192.168.0.87>` |
|
||||
| agent `island_bridge` / `island_guest_addr` | *(absent)* | `vmbr9` / `169.254.253.2/30` |
|
||||
| agent version | `0.95.0` | `0.96.0` |
|
||||
| guest `net1` | *(absent)* | `name=eth1,bridge=vmbr9,ip=169.254.253.2/30` |
|
||||
| guest `bootstrap.json` `local_api.endpoint` | `192.168.0.87:8443` | `169.254.253.1:8443` |
|
||||
| `/etc/network/interfaces` | *(no vmbr9 stanza)* | `+ vmbr9 stanza` |
|
||||
| host firewall (LAN 8443) | *(as-is)* | island-narrowed (LAST) |
|
||||
|
||||
Grab them:
|
||||
```bash
|
||||
grep -oE '"listen_addr":[^,]*|"host_ip":[^,}]*|"island_[a-z_]*":[^,}]*' /etc/felhom-agent/agent.json
|
||||
felhom-agent --version
|
||||
VMID=9201 # <-- the customer guest
|
||||
pct config "$VMID" | grep -E '^net[0-9]'
|
||||
pct exec "$VMID" -- grep -oE '"endpoint":"[^"]*"' /etc/felhom-bootstrap/bootstrap.json 2>/dev/null \
|
||||
|| cat /var/lib/felhom-agent/guests/"$VMID"/bootstrap/bootstrap.json # host-side writable source
|
||||
LAN_IP=$(ip -4 -o addr show vmbr0 | awk '{print $4}' | cut -d/ -f1 | head -1); echo "LAN_IP=$LAN_IP"
|
||||
```
|
||||
|
||||
## 2. Deploy agent 0.96.0 (if not already there)
|
||||
|
||||
Standard sanctioned deploy (build on DooPlex → scp → backup `.bak-<old>` → `install -m0755` → restart).
|
||||
Verify `felhom-agent --version` = 0.96.0, `capabilities self-check … degraded=0`, guest still running.
|
||||
*(Any agent ≥ 0.96.0 binds a configured island `listen_addr`; 0.96.0 also makes future re-provisions on
|
||||
this box island-aware. `configs/` unchanged since 0.93.0 → binary-only.)*
|
||||
|
||||
## 3. Create the island bridge (idempotent, non-disruptive — spike probe P2)
|
||||
|
||||
```bash
|
||||
grep -qE '^\s*iface\s+vmbr9\s' /etc/network/interfaces || { cp -a /etc/network/interfaces /etc/network/interfaces.pre-island.bak
|
||||
cat >> /etc/network/interfaces <<'EOF'
|
||||
|
||||
auto vmbr9
|
||||
iface vmbr9 inet static
|
||||
address 169.254.253.1/30
|
||||
bridge-ports none
|
||||
bridge-stp off
|
||||
bridge-fd 0
|
||||
EOF
|
||||
}
|
||||
ifreload -a
|
||||
ip -4 -o addr show vmbr9 | grep 169.254.253.1 # expect it up
|
||||
ip -4 -o addr show vmbr0 # UNCHANGED
|
||||
ping -c1 -W2 <default-gw> # LAN still reachable
|
||||
```
|
||||
**Abort check:** vmbr0 changed or LAN gw unreachable → `ifreload` after removing the stanza; STOP.
|
||||
|
||||
## 4. Hot-add the guest island NIC (spike probe P3)
|
||||
|
||||
```bash
|
||||
pct set "$VMID" -net1 name=eth1,bridge=vmbr9,ip=169.254.253.2/30 # eth0/LAN untouched
|
||||
sleep 3
|
||||
pct exec "$VMID" -- ip -4 -o addr show eth1 | grep 169.254.253.2 # up
|
||||
pct exec "$VMID" -- ip -4 -o addr show eth0 # LAN leg undisturbed
|
||||
ping -c2 -W2 169.254.253.2 && pct exec "$VMID" -- ping -c2 -W2 169.254.253.1 # both ways
|
||||
```
|
||||
*(If `CT is locked (backup)`: a vzdump is running — wait for it, do NOT `pct unlock`.)*
|
||||
|
||||
## 5. Move the agent bind to the island + PIN LAN DNS (Finding-1)
|
||||
|
||||
Edit `/etc/felhom-agent/agent.json` (JSON-safe; python shown):
|
||||
```bash
|
||||
python3 - <<PY
|
||||
import json; p="/etc/felhom-agent/agent.json"; d=json.load(open(p))
|
||||
d["local_api"]["listen_addr"]="169.254.253.1:8443"
|
||||
d["local_api"]["island_bridge"]="vmbr9"
|
||||
d["local_api"]["island_guest_addr"]="169.254.253.2/30"
|
||||
d.setdefault("lan_resolver",{})["enable"]=True
|
||||
d["lan_resolver"]["host_ip"]="$LAN_IP" # MANDATORY — else DNS follows the bind onto the island and LAN DNS dies
|
||||
json.dump(d,open(p,"w"),indent=2)
|
||||
PY
|
||||
systemctl restart felhom-agent; sleep 4
|
||||
systemctl is-active felhom-agent
|
||||
ss -lntH 'sport = :8443' | awk '{print $4}' # expect 169.254.253.1:8443
|
||||
ss -lunH 'sport = :53' | awk '{print $4}' # expect 127.0.0.1:53 AND <LAN_IP>:53 (NOT the island)
|
||||
```
|
||||
**Abort check:** agent not active, or dnsmasq bound to `169.254.253.1:53` instead of the LAN IP → the
|
||||
`host_ip` pin was missed; fix it and restart, else roll back §7.
|
||||
|
||||
## 6. Point the guest at the island + restart the controller
|
||||
|
||||
```bash
|
||||
python3 - <<PY
|
||||
import json; p="/var/lib/felhom-agent/guests/$VMID/bootstrap/bootstrap.json"; d=json.load(open(p))
|
||||
d["local_api"]["endpoint"]="169.254.253.1:8443"
|
||||
json.dump(d,open(p,"w"))
|
||||
PY
|
||||
pct exec "$VMID" -- docker restart felhom-controller ; sleep 8
|
||||
```
|
||||
|
||||
## 7. VERIFY (all must pass; any app failure → ROLL BACK, STOP)
|
||||
|
||||
```bash
|
||||
# a) control plane over the island — authenticated round-trip
|
||||
pct exec "$VMID" -- python3 - <<PY
|
||||
import json,ssl,urllib.request,urllib.error
|
||||
d=json.load(open("/etc/felhom-bootstrap/bootstrap.json")); tok=d["local_api"]["token"]; ep=d["local_api"]["endpoint"]
|
||||
ctx=ssl.create_default_context(); ctx.check_hostname=False; ctx.verify_mode=ssl.CERT_NONE
|
||||
try: print("island /storage ->", urllib.request.urlopen(urllib.request.Request("https://"+ep+"/storage",headers={"Authorization":"Bearer "+tok}),context=ctx,timeout=8).status)
|
||||
except urllib.error.HTTPError as e: print("island /storage ->", e.code)
|
||||
PY
|
||||
# b) LAN DNS answering on the LAN IP c) apps reachable (the public URL / a container health)
|
||||
dig @"$LAN_IP" +time=2 +tries=1 example.com >/dev/null && echo "LAN DNS OK"
|
||||
pct exec "$VMID" -- docker ps --format '{{.Names}} {{.Status}}' | grep -E 'controller|cloudflared'
|
||||
# d) hub report lands (operator-side): the host's last report is current after a poke
|
||||
```
|
||||
Expected: (a) **HTTP 200**, (b) LAN DNS OK, (c) controller + cloudflared `Up`/healthy, (d) hub current.
|
||||
|
||||
## 8. Firewall — LAST, and only after §7 is green
|
||||
|
||||
With the island bind, nothing listens on the LAN IP anymore (`ss -lnt 'sport = :8443'` shows only the
|
||||
island) — the LAN:8443 surface is already CLOSED by the bind. Optionally apply the island narrowing from
|
||||
`felhom-agent/configs/felhom-localapi-firewall.example` (belt-and-suspenders; changes nothing since vmbr9
|
||||
is portless). **Never do this before §7 passes.**
|
||||
|
||||
## Rollback (any abort criterion, or apps degrade)
|
||||
|
||||
Reverse in order, using the table from §1:
|
||||
1. Guest bootstrap `endpoint` → the LAN value; `docker restart felhom-controller`.
|
||||
2. Agent `listen_addr` → LAN value; remove `island_*`; `lan_resolver.host_ip` → original; restart agent.
|
||||
3. `pct set "$VMID" -delete net1`.
|
||||
4. Remove the `vmbr9` stanza (restore `/etc/network/interfaces.pre-island.bak`); `ifreload -a`.
|
||||
5. (agent binary rollback if needed: `install` the `.bak-<old>`; restart.)
|
||||
Re-run §7's app checks — the box is back on the LAN bind, apps serving.
|
||||
|
||||
## Abort criteria (hard)
|
||||
|
||||
- Any loss of app reachability (public URL / container health) at any step → roll back, STOP, report.
|
||||
- Agent will not start after a step → roll back that step, STOP.
|
||||
- Hub stops receiving reports for > 1 poll after the move → investigate before proceeding; roll back if unresolved.
|
||||
|
||||
## Per-box execution log (append one per box)
|
||||
|
||||
Record: box, date, the filled rollback table, each step's result, §7 evidence (island 200, LAN DNS,
|
||||
apps, hub), firewall state, and whether any rollback fired.
|
||||
|
||||
### drill-r50 (VM 300 on demo-hp) — B1 validation run, 2026-07-25 — PASS
|
||||
Rolled to `r50pre` (clean LAN-literal day-0, agent 0.93.0, `listen_addr=192.168.0.176:8443`, no vmbr9,
|
||||
guest net0-only), then ran this runbook verbatim:
|
||||
- §2 agent 0.93.0 → **0.96.0** (sha `af938601…`, caps ok, guest running).
|
||||
- §3 `vmbr9` up `169.254.253.1/30`; **vmbr0 unchanged**; LAN reachable.
|
||||
- §4 guest `eth1 169.254.253.2/30` hot-added; eth0/LAN undisturbed.
|
||||
- §5 agent bind → `169.254.253.1:8443`; **dnsmasq on `127.0.0.1:53` + `192.168.0.176:53` (LAN), NOT the
|
||||
island** — the `lan_resolver.host_ip` pin held (Finding-1).
|
||||
- §6 bootstrap endpoint → island; controller restarted.
|
||||
- §7 **island `/storage` → HTTP 200**; LAN DNS OK; `felhom-controller` healthy + `traefik` up; hub reports
|
||||
agent 0.96.0 @ 12:29:52. No rollback fired.
|
||||
Runbook validated on the disposable box before any live one.
|
||||
Reference in New Issue
Block a user