felhom.eu/documentation/runbooks/RUNBOOK-local-api-endpoint-drift.md

RUNBOOK — repairing a local_api endpoint drift

When to run: the controller logs [ERROR] bootstrap: local_api endpoint DRIFT — … at startup, or the operator receives a local_api_endpoint_drift event, or the „A tárolókezelő ügynök címe elavult a beállításokban." banner appears. Detection shipped in controller v0.173.0 (R-77).

What drift means: controller.yaml and bootstrap.json disagree on local_api.endpoint. The controller dials controller.yaml. The agent listens wherever it was last configured — which, after the R-50 island migration, is 169.254.253.1:8443.

Why the controller does not fix this itself: auto-reconcile would clobber a correct controller.yaml from a stale bootstrap.json — a half-completed re-provision or a hand-repaired guest would lose a working channel on its next restart, fleet-wide and silently. Which file is authoritative is R-78, unresolved. Until it is ruled, repair is a deliberate operator action.

Operator-present. This edits a live guest's config and restarts its controller. The channel being down also degrades the drive gate, guest-reboot recovery and the quiesce/backup loop, so it is worth doing promptly — but not blind.

---

1. Establish which value is correct — do NOT assume bootstrap.json

bootstrap.json is right when the drift came from a migration or re-provision — the common case. It is wrong if someone hand-repaired controller.yaml after a failed provision. Decide with evidence, not by default:

# What is the agent ACTUALLY listening on? This is the ground truth.
ssh <pve-host> "ss -lntH 'sport = :8443'"      # post-island: 169.254.253.1:8443

The correct endpoint is whatever the agent is bound to. Both files are just claims about it.

2. Compare all three fields, not only the endpoint

mergeLocalAPI replaces the whole block, so a migration may have moved more than the address.

pct exec <VMID> -- python3 - <<'PY'
import json, re, hashlib
CY="/var/lib/docker/volumes/felhom-controller-data/_data/controller.yaml"
BJ="/etc/felhom-bootstrap/bootstrap.json"
def yget(path):
    blk={}; inb=False
    for line in open(path):
        if re.match(r'^local_api:', line): inb=True; continue
        if inb and re.match(r'^\S', line): break
        if inb:
            m=re.match(r'\s+(\w+):\s*(.*)\s*$', line)
            if m: blk[m.group(1)]=m.group(2).strip().strip('"').strip("'")
    return blk
cy=yget(CY); bj=(json.load(open(BJ)).get("local_api") or {})
print("endpoint    cy/bj :", repr(cy.get("endpoint")), repr(bj.get("endpoint")))
print("fingerprint AGREE :", cy.get("fingerprint")==bj.get("fingerprint"))
print("token       AGREE :", cy.get("token")==bj.get("token"))
PY

Never print the fingerprint or token values — booleans only, and never into a committed file.

STOP if fingerprint or token disagree. A moved pin or a rotated token is a different failure: correcting only the address would then fail closed on the pin (agent_channel_pin_mismatch) or 401 (agent_channel_unauthorized). Diagnose that separately before touching anything.

3. Back up, edit the endpoint only, restart

CY=/var/lib/docker/volumes/felhom-controller-data/_data/controller.yaml
pct exec <VMID> -- cp -a $CY $CY.pre-fix.bak
# edit ONLY local_api.endpoint to the value from step 1
pct exec <VMID> -- systemctl restart felhom-controller-bootstrap.service

4. Verify — and know what success looks like

pct exec <VMID> -- docker logs felhom-controller 2>&1 | grep -i "\[channel\]"

Expect SILENCE, not a "recovered" line. On a freshly restarted controller the first probe succeeds and Check returns early (healthy first-obs → no notify), so nothing is logged. A recovered line only appears when a running controller transitions down→up.

Positive confirmation is therefore:

• zero [channel] lines after ~90 s (a broken channel logs transient down within 60 s and DOWN within 120 s);
• the dashboard banner cleared;
• zero new agent_channel_* events hub-side;
• no local_api endpoint DRIFT line at startup (the two files now agree — if they still disagree because you corrected controller.yaml rather than bootstrap.json, the drift alert will keep firing and that is correct: it is telling you the two sources still disagree. Align the other file too, or accept the alert until R-78).

---

Executed instances

date	boxes	before → after	fingerprint/token	outcome
2026-07-26	demo-felhom 9201	192.168.0.162:8443 → 169.254.253.1:8443	both AGREE	channel healthy; 0 [channel] lines in 90 s; banner cleared
2026-07-26	demo-hp 9201	192.168.0.87:8443 → 169.254.253.1:8443	both AGREE	channel healthy; 0 [channel] lines in 90 s

Root cause of both: the R-50 island migration (2026-07-25 12:42–12:46) rewrote bootstrap.json and restarted the controllers, which re-read the untouched controller.yaml. Full analysis: audits/DIAG-agent-channel-2026-07-26.md.