# RUNBOOK — repairing a `local_api` endpoint drift

**When to run:** the controller logs
`[ERROR] bootstrap: local_api endpoint DRIFT — …` at startup, or the operator receives a
`local_api_endpoint_drift` event, or the „A tárolókezelő ügynök címe elavult a beállításokban."
banner appears. Detection shipped in controller **v0.173.0** (R-77).

**What drift means:** `controller.yaml` and `bootstrap.json` disagree on `local_api.endpoint`. The
controller dials **`controller.yaml`**. The agent listens wherever it was last configured — which,
after the R-50 island migration, is `169.254.253.1:8443`.

**Why the controller does not fix this itself:** auto-reconcile would clobber a *correct*
`controller.yaml` from a *stale* `bootstrap.json` — a half-completed re-provision or a hand-repaired
guest would lose a working channel on its next restart, fleet-wide and silently. Which file is
authoritative is **R-78**, unresolved. Until it is ruled, repair is a deliberate operator action.

> **Operator-present.** This edits a live guest's config and restarts its controller. The channel
> being down also degrades the drive gate, guest-reboot recovery and the quiesce/backup loop, so it
> is worth doing promptly — but not blind.

---

## 1. Establish which value is correct — do NOT assume `bootstrap.json`

`bootstrap.json` is right *when the drift came from a migration or re-provision* — the common case.
It is **wrong** if someone hand-repaired `controller.yaml` after a failed provision. Decide with
evidence, not by default:

```bash
# What is the agent ACTUALLY listening on? This is the ground truth.
ssh <pve-host> "ss -lntH 'sport = :8443'"      # post-island: 169.254.253.1:8443
```

**The correct endpoint is whatever the agent is bound to.** Both files are just claims about it.

## 2. Compare all three fields, not only the endpoint

`mergeLocalAPI` replaces the **whole** block, so a migration may have moved more than the address.

```bash
pct exec <VMID> -- python3 - <<'PY'
import json, re, hashlib
CY="/var/lib/docker/volumes/felhom-controller-data/_data/controller.yaml"
BJ="/etc/felhom-bootstrap/bootstrap.json"
def yget(path):
    blk={}; inb=False
    for line in open(path):
        if re.match(r'^local_api:', line): inb=True; continue
        if inb and re.match(r'^\S', line): break
        if inb:
            m=re.match(r'\s+(\w+):\s*(.*)\s*$', line)
            if m: blk[m.group(1)]=m.group(2).strip().strip('"').strip("'")
    return blk
cy=yget(CY); bj=(json.load(open(BJ)).get("local_api") or {})
print("endpoint    cy/bj :", repr(cy.get("endpoint")), repr(bj.get("endpoint")))
print("fingerprint AGREE :", cy.get("fingerprint")==bj.get("fingerprint"))
print("token       AGREE :", cy.get("token")==bj.get("token"))
PY
```

**Never print the fingerprint or token values** — booleans only, and never into a committed file.

> **STOP if `fingerprint` or `token` disagree.** A moved pin or a rotated token is a *different*
> failure: correcting only the address would then fail closed on the pin (`agent_channel_pin_mismatch`)
> or 401 (`agent_channel_unauthorized`). Diagnose that separately before touching anything.

## 3. Back up, edit the endpoint only, restart

```bash
CY=/var/lib/docker/volumes/felhom-controller-data/_data/controller.yaml
pct exec <VMID> -- cp -a $CY $CY.pre-fix.bak
# edit ONLY local_api.endpoint to the value from step 1
pct exec <VMID> -- systemctl restart felhom-controller-bootstrap.service
```

## 4. Verify — and know what success looks like

```bash
pct exec <VMID> -- docker logs felhom-controller 2>&1 | grep -i "\[channel\]"
```

**Expect SILENCE, not a "recovered" line.** On a freshly restarted controller the first probe
succeeds and `Check` returns early (*healthy first-obs → no notify*), so nothing is logged. A
`recovered` line only appears when a *running* controller transitions down→up.

Positive confirmation is therefore:

- **zero** `[channel]` lines after ~90 s (a broken channel logs `transient down` within 60 s and
  `DOWN` within 120 s);
- the dashboard banner cleared;
- **zero** new `agent_channel_*` events hub-side;
- no `local_api endpoint DRIFT` line at startup (the two files now agree — if they still disagree
  because you corrected `controller.yaml` rather than `bootstrap.json`, the drift alert will keep
  firing and that is **correct**: it is telling you the two sources still disagree. Align the other
  file too, or accept the alert until R-78).

---

## Executed instances

| date | boxes | before → after | fingerprint/token | outcome |
|---|---|---|---|---|
| 2026-07-26 | demo-felhom 9201 | `192.168.0.162:8443` → `169.254.253.1:8443` | both AGREE | channel healthy; 0 `[channel]` lines in 90 s; banner cleared |
| 2026-07-26 | demo-hp 9201 | `192.168.0.87:8443` → `169.254.253.1:8443` | both AGREE | channel healthy; 0 `[channel]` lines in 90 s |

Root cause of both: the R-50 island migration (2026-07-25 12:42–12:46) rewrote `bootstrap.json` and
restarted the controllers, which re-read the untouched `controller.yaml`. Full analysis:
`audits/DIAG-agent-channel-2026-07-26.md`.