admin
9cfa619ec3
The allowlist entry is REQUIRED, not cosmetic: handleEvent 400s an unknown event_type, so controller v0.173.0's new drift alert would be silently inert without it. Shipped with the controller that emits it. Docs: - RUNBOOK-local-api-endpoint-drift.md — how to repair a drift, including the step everyone will want to skip (establish which value is CORRECT from what the agent is actually bound to, rather than assuming bootstrap.json wins) and what success looks like (SILENCE, not a "recovered" line, because a fresh controller's healthy first observation is not logged). Records both 2026-07-26 repairs. - ROADMAP: R-77 shipped; R-78 the local_api authority ruling, with the clobber-a-working-channel risk spelled out in BOTH directions so it is not resolved opportunistically; R-79 the whole-surface English-strings sweep; R-80 expected_backup_missed, flagged as likely outranking R-77 because 7.3 days of stale backup materially exceeds the ~1.5-day channel outage, so the causal link the DIAG hedged on cannot be the whole story. - Capability map: note against the drive-wizard row (every agent-backed capability rides this channel) that a silent drift class is now detected. NO row status flips — detection is not prevention.
109 lines
5.0 KiB
Markdown
109 lines
5.0 KiB
Markdown
# RUNBOOK — repairing a `local_api` endpoint drift
|
||
|
||
**When to run:** the controller logs
|
||
`[ERROR] bootstrap: local_api endpoint DRIFT — …` at startup, or the operator receives a
|
||
`local_api_endpoint_drift` event, or the „A tárolókezelő ügynök címe elavult a beállításokban."
|
||
banner appears. Detection shipped in controller **v0.173.0** (R-77).
|
||
|
||
**What drift means:** `controller.yaml` and `bootstrap.json` disagree on `local_api.endpoint`. The
|
||
controller dials **`controller.yaml`**. The agent listens wherever it was last configured — which,
|
||
after the R-50 island migration, is `169.254.253.1:8443`.
|
||
|
||
**Why the controller does not fix this itself:** auto-reconcile would clobber a *correct*
|
||
`controller.yaml` from a *stale* `bootstrap.json` — a half-completed re-provision or a hand-repaired
|
||
guest would lose a working channel on its next restart, fleet-wide and silently. Which file is
|
||
authoritative is **R-78**, unresolved. Until it is ruled, repair is a deliberate operator action.
|
||
|
||
> **Operator-present.** This edits a live guest's config and restarts its controller. The channel
|
||
> being down also degrades the drive gate, guest-reboot recovery and the quiesce/backup loop, so it
|
||
> is worth doing promptly — but not blind.
|
||
|
||
---
|
||
|
||
## 1. Establish which value is correct — do NOT assume `bootstrap.json`
|
||
|
||
`bootstrap.json` is right *when the drift came from a migration or re-provision* — the common case.
|
||
It is **wrong** if someone hand-repaired `controller.yaml` after a failed provision. Decide with
|
||
evidence, not by default:
|
||
|
||
```bash
|
||
# What is the agent ACTUALLY listening on? This is the ground truth.
|
||
ssh <pve-host> "ss -lntH 'sport = :8443'" # post-island: 169.254.253.1:8443
|
||
```
|
||
|
||
**The correct endpoint is whatever the agent is bound to.** Both files are just claims about it.
|
||
|
||
## 2. Compare all three fields, not only the endpoint
|
||
|
||
`mergeLocalAPI` replaces the **whole** block, so a migration may have moved more than the address.
|
||
|
||
```bash
|
||
pct exec <VMID> -- python3 - <<'PY'
|
||
import json, re, hashlib
|
||
CY="/var/lib/docker/volumes/felhom-controller-data/_data/controller.yaml"
|
||
BJ="/etc/felhom-bootstrap/bootstrap.json"
|
||
def yget(path):
|
||
blk={}; inb=False
|
||
for line in open(path):
|
||
if re.match(r'^local_api:', line): inb=True; continue
|
||
if inb and re.match(r'^\S', line): break
|
||
if inb:
|
||
m=re.match(r'\s+(\w+):\s*(.*)\s*$', line)
|
||
if m: blk[m.group(1)]=m.group(2).strip().strip('"').strip("'")
|
||
return blk
|
||
cy=yget(CY); bj=(json.load(open(BJ)).get("local_api") or {})
|
||
print("endpoint cy/bj :", repr(cy.get("endpoint")), repr(bj.get("endpoint")))
|
||
print("fingerprint AGREE :", cy.get("fingerprint")==bj.get("fingerprint"))
|
||
print("token AGREE :", cy.get("token")==bj.get("token"))
|
||
PY
|
||
```
|
||
|
||
**Never print the fingerprint or token values** — booleans only, and never into a committed file.
|
||
|
||
> **STOP if `fingerprint` or `token` disagree.** A moved pin or a rotated token is a *different*
|
||
> failure: correcting only the address would then fail closed on the pin (`agent_channel_pin_mismatch`)
|
||
> or 401 (`agent_channel_unauthorized`). Diagnose that separately before touching anything.
|
||
|
||
## 3. Back up, edit the endpoint only, restart
|
||
|
||
```bash
|
||
CY=/var/lib/docker/volumes/felhom-controller-data/_data/controller.yaml
|
||
pct exec <VMID> -- cp -a $CY $CY.pre-fix.bak
|
||
# edit ONLY local_api.endpoint to the value from step 1
|
||
pct exec <VMID> -- systemctl restart felhom-controller-bootstrap.service
|
||
```
|
||
|
||
## 4. Verify — and know what success looks like
|
||
|
||
```bash
|
||
pct exec <VMID> -- docker logs felhom-controller 2>&1 | grep -i "\[channel\]"
|
||
```
|
||
|
||
**Expect SILENCE, not a "recovered" line.** On a freshly restarted controller the first probe
|
||
succeeds and `Check` returns early (*healthy first-obs → no notify*), so nothing is logged. A
|
||
`recovered` line only appears when a *running* controller transitions down→up.
|
||
|
||
Positive confirmation is therefore:
|
||
|
||
- **zero** `[channel]` lines after ~90 s (a broken channel logs `transient down` within 60 s and
|
||
`DOWN` within 120 s);
|
||
- the dashboard banner cleared;
|
||
- **zero** new `agent_channel_*` events hub-side;
|
||
- no `local_api endpoint DRIFT` line at startup (the two files now agree — if they still disagree
|
||
because you corrected `controller.yaml` rather than `bootstrap.json`, the drift alert will keep
|
||
firing and that is **correct**: it is telling you the two sources still disagree. Align the other
|
||
file too, or accept the alert until R-78).
|
||
|
||
---
|
||
|
||
## Executed instances
|
||
|
||
| date | boxes | before → after | fingerprint/token | outcome |
|
||
|---|---|---|---|---|
|
||
| 2026-07-26 | demo-felhom 9201 | `192.168.0.162:8443` → `169.254.253.1:8443` | both AGREE | channel healthy; 0 `[channel]` lines in 90 s; banner cleared |
|
||
| 2026-07-26 | demo-hp 9201 | `192.168.0.87:8443` → `169.254.253.1:8443` | both AGREE | channel healthy; 0 `[channel]` lines in 90 s |
|
||
|
||
Root cause of both: the R-50 island migration (2026-07-25 12:42–12:46) rewrote `bootstrap.json` and
|
||
restarted the controllers, which re-read the untouched `controller.yaml`. Full analysis:
|
||
`audits/DIAG-agent-channel-2026-07-26.md`.
|