felhom-controller/REPORT.md

# REPORT — Controller→agent channel health-check, v0.90.0

**Task:** the next self-health slice — a periodic check that the controller↔agent local-API channel is
up, classifying failures and alerting the operator + dashboard on a state change. Closes the gap the R1
pin-mismatch incident exposed (the channel was only probed once at startup and only logged).
Spike-proven: `felhom.eu/documentation/audits/SPIKE-controller-agent-channel-health-2026-06-29.md`.

**Baseline:** controller `main` @ `77bccf1` (live 0.89.0) → **v0.90.0** @ `a277b18` (image
`…/felhom-controller:0.90.0`). Companion hub change: felhom-hub **v0.20.0** (allowlist; see below).

## Implementation

- **`internal/channelhealth` (NEW):** a `Checker` over two seams — `Probe` (the channel call) and
  `Sink` (dashboard + notify) — so it is decoupled + table-tested. Each run classifies into
  `up | down:<reason>`.
- **Classifier (spike Q1 map, substring match):**

  | reason | match | first-obs alert? | severity | event type |
  |---|---|---|---|---|
  | pin_mismatch | `TLS pin mismatch` | yes | error | `agent_channel_pin_mismatch` |
  | unauthorized | `HTTP 401` | yes | error | `agent_channel_unauthorized` |
  | misconfigured | `no such host` / `lookup ` | yes | error | `agent_channel_misconfigured` |
  | construction_error | `agentClient()` build error (LATCHES) | yes | error | `agent_channel_construction_error` |
  | unreachable | `connection refused` | **no (N≥2)** | warning | `agent_channel_unreachable` |
  | timeout | `context deadline exceeded` / `i/o timeout` | **no (N≥2)** | warning | `agent_channel_timeout` |
  | unknown | (fallthrough) | no (N≥2) | warning | `agent_channel_unknown` |
  | recovered | down→up transition | — | info | `agent_channel_recovered` |

- **Probe = the PRODUCTION memoized client** (`Server.ProbeAgentChannel` → `s.agentClient()` +
  `Storage()`), **not** a fresh `agentapi.New` per probe (the spike's decisive call: self-heals,
  reflects the disk UI exactly, no transport leak). A construction error (latching `sync.Once` build
  failure) is surfaced distinctly via the `constructionErr` flag.
- **Debounce + state machine:** transient reasons need **N≥2** consecutive down-probes before a
  transition (the ~1s agent-restart gap is suppressed); pin/401/DNS/construction transition on the
  first down. First scheduler observation **seeds** state, no alert. A transition (up→down, down→up,
  reason-change) fires once; steady-down does not re-fire.
- **Operator alert + dashboard:** `Notifier.NotifyAgentChannelDown/Recovered` push an **English,
  operator-only** event (customer can't act on it; the type is not a customer toggle); the hub applies
  its 1 h cooldown. `AlertManager.SetAgentChannelAlert` shows a short **Hungarian** banner whenever the
  channel is down (state-based, idempotent — prepended in `GetAlerts`). **No customer email.**
- **Wiring:** `cmd/controller/main.go` registers a ~60 s `sched.Every("agent-channel-health", …)` job
  (only when `local_api.endpoint` is set), after the webServer is built (it owns the memoized client).
  `probeLocalAPI` stays at startup. A `channelSink` adapter bridges the package to notify+alerts.

## Hub change (was "verify-only" — live test showed it WAS needed)

The controller pushes `agent_channel_*` to `/api/v1/event`, but the hub's `allowedEventTypes`
allowlist rejected them (live: `Event push failed … HTTP 400`). `host_capability_*` didn't hit this
because those are hub-*generated*; `agent_channel_*` are controller-*pushed*. **felhom-hub v0.20.0**
adds the 8 types to the allowlist (operator-only; the dispatcher relays generically — no template
change). No agent change.

## Tests (all green: `go build/vet/test ./...`)

`internal/channelhealth`: classifier per-reason (pin/401/dns/construction), transitions (up→down once,
steady-down no duplicate, down→up recovered, reason-change re-alerts), first-obs seed (born-down shows
on dashboard, no notify), probe-seam-only. **Debounce red-proof** (`TestDebounce_TransientBlipSuppressed`):
one refused → **no** alert; two consecutive → exactly one — a no-debounce impl fails this.

## Live validation (felhom-pve guest 9201) — deployed + verified

- **Startup:** `local-api: channel up (agent 192.168.0.162:8443) — guest 9201, 4 mount(s)` +
  `[scheduler] Registered periodic job: agent-channel-health (every 1m0s)`.
- **Transient (clean `systemctl restart felhom-agent`):** **no** channel-down alert, dashboard stayed
  clean (the ~2s gap + 60s cadence + debounce).
- **Sustained (`systemctl stop felhom-agent`):** the live logs show the debounce + transition exactly:
  `[channel] transient down (unreachable, 1/2) — suppressed` then 60s later
  `[channel] agent channel DOWN (up→down:unreachable)`; the dashboard banner
  *"A tárolókezelő ügynök nem elérhető."* appeared (verified in the served HTML).
- **Recovery (`systemctl start`):** `[channel] agent channel recovered (was down:unreachable)`.
- **Operator relay:** initially `HTTP 400` (hub allowlist) → fixed in hub v0.20.0 → re-validated:
  the `agent_channel_unreachable` + `agent_channel_recovered` events push to the hub HTTP 200
  (`Event pushed: agent_channel_…`).

## NOT changed
The agent; the agentapi pin/transport; `agentClient()` memoization; the disk gate. Detection/surfacing
only.

## Forward note
Remaining self-health leg (backlog): the hub-side leaf-fp comparison (proactively catching an agent
re-key fleet-wide before any user hits it).