felhom-controller/REPORT.md

REPORT — Controller→agent channel health-check, v0.90.0

Task: the next self-health slice — a periodic check that the controller↔agent local-API channel is up, classifying failures and alerting the operator + dashboard on a state change. Closes the gap the R1 pin-mismatch incident exposed (the channel was only probed once at startup and only logged). Spike-proven: felhom.eu/documentation/audits/SPIKE-controller-agent-channel-health-2026-06-29.md.

Baseline: controller main @ 77bccf1 (live 0.89.0) → v0.90.0 @ a277b18 (image …/felhom-controller:0.90.0). Companion hub change: felhom-hub v0.20.0 (allowlist; see below).

Implementation

• internal/channelhealth (NEW): a Checker over two seams — Probe (the channel call) and Sink (dashboard + notify) — so it is decoupled + table-tested. Each run classifies into up | down:<reason>.

• Classifier (spike Q1 map, substring match):

  reason	match	first-obs alert?	severity	event type
  pin_mismatch	TLS pin mismatch	yes	error	agent_channel_pin_mismatch
  unauthorized	HTTP 401	yes	error	agent_channel_unauthorized
  misconfigured	no such host / lookup	yes	error	agent_channel_misconfigured
  construction_error	agentClient() build error (LATCHES)	yes	error	agent_channel_construction_error
  unreachable	connection refused	no (N≥2)	warning	agent_channel_unreachable
  timeout	context deadline exceeded / i/o timeout	no (N≥2)	warning	agent_channel_timeout
  unknown	(fallthrough)	no (N≥2)	warning	agent_channel_unknown
  recovered	down→up transition	—	info	agent_channel_recovered

• Probe = the PRODUCTION memoized client (Server.ProbeAgentChannel → s.agentClient() + Storage()), not a fresh agentapi.New per probe (the spike's decisive call: self-heals, reflects the disk UI exactly, no transport leak). A construction error (latching sync.Once build failure) is surfaced distinctly via the constructionErr flag.

• Debounce + state machine: transient reasons need N≥2 consecutive down-probes before a transition (the ~1s agent-restart gap is suppressed); pin/401/DNS/construction transition on the first down. First scheduler observation seeds state, no alert. A transition (up→down, down→up, reason-change) fires once; steady-down does not re-fire.

• Operator alert + dashboard: Notifier.NotifyAgentChannelDown/Recovered push an English, operator-only event (customer can't act on it; the type is not a customer toggle); the hub applies its 1 h cooldown. AlertManager.SetAgentChannelAlert shows a short Hungarian banner whenever the channel is down (state-based, idempotent — prepended in GetAlerts). No customer email.

• Wiring: cmd/controller/main.go registers a ~60 s sched.Every("agent-channel-health", …) job (only when local_api.endpoint is set), after the webServer is built (it owns the memoized client). probeLocalAPI stays at startup. A channelSink adapter bridges the package to notify+alerts.

Hub change (was "verify-only" — live test showed it WAS needed)

The controller pushes agent_channel_* to /api/v1/event, but the hub's allowedEventTypes allowlist rejected them (live: Event push failed … HTTP 400). host_capability_* didn't hit this because those are hub-generated; agent_channel_* are controller-pushed. felhom-hub v0.20.0 adds the 8 types to the allowlist (operator-only; the dispatcher relays generically — no template change). No agent change.

Tests (all green: go build/vet/test ./...)

internal/channelhealth: classifier per-reason (pin/401/dns/construction), transitions (up→down once, steady-down no duplicate, down→up recovered, reason-change re-alerts), first-obs seed (born-down shows on dashboard, no notify), probe-seam-only. Debounce red-proof (TestDebounce_TransientBlipSuppressed): one refused → no alert; two consecutive → exactly one — a no-debounce impl fails this.

Live validation (felhom-pve guest 9201) — deployed + verified

• Startup: local-api: channel up (agent 192.168.0.162:8443) — guest 9201, 4 mount(s) + [scheduler] Registered periodic job: agent-channel-health (every 1m0s).
• Transient (clean systemctl restart felhom-agent): no channel-down alert, dashboard stayed clean (the ~2s gap + 60s cadence + debounce).
• Sustained (systemctl stop felhom-agent): the live logs show the debounce + transition exactly: [channel] transient down (unreachable, 1/2) — suppressed then 60s later [channel] agent channel DOWN (up→down:unreachable); the dashboard banner "A tárolókezelő ügynök nem elérhető." appeared (verified in the served HTML).
• Recovery (systemctl start): [channel] agent channel recovered (was down:unreachable).
• Operator relay: initially HTTP 400 (hub allowlist) → fixed in hub v0.20.0 → re-validated: the agent_channel_unreachable + agent_channel_recovered events push to the hub HTTP 200 (Event pushed: agent_channel_…).

NOT changed

The agent; the agentapi pin/transport; agentClient() memoization; the disk gate. Detection/surfacing only.

Forward note

Remaining self-health leg (backlog): the hub-side leaf-fp comparison (proactively catching an agent re-key fleet-wide before any user hits it).