diff --git a/REPORT.md b/REPORT.md index 7003111..59a3c64 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,81 +1,45 @@ -# REPORT — Controller→agent channel health-check, v0.90.0 +# REPORT — F2: alert on born/persistent-down channel, v0.91.0 -**Task:** the next self-health slice — a periodic check that the controller↔agent local-API channel is -up, classifying failures and alerting the operator + dashboard on a state change. Closes the gap the R1 -pin-mismatch incident exposed (the channel was only probed once at startup and only logged). -Spike-proven: `felhom.eu/documentation/audits/SPIKE-controller-agent-channel-health-2026-06-29.md`. +**Task:** close F2 (the transition-only alerting gap the full-stack testrun surfaced) — a channel +failure present at **startup/reseed** was dashboard-only, no operator email, forever. Part of the +F2 (Part A) + prevention (Part B) slice; this repo is **Part A (controller)**. -**Baseline:** controller `main` @ `77bccf1` (live 0.89.0) → **v0.90.0** @ `a277b18` (image -`…/felhom-controller:0.90.0`). Companion hub change: felhom-hub **v0.20.0** (allowlist; see below). +**Baseline:** controller `main` @ `77bccf1` (live 0.90.0) → **v0.91.0** @ `b2ad871` (image +`…/felhom-controller:0.91.0`). Companions: hub **v0.21.0** (F2 hub checkers), agent **v0.46.0** (B.1). -## Implementation +## The gap + fix (`internal/channelhealth/checker.go`) -- **`internal/channelhealth` (NEW):** a `Checker` over two seams — `Probe` (the channel call) and - `Sink` (dashboard + notify) — so it is decoupled + table-tested. Each run classifies into - `up | down:`. -- **Classifier (spike Q1 map, substring match):** +Pre-fix `Check()`: a confirmed (past-debounce) down with `prev==""` hit `// first observation seeds, +no alert`, then `prev==newState` every cycle → **never alerted**. A controller that *boots* into a +broken channel (e.g. right after a leaf regen + controller restart) was dashboard-only. - | reason | match | first-obs alert? | severity | event type | - |---|---|---|---|---| - | pin_mismatch | `TLS pin mismatch` | yes | error | `agent_channel_pin_mismatch` | - | unauthorized | `HTTP 401` | yes | error | `agent_channel_unauthorized` | - | misconfigured | `no such host` / `lookup ` | yes | error | `agent_channel_misconfigured` | - | construction_error | `agentClient()` build error (LATCHES) | yes | error | `agent_channel_construction_error` | - | unreachable | `connection refused` | **no (N≥2)** | warning | `agent_channel_unreachable` | - | timeout | `context deadline exceeded` / `i/o timeout` | **no (N≥2)** | warning | `agent_channel_timeout` | - | unknown | (fallthrough) | no (N≥2) | warning | `agent_channel_unknown` | - | recovered | down→up transition | — | info | `agent_channel_recovered` | - -- **Probe = the PRODUCTION memoized client** (`Server.ProbeAgentChannel` → `s.agentClient()` + - `Storage()`), **not** a fresh `agentapi.New` per probe (the spike's decisive call: self-heals, - reflects the disk UI exactly, no transport leak). A construction error (latching `sync.Once` build - failure) is surfaced distinctly via the `constructionErr` flag. -- **Debounce + state machine:** transient reasons need **N≥2** consecutive down-probes before a - transition (the ~1s agent-restart gap is suppressed); pin/401/DNS/construction transition on the - first down. First scheduler observation **seeds** state, no alert. A transition (up→down, down→up, - reason-change) fires once; steady-down does not re-fire. -- **Operator alert + dashboard:** `Notifier.NotifyAgentChannelDown/Recovered` push an **English, - operator-only** event (customer can't act on it; the type is not a customer toggle); the hub applies - its 1 h cooldown. `AlertManager.SetAgentChannelAlert` shows a short **Hungarian** banner whenever the - channel is down (state-based, idempotent — prepended in `GetAlerts`). **No customer email.** -- **Wiring:** `cmd/controller/main.go` registers a ~60 s `sched.Every("agent-channel-health", …)` job - (only when `local_api.endpoint` is set), after the webServer is built (it owns the memoized client). - `probeLocalAPI` stays at startup. A `channelSink` adapter bridges the package to notify+alerts. - -## Hub change (was "verify-only" — live test showed it WAS needed) - -The controller pushes `agent_channel_*` to `/api/v1/event`, but the hub's `allowedEventTypes` -allowlist rejected them (live: `Event push failed … HTTP 400`). `host_capability_*` didn't hit this -because those are hub-*generated*; `agent_channel_*` are controller-*pushed*. **felhom-hub v0.20.0** -adds the 8 types to the allowlist (operator-only; the dispatcher relays generically — no template -change). No agent change. +**Fix — an `alerted` flag** (have we emitted for the CURRENT down-spell?): +- **UP:** `alerted=false` (re-arm); if `prev` was `down:*` → `NotifyRecovered`. +- **DOWN, transient + `consecutiveDown < N`:** unchanged debounce (suppress; unseeded → assume up). +- **DOWN, confirmed:** dashboard down. If `prev` was up/unseeded **or** the reason changed → new spell + → `alerted=false`. Then `if !alerted → NotifyDown; alerted=true`. Removed the `prev==""` + silent-seed-for-down branch. +- Net: non-transient **born-down alerts on cycle 1**; transient born-down still needs N≥2 (cold-boot + agent-not-yet-up race); **healthy** first-obs still silent; reason-change re-alerts; recovery re-arms. ## Tests (all green: `go build/vet/test ./...`) +F2 **born-down red-proof** (`TestF2_BornDownNonTransient_AlertsOnce`: one alert on cycle 1) + companion +(`…OldSeedSilentLogicWouldNotAlert`: the old `prev==""` path would not have alerted); born-down +transient still debounced; recovery re-arms the spell; healthy first-obs silent. The existing +debounce/transition/classify tests still pass. -`internal/channelhealth`: classifier per-reason (pin/401/dns/construction), transitions (up→down once, -steady-down no duplicate, down→up recovered, reason-change re-alerts), first-obs seed (born-down shows -on dashboard, no notify), probe-seam-only. **Debounce red-proof** (`TestDebounce_TransientBlipSuppressed`): -one refused → **no** alert; two consecutive → exactly one — a no-debounce impl fails this. - -## Live validation (felhom-pve guest 9201) — deployed + verified - -- **Startup:** `local-api: channel up (agent 192.168.0.162:8443) — guest 9201, 4 mount(s)` + - `[scheduler] Registered periodic job: agent-channel-health (every 1m0s)`. -- **Transient (clean `systemctl restart felhom-agent`):** **no** channel-down alert, dashboard stayed - clean (the ~2s gap + 60s cadence + debounce). -- **Sustained (`systemctl stop felhom-agent`):** the live logs show the debounce + transition exactly: - `[channel] transient down (unreachable, 1/2) — suppressed` then 60s later - `[channel] agent channel DOWN (up→down:unreachable)`; the dashboard banner - *"A tárolókezelő ügynök nem elérhető."* appeared (verified in the served HTML). -- **Recovery (`systemctl start`):** `[channel] agent channel recovered (was down:unreachable)`. -- **Operator relay:** initially `HTTP 400` (hub allowlist) → fixed in hub v0.20.0 → re-validated: - the `agent_channel_unreachable` + `agent_channel_recovered` events push to the hub HTTP 200 - (`Event pushed: agent_channel_…`). +## Live validation (felhom-pve guest 9201) — PASS +Induced a **born-down non-transient**: regenerated the agent leaf (channel breaks) then restarted the +controller so its FIRST channel observation was `pin_mismatch`: +``` +[channel] agent channel DOWN (unseeded->down:pin_mismatch): … TLS pin mismatch … +Event pushed: agent_channel_pin_mismatch (error) — … (HTTP 200) +``` +**`unseeded->down`** = the born-down case, and it **ALERTED** (~60 s, one cycle) — vs the pre-fix +silent seed. Recovered via R1 leaf restore: `[channel] agent channel recovered` + +`agent_channel_recovered`, leaf==pin MATCH, dashboard banner cleared. This is the capstone's missing +half — **detection at startup, not only on a live transition.** ## NOT changed -The agent; the agentapi pin/transport; `agentClient()` memoization; the disk gate. Detection/surfacing -only. - -## Forward note -Remaining self-health leg (backlog): the hub-side leaf-fp comparison (proactively catching an agent -re-key fleet-wide before any user hits it). +The swap, capability, or drive-gate logic; the debounce threshold; the agentapi pin/transport. Alerting +behaviour only.