docs: REPORT for v0.91.0 F2 born-down alert (live-validated)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ
This commit is contained in:
@@ -1,81 +1,45 @@
|
||||
# REPORT — Controller→agent channel health-check, v0.90.0
|
||||
# REPORT — F2: alert on born/persistent-down channel, v0.91.0
|
||||
|
||||
**Task:** the next self-health slice — a periodic check that the controller↔agent local-API channel is
|
||||
up, classifying failures and alerting the operator + dashboard on a state change. Closes the gap the R1
|
||||
pin-mismatch incident exposed (the channel was only probed once at startup and only logged).
|
||||
Spike-proven: `felhom.eu/documentation/audits/SPIKE-controller-agent-channel-health-2026-06-29.md`.
|
||||
**Task:** close F2 (the transition-only alerting gap the full-stack testrun surfaced) — a channel
|
||||
failure present at **startup/reseed** was dashboard-only, no operator email, forever. Part of the
|
||||
F2 (Part A) + prevention (Part B) slice; this repo is **Part A (controller)**.
|
||||
|
||||
**Baseline:** controller `main` @ `77bccf1` (live 0.89.0) → **v0.90.0** @ `a277b18` (image
|
||||
`…/felhom-controller:0.90.0`). Companion hub change: felhom-hub **v0.20.0** (allowlist; see below).
|
||||
**Baseline:** controller `main` @ `77bccf1` (live 0.90.0) → **v0.91.0** @ `b2ad871` (image
|
||||
`…/felhom-controller:0.91.0`). Companions: hub **v0.21.0** (F2 hub checkers), agent **v0.46.0** (B.1).
|
||||
|
||||
## Implementation
|
||||
## The gap + fix (`internal/channelhealth/checker.go`)
|
||||
|
||||
- **`internal/channelhealth` (NEW):** a `Checker` over two seams — `Probe` (the channel call) and
|
||||
`Sink` (dashboard + notify) — so it is decoupled + table-tested. Each run classifies into
|
||||
`up | down:<reason>`.
|
||||
- **Classifier (spike Q1 map, substring match):**
|
||||
Pre-fix `Check()`: a confirmed (past-debounce) down with `prev==""` hit `// first observation seeds,
|
||||
no alert`, then `prev==newState` every cycle → **never alerted**. A controller that *boots* into a
|
||||
broken channel (e.g. right after a leaf regen + controller restart) was dashboard-only.
|
||||
|
||||
| reason | match | first-obs alert? | severity | event type |
|
||||
|---|---|---|---|---|
|
||||
| pin_mismatch | `TLS pin mismatch` | yes | error | `agent_channel_pin_mismatch` |
|
||||
| unauthorized | `HTTP 401` | yes | error | `agent_channel_unauthorized` |
|
||||
| misconfigured | `no such host` / `lookup ` | yes | error | `agent_channel_misconfigured` |
|
||||
| construction_error | `agentClient()` build error (LATCHES) | yes | error | `agent_channel_construction_error` |
|
||||
| unreachable | `connection refused` | **no (N≥2)** | warning | `agent_channel_unreachable` |
|
||||
| timeout | `context deadline exceeded` / `i/o timeout` | **no (N≥2)** | warning | `agent_channel_timeout` |
|
||||
| unknown | (fallthrough) | no (N≥2) | warning | `agent_channel_unknown` |
|
||||
| recovered | down→up transition | — | info | `agent_channel_recovered` |
|
||||
|
||||
- **Probe = the PRODUCTION memoized client** (`Server.ProbeAgentChannel` → `s.agentClient()` +
|
||||
`Storage()`), **not** a fresh `agentapi.New` per probe (the spike's decisive call: self-heals,
|
||||
reflects the disk UI exactly, no transport leak). A construction error (latching `sync.Once` build
|
||||
failure) is surfaced distinctly via the `constructionErr` flag.
|
||||
- **Debounce + state machine:** transient reasons need **N≥2** consecutive down-probes before a
|
||||
transition (the ~1s agent-restart gap is suppressed); pin/401/DNS/construction transition on the
|
||||
first down. First scheduler observation **seeds** state, no alert. A transition (up→down, down→up,
|
||||
reason-change) fires once; steady-down does not re-fire.
|
||||
- **Operator alert + dashboard:** `Notifier.NotifyAgentChannelDown/Recovered` push an **English,
|
||||
operator-only** event (customer can't act on it; the type is not a customer toggle); the hub applies
|
||||
its 1 h cooldown. `AlertManager.SetAgentChannelAlert` shows a short **Hungarian** banner whenever the
|
||||
channel is down (state-based, idempotent — prepended in `GetAlerts`). **No customer email.**
|
||||
- **Wiring:** `cmd/controller/main.go` registers a ~60 s `sched.Every("agent-channel-health", …)` job
|
||||
(only when `local_api.endpoint` is set), after the webServer is built (it owns the memoized client).
|
||||
`probeLocalAPI` stays at startup. A `channelSink` adapter bridges the package to notify+alerts.
|
||||
|
||||
## Hub change (was "verify-only" — live test showed it WAS needed)
|
||||
|
||||
The controller pushes `agent_channel_*` to `/api/v1/event`, but the hub's `allowedEventTypes`
|
||||
allowlist rejected them (live: `Event push failed … HTTP 400`). `host_capability_*` didn't hit this
|
||||
because those are hub-*generated*; `agent_channel_*` are controller-*pushed*. **felhom-hub v0.20.0**
|
||||
adds the 8 types to the allowlist (operator-only; the dispatcher relays generically — no template
|
||||
change). No agent change.
|
||||
**Fix — an `alerted` flag** (have we emitted for the CURRENT down-spell?):
|
||||
- **UP:** `alerted=false` (re-arm); if `prev` was `down:*` → `NotifyRecovered`.
|
||||
- **DOWN, transient + `consecutiveDown < N`:** unchanged debounce (suppress; unseeded → assume up).
|
||||
- **DOWN, confirmed:** dashboard down. If `prev` was up/unseeded **or** the reason changed → new spell
|
||||
→ `alerted=false`. Then `if !alerted → NotifyDown; alerted=true`. Removed the `prev==""`
|
||||
silent-seed-for-down branch.
|
||||
- Net: non-transient **born-down alerts on cycle 1**; transient born-down still needs N≥2 (cold-boot
|
||||
agent-not-yet-up race); **healthy** first-obs still silent; reason-change re-alerts; recovery re-arms.
|
||||
|
||||
## Tests (all green: `go build/vet/test ./...`)
|
||||
F2 **born-down red-proof** (`TestF2_BornDownNonTransient_AlertsOnce`: one alert on cycle 1) + companion
|
||||
(`…OldSeedSilentLogicWouldNotAlert`: the old `prev==""` path would not have alerted); born-down
|
||||
transient still debounced; recovery re-arms the spell; healthy first-obs silent. The existing
|
||||
debounce/transition/classify tests still pass.
|
||||
|
||||
`internal/channelhealth`: classifier per-reason (pin/401/dns/construction), transitions (up→down once,
|
||||
steady-down no duplicate, down→up recovered, reason-change re-alerts), first-obs seed (born-down shows
|
||||
on dashboard, no notify), probe-seam-only. **Debounce red-proof** (`TestDebounce_TransientBlipSuppressed`):
|
||||
one refused → **no** alert; two consecutive → exactly one — a no-debounce impl fails this.
|
||||
|
||||
## Live validation (felhom-pve guest 9201) — deployed + verified
|
||||
|
||||
- **Startup:** `local-api: channel up (agent 192.168.0.162:8443) — guest 9201, 4 mount(s)` +
|
||||
`[scheduler] Registered periodic job: agent-channel-health (every 1m0s)`.
|
||||
- **Transient (clean `systemctl restart felhom-agent`):** **no** channel-down alert, dashboard stayed
|
||||
clean (the ~2s gap + 60s cadence + debounce).
|
||||
- **Sustained (`systemctl stop felhom-agent`):** the live logs show the debounce + transition exactly:
|
||||
`[channel] transient down (unreachable, 1/2) — suppressed` then 60s later
|
||||
`[channel] agent channel DOWN (up→down:unreachable)`; the dashboard banner
|
||||
*"A tárolókezelő ügynök nem elérhető."* appeared (verified in the served HTML).
|
||||
- **Recovery (`systemctl start`):** `[channel] agent channel recovered (was down:unreachable)`.
|
||||
- **Operator relay:** initially `HTTP 400` (hub allowlist) → fixed in hub v0.20.0 → re-validated:
|
||||
the `agent_channel_unreachable` + `agent_channel_recovered` events push to the hub HTTP 200
|
||||
(`Event pushed: agent_channel_…`).
|
||||
## Live validation (felhom-pve guest 9201) — PASS
|
||||
Induced a **born-down non-transient**: regenerated the agent leaf (channel breaks) then restarted the
|
||||
controller so its FIRST channel observation was `pin_mismatch`:
|
||||
```
|
||||
[channel] agent channel DOWN (unseeded->down:pin_mismatch): … TLS pin mismatch …
|
||||
Event pushed: agent_channel_pin_mismatch (error) — … (HTTP 200)
|
||||
```
|
||||
**`unseeded->down`** = the born-down case, and it **ALERTED** (~60 s, one cycle) — vs the pre-fix
|
||||
silent seed. Recovered via R1 leaf restore: `[channel] agent channel recovered` +
|
||||
`agent_channel_recovered`, leaf==pin MATCH, dashboard banner cleared. This is the capstone's missing
|
||||
half — **detection at startup, not only on a live transition.**
|
||||
|
||||
## NOT changed
|
||||
The agent; the agentapi pin/transport; `agentClient()` memoization; the disk gate. Detection/surfacing
|
||||
only.
|
||||
|
||||
## Forward note
|
||||
Remaining self-health leg (backlog): the hub-side leaf-fp comparison (proactively catching an agent
|
||||
re-key fleet-wide before any user hits it).
|
||||
The swap, capability, or drive-gate logic; the debounce threshold; the agentapi pin/transport. Alerting
|
||||
behaviour only.
|
||||
|
||||
Reference in New Issue
Block a user