diff --git a/CONTEXT.md b/CONTEXT.md index 852c4e6..9d00266 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -7,7 +7,23 @@ > > Ask Claude Code: "Please update CONTEXT.md with what we did today" -Last updated: 2026-06-29 (OPS: R1 restore of agent local-API leaf — controller↔agent service restored) +Last updated: 2026-06-29 (v0.90.0 — controller→agent channel health-check) + +> **2026-06-29 — v0.90.0: controller→agent channel health-check (self-health slice).** A ~60s +> scheduler job (`internal/channelhealth`) probes the local-API channel via the PRODUCTION memoized +> client (`Server.ProbeAgentChannel` → `s.agentClient()` + GET /storage — NOT a fresh client, per the +> spike: self-heals, mirrors the disk UI, no transport leak). Classifies failures (spike Q1 map: +> pin_mismatch/unauthorized/unreachable/timeout/misconfigured/construction_error/unknown), **debounces** +> transient reasons (refused/timeout need N≥2 consecutive so the ~1s agent-restart blip doesn't page; +> pin/401/DNS/construction alert first-obs), seeds the first observation silently, and on a transition +> emits an **English operator-only** event (`Notifier.NotifyAgentChannelDown/Recovered`) + a Hungarian +> dashboard banner (`AlertManager.SetAgentChannelAlert`). Closes the R1-incident gap (channel was only +> checked once at startup). **Required a hub change** (felhom-hub v0.20.0): the `agent_channel_*` event +> types were rejected by the hub's `allowedEventTypes` allowlist (HTTP 400) — controller-pushed events +> are gated, unlike the hub-generated host_* ones. Live-validated: transient restart → no alert; +> sustained stop → debounce(1/2) then DOWN transition + dashboard banner; start → recovered. No agent +> change. Self-health story now: agent watches its own capabilities (v0.44.0) + controller watches its +> link to the agent (this). Backlog: hub-side leaf-fp comparison. > **2026-06-29 — OPS (no code change): controller↔agent TLS leaf-pin mismatch RESOLVED via R1.** > The 2026-06-28 root→non-root agent migration regenerated the agent's local-API leaf diff --git a/REPORT.md b/REPORT.md index c0cbffe..7003111 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,63 +1,81 @@ -# REPORT — App-email gap closure: plaintext listener + split-From, v0.89.0 +# REPORT — Controller→agent channel health-check, v0.90.0 -**Task:** Close the two relay gaps from `FINDING-app-email-rollout-2026-06-29.md` so opportunistic-STARTTLS -clients (cal.com, nextcloud) can use the relay. Controller mechanism + two catalog mappings + live send-tests. -**Locked posture:** plaintext on the single-tenant app Docker bridge is accepted (no TLS/CA work). +**Task:** the next self-health slice — a periodic check that the controller↔agent local-API channel is +up, classifying failures and alerting the operator + dashboard on a state change. Closes the gap the R1 +pin-mismatch incident exposed (the channel was only probed once at startup and only logged). +Spike-proven: `felhom.eu/documentation/audits/SPIKE-controller-agent-channel-health-2026-06-29.md`. -## Baselines (verified) -- felhom-controller `main` @ `6692a2f`, **v0.88.0 → v0.89.0**. app-catalog touched. Hub untouched (N/A). +**Baseline:** controller `main` @ `77bccf1` (live 0.89.0) → **v0.90.0** @ `a277b18` (image +`…/felhom-controller:0.90.0`). Companion hub change: felhom-hub **v0.20.0** (allowlist; see below). -## Files changed (controller) -- `internal/mailrelay/server.go` — **Gap 1:** added a third listener `PlainNoTLSAddr` (default `:2526`), - plaintext, with `TLSConfig` left **nil** so go-smtp does NOT advertise STARTTLS. `:2525`/`:2465` unchanged. - `Start`/`Close`/addr-accessor updated for the third socket. -- `internal/stacks/metadata.go` — **Gap 2:** `SMTPMapping` gains `TLSMode` (`yaml:"tls_mode"`) + - `FromDomainVar` (`yaml:"from_domain_var"`). -- `internal/stacks/mailenv.go` — `smtpEnv` now picks the port via `shimPortForTLSMode` (`""`/`starttls`→2525; - `plaintext`→2526; `implicit-tls`→2465) instead of hardcoded 2525; split-From injection when `FromDomainVar` - is set (`FromVar=` + `FromDomainVar=`), else the existing `@`. -- `internal/config/config.go` — `mail_relay.plain_no_tls_listen` (default `:2526`). `cmd/controller/main.go` — - pass `PlainNoTLSAddr` to the lifecycle Options. +## Implementation -## Green gate (local) -`go build ./... && go vet ./... && go test ./...` — **PASS** (16 packages, 0 failures). -- `TestSMTPEnv_PortByTLSMode` — plaintext→2526, starttls/empty→2525, implicit-tls→2465. **Companion** - `TestShimPortForTLSMode` — plaintext and starttls must map to DIFFERENT ports (fails if the switch is dropped). -- `TestSMTPEnv_SplitFrom` — `from_domain_var` set → two keys (`MAIL_FROM_ADDRESS=nextcloud`+`MAIL_DOMAIN=felhom.eu`), - port 2526. **Companion** — a mapping without `from_domain_var` yields the single `vaultwarden@felhom.eu`. -- `TestServer_PlainNoTLSListener_NoSTARTTLS` — `:2526` server has `TLSConfig==nil` & a real EHLO shows it does - **not** advertise STARTTLS while `:2525` does. +- **`internal/channelhealth` (NEW):** a `Checker` over two seams — `Probe` (the channel call) and + `Sink` (dashboard + notify) — so it is decoupled + table-tested. Each run classifies into + `up | down:`. +- **Classifier (spike Q1 map, substring match):** -## Deployed v0.89.0 + listener -Built `:0.89.0` on 180, deployed to guest 9201 (golden/bootstrap). `docker ps`: `…:0.89.0 Up (healthy)`. Shim -started **three** listeners: `plaintext+STARTTLS on :2525`, `implicit-TLS on :2465`, and the new -`plaintext-only (no STARTTLS) listener on :2526`. + | reason | match | first-obs alert? | severity | event type | + |---|---|---|---|---| + | pin_mismatch | `TLS pin mismatch` | yes | error | `agent_channel_pin_mismatch` | + | unauthorized | `HTTP 401` | yes | error | `agent_channel_unauthorized` | + | misconfigured | `no such host` / `lookup ` | yes | error | `agent_channel_misconfigured` | + | construction_error | `agentClient()` build error (LATCHES) | yes | error | `agent_channel_construction_error` | + | unreachable | `connection refused` | **no (N≥2)** | warning | `agent_channel_unreachable` | + | timeout | `context deadline exceeded` / `i/o timeout` | **no (N≥2)** | warning | `agent_channel_timeout` | + | unknown | (fallthrough) | no (N≥2) | warning | `agent_channel_unknown` | + | recovered | down→up transition | — | info | `agent_channel_recovered` | -## Live send-tests -- **nextcloud (Symfony Mailer) — Scenario B PROVEN:** toggle ON injected `SMTP_HOST=felhom-controller`, - `SMTP_PORT=2526`, `SMTP_SECURE=` (empty), and the **split From** `MAIL_FROM_ADDRESS=nextcloud` + - `MAIL_DOMAIN=felhom.eu`. **§8 env-consumption:** nextcloud applied it on redeploy (`occ config:system:get` - shows `mail_smtphost=felhom-controller mail_smtpport=2526 mail_smtpsecure= mail_from_address=nextcloud - mail_domain=felhom.eu`) — the official image reads these via `getenv()` every boot, **no occ step needed**. A - real password-reset relayed: shim `relayed (from="nextcloud@felhom.eu" rcpts=1 bytes=20430 hub=200)`. The - Symfony client sent **plaintext to :2526 and never attempted STARTTLS** → delivered. (Gap 1 + Gap 2 proven - end-to-end with a real opportunistic-STARTTLS client.) -- **cal.com (Nodemailer) — Scenario A PROVEN:** toggle ON injected `EMAIL_SERVER_HOST=felhom-controller`, - `EMAIL_SERVER_PORT=2526`, `EMAIL_FROM=calcom@felhom.eu` (verified in-container). A real signup (HTTP 201 - "Created user") triggered the verification email: shim `relayed (from="calcom@felhom.eu" rcpts=1 bytes=10260 - hub=200)`. The Nodemailer client sent **plaintext to :2526 and never attempted STARTTLS** → delivered. - (calcom needed three pre-existing template fixes first — see Observations.) -- **No regression (Scenario C):** gitea redeployed under v0.89.0 still injects `SMTP_PORT=2525` + - `smtp+starttls`; a real gitea forgot-password relayed `(from="gitea@felhom.eu" hub=200)` via the unchanged - :2525 STARTTLS path. +- **Probe = the PRODUCTION memoized client** (`Server.ProbeAgentChannel` → `s.agentClient()` + + `Storage()`), **not** a fresh `agentapi.New` per probe (the spike's decisive call: self-heals, + reflects the disk UI exactly, no transport leak). A construction error (latching `sync.Once` build + failure) is surfaced distinctly via the `constructionErr` flag. +- **Debounce + state machine:** transient reasons need **N≥2** consecutive down-probes before a + transition (the ~1s agent-restart gap is suppressed); pin/401/DNS/construction transition on the + first down. First scheduler observation **seeds** state, no alert. A transition (up→down, down→up, + reason-change) fires once; steady-down does not re-fire. +- **Operator alert + dashboard:** `Notifier.NotifyAgentChannelDown/Recovered` push an **English, + operator-only** event (customer can't act on it; the type is not a customer toggle); the hub applies + its 1 h cooldown. `AlertManager.SetAgentChannelAlert` shows a short **Hungarian** banner whenever the + channel is down (state-based, idempotent — prepended in `GetAlerts`). **No customer email.** +- **Wiring:** `cmd/controller/main.go` registers a ~60 s `sched.Every("agent-channel-health", …)` job + (only when `local_api.endpoint` is set), after the webServer is built (it owns the memoized client). + `probeLocalAPI` stays at startup. A `channelSink` adapter bridges the package to notify+alerts. -## Observations -- **Three pre-existing cal.com template bugs** surfaced (never caught — the bad image pin meant calcom never - deployed): (1) image `v4.8.7` doesn't exist as a published image → `v4.6.9`; (2) missing `DATABASE_DIRECT_URL` - (Prisma `migrate deploy` fails → incomplete schema → 500s); (3) healthcheck path `/api/health` is 404 in - v4.x → `/api/auth/providers`. All fixed in the catalog. -- **immich (Gap 3) remains unwired by decision** — it has no SMTP env (admin-UI/DB or `IMMICH_CONFIG_FILE` - JSON); the next email piece is a config-file render+mount mechanism. -- **Other email-capable apps now wire trivially** as `tls_mode: plaintext` (ghost, outline, grafana, …) — no - per-app TLS determination needed under the accepted posture. -- No secrets in any committed file; the Resend key stays hub-side. +## Hub change (was "verify-only" — live test showed it WAS needed) + +The controller pushes `agent_channel_*` to `/api/v1/event`, but the hub's `allowedEventTypes` +allowlist rejected them (live: `Event push failed … HTTP 400`). `host_capability_*` didn't hit this +because those are hub-*generated*; `agent_channel_*` are controller-*pushed*. **felhom-hub v0.20.0** +adds the 8 types to the allowlist (operator-only; the dispatcher relays generically — no template +change). No agent change. + +## Tests (all green: `go build/vet/test ./...`) + +`internal/channelhealth`: classifier per-reason (pin/401/dns/construction), transitions (up→down once, +steady-down no duplicate, down→up recovered, reason-change re-alerts), first-obs seed (born-down shows +on dashboard, no notify), probe-seam-only. **Debounce red-proof** (`TestDebounce_TransientBlipSuppressed`): +one refused → **no** alert; two consecutive → exactly one — a no-debounce impl fails this. + +## Live validation (felhom-pve guest 9201) — deployed + verified + +- **Startup:** `local-api: channel up (agent 192.168.0.162:8443) — guest 9201, 4 mount(s)` + + `[scheduler] Registered periodic job: agent-channel-health (every 1m0s)`. +- **Transient (clean `systemctl restart felhom-agent`):** **no** channel-down alert, dashboard stayed + clean (the ~2s gap + 60s cadence + debounce). +- **Sustained (`systemctl stop felhom-agent`):** the live logs show the debounce + transition exactly: + `[channel] transient down (unreachable, 1/2) — suppressed` then 60s later + `[channel] agent channel DOWN (up→down:unreachable)`; the dashboard banner + *"A tárolókezelő ügynök nem elérhető."* appeared (verified in the served HTML). +- **Recovery (`systemctl start`):** `[channel] agent channel recovered (was down:unreachable)`. +- **Operator relay:** initially `HTTP 400` (hub allowlist) → fixed in hub v0.20.0 → re-validated: + the `agent_channel_unreachable` + `agent_channel_recovered` events push to the hub HTTP 200 + (`Event pushed: agent_channel_…`). + +## NOT changed +The agent; the agentapi pin/transport; `agentClient()` memoization; the disk gate. Detection/surfacing +only. + +## Forward note +Remaining self-health leg (backlog): the hub-side leaf-fp comparison (proactively catching an agent +re-key fleet-wide before any user hits it). diff --git a/controller/README.md b/controller/README.md index 2e1881a..fcf9960 100644 --- a/controller/README.md +++ b/controller/README.md @@ -737,6 +737,19 @@ not just those with HDD data. Non-HDD apps can configure destination, method, an > This replaced a per-call `agentapi.New(...)` that leaked one idle ESTABLISHED socket per call and > exhausted the ephemeral source-port range to the agent's `:8443` after ~5 days (EADDRNOTAVAIL). > Safe because `cfg.LocalAPI` is static per process (a config-apply triggers a graceful self-restart). +> - **(v0.90.0) Channel health-check — `internal/channelhealth`.** A ~60s scheduler job +> (`agent-channel-health`) probes the channel via `Server.ProbeAgentChannel` (the SAME memoized +> client + `GET /storage` — not a fresh client) and classifies failures into +> `up | down:` (pin_mismatch / unauthorized / unreachable / timeout / misconfigured / +> construction_error). Transient reasons (refused/timeout) are **debounced** (N≥2 consecutive) +> so a clean agent restart's ~1s blip doesn't page; pin/401/DNS/construction alert on the first +> down. On a transition it fires an English **operator-only** event +> (`Notifier.NotifyAgentChannelDown/Recovered`, hub 1h cooldown) and sets a Hungarian dashboard +> banner (`AlertManager.SetAgentChannelAlert`). The first observation seeds state silently. This is +> the controller half of the self-health story (the agent watches its own privileged capabilities; +> the controller watches its link to the agent). A construction error (`agentClient()` can't build +> — a latching config fault) is surfaced distinctly. Detection/surfacing only — it never touches the +> pin, transport, or gate. > - The **`StoragePath` registry** (`settings.go`: `AddStoragePath`/default/schedulable/label) is unchanged. > - **(v0.64.0) `AutoDiscoverStoragePaths` is now ADDITIVE** — it no longer bails when the registry is > non-empty; instead it registers only deployed-app paths missing from the registry. It never removes