docs: REPORT + CONTEXT + README for v0.90.0 channel health-check (live-validated, hub v0.20.0 allowlist)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ
This commit is contained in:
+17
-1
@@ -7,7 +7,23 @@
|
||||
>
|
||||
> Ask Claude Code: "Please update CONTEXT.md with what we did today"
|
||||
|
||||
Last updated: 2026-06-29 (OPS: R1 restore of agent local-API leaf — controller↔agent service restored)
|
||||
Last updated: 2026-06-29 (v0.90.0 — controller→agent channel health-check)
|
||||
|
||||
> **2026-06-29 — v0.90.0: controller→agent channel health-check (self-health slice).** A ~60s
|
||||
> scheduler job (`internal/channelhealth`) probes the local-API channel via the PRODUCTION memoized
|
||||
> client (`Server.ProbeAgentChannel` → `s.agentClient()` + GET /storage — NOT a fresh client, per the
|
||||
> spike: self-heals, mirrors the disk UI, no transport leak). Classifies failures (spike Q1 map:
|
||||
> pin_mismatch/unauthorized/unreachable/timeout/misconfigured/construction_error/unknown), **debounces**
|
||||
> transient reasons (refused/timeout need N≥2 consecutive so the ~1s agent-restart blip doesn't page;
|
||||
> pin/401/DNS/construction alert first-obs), seeds the first observation silently, and on a transition
|
||||
> emits an **English operator-only** event (`Notifier.NotifyAgentChannelDown/Recovered`) + a Hungarian
|
||||
> dashboard banner (`AlertManager.SetAgentChannelAlert`). Closes the R1-incident gap (channel was only
|
||||
> checked once at startup). **Required a hub change** (felhom-hub v0.20.0): the `agent_channel_*` event
|
||||
> types were rejected by the hub's `allowedEventTypes` allowlist (HTTP 400) — controller-pushed events
|
||||
> are gated, unlike the hub-generated host_* ones. Live-validated: transient restart → no alert;
|
||||
> sustained stop → debounce(1/2) then DOWN transition + dashboard banner; start → recovered. No agent
|
||||
> change. Self-health story now: agent watches its own capabilities (v0.44.0) + controller watches its
|
||||
> link to the agent (this). Backlog: hub-side leaf-fp comparison.
|
||||
|
||||
> **2026-06-29 — OPS (no code change): controller↔agent TLS leaf-pin mismatch RESOLVED via R1.**
|
||||
> The 2026-06-28 root→non-root agent migration regenerated the agent's local-API leaf
|
||||
|
||||
@@ -1,63 +1,81 @@
|
||||
# REPORT — App-email gap closure: plaintext listener + split-From, v0.89.0
|
||||
# REPORT — Controller→agent channel health-check, v0.90.0
|
||||
|
||||
**Task:** Close the two relay gaps from `FINDING-app-email-rollout-2026-06-29.md` so opportunistic-STARTTLS
|
||||
clients (cal.com, nextcloud) can use the relay. Controller mechanism + two catalog mappings + live send-tests.
|
||||
**Locked posture:** plaintext on the single-tenant app Docker bridge is accepted (no TLS/CA work).
|
||||
**Task:** the next self-health slice — a periodic check that the controller↔agent local-API channel is
|
||||
up, classifying failures and alerting the operator + dashboard on a state change. Closes the gap the R1
|
||||
pin-mismatch incident exposed (the channel was only probed once at startup and only logged).
|
||||
Spike-proven: `felhom.eu/documentation/audits/SPIKE-controller-agent-channel-health-2026-06-29.md`.
|
||||
|
||||
## Baselines (verified)
|
||||
- felhom-controller `main` @ `6692a2f`, **v0.88.0 → v0.89.0**. app-catalog touched. Hub untouched (N/A).
|
||||
**Baseline:** controller `main` @ `77bccf1` (live 0.89.0) → **v0.90.0** @ `a277b18` (image
|
||||
`…/felhom-controller:0.90.0`). Companion hub change: felhom-hub **v0.20.0** (allowlist; see below).
|
||||
|
||||
## Files changed (controller)
|
||||
- `internal/mailrelay/server.go` — **Gap 1:** added a third listener `PlainNoTLSAddr` (default `:2526`),
|
||||
plaintext, with `TLSConfig` left **nil** so go-smtp does NOT advertise STARTTLS. `:2525`/`:2465` unchanged.
|
||||
`Start`/`Close`/addr-accessor updated for the third socket.
|
||||
- `internal/stacks/metadata.go` — **Gap 2:** `SMTPMapping` gains `TLSMode` (`yaml:"tls_mode"`) +
|
||||
`FromDomainVar` (`yaml:"from_domain_var"`).
|
||||
- `internal/stacks/mailenv.go` — `smtpEnv` now picks the port via `shimPortForTLSMode` (`""`/`starttls`→2525;
|
||||
`plaintext`→2526; `implicit-tls`→2465) instead of hardcoded 2525; split-From injection when `FromDomainVar`
|
||||
is set (`FromVar=<local>` + `FromDomainVar=<domain>`), else the existing `<local>@<domain>`.
|
||||
- `internal/config/config.go` — `mail_relay.plain_no_tls_listen` (default `:2526`). `cmd/controller/main.go` —
|
||||
pass `PlainNoTLSAddr` to the lifecycle Options.
|
||||
## Implementation
|
||||
|
||||
## Green gate (local)
|
||||
`go build ./... && go vet ./... && go test ./...` — **PASS** (16 packages, 0 failures).
|
||||
- `TestSMTPEnv_PortByTLSMode` — plaintext→2526, starttls/empty→2525, implicit-tls→2465. **Companion**
|
||||
`TestShimPortForTLSMode` — plaintext and starttls must map to DIFFERENT ports (fails if the switch is dropped).
|
||||
- `TestSMTPEnv_SplitFrom` — `from_domain_var` set → two keys (`MAIL_FROM_ADDRESS=nextcloud`+`MAIL_DOMAIN=felhom.eu`),
|
||||
port 2526. **Companion** — a mapping without `from_domain_var` yields the single `vaultwarden@felhom.eu`.
|
||||
- `TestServer_PlainNoTLSListener_NoSTARTTLS` — `:2526` server has `TLSConfig==nil` & a real EHLO shows it does
|
||||
**not** advertise STARTTLS while `:2525` does.
|
||||
- **`internal/channelhealth` (NEW):** a `Checker` over two seams — `Probe` (the channel call) and
|
||||
`Sink` (dashboard + notify) — so it is decoupled + table-tested. Each run classifies into
|
||||
`up | down:<reason>`.
|
||||
- **Classifier (spike Q1 map, substring match):**
|
||||
|
||||
## Deployed v0.89.0 + listener
|
||||
Built `:0.89.0` on 180, deployed to guest 9201 (golden/bootstrap). `docker ps`: `…:0.89.0 Up (healthy)`. Shim
|
||||
started **three** listeners: `plaintext+STARTTLS on :2525`, `implicit-TLS on :2465`, and the new
|
||||
`plaintext-only (no STARTTLS) listener on :2526`.
|
||||
| reason | match | first-obs alert? | severity | event type |
|
||||
|---|---|---|---|---|
|
||||
| pin_mismatch | `TLS pin mismatch` | yes | error | `agent_channel_pin_mismatch` |
|
||||
| unauthorized | `HTTP 401` | yes | error | `agent_channel_unauthorized` |
|
||||
| misconfigured | `no such host` / `lookup ` | yes | error | `agent_channel_misconfigured` |
|
||||
| construction_error | `agentClient()` build error (LATCHES) | yes | error | `agent_channel_construction_error` |
|
||||
| unreachable | `connection refused` | **no (N≥2)** | warning | `agent_channel_unreachable` |
|
||||
| timeout | `context deadline exceeded` / `i/o timeout` | **no (N≥2)** | warning | `agent_channel_timeout` |
|
||||
| unknown | (fallthrough) | no (N≥2) | warning | `agent_channel_unknown` |
|
||||
| recovered | down→up transition | — | info | `agent_channel_recovered` |
|
||||
|
||||
## Live send-tests
|
||||
- **nextcloud (Symfony Mailer) — Scenario B PROVEN:** toggle ON injected `SMTP_HOST=felhom-controller`,
|
||||
`SMTP_PORT=2526`, `SMTP_SECURE=` (empty), and the **split From** `MAIL_FROM_ADDRESS=nextcloud` +
|
||||
`MAIL_DOMAIN=felhom.eu`. **§8 env-consumption:** nextcloud applied it on redeploy (`occ config:system:get`
|
||||
shows `mail_smtphost=felhom-controller mail_smtpport=2526 mail_smtpsecure= mail_from_address=nextcloud
|
||||
mail_domain=felhom.eu`) — the official image reads these via `getenv()` every boot, **no occ step needed**. A
|
||||
real password-reset relayed: shim `relayed (from="nextcloud@felhom.eu" rcpts=1 bytes=20430 hub=200)`. The
|
||||
Symfony client sent **plaintext to :2526 and never attempted STARTTLS** → delivered. (Gap 1 + Gap 2 proven
|
||||
end-to-end with a real opportunistic-STARTTLS client.)
|
||||
- **cal.com (Nodemailer) — Scenario A PROVEN:** toggle ON injected `EMAIL_SERVER_HOST=felhom-controller`,
|
||||
`EMAIL_SERVER_PORT=2526`, `EMAIL_FROM=calcom@felhom.eu` (verified in-container). A real signup (HTTP 201
|
||||
"Created user") triggered the verification email: shim `relayed (from="calcom@felhom.eu" rcpts=1 bytes=10260
|
||||
hub=200)`. The Nodemailer client sent **plaintext to :2526 and never attempted STARTTLS** → delivered.
|
||||
(calcom needed three pre-existing template fixes first — see Observations.)
|
||||
- **No regression (Scenario C):** gitea redeployed under v0.89.0 still injects `SMTP_PORT=2525` +
|
||||
`smtp+starttls`; a real gitea forgot-password relayed `(from="gitea@felhom.eu" hub=200)` via the unchanged
|
||||
:2525 STARTTLS path.
|
||||
- **Probe = the PRODUCTION memoized client** (`Server.ProbeAgentChannel` → `s.agentClient()` +
|
||||
`Storage()`), **not** a fresh `agentapi.New` per probe (the spike's decisive call: self-heals,
|
||||
reflects the disk UI exactly, no transport leak). A construction error (latching `sync.Once` build
|
||||
failure) is surfaced distinctly via the `constructionErr` flag.
|
||||
- **Debounce + state machine:** transient reasons need **N≥2** consecutive down-probes before a
|
||||
transition (the ~1s agent-restart gap is suppressed); pin/401/DNS/construction transition on the
|
||||
first down. First scheduler observation **seeds** state, no alert. A transition (up→down, down→up,
|
||||
reason-change) fires once; steady-down does not re-fire.
|
||||
- **Operator alert + dashboard:** `Notifier.NotifyAgentChannelDown/Recovered` push an **English,
|
||||
operator-only** event (customer can't act on it; the type is not a customer toggle); the hub applies
|
||||
its 1 h cooldown. `AlertManager.SetAgentChannelAlert` shows a short **Hungarian** banner whenever the
|
||||
channel is down (state-based, idempotent — prepended in `GetAlerts`). **No customer email.**
|
||||
- **Wiring:** `cmd/controller/main.go` registers a ~60 s `sched.Every("agent-channel-health", …)` job
|
||||
(only when `local_api.endpoint` is set), after the webServer is built (it owns the memoized client).
|
||||
`probeLocalAPI` stays at startup. A `channelSink` adapter bridges the package to notify+alerts.
|
||||
|
||||
## Observations
|
||||
- **Three pre-existing cal.com template bugs** surfaced (never caught — the bad image pin meant calcom never
|
||||
deployed): (1) image `v4.8.7` doesn't exist as a published image → `v4.6.9`; (2) missing `DATABASE_DIRECT_URL`
|
||||
(Prisma `migrate deploy` fails → incomplete schema → 500s); (3) healthcheck path `/api/health` is 404 in
|
||||
v4.x → `/api/auth/providers`. All fixed in the catalog.
|
||||
- **immich (Gap 3) remains unwired by decision** — it has no SMTP env (admin-UI/DB or `IMMICH_CONFIG_FILE`
|
||||
JSON); the next email piece is a config-file render+mount mechanism.
|
||||
- **Other email-capable apps now wire trivially** as `tls_mode: plaintext` (ghost, outline, grafana, …) — no
|
||||
per-app TLS determination needed under the accepted posture.
|
||||
- No secrets in any committed file; the Resend key stays hub-side.
|
||||
## Hub change (was "verify-only" — live test showed it WAS needed)
|
||||
|
||||
The controller pushes `agent_channel_*` to `/api/v1/event`, but the hub's `allowedEventTypes`
|
||||
allowlist rejected them (live: `Event push failed … HTTP 400`). `host_capability_*` didn't hit this
|
||||
because those are hub-*generated*; `agent_channel_*` are controller-*pushed*. **felhom-hub v0.20.0**
|
||||
adds the 8 types to the allowlist (operator-only; the dispatcher relays generically — no template
|
||||
change). No agent change.
|
||||
|
||||
## Tests (all green: `go build/vet/test ./...`)
|
||||
|
||||
`internal/channelhealth`: classifier per-reason (pin/401/dns/construction), transitions (up→down once,
|
||||
steady-down no duplicate, down→up recovered, reason-change re-alerts), first-obs seed (born-down shows
|
||||
on dashboard, no notify), probe-seam-only. **Debounce red-proof** (`TestDebounce_TransientBlipSuppressed`):
|
||||
one refused → **no** alert; two consecutive → exactly one — a no-debounce impl fails this.
|
||||
|
||||
## Live validation (felhom-pve guest 9201) — deployed + verified
|
||||
|
||||
- **Startup:** `local-api: channel up (agent 192.168.0.162:8443) — guest 9201, 4 mount(s)` +
|
||||
`[scheduler] Registered periodic job: agent-channel-health (every 1m0s)`.
|
||||
- **Transient (clean `systemctl restart felhom-agent`):** **no** channel-down alert, dashboard stayed
|
||||
clean (the ~2s gap + 60s cadence + debounce).
|
||||
- **Sustained (`systemctl stop felhom-agent`):** the live logs show the debounce + transition exactly:
|
||||
`[channel] transient down (unreachable, 1/2) — suppressed` then 60s later
|
||||
`[channel] agent channel DOWN (up→down:unreachable)`; the dashboard banner
|
||||
*"A tárolókezelő ügynök nem elérhető."* appeared (verified in the served HTML).
|
||||
- **Recovery (`systemctl start`):** `[channel] agent channel recovered (was down:unreachable)`.
|
||||
- **Operator relay:** initially `HTTP 400` (hub allowlist) → fixed in hub v0.20.0 → re-validated:
|
||||
the `agent_channel_unreachable` + `agent_channel_recovered` events push to the hub HTTP 200
|
||||
(`Event pushed: agent_channel_…`).
|
||||
|
||||
## NOT changed
|
||||
The agent; the agentapi pin/transport; `agentClient()` memoization; the disk gate. Detection/surfacing
|
||||
only.
|
||||
|
||||
## Forward note
|
||||
Remaining self-health leg (backlog): the hub-side leaf-fp comparison (proactively catching an agent
|
||||
re-key fleet-wide before any user hits it).
|
||||
|
||||
@@ -737,6 +737,19 @@ not just those with HDD data. Non-HDD apps can configure destination, method, an
|
||||
> This replaced a per-call `agentapi.New(...)` that leaked one idle ESTABLISHED socket per call and
|
||||
> exhausted the ephemeral source-port range to the agent's `:8443` after ~5 days (EADDRNOTAVAIL).
|
||||
> Safe because `cfg.LocalAPI` is static per process (a config-apply triggers a graceful self-restart).
|
||||
> - **(v0.90.0) Channel health-check — `internal/channelhealth`.** A ~60s scheduler job
|
||||
> (`agent-channel-health`) probes the channel via `Server.ProbeAgentChannel` (the SAME memoized
|
||||
> client + `GET /storage` — not a fresh client) and classifies failures into
|
||||
> `up | down:<reason>` (pin_mismatch / unauthorized / unreachable / timeout / misconfigured /
|
||||
> construction_error). Transient reasons (refused/timeout) are **debounced** (N≥2 consecutive)
|
||||
> so a clean agent restart's ~1s blip doesn't page; pin/401/DNS/construction alert on the first
|
||||
> down. On a transition it fires an English **operator-only** event
|
||||
> (`Notifier.NotifyAgentChannelDown/Recovered`, hub 1h cooldown) and sets a Hungarian dashboard
|
||||
> banner (`AlertManager.SetAgentChannelAlert`). The first observation seeds state silently. This is
|
||||
> the controller half of the self-health story (the agent watches its own privileged capabilities;
|
||||
> the controller watches its link to the agent). A construction error (`agentClient()` can't build
|
||||
> — a latching config fault) is surfaced distinctly. Detection/surfacing only — it never touches the
|
||||
> pin, transport, or gate.
|
||||
> - The **`StoragePath` registry** (`settings.go`: `AddStoragePath`/default/schedulable/label) is unchanged.
|
||||
> - **(v0.64.0) `AutoDiscoverStoragePaths` is now ADDITIVE** — it no longer bails when the registry is
|
||||
> non-empty; instead it registers only deployed-app paths missing from the registry. It never removes
|
||||
|
||||
Reference in New Issue
Block a user