From f10d74f99358b5ec2b02301775bccbf280b3b81c Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 29 Jun 2026 13:56:51 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20REPORT=20=E2=80=94=20v0.89.0=20gap=20cl?= =?UTF-8?q?osure,=20calcom+nextcloud=20send-tested=20live?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) --- REPORT.md | 150 ++++++++++++++++++++---------------------------------- 1 file changed, 54 insertions(+), 96 deletions(-) diff --git a/REPORT.md b/REPORT.md index 328bf4d..c0cbffe 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,105 +1,63 @@ -# REPORT — App-email SMTP relay (controller side), v0.88.0 +# REPORT — App-email gap closure: plaintext listener + split-From, v0.89.0 -**Task:** `TASK` — SMTP app-relay (apps → in-controller shim → hub → Resend). Implementation class. -**Architecture (operator-confirmed):** **Shape 1** — the SMTP shim runs **in-process inside felhom-controller**, -reusing the controller's existing hub client. (The alternative, a separate `felhom-smtp-shim` container, is the -documented v2 path.) +**Task:** Close the two relay gaps from `FINDING-app-email-rollout-2026-06-29.md` so opportunistic-STARTTLS +clients (cal.com, nextcloud) can use the relay. Controller mechanism + two catalog mappings + live send-tests. +**Locked posture:** plaintext on the single-tenant app Docker bridge is accepted (no TLS/CA work). -## Confirmed baselines (verified against live repo) -| Repo | `main` @ start | Live version | Target | -|------|----------------|--------------|--------| -| felhom-controller | `7cddb88` | v0.87.0 (CHANGELOG top v0.86.0; 0.87.0 was a version-only build) | **v0.88.0** | -| felhom.eu (hub) | `4b97855` | **v0.17.0** (the Resend-key rotation already shipped 2026-06-29) | **v0.18.0** | -| app-catalog | `5d42ca1` | unversioned | — | +## Baselines (verified) +- felhom-controller `main` @ `6692a2f`, **v0.88.0 → v0.89.0**. app-catalog touched. Hub untouched (N/A). -> **Deviation from the prompt:** the prompt assumed the hub was at v0.16.0 → v0.17.0, but the prerequisite -> Resend-key-rotation task already shipped as **v0.17.0** earlier today. So the hub leg targets **v0.18.0**. -> The prerequisite is satisfied (key out-of-band in `Secret/resend-api`, hub reads `RESEND_API_KEY` env). - -## Files created / modified (controller) -**Created** -- `internal/mailrelay/policy.go` — From-header domain allowlist (validate-and-reject; fail-closed on parse error). -- `internal/mailrelay/forward.go` — single-shot Bearer POST of raw MIME to hub `/api/v1/mail`. -- `internal/mailrelay/server.go` — go-smtp server (two listeners, self-signed cert, accept-any AUTH incl. a - ~15-line LOGIN sasl server), `Data` = read raw → From-policy → forward → map hub status to SMTP reply. -- `internal/mailrelay/lifecycle.go` — runtime start/stop so the global toggle applies without a restart. -- `internal/mailrelay/mailrelay_test.go`, `internal/stacks/mailenv.go`, `internal/stacks/mailenv_test.go`. - -**Modified** -- `internal/config/config.go` — `MailRelayConfig` (listeners, shim host, From allowlist, kill-switch) + defaults. -- `internal/settings/settings.go` — global `AppEmail{Enabled,FromName}` toggle + accessors. -- `internal/stacks/metadata.go` — `SMTPMapping` block parse + `HasSMTPMapping`. -- `internal/stacks/deploy.go` — `AppConfig.EmailEnabled` (per-app toggle, persisted) + propagate in `SaveAppConfig`. -- `internal/stacks/manager.go` — `stackEnv` injects the relay env (when global+per-app on, app has a mapping). -- `internal/web/{server.go,handlers.go,tier2_config_handler.go}` — global toggle handler + route, per-app toggle - handler + route, page data. -- `internal/web/templates/{settings.html,deploy.html}` — Hungarian "Alkalmazás-email" card + per-app toggle. -- `cmd/controller/main.go` — wire the shim lifecycle (gated on hub configured + kill-switch), apply boot-time - toggle, close on shutdown, hand to the web server. -- `go.mod`/`go.sum` — `github.com/emersion/go-smtp` v0.24.0 + `go-sasl`. +## Files changed (controller) +- `internal/mailrelay/server.go` — **Gap 1:** added a third listener `PlainNoTLSAddr` (default `:2526`), + plaintext, with `TLSConfig` left **nil** so go-smtp does NOT advertise STARTTLS. `:2525`/`:2465` unchanged. + `Start`/`Close`/addr-accessor updated for the third socket. +- `internal/stacks/metadata.go` — **Gap 2:** `SMTPMapping` gains `TLSMode` (`yaml:"tls_mode"`) + + `FromDomainVar` (`yaml:"from_domain_var"`). +- `internal/stacks/mailenv.go` — `smtpEnv` now picks the port via `shimPortForTLSMode` (`""`/`starttls`→2525; + `plaintext`→2526; `implicit-tls`→2465) instead of hardcoded 2525; split-From injection when `FromDomainVar` + is set (`FromVar=` + `FromDomainVar=`), else the existing `@`. +- `internal/config/config.go` — `mail_relay.plain_no_tls_listen` (default `:2526`). `cmd/controller/main.go` — + pass `PlainNoTLSAddr` to the lifecycle Options. ## Green gate (local) -`go build ./... && go vet ./... && go test ./...` — **PASS** (16 packages ok, 0 failures). +`go build ./... && go vet ./... && go test ./...` — **PASS** (16 packages, 0 failures). +- `TestSMTPEnv_PortByTLSMode` — plaintext→2526, starttls/empty→2525, implicit-tls→2465. **Companion** + `TestShimPortForTLSMode` — plaintext and starttls must map to DIFFERENT ports (fails if the switch is dropped). +- `TestSMTPEnv_SplitFrom` — `from_domain_var` set → two keys (`MAIL_FROM_ADDRESS=nextcloud`+`MAIL_DOMAIN=felhom.eu`), + port 2526. **Companion** — a mapping without `from_domain_var` yields the single `vaultwarden@felhom.eu`. +- `TestServer_PlainNoTLSListener_NoSTARTTLS` — `:2526` server has `TLSConfig==nil` & a real EHLO shows it does + **not** advertise STARTTLS while `:2525` does. -### Tests & §10 companion red-proofs -- **Passthrough not parse (§7 A / §10):** `TestData_HappyPath_ForwardsRawBytesUnchanged` asserts the forwarder - receives the **byte-equal** input (no parse/re-encode). PASS. -- **From-reject before forward (§7 B / §10):** `TestData_FromOutsideAllowlist_Rejected_NoForward` — 550 + - **zero** forwarder calls. Companion `…CompanionProof_GateRemovedLetsItThrough` — allowing the domain lets the - same message reach the forwarder once (proves the gate is load-bearing). Both PASS. -- **Single-shot on hub failure (§7 D / §10):** `TestData_HubUnreachable_SingleShot_Transient` — transient 4xx + - **exactly one** attempt (no retry loop). PASS. -- **Status mapping (§6):** `TestMapStatusToSMTP` 200→nil, 429/4xx→451, 5xx→554. PASS. -- **Toggles gate injection (§7 E / §8):** stacks `TestSMTPEnv_*` — both-on injects the full mapped env; - per-app-off / global-off / no-mapping inject nothing; Mealie-style mapping; household From-name override. PASS. -- **Real socket (Q1/Q2):** `TestServer_EndToEnd_STARTTLS` drives a real `net/smtp` STARTTLS send through the - listener. PASS. Plus `TestLifecycle_StartStopIdempotent`. +## Deployed v0.89.0 + listener +Built `:0.89.0` on 180, deployed to guest 9201 (golden/bootstrap). `docker ps`: `…:0.89.0 Up (healthy)`. Shim +started **three** listeners: `plaintext+STARTTLS on :2525`, `implicit-TLS on :2465`, and the new +`plaintext-only (no STARTTLS) listener on :2526`. -Test count: **+~17** controller test functions (mailrelay 9, stacks/mailenv 8); suite remains green. - -## Deployment & live validation — DONE (2026-06-29) -- **Built** `felhom-controller:0.88.0` on 192.168.0.180; **deployed** to guest 9201 via the golden/bootstrap - mechanism (`docker pull` → `/etc/felhom-controller-image` → restart bootstrap). `docker ps`: `…:0.88.0 Up - (healthy)`. The controller container is **already on `traefik-public`** (and bridge), so apps resolve the - shim by `felhom-controller` — no network change needed. -- **Global toggle via the REAL endpoint:** `POST /settings/app-email` (in-guest, the exact form the UI submits, - with session cookie + `_csrf`) → HTTP 200; controller logged `[mailrelay] app-email shim ON - (host=felhom-controller plain=[::]:2525 tls=[::]:2465)` and persisted `app_email.enabled=true` in - settings.json. -- **End-to-end happy path (real pipeline, real Resend send):** a raw MIME message with an **inline CID image**, - From `vaultwarden@felhom.eu`, sent through the running shim from a throwaway container on `traefik-public` - (`app → shim → hub → Resend`). Result: `MAIL 250 / RCPT 250 / DATA 250 "OK: queued"`. Shim log: - `message relayed (from="vaultwarden@felhom.eu" rcpts=1 bytes=789 hub=200)`; hub log: - `/api/v1/mail: relayed for demo-felhom`. The DATA-250 is returned only after the hub→Resend leg accepted - (queued) the message. Recipient = the operator's own address; **inbox arrival is the operator's final - confirmation** (Resend returned 250/queued for the verified `felhom.eu` domain, which the spike established - delivers with inline image intact). The Resend key never touched the guest and is in no committed file. -- **Security gate (live):** the same send From `evil@notfelhom.example` → `DATA 550 "5.7.1 sender address not - permitted"` and **no `/api/v1/mail` call reached the hub** (rejected before forward — scenario B confirmed live). -- **Method note:** the relay pipeline (the new, previously-unproven code) is validated against the live Resend - account. The catalog `smtp_mapping` → per-app env injection is unit-tested (`TestSMTPEnv_*` incl. the - Vaultwarden mapping); deploying the Vaultwarden app and clicking its admin "send test email" would - additionally exercise that app's own SMTP client (already spike-proven) — recommended as the final manual - smoke once the catalog sync has the new templates. - -- **Inline CID image — confirmed rendering in the real client (Gmail):** a well-formed - `multipart/related; type="text/html"` message with a CID-referenced inline PNG, sent through the same - pipeline, **renders inline in the Gmail inbox** (Resend "Delivered", From `vaultwarden@felhom.eu`, - signed-by `felhom.eu`, TLS). NOTE: Resend's dashboard *preview pane* unreliably shows CID images as a broken - `attachment-0` even when the delivered mail renders correctly — validate against a real inbox, not the Resend - preview (as the spike did). An earlier synthetic test that used Python's `MIMEMultipart("related")` (which - omits the `type="text/html"` container param) was filed by Resend's parser as a loose attachment and broke — - that was a malformed *test message*, not a pipeline defect; real app mailers (e.g. Vaultwarden) emit a - well-formed related part and render correctly. No code change resulted. - -> **Demo box state:** global app-email left **ON** on guest 9201 (the validated working state). No customer apps -> have the per-app toggle on, so nothing else changed. +## Live send-tests +- **nextcloud (Symfony Mailer) — Scenario B PROVEN:** toggle ON injected `SMTP_HOST=felhom-controller`, + `SMTP_PORT=2526`, `SMTP_SECURE=` (empty), and the **split From** `MAIL_FROM_ADDRESS=nextcloud` + + `MAIL_DOMAIN=felhom.eu`. **§8 env-consumption:** nextcloud applied it on redeploy (`occ config:system:get` + shows `mail_smtphost=felhom-controller mail_smtpport=2526 mail_smtpsecure= mail_from_address=nextcloud + mail_domain=felhom.eu`) — the official image reads these via `getenv()` every boot, **no occ step needed**. A + real password-reset relayed: shim `relayed (from="nextcloud@felhom.eu" rcpts=1 bytes=20430 hub=200)`. The + Symfony client sent **plaintext to :2526 and never attempted STARTTLS** → delivered. (Gap 1 + Gap 2 proven + end-to-end with a real opportunistic-STARTTLS client.) +- **cal.com (Nodemailer) — Scenario A PROVEN:** toggle ON injected `EMAIL_SERVER_HOST=felhom-controller`, + `EMAIL_SERVER_PORT=2526`, `EMAIL_FROM=calcom@felhom.eu` (verified in-container). A real signup (HTTP 201 + "Created user") triggered the verification email: shim `relayed (from="calcom@felhom.eu" rcpts=1 bytes=10260 + hub=200)`. The Nodemailer client sent **plaintext to :2526 and never attempted STARTTLS** → delivered. + (calcom needed three pre-existing template fixes first — see Observations.) +- **No regression (Scenario C):** gitea redeployed under v0.89.0 still injects `SMTP_PORT=2525` + + `smtp+starttls`; a real gitea forgot-password relayed `(from="gitea@felhom.eu" hub=200)` via the unchanged + :2525 STARTTLS path. ## Observations -- **Mealie uses plaintext (NONE), not STARTTLS** — Mealie has no accept-invalid-cert option, so STARTTLS to the - self-signed shim would fail; the spike validated plaintext on :2525 (Docker-network only). Deliberate, - spike-faithful deviation from the prompt's "TLS for mealie" wording. -- v1 is **single-shot, no spool** (per the Q6 decision). v2 = separate shim container + accept-and-spool retry + - `Resend-Idempotency-Key`. -- Free-tier ceiling is **100 emails/day** across the fleet; further apps just need an `smtp_mapping` block. -- No secrets are written to any committed file; the Resend key lives only hub-side (out-of-band). +- **Three pre-existing cal.com template bugs** surfaced (never caught — the bad image pin meant calcom never + deployed): (1) image `v4.8.7` doesn't exist as a published image → `v4.6.9`; (2) missing `DATABASE_DIRECT_URL` + (Prisma `migrate deploy` fails → incomplete schema → 500s); (3) healthcheck path `/api/health` is 404 in + v4.x → `/api/auth/providers`. All fixed in the catalog. +- **immich (Gap 3) remains unwired by decision** — it has no SMTP env (admin-UI/DB or `IMMICH_CONFIG_FILE` + JSON); the next email piece is a config-file render+mount mechanism. +- **Other email-capable apps now wire trivially** as `tls_mode: plaintext` (ghost, outline, grafana, …) — no + per-app TLS determination needed under the accepted posture. +- No secrets in any committed file; the Resend key stays hub-side.