docs(audit): SPIKE findings — on-box SMTP app-relay (apps → shim → hub → Resend)
Verdict READY. Real Vaultwarden test email travelled app → on-box go-smtp shim → relay-leg → Resend → real inbox, From felhom.eu, TLS/DKIM-aligned, Resend key never on the demo guest. Q4 central finding: raw-MIME passthrough via Resend SMTP delivers faithfully; parse-then-API silently drops inline (CID) images — hub leg should be SMTP passthrough, not the structured-API path. Probe code throwaway (not committed); cleaned up from guest 9201 + host 180. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NB4fVgvEurdKgyG8KoQSzC
This commit is contained in:
@@ -0,0 +1,274 @@
|
||||
# SPIKE — on-box SMTP app-relay (apps → shim → hub → Resend)
|
||||
|
||||
**Date:** 2026-06-28
|
||||
**Class:** Spike (empirical validation; no production code shipped). Output is this doc only.
|
||||
**Architecture under test (locked by operator, not re-litigated):** **Relay shape A** — a small SMTP-accepting
|
||||
shim runs on the customer box (Docker-network only, **no Resend key**), and relays onward to the **hub**, which
|
||||
is the only place that holds the Resend key.
|
||||
|
||||
**Verdict (short): READY.** Every mechanism the production design depends on works against real apps and the
|
||||
live Resend account. A real Vaultwarden test email travelled `app → on-box shim → relay-leg → Resend → real
|
||||
inbox`, From `vaultwarden@felhom.eu`, TLS/DKIM-aligned, with the Resend key **never present on the demo guest**.
|
||||
All six Spike Questions answered with evidence below. The one finding that shapes the production design is **Q4**:
|
||||
**raw-MIME passthrough via Resend SMTP delivers faithfully; parse-then-API silently drops inline (CID) images** —
|
||||
so the hub leg should be **SMTP passthrough**, not the existing structured-API path.
|
||||
|
||||
---
|
||||
|
||||
## 1. Confirmed baselines
|
||||
|
||||
| Repo | `main` @ commit | Version | Role in this spike |
|
||||
|------|-----------------|---------|--------------------|
|
||||
| felhom.eu (hub) | `808d0e8` | v0.16.0 | read-only — `hub/internal/notify/dispatcher.go` is the existing Resend **HTTP-API** path (the Q4 reference) |
|
||||
| felhom-controller | `7cddb88` | v0.87.0 | read-only — how apps get env today (`tier2`/`optional_config`) |
|
||||
| felhom-agent | `c3020ee` | v0.43.0 | not touched |
|
||||
|
||||
**Probe libraries (throwaway shim):** `github.com/emersion/go-smtp` **v0.24.0** + `github.com/emersion/go-sasl`
|
||||
`v0.0.0-20241020182733-b788ff22d5a6` (go-sasl ships **no** server-side LOGIN helper — see Q2).
|
||||
|
||||
**Topology used:** demo guest **9201** on `felhom-pve` (Docker host; apps on the `traefik-public` bridge);
|
||||
relay-leg = build/k3s host **192.168.0.180** (where the key legitimately lives). Throwaway probe code lived in
|
||||
`/root/spike-smtp/` (guest) and `~/spike-smtp/` (180); **nothing committed** except this doc.
|
||||
|
||||
**Existing Resend path (the Q4 baseline):** `dispatcher.go:160` `sendEmail()` does `POST https://api.resend.com/emails`
|
||||
with structured JSON (`from`/`to`/`subject`/`text`), `Authorization: Bearer <key>`. It does **not** use Resend SMTP.
|
||||
|
||||
---
|
||||
|
||||
## 2. Per-Spike-Question verdicts (with evidence)
|
||||
|
||||
### Q1 — Server accepts real app SMTP — **YES**
|
||||
|
||||
A ~190-line Go shim (`emersion/go-smtp`) ran as a container `smtp-shim` on `traefik-public`. App containers reach
|
||||
it by Docker-DNS name `smtp-shim`. Minimal viable config: one `smtp.NewServer(backend)` per listener, `Backend`
|
||||
returns a `Session`, `Session.Data(io.Reader)` consumes the raw DATA blob. Two listeners suffice:
|
||||
|
||||
- `:2525` — plaintext, **STARTTLS** offered (`TLSConfig` set), `AllowInsecureAuth=true`
|
||||
- `:2465` — **implicit TLS** (`ListenAndServeTLS()`), self-signed cert generated in-process
|
||||
|
||||
Boot log:
|
||||
```
|
||||
[BOOT] plaintext+STARTTLS listener on :2525
|
||||
[BOOT] implicit-TLS listener on :2465
|
||||
```
|
||||
|
||||
### Q2 — Real-app TLS/AUTH requirements — **shim should offer all three; apps need NO auth**
|
||||
|
||||
Controlled matrix (Python `smtplib`, self-signed cert accepted via `CERT_NONE`):
|
||||
|
||||
| Mode | Port | AUTH | Result |
|
||||
|------|------|------|--------|
|
||||
| plaintext | 2525 | none | PASS |
|
||||
| plaintext | 2525 | PLAIN | PASS |
|
||||
| STARTTLS | 2525 | none | PASS |
|
||||
| STARTTLS | 2525 | LOGIN | PASS |
|
||||
| implicit-TLS | 2465 | PLAIN | PASS |
|
||||
| multipart+attachment | 2525 | none | PASS |
|
||||
|
||||
Then **real Vaultwarden** (`vaultwarden/server:1.33.2-alpine`), exercising its `SMTP_SECURITY` across all three
|
||||
values — each triggered via the admin `POST /admin/test/smtp` (HTTP 200) and captured by the shim:
|
||||
|
||||
| `SMTP_SECURITY` | Port | Shim saw | AUTH | Captured |
|
||||
|-----------------|------|----------|------|----------|
|
||||
| `off` | 2525 | `TLS=false` (plaintext) | none | msg #8, 17912 B |
|
||||
| `starttls` | 2525 | `TLS=false`→`TLS=true` (upgrade) | none | msg #7, 17912 B |
|
||||
| `force_tls` | 2465 | `TLS=true` (implicit) | none | msg #9, 17912 B |
|
||||
|
||||
Plus **Mealie** (`ghcr.io/mealie-recipes/mealie:v3.10.2`, `SMTP_AUTH_STRATEGY=NONE`) → plaintext, no auth,
|
||||
captured msg #10 (21917 B).
|
||||
|
||||
**Conclusions:**
|
||||
- **AUTH is not required.** With no SMTP credentials set, both apps sent **no `AUTH`** at all (`AUTH_PRESENT=false`).
|
||||
The shim should still *accept* AUTH PLAIN/LOGIN (cheaply) so an app configured with dummy creds doesn't break —
|
||||
the shim accepts any credentials and ignores them. (Implementation note: `go-sasl` has `NewPlainServer` but **no**
|
||||
`NewLoginServer`; a ~15-line `loginServer` implementing the `Username:`/`Password:` exchange covers LOGIN.)
|
||||
- **The shim must terminate TLS** to satisfy `force_tls` / `starttls` configs — a **self-signed cert is fine** on
|
||||
the Docker network, but the app must be told to accept it (Vaultwarden: `SMTP_ACCEPT_INVALID_CERTS=true` +
|
||||
`SMTP_ACCEPT_INVALID_HOSTNAMES=true`).
|
||||
- **Simplest production posture:** configure each app **`off` / plaintext / no-auth** to a localhost/Docker-network
|
||||
shim (no cert to manage). Offer STARTTLS + implicit-TLS as well for apps that *force* TLS and can't be set to `off`.
|
||||
|
||||
### Q3 — Capture fidelity — **YES, byte-intact**
|
||||
|
||||
The controlled multipart message captured with envelope + full MIME tree intact: `multipart/mixed`, a base64
|
||||
`text/html` part, and an `application/pdf` attachment with `Content-Disposition: attachment; filename="probe.pdf"`.
|
||||
The **real Vaultwarden** message (17912 B) captured complete: `multipart/alternative` → `text/plain` +
|
||||
`multipart/related` → quoted-printable HTML + **two inline PNGs** (`Content-ID: <logo-gray.png>`, `<mail-github.png>`,
|
||||
base64). Envelope sidecar for the real app:
|
||||
```
|
||||
AUTH_PRESENT=false
|
||||
MAIL_FROM=vaultwarden@felhom.eu
|
||||
RCPT_TO=[felhom.eu@gmail.com]
|
||||
DATA_BYTES=17912
|
||||
```
|
||||
|
||||
### Q4 — Hub-leg fork — **RECOMMEND (a) raw-MIME passthrough via Resend SMTP**
|
||||
|
||||
Both methods were run on the relay-leg host (180) against the live Resend account, fed the **exact captured
|
||||
Vaultwarden message**:
|
||||
|
||||
**(a) Passthrough** — STARTTLS to `smtp.resend.com:587`, `AUTH LOGIN resend / <key>`, then raw `MAIL/RCPT/DATA`:
|
||||
```
|
||||
MAIL FROM<vaultwarden@felhom.eu> -> 250 Accepted
|
||||
RCPT TO<...> -> 250 Accepted
|
||||
DATA -> 250 ff9612a7-89a6-4833-bee8-548cea11edd3 (Resend queued-id in the 250 line)
|
||||
```
|
||||
Delivered email (operator inbox, screenshot): From `Vaultwarden <vaultwarden@felhom.eu>`, "encryption (TLS)",
|
||||
**Vaultwarden logo image renders correctly** — inline CID images preserved.
|
||||
|
||||
**(b) Parse-then-API** — MIME parsed into `from`/`to`/`subject`/`html`/`text`/`attachments`, `POST /emails`:
|
||||
```
|
||||
PARSED ... html=Y text=Y attachments=0 dropped_inline_cids=['<logo-gray.png>', '<mail-github.png>']
|
||||
API POST -> http=200 body={"id":"e6bd7a5a-..."}
|
||||
```
|
||||
Delivered email (same inbox, screenshot): identical headers/TLS, but the header logo is a **broken-image
|
||||
placeholder** — the HTML still references `cid:logo-gray.png`, which the parser dropped (Resend's `attachments`
|
||||
API has no first-class inline-CID story). A clean `multipart/mixed` *attachment* (the PDF probe) **did** survive
|
||||
parse-API intact (`attachments=1`, HTTP 200) — so the loss is specifically **inline/related CID images**, which
|
||||
HTML transactional templates (Vaultwarden, and most app mailers) use heavily.
|
||||
|
||||
**Why passthrough wins:**
|
||||
- **Fidelity:** the message Resend sends is byte-for-byte what the app composed — HTML, encoding, multipart
|
||||
structure, inline images, attachments. No parser to keep in sync with every app's MIME quirks.
|
||||
- **Less fragility:** parse-then-API must re-implement MIME walking, pick the "right" html/text part, re-encode
|
||||
attachments, and handle inline-CID — each a place to silently corrupt mail (proven above).
|
||||
- **Cost:** one Go `net/smtp`/`go-smtp` client on the hub vs. the MIME parser. The hub already holds the key;
|
||||
adding an SMTP client next to the existing `dispatcher.go` API path is small.
|
||||
- The existing `dispatcher.go` **API path stays** for the hub's *own* structured notifications (operator/customer
|
||||
alerts) — those are not raw MIME. App-relay is a **separate** code path.
|
||||
|
||||
### Q5 — From-policy / anti-spoof — **enforceable at relay AND backstopped by Resend**
|
||||
|
||||
Resend rejects an unverified `From` on **both** transports (deliberate wrong-domain send):
|
||||
```
|
||||
# parse-API: API POST -> http=403 {"statusCode":403,
|
||||
# "message":"This API key is not authorized to send emails from evil-notfelhom.example"}
|
||||
# passthrough: MAIL/RCPT 250 Accepted, then
|
||||
# DATA -> 550 This API key is not authorized to send emails from evil-notfelhom.example
|
||||
```
|
||||
So even if the relay-side check were missing, Resend is a hard backstop (the From **header** domain is what's
|
||||
checked, not the SMTP envelope — note the 550 lands at **DATA**, after MAIL/RCPT were accepted).
|
||||
|
||||
**Relay-side enforcement** (the proactive half) is trivial and recommended: the shim/hub parses the `From` header
|
||||
and rejects (or rewrites) any domain not in the allowlist (`felhom.eu`, later `apps.felhom.eu`) **before** dialing
|
||||
Resend — fail fast with a clean SMTP `5xx` to the app, and never spend a Resend call on a doomed message.
|
||||
**Recommendation: validate-and-reject** (don't silently rewrite — rewriting a `From` the app deliberately set hides
|
||||
config errors). The controller sets each app's `From` to e.g. `immich@felhom.eu`; the relay enforces the domain.
|
||||
|
||||
### Q6 — Failure behavior — **apps fail synchronously and surface the exact error; no app-side retry → queue belongs in the shim**
|
||||
|
||||
Two failure modes exercised:
|
||||
|
||||
- **Shim unreachable** (container stopped): Vaultwarden → `HTTP 400`
|
||||
`"SMTP error: Connection error: ... Name does not resolve"`; Mealie → `{"success":false,"error":"[Errno -2]
|
||||
Name or service not known"}`. Both **synchronous, clean, surfaced** — no hang, no silent drop.
|
||||
- **Shim up, rejects at DATA** (probe returned `451 4.4.1 upstream relay (hub) temporarily unavailable`):
|
||||
Vaultwarden → `"SMTP 4xx error: transient error (451): 4.4.1 upstream relay (hub) temporarily unavailable, try
|
||||
again later"` — the app relays the **exact** SMTP code + message to its UI.
|
||||
|
||||
**Conclusion:** apps do a **synchronous** SMTP send tied to a user action (test email, password reset, invite) and
|
||||
have **no spool/retry** of their own — a failure becomes an immediate user-facing error. Therefore, if a transient
|
||||
hub outage must not break user flows, the **retry queue belongs in the on-box shim**: accept (`250`) and spool to
|
||||
the hub with retry. See the recommendation in §6.
|
||||
|
||||
---
|
||||
|
||||
## 3. Recommended shim (Q1/Q2 conclusion)
|
||||
|
||||
- **Library:** `github.com/emersion/go-smtp` (v0.24.0 worked out of the box) + `go-sasl`.
|
||||
- **Listeners:** `:2525` plaintext **with STARTTLS offered**, and `:2465` implicit-TLS (self-signed cert generated
|
||||
at boot; CN/SAN `smtp-shim`). Bind to the Docker-network/localhost interface only — **never** publish to the host
|
||||
or internet.
|
||||
- **AUTH:** advertise PLAIN + LOGIN, **accept any credentials and ignore them** (apps send none, but some refuse to
|
||||
send without an `AUTH` offer). No credential is a secret on the box.
|
||||
- **App config (controller-injected, simplest path):** `SMTP_HOST=<shim>`, `SMTP_PORT=2525`, security **off /
|
||||
plaintext / no-auth**; `From` = `<app>@felhom.eu`. For apps that *force* TLS, point at `:2465` + the app's
|
||||
"accept invalid certs" flag.
|
||||
- The shim **holds no Resend key**; it only validates `From` (Q5) and forwards to the hub.
|
||||
|
||||
## 4. Hub-leg recommendation (Q4)
|
||||
|
||||
**Adopt (a) raw-MIME passthrough via Resend SMTP.** Add a small SMTP client beside the existing `dispatcher.go`
|
||||
API path (which remains for the hub's own structured alerts). The hub endpoint (e.g. `POST /api/v1/mail`, per-box
|
||||
auth) receives the raw message + envelope from the shim and re-emits it to `smtp.resend.com:587` unchanged.
|
||||
**Do not** use parse-then-API for app mail — it silently drops inline CID images (proven) and couples the hub to
|
||||
every app's MIME idiosyncrasies.
|
||||
|
||||
## 5. From-enforcement design (Q5)
|
||||
|
||||
Relay parses the `From` **header** domain and **rejects** (clean SMTP `5xx`) anything outside the allowlist
|
||||
(`felhom.eu`; `apps.felhom.eu` later). Resend is the backstop (`403` API / `550` SMTP at DATA on unverified domain).
|
||||
Prefer reject over rewrite so a misconfigured app surfaces loudly instead of having mail silently re-stamped.
|
||||
|
||||
## 6. Failure / queueing recommendation (Q6)
|
||||
|
||||
Apps surface SMTP failures synchronously with no retry, so a transient hub outage = a failed password-reset for a
|
||||
real user. **Recommend the production shim accept-and-spool with bounded retry to the hub** (small on-disk queue,
|
||||
exponential backoff, a few minutes' TTL), returning `250` to the app once durably queued. If that's deemed
|
||||
over-scope for v1, the fallback is acceptable but worse UX: shim forwards inline and returns the hub's `4xx/5xx`
|
||||
to the app (the app then shows the user an error). Either way the **app needs no change** — the choice is whether
|
||||
the shim absorbs transient hub blips.
|
||||
|
||||
## 7. Per-app SMTP env mapping (bonus → later `.felhom.yml smtp_mapping`)
|
||||
|
||||
**Vaultwarden** (`vaultwarden/server:1.33.2-alpine`) — live-tested, all three modes:
|
||||
|
||||
| Env var | Value used | Semantics |
|
||||
|---------|-----------|-----------|
|
||||
| `SMTP_HOST` | `smtp-shim` | shim DNS name on the app network |
|
||||
| `SMTP_PORT` | `2525` (off/starttls) / `2465` (force_tls) | |
|
||||
| `SMTP_SECURITY` | `off` \| `starttls` \| `force_tls` | TLS mode; `off`=plaintext |
|
||||
| `SMTP_FROM` | `vaultwarden@felhom.eu` | envelope+header From |
|
||||
| `SMTP_FROM_NAME` | `Vaultwarden` | display name |
|
||||
| `SMTP_USERNAME` / `SMTP_PASSWORD` | *(unset)* | omit → app sends no AUTH |
|
||||
| `SMTP_ACCEPT_INVALID_CERTS` | `true` | needed only for `starttls`/`force_tls` to self-signed shim |
|
||||
| `SMTP_ACCEPT_INVALID_HOSTNAMES` | `true` | same |
|
||||
|
||||
**Mealie** (`ghcr.io/mealie-recipes/mealie:v3.10.2`) — live-tested, plaintext/no-auth:
|
||||
|
||||
| Env var | Value used | Semantics |
|
||||
|---------|-----------|-----------|
|
||||
| `SMTP_HOST` | `smtp-shim` | |
|
||||
| `SMTP_PORT` | `2525` | |
|
||||
| `SMTP_AUTH_STRATEGY` | `NONE` | `NONE` \| `TLS` \| `SSL`; `NONE` = no STARTTLS, no auth |
|
||||
| `SMTP_FROM_NAME` | `Mealie` | |
|
||||
| `SMTP_FROM_EMAIL` | `mealie@felhom.eu` | From |
|
||||
| `SMTP_USER` / `SMTP_PASSWORD` | *(unset with `NONE`)* | required if strategy is `TLS`/`SSL` |
|
||||
|
||||
Pattern for the mapping: every app exposes `host` / `port` / `security-mode` / `from` / optional `user`+`pass`. The
|
||||
`smtp_mapping` per template just renames these to the app's specific env keys and pins `host`=shim, `from`=`<app>@felhom.eu`.
|
||||
|
||||
---
|
||||
|
||||
## 8. Verdict
|
||||
|
||||
**READY** to write the production implementation TASK. No blocking gaps. The TASK should specify: (1) the on-box
|
||||
shim (go-smtp, two listeners, accept-any-AUTH, From-allowlist, **accept-and-spool retry to hub**); (2) the hub
|
||||
`POST /api/v1/mail` endpoint doing **raw-SMTP passthrough** to Resend (key stays hub-side); (3) the controller
|
||||
`smtp_mapping` env injection per app (default plaintext/no-auth to the shim, `From=<app>@felhom.eu`).
|
||||
|
||||
## 9. Observations (out of scope — documented, not acted on)
|
||||
|
||||
- **Free-tier ceiling is the 100 emails/day cap**, not the 3,000/mo — a fleet of boxes each sending the odd
|
||||
password-reset/invite will hit 100/day first. Reputation-isolation via a dedicated `apps.felhom.eu` subdomain
|
||||
needs **Resend Pro** (free tier = one verified domain, already `felhom.eu`); deferred, doesn't change the mechanism.
|
||||
- **Resend key is currently committed in plaintext** in this repo's manifests (`manifests/hub.yaml`,
|
||||
`manifests/felhom.secret.yaml`) and lives in k8s `Secret/contact-mailer-config` + `Secret/healthchecks-config`.
|
||||
The whole point of relay-shape-A is keeping that key off boxes — worth also getting it out of git (sealed-secret
|
||||
/ external secret) as a separate hygiene item. (This spike read it **only** on host 180, never echoed it, never
|
||||
put it on the guest, and it appears nowhere in this doc.)
|
||||
- **The committed key is send-scoped:** `GET /emails/{id}` returns `401`, so delivery status can't be polled with
|
||||
it (send `POST` works fine). A read-capable key would let the hub confirm deliveries if that's ever wanted.
|
||||
- **Resend checks the `From` header, not the SMTP envelope** — the unverified-domain `550` lands at **DATA**, after
|
||||
`MAIL FROM`/`RCPT TO` are accepted. A relay doing its own pre-check should parse the header, not the envelope.
|
||||
|
||||
---
|
||||
|
||||
### Evidence index (probe artifacts — throwaway, not committed)
|
||||
|
||||
- Shim: `main.go` (go-smtp, 2 listeners, in-proc self-signed cert, capture to `/capture`), `Dockerfile`.
|
||||
- Drivers: `driver.py` (controlled matrix), `vwtest.sh` / `mealietest.sh` (real-app test-email triggers).
|
||||
- Relay-leg: `relay_stub.py` (`passthrough` | `api` | `status`) on host 180; key sourced inline from
|
||||
`Secret/contact-mailer-config`, never printed.
|
||||
- Captured messages `msg-001..010.eml` (guest `9201:/root/spike-smtp/capture/`); delivered-email screenshots
|
||||
(passthrough = logo renders; parse-API = broken image) confirmed in the operator inbox.
|
||||
Reference in New Issue
Block a user