felhom.eu/documentation/audits/SPIKE-smtp-app-relay-2026-06-28.md

SPIKE — on-box SMTP app-relay (apps → shim → hub → Resend)

Date: 2026-06-28 Class: Spike (empirical validation; no production code shipped). Output is this doc only. Architecture under test (locked by operator, not re-litigated): Relay shape A — a small SMTP-accepting shim runs on the customer box (Docker-network only, no Resend key), and relays onward to the hub, which is the only place that holds the Resend key.

Verdict (short): READY. Every mechanism the production design depends on works against real apps and the live Resend account. A real Vaultwarden test email travelled app → on-box shim → relay-leg → Resend → real inbox, From vaultwarden@felhom.eu, TLS/DKIM-aligned, with the Resend key never present on the demo guest. All six Spike Questions answered with evidence below. The one finding that shapes the production design is Q4: raw-MIME passthrough via Resend SMTP delivers faithfully; parse-then-API silently drops inline (CID) images — so the hub leg should be SMTP passthrough, not the existing structured-API path.

---

1. Confirmed baselines

Repo	main @ commit	Version	Role in this spike
felhom.eu (hub)	808d0e8	v0.16.0	read-only — hub/internal/notify/dispatcher.go is the existing Resend HTTP-API path (the Q4 reference)
felhom-controller	7cddb88	v0.87.0	read-only — how apps get env today (tier2/optional_config)
felhom-agent	c3020ee	v0.43.0	not touched

Probe libraries (throwaway shim): github.com/emersion/go-smtp v0.24.0 + github.com/emersion/go-sasl v0.0.0-20241020182733-b788ff22d5a6 (go-sasl ships no server-side LOGIN helper — see Q2).

Topology used: demo guest 9201 on felhom-pve (Docker host; apps on the traefik-public bridge); relay-leg = build/k3s host 192.168.0.180 (where the key legitimately lives). Throwaway probe code lived in /root/spike-smtp/ (guest) and ~/spike-smtp/ (180); nothing committed except this doc.

Existing Resend path (the Q4 baseline): dispatcher.go:160 sendEmail() does POST https://api.resend.com/emails with structured JSON (from/to/subject/text), Authorization: Bearer <key>. It does not use Resend SMTP.

---

2. Per-Spike-Question verdicts (with evidence)

Q1 — Server accepts real app SMTP — YES

A ~190-line Go shim (emersion/go-smtp) ran as a container smtp-shim on traefik-public. App containers reach it by Docker-DNS name smtp-shim. Minimal viable config: one smtp.NewServer(backend) per listener, Backend returns a Session, Session.Data(io.Reader) consumes the raw DATA blob. Two listeners suffice:

• :2525 — plaintext, STARTTLS offered (TLSConfig set), AllowInsecureAuth=true
• :2465 — implicit TLS (ListenAndServeTLS()), self-signed cert generated in-process

Boot log:

[BOOT] plaintext+STARTTLS listener on :2525
[BOOT] implicit-TLS listener on :2465

Q2 — Real-app TLS/AUTH requirements — shim should offer all three; apps need NO auth

Controlled matrix (Python smtplib, self-signed cert accepted via CERT_NONE):

Mode	Port	AUTH	Result
plaintext	2525	none	PASS
plaintext	2525	PLAIN	PASS
STARTTLS	2525	none	PASS
STARTTLS	2525	LOGIN	PASS
implicit-TLS	2465	PLAIN	PASS
multipart+attachment	2525	none	PASS

Then real Vaultwarden (vaultwarden/server:1.33.2-alpine), exercising its SMTP_SECURITY across all three values — each triggered via the admin POST /admin/test/smtp (HTTP 200) and captured by the shim:

SMTP_SECURITY	Port	Shim saw	AUTH	Captured
off	2525	TLS=false (plaintext)	none	msg #8, 17912 B
starttls	2525	TLS=false→TLS=true (upgrade)	none	msg #7, 17912 B
force_tls	2465	TLS=true (implicit)	none	msg #9, 17912 B

Plus Mealie (ghcr.io/mealie-recipes/mealie:v3.10.2, SMTP_AUTH_STRATEGY=NONE) → plaintext, no auth, captured msg #10 (21917 B).

Conclusions:

• AUTH is not required. With no SMTP credentials set, both apps sent no AUTH at all (AUTH_PRESENT=false). The shim should still accept AUTH PLAIN/LOGIN (cheaply) so an app configured with dummy creds doesn't break — the shim accepts any credentials and ignores them. (Implementation note: go-sasl has NewPlainServer but no NewLoginServer; a ~15-line loginServer implementing the Username:/Password: exchange covers LOGIN.)
• The shim must terminate TLS to satisfy force_tls / starttls configs — a self-signed cert is fine on the Docker network, but the app must be told to accept it (Vaultwarden: SMTP_ACCEPT_INVALID_CERTS=true + SMTP_ACCEPT_INVALID_HOSTNAMES=true).
• Simplest production posture: configure each app off / plaintext / no-auth to a localhost/Docker-network shim (no cert to manage). Offer STARTTLS + implicit-TLS as well for apps that force TLS and can't be set to off.

Q3 — Capture fidelity — YES, byte-intact

The controlled multipart message captured with envelope + full MIME tree intact: multipart/mixed, a base64 text/html part, and an application/pdf attachment with Content-Disposition: attachment; filename="probe.pdf". The real Vaultwarden message (17912 B) captured complete: multipart/alternative → text/plain + multipart/related → quoted-printable HTML + two inline PNGs (Content-ID: <logo-gray.png>, <mail-github.png>, base64). Envelope sidecar for the real app:

AUTH_PRESENT=false
MAIL_FROM=vaultwarden@felhom.eu
RCPT_TO=[felhom.eu@gmail.com]
DATA_BYTES=17912

Q4 — Hub-leg fork — RECOMMEND (a) raw-MIME passthrough via Resend SMTP

Both methods were run on the relay-leg host (180) against the live Resend account, fed the exact captured Vaultwarden message:

(a) Passthrough — STARTTLS to smtp.resend.com:587, AUTH LOGIN resend / <key>, then raw MAIL/RCPT/DATA:

MAIL FROM<vaultwarden@felhom.eu> -> 250 Accepted
RCPT TO<...>                      -> 250 Accepted
DATA -> 250 ff9612a7-89a6-4833-bee8-548cea11edd3   (Resend queued-id in the 250 line)

Delivered email (operator inbox, screenshot): From Vaultwarden <vaultwarden@felhom.eu>, "encryption (TLS)", Vaultwarden logo image renders correctly — inline CID images preserved.

(b) Parse-then-API — MIME parsed into from/to/subject/html/text/attachments, POST /emails:

PARSED ... html=Y text=Y attachments=0 dropped_inline_cids=['<logo-gray.png>', '<mail-github.png>']
API POST -> http=200 body={"id":"e6bd7a5a-..."}

Delivered email (same inbox, screenshot): identical headers/TLS, but the header logo is a broken-image placeholder — the HTML still references cid:logo-gray.png, which the parser dropped (Resend's attachments API has no first-class inline-CID story). A clean multipart/mixed attachment (the PDF probe) did survive parse-API intact (attachments=1, HTTP 200) — so the loss is specifically inline/related CID images, which HTML transactional templates (Vaultwarden, and most app mailers) use heavily.

Why passthrough wins:

• Fidelity: the message Resend sends is byte-for-byte what the app composed — HTML, encoding, multipart structure, inline images, attachments. No parser to keep in sync with every app's MIME quirks.
• Less fragility: parse-then-API must re-implement MIME walking, pick the "right" html/text part, re-encode attachments, and handle inline-CID — each a place to silently corrupt mail (proven above).
• Cost: one Go net/smtp/go-smtp client on the hub vs. the MIME parser. The hub already holds the key; adding an SMTP client next to the existing dispatcher.go API path is small.
• The existing dispatcher.go API path stays for the hub's own structured notifications (operator/customer alerts) — those are not raw MIME. App-relay is a separate code path.

Q5 — From-policy / anti-spoof — enforceable at relay AND backstopped by Resend

Resend rejects an unverified From on both transports (deliberate wrong-domain send):

# parse-API:  API POST -> http=403 {"statusCode":403,
#   "message":"This API key is not authorized to send emails from evil-notfelhom.example"}
# passthrough: MAIL/RCPT 250 Accepted, then
#   DATA -> 550 This API key is not authorized to send emails from evil-notfelhom.example

So even if the relay-side check were missing, Resend is a hard backstop (the From header domain is what's checked, not the SMTP envelope — note the 550 lands at DATA, after MAIL/RCPT were accepted).

Relay-side enforcement (the proactive half) is trivial and recommended: the shim/hub parses the From header and rejects (or rewrites) any domain not in the allowlist (felhom.eu, later apps.felhom.eu) before dialing Resend — fail fast with a clean SMTP 5xx to the app, and never spend a Resend call on a doomed message. Recommendation: validate-and-reject (don't silently rewrite — rewriting a From the app deliberately set hides config errors). The controller sets each app's From to e.g. immich@felhom.eu; the relay enforces the domain.

Q6 — Failure behavior — apps fail synchronously and surface the exact error; no app-side retry → queue belongs in the shim

Two failure modes exercised:

• Shim unreachable (container stopped): Vaultwarden → HTTP 400 "SMTP error: Connection error: ... Name does not resolve"; Mealie → {"success":false,"error":"[Errno -2] Name or service not known"}. Both synchronous, clean, surfaced — no hang, no silent drop.
• Shim up, rejects at DATA (probe returned 451 4.4.1 upstream relay (hub) temporarily unavailable): Vaultwarden → "SMTP 4xx error: transient error (451): 4.4.1 upstream relay (hub) temporarily unavailable, try again later" — the app relays the exact SMTP code + message to its UI.

Conclusion: apps do a synchronous SMTP send tied to a user action (test email, password reset, invite) and have no spool/retry of their own — a failure becomes an immediate user-facing error. Therefore, if a transient hub outage must not break user flows, the retry queue belongs in the on-box shim: accept (250) and spool to the hub with retry. See the recommendation in §6.

---

3. Recommended shim (Q1/Q2 conclusion)

• Library: github.com/emersion/go-smtp (v0.24.0 worked out of the box) + go-sasl.
• Listeners: :2525 plaintext with STARTTLS offered, and :2465 implicit-TLS (self-signed cert generated at boot; CN/SAN smtp-shim). Bind to the Docker-network/localhost interface only — never publish to the host or internet.
• AUTH: advertise PLAIN + LOGIN, accept any credentials and ignore them (apps send none, but some refuse to send without an AUTH offer). No credential is a secret on the box.
• App config (controller-injected, simplest path): SMTP_HOST=<shim>, SMTP_PORT=2525, security off / plaintext / no-auth; From = <app>@felhom.eu. For apps that force TLS, point at :2465 + the app's "accept invalid certs" flag.
• The shim holds no Resend key; it only validates From (Q5) and forwards to the hub.

4. Hub-leg recommendation (Q4)

Adopt (a) raw-MIME passthrough via Resend SMTP. Add a small SMTP client beside the existing dispatcher.go API path (which remains for the hub's own structured alerts). The hub endpoint (e.g. POST /api/v1/mail, per-box auth) receives the raw message + envelope from the shim and re-emits it to smtp.resend.com:587 unchanged. Do not use parse-then-API for app mail — it silently drops inline CID images (proven) and couples the hub to every app's MIME idiosyncrasies.

5. From-enforcement design (Q5)

Relay parses the From header domain and rejects (clean SMTP 5xx) anything outside the allowlist (felhom.eu; apps.felhom.eu later). Resend is the backstop (403 API / 550 SMTP at DATA on unverified domain). Prefer reject over rewrite so a misconfigured app surfaces loudly instead of having mail silently re-stamped.

6. Failure / queueing recommendation (Q6)

Apps surface SMTP failures synchronously with no retry, so a transient hub outage = a failed password-reset for a real user. Recommend the production shim accept-and-spool with bounded retry to the hub (small on-disk queue, exponential backoff, a few minutes' TTL), returning 250 to the app once durably queued. If that's deemed over-scope for v1, the fallback is acceptable but worse UX: shim forwards inline and returns the hub's 4xx/5xx to the app (the app then shows the user an error). Either way the app needs no change — the choice is whether the shim absorbs transient hub blips.

7. Per-app SMTP env mapping (bonus → later .felhom.yml smtp_mapping)

Vaultwarden (vaultwarden/server:1.33.2-alpine) — live-tested, all three modes:

Env var	Value used	Semantics
SMTP_HOST	smtp-shim	shim DNS name on the app network
SMTP_PORT	2525 (off/starttls) / 2465 (force_tls)
SMTP_SECURITY	off | starttls | force_tls	TLS mode; off=plaintext
SMTP_FROM	vaultwarden@felhom.eu	envelope+header From
SMTP_FROM_NAME	Vaultwarden	display name
SMTP_USERNAME / SMTP_PASSWORD	(unset)	omit → app sends no AUTH
SMTP_ACCEPT_INVALID_CERTS	true	needed only for starttls/force_tls to self-signed shim
SMTP_ACCEPT_INVALID_HOSTNAMES	true	same

Mealie (ghcr.io/mealie-recipes/mealie:v3.10.2) — live-tested, plaintext/no-auth:

Env var	Value used	Semantics
SMTP_HOST	smtp-shim
SMTP_PORT	2525
SMTP_AUTH_STRATEGY	NONE	NONE | TLS | SSL; NONE = no STARTTLS, no auth
SMTP_FROM_NAME	Mealie
SMTP_FROM_EMAIL	mealie@felhom.eu	From
SMTP_USER / SMTP_PASSWORD	(unset with NONE)	required if strategy is TLS/SSL

Pattern for the mapping: every app exposes host / port / security-mode / from / optional user+pass. The smtp_mapping per template just renames these to the app's specific env keys and pins host=shim, from=<app>@felhom.eu.

---

8. Verdict

READY to write the production implementation TASK. No blocking gaps. The TASK should specify: (1) the on-box shim (go-smtp, two listeners, accept-any-AUTH, From-allowlist, accept-and-spool retry to hub); (2) the hub POST /api/v1/mail endpoint doing raw-SMTP passthrough to Resend (key stays hub-side); (3) the controller smtp_mapping env injection per app (default plaintext/no-auth to the shim, From=<app>@felhom.eu).

9. Observations (out of scope — documented, not acted on)

• Free-tier ceiling is the 100 emails/day cap, not the 3,000/mo — a fleet of boxes each sending the odd password-reset/invite will hit 100/day first. Reputation-isolation via a dedicated apps.felhom.eu subdomain needs Resend Pro (free tier = one verified domain, already felhom.eu); deferred, doesn't change the mechanism.
• Resend key is currently committed in plaintext in this repo's manifests (manifests/hub.yaml, manifests/felhom.secret.yaml) and lives in k8s Secret/contact-mailer-config + Secret/healthchecks-config. The whole point of relay-shape-A is keeping that key off boxes — worth also getting it out of git (sealed-secret / external secret) as a separate hygiene item. (This spike read it only on host 180, never echoed it, never put it on the guest, and it appears nowhere in this doc.)
• The committed key is send-scoped: GET /emails/{id} returns 401, so delivery status can't be polled with it (send POST works fine). A read-capable key would let the hub confirm deliveries if that's ever wanted.
• Resend checks the From header, not the SMTP envelope — the unverified-domain 550 lands at DATA, after MAIL FROM/RCPT TO are accepted. A relay doing its own pre-check should parse the header, not the envelope.

---

Evidence index (probe artifacts — throwaway, not committed)

• Shim: main.go (go-smtp, 2 listeners, in-proc self-signed cert, capture to /capture), Dockerfile.
• Drivers: driver.py (controlled matrix), vwtest.sh / mealietest.sh (real-app test-email triggers).
• Relay-leg: relay_stub.py (passthrough | api | status) on host 180; key sourced inline from Secret/contact-mailer-config, never printed.
• Captured messages msg-001..010.eml (guest 9201:/root/spike-smtp/capture/); delivered-email screenshots (passthrough = logo renders; parse-API = broken image) confirmed in the operator inbox.