# SPIKE — on-box SMTP app-relay (apps → shim → hub → Resend)

**Date:** 2026-06-28
**Class:** Spike (empirical validation; no production code shipped). Output is this doc only.
**Architecture under test (locked by operator, not re-litigated):** **Relay shape A** — a small SMTP-accepting
shim runs on the customer box (Docker-network only, **no Resend key**), and relays onward to the **hub**, which
is the only place that holds the Resend key.

**Verdict (short): READY.** Every mechanism the production design depends on works against real apps and the
live Resend account. A real Vaultwarden test email travelled `app → on-box shim → relay-leg → Resend → real
inbox`, From `vaultwarden@felhom.eu`, TLS/DKIM-aligned, with the Resend key **never present on the demo guest**.
All six Spike Questions answered with evidence below. The one finding that shapes the production design is **Q4**:
**raw-MIME passthrough via Resend SMTP delivers faithfully; parse-then-API silently drops inline (CID) images** —
so the hub leg should be **SMTP passthrough**, not the existing structured-API path.

---

## 1. Confirmed baselines

| Repo | `main` @ commit | Version | Role in this spike |
|------|-----------------|---------|--------------------|
| felhom.eu (hub) | `808d0e8` | v0.16.0 | read-only — `hub/internal/notify/dispatcher.go` is the existing Resend **HTTP-API** path (the Q4 reference) |
| felhom-controller | `7cddb88` | v0.87.0 | read-only — how apps get env today (`tier2`/`optional_config`) |
| felhom-agent | `c3020ee` | v0.43.0 | not touched |

**Probe libraries (throwaway shim):** `github.com/emersion/go-smtp` **v0.24.0** + `github.com/emersion/go-sasl`
`v0.0.0-20241020182733-b788ff22d5a6` (go-sasl ships **no** server-side LOGIN helper — see Q2).

**Topology used:** demo guest **9201** on `felhom-pve` (Docker host; apps on the `traefik-public` bridge);
relay-leg = build/k3s host **192.168.0.180** (where the key legitimately lives). Throwaway probe code lived in
`/root/spike-smtp/` (guest) and `~/spike-smtp/` (180); **nothing committed** except this doc.

**Existing Resend path (the Q4 baseline):** `dispatcher.go:160` `sendEmail()` does `POST https://api.resend.com/emails`
with structured JSON (`from`/`to`/`subject`/`text`), `Authorization: Bearer <key>`. It does **not** use Resend SMTP.

---

## 2. Per-Spike-Question verdicts (with evidence)

### Q1 — Server accepts real app SMTP — **YES**

A ~190-line Go shim (`emersion/go-smtp`) ran as a container `smtp-shim` on `traefik-public`. App containers reach
it by Docker-DNS name `smtp-shim`. Minimal viable config: one `smtp.NewServer(backend)` per listener, `Backend`
returns a `Session`, `Session.Data(io.Reader)` consumes the raw DATA blob. Two listeners suffice:

- `:2525` — plaintext, **STARTTLS** offered (`TLSConfig` set), `AllowInsecureAuth=true`
- `:2465` — **implicit TLS** (`ListenAndServeTLS()`), self-signed cert generated in-process

Boot log:
```
[BOOT] plaintext+STARTTLS listener on :2525
[BOOT] implicit-TLS listener on :2465
```

### Q2 — Real-app TLS/AUTH requirements — **shim should offer all three; apps need NO auth**

Controlled matrix (Python `smtplib`, self-signed cert accepted via `CERT_NONE`):

| Mode | Port | AUTH | Result |
|------|------|------|--------|
| plaintext | 2525 | none | PASS |
| plaintext | 2525 | PLAIN | PASS |
| STARTTLS | 2525 | none | PASS |
| STARTTLS | 2525 | LOGIN | PASS |
| implicit-TLS | 2465 | PLAIN | PASS |
| multipart+attachment | 2525 | none | PASS |

Then **real Vaultwarden** (`vaultwarden/server:1.33.2-alpine`), exercising its `SMTP_SECURITY` across all three
values — each triggered via the admin `POST /admin/test/smtp` (HTTP 200) and captured by the shim:

| `SMTP_SECURITY` | Port | Shim saw | AUTH | Captured |
|-----------------|------|----------|------|----------|
| `off` | 2525 | `TLS=false` (plaintext) | none | msg #8, 17912 B |
| `starttls` | 2525 | `TLS=false`→`TLS=true` (upgrade) | none | msg #7, 17912 B |
| `force_tls` | 2465 | `TLS=true` (implicit) | none | msg #9, 17912 B |

Plus **Mealie** (`ghcr.io/mealie-recipes/mealie:v3.10.2`, `SMTP_AUTH_STRATEGY=NONE`) → plaintext, no auth,
captured msg #10 (21917 B).

**Conclusions:**
- **AUTH is not required.** With no SMTP credentials set, both apps sent **no `AUTH`** at all (`AUTH_PRESENT=false`).
  The shim should still *accept* AUTH PLAIN/LOGIN (cheaply) so an app configured with dummy creds doesn't break —
  the shim accepts any credentials and ignores them. (Implementation note: `go-sasl` has `NewPlainServer` but **no**
  `NewLoginServer`; a ~15-line `loginServer` implementing the `Username:`/`Password:` exchange covers LOGIN.)
- **The shim must terminate TLS** to satisfy `force_tls` / `starttls` configs — a **self-signed cert is fine** on
  the Docker network, but the app must be told to accept it (Vaultwarden: `SMTP_ACCEPT_INVALID_CERTS=true` +
  `SMTP_ACCEPT_INVALID_HOSTNAMES=true`).
- **Simplest production posture:** configure each app **`off` / plaintext / no-auth** to a localhost/Docker-network
  shim (no cert to manage). Offer STARTTLS + implicit-TLS as well for apps that *force* TLS and can't be set to `off`.

### Q3 — Capture fidelity — **YES, byte-intact**

The controlled multipart message captured with envelope + full MIME tree intact: `multipart/mixed`, a base64
`text/html` part, and an `application/pdf` attachment with `Content-Disposition: attachment; filename="probe.pdf"`.
The **real Vaultwarden** message (17912 B) captured complete: `multipart/alternative` → `text/plain` +
`multipart/related` → quoted-printable HTML + **two inline PNGs** (`Content-ID: <logo-gray.png>`, `<mail-github.png>`,
base64). Envelope sidecar for the real app:
```
AUTH_PRESENT=false
MAIL_FROM=vaultwarden@felhom.eu
RCPT_TO=[felhom.eu@gmail.com]
DATA_BYTES=17912
```

### Q4 — Hub-leg fork — **RECOMMEND (a) raw-MIME passthrough via Resend SMTP**

Both methods were run on the relay-leg host (180) against the live Resend account, fed the **exact captured
Vaultwarden message**:

**(a) Passthrough** — STARTTLS to `smtp.resend.com:587`, `AUTH LOGIN resend / <key>`, then raw `MAIL/RCPT/DATA`:
```
MAIL FROM<vaultwarden@felhom.eu> -> 250 Accepted
RCPT TO<...>                      -> 250 Accepted
DATA -> 250 ff9612a7-89a6-4833-bee8-548cea11edd3   (Resend queued-id in the 250 line)
```
Delivered email (operator inbox, screenshot): From `Vaultwarden <vaultwarden@felhom.eu>`, "encryption (TLS)",
**Vaultwarden logo image renders correctly** — inline CID images preserved.

**(b) Parse-then-API** — MIME parsed into `from`/`to`/`subject`/`html`/`text`/`attachments`, `POST /emails`:
```
PARSED ... html=Y text=Y attachments=0 dropped_inline_cids=['<logo-gray.png>', '<mail-github.png>']
API POST -> http=200 body={"id":"e6bd7a5a-..."}
```
Delivered email (same inbox, screenshot): identical headers/TLS, but the header logo is a **broken-image
placeholder** — the HTML still references `cid:logo-gray.png`, which the parser dropped (Resend's `attachments`
API has no first-class inline-CID story). A clean `multipart/mixed` *attachment* (the PDF probe) **did** survive
parse-API intact (`attachments=1`, HTTP 200) — so the loss is specifically **inline/related CID images**, which
HTML transactional templates (Vaultwarden, and most app mailers) use heavily.

**Why passthrough wins:**
- **Fidelity:** the message Resend sends is byte-for-byte what the app composed — HTML, encoding, multipart
  structure, inline images, attachments. No parser to keep in sync with every app's MIME quirks.
- **Less fragility:** parse-then-API must re-implement MIME walking, pick the "right" html/text part, re-encode
  attachments, and handle inline-CID — each a place to silently corrupt mail (proven above).
- **Cost:** one Go `net/smtp`/`go-smtp` client on the hub vs. the MIME parser. The hub already holds the key;
  adding an SMTP client next to the existing `dispatcher.go` API path is small.
- The existing `dispatcher.go` **API path stays** for the hub's *own* structured notifications (operator/customer
  alerts) — those are not raw MIME. App-relay is a **separate** code path.

### Q5 — From-policy / anti-spoof — **enforceable at relay AND backstopped by Resend**

Resend rejects an unverified `From` on **both** transports (deliberate wrong-domain send):
```
# parse-API:  API POST -> http=403 {"statusCode":403,
#   "message":"This API key is not authorized to send emails from evil-notfelhom.example"}
# passthrough: MAIL/RCPT 250 Accepted, then
#   DATA -> 550 This API key is not authorized to send emails from evil-notfelhom.example
```
So even if the relay-side check were missing, Resend is a hard backstop (the From **header** domain is what's
checked, not the SMTP envelope — note the 550 lands at **DATA**, after MAIL/RCPT were accepted).

**Relay-side enforcement** (the proactive half) is trivial and recommended: the shim/hub parses the `From` header
and rejects (or rewrites) any domain not in the allowlist (`felhom.eu`, later `apps.felhom.eu`) **before** dialing
Resend — fail fast with a clean SMTP `5xx` to the app, and never spend a Resend call on a doomed message.
**Recommendation: validate-and-reject** (don't silently rewrite — rewriting a `From` the app deliberately set hides
config errors). The controller sets each app's `From` to e.g. `immich@felhom.eu`; the relay enforces the domain.

### Q6 — Failure behavior — **apps fail synchronously and surface the exact error; no app-side retry → queue belongs in the shim**

Two failure modes exercised:

- **Shim unreachable** (container stopped): Vaultwarden → `HTTP 400`
  `"SMTP error: Connection error: ... Name does not resolve"`; Mealie → `{"success":false,"error":"[Errno -2]
  Name or service not known"}`. Both **synchronous, clean, surfaced** — no hang, no silent drop.
- **Shim up, rejects at DATA** (probe returned `451 4.4.1 upstream relay (hub) temporarily unavailable`):
  Vaultwarden → `"SMTP 4xx error: transient error (451): 4.4.1 upstream relay (hub) temporarily unavailable, try
  again later"` — the app relays the **exact** SMTP code + message to its UI.

**Conclusion:** apps do a **synchronous** SMTP send tied to a user action (test email, password reset, invite) and
have **no spool/retry** of their own — a failure becomes an immediate user-facing error. Therefore, if a transient
hub outage must not break user flows, the **retry queue belongs in the on-box shim**: accept (`250`) and spool to
the hub with retry. See the recommendation in §6.

---

## 3. Recommended shim (Q1/Q2 conclusion)

- **Library:** `github.com/emersion/go-smtp` (v0.24.0 worked out of the box) + `go-sasl`.
- **Listeners:** `:2525` plaintext **with STARTTLS offered**, and `:2465` implicit-TLS (self-signed cert generated
  at boot; CN/SAN `smtp-shim`). Bind to the Docker-network/localhost interface only — **never** publish to the host
  or internet.
- **AUTH:** advertise PLAIN + LOGIN, **accept any credentials and ignore them** (apps send none, but some refuse to
  send without an `AUTH` offer). No credential is a secret on the box.
- **App config (controller-injected, simplest path):** `SMTP_HOST=<shim>`, `SMTP_PORT=2525`, security **off /
  plaintext / no-auth**; `From` = `<app>@felhom.eu`. For apps that *force* TLS, point at `:2465` + the app's
  "accept invalid certs" flag.
- The shim **holds no Resend key**; it only validates `From` (Q5) and forwards to the hub.

## 4. Hub-leg recommendation (Q4)

**Adopt (a) raw-MIME passthrough via Resend SMTP.** Add a small SMTP client beside the existing `dispatcher.go`
API path (which remains for the hub's own structured alerts). The hub endpoint (e.g. `POST /api/v1/mail`, per-box
auth) receives the raw message + envelope from the shim and re-emits it to `smtp.resend.com:587` unchanged.
**Do not** use parse-then-API for app mail — it silently drops inline CID images (proven) and couples the hub to
every app's MIME idiosyncrasies.

## 5. From-enforcement design (Q5)

Relay parses the `From` **header** domain and **rejects** (clean SMTP `5xx`) anything outside the allowlist
(`felhom.eu`; `apps.felhom.eu` later). Resend is the backstop (`403` API / `550` SMTP at DATA on unverified domain).
Prefer reject over rewrite so a misconfigured app surfaces loudly instead of having mail silently re-stamped.

## 6. Failure / queueing recommendation (Q6)

Apps surface SMTP failures synchronously with no retry, so a transient hub outage = a failed password-reset for a
real user. **Recommend the production shim accept-and-spool with bounded retry to the hub** (small on-disk queue,
exponential backoff, a few minutes' TTL), returning `250` to the app once durably queued. If that's deemed
over-scope for v1, the fallback is acceptable but worse UX: shim forwards inline and returns the hub's `4xx/5xx`
to the app (the app then shows the user an error). Either way the **app needs no change** — the choice is whether
the shim absorbs transient hub blips.

## 7. Per-app SMTP env mapping (bonus → later `.felhom.yml smtp_mapping`)

**Vaultwarden** (`vaultwarden/server:1.33.2-alpine`) — live-tested, all three modes:

| Env var | Value used | Semantics |
|---------|-----------|-----------|
| `SMTP_HOST` | `smtp-shim` | shim DNS name on the app network |
| `SMTP_PORT` | `2525` (off/starttls) / `2465` (force_tls) | |
| `SMTP_SECURITY` | `off` \| `starttls` \| `force_tls` | TLS mode; `off`=plaintext |
| `SMTP_FROM` | `vaultwarden@felhom.eu` | envelope+header From |
| `SMTP_FROM_NAME` | `Vaultwarden` | display name |
| `SMTP_USERNAME` / `SMTP_PASSWORD` | *(unset)* | omit → app sends no AUTH |
| `SMTP_ACCEPT_INVALID_CERTS` | `true` | needed only for `starttls`/`force_tls` to self-signed shim |
| `SMTP_ACCEPT_INVALID_HOSTNAMES` | `true` | same |

**Mealie** (`ghcr.io/mealie-recipes/mealie:v3.10.2`) — live-tested, plaintext/no-auth:

| Env var | Value used | Semantics |
|---------|-----------|-----------|
| `SMTP_HOST` | `smtp-shim` | |
| `SMTP_PORT` | `2525` | |
| `SMTP_AUTH_STRATEGY` | `NONE` | `NONE` \| `TLS` \| `SSL`; `NONE` = no STARTTLS, no auth |
| `SMTP_FROM_NAME` | `Mealie` | |
| `SMTP_FROM_EMAIL` | `mealie@felhom.eu` | From |
| `SMTP_USER` / `SMTP_PASSWORD` | *(unset with `NONE`)* | required if strategy is `TLS`/`SSL` |

Pattern for the mapping: every app exposes `host` / `port` / `security-mode` / `from` / optional `user`+`pass`. The
`smtp_mapping` per template just renames these to the app's specific env keys and pins `host`=shim, `from`=`<app>@felhom.eu`.

---

## 8. Verdict

**READY** to write the production implementation TASK. No blocking gaps. The TASK should specify: (1) the on-box
shim (go-smtp, two listeners, accept-any-AUTH, From-allowlist, **accept-and-spool retry to hub**); (2) the hub
`POST /api/v1/mail` endpoint doing **raw-SMTP passthrough** to Resend (key stays hub-side); (3) the controller
`smtp_mapping` env injection per app (default plaintext/no-auth to the shim, `From=<app>@felhom.eu`).

## 9. Observations (out of scope — documented, not acted on)

- **Free-tier ceiling is the 100 emails/day cap**, not the 3,000/mo — a fleet of boxes each sending the odd
  password-reset/invite will hit 100/day first. Reputation-isolation via a dedicated `apps.felhom.eu` subdomain
  needs **Resend Pro** (free tier = one verified domain, already `felhom.eu`); deferred, doesn't change the mechanism.
- **Resend key is currently committed in plaintext** in this repo's manifests (`manifests/hub.yaml`,
  `manifests/felhom.secret.yaml`) and lives in k8s `Secret/contact-mailer-config` + `Secret/healthchecks-config`.
  The whole point of relay-shape-A is keeping that key off boxes — worth also getting it out of git (sealed-secret
  / external secret) as a separate hygiene item. (This spike read it **only** on host 180, never echoed it, never
  put it on the guest, and it appears nowhere in this doc.)
- **The committed key is send-scoped:** `GET /emails/{id}` returns `401`, so delivery status can't be polled with
  it (send `POST` works fine). A read-capable key would let the hub confirm deliveries if that's ever wanted.
- **Resend checks the `From` header, not the SMTP envelope** — the unverified-domain `550` lands at **DATA**, after
  `MAIL FROM`/`RCPT TO` are accepted. A relay doing its own pre-check should parse the header, not the envelope.

---

### Evidence index (probe artifacts — throwaway, not committed)

- Shim: `main.go` (go-smtp, 2 listeners, in-proc self-signed cert, capture to `/capture`), `Dockerfile`.
- Drivers: `driver.py` (controlled matrix), `vwtest.sh` / `mealietest.sh` (real-app test-email triggers).
- Relay-leg: `relay_stub.py` (`passthrough` | `api` | `status`) on host 180; key sourced inline from
  `Secret/contact-mailer-config`, never printed.
- Captured messages `msg-001..010.eml` (guest `9201:/root/spike-smtp/capture/`); delivered-email screenshots
  (passthrough = logo renders; parse-API = broken image) confirmed in the operator inbox.