# Runbook — out-of-band secrets (felhom-system)

**Model:** out-of-band `kubectl` + this runbook (DECISION model **A**). Secret *values* are **never**
committed to git. The manifests in `manifests/` carry only **placeholders + comments** that point here.
The live values live in the operator's out-of-band store (password manager / escrow) and in the cluster
as Secrets created imperatively with the commands below.

> The `felhom` ArgoCD app syncs the **entire** `manifests/` dir, so any Secret committed there is
> re-applied on every sync. Secrets that must stay out of git are therefore created as objects that are
> **not present in `manifests/`** (ArgoCD does not manage them), e.g. `Secret/resend-api` below.

---

## Resend API key — `Secret/resend-api`

**What uses it (live consumers):**

| Consumer | How it reads the key |
|----------|----------------------|
| `hub` (`Deployment/hub`) | env `RESEND_API_KEY` ← `secretKeyRef: resend-api/RESEND_API_KEY`; the hub binary's `RESEND_API_KEY` env override fills `notifications.resend_api_key` (the committed ConfigMap field is an empty placeholder). |
| `contact-mailer` (`Deployment/contact-mailer`) | env `RESEND_API_KEY` ← `secretKeyRef: resend-api/RESEND_API_KEY`. |
| `healthchecks` (**not currently deployed**) | when deployed, wire `EMAIL_HOST_PASSWORD` to `secretKeyRef: resend-api/RESEND_API_KEY`; do **not** inline the value into `felhom.secret.yaml`. |
| Gmail "Send mail as" (`info@` / `admin@felhom.eu`) | external — the key is the SMTP password; updated manually in Gmail settings, **not** in the cluster. |

**Where the value lives out-of-band:** the operator's password manager, entry "Felhom Resend API key
(app-relay + alerts)". Send-scoped. **Never** written to a file on a host, the guest, or any commit.

### Create / rotate the Secret

The key is available as the `RESEND_API` env var in `kisfenyo`'s shell on the build host
(192.168.0.180). It is exported from `~/.bashrc`, so it is present in an **interactive** shell
(`bash -ic`) but **not** in a plain non-interactive `ssh host 'cmd'`.

Create (or update) the Secret without ever echoing the value — render the Secret YAML in the
(non-sudo) interactive shell where `$RESEND_API` is set, and pipe it into `sudo kubectl apply`:

```bash
# on 192.168.0.180, as kisfenyo:
bash -ic 'kubectl create secret generic resend-api -n felhom-system \
    --from-literal=RESEND_API_KEY="$RESEND_API" \
    --dry-run=client -o yaml' | sudo kubectl apply -f -
```

Then roll the consumers so they pick up the new value:

```bash
sudo kubectl -n felhom-system rollout restart deploy/hub deploy/contact-mailer
sudo kubectl -n felhom-system rollout status  deploy/hub deploy/contact-mailer --timeout=120s
```

### Verify a consumer can send

- **contact-mailer:** submit the website contact form at <https://felhom.eu> (or, if `DEBUG=true`,
  `curl -X POST https://felhom.eu/api/debug/test`) → mail arrives at `info@felhom.eu`.
- **hub:** trigger a notification (or the dispatcher test path) → mail arrives.
- **Gmail:** send a test from `info@felhom.eu` → delivers.

### Rotating away from a compromised key (ordered — load-bearing)

1. Create the **new** send-scoped key in Resend; store it out-of-band; set `RESEND_API` on host 180.
2. Run the create-Secret + rollout above with the new value.
3. Update Gmail "Send mail as" SMTP password to the new key.
4. **Verify every consumer sends on the new key** (above) — do this **before** step 5.
5. Only then: **delete the old key** in Resend. Re-verify one consumer still sends afterwards.

> Never delete the old key before step 4 passes — a premature delete breaks live mail.

---

## Other committed secrets (tracked, NOT yet de-gitted — backlog)

`manifests/felhom.secret.yaml` still commits other plaintext secrets (`healthchecks-config` `SECRET_KEY`
/ `SUPERUSER_PASSWORD`, `umami-config`, `gitea-creds`). These are **out of scope** for the Resend
rotation but are the same hygiene problem; de-git them the same way (out-of-band `Secret/...` +
placeholder) when touched. Tracked here so the gap is visible.