felhom.eu/documentation/runbooks/secrets.md

Runbook — out-of-band secrets (felhom-system)

Model: out-of-band kubectl + this runbook (DECISION model A). Secret values are never committed to git. The manifests in manifests/ carry only placeholders + comments that point here. The live values live in the operator's out-of-band store (password manager / escrow) and in the cluster as Secrets created imperatively with the commands below.

The felhom ArgoCD app syncs the entire manifests/ dir, so any Secret committed there is re-applied on every sync. Secrets that must stay out of git are therefore created as objects that are not present in manifests/ (ArgoCD does not manage them), e.g. Secret/resend-api below.

---

Resend API key — Secret/resend-api

What uses it (live consumers):

Consumer	How it reads the key
hub (Deployment/hub)	env RESEND_API_KEY ← secretKeyRef: resend-api/RESEND_API_KEY; the hub binary's RESEND_API_KEY env override fills notifications.resend_api_key (the committed ConfigMap field is an empty placeholder).
contact-mailer (Deployment/contact-mailer)	env RESEND_API_KEY ← secretKeyRef: resend-api/RESEND_API_KEY.
healthchecks (not currently deployed)	when deployed, wire EMAIL_HOST_PASSWORD to secretKeyRef: resend-api/RESEND_API_KEY; do not inline the value into felhom.secret.yaml.
Gmail "Send mail as" (info@ / admin@felhom.eu)	external — the key is the SMTP password; updated manually in Gmail settings, not in the cluster.

Where the value lives out-of-band: the operator's password manager, entry "Felhom Resend API key (app-relay + alerts)". Send-scoped. Never written to a file on a host, the guest, or any commit.

Create / rotate the Secret

The key is available as the RESEND_API env var in kisfenyo's shell on the build host (192.168.0.180). It is exported from ~/.bashrc, so it is present in an interactive shell (bash -ic) but not in a plain non-interactive ssh host 'cmd'.

Create (or update) the Secret without ever echoing the value — render the Secret YAML in the (non-sudo) interactive shell where $RESEND_API is set, and pipe it into sudo kubectl apply:

# on 192.168.0.180, as kisfenyo:
bash -ic 'kubectl create secret generic resend-api -n felhom-system \
    --from-literal=RESEND_API_KEY="$RESEND_API" \
    --dry-run=client -o yaml' | sudo kubectl apply -f -

Then roll the consumers so they pick up the new value:

sudo kubectl -n felhom-system rollout restart deploy/hub deploy/contact-mailer
sudo kubectl -n felhom-system rollout status  deploy/hub deploy/contact-mailer --timeout=120s

Verify a consumer can send

• contact-mailer: submit the website contact form at https://felhom.eu (or, if DEBUG=true, curl -X POST https://felhom.eu/api/debug/test) → mail arrives at info@felhom.eu.
• hub: trigger a notification (or the dispatcher test path) → mail arrives.
• Gmail: send a test from info@felhom.eu → delivers.

Rotating away from a compromised key (ordered — load-bearing)

1. Create the new send-scoped key in Resend; store it out-of-band; set RESEND_API on host 180.
2. Run the create-Secret + rollout above with the new value.
3. Update Gmail "Send mail as" SMTP password to the new key.
4. Verify every consumer sends on the new key (above) — do this before step 5.
5. Only then: delete the old key in Resend. Re-verify one consumer still sends afterwards.

Never delete the old key before step 4 passes — a premature delete breaks live mail.

---

Other committed secrets (tracked, NOT yet de-gitted — backlog)

manifests/felhom.secret.yaml still commits other plaintext secrets (healthchecks-config SECRET_KEY / SUPERUSER_PASSWORD, umami-config, gitea-creds). These are out of scope for the Resend rotation but are the same hygiene problem; de-git them the same way (out-of-band Secret/... + placeholder) when touched. Tracked here so the gap is visible.