Runbook — out-of-band secrets (felhom-system)

Model: out-of-band kubectl + this runbook (DECISION model A). Secret values are never committed to git. The manifests in manifests/ carry only placeholders + comments that point here. The live values live in the operator's out-of-band store (password manager / escrow) and in the cluster as Secrets created imperatively with the commands below.

The felhom ArgoCD app syncs the entire manifests/ dir, so any Secret committed there is re-applied on every sync. Secrets that must stay out of git are therefore created as objects that are not present in manifests/ (ArgoCD does not manage them), e.g. Secret/resend-api below.

Resend API key — `Secret/resend-api`

What uses it (live consumers):

Consumer	How it reads the key
`hub` (`Deployment/hub`)	env `RESEND_API_KEY` ← `secretKeyRef: resend-api/RESEND_API_KEY`; the hub binary's `RESEND_API_KEY` env override fills `notifications.resend_api_key` (the committed ConfigMap field is an empty placeholder).
`contact-mailer` (`Deployment/contact-mailer`)	env `RESEND_API_KEY` ← `secretKeyRef: resend-api/RESEND_API_KEY`.
`healthchecks` (not currently deployed)	when deployed, wire `EMAIL_HOST_PASSWORD` to `secretKeyRef: resend-api/RESEND_API_KEY`; do not inline the value into `felhom.secret.yaml`.
Gmail "Send mail as" (`info@` / `admin@felhom.eu`)	external — the key is the SMTP password; updated manually in Gmail settings, not in the cluster.

Where the value lives out-of-band: the operator's password manager, entry "Felhom Resend API key (app-relay + alerts)". Send-scoped. Never written to a file on a host, the guest, or any commit.

Create / rotate the Secret

The key is available as the RESEND_API env var in kisfenyo's shell on the build host (192.168.0.180). It is exported from ~/.bashrc, so it is present in an interactive shell (bash -ic) but not in a plain non-interactive ssh host 'cmd'.

Create (or update) the Secret without ever echoing the value — render the Secret YAML in the (non-sudo) interactive shell where $RESEND_API is set, and pipe it into sudo kubectl apply:

# on 192.168.0.180, as kisfenyo:
bash -ic 'kubectl create secret generic resend-api -n felhom-system \
    --from-literal=RESEND_API_KEY="$RESEND_API" \
    --dry-run=client -o yaml' | sudo kubectl apply -f -

Then roll the consumers so they pick up the new value:

sudo kubectl -n felhom-system rollout restart deploy/hub deploy/contact-mailer
sudo kubectl -n felhom-system rollout status  deploy/hub deploy/contact-mailer --timeout=120s

Verify a consumer can send

contact-mailer: submit the website contact form at https://felhom.eu (or, if DEBUG=true, curl -X POST https://felhom.eu/api/debug/test) → mail arrives at info@felhom.eu.
hub: trigger a notification (or the dispatcher test path) → mail arrives.
Gmail: send a test from info@felhom.eu → delivers.

Rotating away from a compromised key (ordered — load-bearing)

Create the new send-scoped key in Resend; store it out-of-band; set RESEND_API on host 180.
Run the create-Secret + rollout above with the new value.
Update Gmail "Send mail as" SMTP password to the new key.
Verify every consumer sends on the new key (above) — do this before step 5.
Only then: delete the old key in Resend. Re-verify one consumer still sends afterwards.

Never delete the old key before step 4 passes — a premature delete breaks live mail.

Operator/global bearer key — `Secret/report-api`

The hub API's global bearer (api.report_api_key) — the operator's own key (e.g. felhom-ops … -hub-key), distinct from the per-customer/per-host keys the hub generates itself. It was COMMITTED in manifests/hub.yaml until v0.53.0 (flagged in the 0.81/0.113 and 0.85/0.120 publish runbooks, incl. a Phase-D screenshot exposure); the manifest now carries a secretKeyRef and scripts/manifest_bearer_gate.py blocks reintroduction. The git-history copy stays alive until the value is ROTATED — de-git alone kills nothing.

What uses it (live consumers of the GLOBAL key):

Consumer	How it reads the key
`hub` (`Deployment/hub`)	env `REPORT_API_KEY` ← `secretKeyRef: report-api/REPORT_API_KEY` (v0.53.0 env override fills `api.report_api_key`; the ConfigMap field is an empty placeholder). Not `optional:` — a missing Secret fails Ready by design.
Operator tooling (`felhom-ops keys upload -hub-key …`, runbook curl probes in break-glass.md / offsite-endpoint.md)	typed per-invocation from the out-of-band store — nothing machine-persisted.
~~`felhom-controller` repo `controller.yaml.example`~~	carried the LITERAL as example text (never a live consumer) — scrubbed 2026-07-13.

Per-customer (customer_configs.api_key) and per-host (hosts.api_key) keys are hub-generated and unaffected by a global-key rotation — no customer box breaks.

Where the value lives out-of-band: the operator's password manager, entry "Felhom hub global bearer (report_api_key)".

Create the Secret (pre-deploy for v0.53.0 — same value, no rotation yet)

Create it with the CURRENT value before syncing the v0.53.0 manifest (the pod refuses to start without it). Render on the build host without echoing the value (file-to-file, the operator-present rule):

# on 192.168.0.180, as kisfenyo — put the current key in a 0600 temp file first (no echo):
kubectl create secret generic report-api -n felhom-system \
    --from-file=REPORT_API_KEY=/dev/stdin < /path/to/keyfile \
    --dry-run=client -o yaml | sudo kubectl apply -f -
shred -u /path/to/keyfile

Rotation (supervised — operator GO required; ordered, load-bearing)

Mint the new key into a 0600 file: openssl rand -hex 32 > keyfile (no terminal echo).
Re-run the create-Secret pipe above with the new file; store the value out-of-band.
sudo kubectl -n felhom-system rollout restart deploy/hub && sudo kubectl -n felhom-system rollout status deploy/hub --timeout=120s
Verify before declaring the old key dead:
- a customer box still reports (per-customer key — proves rotation touched nothing it shouldn't);
- an operator call with the NEW key succeeds (e.g. an authed GET /api/v1/… probe);
- the SAME call with the OLD key returns 401 — only now is the git-history copy dead.
Update the password-manager entry; note the rotation date in the publish-runbook disposition.

Other committed secrets (tracked, NOT yet de-gitted — backlog)

manifests/felhom.secret.yaml still commits other plaintext secrets (healthchecks-config SECRET_KEY / SUPERUSER_PASSWORD, umami-config, gitea-creds). These are out of scope for the Resend rotation but are the same hygiene problem; de-git them the same way (out-of-band Secret/... + placeholder) when touched. Tracked here so the gap is visible.

7.0 KiB Raw Blame History