felhom.eu/documentation/runbooks/secrets.md

Runbook — out-of-band secrets (felhom-system)

Model: out-of-band kubectl + this runbook (DECISION model A). Secret values are never committed to git. The manifests in manifests/ carry only placeholders + comments that point here. The live values live in the operator's out-of-band store (password manager / escrow) and in the cluster as Secrets created imperatively with the commands below.

The felhom ArgoCD app syncs the entire manifests/ dir, so any Secret committed there is re-applied on every sync. Secrets that must stay out of git are therefore created as objects that are not present in manifests/ (ArgoCD does not manage them), e.g. Secret/resend-api below.

---

Resend API key — Secret/resend-api

What uses it (live consumers):

Consumer	How it reads the key
hub (Deployment/hub)	env RESEND_API_KEY ← secretKeyRef: resend-api/RESEND_API_KEY; the hub binary's RESEND_API_KEY env override fills notifications.resend_api_key (the committed ConfigMap field is an empty placeholder).
contact-mailer (Deployment/contact-mailer)	env RESEND_API_KEY ← secretKeyRef: resend-api/RESEND_API_KEY.
healthchecks (not currently deployed)	when deployed, wire EMAIL_HOST_PASSWORD to secretKeyRef: resend-api/RESEND_API_KEY; do not inline the value into felhom.secret.yaml.
Gmail "Send mail as" (info@ / admin@felhom.eu)	external — the key is the SMTP password; updated manually in Gmail settings, not in the cluster.

Where the value lives out-of-band: the operator's password manager, entry "Felhom Resend API key (app-relay + alerts)". Send-scoped. Never written to a file on a host, the guest, or any commit.

Create / rotate the Secret

The key is available as the RESEND_API env var in kisfenyo's shell on the build host (192.168.0.180). It is exported from ~/.bashrc, so it is present in an interactive shell (bash -ic) but not in a plain non-interactive ssh host 'cmd'.

Create (or update) the Secret without ever echoing the value — render the Secret YAML in the (non-sudo) interactive shell where $RESEND_API is set, and pipe it into sudo kubectl apply:

# on 192.168.0.180, as kisfenyo:
bash -ic 'kubectl create secret generic resend-api -n felhom-system \
    --from-literal=RESEND_API_KEY="$RESEND_API" \
    --dry-run=client -o yaml' | sudo kubectl apply -f -

Then roll the consumers so they pick up the new value:

sudo kubectl -n felhom-system rollout restart deploy/hub deploy/contact-mailer
sudo kubectl -n felhom-system rollout status  deploy/hub deploy/contact-mailer --timeout=120s

Verify a consumer can send

• contact-mailer: submit the website contact form at https://felhom.eu (or, if DEBUG=true, curl -X POST https://felhom.eu/api/debug/test) → mail arrives at info@felhom.eu.
• hub: trigger a notification (or the dispatcher test path) → mail arrives.
• Gmail: send a test from info@felhom.eu → delivers.

Rotating away from a compromised key (ordered — load-bearing)

1. Create the new send-scoped key in Resend; store it out-of-band; set RESEND_API on host 180.
2. Run the create-Secret + rollout above with the new value.
3. Update Gmail "Send mail as" SMTP password to the new key.
4. Verify every consumer sends on the new key (above) — do this before step 5.
5. Only then: delete the old key in Resend. Re-verify one consumer still sends afterwards.

Never delete the old key before step 4 passes — a premature delete breaks live mail.

---

Operator/global bearer key — Secret/report-api

The hub API's global bearer (api.report_api_key) — the operator's own key (e.g. felhom-ops … -hub-key), distinct from the per-customer/per-host keys the hub generates itself. It was COMMITTED in manifests/hub.yaml until v0.53.0 (flagged in the 0.81/0.113 and 0.85/0.120 publish runbooks, incl. a Phase-D screenshot exposure); the manifest now carries a secretKeyRef and scripts/manifest_bearer_gate.py blocks reintroduction. The git-history copy stays alive until the value is ROTATED — de-git alone kills nothing.

What uses it (live consumers of the GLOBAL key):

Consumer	How it reads the key
hub (Deployment/hub)	env REPORT_API_KEY ← secretKeyRef: report-api/REPORT_API_KEY (v0.53.0 env override fills api.report_api_key; the ConfigMap field is an empty placeholder). Not optional: — a missing Secret fails Ready by design.
Operator tooling (felhom-ops keys upload -hub-key …, runbook curl probes in break-glass.md / offsite-endpoint.md)	typed per-invocation from the out-of-band store — nothing machine-persisted.
felhom-controller repo controller.yaml.example	carried the LITERAL as example text (never a live consumer) — scrubbed 2026-07-13.

Per-customer (customer_configs.api_key) and per-host (hosts.api_key) keys are hub-generated and unaffected by a global-key rotation — no customer box breaks.

Where the value lives out-of-band: the operator's password manager, entry "Felhom hub global bearer (report_api_key)".

Create the Secret (pre-deploy for v0.53.0 — same value, no rotation yet)

Create it with the CURRENT value before syncing the v0.53.0 manifest (the pod refuses to start without it). Render on the build host without echoing the value (file-to-file, the operator-present rule):

# on 192.168.0.180, as kisfenyo — put the current key in a 0600 temp file first (no echo):
kubectl create secret generic report-api -n felhom-system \
    --from-file=REPORT_API_KEY=/dev/stdin < /path/to/keyfile \
    --dry-run=client -o yaml | sudo kubectl apply -f -
shred -u /path/to/keyfile

Rotation (supervised — operator GO required; ordered, load-bearing)

1. Mint the new key into a 0600 file: openssl rand -hex 32 > keyfile (no terminal echo).
2. Re-run the create-Secret pipe above with the new file; store the value out-of-band.
3. sudo kubectl -n felhom-system rollout restart deploy/hub && sudo kubectl -n felhom-system rollout status deploy/hub --timeout=120s
4. Verify before declaring the old key dead:
   • a customer box still reports (per-customer key — proves rotation touched nothing it shouldn't);
   • an operator call with the NEW key succeeds (e.g. an authed GET /api/v1/… probe);
   • the SAME call with the OLD key returns 401 — only now is the git-history copy dead.
5. Update the password-manager entry; note the rotation date in the publish-runbook disposition.

---

Off-site password sealing key — Secret/offsite-secret-key (hub v0.127.0, decision 69, R-821)

What uses it: hub env OFFSITE_SECRET_KEY (64 hex characters = 32 bytes). It seals every Storage Box sub-account password in one_time_secrets.value; the hub's key registrar opens them to write box keys. Required — the pod does not start without it.

Create (once, before the first v0.127.0 sync) — the value never touches a file or the terminal:

sudo kubectl -n felhom-system create secret generic offsite-secret-key \
  --from-literal=OFFSITE_SECRET_KEY="$(openssl rand -hex 32)"

If it is lost: the sealed passwords cannot be opened. Nothing on the boxes breaks (their keys are installed); the registrar and the daily check fail with offsite_key_audit_failed. Recover per customer with the hub's Re-issue offsite credentials button (the provider resets the password; the hub seals the new one). Keep a copy in the operator's password manager if a Re-issue round is not acceptable.

Rotation: not built. A new key cannot open the old rows; rotate by setting the new key and pressing Re-issue for every off-site customer.

---

DooPlex PBS — the ep0 copy (decision 70, 2026-10-03)

Not k8s Secrets — PBS's own private files on DooPlex, never in git:

File	Holds	Created
/etc/proxmox-backup/remote.cfg (root:backup 0640)	ep0 API token root@pam!dooplex-sync (base64, PBS format)	from the token's one-time output, file → file
/etc/proxmox-backup/notifications-priv.cfg (root:root 0600)	the Resend API key, as the SMTP password of target felhom-operator	from $RESEND_API, file → file

Rotating the Resend key (§ above) must also rewrite notifications-priv.cfg, or the ep0-copy failure mails stop silently. Rotate the ep0 token with proxmox-backup-manager user generate-token on ep0 (delete the old) and rewrite remote.cfg. See runbooks/ep0-datastore-copy.md.

---

Other committed secrets (tracked, NOT yet de-gitted — backlog)

manifests/felhom.secret.yaml still commits other plaintext secrets (healthchecks-config SECRET_KEY / SUPERUSER_PASSWORD, umami-config, gitea-creds). These are out of scope for the Resend rotation but are the same hygiene problem; de-git them the same way (out-of-band Secret/... + placeholder) when touched. Tracked here so the gap is visible.