felhom.eu/documentation/runbooks/secrets.md

Runbook — out-of-band secrets (felhom-system)

Model: out-of-band kubectl + this runbook (DECISION model A). Secret values are never committed to git. The manifests in manifests/ carry only placeholders + comments that point here. The live values live in the operator's out-of-band store (password manager / escrow) and in the cluster as Secrets created imperatively with the commands below.

Corrected 2026-10-09 (R-923): the password manager is Vaultwarden, which runs on DooPlex — it is not out of band for a loss of DooPlex. Every Secret below is also in DooPlex's nightly Secrets export, which since 2026-10-09 rides the encrypted off-site copy; the keys that open it are on the break-glass sheet (break-glass-sheet.md, total-loss-of-dooplex.md).

The felhom ArgoCD app syncs the entire manifests/ dir, so any Secret committed there is re-applied on every sync. Secrets that must stay out of git are therefore created as objects that are not present in manifests/ (ArgoCD does not manage them), e.g. Secret/resend-api below.

---

Resend API key — Secret/resend-api

What uses it (live consumers):

Consumer	How it reads the key
hub (Deployment/hub)	env RESEND_API_KEY ← secretKeyRef: resend-api/RESEND_API_KEY; the hub binary's RESEND_API_KEY env override fills notifications.resend_api_key (the committed ConfigMap field is an empty placeholder).
contact-mailer (Deployment/contact-mailer)	env RESEND_API_KEY ← secretKeyRef: resend-api/RESEND_API_KEY.
healthchecks (not currently deployed)	when deployed, wire EMAIL_HOST_PASSWORD to secretKeyRef: resend-api/RESEND_API_KEY; do not inline the value into felhom.secret.yaml.
Gmail "Send mail as" (info@ / admin@felhom.eu)	external — the key is the SMTP password; updated manually in Gmail settings, not in the cluster.

Where the value lives out-of-band: the operator's password manager, entry "Felhom Resend API key (app-relay + alerts)". Send-scoped. Never written to a file on a host, the guest, or any commit.

Create / rotate the Secret

The key is available as the RESEND_API env var in kisfenyo's shell on the build host (192.168.0.180). It is exported from ~/.bashrc, so it is present in an interactive shell (bash -ic) but not in a plain non-interactive ssh host 'cmd'.

Create (or update) the Secret without ever echoing the value — render the Secret YAML in the (non-sudo) interactive shell where $RESEND_API is set, and pipe it into sudo kubectl apply:

# on 192.168.0.180, as kisfenyo:
bash -ic 'kubectl create secret generic resend-api -n felhom-system \
    --from-literal=RESEND_API_KEY="$RESEND_API" \
    --dry-run=client -o yaml' | sudo kubectl apply -f -

Then roll the consumers so they pick up the new value:

sudo kubectl -n felhom-system rollout restart deploy/hub deploy/contact-mailer
sudo kubectl -n felhom-system rollout status  deploy/hub deploy/contact-mailer --timeout=120s

Verify a consumer can send

• contact-mailer: submit the website contact form at https://felhom.eu (or, if DEBUG=true, curl -X POST https://felhom.eu/api/debug/test) → mail arrives at info@felhom.eu.
• hub: trigger a notification (or the dispatcher test path) → mail arrives.
• Gmail: send a test from info@felhom.eu → delivers.

Rotating away from a compromised key (ordered — load-bearing)

1. Create the new send-scoped key in Resend; store it out-of-band; set RESEND_API on host 180.
2. Run the create-Secret + rollout above with the new value.
3. Update Gmail "Send mail as" SMTP password to the new key.
4. Verify every consumer sends on the new key (above) — do this before step 5.
5. Only then: delete the old key in Resend. Re-verify one consumer still sends afterwards.

Never delete the old key before step 4 passes — a premature delete breaks live mail.

---

Operator/global bearer key — Secret/report-api

The hub API's global bearer (api.report_api_key) — the operator's own key (e.g. felhom-ops … -hub-key), distinct from the per-customer/per-host keys the hub generates itself. It was COMMITTED in manifests/hub.yaml until v0.53.0 (flagged in the 0.81/0.113 and 0.85/0.120 publish runbooks, incl. a Phase-D screenshot exposure); the manifest now carries a secretKeyRef and scripts/manifest_bearer_gate.py blocks reintroduction. The git-history copy stays alive until the value is ROTATED — de-git alone kills nothing.

What uses it (live consumers of the GLOBAL key):

Consumer	How it reads the key
hub (Deployment/hub)	env REPORT_API_KEY ← secretKeyRef: report-api/REPORT_API_KEY (v0.53.0 env override fills api.report_api_key; the ConfigMap field is an empty placeholder). Not optional: — a missing Secret fails Ready by design.
Operator tooling (felhom-ops keys upload -hub-key …, runbook curl probes in break-glass.md / offsite-endpoint.md)	typed per-invocation from the out-of-band store — nothing machine-persisted.
felhom-controller repo controller.yaml.example	carried the LITERAL as example text (never a live consumer) — scrubbed 2026-07-13.

Per-customer (customer_configs.api_key) and per-host (hosts.api_key) keys are hub-generated and unaffected by a global-key rotation — no customer box breaks.

Where the value lives out-of-band: the operator's password manager, entry "Felhom hub global bearer (report_api_key)".

Create the Secret (pre-deploy for v0.53.0 — same value, no rotation yet)

Create it with the CURRENT value before syncing the v0.53.0 manifest (the pod refuses to start without it). Render on the build host without echoing the value (file-to-file, the operator-present rule):

# on 192.168.0.180, as kisfenyo — put the current key in a 0600 temp file first (no echo):
kubectl create secret generic report-api -n felhom-system \
    --from-file=REPORT_API_KEY=/dev/stdin < /path/to/keyfile \
    --dry-run=client -o yaml | sudo kubectl apply -f -
shred -u /path/to/keyfile

Rotation (supervised — operator GO required; ordered, load-bearing)

1. Mint the new key into a 0600 file: openssl rand -hex 32 > keyfile (no terminal echo).
2. Re-run the create-Secret pipe above with the new file; store the value out-of-band.
3. sudo kubectl -n felhom-system rollout restart deploy/hub && sudo kubectl -n felhom-system rollout status deploy/hub --timeout=120s
4. Verify before declaring the old key dead:
   • a customer box still reports (per-customer key — proves rotation touched nothing it shouldn't);
   • an operator call with the NEW key succeeds (e.g. an authed GET /api/v1/… probe);
   • the SAME call with the OLD key returns 401 — only now is the git-history copy dead.
5. Update the password-manager entry; note the rotation date in the publish-runbook disposition.

---

Off-site password sealing key — Secret/offsite-secret-key (hub v0.127.0, decision 69, R-821)

What uses it: hub env OFFSITE_SECRET_KEY (64 hex characters = 32 bytes). It seals every Storage Box sub-account password in one_time_secrets.value; the hub's key registrar opens them to write box keys. Required — the pod does not start without it.

Create (once, before the first v0.127.0 sync) — the value never touches a file or the terminal:

sudo kubectl -n felhom-system create secret generic offsite-secret-key \
  --from-literal=OFFSITE_SECRET_KEY="$(openssl rand -hex 32)"

If it is lost: the sealed passwords cannot be opened. Nothing on the boxes breaks (their keys are installed); the registrar and the daily check fail with offsite_key_audit_failed. Recover per customer with the hub's Re-issue offsite credentials button (the provider resets the password; the hub seals the new one). Keep a copy in the operator's password manager if a Re-issue round is not acceptable.

Rotation: not built. A new key cannot open the old rows; rotate by setting the new key and pressing Re-issue for every off-site customer.

---

DooPlex PBS — the ep0 copy (decision 70, 2026-10-03)

Not k8s Secrets — PBS's own private files on DooPlex, never in git:

File	Holds	Created
/etc/proxmox-backup/remote.cfg (root:backup 0640)	ep0 API token root@pam!dooplex-sync (base64, PBS format)	from the token's one-time output, file → file
/etc/proxmox-backup/notifications-priv.cfg (root:root 0600)	the Resend API key, as the SMTP password of target felhom-operator	from $RESEND_API, file → file

Rotating the Resend key (§ above) must also rewrite notifications-priv.cfg, or the ep0-copy failure mails stop silently. Rotate the ep0 token with proxmox-backup-manager user generate-token on ep0 (delete the old) and rewrite remote.cfg. See runbooks/ep0-datastore-copy.md.

---

felhom.secret.yaml — ROTATED AND DE-GITTED 2026-10-09 (R-925)

What was wrong. manifests/felhom.secret.yaml committed three Secrets in plaintext, and gitea.dooplex.hu serves this repo to anyone on the internet with no login (measured from off-network: a real path returns the file, a nonsense path returns 404). Every committed value was still live — nothing had ever been rotated. Worst of all, gitea-creds was the Gitea admin account password (is_admin: true, /api/v1/admin/users answered 200), and the same string was reused as the admin password in 15 cluster Secrets across the homelab.

What was done, in this order (the order matters — de-gitting alone kills nothing, the history keeps the value):

1. umami-config — new random APP_SECRET and POSTGRES_PASSWORD. The password was changed inside Postgres (ALTER USER umami WITH PASSWORD) as well as in the Secret; the env var alone would not have changed it, because it is only read when the database is first initialised.
2. healthchecks-config — new random SECRET_KEY and SUPERUSER_PASSWORD. Nothing consumes them (there is no healthchecks Deployment), so this was free.
3. gitea-creds — the hub no longer holds the admin password at all. It now holds a scoped Gitea access token (read:package + read:repository). Both scopes are required and were each verified against their real endpoint before the swap: GET /v2/admin/felhom-controller/tags/list (the registry version checker) and GET /admin/felhom-controller/raw/branch/main/controller/configs/controller.yaml.example (the template fetcher — a read:package-only token gets 403 here, which is how the missing scope was found).
4. The Gitea admin password was changed to a new random value and gitea-system/gitea-admin updated. Verified: the new password returns 200, the old published one returns 401.

The values were written to a 0600 file on DooPlex (~/rotated-secrets-2026-10-09.txt) and are to be moved into the password manager and the file deleted. They were never echoed to a terminal.

The file is gone from git and .gitignore's *secret* rule now applies to it (it never did before: .gitignore is not consulted for a file git already tracks, which is why the rule looked broken). scripts/manifest_bearer_gate.py's KNOWN_BACKLOG carve-out was removed in the same commit, so a reintroduction now fails the gate.

Recreating these Secrets (they are no longer in manifests/)

ArgoCD does not manage them any more, exactly like Secret/resend-api above. The felhom app has automated.enabled: false and no prune, so removing the file does not delete them. To recreate one from the password manager, on 192.168.0.180 as kisfenyo, writing the value to a 0600 temp file first so it is never echoed:

# umami-config: APP_SECRET + POSTGRES_PASSWORD.
# If POSTGRES_PASSWORD changes, ALTER USER in the database too, or umami cannot connect:
#   kubectl exec -n felhom-system deploy/umami-db -- psql -U umami -d umami \
#     -c "ALTER USER umami WITH PASSWORD '<new>';"
sudo kubectl create secret generic umami-config -n felhom-system \
  --from-file=APP_SECRET=/path/app_secret --from-file=POSTGRES_PASSWORD=/path/pg_pw \
  --dry-run=client -o yaml | sudo kubectl apply -f -

# gitea-creds: username=admin, password = a SCOPED TOKEN (read:package + read:repository),
# never the admin account password.
sudo kubectl create secret generic gitea-creds -n felhom-system \
  --from-literal=username=admin --from-file=password=/path/token \
  --dry-run=client -o yaml | sudo kubectl apply -f -

# healthchecks-config: only needed if healthchecks is ever deployed; wire EMAIL_HOST_PASSWORD to
# secretKeyRef resend-api/RESEND_API_KEY rather than inlining it (see the Resend section above).

Still owed, and NOT done here (they are outside this repo): the same password is still the admin password in ~12 other namespaces — nextcloud, paperless, bookstack (a database root password), tandoor, calibre, adventurelog, gokapi, qbittorrent, servarr, homepage. Rotating Gitea does not touch those; each is its own login and each is still the published string.