126 lines
7.0 KiB
Markdown
126 lines
7.0 KiB
Markdown
# Runbook — out-of-band secrets (felhom-system)
|
|
|
|
**Model:** out-of-band `kubectl` + this runbook (DECISION model **A**). Secret *values* are **never**
|
|
committed to git. The manifests in `manifests/` carry only **placeholders + comments** that point here.
|
|
The live values live in the operator's out-of-band store (password manager / escrow) and in the cluster
|
|
as Secrets created imperatively with the commands below.
|
|
|
|
> The `felhom` ArgoCD app syncs the **entire** `manifests/` dir, so any Secret committed there is
|
|
> re-applied on every sync. Secrets that must stay out of git are therefore created as objects that are
|
|
> **not present in `manifests/`** (ArgoCD does not manage them), e.g. `Secret/resend-api` below.
|
|
|
|
---
|
|
|
|
## Resend API key — `Secret/resend-api`
|
|
|
|
**What uses it (live consumers):**
|
|
|
|
| Consumer | How it reads the key |
|
|
|----------|----------------------|
|
|
| `hub` (`Deployment/hub`) | env `RESEND_API_KEY` ← `secretKeyRef: resend-api/RESEND_API_KEY`; the hub binary's `RESEND_API_KEY` env override fills `notifications.resend_api_key` (the committed ConfigMap field is an empty placeholder). |
|
|
| `contact-mailer` (`Deployment/contact-mailer`) | env `RESEND_API_KEY` ← `secretKeyRef: resend-api/RESEND_API_KEY`. |
|
|
| `healthchecks` (**not currently deployed**) | when deployed, wire `EMAIL_HOST_PASSWORD` to `secretKeyRef: resend-api/RESEND_API_KEY`; do **not** inline the value into `felhom.secret.yaml`. |
|
|
| Gmail "Send mail as" (`info@` / `admin@felhom.eu`) | external — the key is the SMTP password; updated manually in Gmail settings, **not** in the cluster. |
|
|
|
|
**Where the value lives out-of-band:** the operator's password manager, entry "Felhom Resend API key
|
|
(app-relay + alerts)". Send-scoped. **Never** written to a file on a host, the guest, or any commit.
|
|
|
|
### Create / rotate the Secret
|
|
|
|
The key is available as the `RESEND_API` env var in `kisfenyo`'s shell on the build host
|
|
(192.168.0.180). It is exported from `~/.bashrc`, so it is present in an **interactive** shell
|
|
(`bash -ic`) but **not** in a plain non-interactive `ssh host 'cmd'`.
|
|
|
|
Create (or update) the Secret without ever echoing the value — render the Secret YAML in the
|
|
(non-sudo) interactive shell where `$RESEND_API` is set, and pipe it into `sudo kubectl apply`:
|
|
|
|
```bash
|
|
# on 192.168.0.180, as kisfenyo:
|
|
bash -ic 'kubectl create secret generic resend-api -n felhom-system \
|
|
--from-literal=RESEND_API_KEY="$RESEND_API" \
|
|
--dry-run=client -o yaml' | sudo kubectl apply -f -
|
|
```
|
|
|
|
Then roll the consumers so they pick up the new value:
|
|
|
|
```bash
|
|
sudo kubectl -n felhom-system rollout restart deploy/hub deploy/contact-mailer
|
|
sudo kubectl -n felhom-system rollout status deploy/hub deploy/contact-mailer --timeout=120s
|
|
```
|
|
|
|
### Verify a consumer can send
|
|
|
|
- **contact-mailer:** submit the website contact form at <https://felhom.eu> (or, if `DEBUG=true`,
|
|
`curl -X POST https://felhom.eu/api/debug/test`) → mail arrives at `info@felhom.eu`.
|
|
- **hub:** trigger a notification (or the dispatcher test path) → mail arrives.
|
|
- **Gmail:** send a test from `info@felhom.eu` → delivers.
|
|
|
|
### Rotating away from a compromised key (ordered — load-bearing)
|
|
|
|
1. Create the **new** send-scoped key in Resend; store it out-of-band; set `RESEND_API` on host 180.
|
|
2. Run the create-Secret + rollout above with the new value.
|
|
3. Update Gmail "Send mail as" SMTP password to the new key.
|
|
4. **Verify every consumer sends on the new key** (above) — do this **before** step 5.
|
|
5. Only then: **delete the old key** in Resend. Re-verify one consumer still sends afterwards.
|
|
|
|
> Never delete the old key before step 4 passes — a premature delete breaks live mail.
|
|
|
|
---
|
|
|
|
## Operator/global bearer key — `Secret/report-api`
|
|
|
|
The hub API's global bearer (`api.report_api_key`) — the operator's own key (e.g. `felhom-ops
|
|
… -hub-key`), distinct from the per-customer/per-host keys the hub generates itself. It was
|
|
COMMITTED in `manifests/hub.yaml` until v0.53.0 (flagged in the 0.81/0.113 and 0.85/0.120
|
|
publish runbooks, incl. a Phase-D screenshot exposure); the manifest now carries a
|
|
`secretKeyRef` and `scripts/manifest_bearer_gate.py` blocks reintroduction. **The git-history
|
|
copy stays alive until the value is ROTATED** — de-git alone kills nothing.
|
|
|
|
**What uses it (live consumers of the GLOBAL key):**
|
|
|
|
| Consumer | How it reads the key |
|
|
|----------|----------------------|
|
|
| `hub` (`Deployment/hub`) | env `REPORT_API_KEY` ← `secretKeyRef: report-api/REPORT_API_KEY` (v0.53.0 env override fills `api.report_api_key`; the ConfigMap field is an empty placeholder). **Not `optional:`** — a missing Secret fails Ready by design. |
|
|
| Operator tooling (`felhom-ops keys upload -hub-key …`, runbook curl probes in break-glass.md / offsite-endpoint.md) | typed per-invocation from the out-of-band store — nothing machine-persisted. |
|
|
| ~~`felhom-controller` repo `controller.yaml.example`~~ | carried the LITERAL as example text (never a live consumer) — scrubbed 2026-07-13. |
|
|
|
|
Per-customer (`customer_configs.api_key`) and per-host (`hosts.api_key`) keys are hub-generated
|
|
and **unaffected** by a global-key rotation — no customer box breaks.
|
|
|
|
**Where the value lives out-of-band:** the operator's password manager, entry "Felhom hub
|
|
global bearer (report_api_key)".
|
|
|
|
### Create the Secret (pre-deploy for v0.53.0 — same value, no rotation yet)
|
|
|
|
Create it with the CURRENT value **before** syncing the v0.53.0 manifest (the pod refuses to
|
|
start without it). Render on the build host without echoing the value (file-to-file, the
|
|
operator-present rule):
|
|
|
|
```bash
|
|
# on 192.168.0.180, as kisfenyo — put the current key in a 0600 temp file first (no echo):
|
|
kubectl create secret generic report-api -n felhom-system \
|
|
--from-file=REPORT_API_KEY=/dev/stdin < /path/to/keyfile \
|
|
--dry-run=client -o yaml | sudo kubectl apply -f -
|
|
shred -u /path/to/keyfile
|
|
```
|
|
|
|
### Rotation (supervised — operator GO required; ordered, load-bearing)
|
|
|
|
1. Mint the new key into a 0600 file: `openssl rand -hex 32 > keyfile` (no terminal echo).
|
|
2. Re-run the create-Secret pipe above with the new file; store the value out-of-band.
|
|
3. `sudo kubectl -n felhom-system rollout restart deploy/hub && sudo kubectl -n felhom-system rollout status deploy/hub --timeout=120s`
|
|
4. **Verify before declaring the old key dead:**
|
|
- a customer box still reports (per-customer key — proves rotation touched nothing it shouldn't);
|
|
- an operator call with the NEW key succeeds (e.g. an authed `GET /api/v1/…` probe);
|
|
- the SAME call with the OLD key returns 401 — only now is the git-history copy dead.
|
|
5. Update the password-manager entry; note the rotation date in the publish-runbook disposition.
|
|
|
|
---
|
|
|
|
## Other committed secrets (tracked, NOT yet de-gitted — backlog)
|
|
|
|
`manifests/felhom.secret.yaml` still commits other plaintext secrets (`healthchecks-config` `SECRET_KEY`
|
|
/ `SUPERUSER_PASSWORD`, `umami-config`, `gitea-creds`). These are **out of scope** for the Resend
|
|
rotation but are the same hygiene problem; de-git them the same way (out-of-band `Secret/...` +
|
|
placeholder) when touched. Tracked here so the gap is visible.
|