276 lines
16 KiB
Markdown
276 lines
16 KiB
Markdown
# Runbook — out-of-band secrets (felhom-system)
|
||
|
||
**Model:** out-of-band `kubectl` + this runbook (DECISION model **A**). Secret *values* are **never**
|
||
committed to git. The manifests in `manifests/` carry only **placeholders + comments** that point here.
|
||
The live values live in the operator's out-of-band store (password manager / escrow) and in the cluster
|
||
as Secrets created imperatively with the commands below.
|
||
|
||
> **Corrected 2026-10-09 (R-923):** the password manager is Vaultwarden, which **runs on DooPlex** — it is not out of
|
||
> band for a loss of DooPlex. Every Secret below is also in DooPlex's nightly Secrets export, which since 2026-10-09 rides
|
||
> the encrypted off-site copy; the keys that open it are on the break-glass sheet (`break-glass-sheet.md`,
|
||
> `total-loss-of-dooplex.md`).
|
||
|
||
> The `felhom` ArgoCD app syncs the **entire** `manifests/` dir, so any Secret committed there is
|
||
> re-applied on every sync. Secrets that must stay out of git are therefore created as objects that are
|
||
> **not present in `manifests/`** (ArgoCD does not manage them), e.g. `Secret/resend-api` below.
|
||
|
||
---
|
||
|
||
## Resend API key — `Secret/resend-api`
|
||
|
||
**What uses it (live consumers):**
|
||
|
||
| Consumer | How it reads the key |
|
||
|----------|----------------------|
|
||
| `hub` (`Deployment/hub`) | env `RESEND_API_KEY` ← `secretKeyRef: resend-api/RESEND_API_KEY`; the hub binary's `RESEND_API_KEY` env override fills `notifications.resend_api_key` (the committed ConfigMap field is an empty placeholder). |
|
||
| `contact-mailer` (`Deployment/contact-mailer`) | env `RESEND_API_KEY` ← `secretKeyRef: resend-api/RESEND_API_KEY`. |
|
||
| `healthchecks` (**not currently deployed**) | when deployed, wire `EMAIL_HOST_PASSWORD` to `secretKeyRef: resend-api/RESEND_API_KEY`; do **not** inline the value into `felhom.secret.yaml`. |
|
||
| Gmail "Send mail as" (`info@` / `admin@felhom.eu`) | external — the key is the SMTP password; updated manually in Gmail settings, **not** in the cluster. |
|
||
|
||
**Where the value lives out-of-band:** the operator's password manager, entry "Felhom Resend API key
|
||
(app-relay + alerts)". Send-scoped. **Never** written to a file on a host, the guest, or any commit.
|
||
|
||
### Create / rotate the Secret
|
||
|
||
The key is available as the `RESEND_API` env var in `kisfenyo`'s shell on the build host
|
||
(192.168.0.180). It is exported from `~/.bashrc`, so it is present in an **interactive** shell
|
||
(`bash -ic`) but **not** in a plain non-interactive `ssh host 'cmd'`.
|
||
|
||
Create (or update) the Secret without ever echoing the value — render the Secret YAML in the
|
||
(non-sudo) interactive shell where `$RESEND_API` is set, and pipe it into `sudo kubectl apply`:
|
||
|
||
```bash
|
||
# on 192.168.0.180, as kisfenyo:
|
||
bash -ic 'kubectl create secret generic resend-api -n felhom-system \
|
||
--from-literal=RESEND_API_KEY="$RESEND_API" \
|
||
--dry-run=client -o yaml' | sudo kubectl apply -f -
|
||
```
|
||
|
||
Then roll the consumers so they pick up the new value:
|
||
|
||
```bash
|
||
sudo kubectl -n felhom-system rollout restart deploy/hub deploy/contact-mailer
|
||
sudo kubectl -n felhom-system rollout status deploy/hub deploy/contact-mailer --timeout=120s
|
||
```
|
||
|
||
### Verify a consumer can send
|
||
|
||
- **contact-mailer:** submit the website contact form at <https://felhom.eu> (or, if `DEBUG=true`,
|
||
`curl -X POST https://felhom.eu/api/debug/test`) → mail arrives at `info@felhom.eu`.
|
||
- **hub:** trigger a notification (or the dispatcher test path) → mail arrives.
|
||
- **Gmail:** send a test from `info@felhom.eu` → delivers.
|
||
|
||
### Rotating away from a compromised key (ordered — load-bearing)
|
||
|
||
1. Create the **new** send-scoped key in Resend; store it out-of-band; set `RESEND_API` on host 180.
|
||
2. Run the create-Secret + rollout above with the new value.
|
||
3. Update Gmail "Send mail as" SMTP password to the new key.
|
||
4. **Verify every consumer sends on the new key** (above) — do this **before** step 5.
|
||
5. Only then: **delete the old key** in Resend. Re-verify one consumer still sends afterwards.
|
||
|
||
> Never delete the old key before step 4 passes — a premature delete breaks live mail.
|
||
|
||
---
|
||
|
||
## Operator/global bearer key — `Secret/report-api`
|
||
|
||
The hub API's global bearer (`api.report_api_key`) — the operator's own key (e.g. `felhom-ops
|
||
… -hub-key`), distinct from the per-customer/per-host keys the hub generates itself. It was
|
||
COMMITTED in `manifests/hub.yaml` until v0.53.0 (flagged in the 0.81/0.113 and 0.85/0.120
|
||
publish runbooks, incl. a Phase-D screenshot exposure); the manifest now carries a
|
||
`secretKeyRef` and `scripts/manifest_bearer_gate.py` blocks reintroduction. **The git-history
|
||
copy stays alive until the value is ROTATED** — de-git alone kills nothing.
|
||
|
||
**What uses it (live consumers of the GLOBAL key):**
|
||
|
||
| Consumer | How it reads the key |
|
||
|----------|----------------------|
|
||
| `hub` (`Deployment/hub`) | env `REPORT_API_KEY` ← `secretKeyRef: report-api/REPORT_API_KEY` (v0.53.0 env override fills `api.report_api_key`; the ConfigMap field is an empty placeholder). **Not `optional:`** — a missing Secret fails Ready by design. |
|
||
| Operator tooling (`felhom-ops keys upload -hub-key …`, runbook curl probes in break-glass.md / offsite-endpoint.md) | typed per-invocation from the out-of-band store — nothing machine-persisted. |
|
||
| ~~`felhom-controller` repo `controller.yaml.example`~~ | carried the LITERAL as example text (never a live consumer) — scrubbed 2026-07-13. |
|
||
|
||
Per-customer (`customer_configs.api_key`) and per-host (`hosts.api_key`) keys are hub-generated
|
||
and **unaffected** by a global-key rotation — no customer box breaks.
|
||
|
||
**Where the value lives out-of-band:** the operator's password manager, entry "Felhom hub
|
||
global bearer (report_api_key)".
|
||
|
||
### Create the Secret (pre-deploy for v0.53.0 — same value, no rotation yet)
|
||
|
||
Create it with the CURRENT value **before** syncing the v0.53.0 manifest (the pod refuses to
|
||
start without it). Render on the build host without echoing the value (file-to-file, the
|
||
operator-present rule):
|
||
|
||
```bash
|
||
# on 192.168.0.180, as kisfenyo — put the current key in a 0600 temp file first (no echo):
|
||
kubectl create secret generic report-api -n felhom-system \
|
||
--from-file=REPORT_API_KEY=/dev/stdin < /path/to/keyfile \
|
||
--dry-run=client -o yaml | sudo kubectl apply -f -
|
||
shred -u /path/to/keyfile
|
||
```
|
||
|
||
### Rotation (supervised — operator GO required; ordered, load-bearing)
|
||
|
||
1. Mint the new key into a 0600 file: `openssl rand -hex 32 > keyfile` (no terminal echo).
|
||
2. Re-run the create-Secret pipe above with the new file; store the value out-of-band.
|
||
3. `sudo kubectl -n felhom-system rollout restart deploy/hub && sudo kubectl -n felhom-system rollout status deploy/hub --timeout=120s`
|
||
4. **Verify before declaring the old key dead:**
|
||
- a customer box still reports (per-customer key — proves rotation touched nothing it shouldn't);
|
||
- an operator call with the NEW key succeeds (e.g. an authed `GET /api/v1/…` probe);
|
||
- the SAME call with the OLD key returns 401 — only now is the git-history copy dead.
|
||
5. Update the password-manager entry; note the rotation date in the publish-runbook disposition.
|
||
|
||
**Rotated 2026-10-12** (operator GO; security review, R-950): steps 1–4 done. Check after the restart: the new
|
||
key answered 200 and the old key 403/401 — the old value is dead, including every copy in git history. All three boxes
|
||
reported normally. Step 5 (the password manager) is the operator's.
|
||
|
||
---
|
||
|
||
## Off-site password sealing key — `Secret/offsite-secret-key` (hub v0.127.0, decision 69, R-821)
|
||
|
||
**What uses it:** `hub` env `OFFSITE_SECRET_KEY` (64 hex characters = 32 bytes). It seals every Storage Box
|
||
sub-account password in `one_time_secrets.value`; the hub's key registrar opens them to write box keys.
|
||
**Required** — the pod does not start without it.
|
||
|
||
**Create (once, before the first v0.127.0 sync) — the value never touches a file or the terminal:**
|
||
|
||
```bash
|
||
sudo kubectl -n felhom-system create secret generic offsite-secret-key \
|
||
--from-literal=OFFSITE_SECRET_KEY="$(openssl rand -hex 32)"
|
||
```
|
||
|
||
**If it is lost:** the sealed passwords cannot be opened. Nothing on the boxes breaks (their keys are installed);
|
||
the registrar and the daily check fail with `offsite_key_audit_failed`. Recover per customer with the hub's
|
||
**Re-issue offsite credentials** button (the provider resets the password; the hub seals the new one). Keep a copy
|
||
in the operator's password manager if a Re-issue round is not acceptable.
|
||
|
||
**Rotation:** not built. A new key cannot open the old rows; rotate by setting the new key and pressing Re-issue
|
||
for every off-site customer.
|
||
|
||
---
|
||
|
||
## DooPlex PBS — the ep0 copy (decision 70, 2026-10-03)
|
||
|
||
Not k8s Secrets — PBS's own private files on DooPlex, never in git:
|
||
|
||
| File | Holds | Created |
|
||
|---|---|---|
|
||
| `/etc/proxmox-backup/remote.cfg` (root:backup 0640) | ep0 API token `root@pam!dooplex-sync` (base64, PBS format) | from the token's one-time output, file → file |
|
||
| `/etc/proxmox-backup/notifications-priv.cfg` (root:root 0600) | the Resend API key, as the SMTP password of target `felhom-operator` | from `$RESEND_API`, file → file |
|
||
|
||
**Rotating the Resend key (§ above) must also rewrite `notifications-priv.cfg`**, or the ep0-copy failure mails stop
|
||
silently. Rotate the ep0 token with `proxmox-backup-manager user generate-token` on ep0 (delete the old) and rewrite
|
||
`remote.cfg`. See `runbooks/ep0-datastore-copy.md`.
|
||
|
||
---
|
||
|
||
## `felhom.secret.yaml` — ROTATED AND DE-GITTED 2026-10-09 (R-925)
|
||
|
||
**What was wrong.** `manifests/felhom.secret.yaml` committed three Secrets in plaintext, and
|
||
`gitea.dooplex.hu` serves this repo to **anyone on the internet with no login** (measured from
|
||
off-network: a real path returns the file, a nonsense path returns 404). Every committed value was
|
||
still live — nothing had ever been rotated. Worst of all, `gitea-creds` was the **Gitea `admin`
|
||
account password** (`is_admin: true`, `/api/v1/admin/users` answered 200), and the *same string* was
|
||
reused as the admin password in **15 cluster Secrets** across the homelab.
|
||
|
||
**What was done, in this order** (the order matters — de-gitting alone kills nothing, the history
|
||
keeps the value):
|
||
|
||
1. **`umami-config`** — new random `APP_SECRET` and `POSTGRES_PASSWORD`. The password was changed
|
||
**inside Postgres** (`ALTER USER umami WITH PASSWORD`) as well as in the Secret; the env var alone
|
||
would not have changed it, because it is only read when the database is first initialised.
|
||
2. **`healthchecks-config`** — new random `SECRET_KEY` and `SUPERUSER_PASSWORD`. Nothing consumes
|
||
them (there is no healthchecks Deployment), so this was free.
|
||
3. **`gitea-creds`** — the hub no longer holds the admin password at all. It now holds a **scoped
|
||
Gitea access token** (`read:package` + `read:repository`). Both scopes are required and were each
|
||
verified against their real endpoint *before* the swap:
|
||
`GET /v2/admin/felhom-controller/tags/list` (the registry version checker) and
|
||
`GET /admin/felhom-controller/raw/branch/main/controller/configs/controller.yaml.example`
|
||
(the template fetcher — a `read:package`-only token gets **403** here, which is how the missing
|
||
scope was found).
|
||
4. **The Gitea `admin` password** was changed to a new random value and
|
||
`gitea-system/gitea-admin` updated. Verified: the new password returns **200**, the old published
|
||
one returns **401**.
|
||
|
||
**The values** were written to a 0600 file on DooPlex (`~/rotated-secrets-2026-10-09.txt`) and are to
|
||
be moved into the password manager and the file deleted. They were never echoed to a terminal.
|
||
|
||
**The file is gone from git** and `.gitignore`'s `*secret*` rule now applies to it (it never did
|
||
before: **`.gitignore` is not consulted for a file git already tracks**, which is why the rule looked
|
||
broken). `scripts/manifest_bearer_gate.py`'s `KNOWN_BACKLOG` carve-out was removed in the same
|
||
commit, so a reintroduction now fails the gate.
|
||
|
||
### Recreating these Secrets (they are no longer in `manifests/`)
|
||
|
||
ArgoCD does not manage them any more, exactly like `Secret/resend-api` above. The `felhom` app has
|
||
`automated.enabled: false` and no prune, so removing the file does **not** delete them. To recreate
|
||
one from the password manager, on 192.168.0.180 as `kisfenyo`, writing the value to a 0600 temp file
|
||
first so it is never echoed:
|
||
|
||
```bash
|
||
# umami-config: APP_SECRET + POSTGRES_PASSWORD.
|
||
# If POSTGRES_PASSWORD changes, ALTER USER in the database too, or umami cannot connect:
|
||
# kubectl exec -n felhom-system deploy/umami-db -- psql -U umami -d umami \
|
||
# -c "ALTER USER umami WITH PASSWORD '<new>';"
|
||
sudo kubectl create secret generic umami-config -n felhom-system \
|
||
--from-file=APP_SECRET=/path/app_secret --from-file=POSTGRES_PASSWORD=/path/pg_pw \
|
||
--dry-run=client -o yaml | sudo kubectl apply -f -
|
||
|
||
# gitea-creds: username=admin, password = a SCOPED TOKEN (read:package + read:repository),
|
||
# never the admin account password.
|
||
sudo kubectl create secret generic gitea-creds -n felhom-system \
|
||
--from-literal=username=admin --from-file=password=/path/token \
|
||
--dry-run=client -o yaml | sudo kubectl apply -f -
|
||
|
||
# healthchecks-config: only needed if healthchecks is ever deployed; wire EMAIL_HOST_PASSWORD to
|
||
# secretKeyRef resend-api/RESEND_API_KEY rather than inlining it (see the Resend section above).
|
||
```
|
||
|
||
### Still owed — the same password opens 12 more services (operator's, 2026-10-09 ruling)
|
||
|
||
Rotating Gitea does **not** touch these: each is its own login and each is still the exact string that
|
||
was published. The operator chose to do these himself; CC's scope stopped at the Felhom boundary.
|
||
|
||
**Measured 2026-10-09 after the Felhom rotation** (hash comparison against the value still served from
|
||
git history — the three CC rotated are confirmed absent from this list):
|
||
|
||
```
|
||
[ ] adventurelog-system adventurelog-admin password
|
||
[ ] bookstack-system bookstack-db root-password <-- DATABASE root, do first
|
||
[ ] calibre-system calibre-auth password
|
||
[ ] fileshare-system gokapi-app admin-password
|
||
[ ] homepage-system homepage-secrets calibreweb-pass
|
||
[ ] homepage-system homepage-secrets qbittorrent-pass
|
||
[ ] mediaserver-system qbittorrent-admin password
|
||
[ ] nextcloud-system nextcloud nextcloud-password
|
||
[ ] paperless-system paperless-admin password
|
||
[ ] servarr-system download-client-credentials qbittorrent-password
|
||
[ ] servarr-system servarr-credentials password
|
||
[ ] tandoor-system tandoor-admin password
|
||
```
|
||
|
||
**These are not LAN-only.** `nextcloud`, `paperless`, `bookstack`, `tandoor`, `calibre`,
|
||
`adventurelog`, `fileshare`, `plex`, the whole `servarr` set and `homepage` all have
|
||
`*.dooplex.hu` ingresses; spot-checked in **public DNS** — `nextcloud/paperless/bookstack/qbittorrent
|
||
.dooplex.hu` all resolve to the public address and answer HTTPS with a login page. No login was
|
||
attempted; reachability is the point.
|
||
|
||
**Two traps when rotating these**, both learned on the Felhom side the same day:
|
||
|
||
1. **A database password is not changed by editing the Secret.** `bookstack-db/root-password` is read
|
||
when the database is first initialised; afterwards it lives in the database. Change it *in the
|
||
engine* and in the Secret, then restart — exactly as `umami-config/POSTGRES_PASSWORD` had to be.
|
||
2. **Check the app can still RESTART before you trust it.** Patching a Secret does nothing until the
|
||
pod restarts, and a pod that has run for months may not come back: umami ran 124 days at
|
||
`512Mi` but was OOMKilled on every restart attempt. Rotate when you can watch it.
|
||
|
||
### Repository visibility — DECIDED 2026-10-09: stays public for now
|
||
|
||
The operator ruled that `felhom.eu` stays anonymously readable for the moment. Making it private
|
||
breaks two live paths that clone it with **no credentials**: the website git-sync in
|
||
`manifests/webpage.yaml`, and the installer fetched from the `installer-v…` tag by every new box
|
||
(R-110). Both would have to be given credentials first. The standing consequence of that ruling:
|
||
**no secret may ever enter this repo again** — which `manifest_bearer_gate.py` now enforces with no
|
||
exemption. If the decision is ever reversed, the 32 `<!-- source: … -->` comments served on
|
||
`/adatkezeles` must be stripped in the same change, because they are a map of the repo.
|