hub + manifests: operator bearer out of git (secretKeyRef report-api)

The global bearer (api.report_api_key) leaves manifests/hub.yaml: the
ConfigMap field is an empty placeholder; the Deployment injects
REPORT_API_KEY from out-of-band Secret/report-api (deliberately NOT
optional - a missing Secret fails Ready rather than booting a hub with an
empty bearer). main.go gains the REPORT_API_KEY env override (the
RESEND_API_KEY twin). scripts/manifest_secret_gate.py blocks bearer-shaped
(64-hex) literals across manifests/ (felhom.secret.yaml's pre-existing
umami APP_SECRET is a visible non-fatal KNOWN-BACKLOG line - secrets.md
backlog, out of scope here).

Ordering (load-bearing): create Secret/report-api with the CURRENT value
BEFORE syncing; this manifest change rides the same sync as the 0.53.0
image (older images ignore the env override). The exposed git-history copy
dies only with the supervised ROTATION - procedure + consumer list in
documentation/runbooks/secrets.md ("Operator/global bearer key"); both
publish runbooks' ROTATION notes now point at that disposition. The
controller repo's example-config copy of the literal is scrubbed
(felhom-controller 51c871a).

Red-proof E: reintroducing a 64-hex literal into manifests/hub.yaml ->
gate exit 1 (recorded); restored clean.
This commit is contained in:
2026-07-13 14:57:09 +02:00
parent 36c72138f1
commit 80aca3a3a1
6 changed files with 87 additions and 15 deletions
@@ -115,6 +115,9 @@ rollout OK, live env verified 0.113.0, clean hub startup log. Demo: no churn (al
one of the three known committed secrets on the rotate+de-git backlog) and was additionally
exposed in a session screenshot during Phase D. Rotate it + move to a `secretKeyRef` (the
Resend-key pattern, documentation/runbooks/secrets.md) as the next operator hygiene task.
**DISPOSITION 2026-07-13 (hub v0.53.0):** de-git DONE (`secretKeyRef: report-api/REPORT_API_KEY`
+ `scripts/manifest_secret_gate.py`); the value rotation is the supervised procedure in
documentation/runbooks/secrets.md §"Operator/global bearer key".
## END STATE
@@ -92,9 +92,13 @@ floor lands his controller at 0.120 automatically.
- No key material or tokens in any transcript; publish creds stayed as `$(cat ~/.gitea-token)`
env-substitution on 180; drill-VM token shredded; saved-log leak grep 0.
- **⚠ ROTATION STILL DUE:** the hub operator bearer key is COMMITTED in git (`manifests/hub.yaml`) —
- ~~**⚠ ROTATION STILL DUE:** the hub operator bearer key is COMMITTED in git (`manifests/hub.yaml`) —
carried over from the 0.81/0.113 train. Rotate + move to a `secretKeyRef` (Resend-key pattern,
documentation/runbooks/secrets.md) as an operator hygiene task.
documentation/runbooks/secrets.md) as an operator hygiene task.~~
**DISPOSITION 2026-07-13 (hub v0.53.0):** moved to `secretKeyRef: report-api/REPORT_API_KEY`
(literal removed from the manifest; `scripts/manifest_secret_gate.py` blocks reintroduction).
The value ROTATION itself is the supervised step in documentation/runbooks/secrets.md
§"Operator/global bearer key" — the git-history copy is dead only after it runs.
## END STATE
+50
View File
@@ -67,6 +67,56 @@ sudo kubectl -n felhom-system rollout status deploy/hub deploy/contact-mailer -
---
## Operator/global bearer key — `Secret/report-api`
The hub API's global bearer (`api.report_api_key`) — the operator's own key (e.g. `felhom-ops
… -hub-key`), distinct from the per-customer/per-host keys the hub generates itself. It was
COMMITTED in `manifests/hub.yaml` until v0.53.0 (flagged in the 0.81/0.113 and 0.85/0.120
publish runbooks, incl. a Phase-D screenshot exposure); the manifest now carries a
`secretKeyRef` and `scripts/manifest_secret_gate.py` blocks reintroduction. **The git-history
copy stays alive until the value is ROTATED** — de-git alone kills nothing.
**What uses it (live consumers of the GLOBAL key):**
| Consumer | How it reads the key |
|----------|----------------------|
| `hub` (`Deployment/hub`) | env `REPORT_API_KEY``secretKeyRef: report-api/REPORT_API_KEY` (v0.53.0 env override fills `api.report_api_key`; the ConfigMap field is an empty placeholder). **Not `optional:`** — a missing Secret fails Ready by design. |
| Operator tooling (`felhom-ops keys upload -hub-key …`, runbook curl probes in break-glass.md / offsite-endpoint.md) | typed per-invocation from the out-of-band store — nothing machine-persisted. |
| ~~`felhom-controller` repo `controller.yaml.example`~~ | carried the LITERAL as example text (never a live consumer) — scrubbed 2026-07-13. |
Per-customer (`customer_configs.api_key`) and per-host (`hosts.api_key`) keys are hub-generated
and **unaffected** by a global-key rotation — no customer box breaks.
**Where the value lives out-of-band:** the operator's password manager, entry "Felhom hub
global bearer (report_api_key)".
### Create the Secret (pre-deploy for v0.53.0 — same value, no rotation yet)
Create it with the CURRENT value **before** syncing the v0.53.0 manifest (the pod refuses to
start without it). Render on the build host without echoing the value (file-to-file, the
operator-present rule):
```bash
# on 192.168.0.180, as kisfenyo — put the current key in a 0600 temp file first (no echo):
kubectl create secret generic report-api -n felhom-system \
--from-file=REPORT_API_KEY=/dev/stdin < /path/to/keyfile \
--dry-run=client -o yaml | sudo kubectl apply -f -
shred -u /path/to/keyfile
```
### Rotation (supervised — operator GO required; ordered, load-bearing)
1. Mint the new key into a 0600 file: `openssl rand -hex 32 > keyfile` (no terminal echo).
2. Re-run the create-Secret pipe above with the new file; store the value out-of-band.
3. `sudo kubectl -n felhom-system rollout restart deploy/hub && sudo kubectl -n felhom-system rollout status deploy/hub --timeout=120s`
4. **Verify before declaring the old key dead:**
- a customer box still reports (per-customer key — proves rotation touched nothing it shouldn't);
- an operator call with the NEW key succeeds (e.g. an authed `GET /api/v1/…` probe);
- the SAME call with the OLD key returns 401 — only now is the git-history copy dead.
5. Update the password-manager entry; note the rotation date in the publish-runbook disposition.
---
## Other committed secrets (tracked, NOT yet de-gitted — backlog)
`manifests/felhom.secret.yaml` still commits other plaintext secrets (`healthchecks-config` `SECRET_KEY`