From 80aca3a3a1a4746a0d9988d3d65490fe4ffc65ae Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 13 Jul 2026 14:57:09 +0200 Subject: [PATCH] hub + manifests: operator bearer out of git (secretKeyRef report-api) The global bearer (api.report_api_key) leaves manifests/hub.yaml: the ConfigMap field is an empty placeholder; the Deployment injects REPORT_API_KEY from out-of-band Secret/report-api (deliberately NOT optional - a missing Secret fails Ready rather than booting a hub with an empty bearer). main.go gains the REPORT_API_KEY env override (the RESEND_API_KEY twin). scripts/manifest_secret_gate.py blocks bearer-shaped (64-hex) literals across manifests/ (felhom.secret.yaml's pre-existing umami APP_SECRET is a visible non-fatal KNOWN-BACKLOG line - secrets.md backlog, out of scope here). Ordering (load-bearing): create Secret/report-api with the CURRENT value BEFORE syncing; this manifest change rides the same sync as the 0.53.0 image (older images ignore the env override). The exposed git-history copy dies only with the supervised ROTATION - procedure + consumer list in documentation/runbooks/secrets.md ("Operator/global bearer key"); both publish runbooks' ROTATION notes now point at that disposition. The controller repo's example-config copy of the literal is scrubbed (felhom-controller 51c871a). Red-proof E: reintroducing a 64-hex literal into manifests/hub.yaml -> gate exit 1 (recorded); restored clean. --- REUSE.md | 2 +- .../RUNBOOK-publish-0.81-0.113-2026-07-11.md | 3 ++ .../RUNBOOK-publish-0.85-0.120-2026-07-12.md | 8 ++- documentation/runbooks/secrets.md | 50 +++++++++++++++++++ hub/cmd/hub/main.go | 7 +++ manifests/hub.yaml | 32 +++++++----- 6 files changed, 87 insertions(+), 15 deletions(-) diff --git a/REUSE.md b/REUSE.md index 9812e60..479daf2 100644 --- a/REUSE.md +++ b/REUSE.md @@ -132,7 +132,7 @@ | Install-profile gate (shell) | scripts/felhom-host-install.sh `--mode appliance\|byo` (GL-2, v1.10.0) | Mandatory-flag profile (no default), refusals at argv time BEFORE any prompt/step, risky step gated at its CALL SITE (one auditable place — never a branch inside the step), mode persisted to state.json + resume-mismatch refusal, `FELHOM_INSTALL_STATE_DIR` override for harness isolation. Harness: scripts/hostinstall-mode-harness.sh (static refusal matrix + grep-invariants + PVE dry-transcript tier; red-proofs run against a mutated scratch copy). | | Disclosure↔uninstall parity (shell) | scripts/felhom-host-install.sh `_uninstall_statement` + harness GL4-D (v1.11.0) | Every host artifact the byo disclosure names must be removed OR explicitly listed KEPT by `run_uninstall`; the harness greps the parity (token list). New install-time artifact ⇒ add its removal + disclosure line + parity token in the SAME commit. Drive data rule: plain `umount` only, never `-l`/`-f`, never any format op under /mnt/felhom-drives. | | Website deploy (manifest) | manifests/webpage.yaml | git-sync sidecar (sparse-checkout `/website/` + `/scripts/`, `--link=current`) + init container waits for first sync; nginx serves `current/website`; push to main = deployed, no image build. | -| Secret handling (manifest) | manifests/hub.yaml (env, ~L142) | Secrets via `secretKeyRef` to OUT-OF-BAND secrets created per documentation/runbooks/secrets.md — never inline stringData (see §3). ERRATA (2026-07-03): only `resend-api` is truly out-of-band today; `gitea-creds` is COMMITTED in manifests/felhom.secret.yaml AND live-consumed by hub.yaml — rotation + de-git is a pending operator task (spike SPIKE-a1 appendix). | +| Secret handling (manifest) | manifests/hub.yaml (env, ~L142) | Secrets via `secretKeyRef` to OUT-OF-BAND secrets created per documentation/runbooks/secrets.md — never inline stringData (see §3). `report-api` (the operator bearer, v0.53.0) is deliberately NOT `optional:` — a missing Secret fails Ready instead of booting an unauthenticatable hub. `scripts/manifest_secret_gate.py` (run after ANY manifests/ change) blocks bearer-shaped (64-hex) literals. ERRATA (2026-07-03): `gitea-creds` is COMMITTED in manifests/felhom.secret.yaml AND live-consumed by hub.yaml — rotation + de-git is a pending operator task (spike SPIKE-a1 appendix). | | Hub deploy (GitOps) | manifests/hub.yaml `image:` (~L129) | Pinned explicit tag, bumped in git, deliberate ArgoCD sync (auto-sync OFF). Code push alone deploys nothing. | ## 3. Dangerous lookalikes — do NOT reuse diff --git a/documentation/pilot/RUNBOOK-publish-0.81-0.113-2026-07-11.md b/documentation/pilot/RUNBOOK-publish-0.81-0.113-2026-07-11.md index d477596..3e45387 100644 --- a/documentation/pilot/RUNBOOK-publish-0.81-0.113-2026-07-11.md +++ b/documentation/pilot/RUNBOOK-publish-0.81-0.113-2026-07-11.md @@ -115,6 +115,9 @@ rollout OK, live env verified 0.113.0, clean hub startup log. Demo: no churn (al one of the three known committed secrets on the rotate+de-git backlog) and was additionally exposed in a session screenshot during Phase D. Rotate it + move to a `secretKeyRef` (the Resend-key pattern, documentation/runbooks/secrets.md) as the next operator hygiene task. + **DISPOSITION 2026-07-13 (hub v0.53.0):** de-git DONE (`secretKeyRef: report-api/REPORT_API_KEY` + + `scripts/manifest_secret_gate.py`); the value rotation is the supervised procedure in + documentation/runbooks/secrets.md §"Operator/global bearer key". ## END STATE diff --git a/documentation/pilot/RUNBOOK-publish-0.85-0.120-2026-07-12.md b/documentation/pilot/RUNBOOK-publish-0.85-0.120-2026-07-12.md index 099c00f..1706474 100644 --- a/documentation/pilot/RUNBOOK-publish-0.85-0.120-2026-07-12.md +++ b/documentation/pilot/RUNBOOK-publish-0.85-0.120-2026-07-12.md @@ -92,9 +92,13 @@ floor lands his controller at 0.120 automatically. - No key material or tokens in any transcript; publish creds stayed as `$(cat ~/.gitea-token)` env-substitution on 180; drill-VM token shredded; saved-log leak grep 0. -- **⚠ ROTATION STILL DUE:** the hub operator bearer key is COMMITTED in git (`manifests/hub.yaml`) — +- ~~**⚠ ROTATION STILL DUE:** the hub operator bearer key is COMMITTED in git (`manifests/hub.yaml`) — carried over from the 0.81/0.113 train. Rotate + move to a `secretKeyRef` (Resend-key pattern, - documentation/runbooks/secrets.md) as an operator hygiene task. + documentation/runbooks/secrets.md) as an operator hygiene task.~~ + **DISPOSITION 2026-07-13 (hub v0.53.0):** moved to `secretKeyRef: report-api/REPORT_API_KEY` + (literal removed from the manifest; `scripts/manifest_secret_gate.py` blocks reintroduction). + The value ROTATION itself is the supervised step in documentation/runbooks/secrets.md + §"Operator/global bearer key" — the git-history copy is dead only after it runs. ## END STATE diff --git a/documentation/runbooks/secrets.md b/documentation/runbooks/secrets.md index 9292ed1..a9dd3ba 100644 --- a/documentation/runbooks/secrets.md +++ b/documentation/runbooks/secrets.md @@ -67,6 +67,56 @@ sudo kubectl -n felhom-system rollout status deploy/hub deploy/contact-mailer - --- +## Operator/global bearer key — `Secret/report-api` + +The hub API's global bearer (`api.report_api_key`) — the operator's own key (e.g. `felhom-ops +… -hub-key`), distinct from the per-customer/per-host keys the hub generates itself. It was +COMMITTED in `manifests/hub.yaml` until v0.53.0 (flagged in the 0.81/0.113 and 0.85/0.120 +publish runbooks, incl. a Phase-D screenshot exposure); the manifest now carries a +`secretKeyRef` and `scripts/manifest_secret_gate.py` blocks reintroduction. **The git-history +copy stays alive until the value is ROTATED** — de-git alone kills nothing. + +**What uses it (live consumers of the GLOBAL key):** + +| Consumer | How it reads the key | +|----------|----------------------| +| `hub` (`Deployment/hub`) | env `REPORT_API_KEY` ← `secretKeyRef: report-api/REPORT_API_KEY` (v0.53.0 env override fills `api.report_api_key`; the ConfigMap field is an empty placeholder). **Not `optional:`** — a missing Secret fails Ready by design. | +| Operator tooling (`felhom-ops keys upload -hub-key …`, runbook curl probes in break-glass.md / offsite-endpoint.md) | typed per-invocation from the out-of-band store — nothing machine-persisted. | +| ~~`felhom-controller` repo `controller.yaml.example`~~ | carried the LITERAL as example text (never a live consumer) — scrubbed 2026-07-13. | + +Per-customer (`customer_configs.api_key`) and per-host (`hosts.api_key`) keys are hub-generated +and **unaffected** by a global-key rotation — no customer box breaks. + +**Where the value lives out-of-band:** the operator's password manager, entry "Felhom hub +global bearer (report_api_key)". + +### Create the Secret (pre-deploy for v0.53.0 — same value, no rotation yet) + +Create it with the CURRENT value **before** syncing the v0.53.0 manifest (the pod refuses to +start without it). Render on the build host without echoing the value (file-to-file, the +operator-present rule): + +```bash +# on 192.168.0.180, as kisfenyo — put the current key in a 0600 temp file first (no echo): +kubectl create secret generic report-api -n felhom-system \ + --from-file=REPORT_API_KEY=/dev/stdin < /path/to/keyfile \ + --dry-run=client -o yaml | sudo kubectl apply -f - +shred -u /path/to/keyfile +``` + +### Rotation (supervised — operator GO required; ordered, load-bearing) + +1. Mint the new key into a 0600 file: `openssl rand -hex 32 > keyfile` (no terminal echo). +2. Re-run the create-Secret pipe above with the new file; store the value out-of-band. +3. `sudo kubectl -n felhom-system rollout restart deploy/hub && sudo kubectl -n felhom-system rollout status deploy/hub --timeout=120s` +4. **Verify before declaring the old key dead:** + - a customer box still reports (per-customer key — proves rotation touched nothing it shouldn't); + - an operator call with the NEW key succeeds (e.g. an authed `GET /api/v1/…` probe); + - the SAME call with the OLD key returns 401 — only now is the git-history copy dead. +5. Update the password-manager entry; note the rotation date in the publish-runbook disposition. + +--- + ## Other committed secrets (tracked, NOT yet de-gitted — backlog) `manifests/felhom.secret.yaml` still commits other plaintext secrets (`healthchecks-config` `SECRET_KEY` diff --git a/hub/cmd/hub/main.go b/hub/cmd/hub/main.go index 701b6de..1148b88 100644 --- a/hub/cmd/hub/main.go +++ b/hub/cmd/hub/main.go @@ -140,6 +140,13 @@ func main() { if v := os.Getenv("RESEND_API_KEY"); v != "" { cfg.Notifications.ResendAPIKey = v } + // The operator/global bearer key (api.report_api_key) is sourced from Secret/report-api + // (env REPORT_API_KEY) since v0.53.0 — the ConfigMap field is an empty placeholder (the + // previously-committed literal is dead once rotated; see the publish-runbook ROTATION + // notes). Same pattern as RESEND_API_KEY above. + if v := os.Getenv("REPORT_API_KEY"); v != "" { + cfg.API.ReportAPIKey = v + } // Ensure data dir exists os.MkdirAll(cfg.Server.DataDir, 0755) diff --git a/manifests/hub.yaml b/manifests/hub.yaml index 61f8ed9..4e10452 100644 --- a/manifests/hub.yaml +++ b/manifests/hub.yaml @@ -15,14 +15,12 @@ # htpasswd -nbBC 10 "" "your-password" | cut -d: -f2 # Update the ConfigMap password_hash field below. # -# 3. Generate a report API key (shared secret for controllers): -# openssl rand -hex 32 -# Update the ConfigMap report_api_key field below. -# Then add the same key to each customer's controller.yaml: -# hub: -# enabled: true -# url: "https://hub.felhom.eu" -# api_key: "" +# 3. Create the operator/global bearer key Secret (out-of-band, NEVER committed): +# openssl rand -hex 32 # mint +# kubectl -n felhom-system create secret generic report-api \ +# --from-literal=REPORT_API_KEY= +# (Customer boxes use per-customer/per-host keys generated by the hub — the global +# key is the operator's own, e.g. felhom-ops -hub-key.) # # 4. Apply this manifest: # kubectl apply -f manifests/hub.yaml @@ -71,10 +69,11 @@ data: # Generate: htpasswd -nbBC 10 "" "your-password" | cut -d: -f2 password_hash: "$2y$10$N5.O9jBnc.1tIlJT/irx3OlVjJQemlCHRnfqIJg/EyZofnzXSCpeG" api: - # Shared secret for controller → hub report push - # Generate: openssl rand -hex 32 - # Must match hub.api_key in each customer's controller.yaml - report_api_key: "094091de545ce28795c47ac2158fc30750db5c24a621c49329b001ee8db57fb8" + # Operator/global bearer key. NOT stored here since v0.53.0 — injected at runtime from + # Secret/report-api via the REPORT_API_KEY env var (see Deployment below). The Secret is + # created out-of-band and NOT committed (documentation/runbooks/secrets.md); the previously + # committed literal is retired by ROTATION (see the publish-runbook notes). Leave empty. + report_api_key: "" retention: max_days: 90 prune_schedule: "04:30" @@ -144,6 +143,15 @@ spec: secretKeyRef: name: resend-api key: RESEND_API_KEY + # Operator/global bearer key — injected from the out-of-band Secret/report-api + # (NOT committed; documentation/runbooks/secrets.md). Deliberately NOT optional: + # a missing Secret must fail the pod Ready rather than boot an unauthenticatable + # hub with an empty bearer key. Create the Secret BEFORE syncing this manifest. + - name: REPORT_API_KEY + valueFrom: + secretKeyRef: + name: report-api + key: REPORT_API_KEY - name: REGISTRY_USERNAME valueFrom: secretKeyRef: