hub + manifests: operator bearer out of git (secretKeyRef report-api)

The global bearer (api.report_api_key) leaves manifests/hub.yaml: the
ConfigMap field is an empty placeholder; the Deployment injects
REPORT_API_KEY from out-of-band Secret/report-api (deliberately NOT
optional - a missing Secret fails Ready rather than booting a hub with an
empty bearer). main.go gains the REPORT_API_KEY env override (the
RESEND_API_KEY twin). scripts/manifest_secret_gate.py blocks bearer-shaped
(64-hex) literals across manifests/ (felhom.secret.yaml's pre-existing
umami APP_SECRET is a visible non-fatal KNOWN-BACKLOG line - secrets.md
backlog, out of scope here).

Ordering (load-bearing): create Secret/report-api with the CURRENT value
BEFORE syncing; this manifest change rides the same sync as the 0.53.0
image (older images ignore the env override). The exposed git-history copy
dies only with the supervised ROTATION - procedure + consumer list in
documentation/runbooks/secrets.md ("Operator/global bearer key"); both
publish runbooks' ROTATION notes now point at that disposition. The
controller repo's example-config copy of the literal is scrubbed
(felhom-controller 51c871a).

Red-proof E: reintroducing a 64-hex literal into manifests/hub.yaml ->
gate exit 1 (recorded); restored clean.

REUSE.md

@@ -132,7 +132,7 @@
 | Install-profile gate (shell) | scripts/felhom-host-install.sh `--mode appliance\|byo` (GL-2, v1.10.0) | Mandatory-flag profile (no default), refusals at argv time BEFORE any prompt/step, risky step gated at its CALL SITE (one auditable place — never a branch inside the step), mode persisted to state.json + resume-mismatch refusal, `FELHOM_INSTALL_STATE_DIR` override for harness isolation. Harness: scripts/hostinstall-mode-harness.sh (static refusal matrix + grep-invariants + PVE dry-transcript tier; red-proofs run against a mutated scratch copy). |
 | Disclosure↔uninstall parity (shell) | scripts/felhom-host-install.sh `_uninstall_statement` + harness GL4-D (v1.11.0) | Every host artifact the byo disclosure names must be removed OR explicitly listed KEPT by `run_uninstall`; the harness greps the parity (token list). New install-time artifact ⇒ add its removal + disclosure line + parity token in the SAME commit. Drive data rule: plain `umount` only, never `-l`/`-f`, never any format op under /mnt/felhom-drives. |
 | Website deploy (manifest) | manifests/webpage.yaml | git-sync sidecar (sparse-checkout `/website/` + `/scripts/`, `--link=current`) + init container waits for first sync; nginx serves `current/website`; push to main = deployed, no image build. |
-| Secret handling (manifest) | manifests/hub.yaml (env, ~L142) | Secrets via `secretKeyRef` to OUT-OF-BAND secrets created per documentation/runbooks/secrets.md — never inline stringData (see §3). ERRATA (2026-07-03): only `resend-api` is truly out-of-band today; `gitea-creds` is COMMITTED in manifests/felhom.secret.yaml AND live-consumed by hub.yaml — rotation + de-git is a pending operator task (spike SPIKE-a1 appendix). |
+| Secret handling (manifest) | manifests/hub.yaml (env, ~L142) | Secrets via `secretKeyRef` to OUT-OF-BAND secrets created per documentation/runbooks/secrets.md — never inline stringData (see §3). `report-api` (the operator bearer, v0.53.0) is deliberately NOT `optional:` — a missing Secret fails Ready instead of booting an unauthenticatable hub. `scripts/manifest_secret_gate.py` (run after ANY manifests/ change) blocks bearer-shaped (64-hex) literals. ERRATA (2026-07-03): `gitea-creds` is COMMITTED in manifests/felhom.secret.yaml AND live-consumed by hub.yaml — rotation + de-git is a pending operator task (spike SPIKE-a1 appendix). |
 | Hub deploy (GitOps) | manifests/hub.yaml `image:` (~L129) | Pinned explicit tag, bumped in git, deliberate ArgoCD sync (auto-sync OFF). Code push alone deploys nothing. |
 
 ## 3. Dangerous lookalikes — do NOT reuse

documentation/pilot/RUNBOOK-publish-0.81-0.113-2026-07-11.md

@@ -115,6 +115,9 @@ rollout OK, live env verified 0.113.0, clean hub startup log. Demo: no churn (al
   one of the three known committed secrets on the rotate+de-git backlog) and was additionally
   exposed in a session screenshot during Phase D. Rotate it + move to a `secretKeyRef` (the
   Resend-key pattern, documentation/runbooks/secrets.md) as the next operator hygiene task.
+  **DISPOSITION 2026-07-13 (hub v0.53.0):** de-git DONE (`secretKeyRef: report-api/REPORT_API_KEY`
+  + `scripts/manifest_secret_gate.py`); the value rotation is the supervised procedure in
+  documentation/runbooks/secrets.md §"Operator/global bearer key".
 
 ## END STATE
 

documentation/pilot/RUNBOOK-publish-0.85-0.120-2026-07-12.md

@@ -92,9 +92,13 @@ floor lands his controller at 0.120 automatically.
 
 - No key material or tokens in any transcript; publish creds stayed as `$(cat ~/.gitea-token)`
   env-substitution on 180; drill-VM token shredded; saved-log leak grep 0.
-- **⚠ ROTATION STILL DUE:** the hub operator bearer key is COMMITTED in git (`manifests/hub.yaml`) —
+- ~~**⚠ ROTATION STILL DUE:** the hub operator bearer key is COMMITTED in git (`manifests/hub.yaml`) —
   carried over from the 0.81/0.113 train. Rotate + move to a `secretKeyRef` (Resend-key pattern,
-  documentation/runbooks/secrets.md) as an operator hygiene task.
+  documentation/runbooks/secrets.md) as an operator hygiene task.~~
+  **DISPOSITION 2026-07-13 (hub v0.53.0):** moved to `secretKeyRef: report-api/REPORT_API_KEY`
+  (literal removed from the manifest; `scripts/manifest_secret_gate.py` blocks reintroduction).
+  The value ROTATION itself is the supervised step in documentation/runbooks/secrets.md
+  §"Operator/global bearer key" — the git-history copy is dead only after it runs.
 
 ## END STATE
 

documentation/runbooks/secrets.md

@@ -67,6 +67,56 @@ sudo kubectl -n felhom-system rollout status  deploy/hub deploy/contact-mailer -
 
 ---
 
+## Operator/global bearer key — `Secret/report-api`
+
+The hub API's global bearer (`api.report_api_key`) — the operator's own key (e.g. `felhom-ops
+… -hub-key`), distinct from the per-customer/per-host keys the hub generates itself. It was
+COMMITTED in `manifests/hub.yaml` until v0.53.0 (flagged in the 0.81/0.113 and 0.85/0.120
+publish runbooks, incl. a Phase-D screenshot exposure); the manifest now carries a
+`secretKeyRef` and `scripts/manifest_secret_gate.py` blocks reintroduction. **The git-history
+copy stays alive until the value is ROTATED** — de-git alone kills nothing.
+
+**What uses it (live consumers of the GLOBAL key):**
+
+| Consumer | How it reads the key |
+|----------|----------------------|
+| `hub` (`Deployment/hub`) | env `REPORT_API_KEY` ← `secretKeyRef: report-api/REPORT_API_KEY` (v0.53.0 env override fills `api.report_api_key`; the ConfigMap field is an empty placeholder). **Not `optional:`** — a missing Secret fails Ready by design. |
+| Operator tooling (`felhom-ops keys upload -hub-key …`, runbook curl probes in break-glass.md / offsite-endpoint.md) | typed per-invocation from the out-of-band store — nothing machine-persisted. |
+| ~~`felhom-controller` repo `controller.yaml.example`~~ | carried the LITERAL as example text (never a live consumer) — scrubbed 2026-07-13. |
+
+Per-customer (`customer_configs.api_key`) and per-host (`hosts.api_key`) keys are hub-generated
+and **unaffected** by a global-key rotation — no customer box breaks.
+
+**Where the value lives out-of-band:** the operator's password manager, entry "Felhom hub
+global bearer (report_api_key)".
+
+### Create the Secret (pre-deploy for v0.53.0 — same value, no rotation yet)
+
+Create it with the CURRENT value **before** syncing the v0.53.0 manifest (the pod refuses to
+start without it). Render on the build host without echoing the value (file-to-file, the
+operator-present rule):
+
+```bash
+# on 192.168.0.180, as kisfenyo — put the current key in a 0600 temp file first (no echo):
+kubectl create secret generic report-api -n felhom-system \
+    --from-file=REPORT_API_KEY=/dev/stdin < /path/to/keyfile \
+    --dry-run=client -o yaml | sudo kubectl apply -f -
+shred -u /path/to/keyfile
+```
+
+### Rotation (supervised — operator GO required; ordered, load-bearing)
+
+1. Mint the new key into a 0600 file: `openssl rand -hex 32 > keyfile` (no terminal echo).
+2. Re-run the create-Secret pipe above with the new file; store the value out-of-band.
+3. `sudo kubectl -n felhom-system rollout restart deploy/hub && sudo kubectl -n felhom-system rollout status deploy/hub --timeout=120s`
+4. **Verify before declaring the old key dead:**
+   - a customer box still reports (per-customer key — proves rotation touched nothing it shouldn't);
+   - an operator call with the NEW key succeeds (e.g. an authed `GET /api/v1/…` probe);
+   - the SAME call with the OLD key returns 401 — only now is the git-history copy dead.
+5. Update the password-manager entry; note the rotation date in the publish-runbook disposition.
+
+---
+
 ## Other committed secrets (tracked, NOT yet de-gitted — backlog)
 
 `manifests/felhom.secret.yaml` still commits other plaintext secrets (`healthchecks-config` `SECRET_KEY`

hub/cmd/hub/main.go

@@ -140,6 +140,13 @@ func main() {
 	if v := os.Getenv("RESEND_API_KEY"); v != "" {
 		cfg.Notifications.ResendAPIKey = v
 	}
+	// The operator/global bearer key (api.report_api_key) is sourced from Secret/report-api
+	// (env REPORT_API_KEY) since v0.53.0 — the ConfigMap field is an empty placeholder (the
+	// previously-committed literal is dead once rotated; see the publish-runbook ROTATION
+	// notes). Same pattern as RESEND_API_KEY above.
+	if v := os.Getenv("REPORT_API_KEY"); v != "" {
+		cfg.API.ReportAPIKey = v
+	}
 
 	// Ensure data dir exists
 	os.MkdirAll(cfg.Server.DataDir, 0755)

manifests/hub.yaml

@@ -15,14 +15,12 @@
 #        htpasswd -nbBC 10 "" "your-password" | cut -d: -f2
 #      Update the ConfigMap password_hash field below.
 #
-#   3. Generate a report API key (shared secret for controllers):
+#   3. Create the operator/global bearer key Secret (out-of-band, NEVER committed):
-#        openssl rand -hex 32
+#        openssl rand -hex 32   # mint
-#      Update the ConfigMap report_api_key field below.
+#        kubectl -n felhom-system create secret generic report-api \
-#      Then add the same key to each customer's controller.yaml:
+#          --from-literal=REPORT_API_KEY=<minted-key>
-#        hub:
+#      (Customer boxes use per-customer/per-host keys generated by the hub — the global
-#          enabled: true
+#       key is the operator's own, e.g. felhom-ops -hub-key.)
-#          url: "https://hub.felhom.eu"
-#          api_key: "<same-key>"
 #
 #   4. Apply this manifest:
 #        kubectl apply -f manifests/hub.yaml
@@ -71,10 +69,11 @@ data:
       # Generate: htpasswd -nbBC 10 "" "your-password" | cut -d: -f2
       password_hash: "$2y$10$N5.O9jBnc.1tIlJT/irx3OlVjJQemlCHRnfqIJg/EyZofnzXSCpeG"
     api:
-      # Shared secret for controller → hub report push
+      # Operator/global bearer key. NOT stored here since v0.53.0 — injected at runtime from
-      # Generate: openssl rand -hex 32
+      # Secret/report-api via the REPORT_API_KEY env var (see Deployment below). The Secret is
-      # Must match hub.api_key in each customer's controller.yaml
+      # created out-of-band and NOT committed (documentation/runbooks/secrets.md); the previously
-      report_api_key: "094091de545ce28795c47ac2158fc30750db5c24a621c49329b001ee8db57fb8"
+      # committed literal is retired by ROTATION (see the publish-runbook notes). Leave empty.
+      report_api_key: ""
     retention:
       max_days: 90
       prune_schedule: "04:30"
@@ -144,6 +143,15 @@ spec:
                 secretKeyRef:
                   name: resend-api
                   key: RESEND_API_KEY
+            # Operator/global bearer key — injected from the out-of-band Secret/report-api
+            # (NOT committed; documentation/runbooks/secrets.md). Deliberately NOT optional:
+            # a missing Secret must fail the pod Ready rather than boot an unauthenticatable
+            # hub with an empty bearer key. Create the Secret BEFORE syncing this manifest.
+            - name: REPORT_API_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: report-api
+                  key: REPORT_API_KEY
             - name: REGISTRY_USERNAME
               valueFrom:
                 secretKeyRef: