hub + manifests: operator bearer out of git (secretKeyRef report-api)
The global bearer (api.report_api_key) leaves manifests/hub.yaml: the
ConfigMap field is an empty placeholder; the Deployment injects
REPORT_API_KEY from out-of-band Secret/report-api (deliberately NOT
optional - a missing Secret fails Ready rather than booting a hub with an
empty bearer). main.go gains the REPORT_API_KEY env override (the
RESEND_API_KEY twin). scripts/manifest_secret_gate.py blocks bearer-shaped
(64-hex) literals across manifests/ (felhom.secret.yaml's pre-existing
umami APP_SECRET is a visible non-fatal KNOWN-BACKLOG line - secrets.md
backlog, out of scope here).
Ordering (load-bearing): create Secret/report-api with the CURRENT value
BEFORE syncing; this manifest change rides the same sync as the 0.53.0
image (older images ignore the env override). The exposed git-history copy
dies only with the supervised ROTATION - procedure + consumer list in
documentation/runbooks/secrets.md ("Operator/global bearer key"); both
publish runbooks' ROTATION notes now point at that disposition. The
controller repo's example-config copy of the literal is scrubbed
(felhom-controller 51c871a).
Red-proof E: reintroducing a 64-hex literal into manifests/hub.yaml ->
gate exit 1 (recorded); restored clean.
This commit is contained in:
@@ -132,7 +132,7 @@
|
||||
| Install-profile gate (shell) | scripts/felhom-host-install.sh `--mode appliance\|byo` (GL-2, v1.10.0) | Mandatory-flag profile (no default), refusals at argv time BEFORE any prompt/step, risky step gated at its CALL SITE (one auditable place — never a branch inside the step), mode persisted to state.json + resume-mismatch refusal, `FELHOM_INSTALL_STATE_DIR` override for harness isolation. Harness: scripts/hostinstall-mode-harness.sh (static refusal matrix + grep-invariants + PVE dry-transcript tier; red-proofs run against a mutated scratch copy). |
|
||||
| Disclosure↔uninstall parity (shell) | scripts/felhom-host-install.sh `_uninstall_statement` + harness GL4-D (v1.11.0) | Every host artifact the byo disclosure names must be removed OR explicitly listed KEPT by `run_uninstall`; the harness greps the parity (token list). New install-time artifact ⇒ add its removal + disclosure line + parity token in the SAME commit. Drive data rule: plain `umount` only, never `-l`/`-f`, never any format op under /mnt/felhom-drives. |
|
||||
| Website deploy (manifest) | manifests/webpage.yaml | git-sync sidecar (sparse-checkout `/website/` + `/scripts/`, `--link=current`) + init container waits for first sync; nginx serves `current/website`; push to main = deployed, no image build. |
|
||||
| Secret handling (manifest) | manifests/hub.yaml (env, ~L142) | Secrets via `secretKeyRef` to OUT-OF-BAND secrets created per documentation/runbooks/secrets.md — never inline stringData (see §3). ERRATA (2026-07-03): only `resend-api` is truly out-of-band today; `gitea-creds` is COMMITTED in manifests/felhom.secret.yaml AND live-consumed by hub.yaml — rotation + de-git is a pending operator task (spike SPIKE-a1 appendix). |
|
||||
| Secret handling (manifest) | manifests/hub.yaml (env, ~L142) | Secrets via `secretKeyRef` to OUT-OF-BAND secrets created per documentation/runbooks/secrets.md — never inline stringData (see §3). `report-api` (the operator bearer, v0.53.0) is deliberately NOT `optional:` — a missing Secret fails Ready instead of booting an unauthenticatable hub. `scripts/manifest_secret_gate.py` (run after ANY manifests/ change) blocks bearer-shaped (64-hex) literals. ERRATA (2026-07-03): `gitea-creds` is COMMITTED in manifests/felhom.secret.yaml AND live-consumed by hub.yaml — rotation + de-git is a pending operator task (spike SPIKE-a1 appendix). |
|
||||
| Hub deploy (GitOps) | manifests/hub.yaml `image:` (~L129) | Pinned explicit tag, bumped in git, deliberate ArgoCD sync (auto-sync OFF). Code push alone deploys nothing. |
|
||||
|
||||
## 3. Dangerous lookalikes — do NOT reuse
|
||||
|
||||
@@ -115,6 +115,9 @@ rollout OK, live env verified 0.113.0, clean hub startup log. Demo: no churn (al
|
||||
one of the three known committed secrets on the rotate+de-git backlog) and was additionally
|
||||
exposed in a session screenshot during Phase D. Rotate it + move to a `secretKeyRef` (the
|
||||
Resend-key pattern, documentation/runbooks/secrets.md) as the next operator hygiene task.
|
||||
**DISPOSITION 2026-07-13 (hub v0.53.0):** de-git DONE (`secretKeyRef: report-api/REPORT_API_KEY`
|
||||
+ `scripts/manifest_secret_gate.py`); the value rotation is the supervised procedure in
|
||||
documentation/runbooks/secrets.md §"Operator/global bearer key".
|
||||
|
||||
## END STATE
|
||||
|
||||
|
||||
@@ -92,9 +92,13 @@ floor lands his controller at 0.120 automatically.
|
||||
|
||||
- No key material or tokens in any transcript; publish creds stayed as `$(cat ~/.gitea-token)`
|
||||
env-substitution on 180; drill-VM token shredded; saved-log leak grep 0.
|
||||
- **⚠ ROTATION STILL DUE:** the hub operator bearer key is COMMITTED in git (`manifests/hub.yaml`) —
|
||||
- ~~**⚠ ROTATION STILL DUE:** the hub operator bearer key is COMMITTED in git (`manifests/hub.yaml`) —
|
||||
carried over from the 0.81/0.113 train. Rotate + move to a `secretKeyRef` (Resend-key pattern,
|
||||
documentation/runbooks/secrets.md) as an operator hygiene task.
|
||||
documentation/runbooks/secrets.md) as an operator hygiene task.~~
|
||||
**DISPOSITION 2026-07-13 (hub v0.53.0):** moved to `secretKeyRef: report-api/REPORT_API_KEY`
|
||||
(literal removed from the manifest; `scripts/manifest_secret_gate.py` blocks reintroduction).
|
||||
The value ROTATION itself is the supervised step in documentation/runbooks/secrets.md
|
||||
§"Operator/global bearer key" — the git-history copy is dead only after it runs.
|
||||
|
||||
## END STATE
|
||||
|
||||
|
||||
@@ -67,6 +67,56 @@ sudo kubectl -n felhom-system rollout status deploy/hub deploy/contact-mailer -
|
||||
|
||||
---
|
||||
|
||||
## Operator/global bearer key — `Secret/report-api`
|
||||
|
||||
The hub API's global bearer (`api.report_api_key`) — the operator's own key (e.g. `felhom-ops
|
||||
… -hub-key`), distinct from the per-customer/per-host keys the hub generates itself. It was
|
||||
COMMITTED in `manifests/hub.yaml` until v0.53.0 (flagged in the 0.81/0.113 and 0.85/0.120
|
||||
publish runbooks, incl. a Phase-D screenshot exposure); the manifest now carries a
|
||||
`secretKeyRef` and `scripts/manifest_secret_gate.py` blocks reintroduction. **The git-history
|
||||
copy stays alive until the value is ROTATED** — de-git alone kills nothing.
|
||||
|
||||
**What uses it (live consumers of the GLOBAL key):**
|
||||
|
||||
| Consumer | How it reads the key |
|
||||
|----------|----------------------|
|
||||
| `hub` (`Deployment/hub`) | env `REPORT_API_KEY` ← `secretKeyRef: report-api/REPORT_API_KEY` (v0.53.0 env override fills `api.report_api_key`; the ConfigMap field is an empty placeholder). **Not `optional:`** — a missing Secret fails Ready by design. |
|
||||
| Operator tooling (`felhom-ops keys upload -hub-key …`, runbook curl probes in break-glass.md / offsite-endpoint.md) | typed per-invocation from the out-of-band store — nothing machine-persisted. |
|
||||
| ~~`felhom-controller` repo `controller.yaml.example`~~ | carried the LITERAL as example text (never a live consumer) — scrubbed 2026-07-13. |
|
||||
|
||||
Per-customer (`customer_configs.api_key`) and per-host (`hosts.api_key`) keys are hub-generated
|
||||
and **unaffected** by a global-key rotation — no customer box breaks.
|
||||
|
||||
**Where the value lives out-of-band:** the operator's password manager, entry "Felhom hub
|
||||
global bearer (report_api_key)".
|
||||
|
||||
### Create the Secret (pre-deploy for v0.53.0 — same value, no rotation yet)
|
||||
|
||||
Create it with the CURRENT value **before** syncing the v0.53.0 manifest (the pod refuses to
|
||||
start without it). Render on the build host without echoing the value (file-to-file, the
|
||||
operator-present rule):
|
||||
|
||||
```bash
|
||||
# on 192.168.0.180, as kisfenyo — put the current key in a 0600 temp file first (no echo):
|
||||
kubectl create secret generic report-api -n felhom-system \
|
||||
--from-file=REPORT_API_KEY=/dev/stdin < /path/to/keyfile \
|
||||
--dry-run=client -o yaml | sudo kubectl apply -f -
|
||||
shred -u /path/to/keyfile
|
||||
```
|
||||
|
||||
### Rotation (supervised — operator GO required; ordered, load-bearing)
|
||||
|
||||
1. Mint the new key into a 0600 file: `openssl rand -hex 32 > keyfile` (no terminal echo).
|
||||
2. Re-run the create-Secret pipe above with the new file; store the value out-of-band.
|
||||
3. `sudo kubectl -n felhom-system rollout restart deploy/hub && sudo kubectl -n felhom-system rollout status deploy/hub --timeout=120s`
|
||||
4. **Verify before declaring the old key dead:**
|
||||
- a customer box still reports (per-customer key — proves rotation touched nothing it shouldn't);
|
||||
- an operator call with the NEW key succeeds (e.g. an authed `GET /api/v1/…` probe);
|
||||
- the SAME call with the OLD key returns 401 — only now is the git-history copy dead.
|
||||
5. Update the password-manager entry; note the rotation date in the publish-runbook disposition.
|
||||
|
||||
---
|
||||
|
||||
## Other committed secrets (tracked, NOT yet de-gitted — backlog)
|
||||
|
||||
`manifests/felhom.secret.yaml` still commits other plaintext secrets (`healthchecks-config` `SECRET_KEY`
|
||||
|
||||
@@ -140,6 +140,13 @@ func main() {
|
||||
if v := os.Getenv("RESEND_API_KEY"); v != "" {
|
||||
cfg.Notifications.ResendAPIKey = v
|
||||
}
|
||||
// The operator/global bearer key (api.report_api_key) is sourced from Secret/report-api
|
||||
// (env REPORT_API_KEY) since v0.53.0 — the ConfigMap field is an empty placeholder (the
|
||||
// previously-committed literal is dead once rotated; see the publish-runbook ROTATION
|
||||
// notes). Same pattern as RESEND_API_KEY above.
|
||||
if v := os.Getenv("REPORT_API_KEY"); v != "" {
|
||||
cfg.API.ReportAPIKey = v
|
||||
}
|
||||
|
||||
// Ensure data dir exists
|
||||
os.MkdirAll(cfg.Server.DataDir, 0755)
|
||||
|
||||
+20
-12
@@ -15,14 +15,12 @@
|
||||
# htpasswd -nbBC 10 "" "your-password" | cut -d: -f2
|
||||
# Update the ConfigMap password_hash field below.
|
||||
#
|
||||
# 3. Generate a report API key (shared secret for controllers):
|
||||
# openssl rand -hex 32
|
||||
# Update the ConfigMap report_api_key field below.
|
||||
# Then add the same key to each customer's controller.yaml:
|
||||
# hub:
|
||||
# enabled: true
|
||||
# url: "https://hub.felhom.eu"
|
||||
# api_key: "<same-key>"
|
||||
# 3. Create the operator/global bearer key Secret (out-of-band, NEVER committed):
|
||||
# openssl rand -hex 32 # mint
|
||||
# kubectl -n felhom-system create secret generic report-api \
|
||||
# --from-literal=REPORT_API_KEY=<minted-key>
|
||||
# (Customer boxes use per-customer/per-host keys generated by the hub — the global
|
||||
# key is the operator's own, e.g. felhom-ops -hub-key.)
|
||||
#
|
||||
# 4. Apply this manifest:
|
||||
# kubectl apply -f manifests/hub.yaml
|
||||
@@ -71,10 +69,11 @@ data:
|
||||
# Generate: htpasswd -nbBC 10 "" "your-password" | cut -d: -f2
|
||||
password_hash: "$2y$10$N5.O9jBnc.1tIlJT/irx3OlVjJQemlCHRnfqIJg/EyZofnzXSCpeG"
|
||||
api:
|
||||
# Shared secret for controller → hub report push
|
||||
# Generate: openssl rand -hex 32
|
||||
# Must match hub.api_key in each customer's controller.yaml
|
||||
report_api_key: "094091de545ce28795c47ac2158fc30750db5c24a621c49329b001ee8db57fb8"
|
||||
# Operator/global bearer key. NOT stored here since v0.53.0 — injected at runtime from
|
||||
# Secret/report-api via the REPORT_API_KEY env var (see Deployment below). The Secret is
|
||||
# created out-of-band and NOT committed (documentation/runbooks/secrets.md); the previously
|
||||
# committed literal is retired by ROTATION (see the publish-runbook notes). Leave empty.
|
||||
report_api_key: ""
|
||||
retention:
|
||||
max_days: 90
|
||||
prune_schedule: "04:30"
|
||||
@@ -144,6 +143,15 @@ spec:
|
||||
secretKeyRef:
|
||||
name: resend-api
|
||||
key: RESEND_API_KEY
|
||||
# Operator/global bearer key — injected from the out-of-band Secret/report-api
|
||||
# (NOT committed; documentation/runbooks/secrets.md). Deliberately NOT optional:
|
||||
# a missing Secret must fail the pod Ready rather than boot an unauthenticatable
|
||||
# hub with an empty bearer key. Create the Secret BEFORE syncing this manifest.
|
||||
- name: REPORT_API_KEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: report-api
|
||||
key: REPORT_API_KEY
|
||||
- name: REGISTRY_USERNAME
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
|
||||
Reference in New Issue
Block a user