6 Commits

Author SHA1 Message Date
admin 3efb1b1af1 docs: hub v0.53.0 closing bundle - CHANGELOG + CONTEXT (both operator rulings verbatim) + take-two F-14 disposition (SHIPPED, gated on escrow-acked deletion provenance) 2026-07-13 15:02:44 +02:00
admin 3682303131 scripts: manifest bearer gate lands as manifest_bearer_gate.py (the .gitignore *secret* pattern silently un-tracks any 'secret'-named gate file - the previous commit's gate never made it into git); references updated 2026-07-13 14:59:31 +02:00
admin 80aca3a3a1 hub + manifests: operator bearer out of git (secretKeyRef report-api)
The global bearer (api.report_api_key) leaves manifests/hub.yaml: the
ConfigMap field is an empty placeholder; the Deployment injects
REPORT_API_KEY from out-of-band Secret/report-api (deliberately NOT
optional - a missing Secret fails Ready rather than booting a hub with an
empty bearer). main.go gains the REPORT_API_KEY env override (the
RESEND_API_KEY twin). scripts/manifest_secret_gate.py blocks bearer-shaped
(64-hex) literals across manifests/ (felhom.secret.yaml's pre-existing
umami APP_SECRET is a visible non-fatal KNOWN-BACKLOG line - secrets.md
backlog, out of scope here).

Ordering (load-bearing): create Secret/report-api with the CURRENT value
BEFORE syncing; this manifest change rides the same sync as the 0.53.0
image (older images ignore the env override). The exposed git-history copy
dies only with the supervised ROTATION - procedure + consumer list in
documentation/runbooks/secrets.md ("Operator/global bearer key"); both
publish runbooks' ROTATION notes now point at that disposition. The
controller repo's example-config copy of the literal is scrubbed
(felhom-controller 51c871a).

Red-proof E: reintroducing a 64-hex literal into manifests/hub.yaml ->
gate exit 1 (recorded); restored clean.
2026-07-13 14:57:09 +02:00
admin 36c72138f1 hub: dead-host roll-up honesty - customer status folds worst expected host
Customer status (dashboard row, /configs list, detail header + strip) is
now worst(controllerDerived, hostStatusOf(each expected host)) via the ONE
staleness definition (Server.hostStatus, hosts.go - shared with the
HostStalenessChecker; no second threshold). Any host down/stale caps the
customer at WARN with a cause chip naming the host ("host down: <id>");
pending (never-reported) hosts worsen only once the customer has reported
(onboarding exclusion). The three previously-inlined controller-status
chains collapse into controllerStatus() (rollup.go). Display + derivation
only - checker alerting untouched.

Live shape pinned (drill-1 / Peti cluster): host down 23h + controller
report minutes old rendered a GREEN row - TestRollup_DeadHostMasking now
fails that exact outcome. Red-proof: short-circuiting foldHostStatus to
controller-only flips C + two D subtests red ("dashboard row is GREEN
over a 23h-dead host").
2026-07-13 14:52:04 +02:00
admin 04861a7ed3 hub: F-14 part 2 — gated auto-Reissue on the PBS-DR enable dead-end
pbsdrProvisionAtom, on tenantsync token_exists: consult the customer's
MOST RECENT host-deletion record. escrow_acked=true -> invoke the
EXISTING tenantsync Reissue op, store the audit event (pbsdr_auto_reissue,
hub-source, "Previous key destroyed (acknowledged deletion) - credentials
re-issued automatically."), proceed to secret+descriptor as a normal
provision. No record / un-acked -> the pre-existing refusal, byte-unchanged
(never-silently-re-key law; operator ruling 2026-07-13).
Scenario A test drives the REAL handler path over a REAL escrow-ack delete;
scenario B pins the exact non-effect (zero Reissue calls, no state).
Red-proof: bypassing the gate flips both B subtests to 303 (silent re-key).
2026-07-13 14:44:42 +02:00
admin 2321077800 hub: F-14 part 1 — host-deletion provenance (host_deletions, in-tx)
Every DeleteHost writes a provenance row INSIDE the cascade tx: host_id,
customer_id, deleted_at, escrow_acked. escrow_acked = ack given over a
PRESENT escrow row (acknowledged destruction, not a vacuous checkbox).
LatestHostDeletion(customer) serves the F-14 gate — newest record only,
so an old acked deletion never whitelists a newer un-acked one. No
backfill by design: pre-record deletions stay on the manual path.
Red-proof: dropping the in-tx INSERT fails TestDeleteHost_ProvenanceRecord
("no deletion record written") + the part-2 scenario-A test.
2026-07-13 14:44:42 +02:00
22 changed files with 818 additions and 65 deletions
+22
View File
@@ -3,6 +3,28 @@
> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
- **2026-07-13 — CLOSING BUNDLE SHIPPED: hub v0.53.0** — the last engineering items on the
pre-tester board: F-14 gated auto-Reissue (deletion provenance `host_deletions` in-tx +
`token_exists` gate → the EXISTING Reissue op + `pbsdr_auto_reissue` audit event; no record /
un-acked → refusal byte-unchanged), dead-host roll-up honesty (customer status = worst of
controller-derived + per-host `hostStatus` on dashboard//configs/detail, cause chip names the
host; the Peti 23h-masking shape is the pinned fixture), operator bearer OUT OF GIT
(`Secret/report-api` secretKeyRef non-optional + `REPORT_API_KEY` env override +
`scripts/manifest_bearer_gate.py`; VALUE ROTATION = supervised step in
documentation/runbooks/secrets.md — the git-history copy is dead only after it).
**Two operator rulings (2026-07-13), recorded verbatim — future sessions must treat these as
settled:**
1. ***F-14:** auto-Reissue of offsite credentials is permitted ONLY when the hub's own deletion
record shows the tenancy's owning host was removed through the escrow-ack flow — acknowledged
destruction is not silent re-keying. No record → manual Re-issue stays the only path.*
2. ***Roll-up:** a customer's status may never look better than its worst expected host. Single
staleness definition (`hostStatusOf`) — no second threshold anywhere.*
Gate additions: `python scripts/manifest_bearer_gate.py` after ANY manifests/ change (64-hex
bearer shapes; felhom.secret.yaml carve-out = visible KNOWN-BACKLOG line). GOTCHA: the repo
.gitignore `*secret*` pattern silently un-tracks any "secret"-named script — hence the
"bearer" name. The controller repo's controller.yaml.example carried the REAL bearer literal —
scrubbed (felhom-controller 51c871a).
- **2026-07-13 — POLISH BATCH SHIPPED: hub v0.52.0 (+ controller v0.123.0)** — take-two F-15 - **2026-07-13 — POLISH BATCH SHIPPED: hub v0.52.0 (+ controller v0.123.0)** — take-two F-15
CLOSED (reset-request response carries the rotated code hash — ACK shape/guarantee — emailed CLOSED (reset-request response carries the rotated code hash — ACK shape/guarantee — emailed
codes work immediately; live 1 s apply + first-try accept on Viktor's exact failure path) and codes work immediately; live 1 s apply + first-try accept on Viktor's exact failure path) and
+1 -1
View File
@@ -132,7 +132,7 @@
| Install-profile gate (shell) | scripts/felhom-host-install.sh `--mode appliance\|byo` (GL-2, v1.10.0) | Mandatory-flag profile (no default), refusals at argv time BEFORE any prompt/step, risky step gated at its CALL SITE (one auditable place — never a branch inside the step), mode persisted to state.json + resume-mismatch refusal, `FELHOM_INSTALL_STATE_DIR` override for harness isolation. Harness: scripts/hostinstall-mode-harness.sh (static refusal matrix + grep-invariants + PVE dry-transcript tier; red-proofs run against a mutated scratch copy). | | Install-profile gate (shell) | scripts/felhom-host-install.sh `--mode appliance\|byo` (GL-2, v1.10.0) | Mandatory-flag profile (no default), refusals at argv time BEFORE any prompt/step, risky step gated at its CALL SITE (one auditable place — never a branch inside the step), mode persisted to state.json + resume-mismatch refusal, `FELHOM_INSTALL_STATE_DIR` override for harness isolation. Harness: scripts/hostinstall-mode-harness.sh (static refusal matrix + grep-invariants + PVE dry-transcript tier; red-proofs run against a mutated scratch copy). |
| Disclosure↔uninstall parity (shell) | scripts/felhom-host-install.sh `_uninstall_statement` + harness GL4-D (v1.11.0) | Every host artifact the byo disclosure names must be removed OR explicitly listed KEPT by `run_uninstall`; the harness greps the parity (token list). New install-time artifact ⇒ add its removal + disclosure line + parity token in the SAME commit. Drive data rule: plain `umount` only, never `-l`/`-f`, never any format op under /mnt/felhom-drives. | | Disclosure↔uninstall parity (shell) | scripts/felhom-host-install.sh `_uninstall_statement` + harness GL4-D (v1.11.0) | Every host artifact the byo disclosure names must be removed OR explicitly listed KEPT by `run_uninstall`; the harness greps the parity (token list). New install-time artifact ⇒ add its removal + disclosure line + parity token in the SAME commit. Drive data rule: plain `umount` only, never `-l`/`-f`, never any format op under /mnt/felhom-drives. |
| Website deploy (manifest) | manifests/webpage.yaml | git-sync sidecar (sparse-checkout `/website/` + `/scripts/`, `--link=current`) + init container waits for first sync; nginx serves `current/website`; push to main = deployed, no image build. | | Website deploy (manifest) | manifests/webpage.yaml | git-sync sidecar (sparse-checkout `/website/` + `/scripts/`, `--link=current`) + init container waits for first sync; nginx serves `current/website`; push to main = deployed, no image build. |
| Secret handling (manifest) | manifests/hub.yaml (env, ~L142) | Secrets via `secretKeyRef` to OUT-OF-BAND secrets created per documentation/runbooks/secrets.md — never inline stringData (see §3). ERRATA (2026-07-03): only `resend-api` is truly out-of-band today; `gitea-creds` is COMMITTED in manifests/felhom.secret.yaml AND live-consumed by hub.yaml — rotation + de-git is a pending operator task (spike SPIKE-a1 appendix). | | Secret handling (manifest) | manifests/hub.yaml (env, ~L142) | Secrets via `secretKeyRef` to OUT-OF-BAND secrets created per documentation/runbooks/secrets.md — never inline stringData (see §3). `report-api` (the operator bearer, v0.53.0) is deliberately NOT `optional:` — a missing Secret fails Ready instead of booting an unauthenticatable hub. `scripts/manifest_bearer_gate.py` (run after ANY manifests/ change) blocks bearer-shaped (64-hex) literals. ERRATA (2026-07-03): `gitea-creds` is COMMITTED in manifests/felhom.secret.yaml AND live-consumed by hub.yaml — rotation + de-git is a pending operator task (spike SPIKE-a1 appendix). |
| Hub deploy (GitOps) | manifests/hub.yaml `image:` (~L129) | Pinned explicit tag, bumped in git, deliberate ArgoCD sync (auto-sync OFF). Code push alone deploys nothing. | | Hub deploy (GitOps) | manifests/hub.yaml `image:` (~L129) | Pinned explicit tag, bumped in git, deliberate ArgoCD sync (auto-sync OFF). Code push alone deploys nothing. |
## 3. Dangerous lookalikes — do NOT reuse ## 3. Dangerous lookalikes — do NOT reuse
@@ -59,7 +59,7 @@ capability chip table (v0.51.0) live.
| # | Sev | Finding | Direction | | # | Sev | Finding | Direction |
|---|---|---|---| |---|---|---|---|
| **F-14** | MEDIUM | Host delete + re-enroll while the ep0 tenancy survives = DR re-attach dead-end: auto-provision AND config save hard-error `token_exists`; "Re-issue PBS credentials" 400s (requires the descriptor the deleted host took with it). Recovery today = manual ep0 root token-delete (done during this reset) | candidate: flag-ON + no-descriptor + token-exists ⇒ auto-Reissue (re-adopt; the old secret is unrecoverable anyway) — needs an operator ruling vs the never-silently-re-key law | | **F-14** | MEDIUM | Host delete + re-enroll while the ep0 tenancy survives = DR re-attach dead-end: auto-provision AND config save hard-error `token_exists`; "Re-issue PBS credentials" 400s (requires the descriptor the deleted host took with it). Recovery today = manual ep0 root token-delete (done during this reset) | **SHIPPED 2026-07-13** (hub v0.53.0, operator ruling): auto-Reissue permitted ONLY when the hub's own deletion record (`host_deletions`, written in the DeleteHost tx) shows the owning host was removed through the escrow-ack flow — acknowledged destruction, not silent re-keying; no record (incl. THIS drill's pre-record reset) / un-acked → the refusal + manual path, byte-unchanged. Scenario A/B tests + red-proofs; live validation = the next real host-reset cycle (fixtures carry it until then) |
| **F-15** | MEDIUM/UX — **operator-flagged must-fix** | Claim/reset code NOT immediately usable: the hub rotates the hash at reset-request, but the box learns it only on its next report ACK (~15 min). Viktor hit it live ("Hibás vagy lejárt kód" with a fresh code) | **SHIPPED 2026-07-13** (hub v0.52.0 + controller v0.123.0): the reset-request response carries the rotated hash, applied via the ACK's generation-guarded consumer. Live re-run of the exact failure path: applied 1 s after the request, code accepted first try | | **F-15** | MEDIUM/UX — **operator-flagged must-fix** | Claim/reset code NOT immediately usable: the hub rotates the hash at reset-request, but the box learns it only on its next report ACK (~15 min). Viktor hit it live ("Hibás vagy lejárt kód" with a fresh code) | **SHIPPED 2026-07-13** (hub v0.52.0 + controller v0.123.0): the reset-request response carries the rotated hash, applied via the ACK's generation-guarded consumer. Live re-run of the exact failure path: applied 1 s after the request, code accepted first try |
| **F-16** | LOW | Native `confirm()` on hub buttons (offsite/PBS re-issue, freeze) froze CC's browser automation — the hub-side siblings of drill F-11 | **SHIPPED 2026-07-13** (hub v0.52.0 + controller v0.123.0): inline "Igen/Mégse" two-step everywhere, both repos; `hub_confirm_gate.py`/`native_confirm_gate.py` enforce zero native confirms. Live: the offsite re-issue completed under automation without freezing | | **F-16** | LOW | Native `confirm()` on hub buttons (offsite/PBS re-issue, freeze) froze CC's browser automation — the hub-side siblings of drill F-11 | **SHIPPED 2026-07-13** (hub v0.52.0 + controller v0.123.0): inline "Igen/Mégse" two-step everywhere, both repos; `hub_confirm_gate.py`/`native_confirm_gate.py` enforce zero native confirms. Live: the offsite re-issue completed under automation without freezing |
| obs. | — | A re-provisioned guest over an EXISTING offsite repo needs the old repo password (gone with the deleted escrow) or a repo wipe — this IS the S5 DR-restore scenario, already queued as its own drill; the wipe was the correct reset action here, not a product gap | S5 drill | | obs. | — | A re-provisioned guest over an EXISTING offsite repo needs the old repo password (gone with the deleted escrow) or a repo wipe — this IS the S5 DR-restore scenario, already queued as its own drill; the wipe was the correct reset action here, not a product gap | S5 drill |
@@ -115,6 +115,9 @@ rollout OK, live env verified 0.113.0, clean hub startup log. Demo: no churn (al
one of the three known committed secrets on the rotate+de-git backlog) and was additionally one of the three known committed secrets on the rotate+de-git backlog) and was additionally
exposed in a session screenshot during Phase D. Rotate it + move to a `secretKeyRef` (the exposed in a session screenshot during Phase D. Rotate it + move to a `secretKeyRef` (the
Resend-key pattern, documentation/runbooks/secrets.md) as the next operator hygiene task. Resend-key pattern, documentation/runbooks/secrets.md) as the next operator hygiene task.
**DISPOSITION 2026-07-13 (hub v0.53.0):** de-git DONE (`secretKeyRef: report-api/REPORT_API_KEY`
+ `scripts/manifest_bearer_gate.py`); the value rotation is the supervised procedure in
documentation/runbooks/secrets.md §"Operator/global bearer key".
## END STATE ## END STATE
@@ -92,9 +92,13 @@ floor lands his controller at 0.120 automatically.
- No key material or tokens in any transcript; publish creds stayed as `$(cat ~/.gitea-token)` - No key material or tokens in any transcript; publish creds stayed as `$(cat ~/.gitea-token)`
env-substitution on 180; drill-VM token shredded; saved-log leak grep 0. env-substitution on 180; drill-VM token shredded; saved-log leak grep 0.
- **⚠ ROTATION STILL DUE:** the hub operator bearer key is COMMITTED in git (`manifests/hub.yaml`) — - ~~**⚠ ROTATION STILL DUE:** the hub operator bearer key is COMMITTED in git (`manifests/hub.yaml`) —
carried over from the 0.81/0.113 train. Rotate + move to a `secretKeyRef` (Resend-key pattern, carried over from the 0.81/0.113 train. Rotate + move to a `secretKeyRef` (Resend-key pattern,
documentation/runbooks/secrets.md) as an operator hygiene task. documentation/runbooks/secrets.md) as an operator hygiene task.~~
**DISPOSITION 2026-07-13 (hub v0.53.0):** moved to `secretKeyRef: report-api/REPORT_API_KEY`
(literal removed from the manifest; `scripts/manifest_bearer_gate.py` blocks reintroduction).
The value ROTATION itself is the supervised step in documentation/runbooks/secrets.md
§"Operator/global bearer key" — the git-history copy is dead only after it runs.
## END STATE ## END STATE
+50
View File
@@ -67,6 +67,56 @@ sudo kubectl -n felhom-system rollout status deploy/hub deploy/contact-mailer -
--- ---
## Operator/global bearer key — `Secret/report-api`
The hub API's global bearer (`api.report_api_key`) — the operator's own key (e.g. `felhom-ops
… -hub-key`), distinct from the per-customer/per-host keys the hub generates itself. It was
COMMITTED in `manifests/hub.yaml` until v0.53.0 (flagged in the 0.81/0.113 and 0.85/0.120
publish runbooks, incl. a Phase-D screenshot exposure); the manifest now carries a
`secretKeyRef` and `scripts/manifest_bearer_gate.py` blocks reintroduction. **The git-history
copy stays alive until the value is ROTATED** — de-git alone kills nothing.
**What uses it (live consumers of the GLOBAL key):**
| Consumer | How it reads the key |
|----------|----------------------|
| `hub` (`Deployment/hub`) | env `REPORT_API_KEY``secretKeyRef: report-api/REPORT_API_KEY` (v0.53.0 env override fills `api.report_api_key`; the ConfigMap field is an empty placeholder). **Not `optional:`** — a missing Secret fails Ready by design. |
| Operator tooling (`felhom-ops keys upload -hub-key …`, runbook curl probes in break-glass.md / offsite-endpoint.md) | typed per-invocation from the out-of-band store — nothing machine-persisted. |
| ~~`felhom-controller` repo `controller.yaml.example`~~ | carried the LITERAL as example text (never a live consumer) — scrubbed 2026-07-13. |
Per-customer (`customer_configs.api_key`) and per-host (`hosts.api_key`) keys are hub-generated
and **unaffected** by a global-key rotation — no customer box breaks.
**Where the value lives out-of-band:** the operator's password manager, entry "Felhom hub
global bearer (report_api_key)".
### Create the Secret (pre-deploy for v0.53.0 — same value, no rotation yet)
Create it with the CURRENT value **before** syncing the v0.53.0 manifest (the pod refuses to
start without it). Render on the build host without echoing the value (file-to-file, the
operator-present rule):
```bash
# on 192.168.0.180, as kisfenyo — put the current key in a 0600 temp file first (no echo):
kubectl create secret generic report-api -n felhom-system \
--from-file=REPORT_API_KEY=/dev/stdin < /path/to/keyfile \
--dry-run=client -o yaml | sudo kubectl apply -f -
shred -u /path/to/keyfile
```
### Rotation (supervised — operator GO required; ordered, load-bearing)
1. Mint the new key into a 0600 file: `openssl rand -hex 32 > keyfile` (no terminal echo).
2. Re-run the create-Secret pipe above with the new file; store the value out-of-band.
3. `sudo kubectl -n felhom-system rollout restart deploy/hub && sudo kubectl -n felhom-system rollout status deploy/hub --timeout=120s`
4. **Verify before declaring the old key dead:**
- a customer box still reports (per-customer key — proves rotation touched nothing it shouldn't);
- an operator call with the NEW key succeeds (e.g. an authed `GET /api/v1/…` probe);
- the SAME call with the OLD key returns 401 — only now is the git-history copy dead.
5. Update the password-manager entry; note the rotation date in the publish-runbook disposition.
---
## Other committed secrets (tracked, NOT yet de-gitted — backlog) ## Other committed secrets (tracked, NOT yet de-gitted — backlog)
`manifests/felhom.secret.yaml` still commits other plaintext secrets (`healthchecks-config` `SECRET_KEY` `manifests/felhom.secret.yaml` still commits other plaintext secrets (`healthchecks-config` `SECRET_KEY`
+36 -1
View File
@@ -1,6 +1,41 @@
# Felhom Hub — Changelog # Felhom Hub — Changelog
## v0.52.0 — polish batch: F-15 instant reset codes + F-16 inline confirms (2026-07-13) ## v0.53.0 — closing bundle: F-14 gated auto-Reissue + dead-host roll-up honesty + bearer out of git (2026-07-13)
The last engineering items on the pre-tester board. Two operator rulings in force (CONTEXT.md):
F-14 auto-re-issue only on a recorded escrow-acked deletion; customer status never better than
its worst expected host.
- **F-14 deletion provenance + gated auto-Reissue** (take-two MEDIUM: host delete + re-enroll
with a surviving ep0 tenancy = DR re-attach dead-end, `token_exists` on both auto-provision
and config save). New `host_deletions` table — host_id, customer_id, deleted_at,
`escrow_acked` (= ack given over a PRESENT escrow row) — written INSIDE the DeleteHost
transaction; NO backfill (pre-record deletions keep the manual path by design). The provision
atom, on `token_exists`, reads the customer's MOST RECENT deletion record: escrow_acked →
invoke the EXISTING tenantsync Reissue op, store the `pbsdr_auto_reissue` audit event
("Previous key destroyed (acknowledged deletion) — credentials re-issued automatically."),
proceed; no record / un-acked → the pre-existing refusal byte-unchanged (never-silently-re-key
law). Red-proofs: provenance-write drop → scenario-A fails; gate bypass → scenario-B's
zero-reissue assertions fail (silent re-key visible as a 303).
- **Dead-host roll-up honesty** (drill-1 observation, live on the Peti cluster: proxmox1 down
23h behind a GREEN customer row — controller reports ride the internet, independent of the
agent). Customer status on the dashboard, /configs list and customer detail (header + strip)
is now `worst(controllerDerived, hostStatusOf(each expected host))` via THE single staleness
definition (`Server.hostStatus`; no second threshold anywhere): any host down/stale caps the
customer at WARN with a cause chip naming the host ("host down: <id>"); pending hosts worsen
only after the customer has ever reported (onboarding exclusion). The three inlined
controller-status chains collapsed into `controllerStatus()` (rollup.go). Display +
derivation only — HostStalenessChecker alerting untouched. Red-proof: fold removal → the
exact Peti fixture renders green → TestRollup_DeadHostMasking fails.
- **Operator bearer out of git** (the two publish runbooks' ROTATION item): `manifests/hub.yaml`
no longer commits `report_api_key` — the Deployment injects `REPORT_API_KEY` from out-of-band
`Secret/report-api` (deliberately NOT `optional:` — a missing Secret fails Ready instead of
booting an unauthenticatable hub); main.go gains the env override (RESEND_API_KEY twin). New
gate `scripts/manifest_bearer_gate.py` blocks bearer-shaped (64-hex) literals in manifests/
(red-proven: reintroduction → exit 1). The controller repo's example-config copy of the
literal is scrubbed. The exposed git-history value dies with the SUPERVISED rotation —
procedure + full consumer list in documentation/runbooks/secrets.md §"Operator/global bearer
key" (per-customer/per-host keys unaffected).
Hub half of the polish batch (take-two findings F-15/F-16). Companion: controller v0.123.0. Hub half of the polish batch (take-two findings F-15/F-16). Companion: controller v0.123.0.
+7
View File
@@ -140,6 +140,13 @@ func main() {
if v := os.Getenv("RESEND_API_KEY"); v != "" { if v := os.Getenv("RESEND_API_KEY"); v != "" {
cfg.Notifications.ResendAPIKey = v cfg.Notifications.ResendAPIKey = v
} }
// The operator/global bearer key (api.report_api_key) is sourced from Secret/report-api
// (env REPORT_API_KEY) since v0.53.0 — the ConfigMap field is an empty placeholder (the
// previously-committed literal is dead once rotated; see the publish-runbook ROTATION
// notes). Same pattern as RESEND_API_KEY above.
if v := os.Getenv("REPORT_API_KEY"); v != "" {
cfg.API.ReportAPIKey = v
}
// Ensure data dir exists // Ensure data dir exists
os.MkdirAll(cfg.Server.DataDir, 0755) os.MkdirAll(cfg.Server.DataDir, 0755)
+83
View File
@@ -174,6 +174,89 @@ func TestDeleteHost_EscrowFlagSemantics(t *testing.T) {
} }
} }
// v0.53.0 F-14 provenance — the deletion record is written IN the delete tx, with
// escrow_acked reflecting an ACTUAL acknowledged destruction (ack over a present escrow).
// RED-PROOF (Part 1): dropping the provenance INSERT from DeleteHost fails the acked case
// (LatestHostDeletion returns nil — the gate finds nothing).
func TestDeleteHost_ProvenanceRecord(t *testing.T) {
s := newTestStore(t)
// Escrow-ack delete → record with escrow_acked = true.
seedHostWithArtifacts(t, s, "prov-acked", "cust-f14")
if err := s.DeleteHost("prov-acked", true); err != nil {
t.Fatalf("DeleteHost: %v", err)
}
rec, err := s.LatestHostDeletion("cust-f14")
if err != nil {
t.Fatalf("LatestHostDeletion: %v", err)
}
if rec == nil {
t.Fatal("no deletion record written by the escrow-ack delete")
}
if rec.HostID != "prov-acked" || rec.CustomerID != "cust-f14" || !rec.EscrowAcked {
t.Errorf("record = %+v, want host=prov-acked customer=cust-f14 escrow_acked=true", rec)
}
if rec.DeletedAt.IsZero() {
t.Error("deleted_at not populated")
}
// Delete WITHOUT escrow (none present) → record exists but escrow_acked = false, even
// though deleteEscrow=true was passed: ticking the box over NOTHING is not an
// acknowledged destruction.
if err := s.UpsertHost(&Host{HostID: "prov-noescrow", CustomerID: "cust-noesc", APIKey: "k"}); err != nil {
t.Fatal(err)
}
if err := s.DeleteHost("prov-noescrow", true); err != nil {
t.Fatalf("DeleteHost: %v", err)
}
rec, err = s.LatestHostDeletion("cust-noesc")
if err != nil || rec == nil {
t.Fatalf("LatestHostDeletion = %+v, %v; want a record", rec, err)
}
if rec.EscrowAcked {
t.Error("escrow_acked = true for a host with NO escrow row — vacuous ack must record false")
}
// The refused delete (escrow present, no ack) writes NOTHING — the tx never ran.
seedHostWithArtifacts(t, s, "prov-refused", "cust-refused")
if err := s.DeleteHost("prov-refused", false); !errors.Is(err, ErrHostEscrowPresent) {
t.Fatalf("expected escrow refusal, got %v", err)
}
if rec, _ := s.LatestHostDeletion("cust-refused"); rec != nil {
t.Errorf("refused delete wrote a provenance record: %+v", rec)
}
// Customer with no deletions ever → nil, nil (the pre-v0.53.0 shape — manual path).
if rec, err := s.LatestHostDeletion("cust-never"); err != nil || rec != nil {
t.Errorf("LatestHostDeletion(no deletions) = %+v, %v; want nil, nil", rec, err)
}
}
// The gate reads the MOST RECENT record: an old acked deletion must not whitelist a newer
// un-acked one (the F-14 law is about the deletion that orphaned the CURRENT tenancy).
func TestLatestHostDeletion_NewestWins(t *testing.T) {
s := newTestStore(t)
seedHostWithArtifacts(t, s, "gen1-host", "cust-seq")
if err := s.DeleteHost("gen1-host", true); err != nil { // acked
t.Fatal(err)
}
if err := s.UpsertHost(&Host{HostID: "gen2-host", CustomerID: "cust-seq", APIKey: "k2"}); err != nil {
t.Fatal(err)
}
if err := s.DeleteHost("gen2-host", false); err != nil { // no escrow → un-acked record
t.Fatal(err)
}
rec, err := s.LatestHostDeletion("cust-seq")
if err != nil || rec == nil {
t.Fatalf("LatestHostDeletion = %+v, %v", rec, err)
}
if rec.HostID != "gen2-host" || rec.EscrowAcked {
t.Errorf("latest record = %+v, want the NEWER un-acked gen2-host row", rec)
}
}
func TestCountHostArtifacts(t *testing.T) { func TestCountHostArtifacts(t *testing.T) {
s := newTestStore(t) s := newTestStore(t)
seedHostWithArtifacts(t, s, "impact-host", "cust-e") seedHostWithArtifacts(t, s, "impact-host", "cust-e")
+82 -5
View File
@@ -555,6 +555,27 @@ func (s *Store) migrate() error {
return err return err
} }
// v0.53.0 — host-deletion provenance (F-14, operator ruling 2026-07-13): one row per DeleteHost,
// written INSIDE the delete transaction. escrow_acked records whether the host was removed
// through the escrow-ack flow (the operator explicitly acknowledged destroying a PRESENT escrow
// row — acknowledged key destruction). The PBS-DR enable path may auto-re-issue a surviving ep0
// tenancy ONLY when the customer's most recent record here has escrow_acked=1; no record (all
// pre-v0.53.0 deletions — deliberately NO backfill) or an un-acked record keeps the manual
// re-issue path the only one (never-silently-re-key law).
_, err = s.db.Exec(`
CREATE TABLE IF NOT EXISTS host_deletions (
id INTEGER PRIMARY KEY AUTOINCREMENT,
host_id TEXT NOT NULL,
customer_id TEXT NOT NULL,
deleted_at DATETIME NOT NULL DEFAULT (datetime('now')),
escrow_acked INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX IF NOT EXISTS idx_host_deletions_customer ON host_deletions(customer_id, id DESC);
`)
if err != nil {
return err
}
// v0.51.0 dr_tier one-time legacy backfill — see the ALTER above; runs last so every table // v0.51.0 dr_tier one-time legacy backfill — see the ALTER above; runs last so every table
// it touches (hosts, customer_configs) exists on a fresh DB too (where it finds nothing). // it touches (hosts, customer_configs) exists on a fresh DB too (where it finds nothing).
if drTierAlterErr == nil { if drTierAlterErr == nil {
@@ -1893,19 +1914,22 @@ func (s *Store) CountHostArtifacts(hostID string) (HostArtifacts, error) {
// reconciler's 5-minute declarative full-list push converges the endpoint after the row // reconciler's 5-minute declarative full-list push converges the endpoint after the row
// disappears — no bump, no reconciler change. log_bundle rows die by scope_id == host_id // disappears — no bump, no reconciler change. log_bundle rows die by scope_id == host_id
// (agent channel); customer-scoped bundles (scope_id == customer_id) are NOT touched. // (agent channel); customer-scoped bundles (scope_id == customer_id) are NOT touched.
//
// v0.53.0 (F-14 provenance): every delete also writes a host_deletions row IN THE SAME tx.
// escrow_acked = deleteEscrow AND an escrow row was actually present — "removed through the
// escrow-ack flow" means an acknowledged destruction happened, not merely that the checkbox
// was ticked over nothing.
func (s *Store) DeleteHost(hostID string, deleteEscrow bool) error { func (s *Store) DeleteHost(hostID string, deleteEscrow bool) error {
if hostID == "" { if hostID == "" {
return fmt.Errorf("DeleteHost: empty host_id") return fmt.Errorf("DeleteHost: empty host_id")
} }
if !deleteEscrow { var escrowPresent int
var n int if err := s.db.QueryRow(`SELECT EXISTS(SELECT 1 FROM host_escrow WHERE host_id = ?)`, hostID).Scan(&escrowPresent); err != nil {
if err := s.db.QueryRow(`SELECT EXISTS(SELECT 1 FROM host_escrow WHERE host_id = ?)`, hostID).Scan(&n); err != nil {
return fmt.Errorf("DeleteHost %s: escrow check: %w", hostID, err) return fmt.Errorf("DeleteHost %s: escrow check: %w", hostID, err)
} }
if n != 0 { if !deleteEscrow && escrowPresent != 0 {
return ErrHostEscrowPresent return ErrHostEscrowPresent
} }
}
tx, err := s.db.Begin() tx, err := s.db.Begin()
if err != nil { if err != nil {
@@ -1913,6 +1937,25 @@ func (s *Store) DeleteHost(hostID string, deleteEscrow bool) error {
} }
defer tx.Rollback() defer tx.Rollback()
// Provenance first (reads the host row this tx is about to delete). A host_id that has no
// row deletes nothing anyway — skip the record rather than inventing an empty customer_id.
var customerID string
switch err := tx.QueryRow(`SELECT customer_id FROM hosts WHERE host_id = ?`, hostID).Scan(&customerID); err {
case nil:
acked := 0
if deleteEscrow && escrowPresent != 0 {
acked = 1
}
if _, err := tx.Exec(`INSERT INTO host_deletions (host_id, customer_id, escrow_acked) VALUES (?, ?, ?)`,
hostID, customerID, acked); err != nil {
return fmt.Errorf("DeleteHost %s: provenance record: %w", hostID, err)
}
case sql.ErrNoRows:
// no host row — fall through, the deletes below are no-ops
default:
return fmt.Errorf("DeleteHost %s: customer lookup: %w", hostID, err)
}
stmts := []string{ stmts := []string{
`DELETE FROM guests WHERE host_id = ?`, `DELETE FROM guests WHERE host_id = ?`,
`DELETE FROM host_reports WHERE host_id = ?`, `DELETE FROM host_reports WHERE host_id = ?`,
@@ -1935,6 +1978,40 @@ func (s *Store) DeleteHost(hostID string, deleteEscrow bool) error {
return tx.Commit() return tx.Commit()
} }
// HostDeletion is one host-removal provenance record (v0.53.0, F-14). EscrowAcked means the
// operator removed the host through the escrow-ack flow — an acknowledged destruction of the
// host's key custody, the ONLY state that permits the PBS-DR auto-re-issue.
type HostDeletion struct {
HostID string
CustomerID string
DeletedAt time.Time
EscrowAcked bool
}
// LatestHostDeletion returns the customer's MOST RECENT host-deletion record (nil when the
// customer has none — every pre-v0.53.0 deletion, by design: no backfill invents provenance).
// The latest record is the one that orphaned a surviving ep0 tenancy, so the F-14 gate reads
// exactly this row — an older acked record must not whitelist a newer un-acked deletion.
func (s *Store) LatestHostDeletion(customerID string) (*HostDeletion, error) {
var d HostDeletion
var deletedAt string
var acked int
err := s.db.QueryRow(`
SELECT host_id, customer_id, deleted_at, escrow_acked
FROM host_deletions WHERE customer_id = ?
ORDER BY id DESC LIMIT 1`, customerID,
).Scan(&d.HostID, &d.CustomerID, &deletedAt, &acked)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, err
}
d.DeletedAt = parseSQLiteTime(deletedAt)
d.EscrowAcked = acked != 0
return &d, nil
}
// UpsertHost creates or updates a host identity (used by the admin mint). On // UpsertHost creates or updates a host identity (used by the admin mint). On
// conflict it updates only operator-settable identity fields + updated_at; it does // conflict it updates only operator-settable identity fields + updated_at; it does
// NOT touch the reality columns (agent_version/last_report_at) or the inert intent // NOT touch the reality columns (agent_version/last_report_at) or the inert intent
+21 -22
View File
@@ -53,6 +53,7 @@ type customerListEntry struct {
HasConfig bool HasConfig bool
IsBlocked bool IsBlocked bool
OverallStatus string // ok, warn, down, disabled, pending, "" if no reports OverallStatus string // ok, warn, down, disabled, pending, "" if no reports
HostCause string // v0.53.0 roll-up: "" or "host down|stale|pending: <id>"
ControllerVersion string ControllerVersion string
TimeSinceReport time.Duration TimeSinceReport time.Duration
ConfigCreatedAt time.Time ConfigCreatedAt time.Time
@@ -95,20 +96,13 @@ func (s *Server) handleConfigList(w http.ResponseWriter, r *http.Request) {
} }
for _, c := range customers { for _, c := range customers {
status := "ok" // Controller-derived status + the v0.53.0 dead-host roll-up (rollup.go).
if c.HealthStatus == "disabled" { status, hostCause := s.foldHostStatus(c.CustomerID, controllerStatus(&c), true)
status = "disabled"
} else if c.TimeSinceReport > time.Hour {
status = "down"
} else if c.TimeSinceReport > 30*time.Minute || c.HealthStatus == "warn" {
status = "warn"
} else if c.HealthStatus == "fail" {
status = "down"
}
if entry, ok := merged[c.CustomerID]; ok { if entry, ok := merged[c.CustomerID]; ok {
// Config exists — enrich with report data // Config exists — enrich with report data
entry.OverallStatus = status entry.OverallStatus = status
entry.HostCause = hostCause
entry.ControllerVersion = c.ControllerVersion entry.ControllerVersion = c.ControllerVersion
entry.TimeSinceReport = c.TimeSinceReport entry.TimeSinceReport = c.TimeSinceReport
if entry.CustomerName == "" { if entry.CustomerName == "" {
@@ -120,12 +114,21 @@ func (s *Server) handleConfigList(w http.ResponseWriter, r *http.Request) {
CustomerID: c.CustomerID, CustomerID: c.CustomerID,
CustomerName: c.CustomerName, CustomerName: c.CustomerName,
OverallStatus: status, OverallStatus: status,
HostCause: hostCause,
ControllerVersion: c.ControllerVersion, ControllerVersion: c.ControllerVersion,
TimeSinceReport: c.TimeSinceReport, TimeSinceReport: c.TimeSinceReport,
} }
} }
} }
// Config-only customers (no reports yet): the roll-up still applies — a down/stale host
// must not hide behind the muted no-reports dash; only never-reported hosts are excluded.
for _, e := range merged {
if e.OverallStatus == "" {
e.OverallStatus, e.HostCause = s.foldHostStatus(e.CustomerID, "", false)
}
}
// Phase 2 floor: resolve each customer's effective floor (override else global) + below-floor flag. // Phase 2 floor: resolve each customer's effective floor (override else global) + below-floor flag.
globalFloor := s.store.GetGlobalMinControllerVersion() globalFloor := s.store.GetGlobalMinControllerVersion()
for _, e := range merged { for _, e := range merged {
@@ -198,21 +201,15 @@ func (s *Server) handleCustomerUnified(w http.ResponseWriter, r *http.Request, c
json.Unmarshal([]byte(cfg.ConfigJSON), &overrides) json.Unmarshal([]byte(cfg.ConfigJSON), &overrides)
} }
// Overall status // Overall status: controller-derived + the v0.53.0 dead-host roll-up (rollup.go). The
// blocked override stays LAST (administrative state wins the token); the host cause chip
// renders regardless so the header says WHICH host is the problem.
overallStatus := "pending" overallStatus := "pending"
if customer != nil { if customer != nil {
if customer.HealthStatus == "disabled" { overallStatus = controllerStatus(customer)
overallStatus = "disabled"
} else if customer.TimeSinceReport > time.Hour {
overallStatus = "down"
} else if customer.TimeSinceReport > 30*time.Minute || customer.HealthStatus == "warn" {
overallStatus = "warn"
} else if customer.HealthStatus == "fail" {
overallStatus = "down"
} else {
overallStatus = "ok"
}
} }
var hostCause string
overallStatus, hostCause = s.foldHostStatus(customerID, overallStatus, customer != nil)
if cfg != nil && cfg.Status == "blocked" { if cfg != nil && cfg.Status == "blocked" {
overallStatus = "blocked" overallStatus = "blocked"
} }
@@ -290,6 +287,7 @@ func (s *Server) handleCustomerUnified(w http.ResponseWriter, r *http.Request, c
Customer *store.CustomerSummary Customer *store.CustomerSummary
Report map[string]interface{} Report map[string]interface{}
OverallStatus string OverallStatus string
HostCause string // v0.53.0 roll-up: "" or "host down|stale|pending: <id>"
LatestVersion string LatestVersion string
UpdateAvailable bool UpdateAvailable bool
@@ -386,6 +384,7 @@ func (s *Server) handleCustomerUnified(w http.ResponseWriter, r *http.Request, c
Customer: customer, Customer: customer,
Report: report, Report: report,
OverallStatus: overallStatus, OverallStatus: overallStatus,
HostCause: hostCause,
LatestVersion: latestVersion, LatestVersion: latestVersion,
UpdateAvailable: updateAvailable, UpdateAvailable: updateAvailable,
+29 -2
View File
@@ -205,11 +205,38 @@ func (s *Server) pbsdrProvisionAtom(ctx context.Context, customerID string, host
defer cancel() defer cancel()
res, err := s.tenantsync.Provision(ctx, customerID) res, err := s.tenantsync.Provision(ctx, customerID)
if errors.Is(err, tenantsync.ErrTokenExists) { if errors.Is(err, tenantsync.ErrTokenExists) {
// ep0 has a token but the hub has no descriptor — state mismatch (lost hub state or a // ep0 has a token but the hub has no descriptor — state mismatch (lost hub state, a
// half-torn earlier attempt). Never silently re-key: the operator decides via Re-issue. // half-torn earlier attempt, or the F-14 shape: host deleted, tenancy survived).
//
// F-14 gate (operator ruling 2026-07-13): auto-re-issue is permitted ONLY when the
// hub's own deletion record shows the tenancy's owning host — the customer's most
// recent host deletion — was removed through the escrow-ack flow. Acknowledged
// destruction is not silent re-keying; the old secret went down with the acked host.
// No record / un-acked record → the refusal below, byte-unchanged (manual path).
rec, derr := s.store.LatestHostDeletion(customerID)
if derr != nil {
return "", fmt.Errorf("pbsdr: deletion-provenance lookup for %s: %w", customerID, derr)
}
if rec == nil || !rec.EscrowAcked {
return "", fmt.Errorf("the endpoint already holds a PBS token for %s but the hub has no descriptor — use the explicit \"Re-issue PBS credentials\" action", customerID) return "", fmt.Errorf("the endpoint already holds a PBS token for %s but the hub has no descriptor — use the explicit \"Re-issue PBS credentials\" action", customerID)
} }
res, err = s.tenantsync.Reissue(ctx, customerID) // the EXISTING re-issue op — no new endpoint interaction
if err != nil { if err != nil {
return "", fmt.Errorf("pbsdr: F-14 auto re-issue for %s: %w", customerID, err)
}
note := "Previous key destroyed (acknowledged deletion) — credentials re-issued automatically."
details, _ := json.Marshal(map[string]string{
"deleted_host": rec.HostID,
"deleted_at": rec.DeletedAt.UTC().Format(time.RFC3339),
"new_host": host.HostID,
"token_id": res.TokenID,
})
if _, eerr := s.store.SaveEvent(customerID, "pbsdr_auto_reissue", "info", note, string(details), "hub"); eerr != nil {
s.logger.Printf("[WARN] pbsdr: F-14 audit event for %s not stored: %v", customerID, eerr)
}
s.logger.Printf("[INFO] pbsdr F-14 auto re-issue for %s: owning host %s removed via escrow-ack flow (%s) — %s",
customerID, rec.HostID, rec.DeletedAt.UTC().Format(time.RFC3339), note)
} else if err != nil {
return "", err return "", err
} }
+114 -1
View File
@@ -26,7 +26,8 @@ import (
type fakeTenancy struct { type fakeTenancy struct {
provisionCalls int provisionCalls int
reissueCalls int reissueCalls int
err error err error // both ops fail with this
provisionErr error // Provision-only failure (the F-14 token_exists shape: reissue still works)
secret string secret string
} }
@@ -45,6 +46,9 @@ func (f *fakeTenancy) Provision(ctx context.Context, customerID string) (*tenant
if f.err != nil { if f.err != nil {
return nil, f.err return nil, f.err
} }
if f.provisionErr != nil {
return nil, f.provisionErr
}
return f.result(customerID), nil return f.result(customerID), nil
} }
@@ -372,6 +376,115 @@ func TestPBSDR_StorageIDChangeUpdatesDescriptorOnly(t *testing.T) {
} }
} }
// F-14 scenario A (v0.53.0, operator ruling 2026-07-13): a tenancy orphaned by an ESCROW-ACKED
// host delete → the enable path auto-re-issues via the EXISTING tenantsync re-issue op, writes
// the audit event, and provisioning proceeds to a full descriptor + consume-once secret.
// RED-PROOF (Part 1): dropping the provenance write from DeleteHost's tx → the gate finds no
// record → this test fails with the 502 refusal.
func TestPBSDR_F14AutoReissueOnAckedDeletion(t *testing.T) {
fake := &fakeTenancy{provisionErr: tenantsync.ErrTokenExists, secret: "REISSUED-SECRET"}
s, st, logBuf := newPBSDRServer(t, fake)
// The F-14 history: the tenancy's owning host was deleted through the escrow-ack flow
// (real DeleteHost, real escrow row — no hand-set provenance).
if err := st.UpsertHost(&store.Host{HostID: "peti-00-dead", CustomerID: "peti", APIKey: "oldkey"}); err != nil {
t.Fatal(err)
}
if err := st.SaveHostEscrow("peti-00-dead", []byte("opaque"), "fp", "posture", "2026-07-01T00:00:00Z", ""); err != nil {
t.Fatal(err)
}
if err := st.DeleteHost("peti-00-dead", true); err != nil {
t.Fatalf("escrow-ack delete: %v", err)
}
rr := postUpdate(t, s, url.Values{"dr_tier": {"on"}})
if rr.Code != 303 {
t.Fatalf("enable with acked deletion record = %d (%s), want 303 (auto re-issue proceeds)", rr.Code, rr.Body.String())
}
if fake.provisionCalls != 1 || fake.reissueCalls != 1 {
t.Errorf("calls = provision %d / reissue %d, want 1/1 (the EXISTING re-issue op, once)", fake.provisionCalls, fake.reissueCalls)
}
// Full effect: descriptor landed, generation bumped, fresh consume-once secret staged.
desc, desiredJSON, gen := hostState(t, st)
if desc == nil || !desc.Enabled || desc.Namespace != "peti" || desc.TokenID != "felhom@pbs!peti" {
t.Fatalf("descriptor after auto re-issue = %+v (json %s)", desc, desiredJSON)
}
if gen != 1 {
t.Errorf("generation = %d, want 1", gen)
}
if got, err := st.ConsumeHostPBSSecret("peti-01"); err != nil || got != "REISSUED-SECRET" {
t.Fatalf("consume-once secret = (%q, %v), want the re-issued secret", got, err)
}
// The audit line: a stored hub-source event carrying the operator note.
events, err := st.GetRecentEvents("peti", 10)
if err != nil {
t.Fatalf("events: %v", err)
}
var audit *store.Event
for i := range events {
if events[i].EventType == "pbsdr_auto_reissue" {
audit = &events[i]
}
}
if audit == nil {
t.Fatal("no pbsdr_auto_reissue audit event stored")
}
if audit.Source != "hub" || !strings.Contains(audit.Message, "Previous key destroyed (acknowledged deletion)") {
t.Errorf("audit event = %+v, want hub-source with the operator note", audit)
}
if !strings.Contains(audit.DetailsJSON, "peti-00-dead") {
t.Errorf("audit details lack the deleted host: %s", audit.DetailsJSON)
}
if strings.Contains(logBuf.String(), "REISSUED-SECRET") {
t.Error("secret leaked into the hub log")
}
}
// F-14 scenario B (the never-silently-re-key law): a surviving tenancy WITHOUT an acked
// deletion record — none at all, or the latest one un-acked — keeps the current refusal and
// records ZERO re-issue calls (the exact non-effect).
// RED-PROOF (§10 B): removing the record check in pbsdrProvisionAtom (auto-reissue
// unconditionally) → the zero-reissue assertions here fail.
func TestPBSDR_F14NoRecordNeverRekeys(t *testing.T) {
assertRefused := func(t *testing.T, fake *fakeTenancy, s *Server, st *store.Store) {
t.Helper()
rr := postUpdate(t, s, url.Values{"dr_tier": {"on"}})
if rr.Code != 502 || !strings.Contains(rr.Body.String(), "Re-issue PBS credentials") {
t.Fatalf("save = %d (%s), want the byte-unchanged 502 refusal", rr.Code, rr.Body.String())
}
if fake.reissueCalls != 0 {
t.Errorf("reissue calls = %d, want 0 — SILENT RE-KEY", fake.reissueCalls)
}
if desc, _, gen := hostState(t, st); desc != nil || gen != 0 {
t.Errorf("state written on refusal: desc=%+v gen=%d", desc, gen)
}
if _, err := st.ConsumeHostPBSSecret("peti-01"); err != sql.ErrNoRows {
t.Errorf("a secret was staged on refusal (err %v)", err)
}
}
t.Run("no deletion record at all", func(t *testing.T) {
fake := &fakeTenancy{provisionErr: tenantsync.ErrTokenExists, secret: "S"}
s, st, _ := newPBSDRServer(t, fake)
assertRefused(t, fake, s, st)
})
t.Run("latest record un-acked", func(t *testing.T) {
fake := &fakeTenancy{provisionErr: tenantsync.ErrTokenExists, secret: "S"}
s, st, _ := newPBSDRServer(t, fake)
// A deletion happened, but NOT through the escrow-ack flow (no escrow row → acked=false).
if err := st.UpsertHost(&store.Host{HostID: "peti-00-dead", CustomerID: "peti", APIKey: "oldkey"}); err != nil {
t.Fatal(err)
}
if err := st.DeleteHost("peti-00-dead", true); err != nil {
t.Fatal(err)
}
assertRefused(t, fake, s, st)
})
}
func TestPBSDR_Reissue(t *testing.T) { func TestPBSDR_Reissue(t *testing.T) {
fake := &fakeTenancy{secret: "OLD-SECRET"} fake := &fakeTenancy{secret: "OLD-SECRET"}
s, st, logBuf := newPBSDRServer(t, fake) s, st, logBuf := newPBSDRServer(t, fake)
+1
View File
@@ -86,6 +86,7 @@ func TestTemplates_DashboardCriticalBadge(t *testing.T) {
type dashboardCustomer struct { type dashboardCustomer struct {
store.CustomerSummary store.CustomerSummary
OverallStatus string OverallStatus string
HostCause string
BackupAge string BackupAge string
EventCriticals int EventCriticals int
EventErrors int EventErrors int
+78
View File
@@ -0,0 +1,78 @@
package web
// Dead-host roll-up honesty (v0.53.0, drill-1 observation; operator ruling 2026-07-13): a
// customer's status may never look better than its worst expected host. The customer roll-up
// derives from CONTROLLER reports, which reach the hub independently of the host agent — so a
// host DOWN for 23 hours hid behind a green customer row as long as the guest kept reporting
// (the live Peti-cluster shape: proxmox1 down 23h, fresh reports through proxmox2).
//
// foldHostStatus worsens the controller-derived status with per-host staleness via
// (*Server).hostStatus — THE single staleness definition (hosts.go; the same thresholds the
// HostStalenessChecker alerts on — no second definition anywhere). Display + derivation only:
// checker alerting is untouched.
import (
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// controllerStatus is the controller-report-derived customer status — the pre-roll-up chain the
// dashboard, the /configs list and the customer detail all inlined verbatim; this is now the
// ONE copy. Behavior-preserving: the branch order (incl. fail-after-warn) is the historical one.
func controllerStatus(c *store.CustomerSummary) string {
switch {
case c.HealthStatus == "disabled":
return "disabled"
case c.TimeSinceReport > time.Hour:
return "down"
case c.TimeSinceReport > 30*time.Minute || c.HealthStatus == "warn":
return "warn"
case c.HealthStatus == "fail":
return "down"
default:
return "ok"
}
}
// hostFoldRank orders host states by badness for the worst-host pick. "ok" ranks 0 (never folds).
var hostFoldRank = map[string]int{"down": 3, "stale": 2, "pending": 1}
// hostFoldLabel is the operator-facing cause chip prefix per worst-host state.
var hostFoldLabel = map[string]string{"down": "host down", "stale": "host stale", "pending": "host pending"}
// foldHostStatus folds the customer's expected hosts into a controller-derived status:
// worst(controllerDerived, hostStatusOf(each host)). Any host down/stale caps the customer at
// WARN (a green row over a dead host is the masking bug); the returned cause names the state
// AND the host ("host down: <id>") so the detail header says WHICH host. "pending" hosts
// (enrolled, never reported) worsen only after initial onboarding — customerHasReported=false
// (the customer has never reported) excludes them, a half-installed box is not an incident.
// Statuses worse than warn (down) and administrative ones (disabled/blocked) keep their own
// token; the cause chip still surfaces the host signal. Read errors degrade to the unfolded
// status — the page must render.
func (s *Server) foldHostStatus(customerID, base string, customerHasReported bool) (status, cause string) {
hosts, err := s.store.ListHostsByCustomer(customerID)
if err != nil {
s.logger.Printf("[ERROR] roll-up: ListHostsByCustomer %s: %v", customerID, err)
return base, ""
}
worst, worstHost := "", ""
for i := range hosts {
hs := s.hostStatus(hosts[i].LastReportAt)
if hs == "pending" && !customerHasReported {
continue
}
if hostFoldRank[hs] > hostFoldRank[worst] {
worst, worstHost = hs, hosts[i].HostID
}
}
if worst == "" {
return base, ""
}
cause = hostFoldLabel[worst] + ": " + worstHost
// The fold worsens, never improves: ok / pending / no-report ("") cap at warn.
if base == "ok" || base == "pending" || base == "" {
return "warn", cause
}
return base, cause
}
+194
View File
@@ -0,0 +1,194 @@
package web
// v0.53.0 dead-host roll-up honesty (drill-1 masking observation; operator ruling 2026-07-13).
// Scenario C is the LIVE Peti-cluster shape: a host down 23h while the controller keeps
// reporting through the internet — pre-fix the customer row rendered GREEN (the wrong outcome
// these tests pin). Scenario D bounds the fold: all-ok is a no-regression pass-through, a stale
// host warns with its own chip, and pending hosts worsen only after onboarding.
// RED-PROOF (§10 C): short-circuiting foldHostStatus to `return base, ""` (controller-only
// derivation) → the C assertions fail with the green row / missing chip visible.
import (
"database/sql"
"fmt"
"io"
"log"
"net/http/httptest"
"path/filepath"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// newRollupServer builds a server whose store DB path is known, so tests can backdate host
// reports over a second connection (the monitor host_staleness_test pattern).
func newRollupServer(t *testing.T) (*Server, *store.Store, *sql.DB) {
t.Helper()
path := filepath.Join(t.TempDir(), "t.db")
st, err := store.New(path, log.New(io.Discard, "", 0))
if err != nil {
t.Fatalf("store.New: %v", err)
}
t.Cleanup(func() { st.Close() })
db, err := sql.Open("sqlite", path)
if err != nil {
t.Fatalf("sql.Open: %v", err)
}
t.Cleanup(func() { db.Close() })
// staleThreshold 30m → host "stale" past 30m, "down" past 60m (the single definition).
s := New(st, "", "", "test", 30*time.Minute, log.New(io.Discard, "", 0))
return s, st, db
}
func backdateHost(t *testing.T, db *sql.DB, hostID string, minutesAgo int) {
t.Helper()
if _, err := db.Exec(`UPDATE hosts SET last_report_at = datetime('now', ?) WHERE host_id = ?`,
fmt.Sprintf("-%d minutes", minutesAgo), hostID); err != nil {
t.Fatal(err)
}
}
func renderDashboard(t *testing.T, s *Server) string {
t.Helper()
rr := httptest.NewRecorder()
s.handleDashboard(rr, httptest.NewRequest("GET", "/", nil))
if rr.Code != 200 {
t.Fatalf("dashboard = %d", rr.Code)
}
return rr.Body.String()
}
func renderCustomer(t *testing.T, s *Server, customerID string) string {
t.Helper()
rr := httptest.NewRecorder()
s.handleCustomerUnified(rr, httptest.NewRequest("GET", "/customers/"+customerID, nil), customerID)
if rr.Code != 200 {
t.Fatalf("customer detail = %d", rr.Code)
}
return rr.Body.String()
}
// Scenario C — the Peti shape: proxmox1 down 23h, fresh controller report → the customer row
// must read WARN with the cause chip naming the host; the detail header must say WHICH host.
// WRONG outcome (pre-fix): a green row.
func TestRollup_DeadHostMasking(t *testing.T) {
s, st, db := newRollupServer(t)
if err := st.SaveCustomerConfig(&store.CustomerConfig{
CustomerID: "acme", CustomerName: "Acme", APIKey: "k", RetrievalPassword: "p",
}); err != nil {
t.Fatal(err)
}
// Fresh controller report (the guest reports hub-direct, independent of the host agent).
if err := st.SaveReport("acme", []byte(`{"customer_id":"acme","customer_name":"Acme"}`)); err != nil {
t.Fatal(err)
}
// The host: reported once, then silent for 23h → "down" by THE definition.
if err := st.UpsertHost(&store.Host{HostID: "proxmox1", CustomerID: "acme", APIKey: "hk"}); err != nil {
t.Fatal(err)
}
if err := st.SaveHostReport("proxmox1", "acme", []byte(`{}`), store.HostReportDenorm{}); err != nil {
t.Fatal(err)
}
backdateHost(t, db, "proxmox1", 23*60)
body := renderDashboard(t, s)
if !strings.Contains(body, "host down: proxmox1") {
t.Error("dashboard row lacks the cause chip \"host down: proxmox1\"")
}
if strings.Contains(body, "status-badge-ok") {
t.Error("dashboard row is GREEN over a 23h-dead host — the exact masking bug")
}
if !strings.Contains(body, `status-badge-warn">host down: proxmox1`) {
t.Error("cause chip not rendered as a warn badge")
}
// Detail header: WHICH host.
detail := renderCustomer(t, s, "acme")
if !strings.Contains(detail, "host down: proxmox1") {
t.Error("customer detail header does not name the down host")
}
}
// Scenario D — boundaries: all-ok pass-through (no regression), stale-host warn chip, and the
// onboarding rule for pending hosts.
func TestRollup_Boundaries(t *testing.T) {
t.Run("all hosts ok leaves controller status untouched", func(t *testing.T) {
s, st, _ := newRollupServer(t)
if err := st.SaveReport("acme", []byte(`{"customer_id":"acme"}`)); err != nil {
t.Fatal(err)
}
if err := st.UpsertHost(&store.Host{HostID: "h-ok", CustomerID: "acme", APIKey: "hk"}); err != nil {
t.Fatal(err)
}
if err := st.SaveHostReport("h-ok", "acme", []byte(`{}`), store.HostReportDenorm{}); err != nil {
t.Fatal(err)
}
body := renderDashboard(t, s)
if !strings.Contains(body, "status-badge-ok") {
t.Error("healthy customer + healthy host must stay OK")
}
if strings.Contains(body, "host down") || strings.Contains(body, "host stale") || strings.Contains(body, "host pending") {
t.Error("cause chip rendered with every host ok")
}
})
t.Run("single stale host caps at warn with stale chip", func(t *testing.T) {
s, st, db := newRollupServer(t)
if err := st.SaveReport("acme", []byte(`{"customer_id":"acme"}`)); err != nil {
t.Fatal(err)
}
if err := st.UpsertHost(&store.Host{HostID: "h-stale", CustomerID: "acme", APIKey: "hk"}); err != nil {
t.Fatal(err)
}
if err := st.SaveHostReport("h-stale", "acme", []byte(`{}`), store.HostReportDenorm{}); err != nil {
t.Fatal(err)
}
backdateHost(t, db, "h-stale", 45) // between stale (30m) and down (60m)
body := renderDashboard(t, s)
if !strings.Contains(body, "host stale: h-stale") {
t.Error("stale-host cause chip missing")
}
if strings.Contains(body, "status-badge-ok") {
t.Error("customer stayed green over a stale host")
}
})
t.Run("pending host during onboarding does not worsen", func(t *testing.T) {
s, st, _ := newRollupServer(t)
// Config-only customer (never reported) + freshly-minted host (never reported).
if err := st.SaveCustomerConfig(&store.CustomerConfig{
CustomerID: "newbie", CustomerName: "Newbie", APIKey: "k", RetrievalPassword: "p",
}); err != nil {
t.Fatal(err)
}
if err := st.UpsertHost(&store.Host{HostID: "h-new", CustomerID: "newbie", APIKey: "hk"}); err != nil {
t.Fatal(err)
}
body := renderDashboard(t, s)
if !strings.Contains(body, "status-badge-pending") {
t.Error("onboarding customer must render PENDING")
}
if strings.Contains(body, "host pending") {
t.Error("a never-reported host worsened a never-reported customer (onboarding exclusion violated)")
}
})
t.Run("pending host after onboarding worsens", func(t *testing.T) {
s, st, _ := newRollupServer(t)
if err := st.SaveReport("acme", []byte(`{"customer_id":"acme"}`)); err != nil {
t.Fatal(err)
}
if err := st.UpsertHost(&store.Host{HostID: "h-silent", CustomerID: "acme", APIKey: "hk"}); err != nil {
t.Fatal(err)
}
body := renderDashboard(t, s)
if !strings.Contains(body, "host pending: h-silent") {
t.Error("a never-reported host must worsen a LIVE customer (post-onboarding)")
}
if strings.Contains(body, "status-badge-ok") {
t.Error("customer stayed green over a never-reported host")
}
})
}
+7 -12
View File
@@ -611,6 +611,7 @@ func (s *Server) handleDashboard(w http.ResponseWriter, r *http.Request) {
type dashboardCustomer struct { type dashboardCustomer struct {
store.CustomerSummary store.CustomerSummary
OverallStatus string // "ok", "warn", "down", "pending" OverallStatus string // "ok", "warn", "down", "pending"
HostCause string // v0.53.0 roll-up: "" or "host down|stale|pending: <id>"
BackupAge string BackupAge string
EventCriticals int EventCriticals int
EventErrors int EventErrors int
@@ -629,18 +630,9 @@ func (s *Server) handleDashboard(w http.ResponseWriter, r *http.Request) {
seen[c.CustomerID] = true seen[c.CustomerID] = true
dc := dashboardCustomer{CustomerSummary: c} dc := dashboardCustomer{CustomerSummary: c}
// Determine overall status // Controller-derived status + the v0.53.0 dead-host roll-up (rollup.go): a customer
if c.HealthStatus == "disabled" { // may never look better than its worst expected host.
dc.OverallStatus = "disabled" dc.OverallStatus, dc.HostCause = s.foldHostStatus(c.CustomerID, controllerStatus(&c), true)
} else if c.TimeSinceReport > time.Hour {
dc.OverallStatus = "down"
} else if c.TimeSinceReport > 30*time.Minute || c.HealthStatus == "warn" {
dc.OverallStatus = "warn"
} else if c.HealthStatus == "fail" {
dc.OverallStatus = "down"
} else {
dc.OverallStatus = "ok"
}
// Backup age // Backup age
if c.BackupLastSnapshot != nil { if c.BackupLastSnapshot != nil {
@@ -672,6 +664,9 @@ func (s *Server) handleDashboard(w http.ResponseWriter, r *http.Request) {
OverallStatus: "pending", OverallStatus: "pending",
BackupAge: "", BackupAge: "",
} }
// Roll-up for the never-reported customer too: a down/stale host worsens even during
// onboarding — only never-reported ("pending") hosts are excluded here.
dc.OverallStatus, dc.HostCause = s.foldHostStatus(cfg.CustomerID, dc.OverallStatus, false)
data = append(data, dc) data = append(data, dc)
} }
+1
View File
@@ -63,6 +63,7 @@
<span class="status-badge status-badge-{{.OverallStatus}}"> <span class="status-badge status-badge-{{.OverallStatus}}">
{{if eq .OverallStatus "ok"}}OK{{else if eq .OverallStatus "warn"}}WARN{{else if eq .OverallStatus "down"}}DOWN{{else if eq .OverallStatus "disabled"}}PAUSED{{else if eq .OverallStatus "pending"}}PENDING{{else}}{{.OverallStatus}}{{end}} {{if eq .OverallStatus "ok"}}OK{{else if eq .OverallStatus "warn"}}WARN{{else if eq .OverallStatus "down"}}DOWN{{else if eq .OverallStatus "disabled"}}PAUSED{{else if eq .OverallStatus "pending"}}PENDING{{else}}{{.OverallStatus}}{{end}}
</span> </span>
{{if .HostCause}}<span class="status-badge status-badge-warn">{{.HostCause}}</span>{{end}}
{{else}} {{else}}
<span class="text-muted"></span> <span class="text-muted"></span>
{{end}} {{end}}
@@ -25,6 +25,7 @@
<h1> <h1>
<span class="status-dot status-dot-{{statusColor .OverallStatus}}"></span> <span class="status-dot status-dot-{{statusColor .OverallStatus}}"></span>
{{if .CustomerName}}{{.CustomerName}}{{else}}{{.CustomerID}}{{end}} {{if .CustomerName}}{{.CustomerName}}{{else}}{{.CustomerID}}{{end}}
{{if .HostCause}}<span class="status-badge status-badge-warn">{{.HostCause}}</span>{{end}}
</h1> </h1>
{{if .HasReports}} {{if .HasReports}}
<p class="subtitle"> <p class="subtitle">
@@ -69,6 +70,7 @@
<div class="summary-strip"> <div class="summary-strip">
<span class="strip-name">{{if .CustomerName}}{{.CustomerName}}{{else}}{{.CustomerID}}{{end}}</span> <span class="strip-name">{{if .CustomerName}}{{.CustomerName}}{{else}}{{.CustomerID}}{{end}}</span>
<span class="strip-item"><span class="status-dot status-dot-{{statusColor .OverallStatus}}"></span> {{.OverallStatus}}</span> <span class="strip-item"><span class="status-dot status-dot-{{statusColor .OverallStatus}}"></span> {{.OverallStatus}}</span>
{{if .HostCause}}<span class="strip-item"><span class="status-badge status-badge-warn">{{.HostCause}}</span></span>{{end}}
{{if .HasReports}} {{if .HasReports}}
<span class="strip-item">Controller <code>{{.Customer.ControllerVersion}}</code></span> <span class="strip-item">Controller <code>{{.Customer.ControllerVersion}}</code></span>
<span class="strip-item">Last report {{timeAgo .Customer.ReceivedAt}}</span> <span class="strip-item">Last report {{timeAgo .Customer.ReceivedAt}}</span>
@@ -54,6 +54,7 @@
<span class="status-badge status-badge-{{.OverallStatus}}"> <span class="status-badge status-badge-{{.OverallStatus}}">
{{if eq .OverallStatus "ok"}}OK{{else if eq .OverallStatus "warn"}}WARN{{else if eq .OverallStatus "disabled"}}PAUSED{{else if eq .OverallStatus "pending"}}PENDING{{else}}DOWN{{end}} {{if eq .OverallStatus "ok"}}OK{{else if eq .OverallStatus "warn"}}WARN{{else if eq .OverallStatus "disabled"}}PAUSED{{else if eq .OverallStatus "pending"}}PENDING{{else}}DOWN{{end}}
</span> </span>
{{if .HostCause}}<span class="status-badge status-badge-warn">{{.HostCause}}</span>{{end}}
</td> </td>
<td>{{if eq .OverallStatus "pending"}}—{{else}}{{if gt (add (add .EventCriticals .EventErrors) .EventWarnings) 0}}{{if gt .EventCriticals 0}}<span class="severity-badge severity-critical">{{.EventCriticals}}</span>{{end}}{{if gt .EventErrors 0}}<span class="severity-badge severity-error">{{.EventErrors}}</span>{{end}}{{if gt .EventWarnings 0}}<span class="severity-badge severity-warning">{{.EventWarnings}}</span>{{end}}{{else}}<span class="text-muted"></span>{{end}}{{end}}</td> <td>{{if eq .OverallStatus "pending"}}—{{else}}{{if gt (add (add .EventCriticals .EventErrors) .EventWarnings) 0}}{{if gt .EventCriticals 0}}<span class="severity-badge severity-critical">{{.EventCriticals}}</span>{{end}}{{if gt .EventErrors 0}}<span class="severity-badge severity-error">{{.EventErrors}}</span>{{end}}{{if gt .EventWarnings 0}}<span class="severity-badge severity-warning">{{.EventWarnings}}</span>{{end}}{{else}}<span class="text-muted"></span>{{end}}{{end}}</td>
<td>{{if eq .OverallStatus "pending"}}—{{else}}{{timeAgo .ReceivedAt}}{{end}}</td> <td>{{if eq .OverallStatus "pending"}}—{{else}}{{timeAgo .ReceivedAt}}{{end}}</td>
+20 -12
View File
@@ -15,14 +15,12 @@
# htpasswd -nbBC 10 "" "your-password" | cut -d: -f2 # htpasswd -nbBC 10 "" "your-password" | cut -d: -f2
# Update the ConfigMap password_hash field below. # Update the ConfigMap password_hash field below.
# #
# 3. Generate a report API key (shared secret for controllers): # 3. Create the operator/global bearer key Secret (out-of-band, NEVER committed):
# openssl rand -hex 32 # openssl rand -hex 32 # mint
# Update the ConfigMap report_api_key field below. # kubectl -n felhom-system create secret generic report-api \
# Then add the same key to each customer's controller.yaml: # --from-literal=REPORT_API_KEY=<minted-key>
# hub: # (Customer boxes use per-customer/per-host keys generated by the hub — the global
# enabled: true # key is the operator's own, e.g. felhom-ops -hub-key.)
# url: "https://hub.felhom.eu"
# api_key: "<same-key>"
# #
# 4. Apply this manifest: # 4. Apply this manifest:
# kubectl apply -f manifests/hub.yaml # kubectl apply -f manifests/hub.yaml
@@ -71,10 +69,11 @@ data:
# Generate: htpasswd -nbBC 10 "" "your-password" | cut -d: -f2 # Generate: htpasswd -nbBC 10 "" "your-password" | cut -d: -f2
password_hash: "$2y$10$N5.O9jBnc.1tIlJT/irx3OlVjJQemlCHRnfqIJg/EyZofnzXSCpeG" password_hash: "$2y$10$N5.O9jBnc.1tIlJT/irx3OlVjJQemlCHRnfqIJg/EyZofnzXSCpeG"
api: api:
# Shared secret for controller → hub report push # Operator/global bearer key. NOT stored here since v0.53.0 — injected at runtime from
# Generate: openssl rand -hex 32 # Secret/report-api via the REPORT_API_KEY env var (see Deployment below). The Secret is
# Must match hub.api_key in each customer's controller.yaml # created out-of-band and NOT committed (documentation/runbooks/secrets.md); the previously
report_api_key: "094091de545ce28795c47ac2158fc30750db5c24a621c49329b001ee8db57fb8" # committed literal is retired by ROTATION (see the publish-runbook notes). Leave empty.
report_api_key: ""
retention: retention:
max_days: 90 max_days: 90
prune_schedule: "04:30" prune_schedule: "04:30"
@@ -144,6 +143,15 @@ spec:
secretKeyRef: secretKeyRef:
name: resend-api name: resend-api
key: RESEND_API_KEY key: RESEND_API_KEY
# Operator/global bearer key — injected from the out-of-band Secret/report-api
# (NOT committed; documentation/runbooks/secrets.md). Deliberately NOT optional:
# a missing Secret must fail the pod Ready rather than boot an unauthenticatable
# hub with an empty bearer key. Create the Secret BEFORE syncing this manifest.
- name: REPORT_API_KEY
valueFrom:
secretKeyRef:
name: report-api
key: REPORT_API_KEY
- name: REGISTRY_USERNAME - name: REGISTRY_USERNAME
valueFrom: valueFrom:
secretKeyRef: secretKeyRef:
+53
View File
@@ -0,0 +1,53 @@
# -*- coding: utf-8 -*-
"""Manifest bearer-literal gate (v0.53.0, part of the hub bearer de-git) — no bearer-shaped
literal (64 hex chars, the `openssl rand -hex 32` shape every felhom bearer/API key uses) may
appear ANYWHERE in manifests/, comments included. Secrets ride out-of-band `kubectl create
secret` + secretKeyRef (documentation/runbooks/secrets.md); the manifests carry only
placeholders. The other known committed secrets in felhom.secret.yaml (passwords, non-hex
shapes) are a tracked backlog item (secrets.md) and are NOT matched by this gate extend the
patterns when they are de-gitted.
Run from the repo root: python scripts/manifest_bearer_gate.py
Exit 1 on any hit.
(Named "bearer", not "secret": the repo .gitignore's `*secret*` pattern — which guards real
secret files would silently un-track a gate with "secret" in its filename.)
"""
import io, os, re, sys
ROOT = "manifests"
# 64 hex chars with no hex/word neighbors (so longer blobs and sha256-of-file hexes embedded in
# longer strings still match at 64+, but ordinary short ids never do).
BEARER = re.compile(r"(?<![0-9a-fA-F])[0-9a-fA-F]{64}(?![0-9a-fA-F])")
# KNOWN BACKLOG (non-fatal, stays VISIBLE): felhom.secret.yaml commits pre-existing secrets
# (umami APP_SECRET is 64-hex) tracked for de-git in documentation/runbooks/secrets.md — out of
# the bearer-de-git scope (2026-07-13 operator ruling batch). Remove this carve-out when that
# file is cleaned; new bearer literals must NOT be hidden behind it.
KNOWN_BACKLOG = {"felhom.secret.yaml"}
def main():
total = 0
for fn in sorted(os.listdir(ROOT)):
if not fn.endswith((".yaml", ".yml")):
continue
path = os.path.join(ROOT, fn)
for lineno, line in enumerate(io.open(path, encoding="utf-8", errors="replace"), 1):
for m in BEARER.finditer(line):
masked = m.group(0)[:8] + "..." + m.group(0)[-4:]
if fn in KNOWN_BACKLOG:
print("%s:%d KNOWN-BACKLOG committed secret %s (secrets.md de-git backlog; not this gate's failure)"
% (path, lineno, masked))
continue
total += 1
print("%s:%d bearer-shaped literal %s" % (path, lineno, masked))
if total:
print("MANIFEST BEARER GATE FAILED: %d bearer-shaped literal(s) in manifests/" % total)
sys.exit(1)
print("manifest bearer gate OK - no bearer-shaped literals in manifests/")
if __name__ == "__main__":
main()