docs: REPORT overwrite - hub v0.53.0 closing bundle (step-0 citation, four red-proofs, scenario E live legs, s13.2 roll-up before/after/recovery evidence, consumer list, operator STOP points)

REPORT.md

@@ -2,47 +2,167 @@
 
 > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
 
-## Polish batch — hub v0.52.0 (F-15 instant reset codes + F-16 inline confirms) — 2026-07-13
+## Hub closing bundle — v0.53.0 (F-14 gated auto-Reissue + dead-host roll-up + bearer out of git) — 2026-07-13
 
-Hub half of the polish batch (take-two findings F-15/F-16); companion controller v0.123.0.
-Commits `bad9203` (F-15), `5d66c62` (F-16), `0e05c69` (manifest 0.52.0).
+The last engineering items on the pre-tester board. Commits `2321077` (part 1), `04861a7`
+(part 2), `36c7213` (part 3), `80aca3a` + `3682303` (part 4 + gate re-land), `3efb1b1` (docs),
+`3616dcb` (manifest 0.53.0); felhom-controller `51c871a` (example-config scrub).
+Deployed: **felhom-hub:0.53.0**, ArgoCD Synced/Healthy, rollout clean, startup log clean.
 
-### What shipped
+### Baselines (re-confirmed live before work)
 
-1. **F-15** (`internal/api/handler.go`): `POST /api/v1/claim/reset-request` returns the ACTIVE
-   claim state — `{claim: {code_hash, generation, issued_at}}`, the report ACK's exact shape and
-   bcrypt-only guarantee — so the box applies the rotated hash in the SAME request cycle and the
-   emailed code works immediately (was: next-ACK, ~15 min — Viktor's take-two live failure).
-   Served on every authorized outcome; a cap-reached refusal returns the unrotated row (a
-   controller-side no-op by generation). The operator "Kód újraküldése" keeps its ACK lag —
-   its flash + confirm copy now state it; the unmapped `claim-resent`/`claim-resend-failed`
-   flash codes finally render.
-2. **F-16** (`templates/inline_confirm.html` + 4 templates + `scripts/hub_confirm_gate.py`):
-   every native `confirm()` in the hub UI replaced by the inline "question + Igen/Mégse"
-   two-step (`felhomConfirm` / `data-confirm` + `requestSubmit`, so `formaction` buttons riding
-   the parent form still work). Danger-zone typed-confirm untouched by design. Gate enforces
-   zero native confirm/prompt.
+- Hub v0.52.0 at head (`74fa61c`, past the 0.126.x doc commits); manifest image `0.52.0`.
+- The operator bearer literal committed at `manifests/hub.yaml:77` (the runbook-flagged item).
 
-### Tests + red-proofs
+### Step-0 citation — where the tenancy↔host linkage lives
 
-- `TestClaimResetRequest_ResponseCarriesRotatedHash` (rotated hash == stored row, != pre-rotation
-  row, bcrypt-only). **Red-proof:** handler mutated to serve the pre-rotation row → FAIL
-  "response generation = 1, want the rotated one (stored 2)". Restored → green.
-- `hub_confirm_gate.py` **red-proof:** one `onclick="return confirm('redproof')"` reintroduced →
-  exit 1; restored → 0. Full hub green gate (build+vet+test) before each commit.
+The ep0 tenancy is **customer-keyed** (`tenantsync.Provision(ctx, customerID)`; namespace +
+`felhom@pbs!<customer>` token per customer — `hub/internal/tenantsync/client.go`). Its only
+host-side anchors are (a) the `pbs_dr` descriptor inside `hosts.desired_json`
+(`hub/internal/web/pbsdr.go:63-92` read/merge) and (b) the consume-once secret in
+`host_pbs_secrets` keyed by host_id (`hub/internal/store/pbsdr.go:10-17`). **Both die inside the
+DeleteHost cascade** (`hub/internal/store/store.go` — `host_pbs_secrets` and `hosts` deletes in
+the one-tx statement list), so no surviving row could carry a provenance column → a dedicated
+`host_deletions` table is the justified shape. The F-14 gate matches on `customer_id` because
+the tenancy is customer-scoped; it reads the MOST RECENT record so an old acked deletion never
+whitelists a newer un-acked one.
 
-### Deployed + live-validated
+### Part 1 — deletion provenance (`host_deletions`)
 
-- ArgoCD app `felhom` **Synced/Healthy**, deploy/hub image `felhom-hub:0.52.0`, clean startup log.
-- **F-15 live:** drill-box reset request → controller cached the response hash **1 s** after the
-  POST; Viktor re-ran his exact take-two failure path — emailed code **accepted immediately**.
-- **F-16 live (its own regression test):** "Re-issue offsite credentials" on demo-vm-felhom
-  driven fully by browser automation — inline strip rendered, Igen → `?flash=offsite_reissued`,
-  **zero CDP freezing** (with v0.51.0 this exact click froze the session).
+- Schema: `id, host_id, customer_id, deleted_at, escrow_acked` + customer index
+  (store.go, additive v0.53.0 block). **No backfill** — pre-record deletions (incl. the
+  take-two drill's own ep0 residue, already manually cleared) legitimately fall to the manual
+  path.
+- Written **inside** the DeleteHost transaction, before the cascade deletes read the host row
+  away. `escrow_acked = deleteEscrow AND an escrow row was actually present` — "removed through
+  the escrow-ack flow" means an acknowledged destruction happened; ticking the box over nothing
+  records `false` (justification: the ruling's operative fact is destroyed-with-acknowledgement,
+  not checkbox state; pinned by test).
+- `LatestHostDeletion(customerID)` — newest record only.
+
+### Part 2 — F-14 gated auto-Reissue
+
+`pbsdrProvisionAtom` (pbsdr.go), on `tenantsync.ErrTokenExists`: reads the customer's latest
+deletion record. `escrow_acked=true` → invokes the **existing** `tenantsync.Reissue` op (no new
+endpoint interaction), stores the `pbsdr_auto_reissue` hub-source audit event with the operator
+note **"Previous key destroyed (acknowledged deletion) — credentials re-issued automatically."**
+(+ deleted_host/deleted_at/new_host/token_id details), logs INFO, and proceeds into the normal
+secret+descriptor+bump tail. No record / un-acked → the pre-existing refusal string,
+**byte-unchanged**, and the manual Re-issue button path untouched. Both rulings recorded
+verbatim in CONTEXT.md; take-two F-14 row dispositioned SHIPPED (live validation = the next
+real host-reset cycle, per spec option 4 — fixtures + red-proofs carry it until then).
+
+### Part 3 — dead-host roll-up honesty
+
+- `hub/internal/web/rollup.go`: `controllerStatus()` (the ONE copy of the previously
+  triple-inlined controller-derived chain — behavior-preserving) +
+  `(s *Server) foldHostStatus(customerID, base, customerHasReported)` =
+  `worst(controllerDerived, hostStatusOf(each expected host))` via **`Server.hostStatus`
+  only** (hosts.go — the single staleness definition shared with HostStalenessChecker; no
+  second threshold introduced anywhere).
+- Mapping: host `down`/`stale` → customer capped at **warn** + cause chip
+  `host down|stale: <host_id>`; `pending` hosts worsen only once the customer has ever
+  reported (customer-summary row exists — the durable proxy for "never had an ok report";
+  reports prune at 90d so a per-report query can't be the definition). Statuses already worse
+  (down) or administrative (disabled/blocked) keep their token; the chip still names the host.
+- Wired at all three derivation sites: dashboard (server.go), /configs list (configs.go),
+  customer detail header + summary strip (configs.go + customer_unified.html). Chip renders as
+  an existing `status-badge status-badge-warn` token (no new CSS). Display + derivation only —
+  checker alerting untouched. `hub_confirm_gate.py` green.
+
+### Part 4 — bearer out of git
+
+- `manifests/hub.yaml`: ConfigMap `report_api_key` → empty placeholder; Deployment env
+  `REPORT_API_KEY` ← `secretKeyRef: report-api/REPORT_API_KEY`, **deliberately not
+  `optional:`** (missing Secret must fail Ready). `cmd/hub/main.go`: `REPORT_API_KEY` env
+  override (the RESEND_API_KEY twin) — the hub read the key from the YAML config, so this
+  minimal adaptation was required (cited: main.go env-override block, Config.API.ReportAPIKey).
+- Gate: `scripts/manifest_bearer_gate.py` — 64-hex bearer shapes anywhere in `manifests/`,
+  exit 1; `felhom.secret.yaml`'s pre-existing umami APP_SECRET prints as a **visible non-fatal
+  KNOWN-BACKLOG line** (secrets.md backlog, out of scope — no history rewrite, no umami touch).
+  GOTCHA found: the repo `.gitignore` `*secret*` pattern silently un-tracked the gate's first
+  name (`manifest_secret_gate.py` never landed in commit `80aca3a`) — re-landed as
+  `manifest_bearer_gate.py` (`3682303`).
+- **Consumer list (the GLOBAL key only — per-customer/per-host keys are hub-generated and
+  unaffected):** (1) the hub itself (now via Secret/report-api); (2) operator-typed tooling —
+  `felhom-ops … -hub-key` (GL-6/PBS-DR runbooks) and the runbook curl probes
+  (break-glass.md §, offsite-endpoint.md §§) — all reference it "out-of-band", nothing
+  machine-persisted; (3) `felhom-controller/controller/configs/controller.yaml.example` carried
+  the real literal as example text (never a live consumer) — **scrubbed** (`51c871a`).
+- Both publish runbooks' ROTATION notes now carry the 2026-07-13 disposition; full
+  create/rotate/verify procedure in `documentation/runbooks/secrets.md` §"Operator/global
+  bearer key".
+
+### Deploy + live validation (§13)
+
+- Image `felhom-hub:0.53.0` built+pushed on 180; **Secret/report-api created FIRST**
+  (value extracted from git history on 180 straight into a 0600 file — never echoed —
+  `kubectl create secret`, file shredded); manifest bump `3616dcb`; ArgoCD hard-refresh +
+  deliberate sync → **Synced/Healthy**, rollout OK, image verified, startup log clean.
+- **Scenario E live:** authed probe with the Secret-provided value → **200**; wrong key →
+  **401**; /healthz 200. Fail-Ready property verified non-disruptively: a throwaway pod with
+  the same non-optional `secretKeyRef` shape against an absent Secret name →
+  `CreateContainerConfigError` (container never starts); the live manifest's `report-api` ref
+  carries zero `optional:` flags.
+- **§13.2 roll-up live leg (drill host `demo-vm-felhom-2f4b00`, hostname drill-day0) —
+  the exact masking shape, disproven live, then recovered (all via the logged-in operator
+  browser session; screenshots recorded):**
+  - BEFORE (13:0x Z): dashboard "Demo in VM = **OK**, no chip"; /hosts host **ONLINE**.
+  - `systemctl stop felhom-agent` at **13:09:28Z**; the guest controller kept reporting.
+  - AFTER (~13:45Z, past the 30m stale threshold): dashboard row
+    **"Demo in VM = WARN + chip `HOST STALE: DEMO-VM-FELHOM-2F4B00`"** with the controller's
+    last-seen at **1 min ago** — pre-fix this row was green (the drill-1/Peti masking bug).
+    Customer detail header + summary strip both carry the same chip → the header says WHICH
+    host; the controller Health card honestly still reads STATUS: OK (the warn comes from the
+    host fold, correctly attributed). The HostStalenessChecker independently emitted its
+    warning event (events badge 2→4) — alerting untouched, as specified.
+  - RECOVERY: `systemctl start felhom-agent` at **13:46:22Z** → first heartbeat → row back to
+    **OK, chip gone** by 13:47Z. Both states + recovery recorded.
+
+### Red-proofs (all run → FAIL → restored; `git diff` clean after each)
+
+| # | Mutation | Failing evidence |
+|---|---|---|
+| Part 1 | provenance INSERT dropped from the DeleteHost tx | `TestDeleteHost_ProvenanceRecord`: "no deletion record written by the escrow-ack delete"; scenario A: 502 refusal instead of 303 (gate found nothing) |
+| B | gate bypassed (auto-reissue unconditionally, nil-safe) | both `TestPBSDR_F14NoRecordNeverRekeys` subtests: save = **303** (silent re-key proceeded) instead of the 502 refusal |
+| C | `foldHostStatus` short-circuited to controller-only | `TestRollup_DeadHostMasking`: "dashboard row is GREEN over a 23h-dead host" + missing chip + detail header silent; 2 boundary subtests also red |
+| E | 64-hex literal reintroduced into manifests/hub.yaml | `manifest_bearer_gate.py` exit **1**, literal reported masked (`00000000...0000`) |
+
+### Tests (scenarios A–E)
+
+- A: `TestPBSDR_F14AutoReissueOnAckedDeletion` — real escrow-ack DeleteHost, real handler path;
+  asserts reissue call count, descriptor, generation, consume-once secret, the stored audit
+  event + note text, and zero secret leakage into logs.
+- B: `TestPBSDR_F14NoRecordNeverRekeys` (no record / latest un-acked) — the byte-unchanged 502
+  + **zero** Reissue calls + zero state writes (the exact non-effect).
+- C: `TestRollup_DeadHostMasking` — the Peti fixture (host backdated 23h, controller report
+  fresh) rendered through the REAL dashboard + customer handlers.
+- D: `TestRollup_Boundaries` — all-ok pass-through, stale-host warn chip, onboarding pending
+  exclusion, post-onboarding pending worsening.
+- E: gate (committed) + the live auth/fail-Ready legs above.
+- Full hub green gate: `go build ./... && go vet ./... && go test ./...` — all packages ok.
 
 ### Observations
 
-- Take-two §4 dispositions updated: F-15 and F-16 SHIPPED (this batch); the zero-toggle obs.
-  shipped controller-side (v0.123.0).
-- Remaining from take-two: **F-14** (tenancy re-attach dead-end — needs the operator ruling on
-  auto-Reissue vs never-silently-re-key) and the S5 DR-restore drill.
+- **Peti's box is BACK ONLINE** (dashboard: Peti Proxmox OK, controller **0.115.0**, agent
+  **0.81.0**, fresh reports; 5/5/5 event badges). The parked publish-train items (Peti D/E/G:
+  sign agent_update 0.79→0.85 — his agent already shows 0.81 —, journal group, G1 temp-creds
+  deletion) look actionable; his `motioneye` storage target reads **100% full** (WORST STORAGE
+  on /hosts). Operator follow-up.
+- The spec's "F-13 reference resolved as typo-or-assigned": the take-two record's findings run
+  F-14/F-15/F-16; **F-13 is the CAMPAIGN-3 nfs4-under-mp8 finding** (separate, still open) —
+  the numbering is continuous across records, no typo, nothing to reassign.
+- The "remove from system" decommission finisher stays parked (untouched by this bundle).
+- The `.gitignore *secret*` un-tracking gotcha (above) is recorded in CONTEXT.md — any future
+  "secret"-named script would silently never land.
+
+### Operator steps remaining (STOP points — not executed)
+
+1. **Bearer value ROTATION** (Viktor GO + his out-of-band consumers): the supervised procedure
+   in documentation/runbooks/secrets.md §"Operator/global bearer key" — mint → Secret update →
+   hub rollout → verify new-key 200 / old-key 401 / customer boxes unaffected. Until it runs,
+   the git-history copy of the old value is still live (it currently IS the Secret's value —
+   deployed same-value first so nothing broke).
+2. **F-14 full live leg** (optional per spec §13.4): a complete host-reset cycle on the drill
+   box reproduces the dead-end end-to-end; the scenario A/B fixtures + red-proofs carry it
+   until the next real reset validates it in production terms.