felhom.eu/REPORT.md

felhom.eu — task reports

Overwrite this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in hub/CHANGELOG.md; the scripts history lives in scripts/CHANGELOG.md.

Hub closing bundle — v0.53.0 (F-14 gated auto-Reissue + dead-host roll-up + bearer out of git) — 2026-07-13

The last engineering items on the pre-tester board. Commits 2321077 (part 1), 04861a7 (part 2), 36c7213 (part 3), 80aca3a + 3682303 (part 4 + gate re-land), 3efb1b1 (docs), 3616dcb (manifest 0.53.0); felhom-controller 51c871a (example-config scrub). Deployed: felhom-hub:0.53.0, ArgoCD Synced/Healthy, rollout clean, startup log clean.

Baselines (re-confirmed live before work)

• Hub v0.52.0 at head (74fa61c, past the 0.126.x doc commits); manifest image 0.52.0.
• The operator bearer literal committed at manifests/hub.yaml:77 (the runbook-flagged item).

Step-0 citation — where the tenancy↔host linkage lives

The ep0 tenancy is customer-keyed (tenantsync.Provision(ctx, customerID); namespace + felhom@pbs!<customer> token per customer — hub/internal/tenantsync/client.go). Its only host-side anchors are (a) the pbs_dr descriptor inside hosts.desired_json (hub/internal/web/pbsdr.go:63-92 read/merge) and (b) the consume-once secret in host_pbs_secrets keyed by host_id (hub/internal/store/pbsdr.go:10-17). Both die inside the DeleteHost cascade (hub/internal/store/store.go — host_pbs_secrets and hosts deletes in the one-tx statement list), so no surviving row could carry a provenance column → a dedicated host_deletions table is the justified shape. The F-14 gate matches on customer_id because the tenancy is customer-scoped; it reads the MOST RECENT record so an old acked deletion never whitelists a newer un-acked one.

Part 1 — deletion provenance (host_deletions)

• Schema: id, host_id, customer_id, deleted_at, escrow_acked + customer index (store.go, additive v0.53.0 block). No backfill — pre-record deletions (incl. the take-two drill's own ep0 residue, already manually cleared) legitimately fall to the manual path.
• Written inside the DeleteHost transaction, before the cascade deletes read the host row away. escrow_acked = deleteEscrow AND an escrow row was actually present — "removed through the escrow-ack flow" means an acknowledged destruction happened; ticking the box over nothing records false (justification: the ruling's operative fact is destroyed-with-acknowledgement, not checkbox state; pinned by test).
• LatestHostDeletion(customerID) — newest record only.

Part 2 — F-14 gated auto-Reissue

pbsdrProvisionAtom (pbsdr.go), on tenantsync.ErrTokenExists: reads the customer's latest deletion record. escrow_acked=true → invokes the existing tenantsync.Reissue op (no new endpoint interaction), stores the pbsdr_auto_reissue hub-source audit event with the operator note "Previous key destroyed (acknowledged deletion) — credentials re-issued automatically." (+ deleted_host/deleted_at/new_host/token_id details), logs INFO, and proceeds into the normal secret+descriptor+bump tail. No record / un-acked → the pre-existing refusal string, byte-unchanged, and the manual Re-issue button path untouched. Both rulings recorded verbatim in CONTEXT.md; take-two F-14 row dispositioned SHIPPED (live validation = the next real host-reset cycle, per spec option 4 — fixtures + red-proofs carry it until then).

Part 3 — dead-host roll-up honesty

• hub/internal/web/rollup.go: controllerStatus() (the ONE copy of the previously triple-inlined controller-derived chain — behavior-preserving) + (s *Server) foldHostStatus(customerID, base, customerHasReported) = worst(controllerDerived, hostStatusOf(each expected host)) via Server.hostStatus only (hosts.go — the single staleness definition shared with HostStalenessChecker; no second threshold introduced anywhere).
• Mapping: host down/stale → customer capped at warn + cause chip host down|stale: <host_id>; pending hosts worsen only once the customer has ever reported (customer-summary row exists — the durable proxy for "never had an ok report"; reports prune at 90d so a per-report query can't be the definition). Statuses already worse (down) or administrative (disabled/blocked) keep their token; the chip still names the host.
• Wired at all three derivation sites: dashboard (server.go), /configs list (configs.go), customer detail header + summary strip (configs.go + customer_unified.html). Chip renders as an existing status-badge status-badge-warn token (no new CSS). Display + derivation only — checker alerting untouched. hub_confirm_gate.py green.

Part 4 — bearer out of git

• manifests/hub.yaml: ConfigMap report_api_key → empty placeholder; Deployment env REPORT_API_KEY ← secretKeyRef: report-api/REPORT_API_KEY, deliberately not optional: (missing Secret must fail Ready). cmd/hub/main.go: REPORT_API_KEY env override (the RESEND_API_KEY twin) — the hub read the key from the YAML config, so this minimal adaptation was required (cited: main.go env-override block, Config.API.ReportAPIKey).
• Gate: scripts/manifest_bearer_gate.py — 64-hex bearer shapes anywhere in manifests/, exit 1; felhom.secret.yaml's pre-existing umami APP_SECRET prints as a visible non-fatal KNOWN-BACKLOG line (secrets.md backlog, out of scope — no history rewrite, no umami touch). GOTCHA found: the repo .gitignore *secret* pattern silently un-tracked the gate's first name (manifest_secret_gate.py never landed in commit 80aca3a) — re-landed as manifest_bearer_gate.py (3682303).
• Consumer list (the GLOBAL key only — per-customer/per-host keys are hub-generated and unaffected): (1) the hub itself (now via Secret/report-api); (2) operator-typed tooling — felhom-ops … -hub-key (GL-6/PBS-DR runbooks) and the runbook curl probes (break-glass.md §, offsite-endpoint.md §§) — all reference it "out-of-band", nothing machine-persisted; (3) felhom-controller/controller/configs/controller.yaml.example carried the real literal as example text (never a live consumer) — scrubbed (51c871a).
• Both publish runbooks' ROTATION notes now carry the 2026-07-13 disposition; full create/rotate/verify procedure in documentation/runbooks/secrets.md §"Operator/global bearer key".

Deploy + live validation (§13)

• Image felhom-hub:0.53.0 built+pushed on 180; Secret/report-api created FIRST (value extracted from git history on 180 straight into a 0600 file — never echoed — kubectl create secret, file shredded); manifest bump 3616dcb; ArgoCD hard-refresh + deliberate sync → Synced/Healthy, rollout OK, image verified, startup log clean.
• Scenario E live: authed probe with the Secret-provided value → 200; wrong key → 401; /healthz 200. Fail-Ready property verified non-disruptively: a throwaway pod with the same non-optional secretKeyRef shape against an absent Secret name → CreateContainerConfigError (container never starts); the live manifest's report-api ref carries zero optional: flags.
• §13.2 roll-up live leg (drill host demo-vm-felhom-2f4b00, hostname drill-day0) — the exact masking shape, disproven live, then recovered (all via the logged-in operator browser session; screenshots recorded):
  • BEFORE (13:0x Z): dashboard "Demo in VM = OK, no chip"; /hosts host ONLINE.
  • systemctl stop felhom-agent at 13:09:28Z; the guest controller kept reporting.
  • AFTER (~13:45Z, past the 30m stale threshold): dashboard row "Demo in VM = WARN + chip HOST STALE: DEMO-VM-FELHOM-2F4B00" with the controller's last-seen at 1 min ago — pre-fix this row was green (the drill-1/Peti masking bug). Customer detail header + summary strip both carry the same chip → the header says WHICH host; the controller Health card honestly still reads STATUS: OK (the warn comes from the host fold, correctly attributed). The HostStalenessChecker independently emitted its warning event (events badge 2→4) — alerting untouched, as specified.
  • RECOVERY: systemctl start felhom-agent at 13:46:22Z → first heartbeat → row back to OK, chip gone by 13:47Z. Both states + recovery recorded.

Red-proofs (all run → FAIL → restored; git diff clean after each)

#	Mutation	Failing evidence
Part 1	provenance INSERT dropped from the DeleteHost tx	TestDeleteHost_ProvenanceRecord: "no deletion record written by the escrow-ack delete"; scenario A: 502 refusal instead of 303 (gate found nothing)
B	gate bypassed (auto-reissue unconditionally, nil-safe)	both TestPBSDR_F14NoRecordNeverRekeys subtests: save = 303 (silent re-key proceeded) instead of the 502 refusal
C	foldHostStatus short-circuited to controller-only	TestRollup_DeadHostMasking: "dashboard row is GREEN over a 23h-dead host" + missing chip + detail header silent; 2 boundary subtests also red
E	64-hex literal reintroduced into manifests/hub.yaml	manifest_bearer_gate.py exit 1, literal reported masked (00000000...0000)

Tests (scenarios A–E)

• A: TestPBSDR_F14AutoReissueOnAckedDeletion — real escrow-ack DeleteHost, real handler path; asserts reissue call count, descriptor, generation, consume-once secret, the stored audit event + note text, and zero secret leakage into logs.
• B: TestPBSDR_F14NoRecordNeverRekeys (no record / latest un-acked) — the byte-unchanged 502
  • zero Reissue calls + zero state writes (the exact non-effect).
• C: TestRollup_DeadHostMasking — the Peti fixture (host backdated 23h, controller report fresh) rendered through the REAL dashboard + customer handlers.
• D: TestRollup_Boundaries — all-ok pass-through, stale-host warn chip, onboarding pending exclusion, post-onboarding pending worsening.
• E: gate (committed) + the live auth/fail-Ready legs above.
• Full hub green gate: go build ./... && go vet ./... && go test ./... — all packages ok.

Observations

• Peti's box is BACK ONLINE (dashboard: Peti Proxmox OK, controller 0.115.0, agent 0.81.0, fresh reports; 5/5/5 event badges). The parked publish-train items (Peti D/E/G: sign agent_update 0.79→0.85 — his agent already shows 0.81 —, journal group, G1 temp-creds deletion) look actionable; his motioneye storage target reads 100% full (WORST STORAGE on /hosts). Operator follow-up.
• The spec's "F-13 reference resolved as typo-or-assigned": the take-two record's findings run F-14/F-15/F-16; F-13 is the CAMPAIGN-3 nfs4-under-mp8 finding (separate, still open) — the numbering is continuous across records, no typo, nothing to reassign.
• The "remove from system" decommission finisher stays parked (untouched by this bundle).
• The .gitignore *secret* un-tracking gotcha (above) is recorded in CONTEXT.md — any future "secret"-named script would silently never land.

Operator steps remaining (STOP points — not executed)

1. Bearer value ROTATION (Viktor GO + his out-of-band consumers): the supervised procedure in documentation/runbooks/secrets.md §"Operator/global bearer key" — mint → Secret update → hub rollout → verify new-key 200 / old-key 401 / customer boxes unaffected. Until it runs, the git-history copy of the old value is still live (it currently IS the Secret's value — deployed same-value first so nothing broke).
2. F-14 full live leg (optional per spec §13.4): a complete host-reset cycle on the drill box reproduces the dead-end end-to-end; the scenario A/B fixtures + red-proofs carry it until the next real reset validates it in production terms.