felhom.eu/REPORT.md

felhom.eu — task reports

Overwrite this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in hub/CHANGELOG.md; the scripts history lives in scripts/CHANGELOG.md.

PBS-DR self-heal reconciler (hub v0.56.0) — 2026-07-15

Task: implement SPIKE-pbsdr-selfheal-2026-07-15 (e8f8c44) — a hub periodic reconciler that re-stages a consumable PBS-DR secret for boxes stuck in waiting_secret/consumed_failed after losing their agent-side converged marker (re-install / restore / snapshot rollback onto a stable host_id). Class: risky/supervised. Repo: felhom.eu (hub/) only — no agent change (the spike proved the agent already self-heals once a secret is consumable).

Baselines: hub main e8f8c44 → this work; hub live image v0.55.0 → v0.56.0. Green gate (go build/vet/test ./... in hub/) passing across the module.

What shipped

• internal/store/pbsdr.go — RestageHostPBSSecret(hostID) (bool, error): clears consumed_at IFF a secret row exists (no INSERT, no value change, no generation bump); restaged=false → no row → the caller escalates. PBSDRHealStates(): one query joining each host's descriptor enable/provision flags to its latest report pbs_dr.state + id (mirrors GetHostOOBStates).
• internal/pbsdrheal/reconciler.go (new) — 5-min reconciler (wgsync shape). Work set = descriptor enabled + provisioned. Per host, on the latest report pbs_dr.state held across a debounce (≥2 distinct reports): waiting_secret → re-stage the stored secret (no ep0 call, no gen bump), or escalate to Re-issue if none stored; consumed_failed → escalate to Re-issue only. Converged/disabled/verify_failed/unprovisioned/DR-OFF = pure no-op. Distinct audit events per heal. PBSDRHEAL_ONLY_HOST scopes a supervised rollout.
• internal/web/pbsdr.go — ReissuePBSDR(ctx, customerID): the non-HTTP core of the operator Re-issue button, now the reconciler's escalation seam. The operator handler is unchanged (303/400 codes stay test-pinned).
• cmd/hub/main.go — reconciler started unconditionally; PBSDRHEAL_ONLY_HOST env read.

Tests (green) + red-proofs (verified): internal/pbsdrheal/reconciler_test.go (Scenarios A–F + scope + no-re-heal, real store + fake action seam, audit events asserted); internal/store/pbsdr_test.go (re-stage semantics + no-generation-bump guard + PBSDRHealStates parsing). All six §10 red-proofs (A, A-gen, B, C, D, E) failed exactly their guard test then reverted.

Deploy + live validation: built felhom-hub:0.56.0, manifest bump + ArgoCD sync (Synced/Healthy); first rollout scoped via PBSDRHEAL_ONLY_HOST=demo-vm-felhom-2f4b00 (STOP discipline). Drill live-leg (qm300 rolled to post_day0_golden136 → stuck waiting_secret): the reconciler observed waiting_secret across two reports and at 17:10:00 UTC re-staged the stored secret (event pbsdr_selfheal_restaged; no ep0 token, no gen bump) → agent re-consumed 17:10:26 → converged applied 17:10:28, hands-free. The demo host demo-felhom-01 was never touched (out of scope). Fleet-wide widening (remove PBSDRHEAL_ONLY_HOST) is a deliberate operator follow-up, not done this session.

⚠️ ARCHITECTURE IMPACT: before this, no automatic recovery existed for a box re-installed/restored onto its stable host_id — PBS-DR (hence escrow + offsite) stayed silently unconverged until a manual operator Re-issue. This reconciler closes that gap.