admin
6218e7919d
Implements SPIKE-pbsdr-selfheal-2026-07-15 (e8f8c44). A box re-installed/rolled
back onto its stable host_id loses its agent-side converged marker; the hub
keeps the enabled descriptor + a CONSUMED one-time secret, the WG peer persists
(changed==false, cascade can't re-fire), so the agent sits in waiting_secret
forever. The missing piece is a consumable secret, not the descriptor.
New internal/pbsdrheal reconciler (5m, wgsync shape): for enabled+provisioned
hosts whose latest report pbs_dr.state is a stuck state past a >=2-distinct-report
debounce, re-stage the stored secret (store.RestageHostPBSSecret: clear
consumed_at, no ep0 call, NO generation bump); escalate to Re-issue (web
ReissuePBSDR) only when no secret is stored or the agent reports consumed_failed.
Converged/disabled/verify_failed/DR-OFF = no-op. PBSDRHEAL_ONLY_HOST scopes a
supervised rollout. Scenarios A-F + all six red-proofs verified. No agent change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
3.3 KiB
felhom.eu — task reports
Overwrite this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in hub/CHANGELOG.md; the scripts history lives in scripts/CHANGELOG.md.
PBS-DR self-heal reconciler (hub v0.56.0) — 2026-07-15
Task: implement SPIKE-pbsdr-selfheal-2026-07-15 (e8f8c44) — a hub periodic reconciler that
re-stages a consumable PBS-DR secret for boxes stuck in waiting_secret/consumed_failed after
losing their agent-side converged marker (re-install / restore / snapshot rollback onto a stable
host_id). Class: risky/supervised. Repo: felhom.eu (hub/) only — no agent change (the spike
proved the agent already self-heals once a secret is consumable).
Baselines: hub main e8f8c44 → this work; hub live image v0.55.0 → v0.56.0. Green gate
(go build/vet/test ./... in hub/) passing across the module.
What shipped
internal/store/pbsdr.go—RestageHostPBSSecret(hostID) (bool, error): clearsconsumed_atIFF a secret row exists (no INSERT, no value change, no generation bump);restaged=false→ no row → the caller escalates.PBSDRHealStates(): one query joining each host's descriptor enable/provision flags to its latest reportpbs_dr.state+ id (mirrorsGetHostOOBStates).internal/pbsdrheal/reconciler.go(new) — 5-min reconciler (wgsync shape). Work set = descriptor enabled + provisioned. Per host, on the latest reportpbs_dr.stateheld across a debounce (≥2 distinct reports):waiting_secret→ re-stage the stored secret (no ep0 call, no gen bump), or escalate to Re-issue if none stored;consumed_failed→ escalate to Re-issue only. Converged/disabled/verify_failed/unprovisioned/DR-OFF = pure no-op. Distinct audit events per heal.PBSDRHEAL_ONLY_HOSTscopes a supervised rollout.internal/web/pbsdr.go—ReissuePBSDR(ctx, customerID): the non-HTTP core of the operator Re-issue button, now the reconciler's escalation seam. The operator handler is unchanged (303/400 codes stay test-pinned).cmd/hub/main.go— reconciler started unconditionally;PBSDRHEAL_ONLY_HOSTenv read.
Tests (green) + red-proofs (verified): internal/pbsdrheal/reconciler_test.go (Scenarios A–F +
scope + no-re-heal, real store + fake action seam, audit events asserted); internal/store/pbsdr_test.go
(re-stage semantics + no-generation-bump guard + PBSDRHealStates parsing). All six §10 red-proofs
(A, A-gen, B, C, D, E) failed exactly their guard test then reverted.
Deploy + live validation: built felhom-hub:0.56.0, manifest bump + ArgoCD sync; first rollout
scoped to the drill guest via PBSDRHEAL_ONLY_HOST=demo-vm-felhom-2f4b00 (STOP discipline — the
fleet-wide widening is a deliberate follow-up). Drill live-leg: the drill's PBS-DR secret was rolled
back to a stuck waiting_secret state; the reconciler re-staged it (audit event) → the agent
consumed → PBS-DR GREEN with no operator click. (See CHANGELOG / chat report for the exact
outcome.)
⚠️ ARCHITECTURE IMPACT: before this, no automatic recovery existed for a box re-installed/restored
onto its stable host_id — PBS-DR (hence escrow + offsite) stayed silently unconverged until a manual
operator Re-issue. This reconciler closes that gap.