admin
6218e7919d
Implements SPIKE-pbsdr-selfheal-2026-07-15 (e8f8c44). A box re-installed/rolled
back onto its stable host_id loses its agent-side converged marker; the hub
keeps the enabled descriptor + a CONSUMED one-time secret, the WG peer persists
(changed==false, cascade can't re-fire), so the agent sits in waiting_secret
forever. The missing piece is a consumable secret, not the descriptor.
New internal/pbsdrheal reconciler (5m, wgsync shape): for enabled+provisioned
hosts whose latest report pbs_dr.state is a stuck state past a >=2-distinct-report
debounce, re-stage the stored secret (store.RestageHostPBSSecret: clear
consumed_at, no ep0 call, NO generation bump); escalate to Re-issue (web
ReissuePBSDR) only when no secret is stored or the agent reports consumed_failed.
Converged/disabled/verify_failed/DR-OFF = no-op. PBSDRHEAL_ONLY_HOST scopes a
supervised rollout. Scenarios A-F + all six red-proofs verified. No agent change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
47 lines
3.3 KiB
Markdown
47 lines
3.3 KiB
Markdown
# felhom.eu — task reports
|
||
|
||
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
|
||
|
||
## PBS-DR self-heal reconciler (hub v0.56.0) — 2026-07-15
|
||
|
||
**Task:** implement `SPIKE-pbsdr-selfheal-2026-07-15` (`e8f8c44`) — a hub periodic reconciler that
|
||
re-stages a consumable PBS-DR secret for boxes stuck in `waiting_secret`/`consumed_failed` after
|
||
losing their agent-side converged marker (re-install / restore / snapshot rollback onto a stable
|
||
`host_id`). Class: risky/supervised. Repo: `felhom.eu` (`hub/`) only — **no agent change** (the spike
|
||
proved the agent already self-heals once a secret is consumable).
|
||
|
||
**Baselines:** hub main `e8f8c44` → this work; hub live image v0.55.0 → **v0.56.0**. Green gate
|
||
(`go build/vet/test ./...` in `hub/`) passing across the module.
|
||
|
||
**What shipped**
|
||
- `internal/store/pbsdr.go` — `RestageHostPBSSecret(hostID) (bool, error)`: clears `consumed_at` IFF
|
||
a secret row exists (no INSERT, no value change, **no generation bump**); `restaged=false` → no row
|
||
→ the caller escalates. `PBSDRHealStates()`: one query joining each host's descriptor
|
||
enable/provision flags to its latest report `pbs_dr.state` + id (mirrors `GetHostOOBStates`).
|
||
- `internal/pbsdrheal/reconciler.go` (new) — 5-min reconciler (wgsync shape). Work set = descriptor
|
||
**enabled + provisioned**. Per host, on the latest report `pbs_dr.state` held across a **debounce
|
||
(≥2 distinct reports)**: `waiting_secret` → re-stage the stored secret (no ep0 call, no gen bump),
|
||
or escalate to Re-issue if none stored; `consumed_failed` → escalate to Re-issue only.
|
||
Converged/`disabled`/`verify_failed`/unprovisioned/DR-OFF = pure no-op. Distinct audit events per
|
||
heal. `PBSDRHEAL_ONLY_HOST` scopes a supervised rollout.
|
||
- `internal/web/pbsdr.go` — `ReissuePBSDR(ctx, customerID)`: the non-HTTP core of the operator
|
||
Re-issue button, now the reconciler's escalation seam. The operator handler is unchanged (303/400
|
||
codes stay test-pinned).
|
||
- `cmd/hub/main.go` — reconciler started unconditionally; `PBSDRHEAL_ONLY_HOST` env read.
|
||
|
||
**Tests (green) + red-proofs (verified):** `internal/pbsdrheal/reconciler_test.go` (Scenarios A–F +
|
||
scope + no-re-heal, real store + fake action seam, audit events asserted); `internal/store/pbsdr_test.go`
|
||
(re-stage semantics + no-generation-bump guard + `PBSDRHealStates` parsing). All six §10 red-proofs
|
||
(A, A-gen, B, C, D, E) failed exactly their guard test then reverted.
|
||
|
||
**Deploy + live validation:** built `felhom-hub:0.56.0`, manifest bump + ArgoCD sync; first rollout
|
||
scoped to the drill guest via `PBSDRHEAL_ONLY_HOST=demo-vm-felhom-2f4b00` (STOP discipline — the
|
||
fleet-wide widening is a deliberate follow-up). Drill live-leg: the drill's PBS-DR secret was rolled
|
||
back to a stuck `waiting_secret` state; the reconciler re-staged it (audit event) → the agent
|
||
consumed → PBS-DR GREEN with **no operator click**. (See CHANGELOG / chat report for the exact
|
||
outcome.)
|
||
|
||
**⚠️ ARCHITECTURE IMPACT:** before this, no automatic recovery existed for a box re-installed/restored
|
||
onto its stable `host_id` — PBS-DR (hence escrow + offsite) stayed silently unconverged until a manual
|
||
operator Re-issue. This reconciler closes that gap.
|