admin
65544ae689
Reconciler observed waiting_secret across two reports (16:52+17:07 UTC), re-staged the stored secret at 17:10:00 (no ep0 token, no gen bump); agent re-consumed 17:10:26 and converged (applied) 17:10:28 — no operator click. Demo host untouched (scoped). Fleet widening remains a deliberate follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
49 lines
3.5 KiB
Markdown
49 lines
3.5 KiB
Markdown
# felhom.eu — task reports
|
||
|
||
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
|
||
|
||
## PBS-DR self-heal reconciler (hub v0.56.0) — 2026-07-15
|
||
|
||
**Task:** implement `SPIKE-pbsdr-selfheal-2026-07-15` (`e8f8c44`) — a hub periodic reconciler that
|
||
re-stages a consumable PBS-DR secret for boxes stuck in `waiting_secret`/`consumed_failed` after
|
||
losing their agent-side converged marker (re-install / restore / snapshot rollback onto a stable
|
||
`host_id`). Class: risky/supervised. Repo: `felhom.eu` (`hub/`) only — **no agent change** (the spike
|
||
proved the agent already self-heals once a secret is consumable).
|
||
|
||
**Baselines:** hub main `e8f8c44` → this work; hub live image v0.55.0 → **v0.56.0**. Green gate
|
||
(`go build/vet/test ./...` in `hub/`) passing across the module.
|
||
|
||
**What shipped**
|
||
- `internal/store/pbsdr.go` — `RestageHostPBSSecret(hostID) (bool, error)`: clears `consumed_at` IFF
|
||
a secret row exists (no INSERT, no value change, **no generation bump**); `restaged=false` → no row
|
||
→ the caller escalates. `PBSDRHealStates()`: one query joining each host's descriptor
|
||
enable/provision flags to its latest report `pbs_dr.state` + id (mirrors `GetHostOOBStates`).
|
||
- `internal/pbsdrheal/reconciler.go` (new) — 5-min reconciler (wgsync shape). Work set = descriptor
|
||
**enabled + provisioned**. Per host, on the latest report `pbs_dr.state` held across a **debounce
|
||
(≥2 distinct reports)**: `waiting_secret` → re-stage the stored secret (no ep0 call, no gen bump),
|
||
or escalate to Re-issue if none stored; `consumed_failed` → escalate to Re-issue only.
|
||
Converged/`disabled`/`verify_failed`/unprovisioned/DR-OFF = pure no-op. Distinct audit events per
|
||
heal. `PBSDRHEAL_ONLY_HOST` scopes a supervised rollout.
|
||
- `internal/web/pbsdr.go` — `ReissuePBSDR(ctx, customerID)`: the non-HTTP core of the operator
|
||
Re-issue button, now the reconciler's escalation seam. The operator handler is unchanged (303/400
|
||
codes stay test-pinned).
|
||
- `cmd/hub/main.go` — reconciler started unconditionally; `PBSDRHEAL_ONLY_HOST` env read.
|
||
|
||
**Tests (green) + red-proofs (verified):** `internal/pbsdrheal/reconciler_test.go` (Scenarios A–F +
|
||
scope + no-re-heal, real store + fake action seam, audit events asserted); `internal/store/pbsdr_test.go`
|
||
(re-stage semantics + no-generation-bump guard + `PBSDRHealStates` parsing). All six §10 red-proofs
|
||
(A, A-gen, B, C, D, E) failed exactly their guard test then reverted.
|
||
|
||
**Deploy + live validation:** built `felhom-hub:0.56.0`, manifest bump + ArgoCD sync (Synced/Healthy);
|
||
first rollout scoped via `PBSDRHEAL_ONLY_HOST=demo-vm-felhom-2f4b00` (STOP discipline). Drill live-leg
|
||
(qm300 rolled to `post_day0_golden136` → stuck `waiting_secret`): the reconciler observed
|
||
`waiting_secret` across two reports and at **17:10:00 UTC re-staged** the stored secret (event
|
||
`pbsdr_selfheal_restaged`; no ep0 token, no gen bump) → agent **re-consumed 17:10:26** → **converged
|
||
`applied` 17:10:28**, hands-free. The demo host `demo-felhom-01` was never touched (out of scope).
|
||
**Fleet-wide widening (remove `PBSDRHEAL_ONLY_HOST`) is a deliberate operator follow-up**, not done
|
||
this session.
|
||
|
||
**⚠️ ARCHITECTURE IMPACT:** before this, no automatic recovery existed for a box re-installed/restored
|
||
onto its stable `host_id` — PBS-DR (hence escrow + offsite) stayed silently unconverged until a manual
|
||
operator Re-issue. This reconciler closes that gap.
|