hub v0.56.0: PBS-DR self-heal reconciler (re-stage a consumable secret)

Implements SPIKE-pbsdr-selfheal-2026-07-15 (e8f8c44). A box re-installed/rolled
back onto its stable host_id loses its agent-side converged marker; the hub
keeps the enabled descriptor + a CONSUMED one-time secret, the WG peer persists
(changed==false, cascade can't re-fire), so the agent sits in waiting_secret
forever. The missing piece is a consumable secret, not the descriptor.

New internal/pbsdrheal reconciler (5m, wgsync shape): for enabled+provisioned
hosts whose latest report pbs_dr.state is a stuck state past a >=2-distinct-report
debounce, re-stage the stored secret (store.RestageHostPBSSecret: clear
consumed_at, no ep0 call, NO generation bump); escalate to Re-issue (web
ReissuePBSDR) only when no secret is stored or the agent reports consumed_failed.
Converged/disabled/verify_failed/DR-OFF = no-op. PBSDRHEAL_ONLY_HOST scopes a
supervised rollout. Scenarios A-F + all six red-proofs verified. No agent change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn
This commit is contained in:
2026-07-15 18:27:39 +02:00
parent e8f8c441fa
commit 6218e7919d
9 changed files with 878 additions and 43 deletions
+37
View File
@@ -1,5 +1,42 @@
# Felhom Hub — Changelog
## v0.56.0 — PBS-DR self-heal reconciler (re-stage a consumable secret) (2026-07-15)
Implements `SPIKE-pbsdr-selfheal-2026-07-15` (`e8f8c44`). **⚠️ ARCHITECTURE IMPACT:** before this,
there was **no automatic recovery** for the commonest real event — a customer box re-installed /
restored / rolled back onto its **stable `host_id`**. The hub kept the durable `pbs_dr` descriptor
(enabled) + a durable **consumed** one-time secret; the WG peer still existed (same pubkey →
`changed==false`, so the provision cascade could not re-fire) and `applyPBSDR`'s "already
provisioned → no-op" meant even a config re-save minted nothing. The agent sat in
`pbs_dr.state="waiting_secret"` forever — PBS-DR never converged, so escrow could not run and offsite
never armed. The spike proved (SQ-2b) the **missing piece is a consumable secret, not the
descriptor**: re-staging the stored secret converged a stuck box in one ~30 s agent tick, using the
existing ep0 token, **zero churn**. This reconciler closes that gap.
- **`internal/pbsdrheal/reconciler.go` (new):** a hub periodic reconciler (5 min, `wgsync` shape).
For each host whose descriptor is **enabled + provisioned** and whose **latest report**
`pbs_dr.state` is a stuck state held across a **debounce** (≥2 distinct reports — so a box briefly
`waiting_secret` between provision and its first consume is not touched): `waiting_secret`
**re-stage the stored secret** (no ep0 call, **no generation bump**); no stored secret →
**escalate to Re-issue**; `consumed_failed`**escalate to Re-issue only** (a re-stage would
re-feed a burned secret). A converged/`disabled`/`verify_failed`/unprovisioned/DR-OFF host is a
**pure no-op**. Each heal emits a distinct audit event (`pbsdr_selfheal_restaged` /
`_reissued` / `_consumed_failed`). Reads the hub DB only — never the box.
- **`internal/store/pbsdr.go`:** `RestageHostPBSSecret(hostID) (bool, error)` — clears `consumed_at`
IFF a row exists (no INSERT, no value change, **no generation bump**); `restaged=false` → the
caller escalates. `PBSDRHealStates()` — one query joining each host's descriptor enable/provision
flags to its latest report's `pbs_dr.state` + id (mirrors `GetHostOOBStates`).
- **`internal/web/pbsdr.go`:** `ReissuePBSDR(ctx, customerID)` — the non-HTTP core of the operator
Re-issue button (tenantsync reissue → fresh consume-once secret → descriptor refresh + gen bump),
now the reconciler's escalation seam. The operator handler is unchanged.
- **`cmd/hub/main.go`:** the reconciler is started unconditionally (the primary re-stage heal needs
no endpoint). **`PBSDRHEAL_ONLY_HOST`** env scopes a supervised first rollout to one host
(empty = whole fleet — the steady state).
- **Tests:** `internal/pbsdrheal/reconciler_test.go` (Scenarios AF + scope + no-re-heal, real store
+ fake action seam) and `internal/store/pbsdr_test.go` (re-stage semantics + the **no-generation-
bump** guard + `PBSDRHealStates` parsing). All six §10 red-proofs verified (mutation → FAIL →
revert). No agent changes (the agent already self-heals once a secret is consumable).
## v0.55.0 — accept the offbox_enlarge_blocked event (Task 3a-fix delivery chain) (2026-07-15)
The controller (v0.134.1) sends an `offbox_enlarge_blocked` warning when an app's enlarged offsite