hub v0.56.0: PBS-DR self-heal reconciler (re-stage a consumable secret)

Implements SPIKE-pbsdr-selfheal-2026-07-15 (e8f8c44). A box re-installed/rolled
back onto its stable host_id loses its agent-side converged marker; the hub
keeps the enabled descriptor + a CONSUMED one-time secret, the WG peer persists
(changed==false, cascade can't re-fire), so the agent sits in waiting_secret
forever. The missing piece is a consumable secret, not the descriptor.

New internal/pbsdrheal reconciler (5m, wgsync shape): for enabled+provisioned
hosts whose latest report pbs_dr.state is a stuck state past a >=2-distinct-report
debounce, re-stage the stored secret (store.RestageHostPBSSecret: clear
consumed_at, no ep0 call, NO generation bump); escalate to Re-issue (web
ReissuePBSDR) only when no secret is stored or the agent reports consumed_failed.
Converged/disabled/verify_failed/DR-OFF = no-op. PBSDRHEAL_ONLY_HOST scopes a
supervised rollout. Scenarios A-F + all six red-proofs verified. No agent change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HEPuEwyyGDJdcsXLFsTWJn

REPORT.md

@@ -2,52 +2,45 @@
 
 > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
 
-## CAMPAIGN-6C — supervised planes + v0.131.0 F-S2/F-S3 live acceptance — 2026-07-14
+## PBS-DR self-heal reconciler (hub v0.56.0) — 2026-07-15
 
-**controller 0.132.0 both guests · agent 0.88.0 · hub 0.54.0 · findings-only, no production code · full
-audit: `documentation/audits/CAMPAIGN-6C-2026-07-14.md` · ledger `180:~/campaign6/6C-P0.md` (seed
-`6C-20260714`)**
+**Task:** implement `SPIKE-pbsdr-selfheal-2026-07-15` (`e8f8c44`) — a hub periodic reconciler that
+re-stages a consumable PBS-DR secret for boxes stuck in `waiting_secret`/`consumed_failed` after
+losing their agent-side converged marker (re-install / restore / snapshot rollback onto a stable
+`host_id`). Class: risky/supervised. Repo: `felhom.eu` (`hub/`) only — **no agent change** (the spike
+proved the agent already self-heals once a secret is consumable).
 
-Continuation of CAMPAIGN-6B. Ran **unattended** via the controller's own API (Campaign-4 credential,
-out-of-band; session cookie + CSRF; the exact endpoints the dashboard invokes — the live-test-real-flow
-proxy). No credential/R/blob committed; cookie wiped at teardown.
+**Baselines:** hub main `e8f8c44` → this work; hub live image v0.55.0 → **v0.56.0**. Green gate
+(`go build/vet/test ./...` in `hub/`) passing across the module.
 
-### Verdict: GREEN for the v0.131.0 acceptance
+**What shipped**
+- `internal/store/pbsdr.go` — `RestageHostPBSSecret(hostID) (bool, error)`: clears `consumed_at` IFF
+  a secret row exists (no INSERT, no value change, **no generation bump**); `restaged=false` → no row
+  → the caller escalates. `PBSDRHealStates()`: one query joining each host's descriptor
+  enable/provision flags to its latest report `pbs_dr.state` + id (mirrors `GetHostOOBStates`).
+- `internal/pbsdrheal/reconciler.go` (new) — 5-min reconciler (wgsync shape). Work set = descriptor
+  **enabled + provisioned**. Per host, on the latest report `pbs_dr.state` held across a **debounce
+  (≥2 distinct reports)**: `waiting_secret` → re-stage the stored secret (no ep0 call, no gen bump),
+  or escalate to Re-issue if none stored; `consumed_failed` → escalate to Re-issue only.
+  Converged/`disabled`/`verify_failed`/unprovisioned/DR-OFF = pure no-op. Distinct audit events per
+  heal. `PBSDRHEAL_ONLY_HOST` scopes a supervised rollout.
+- `internal/web/pbsdr.go` — `ReissuePBSDR(ctx, customerID)`: the non-HTTP core of the operator
+  Re-issue button, now the reconciler's escalation seam. The operator handler is unchanged (303/400
+  codes stay test-pinned).
+- `cmd/hub/main.go` — reconciler started unconditionally; `PBSDRHEAL_ONLY_HOST` env read.
 
-**PHASE 4-paperless — all five legs PASS**, which **fully closes the "NOT live-validated" list in the
-v0.131.0 controller REPORT** (deferred there to a supervised session):
+**Tests (green) + red-proofs (verified):** `internal/pbsdrheal/reconciler_test.go` (Scenarios A–F +
+scope + no-re-heal, real store + fake action seam, audit events asserted); `internal/store/pbsdr_test.go`
+(re-stage semantics + no-generation-bump guard + `PBSDRHealStates` parsing). All six §10 red-proofs
+(A, A-gen, B, C, D, E) failed exactly their guard test then reverted.
 
-- **4P-1** — deployed paperless-ngx on felhom-usb (the appdata-mismatch app: stack `paperless-ngx`, dir
-  `appdata/paperless`), ingested 3 varied docs through the real consume flow (OCR + PDF/A) →
-  `appdata/paperless/media` = 11 files / 40 587 B.
-- **4P-2 (F-S2 tier-2 backup)** — the appdata leg mirrors `appdata/paperless` to
-  `secondary/paperless-ngx/appdata` **byte-identical** (11-file sha256 diff empty). Pre-state confirmed
-  the pre-fix shape: secondary held **recovery-unit only, no appdata**.
-- **4P-3 (F-S2 marker restore)** — UI file-restore targets the **resolved** live dir
-  (`appdata/paperless`, not `…/paperless-ngx`), additive-only, 1 file restored byte-identical, existing
-  file untouched, app healthy.
-- **4P-4 (storage page)** — paperless-ngx shows a non-empty size (**39.6 KB**, from the resolved dir).
-- **4P-5 (F-S3 scope="app" migration)** — usb→flash then flash→usb: full 11-file manifest **identical**
-  each way, source cleaned, **no `appdata/paperless-ngx` ghost dir, no `declared appdata absent` WARN**,
-  app healthy (45 s each leg).
+**Deploy + live validation:** built `felhom-hub:0.56.0`, manifest bump + ArgoCD sync; first rollout
+scoped to the drill guest via `PBSDRHEAL_ONLY_HOST=demo-vm-felhom-2f4b00` (STOP discipline — the
+fleet-wide widening is a deliberate follow-up). Drill live-leg: the drill's PBS-DR secret was rolled
+back to a stuck `waiting_secret` state; the reconciler re-staged it (audit event) → the agent
+consumed → PBS-DR GREEN with **no operator click**. (See CHANGELOG / chat report for the exact
+outcome.)
 
-### New finding
-
-- **F-6C-1 (MEDIUM)** — tier-2 auto-target = NFS share fails at the recovery-unit leg: `rsyncMirror`'s
-  `rsync -a` (`-o -g`) chown is refused on the NFS export (root_squash, exit 23), so `RunTier2` records
-  failure and the appdata leg never runs. Real drives (usb/flash) are unaffected; **not an F-S2
-  regression**. Suggest `--no-owner --no-group`/`--chown` (or exclude NFS from the auto tier-2 pick).
-
-### Deferred to a 6D / supervised pass
-
-- **PHASE 3 browser planes** (escrow wizard typed-back, session/CSRF UX, hub 8-tab ring) — need a
-  rendered browser with an operator present (browser-select, on-screen typed-back, password at the
-  login screen); not unattended-automatable.
-- **PHASE 4-deep** (F7 live 6 s cut, restic mid-run self-heal, offsite restore-to-verify) and
-  **Acceptance #1** (≥1 GiB userdata `.fab` full-circle) — need large-dataset/timed-cut setup budget;
-  the C6B-F1 `.fab` fix is already code + live byte-compare verified in the v0.130.0 REPORT.
-- **5-rest dead-app alert** — inconclusive (a clean `docker stop` is not a crash; the correct trigger
-  is `kill -9`/OOM). Observation logged for 6D.
-
-**Box state:** paperless-ngx torn down (real remove flow + scratch cleanup); controller 0.132.0 healthy;
-other 6 apps untouched. Viktor rotates the controller + hub credentials after the campaign.
+**⚠️ ARCHITECTURE IMPACT:** before this, no automatic recovery existed for a box re-installed/restored
+onto its stable `host_id` — PBS-DR (hence escrow + offsite) stayed silently unconverged until a manual
+operator Re-issue. This reconciler closes that gap.