felhom.eu/documentation/audits/SPIKE-pbsdr-selfheal-2026-07-15.md

# SPIKE — PBS-DR day-0 self-heal: root cause + re-trigger validation — 2026-07-15

**Class:** spike (findings only — no production code, config, or schema edits; the only non-scratch
write is this doc + its commit to `felhom.eu` main). Gates the **PBS-DR self-heal TASK** spec, which
is BLOCKED until these verdicts land. All hub-DB mutations were throwaway probes scoped to the drill
host; every one is logged + reverted/self-reverted below (§Mutations). Evidence sink:
`180:~/spikes/pbsdr-selfheal-2026-07-15/`.

## Verdict up front

1. **ROOT CAUSE (SQ-1) — the prompt's surviving theory is REFUTED by live evidence.** The descriptor
   is **NOT** absent. On the drill host the `pbs_dr` descriptor is **present and enabled** in
   `desired_json`, the WG peer **exists**, and the fingerprint **verifies over the tunnel**. The
   block is that the **one-time PBS secret was consumed on 2026-07-12 and no code path ever re-mints
   or re-stages it** for a host that has since lost its agent-side converged marker (fresh day-0 /
   re-install / snapshot rollback). The agent sits in `pbs_dr.state="waiting_secret"` **forever**,
   re-applying the descriptor every 60 s tick but unable to consume a burned secret. The reused-peer
   angle is real but its consequence is *not* "descriptor never written" — it is "the WG-registration
   cascade cannot re-fire (same pubkey → `changed==false`), so nothing re-mints a consumable secret."

2. **THE MISSING PIECE IS A CONSUMABLE SECRET, NOT THE DESCRIPTOR (SQ-2, live-proven).** Re-asserting
   the descriptor alone (generation bump) converges **nothing** (negative control). Staging **any**
   consumable secret converges the box in **one tick**: re-exposing the *existing, still-valid* ep0
   token secret (`consumed_at → NULL`, **zero ep0 churn**) made the agent verify → consume → create
   the `felhom-pbs` storage entry → mark converged within ~30 s. The operator **"Re-issue PBS
   credentials"** action IS the manual recovery; its DB effect is a **superset** of that proven
   re-stage (fresh secret + descriptor bump).

3. **A PERIODIC RE-TRIGGER IS SAFE — IF it re-serves a secret and never blind-timer-reissues (SQ-3).**
   Re-asserting a *converged* descriptor is a clean idempotent no-op (live-proven: marker untouched,
   no re-consume, no generation thrash — the `descriptorHash` marker gates it). Re-running the
   **provision atom** does NOT churn ep0 (`Provision` short-circuits with `ErrTokenExists`, mints
   nothing) — but it also does NOT fix a stuck host (it refuses with "use Re-issue"). **`Reissue`
   rotates the ep0 token every call** — safe against orphans (delete+recreate) but a blind-timer
   `Reissue` = descriptor-hash thrash + constant re-consume. Safe design: **gate on the agent's
   reported `waiting_secret`/`consumed_failed` state, re-stage the stored secret first (no churn, no
   gen bump), escalate to `Reissue` only if that fails.**

4. **⚠️ ARCHITECTURE IMPACT.** There is **no automatic recovery** for the single most common
   production event — a customer box re-installed / restored / rolled back onto its stable `host_id`.
   The hub keeps the durable descriptor + a durable *consumed* secret; the WG hook can't re-fire;
   `applyPBSDR`'s "already provisioned → no-op" branch means even a config re-save won't re-mint. The
   PBS-DR tier silently stays unconverged (and therefore escrow + offsite never arm) until an operator
   notices and clicks Re-issue. This is the P-DAY0-DEEP block, generalized.

---

## 0. Baselines (live-verified at session start)

| Item | Value | How verified |
|---|---|---|
| `felhom.eu` main head | **`aff272c4be40da33855c0c45f7e66f407f33c99c`** (CAMPAIGN-6D P-DAY0 doc) | `git log -1` local == pulled on 180 |
| Hub live image / version | **`felhom-hub:0.55.0`** (`hub-6fddb45766-k9f7f`, Running 9 h) | `kubectl get deploy hub -o jsonpath`; startup log `felhom-hub 0.55.0 starting` |
| Agent live version (drill) | **`felhom-agent 0.88.0`** | `felhom-agent --version` on drill VM `192.168.0.152` |
| Controller golden | **0.136.0** (not exercised here) | baseline per prompt |
| Drill host (hub `host_id`) | **`demo-vm-felhom-2f4b00`** / customer `demo-vm-felhom`, gen 3 at start | `hosts` table |
| Drill guest (Proxmox) | **VM 300 `drill-day0`** on demo-pve (`192.168.0.152`, MAC `bc:24:11:85:98:7b`) | `qm status 300`; ARP on felhom-pve |
| Rollback point | snapshot **`post_day0_golden136`** (2026-07-15 15:32:02) — *"golden 0.136.0 + agent 0.88.0 … escrow pending (no DR chain)"* | `qm listsnapshot 300` |

**Read seam (both directions):** a throwaway pure-Go tool `hubdbpoke` (built on 180 inside the hub
module so `modernc.org/sqlite` resolves; `CGO_ENABLED=0`), `kubectl cp`'d into the hub pod, run
against `/data/hub.db` — `q` = read query, `e` = exec. Removed from the pod at session end; source
never committed. Secret **values** were never selected (only `length(value)` + timestamps).

---

## The two theories the reviewer already refuted (restated with file:line, both CONFIRMED here)

- **(a) "the agent lacks self-heal" — FALSE.** `pbsdr/loop.go:62-77` re-runs `Manager.Apply` on every
  60 s tick *and* on each desired-state nudge. `pbsdr/manager.go:265-272` (verify-pin-before-consume):
  a probe failure leaves the one-time secret **untouched** and returns, so a slow/flapping tunnel
  self-heals **as long as a consumable secret is in hand**. Live-proven: the drill agent oscillated
  through `verify_failed` (tunnel i/o timeout) and recovered without operator help — see §SQ-1 note.

- **(b) "the hub provision atom probes the customer tunnel" — FALSE.** `web/pbsdr.go:184-198`: the
  atom's preconditions are all **hub-DB-local** (`tenantsync!=nil`, `GetWGPeerForHost`,
  `GetWGEndpoint`). It never dials the box. The `i/o timeout` in the field is therefore the **agent's**
  PBS fingerprint probe (`dial 10.77.0.1:8007`), not a hub failure — confirmed verbatim in the drill
  agent journal (§SQ-1).

---

## SQ-1 — delivery-path root cause

**Question:** is the descriptor's only write path the first-WG cascade — and on the drill host, is
the descriptor absent (cascade never fired) or present-but-something-else?

**Probes (hubdbpoke `q` against live `/data/hub.db`, scoped `host_id='demo-vm-felhom-2f4b00'`):**

| # | Query | Verbatim result |
|---|---|---|
| a | `wg_peers WHERE host_id=…` | 1 row: pubkey `zxy+y029eYri+g8HhP4qGwep/Nmv8VR7l3iyf39ovmo=`, ip `10.77.0.3`, created `2026-07-12 19:50:52`, updated `2026-07-15 13:14:43` |
| b | `desired_json` | `{"pbs_dr":{"enabled":true,"storage_id":"felhom-pbs","pbs_tunnel_ip":"10.77.0.1","datastore":"felhom-offsite","namespace":"demo-vm-felhom","token_id":"felhom@pbs!demo-vm-felhom","fingerprint":"c6:07:…:fd"}}` |
| c | `host_pbs_secrets` (len only) | 1 row: `val_len=36`, created `2026-07-12 19:50:53`, **`consumed_at=2026-07-12 20:05:53`** |
| d | agent pbsdr status (latest host-report) | `"pbs_dr":{"state":"waiting_secret","storage_id":"felhom-pbs","namespace":"demo-vm-felhom","message":"verified; no unconsumed token secret staged on the hub"}` |
| e | `customer_configs.dr_tier` | `1` (DR tier ON) |

**Agent journal (drill VM, local CEST = UTC+2), verbatim:**
```
17:07:26 WARN pbsdr: PBS fingerprint verify failed BEFORE consume (nothing consumed; retrying)
         err="pbs: fingerprint probe dial 10.77.0.1:8007: i/o timeout"
15:14:42 INFO pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)
```
At session time the tunnel was **up** (`wg show`: handshake 10 s ago; `10.77.0.1:8007` OPEN 3/3). No
`marker.json` and no `felhom-pbs` entry in `/etc/pve/storage.cfg` existed → the agent had **never**
gotten past consume on this box's current life.

**Forensic tie-off (reused peer):** the agent's own WG pubkey on the drill VM is
`zxy+y029eYri+g8HhP4qGwep/Nmv8VR7l3iyf39ovmo=` — **identical** to the stored `wg_peers` row. The key
survives the snapshot/re-install, so on re-registration `RegisterWGPeerForHost` returns
`changed==false` → the `if changed { … wgRegisteredHook(…) }` block (`api/wg.go:288-305`) never fires
→ `PBSDRAutoProvision` (`web/pbsdr.go:274-300`) never runs → no new secret.

**Verdict:** the descriptor is **PRESENT and enabled** (delivered correctly at first enrollment on
07-12, and durable in `desired_json` ever since). The block is that the **one-time secret was consumed
on 07-12 and is never re-minted/re-staged** after the box lost its converged marker (the
`post_day0_golden136` snapshot is a *fresh* day-0 with "no DR chain", branched off `pre-day0-clean`).
The agent is permanently `waiting_secret`. The transient `verify_failed`/i/o-timeout is a separate,
self-healing environmental flap of the nested drill VM's WG tunnel — **not** the root cause.

---

## SQ-2 — prove the gap directly + validate the re-trigger

The literal SQ-2a premise ("`host_pbs_secrets` holds an UNCONSUMED secret") is **false** — it is
consumed. Two probes isolate the exact missing piece.

**SQ-2a′ (negative control — descriptor re-assert only).** MUT-1: `desired_generation` 3→4 (re-serve
the identical descriptor; secret untouched). Result: the agent kept reporting `waiting_secret`; the
descriptor had *already* been re-applied on every 60 s tick since rollback with no convergence. **A
generation bump / descriptor re-delivery changes nothing.** → descriptor delivery is **not** the gap.

**SQ-2b′ (stage a consumable secret — zero ep0 churn).** Tunnel confirmed up first. MUT-2:
`host_pbs_secrets.consumed_at → NULL` (re-expose the *existing* still-valid ep0 token secret; no
`tenantsync` call, no new token). Within ~30 s the drill agent journal showed, verbatim:
```
17:44:51 INFO pbsdr: one-time token secret consumed (single-use; value withheld from logs) storage_id=felhom-pbs secret_len=36
17:44:53 INFO pbsdr: converged state=applied storage_id=felhom-pbs
```
`marker.json` = `{"hash":"f1059b54…","state":"applied","applied_at":"2026-07-15T15:44:53Z"}`;
`felhom-pbs` now PRESENT in `/etc/pve/storage.cfg`; hub `consumed_at` self-re-set to `15:44:51`
(single-use tx). **The agent self-heals fully the instant a consumable secret exists** — the descriptor
was never the missing thing.

**Re-issue (SQ-2b, the literal operator action):** validated from code rather than fired live — the
web handler `POST /configs/{id}/pbsdr-reissue` (`server.go:456-463` → `handlePBSDRReissue`,
`web/pbsdr.go:305-355`) sits behind the password-gated operator session + CSRF (not drivable
headless), and a live run would rotate an ep0 token against the STOP condition. Its DB effect is a
**superset** of the proven re-stage: `tenantsync.Reissue` (fresh token) → `SaveHostPBSSecret` (new
secret, `consumed_at` reset) → descriptor rewrite with the **new** `token_id`/`fingerprint` + bump
(`web/pbsdr.go:335-346`). Since a fresh consumable secret is exactly what SQ-2b′ proved sufficient,
Re-issue converges a reused-peer guest. Use it as the escalation when the stored secret is stale.

**Verdict:** descriptor-only delivery converges **NO**; staging a consumable secret converges **YES**
(one tick, existing token, no churn); Re-issue converges **YES** (superset, code-proven). **The fix is
to re-serve a consumable secret** — re-stage the stored one (least-churn) or Re-issue (escalation) —
**not** to re-assert the descriptor.

---

## SQ-3 — is a periodic re-trigger safe?

**SQ-3a — re-serve for an ALREADY-CONVERGED host (live-proven).** MUT-3: `desired_generation` 4→5 on
the now-converged host; waited ~75 s (≥1 tick). Result: `marker.json` **unchanged**
(`applied_at` still `15:44:53Z`), **no** new `consumed`/`converged` journal line, hub `consumed_at`
**unchanged** (`15:44:51`). The `descriptorHash` idempotency short-circuit (`manager.go:233-238`:
`mk.Hash==h && cf==nil` → set status + return) makes re-serving the same descriptor a **clean no-op**
— no re-consume, no re-adopt, **no generation thrash on the agent side**.

**SQ-3b — the one-shot-secret interaction + ep0 churn (code-derived; STOP-condition forbids spraying
tokens).**
- `store/pbsdr.go:10-17` `SaveHostPBSSecret` = last-write-wins, resets `consumed_at`. It is only
  reached **after** a successful `Provision`/`Reissue`.
- `tenantsync/client.go:93-95,151-155` `Provision` returns typed **`ErrTokenExists`** for an existing
  token — it **mints nothing** and never reaches `SaveHostPBSSecret`. So **re-running the provision
  atom does NOT churn ep0 and does NOT stomp** — but (per `web/pbsdr.go:206-222`) for a
  provisioned-and-not-acked-deleted host it **returns a loud "use Re-issue" error and does not fix
  the stuck host**. Re-running the atom is therefore the *wrong* re-trigger for this root cause.
- `tenantsync/client.go:97-100` `Reissue` = delete+recreate the token (one rotation per call; no
  orphan accumulation). A **blind-timer** `Reissue` would rotate the token every tick → the descriptor
  hash (`token_id`/`fingerprint`) changes every tick → the agent re-consumes + re-reconciles every
  tick = **generation + apply thrash**. This is the real hazard to avoid.
- **Least-churn primary:** re-stage the stored secret (`consumed_at → NULL`). Zero ep0 interaction, no
  hash change; proven in SQ-2b′. Valid whenever the stored token is still good on ep0 (the common
  re-install/rollback case — the token was never deleted).

**SQ-3c — the `consumed-failed` dead-end (code-derived from `manager.go`).**
- Same-hash re-serve **stays put**: with `consumed-failed.json` present and `cf.Hash==h`, the
  idempotent short-circuit is skipped (`manager.go:235`), and when `ConsumePBSToken` returns
  `ErrNoPBSSecret` the `cf.Hash==h` branch (`:275-281`) re-asserts the **LOUD** `consumed_failed`
  state — never a silent burned-secret retry. Matches the package law comment (`:17-19`).
- **Re-issue breaks out** because it does two things at once: stages a **fresh** secret **and** changes
  the descriptor (new `token_id`/`fingerprint` → new hash `h'`). With `h'≠cf.Hash`, the dead-end gate
  no longer matches → the agent re-attempts, consume now succeeds, `finishConverged` runs and
  **`os.Remove(consumedFailedPath())`** clears the dead-end (`manager.go:348-362`).
- Note (design input): **re-staging the stored secret also recovers `consumed_failed`** *when the
  stored token is still valid* — because `ConsumePBSToken` then **succeeds** (not `ErrNoPBSSecret`), so
  the dead-end branch is never entered and the apply proceeds. Only a genuinely stale/rotated ep0
  token forces the `Reissue` escalation.

**Verdict:** a safe re-trigger = **re-assert-secret, gated on the agent's reported stuck state, never
blind-timer-reissue.** Re-staging the stored secret needs **no** generation bump (SQ-2b′ converged on
the agent's own ticker); a generation bump must be reserved for an **actual descriptor content
change** (Reissue / storage-id edit) — reuse the existing `readPBSDR`+compare no-op discipline
(`web/pbsdr.go:148-164`).

---

## SQ-4 — where the re-trigger lives + the condition

**Recommended design (spike-proven, one paragraph):** add a **hub periodic reconciler** (mirroring
`wgsync/reconciler.go`) that, for each host with `dr_tier=1` and an **enabled `pbs_dr` descriptor**,
reads the latest host-report `pbs_dr.state`; when that state is **`waiting_secret` or `consumed_failed`
sustained across ≥1 report cycle** (debounce, so a transient `verify_failed` tunnel flap is *not*
acted on), it **re-stages the stored one-time secret** (`consumed_at → NULL`, **no generation bump**)
and records an event. If the host is still stuck after N recon_cycles (stored token stale →
`consumed_failed` persists), it **escalates once to `Reissue`** (fresh token + descriptor bump). The
capability/host-report path is the only trigger that can observe "DR-ON but agent reports pbsdr
inactive" — the WG hook is structurally unable to help (same pubkey → `changed==false`), and
config-save is operator-manual. **The one spike-proven reason this is correct:** convergence requires
only a *consumable secret*, and staging the existing stored secret is a zero-ep0-churn, no-hash-change,
no-gen-bump operation that the agent picks up on its own 60 s tick (SQ-2b′) and that no-ops cleanly
once converged (SQ-3a) — so the reconciler self-limits to exactly the stuck hosts and stops the moment
they converge.

**Generation-bump discipline (exact condition):** bump **only** on a real descriptor **content**
change (Reissue's new `token_id`/`fingerprint`, or a `storage_id` edit) — never for a secret re-stage.
This is the existing `applyPBSDR` precedent: `readPBSDR` + field compare, no-op-if-equal
(`web/pbsdr.go:148-164`); the agent re-fetches on generation change, so a spurious bump every tick is
an agent-refetch loop.

### Design inputs for the PBS-DR self-heal TASK

| Dimension | Spike-proven answer |
|---|---|
| **Root cause** | Durable hub descriptor + durable *consumed* one-time secret + stable `host_id`, after the agent loses its converged marker (re-install / rollback / disk loss). No path re-mints/re-stages the secret; WG cascade can't re-fire (`changed==false`); `applyPBSDR` "already provisioned" no-ops. Agent stuck in `waiting_secret`. |
| **Safe re-trigger shape** | Re-serve a **consumable secret**, gated on the agent's reported `waiting_secret`/`consumed_failed` (debounced ≥1 cycle). Primary: re-stage stored secret (`consumed_at→NULL`, no ep0 call, no gen bump). Escalation (stored token stale): `Reissue` once. **Never** re-run the provision atom (refuses, doesn't fix) and **never** blind-timer `Reissue` (hash/gen thrash). |
| **Trigger location** | Hub **periodic reconciler** reading host-report `pbs_dr.state` (like `wgsync/reconciler.go`). Not the WG hook (structurally can't re-fire); not config-save (manual). |
| **Gen-bump condition** | Only on a real descriptor **content** change (Reissue/storage-id edit) via `readPBSDR`+compare no-op discipline (`web/pbsdr.go:148-164`). A secret re-stage bumps **nothing**. |
| **Orphan-token cleanup** | **None needed** for the primary path (re-stage = no ep0 interaction) or repeat `Provision` (`ErrTokenExists`, mints nothing). `Reissue` is delete+recreate (no accumulation). The only orphan risk is a *blind-timer* Reissue design — explicitly excluded above. |

---

## Mutations made + reverts (all scoped `host_id='demo-vm-felhom-2f4b00'`; SELECT-verified 1 row before each)

| ID | Mutation | Revert |
|---|---|---|
| MUT-1 | `UPDATE hosts SET desired_generation+1` (3→4) — SQ-2a′ | none needed (monotonic counter; agent compares, absolute value irrelevant) |
| MUT-2 | `UPDATE host_pbs_secrets SET consumed_at=NULL WHERE … AND consumed_at IS NOT NULL` — SQ-2b′ re-stage of the **existing** token secret (no ep0 churn) | **self-reverted**: agent re-consumed at `15:44:51Z`, `consumed_at` re-set by `ConsumeHostPBSSecret` |
| MUT-3 | `UPDATE hosts SET desired_generation+1` (4→5) — SQ-3a | none needed (monotonic) |

**Net DB delta vs spike start:** `host_pbs_secrets.consumed_at` `2026-07-12T20:05:53Z` →
`2026-07-15T15:44:51Z` (secret re-consumed *by real convergence*); `hosts.desired_generation` 3 → 5.
**No orphaned/unconsumed secrets, no ep0 token churn** (existing token reused throughout). No non-drill
host touched (every statement `WHERE host_id='demo-vm-felhom-2f4b00'`; the demo host
`demo-felhom-01` was never in a WHERE clause). Throwaway `hubdbpoke` binary removed from the hub pod.

## Guest end-state

**LEFT CONVERGED** (not rolled back). PBS-DR on the drill guest is now genuinely `state=applied`
(`felhom-pbs` storage entry created, marker `f1059b54…`) — a real side-benefit: the P-DAY0-DEEP
PBS-DR leg is now GREEN on the drill (escrow ceremony + offsite arming remain separate follow-ups).
Rollback to `post_day0_golden136` is available if a pristine "no DR chain" fixture is wanted again.

## STOP conditions honored

Every WHERE clause proven to hit only the drill host (SELECT-before-UPDATE, `count(*)=1`). No code
fix attempted (that is the spec). No ep0 token churn — the primary proof deliberately reused the
existing token; `Reissue` was code-validated, not fired. No non-drill/production host touched.