hub v0.68.0 — auth_failed self-heal, consumed_at honesty gauge, wrapper drift (R-39 + R-50b(a))

Completes the hub half of R-39's fleet fix on top of the generation core (c484aa2).

pbsdrheal gains an auth_failed TRIGGER — a new trigger in the existing machine, not a
new machine. A box whose credential PBS rejects escalates to a fresh mint, never a
re-stage (which would re-feed the secret PBS just rejected), through the EXISTING damper:
a 401 flap must not become a secret-minting chain. With the generation stamp this closes
the loop end to end — agent proves the 401, hub re-keys, generation advances, descriptor
hash moves, agent re-consumes.

consumed_at honesty gauge: a staged secret still unconsumed past a 15-minute grace while
the box reports `applied` is surfaced with its own event. That is the exact 2026-07-18
fingerprint and a disagreement no single tier can see alone. Deliberately a SURFACE, not
a heal — auto-re-issuing on it would mint a second secret on top of an unconsumed one,
which is the mint/consume race R-39(a) already recorded. One event per distinct report,
and an honestly-stuck box does not double-report (its unconsumed secret is the symptom
being healed, not a contradiction).

R-50b(a): ArtifactManifest.WrapperSHA256 + operator field + host-page drift surface. The
PBS wrapper is root-owned 0755 and the pinned sudoers vector, yet installed unversioned
from raw/branch/main and absent from every manifest. Agents >=0.91.0 report the installed
hash; a mismatch is surfaced. An unknown on EITHER side reads as quiet, never as drift —
lighting every host amber on rollout day is how a warning becomes background noise. The
delivery channel itself stays R-50b(b)/(c).

Compatibility unchanged: safe for 0.90.0 agents (unknown JSON key dropped); the re-arm
and auth-honesty guarantees need agent >=0.91.0, so MinAgent moves only after the fleet
has self-updated.

Tests: auth_failed escalate/debounce/recovery-forgets-streak; honesty gauge incl. grace
window, the restage edge (consumed_at deliberately NULLed), consumed-never-alarms, and
honest-stuck-no-double-report; wrapper drift incl. both unknown directions. Red-proof run
at the assertion level: removing the auth_failed arm fails the escalation tests with
reissues=0.

hub/CHANGELOG.md

@@ -1,5 +1,69 @@
 # Felhom Hub — Changelog
 
+## v0.68.0 — a credential re-issue finally re-arms the box (R-39 fleet fix) + wrapper drift is visible (R-50b(a)) (2026-07-21)
+
+**Coupling, stated honestly: this release is SAFE for agents at 0.90.0** — the new descriptor field
+is an unknown JSON key to them; they drop it and behave exactly as today (inert, not breaking).
+**The re-arm and auth-honesty guarantees require agent >= 0.91.0.** Raise MinAgent to 0.91.0 only
+after the fleet's agents have self-updated.
+
+### The defect (R-39, fleet half)
+
+An ep0 credential re-issue re-keys the **secret of an existing token**. `token_id`, `fingerprint`,
+`datastore` and `namespace` all come back byte-identical — only the side-table `host_pbs_secrets`
+row rotates. The agent's re-apply trigger is a change in the **descriptor content hash**
+(`felhom-agent internal/pbsdr/manager.go` `descriptorHash`). Same hash → the converged agent
+short-circuits → the fresh secret is never consumed → the box keeps presenting a revoked credential
+→ **401 forever, while both tiers report `applied`**. Proven on the N100 2026-07-18: the agent's
+`consumed-failed.json` hash was byte-identical to the `marker.json` written two minutes before the
+re-issue.
+
+### The fix
+
+- **`host_pbs_secrets.generation`** — a monotonic per-host counter advanced by every fresh **mint**
+  and by nothing else, stamped into the descriptor as `secret_generation`. That is now the only
+  field a re-key moves, and it is what re-arms the agent.
+  - **A re-stage deliberately does not advance it**: it re-arms the *same* secret, the descriptor
+    content genuinely has not changed, and a bump would cause a pointless agent refetch loop.
+  - `omitempty` is load-bearing — emitting a zero into every pre-existing descriptor would itself be
+    a fleet-wide spurious re-apply.
+  - **Deviation from the spec, deliberate:** the brief said to reuse "the new row's id … no schema
+    change". There is no row id — the table is keyed by `host_id` and UPSERTed last-write-wins, so a
+    new row never exists, and `created_at` collides for two mints in one second. An additive counter
+    column (existing idempotent `ALTER TABLE` idiom) is the only monotonic source available.
+- **`pbsdrheal` gains an `auth_failed` trigger** — a NEW trigger in the existing machine, not a new
+  machine. A box whose credential PBS rejects is escalated to a fresh mint (never a re-stage: that
+  re-feeds the secret PBS just rejected), through the **existing damping** — a 401 flap must not
+  become a secret-minting chain. This closes the loop end to end: agent proves the 401 → hub re-keys
+  → generation advances → descriptor hash moves → agent re-consumes.
+- **`consumed_at` honesty gauge** — a staged secret left **unconsumed** past a 15-minute grace while
+  the box reports `applied` is surfaced loudly with its own event. This is the exact July-18
+  fingerprint and a disagreement **no single tier can detect alone**. Deliberately a *surface*, not
+  a heal: auto-re-issuing here would mint a second secret on top of an unconsumed one — the
+  mint/consume race R-39(a) already recorded.
+- **Corrected a comment that stated a falsehood**: `ReissuePBSDR` claimed it refreshed the descriptor
+  "with the NEW token_id/fingerprint". That is false for a re-key, and believing it is why nobody
+  expected the descriptor to come back identical.
+
+### R-50b(a) — wrapper drift is answerable
+
+`ArtifactManifest.WrapperSHA256` + an operator field. Unlike the agent binary and the golden, the
+PBS-DR wrapper is installed from `raw/branch/main` — unversioned, unpinned, absent from every
+manifest — yet it is root-owned 0755 and the pinned sudoers vector. Agents (>= 0.91.0) report the
+installed file's hash and the host page surfaces a mismatch. **An unknown on either side reads as
+quiet, never as drift** — lighting every host amber on rollout day is how a warning becomes noise.
+This does not fix the delivery channel; that stays R-50b(b)/(c).
+
+### Tests
+
+Store-level generation monotonicity, per-host isolation and restage-leaves-it-alone; descriptor
+byte-change, `omitempty` and sibling-key round-trip; a **flow-level** test driving `ReissuePBSDR`
+against a fake that models a real re-key; `auth_failed` escalate/debounce/recovery; the honesty
+gauge incl. its grace window, the restage edge, the consumed case and the honest-stuck case; wrapper
+drift incl. both unknown directions. **Two red-proofs run at the assertion level** (not the
+compiler): removing the generation stamp makes the flow test fail with both byte-identical blocks
+printed; removing the `auth_failed` arm makes the escalation tests fail with `reissues=0`.
+
 ## v0.67.0 — the hub stops keeping things to itself: auto-minted self-bind link, post-RESET staleness, unprovisioned-offsite warning (2026-07-18)
 
 Four small items, each one a case where the hub already knew something and said nothing. Green:

hub/internal/pbsdrheal/reconciler.go

@@ -33,8 +33,15 @@ import (
 // (the secret is untouched by verify-before-consume), self-heals when the tunnel recovers, and must
 // stay LOUD for the operator if it does not — re-staging a secret would not help it.
 const (
-	stateWaitingSecret = "waiting_secret"
+	stateWaitingSecret  = "waiting_secret"
 	stateConsumedFailed = "consumed_failed"
+	// stateAuthFailed (R-39, agent >= 0.91.0) — the box HAS a credential, the descriptor is applied,
+	// and PBS rejects it with 401. Before 0.91.0 this state could not exist: the agent's verify loop
+	// read the credential file directly as non-root, always failed with "permission denied", and
+	// skipped — so an applied-and-dead tier was invisible to both tiers. It is healed like
+	// consumed_failed (escalate to a fresh mint), never by a re-stage: re-staging re-feeds the SAME
+	// secret PBS just rejected.
+	stateAuthFailed = "auth_failed"
 )
 
 // Audit event types (store.SaveEvent; hub-internal, not gated by allowedEventTypes). Distinct per
@@ -43,8 +50,18 @@ const (
 	eventRestaged       = "pbsdr_selfheal_restaged"        // re-armed the stored secret (routine)
 	eventReissued       = "pbsdr_selfheal_reissued"        // no stored secret → minted a fresh one
 	eventConsumedFailed = "pbsdr_selfheal_consumed_failed" // burned secret → minted a fresh one (a real problem was remediated)
+	eventAuthFailed     = "pbsdr_selfheal_auth_failed"     // PBS rejected the box's credential (401) → minted a fresh one
+	eventUnconsumed     = "pbsdr_unconsumed_secret"        // staged secret never consumed under an `applied` box (surfaced, NOT auto-healed)
 )
 
+// unconsumedGrace bounds how long a staged-but-unconsumed secret is NORMAL before it is a lie.
+//
+// A fresh mint (or a deliberate re-stage) is consumed on the agent's next tick — well inside one
+// ~15-minute report cycle. Beyond this window, an unconsumed secret sitting under a box that reports
+// `applied` is the exact fingerprint of the 2026-07-18 N100 failure: hub minted, agent
+// short-circuited, both tiers green, box serving a revoked credential.
+const unconsumedGrace = 15 * time.Minute
+
 // Actions is the mutation seam — fakes in tests count calls without SSH/ep0. Restage flips a stored
 // secret's consumed flag (returns restaged=false when NO row exists → the caller escalates). Reissue
 // mints a fresh ep0 token + stores a fresh consume-once secret + bumps the descriptor.
@@ -71,7 +88,9 @@ func (a storeActions) Reissue(ctx context.Context, customerID string) error {
 }
 
 // NewActions builds the production mutation seam.
-func NewActions(st *store.Store, reissuer Reissuer) Actions { return storeActions{st: st, reissuer: reissuer} }
+func NewActions(st *store.Store, reissuer Reissuer) Actions {
+	return storeActions{st: st, reissuer: reissuer}
+}
 
 // debounceState tracks, per host, the last DISTINCT report observed and how many consecutive
 // distinct reports it has held the current stuck state — so a fresh box that briefly shows
@@ -86,15 +105,18 @@ type debounceState struct {
 // Reconciler re-arms stuck PBS-DR hosts. DECLARATIVE + IDEMPOTENT: a tick over a converged fleet
 // writes nothing (Scenario C). It reads the hub DB (the source of truth) — never the box.
 type Reconciler struct {
-	store    *store.Store
-	act      Actions
-	interval time.Duration
-	debounceReports int // distinct stuck reports required before healing (default 2)
-	onlyHost string // "" = whole fleet; non-empty restricts the work set to one host (supervised rollout)
-	trigger  chan struct{}
-	logger   *log.Logger
+	store           *store.Store
+	act             Actions
+	interval        time.Duration
+	debounceReports int    // distinct stuck reports required before healing (default 2)
+	onlyHost        string // "" = whole fleet; non-empty restricts the work set to one host (supervised rollout)
+	trigger         chan struct{}
+	logger          *log.Logger
 
 	deb map[string]debounceState
+	// unconsumedSeen remembers the last report id already surfaced per host (Scenario F), so a
+	// sustained disagreement produces one event per fresh report rather than one per tick.
+	unconsumedSeen map[string]int64
 }
 
 // NewReconciler builds the reconciler. interval defaults to 5m, debounceReports to 2.
@@ -110,6 +132,7 @@ func NewReconciler(st *store.Store, act Actions, logger *log.Logger) *Reconciler
 		trigger:         make(chan struct{}, 1),
 		logger:          logger,
 		deb:             map[string]debounceState{},
+		unconsumedSeen:  map[string]int64{},
 	}
 }
 
@@ -161,6 +184,14 @@ func (r *Reconciler) reconcileOnce(ctx context.Context) {
 			delete(r.deb, row.HostID)
 			continue
 		}
+		// R-39 consumed_at HONESTY (Scenario F). Deliberately a SURFACE, not a heal: the remediation
+		// for a genuinely stuck box is the auth_failed / waiting_secret / consumed_failed machinery
+		// above, driven by what the BOX reports. This check catches the disagreement itself — the hub
+		// staged a credential the box never took, while the box claims to be applied — which is a
+		// state no single tier can detect alone. Auto-re-issuing on it would mint a second secret on
+		// top of an unconsumed one: exactly the mint/consume race R-39(a) already recorded.
+		r.checkUnconsumed(row)
+
 		switch row.ReportedState {
 		case stateWaitingSecret:
 			if r.confirm(row) {
@@ -170,6 +201,10 @@ func (r *Reconciler) reconcileOnce(ctx context.Context) {
 			if r.confirm(row) {
 				r.healConsumedFailed(ctx, row)
 			}
+		case stateAuthFailed:
+			if r.confirm(row) {
+				r.healAuthFailed(ctx, row)
+			}
 		default:
 			// applied | adopted | disabled | verify_failed | "" (no report) | anything else → no-op.
 			delete(r.deb, row.HostID)
@@ -183,6 +218,31 @@ func (r *Reconciler) reconcileOnce(ctx context.Context) {
 	}
 }
 
+// checkUnconsumed surfaces the applied-but-never-consumed disagreement (Scenario F). One event per
+// distinct report, so a sustained state does not spam the audit log every tick.
+func (r *Reconciler) checkUnconsumed(row store.PBSDRHealRow) {
+	if row.SecretUnconsumedFor <= unconsumedGrace {
+		return // no staged secret, already consumed, or still inside the normal pickup window
+	}
+	// Only meaningful when the box claims to be converged. A box that honestly reports
+	// waiting_secret / consumed_failed / auth_failed is already being healed above and its unconsumed
+	// secret is the SYMPTOM being fixed, not a contradiction.
+	if row.ReportedState != "applied" && row.ReportedState != "adopted" {
+		return
+	}
+	if st, ok := r.unconsumedSeen[row.HostID]; ok && st == row.ReportID {
+		return // already surfaced for this exact report
+	}
+	if r.unconsumedSeen == nil {
+		r.unconsumedSeen = map[string]int64{}
+	}
+	r.unconsumedSeen[row.HostID] = row.ReportID
+	r.logger.Printf("[WARN] pbsdrheal: host %s (customer %s) reports pbs_dr=%s while a staged one-time secret has been UNCONSUMED for %s — the box is not using the credential the hub issued (R-39 fingerprint)",
+		row.HostID, row.CustomerID, row.ReportedState, row.SecretUnconsumedFor.Round(time.Minute))
+	r.event(row.CustomerID, eventUnconsumed, "warning",
+		"PBS-DR honesty check: the box reports the DR tier as applied, but a one-time credential issued by the hub has never been consumed. The tier may be authenticating with a superseded credential.")
+}
+
 // confirm advances the per-host debounce and reports whether the stuck state has held across
 // >= debounceReports DISTINCT reports. A re-observed same report (same reportID) never advances the
 // streak — the debounce counts fresh evidence, not reconciler ticks.
@@ -235,6 +295,24 @@ func (r *Reconciler) healConsumedFailed(ctx context.Context, row store.PBSDRHeal
 	}
 }
 
+// healAuthFailed escalates a box whose credential PBS rejects (401) to a fresh mint. This is the leg
+// that closes the R-39 loop end to end: the agent now PROVES the credential is dead instead of
+// silently skipping, the hub re-keys, the fresh mint advances the secret generation, the descriptor
+// hash moves, and the agent finally re-consumes (Scenario A). Before this, an applied-and-401 tier
+// stayed green forever and would have surfaced first at a real restore.
+//
+// A re-stage is deliberately NOT attempted: the stored secret IS the one PBS just rejected, so
+// re-arming it would burn a tick and change nothing. Damping is the SHARED confirm() — a 401 flap
+// must not turn into a secret-minting chain.
+func (r *Reconciler) healAuthFailed(ctx context.Context, row store.PBSDRHealRow) {
+	r.logger.Printf("[WARN] pbsdrheal: host %s (customer %s) reports auth_failed (PBS rejects its credential) — escalating to Re-issue", row.HostID, row.CustomerID)
+	if r.reissue(ctx, row) {
+		r.event(row.CustomerID, eventAuthFailed, "warning",
+			"PBS-DR self-heal: a box reported auth_failed (the DR endpoint rejected its stored credential); re-issued fresh endpoint credentials so the agent can re-consume and converge.")
+		r.resetAfterHeal(row)
+	}
+}
+
 // reissue runs the escalation; returns true on success (the caller then records the audit event).
 func (r *Reconciler) reissue(ctx context.Context, row store.PBSDRHealRow) bool {
 	if err := r.act.Reissue(ctx, row.CustomerID); err != nil {

hub/internal/pbsdrheal/reconciler_test.go

@@ -12,6 +12,7 @@ import (
 	"path/filepath"
 	"sync"
 	"testing"
+	"time"
 
 	"gitea.dooplex.hu/admin/felhom-hub/internal/store"
 )
@@ -75,6 +76,16 @@ func seedReport(t *testing.T, st *store.Store, hostID, customerID, state string)
 	}
 }
 
+// ageSecret back-dates a staged secret's created_at so grace-window behaviour is testable without
+// sleeping. Writes the house SQLite datetime format.
+func ageSecret(t *testing.T, st *store.Store, hostID string, by time.Duration) {
+	t.Helper()
+	when := time.Now().UTC().Add(-by).Format("2006-01-02 15:04:05")
+	if err := st.SetHostPBSSecretCreatedAtForTest(hostID, when); err != nil {
+		t.Fatalf("age secret %s: %v", hostID, err)
+	}
+}
+
 func boolStr(b bool) string {
 	if b {
 		return "true"
@@ -228,7 +239,7 @@ func TestScenarioE_ConsumedFailedEscalates(t *testing.T) {
 // F — DR-OFF (disabled descriptor) and enabled-but-unprovisioned hosts are never in the work set.
 func TestScenarioF_OutOfScopeHostsUntouched(t *testing.T) {
 	st := newHealStore(t)
-	seedHost(t, st, "hOff", "cOff", false, true)   // descriptor disabled
+	seedHost(t, st, "hOff", "cOff", false, true) // descriptor disabled
 	seedReport(t, st, "hOff", "cOff", stateWaitingSecret)
 	seedHost(t, st, "hUnprov", "cUnprov", true, false) // enabled but never provisioned
 	seedReport(t, st, "hUnprov", "cUnprov", stateWaitingSecret)
@@ -274,3 +285,190 @@ func TestNoReHealOnSameReport(t *testing.T) {
 		t.Fatalf("re-healed the same stuck report: restages=%d, want 1", fake.restages())
 	}
 }
+
+// R-39 — auth_failed is escalated to a fresh mint, never re-staged.
+//
+// This is the leg that closes the loop end to end: the agent (>=0.91.0) now PROVES the credential is
+// rejected instead of silently skipping, and the hub re-keys, which advances the secret generation,
+// which moves the descriptor hash, which finally re-arms the agent. A re-stage would re-feed the very
+// secret PBS just rejected.
+//
+// COMPANION RED-PROOF (run + recorded): delete the `case stateAuthFailed` arm → the state falls to
+// the default no-op branch and this test FAILS with reissues=0.
+func TestR39_AuthFailedEscalatesToReissue(t *testing.T) {
+	st := newHealStore(t)
+	seedHost(t, st, "hAF", "cAF", true, true)
+	seedReport(t, st, "hAF", "cAF", stateAuthFailed)
+	fake := &fakeActions{restageResult: true} // a re-stageable secret exists — it must STILL not be used
+	r := newRec(st, fake)
+
+	r.reconcileOnce(context.Background())
+
+	if fake.restages() != 0 {
+		t.Errorf("restages=%d, want 0 — re-staging re-feeds the credential PBS just rejected", fake.restages())
+	}
+	if fake.reissues() != 1 {
+		t.Fatalf("reissues=%d, want 1 — an applied-and-401 tier must be re-keyed, not left green", fake.reissues())
+	}
+	if got := eventTypes(t, st, "cAF"); len(got) != 1 || got[0] != eventAuthFailed {
+		t.Errorf("events = %v, want [%s] (distinct auth_failed signal, not borrowed from consumed_failed)", got, eventAuthFailed)
+	}
+}
+
+// The damper is the burn protection: a 401 flap must not become a secret-minting chain. auth_failed
+// uses the SAME confirm() debounce as the other triggers — one report is not enough.
+func TestR39_AuthFailedIsDebounced(t *testing.T) {
+	st := newHealStore(t)
+	seedHost(t, st, "hAF2", "cAF2", true, true)
+	fake := &fakeActions{restageResult: true}
+	r := newRec(st, fake)
+	r.debounceReports = 2 // require two DISTINCT stuck reports
+
+	seedReport(t, st, "hAF2", "cAF2", stateAuthFailed)
+	r.reconcileOnce(context.Background())
+	if fake.reissues() != 0 {
+		t.Fatalf("re-issued on the FIRST auth_failed report (reissues=%d) — the damper is bypassed", fake.reissues())
+	}
+
+	seedReport(t, st, "hAF2", "cAF2", stateAuthFailed) // a second, distinct report
+	r.reconcileOnce(context.Background())
+	if fake.reissues() != 1 {
+		t.Fatalf("reissues=%d after two distinct auth_failed reports, want 1", fake.reissues())
+	}
+
+	// And the same confirmed report must not re-heal on the next tick.
+	r.reconcileOnce(context.Background())
+	if fake.reissues() != 1 {
+		t.Errorf("re-healed on a stale report: reissues=%d, want 1", fake.reissues())
+	}
+}
+
+// A host that recovers (auth_failed → applied) is a pure no-op, and its streak is forgotten.
+func TestR39_AuthFailedRecoveryIsNoOp(t *testing.T) {
+	st := newHealStore(t)
+	seedHost(t, st, "hAF3", "cAF3", true, true)
+	fake := &fakeActions{restageResult: true}
+	r := newRec(st, fake)
+	r.debounceReports = 2 // so a single stuck report is not yet actionable
+
+	seedReport(t, st, "hAF3", "cAF3", stateAuthFailed)
+	r.reconcileOnce(context.Background())        // streak 1 — below the damper, nothing done
+	seedReport(t, st, "hAF3", "cAF3", "applied") // the box healed on its own
+	r.reconcileOnce(context.Background())
+
+	if fake.reissues() != 0 || fake.restages() != 0 {
+		t.Errorf("a recovered host was acted on: restages=%d reissues=%d, want 0/0", fake.restages(), fake.reissues())
+	}
+
+	// The streak must be FORGOTTEN, not merely paused: a later single auth_failed must start over
+	// rather than instantly tripping the damper it had half-filled before recovering.
+	seedReport(t, st, "hAF3", "cAF3", stateAuthFailed)
+	r.reconcileOnce(context.Background())
+	if fake.reissues() != 0 {
+		t.Errorf("a stale pre-recovery streak survived and fired early: reissues=%d, want 0", fake.reissues())
+	}
+}
+
+// R-39 Scenario F — the consumed_at HONESTY gauge.
+//
+// The July-18 fingerprint is a disagreement no single tier can see: the hub staged a credential, the
+// box never took it, and the box nonetheless reports `applied`. This surfaces it. It deliberately
+// does NOT auto-heal — minting on top of an unconsumed secret is the mint/consume race R-39(a)
+// already recorded.
+//
+// COMPANION RED-PROOF (run + recorded): delete the r.checkUnconsumed(row) call → no event is written
+// and TestR39_UnconsumedSecretUnderAppliedIsSurfaced FAILS.
+func TestR39_UnconsumedSecretUnderAppliedIsSurfaced(t *testing.T) {
+	st := newHealStore(t)
+	seedHost(t, st, "hU", "cU", true, true)
+	seedReport(t, st, "hU", "cU", "applied") // the box claims converged
+	if _, err := st.SaveHostPBSSecret("hU", "never-taken"); err != nil {
+		t.Fatalf("mint: %v", err)
+	}
+	// Age the staged secret past the grace window.
+	ageSecret(t, st, "hU", 3*time.Hour)
+
+	fake := &fakeActions{restageResult: true}
+	r := newRec(st, fake)
+	r.reconcileOnce(context.Background())
+
+	if got := eventTypes(t, st, "cU"); len(got) != 1 || got[0] != eventUnconsumed {
+		t.Fatalf("events = %v, want [%s]", got, eventUnconsumed)
+	}
+	// SURFACE, not heal: an applied box is not in any heal arm, so nothing was minted or re-staged.
+	if fake.reissues() != 0 || fake.restages() != 0 {
+		t.Errorf("the honesty gauge must not mutate: restages=%d reissues=%d, want 0/0", fake.restages(), fake.reissues())
+	}
+
+	// One event per DISTINCT report — a sustained disagreement must not spam the audit log.
+	r.reconcileOnce(context.Background())
+	if got := eventTypes(t, st, "cU"); len(got) != 1 {
+		t.Errorf("re-surfaced on the same report: %d events, want 1", len(got))
+	}
+}
+
+// A freshly staged secret (or a just-restaged one) is NORMAL inside the grace window and must not
+// alarm — RestageHostPBSSecret deliberately sets consumed_at back to NULL.
+func TestR39_FreshlyStagedSecretDoesNotAlarm(t *testing.T) {
+	st := newHealStore(t)
+	seedHost(t, st, "hF", "cF", true, true)
+	seedReport(t, st, "hF", "cF", "applied")
+	if _, err := st.SaveHostPBSSecret("hF", "just-minted"); err != nil {
+		t.Fatalf("mint: %v", err)
+	}
+	// A deliberate re-stage clears consumed_at — the Scenario-F edge case.
+	if _, err := st.ConsumeHostPBSSecret("hF"); err != nil {
+		t.Fatalf("consume: %v", err)
+	}
+	if _, err := st.RestageHostPBSSecret("hF"); err != nil {
+		t.Fatalf("restage: %v", err)
+	}
+
+	r := newRec(st, &fakeActions{restageResult: true})
+	r.reconcileOnce(context.Background())
+
+	if got := eventTypes(t, st, "cF"); len(got) != 0 {
+		t.Errorf("a just-staged secret alarmed inside the grace window: %v", got)
+	}
+}
+
+// A CONSUMED secret never alarms, however old.
+func TestR39_ConsumedSecretNeverAlarms(t *testing.T) {
+	st := newHealStore(t)
+	seedHost(t, st, "hC", "cC", true, true)
+	seedReport(t, st, "hC", "cC", "applied")
+	if _, err := st.SaveHostPBSSecret("hC", "taken"); err != nil {
+		t.Fatalf("mint: %v", err)
+	}
+	if _, err := st.ConsumeHostPBSSecret("hC"); err != nil {
+		t.Fatalf("consume: %v", err)
+	}
+	ageSecret(t, st, "hC", 30*24*time.Hour)
+
+	r := newRec(st, &fakeActions{restageResult: true})
+	r.reconcileOnce(context.Background())
+	if got := eventTypes(t, st, "cC"); len(got) != 0 {
+		t.Errorf("a consumed secret alarmed: %v", got)
+	}
+}
+
+// A box that HONESTLY reports a stuck state is already being healed; its unconsumed secret is the
+// symptom, not a contradiction, so the gauge stays quiet and only the heal event is written.
+func TestR39_HonestStuckStateDoesNotDoubleReport(t *testing.T) {
+	st := newHealStore(t)
+	seedHost(t, st, "hH", "cH", true, true)
+	seedReport(t, st, "hH", "cH", stateWaitingSecret)
+	if _, err := st.SaveHostPBSSecret("hH", "staged"); err != nil {
+		t.Fatalf("mint: %v", err)
+	}
+	ageSecret(t, st, "hH", 3*time.Hour)
+
+	r := newRec(st, &fakeActions{restageResult: true})
+	r.reconcileOnce(context.Background())
+
+	for _, e := range eventTypes(t, st, "cH") {
+		if e == eventUnconsumed {
+			t.Error("an honestly-stuck box must not also raise the honesty gauge — it is the symptom being healed")
+		}
+	}
+}

hub/internal/store/pbsdr.go

@@ -3,6 +3,7 @@ package store
 import (
 	"database/sql"
 	"encoding/json"
+	"time"
 )
 
 // PBS DR tier (SLICE 1): the HOST-scoped one-time PBS token secret — the host/agent twin of the
@@ -102,6 +103,17 @@ type PBSDRHealRow struct {
 	DescriptorProvisioned bool   // desired_json pbs_dr.namespace != "" (was provisioned, not a bare enable)
 	ReportedState         string // latest report pbs_dr.state ("" = no report / no stanza)
 	ReportID              int64  // id of that latest host_report (0 = none); the debounce distinctness key
+
+	// R-39 consumed_at honesty (Scenario F). SecretUnconsumedFor is how long an UNCONSUMED secret has
+	// been staged for this host (0 when none is staged, or when it has already been consumed).
+	//
+	// A staged-but-unconsumed secret sitting under a box that reports `applied` is the exact
+	// fingerprint of the 2026-07-18 N100 failure: the hub minted, the agent short-circuited, and both
+	// tiers reported success while the box served a revoked credential. Briefly unconsumed is NORMAL
+	// (a fresh mint, or a deliberate re-stage, is consumed on the agent's next tick) — only a
+	// SUSTAINED one is a lie, which is why the caller applies a grace window rather than alarming on
+	// presence alone.
+	SecretUnconsumedFor time.Duration
 }
 
 // PBSDRHealStates returns one row per host: its descriptor enable/provision flags + the agent's
@@ -110,11 +122,13 @@ type PBSDRHealRow struct {
 // (the reconciler filters to enabled+provisioned hosts).
 func (s *Store) PBSDRHealStates() ([]PBSDRHealRow, error) {
 	rows, err := s.db.Query(`
-		SELECT h.host_id, h.customer_id, h.desired_json, latest.mx, hr.report_json
+		SELECT h.host_id, h.customer_id, h.desired_json, latest.mx, hr.report_json,
+		       ps.created_at, ps.consumed_at
 		FROM hosts h
 		LEFT JOIN (SELECT host_id, MAX(id) AS mx FROM host_reports GROUP BY host_id) latest
 		       ON latest.host_id = h.host_id
-		LEFT JOIN host_reports hr ON hr.id = latest.mx`)
+		LEFT JOIN host_reports hr ON hr.id = latest.mx
+		LEFT JOIN host_pbs_secrets ps ON ps.host_id = h.host_id`)
 	if err != nil {
 		return nil, err
 	}
@@ -123,8 +137,9 @@ func (s *Store) PBSDRHealStates() ([]PBSDRHealRow, error) {
 	for rows.Next() {
 		var hostID, customerID, desiredJSON string
 		var reportID sql.NullInt64
-		var reportJSON sql.NullString
-		if err := rows.Scan(&hostID, &customerID, &desiredJSON, &reportID, &reportJSON); err != nil {
+		var reportJSON, secretCreatedAt, secretConsumedAt sql.NullString
+		if err := rows.Scan(&hostID, &customerID, &desiredJSON, &reportID, &reportJSON,
+			&secretCreatedAt, &secretConsumedAt); err != nil {
 			return nil, err
 		}
 		r := PBSDRHealRow{HostID: hostID, CustomerID: customerID}
@@ -151,7 +166,27 @@ func (s *Store) PBSDRHealStates() ([]PBSDRHealRow, error) {
 				r.ReportedState = rr.PBSDR.State
 			}
 		}
+		// Unconsumed-secret age: only when a secret is staged AND still unconsumed.
+		if secretCreatedAt.Valid && !secretConsumedAt.Valid {
+			// SQLite datetime('now') is UTC and format-varied — parseSQLiteTime is the house parser
+			// (it returns the zero time on an unparseable value, which we treat as "unknown", never
+			// as "infinitely stale").
+			if created := parseSQLiteTime(secretCreatedAt.String); !created.IsZero() {
+				if age := time.Since(created); age > 0 {
+					r.SecretUnconsumedFor = age
+				}
+			}
+		}
 		out = append(out, r)
 	}
 	return out, rows.Err()
 }
+
+// SetHostPBSSecretCreatedAtForTest back-dates a staged secret's created_at. TEST-ONLY seam: the
+// grace-window behaviour of the consumed_at honesty gauge (R-39 Scenario F) is otherwise only
+// observable by sleeping for 15 minutes. It touches nothing else — not the value, not consumed_at,
+// not the generation.
+func (s *Store) SetHostPBSSecretCreatedAtForTest(hostID, sqliteTime string) error {
+	_, err := s.db.Exec(`UPDATE host_pbs_secrets SET created_at = ? WHERE host_id = ?`, sqliteTime, hostID)
+	return err
+}

hub/internal/store/store.go

@@ -1476,6 +1476,17 @@ type ArtifactManifest struct {
 	// box whose agent is below it (Part D) — mechanising the publish-train "agent BEFORE controller
 	// floor" rule instead of leaving it to operator discipline.
 	MinAgent string `json:"min_agent"`
+	// WrapperSHA256 is the sha256 of the PBS-DR apply wrapper (configs/felhom-pbs-apply) the operator
+	// has vouched (R-50b(a), v0.68.0).
+	//
+	// Unlike the agent binary and the golden, this artifact is installed from
+	// `raw/branch/main` by felhom-host-install.sh — UNVERSIONED, with no tag, no pin and no checksum.
+	// It is a root-owned 0755 file and the pinned sudoers vector for the PBS storage verbs, so "which
+	// wrapper is on this host?" was previously unanswerable from any manifest: two hosts installed a
+	// week apart could carry different privileged code while reporting the same agent version.
+	// Recording the hash here does not fix the delivery channel (that is R-50b(b)/(c)) — it makes
+	// DRIFT VISIBLE, which is the cheap honest first step.
+	WrapperSHA256 string `json:"wrapper_sha256"`
 }
 
 // hub_settings keys for the artifact manifest (BUNDLE slice). Stored as discrete key/value rows in
@@ -1487,6 +1498,7 @@ const (
 	settingArtifactGoldenVersion = "artifact_golden_version"
 	settingArtifactGoldenSHA256  = "artifact_golden_sha256"
 	settingArtifactMinAgent      = "artifact_min_agent"
+	settingArtifactWrapperSHA256 = "artifact_wrapper_sha256" // R-50b(a): the vouched felhom-pbs-apply hash
 )
 
 // settingOperatorPasswordHash is the hub_settings key for the operator login password bcrypt hash,
@@ -1534,6 +1546,7 @@ func (s *Store) GetArtifactManifest() ArtifactManifest {
 		GoldenVersion: s.getSetting(settingArtifactGoldenVersion),
 		GoldenSHA256:  s.getSetting(settingArtifactGoldenSHA256),
 		MinAgent:      s.getSetting(settingArtifactMinAgent),
+		WrapperSHA256: s.getSetting(settingArtifactWrapperSHA256),
 	}
 }
 
@@ -1552,7 +1565,10 @@ func (s *Store) SetArtifactManifest(m ArtifactManifest) error {
 	if err := s.setSetting(settingArtifactGoldenSHA256, m.GoldenSHA256); err != nil {
 		return err
 	}
-	return s.setSetting(settingArtifactMinAgent, m.MinAgent)
+	if err := s.setSetting(settingArtifactMinAgent, m.MinAgent); err != nil {
+		return err
+	}
+	return s.setSetting(settingArtifactWrapperSHA256, m.WrapperSHA256)
 }
 
 // EffectiveMinControllerVersion resolves the floor that actually applies to a customer: the

hub/internal/web/configs.go

@@ -1019,6 +1019,12 @@ func normalizeSHA256(raw string) (string, bool) {
 // that version's sha256 from Gitea itself (never trusting a client-supplied checksum), so there is no
 // hand-copied sha to get wrong. When no Gitea client is configured (no registry creds) it falls back
 // to the submitted sha256 (legacy manual path). Empty version clears that artifact.
+// sha256HexRe validates an operator-typed sha256 (R-50b(a) wrapper hash): exactly 64 lowercase hex
+// characters. The agent/golden hashes are resolved from the package registry instead, so this is the
+// only manifest field a human types by hand — and a truncated paste must be refused, not stored as a
+// hash that can never match.
+var sha256HexRe = regexp.MustCompile(`^[0-9a-f]{64}$`)
+
 func (s *Server) handleSetArtifacts(w http.ResponseWriter, r *http.Request) {
 	if err := r.ParseForm(); err != nil {
 		http.Error(w, "Bad request", http.StatusBadRequest)
@@ -1037,18 +1043,27 @@ func (s *Server) handleSetArtifacts(w http.ResponseWriter, r *http.Request) {
 		http.Redirect(w, r, "/configuration?flash=artifact_sha_invalid", http.StatusSeeOther)
 		return
 	}
+	// R-50b(a): the PBS-DR wrapper hash is operator-typed, not resolved from the package registry —
+	// unlike the agent binary and the golden, this artifact is not published there at all. It is
+	// installed from raw/branch/main, which is exactly the drift this field makes visible.
+	wrapperSHA := strings.ToLower(strings.TrimSpace(r.FormValue("wrapper_sha256")))
+	if wrapperSHA != "" && !sha256HexRe.MatchString(wrapperSHA) {
+		http.Redirect(w, r, "/configuration?flash=artifact_sha_invalid", http.StatusSeeOther)
+		return
+	}
 	if err := s.store.SetArtifactManifest(store.ArtifactManifest{
 		AgentVersion:  agentVer,
 		AgentSHA256:   agentSHA,
 		GoldenVersion: goldenVer,
 		GoldenSHA256:  goldenSHA,
 		MinAgent:      minAgent,
+		WrapperSHA256: wrapperSHA,
 	}); err != nil {
 		s.logger.Printf("[ERROR] Failed to set artifact manifest: %v", err)
 		http.Error(w, "Internal error", http.StatusInternalServerError)
 		return
 	}
-	s.logger.Printf("[INFO] Artifact manifest set: agent=%s golden=%s min_agent=%q", agentVer, goldenVer, minAgent)
+	s.logger.Printf("[INFO] Artifact manifest set: agent=%s golden=%s min_agent=%q wrapper_sha=%t", agentVer, goldenVer, minAgent, wrapperSHA != "")
 	// Agent-plane immediate-sync (Direction-2a, v0.59.0): a MinAgent-floor / vouched-agent change is
 	// a fleet-wide agent-plane intent shift. Fire-and-forget nudge every box so it re-reports at
 	// once (the self-update train's signed op / floor re-evaluation lands in seconds, not ≤15 min).

hub/internal/web/hosts.go

@@ -356,6 +356,49 @@ func (s *Server) handleHostsList(w http.ResponseWriter, r *http.Request) {
 // hostDetailData assembles the view-model map the shared host_detail_body sub-template
 // renders — used by BOTH the standalone /hosts/{id} page and the customer page's Host tab
 // (v0.47.0). Booleans/counts only for DR/escrow; never api_key or blob contents.
+// wrapperDrift compares the PBS-DR wrapper hash a host REPORTS against the one the operator vouched
+// in the artifact manifest (R-50b(a), v0.68.0).
+//
+// The wrapper is a root-owned 0755 file installed from `raw/branch/main` — unversioned, unpinned and
+// absent from every manifest until now, so two hosts installed a week apart could carry different
+// privileged code while reporting the same agent version. This does not fix the delivery channel
+// (R-50b(b)/(c)); it makes drift VISIBLE, which is what was missing.
+//
+// Returns ("", "") when either side is unknown: a hub that has not vouched a hash, or an agent below
+// 0.91.0 that does not report one, is NOT drift — treating "unknown" as "mismatch" would light every
+// host amber on the day this ships and teach the operator to ignore it.
+func (s *Server) wrapperDrift(reportJSON string) (drift string, reported string) {
+	reported = parseReportedWrapperSHA(reportJSON)
+	return compareWrapperSHA(reported, s.store.GetArtifactManifest().WrapperSHA256), reported
+}
+
+// compareWrapperSHA is the pure comparison: "" (quiet) when either side is unknown, else ok/mismatch.
+func compareWrapperSHA(reported, vouched string) string {
+	if reported == "" || vouched == "" {
+		return ""
+	}
+	if !strings.EqualFold(reported, vouched) {
+		return "mismatch"
+	}
+	return "ok"
+}
+
+// parseReportedWrapperSHA pulls host.wrapper_sha256 out of a host report ("" when absent).
+func parseReportedWrapperSHA(reportJSON string) string {
+	if strings.TrimSpace(reportJSON) == "" {
+		return ""
+	}
+	var doc struct {
+		Host struct {
+			WrapperSHA256 string `json:"wrapper_sha256"`
+		} `json:"host"`
+	}
+	if json.Unmarshal([]byte(reportJSON), &doc) != nil {
+		return ""
+	}
+	return strings.ToLower(strings.TrimSpace(doc.Host.WrapperSHA256))
+}
+
 func (s *Server) hostDetailData(host *store.Host, r *http.Request) map[string]interface{} {
 	status := s.hostStatus(host.LastReportAt)
 
@@ -369,6 +412,7 @@ func (s *Server) hostDetailData(host *store.Host, r *http.Request) map[string]in
 
 	reportJSON, _ := s.store.GetLatestHostReportJSON(host.CustomerID)
 	vitals := parseHostVitals(reportJSON)
+	wrapperDrift, reportedWrapperSHA := s.wrapperDrift(reportJSON)
 	storageTargets := parseHostStorageTargets(reportJSON)
 	sort.Slice(storageTargets, func(i, j int) bool { return storageTargets[i].Name < storageTargets[j].Name })
 	// v0.51.0: capability chips — non-ok first (what the operator needs to see), then by name.
@@ -395,28 +439,31 @@ func (s *Server) hostDetailData(host *store.Host, r *http.Request) map[string]in
 	escrow, _ := s.store.GetHostEscrow(host.HostID)
 
 	return map[string]interface{}{
-		"HostID":            host.HostID,
-		"CustomerID":        host.CustomerID,
-		"CustomerName":      s.customerName(host.CustomerID),
-		"AgentVersion":      host.AgentVersion,
-		"CreatedAt":         host.CreatedAt,
-		"Status":            status,
-		"StatusLabel":       hostStatusLabel(status),
-		"StatusClass":       hostStatusClass(status),
-		"LastReportAt":      host.LastReportAt,
-		"HasReport":         host.LastReportAt != nil,
-		"RecoveryMode":      host.InRecoveryMode(time.Now()),
-		"RecoveryUntil":     host.RecoveryModeUntil,
-		"DesiredGeneration": host.DesiredGeneration,
-		"Vitals":            vitals,
-		"Guests":            guests,
-		"GuestRunning":      guestRunning,
-		"GuestTotal":        len(guests),
-		"StorageTargets":    storageTargets,
-		"Capabilities":      capabilities,
-		"NeedsDRMigration":  capabilitiesNeedDRMigration(capabilities),
-		"DRPresent":         drBundle != nil,
-		"EscrowPresent":     escrow != nil,
+		"WrapperDrift":       wrapperDrift,
+		"ReportedWrapperSHA": reportedWrapperSHA,
+		"VouchedWrapperSHA":  s.store.GetArtifactManifest().WrapperSHA256,
+		"HostID":             host.HostID,
+		"CustomerID":         host.CustomerID,
+		"CustomerName":       s.customerName(host.CustomerID),
+		"AgentVersion":       host.AgentVersion,
+		"CreatedAt":          host.CreatedAt,
+		"Status":             status,
+		"StatusLabel":        hostStatusLabel(status),
+		"StatusClass":        hostStatusClass(status),
+		"LastReportAt":       host.LastReportAt,
+		"HasReport":          host.LastReportAt != nil,
+		"RecoveryMode":       host.InRecoveryMode(time.Now()),
+		"RecoveryUntil":      host.RecoveryModeUntil,
+		"DesiredGeneration":  host.DesiredGeneration,
+		"Vitals":             vitals,
+		"Guests":             guests,
+		"GuestRunning":       guestRunning,
+		"GuestTotal":         len(guests),
+		"StorageTargets":     storageTargets,
+		"Capabilities":       capabilities,
+		"NeedsDRMigration":   capabilitiesNeedDRMigration(capabilities),
+		"DRPresent":          drBundle != nil,
+		"EscrowPresent":      escrow != nil,
 		// v0.60.0 Part B: retained superseded escrow blobs (data-first — old passphrases stay
 		// R-recoverable). Operator-only surface.
 		"SupersededEscrowCount": func() int { n, _ := s.store.CountSupersededEscrow(host.HostID); return n }(),

hub/internal/web/templates/configuration.html

@@ -180,6 +180,11 @@
                 <label style="font-size: 0.9em; color: #cbd5e1;">Min agent</label>
                 <input type="text" name="min_agent" value="{{.Artifacts.MinAgent}}" placeholder="e.g. 0.81.0 (blank = uncoupled)" style="padding: 0.3em 0.5em;">
                 <span style="font-size: 0.8em; color: #94a6bf;">The golden's controller CHANGELOG <code>MinAgent:</code>. The hub HOLDS the floor for any box whose agent is below this — blank = uncoupled release, no gating.</span>
+            </div>
+            <div style="margin-bottom: 0.75em;">
+                <label style="display:inline-block; min-width: 12em;">PBS wrapper sha256</label>
+                <input type="text" name="wrapper_sha256" value="{{.Artifacts.WrapperSHA256}}" placeholder="64 hex chars (blank = not vouched)" style="padding: 0.3em 0.5em; width: 34em;">
+                <span style="font-size: 0.8em; color: #94a6bf;">sha256 of <code>configs/felhom-pbs-apply</code> (R-50b). Unlike the agent and golden, this root-owned wrapper is installed from <code>raw/branch/main</code> — unversioned and unpinned. Recording it here does not fix the channel; it makes host drift <em>visible</em>: agents report the installed file's hash and a mismatch is surfaced on the host.</span>
                 <span></span><span></span>
                 <button class="btn btn-sm" type="submit" style="justify-self: start;">Save artifact manifest</button>
             </form>

hub/internal/web/templates/host_detail_body.html

@@ -22,6 +22,17 @@
                     <span class="label">Agent Version</span>
                     <span class="value">{{if .AgentVersion}}<code>{{.AgentVersion}}</code>{{else}}—{{end}}</span>
                 </div>
+                {{if .WrapperDrift}}
+                <div class="info-item">
+                    <span class="label">PBS wrapper</span>
+                    {{if eq .WrapperDrift "ok"}}
+                    <span class="value">matches vouched <code>{{slice .ReportedWrapperSHA 0 12}}…</code></span>
+                    {{else}}
+                    <span class="value" style="color: var(--yellow)">DRIFT — installed <code>{{slice .ReportedWrapperSHA 0 12}}…</code>, vouched <code>{{slice .VouchedWrapperSHA 0 12}}…</code><br>
+                        <span style="font-size:.85em">/usr/local/sbin/felhom-pbs-apply differs from the manifest. It is delivered unversioned from <code>main</code> (R-50b), so this host may be running privileged code from a different commit.</span></span>
+                    {{end}}
+                </div>
+                {{end}}
                 <div class="info-item">
                     <span class="label">Enrolled</span>
                     <span class="value">{{timeAgo .CreatedAt}}</span>

hub/internal/web/wrapper_drift_test.go

@@ -0,0 +1,54 @@
+package web
+
+import "testing"
+
+// R-50b(a) — wrapper drift must be VISIBLE, and "unknown" must never read as "mismatch".
+//
+// The wrapper is root-owned, 0755, and installed from raw/branch/main: unversioned, unpinned, and
+// absent from every manifest until v0.68.0. This is the surface that makes drift answerable.
+func TestParseReportedWrapperSHA(t *testing.T) {
+	cases := []struct {
+		name, report, want string
+	}{
+		{"present", `{"host":{"wrapper_sha256":"AABBCC"}}`, "aabbcc"},
+		{"lowercased and trimmed", `{"host":{"wrapper_sha256":"  AaBb  "}}`, "aabb"},
+		{"absent key", `{"host":{"agent_version":"0.91.0"}}`, ""},
+		{"no host stanza", `{"guests":[]}`, ""},
+		{"empty report", ``, ""},
+		{"malformed json", `{nope`, ""},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			if got := parseReportedWrapperSHA(tc.report); got != tc.want {
+				t.Errorf("parseReportedWrapperSHA(%q) = %q, want %q", tc.report, got, tc.want)
+			}
+		})
+	}
+}
+
+// The comparison itself. The load-bearing case is the pair of UNKNOWNS: an un-vouched hub, or an
+// agent below 0.91.0 that reports nothing, must be quiet — lighting every host amber on rollout day
+// is how a warning gets trained into background noise.
+func TestWrapperDriftComparison(t *testing.T) {
+	const vouched = "1111111111111111111111111111111111111111111111111111111111111111"
+	const other = "2222222222222222222222222222222222222222222222222222222222222222"
+
+	cases := []struct {
+		name, reported, vouched, want string
+	}{
+		{"match", vouched, vouched, "ok"},
+		{"match is case-insensitive", "AAAA", "aaaa", "ok"},
+		{"mismatch", other, vouched, "mismatch"},
+		{"agent reports nothing (pre-0.91.0) → quiet", "", vouched, ""},
+		{"hub has vouched nothing → quiet", vouched, "", ""},
+		{"neither side known → quiet", "", "", ""},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			got := compareWrapperSHA(tc.reported, tc.vouched)
+			if got != tc.want {
+				t.Errorf("compareWrapperSHA(%q, %q) = %q, want %q", tc.reported, tc.vouched, got, tc.want)
+			}
+		})
+	}
+}