v0.91.0 — the DR tier can no longer be applied and dead at the same time (R-39 + R-50b(a))

Closes the agent half of R-39's fleet fix. Requires hub >=0.68.0 for the re-arm signal;
that hub is safe for 0.90.0 agents (unknown key dropped), so it deploys first.

Three compounding defects let a box report `applied` while every PBS request 401'd:

1. The re-key was INVISIBLE. An ep0 re-issue rotates the secret of an existing token, so
   token_id/fingerprint/datastore/namespace come back byte-identical and the descriptor
   content hash never moved — the converged agent short-circuited and never consumed the
   fresh secret. WirePBSDR.SecretGeneration (field-exact with the hub) is what moves the
   hash now, because descriptorHash marshals this struct.

2. The agent could not READ its own credential. It writes /etc/pve/priv/storage/<id>.pw
   through the root wrapper, but that dir is 0700 root:www-data and the wrapper had no
   read verb — so the target resolver got "permission denied" every cycle, warned, and
   skipped. The one loop that could have caught the 401 was blind BY CONSTRUCTION. Adds a
   narrow `read` verb (+ exactly one sudoers line, + a pbsdr-read capability row): one
   secret to stdout, no network, no mutation, never in argv (sudo logs argv), traversal
   refused by the id grammar, the dir allowlist AND a resolved-path prefix assertion.

3. Nothing probed AUTHENTICATION. pbs.ProbeAuth (GET /version + an ErrUnauthorized
   sentinel) runs on the 15-minute collect path and its verdict becomes a loud
   `auth_failed` the hub escalates to a fresh mint. /version needs no datastore, namespace
   or privilege, so a 401 means the CREDENTIAL is bad; 403 is deliberately NOT treated as
   unauthorized, since re-keying a too-narrow token would mint forever without fixing
   anything. A transport error is UNKNOWN, never a rejection — otherwise every network
   blip burns a credential. Recovery self-clears.

R-50b(a): the report now carries the installed wrapper's sha256 so drift against the
vouched manifest value is answerable. Empty = unknown, never drift.

Three red-proofs, all at the assertion level. Removing SecretGeneration fails the re-arm
test with "consume calls=1, want 2". Swallowing the probe result leaves State:applied
AuthFailed:false — the July-18 shape exactly. Notably, deleting the wrapper's id charset
guard alone does NOT open a traversal hole (readlink + the prefix assertion still catch
it), so the isolating red-proof removes BOTH and shows the out-of-tree secret printed —
the layering is real, and a single-guard red-proof would have passed vacuously.

internal/pbs/client.go

@@ -3,6 +3,7 @@ package pbs
 import (
 	"context"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"io"
 	"net/http"
@@ -193,6 +194,46 @@ func NodeFromUPID(upid string) string {
 	return parts[1]
 }
 
+// ErrUnauthorized is returned by ProbeAuth when PBS REJECTS the credential (HTTP 401).
+//
+// It is a distinct sentinel because a rejected credential and an unreachable server demand opposite
+// responses: 401 is terminal until the credential is replaced (the hub must re-key), while a dial
+// error is transient and must NOT trigger a re-issue — mistaking one for the other would either
+// leave a dead tier green (the R-39 failure) or burn a fresh secret on every network blip.
+var ErrUnauthorized = errors.New("pbs: unauthorized (401) — the token secret is not accepted")
+
+// ProbeAuth asks PBS the cheapest question that requires authentication: GET /version.
+//
+// WHY THIS EXISTS (R-39 leg c). The DR tier could be `applied` and dead at the same time: PVE holds
+// a storage entry, the agent's marker says converged, and every PBS request 401s because the entry
+// is pinned to a superseded credential. Nothing noticed, because the one loop that could — the
+// 15-minute PBS verify loop — could not even READ the credential to test it (the non-root agent
+// writes /etc/pve/priv/storage/<id>.pw through a root wrapper and had no read verb). With the
+// wrapper's `read` verb this probe finally closes that gap, and its result becomes a LOUD
+// `auth_failed` state the hub self-heals instead of a Warn-and-skip.
+//
+// /version is deliberate: it needs no datastore, no namespace and no privileges beyond a valid
+// token, so a 401 here means the CREDENTIAL is bad — not that a datastore is missing or an ACL is
+// too narrow. That distinction is what makes the state safe to auto-remediate.
+func (c *Client) ProbeAuth(ctx context.Context) error {
+	err := c.do(ctx, http.MethodGet, "/version", nil)
+	if err == nil {
+		return nil
+	}
+	if isUnauthorized(err) {
+		return ErrUnauthorized
+	}
+	return err
+}
+
+// isUnauthorized classifies a doBody error as an authentication rejection. doBody formats non-2xx as
+// "... -> HTTP <code>: <body>", so the code is matched on that shape. 403 is deliberately NOT
+// included: a valid token with too narrow an ACL is a permissions problem, and re-keying it would
+// mint credentials forever without fixing anything.
+func isUnauthorized(err error) bool {
+	return err != nil && strings.Contains(err.Error(), "-> HTTP 401")
+}
+
 // post performs a form-encoded POST (PBS mutating ops take form params).
 func (c *Client) post(ctx context.Context, path string, form url.Values, out any) error {
 	return c.doBody(ctx, http.MethodPost, path, strings.NewReader(form.Encode()), "application/x-www-form-urlencoded", out)

internal/pbs/live_reporter.go

@@ -2,6 +2,7 @@ package pbs
 
 import (
 	"context"
+	"errors"
 	"log/slog"
 	"time"
 
@@ -32,8 +33,31 @@ type LiveSnapshotReporter struct {
 	// listSnapshots is the production→PBS seam, overridable in tests so no live PBS is needed.
 	// Default = liveListSnapshots (one Snapshots() GET, converted via Snapshot.ToHub()).
 	listSnapshots func(ctx context.Context, t Target) ([]hub.PBSSnapshot, error)
+
+	// probeAuth is the R-39 credential probe seam (default = (*Client).ProbeAuth). Overridable so the
+	// auth-honesty path is testable with no PBS.
+	probeAuth func(ctx context.Context, t Target) error
+	// authSink receives every probe result. nil = nobody is listening (the probe is then skipped
+	// entirely — no point paying for a request nothing consumes).
+	authSink AuthSink
 }
 
+// AuthSink receives the result of each per-storage credential probe (R-39 leg c).
+//
+// It exists so the DR bridge can turn a 401 into a LOUD `auth_failed` state instead of the Warn-and-
+// skip that made an applied-but-dead tier invisible. Deliberately a plain interface taking a bool
+// rather than the error: the consumer (internal/pbsdr) must not have to import this package just to
+// test a sentinel.
+type AuthSink interface {
+	// NoteAuthResult reports one storage's credential health. unauthorized=true means PBS REJECTED
+	// the credential (401) — terminal until it is replaced. A transport error is unauthorized=false
+	// with a non-empty detail: unknown, not dead, and never a reason to re-key.
+	NoteAuthResult(storageID string, unauthorized bool, detail string)
+}
+
+// SetAuthSink wires the credential-probe consumer. Without it the reporter does not probe at all.
+func (r *LiveSnapshotReporter) SetAuthSink(s AuthSink) { r.authSink = s }
+
 // NewLiveSnapshotReporter builds a live reporter sharing store with the verify loop. A zero timeout
 // falls back to DefaultLiveSnapshotTimeout; a nil logger to slog.Default.
 func NewLiveSnapshotReporter(targets Targets, store *SnapshotStore, timeout time.Duration, log *slog.Logger) *LiveSnapshotReporter {
@@ -49,6 +73,7 @@ func NewLiveSnapshotReporter(targets Targets, store *SnapshotStore, timeout time
 		timeout:       timeout,
 		log:           log,
 		listSnapshots: liveListSnapshots,
+		probeAuth:     func(ctx context.Context, t Target) error { return t.Client.ProbeAuth(ctx) },
 	}
 }
 
@@ -66,6 +91,30 @@ func liveListSnapshots(ctx context.Context, t Target) ([]hub.PBSSnapshot, error)
 	return out, nil
 }
 
+// probeAuthAndReport runs the credential probe for one target and forwards the verdict to the sink.
+// No sink → no probe (nothing would consume it). Never fails the collect: a report must still go out.
+func (r *LiveSnapshotReporter) probeAuthAndReport(ctx context.Context, t Target) {
+	if r.authSink == nil || r.probeAuth == nil {
+		return
+	}
+	err := r.probeAuth(ctx, t)
+	switch {
+	case err == nil:
+		r.authSink.NoteAuthResult(t.StorageID, false, "")
+	case errors.Is(err, ErrUnauthorized):
+		// The one case that is TERMINAL and actionable: the credential is rejected, not the network.
+		r.log.Error("pbs: the DR endpoint REJECTED this box's credential (401) — the storage entry is "+
+			"pinned to a superseded secret and every backup/restore against it will fail",
+			"storage", t.StorageID, "datastore", t.Datastore)
+		r.authSink.NoteAuthResult(t.StorageID, true, "PBS rejected the stored credential (401)")
+	default:
+		// Unreachable/timeout/TLS — UNKNOWN, not dead. Reporting this as unauthorized would re-key a
+		// perfectly good credential on every network blip.
+		r.log.Debug("pbs: credential probe inconclusive (not a rejection)", "storage", t.StorageID, "err", err)
+		r.authSink.NoteAuthResult(t.StorageID, false, err.Error())
+	}
+}
+
 // PBSSnapshots implements hub.PBSReporter with a single bounded, live pass (last-known-good
 // fallback). Non-nil result so it marshals as [].
 func (r *LiveSnapshotReporter) PBSSnapshots(ctx context.Context) []hub.PBSSnapshot {
@@ -81,6 +130,12 @@ func (r *LiveSnapshotReporter) PBSSnapshots(ctx context.Context) []hub.PBSSnapsh
 
 	out := []hub.PBSSnapshot{}
 	for _, t := range targets {
+		// R-39 leg (c): prove the credential BEFORE interpreting anything else about this datastore.
+		// A snapshot list that fails with 401 used to look identical to "PBS is busy" — which is how
+		// an applied-and-dead tier stayed green. The probe runs on this 15-minute collect path (not
+		// the 6 h verify cadence) because that is how fast the hub can react.
+		r.probeAuthAndReport(childCtx, t)
+
 		snaps, err := r.listSnapshots(childCtx, t)
 		if err != nil {
 			// Per-datastore live failure → that datastore's last-known-good (does NOT clobber it).

internal/pbs/verify.go

@@ -16,6 +16,10 @@ const DefaultVerifyCadence = 6 * time.Hour
 type Target struct {
 	Datastore string
 	Client    *Client
+	// StorageID is the PVE storage-entry id this target came from. Carried so an auth failure can
+	// NAME the storage the operator has to fix, and so the pbsdr bridge can match the failure to its
+	// own descriptor (a host may hold several PBS storages).
+	StorageID string
 }
 
 // Targets resolves the current set of PBS datastores to verify (re-derived each cycle from