v0.91.0 — the DR tier can no longer be applied and dead at the same time (R-39 + R-50b(a))

Closes the agent half of R-39's fleet fix. Requires hub >=0.68.0 for the re-arm signal;
that hub is safe for 0.90.0 agents (unknown key dropped), so it deploys first.

Three compounding defects let a box report `applied` while every PBS request 401'd:

1. The re-key was INVISIBLE. An ep0 re-issue rotates the secret of an existing token, so
   token_id/fingerprint/datastore/namespace come back byte-identical and the descriptor
   content hash never moved — the converged agent short-circuited and never consumed the
   fresh secret. WirePBSDR.SecretGeneration (field-exact with the hub) is what moves the
   hash now, because descriptorHash marshals this struct.

2. The agent could not READ its own credential. It writes /etc/pve/priv/storage/<id>.pw
   through the root wrapper, but that dir is 0700 root:www-data and the wrapper had no
   read verb — so the target resolver got "permission denied" every cycle, warned, and
   skipped. The one loop that could have caught the 401 was blind BY CONSTRUCTION. Adds a
   narrow `read` verb (+ exactly one sudoers line, + a pbsdr-read capability row): one
   secret to stdout, no network, no mutation, never in argv (sudo logs argv), traversal
   refused by the id grammar, the dir allowlist AND a resolved-path prefix assertion.

3. Nothing probed AUTHENTICATION. pbs.ProbeAuth (GET /version + an ErrUnauthorized
   sentinel) runs on the 15-minute collect path and its verdict becomes a loud
   `auth_failed` the hub escalates to a fresh mint. /version needs no datastore, namespace
   or privilege, so a 401 means the CREDENTIAL is bad; 403 is deliberately NOT treated as
   unauthorized, since re-keying a too-narrow token would mint forever without fixing
   anything. A transport error is UNKNOWN, never a rejection — otherwise every network
   blip burns a credential. Recovery self-clears.

R-50b(a): the report now carries the installed wrapper's sha256 so drift against the
vouched manifest value is answerable. Empty = unknown, never drift.

Three red-proofs, all at the assertion level. Removing SecretGeneration fails the re-arm
test with "consume calls=1, want 2". Swallowing the probe result leaves State:applied
AuthFailed:false — the July-18 shape exactly. Notably, deleting the wrapper's id charset
guard alone does NOT open a traversal hole (readlink + the prefix assertion still catch
it), so the isolating red-proof removes BOTH and shows the out-of-tree secret printed —
the layering is real, and a single-guard red-proof would have passed vacuously.
This commit is contained in:
2026-07-21 10:12:31 +02:00
parent 8c55ac7fda
commit b2ca63ee9f
14 changed files with 784 additions and 8 deletions
+41
View File
@@ -3,6 +3,7 @@ package pbs
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
@@ -193,6 +194,46 @@ func NodeFromUPID(upid string) string {
return parts[1]
}
// ErrUnauthorized is returned by ProbeAuth when PBS REJECTS the credential (HTTP 401).
//
// It is a distinct sentinel because a rejected credential and an unreachable server demand opposite
// responses: 401 is terminal until the credential is replaced (the hub must re-key), while a dial
// error is transient and must NOT trigger a re-issue — mistaking one for the other would either
// leave a dead tier green (the R-39 failure) or burn a fresh secret on every network blip.
var ErrUnauthorized = errors.New("pbs: unauthorized (401) — the token secret is not accepted")
// ProbeAuth asks PBS the cheapest question that requires authentication: GET /version.
//
// WHY THIS EXISTS (R-39 leg c). The DR tier could be `applied` and dead at the same time: PVE holds
// a storage entry, the agent's marker says converged, and every PBS request 401s because the entry
// is pinned to a superseded credential. Nothing noticed, because the one loop that could — the
// 15-minute PBS verify loop — could not even READ the credential to test it (the non-root agent
// writes /etc/pve/priv/storage/<id>.pw through a root wrapper and had no read verb). With the
// wrapper's `read` verb this probe finally closes that gap, and its result becomes a LOUD
// `auth_failed` state the hub self-heals instead of a Warn-and-skip.
//
// /version is deliberate: it needs no datastore, no namespace and no privileges beyond a valid
// token, so a 401 here means the CREDENTIAL is bad — not that a datastore is missing or an ACL is
// too narrow. That distinction is what makes the state safe to auto-remediate.
func (c *Client) ProbeAuth(ctx context.Context) error {
err := c.do(ctx, http.MethodGet, "/version", nil)
if err == nil {
return nil
}
if isUnauthorized(err) {
return ErrUnauthorized
}
return err
}
// isUnauthorized classifies a doBody error as an authentication rejection. doBody formats non-2xx as
// "... -> HTTP <code>: <body>", so the code is matched on that shape. 403 is deliberately NOT
// included: a valid token with too narrow an ACL is a permissions problem, and re-keying it would
// mint credentials forever without fixing anything.
func isUnauthorized(err error) bool {
return err != nil && strings.Contains(err.Error(), "-> HTTP 401")
}
// post performs a form-encoded POST (PBS mutating ops take form params).
func (c *Client) post(ctx context.Context, path string, form url.Values, out any) error {
return c.doBody(ctx, http.MethodPost, path, strings.NewReader(form.Encode()), "application/x-www-form-urlencoded", out)