v0.91.0 — the DR tier can no longer be applied and dead at the same time (R-39 + R-50b(a))
Closes the agent half of R-39's fleet fix. Requires hub >=0.68.0 for the re-arm signal; that hub is safe for 0.90.0 agents (unknown key dropped), so it deploys first. Three compounding defects let a box report `applied` while every PBS request 401'd: 1. The re-key was INVISIBLE. An ep0 re-issue rotates the secret of an existing token, so token_id/fingerprint/datastore/namespace come back byte-identical and the descriptor content hash never moved — the converged agent short-circuited and never consumed the fresh secret. WirePBSDR.SecretGeneration (field-exact with the hub) is what moves the hash now, because descriptorHash marshals this struct. 2. The agent could not READ its own credential. It writes /etc/pve/priv/storage/<id>.pw through the root wrapper, but that dir is 0700 root:www-data and the wrapper had no read verb — so the target resolver got "permission denied" every cycle, warned, and skipped. The one loop that could have caught the 401 was blind BY CONSTRUCTION. Adds a narrow `read` verb (+ exactly one sudoers line, + a pbsdr-read capability row): one secret to stdout, no network, no mutation, never in argv (sudo logs argv), traversal refused by the id grammar, the dir allowlist AND a resolved-path prefix assertion. 3. Nothing probed AUTHENTICATION. pbs.ProbeAuth (GET /version + an ErrUnauthorized sentinel) runs on the 15-minute collect path and its verdict becomes a loud `auth_failed` the hub escalates to a fresh mint. /version needs no datastore, namespace or privilege, so a 401 means the CREDENTIAL is bad; 403 is deliberately NOT treated as unauthorized, since re-keying a too-narrow token would mint forever without fixing anything. A transport error is UNKNOWN, never a rejection — otherwise every network blip burns a credential. Recovery self-clears. R-50b(a): the report now carries the installed wrapper's sha256 so drift against the vouched manifest value is answerable. Empty = unknown, never drift. Three red-proofs, all at the assertion level. Removing SecretGeneration fails the re-arm test with "consume calls=1, want 2". Swallowing the probe result leaves State:applied AuthFailed:false — the July-18 shape exactly. Notably, deleting the wrapper's id charset guard alone does NOT open a traversal hole (readlink + the prefix assertion still catch it), so the isolating red-proof removes BOTH and shows the out-of-tree secret printed — the layering is real, and a single-guard red-proof would have passed vacuously.
This commit is contained in:
@@ -3,6 +3,7 @@ package pbs
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
@@ -193,6 +194,46 @@ func NodeFromUPID(upid string) string {
|
||||
return parts[1]
|
||||
}
|
||||
|
||||
// ErrUnauthorized is returned by ProbeAuth when PBS REJECTS the credential (HTTP 401).
|
||||
//
|
||||
// It is a distinct sentinel because a rejected credential and an unreachable server demand opposite
|
||||
// responses: 401 is terminal until the credential is replaced (the hub must re-key), while a dial
|
||||
// error is transient and must NOT trigger a re-issue — mistaking one for the other would either
|
||||
// leave a dead tier green (the R-39 failure) or burn a fresh secret on every network blip.
|
||||
var ErrUnauthorized = errors.New("pbs: unauthorized (401) — the token secret is not accepted")
|
||||
|
||||
// ProbeAuth asks PBS the cheapest question that requires authentication: GET /version.
|
||||
//
|
||||
// WHY THIS EXISTS (R-39 leg c). The DR tier could be `applied` and dead at the same time: PVE holds
|
||||
// a storage entry, the agent's marker says converged, and every PBS request 401s because the entry
|
||||
// is pinned to a superseded credential. Nothing noticed, because the one loop that could — the
|
||||
// 15-minute PBS verify loop — could not even READ the credential to test it (the non-root agent
|
||||
// writes /etc/pve/priv/storage/<id>.pw through a root wrapper and had no read verb). With the
|
||||
// wrapper's `read` verb this probe finally closes that gap, and its result becomes a LOUD
|
||||
// `auth_failed` state the hub self-heals instead of a Warn-and-skip.
|
||||
//
|
||||
// /version is deliberate: it needs no datastore, no namespace and no privileges beyond a valid
|
||||
// token, so a 401 here means the CREDENTIAL is bad — not that a datastore is missing or an ACL is
|
||||
// too narrow. That distinction is what makes the state safe to auto-remediate.
|
||||
func (c *Client) ProbeAuth(ctx context.Context) error {
|
||||
err := c.do(ctx, http.MethodGet, "/version", nil)
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
if isUnauthorized(err) {
|
||||
return ErrUnauthorized
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// isUnauthorized classifies a doBody error as an authentication rejection. doBody formats non-2xx as
|
||||
// "... -> HTTP <code>: <body>", so the code is matched on that shape. 403 is deliberately NOT
|
||||
// included: a valid token with too narrow an ACL is a permissions problem, and re-keying it would
|
||||
// mint credentials forever without fixing anything.
|
||||
func isUnauthorized(err error) bool {
|
||||
return err != nil && strings.Contains(err.Error(), "-> HTTP 401")
|
||||
}
|
||||
|
||||
// post performs a form-encoded POST (PBS mutating ops take form params).
|
||||
func (c *Client) post(ctx context.Context, path string, form url.Values, out any) error {
|
||||
return c.doBody(ctx, http.MethodPost, path, strings.NewReader(form.Encode()), "application/x-www-form-urlencoded", out)
|
||||
|
||||
Reference in New Issue
Block a user