v0.91.0 — the DR tier can no longer be applied and dead at the same time (R-39 + R-50b(a))

Closes the agent half of R-39's fleet fix. Requires hub >=0.68.0 for the re-arm signal;
that hub is safe for 0.90.0 agents (unknown key dropped), so it deploys first.

Three compounding defects let a box report `applied` while every PBS request 401'd:

1. The re-key was INVISIBLE. An ep0 re-issue rotates the secret of an existing token, so
   token_id/fingerprint/datastore/namespace come back byte-identical and the descriptor
   content hash never moved — the converged agent short-circuited and never consumed the
   fresh secret. WirePBSDR.SecretGeneration (field-exact with the hub) is what moves the
   hash now, because descriptorHash marshals this struct.

2. The agent could not READ its own credential. It writes /etc/pve/priv/storage/<id>.pw
   through the root wrapper, but that dir is 0700 root:www-data and the wrapper had no
   read verb — so the target resolver got "permission denied" every cycle, warned, and
   skipped. The one loop that could have caught the 401 was blind BY CONSTRUCTION. Adds a
   narrow `read` verb (+ exactly one sudoers line, + a pbsdr-read capability row): one
   secret to stdout, no network, no mutation, never in argv (sudo logs argv), traversal
   refused by the id grammar, the dir allowlist AND a resolved-path prefix assertion.

3. Nothing probed AUTHENTICATION. pbs.ProbeAuth (GET /version + an ErrUnauthorized
   sentinel) runs on the 15-minute collect path and its verdict becomes a loud
   `auth_failed` the hub escalates to a fresh mint. /version needs no datastore, namespace
   or privilege, so a 401 means the CREDENTIAL is bad; 403 is deliberately NOT treated as
   unauthorized, since re-keying a too-narrow token would mint forever without fixing
   anything. A transport error is UNKNOWN, never a rejection — otherwise every network
   blip burns a credential. Recovery self-clears.

R-50b(a): the report now carries the installed wrapper's sha256 so drift against the
vouched manifest value is answerable. Empty = unknown, never drift.

Three red-proofs, all at the assertion level. Removing SecretGeneration fails the re-arm
test with "consume calls=1, want 2". Swallowing the probe result leaves State:applied
AuthFailed:false — the July-18 shape exactly. Notably, deleting the wrapper's id charset
guard alone does NOT open a traversal hole (readlink + the prefix assertion still catch
it), so the isolating red-proof removes BOTH and shows the out-of-tree secret printed —
the layering is real, and a single-guard red-proof would have passed vacuously.
This commit is contained in:
2026-07-21 10:12:31 +02:00
parent 8c55ac7fda
commit b2ca63ee9f
14 changed files with 784 additions and 8 deletions
+4
View File
@@ -161,6 +161,10 @@ var manifest = []Capability{
{"pbsdr-create", "PBS DR storage-entry create (K autogen)", "/usr/local/sbin/felhom-pbs-apply", []string{"create", "felhom-pbs", "10.77.0.1", "felhom-offsite", "ns0", "felhom@pbs!ns0", reprFingerprint, "/etc/pve/priv/storage"}, false, ""},
{"pbsdr-reconcile", "PBS DR storage-entry reconcile (set-only)", "/usr/local/sbin/felhom-pbs-apply", []string{"reconcile", "felhom-pbs", "10.77.0.1", "ns0", "felhom@pbs!ns0", reprFingerprint, "/etc/pve/priv/storage"}, false, ""},
{"pbsdr-grant", "PBS DR storage ACL self-grant", "/usr/local/sbin/felhom-pbs-apply", []string{"grant", "felhom-pbs"}, false, ""},
// R-39 leg (b), v0.91.0: the credential READ path. Its absence is what made the PBS verify loop
// permanently blind to an applied-but-401 tier, so a host missing this verb is DEGRADED in a way
// that matters — it cannot detect the failure this whole tier exists to survive.
{"pbsdr-read", "PBS DR credential read (verify-loop auth probe)", "/usr/local/sbin/felhom-pbs-apply", []string{"read", "felhom-pbs", "/etc/pve/priv/storage"}, false, ""},
// ---- Escrow ceremony (FELHOM_ESCROW, controller-driven, v0.88.0). Critical: the customer
// wizard's whole run path IS this one grant — a dropped line silently breaks every ceremony.
+5 -3
View File
@@ -132,7 +132,9 @@ func TestProbe_GateOffHealthyIsInactive(t *testing.T) {
statuses := p.Probe(context.Background())
// v0.88.0: escrow-ceremony joins the gate EXPLICITLY (non-pbsdr name, GatedBy literal) —
// the ceremony only exists behind the DR tier (no PBS key, no ceremony).
for _, name := range []string{"pbsdr-create", "pbsdr-reconcile", "pbsdr-grant", "escrow-ceremony"} {
// v0.91.0: pbsdr-read (the R-39 credential-read verb) rides the same `pbsdr-` prefix gate — a new
// pbsdr-* op is gated by construction, which is exactly the property this list is here to hold.
for _, name := range []string{"pbsdr-create", "pbsdr-reconcile", "pbsdr-grant", "pbsdr-read", "escrow-ceremony"} {
s := find(statuses, name)
if s.Status != StatusInactive || s.Reason != ReasonInactive {
t.Fatalf("%s = %+v, want inactive/%q", name, s, ReasonInactive)
@@ -147,8 +149,8 @@ func TestProbe_GateOffHealthyIsInactive(t *testing.T) {
if len(degraded) != 0 {
t.Fatalf("inactive leaked into degraded: %+v", degraded)
}
if ok != total-4 {
t.Fatalf("ok=%d total=%d, want exactly the 4 gated ones non-ok", ok, total)
if ok != total-5 {
t.Fatalf("ok=%d total=%d, want exactly the 5 gated ones non-ok", ok, total)
}
}