v0.91.0 — the DR tier can no longer be applied and dead at the same time (R-39 + R-50b(a))

Closes the agent half of R-39's fleet fix. Requires hub >=0.68.0 for the re-arm signal;
that hub is safe for 0.90.0 agents (unknown key dropped), so it deploys first.

Three compounding defects let a box report `applied` while every PBS request 401'd:

1. The re-key was INVISIBLE. An ep0 re-issue rotates the secret of an existing token, so
   token_id/fingerprint/datastore/namespace come back byte-identical and the descriptor
   content hash never moved — the converged agent short-circuited and never consumed the
   fresh secret. WirePBSDR.SecretGeneration (field-exact with the hub) is what moves the
   hash now, because descriptorHash marshals this struct.

2. The agent could not READ its own credential. It writes /etc/pve/priv/storage/<id>.pw
   through the root wrapper, but that dir is 0700 root:www-data and the wrapper had no
   read verb — so the target resolver got "permission denied" every cycle, warned, and
   skipped. The one loop that could have caught the 401 was blind BY CONSTRUCTION. Adds a
   narrow `read` verb (+ exactly one sudoers line, + a pbsdr-read capability row): one
   secret to stdout, no network, no mutation, never in argv (sudo logs argv), traversal
   refused by the id grammar, the dir allowlist AND a resolved-path prefix assertion.

3. Nothing probed AUTHENTICATION. pbs.ProbeAuth (GET /version + an ErrUnauthorized
   sentinel) runs on the 15-minute collect path and its verdict becomes a loud
   `auth_failed` the hub escalates to a fresh mint. /version needs no datastore, namespace
   or privilege, so a 401 means the CREDENTIAL is bad; 403 is deliberately NOT treated as
   unauthorized, since re-keying a too-narrow token would mint forever without fixing
   anything. A transport error is UNKNOWN, never a rejection — otherwise every network
   blip burns a credential. Recovery self-clears.

R-50b(a): the report now carries the installed wrapper's sha256 so drift against the
vouched manifest value is answerable. Empty = unknown, never drift.

Three red-proofs, all at the assertion level. Removing SecretGeneration fails the re-arm
test with "consume calls=1, want 2". Swallowing the probe result leaves State:applied
AuthFailed:false — the July-18 shape exactly. Notably, deleting the wrapper's id charset
guard alone does NOT open a traversal hole (readlink + the prefix assertion still catch
it), so the isolating red-proof removes BOTH and shows the out-of-tree secret printed —
the layering is real, and a single-guard red-proof would have passed vacuously.
This commit is contained in:
2026-07-21 10:12:31 +02:00
parent 8c55ac7fda
commit b2ca63ee9f
14 changed files with 784 additions and 8 deletions
+50
View File
@@ -141,6 +141,56 @@ func (m *Manager) DRConfigured() bool {
return m.loadMarker() != nil
}
// NoteAuthResult implements pbs.AuthSink (R-39 leg c): the credential probe's verdict for one
// storage, turned into the DR bridge's reported state.
//
// This is the leg that makes `applied` mean something. Until v0.91.0 the agent could not read the
// credential it had written (root-only path, no wrapper read verb), so a tier pinned to a superseded
// secret reported `applied` forever while every PBS request 401'd — and the hub, seeing `applied`,
// had no reason to re-key. Now a rejection becomes a LOUD `auth_failed` that pbsdrheal escalates to
// a fresh mint; the fresh mint advances the secret generation; the descriptor hash moves; and Apply
// finally re-consumes.
//
// Rules that keep it safe:
// - Only a REJECTION (401) sets the state. An unreachable PBS is UNKNOWN and must never re-key.
// - Only the storage this box's descriptor actually names is considered; a host may carry other
// PBS entries that are none of the DR tier's business.
// - Recovery is self-clearing: a subsequent successful probe restores the converged state from the
// marker, so the operator does not have to acknowledge a fault that fixed itself.
func (m *Manager) NoteAuthResult(storageID string, unauthorized bool, detail string) {
m.mu.Lock()
st := m.status
m.mu.Unlock()
// No descriptor seen yet, or this is not our storage → not our business.
if st == nil || st.StorageID == "" || storageID == "" || st.StorageID != storageID {
return
}
if unauthorized {
if st.State == "auth_failed" {
return // already loud; do not churn the report
}
m.logger.Error("pbsdr: the DR endpoint REJECTED this box's credential — the tier is applied and DEAD",
"storage_id", storageID, "previous_state", st.State)
m.setStatus(&hub.PBSDRStatus{
State: "auth_failed", StorageID: st.StorageID, Namespace: st.Namespace, AppliedAt: st.AppliedAt,
AuthFailed: true,
Message: detail + " — awaiting fresh credentials from the hub (automatic)",
})
return
}
// A clean probe clears a previously-loud auth failure by restoring the converged marker state.
if st.State == "auth_failed" && detail == "" {
mk := m.loadMarker()
restored := "applied"
appliedAt := st.AppliedAt
if mk != nil {
restored, appliedAt = mk.State, mk.AppliedAt
}
m.logger.Info("pbsdr: credential accepted again — clearing auth_failed", "storage_id", storageID, "state", restored)
m.setStatus(&hub.PBSDRStatus{State: restored, StorageID: st.StorageID, Namespace: st.Namespace, AppliedAt: appliedAt})
}
}
// descriptorHash is the idempotency key: sha256 of the canonical (struct-ordered) JSON.
func descriptorHash(b *hub.WirePBSDR) string {
j, _ := json.Marshal(b)