v0.91.0 — the DR tier can no longer be applied and dead at the same time (R-39 + R-50b(a))
Closes the agent half of R-39's fleet fix. Requires hub >=0.68.0 for the re-arm signal; that hub is safe for 0.90.0 agents (unknown key dropped), so it deploys first. Three compounding defects let a box report `applied` while every PBS request 401'd: 1. The re-key was INVISIBLE. An ep0 re-issue rotates the secret of an existing token, so token_id/fingerprint/datastore/namespace come back byte-identical and the descriptor content hash never moved — the converged agent short-circuited and never consumed the fresh secret. WirePBSDR.SecretGeneration (field-exact with the hub) is what moves the hash now, because descriptorHash marshals this struct. 2. The agent could not READ its own credential. It writes /etc/pve/priv/storage/<id>.pw through the root wrapper, but that dir is 0700 root:www-data and the wrapper had no read verb — so the target resolver got "permission denied" every cycle, warned, and skipped. The one loop that could have caught the 401 was blind BY CONSTRUCTION. Adds a narrow `read` verb (+ exactly one sudoers line, + a pbsdr-read capability row): one secret to stdout, no network, no mutation, never in argv (sudo logs argv), traversal refused by the id grammar, the dir allowlist AND a resolved-path prefix assertion. 3. Nothing probed AUTHENTICATION. pbs.ProbeAuth (GET /version + an ErrUnauthorized sentinel) runs on the 15-minute collect path and its verdict becomes a loud `auth_failed` the hub escalates to a fresh mint. /version needs no datastore, namespace or privilege, so a 401 means the CREDENTIAL is bad; 403 is deliberately NOT treated as unauthorized, since re-keying a too-narrow token would mint forever without fixing anything. A transport error is UNKNOWN, never a rejection — otherwise every network blip burns a credential. Recovery self-clears. R-50b(a): the report now carries the installed wrapper's sha256 so drift against the vouched manifest value is answerable. Empty = unknown, never drift. Three red-proofs, all at the assertion level. Removing SecretGeneration fails the re-arm test with "consume calls=1, want 2". Swallowing the probe result leaves State:applied AuthFailed:false — the July-18 shape exactly. Notably, deleting the wrapper's id charset guard alone does NOT open a traversal hole (readlink + the prefix assertion still catch it), so the isolating red-proof removes BOTH and shows the out-of-tree secret printed — the layering is real, and a single-guard red-proof would have passed vacuously.
This commit is contained in:
@@ -141,6 +141,56 @@ func (m *Manager) DRConfigured() bool {
|
||||
return m.loadMarker() != nil
|
||||
}
|
||||
|
||||
// NoteAuthResult implements pbs.AuthSink (R-39 leg c): the credential probe's verdict for one
|
||||
// storage, turned into the DR bridge's reported state.
|
||||
//
|
||||
// This is the leg that makes `applied` mean something. Until v0.91.0 the agent could not read the
|
||||
// credential it had written (root-only path, no wrapper read verb), so a tier pinned to a superseded
|
||||
// secret reported `applied` forever while every PBS request 401'd — and the hub, seeing `applied`,
|
||||
// had no reason to re-key. Now a rejection becomes a LOUD `auth_failed` that pbsdrheal escalates to
|
||||
// a fresh mint; the fresh mint advances the secret generation; the descriptor hash moves; and Apply
|
||||
// finally re-consumes.
|
||||
//
|
||||
// Rules that keep it safe:
|
||||
// - Only a REJECTION (401) sets the state. An unreachable PBS is UNKNOWN and must never re-key.
|
||||
// - Only the storage this box's descriptor actually names is considered; a host may carry other
|
||||
// PBS entries that are none of the DR tier's business.
|
||||
// - Recovery is self-clearing: a subsequent successful probe restores the converged state from the
|
||||
// marker, so the operator does not have to acknowledge a fault that fixed itself.
|
||||
func (m *Manager) NoteAuthResult(storageID string, unauthorized bool, detail string) {
|
||||
m.mu.Lock()
|
||||
st := m.status
|
||||
m.mu.Unlock()
|
||||
// No descriptor seen yet, or this is not our storage → not our business.
|
||||
if st == nil || st.StorageID == "" || storageID == "" || st.StorageID != storageID {
|
||||
return
|
||||
}
|
||||
if unauthorized {
|
||||
if st.State == "auth_failed" {
|
||||
return // already loud; do not churn the report
|
||||
}
|
||||
m.logger.Error("pbsdr: the DR endpoint REJECTED this box's credential — the tier is applied and DEAD",
|
||||
"storage_id", storageID, "previous_state", st.State)
|
||||
m.setStatus(&hub.PBSDRStatus{
|
||||
State: "auth_failed", StorageID: st.StorageID, Namespace: st.Namespace, AppliedAt: st.AppliedAt,
|
||||
AuthFailed: true,
|
||||
Message: detail + " — awaiting fresh credentials from the hub (automatic)",
|
||||
})
|
||||
return
|
||||
}
|
||||
// A clean probe clears a previously-loud auth failure by restoring the converged marker state.
|
||||
if st.State == "auth_failed" && detail == "" {
|
||||
mk := m.loadMarker()
|
||||
restored := "applied"
|
||||
appliedAt := st.AppliedAt
|
||||
if mk != nil {
|
||||
restored, appliedAt = mk.State, mk.AppliedAt
|
||||
}
|
||||
m.logger.Info("pbsdr: credential accepted again — clearing auth_failed", "storage_id", storageID, "state", restored)
|
||||
m.setStatus(&hub.PBSDRStatus{State: restored, StorageID: st.StorageID, Namespace: st.Namespace, AppliedAt: appliedAt})
|
||||
}
|
||||
}
|
||||
|
||||
// descriptorHash is the idempotency key: sha256 of the canonical (struct-ordered) JSON.
|
||||
func descriptorHash(b *hub.WirePBSDR) string {
|
||||
j, _ := json.Marshal(b)
|
||||
|
||||
Reference in New Issue
Block a user