admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

hub v0.68.0 — auth_failed self-heal, consumed_at honesty gauge, wrapper drift (R-39 + R-50b(a))

Browse Source 
Completes the hub half of R-39's fleet fix on top of the generation core (c484aa2).

pbsdrheal gains an auth_failed TRIGGER — a new trigger in the existing machine, not a
new machine. A box whose credential PBS rejects escalates to a fresh mint, never a
re-stage (which would re-feed the secret PBS just rejected), through the EXISTING damper:
a 401 flap must not become a secret-minting chain. With the generation stamp this closes
the loop end to end — agent proves the 401, hub re-keys, generation advances, descriptor
hash moves, agent re-consumes.

consumed_at honesty gauge: a staged secret still unconsumed past a 15-minute grace while
the box reports `applied` is surfaced with its own event. That is the exact 2026-07-18
fingerprint and a disagreement no single tier can see alone. Deliberately a SURFACE, not
a heal — auto-re-issuing on it would mint a second secret on top of an unconsumed one,
which is the mint/consume race R-39(a) already recorded. One event per distinct report,
and an honestly-stuck box does not double-report (its unconsumed secret is the symptom
being healed, not a contradiction).

R-50b(a): ArtifactManifest.WrapperSHA256 + operator field + host-page drift surface. The
PBS wrapper is root-owned 0755 and the pinned sudoers vector, yet installed unversioned
from raw/branch/main and absent from every manifest. Agents >=0.91.0 report the installed
hash; a mismatch is surfaced. An unknown on EITHER side reads as quiet, never as drift —
lighting every host amber on rollout day is how a warning becomes background noise. The
delivery channel itself stays R-50b(b)/(c).

Compatibility unchanged: safe for 0.90.0 agents (unknown JSON key dropped); the re-arm
and auth-honesty guarantees need agent >=0.91.0, so MinAgent moves only after the fleet
has self-updated.

Tests: auth_failed escalate/debounce/recovery-forgets-streak; honesty gauge incl. grace
window, the restage edge (consumed_at deliberately NULLed), consumed-never-alarms, and
honest-stuck-no-double-report; wrapper drift incl. both unknown directions. Red-proof run
at the assertion level: removing the auth_failed arm fails the escalation tests with
reissues=0.
This commit is contained in:
authentik Default Admin
2026-07-21 10:01:35 +02:00
parent c484aa204e
commit 107f74ea3c
10 changed files with 561 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files
hub/internal/pbsdrheal/reconciler.go
+87 -9
View File
@@ -33,8 +33,15 @@ import (
// (the secret is untouched by verify-before-consume), self-heals when the tunnel recovers, and must
// stay LOUD for the operator if it does not — re-staging a secret would not help it.
const (
stateWaitingSecret = "waiting_secret"
stateWaitingSecret = "waiting_secret"
stateConsumedFailed = "consumed_failed"
// stateAuthFailed (R-39, agent >= 0.91.0) — the box HAS a credential, the descriptor is applied,
// and PBS rejects it with 401. Before 0.91.0 this state could not exist: the agent's verify loop
// read the credential file directly as non-root, always failed with "permission denied", and
// skipped — so an applied-and-dead tier was invisible to both tiers. It is healed like
// consumed_failed (escalate to a fresh mint), never by a re-stage: re-staging re-feeds the SAME
// secret PBS just rejected.
stateAuthFailed = "auth_failed"
)
// Audit event types (store.SaveEvent; hub-internal, not gated by allowedEventTypes). Distinct per
@@ -43,8 +50,18 @@ const (
eventRestaged = "pbsdr_selfheal_restaged" // re-armed the stored secret (routine)
eventReissued = "pbsdr_selfheal_reissued" // no stored secret → minted a fresh one
eventConsumedFailed = "pbsdr_selfheal_consumed_failed" // burned secret → minted a fresh one (a real problem was remediated)
eventAuthFailed = "pbsdr_selfheal_auth_failed" // PBS rejected the box's credential (401) → minted a fresh one
eventUnconsumed = "pbsdr_unconsumed_secret" // staged secret never consumed under an `applied` box (surfaced, NOT auto-healed)
)
// unconsumedGrace bounds how long a staged-but-unconsumed secret is NORMAL before it is a lie.
//
// A fresh mint (or a deliberate re-stage) is consumed on the agent's next tick — well inside one
// ~15-minute report cycle. Beyond this window, an unconsumed secret sitting under a box that reports
// `applied` is the exact fingerprint of the 2026-07-18 N100 failure: hub minted, agent
// short-circuited, both tiers green, box serving a revoked credential.
const unconsumedGrace = 15 * time.Minute
// Actions is the mutation seam — fakes in tests count calls without SSH/ep0. Restage flips a stored
// secret's consumed flag (returns restaged=false when NO row exists → the caller escalates). Reissue
// mints a fresh ep0 token + stores a fresh consume-once secret + bumps the descriptor.
@@ -71,7 +88,9 @@ func (a storeActions) Reissue(ctx context.Context, customerID string) error {
}
// NewActions builds the production mutation seam.
func NewActions(st *store.Store, reissuer Reissuer) Actions { return storeActions{st: st, reissuer: reissuer} }
func NewActions(st *store.Store, reissuer Reissuer) Actions {
return storeActions{st: st, reissuer: reissuer}
}
// debounceState tracks, per host, the last DISTINCT report observed and how many consecutive
// distinct reports it has held the current stuck state — so a fresh box that briefly shows
@@ -86,15 +105,18 @@ type debounceState struct {
// Reconciler re-arms stuck PBS-DR hosts. DECLARATIVE + IDEMPOTENT: a tick over a converged fleet
// writes nothing (Scenario C). It reads the hub DB (the source of truth) — never the box.
type Reconciler struct {
store *store.Store
act Actions
interval time.Duration
debounceReports int // distinct stuck reports required before healing (default 2)
onlyHost string // "" = whole fleet; non-empty restricts the work set to one host (supervised rollout)
trigger chan struct{}
logger *log.Logger
store *store.Store
act Actions
interval time.Duration
debounceReports int // distinct stuck reports required before healing (default 2)
onlyHost string // "" = whole fleet; non-empty restricts the work set to one host (supervised rollout)
trigger chan struct{}
logger *log.Logger
deb map[string]debounceState
// unconsumedSeen remembers the last report id already surfaced per host (Scenario F), so a
// sustained disagreement produces one event per fresh report rather than one per tick.
unconsumedSeen map[string]int64
}
// NewReconciler builds the reconciler. interval defaults to 5m, debounceReports to 2.
@@ -110,6 +132,7 @@ func NewReconciler(st *store.Store, act Actions, logger *log.Logger) *Reconciler
trigger: make(chan struct{}, 1),
logger: logger,
deb: map[string]debounceState{},
unconsumedSeen: map[string]int64{},
}
}
@@ -161,6 +184,14 @@ func (r *Reconciler) reconcileOnce(ctx context.Context) {
delete(r.deb, row.HostID)
continue
}
// R-39 consumed_at HONESTY (Scenario F). Deliberately a SURFACE, not a heal: the remediation
// for a genuinely stuck box is the auth_failed / waiting_secret / consumed_failed machinery
// above, driven by what the BOX reports. This check catches the disagreement itself — the hub
// staged a credential the box never took, while the box claims to be applied — which is a
// state no single tier can detect alone. Auto-re-issuing on it would mint a second secret on
// top of an unconsumed one: exactly the mint/consume race R-39(a) already recorded.
r.checkUnconsumed(row)
switch row.ReportedState {
case stateWaitingSecret:
if r.confirm(row) {
@@ -170,6 +201,10 @@ func (r *Reconciler) reconcileOnce(ctx context.Context) {
if r.confirm(row) {
r.healConsumedFailed(ctx, row)
}
case stateAuthFailed:
if r.confirm(row) {
r.healAuthFailed(ctx, row)
}
default:
// applied | adopted | disabled | verify_failed | "" (no report) | anything else → no-op.
delete(r.deb, row.HostID)
@@ -183,6 +218,31 @@ func (r *Reconciler) reconcileOnce(ctx context.Context) {
}
}
// checkUnconsumed surfaces the applied-but-never-consumed disagreement (Scenario F). One event per
// distinct report, so a sustained state does not spam the audit log every tick.
func (r *Reconciler) checkUnconsumed(row store.PBSDRHealRow) {
if row.SecretUnconsumedFor <= unconsumedGrace {
return // no staged secret, already consumed, or still inside the normal pickup window
}
// Only meaningful when the box claims to be converged. A box that honestly reports
// waiting_secret / consumed_failed / auth_failed is already being healed above and its unconsumed
// secret is the SYMPTOM being fixed, not a contradiction.
if row.ReportedState != "applied" && row.ReportedState != "adopted" {
return
}
if st, ok := r.unconsumedSeen[row.HostID]; ok && st == row.ReportID {
return // already surfaced for this exact report
}
if r.unconsumedSeen == nil {
r.unconsumedSeen = map[string]int64{}
}
r.unconsumedSeen[row.HostID] = row.ReportID
r.logger.Printf("[WARN] pbsdrheal: host %s (customer %s) reports pbs_dr=%s while a staged one-time secret has been UNCONSUMED for %s — the box is not using the credential the hub issued (R-39 fingerprint)",
row.HostID, row.CustomerID, row.ReportedState, row.SecretUnconsumedFor.Round(time.Minute))
r.event(row.CustomerID, eventUnconsumed, "warning",
"PBS-DR honesty check: the box reports the DR tier as applied, but a one-time credential issued by the hub has never been consumed. The tier may be authenticating with a superseded credential.")
}
// confirm advances the per-host debounce and reports whether the stuck state has held across
// >= debounceReports DISTINCT reports. A re-observed same report (same reportID) never advances the
// streak — the debounce counts fresh evidence, not reconciler ticks.
@@ -235,6 +295,24 @@ func (r *Reconciler) healConsumedFailed(ctx context.Context, row store.PBSDRHeal
}
}
// healAuthFailed escalates a box whose credential PBS rejects (401) to a fresh mint. This is the leg
// that closes the R-39 loop end to end: the agent now PROVES the credential is dead instead of
// silently skipping, the hub re-keys, the fresh mint advances the secret generation, the descriptor
// hash moves, and the agent finally re-consumes (Scenario A). Before this, an applied-and-401 tier
// stayed green forever and would have surfaced first at a real restore.
//
// A re-stage is deliberately NOT attempted: the stored secret IS the one PBS just rejected, so
// re-arming it would burn a tick and change nothing. Damping is the SHARED confirm() — a 401 flap
// must not turn into a secret-minting chain.
func (r *Reconciler) healAuthFailed(ctx context.Context, row store.PBSDRHealRow) {
r.logger.Printf("[WARN] pbsdrheal: host %s (customer %s) reports auth_failed (PBS rejects its credential) — escalating to Re-issue", row.HostID, row.CustomerID)
if r.reissue(ctx, row) {
r.event(row.CustomerID, eventAuthFailed, "warning",
"PBS-DR self-heal: a box reported auth_failed (the DR endpoint rejected its stored credential); re-issued fresh endpoint credentials so the agent can re-consume and converge.")
r.resetAfterHeal(row)
}
}
// reissue runs the escalation; returns true on success (the caller then records the audit event).
func (r *Reconciler) reissue(ctx context.Context, row store.PBSDRHealRow) bool {
if err := r.act.Reissue(ctx, row.CustomerID); err != nil {