admin/felhom-controller
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

F-CRIT-1 + F-A1: one alarm that never fired, one that fired wrongly (v0.179.0)

Browse Source 
F-CRIT-1 — an app that failed to restart after a quiesce never alarmed, for two
independent reasons, either of which alone kept it dead:
  1. restartAll returned nothing, so the failure was logged and dropped and no
     caller could learn a customer's app had not come back. It now returns the
     stacks that failed; both call sites record the outcome.
  2. classifyRunStates whitelists StateStopped on invariant I1 ('StateStopped
     means the user stopped it'). The quiesce loop stops stacks by the same
     compose-down path, so a failed restart is also StateStopped and was
     whitelisted into silence. Loop.FailedRestarts() is now the only thing that
     lifts the whitelist, so genuine user stops stay silent (v0.164.0 pinned).

F-A1 — HTTP 409 is the agent's single-flight gate refusing while a restore-test
holds it, not a failure. agentapi now returns a typed *StatusError on POST, the
adapter maps 409 -> quiesce.ErrTierBusy, and the loop defers: no breaker, no
event, no operator email, tier stays DUE.

Two traps avoided. Silence: contention outliving contentionAlarmAfter (3h, set
by the agent's own 120m PBS restore-test ceiling) raises its own BLOCKED signal.
App thrash: removing the failure treatment also removes the breaker's deferral,
so a contended tier is dropped BEFORE anything stops (contentionRetryAfter 15m,
against a 12m01s longest observed restore-test).

Three comments corrected; the invariant rule added to both CLAUDE.md copies.
Six red-proofs, all observed failing.
This commit is contained in:
authentik Default Admin
2026-07-28 08:50:11 +02:00
parent 8f46495426
commit 079265ad8e
10 changed files with 884 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files
controller/internal/quiesce/quiesce.go
+84 -6
View File
@@ -105,6 +105,10 @@ type Loop struct {
// breaker (R-88) defers the QUIESCE for a tier whose backups keep failing, so a broken target
// cannot stop the customer's apps every 5 minutes forever. Scheduled path only — see breaker.go.
breaker *failureBreaker
// contention (F-A1) tracks per-tier HTTP 409 runs. A refusal is NOT a failure, so it must never
// touch the breaker — but it still has to defer the quiesce (else the apps are stopped every
// poll for the whole restore-test) and it must alarm if it never ends. See contention.go.
contention *contentionTracker
// tierNotify (R-97a) reports a tier's backup outcome to the hub. nil = not wired (pre-provisioning).
// Init-only: set once at startup via SetTierNotifier, before Run.
tierNotify TierNotifier
@@ -112,6 +116,9 @@ type Loop struct {
// SuppressedStacks so an app WE stopped is not reported to the customer as broken.
suppressMu sync.Mutex
suppressed map[string]time.Time
// restartFailed (F-CRIT-1) is the set of stacks this loop stopped and could NOT restart. Guarded
// by suppressMu — same concern, same lock. See suppress.go.
restartFailed map[string]struct{}
}
// SetTierNotifier wires the hub-event seam. INIT-ONLY — call once at startup, before Run.
@@ -143,7 +150,8 @@ func New(o Options) *Loop {
poll: o.Poll, statusPoll: o.StatusPoll, maxQuiesce: o.MaxQuiesce,
logger: o.Logger, now: time.Now,
windowStartFn: o.WindowStartFn, cadence: o.Cadence,
breaker: newFailureBreaker(),
breaker: newFailureBreaker(),
contention: newContentionTracker(),
}
}
@@ -157,7 +165,7 @@ func (l *Loop) Recover() {
}
l.logger.Printf("[WARN] [quiesce] crash recovery: a quiesce was in progress (job %q, %d stack(s) stopped) — restarting them",
m.JobID, len(m.StoppedStacks))
l.restartAll(m.StoppedStacks)
l.noteRestartOutcome(m.StoppedStacks, l.restartAll(m.StoppedStacks))
if err := l.clearMarker(); err != nil {
l.logger.Printf("[ERROR] [quiesce] crash recovery: clear marker: %v", err)
}
@@ -256,9 +264,38 @@ func (l *Loop) noteTierFailure(target, label, errMsg string) {
}
}
// noteTierContention records a 409 refusal: log it as contention, defer the tier, and — only if
// contention has run past contentionAlarmAfter — raise it as a fault.
//
// The headline deliberately says BLOCKED, not FAILED. The operator's action for "a concurrent
// operation has held the agent for three hours" is entirely different from "the backup errored",
// and the whole point of F-A1 is that the two stopped being distinguishable.
//
// It reuses the existing `whole_guest_backup_failed` event type rather than inventing one: a new
// type would need the hub's allowedEventTypes + customerMessages pair changed, i.e. a wire change,
// which this fix deliberately does not make.
func (l *Loop) noteTierContention(target, label string) {
elapsed, alarmNow := l.contention.note(target, l.now())
l.logger.Printf("[INFO] [quiesce] tier %s is BUSY — the agent refused the backup because a concurrent heavy operation holds it. This is contention, NOT a failure: the tier stays due and retries in %s (contended for %s)",
label, contentionRetryAfter, elapsed.Round(time.Second))
if !alarmNow {
return
}
l.logger.Printf("[ERROR] [quiesce] tier %s has been BLOCKED by a concurrent operation for over %s — that exceeds the agent's own restore-test ceiling, so the gate is stuck, not busy",
label, contentionAlarmAfter)
if l.tierNotify != nil {
l.tierNotify.BackupFailed(label,
fmt.Sprintf("Whole-guest backup BLOCKED on the %s tier — a concurrent operation has held the agent for over %s and the backup has still not run", label, contentionAlarmAfter),
fmt.Sprintf("contention (HTTP 409) unresolved for %s; exceeds the %s restore-test ceiling", elapsed.Round(time.Second), contentionAlarmAfter))
}
}
// noteTierSuccess clears any backoff. Quiet unless there was something to clear — a line per healthy
// backup would be noise, but a recovery is worth one.
func (l *Loop) noteTierSuccess(target, label string) {
if l.contention.clear(target) {
l.logger.Printf("[INFO] [quiesce] tier %s is no longer contended — the concurrent operation released the agent", label)
}
// recordSuccess's bool is the edge: true only when there WAS a backoff to clear. That is exactly
// the recovery edge — an operator told a tier broke must also be told it healed, and a line (or
// an event) per healthy backup would be noise.
@@ -285,6 +322,15 @@ func (l *Loop) dropBackedOffTiers(tiers []dueTier) []dueTier {
tierLabel(t.target), l.breaker.failuresFor(t.target), until.Format(time.RFC3339))
continue
}
// F-A1: a CONTENDED tier is dropped here too, for the same reason R-88 drops a failing one —
// before any stack is stopped. Without this, removing the (wrong) failure treatment would
// leave the loop re-quiescing every 5 minutes for the whole restore-test, stopping the
// customer's apps each time: worse than the bug being fixed.
if until, busy := l.contention.blocked(t.target, now); busy {
l.logger.Printf("[DEBUG] [quiesce] tier %s is BUSY (a concurrent heavy operation holds the agent) — not quiescing until %s; the tier stays due",
tierLabel(t.target), until.Format(time.RFC3339))
continue
}
kept = append(kept, t)
}
return kept
@@ -423,7 +469,9 @@ func (l *Loop) quiesceAndPollTiers(ctx context.Context, tiers []dueTier) error {
}
unquiesced = true
l.logger.Printf("[INFO] [quiesce] unquiescing (%s): restarting %d stack(s)", reason, len(running))
l.restartAll(running)
// F-CRIT-1: record which stacks came back and which did not, so the app-down classifier can
// tell "the user stopped this" from "we stopped it and could not restart it".
l.noteRestartOutcome(running, l.restartAll(running))
// R-97b: start the grace clock AFTER the restart call, so the window measures time the app
// has actually had to come up rather than time it spent stopped.
l.markUnquiesced(running)
@@ -453,8 +501,20 @@ func (l *Loop) quiesceAndPollTiers(ctx context.Context, tiers []dueTier) error {
last := i == len(tiers)-1
jobID, err := l.startBackupOn(ctx, t.target)
if errors.Is(err, ErrTierBusy) {
// F-A1: the agent's single-flight gate refused (HTTP 409). Nothing is broken — a
// restore-test or another backup holds it. This is CONTENTION, not failure: no breaker,
// no whole_guest_backup_failed, no operator email. The tier stays DUE and retries.
l.noteTierContention(t.target, label)
if last {
unquiesce("last tier is busy — deferring to a later cycle")
}
continue
}
if err != nil {
l.logger.Printf("[ERROR] [quiesce] start backup on tier %s: %v", label, err)
// A REAL error ends any contention run: whatever the gate was doing, this is a fault now.
l.contention.clear(t.target)
l.noteTierFailure(t.target, label, err.Error())
if firstErr == nil {
firstErr = fmt.Errorf("start backup on %s: %w", label, err)
@@ -465,6 +525,9 @@ func (l *Loop) quiesceAndPollTiers(ctx context.Context, tiers []dueTier) error {
}
continue
}
// The tier started, so it is not contended. Clear before the poll so a long backup cannot be
// mistaken for a stuck gate.
l.contention.clear(t.target)
marker.JobID = jobID
_ = l.writeMarker(marker) // best-effort: record the CURRENT tier's job id for diagnosis
l.logger.Printf("[INFO] [quiesce] tier %s: backup job %s started — polling", label, jobID)
@@ -488,8 +551,14 @@ func (l *Loop) quiesceAndPollTiers(ctx context.Context, tiers []dueTier) error {
// offsite snapshot legitimately runs for hours). The app is already back up. We must
// NOT start the next tier: ONE BACKUP AT A TIME PER GUEST (operator ruling
// 2026-07-26) — vzdump still holds the guest lock, so a second start would be refused
// by the agent (409) or fail on the lock and record a spurious failure. The remaining
// tiers simply run on a later poll, once this one has finished.
// by the agent (409) or fail on the lock. The remaining tiers simply run on a later
// poll, once this one has finished.
//
// CORRECTED v0.179.0 (F-A1): this used to say the 409 would "record a spurious
// failure". Until v0.179.0 that was not a hypothetical the comment was guarding
// against — it was what the start path ACTUALLY did, on every 409, on both boxes.
// A 409 is now handled as CONTENTION (see contention.go): no breaker, no operator
// email, the tier stays due. So the outcome this comment feared no longer exists.
l.logger.Printf("[INFO] [quiesce] tier %s still running past the quiesce bound — deferring %d remaining tier(s) to a later cycle",
label, len(tiers)-i-1)
break
@@ -648,12 +717,21 @@ func within(p, start, span int) bool {
return mod1440(p-start) < span
}
func (l *Loop) restartAll(stacks []string) {
// restartAll restarts the given stacks and RETURNS the ones that failed.
//
// F-CRIT-1 cause 1: this used to return nothing. The error was logged and dropped on the spot, so
// no caller could learn that a customer's app had not come back — and the classifier downstream
// therefore had nothing to key on. A restart failure is the single most important fact this loop
// produces; it must leave the function.
func (l *Loop) restartAll(stacks []string) []string {
var failed []string
for _, s := range stacks {
if err := l.stacks.StartStack(s); err != nil {
l.logger.Printf("[ERROR] [quiesce] restart %s: %v", s, err)
failed = append(failed, s)
}
}
return failed
}
// ---- marker persistence (atomic, 0600) --------------------------------------------------