F-CRIT-1 + F-A1: one alarm that never fired, one that fired wrongly (v0.179.0)
F-CRIT-1 — an app that failed to restart after a quiesce never alarmed, for two
independent reasons, either of which alone kept it dead:
1. restartAll returned nothing, so the failure was logged and dropped and no
caller could learn a customer's app had not come back. It now returns the
stacks that failed; both call sites record the outcome.
2. classifyRunStates whitelists StateStopped on invariant I1 ('StateStopped
means the user stopped it'). The quiesce loop stops stacks by the same
compose-down path, so a failed restart is also StateStopped and was
whitelisted into silence. Loop.FailedRestarts() is now the only thing that
lifts the whitelist, so genuine user stops stay silent (v0.164.0 pinned).
F-A1 — HTTP 409 is the agent's single-flight gate refusing while a restore-test
holds it, not a failure. agentapi now returns a typed *StatusError on POST, the
adapter maps 409 -> quiesce.ErrTierBusy, and the loop defers: no breaker, no
event, no operator email, tier stays DUE.
Two traps avoided. Silence: contention outliving contentionAlarmAfter (3h, set
by the agent's own 120m PBS restore-test ceiling) raises its own BLOCKED signal.
App thrash: removing the failure treatment also removes the breaker's deferral,
so a contended tier is dropped BEFORE anything stops (contentionRetryAfter 15m,
against a 12m01s longest observed restore-test).
Three comments corrected; the invariant rule added to both CLAUDE.md copies.
Six red-proofs, all observed failing.
This commit is contained in:
@@ -88,3 +88,63 @@ func (l *Loop) SuppressedStacks() map[string]bool {
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ---- F-CRIT-1: the loop remembers which stacks it FAILED to restart -------------------------
|
||||
//
|
||||
// THE BUG THIS EXISTS TO KILL. `classifyRunStates` whitelists `StateStopped` because v0.164.0
|
||||
// (correctly) refused to alarm on a deliberate user stop. That rests on invariant I1: "a deployed
|
||||
// stack with zero containers was stopped by the user". **The quiesce loop broke I1** — it stops
|
||||
// stacks by the same `docker compose down` path, so a stack this loop stopped and then FAILED to
|
||||
// restart is also `StateStopped`, and was therefore whitelisted into total silence. Campaign 8
|
||||
// observed exactly that: a customer's app dead indefinitely, no banner, no event, no email, while
|
||||
// the dead-app scanner ran 11 times.
|
||||
//
|
||||
// No state test can separate the two cases — they are byte-identical on the Docker side. The
|
||||
// distinguishing fact is not in the state at all: it is that WE tried to restart it and could not.
|
||||
// That fact now leaves `restartAll`, and is remembered here.
|
||||
//
|
||||
// WHY A SET AND NOT A TIMESTAMP. The flag is only ever consulted for a stack that is ALREADY in a
|
||||
// down state, so a stale entry cannot manufacture an alarm on a healthy app: if the operator fixes
|
||||
// the app and starts it by any route, the stack is no longer down and the classifier never reaches
|
||||
// this. The entry is cleared the moment a later restart of that stack succeeds.
|
||||
|
||||
// noteRestartOutcome records the result of one restart pass: `failed` are the stacks that would not
|
||||
// start, and everything else in `attempted` is cleared. Clearing on success is what stops a fixed
|
||||
// app from carrying its old failure forever.
|
||||
func (l *Loop) noteRestartOutcome(attempted, failed []string) {
|
||||
if l == nil {
|
||||
return
|
||||
}
|
||||
bad := make(map[string]bool, len(failed))
|
||||
for _, n := range failed {
|
||||
bad[n] = true
|
||||
}
|
||||
l.suppressMu.Lock()
|
||||
defer l.suppressMu.Unlock()
|
||||
if l.restartFailed == nil {
|
||||
l.restartFailed = map[string]struct{}{}
|
||||
}
|
||||
for _, n := range attempted {
|
||||
if bad[n] {
|
||||
l.restartFailed[n] = struct{}{}
|
||||
} else {
|
||||
delete(l.restartFailed, n)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// FailedRestarts returns the stacks this loop stopped and could not restart. Nil-safe on a nil
|
||||
// *Loop for the same reason SuppressedStacks is: an unprovisioned guest has no loop and must report
|
||||
// nothing rather than force a branch on the caller.
|
||||
func (l *Loop) FailedRestarts() map[string]bool {
|
||||
if l == nil {
|
||||
return nil
|
||||
}
|
||||
l.suppressMu.Lock()
|
||||
defer l.suppressMu.Unlock()
|
||||
out := make(map[string]bool, len(l.restartFailed))
|
||||
for n := range l.restartFailed {
|
||||
out[n] = true
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user