F-CRIT-1 + F-A1: one alarm that never fired, one that fired wrongly (v0.179.0)

F-CRIT-1 — an app that failed to restart after a quiesce never alarmed, for two
independent reasons, either of which alone kept it dead:
  1. restartAll returned nothing, so the failure was logged and dropped and no
     caller could learn a customer's app had not come back. It now returns the
     stacks that failed; both call sites record the outcome.
  2. classifyRunStates whitelists StateStopped on invariant I1 ('StateStopped
     means the user stopped it'). The quiesce loop stops stacks by the same
     compose-down path, so a failed restart is also StateStopped and was
     whitelisted into silence. Loop.FailedRestarts() is now the only thing that
     lifts the whitelist, so genuine user stops stay silent (v0.164.0 pinned).

F-A1 — HTTP 409 is the agent's single-flight gate refusing while a restore-test
holds it, not a failure. agentapi now returns a typed *StatusError on POST, the
adapter maps 409 -> quiesce.ErrTierBusy, and the loop defers: no breaker, no
event, no operator email, tier stays DUE.

Two traps avoided. Silence: contention outliving contentionAlarmAfter (3h, set
by the agent's own 120m PBS restore-test ceiling) raises its own BLOCKED signal.
App thrash: removing the failure treatment also removes the breaker's deferral,
so a contended tier is dropped BEFORE anything stops (contentionRetryAfter 15m,
against a 12m01s longest observed restore-test).

Three comments corrected; the invariant rule added to both CLAUDE.md copies.
Six red-proofs, all observed failing.
This commit is contained in:
2026-07-28 08:50:11 +02:00
parent 8f46495426
commit 079265ad8e
10 changed files with 884 additions and 29 deletions
+60
View File
@@ -88,3 +88,63 @@ func (l *Loop) SuppressedStacks() map[string]bool {
}
return out
}
// ---- F-CRIT-1: the loop remembers which stacks it FAILED to restart -------------------------
//
// THE BUG THIS EXISTS TO KILL. `classifyRunStates` whitelists `StateStopped` because v0.164.0
// (correctly) refused to alarm on a deliberate user stop. That rests on invariant I1: "a deployed
// stack with zero containers was stopped by the user". **The quiesce loop broke I1** — it stops
// stacks by the same `docker compose down` path, so a stack this loop stopped and then FAILED to
// restart is also `StateStopped`, and was therefore whitelisted into total silence. Campaign 8
// observed exactly that: a customer's app dead indefinitely, no banner, no event, no email, while
// the dead-app scanner ran 11 times.
//
// No state test can separate the two cases — they are byte-identical on the Docker side. The
// distinguishing fact is not in the state at all: it is that WE tried to restart it and could not.
// That fact now leaves `restartAll`, and is remembered here.
//
// WHY A SET AND NOT A TIMESTAMP. The flag is only ever consulted for a stack that is ALREADY in a
// down state, so a stale entry cannot manufacture an alarm on a healthy app: if the operator fixes
// the app and starts it by any route, the stack is no longer down and the classifier never reaches
// this. The entry is cleared the moment a later restart of that stack succeeds.
// noteRestartOutcome records the result of one restart pass: `failed` are the stacks that would not
// start, and everything else in `attempted` is cleared. Clearing on success is what stops a fixed
// app from carrying its old failure forever.
func (l *Loop) noteRestartOutcome(attempted, failed []string) {
if l == nil {
return
}
bad := make(map[string]bool, len(failed))
for _, n := range failed {
bad[n] = true
}
l.suppressMu.Lock()
defer l.suppressMu.Unlock()
if l.restartFailed == nil {
l.restartFailed = map[string]struct{}{}
}
for _, n := range attempted {
if bad[n] {
l.restartFailed[n] = struct{}{}
} else {
delete(l.restartFailed, n)
}
}
}
// FailedRestarts returns the stacks this loop stopped and could not restart. Nil-safe on a nil
// *Loop for the same reason SuppressedStacks is: an unprovisioned guest has no loop and must report
// nothing rather than force a branch on the caller.
func (l *Loop) FailedRestarts() map[string]bool {
if l == nil {
return nil
}
l.suppressMu.Lock()
defer l.suppressMu.Unlock()
out := make(map[string]bool, len(l.restartFailed))
for n := range l.restartFailed {
out[n] = true
}
return out
}