F-CRIT-1 + F-A1: one alarm that never fired, one that fired wrongly (v0.179.0)

F-CRIT-1 — an app that failed to restart after a quiesce never alarmed, for two
independent reasons, either of which alone kept it dead:
  1. restartAll returned nothing, so the failure was logged and dropped and no
     caller could learn a customer's app had not come back. It now returns the
     stacks that failed; both call sites record the outcome.
  2. classifyRunStates whitelists StateStopped on invariant I1 ('StateStopped
     means the user stopped it'). The quiesce loop stops stacks by the same
     compose-down path, so a failed restart is also StateStopped and was
     whitelisted into silence. Loop.FailedRestarts() is now the only thing that
     lifts the whitelist, so genuine user stops stay silent (v0.164.0 pinned).

F-A1 — HTTP 409 is the agent's single-flight gate refusing while a restore-test
holds it, not a failure. agentapi now returns a typed *StatusError on POST, the
adapter maps 409 -> quiesce.ErrTierBusy, and the loop defers: no breaker, no
event, no operator email, tier stays DUE.

Two traps avoided. Silence: contention outliving contentionAlarmAfter (3h, set
by the agent's own 120m PBS restore-test ceiling) raises its own BLOCKED signal.
App thrash: removing the failure treatment also removes the breaker's deferral,
so a contended tier is dropped BEFORE anything stops (contentionRetryAfter 15m,
against a 12m01s longest observed restore-test).

Three comments corrected; the invariant rule added to both CLAUDE.md copies.
Six red-proofs, all observed failing.
This commit is contained in:
2026-07-28 08:50:11 +02:00
parent 8f46495426
commit 079265ad8e
10 changed files with 884 additions and 29 deletions
+21 -4
View File
@@ -1089,13 +1089,28 @@ func (c *Client) HostMetrics(ctx context.Context) (HostMetricsResponse, error) {
// StatusError is a non-2xx agent HTTP status surfaced as a TYPED error (same text the old
// fmt.Errorf produced). errors.As-able — the capability probe (features.go) keys on Code 404 to
// distinguish "this agent predates the route" from every other failure. Never match the string.
// StatusError is a non-2xx response from the agent, carrying the STATUS CODE so callers can react
// to specific ones rather than string-matching an error message.
//
// F-A1: this exists on the POST path because HTTP 409 from `POST /backup` is not a failure — it is
// the agent's R-85 single-flight gate correctly refusing while a restore-test holds it. Treating
// that refusal as a tier failure armed the breaker and emailed the operator about a backup that was
// never actually broken. The controller now needs to tell 409 apart from a real error, and a typed
// code is the only honest way to do that.
type StatusError struct {
Path string
Code int
// Method is the HTTP method. Empty means GET, so the message stays byte-identical for the
// pre-existing GET call sites.
Method string
Path string
Code int
}
func (e *StatusError) Error() string {
return fmt.Sprintf("agentapi: GET %s: HTTP %d", e.Path, e.Code)
m := e.Method
if m == "" {
m = http.MethodGet
}
return fmt.Sprintf("agentapi: %s %s: HTTP %d", m, e.Path, e.Code)
}
// get issues an authenticated GET and unwraps the {ok,data,error} envelope.
@@ -1152,7 +1167,9 @@ func (c *Client) post(ctx context.Context, path string, body any) (json.RawMessa
logx.Debugf(c.logger, "[agentapi] POST %s -> %d (%dms)", path, resp.StatusCode, time.Since(start).Milliseconds())
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusAccepted {
return nil, fmt.Errorf("agentapi: POST %s: HTTP %d", path, resp.StatusCode)
// Typed, not fmt.Errorf: callers must be able to distinguish 409 (the agent's single-flight
// gate refusing — contention, not failure) from a genuine 5xx. See StatusError.
return nil, &StatusError{Method: http.MethodPost, Path: path, Code: resp.StatusCode}
}
var env apiResponse
if err := json.Unmarshal(raw, &env); err != nil {