F-CRIT-1 + F-A1: one alarm that never fired, one that fired wrongly (v0.179.0)
F-CRIT-1 — an app that failed to restart after a quiesce never alarmed, for two
independent reasons, either of which alone kept it dead:
1. restartAll returned nothing, so the failure was logged and dropped and no
caller could learn a customer's app had not come back. It now returns the
stacks that failed; both call sites record the outcome.
2. classifyRunStates whitelists StateStopped on invariant I1 ('StateStopped
means the user stopped it'). The quiesce loop stops stacks by the same
compose-down path, so a failed restart is also StateStopped and was
whitelisted into silence. Loop.FailedRestarts() is now the only thing that
lifts the whitelist, so genuine user stops stay silent (v0.164.0 pinned).
F-A1 — HTTP 409 is the agent's single-flight gate refusing while a restore-test
holds it, not a failure. agentapi now returns a typed *StatusError on POST, the
adapter maps 409 -> quiesce.ErrTierBusy, and the loop defers: no breaker, no
event, no operator email, tier stays DUE.
Two traps avoided. Silence: contention outliving contentionAlarmAfter (3h, set
by the agent's own 120m PBS restore-test ceiling) raises its own BLOCKED signal.
App thrash: removing the failure treatment also removes the breaker's deferral,
so a contended tier is dropped BEFORE anything stops (contentionRetryAfter 15m,
against a 12m01s longest observed restore-test).
Three comments corrected; the invariant rule added to both CLAUDE.md copies.
Six red-proofs, all observed failing.
This commit is contained in:
@@ -1089,13 +1089,28 @@ func (c *Client) HostMetrics(ctx context.Context) (HostMetricsResponse, error) {
|
||||
// StatusError is a non-2xx agent HTTP status surfaced as a TYPED error (same text the old
|
||||
// fmt.Errorf produced). errors.As-able — the capability probe (features.go) keys on Code 404 to
|
||||
// distinguish "this agent predates the route" from every other failure. Never match the string.
|
||||
// StatusError is a non-2xx response from the agent, carrying the STATUS CODE so callers can react
|
||||
// to specific ones rather than string-matching an error message.
|
||||
//
|
||||
// F-A1: this exists on the POST path because HTTP 409 from `POST /backup` is not a failure — it is
|
||||
// the agent's R-85 single-flight gate correctly refusing while a restore-test holds it. Treating
|
||||
// that refusal as a tier failure armed the breaker and emailed the operator about a backup that was
|
||||
// never actually broken. The controller now needs to tell 409 apart from a real error, and a typed
|
||||
// code is the only honest way to do that.
|
||||
type StatusError struct {
|
||||
Path string
|
||||
Code int
|
||||
// Method is the HTTP method. Empty means GET, so the message stays byte-identical for the
|
||||
// pre-existing GET call sites.
|
||||
Method string
|
||||
Path string
|
||||
Code int
|
||||
}
|
||||
|
||||
func (e *StatusError) Error() string {
|
||||
return fmt.Sprintf("agentapi: GET %s: HTTP %d", e.Path, e.Code)
|
||||
m := e.Method
|
||||
if m == "" {
|
||||
m = http.MethodGet
|
||||
}
|
||||
return fmt.Sprintf("agentapi: %s %s: HTTP %d", m, e.Path, e.Code)
|
||||
}
|
||||
|
||||
// get issues an authenticated GET and unwraps the {ok,data,error} envelope.
|
||||
@@ -1152,7 +1167,9 @@ func (c *Client) post(ctx context.Context, path string, body any) (json.RawMessa
|
||||
logx.Debugf(c.logger, "[agentapi] POST %s -> %d (%dms)", path, resp.StatusCode, time.Since(start).Milliseconds())
|
||||
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
|
||||
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusAccepted {
|
||||
return nil, fmt.Errorf("agentapi: POST %s: HTTP %d", path, resp.StatusCode)
|
||||
// Typed, not fmt.Errorf: callers must be able to distinguish 409 (the agent's single-flight
|
||||
// gate refusing — contention, not failure) from a genuine 5xx. See StatusError.
|
||||
return nil, &StatusError{Method: http.MethodPost, Path: path, Code: resp.StatusCode}
|
||||
}
|
||||
var env apiResponse
|
||||
if err := json.Unmarshal(raw, &env); err != nil {
|
||||
|
||||
Reference in New Issue
Block a user