F-CRIT-1 + F-A1: one alarm that never fired, one that fired wrongly (v0.179.0)

F-CRIT-1 — an app that failed to restart after a quiesce never alarmed, for two
independent reasons, either of which alone kept it dead:
  1. restartAll returned nothing, so the failure was logged and dropped and no
     caller could learn a customer's app had not come back. It now returns the
     stacks that failed; both call sites record the outcome.
  2. classifyRunStates whitelists StateStopped on invariant I1 ('StateStopped
     means the user stopped it'). The quiesce loop stops stacks by the same
     compose-down path, so a failed restart is also StateStopped and was
     whitelisted into silence. Loop.FailedRestarts() is now the only thing that
     lifts the whitelist, so genuine user stops stay silent (v0.164.0 pinned).

F-A1 — HTTP 409 is the agent's single-flight gate refusing while a restore-test
holds it, not a failure. agentapi now returns a typed *StatusError on POST, the
adapter maps 409 -> quiesce.ErrTierBusy, and the loop defers: no breaker, no
event, no operator email, tier stays DUE.

Two traps avoided. Silence: contention outliving contentionAlarmAfter (3h, set
by the agent's own 120m PBS restore-test ceiling) raises its own BLOCKED signal.
App thrash: removing the failure treatment also removes the breaker's deferral,
so a contended tier is dropped BEFORE anything stops (contentionRetryAfter 15m,
against a 12m01s longest observed restore-test).

Three comments corrected; the invariant rule added to both CLAUDE.md copies.
Six red-proofs, all observed failing.
This commit is contained in:
2026-07-28 08:50:11 +02:00
parent 8f46495426
commit 079265ad8e
10 changed files with 884 additions and 29 deletions
+66
View File
@@ -1,5 +1,71 @@
## Changelog
### v0.179.0 — F-CRIT-1 + F-A1: one alarm that never fired, one that fired wrongly (2026-07-28)
Both Campaign 8 findings live in `internal/quiesce` and its `classifyRunStates` consumer, and both
are "the alarm is wrong" — one missing, one spurious. Fixed together, one pass over the same code.
#### F-CRIT-1 — an app that failed to restart after a quiesce NEVER alarmed. **Two independent causes; either alone kept it dead.**
*Cause 1 — the outcome was thrown away.* `restartAll` returned nothing; a failed `StartStack` was
logged and dropped on the spot, so no caller could learn a customer's app had not come back. It now
**returns the stacks that failed**, and both call sites (the cycle's unquiesce and crash recovery)
record the result.
*Cause 2 — a documented invariant the quiesce path had made false.* `classifyRunStates` whitelists
`StateStopped` because v0.164.0 (correctly) refused to alarm on deliberate user stops, resting on
I1: *"StateStopped means deployed, deliberately stopped by the user."* **The quiesce loop stops
stacks by the same `docker compose down` path**, so a stack it stopped and then failed to restart is
also `StateStopped` — byte-identical to a user stop on the Docker side — and was whitelisted into
total silence. Campaign 8 watched a customer app sit dead indefinitely with no banner, no event and
no email while the dead-app scanner ran over it 11 times.
No state test can separate the two; they *are* the same state. The distinguishing fact is that the
loop **tried to restart it and could not**, which it now reports via `Loop.FailedRestarts()`. That
set is the only thing that lifts the whitelist, so genuine user stops stay silent (pinned by
`TestClassifyRunStates_UserStopStillSilent`).
*Why the existing tests missed it:* R-97b's Scenario F asserted that **suppression expires**. It
never asserted that **an alarm follows**. The suppression lifted correctly and the whitelist ate the
alarm one layer down — a green, red-proofed suite over a production path broken two ways.
#### F-A1 — a correct refusal reported as a failure
HTTP 409 from `POST /backup` is the agent's R-85 single-flight gate refusing while a restore-test
holds it. The start path had no 409 branch, so it called `noteTierFailure`: the R-88 breaker armed
and `whole_guest_backup_failed` was emailed. Campaign 8 saw it on both boxes in the same minute.
At real cadences a ~12-minute restore-test against a daily backup collides roughly once per 420
guest-days — about **every 4 days on a 100-guest fleet, forever** — which trains the operator to
ignore the alarm and quietly undoes R-97a.
409 is now **contention, not failure**: `agentapi` returns a typed `*StatusError` on POST, the
adapter maps 409 → `quiesce.ErrTierBusy` (the same seam that maps 404 → `ErrTiersUnsupported`), and
the loop defers instead of failing. No breaker, no event, no email; the tier stays **DUE**.
**Two traps this deliberately avoids.** *Silence:* "just ignore 409" would let a wedged
restore-test block backups forever with nobody told — so contention lasting past
`contentionAlarmAfter` (**3h**) raises its own signal, headlined **BLOCKED**, not FAILED. The bound
is set by the agent's own ceiling, not taste: its PBS restore-test task is capped at 120 minutes, so
contention outliving that is a stuck gate rather than a busy one; 3h adds margin and is 15× the
longest contention actually observed (12m01s). *App thrash:* removing the failure treatment also
removes the breaker's deferral, which had been (accidentally) preventing a re-quiesce every 5
minutes. Without a replacement the customer's apps would be stopped and restarted on **every poll**
for the whole restore-test — worse than the bug. A contended tier is therefore dropped from the due
set **before anything stops**, on a `contentionRetryAfter` of 15m (the longest observed restore-test
is 12m01s; the agent's local restore-test wait is 10m).
#### Comments corrected (three of the six)
`classifyRunStates`' I1 now states what `StateStopped` actually means and names the quiesce path;
`quiesce.go`'s "would record a spurious failure" says that this was not hypothetical until now; and
the agent's `inflight.go` "a caller that cannot acquire DEFERS" records that this was true of the
restore-test caller and not the backup caller. A standing rule was added to both copies of
`CLAUDE.md`: **a comment asserting an invariant needs a test pinning it, or it is a wish.**
Files: `internal/quiesce/{quiesce.go,suppress.go,contention.go (new)}`,
`internal/agentapi/client.go`, `cmd/controller/main.go`, plus new tests
`internal/quiesce/{failed_restart_test.go,contention_test.go}` and
`cmd/controller/failed_restart_classify_test.go`. No wire/contract change; no agent behaviour change.
### v0.178.0 — R-88 Part 2 (controller) + R-97c comment fix (2026-07-27) — **MinAgent: 0.105.0** for the age_state semantics
**The safety valve now needs a licence.** `scheduledRunAllowed` fired on ANY nil age — "no recorded