admin
e168600148
Both marked SHIPPED + PROVEN-LIVE in OPEN-ITEMS and the campaign doc. All three of Campaign 8's alarm findings are now closed (F-CRIT-1, F-CRIT-2, F-A1). Adds the standing rule earned by this arc to the versioned workspace CLAUDE.md: a comment asserting an invariant needs a test pinning it, or it is a wish — with all six shipped-false-guarantee instances catalogued, and the corollary that a test should assert the CONSEQUENCE (does the alarm fire?) not the MECHANISM (does suppression expire?).
3.8 KiB
REPORT — F-CRIT-1 + F-A1 fixed (controller v0.179.0, 2026-07-28)
Docs here. Implementation, all six red-proofs and the full live replay live in
felhom-controller/REPORT.md. The campaign that found both:
documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md.
What changed
F-CRIT-1 — an app that failed to restart after a quiesce never alarmed, for two independent
reasons, either of which alone kept it dead: restartAll returned nothing (the failure was logged
and dropped), and classifyRunStates whitelisted StateStopped on invariant I1 ("the user stopped
it") — which the quiesce loop had made false by stopping stacks the same docker compose down way.
A failed restart and a user stop are the same Docker state; the only difference is that the loop
tried and could not, now surfaced by Loop.FailedRestarts().
F-A1 — HTTP 409 is the agent's single-flight gate refusing while a restore-test holds it, not a failure. It is now contention: the tier stays DUE, is dropped before anything stops, and unending contention raises its own BLOCKED signal rather than going silent.
Bounds, justified against measured reality
contentionRetryAfter15m — longest restore-test observed on the fleet is 12m01s; the agent's local restore-test wait is 10m. Caps app-stop churn at 4/hour instead of 12/hour.contentionAlarmAfter3h — the agent's own PBS restore-test task is capped at 120 minutes, so contention outliving that is a stuck gate, not a busy one. 3h adds margin and is 15× the longest contention actually observed.
Verified live, with the hub DB as arbiter — not from logs
Same box, same day, same event type; the only difference is 409 versus a genuine error:
| injection | operator emails (demo-hp) |
|---|---|
| 409 contention | 8 → 8 (none) |
| real transport failure | 8 → 9 |
And for F-CRIT-1: the failed restart alarmed 9 seconds after grace expiry with the dashboard
banner naming the (stopped) state, while a deliberate user stop on the same box stayed silent
through 9 dead-app scans (the positive observable that the silence is suppression, not a dead
detector).
The rule this arc earned
Added to both copies of CLAUDE.md (live + documentation/runbooks/workspace-CLAUDE.md):
a comment asserting an invariant needs a test pinning it, or it is a wish. Six instances in this
project have shipped guarantees the code did not provide — EffectiveProtected, newestArchiveOn,
the R-97a operator-only claim, classifyRunStates' I1, inflight.go's defer claim, and
quiesce.go's spurious-failure claim. Two were found only on live hardware, and one of those had a
green, red-proofed test suite over a production path broken two independent ways.
Corollary recorded with it: prefer a test that asserts the consequence (does the alarm fire?) over one that asserts the mechanism (does suppression expire?). R-97b's Scenario F proved the mechanism; the consequence was still broken.
Docs touched
documentation/backlog/OPEN-ITEMS.md— F-CRIT-1 and F-A1 → SHIPPED + PROVEN-LIVE.documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md— both rows struck through, closing section added. All three of the campaign's alarm findings are now closed (F-CRIT-1, F-CRIT-2, F-A1).documentation/runbooks/workspace-CLAUDE.md— the invariant rule.
Still open, highest first
Fault 4 (restic transport interruption) — four injection approaches were defeated by guest-bridged networking, and it is now the most valuable follow-up: F-CRIT-2 answered the phantom question for PBS and left the identical question open for restic. Then R-99 (prune never removes phantoms) and F-LEAK (a failed restore-test cannot destroy its own scratch guest — observed again during this work).