admin
e168600148
Both marked SHIPPED + PROVEN-LIVE in OPEN-ITEMS and the campaign doc. All three of Campaign 8's alarm findings are now closed (F-CRIT-1, F-CRIT-2, F-A1). Adds the standing rule earned by this arc to the versioned workspace CLAUDE.md: a comment asserting an invariant needs a test pinning it, or it is a wish — with all six shipped-false-guarantee instances catalogued, and the corollary that a test should assert the CONSEQUENCE (does the alarm fire?) not the MECHANISM (does suppression expire?).
64 lines
3.8 KiB
Markdown
64 lines
3.8 KiB
Markdown
# REPORT — F-CRIT-1 + F-A1 fixed (controller v0.179.0, 2026-07-28)
|
||
|
||
Docs here. Implementation, all six red-proofs and the full live replay live in
|
||
`felhom-controller/REPORT.md`. The campaign that found both:
|
||
`documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`.
|
||
|
||
## What changed
|
||
**F-CRIT-1** — an app that failed to restart after a quiesce never alarmed, for two independent
|
||
reasons, either of which alone kept it dead: `restartAll` returned nothing (the failure was logged
|
||
and dropped), and `classifyRunStates` whitelisted `StateStopped` on invariant I1 ("the user stopped
|
||
it") — which the quiesce loop had made false by stopping stacks the same `docker compose down` way.
|
||
A failed restart and a user stop are the *same* Docker state; the only difference is that the loop
|
||
tried and could not, now surfaced by `Loop.FailedRestarts()`.
|
||
|
||
**F-A1** — HTTP 409 is the agent's single-flight gate refusing while a restore-test holds it, not a
|
||
failure. It is now contention: the tier stays DUE, is dropped before anything stops, and unending
|
||
contention raises its own **BLOCKED** signal rather than going silent.
|
||
|
||
## Bounds, justified against measured reality
|
||
- `contentionRetryAfter` **15m** — longest restore-test observed on the fleet is 12m01s; the agent's
|
||
local restore-test wait is 10m. Caps app-stop churn at 4/hour instead of 12/hour.
|
||
- `contentionAlarmAfter` **3h** — the agent's own PBS restore-test task is capped at 120 minutes, so
|
||
contention outliving that is a *stuck* gate, not a busy one. 3h adds margin and is 15× the longest
|
||
contention actually observed.
|
||
|
||
## Verified live, with the hub DB as arbiter — not from logs
|
||
Same box, same day, same event type; the only difference is 409 versus a genuine error:
|
||
|
||
| injection | operator emails (demo-hp) |
|
||
|---|---|
|
||
| **409 contention** | 8 → **8** (none) |
|
||
| **real transport failure** | 8 → **9** |
|
||
|
||
And for F-CRIT-1: the failed restart alarmed **9 seconds** after grace expiry with the dashboard
|
||
banner naming the `(stopped)` state, while a **deliberate** user stop on the same box stayed silent
|
||
through **9** dead-app scans (the positive observable that the silence is suppression, not a dead
|
||
detector).
|
||
|
||
## The rule this arc earned
|
||
Added to **both** copies of `CLAUDE.md` (live + `documentation/runbooks/workspace-CLAUDE.md`):
|
||
**a comment asserting an invariant needs a test pinning it, or it is a wish.** Six instances in this
|
||
project have shipped guarantees the code did not provide — `EffectiveProtected`, `newestArchiveOn`,
|
||
the R-97a operator-only claim, `classifyRunStates`' I1, `inflight.go`'s defer claim, and
|
||
`quiesce.go`'s spurious-failure claim. Two were found only on live hardware, and one of those had a
|
||
green, red-proofed test suite over a production path broken two independent ways.
|
||
|
||
Corollary recorded with it: prefer a test that asserts the **consequence** (does the alarm fire?)
|
||
over one that asserts the **mechanism** (does suppression expire?). R-97b's Scenario F proved the
|
||
mechanism; the consequence was still broken.
|
||
|
||
## Docs touched
|
||
- `documentation/backlog/OPEN-ITEMS.md` — F-CRIT-1 and F-A1 → SHIPPED + PROVEN-LIVE.
|
||
- `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md` — both rows struck through, closing
|
||
section added. **All three of the campaign's alarm findings are now closed** (F-CRIT-1, F-CRIT-2,
|
||
F-A1).
|
||
- `documentation/runbooks/workspace-CLAUDE.md` — the invariant rule.
|
||
|
||
## Still open, highest first
|
||
**Fault 4** (restic transport interruption) — four injection approaches were defeated by
|
||
guest-bridged networking, and it is now the most valuable follow-up: F-CRIT-2 answered the phantom
|
||
question for PBS and left the identical question open for restic. Then **R-99** (prune never removes
|
||
phantoms) and **F-LEAK** (a failed restore-test cannot destroy its own scratch guest — observed
|
||
again during this work).
|