diff --git a/REPORT-fcrit1-fa1.md b/REPORT-fcrit1-fa1.md new file mode 100644 index 0000000..3e37816 --- /dev/null +++ b/REPORT-fcrit1-fa1.md @@ -0,0 +1,63 @@ +# REPORT — F-CRIT-1 + F-A1 fixed (controller v0.179.0, 2026-07-28) + +Docs here. Implementation, all six red-proofs and the full live replay live in +`felhom-controller/REPORT.md`. The campaign that found both: +`documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`. + +## What changed +**F-CRIT-1** — an app that failed to restart after a quiesce never alarmed, for two independent +reasons, either of which alone kept it dead: `restartAll` returned nothing (the failure was logged +and dropped), and `classifyRunStates` whitelisted `StateStopped` on invariant I1 ("the user stopped +it") — which the quiesce loop had made false by stopping stacks the same `docker compose down` way. +A failed restart and a user stop are the *same* Docker state; the only difference is that the loop +tried and could not, now surfaced by `Loop.FailedRestarts()`. + +**F-A1** — HTTP 409 is the agent's single-flight gate refusing while a restore-test holds it, not a +failure. It is now contention: the tier stays DUE, is dropped before anything stops, and unending +contention raises its own **BLOCKED** signal rather than going silent. + +## Bounds, justified against measured reality +- `contentionRetryAfter` **15m** — longest restore-test observed on the fleet is 12m01s; the agent's + local restore-test wait is 10m. Caps app-stop churn at 4/hour instead of 12/hour. +- `contentionAlarmAfter` **3h** — the agent's own PBS restore-test task is capped at 120 minutes, so + contention outliving that is a *stuck* gate, not a busy one. 3h adds margin and is 15× the longest + contention actually observed. + +## Verified live, with the hub DB as arbiter — not from logs +Same box, same day, same event type; the only difference is 409 versus a genuine error: + +| injection | operator emails (demo-hp) | +|---|---| +| **409 contention** | 8 → **8** (none) | +| **real transport failure** | 8 → **9** | + +And for F-CRIT-1: the failed restart alarmed **9 seconds** after grace expiry with the dashboard +banner naming the `(stopped)` state, while a **deliberate** user stop on the same box stayed silent +through **9** dead-app scans (the positive observable that the silence is suppression, not a dead +detector). + +## The rule this arc earned +Added to **both** copies of `CLAUDE.md` (live + `documentation/runbooks/workspace-CLAUDE.md`): +**a comment asserting an invariant needs a test pinning it, or it is a wish.** Six instances in this +project have shipped guarantees the code did not provide — `EffectiveProtected`, `newestArchiveOn`, +the R-97a operator-only claim, `classifyRunStates`' I1, `inflight.go`'s defer claim, and +`quiesce.go`'s spurious-failure claim. Two were found only on live hardware, and one of those had a +green, red-proofed test suite over a production path broken two independent ways. + +Corollary recorded with it: prefer a test that asserts the **consequence** (does the alarm fire?) +over one that asserts the **mechanism** (does suppression expire?). R-97b's Scenario F proved the +mechanism; the consequence was still broken. + +## Docs touched +- `documentation/backlog/OPEN-ITEMS.md` — F-CRIT-1 and F-A1 → SHIPPED + PROVEN-LIVE. +- `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md` — both rows struck through, closing + section added. **All three of the campaign's alarm findings are now closed** (F-CRIT-1, F-CRIT-2, + F-A1). +- `documentation/runbooks/workspace-CLAUDE.md` — the invariant rule. + +## Still open, highest first +**Fault 4** (restic transport interruption) — four injection approaches were defeated by +guest-bridged networking, and it is now the most valuable follow-up: F-CRIT-2 answered the phantom +question for PBS and left the identical question open for restic. Then **R-99** (prune never removes +phantoms) and **F-LEAK** (a failed restore-test cannot destroy its own scratch guest — observed +again during this work). diff --git a/documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md b/documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md index f97d3e3..a0238c9 100644 --- a/documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md +++ b/documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md @@ -265,9 +265,9 @@ Neither restored guest was booted — deliberately, see F-OPS below. | # | Finding | Severity | DEFECT / ARTIFACT | |---|---|---|---| -| **F-CRIT-1** | An app that fails to restart after a quiesce **never alarms** | **HIGH** | **DEFECT** (compression-independent) | +| ~~**F-CRIT-1**~~ | ~~An app that fails to restart after a quiesce **never alarms**~~ — **FIXED, controller v0.179.0 (2026-07-28), both causes, replayed live** | ~~HIGH~~ | DEFECT — closed | | ~~**F-CRIT-2**~~ | ~~A failed offsite backup leaves a phantom snapshot that **resets tier freshness**~~ — **FIXED, agent v0.106.0 (2026-07-28), re-tested live by replaying fault 2** | ~~HIGH~~ | DEFECT — closed | -| **F-A1** | A restore-test in progress makes a healthy backup report as FAILED, arms the breaker and pages the operator | MEDIUM | **DEFECT** in behaviour / ARTIFACT in frequency | +| ~~**F-A1**~~ | ~~A restore-test in progress makes a healthy backup report as FAILED~~ — **FIXED, controller v0.179.0 (2026-07-28); hub DB shows 0 spurious emails, real failures still alarm** | ~~MEDIUM~~ | DEFECT — closed | | **F-HUB** | The hub **dropped an event** under concurrent load (`SQLITE_BUSY`), no retry, cause unnamed | MEDIUM | DEFECT (frequency compression-influenced) | | **F-LEAK** | A **failed** restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); leaks are never reclaimed and the 10-slot VMID band shrinks silently | MEDIUM | DEFECT (root-caused by fault 18) | | **F-REBOOT** | A guest rebooted during its backup **does not come back** — shutdown completes, start never happens, no self-heal | MEDIUM | DEFECT | @@ -336,6 +336,24 @@ Frequency is an ARTIFACT; the behaviour is a DEFECT — at real cadences a ~12-m against a daily backup collides roughly once per 420 guest-days, i.e. **every ~4 days on a 100-guest fleet**, forever training R-97a's alarm into noise. +### F-CRIT-1 and F-A1 — FIXED (controller v0.179.0, 2026-07-28) +**F-CRIT-1**, both causes: `restartAll` now returns the stacks that failed, and `classifyRunStates` +lifts the `StateStopped` whitelist for exactly those (`Loop.FailedRestarts()`). I1 is restated +truthfully — `StateStopped` means "deployed with zero containers", which is EITHER a user stop OR a +failed quiesce restart, and only the loop's own knowledge separates them. + +**F-A1**: HTTP 409 is contention, not failure — typed `*StatusError` on POST, mapped to +`quiesce.ErrTierBusy` at the adapter. The tier stays DUE and is dropped before anything stops +(`contentionRetryAfter` 15m), and contention outliving the agent's own 120m PBS restore-test ceiling +raises a **BLOCKED** signal at 3h so a wedged gate is never silent. + +Replayed live on demo-hp with the hub DB as arbiter: the failed restart alarmed 9 s after grace +expiry with the banner naming `(stopped)`; a deliberate user stop stayed silent through 9 dead-app +scans; a 409 produced **0** operator emails (8 → 8) while a real transport failure produced **1** +(8 → 9). Six red-proofs. Detail: `felhom-controller/REPORT.md`. + +**All three of Campaign 8's HIGH/MEDIUM alarm findings are now closed** (F-CRIT-1, F-CRIT-2, F-A1). + ### F-CRIT-2 — FIXED (agent v0.106.0, 2026-07-28) `NewestArchiveTime` now counts only *plausibly complete* entries, via a measured size floor (`minPlausibleArchiveBytes` = 1 MiB; smallest real backup on the fleet is 612,397,450 B, so 584x diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index d451cbb..87cc882 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -24,7 +24,8 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-97** | ~~Whole-guest backup tier had no hub signal; quiesce blamed the apps~~ | **SHIPPED** (controller v0.177.0 + hub v0.78.0/v0.79.0, 2026-07-27) | — | v0.79.0 (R-97c) replaced a FALSE operator-only comment with a real `operatorOnlyEvents` register | — | | **F-CRIT-2** | ~~A failed offsite backup left a phantom snapshot (1 B, manifest-less, NEWEST) that RESET the tier's freshness clock — 7 days silent on the real 168h cadence, invisible to both the R-88 breaker and the hub deadline monitor~~ | **SHIPPED + PROVEN-LIVE** (agent v0.106.0, 2026-07-28) | — | `NewestArchiveTime` now counts only plausibly-complete entries (measured 1 MiB floor; undecidable ⇒ not counted). Campaign fault 2 replayed on demo-hp: phantom rejected + logged once, tier correctly DUE and backed up, and **no thrash** on the inverse | — | | **R-99** | Server-side prune **never removes** a phantom snapshot. Confirmed it does NOT count them toward `keep-last` (dry-run kept 2 real + the phantom) so there is **no retention/data-loss bug** — but one accumulates per aborted upload, forever | READY (S) | — | Decide a cleanup path. Deletion on a **customer** datastore is a separate ruling — detection shipped, removal deliberately not automated | CC | -| **F-CRIT-1** | An app that **fails to restart** after a quiesce never alarms on any channel. Two independent causes: `restartAll` discards the error (returns nothing), and `StateStopped` is whitelisted at `main.go:1237` on invariant I1, which is false for the failed-restart path | **READY — HIGHEST** | — | Campaign 8's other HIGH finding, untouched by the F-CRIT-2 fix. Evidence: `audits/CAMPAIGN-8-backup-restore-2026-07-27.md` | CC | +| **F-CRIT-1** | ~~An app that **fails to restart** after a quiesce never alarms on any channel — `restartAll` discarded the error AND `StateStopped` was whitelisted on invariant I1, which the quiesce path had made false~~ | **SHIPPED + PROVEN-LIVE** (controller v0.179.0, 2026-07-28) | — | Both causes fixed. Live on demo-hp: alarmed 9s after grace expiry, banner shows `(stopped)`; a deliberate user stop stayed silent through 9 dead-app scans | — | +| **F-A1** | ~~A restore-test in flight made a healthy backup report as FAILED (HTTP 409 read as a tier failure): breaker armed + operator emailed, on both boxes~~ | **SHIPPED + PROVEN-LIVE** (controller v0.179.0, 2026-07-28) | — | 409 → contention: tier stays DUE, dropped before anything stops (15m), and BLOCKED alarm if contention outlives the agent's 120m ceiling (3h). Hub DB: 409 → **0** operator emails, real failure → **1** | — | | **R-89** | Retention as a per-customer **commercial** policy on the hub | READY (increment 2) | — | Policy object + reconciler → ep0 prune job; keep box tokens write-only | CC | | **R-92** | Hub PBS-DR gauge is 0.1 GB-granular — small deltas unverifiable | READY (XS) | — | Widen precision when retention becomes customer-visible | CC | | **R-93** | `drill-r50` is both a blocked customer and the only drift fixture | READY (XS) | — | Retire it for a synthetic fixture, or unblock + silence per-customer | CC | diff --git a/documentation/runbooks/workspace-CLAUDE.md b/documentation/runbooks/workspace-CLAUDE.md index 031beae..4b391a9 100644 --- a/documentation/runbooks/workspace-CLAUDE.md +++ b/documentation/runbooks/workspace-CLAUDE.md @@ -147,3 +147,28 @@ Kept so the old environment can be revived; **not the current setup**. - Skills were installed as Windows junctions (`mklink /J`) rather than POSIX symlinks. - `claude-in-chrome` browser automation WAS available there (attaching only to sessions started after the bridge connected). + +### A comment asserting an invariant needs a test pinning it, or it is a wish + +**Six instances in this project have shipped guarantees the code did not provide** — each survived +review because the comment read as settled: + +| # | Comment | What it claimed | What the code did | +|---|---|---|---| +| 1 | `EffectiveProtected` | a stack was protected | it was not — the samba false alarm | +| 2 | `newestArchiveOn` | *"errors degrade to unknown, never to no-backup"* | the `(time,bool)` signature made that impossible (R-88 Part 2) | +| 3 | R-97a operator-only | the event *"cannot be routed to a customer"* | only configuration stopped it; fixed by a real `operatorOnlyEvents` register | +| 4 | `classifyRunStates` I1 | *"StateStopped means deliberately stopped by the user"* | quiesce stops stacks the same way — a failed restart was silent (F-CRIT-1) | +| 5 | `inflight.go` | *"a caller that cannot acquire DEFERS"* | the backup caller recorded a failure and paged the operator (F-A1) | +| 6 | `quiesce.go` | the agent's 409 *prevents* "a spurious failure" | on the start path it produced one (F-A1) | + +Two of these (4 and 5/6) were found by Campaign 8 **on live hardware**, not by review or unit tests +— #4 had a green, red-proofed test suite over a production path that was broken two independent +ways. So: + +- If a comment states an invariant, **name the test that pins it**, or write one. +- If an invariant has a stated dependency (*"if either invariant changes, revisit this"*), that is + not a safeguard — nobody revisits. Pin it with a test that fails when the dependency moves. +- Prefer a test that asserts the **consequence** (does the alarm fire?) over one that asserts the + **mechanism** (does suppression expire?). R-97b's Scenario F proved the mechanism and the + consequence was still broken.