admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: F-CRIT-1 + F-A1 shipped (controller v0.179.0); invariant rule

Browse Source 
Both marked SHIPPED + PROVEN-LIVE in OPEN-ITEMS and the campaign doc. All three
of Campaign 8's alarm findings are now closed (F-CRIT-1, F-CRIT-2, F-A1).

Adds the standing rule earned by this arc to the versioned workspace CLAUDE.md:
a comment asserting an invariant needs a test pinning it, or it is a wish — with
all six shipped-false-guarantee instances catalogued, and the corollary that a
test should assert the CONSEQUENCE (does the alarm fire?) not the MECHANISM
(does suppression expire?).
This commit is contained in:
authentik Default Admin
2026-07-28 09:48:51 +02:00
parent d0cec9d3f3
commit e168600148
4 changed files with 110 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT-fcrit1-fa1.md
+63
View File
@@ -0,0 +1,63 @@
# REPORT — F-CRIT-1 + F-A1 fixed (controller v0.179.0, 2026-07-28)
Docs here. Implementation, all six red-proofs and the full live replay live in
`felhom-controller/REPORT.md`. The campaign that found both:
`documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`.
## What changed
**F-CRIT-1** — an app that failed to restart after a quiesce never alarmed, for two independent
reasons, either of which alone kept it dead: `restartAll` returned nothing (the failure was logged
and dropped), and `classifyRunStates` whitelisted `StateStopped` on invariant I1 ("the user stopped
it") — which the quiesce loop had made false by stopping stacks the same `docker compose down` way.
A failed restart and a user stop are the *same* Docker state; the only difference is that the loop
tried and could not, now surfaced by `Loop.FailedRestarts()`.
**F-A1** — HTTP 409 is the agent's single-flight gate refusing while a restore-test holds it, not a
failure. It is now contention: the tier stays DUE, is dropped before anything stops, and unending
contention raises its own **BLOCKED** signal rather than going silent.
## Bounds, justified against measured reality
- `contentionRetryAfter` **15m** — longest restore-test observed on the fleet is 12m01s; the agent's
local restore-test wait is 10m. Caps app-stop churn at 4/hour instead of 12/hour.
- `contentionAlarmAfter` **3h** — the agent's own PBS restore-test task is capped at 120 minutes, so
contention outliving that is a *stuck* gate, not a busy one. 3h adds margin and is 15× the longest
contention actually observed.
## Verified live, with the hub DB as arbiter — not from logs
Same box, same day, same event type; the only difference is 409 versus a genuine error:
| injection | operator emails (demo-hp) |
|---|---|
| **409 contention** | 8 → **8** (none) |
| **real transport failure** | 8 → **9** |
And for F-CRIT-1: the failed restart alarmed **9 seconds** after grace expiry with the dashboard
banner naming the `(stopped)` state, while a **deliberate** user stop on the same box stayed silent
through **9** dead-app scans (the positive observable that the silence is suppression, not a dead
detector).
## The rule this arc earned
Added to **both** copies of `CLAUDE.md` (live + `documentation/runbooks/workspace-CLAUDE.md`):
**a comment asserting an invariant needs a test pinning it, or it is a wish.** Six instances in this
project have shipped guarantees the code did not provide — `EffectiveProtected`, `newestArchiveOn`,
the R-97a operator-only claim, `classifyRunStates`' I1, `inflight.go`'s defer claim, and
`quiesce.go`'s spurious-failure claim. Two were found only on live hardware, and one of those had a
green, red-proofed test suite over a production path broken two independent ways.
Corollary recorded with it: prefer a test that asserts the **consequence** (does the alarm fire?)
over one that asserts the **mechanism** (does suppression expire?). R-97b's Scenario F proved the
mechanism; the consequence was still broken.
## Docs touched
- `documentation/backlog/OPEN-ITEMS.md` — F-CRIT-1 and F-A1 → SHIPPED + PROVEN-LIVE.
- `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md` — both rows struck through, closing
section added. **All three of the campaign's alarm findings are now closed** (F-CRIT-1, F-CRIT-2,
F-A1).
- `documentation/runbooks/workspace-CLAUDE.md` — the invariant rule.
## Still open, highest first
**Fault 4** (restic transport interruption) — four injection approaches were defeated by
guest-bridged networking, and it is now the most valuable follow-up: F-CRIT-2 answered the phantom
question for PBS and left the identical question open for restic. Then **R-99** (prune never removes
phantoms) and **F-LEAK** (a failed restore-test cannot destroy its own scratch guest — observed
again during this work).
documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md
+20 -2
View File
@@ -265,9 +265,9 @@ Neither restored guest was booted — deliberately, see F-OPS below.
| # | Finding | Severity | DEFECT / ARTIFACT |
|---|---|---|---|
| **F-CRIT-1** | An app that fails to restart after a quiesce **never alarms** | **HIGH** | **DEFECT** (compression-independent) |
| ~~**F-CRIT-1**~~ | ~~An app that fails to restart after a quiesce **never alarms**~~ — **FIXED, controller v0.179.0 (2026-07-28), both causes, replayed live** | ~~HIGH~~ | DEFECT — closed |
| ~~**F-CRIT-2**~~ | ~~A failed offsite backup leaves a phantom snapshot that **resets tier freshness**~~ — **FIXED, agent v0.106.0 (2026-07-28), re-tested live by replaying fault 2** | ~~HIGH~~ | DEFECT — closed |
| **F-A1** | A restore-test in progress makes a healthy backup report as FAILED, arms the breaker and pages the operator | MEDIUM | **DEFECT** in behaviour / ARTIFACT in frequency |
| ~~**F-A1**~~ | ~~A restore-test in progress makes a healthy backup report as FAILED~~ — **FIXED, controller v0.179.0 (2026-07-28); hub DB shows 0 spurious emails, real failures still alarm** | ~~MEDIUM~~ | DEFECT — closed |
| **F-HUB** | The hub **dropped an event** under concurrent load (`SQLITE_BUSY`), no retry, cause unnamed | MEDIUM | DEFECT (frequency compression-influenced) |
| **F-LEAK** | A **failed** restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); leaks are never reclaimed and the 10-slot VMID band shrinks silently | MEDIUM | DEFECT (root-caused by fault 18) |
| **F-REBOOT** | A guest rebooted during its backup **does not come back** — shutdown completes, start never happens, no self-heal | MEDIUM | DEFECT |
@@ -336,6 +336,24 @@ Frequency is an ARTIFACT; the behaviour is a DEFECT — at real cadences a ~12-m
against a daily backup collides roughly once per 420 guest-days, i.e. **every ~4 days on a
100-guest fleet**, forever training R-97a's alarm into noise.
### F-CRIT-1 and F-A1 — FIXED (controller v0.179.0, 2026-07-28)
**F-CRIT-1**, both causes: `restartAll` now returns the stacks that failed, and `classifyRunStates`
lifts the `StateStopped` whitelist for exactly those (`Loop.FailedRestarts()`). I1 is restated
truthfully — `StateStopped` means "deployed with zero containers", which is EITHER a user stop OR a
failed quiesce restart, and only the loop's own knowledge separates them.
**F-A1**: HTTP 409 is contention, not failure — typed `*StatusError` on POST, mapped to
`quiesce.ErrTierBusy` at the adapter. The tier stays DUE and is dropped before anything stops
(`contentionRetryAfter` 15m), and contention outliving the agent's own 120m PBS restore-test ceiling
raises a **BLOCKED** signal at 3h so a wedged gate is never silent.
Replayed live on demo-hp with the hub DB as arbiter: the failed restart alarmed 9 s after grace
expiry with the banner naming `(stopped)`; a deliberate user stop stayed silent through 9 dead-app
scans; a 409 produced **0** operator emails (8 → 8) while a real transport failure produced **1**
(8 → 9). Six red-proofs. Detail: `felhom-controller/REPORT.md`.
**All three of Campaign 8's HIGH/MEDIUM alarm findings are now closed** (F-CRIT-1, F-CRIT-2, F-A1).
### F-CRIT-2 — FIXED (agent v0.106.0, 2026-07-28)
`NewestArchiveTime` now counts only *plausibly complete* entries, via a measured size floor
(`minPlausibleArchiveBytes` = 1 MiB; smallest real backup on the fleet is 612,397,450 B, so 584x
documentation/backlog/OPEN-ITEMS.md
+2 -1
View File
@@ -24,7 +24,8 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
| **R-97** | ~~Whole-guest backup tier had no hub signal; quiesce blamed the apps~~ | **SHIPPED** (controller v0.177.0 + hub v0.78.0/v0.79.0, 2026-07-27) | — | v0.79.0 (R-97c) replaced a FALSE operator-only comment with a real `operatorOnlyEvents` register | — |
| **F-CRIT-2** | ~~A failed offsite backup left a phantom snapshot (1 B, manifest-less, NEWEST) that RESET the tier's freshness clock — 7 days silent on the real 168h cadence, invisible to both the R-88 breaker and the hub deadline monitor~~ | **SHIPPED + PROVEN-LIVE** (agent v0.106.0, 2026-07-28) | — | `NewestArchiveTime` now counts only plausibly-complete entries (measured 1 MiB floor; undecidable ⇒ not counted). Campaign fault 2 replayed on demo-hp: phantom rejected + logged once, tier correctly DUE and backed up, and **no thrash** on the inverse | — |
| **R-99** | Server-side prune **never removes** a phantom snapshot. Confirmed it does NOT count them toward `keep-last` (dry-run kept 2 real + the phantom) so there is **no retention/data-loss bug** — but one accumulates per aborted upload, forever | READY (S) | — | Decide a cleanup path. Deletion on a **customer** datastore is a separate ruling — detection shipped, removal deliberately not automated | CC |
| **F-CRIT-1** | An app that **fails to restart** after a quiesce never alarms on any channel. Two independent causes: `restartAll` discards the error (returns nothing), and `StateStopped` is whitelisted at `main.go:1237` on invariant I1, which is false for the failed-restart path | **READY — HIGHEST** | — | Campaign 8's other HIGH finding, untouched by the F-CRIT-2 fix. Evidence: `audits/CAMPAIGN-8-backup-restore-2026-07-27.md` | CC |
| **F-CRIT-1** | ~~An app that **fails to restart** after a quiesce never alarms on any channel — `restartAll` discarded the error AND `StateStopped` was whitelisted on invariant I1, which the quiesce path had made false~~ | **SHIPPED + PROVEN-LIVE** (controller v0.179.0, 2026-07-28) | — | Both causes fixed. Live on demo-hp: alarmed 9s after grace expiry, banner shows `(stopped)`; a deliberate user stop stayed silent through 9 dead-app scans | — |
| **F-A1** | ~~A restore-test in flight made a healthy backup report as FAILED (HTTP 409 read as a tier failure): breaker armed + operator emailed, on both boxes~~ | **SHIPPED + PROVEN-LIVE** (controller v0.179.0, 2026-07-28) | — | 409 → contention: tier stays DUE, dropped before anything stops (15m), and BLOCKED alarm if contention outlives the agent's 120m ceiling (3h). Hub DB: 409 → **0** operator emails, real failure → **1** | — |
| **R-89** | Retention as a per-customer **commercial** policy on the hub | READY (increment 2) | — | Policy object + reconciler → ep0 prune job; keep box tokens write-only | CC |
| **R-92** | Hub PBS-DR gauge is 0.1 GB-granular — small deltas unverifiable | READY (XS) | — | Widen precision when retention becomes customer-visible | CC |
| **R-93** | `drill-r50` is both a blocked customer and the only drift fixture | READY (XS) | — | Retire it for a synthetic fixture, or unblock + silence per-customer | CC |
documentation/runbooks/workspace-CLAUDE.md
+25
View File
@@ -147,3 +147,28 @@ Kept so the old environment can be revived; **not the current setup**.
- Skills were installed as Windows junctions (`mklink /J`) rather than POSIX symlinks.
- `claude-in-chrome` browser automation WAS available there (attaching only to sessions started
after the bridge connected).
### A comment asserting an invariant needs a test pinning it, or it is a wish
**Six instances in this project have shipped guarantees the code did not provide** — each survived
review because the comment read as settled:
| # | Comment | What it claimed | What the code did |
|---|---|---|---|
| 1 | `EffectiveProtected` | a stack was protected | it was not — the samba false alarm |
| 2 | `newestArchiveOn` | *"errors degrade to unknown, never to no-backup"* | the `(time,bool)` signature made that impossible (R-88 Part 2) |
| 3 | R-97a operator-only | the event *"cannot be routed to a customer"* | only configuration stopped it; fixed by a real `operatorOnlyEvents` register |
| 4 | `classifyRunStates` I1 | *"StateStopped means deliberately stopped by the user"* | quiesce stops stacks the same way — a failed restart was silent (F-CRIT-1) |
| 5 | `inflight.go` | *"a caller that cannot acquire DEFERS"* | the backup caller recorded a failure and paged the operator (F-A1) |
| 6 | `quiesce.go` | the agent's 409 *prevents* "a spurious failure" | on the start path it produced one (F-A1) |
Two of these (4 and 5/6) were found by Campaign 8 **on live hardware**, not by review or unit tests
— #4 had a green, red-proofed test suite over a production path that was broken two independent
ways. So:
- If a comment states an invariant, **name the test that pins it**, or write one.
- If an invariant has a stated dependency (*"if either invariant changes, revisit this"*), that is
not a safeguard — nobody revisits. Pin it with a test that fails when the dependency moves.
- Prefer a test that asserts the **consequence** (does the alarm fire?) over one that asserts the
**mechanism** (does suppression expire?). R-97b's Scenario F proved the mechanism and the
consequence was still broken.