admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: F-CRIT-1 + F-A1 shipped (controller v0.179.0); invariant rule

Browse Source 
Both marked SHIPPED + PROVEN-LIVE in OPEN-ITEMS and the campaign doc. All three
of Campaign 8's alarm findings are now closed (F-CRIT-1, F-CRIT-2, F-A1).

Adds the standing rule earned by this arc to the versioned workspace CLAUDE.md:
a comment asserting an invariant needs a test pinning it, or it is a wish — with
all six shipped-false-guarantee instances catalogued, and the corollary that a
test should assert the CONSEQUENCE (does the alarm fire?) not the MECHANISM
(does suppression expire?).
This commit is contained in:
authentik Default Admin
2026-07-28 09:48:51 +02:00
parent d0cec9d3f3
commit e168600148
4 changed files with 110 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md
+20 -2
View File
@@ -265,9 +265,9 @@ Neither restored guest was booted — deliberately, see F-OPS below.
| # | Finding | Severity | DEFECT / ARTIFACT |
|---|---|---|---|
| **F-CRIT-1** | An app that fails to restart after a quiesce **never alarms** | **HIGH** | **DEFECT** (compression-independent) |
| ~~**F-CRIT-1**~~ | ~~An app that fails to restart after a quiesce **never alarms**~~ — **FIXED, controller v0.179.0 (2026-07-28), both causes, replayed live** | ~~HIGH~~ | DEFECT — closed |
| ~~**F-CRIT-2**~~ | ~~A failed offsite backup leaves a phantom snapshot that **resets tier freshness**~~ — **FIXED, agent v0.106.0 (2026-07-28), re-tested live by replaying fault 2** | ~~HIGH~~ | DEFECT — closed |
| **F-A1** | A restore-test in progress makes a healthy backup report as FAILED, arms the breaker and pages the operator | MEDIUM | **DEFECT** in behaviour / ARTIFACT in frequency |
| ~~**F-A1**~~ | ~~A restore-test in progress makes a healthy backup report as FAILED~~ — **FIXED, controller v0.179.0 (2026-07-28); hub DB shows 0 spurious emails, real failures still alarm** | ~~MEDIUM~~ | DEFECT — closed |
| **F-HUB** | The hub **dropped an event** under concurrent load (`SQLITE_BUSY`), no retry, cause unnamed | MEDIUM | DEFECT (frequency compression-influenced) |
| **F-LEAK** | A **failed** restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); leaks are never reclaimed and the 10-slot VMID band shrinks silently | MEDIUM | DEFECT (root-caused by fault 18) |
| **F-REBOOT** | A guest rebooted during its backup **does not come back** — shutdown completes, start never happens, no self-heal | MEDIUM | DEFECT |
@@ -336,6 +336,24 @@ Frequency is an ARTIFACT; the behaviour is a DEFECT — at real cadences a ~12-m
against a daily backup collides roughly once per 420 guest-days, i.e. **every ~4 days on a
100-guest fleet**, forever training R-97a's alarm into noise.
### F-CRIT-1 and F-A1 — FIXED (controller v0.179.0, 2026-07-28)
**F-CRIT-1**, both causes: `restartAll` now returns the stacks that failed, and `classifyRunStates`
lifts the `StateStopped` whitelist for exactly those (`Loop.FailedRestarts()`). I1 is restated
truthfully — `StateStopped` means "deployed with zero containers", which is EITHER a user stop OR a
failed quiesce restart, and only the loop's own knowledge separates them.
**F-A1**: HTTP 409 is contention, not failure — typed `*StatusError` on POST, mapped to
`quiesce.ErrTierBusy` at the adapter. The tier stays DUE and is dropped before anything stops
(`contentionRetryAfter` 15m), and contention outliving the agent's own 120m PBS restore-test ceiling
raises a **BLOCKED** signal at 3h so a wedged gate is never silent.
Replayed live on demo-hp with the hub DB as arbiter: the failed restart alarmed 9 s after grace
expiry with the banner naming `(stopped)`; a deliberate user stop stayed silent through 9 dead-app
scans; a 409 produced **0** operator emails (8 → 8) while a real transport failure produced **1**
(8 → 9). Six red-proofs. Detail: `felhom-controller/REPORT.md`.
**All three of Campaign 8's HIGH/MEDIUM alarm findings are now closed** (F-CRIT-1, F-CRIT-2, F-A1).
### F-CRIT-2 — FIXED (agent v0.106.0, 2026-07-28)
`NewestArchiveTime` now counts only *plausibly complete* entries, via a measured size floor
(`minPlausibleArchiveBytes` = 1 MiB; smallest real backup on the fleet is 612,397,450 B, so 584x
documentation/backlog/OPEN-ITEMS.md
+2 -1
View File
@@ -24,7 +24,8 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
| **R-97** | ~~Whole-guest backup tier had no hub signal; quiesce blamed the apps~~ | **SHIPPED** (controller v0.177.0 + hub v0.78.0/v0.79.0, 2026-07-27) | — | v0.79.0 (R-97c) replaced a FALSE operator-only comment with a real `operatorOnlyEvents` register | — |
| **F-CRIT-2** | ~~A failed offsite backup left a phantom snapshot (1 B, manifest-less, NEWEST) that RESET the tier's freshness clock — 7 days silent on the real 168h cadence, invisible to both the R-88 breaker and the hub deadline monitor~~ | **SHIPPED + PROVEN-LIVE** (agent v0.106.0, 2026-07-28) | — | `NewestArchiveTime` now counts only plausibly-complete entries (measured 1 MiB floor; undecidable ⇒ not counted). Campaign fault 2 replayed on demo-hp: phantom rejected + logged once, tier correctly DUE and backed up, and **no thrash** on the inverse | — |
| **R-99** | Server-side prune **never removes** a phantom snapshot. Confirmed it does NOT count them toward `keep-last` (dry-run kept 2 real + the phantom) so there is **no retention/data-loss bug** — but one accumulates per aborted upload, forever | READY (S) | — | Decide a cleanup path. Deletion on a **customer** datastore is a separate ruling — detection shipped, removal deliberately not automated | CC |
| **F-CRIT-1** | An app that **fails to restart** after a quiesce never alarms on any channel. Two independent causes: `restartAll` discards the error (returns nothing), and `StateStopped` is whitelisted at `main.go:1237` on invariant I1, which is false for the failed-restart path | **READY — HIGHEST** | — | Campaign 8's other HIGH finding, untouched by the F-CRIT-2 fix. Evidence: `audits/CAMPAIGN-8-backup-restore-2026-07-27.md` | CC |
| **F-CRIT-1** | ~~An app that **fails to restart** after a quiesce never alarms on any channel — `restartAll` discarded the error AND `StateStopped` was whitelisted on invariant I1, which the quiesce path had made false~~ | **SHIPPED + PROVEN-LIVE** (controller v0.179.0, 2026-07-28) | — | Both causes fixed. Live on demo-hp: alarmed 9s after grace expiry, banner shows `(stopped)`; a deliberate user stop stayed silent through 9 dead-app scans | — |
| **F-A1** | ~~A restore-test in flight made a healthy backup report as FAILED (HTTP 409 read as a tier failure): breaker armed + operator emailed, on both boxes~~ | **SHIPPED + PROVEN-LIVE** (controller v0.179.0, 2026-07-28) | — | 409 → contention: tier stays DUE, dropped before anything stops (15m), and BLOCKED alarm if contention outlives the agent's 120m ceiling (3h). Hub DB: 409 → **0** operator emails, real failure → **1** | — |
| **R-89** | Retention as a per-customer **commercial** policy on the hub | READY (increment 2) | — | Policy object + reconciler → ep0 prune job; keep box tokens write-only | CC |
| **R-92** | Hub PBS-DR gauge is 0.1 GB-granular — small deltas unverifiable | READY (XS) | — | Widen precision when retention becomes customer-visible | CC |
| **R-93** | `drill-r50` is both a blocked customer and the only drift fixture | READY (XS) | — | Retire it for a synthetic fixture, or unblock + silence per-customer | CC |
documentation/runbooks/workspace-CLAUDE.md
+25
View File
@@ -147,3 +147,28 @@ Kept so the old environment can be revived; **not the current setup**.
- Skills were installed as Windows junctions (`mklink /J`) rather than POSIX symlinks.
- `claude-in-chrome` browser automation WAS available there (attaching only to sessions started
after the bridge connected).
### A comment asserting an invariant needs a test pinning it, or it is a wish
**Six instances in this project have shipped guarantees the code did not provide** — each survived
review because the comment read as settled:
| # | Comment | What it claimed | What the code did |
|---|---|---|---|
| 1 | `EffectiveProtected` | a stack was protected | it was not — the samba false alarm |
| 2 | `newestArchiveOn` | *"errors degrade to unknown, never to no-backup"* | the `(time,bool)` signature made that impossible (R-88 Part 2) |
| 3 | R-97a operator-only | the event *"cannot be routed to a customer"* | only configuration stopped it; fixed by a real `operatorOnlyEvents` register |
| 4 | `classifyRunStates` I1 | *"StateStopped means deliberately stopped by the user"* | quiesce stops stacks the same way — a failed restart was silent (F-CRIT-1) |
| 5 | `inflight.go` | *"a caller that cannot acquire DEFERS"* | the backup caller recorded a failure and paged the operator (F-A1) |
| 6 | `quiesce.go` | the agent's 409 *prevents* "a spurious failure" | on the start path it produced one (F-A1) |
Two of these (4 and 5/6) were found by Campaign 8 **on live hardware**, not by review or unit tests
— #4 had a green, red-proofed test suite over a production path that was broken two independent
ways. So:
- If a comment states an invariant, **name the test that pins it**, or write one.
- If an invariant has a stated dependency (*"if either invariant changes, revisit this"*), that is
not a safeguard — nobody revisits. Pin it with a test that fails when the dependency moves.
- Prefer a test that asserts the **consequence** (does the alarm fire?) over one that asserts the
**mechanism** (does suppression expire?). R-97b's Scenario F proved the mechanism and the
consequence was still broken.