diff --git a/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md b/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md index f1af43b..d87c128 100644 --- a/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md +++ b/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md @@ -292,8 +292,10 @@ when the run started. ### R-157 — bootrecon's start-ONCE sweep acts on an unsettled snapshot; the boot orphan survives -**New. Register grepped: R-52 is the feature, nothing covers its timing.** Reproduced twice, in two -independent runs, at the same cycle (seeded permutation). Full evidence: +**New. Register grepped: R-52 is the feature, nothing covers its timing.** **INTERMITTENT — 3 of 6 +hard resets (~50%)** across two runs, with an identical signature each time. (An earlier draft called +this deterministic because the failing cycle numbers matched; they matched only because the runner's +RNG is seeded. Intermittency is what a race predicts and a wrong predicate does not.) Full evidence: `../tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md`. A `qm reset` mid-backup brought everything back **except the app half of the DB-backed stack**: diff --git a/documentation/tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md b/documentation/tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md index 71e8c24..91cedd0 100644 --- a/documentation/tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md +++ b/documentation/tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md @@ -1,6 +1,15 @@ # R-157 — bootrecon's start-ONCE sweep acts on an unsettled snapshot, so the boot orphan it exists to recover survives -**Reproduced twice, in two independent runs, at the same cycle.** Controller **0.188.0**, agent 0.119.0. +**INTERMITTENT — 3 failures in 6 hard resets (~50%), across two independent runs.** Controller +**0.188.0**, agent 0.119.0. + +> **Correction to this document's first version.** It said the failure reproduced "at the same cycle +> in both runs — deterministic, not a coincidence". That was wrong, and the correction matters because +> it changes what kind of bug this is. The cycle numbers matched only because the runner's RNG is +> seeded, so both runs drew the same permutation; the *failure* is a coin flip. Run 2b's four hard +> resets went **PASS (c2) → FAIL (c10) → PASS (c18) → FAIL (c26)**, and run 2a went PASS (c2) → +> FAIL (c10). **Intermittency is what a race predicts** — a wrong predicate would fail every time — +> so this strengthens the root cause below rather than weakening it. R-52 built `internal/bootrecon` for exactly this failure (audit F5: `immich-server` and `calibre-web` left `Exited` after an interrupted boot, ten siblings back, those two **still down ~18 h later** — @@ -79,8 +88,11 @@ The preceding two violations in this campaign **were** harness defects and are d - only the app half of a two-container stack is missing, while its database is healthy; - the fixed harness reported it through the check written for precisely this (*"apps did not return within 10 min — that IS the failure"*), not through a seed or canary path; -- it reproduced at cycle 10 in **run 2a and run 2b**. The runner's RNG is seeded (`Random(20260801)`), - so both runs drew the same permutation — the reproduction is deterministic, not a coincidence. +- it recurred **3 times out of 6 hard resets** with an identical signature every time: `rallly` + `Exited (255)` with `rallly-postgres` healthy, `bootrecon` reporting "no boot-orphaned apps" ~5 s + after start, and the container count still churning **after** the sweep. Third occurrence + (2026-08-02 01:05): `Status refresh` 8 → bootrecon at 01:05:13 → refresh 8 → **7** → 8 at + 01:05:19-20. ## Shape of a fix — not applied (the fences forbid it, and a fix mid-run proves a version that did not exist)