Campaign 10: correct R-157 — the failure is INTERMITTENT (3 of 6), not deterministic
The first write-up said R-157 reproduced "at the same cycle in both runs - deterministic, not a coincidence". Wrong. The cycle numbers matched only because the runner's RNG is seeded so both runs drew the same permutation. The failure itself is a coin flip: run 2b's four hard resets went PASS(c2), FAIL(c10), PASS(c18), FAIL(c26); run 2a went PASS(c2), FAIL(c10). Three failures in six. The correction matters because it changes what kind of bug this is, and it strengthens rather than weakens the root cause: intermittency is exactly what a race against container-state settling predicts, whereas a wrong predicate would fail every time. Signature is identical on all three occurrences: rallly Exited 255 with rallly-postgres healthy, bootrecon reporting "no boot-orphaned apps" about 5s after controller start, and the container count still churning after the sweep (third occurrence 01:05: refresh 8, bootrecon 01:05:13, then 8 -> 7 -> 8).
This commit is contained in:
@@ -292,8 +292,10 @@ when the run started.
|
||||
|
||||
### R-157 — bootrecon's start-ONCE sweep acts on an unsettled snapshot; the boot orphan survives
|
||||
|
||||
**New. Register grepped: R-52 is the feature, nothing covers its timing.** Reproduced twice, in two
|
||||
independent runs, at the same cycle (seeded permutation). Full evidence:
|
||||
**New. Register grepped: R-52 is the feature, nothing covers its timing.** **INTERMITTENT — 3 of 6
|
||||
hard resets (~50%)** across two runs, with an identical signature each time. (An earlier draft called
|
||||
this deterministic because the failing cycle numbers matched; they matched only because the runner's
|
||||
RNG is seeded. Intermittency is what a race predicts and a wrong predicate does not.) Full evidence:
|
||||
`../tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md`.
|
||||
|
||||
A `qm reset` mid-backup brought everything back **except the app half of the DB-backed stack**:
|
||||
|
||||
+15
-3
@@ -1,6 +1,15 @@
|
||||
# R-157 — bootrecon's start-ONCE sweep acts on an unsettled snapshot, so the boot orphan it exists to recover survives
|
||||
|
||||
**Reproduced twice, in two independent runs, at the same cycle.** Controller **0.188.0**, agent 0.119.0.
|
||||
**INTERMITTENT — 3 failures in 6 hard resets (~50%), across two independent runs.** Controller
|
||||
**0.188.0**, agent 0.119.0.
|
||||
|
||||
> **Correction to this document's first version.** It said the failure reproduced "at the same cycle
|
||||
> in both runs — deterministic, not a coincidence". That was wrong, and the correction matters because
|
||||
> it changes what kind of bug this is. The cycle numbers matched only because the runner's RNG is
|
||||
> seeded, so both runs drew the same permutation; the *failure* is a coin flip. Run 2b's four hard
|
||||
> resets went **PASS (c2) → FAIL (c10) → PASS (c18) → FAIL (c26)**, and run 2a went PASS (c2) →
|
||||
> FAIL (c10). **Intermittency is what a race predicts** — a wrong predicate would fail every time —
|
||||
> so this strengthens the root cause below rather than weakening it.
|
||||
|
||||
R-52 built `internal/bootrecon` for exactly this failure (audit F5: `immich-server` and `calibre-web`
|
||||
left `Exited` after an interrupted boot, ten siblings back, those two **still down ~18 h later** —
|
||||
@@ -79,8 +88,11 @@ The preceding two violations in this campaign **were** harness defects and are d
|
||||
- only the app half of a two-container stack is missing, while its database is healthy;
|
||||
- the fixed harness reported it through the check written for precisely this
|
||||
(*"apps did not return within 10 min — that IS the failure"*), not through a seed or canary path;
|
||||
- it reproduced at cycle 10 in **run 2a and run 2b**. The runner's RNG is seeded (`Random(20260801)`),
|
||||
so both runs drew the same permutation — the reproduction is deterministic, not a coincidence.
|
||||
- it recurred **3 times out of 6 hard resets** with an identical signature every time: `rallly`
|
||||
`Exited (255)` with `rallly-postgres` healthy, `bootrecon` reporting "no boot-orphaned apps" ~5 s
|
||||
after start, and the container count still churning **after** the sweep. Third occurrence
|
||||
(2026-08-02 01:05): `Status refresh` 8 → bootrecon at 01:05:13 → refresh 8 → **7** → 8 at
|
||||
01:05:19-20.
|
||||
|
||||
## Shape of a fix — not applied (the fences forbid it, and a fix mid-run proves a version that did not exist)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user