From 3d4c5365c1de64ebd2c6ce7c93d7d4e1bfd2d893 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sun, 2 Aug 2026 03:17:03 +0200 Subject: [PATCH] =?UTF-8?q?Campaign=2010:=20correct=20R-157=20=E2=80=94=20?= =?UTF-8?q?the=20failure=20is=20INTERMITTENT=20(3=20of=206),=20not=20deter?= =?UTF-8?q?ministic?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first write-up said R-157 reproduced "at the same cycle in both runs - deterministic, not a coincidence". Wrong. The cycle numbers matched only because the runner's RNG is seeded so both runs drew the same permutation. The failure itself is a coin flip: run 2b's four hard resets went PASS(c2), FAIL(c10), PASS(c18), FAIL(c26); run 2a went PASS(c2), FAIL(c10). Three failures in six. The correction matters because it changes what kind of bug this is, and it strengthens rather than weakens the root cause: intermittency is exactly what a race against container-state settling predicts, whereas a wrong predicate would fail every time. Signature is identical on all three occurrences: rallly Exited 255 with rallly-postgres healthy, bootrecon reporting "no boot-orphaned apps" about 5s after controller start, and the container count still churning after the sweep (third occurrence 01:05: refresh 8, bootrecon 01:05:13, then 8 -> 7 -> 8). --- .../CAMPAIGN-10-two-storage-soak-2026-07-31.md | 6 ++++-- .../r157-bootrecon-start-once-race.md | 18 +++++++++++++++--- 2 files changed, 19 insertions(+), 5 deletions(-) diff --git a/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md b/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md index f1af43b..d87c128 100644 --- a/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md +++ b/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md @@ -292,8 +292,10 @@ when the run started. ### R-157 — bootrecon's start-ONCE sweep acts on an unsettled snapshot; the boot orphan survives -**New. Register grepped: R-52 is the feature, nothing covers its timing.** Reproduced twice, in two -independent runs, at the same cycle (seeded permutation). Full evidence: +**New. Register grepped: R-52 is the feature, nothing covers its timing.** **INTERMITTENT — 3 of 6 +hard resets (~50%)** across two runs, with an identical signature each time. (An earlier draft called +this deterministic because the failing cycle numbers matched; they matched only because the runner's +RNG is seeded. Intermittency is what a race predicts and a wrong predicate does not.) Full evidence: `../tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md`. A `qm reset` mid-backup brought everything back **except the app half of the DB-backed stack**: diff --git a/documentation/tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md b/documentation/tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md index 71e8c24..91cedd0 100644 --- a/documentation/tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md +++ b/documentation/tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md @@ -1,6 +1,15 @@ # R-157 — bootrecon's start-ONCE sweep acts on an unsettled snapshot, so the boot orphan it exists to recover survives -**Reproduced twice, in two independent runs, at the same cycle.** Controller **0.188.0**, agent 0.119.0. +**INTERMITTENT — 3 failures in 6 hard resets (~50%), across two independent runs.** Controller +**0.188.0**, agent 0.119.0. + +> **Correction to this document's first version.** It said the failure reproduced "at the same cycle +> in both runs — deterministic, not a coincidence". That was wrong, and the correction matters because +> it changes what kind of bug this is. The cycle numbers matched only because the runner's RNG is +> seeded, so both runs drew the same permutation; the *failure* is a coin flip. Run 2b's four hard +> resets went **PASS (c2) → FAIL (c10) → PASS (c18) → FAIL (c26)**, and run 2a went PASS (c2) → +> FAIL (c10). **Intermittency is what a race predicts** — a wrong predicate would fail every time — +> so this strengthens the root cause below rather than weakening it. R-52 built `internal/bootrecon` for exactly this failure (audit F5: `immich-server` and `calibre-web` left `Exited` after an interrupted boot, ten siblings back, those two **still down ~18 h later** — @@ -79,8 +88,11 @@ The preceding two violations in this campaign **were** harness defects and are d - only the app half of a two-container stack is missing, while its database is healthy; - the fixed harness reported it through the check written for precisely this (*"apps did not return within 10 min — that IS the failure"*), not through a seed or canary path; -- it reproduced at cycle 10 in **run 2a and run 2b**. The runner's RNG is seeded (`Random(20260801)`), - so both runs drew the same permutation — the reproduction is deterministic, not a coincidence. +- it recurred **3 times out of 6 hard resets** with an identical signature every time: `rallly` + `Exited (255)` with `rallly-postgres` healthy, `bootrecon` reporting "no boot-orphaned apps" ~5 s + after start, and the container count still churning **after** the sweep. Third occurrence + (2026-08-02 01:05): `Status refresh` 8 → bootrecon at 01:05:13 → refresh 8 → **7** → 8 at + 01:05:19-20. ## Shape of a fix — not applied (the fences forbid it, and a fix mid-run proves a version that did not exist)