admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Campaign 10: correct R-157 — the failure is INTERMITTENT (3 of 6), not deterministic

Browse Source 
The first write-up said R-157 reproduced "at the same cycle in both runs -
deterministic, not a coincidence". Wrong. The cycle numbers matched only because
the runner's RNG is seeded so both runs drew the same permutation. The failure
itself is a coin flip: run 2b's four hard resets went PASS(c2), FAIL(c10),
PASS(c18), FAIL(c26); run 2a went PASS(c2), FAIL(c10). Three failures in six.

The correction matters because it changes what kind of bug this is, and it
strengthens rather than weakens the root cause: intermittency is exactly what a
race against container-state settling predicts, whereas a wrong predicate would
fail every time.

Signature is identical on all three occurrences: rallly Exited 255 with
rallly-postgres healthy, bootrecon reporting "no boot-orphaned apps" about 5s
after controller start, and the container count still churning after the sweep
(third occurrence 01:05: refresh 8, bootrecon 01:05:13, then 8 -> 7 -> 8).
This commit is contained in:
authentik Default Admin
2026-08-02 03:17:03 +02:00
parent 7f6b00375b
commit 3d4c5365c1
2 changed files with 19 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md
+4 -2
View File
@@ -292,8 +292,10 @@ when the run started.
### R-157 — bootrecon's start-ONCE sweep acts on an unsettled snapshot; the boot orphan survives ### R-157 — bootrecon's start-ONCE sweep acts on an unsettled snapshot; the boot orphan survives
**New. Register grepped: R-52 is the feature, nothing covers its timing.** Reproduced twice, in two **New. Register grepped: R-52 is the feature, nothing covers its timing.** **INTERMITTENT — 3 of 6
independent runs, at the same cycle (seeded permutation). Full evidence: hard resets (~50%)** across two runs, with an identical signature each time. (An earlier draft called
this deterministic because the failing cycle numbers matched; they matched only because the runner's
RNG is seeded. Intermittency is what a race predicts and a wrong predicate does not.) Full evidence:
`../tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md`. `../tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md`.
A `qm reset` mid-backup brought everything back **except the app half of the DB-backed stack**: A `qm reset` mid-backup brought everything back **except the app half of the DB-backed stack**:
documentation/tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md
+15 -3
View File
@@ -1,6 +1,15 @@
# R-157 — bootrecon's start-ONCE sweep acts on an unsettled snapshot, so the boot orphan it exists to recover survives # R-157 — bootrecon's start-ONCE sweep acts on an unsettled snapshot, so the boot orphan it exists to recover survives
**Reproduced twice, in two independent runs, at the same cycle.** Controller **0.188.0**, agent 0.119.0. **INTERMITTENT — 3 failures in 6 hard resets (~50%), across two independent runs.** Controller
**0.188.0**, agent 0.119.0.
> **Correction to this document's first version.** It said the failure reproduced "at the same cycle
> in both runs — deterministic, not a coincidence". That was wrong, and the correction matters because
> it changes what kind of bug this is. The cycle numbers matched only because the runner's RNG is
> seeded, so both runs drew the same permutation; the *failure* is a coin flip. Run 2b's four hard
> resets went **PASS (c2) → FAIL (c10) → PASS (c18) → FAIL (c26)**, and run 2a went PASS (c2) →
> FAIL (c10). **Intermittency is what a race predicts** — a wrong predicate would fail every time —
> so this strengthens the root cause below rather than weakening it.
R-52 built `internal/bootrecon` for exactly this failure (audit F5: `immich-server` and `calibre-web` R-52 built `internal/bootrecon` for exactly this failure (audit F5: `immich-server` and `calibre-web`
left `Exited` after an interrupted boot, ten siblings back, those two **still down ~18 h later** — left `Exited` after an interrupted boot, ten siblings back, those two **still down ~18 h later** —
@@ -79,8 +88,11 @@ The preceding two violations in this campaign **were** harness defects and are d
- only the app half of a two-container stack is missing, while its database is healthy; - only the app half of a two-container stack is missing, while its database is healthy;
- the fixed harness reported it through the check written for precisely this - the fixed harness reported it through the check written for precisely this
(*"apps did not return within 10 min — that IS the failure"*), not through a seed or canary path; (*"apps did not return within 10 min — that IS the failure"*), not through a seed or canary path;
- it reproduced at cycle 10 in **run 2a and run 2b**. The runner's RNG is seeded (`Random(20260801)`), - it recurred **3 times out of 6 hard resets** with an identical signature every time: `rallly`
so both runs drew the same permutation — the reproduction is deterministic, not a coincidence. `Exited (255)` with `rallly-postgres` healthy, `bootrecon` reporting "no boot-orphaned apps" ~5 s
after start, and the container count still churning **after** the sweep. Third occurrence
(2026-08-02 01:05): `Status refresh` 8 → bootrecon at 01:05:13 → refresh 8 → **7** → 8 at
01:05:19-20.
## Shape of a fix — not applied (the fences forbid it, and a fix mid-run proves a version that did not exist) ## Shape of a fix — not applied (the fences forbid it, and a fix mid-run proves a version that did not exist)