Campaign 10: R-157 — bootrecon's start-once sweep misses the boot orphan it exists to recover
Reproduced twice, two independent runs, same cycle (the runner's RNG is seeded so both drew the same permutation - deterministic, not coincidence). A hard reset mid-backup brought everything back except the app half of the DB-backed stack: rallly left Exited 255, oom=false, restarts=0, its own log ending "Ready" - it died healthy - while rallly-postgres returned healthy. 20:28:13 Status refresh: 8 containers across 55 stacks <-- docker ps -a shows NINE 20:28:18 [bootrecon] Boot reconciliation: no boot-orphaned apps 20:28:25 Status refresh: 7 ... 8 containers <-- still churning AFTER the sweep 20:39:14 [deadapp] 20 scans, 5 deployed evaluated, 1 currently down The predicate is sound: once settled the controller reports rallly state=degraded containers=2, and IsDownState includes StateDegraded, so len>0 && IsDownState holds. The SNAPSHOT was wrong. bootrecon fires as a goroutine ~5s after start while docker is still restoring containers, and is start-once by design, so it never re-checks. Consequence: the app stays down indefinitely. Detection is perfect and recovery never happens - R-52's original shape, an alarm with no recovery. Not fixed. Distinguished from this campaign's two earlier HARNESS defects: both drives bound, every other app returned incl. the drive-backed one, only the app half of a two-container stack missing while its DB is healthy, and it surfaced through the fixed check written for exactly this.
This commit is contained in:
@@ -290,6 +290,32 @@ defence is a catalog gate asserting each template's mounted volume is a path the
|
||||
**Not fixed here** — the fences forbid it, and a fix mid-run would prove a version that did not exist
|
||||
when the run started.
|
||||
|
||||
### R-157 — bootrecon's start-ONCE sweep acts on an unsettled snapshot; the boot orphan survives
|
||||
|
||||
**New. Register grepped: R-52 is the feature, nothing covers its timing.** Reproduced twice, in two
|
||||
independent runs, at the same cycle (seeded permutation). Full evidence:
|
||||
`../tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md`.
|
||||
|
||||
A `qm reset` mid-backup brought everything back **except the app half of the DB-backed stack**:
|
||||
`rallly` left `Exited (255)` (`oom=false`, `restarts=0`, its own log ending `✓ Ready` — it died
|
||||
healthy) while `rallly-postgres` returned healthy.
|
||||
|
||||
20:28:13 Status refresh: 8 containers across 55 stacks <-- docker ps -a shows NINE
|
||||
20:28:18 [bootrecon] Boot reconciliation: no boot-orphaned apps (nothing to start)
|
||||
20:28:25 Status refresh: 7 containers … 8 containers <-- still churning AFTER the sweep
|
||||
20:39:14 [deadapp] 20 scans since boot, 5 deployed app(s) evaluated, 1 currently down
|
||||
|
||||
The predicate is not at fault — once settled the controller reports `rallly state=degraded
|
||||
containers=2`, and `IsDownState` includes `StateDegraded` (`manager.go:55`), so
|
||||
`len(Containers)>0 && IsDownState(State)` holds. **The snapshot was**: bootrecon fires as
|
||||
`go runBootReconcile(...)` (`cmd/controller/main.go:236`) ~5 s after start, while docker is still
|
||||
restoring containers, and is start-once by design so it never re-checks.
|
||||
|
||||
**Consequence: the app stays down indefinitely.** Detection is perfect and recovery never happens —
|
||||
exactly R-52's original shape, *an alarm with no recovery* (F5: two apps Exited for ~18 h). In run 2a
|
||||
the app only returned because a later campaign atom redeployed it. **Not fixed** — the fences forbid
|
||||
it, and the settle-condition fix belongs with a test that pins the *consequence*.
|
||||
|
||||
### 5.2 — Investigated and DISPROVED: `/api/disks` reports `state=attached` for an absent drive
|
||||
|
||||
Recorded because the *chase* is the useful artefact, not because anything is broken.
|
||||
|
||||
Reference in New Issue
Block a user