Campaign 10: R-157 — bootrecon's start-once sweep misses the boot orphan it exists to recover

Reproduced twice, two independent runs, same cycle (the runner's RNG is seeded so
both drew the same permutation - deterministic, not coincidence).

A hard reset mid-backup brought everything back except the app half of the
DB-backed stack: rallly left Exited 255, oom=false, restarts=0, its own log ending
"Ready" - it died healthy - while rallly-postgres returned healthy.

  20:28:13 Status refresh: 8 containers across 55 stacks   <-- docker ps -a shows NINE
  20:28:18 [bootrecon] Boot reconciliation: no boot-orphaned apps
  20:28:25 Status refresh: 7 ... 8 containers              <-- still churning AFTER the sweep
  20:39:14 [deadapp] 20 scans, 5 deployed evaluated, 1 currently down

The predicate is sound: once settled the controller reports rallly state=degraded
containers=2, and IsDownState includes StateDegraded, so len>0 && IsDownState
holds. The SNAPSHOT was wrong. bootrecon fires as a goroutine ~5s after start
while docker is still restoring containers, and is start-once by design, so it
never re-checks.

Consequence: the app stays down indefinitely. Detection is perfect and recovery
never happens - R-52's original shape, an alarm with no recovery. Not fixed.

Distinguished from this campaign's two earlier HARNESS defects: both drives bound,
every other app returned incl. the drive-backed one, only the app half of a
two-container stack missing while its DB is healthy, and it surfaced through the
fixed check written for exactly this.
This commit is contained in:
2026-08-01 22:45:01 +02:00
parent 80db2c103a
commit 7f6b00375b
2 changed files with 123 additions and 0 deletions
@@ -290,6 +290,32 @@ defence is a catalog gate asserting each template's mounted volume is a path the
**Not fixed here** — the fences forbid it, and a fix mid-run would prove a version that did not exist
when the run started.
### R-157 — bootrecon's start-ONCE sweep acts on an unsettled snapshot; the boot orphan survives
**New. Register grepped: R-52 is the feature, nothing covers its timing.** Reproduced twice, in two
independent runs, at the same cycle (seeded permutation). Full evidence:
`../tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md`.
A `qm reset` mid-backup brought everything back **except the app half of the DB-backed stack**:
`rallly` left `Exited (255)` (`oom=false`, `restarts=0`, its own log ending `✓ Ready` — it died
healthy) while `rallly-postgres` returned healthy.
20:28:13 Status refresh: 8 containers across 55 stacks <-- docker ps -a shows NINE
20:28:18 [bootrecon] Boot reconciliation: no boot-orphaned apps (nothing to start)
20:28:25 Status refresh: 7 containers … 8 containers <-- still churning AFTER the sweep
20:39:14 [deadapp] 20 scans since boot, 5 deployed app(s) evaluated, 1 currently down
The predicate is not at fault — once settled the controller reports `rallly state=degraded
containers=2`, and `IsDownState` includes `StateDegraded` (`manager.go:55`), so
`len(Containers)>0 && IsDownState(State)` holds. **The snapshot was**: bootrecon fires as
`go runBootReconcile(...)` (`cmd/controller/main.go:236`) ~5 s after start, while docker is still
restoring containers, and is start-once by design so it never re-checks.
**Consequence: the app stays down indefinitely.** Detection is perfect and recovery never happens —
exactly R-52's original shape, *an alarm with no recovery* (F5: two apps Exited for ~18 h). In run 2a
the app only returned because a later campaign atom redeployed it. **Not fixed** — the fences forbid
it, and the settle-condition fix belongs with a test that pins the *consequence*.
### 5.2 — Investigated and DISPROVED: `/api/disks` reports `state=attached` for an absent drive
Recorded because the *chase* is the useful artefact, not because anything is broken.