The 4th hard-reset failure had a different signature, verified not assumed: all of
rallly healthy, papra missing entirely with state=stopped deployed=True
containers=0. Zero containers is exactly what bootrecon deliberately never touches,
because the UI's Stop is compose down which removes containers - but a hard reset
landing during a compose operation produces the identical state. The signature the
safety rule depends on cannot distinguish the two.
Worse: in that state the deadapp check reported 0 currently down while a deployed
app was not running. No app_start_failed, no banner. That is the workspace's own
false-invariant #4 (F-CRIT-1, StateStopped assumed deliberate) recurring through a
hard reset rather than quiesce. NOT filed as new - CLAUDE.md already records it -
but confirmed live on 0.188.0 via a new path.
papra returned after ~15 min, later than the harness's 10-min window, so this
instance was slow rather than permanent and the doc says so. What restarted it is
not established.
Mechanism A (Exited, missed by the unsettled snapshot) alarms; mechanism B (zero
containers) is invisible on every channel. A settle-condition fix closes A only.
The first write-up said R-157 reproduced "at the same cycle in both runs -
deterministic, not a coincidence". Wrong. The cycle numbers matched only because
the runner's RNG is seeded so both runs drew the same permutation. The failure
itself is a coin flip: run 2b's four hard resets went PASS(c2), FAIL(c10),
PASS(c18), FAIL(c26); run 2a went PASS(c2), FAIL(c10). Three failures in six.
The correction matters because it changes what kind of bug this is, and it
strengthens rather than weakens the root cause: intermittency is exactly what a
race against container-state settling predicts, whereas a wrong predicate would
fail every time.
Signature is identical on all three occurrences: rallly Exited 255 with
rallly-postgres healthy, bootrecon reporting "no boot-orphaned apps" about 5s
after controller start, and the container count still churning after the sweep
(third occurrence 01:05: refresh 8, bootrecon 01:05:13, then 8 -> 7 -> 8).
Reproduced twice, two independent runs, same cycle (the runner's RNG is seeded so
both drew the same permutation - deterministic, not coincidence).
A hard reset mid-backup brought everything back except the app half of the
DB-backed stack: rallly left Exited 255, oom=false, restarts=0, its own log ending
"Ready" - it died healthy - while rallly-postgres returned healthy.
20:28:13 Status refresh: 8 containers across 55 stacks <-- docker ps -a shows NINE
20:28:18 [bootrecon] Boot reconciliation: no boot-orphaned apps
20:28:25 Status refresh: 7 ... 8 containers <-- still churning AFTER the sweep
20:39:14 [deadapp] 20 scans, 5 deployed evaluated, 1 currently down
The predicate is sound: once settled the controller reports rallly state=degraded
containers=2, and IsDownState includes StateDegraded, so len>0 && IsDownState
holds. The SNAPSHOT was wrong. bootrecon fires as a goroutine ~5s after start
while docker is still restoring containers, and is start-once by design, so it
never re-checks.
Consequence: the app stays down indefinitely. Detection is perfect and recovery
never happens - R-52's original shape, an alarm with no recovery. Not fixed.
Distinguished from this campaign's two earlier HARNESS defects: both drives bound,
every other app returned incl. the drive-backed one, only the app half of a
two-container stack missing while its DB is healthy, and it surfaced through the
fixed check written for exactly this.
admin