R-265: a CI run failed with NO LOG, and the alarm points at a log that does not exist
gates / gates (push) Successful in 20s

Run 264 (650cc8a, a DOCUMENTATION-ONLY commit) failed between two greens of byte-identical gate code.
Not waved away as a flake, because this project's own record is that a "known flake" can be a true
positive.

MEASURED. Every other run this session: 18-34s, log present (HTTP 200). Run 264: 834s (07:12:40 ->
07:26:34 UTC) and GET /actions/jobs/264/logs returns HTTP 500 - "264.log.zst: file does not exist".
The act-runner pod never restarted (0 restarts, 5d17h), so the job hung and was reaped; the runner
did not die.

NOT A GATE FINDING, on four independent facts: the diff from the green before it is Markdown only;
the same content is green two commits later (265, 33s); the gate code is identical across 263/264/265;
and 260-262, which WERE real gate failures, each failed in under 35s WITH a log.

THE CAUSE OF THE HANG IS UNDETERMINED and is deliberately recorded as such. DooPlex was doing heavy
work in that window (139 MB kubectl cp, a go run compiling the whole hub module), which is a
plausible contention story - but 40 cores at load ~5 does not establish it, so it is filed as a
hypothesis rather than asserted as a cause.

THE FINDING THAT MATTERS IS SECOND-ORDER, and it is gates.yml's own purpose turned against it. The
workflow exists because "a detector nobody hears is the defect R-29 filed, rebuilt one layer up", and
its alarm mail says "The failing gate names itself in the run log." There is no run log. An operator
following that sentence finds nothing and cannot tell a reap from a conviction. Unverified and worse:
the alarm step is `if: failure()` and whether it ran at all for a reaped job is unknown - if it did
not, this was a red CI that alarmed nobody.

Fix shapes recorded, none built: surface duration + log-presence in the alarm; an explicit
timeout-minutes under the reap so it fails fast and loudly WITH a log; and one deliberate test of
whether the alarm fires on a reaped job, because until that runs, "CI alarms on failure" is an
assumption.
This commit is contained in:
2026-08-08 09:29:51 +02:00
parent dd55a3f98c
commit 4f5784894e
2 changed files with 14 additions and 0 deletions
+12
View File
@@ -107,6 +107,18 @@ that is the fail-open shape and would leave it running in neither home (R-29).
mail before run **263** went green. The alarm working is the system behaving correctly; the noise was
mine.
**A FOURTH red run, 264, was NOT one of mine and is filed as R-265.** It sat between two greens on a
**documentation-only** commit, ran **834 s** against 1834 s for every other run in the session, and
**persisted no log at all** (`jobs/264/logs` → HTTP 500, *file does not exist*). The runner pod never
restarted, so the job hung and was reaped rather than the runner dying. Not a gate finding — the diff
was Markdown, the gate code was byte-identical to the two greens around it, and the same content is
green at run 265. **The cause of the hang is undetermined and is not guessed at**; DooPlex was busy in
that window with this session's own live-validation work, but the box has 40 cores at load ~5, so
that is a hypothesis, not a cause. **The reusable finding is second-order:** the alarm mail tells the
operator "the failing gate names itself in the run log", and here there is no run log — so a reap is
indistinguishable from a conviction, and whether the alarm fired at all for a reaped job is
**unverified**. R-265 carries it.
## 6. What `oobDegraded` says when it fails
```