admin
4f5784894e
Run 264 (650cc8a, a DOCUMENTATION-ONLY commit) failed between two greens of byte-identical gate code.
Not waved away as a flake, because this project's own record is that a "known flake" can be a true
positive.
MEASURED. Every other run this session: 18-34s, log present (HTTP 200). Run 264: 834s (07:12:40 ->
07:26:34 UTC) and GET /actions/jobs/264/logs returns HTTP 500 - "264.log.zst: file does not exist".
The act-runner pod never restarted (0 restarts, 5d17h), so the job hung and was reaped; the runner
did not die.
NOT A GATE FINDING, on four independent facts: the diff from the green before it is Markdown only;
the same content is green two commits later (265, 33s); the gate code is identical across 263/264/265;
and 260-262, which WERE real gate failures, each failed in under 35s WITH a log.
THE CAUSE OF THE HANG IS UNDETERMINED and is deliberately recorded as such. DooPlex was doing heavy
work in that window (139 MB kubectl cp, a go run compiling the whole hub module), which is a
plausible contention story - but 40 cores at load ~5 does not establish it, so it is filed as a
hypothesis rather than asserted as a cause.
THE FINDING THAT MATTERS IS SECOND-ORDER, and it is gates.yml's own purpose turned against it. The
workflow exists because "a detector nobody hears is the defect R-29 filed, rebuilt one layer up", and
its alarm mail says "The failing gate names itself in the run log." There is no run log. An operator
following that sentence finds nothing and cannot tell a reap from a conviction. Unverified and worse:
the alarm step is `if: failure()` and whether it ran at all for a reaped job is unknown - if it did
not, this was a red CI that alarmed nobody.
Fix shapes recorded, none built: surface duration + log-presence in the alarm; an explicit
timeout-minutes under the reap so it fails fast and loudly WITH a log; and one deliberate test of
whether the alarm fires on a reaped job, because until that runs, "CI alarms on failure" is an
assumption.
documentation/backlog/
OPEN-ITEMS.md is the register of open work and the file to read first — it holds only what is
open, one row per item, every row with a state and an owner. ROADMAP.md is the full history and
reasoning behind the R-n IDs, including shipped and killed items; an ID is minted there, and a new
instance of an existing item attaches to that ID rather than getting its own.
The rest of this folder: verified-LIVE findings with implementable fix plans that are not yet
implemented. Preserved here
(instead of on git branches) per the trunk-based, no-branches rule — the fix itself is implemented later
directly on main, during a normal/supervised session.
-
FIX-M18-NOTES.md — dump re-validation runs every 5 min (perf). FIXED in controller v0.62.0 @
f8afe5c(2026-06-14). (was on the deletedfelhom-controllerbranchfix/m18-dump-validation-cache.) -
FIX-M19-NOTES.md —
deriveStackNamemisattribution edge (low-incidence correctness). FIXED in controller v0.62.0 @6bab68b(2026-06-14). (was on the deleted branchfix/m19-stackname-crossref.) -
FOLLOWUP-golden-default-controller-tag.md — the golden bakes a stale controller (
:0.43.0when queued; had rotted again to:0.85.1by resolution). FIXED in felhom-agent @ceca355(2026-07-03):build-golden.shv2.0.0 makes the controller tag a MANDATORY argument (a required arg cannot rot) and golden 0.98.3 was baked + clean-room-validated (bake → first-boot-current → self-manage → app deploy, on the drill VM — no supervised touch of live guests needed) + published + vouched. Evidence:../audits/DRILL-golden-098-2026-07-03.md.
Related: the live-drive fixspec (../audits/live-drive-fixspec-2026-06-14.md) carries the deferred
supervised items F9 (HDD provisioning/guest-attach), F20-BUG2 (durable_id scheme), F20-BUG3 (async
mkfs) — to be implemented in the agent/golden supervised session.