Run 264 (650cc8a, a DOCUMENTATION-ONLY commit) failed between two greens of byte-identical gate code.
Not waved away as a flake, because this project's own record is that a "known flake" can be a true
positive.
MEASURED. Every other run this session: 18-34s, log present (HTTP 200). Run 264: 834s (07:12:40 ->
07:26:34 UTC) and GET /actions/jobs/264/logs returns HTTP 500 - "264.log.zst: file does not exist".
The act-runner pod never restarted (0 restarts, 5d17h), so the job hung and was reaped; the runner
did not die.
NOT A GATE FINDING, on four independent facts: the diff from the green before it is Markdown only;
the same content is green two commits later (265, 33s); the gate code is identical across 263/264/265;
and 260-262, which WERE real gate failures, each failed in under 35s WITH a log.
THE CAUSE OF THE HANG IS UNDETERMINED and is deliberately recorded as such. DooPlex was doing heavy
work in that window (139 MB kubectl cp, a go run compiling the whole hub module), which is a
plausible contention story - but 40 cores at load ~5 does not establish it, so it is filed as a
hypothesis rather than asserted as a cause.
THE FINDING THAT MATTERS IS SECOND-ORDER, and it is gates.yml's own purpose turned against it. The
workflow exists because "a detector nobody hears is the defect R-29 filed, rebuilt one layer up", and
its alarm mail says "The failing gate names itself in the run log." There is no run log. An operator
following that sentence finds nothing and cannot tell a reap from a conviction. Unverified and worse:
the alarm step is `if: failure()` and whether it ran at all for a reaped job is unknown - if it did
not, this was a red CI that alarmed nobody.
Fix shapes recorded, none built: surface duration + log-presence in the alarm; an explicit
timeout-minutes under the reap so it fails fast and loudly WITH a log; and one deliberate test of
whether the alarm fires on a reaped job, because until that runs, "CI alarms on failure" is an
assumption.
The gate's third instrument defect and the reusable lesson underneath it: a gate needing a sibling
clone passes on a workstation and is INCONCLUSIVE in CI, and a gate must not shell out to a tool the
CI image may not have. Three red runs (260-262), each of which mailed the operator, before 263 went
green. CONTEXT S-38 gains the two-homes rule.
oobDegraded tested five things and the sixth never arrived.
The agent has emitted `operator_key_configured` on every heartbeat since v0.72.0 — the SAME version
that introduced the `oob` stanza carrying it — and store.HostOOBRow mirrored five of the agent's
eight OOB fields. With no field for it, encoding/json discarded the fact on arrival, so a box with
felhom-sshd active, reachable, a valid config and a configured peer reported `ok` with NO OPERATOR
KEY INSTALLED AT ALL. Not a wrong answer: an answer to a question nobody was asking.
`operator_peer_configured`, which the hub did read, only says the peer IP is in desired-state — that
OOB is MEANT to work, not that entry is possible.
Now decoded: operator_key_configured, plus wg_handshake_age_s and healed_at. The last two ride the
ALERT TEXT and are deliberately NOT in the predicate — widening a check beyond the fact that is now
arriving is how a check stops being read.
SCENARIO F, decided on a measurement rather than a preference. operator_key_configured decodes as a
POINTER: nil = the agent never said, reported distinctly and never as ok. The version gate was
rejected because the field and its stanza shipped in the SAME agent version (v0.72.0), so a stanza
without the field cannot come from any released agent; the fleet is 0.113.0/0.127.0 and the vouched
floor is 0.127.0. Handled explicitly anyway and pinned, because "cannot happen" is a claim this
project has been burned by.
THE MESSAGE NAMES THE FAULT. oobDegradedReason is the single source for both predicate and text, so
the alert can never name a different fault from the one that fired. The old form derived it
separately and had a vocabulary of two — unreachable, or config invalid — with no way to say the key
is missing. The operator reads this at 07:00.
TESTS DRIVE THE DECODE BOUNDARY. Every hub OOB test before this built a HostOOBRow by hand, and a
test written that way CANNOT SEE A FIELD THAT NEVER DECODES — which is how this held a green suite
for five weeks. The pre-existing fixture oobReport() also omitted the field, so those scenarios ran
against a report shape no released agent produces (same family as R-262). Both fixed.
Red-proofs, 8 expected outcomes and 0 wrong, each with the mutation asserted applied: dropping the
field returns the false ok; an unconditional check alerts a healthy box; unknown-as-ok restores the
silent pass.
G-1 CLOSED — scripts/wire_contract_gate.py shipped as ranked, built BEFORE the fixes and seen
failing on 40 fields (documentation/tests/wire-contract-gate-2026-08-08/BEFORE.md). Two instrument
defects the control caught first: a substring false negative (grep -F healed_at matched
privsep_healed_at) and treating dr_recipe as wholly opaque when its top-level sections ARE decoded
through an allow-list that already cost offsite_restic (R-122).
The prompt for this session said "465 emitted tags, eight unreachable". Checked against the repo:
R-260 said "at least eight DECISION-BEARING facts", never eight tags. The real count is 40.
R-260 CLOSED (class gated, sharpest instance fixed). R-247 CLOSED (controller v0.209.0). R-264
MINTED and OPEN — the 21 facts with no consumer, allowlisted with reasons so that gating the class
could not be mistaken for deciding them. Still open and named: R-246, R-255..R-259, R-261..R-263,
and C7's test-comment half.
Capability map checked: it claims OOB access is implemented, never monitored, so no row was untrue;
what was untrue sat one layer down and the row now records it.
repo_gates --fast: all 8 OK. go build/vet/test green in hub, run separately from this commit.
admin