admin
4f5784894e
Run 264 (650cc8a, a DOCUMENTATION-ONLY commit) failed between two greens of byte-identical gate code.
Not waved away as a flake, because this project's own record is that a "known flake" can be a true
positive.
MEASURED. Every other run this session: 18-34s, log present (HTTP 200). Run 264: 834s (07:12:40 ->
07:26:34 UTC) and GET /actions/jobs/264/logs returns HTTP 500 - "264.log.zst: file does not exist".
The act-runner pod never restarted (0 restarts, 5d17h), so the job hung and was reaped; the runner
did not die.
NOT A GATE FINDING, on four independent facts: the diff from the green before it is Markdown only;
the same content is green two commits later (265, 33s); the gate code is identical across 263/264/265;
and 260-262, which WERE real gate failures, each failed in under 35s WITH a log.
THE CAUSE OF THE HANG IS UNDETERMINED and is deliberately recorded as such. DooPlex was doing heavy
work in that window (139 MB kubectl cp, a go run compiling the whole hub module), which is a
plausible contention story - but 40 cores at load ~5 does not establish it, so it is filed as a
hypothesis rather than asserted as a cause.
THE FINDING THAT MATTERS IS SECOND-ORDER, and it is gates.yml's own purpose turned against it. The
workflow exists because "a detector nobody hears is the defect R-29 filed, rebuilt one layer up", and
its alarm mail says "The failing gate names itself in the run log." There is no run log. An operator
following that sentence finds nothing and cannot tell a reap from a conviction. Unverified and worse:
the alarm step is `if: failure()` and whether it ran at all for a reaped job is unknown - if it did
not, this was a red CI that alarmed nobody.
Fix shapes recorded, none built: surface duration + log-presence in the alarm; an explicit
timeout-minutes under the reap so it fails fast and loudly WITH a log; and one deliberate test of
whether the alarm fires on a reaped job, because until that runs, "CI alarms on failure" is an
assumption.
12 KiB
REPORT — G-1: a gate for the dropped field, then the fields it found (2026-08-08)
A non-overwritten REPORT-<topic>.md sibling, per CLAUDE.md:82-87 — a parallel session shares this
clone and the shared REPORT.md was not touched.
1. The gate's output on today's tree — failing, before anything was fixed
This is the session. Captured verbatim in
documentation/tests/wire-contract-gate-2026-08-08/BEFORE.md:
wire-contract gate — 210 tag(s) checked across 3 declared wire(s); 51 skipped
WIRE-CONTRACT GATE FAILED: 40 emitted field(s) cannot be received.
It named every one, with its emit path and its direction, and re-found escrow_stale (R-247) and
every field R-260 listed. Had it been green, the gate would not work and that would have been the
finding — which is not hypothetical: the night before, deadcode was rejected for the neighbouring
C6 class for exactly that reason.
⚠ A count this session's prompt got wrong. The prompt said "465 emitted tags, eight unreachable". R-260's wording was "at least eight decision-bearing facts", never eight tags in total. Measured: 40 on the three declared wires. Checked against the repo, not quoted — the prompt's own rule 6, and the second prompt claim caught that way this week.
2. The forty, by disposition
| # | field(s) | direction | decision | what changed |
|---|---|---|---|---|
| 1 | oob.operator_key_configured |
agent → hub | receive and act | decoded (pointer); oobDegraded fails on a missing key and the alert names it |
| 2 | oob.wg_handshake_age_s, oob.healed_at |
agent → hub | receive, message only | in HostOOBRow + the event payload; deliberately NOT in the predicate |
| 3 | escrow_stale |
hub → controller | receive and act | report.EscrowStatus.Stale; withheld-hash told apart from hash-less. R-247 |
| 4 | 12 host/system metric fields | both → hub | no consumer wanted — redundant | allowlisted: the hub bands on the *_percent figures from the same stanzas |
| 5 | guests.spec.{disk_bytes,memory_bytes} |
agent → hub | redundant | sizing is hub-owned intent, not mirrored reality |
| 6 | storage_targets.smart.model_name |
agent → hub | redundant | a display label; smart.health + every banded counter ARE decoded |
| 7 | wireguard.last_handshake_age_s |
agent → hub | redundant | wgsync reconciles from its own state |
| 8 | 21 fields (guest_net+7, selfupdate_pending+1, healed_recently, applied_at, mount_parity/_inventory, config_hash, reporting_disabled, stacks, migrated_to, last_db_dump, last_integrity_check) |
both → hub | no consumer today, one arguably owed | allowlisted against R-264, OPEN. Allowlisting is not deciding, and the entries say so |
Full per-field reasons are in the gate's own ALLOWLIST, each a claim someone can re-check.
3. Scenario F — the choice, and why
Unknown is reported distinctly and is never ok. operator_key_configured decodes as a
pointer: nil = the agent never said, which is not a value.
The version gate the prompt thought "probably right" was rejected on a measurement: the field and
the oob stanza that carries it shipped in the same agent version (v0.72.0, 2026-07-05), so a
stanza without the field cannot come from any released agent. The live fleet is 0.113.0 and 0.127.0;
the vouched floor is 0.127.0. Building version-gating machinery the hub does not otherwise have, for a
state no box can be in, is cost without cover. The case is still handled explicitly and pinned by a
test, because "cannot happen" is a claim this project has been burned by.
4. R-247 — CLOSED
The field is received, and reconcileEscrowed tells a withheld hash from a hash-less one.
Controller v0.209.0.
Deliberately not folded in, and said rather than skipped: the wrong flag on demo-hp is an
operator act hub-side (R-246, still open), and the customer-facing Hungarian card copy is
unchanged — that is UI work with its own review path.
5. The gate's blind spots, and its self-test
Published in the module docstring and in the gate's own output, because Campaign 12's C1 guard turned out blind to one of the three shapes it was written for:
- generic tag names are not checked (
name,state,status, …) — a repo-wide string test says nothing about them, so a drop of a generically-named field is missed; the gate under-reports rather than over-reports; - reachability of a NAME is not use of a VALUE;
- only declared ROOTS are covered — the hub's desired-state (raw stored JSON, no typed emitter) and the agent's local API (no single root) are not;
- it reads source, not traffic; test files and
testdata/are excluded on the receiving side deliberately (a tag present only in a fixture is not decodable — which is R-262 exactly).
--selftest plants an unreachable tag on a real root in a throwaway copy and asserts conviction:
exit 1, planted tag named; unplanted tree exit 0.
THREE instrument defects this gate's own controls caught before it was trusted. None was found by review; each was found by making the gate prove something.
- A substring false negative —
grep -F healed_atalso matchedprivsep_healed_at. R-260 namedhealed_at, so its absence from the output was the tell. Whole-token now; 40, not 39. dr_recipeis not wholly opaque — its top-level section keys ARE decoded, through allow-lists that already swallowedoffsite_resticfor months (R-122). Now opaque only below depth 1.- The search shelled out to
grepand read its failure as a finding. CI convicted all 174 checked tags while the pre-push hook was green. The CI runner's image carries python3 and git and deliberately little else, and itsgrepdoes not support--include, so stdout was empty and empty was read as "absent". A gate that silently turns a tool failure into a finding is worse than no gate, and its green would have been as untrustworthy as its red. Removed the dependency rather than working around it: the search is pure Python now, one token index per receiving repo.
The BEFORE capture was RE-VERIFIED, not re-generated — the stronger claim. All 40 recorded fields were re-tested against the new implementation: agree=40, disagree=0, i.e. exactly the four this session fixed are now present and the other 36 still absent. The number stands under both implementations.
And the reusable half, which is about the gates and not about this gate. The pre-push hook runs on
a workstation where every sibling repo is a real clone; CI checks out one repo, shallow. A gate
that needs a sibling passes locally and is INCONCLUSIVE in CI — the two automated homes are not
interchangeable, and a new gate must be checked in BOTH. The workflow's own alarm mail says a
hook-versus-CI disagreement "outranks whatever the push was for"; it did. Fixed by fetching the agent
clone in CI (.gitea/workflows/gates.yml), never by letting the gate skip when a sibling is absent —
that is the fail-open shape and would leave it running in neither home (R-29).
Cost, stated plainly: three CI runs went red (260, 261, 262) and each sent the operator an alarm mail before run 263 went green. The alarm working is the system behaving correctly; the noise was mine.
A FOURTH red run, 264, was NOT one of mine and is filed as R-265. It sat between two greens on a
documentation-only commit, ran 834 s against 18–34 s for every other run in the session, and
persisted no log at all (jobs/264/logs → HTTP 500, file does not exist). The runner pod never
restarted, so the job hung and was reaped rather than the runner dying. Not a gate finding — the diff
was Markdown, the gate code was byte-identical to the two greens around it, and the same content is
green at run 265. The cause of the hang is undetermined and is not guessed at; DooPlex was busy in
that window with this session's own live-validation work, but the box has 40 cores at load ~5, so
that is a hypothesis, not a cause. The reusable finding is second-order: the alarm mail tells the
operator "the failing gate names itself in the run log", and here there is no run log — so a reap is
indistinguishable from a conviction, and whether the alarm fired at all for a reaped job is
unverified. R-265 carries it.
6. What oobDegraded says when it fails
Host <id>: OPERATOR ACCESS DEGRADED — the operator's authorized_key is NOT installed —
felhom-sshd is up and answering, and nobody can log in through it. The break-glass net
(auto-heal + vaulted root@pam console) is still under the box.
and for the unreachable-but-handled unknown:
… — the agent reports operator access but is too old to say whether the operator key is
installed (pre-v0.72.0) — treat entry as UNPROVEN, not working. …
oobDegradedReason is now the single source for both the predicate and the text, so the message can
never name a different fault from the one that fired. The old form derived it separately and had a
vocabulary of two.
7. Tests and red-proofs
New: hub/internal/store/host_oob_decode_test.go (4 tests, raw JSON at the decode boundary),
hub/internal/monitor/host_oob_operatorkey_test.go (6), plus two end-to-end tests in
host_oob_test.go driving JSON → store → checker → event.
Red-proofs — 8 expected outcomes, 0 wrong, each with the mutation asserted applied:
| mutation | assertion it applied | outcome |
|---|---|---|
| the gate on today's tree | — | RED, naming all 40 ✔ |
| planted unreachable tag (post-fix) | self-test reports the planted tag by name | RED on the plant, GREEN unplanted ✔ |
drop operator_key_configured from the decoder |
json-tag occurrences in the decoder 2 → 1 | RED — the false ok returns ✔ |
| make the check unconditional | MUTATED unconditional degrade marker present |
RED — a healthy box alerts ✔ |
treat unknown as ok |
MUTATED: unknown is silently ok again marker present |
RED — the silent pass returns ✔ |
| all three restored | — | GREEN ✔ |
The pre-existing fixture was part of the defect and was fixed too: oobReport() omitted
operator_key_configured, so every earlier scenario ran against a report shape no released agent
produces. Same family as R-262.
8. The capability-map row about operator access
Checked, and it was NOT claiming something untrue. 00-capability-map.md:127 claims OOB operator
access is implemented, never that it is monitored, so no correction was owed. What was untrue sat
one layer down — the hub's own health check could not see the key — and the row now records that,
with the fix and the tests that pin it.
9. Gates, and what remains
python3 scripts/repo_gates.py --fast → all 8 OK, including the new wire-contract and
golden-currency. go build ./... && go vet ./... && go test ./... green in hub and
controller (run separately from every commit). No --no-verify anywhere.
The one gate failure that remains is not a failure of this work: golden 0.208.0 is baked and byte-verified but still not vouched, so fresh installs receive 0.207.0. That is R-242's untouched half and one operator Save.
10. Register
R-260 CLOSED (class gated + sharpest instance fixed), R-247 CLOSED, G-1 CLOSED in
ROADMAP.md. R-264 minted and OPEN — the twenty-one facts with no consumer, split out so that
gating the class could not be mistaken for deciding them. Highest ID moved R-263 → R-264.
Explicitly still open: R-246, R-255, R-256, R-257, R-258, R-259, R-261, R-262, R-263, and C7's test-comment half, which Campaign 12 recorded as owed, not done.
11. Observations — noticed, NOT acted on
stacksis the whole per-stack report object and the hub decodes none of it. The largest single unconsumed structure on the controller wire; folded into R-264 rather than sized here.- The hub has no version-gating machinery for report fields at all. Not needed today (see §3), but the next additive field whose emitter and stanza do not ship together will need it, and there is no convention to reach for.
backup.last_db_dump/last_integrity_checkare backup-integrity timestamps the hub cannot see — the "presence is not success" neighbourhood, and worth ranking first inside R-264 after guest_net.- The gate cannot cover the hub's desired-state wire because it is served as raw stored JSON. That is the one remaining hub→box direction with no contract check of any kind.