admin
4f5784894e
gates / gates (push) Successful in 20s
Run 264 (650cc8a, a DOCUMENTATION-ONLY commit) failed between two greens of byte-identical gate code.
Not waved away as a flake, because this project's own record is that a "known flake" can be a true
positive.
MEASURED. Every other run this session: 18-34s, log present (HTTP 200). Run 264: 834s (07:12:40 ->
07:26:34 UTC) and GET /actions/jobs/264/logs returns HTTP 500 - "264.log.zst: file does not exist".
The act-runner pod never restarted (0 restarts, 5d17h), so the job hung and was reaped; the runner
did not die.
NOT A GATE FINDING, on four independent facts: the diff from the green before it is Markdown only;
the same content is green two commits later (265, 33s); the gate code is identical across 263/264/265;
and 260-262, which WERE real gate failures, each failed in under 35s WITH a log.
THE CAUSE OF THE HANG IS UNDETERMINED and is deliberately recorded as such. DooPlex was doing heavy
work in that window (139 MB kubectl cp, a go run compiling the whole hub module), which is a
plausible contention story - but 40 cores at load ~5 does not establish it, so it is filed as a
hypothesis rather than asserted as a cause.
THE FINDING THAT MATTERS IS SECOND-ORDER, and it is gates.yml's own purpose turned against it. The
workflow exists because "a detector nobody hears is the defect R-29 filed, rebuilt one layer up", and
its alarm mail says "The failing gate names itself in the run log." There is no run log. An operator
following that sentence finds nothing and cannot tell a reap from a conviction. Unverified and worse:
the alarm step is `if: failure()` and whether it ran at all for a reaped job is unknown - if it did
not, this was a red CI that alarmed nobody.
Fix shapes recorded, none built: surface duration + log-presence in the alarm; an explicit
timeout-minutes under the reap so it fails fast and loudly WITH a log; and one deliberate test of
whether the alarm fires on a reaped job, because until that runs, "CI alarms on failure" is an
assumption.
200 lines
12 KiB
Markdown
200 lines
12 KiB
Markdown
# REPORT — G-1: a gate for the dropped field, then the fields it found (2026-08-08)
|
||
|
||
*A non-overwritten `REPORT-<topic>.md` sibling, per `CLAUDE.md:82-87` — a parallel session shares this
|
||
clone and the shared `REPORT.md` was not touched.*
|
||
|
||
## 1. The gate's output on today's tree — failing, before anything was fixed
|
||
|
||
**This is the session.** Captured verbatim in
|
||
`documentation/tests/wire-contract-gate-2026-08-08/BEFORE.md`:
|
||
|
||
```
|
||
wire-contract gate — 210 tag(s) checked across 3 declared wire(s); 51 skipped
|
||
WIRE-CONTRACT GATE FAILED: 40 emitted field(s) cannot be received.
|
||
```
|
||
|
||
It named every one, with its emit path and its direction, and re-found **`escrow_stale`** (R-247) and
|
||
every field R-260 listed. Had it been green, the gate would not work and *that* would have been the
|
||
finding — which is not hypothetical: the night before, `deadcode` was rejected for the neighbouring
|
||
C6 class for exactly that reason.
|
||
|
||
**⚠ A count this session's prompt got wrong.** The prompt said *"465 emitted tags, eight
|
||
unreachable"*. R-260's wording was "at least eight **decision-bearing** facts", never eight tags in
|
||
total. Measured: **40** on the three declared wires. Checked against the repo, not quoted — the
|
||
prompt's own rule 6, and the second prompt claim caught that way this week.
|
||
|
||
## 2. The forty, by disposition
|
||
|
||
| # | field(s) | direction | decision | what changed |
|
||
|---|---|---|---|---|
|
||
| 1 | `oob.operator_key_configured` | agent → hub | **receive and act** | decoded (pointer); `oobDegraded` fails on a missing key and the alert names it |
|
||
| 2 | `oob.wg_handshake_age_s`, `oob.healed_at` | agent → hub | **receive, message only** | in `HostOOBRow` + the event payload; deliberately NOT in the predicate |
|
||
| 3 | `escrow_stale` | hub → controller | **receive and act** | `report.EscrowStatus.Stale`; withheld-hash told apart from hash-less. **R-247** |
|
||
| 4 | 12 host/system metric fields | both → hub | **no consumer wanted — redundant** | allowlisted: the hub bands on the `*_percent` figures from the same stanzas |
|
||
| 5 | `guests.spec.{disk_bytes,memory_bytes}` | agent → hub | **redundant** | sizing is hub-owned intent, not mirrored reality |
|
||
| 6 | `storage_targets.smart.model_name` | agent → hub | **redundant** | a display label; `smart.health` + every banded counter ARE decoded |
|
||
| 7 | `wireguard.last_handshake_age_s` | agent → hub | **redundant** | wgsync reconciles from its own state |
|
||
| 8 | 21 fields (`guest_net`+7, `selfupdate_pending`+1, `healed_recently`, `applied_at`, `mount_parity`/`_inventory`, `config_hash`, `reporting_disabled`, `stacks`, `migrated_to`, `last_db_dump`, `last_integrity_check`) | both → hub | **no consumer today, one arguably owed** | allowlisted **against R-264, OPEN**. Allowlisting is not deciding, and the entries say so |
|
||
|
||
Full per-field reasons are in the gate's own `ALLOWLIST`, each a claim someone can re-check.
|
||
|
||
## 3. Scenario F — the choice, and why
|
||
|
||
**Unknown is reported distinctly and is never `ok`.** `operator_key_configured` decodes as a
|
||
**pointer**: nil = the agent never said, which is not a value.
|
||
|
||
The version gate the prompt thought "probably right" was **rejected on a measurement**: the field and
|
||
the `oob` stanza that carries it shipped in the **same** agent version (v0.72.0, 2026-07-05), so a
|
||
stanza without the field cannot come from any released agent. The live fleet is 0.113.0 and 0.127.0;
|
||
the vouched floor is 0.127.0. Building version-gating machinery the hub does not otherwise have, for a
|
||
state no box can be in, is cost without cover. The case is still handled explicitly and pinned by a
|
||
test, because "cannot happen" is a claim this project has been burned by.
|
||
|
||
## 4. R-247 — CLOSED
|
||
|
||
The field is received, and `reconcileEscrowed` tells a **withheld** hash from a **hash-less** one.
|
||
Controller v0.209.0.
|
||
|
||
**Deliberately not folded in, and said rather than skipped:** the wrong flag on `demo-hp` is an
|
||
operator act hub-side (**R-246**, still open), and the customer-facing Hungarian card copy is
|
||
unchanged — that is UI work with its own review path.
|
||
|
||
## 5. The gate's blind spots, and its self-test
|
||
|
||
Published in the module docstring **and** in the gate's own output, because Campaign 12's C1 guard
|
||
turned out blind to one of the three shapes it was written for:
|
||
|
||
- **generic tag names are not checked** (`name`, `state`, `status`, …) — a repo-wide string test says
|
||
nothing about them, so a drop of a generically-named field is **missed**; the gate under-reports
|
||
rather than over-reports;
|
||
- **reachability of a NAME is not use of a VALUE**;
|
||
- **only declared ROOTS are covered** — the hub's desired-state (raw stored JSON, no typed emitter)
|
||
and the agent's local API (no single root) are **not**;
|
||
- it reads source, not traffic; test files and `testdata/` are excluded on the receiving side
|
||
deliberately (a tag present only in a fixture is not decodable — which is R-262 exactly).
|
||
|
||
`--selftest` plants an unreachable tag on a real root in a throwaway copy and asserts conviction:
|
||
**exit 1, planted tag named**; unplanted tree **exit 0**.
|
||
|
||
**THREE instrument defects this gate's own controls caught before it was trusted.** None was found
|
||
by review; each was found by making the gate prove something.
|
||
|
||
1. **A substring false negative** — `grep -F healed_at` also matched `privsep_healed_at`. R-260 named
|
||
`healed_at`, so its absence from the output was the tell. Whole-token now; 40, not 39.
|
||
2. **`dr_recipe` is not wholly opaque** — its top-level section keys ARE decoded, through allow-lists
|
||
that already swallowed `offsite_restic` for months (R-122). Now opaque only **below depth 1**.
|
||
3. **The search shelled out to `grep` and read its failure as a finding.** CI convicted **all 174**
|
||
checked tags while the pre-push hook was green. The CI runner's image carries python3 and git and
|
||
deliberately little else, and its `grep` does not support `--include`, so stdout was empty and
|
||
empty was read as "absent". **A gate that silently turns a tool failure into a finding is worse
|
||
than no gate**, and its green would have been as untrustworthy as its red. Removed the dependency
|
||
rather than working around it: the search is pure Python now, one token index per receiving repo.
|
||
|
||
**The BEFORE capture was RE-VERIFIED, not re-generated** — the stronger claim. All 40 recorded fields
|
||
were re-tested against the new implementation: **agree=40, disagree=0**, i.e. exactly the four this
|
||
session fixed are now present and the other 36 still absent. The number stands under both
|
||
implementations.
|
||
|
||
**And the reusable half, which is about the gates and not about this gate.** The pre-push hook runs on
|
||
a workstation where every sibling repo is a real clone; CI checks out one repo, shallow. **A gate
|
||
that needs a sibling passes locally and is INCONCLUSIVE in CI — the two automated homes are not
|
||
interchangeable, and a new gate must be checked in BOTH.** The workflow's own alarm mail says a
|
||
hook-versus-CI disagreement "outranks whatever the push was for"; it did. Fixed by fetching the agent
|
||
clone in CI (`.gitea/workflows/gates.yml`), never by letting the gate skip when a sibling is absent —
|
||
that is the fail-open shape and would leave it running in neither home (R-29).
|
||
|
||
**Cost, stated plainly:** three CI runs went red (260, 261, 262) and each sent the operator an alarm
|
||
mail before run **263** went green. The alarm working is the system behaving correctly; the noise was
|
||
mine.
|
||
|
||
**A FOURTH red run, 264, was NOT one of mine and is filed as R-265.** It sat between two greens on a
|
||
**documentation-only** commit, ran **834 s** against 18–34 s for every other run in the session, and
|
||
**persisted no log at all** (`jobs/264/logs` → HTTP 500, *file does not exist*). The runner pod never
|
||
restarted, so the job hung and was reaped rather than the runner dying. Not a gate finding — the diff
|
||
was Markdown, the gate code was byte-identical to the two greens around it, and the same content is
|
||
green at run 265. **The cause of the hang is undetermined and is not guessed at**; DooPlex was busy in
|
||
that window with this session's own live-validation work, but the box has 40 cores at load ~5, so
|
||
that is a hypothesis, not a cause. **The reusable finding is second-order:** the alarm mail tells the
|
||
operator "the failing gate names itself in the run log", and here there is no run log — so a reap is
|
||
indistinguishable from a conviction, and whether the alarm fired at all for a reaped job is
|
||
**unverified**. R-265 carries it.
|
||
|
||
## 6. What `oobDegraded` says when it fails
|
||
|
||
```
|
||
Host <id>: OPERATOR ACCESS DEGRADED — the operator's authorized_key is NOT installed —
|
||
felhom-sshd is up and answering, and nobody can log in through it. The break-glass net
|
||
(auto-heal + vaulted root@pam console) is still under the box.
|
||
```
|
||
|
||
and for the unreachable-but-handled unknown:
|
||
|
||
```
|
||
… — the agent reports operator access but is too old to say whether the operator key is
|
||
installed (pre-v0.72.0) — treat entry as UNPROVEN, not working. …
|
||
```
|
||
|
||
`oobDegradedReason` is now the single source for both the predicate and the text, so the message can
|
||
never name a different fault from the one that fired. The old form derived it separately and had a
|
||
vocabulary of two.
|
||
|
||
## 7. Tests and red-proofs
|
||
|
||
New: `hub/internal/store/host_oob_decode_test.go` (4 tests, raw JSON at the decode boundary),
|
||
`hub/internal/monitor/host_oob_operatorkey_test.go` (6), plus two end-to-end tests in
|
||
`host_oob_test.go` driving JSON → store → checker → event.
|
||
|
||
**Red-proofs — 8 expected outcomes, 0 wrong, each with the mutation asserted applied:**
|
||
|
||
| mutation | assertion it applied | outcome |
|
||
|---|---|---|
|
||
| the gate on today's tree | — | **RED, naming all 40** ✔ |
|
||
| planted unreachable tag (post-fix) | self-test reports the planted tag by name | **RED on the plant, GREEN unplanted** ✔ |
|
||
| drop `operator_key_configured` from the decoder | json-tag occurrences in the decoder 2 → 1 | **RED — the false `ok` returns** ✔ |
|
||
| make the check unconditional | `MUTATED unconditional degrade` marker present | **RED — a healthy box alerts** ✔ |
|
||
| treat unknown as `ok` | `MUTATED: unknown is silently ok again` marker present | **RED — the silent pass returns** ✔ |
|
||
| all three restored | — | **GREEN** ✔ |
|
||
|
||
**The pre-existing fixture was part of the defect and was fixed too:** `oobReport()` omitted
|
||
`operator_key_configured`, so every earlier scenario ran against a report shape **no released agent
|
||
produces**. Same family as R-262.
|
||
|
||
## 8. The capability-map row about operator access
|
||
|
||
**Checked, and it was NOT claiming something untrue.** `00-capability-map.md:127` claims OOB operator
|
||
access is *implemented*, never that it is *monitored*, so no correction was owed. What was untrue sat
|
||
one layer down — the hub's own health check could not see the key — and the row now records that,
|
||
with the fix and the tests that pin it.
|
||
|
||
## 9. Gates, and what remains
|
||
|
||
`python3 scripts/repo_gates.py --fast` → **all 8 OK**, including the new `wire-contract` and
|
||
`golden-currency`. `go build ./... && go vet ./... && go test ./...` green in **hub** and
|
||
**controller** (run separately from every commit). **No `--no-verify` anywhere.**
|
||
|
||
**The one gate failure that remains is not a failure of this work:** golden **0.208.0** is baked and
|
||
byte-verified but **still not vouched**, so fresh installs receive 0.207.0. That is R-242's untouched
|
||
half and one operator Save.
|
||
|
||
## 10. Register
|
||
|
||
**R-260 CLOSED** (class gated + sharpest instance fixed), **R-247 CLOSED**, **G-1 CLOSED** in
|
||
`ROADMAP.md`. **R-264 minted and OPEN** — the twenty-one facts with no consumer, split out so that
|
||
gating the class could not be mistaken for deciding them. **Highest ID moved R-263 → R-264.**
|
||
|
||
Explicitly still open: R-246, R-255, R-256, R-257, R-258, R-259, R-261, R-262, R-263, and **C7's
|
||
test-comment half**, which Campaign 12 recorded as *owed, not done*.
|
||
|
||
## 11. Observations — noticed, NOT acted on
|
||
|
||
1. **`stacks` is the whole per-stack report object and the hub decodes none of it.** The largest
|
||
single unconsumed structure on the controller wire; folded into R-264 rather than sized here.
|
||
2. **The hub has no version-gating machinery for report fields at all.** Not needed today (see §3),
|
||
but the next additive field whose emitter and stanza do *not* ship together will need it, and
|
||
there is no convention to reach for.
|
||
3. **`backup.last_db_dump` / `last_integrity_check` are backup-integrity timestamps the hub cannot
|
||
see** — the "presence is not success" neighbourhood, and worth ranking first inside R-264 after
|
||
guest_net.
|
||
4. **The gate cannot cover the hub's desired-state wire** because it is served as raw stored JSON.
|
||
That is the one remaining hub→box direction with no contract check of any kind.
|