650cc8a4a7
gates / gates (push) Failing after 13m54s
The gate's third instrument defect and the reusable lesson underneath it: a gate needing a sibling clone passes on a workstation and is INCONCLUSIVE in CI, and a gate must not shell out to a tool the CI image may not have. Three red runs (260-262), each of which mailed the operator, before 263 went green. CONTEXT S-38 gains the two-homes rule.
188 lines
12 KiB
Markdown
188 lines
12 KiB
Markdown
# REPORT — G-1: a gate for the dropped field, then the fields it found (2026-08-08)
|
|
|
|
*A non-overwritten `REPORT-<topic>.md` sibling, per `CLAUDE.md:82-87` — a parallel session shares this
|
|
clone and the shared `REPORT.md` was not touched.*
|
|
|
|
## 1. The gate's output on today's tree — failing, before anything was fixed
|
|
|
|
**This is the session.** Captured verbatim in
|
|
`documentation/tests/wire-contract-gate-2026-08-08/BEFORE.md`:
|
|
|
|
```
|
|
wire-contract gate — 210 tag(s) checked across 3 declared wire(s); 51 skipped
|
|
WIRE-CONTRACT GATE FAILED: 40 emitted field(s) cannot be received.
|
|
```
|
|
|
|
It named every one, with its emit path and its direction, and re-found **`escrow_stale`** (R-247) and
|
|
every field R-260 listed. Had it been green, the gate would not work and *that* would have been the
|
|
finding — which is not hypothetical: the night before, `deadcode` was rejected for the neighbouring
|
|
C6 class for exactly that reason.
|
|
|
|
**⚠ A count this session's prompt got wrong.** The prompt said *"465 emitted tags, eight
|
|
unreachable"*. R-260's wording was "at least eight **decision-bearing** facts", never eight tags in
|
|
total. Measured: **40** on the three declared wires. Checked against the repo, not quoted — the
|
|
prompt's own rule 6, and the second prompt claim caught that way this week.
|
|
|
|
## 2. The forty, by disposition
|
|
|
|
| # | field(s) | direction | decision | what changed |
|
|
|---|---|---|---|---|
|
|
| 1 | `oob.operator_key_configured` | agent → hub | **receive and act** | decoded (pointer); `oobDegraded` fails on a missing key and the alert names it |
|
|
| 2 | `oob.wg_handshake_age_s`, `oob.healed_at` | agent → hub | **receive, message only** | in `HostOOBRow` + the event payload; deliberately NOT in the predicate |
|
|
| 3 | `escrow_stale` | hub → controller | **receive and act** | `report.EscrowStatus.Stale`; withheld-hash told apart from hash-less. **R-247** |
|
|
| 4 | 12 host/system metric fields | both → hub | **no consumer wanted — redundant** | allowlisted: the hub bands on the `*_percent` figures from the same stanzas |
|
|
| 5 | `guests.spec.{disk_bytes,memory_bytes}` | agent → hub | **redundant** | sizing is hub-owned intent, not mirrored reality |
|
|
| 6 | `storage_targets.smart.model_name` | agent → hub | **redundant** | a display label; `smart.health` + every banded counter ARE decoded |
|
|
| 7 | `wireguard.last_handshake_age_s` | agent → hub | **redundant** | wgsync reconciles from its own state |
|
|
| 8 | 21 fields (`guest_net`+7, `selfupdate_pending`+1, `healed_recently`, `applied_at`, `mount_parity`/`_inventory`, `config_hash`, `reporting_disabled`, `stacks`, `migrated_to`, `last_db_dump`, `last_integrity_check`) | both → hub | **no consumer today, one arguably owed** | allowlisted **against R-264, OPEN**. Allowlisting is not deciding, and the entries say so |
|
|
|
|
Full per-field reasons are in the gate's own `ALLOWLIST`, each a claim someone can re-check.
|
|
|
|
## 3. Scenario F — the choice, and why
|
|
|
|
**Unknown is reported distinctly and is never `ok`.** `operator_key_configured` decodes as a
|
|
**pointer**: nil = the agent never said, which is not a value.
|
|
|
|
The version gate the prompt thought "probably right" was **rejected on a measurement**: the field and
|
|
the `oob` stanza that carries it shipped in the **same** agent version (v0.72.0, 2026-07-05), so a
|
|
stanza without the field cannot come from any released agent. The live fleet is 0.113.0 and 0.127.0;
|
|
the vouched floor is 0.127.0. Building version-gating machinery the hub does not otherwise have, for a
|
|
state no box can be in, is cost without cover. The case is still handled explicitly and pinned by a
|
|
test, because "cannot happen" is a claim this project has been burned by.
|
|
|
|
## 4. R-247 — CLOSED
|
|
|
|
The field is received, and `reconcileEscrowed` tells a **withheld** hash from a **hash-less** one.
|
|
Controller v0.209.0.
|
|
|
|
**Deliberately not folded in, and said rather than skipped:** the wrong flag on `demo-hp` is an
|
|
operator act hub-side (**R-246**, still open), and the customer-facing Hungarian card copy is
|
|
unchanged — that is UI work with its own review path.
|
|
|
|
## 5. The gate's blind spots, and its self-test
|
|
|
|
Published in the module docstring **and** in the gate's own output, because Campaign 12's C1 guard
|
|
turned out blind to one of the three shapes it was written for:
|
|
|
|
- **generic tag names are not checked** (`name`, `state`, `status`, …) — a repo-wide string test says
|
|
nothing about them, so a drop of a generically-named field is **missed**; the gate under-reports
|
|
rather than over-reports;
|
|
- **reachability of a NAME is not use of a VALUE**;
|
|
- **only declared ROOTS are covered** — the hub's desired-state (raw stored JSON, no typed emitter)
|
|
and the agent's local API (no single root) are **not**;
|
|
- it reads source, not traffic; test files and `testdata/` are excluded on the receiving side
|
|
deliberately (a tag present only in a fixture is not decodable — which is R-262 exactly).
|
|
|
|
`--selftest` plants an unreachable tag on a real root in a throwaway copy and asserts conviction:
|
|
**exit 1, planted tag named**; unplanted tree **exit 0**.
|
|
|
|
**THREE instrument defects this gate's own controls caught before it was trusted.** None was found
|
|
by review; each was found by making the gate prove something.
|
|
|
|
1. **A substring false negative** — `grep -F healed_at` also matched `privsep_healed_at`. R-260 named
|
|
`healed_at`, so its absence from the output was the tell. Whole-token now; 40, not 39.
|
|
2. **`dr_recipe` is not wholly opaque** — its top-level section keys ARE decoded, through allow-lists
|
|
that already swallowed `offsite_restic` for months (R-122). Now opaque only **below depth 1**.
|
|
3. **The search shelled out to `grep` and read its failure as a finding.** CI convicted **all 174**
|
|
checked tags while the pre-push hook was green. The CI runner's image carries python3 and git and
|
|
deliberately little else, and its `grep` does not support `--include`, so stdout was empty and
|
|
empty was read as "absent". **A gate that silently turns a tool failure into a finding is worse
|
|
than no gate**, and its green would have been as untrustworthy as its red. Removed the dependency
|
|
rather than working around it: the search is pure Python now, one token index per receiving repo.
|
|
|
|
**The BEFORE capture was RE-VERIFIED, not re-generated** — the stronger claim. All 40 recorded fields
|
|
were re-tested against the new implementation: **agree=40, disagree=0**, i.e. exactly the four this
|
|
session fixed are now present and the other 36 still absent. The number stands under both
|
|
implementations.
|
|
|
|
**And the reusable half, which is about the gates and not about this gate.** The pre-push hook runs on
|
|
a workstation where every sibling repo is a real clone; CI checks out one repo, shallow. **A gate
|
|
that needs a sibling passes locally and is INCONCLUSIVE in CI — the two automated homes are not
|
|
interchangeable, and a new gate must be checked in BOTH.** The workflow's own alarm mail says a
|
|
hook-versus-CI disagreement "outranks whatever the push was for"; it did. Fixed by fetching the agent
|
|
clone in CI (`.gitea/workflows/gates.yml`), never by letting the gate skip when a sibling is absent —
|
|
that is the fail-open shape and would leave it running in neither home (R-29).
|
|
|
|
**Cost, stated plainly:** three CI runs went red (260, 261, 262) and each sent the operator an alarm
|
|
mail before run **263** went green. The alarm working is the system behaving correctly; the noise was
|
|
mine.
|
|
|
|
## 6. What `oobDegraded` says when it fails
|
|
|
|
```
|
|
Host <id>: OPERATOR ACCESS DEGRADED — the operator's authorized_key is NOT installed —
|
|
felhom-sshd is up and answering, and nobody can log in through it. The break-glass net
|
|
(auto-heal + vaulted root@pam console) is still under the box.
|
|
```
|
|
|
|
and for the unreachable-but-handled unknown:
|
|
|
|
```
|
|
… — the agent reports operator access but is too old to say whether the operator key is
|
|
installed (pre-v0.72.0) — treat entry as UNPROVEN, not working. …
|
|
```
|
|
|
|
`oobDegradedReason` is now the single source for both the predicate and the text, so the message can
|
|
never name a different fault from the one that fired. The old form derived it separately and had a
|
|
vocabulary of two.
|
|
|
|
## 7. Tests and red-proofs
|
|
|
|
New: `hub/internal/store/host_oob_decode_test.go` (4 tests, raw JSON at the decode boundary),
|
|
`hub/internal/monitor/host_oob_operatorkey_test.go` (6), plus two end-to-end tests in
|
|
`host_oob_test.go` driving JSON → store → checker → event.
|
|
|
|
**Red-proofs — 8 expected outcomes, 0 wrong, each with the mutation asserted applied:**
|
|
|
|
| mutation | assertion it applied | outcome |
|
|
|---|---|---|
|
|
| the gate on today's tree | — | **RED, naming all 40** ✔ |
|
|
| planted unreachable tag (post-fix) | self-test reports the planted tag by name | **RED on the plant, GREEN unplanted** ✔ |
|
|
| drop `operator_key_configured` from the decoder | json-tag occurrences in the decoder 2 → 1 | **RED — the false `ok` returns** ✔ |
|
|
| make the check unconditional | `MUTATED unconditional degrade` marker present | **RED — a healthy box alerts** ✔ |
|
|
| treat unknown as `ok` | `MUTATED: unknown is silently ok again` marker present | **RED — the silent pass returns** ✔ |
|
|
| all three restored | — | **GREEN** ✔ |
|
|
|
|
**The pre-existing fixture was part of the defect and was fixed too:** `oobReport()` omitted
|
|
`operator_key_configured`, so every earlier scenario ran against a report shape **no released agent
|
|
produces**. Same family as R-262.
|
|
|
|
## 8. The capability-map row about operator access
|
|
|
|
**Checked, and it was NOT claiming something untrue.** `00-capability-map.md:127` claims OOB operator
|
|
access is *implemented*, never that it is *monitored*, so no correction was owed. What was untrue sat
|
|
one layer down — the hub's own health check could not see the key — and the row now records that,
|
|
with the fix and the tests that pin it.
|
|
|
|
## 9. Gates, and what remains
|
|
|
|
`python3 scripts/repo_gates.py --fast` → **all 8 OK**, including the new `wire-contract` and
|
|
`golden-currency`. `go build ./... && go vet ./... && go test ./...` green in **hub** and
|
|
**controller** (run separately from every commit). **No `--no-verify` anywhere.**
|
|
|
|
**The one gate failure that remains is not a failure of this work:** golden **0.208.0** is baked and
|
|
byte-verified but **still not vouched**, so fresh installs receive 0.207.0. That is R-242's untouched
|
|
half and one operator Save.
|
|
|
|
## 10. Register
|
|
|
|
**R-260 CLOSED** (class gated + sharpest instance fixed), **R-247 CLOSED**, **G-1 CLOSED** in
|
|
`ROADMAP.md`. **R-264 minted and OPEN** — the twenty-one facts with no consumer, split out so that
|
|
gating the class could not be mistaken for deciding them. **Highest ID moved R-263 → R-264.**
|
|
|
|
Explicitly still open: R-246, R-255, R-256, R-257, R-258, R-259, R-261, R-262, R-263, and **C7's
|
|
test-comment half**, which Campaign 12 recorded as *owed, not done*.
|
|
|
|
## 11. Observations — noticed, NOT acted on
|
|
|
|
1. **`stacks` is the whole per-stack report object and the hub decodes none of it.** The largest
|
|
single unconsumed structure on the controller wire; folded into R-264 rather than sized here.
|
|
2. **The hub has no version-gating machinery for report fields at all.** Not needed today (see §3),
|
|
but the next additive field whose emitter and stanza do *not* ship together will need it, and
|
|
there is no convention to reach for.
|
|
3. **`backup.last_db_dump` / `last_integrity_check` are backup-integrity timestamps the hub cannot
|
|
see** — the "presence is not success" neighbourhood, and worth ranking first inside R-264 after
|
|
guest_net.
|
|
4. **The gate cannot cover the hub's desired-state wire** because it is served as raw stored JSON.
|
|
That is the one remaining hub→box direction with no contract check of any kind.
|