admin
b080ecf411
oobDegraded tested five things and the sixth never arrived. The agent has emitted `operator_key_configured` on every heartbeat since v0.72.0 — the SAME version that introduced the `oob` stanza carrying it — and store.HostOOBRow mirrored five of the agent's eight OOB fields. With no field for it, encoding/json discarded the fact on arrival, so a box with felhom-sshd active, reachable, a valid config and a configured peer reported `ok` with NO OPERATOR KEY INSTALLED AT ALL. Not a wrong answer: an answer to a question nobody was asking. `operator_peer_configured`, which the hub did read, only says the peer IP is in desired-state — that OOB is MEANT to work, not that entry is possible. Now decoded: operator_key_configured, plus wg_handshake_age_s and healed_at. The last two ride the ALERT TEXT and are deliberately NOT in the predicate — widening a check beyond the fact that is now arriving is how a check stops being read. SCENARIO F, decided on a measurement rather than a preference. operator_key_configured decodes as a POINTER: nil = the agent never said, reported distinctly and never as ok. The version gate was rejected because the field and its stanza shipped in the SAME agent version (v0.72.0), so a stanza without the field cannot come from any released agent; the fleet is 0.113.0/0.127.0 and the vouched floor is 0.127.0. Handled explicitly anyway and pinned, because "cannot happen" is a claim this project has been burned by. THE MESSAGE NAMES THE FAULT. oobDegradedReason is the single source for both predicate and text, so the alert can never name a different fault from the one that fired. The old form derived it separately and had a vocabulary of two — unreachable, or config invalid — with no way to say the key is missing. The operator reads this at 07:00. TESTS DRIVE THE DECODE BOUNDARY. Every hub OOB test before this built a HostOOBRow by hand, and a test written that way CANNOT SEE A FIELD THAT NEVER DECODES — which is how this held a green suite for five weeks. The pre-existing fixture oobReport() also omitted the field, so those scenarios ran against a report shape no released agent produces (same family as R-262). Both fixed. Red-proofs, 8 expected outcomes and 0 wrong, each with the mutation asserted applied: dropping the field returns the false ok; an unconditional check alerts a healthy box; unknown-as-ok restores the silent pass. G-1 CLOSED — scripts/wire_contract_gate.py shipped as ranked, built BEFORE the fixes and seen failing on 40 fields (documentation/tests/wire-contract-gate-2026-08-08/BEFORE.md). Two instrument defects the control caught first: a substring false negative (grep -F healed_at matched privsep_healed_at) and treating dr_recipe as wholly opaque when its top-level sections ARE decoded through an allow-list that already cost offsite_restic (R-122). The prompt for this session said "465 emitted tags, eight unreachable". Checked against the repo: R-260 said "at least eight DECISION-BEARING facts", never eight tags. The real count is 40. R-260 CLOSED (class gated, sharpest instance fixed). R-247 CLOSED (controller v0.209.0). R-264 MINTED and OPEN — the 21 facts with no consumer, allowlisted with reasons so that gating the class could not be mistaken for deciding them. Still open and named: R-246, R-255..R-259, R-261..R-263, and C7's test-comment half. Capability map checked: it claims OOB access is implemented, never monitored, so no row was untrue; what was untrue sat one layer down and the row now records it. repo_gates --fast: all 8 OK. go build/vet/test green in hub, run separately from this commit.
164 lines
9.6 KiB
Markdown
164 lines
9.6 KiB
Markdown
# REPORT — G-1: a gate for the dropped field, then the fields it found (2026-08-08)
|
|
|
|
*A non-overwritten `REPORT-<topic>.md` sibling, per `CLAUDE.md:82-87` — a parallel session shares this
|
|
clone and the shared `REPORT.md` was not touched.*
|
|
|
|
## 1. The gate's output on today's tree — failing, before anything was fixed
|
|
|
|
**This is the session.** Captured verbatim in
|
|
`documentation/tests/wire-contract-gate-2026-08-08/BEFORE.md`:
|
|
|
|
```
|
|
wire-contract gate — 210 tag(s) checked across 3 declared wire(s); 51 skipped
|
|
WIRE-CONTRACT GATE FAILED: 40 emitted field(s) cannot be received.
|
|
```
|
|
|
|
It named every one, with its emit path and its direction, and re-found **`escrow_stale`** (R-247) and
|
|
every field R-260 listed. Had it been green, the gate would not work and *that* would have been the
|
|
finding — which is not hypothetical: the night before, `deadcode` was rejected for the neighbouring
|
|
C6 class for exactly that reason.
|
|
|
|
**⚠ A count this session's prompt got wrong.** The prompt said *"465 emitted tags, eight
|
|
unreachable"*. R-260's wording was "at least eight **decision-bearing** facts", never eight tags in
|
|
total. Measured: **40** on the three declared wires. Checked against the repo, not quoted — the
|
|
prompt's own rule 6, and the second prompt claim caught that way this week.
|
|
|
|
## 2. The forty, by disposition
|
|
|
|
| # | field(s) | direction | decision | what changed |
|
|
|---|---|---|---|---|
|
|
| 1 | `oob.operator_key_configured` | agent → hub | **receive and act** | decoded (pointer); `oobDegraded` fails on a missing key and the alert names it |
|
|
| 2 | `oob.wg_handshake_age_s`, `oob.healed_at` | agent → hub | **receive, message only** | in `HostOOBRow` + the event payload; deliberately NOT in the predicate |
|
|
| 3 | `escrow_stale` | hub → controller | **receive and act** | `report.EscrowStatus.Stale`; withheld-hash told apart from hash-less. **R-247** |
|
|
| 4 | 12 host/system metric fields | both → hub | **no consumer wanted — redundant** | allowlisted: the hub bands on the `*_percent` figures from the same stanzas |
|
|
| 5 | `guests.spec.{disk_bytes,memory_bytes}` | agent → hub | **redundant** | sizing is hub-owned intent, not mirrored reality |
|
|
| 6 | `storage_targets.smart.model_name` | agent → hub | **redundant** | a display label; `smart.health` + every banded counter ARE decoded |
|
|
| 7 | `wireguard.last_handshake_age_s` | agent → hub | **redundant** | wgsync reconciles from its own state |
|
|
| 8 | 21 fields (`guest_net`+7, `selfupdate_pending`+1, `healed_recently`, `applied_at`, `mount_parity`/`_inventory`, `config_hash`, `reporting_disabled`, `stacks`, `migrated_to`, `last_db_dump`, `last_integrity_check`) | both → hub | **no consumer today, one arguably owed** | allowlisted **against R-264, OPEN**. Allowlisting is not deciding, and the entries say so |
|
|
|
|
Full per-field reasons are in the gate's own `ALLOWLIST`, each a claim someone can re-check.
|
|
|
|
## 3. Scenario F — the choice, and why
|
|
|
|
**Unknown is reported distinctly and is never `ok`.** `operator_key_configured` decodes as a
|
|
**pointer**: nil = the agent never said, which is not a value.
|
|
|
|
The version gate the prompt thought "probably right" was **rejected on a measurement**: the field and
|
|
the `oob` stanza that carries it shipped in the **same** agent version (v0.72.0, 2026-07-05), so a
|
|
stanza without the field cannot come from any released agent. The live fleet is 0.113.0 and 0.127.0;
|
|
the vouched floor is 0.127.0. Building version-gating machinery the hub does not otherwise have, for a
|
|
state no box can be in, is cost without cover. The case is still handled explicitly and pinned by a
|
|
test, because "cannot happen" is a claim this project has been burned by.
|
|
|
|
## 4. R-247 — CLOSED
|
|
|
|
The field is received, and `reconcileEscrowed` tells a **withheld** hash from a **hash-less** one.
|
|
Controller v0.209.0.
|
|
|
|
**Deliberately not folded in, and said rather than skipped:** the wrong flag on `demo-hp` is an
|
|
operator act hub-side (**R-246**, still open), and the customer-facing Hungarian card copy is
|
|
unchanged — that is UI work with its own review path.
|
|
|
|
## 5. The gate's blind spots, and its self-test
|
|
|
|
Published in the module docstring **and** in the gate's own output, because Campaign 12's C1 guard
|
|
turned out blind to one of the three shapes it was written for:
|
|
|
|
- **generic tag names are not checked** (`name`, `state`, `status`, …) — a repo-wide string test says
|
|
nothing about them, so a drop of a generically-named field is **missed**; the gate under-reports
|
|
rather than over-reports;
|
|
- **reachability of a NAME is not use of a VALUE**;
|
|
- **only declared ROOTS are covered** — the hub's desired-state (raw stored JSON, no typed emitter)
|
|
and the agent's local API (no single root) are **not**;
|
|
- it reads source, not traffic; test files and `testdata/` are excluded on the receiving side
|
|
deliberately (a tag present only in a fixture is not decodable — which is R-262 exactly).
|
|
|
|
`--selftest` plants an unreachable tag on a real root in a throwaway copy and asserts conviction:
|
|
**exit 1, planted tag named**; unplanted tree **exit 0**.
|
|
|
|
**Two instrument defects the CONTROL caught before the gate was trusted:**
|
|
|
|
1. **A substring false negative** — `grep -F healed_at` also matched `privsep_healed_at`. R-260 named
|
|
`healed_at`, so its absence from the output was the tell. Now a whole-token regex; 40 not 39.
|
|
2. **`dr_recipe` is not wholly opaque** — its top-level section keys ARE decoded, through allow-lists
|
|
that already swallowed `offsite_restic` for months (R-122). Now opaque only **below depth 1**.
|
|
|
|
## 6. What `oobDegraded` says when it fails
|
|
|
|
```
|
|
Host <id>: OPERATOR ACCESS DEGRADED — the operator's authorized_key is NOT installed —
|
|
felhom-sshd is up and answering, and nobody can log in through it. The break-glass net
|
|
(auto-heal + vaulted root@pam console) is still under the box.
|
|
```
|
|
|
|
and for the unreachable-but-handled unknown:
|
|
|
|
```
|
|
… — the agent reports operator access but is too old to say whether the operator key is
|
|
installed (pre-v0.72.0) — treat entry as UNPROVEN, not working. …
|
|
```
|
|
|
|
`oobDegradedReason` is now the single source for both the predicate and the text, so the message can
|
|
never name a different fault from the one that fired. The old form derived it separately and had a
|
|
vocabulary of two.
|
|
|
|
## 7. Tests and red-proofs
|
|
|
|
New: `hub/internal/store/host_oob_decode_test.go` (4 tests, raw JSON at the decode boundary),
|
|
`hub/internal/monitor/host_oob_operatorkey_test.go` (6), plus two end-to-end tests in
|
|
`host_oob_test.go` driving JSON → store → checker → event.
|
|
|
|
**Red-proofs — 8 expected outcomes, 0 wrong, each with the mutation asserted applied:**
|
|
|
|
| mutation | assertion it applied | outcome |
|
|
|---|---|---|
|
|
| the gate on today's tree | — | **RED, naming all 40** ✔ |
|
|
| planted unreachable tag (post-fix) | self-test reports the planted tag by name | **RED on the plant, GREEN unplanted** ✔ |
|
|
| drop `operator_key_configured` from the decoder | json-tag occurrences in the decoder 2 → 1 | **RED — the false `ok` returns** ✔ |
|
|
| make the check unconditional | `MUTATED unconditional degrade` marker present | **RED — a healthy box alerts** ✔ |
|
|
| treat unknown as `ok` | `MUTATED: unknown is silently ok again` marker present | **RED — the silent pass returns** ✔ |
|
|
| all three restored | — | **GREEN** ✔ |
|
|
|
|
**The pre-existing fixture was part of the defect and was fixed too:** `oobReport()` omitted
|
|
`operator_key_configured`, so every earlier scenario ran against a report shape **no released agent
|
|
produces**. Same family as R-262.
|
|
|
|
## 8. The capability-map row about operator access
|
|
|
|
**Checked, and it was NOT claiming something untrue.** `00-capability-map.md:127` claims OOB operator
|
|
access is *implemented*, never that it is *monitored*, so no correction was owed. What was untrue sat
|
|
one layer down — the hub's own health check could not see the key — and the row now records that,
|
|
with the fix and the tests that pin it.
|
|
|
|
## 9. Gates, and what remains
|
|
|
|
`python3 scripts/repo_gates.py --fast` → **all 8 OK**, including the new `wire-contract` and
|
|
`golden-currency`. `go build ./... && go vet ./... && go test ./...` green in **hub** and
|
|
**controller** (run separately from every commit). **No `--no-verify` anywhere.**
|
|
|
|
**The one gate failure that remains is not a failure of this work:** golden **0.208.0** is baked and
|
|
byte-verified but **still not vouched**, so fresh installs receive 0.207.0. That is R-242's untouched
|
|
half and one operator Save.
|
|
|
|
## 10. Register
|
|
|
|
**R-260 CLOSED** (class gated + sharpest instance fixed), **R-247 CLOSED**, **G-1 CLOSED** in
|
|
`ROADMAP.md`. **R-264 minted and OPEN** — the twenty-one facts with no consumer, split out so that
|
|
gating the class could not be mistaken for deciding them. **Highest ID moved R-263 → R-264.**
|
|
|
|
Explicitly still open: R-246, R-255, R-256, R-257, R-258, R-259, R-261, R-262, R-263, and **C7's
|
|
test-comment half**, which Campaign 12 recorded as *owed, not done*.
|
|
|
|
## 11. Observations — noticed, NOT acted on
|
|
|
|
1. **`stacks` is the whole per-stack report object and the hub decodes none of it.** The largest
|
|
single unconsumed structure on the controller wire; folded into R-264 rather than sized here.
|
|
2. **The hub has no version-gating machinery for report fields at all.** Not needed today (see §3),
|
|
but the next additive field whose emitter and stanza do *not* ship together will need it, and
|
|
there is no convention to reach for.
|
|
3. **`backup.last_db_dump` / `last_integrity_check` are backup-integrity timestamps the hub cannot
|
|
see** — the "presence is not success" neighbourhood, and worth ranking first inside R-264 after
|
|
guest_net.
|
|
4. **The gate cannot cover the hub's desired-state wire** because it is served as raw stored JSON.
|
|
That is the one remaining hub→box direction with no contract check of any kind.
|