b080ecf411
oobDegraded tested five things and the sixth never arrived. The agent has emitted `operator_key_configured` on every heartbeat since v0.72.0 — the SAME version that introduced the `oob` stanza carrying it — and store.HostOOBRow mirrored five of the agent's eight OOB fields. With no field for it, encoding/json discarded the fact on arrival, so a box with felhom-sshd active, reachable, a valid config and a configured peer reported `ok` with NO OPERATOR KEY INSTALLED AT ALL. Not a wrong answer: an answer to a question nobody was asking. `operator_peer_configured`, which the hub did read, only says the peer IP is in desired-state — that OOB is MEANT to work, not that entry is possible. Now decoded: operator_key_configured, plus wg_handshake_age_s and healed_at. The last two ride the ALERT TEXT and are deliberately NOT in the predicate — widening a check beyond the fact that is now arriving is how a check stops being read. SCENARIO F, decided on a measurement rather than a preference. operator_key_configured decodes as a POINTER: nil = the agent never said, reported distinctly and never as ok. The version gate was rejected because the field and its stanza shipped in the SAME agent version (v0.72.0), so a stanza without the field cannot come from any released agent; the fleet is 0.113.0/0.127.0 and the vouched floor is 0.127.0. Handled explicitly anyway and pinned, because "cannot happen" is a claim this project has been burned by. THE MESSAGE NAMES THE FAULT. oobDegradedReason is the single source for both predicate and text, so the alert can never name a different fault from the one that fired. The old form derived it separately and had a vocabulary of two — unreachable, or config invalid — with no way to say the key is missing. The operator reads this at 07:00. TESTS DRIVE THE DECODE BOUNDARY. Every hub OOB test before this built a HostOOBRow by hand, and a test written that way CANNOT SEE A FIELD THAT NEVER DECODES — which is how this held a green suite for five weeks. The pre-existing fixture oobReport() also omitted the field, so those scenarios ran against a report shape no released agent produces (same family as R-262). Both fixed. Red-proofs, 8 expected outcomes and 0 wrong, each with the mutation asserted applied: dropping the field returns the false ok; an unconditional check alerts a healthy box; unknown-as-ok restores the silent pass. G-1 CLOSED — scripts/wire_contract_gate.py shipped as ranked, built BEFORE the fixes and seen failing on 40 fields (documentation/tests/wire-contract-gate-2026-08-08/BEFORE.md). Two instrument defects the control caught first: a substring false negative (grep -F healed_at matched privsep_healed_at) and treating dr_recipe as wholly opaque when its top-level sections ARE decoded through an allow-list that already cost offsite_restic (R-122). The prompt for this session said "465 emitted tags, eight unreachable". Checked against the repo: R-260 said "at least eight DECISION-BEARING facts", never eight tags. The real count is 40. R-260 CLOSED (class gated, sharpest instance fixed). R-247 CLOSED (controller v0.209.0). R-264 MINTED and OPEN — the 21 facts with no consumer, allowlisted with reasons so that gating the class could not be mistaken for deciding them. Still open and named: R-246, R-255..R-259, R-261..R-263, and C7's test-comment half. Capability map checked: it claims OOB access is implemented, never monitored, so no row was untrue; what was untrue sat one layer down and the row now records it. repo_gates --fast: all 8 OK. go build/vet/test green in hub, run separately from this commit.
111 lines
4.9 KiB
Go
111 lines
4.9 KiB
Go
package store
|
|
|
|
import "encoding/json"
|
|
|
|
// HostOOBRow is the latest operator-access (OOB) state per host (TASK H1), parsed from the newest
|
|
// host_report. Present is false when the agent sent no oob stanza (pre-H1 / feature off) → never
|
|
// alerted.
|
|
//
|
|
// ⚠ THIS STRUCT USED TO MIRROR FIVE OF THE AGENT'S EIGHT OOB FIELDS, and the three it dropped
|
|
// included the one that decides the question the OOB checker exists to answer (R-260, G-1). The
|
|
// agent has emitted `operator_key_configured` on every heartbeat since v0.72.0 — the same version
|
|
// that introduced the stanza itself — and having no field for it here meant encoding/json discarded
|
|
// it on arrival, so `oobDegraded` reported a box with felhom-sshd active, reachable, a valid config,
|
|
// a configured peer and NO OPERATOR KEY INSTALLED as `ok`. The agent knew and said so.
|
|
type HostOOBRow struct {
|
|
HostID string
|
|
CustomerID string
|
|
Present bool
|
|
FelhomSshdActive bool
|
|
FelhomSshdPort int
|
|
Reachable bool
|
|
ConfigInvalid bool
|
|
OperatorPeerConfigured bool
|
|
// OperatorKeyConfigured — the operator's authorized_key is actually installed on the box.
|
|
// `operator_peer_configured` above is NOT a substitute: that one only says the peer IP is in
|
|
// desired-state, i.e. that OOB is MEANT to work. This says the credential that actually grants
|
|
// entry is there.
|
|
OperatorKeyConfigured bool
|
|
// OperatorKeyReported distinguishes "the agent said false" from "the agent never said".
|
|
// Absence must never read as "the key is installed" — that is this defect returning through the
|
|
// version door, and this project has watched an absence read as a fact four times.
|
|
//
|
|
// It cannot be false while Present is true for any RELEASED agent: the field and the stanza
|
|
// shipped together in v0.72.0 (2026-07-05), and the vouched floor is far above it. That is a
|
|
// claim, so it is pinned by TestOOBDecode_StanzaWithoutOperatorKey_IsNotSilentlyOK rather than
|
|
// left as a comment.
|
|
OperatorKeyReported bool
|
|
// WGHandshakeAgeS / HealedAt are carried for the ALERT MESSAGE, not for the predicate — the
|
|
// operator reads this at 07:00 and needs the context, but a check that starts failing for
|
|
// reasons nobody asked for is how a check stops being read. nil / "" when not reported.
|
|
WGHandshakeAgeS *int64
|
|
HealedAt string
|
|
}
|
|
|
|
// decodeOOBInto parses the `oob` stanza of one raw host report into r.
|
|
//
|
|
// It is a named function rather than an inline literal SO THAT TESTS CAN DRIVE THE REAL DECODE
|
|
// BOUNDARY. A test that constructs HostOOBRow by hand cannot see a field that never decodes, which
|
|
// is the entire class of defect this exists because of (R-260): every hub test asked what the
|
|
// checker did with a row, none asked whether the row could carry the fact.
|
|
func decodeOOBInto(r *HostOOBRow, reportJSON string) error {
|
|
var body struct {
|
|
OOB *struct {
|
|
FelhomSshdActive bool `json:"felhom_sshd_active"`
|
|
FelhomSshdPort int `json:"felhom_sshd_port"`
|
|
Reachable bool `json:"reachable"`
|
|
ConfigInvalid bool `json:"config_invalid"`
|
|
OperatorPeerConfigured bool `json:"operator_peer_configured"`
|
|
OperatorKeyConfigured *bool `json:"operator_key_configured"`
|
|
WGHandshakeAgeS *int64 `json:"wg_handshake_age_s"`
|
|
HealedAt string `json:"healed_at"`
|
|
} `json:"oob"`
|
|
}
|
|
err := json.Unmarshal([]byte(reportJSON), &body)
|
|
if body.OOB == nil {
|
|
return err
|
|
}
|
|
r.Present = true
|
|
r.FelhomSshdActive = body.OOB.FelhomSshdActive
|
|
r.FelhomSshdPort = body.OOB.FelhomSshdPort
|
|
r.Reachable = body.OOB.Reachable
|
|
r.ConfigInvalid = body.OOB.ConfigInvalid
|
|
r.OperatorPeerConfigured = body.OOB.OperatorPeerConfigured
|
|
// A POINTER, deliberately: nil means the agent never said, which is not the same as saying no.
|
|
if body.OOB.OperatorKeyConfigured != nil {
|
|
r.OperatorKeyReported = true
|
|
r.OperatorKeyConfigured = *body.OOB.OperatorKeyConfigured
|
|
}
|
|
r.WGHandshakeAgeS = body.OOB.WGHandshakeAgeS
|
|
r.HealedAt = body.OOB.HealedAt
|
|
return err
|
|
}
|
|
|
|
// GetHostOOBStates returns the latest oob stanza per host (mirrors GetHostMgmtPlaneStates). A report
|
|
// without the stanza yields Present=false; malformed JSON degrades to zero values, never an error.
|
|
func (s *Store) GetHostOOBStates() ([]HostOOBRow, error) {
|
|
rows, err := s.db.Query(`
|
|
SELECT hr.host_id, hr.customer_id, hr.report_json
|
|
FROM host_reports hr
|
|
JOIN (SELECT host_id, MAX(id) AS mx FROM host_reports GROUP BY host_id) latest
|
|
ON hr.id = latest.mx`)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
defer rows.Close()
|
|
var out []HostOOBRow
|
|
for rows.Next() {
|
|
var r HostOOBRow
|
|
var reportJSON string
|
|
if err := rows.Scan(&r.HostID, &r.CustomerID, &reportJSON); err != nil {
|
|
return nil, err
|
|
}
|
|
if err := decodeOOBInto(&r, reportJSON); err != nil {
|
|
// malformed JSON degrades to zero values, never an error (documented above)
|
|
_ = err
|
|
}
|
|
out = append(out, r)
|
|
}
|
|
return out, rows.Err()
|
|
}
|