hub v0.99.0 — the hub can see whether the operator can get in (R-260); G-1 gate closes, R-247 closes
oobDegraded tested five things and the sixth never arrived. The agent has emitted `operator_key_configured` on every heartbeat since v0.72.0 — the SAME version that introduced the `oob` stanza carrying it — and store.HostOOBRow mirrored five of the agent's eight OOB fields. With no field for it, encoding/json discarded the fact on arrival, so a box with felhom-sshd active, reachable, a valid config and a configured peer reported `ok` with NO OPERATOR KEY INSTALLED AT ALL. Not a wrong answer: an answer to a question nobody was asking. `operator_peer_configured`, which the hub did read, only says the peer IP is in desired-state — that OOB is MEANT to work, not that entry is possible. Now decoded: operator_key_configured, plus wg_handshake_age_s and healed_at. The last two ride the ALERT TEXT and are deliberately NOT in the predicate — widening a check beyond the fact that is now arriving is how a check stops being read. SCENARIO F, decided on a measurement rather than a preference. operator_key_configured decodes as a POINTER: nil = the agent never said, reported distinctly and never as ok. The version gate was rejected because the field and its stanza shipped in the SAME agent version (v0.72.0), so a stanza without the field cannot come from any released agent; the fleet is 0.113.0/0.127.0 and the vouched floor is 0.127.0. Handled explicitly anyway and pinned, because "cannot happen" is a claim this project has been burned by. THE MESSAGE NAMES THE FAULT. oobDegradedReason is the single source for both predicate and text, so the alert can never name a different fault from the one that fired. The old form derived it separately and had a vocabulary of two — unreachable, or config invalid — with no way to say the key is missing. The operator reads this at 07:00. TESTS DRIVE THE DECODE BOUNDARY. Every hub OOB test before this built a HostOOBRow by hand, and a test written that way CANNOT SEE A FIELD THAT NEVER DECODES — which is how this held a green suite for five weeks. The pre-existing fixture oobReport() also omitted the field, so those scenarios ran against a report shape no released agent produces (same family as R-262). Both fixed. Red-proofs, 8 expected outcomes and 0 wrong, each with the mutation asserted applied: dropping the field returns the false ok; an unconditional check alerts a healthy box; unknown-as-ok restores the silent pass. G-1 CLOSED — scripts/wire_contract_gate.py shipped as ranked, built BEFORE the fixes and seen failing on 40 fields (documentation/tests/wire-contract-gate-2026-08-08/BEFORE.md). Two instrument defects the control caught first: a substring false negative (grep -F healed_at matched privsep_healed_at) and treating dr_recipe as wholly opaque when its top-level sections ARE decoded through an allow-list that already cost offsite_restic (R-122). The prompt for this session said "465 emitted tags, eight unreachable". Checked against the repo: R-260 said "at least eight DECISION-BEARING facts", never eight tags. The real count is 40. R-260 CLOSED (class gated, sharpest instance fixed). R-247 CLOSED (controller v0.209.0). R-264 MINTED and OPEN — the 21 facts with no consumer, allowlisted with reasons so that gating the class could not be mistaken for deciding them. Still open and named: R-246, R-255..R-259, R-261..R-263, and C7's test-comment half. Capability map checked: it claims OOB access is implemented, never monitored, so no row was untrue; what was untrue sat one layer down and the row now records it. repo_gates --fast: all 8 OK. go build/vet/test green in hub, run separately from this commit.
This commit is contained in:
@@ -4,13 +4,30 @@ import (
|
||||
"io"
|
||||
"log"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
_ "modernc.org/sqlite"
|
||||
)
|
||||
|
||||
// oobReport builds a host report's `oob` stanza AS A RELEASED AGENT ACTUALLY SENDS IT.
|
||||
//
|
||||
// ⚠ IT DID NOT, UNTIL 2026-08-08 (R-260), and that mattered. This helper omitted
|
||||
// `operator_key_configured` entirely, so every scenario below was driven by a report shape NO
|
||||
// RELEASED AGENT PRODUCES — the field and the stanza that carries it shipped together in agent
|
||||
// v0.72.0. A fixture that is not the wire cannot detect a field missing from the wire's receiver,
|
||||
// which is one reason the gap survived five weeks under a green suite. Same family as R-262, where
|
||||
// the cross-repo golden omits the two fields whose drift its key-set test is supposed to guard.
|
||||
//
|
||||
// The key defaults to INSTALLED here so the existing scenarios keep their original meaning
|
||||
// (they vary the service, not the credential); the key's own scenarios live in
|
||||
// host_oob_operatorkey_test.go, and the never-reported case is exercised by oobReportNoKeyField.
|
||||
func oobReport(active, reachable, configInvalid, operatorConfigured bool) []byte {
|
||||
return oobReportWithKey(active, reachable, configInvalid, operatorConfigured, true)
|
||||
}
|
||||
|
||||
func oobReportWithKey(active, reachable, configInvalid, operatorConfigured, keyInstalled bool) []byte {
|
||||
b := func(v bool) string {
|
||||
if v {
|
||||
return "true"
|
||||
@@ -20,7 +37,16 @@ func oobReport(active, reachable, configInvalid, operatorConfigured bool) []byte
|
||||
return []byte(`{"host_id":"h1","oob":{"felhom_sshd_active":` + b(active) +
|
||||
`,"felhom_sshd_port":8822,"reachable":` + b(reachable) +
|
||||
`,"config_invalid":` + b(configInvalid) +
|
||||
`,"operator_peer_configured":` + b(operatorConfigured) + `}}`)
|
||||
`,"operator_peer_configured":` + b(operatorConfigured) +
|
||||
`,"operator_key_configured":` + b(keyInstalled) + `}}`)
|
||||
}
|
||||
|
||||
// oobReportNoKeyField is the pre-v0.72.0 shape: an `oob` stanza with no operator_key_configured key
|
||||
// at all. No released agent sends it; it exists so Scenario F is tested against real JSON rather
|
||||
// than against a hand-set struct field.
|
||||
func oobReportNoKeyField() []byte {
|
||||
return []byte(`{"host_id":"h1","oob":{"felhom_sshd_active":true,"felhom_sshd_port":8822,` +
|
||||
`"reachable":true,"config_invalid":false,"operator_peer_configured":true}}`)
|
||||
}
|
||||
|
||||
func newOOBStore(t *testing.T) *store.Store {
|
||||
@@ -101,3 +127,61 @@ func TestHostOOB_NoStanzaIgnored(t *testing.T) {
|
||||
t.Fatalf("no oob stanza must not alert, got %v", events)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHostOOB_MissingOperatorKey_EndToEnd drives the WHOLE path — raw report JSON → SaveHostReport
|
||||
// → GetHostOOBStates → the checker → the emitted event — for the box this session exists for.
|
||||
//
|
||||
// The per-predicate tests in host_oob_operatorkey_test.go set the row's fields directly, which is
|
||||
// fine for the verdict but cannot prove the fact SURVIVES THE DECODE. This one can, and it is the
|
||||
// test that would have failed before R-260.
|
||||
func TestHostOOB_MissingOperatorKey_EndToEnd(t *testing.T) {
|
||||
st := newOOBStore(t)
|
||||
// healthy, key installed → seeded clean
|
||||
st.SaveHostReport("h1", "c1", oobReport(true, true, false, true), store.HostReportDenorm{})
|
||||
var events []string
|
||||
var msgs []string
|
||||
c := NewHostOOBChecker(st, func(_, et, _, msg, _, _ string) {
|
||||
events = append(events, et)
|
||||
msgs = append(msgs, msg)
|
||||
}, log.New(io.Discard, "", 0))
|
||||
if c.IsDegraded("h1") {
|
||||
t.Fatal("healthy host with the key installed seeded degraded")
|
||||
}
|
||||
|
||||
// everything still up — only the operator's key is gone
|
||||
st.SaveHostReport("h1", "c1", oobReportWithKey(true, true, false, true, false), store.HostReportDenorm{})
|
||||
c.Check()
|
||||
if len(events) != 1 || events[0] != "oob_degraded" {
|
||||
t.Fatalf("a box whose operator key vanished must alert exactly once; got %v", events)
|
||||
}
|
||||
if !strings.Contains(msgs[0], "authorized_key") {
|
||||
t.Errorf("the alert must NAME the missing key — the operator reads this at 07:00 and needs to know which of five things is wrong; got %q", msgs[0])
|
||||
}
|
||||
|
||||
// the key comes back → recovered
|
||||
st.SaveHostReport("h1", "c1", oobReport(true, true, false, true), store.HostReportDenorm{})
|
||||
c.Check()
|
||||
if len(events) != 2 || events[1] != "oob_recovered" {
|
||||
t.Fatalf("reinstalling the key must recover; got %v", events)
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario F end to end: an `oob` stanza with no operator_key_configured key at all must NOT be a
|
||||
// silent ok, and must be reported distinctly from a known-missing key.
|
||||
func TestHostOOB_NoKeyField_IsNotSilentlyOK_EndToEnd(t *testing.T) {
|
||||
st := newOOBStore(t)
|
||||
st.SaveHostReport("h1", "c1", oobReportNoKeyField(), store.HostReportDenorm{})
|
||||
rows, err := st.GetHostOOBStates()
|
||||
if err != nil || len(rows) != 1 {
|
||||
t.Fatalf("GetHostOOBStates: %v rows=%d", err, len(rows))
|
||||
}
|
||||
if rows[0].OperatorKeyReported {
|
||||
t.Fatal("a stanza with no operator_key_configured decoded as though the agent had reported one")
|
||||
}
|
||||
if !oobDegraded(rows[0]) {
|
||||
t.Fatal("an agent too old to report the key was treated as ok — absence read as a fact, which is this defect through the version door")
|
||||
}
|
||||
if got := oobDegradedReason(rows[0]); !strings.Contains(got, "too old") {
|
||||
t.Errorf("the unknown case must say so distinctly; got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user