hub v0.99.0 — the hub can see whether the operator can get in (R-260); G-1 gate closes, R-247 closes
oobDegraded tested five things and the sixth never arrived. The agent has emitted `operator_key_configured` on every heartbeat since v0.72.0 — the SAME version that introduced the `oob` stanza carrying it — and store.HostOOBRow mirrored five of the agent's eight OOB fields. With no field for it, encoding/json discarded the fact on arrival, so a box with felhom-sshd active, reachable, a valid config and a configured peer reported `ok` with NO OPERATOR KEY INSTALLED AT ALL. Not a wrong answer: an answer to a question nobody was asking. `operator_peer_configured`, which the hub did read, only says the peer IP is in desired-state — that OOB is MEANT to work, not that entry is possible. Now decoded: operator_key_configured, plus wg_handshake_age_s and healed_at. The last two ride the ALERT TEXT and are deliberately NOT in the predicate — widening a check beyond the fact that is now arriving is how a check stops being read. SCENARIO F, decided on a measurement rather than a preference. operator_key_configured decodes as a POINTER: nil = the agent never said, reported distinctly and never as ok. The version gate was rejected because the field and its stanza shipped in the SAME agent version (v0.72.0), so a stanza without the field cannot come from any released agent; the fleet is 0.113.0/0.127.0 and the vouched floor is 0.127.0. Handled explicitly anyway and pinned, because "cannot happen" is a claim this project has been burned by. THE MESSAGE NAMES THE FAULT. oobDegradedReason is the single source for both predicate and text, so the alert can never name a different fault from the one that fired. The old form derived it separately and had a vocabulary of two — unreachable, or config invalid — with no way to say the key is missing. The operator reads this at 07:00. TESTS DRIVE THE DECODE BOUNDARY. Every hub OOB test before this built a HostOOBRow by hand, and a test written that way CANNOT SEE A FIELD THAT NEVER DECODES — which is how this held a green suite for five weeks. The pre-existing fixture oobReport() also omitted the field, so those scenarios ran against a report shape no released agent produces (same family as R-262). Both fixed. Red-proofs, 8 expected outcomes and 0 wrong, each with the mutation asserted applied: dropping the field returns the false ok; an unconditional check alerts a healthy box; unknown-as-ok restores the silent pass. G-1 CLOSED — scripts/wire_contract_gate.py shipped as ranked, built BEFORE the fixes and seen failing on 40 fields (documentation/tests/wire-contract-gate-2026-08-08/BEFORE.md). Two instrument defects the control caught first: a substring false negative (grep -F healed_at matched privsep_healed_at) and treating dr_recipe as wholly opaque when its top-level sections ARE decoded through an allow-list that already cost offsite_restic (R-122). The prompt for this session said "465 emitted tags, eight unreachable". Checked against the repo: R-260 said "at least eight DECISION-BEARING facts", never eight tags. The real count is 40. R-260 CLOSED (class gated, sharpest instance fixed). R-247 CLOSED (controller v0.209.0). R-264 MINTED and OPEN — the 21 facts with no consumer, allowlisted with reasons so that gating the class could not be mistaken for deciding them. Still open and named: R-246, R-255..R-259, R-261..R-263, and C7's test-comment half. Capability map checked: it claims OOB access is implemented, never monitored, so no row was untrue; what was untrue sat one layer down and the row now records it. repo_gates --fast: all 8 OK. go build/vet/test green in hub, run separately from this commit.
This commit is contained in:
@@ -1,3 +1,48 @@
|
||||
## v0.99.0 — the hub can finally see whether the operator can get in (2026-08-08, R-260 / G-1)
|
||||
|
||||
**`oobDegraded` tested five things and the sixth never arrived.** The agent has emitted
|
||||
`operator_key_configured` on every heartbeat since v0.72.0 — the same version that introduced the
|
||||
`oob` stanza carrying it — and `store.HostOOBRow` mirrored five of the agent's eight OOB fields. With
|
||||
no field for it, `encoding/json` discarded the fact on arrival, so a box with felhom-sshd **active,
|
||||
reachable, a valid config and a configured peer** reported `ok` **with no operator key installed at
|
||||
all**. That is not a wrong answer; it is an answer to a question nobody was asking.
|
||||
|
||||
`operator_peer_configured`, which the hub did read, is not a substitute: it says the peer IP is in
|
||||
desired-state — that OOB is MEANT to work — not that entry is possible.
|
||||
|
||||
**Decoded now:** `operator_key_configured`, plus `wg_handshake_age_s` and `healed_at`. The last two
|
||||
are carried for the ALERT TEXT and are deliberately NOT in the predicate — a check that starts
|
||||
failing for reasons nobody asked for is how a check stops being read.
|
||||
|
||||
**`operator_key_configured` decodes as a POINTER.** nil = the agent never said, which is not the same
|
||||
as saying no. Absence must never read as "the key is installed" — that is the same defect returning
|
||||
through the version door, and this project has watched an absence read as a fact four times. A stanza
|
||||
without the field cannot come from any released agent (field and stanza shipped together in v0.72.0,
|
||||
and the vouched floor is far above it), so it is handled explicitly rather than assumed impossible,
|
||||
and reported DISTINCTLY from a known-missing key.
|
||||
|
||||
**The message names the fault.** `oobDegradedReason` is now the single source for both the predicate
|
||||
and the alert, so the text can never name a different fault from the one that fired. The old form
|
||||
derived the reason separately and had a vocabulary of exactly two — "unreachable" or "config
|
||||
invalid" — with no way to say "the key is missing". The operator reads this at 07:00 and needs to
|
||||
know which of five things is wrong.
|
||||
|
||||
**Tests drive the DECODE BOUNDARY, not a hand-built struct.** Every hub OOB test before this one
|
||||
constructed a `HostOOBRow` itself and asked what the checker did with it — a test written that way
|
||||
cannot see a field that never decodes, which is exactly why this survived five weeks under a green
|
||||
suite. `host_oob_decode_test.go` feeds raw report JSON; `TestHostOOB_MissingOperatorKey_EndToEnd` and
|
||||
`TestHostOOB_NoKeyField_IsNotSilentlyOK_EndToEnd` drive the whole path from JSON to emitted event.
|
||||
|
||||
**The existing fixture was part of the problem and was fixed too:** `oobReport()` omitted
|
||||
`operator_key_configured` entirely, so every pre-existing scenario ran against a report shape **no
|
||||
released agent produces**. Same family as R-262.
|
||||
|
||||
Red-proofs, each with the mutation asserted applied: dropping the field from the decoder returns the
|
||||
false `ok`; making the check unconditional alerts a healthy box; treating unknown as ok restores the
|
||||
silent pass.
|
||||
|
||||
**Gate:** `felhom.eu/scripts/wire_contract_gate.py` (G-1), `--fast`, registered in `repo_gates.py`.
|
||||
|
||||
## v0.98.0 — drop the retained recovery package when the box says its set-aside history is gone (2026-08-07, R-241)
|
||||
|
||||
The hub half of the controller's abandonment countdown, and the **only** reason the hub was touched
|
||||
|
||||
@@ -58,18 +58,58 @@ func NewHostOOBChecker(s *store.Store, onEvent EventNotifyFunc, logger *log.Logg
|
||||
}
|
||||
|
||||
// oobDegraded is the degraded predicate: config invalid, OR (OOB meant to work — operator peer
|
||||
// configured — AND felhom-sshd is not active/reachable).
|
||||
// configured — AND felhom-sshd is not active/reachable OR the operator's key is not installed).
|
||||
//
|
||||
// THE KEY CLAUSE IS NEW (R-260, G-1) AND IT IS THE POINT. Until 2026-08-08 this predicate tested
|
||||
// five things and the sixth — whether the credential that actually grants entry exists — never
|
||||
// arrived, because the hub's decoder had no field for it. A box with the service active, reachable,
|
||||
// a valid config and a configured peer reported `ok` with NO OPERATOR KEY INSTALLED. That is not a
|
||||
// wrong answer; it is an answer to a question nobody was asking.
|
||||
//
|
||||
// It is deliberately gated on OperatorPeerConfigured, exactly like the reachability clause: a box
|
||||
// where OOB was never set up is not "broken", and widening this check beyond the fact that is now
|
||||
// arriving is how a check stops being read.
|
||||
func oobDegraded(r store.HostOOBRow) bool {
|
||||
return oobDegradedReason(r) != ""
|
||||
}
|
||||
|
||||
// oobDegradedReason returns the SPECIFIC reason a host's operator access is degraded, or "" when it
|
||||
// is not. The reason is separated from the boolean because the operator reads the alert at 07:00 and
|
||||
// needs to know WHICH of the things this checks is wrong — "out-of-band access degraded" is true and
|
||||
// useless.
|
||||
func oobDegradedReason(r store.HostOOBRow) string {
|
||||
if !r.Present {
|
||||
return false
|
||||
return ""
|
||||
}
|
||||
if r.ConfigInvalid {
|
||||
return true
|
||||
return "felhom-sshd config invalid (sshd -t fails)"
|
||||
}
|
||||
if r.OperatorPeerConfigured && (!r.FelhomSshdActive || !r.Reachable) {
|
||||
return true
|
||||
if !r.OperatorPeerConfigured {
|
||||
return ""
|
||||
}
|
||||
return false
|
||||
if !r.FelhomSshdActive {
|
||||
return "felhom-sshd is not active"
|
||||
}
|
||||
if !r.Reachable {
|
||||
return "felhom-sshd unreachable (local dial to the OOB port fails)"
|
||||
}
|
||||
// The operator key, and the two ways it can be wrong. Both are reported distinctly, and NEITHER
|
||||
// is a silent ok — an absence read as "the key is installed" is precisely the defect this clause
|
||||
// was added to end, arriving through the version door instead.
|
||||
if !r.OperatorKeyReported {
|
||||
// Unreachable for any released agent: `operator_key_configured` and the `oob` stanza that
|
||||
// carries it shipped together in agent v0.72.0 (2026-07-05), so a stanza without the field
|
||||
// cannot come from a version anyone runs — the vouched floor is far above it. Handled
|
||||
// explicitly anyway, because "cannot happen" is the kind of claim this project has been
|
||||
// burned by, and pinned by TestOOBDegraded_StanzaWithoutKeyField_IsNotOK.
|
||||
return "the agent reports operator access but is too old to say whether the operator key is " +
|
||||
"installed (pre-v0.72.0) — treat entry as UNPROVEN, not working"
|
||||
}
|
||||
if !r.OperatorKeyConfigured {
|
||||
return "the operator's authorized_key is NOT installed — felhom-sshd is up and answering, " +
|
||||
"and nobody can log in through it"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// Check evaluates all hosts and emits oob_degraded / oob_recovered on transitions.
|
||||
@@ -121,22 +161,36 @@ func (c *HostOOBChecker) IsDegraded(hostID string) bool {
|
||||
func (c *HostOOBChecker) emit(row store.HostOOBRow, eventType, severity string) {
|
||||
var msg string
|
||||
if eventType == "oob_degraded" {
|
||||
reason := "felhom-sshd unreachable"
|
||||
if row.ConfigInvalid {
|
||||
reason = "felhom-sshd config invalid (sshd -t fails)"
|
||||
// The reason comes from the predicate itself, so the message can never name a different
|
||||
// fault from the one that fired. The old form derived it separately and could only ever say
|
||||
// "unreachable" or "config invalid" — it had no vocabulary for a missing operator key,
|
||||
// which is the fault this checker most needs to be able to name (R-260).
|
||||
reason := oobDegradedReason(row)
|
||||
if reason == "" {
|
||||
reason = "operator access degraded"
|
||||
}
|
||||
msg = "Host " + row.HostID + ": OPERATOR ACCESS DEGRADED — " + reason +
|
||||
". The break-glass net (auto-heal + vaulted root@pam console) is still under the box."
|
||||
} else {
|
||||
msg = "Host " + row.HostID + ": operator access recovered (felhom-sshd reachable again)."
|
||||
}
|
||||
details, _ := json.Marshal(map[string]any{
|
||||
det := map[string]any{
|
||||
"host_id": row.HostID,
|
||||
"felhom_sshd_port": row.FelhomSshdPort,
|
||||
"active": row.FelhomSshdActive,
|
||||
"reachable": row.Reachable,
|
||||
"config_invalid": row.ConfigInvalid,
|
||||
})
|
||||
// R-260: carried for context, NOT consulted by the predicate.
|
||||
"operator_key_configured": row.OperatorKeyConfigured,
|
||||
"operator_key_reported": row.OperatorKeyReported,
|
||||
}
|
||||
if row.WGHandshakeAgeS != nil {
|
||||
det["wg_handshake_age_s"] = *row.WGHandshakeAgeS
|
||||
}
|
||||
if row.HealedAt != "" {
|
||||
det["healed_at"] = row.HealedAt
|
||||
}
|
||||
details, _ := json.Marshal(det)
|
||||
c.logger.Printf("[%s] Host OOB: %s (%s)", map[string]string{"warning": "WARN", "info": "INFO"}[severity], row.HostID, eventType)
|
||||
if _, err := c.store.SaveEvent(row.CustomerID, eventType, severity, msg, string(details), "hub"); err != nil {
|
||||
c.logger.Printf("[WARN] save %s for %s: %v", eventType, row.HostID, err)
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
package monitor
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
// Scenarios D, E and F of the R-260 / G-1 session, asserted on the CONSEQUENCE (does the check
|
||||
// report degraded, and does it name the right thing) rather than on the mechanism.
|
||||
|
||||
func healthyRow() store.HostOOBRow {
|
||||
return store.HostOOBRow{
|
||||
HostID: "h1", CustomerID: "c1", Present: true,
|
||||
FelhomSshdActive: true, Reachable: true, ConfigInvalid: false,
|
||||
OperatorPeerConfigured: true,
|
||||
OperatorKeyConfigured: true, OperatorKeyReported: true,
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario D — the box this session exists for: everything up, no operator key.
|
||||
func TestOOBDegraded_NoOperatorKey_IsDegradedAndSaysSo(t *testing.T) {
|
||||
r := healthyRow()
|
||||
r.OperatorKeyConfigured = false // reported, and false
|
||||
|
||||
if !oobDegraded(r) {
|
||||
t.Fatal("a box with felhom-sshd active, reachable, a valid config, a configured peer and NO OPERATOR KEY reported ok — that is R-260, the exact question this check exists to answer")
|
||||
}
|
||||
reason := oobDegradedReason(r)
|
||||
// §7.4: the operator reads this at 07:00 and must know WHICH of the five things is wrong.
|
||||
if !strings.Contains(reason, "authorized_key") {
|
||||
t.Errorf("the reason must NAME the missing operator key, not just say access is degraded; got %q", reason)
|
||||
}
|
||||
for _, wrong := range []string{"unreachable", "config invalid"} {
|
||||
if strings.Contains(reason, wrong) {
|
||||
t.Errorf("the reason names %q, which is not what is wrong: %q", wrong, reason)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario E — a healthy box must be unchanged. No new alert on a box that is fine.
|
||||
func TestOOBDegraded_HealthyBoxUnchanged(t *testing.T) {
|
||||
if oobDegraded(healthyRow()) {
|
||||
t.Fatalf("a fully healthy box alerted: %q", oobDegradedReason(healthyRow()))
|
||||
}
|
||||
if got := oobDegradedReason(healthyRow()); got != "" {
|
||||
t.Errorf("healthy box produced a reason %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario F — an agent too old to report the field. Absence must NOT read as "the key is
|
||||
// installed". This project has watched an absence read as a fact four times.
|
||||
func TestOOBDegraded_StanzaWithoutKeyField_IsNotOK(t *testing.T) {
|
||||
r := healthyRow()
|
||||
r.OperatorKeyReported = false
|
||||
r.OperatorKeyConfigured = false // the zero value an absent field leaves behind
|
||||
|
||||
if !oobDegraded(r) {
|
||||
t.Fatal("an agent that never said whether the operator key is installed was reported ok — that is the defect returning through the version door")
|
||||
}
|
||||
reason := oobDegradedReason(r)
|
||||
if !strings.Contains(reason, "too old") || !strings.Contains(reason, "UNPROVEN") {
|
||||
t.Errorf("the unknown case must be reported DISTINCTLY from a known-missing key, and must not claim the key is absent; got %q", reason)
|
||||
}
|
||||
// and it must NOT be worded as the known-missing-key case
|
||||
if strings.Contains(reason, "nobody can log in") {
|
||||
t.Errorf("unknown was reported as if it were a known-missing key: %q", reason)
|
||||
}
|
||||
}
|
||||
|
||||
// The peer gate is preserved: a box where OOB was never set up is not "broken", and this session
|
||||
// must not widen the check beyond the fact that is now arriving.
|
||||
func TestOOBDegraded_NoOperatorPeer_StillNotEvaluated(t *testing.T) {
|
||||
r := healthyRow()
|
||||
r.OperatorPeerConfigured = false
|
||||
r.OperatorKeyConfigured = false
|
||||
r.OperatorKeyReported = true
|
||||
if oobDegraded(r) {
|
||||
t.Errorf("a box with no operator peer configured must not alert on a missing key: %q", oobDegradedReason(r))
|
||||
}
|
||||
}
|
||||
|
||||
// The pre-existing faults must still fire, and still name themselves correctly.
|
||||
func TestOOBDegraded_ExistingFaultsUnchanged(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
mutate func(*store.HostOOBRow)
|
||||
expect string
|
||||
}{
|
||||
{"config invalid", func(r *store.HostOOBRow) { r.ConfigInvalid = true }, "config invalid"},
|
||||
{"not active", func(r *store.HostOOBRow) { r.FelhomSshdActive = false }, "not active"},
|
||||
{"unreachable", func(r *store.HostOOBRow) { r.Reachable = false }, "unreachable"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
r := healthyRow()
|
||||
tc.mutate(&r)
|
||||
if !oobDegraded(r) {
|
||||
t.Fatalf("%s no longer degrades", tc.name)
|
||||
}
|
||||
if got := oobDegradedReason(r); !strings.Contains(got, tc.expect) {
|
||||
t.Errorf("reason %q does not contain %q", got, tc.expect)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A host with no oob stanza at all is never evaluated — pre-H1 or feature off.
|
||||
func TestOOBDegraded_AbsentStanzaNeverAlerts(t *testing.T) {
|
||||
if oobDegraded(store.HostOOBRow{HostID: "h1"}) {
|
||||
t.Error("a host with no oob stanza alerted")
|
||||
}
|
||||
}
|
||||
@@ -4,13 +4,30 @@ import (
|
||||
"io"
|
||||
"log"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
_ "modernc.org/sqlite"
|
||||
)
|
||||
|
||||
// oobReport builds a host report's `oob` stanza AS A RELEASED AGENT ACTUALLY SENDS IT.
|
||||
//
|
||||
// ⚠ IT DID NOT, UNTIL 2026-08-08 (R-260), and that mattered. This helper omitted
|
||||
// `operator_key_configured` entirely, so every scenario below was driven by a report shape NO
|
||||
// RELEASED AGENT PRODUCES — the field and the stanza that carries it shipped together in agent
|
||||
// v0.72.0. A fixture that is not the wire cannot detect a field missing from the wire's receiver,
|
||||
// which is one reason the gap survived five weeks under a green suite. Same family as R-262, where
|
||||
// the cross-repo golden omits the two fields whose drift its key-set test is supposed to guard.
|
||||
//
|
||||
// The key defaults to INSTALLED here so the existing scenarios keep their original meaning
|
||||
// (they vary the service, not the credential); the key's own scenarios live in
|
||||
// host_oob_operatorkey_test.go, and the never-reported case is exercised by oobReportNoKeyField.
|
||||
func oobReport(active, reachable, configInvalid, operatorConfigured bool) []byte {
|
||||
return oobReportWithKey(active, reachable, configInvalid, operatorConfigured, true)
|
||||
}
|
||||
|
||||
func oobReportWithKey(active, reachable, configInvalid, operatorConfigured, keyInstalled bool) []byte {
|
||||
b := func(v bool) string {
|
||||
if v {
|
||||
return "true"
|
||||
@@ -20,7 +37,16 @@ func oobReport(active, reachable, configInvalid, operatorConfigured bool) []byte
|
||||
return []byte(`{"host_id":"h1","oob":{"felhom_sshd_active":` + b(active) +
|
||||
`,"felhom_sshd_port":8822,"reachable":` + b(reachable) +
|
||||
`,"config_invalid":` + b(configInvalid) +
|
||||
`,"operator_peer_configured":` + b(operatorConfigured) + `}}`)
|
||||
`,"operator_peer_configured":` + b(operatorConfigured) +
|
||||
`,"operator_key_configured":` + b(keyInstalled) + `}}`)
|
||||
}
|
||||
|
||||
// oobReportNoKeyField is the pre-v0.72.0 shape: an `oob` stanza with no operator_key_configured key
|
||||
// at all. No released agent sends it; it exists so Scenario F is tested against real JSON rather
|
||||
// than against a hand-set struct field.
|
||||
func oobReportNoKeyField() []byte {
|
||||
return []byte(`{"host_id":"h1","oob":{"felhom_sshd_active":true,"felhom_sshd_port":8822,` +
|
||||
`"reachable":true,"config_invalid":false,"operator_peer_configured":true}}`)
|
||||
}
|
||||
|
||||
func newOOBStore(t *testing.T) *store.Store {
|
||||
@@ -101,3 +127,61 @@ func TestHostOOB_NoStanzaIgnored(t *testing.T) {
|
||||
t.Fatalf("no oob stanza must not alert, got %v", events)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHostOOB_MissingOperatorKey_EndToEnd drives the WHOLE path — raw report JSON → SaveHostReport
|
||||
// → GetHostOOBStates → the checker → the emitted event — for the box this session exists for.
|
||||
//
|
||||
// The per-predicate tests in host_oob_operatorkey_test.go set the row's fields directly, which is
|
||||
// fine for the verdict but cannot prove the fact SURVIVES THE DECODE. This one can, and it is the
|
||||
// test that would have failed before R-260.
|
||||
func TestHostOOB_MissingOperatorKey_EndToEnd(t *testing.T) {
|
||||
st := newOOBStore(t)
|
||||
// healthy, key installed → seeded clean
|
||||
st.SaveHostReport("h1", "c1", oobReport(true, true, false, true), store.HostReportDenorm{})
|
||||
var events []string
|
||||
var msgs []string
|
||||
c := NewHostOOBChecker(st, func(_, et, _, msg, _, _ string) {
|
||||
events = append(events, et)
|
||||
msgs = append(msgs, msg)
|
||||
}, log.New(io.Discard, "", 0))
|
||||
if c.IsDegraded("h1") {
|
||||
t.Fatal("healthy host with the key installed seeded degraded")
|
||||
}
|
||||
|
||||
// everything still up — only the operator's key is gone
|
||||
st.SaveHostReport("h1", "c1", oobReportWithKey(true, true, false, true, false), store.HostReportDenorm{})
|
||||
c.Check()
|
||||
if len(events) != 1 || events[0] != "oob_degraded" {
|
||||
t.Fatalf("a box whose operator key vanished must alert exactly once; got %v", events)
|
||||
}
|
||||
if !strings.Contains(msgs[0], "authorized_key") {
|
||||
t.Errorf("the alert must NAME the missing key — the operator reads this at 07:00 and needs to know which of five things is wrong; got %q", msgs[0])
|
||||
}
|
||||
|
||||
// the key comes back → recovered
|
||||
st.SaveHostReport("h1", "c1", oobReport(true, true, false, true), store.HostReportDenorm{})
|
||||
c.Check()
|
||||
if len(events) != 2 || events[1] != "oob_recovered" {
|
||||
t.Fatalf("reinstalling the key must recover; got %v", events)
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario F end to end: an `oob` stanza with no operator_key_configured key at all must NOT be a
|
||||
// silent ok, and must be reported distinctly from a known-missing key.
|
||||
func TestHostOOB_NoKeyField_IsNotSilentlyOK_EndToEnd(t *testing.T) {
|
||||
st := newOOBStore(t)
|
||||
st.SaveHostReport("h1", "c1", oobReportNoKeyField(), store.HostReportDenorm{})
|
||||
rows, err := st.GetHostOOBStates()
|
||||
if err != nil || len(rows) != 1 {
|
||||
t.Fatalf("GetHostOOBStates: %v rows=%d", err, len(rows))
|
||||
}
|
||||
if rows[0].OperatorKeyReported {
|
||||
t.Fatal("a stanza with no operator_key_configured decoded as though the agent had reported one")
|
||||
}
|
||||
if !oobDegraded(rows[0]) {
|
||||
t.Fatal("an agent too old to report the key was treated as ok — absence read as a fact, which is this defect through the version door")
|
||||
}
|
||||
if got := oobDegradedReason(rows[0]); !strings.Contains(got, "too old") {
|
||||
t.Errorf("the unknown case must say so distinctly; got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -5,6 +5,13 @@ import "encoding/json"
|
||||
// HostOOBRow is the latest operator-access (OOB) state per host (TASK H1), parsed from the newest
|
||||
// host_report. Present is false when the agent sent no oob stanza (pre-H1 / feature off) → never
|
||||
// alerted.
|
||||
//
|
||||
// ⚠ THIS STRUCT USED TO MIRROR FIVE OF THE AGENT'S EIGHT OOB FIELDS, and the three it dropped
|
||||
// included the one that decides the question the OOB checker exists to answer (R-260, G-1). The
|
||||
// agent has emitted `operator_key_configured` on every heartbeat since v0.72.0 — the same version
|
||||
// that introduced the stanza itself — and having no field for it here meant encoding/json discarded
|
||||
// it on arrival, so `oobDegraded` reported a box with felhom-sshd active, reachable, a valid config,
|
||||
// a configured peer and NO OPERATOR KEY INSTALLED as `ok`. The agent knew and said so.
|
||||
type HostOOBRow struct {
|
||||
HostID string
|
||||
CustomerID string
|
||||
@@ -14,6 +21,64 @@ type HostOOBRow struct {
|
||||
Reachable bool
|
||||
ConfigInvalid bool
|
||||
OperatorPeerConfigured bool
|
||||
// OperatorKeyConfigured — the operator's authorized_key is actually installed on the box.
|
||||
// `operator_peer_configured` above is NOT a substitute: that one only says the peer IP is in
|
||||
// desired-state, i.e. that OOB is MEANT to work. This says the credential that actually grants
|
||||
// entry is there.
|
||||
OperatorKeyConfigured bool
|
||||
// OperatorKeyReported distinguishes "the agent said false" from "the agent never said".
|
||||
// Absence must never read as "the key is installed" — that is this defect returning through the
|
||||
// version door, and this project has watched an absence read as a fact four times.
|
||||
//
|
||||
// It cannot be false while Present is true for any RELEASED agent: the field and the stanza
|
||||
// shipped together in v0.72.0 (2026-07-05), and the vouched floor is far above it. That is a
|
||||
// claim, so it is pinned by TestOOBDecode_StanzaWithoutOperatorKey_IsNotSilentlyOK rather than
|
||||
// left as a comment.
|
||||
OperatorKeyReported bool
|
||||
// WGHandshakeAgeS / HealedAt are carried for the ALERT MESSAGE, not for the predicate — the
|
||||
// operator reads this at 07:00 and needs the context, but a check that starts failing for
|
||||
// reasons nobody asked for is how a check stops being read. nil / "" when not reported.
|
||||
WGHandshakeAgeS *int64
|
||||
HealedAt string
|
||||
}
|
||||
|
||||
// decodeOOBInto parses the `oob` stanza of one raw host report into r.
|
||||
//
|
||||
// It is a named function rather than an inline literal SO THAT TESTS CAN DRIVE THE REAL DECODE
|
||||
// BOUNDARY. A test that constructs HostOOBRow by hand cannot see a field that never decodes, which
|
||||
// is the entire class of defect this exists because of (R-260): every hub test asked what the
|
||||
// checker did with a row, none asked whether the row could carry the fact.
|
||||
func decodeOOBInto(r *HostOOBRow, reportJSON string) error {
|
||||
var body struct {
|
||||
OOB *struct {
|
||||
FelhomSshdActive bool `json:"felhom_sshd_active"`
|
||||
FelhomSshdPort int `json:"felhom_sshd_port"`
|
||||
Reachable bool `json:"reachable"`
|
||||
ConfigInvalid bool `json:"config_invalid"`
|
||||
OperatorPeerConfigured bool `json:"operator_peer_configured"`
|
||||
OperatorKeyConfigured *bool `json:"operator_key_configured"`
|
||||
WGHandshakeAgeS *int64 `json:"wg_handshake_age_s"`
|
||||
HealedAt string `json:"healed_at"`
|
||||
} `json:"oob"`
|
||||
}
|
||||
err := json.Unmarshal([]byte(reportJSON), &body)
|
||||
if body.OOB == nil {
|
||||
return err
|
||||
}
|
||||
r.Present = true
|
||||
r.FelhomSshdActive = body.OOB.FelhomSshdActive
|
||||
r.FelhomSshdPort = body.OOB.FelhomSshdPort
|
||||
r.Reachable = body.OOB.Reachable
|
||||
r.ConfigInvalid = body.OOB.ConfigInvalid
|
||||
r.OperatorPeerConfigured = body.OOB.OperatorPeerConfigured
|
||||
// A POINTER, deliberately: nil means the agent never said, which is not the same as saying no.
|
||||
if body.OOB.OperatorKeyConfigured != nil {
|
||||
r.OperatorKeyReported = true
|
||||
r.OperatorKeyConfigured = *body.OOB.OperatorKeyConfigured
|
||||
}
|
||||
r.WGHandshakeAgeS = body.OOB.WGHandshakeAgeS
|
||||
r.HealedAt = body.OOB.HealedAt
|
||||
return err
|
||||
}
|
||||
|
||||
// GetHostOOBStates returns the latest oob stanza per host (mirrors GetHostMgmtPlaneStates). A report
|
||||
@@ -35,23 +100,9 @@ func (s *Store) GetHostOOBStates() ([]HostOOBRow, error) {
|
||||
if err := rows.Scan(&r.HostID, &r.CustomerID, &reportJSON); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var body struct {
|
||||
OOB *struct {
|
||||
FelhomSshdActive bool `json:"felhom_sshd_active"`
|
||||
FelhomSshdPort int `json:"felhom_sshd_port"`
|
||||
Reachable bool `json:"reachable"`
|
||||
ConfigInvalid bool `json:"config_invalid"`
|
||||
OperatorPeerConfigured bool `json:"operator_peer_configured"`
|
||||
} `json:"oob"`
|
||||
}
|
||||
_ = json.Unmarshal([]byte(reportJSON), &body)
|
||||
if body.OOB != nil {
|
||||
r.Present = true
|
||||
r.FelhomSshdActive = body.OOB.FelhomSshdActive
|
||||
r.FelhomSshdPort = body.OOB.FelhomSshdPort
|
||||
r.Reachable = body.OOB.Reachable
|
||||
r.ConfigInvalid = body.OOB.ConfigInvalid
|
||||
r.OperatorPeerConfigured = body.OOB.OperatorPeerConfigured
|
||||
if err := decodeOOBInto(&r, reportJSON); err != nil {
|
||||
// malformed JSON degrades to zero values, never an error (documented above)
|
||||
_ = err
|
||||
}
|
||||
out = append(out, r)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
package store
|
||||
|
||||
import "testing"
|
||||
|
||||
// These tests drive decodeOOBInto — THE REAL DECODE BOUNDARY — with raw report JSON, not a
|
||||
// hand-built HostOOBRow.
|
||||
//
|
||||
// That distinction is the entire point of R-260 and is not a style preference. Every hub test that
|
||||
// touched OOB before this file constructed a HostOOBRow itself and asked what the checker did with
|
||||
// it. A test written that way CANNOT SEE A FIELD THAT NEVER DECODES: it hands the struct the value
|
||||
// the production path would have discarded, and passes. The defect lived under a green suite for
|
||||
// five weeks because of exactly that.
|
||||
|
||||
const oobHealthyJSON = `{"host_id":"h1","oob":{
|
||||
"felhom_sshd_active":true,"felhom_sshd_port":8822,"reachable":true,
|
||||
"config_invalid":false,"operator_peer_configured":true,
|
||||
"operator_key_configured":true,"wg_handshake_age_s":42,"healed_at":"2026-08-08T01:02:03Z"}}`
|
||||
|
||||
// The shape the agent sends for the box this whole session is about: everything up, and the key
|
||||
// that actually grants entry absent.
|
||||
const oobNoKeyJSON = `{"host_id":"h1","oob":{
|
||||
"felhom_sshd_active":true,"felhom_sshd_port":8822,"reachable":true,
|
||||
"config_invalid":false,"operator_peer_configured":true,
|
||||
"operator_key_configured":false}}`
|
||||
|
||||
// An `oob` stanza with NO operator_key_configured key at all. Unreachable for any released agent —
|
||||
// the field and the stanza shipped together in v0.72.0 — but absence must be structurally
|
||||
// distinguishable from a reported false, or this defect returns through the version door.
|
||||
const oobLegacyNoFieldJSON = `{"host_id":"h1","oob":{
|
||||
"felhom_sshd_active":true,"felhom_sshd_port":8822,"reachable":true,
|
||||
"config_invalid":false,"operator_peer_configured":true}}`
|
||||
|
||||
func TestOOBDecode_CarriesOperatorKeyConfigured(t *testing.T) {
|
||||
var r HostOOBRow
|
||||
if err := decodeOOBInto(&r, oobHealthyJSON); err != nil {
|
||||
t.Fatalf("decode: %v", err)
|
||||
}
|
||||
if !r.Present {
|
||||
t.Fatal("stanza present but Present=false")
|
||||
}
|
||||
// THE ASSERTION THIS FILE EXISTS FOR. Before R-260 the struct had no field, so this line does
|
||||
// not compile against the old code — which is the red-proof, recorded rather than described.
|
||||
if !r.OperatorKeyConfigured {
|
||||
t.Error("operator_key_configured=true on the wire did not reach the row — it is being dropped at the decode boundary, which is R-260")
|
||||
}
|
||||
if !r.OperatorKeyReported {
|
||||
t.Error("the field was on the wire; OperatorKeyReported must be true")
|
||||
}
|
||||
if r.WGHandshakeAgeS == nil || *r.WGHandshakeAgeS != 42 {
|
||||
t.Errorf("wg_handshake_age_s did not decode: %v", r.WGHandshakeAgeS)
|
||||
}
|
||||
if r.HealedAt != "2026-08-08T01:02:03Z" {
|
||||
t.Errorf("healed_at did not decode: %q", r.HealedAt)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOOBDecode_ReportedFalseIsDistinctFromNotReported(t *testing.T) {
|
||||
var no HostOOBRow
|
||||
if err := decodeOOBInto(&no, oobNoKeyJSON); err != nil {
|
||||
t.Fatalf("decode: %v", err)
|
||||
}
|
||||
if no.OperatorKeyConfigured {
|
||||
t.Error("operator_key_configured=false decoded as true")
|
||||
}
|
||||
if !no.OperatorKeyReported {
|
||||
t.Error("the agent DID say false; that must be distinguishable from never saying")
|
||||
}
|
||||
|
||||
var legacy HostOOBRow
|
||||
if err := decodeOOBInto(&legacy, oobLegacyNoFieldJSON); err != nil {
|
||||
t.Fatalf("decode: %v", err)
|
||||
}
|
||||
if legacy.OperatorKeyReported {
|
||||
t.Error("no operator_key_configured key on the wire, yet OperatorKeyReported=true — absence is being read as a statement")
|
||||
}
|
||||
// Both are `false`; only the Reported flag tells them apart. If a future change collapses the
|
||||
// pointer to a plain bool, this is the assertion that fails.
|
||||
if no.OperatorKeyConfigured != legacy.OperatorKeyConfigured {
|
||||
t.Fatal("precondition of this test changed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOOBDecode_AbsentStanzaIsNotPresent(t *testing.T) {
|
||||
var r HostOOBRow
|
||||
_ = decodeOOBInto(&r, `{"host_id":"h1"}`)
|
||||
if r.Present {
|
||||
t.Error("no oob stanza, yet Present=true")
|
||||
}
|
||||
if r.OperatorKeyReported {
|
||||
t.Error("no oob stanza, yet OperatorKeyReported=true")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOOBDecode_MalformedDegradesToZeroNeverPanics(t *testing.T) {
|
||||
var r HostOOBRow
|
||||
_ = decodeOOBInto(&r, `{"oob":`) // truncated
|
||||
if r.Present {
|
||||
t.Error("malformed JSON must not yield Present=true")
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user