hub v0.99.0 — the hub can see whether the operator can get in (R-260); G-1 gate closes, R-247 closes

oobDegraded tested five things and the sixth never arrived.

The agent has emitted `operator_key_configured` on every heartbeat since v0.72.0 — the SAME version
that introduced the `oob` stanza carrying it — and store.HostOOBRow mirrored five of the agent's
eight OOB fields. With no field for it, encoding/json discarded the fact on arrival, so a box with
felhom-sshd active, reachable, a valid config and a configured peer reported `ok` with NO OPERATOR
KEY INSTALLED AT ALL. Not a wrong answer: an answer to a question nobody was asking.
`operator_peer_configured`, which the hub did read, only says the peer IP is in desired-state — that
OOB is MEANT to work, not that entry is possible.

Now decoded: operator_key_configured, plus wg_handshake_age_s and healed_at. The last two ride the
ALERT TEXT and are deliberately NOT in the predicate — widening a check beyond the fact that is now
arriving is how a check stops being read.

SCENARIO F, decided on a measurement rather than a preference. operator_key_configured decodes as a
POINTER: nil = the agent never said, reported distinctly and never as ok. The version gate was
rejected because the field and its stanza shipped in the SAME agent version (v0.72.0), so a stanza
without the field cannot come from any released agent; the fleet is 0.113.0/0.127.0 and the vouched
floor is 0.127.0. Handled explicitly anyway and pinned, because "cannot happen" is a claim this
project has been burned by.

THE MESSAGE NAMES THE FAULT. oobDegradedReason is the single source for both predicate and text, so
the alert can never name a different fault from the one that fired. The old form derived it
separately and had a vocabulary of two — unreachable, or config invalid — with no way to say the key
is missing. The operator reads this at 07:00.

TESTS DRIVE THE DECODE BOUNDARY. Every hub OOB test before this built a HostOOBRow by hand, and a
test written that way CANNOT SEE A FIELD THAT NEVER DECODES — which is how this held a green suite
for five weeks. The pre-existing fixture oobReport() also omitted the field, so those scenarios ran
against a report shape no released agent produces (same family as R-262). Both fixed.

Red-proofs, 8 expected outcomes and 0 wrong, each with the mutation asserted applied: dropping the
field returns the false ok; an unconditional check alerts a healthy box; unknown-as-ok restores the
silent pass.

G-1 CLOSED — scripts/wire_contract_gate.py shipped as ranked, built BEFORE the fixes and seen
failing on 40 fields (documentation/tests/wire-contract-gate-2026-08-08/BEFORE.md). Two instrument
defects the control caught first: a substring false negative (grep -F healed_at matched
privsep_healed_at) and treating dr_recipe as wholly opaque when its top-level sections ARE decoded
through an allow-list that already cost offsite_restic (R-122).

The prompt for this session said "465 emitted tags, eight unreachable". Checked against the repo:
R-260 said "at least eight DECISION-BEARING facts", never eight tags. The real count is 40.

R-260 CLOSED (class gated, sharpest instance fixed). R-247 CLOSED (controller v0.209.0). R-264
MINTED and OPEN — the 21 facts with no consumer, allowlisted with reasons so that gating the class
could not be mistaken for deciding them. Still open and named: R-246, R-255..R-259, R-261..R-263,
and C7's test-comment half.

Capability map checked: it claims OOB access is implemented, never monitored, so no row was untrue;
what was untrue sat one layer down and the row now records it.

repo_gates --fast: all 8 OK. go build/vet/test green in hub, run separately from this commit.
This commit is contained in:
2026-08-08 08:47:02 +02:00
parent 560f0d4451
commit b080ecf411
13 changed files with 836 additions and 58 deletions
+65 -11
View File
@@ -58,18 +58,58 @@ func NewHostOOBChecker(s *store.Store, onEvent EventNotifyFunc, logger *log.Logg
}
// oobDegraded is the degraded predicate: config invalid, OR (OOB meant to work — operator peer
// configured — AND felhom-sshd is not active/reachable).
// configured — AND felhom-sshd is not active/reachable OR the operator's key is not installed).
//
// THE KEY CLAUSE IS NEW (R-260, G-1) AND IT IS THE POINT. Until 2026-08-08 this predicate tested
// five things and the sixth — whether the credential that actually grants entry exists — never
// arrived, because the hub's decoder had no field for it. A box with the service active, reachable,
// a valid config and a configured peer reported `ok` with NO OPERATOR KEY INSTALLED. That is not a
// wrong answer; it is an answer to a question nobody was asking.
//
// It is deliberately gated on OperatorPeerConfigured, exactly like the reachability clause: a box
// where OOB was never set up is not "broken", and widening this check beyond the fact that is now
// arriving is how a check stops being read.
func oobDegraded(r store.HostOOBRow) bool {
return oobDegradedReason(r) != ""
}
// oobDegradedReason returns the SPECIFIC reason a host's operator access is degraded, or "" when it
// is not. The reason is separated from the boolean because the operator reads the alert at 07:00 and
// needs to know WHICH of the things this checks is wrong — "out-of-band access degraded" is true and
// useless.
func oobDegradedReason(r store.HostOOBRow) string {
if !r.Present {
return false
return ""
}
if r.ConfigInvalid {
return true
return "felhom-sshd config invalid (sshd -t fails)"
}
if r.OperatorPeerConfigured && (!r.FelhomSshdActive || !r.Reachable) {
return true
if !r.OperatorPeerConfigured {
return ""
}
return false
if !r.FelhomSshdActive {
return "felhom-sshd is not active"
}
if !r.Reachable {
return "felhom-sshd unreachable (local dial to the OOB port fails)"
}
// The operator key, and the two ways it can be wrong. Both are reported distinctly, and NEITHER
// is a silent ok — an absence read as "the key is installed" is precisely the defect this clause
// was added to end, arriving through the version door instead.
if !r.OperatorKeyReported {
// Unreachable for any released agent: `operator_key_configured` and the `oob` stanza that
// carries it shipped together in agent v0.72.0 (2026-07-05), so a stanza without the field
// cannot come from a version anyone runs — the vouched floor is far above it. Handled
// explicitly anyway, because "cannot happen" is the kind of claim this project has been
// burned by, and pinned by TestOOBDegraded_StanzaWithoutKeyField_IsNotOK.
return "the agent reports operator access but is too old to say whether the operator key is " +
"installed (pre-v0.72.0) — treat entry as UNPROVEN, not working"
}
if !r.OperatorKeyConfigured {
return "the operator's authorized_key is NOT installed — felhom-sshd is up and answering, " +
"and nobody can log in through it"
}
return ""
}
// Check evaluates all hosts and emits oob_degraded / oob_recovered on transitions.
@@ -121,22 +161,36 @@ func (c *HostOOBChecker) IsDegraded(hostID string) bool {
func (c *HostOOBChecker) emit(row store.HostOOBRow, eventType, severity string) {
var msg string
if eventType == "oob_degraded" {
reason := "felhom-sshd unreachable"
if row.ConfigInvalid {
reason = "felhom-sshd config invalid (sshd -t fails)"
// The reason comes from the predicate itself, so the message can never name a different
// fault from the one that fired. The old form derived it separately and could only ever say
// "unreachable" or "config invalid" — it had no vocabulary for a missing operator key,
// which is the fault this checker most needs to be able to name (R-260).
reason := oobDegradedReason(row)
if reason == "" {
reason = "operator access degraded"
}
msg = "Host " + row.HostID + ": OPERATOR ACCESS DEGRADED — " + reason +
". The break-glass net (auto-heal + vaulted root@pam console) is still under the box."
} else {
msg = "Host " + row.HostID + ": operator access recovered (felhom-sshd reachable again)."
}
details, _ := json.Marshal(map[string]any{
det := map[string]any{
"host_id": row.HostID,
"felhom_sshd_port": row.FelhomSshdPort,
"active": row.FelhomSshdActive,
"reachable": row.Reachable,
"config_invalid": row.ConfigInvalid,
})
// R-260: carried for context, NOT consulted by the predicate.
"operator_key_configured": row.OperatorKeyConfigured,
"operator_key_reported": row.OperatorKeyReported,
}
if row.WGHandshakeAgeS != nil {
det["wg_handshake_age_s"] = *row.WGHandshakeAgeS
}
if row.HealedAt != "" {
det["healed_at"] = row.HealedAt
}
details, _ := json.Marshal(det)
c.logger.Printf("[%s] Host OOB: %s (%s)", map[string]string{"warning": "WARN", "info": "INFO"}[severity], row.HostID, eventType)
if _, err := c.store.SaveEvent(row.CustomerID, eventType, severity, msg, string(details), "hub"); err != nil {
c.logger.Printf("[WARN] save %s for %s: %v", eventType, row.HostID, err)