hub v0.105.0: the third name, a machine told to be quiet, and a guard for the hub's own words
gates / gates (push) Successful in 17s
gates / gates (push) Successful in 17s
Hub only. No controller change, no agent change, no wire change — nothing to bake. demo-hp untouched: the operator is re-deploying it this evening. R-323 — the five-word phrase is „Tulajdonosi jelmondat". It was „Visszaállító jelszó": one word from the name retired last week, and false besides — it restores nothing, it proves the account owns the box being bound. Five sites, all in the hub; felhom-controller and felhom-agent carry the name nowhere, so no halt and no bake. Both suggested names were rejected with reasons: „Fiókjelszó" would collide with the dashboard login (a DIFFERENT real secret), and „Összekötési jelszó" would leave the two factors on this page separated only by kód-versus-jelszó — the exact shape being removed, since the other factor is the „Párosító kód". The chosen name differs on both axes, stem and noun. Naming only; the acceptance pin drives the real handler. R-324 — the hub's customer copy is under a guard for the first time. Retired names banned across all 95 hub files; retrieval stems registered in four declared customer surfaces. The selftest found a defect in its own instrument on the first run. One shared vocabulary in scripts/, drift-checked into the controller gate rather than copied (R-325 removes the scaffold). R-321 — a machine we told to be quiet is no longer reported as dead, and it was two doors, not one: because the state is RECORDED rather than deleted, the morning deadline check can skip it too. A deleted state returns "", which is not "down" — R-195's shape returning through a second door. The clock runs from the report the hub can see, so re-enabling starts it there and emits no recovery for an outage that never happened. Three red-proofs; the one that matters showed a genuinely dead machine sitting at "disabled" when the suppression was made unconditional. R-326 — "which claims are unproven" is answerable by a command now. The nine I have been repeating was the count of claims the 9 August pass DOWNGRADED, not the count of unproven ones. The real figures: 55 claims, 23 walked, 32 not — and only 6 of those 32 cite evidence. Its first run found a stale claim (R-327).
This commit is contained in:
@@ -3,6 +3,7 @@ package monitor
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
@@ -13,6 +14,27 @@ import (
|
||||
// to trigger notification dispatch. Keeps monitor decoupled from notify.
|
||||
type EventNotifyFunc func(customerID, eventType, severity, message, detailsJSON, source string)
|
||||
|
||||
// StateDisabled is the state of a customer whose box we DELIBERATELY told to stop reporting.
|
||||
//
|
||||
// R-321. Switching a box's hub reporting off is a supported product state: the controller sends one
|
||||
// final report carrying health.status = "disabled" (felhom-controller
|
||||
// `cmd/controller/main.go:1253`) and then goes quiet BY DESIGN. That status is parsed and persisted
|
||||
// (`store.go:953-955` → `reports.health_status`), reaches this checker on every pass inside
|
||||
// `CustomerSummary.HealthStatus` (`store.go:40`), and the operator roll-up already renders such a
|
||||
// customer as `disabled` (`web/rollup.go:25`).
|
||||
//
|
||||
// This checker ignored it and measured only the AGE of the last report — so a machine we asked to be
|
||||
// quiet went stale at 30 minutes, down at 60, and e-mailed the operator twice about an outage we
|
||||
// caused on purpose. That is the false alarm that teaches people to ignore the channel, and it is a
|
||||
// sibling of R-195, where the guard that should have covered a customer was keyed off the wrong fact.
|
||||
//
|
||||
// It is a STATE here rather than a deletion (the `blocked` branch below deletes) because other
|
||||
// checkers consult GetState: CheckBackupDeadlines skips a customer that is "down", and a DELETED
|
||||
// state returns "" — which is not "down", so a disabled box would have gone on alarming
|
||||
// `expected_backup_missed` from a different function. Exactly the R-195 shape returning through a
|
||||
// second door.
|
||||
const StateDisabled = "disabled"
|
||||
|
||||
// StalenessChecker monitors customer report freshness and generates
|
||||
// node_stale / node_down / node_recovered events on state transitions.
|
||||
type StalenessChecker struct {
|
||||
@@ -94,6 +116,27 @@ func (sc *StalenessChecker) Check() {
|
||||
continue
|
||||
}
|
||||
|
||||
// R-321 — a machine we told to be quiet is not a machine that died.
|
||||
//
|
||||
// The age-based transition is skipped entirely; no stale, no down, no e-mail. The state is
|
||||
// RECORDED rather than deleted so the deliberate silence is visible to anything that asks
|
||||
// (see StateDisabled). `downtimeStart` is cleared on ENTRY so that a later, genuine outage
|
||||
// cannot compute its duration from a clock that started before we asked for the silence.
|
||||
//
|
||||
// The discriminator is the box's OWN last word, not an inference: it said `disabled` on the
|
||||
// way out. LIMIT, stated rather than hidden: if reporting is re-enabled and the box then
|
||||
// fails to report at all, the hub still sees only that final `disabled` report and keeps
|
||||
// suppressing. The hub cannot distinguish that from "still switched off" — its view changes
|
||||
// only when a report arrives. This is why the state is made VISIBLE: an operator who
|
||||
// re-enabled a box and still sees `disabled` is being told it has not come back.
|
||||
if strings.EqualFold(c.HealthStatus, StateDisabled) {
|
||||
if sc.states[c.CustomerID] != StateDisabled {
|
||||
sc.states[c.CustomerID] = StateDisabled
|
||||
delete(sc.downtimeStart, c.CustomerID)
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
age := time.Since(c.ReceivedAt)
|
||||
var newState string
|
||||
switch {
|
||||
@@ -106,8 +149,20 @@ func (sc *StalenessChecker) Check() {
|
||||
}
|
||||
|
||||
oldState := sc.states[c.CustomerID]
|
||||
if oldState == "" {
|
||||
// New customer — set state without event
|
||||
if oldState == "" || oldState == StateDisabled {
|
||||
// New customer — set state without event.
|
||||
//
|
||||
// R-321 adds the re-enabled case to the SAME branch, deliberately. A box coming back
|
||||
// from a deliberate silence is a first observation, not a recovery: emitting here would
|
||||
// send `node_recovered` for an outage that never happened — and would do it every time
|
||||
// an operator switched reporting back on.
|
||||
//
|
||||
// THE RE-ENABLEMENT CLOCK, and this is the judgement: it is the age of the report the
|
||||
// hub can actually see. For a box that reports on re-enabling, that report IS the
|
||||
// re-enablement, so the clock starts there and scenario D (re-enabled, then genuinely
|
||||
// quiet) alarms on a normal schedule timed from the moment it came back. Timing from the
|
||||
// last report BEFORE the switch-off would fire an instant stale/down for a quiet period
|
||||
// we asked for — a false alarm produced by fixing false alarms.
|
||||
sc.states[c.CustomerID] = newState
|
||||
continue
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user