hub v0.105.0: the third name, a machine told to be quiet, and a guard for the hub's own words

Hub only. No controller change, no agent change, no wire change — nothing to bake.
demo-hp untouched: the operator is re-deploying it this evening.

R-323 — the five-word phrase is „Tulajdonosi jelmondat". It was „Visszaállító
jelszó": one word from the name retired last week, and false besides — it restores
nothing, it proves the account owns the box being bound. Five sites, all in the hub;
felhom-controller and felhom-agent carry the name nowhere, so no halt and no bake.
Both suggested names were rejected with reasons: „Fiókjelszó" would collide with the
dashboard login (a DIFFERENT real secret), and „Összekötési jelszó" would leave the
two factors on this page separated only by kód-versus-jelszó — the exact shape being
removed, since the other factor is the „Párosító kód". The chosen name differs on
both axes, stem and noun. Naming only; the acceptance pin drives the real handler.

R-324 — the hub's customer copy is under a guard for the first time. Retired names
banned across all 95 hub files; retrieval stems registered in four declared customer
surfaces. The selftest found a defect in its own instrument on the first run. One
shared vocabulary in scripts/, drift-checked into the controller gate rather than
copied (R-325 removes the scaffold).

R-321 — a machine we told to be quiet is no longer reported as dead, and it was two
doors, not one: because the state is RECORDED rather than deleted, the morning
deadline check can skip it too. A deleted state returns "", which is not "down" —
R-195's shape returning through a second door. The clock runs from the report the hub
can see, so re-enabling starts it there and emits no recovery for an outage that never
happened. Three red-proofs; the one that matters showed a genuinely dead machine
sitting at "disabled" when the suppression was made unconditional.

R-326 — "which claims are unproven" is answerable by a command now. The nine I have
been repeating was the count of claims the 9 August pass DOWNGRADED, not the count of
unproven ones. The real figures: 55 claims, 23 walked, 32 not — and only 6 of those 32
cite evidence. Its first run found a stale claim (R-327).

hub/internal/monitor/deadline.go

@@ -324,6 +324,16 @@ func init() {
 //
 // Customers whose nodes are "down" (no report in >1h) are skipped — they
 // already have staleness events.
+// stalenessState reads a customer's staleness state, tolerating a nil checker. Named rather than
+// inlined so the two states this function skips on are visible in one place — a second caller adding
+// a third state must not have to rediscover that the nil check exists.
+func stalenessState(staleness *StalenessChecker, customerID string) string {
+	if staleness == nil {
+		return ""
+	}
+	return staleness.GetState(customerID)
+}
+
 func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent EventNotifyFunc, logger *log.Logger) {
 	customerIDs, err := s.GetActiveCustomerIDs()
 	if err != nil {
@@ -339,8 +349,15 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
 	var backupMissed, dbdumpMissed, skipped, deferred, unbound int
 
 	for _, id := range customerIDs {
-		// Skip nodes that are down — they already have staleness events
-		if staleness != nil && staleness.GetState(id) == "down" {
+		// Skip nodes that are down — they already have staleness events.
+		//
+		// R-321 adds StateDisabled to the same skip, and it is NOT cosmetic. This check is a second
+		// door onto the same false alarm: a box whose reporting we switched off deliberately is not
+		// "down" (the staleness checker suppresses that), so without this it would keep e-mailing
+		// `expected_backup_missed` / `expected_dbdump_missed` every morning about a machine we asked
+		// to be quiet. That is R-195's shape exactly — a skip keyed off the wrong fact missing the
+		// customer it would most obviously cover — which is why both doors are closed together.
+		if st := stalenessState(staleness, id); st == "down" || st == StateDisabled {
 			skipped++
 			continue
 		}

hub/internal/monitor/staleness.go

@@ -3,6 +3,7 @@ package monitor
 import (
 	"fmt"
 	"log"
+	"strings"
 	"sync"
 	"time"
 
@@ -13,6 +14,27 @@ import (
 // to trigger notification dispatch. Keeps monitor decoupled from notify.
 type EventNotifyFunc func(customerID, eventType, severity, message, detailsJSON, source string)
 
+// StateDisabled is the state of a customer whose box we DELIBERATELY told to stop reporting.
+//
+// R-321. Switching a box's hub reporting off is a supported product state: the controller sends one
+// final report carrying health.status = "disabled" (felhom-controller
+// `cmd/controller/main.go:1253`) and then goes quiet BY DESIGN. That status is parsed and persisted
+// (`store.go:953-955` → `reports.health_status`), reaches this checker on every pass inside
+// `CustomerSummary.HealthStatus` (`store.go:40`), and the operator roll-up already renders such a
+// customer as `disabled` (`web/rollup.go:25`).
+//
+// This checker ignored it and measured only the AGE of the last report — so a machine we asked to be
+// quiet went stale at 30 minutes, down at 60, and e-mailed the operator twice about an outage we
+// caused on purpose. That is the false alarm that teaches people to ignore the channel, and it is a
+// sibling of R-195, where the guard that should have covered a customer was keyed off the wrong fact.
+//
+// It is a STATE here rather than a deletion (the `blocked` branch below deletes) because other
+// checkers consult GetState: CheckBackupDeadlines skips a customer that is "down", and a DELETED
+// state returns "" — which is not "down", so a disabled box would have gone on alarming
+// `expected_backup_missed` from a different function. Exactly the R-195 shape returning through a
+// second door.
+const StateDisabled = "disabled"
+
 // StalenessChecker monitors customer report freshness and generates
 // node_stale / node_down / node_recovered events on state transitions.
 type StalenessChecker struct {
@@ -94,6 +116,27 @@ func (sc *StalenessChecker) Check() {
 			continue
 		}
 
+		// R-321 — a machine we told to be quiet is not a machine that died.
+		//
+		// The age-based transition is skipped entirely; no stale, no down, no e-mail. The state is
+		// RECORDED rather than deleted so the deliberate silence is visible to anything that asks
+		// (see StateDisabled). `downtimeStart` is cleared on ENTRY so that a later, genuine outage
+		// cannot compute its duration from a clock that started before we asked for the silence.
+		//
+		// The discriminator is the box's OWN last word, not an inference: it said `disabled` on the
+		// way out. LIMIT, stated rather than hidden: if reporting is re-enabled and the box then
+		// fails to report at all, the hub still sees only that final `disabled` report and keeps
+		// suppressing. The hub cannot distinguish that from "still switched off" — its view changes
+		// only when a report arrives. This is why the state is made VISIBLE: an operator who
+		// re-enabled a box and still sees `disabled` is being told it has not come back.
+		if strings.EqualFold(c.HealthStatus, StateDisabled) {
+			if sc.states[c.CustomerID] != StateDisabled {
+				sc.states[c.CustomerID] = StateDisabled
+				delete(sc.downtimeStart, c.CustomerID)
+			}
+			continue
+		}
+
 		age := time.Since(c.ReceivedAt)
 		var newState string
 		switch {
@@ -106,8 +149,20 @@ func (sc *StalenessChecker) Check() {
 		}
 
 		oldState := sc.states[c.CustomerID]
-		if oldState == "" {
-			// New customer — set state without event
+		if oldState == "" || oldState == StateDisabled {
+			// New customer — set state without event.
+			//
+			// R-321 adds the re-enabled case to the SAME branch, deliberately. A box coming back
+			// from a deliberate silence is a first observation, not a recovery: emitting here would
+			// send `node_recovered` for an outage that never happened — and would do it every time
+			// an operator switched reporting back on.
+			//
+			// THE RE-ENABLEMENT CLOCK, and this is the judgement: it is the age of the report the
+			// hub can actually see. For a box that reports on re-enabling, that report IS the
+			// re-enablement, so the clock starts there and scenario D (re-enabled, then genuinely
+			// quiet) alarms on a normal schedule timed from the moment it came back. Timing from the
+			// last report BEFORE the switch-off would fire an instant stale/down for a quiet period
+			// we asked for — a false alarm produced by fixing false alarms.
 			sc.states[c.CustomerID] = newState
 			continue
 		}

hub/internal/monitor/staleness_disabled_test.go

@@ -0,0 +1,226 @@
+package monitor
+
+// R-321 — A MACHINE WE TOLD TO BE QUIET IS NOT A MACHINE THAT DIED.
+//
+// Switching a box's hub reporting off is a supported product state. The controller sends one final
+// report carrying health.status = "disabled" and then goes quiet BY DESIGN. The hub parses and stores
+// that status, and the operator roll-up already renders such a customer as `disabled` — but this
+// checker measured only the AGE of the last report, so the machine went stale at 30 minutes, down at
+// 60, and e-mailed the operator twice about an outage we caused on purpose.
+//
+// The discriminator is the box's OWN last word, not an inference, and it is in the data this checker
+// already reads (`CustomerSummary.HealthStatus`). Suppressing on a guess would be worse than the
+// false alarm it removes; suppressing on the machine's own declaration is not a guess.
+
+import (
+	"database/sql"
+	"io"
+	"log"
+	"path/filepath"
+	"testing"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-hub/internal/store"
+	_ "modernc.org/sqlite"
+)
+
+// seedStalenessCustomer creates a store with one customer and one report of the given health status,
+// backdated by `age`. Reports are inserted through the REAL SaveReport path so the health_status
+// denormalization is exercised rather than hand-set — a test that writes the column directly cannot
+// see a decode that never happens, which is the R-260 class.
+func seedStalenessCustomer(t *testing.T, health string, age time.Duration) (*store.Store, string) {
+	t.Helper()
+	path := filepath.Join(t.TempDir(), "t.db")
+	st, err := store.New(path, log.New(io.Discard, "", 0))
+	if err != nil {
+		t.Fatalf("store.New: %v", err)
+	}
+	t.Cleanup(func() { st.Close() })
+	if err := st.SaveCustomerConfig(&store.CustomerConfig{
+		CustomerID: "c1", APIKey: "ck", RetrievalPassword: "p", Status: "active",
+	}); err != nil {
+		t.Fatalf("SaveCustomerConfig: %v", err)
+	}
+	saveReportAged(t, st, path, health, age)
+	return st, path
+}
+
+// saveReportAged saves a controller report with the given health status and then backdates its
+// received_at. The backdate is a raw UPDATE because there is no production path that writes an old
+// timestamp — which is exactly why it is confined to this one helper.
+func saveReportAged(t *testing.T, st *store.Store, path, health string, age time.Duration) {
+	t.Helper()
+	body := `{"customer_id":"c1","health":{"status":"` + health + `"}}`
+	if err := st.SaveReport("c1", []byte(body)); err != nil {
+		t.Fatalf("SaveReport: %v", err)
+	}
+	if age <= 0 {
+		return
+	}
+	// Backdating goes through a SECOND connection to the same file rather than a test-only method on
+	// Store. No production path writes an old received_at, so exposing one would widen the store's
+	// API for a fixture — and this keeps the ageing confined to the one helper that needs it.
+	db, err := sql.Open("sqlite", path)
+	if err != nil {
+		t.Fatalf("open for backdate: %v", err)
+	}
+	defer db.Close()
+	// GetCustomers selects the report with MAX(received_at), not MAX(id) — so backdating only the
+	// row just written would leave an EARLIER, fresher-looking row as the one the checker reads, and
+	// the fixture would quietly test the wrong report. (Found by two scenarios failing on a change
+	// that was correct: the instrument was wrong, not the code.) Every older row is therefore pushed
+	// further back, so the row just written is unambiguously the latest.
+	when := time.Now().UTC().Add(-age)
+	newest := when.Format("2006-01-02 15:04:05")
+	older := when.Add(-time.Hour).Format("2006-01-02 15:04:05")
+	if _, err := db.Exec(
+		`UPDATE reports SET received_at = ? WHERE customer_id = 'c1' AND id < (SELECT MAX(id) FROM reports WHERE customer_id = 'c1')`,
+		older); err != nil {
+		t.Fatalf("age the older rows: %v", err)
+	}
+	if _, err := db.Exec(
+		`UPDATE reports SET received_at = ? WHERE id = (SELECT MAX(id) FROM reports WHERE customer_id = 'c1')`,
+		newest); err != nil {
+		t.Fatalf("backdate: %v", err)
+	}
+}
+
+// newChecker builds a checker with a 30-minute threshold and records every event it emits.
+func newChecker(t *testing.T, st *store.Store) (*StalenessChecker, *[]string) {
+	t.Helper()
+	var events []string
+	sc := NewStalenessChecker(st, 30*time.Minute, func(cid, et, sev, msg, det, src string) {
+		events = append(events, et)
+	}, log.New(io.Discard, "", 0))
+	return sc, &events
+}
+
+// ── A — a machine deliberately silent for days ──────────────────────────────────────────────────
+//
+// WRONG OUTCOME GUARDED: stale, then down, then two e-mails — which is what happens today.
+func TestStaleness_A_DeliberatelySilentDoesNotAlarm(t *testing.T) {
+	// The box is HEALTHY and OBSERVED first, then switched off. Seeding it already-disabled would
+	// make this test pass vacuously: the checker's new-customer branch sets the first state without
+	// an event, so a customer that was never seen healthy can never emit a transition — and the
+	// assertion below would hold even with the suppression deleted. (Confirmed: it did. The
+	// red-proof for this scenario passed on the state assertion alone until this line was added.)
+	st, path := seedStalenessCustomer(t, "ok", 0)
+	sc, events := newChecker(t, st)
+	sc.Check()
+	if sc.GetState("c1") != "ok" {
+		t.Fatalf("setup: the machine should start observed-healthy, got %q", sc.GetState("c1"))
+	}
+
+	// Reporting is switched off. The box says so on the way out, then goes quiet for three days.
+	saveReportAged(t, st, path, "disabled", 72*time.Hour)
+
+	for i := 0; i < 3; i++ { // several passes: a suppression that only holds once is not a suppression
+		sc.Check()
+	}
+
+	if len(*events) != 0 {
+		t.Fatalf("a deliberately-disabled machine emitted %v — three days quiet BY REQUEST", *events)
+	}
+	// The silence must be VISIBLE, not merely un-alarmed: quiet-on-purpose and quiet-by-accident
+	// must not look identical, and an unknown is never drawn as healthy.
+	if got := sc.GetState("c1"); got != StateDisabled {
+		t.Errorf("state = %q, want %q — the deliberate silence is invisible", got, StateDisabled)
+	}
+}
+
+// ── B — a machine that simply stopped reporting ─────────────────────────────────────────────────
+//
+// WRONG OUTCOME GUARDED: silence mistaken for a deliberate switch-off — THE ALARM THAT MATTERS,
+// suppressed. This is the one that must never regress.
+func TestStaleness_B_GenuineSilenceStillAlarms(t *testing.T) {
+	st, path := seedStalenessCustomer(t, "ok", 0)
+	sc, events := newChecker(t, st)
+	sc.Check() // first observation: healthy, no event
+
+	// The same customer now goes quiet for real.
+	saveReportAged(t, st, path, "ok", 3*time.Hour)
+	sc.Check()
+
+	if got := sc.GetState("c1"); got != "down" {
+		t.Fatalf("a genuinely silent machine is %q, want \"down\" — a real alarm was swallowed", got)
+	}
+	if len(*events) == 0 {
+		t.Fatal("a genuinely silent machine emitted NO event — the suppression is over-broad")
+	}
+}
+
+// ── C — a disabled machine that is re-enabled and reports promptly ──────────────────────────────
+//
+// WRONG OUTCOME GUARDED: a recovery e-mail for an outage that never happened.
+func TestStaleness_C_ReEnabledReportsCleanlyWithNoRecoveryEvent(t *testing.T) {
+	st, path := seedStalenessCustomer(t, "disabled", 72*time.Hour)
+	sc, events := newChecker(t, st)
+	sc.Check()
+	if sc.GetState("c1") != StateDisabled {
+		t.Fatalf("setup: expected the disabled state, got %q", sc.GetState("c1"))
+	}
+
+	// Reporting is switched back on and the box reports immediately.
+	saveReportAged(t, st, path, "ok", 0)
+	sc.Check()
+
+	if len(*events) != 0 {
+		t.Fatalf("re-enabling emitted %v — there was no outage to recover from", *events)
+	}
+	if got := sc.GetState("c1"); got != "ok" {
+		t.Errorf("state after re-enable = %q, want \"ok\"", got)
+	}
+}
+
+// ── D — a disabled machine that is re-enabled and then goes genuinely quiet ─────────────────────
+//
+// WRONG OUTCOME GUARDED: permanently silenced because it was once disabled. Also pins THE CLOCK
+// JUDGEMENT: timing runs from the report the box sent on re-enabling, not from the last report
+// before the switch-off — otherwise coming back would fire an instant false alarm about a quiet
+// period we asked for.
+func TestStaleness_D_ReEnabledThenQuietAlarmsFromReEnablement(t *testing.T) {
+	st, path := seedStalenessCustomer(t, "disabled", 72*time.Hour)
+	sc, events := newChecker(t, st)
+	sc.Check()
+
+	// Re-enabled, reports promptly — the clean first observation.
+	saveReportAged(t, st, path, "ok", 0)
+	sc.Check()
+	if len(*events) != 0 {
+		t.Fatalf("the re-enable itself emitted %v", *events)
+	}
+
+	// …and then goes quiet for real. Timed from THIS report, it is down.
+	saveReportAged(t, st, path, "ok", 3*time.Hour)
+	sc.Check()
+
+	if got := sc.GetState("c1"); got != "down" {
+		t.Fatalf("state = %q, want \"down\" — a once-disabled machine was silenced for ever", got)
+	}
+	if len(*events) == 0 {
+		t.Fatal("no event for a machine that genuinely died after being re-enabled")
+	}
+}
+
+// The deadline check is a SECOND DOOR onto the same false alarm: a disabled box is not "down", so
+// without its own skip it would keep e-mailing expected_backup_missed every morning. R-195's shape,
+// which is why both doors are closed together.
+func TestStaleness_DisabledIsAlsoSkippedByTheDeadlineCheck(t *testing.T) {
+	st, _ := seedStalenessCustomer(t, "disabled", 72*time.Hour)
+	if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
+		t.Fatalf("UpsertHost: %v", err)
+	}
+	sc, _ := newChecker(t, st)
+	sc.Check()
+
+	var events []string
+	CheckBackupDeadlines(st, sc, func(cid, et, sev, msg, det, src string) {
+		events = append(events, et)
+	}, log.New(io.Discard, "", 0))
+
+	for _, e := range events {
+		if e == "expected_backup_missed" || e == "expected_dbdump_missed" {
+			t.Fatalf("the deadline check alarmed on a deliberately-disabled machine: %v", events)
+		}
+	}
+}