R-195: a customer with no machine ever bound does not alarm (hub v0.92.0) + R-193/R-192 spike
gates / gates (push) Successful in 7s

Part 4 (ships): `david` — a prospective customer with hosts=0, host_deletions=0,
reports=0 — e-mailed an expected_dbdump_missed ERROR at 03:00 UTC three mornings
running. The existing down-skip could never cover it: it reads the staleness
checker's state, which is seeded from a query over the `reports` table, so a
customer that never reported has no state at all and GetState() returns "" rather
than "down". store.HasEverBoundHost (hosts row OR host_deletions tombstone) is
consulted once per customer at the top of the deadline loop. The discriminator is
"was a host EVER bound", never "has a report arrived" — a box installed and never
heard from is a real fault and keeps alarming. Fail-OPEN on a read error. Red-proof
observed: removing the guard fails with `got [expected_dbdump_missed]`, verbatim the
event david sent.

Parts 0-3 (spike, NO production code for R-193/R-192):
audits/SPIKE-offsite-credential-recovery-2026-08-04.md establishes that the one-shot
provider password is the RECOVERABLE secret and the restic repository password is the
irreplaceable one — and that a guest rebuild mints a fresh one, orphaning the previous
off-site history. Measured without touching a box, by comparing
host_escrow.restic_pw_sha256 against host_escrow_superseded: BOTH demo boxes changed
(demo-hp 15 snapshots / 40.9 MB, demo-felhom 36 snapshots / 1.14 GB). demo-felhom's
"lucky" 76-second recovery restored delivery and not the repository, silently, for 13h.
ReissueCredentials does NOT rotate the restic password (R-39's record and two hub
comments are wrong -> R-196); candidate (b) is not implementable against a
zero-knowledge escrow; candidate (a) already exists as F3 and is wired to the wrong
event. Ends in ranked options and an unanswered question for the operator.

R-195 SHIPPED; R-196 + R-197 filed; R-192 + R-193 updated, neither closed.
This commit is contained in:
2026-08-04 11:04:39 +02:00
parent f456835bbc
commit 7fff45d688
9 changed files with 815 additions and 6 deletions
+45
View File
@@ -1,3 +1,48 @@
## v0.92.0 — a customer with no machine ever bound does not alarm (2026-08-04, R-195)
`david` is a real prospective customer whose record was created 2026-08-01 with **no host ever bound**
`hosts=0`, `host_deletions=0`, `host_reports=0`, `reports=0` — and it e-mailed an
`expected_dbdump_missed` **error** at 03:00 UTC on 2026-08-02, 08-03 and 08-04. Nothing has ever been
expected of that customer, so the honest verdict is UNKNOWN, not missed.
**The mechanism is the interesting half, because the guard that should have covered this is keyed off
the wrong thing.** `CheckBackupDeadlines` already skips nodes that are `down`, and that skip reads
`StalenessChecker.GetState()`, whose map is seeded from `store.GetCustomers()` — **a query over the
`reports` table**. A customer with zero reports is in no row, gets no state at all, and `GetState()`
returns `""` rather than `"down"`. **The skip misses exactly the customer it would most obviously
cover.** Corroborated on live data: `peti-felhom` is active with a host deleted 2026-07-15 and does
*not* alarm, because it has 482 old reports and is therefore `down`. The backup half was already safe
(`reportJSON == ""` → skip); the DB-dump half had no guard at all.
- **`store.HasEverBoundHost(customerID)`** — a live `hosts` row **OR** a `host_deletions` tombstone.
Consulted once per customer at the top of the deadline loop; a false verdict skips the customer with
an INFO line and its own counter in the summary.
- **The discriminator is "was a host EVER bound", NOT "has a report arrived", and that is the whole
design.** A box that was installed, bound and then went silent is a real fault and must keep
alarming — it has a `hosts` row, so it is judged. Only a customer that never had a machine is
UNKNOWN. The tombstone is included for the same reason: a customer whose host was deleted *had* one,
and that judgement belongs to the staleness checker, not to this predicate.
- **Fail-OPEN on a read error.** An unreadable binding must never SUPPRESS a real alarm; it logs and
judges anyway.
- **The anchored-verdict structure is untouched** (R-81/v0.73.0). This is the same invariant — absence
is UNKNOWN until an anchor elapses — applied one level up, at the question of whether there is a
subject at all.
**Tests, including the one that matters.** `TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms`
is a real shape — a machine installed and never phoned home — and it is indistinguishable from `david`
on every signal except the one the guard reads. If the guard is ever "simplified" to key off report
presence, customer age or a name pattern, it goes red. Plus a bound-then-went-quiet case, a
deleted-host case, and the predicate's own three inputs. **Companion red-proof observed:** removing the
guard fails `TestCheckBackupDeadlines_NeverBoundHost_Silent` with `got [expected_dbdump_missed]`
verbatim the event `david` sent three mornings running. Restored after.
**`david`'s record was not modified.** The record is correct; the alarm was what was wrong.
**Shipped alongside a spike that deliberately shipped NO code** —
`documentation/audits/SPIKE-offsite-credential-recovery-2026-08-04.md` (R-193/R-192): what is one-shot
in the offsite path, and what a guest rebuild really costs. It ends in ranked options and a question
for the operator; R-196 and R-197 were minted from it and are not implemented here.
## v0.91.1 — observation may only WIDEN a tier's window, never tighten it (2026-08-03, R-86 Part 2)
**Found by checking v0.91.0 against the live box before trusting it, not by review.** demo-felhom's
+34 -3
View File
@@ -336,7 +336,7 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
midnightBudapest := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, budapest)
sinceUTC := midnightBudapest.UTC()
var backupMissed, dbdumpMissed, skipped, deferred int
var backupMissed, dbdumpMissed, skipped, deferred, unbound int
for _, id := range customerIDs {
// Skip nodes that are down — they already have staleness events
@@ -350,6 +350,37 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
continue
}
// ── R-195: a customer with NO machine EVER bound is UNKNOWN, not missed ────────────────
//
// Both verdicts below ask "did the thing we expect every day happen?". For a customer
// that has never had a machine bound, nothing has ever been expected, so the honest
// answer is UNKNOWN — the same invariant assessBackupFreshness states above, applied one
// level up, at the question of whether there is a subject at all.
//
// The discriminator is "was a host EVER bound", NOT "has a report arrived". That is the
// case this check must not break: a box that was installed, bound, and then went silent
// has a real fault and must keep alarming. It is bound, so it is judged.
//
// WHY THIS WAS REACHABLE AT ALL, measured 2026-08-04: the down-skip above is what
// protects every other silent customer, and it reads the staleness checker's state — which
// is seeded from the `reports` table (store.GetCustomers). A customer that has NEVER
// reported appears in no report row, so it gets no staleness state at all and GetState()
// returns "" rather than "down". The skip misses exactly the customer it would most
// obviously cover, and the DB-dump half below then fires every night: `david`, a
// prospective customer whose record was created 2026-08-01 with no machine ever bound,
// e-mailed an expected_dbdump_missed ERROR at 03:00 UTC on three consecutive days.
//
// Fail-open on a read error: an unreadable binding must never SUPPRESS a real alarm.
if bound, berr := s.HasEverBoundHost(id); berr != nil {
logger.Printf("[WARN] Deadline check: failed to read host binding for %s (judging anyway): %v", id, berr)
} else if !bound {
// Visible, per the v0.73.0 Part-7 precedent below: a quiet check must never be
// indistinguishable from a check that did not run. Once daily, one line per customer.
logger.Printf("[INFO] Deadline check: %s has no host EVER bound — all deadline verdicts UNKNOWN (no alarm)", id)
unbound++
continue
}
// Backup freshness from the agent's host-report (PBS snapshots + vzdump),
// the authoritative offsite-backup signal post-slice-8C.
reportJSON, rerr := s.GetLatestHostReportJSON(id)
@@ -423,6 +454,6 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
}
}
logger.Printf("[INFO] Deadline check: %d customers, %d backup missed, %d backup unknown (deferred), %d dbdump missed, %d skipped (down)",
len(customerIDs), backupMissed, deferred, dbdumpMissed, skipped)
logger.Printf("[INFO] Deadline check: %d customers, %d backup missed, %d backup unknown (deferred), %d dbdump missed, %d skipped (down), %d unknown (no host ever bound)",
len(customerIDs), backupMissed, deferred, dbdumpMissed, skipped, unbound)
}
@@ -0,0 +1,148 @@
package monitor
import (
"io"
"log"
"path/filepath"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// R-195 — a customer with NO machine EVER bound must not alarm; a customer WITH one must.
//
// Origin, measured on the live hub 2026-08-04: `david` is a prospective customer whose record was
// created 2026-08-01 16:51:49 with no host ever bound (hosts=0, host_deletions=0, host_reports=0,
// reports=0). It e-mailed an `expected_dbdump_missed` ERROR at 03:00 UTC on 08-02, 08-03 and 08-04.
//
// The mechanism, established at source: the down-skip in CheckBackupDeadlines reads
// StalenessChecker.GetState(), whose map is seeded from store.GetCustomers() — a query over the
// `reports` table. A customer with zero reports is in no row, so it has no state, and GetState()
// returns "" rather than "down". The skip that protects every other silent customer misses the one
// that never reported at all.
//
// These tests pin BOTH halves. A suite that only proved the silence would pass against an
// implementation that never alarms, which is strictly worse than the defect it replaces.
// newUnboundStore creates a store holding ONE active customer and NO host row at all.
func newUnboundStore(t *testing.T) *store.Store {
t.Helper()
st, err := store.New(filepath.Join(t.TempDir(), "test.db"), log.New(io.Discard, "", 0))
if err != nil {
t.Fatalf("store.New: %v", err)
}
t.Cleanup(func() { st.Close() })
if err := st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c1", APIKey: "ck", RetrievalPassword: "p"}); err != nil {
t.Fatalf("SaveCustomerConfig: %v", err)
}
return st
}
// ── Half 1: the silence ────────────────────────────────────────────────────────────────────────
// TestCheckBackupDeadlines_NeverBoundHost_Silent is the david case.
//
// COMPANION RED-PROOF (observed): deleting the HasEverBoundHost guard from CheckBackupDeadlines
// makes this test fail with
//
// deadline_unbound_test.go: a customer with NO host ever bound must raise NOTHING;
// got [expected_dbdump_missed]
//
// which is verbatim the event `david` e-mailed three mornings running. Restored after.
func TestCheckBackupDeadlines_NeverBoundHost_Silent(t *testing.T) {
st := newUnboundStore(t)
// No UpsertHost, no host-report, no db_dump_completed event — nothing has ever been expected.
got := runDeadline(t, st)
if len(got) != 0 {
t.Fatalf("a customer with NO host ever bound must raise NOTHING; got %v", got)
}
}
// TestHasEverBoundHost pins the predicate itself across its three inputs, because the whole
// behaviour above turns on it and a predicate that answered `false` for everything would make the
// test above pass while silencing the entire fleet.
func TestHasEverBoundHost(t *testing.T) {
st := newUnboundStore(t)
if bound, err := st.HasEverBoundHost("c1"); err != nil || bound {
t.Fatalf("no host rows → want (false,nil); got (%v,%v)", bound, err)
}
if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
t.Fatalf("UpsertHost: %v", err)
}
if bound, err := st.HasEverBoundHost("c1"); err != nil || !bound {
t.Fatalf("live host row → want (true,nil); got (%v,%v)", bound, err)
}
// An unknown customer is never bound — the predicate must not answer from another customer's rows.
if bound, err := st.HasEverBoundHost("nobody"); err != nil || bound {
t.Fatalf("unknown customer → want (false,nil); got (%v,%v)", bound, err)
}
}
// ── Half 2: THE RED-PROOF THAT MATTERS — a bound machine must still alarm ───────────────────────
// TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms is the case the change could break,
// and it is a real shape: a machine that was installed and bound and never phoned home. It has a
// `hosts` row and zero reports — indistinguishable from `david` on every signal EXCEPT the one the
// guard discriminates on. If the guard is ever "simplified" to key off report presence, customer
// age or a name pattern, this test goes red.
func TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms(t *testing.T) {
st := newUnboundStore(t)
if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
t.Fatalf("UpsertHost: %v", err)
}
// No host-report and no db_dump_completed — the box was bound and never said anything.
got := runDeadline(t, st)
if !has(got, "expected_dbdump_missed") {
t.Fatalf("a BOUND machine that never reported is a real fault and must still alarm; got %v", got)
}
}
// TestCheckBackupDeadlines_BoundThenWentQuiet_StillAlarms: the machine reported once, days ago,
// and stopped. Nothing about the guard may suppress that.
//
// The staleness checker's down-skip is nil here (runDeadline passes nil), which is deliberate: it
// isolates THIS guard. In production a genuinely down node is skipped by staleness and gets its own
// node_down event — that path is unchanged and is not what this test is about.
func TestCheckBackupDeadlines_BoundThenWentQuiet_StillAlarms(t *testing.T) {
st := newUnboundStore(t)
if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
t.Fatalf("UpsertHost: %v", err)
}
// One host-report whose newest backup evidence is 9 days old, and a db dump that last
// completed 5 days ago (i.e. not since midnight).
report := hostReportJSON(t, [][2]string{{rfc(-9 * 24 * time.Hour), "ok"}}, nil)
if err := st.SaveHostReport("h1", "c1", []byte(report), store.HostReportDenorm{}); err != nil {
t.Fatalf("SaveHostReport: %v", err)
}
got := runDeadline(t, st)
if !has(got, "expected_backup_missed") {
t.Fatalf("a bound machine that went quiet with stale backups must still raise expected_backup_missed; got %v", got)
}
if !has(got, "expected_dbdump_missed") {
t.Fatalf("a bound machine that went quiet must still raise expected_dbdump_missed; got %v", got)
}
}
// TestCheckBackupDeadlines_DeletedHost_StillJudged: the host row is gone but a tombstone remains
// (peti-felhom's live shape). The customer HAD a machine, so this check must not take over the
// judgement — it hands off to the staleness down-skip exactly as before the change.
func TestCheckBackupDeadlines_DeletedHost_StillJudged(t *testing.T) {
st := newUnboundStore(t)
if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
t.Fatalf("UpsertHost: %v", err)
}
if err := st.DeleteHost("h1", false); err != nil {
t.Fatalf("DeleteHost: %v", err)
}
if bound, err := st.HasEverBoundHost("c1"); err != nil || !bound {
t.Fatalf("a DELETED host is still a machine that was once bound → want (true,nil); got (%v,%v)", bound, err)
}
got := runDeadline(t, st)
if !has(got, "expected_dbdump_missed") {
t.Fatalf("a customer whose host was deleted was still bound and stays judged here; got %v", got)
}
}
+22
View File
@@ -2262,6 +2262,28 @@ func (s *Store) GetHostByCustomer(customerID string) (*Host, error) {
return h, err
}
// HasEverBoundHost reports whether a machine was EVER bound to this customer — a live row in
// `hosts` OR a tombstone in `host_deletions`. It answers "was anything ever expected of this
// customer", which is the question the deadline verdicts actually need (R-195).
//
// It is deliberately NOT "has a report arrived", and the distinction is the whole point: a box
// that was installed, bound, and then went silent IS bound, and its silence is a real fault that
// must keep alarming. Only a customer that never had a machine at all is UNKNOWN.
//
// `host_deletions` is included because a customer whose host was removed HAD one — the deadline
// caller reaches its down-skip for that shape, and this predicate must not quietly take over a
// judgement the staleness checker owns.
func (s *Store) HasEverBoundHost(customerID string) (bool, error) {
var n int
if err := s.db.QueryRow(
`SELECT EXISTS(SELECT 1 FROM hosts WHERE customer_id = ?)
OR EXISTS(SELECT 1 FROM host_deletions WHERE customer_id = ?)`,
customerID, customerID).Scan(&n); err != nil {
return false, err
}
return n != 0, nil
}
// ListHostsByCustomer returns the customer's hosts ordered by host_id (v0.47.0 — the
// customer page's Host tab is a LIST by design: 1 host today, N for a later HA cluster).
// Uses the idx_hosts_customer index.