R-195: a customer with no machine ever bound does not alarm (hub v0.92.0) + R-193/R-192 spike
gates / gates (push) Successful in 7s

Part 4 (ships): `david` — a prospective customer with hosts=0, host_deletions=0,
reports=0 — e-mailed an expected_dbdump_missed ERROR at 03:00 UTC three mornings
running. The existing down-skip could never cover it: it reads the staleness
checker's state, which is seeded from a query over the `reports` table, so a
customer that never reported has no state at all and GetState() returns "" rather
than "down". store.HasEverBoundHost (hosts row OR host_deletions tombstone) is
consulted once per customer at the top of the deadline loop. The discriminator is
"was a host EVER bound", never "has a report arrived" — a box installed and never
heard from is a real fault and keeps alarming. Fail-OPEN on a read error. Red-proof
observed: removing the guard fails with `got [expected_dbdump_missed]`, verbatim the
event david sent.

Parts 0-3 (spike, NO production code for R-193/R-192):
audits/SPIKE-offsite-credential-recovery-2026-08-04.md establishes that the one-shot
provider password is the RECOVERABLE secret and the restic repository password is the
irreplaceable one — and that a guest rebuild mints a fresh one, orphaning the previous
off-site history. Measured without touching a box, by comparing
host_escrow.restic_pw_sha256 against host_escrow_superseded: BOTH demo boxes changed
(demo-hp 15 snapshots / 40.9 MB, demo-felhom 36 snapshots / 1.14 GB). demo-felhom's
"lucky" 76-second recovery restored delivery and not the repository, silently, for 13h.
ReissueCredentials does NOT rotate the restic password (R-39's record and two hub
comments are wrong -> R-196); candidate (b) is not implementable against a
zero-knowledge escrow; candidate (a) already exists as F3 and is wired to the wrong
event. Ends in ranked options and an unanswered question for the operator.

R-195 SHIPPED; R-196 + R-197 filed; R-192 + R-193 updated, neither closed.
This commit is contained in:
2026-08-04 11:04:39 +02:00
parent f456835bbc
commit 7fff45d688
9 changed files with 815 additions and 6 deletions
+34 -3
View File
@@ -336,7 +336,7 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
midnightBudapest := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, budapest)
sinceUTC := midnightBudapest.UTC()
var backupMissed, dbdumpMissed, skipped, deferred int
var backupMissed, dbdumpMissed, skipped, deferred, unbound int
for _, id := range customerIDs {
// Skip nodes that are down — they already have staleness events
@@ -350,6 +350,37 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
continue
}
// ── R-195: a customer with NO machine EVER bound is UNKNOWN, not missed ────────────────
//
// Both verdicts below ask "did the thing we expect every day happen?". For a customer
// that has never had a machine bound, nothing has ever been expected, so the honest
// answer is UNKNOWN — the same invariant assessBackupFreshness states above, applied one
// level up, at the question of whether there is a subject at all.
//
// The discriminator is "was a host EVER bound", NOT "has a report arrived". That is the
// case this check must not break: a box that was installed, bound, and then went silent
// has a real fault and must keep alarming. It is bound, so it is judged.
//
// WHY THIS WAS REACHABLE AT ALL, measured 2026-08-04: the down-skip above is what
// protects every other silent customer, and it reads the staleness checker's state — which
// is seeded from the `reports` table (store.GetCustomers). A customer that has NEVER
// reported appears in no report row, so it gets no staleness state at all and GetState()
// returns "" rather than "down". The skip misses exactly the customer it would most
// obviously cover, and the DB-dump half below then fires every night: `david`, a
// prospective customer whose record was created 2026-08-01 with no machine ever bound,
// e-mailed an expected_dbdump_missed ERROR at 03:00 UTC on three consecutive days.
//
// Fail-open on a read error: an unreadable binding must never SUPPRESS a real alarm.
if bound, berr := s.HasEverBoundHost(id); berr != nil {
logger.Printf("[WARN] Deadline check: failed to read host binding for %s (judging anyway): %v", id, berr)
} else if !bound {
// Visible, per the v0.73.0 Part-7 precedent below: a quiet check must never be
// indistinguishable from a check that did not run. Once daily, one line per customer.
logger.Printf("[INFO] Deadline check: %s has no host EVER bound — all deadline verdicts UNKNOWN (no alarm)", id)
unbound++
continue
}
// Backup freshness from the agent's host-report (PBS snapshots + vzdump),
// the authoritative offsite-backup signal post-slice-8C.
reportJSON, rerr := s.GetLatestHostReportJSON(id)
@@ -423,6 +454,6 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
}
}
logger.Printf("[INFO] Deadline check: %d customers, %d backup missed, %d backup unknown (deferred), %d dbdump missed, %d skipped (down)",
len(customerIDs), backupMissed, deferred, dbdumpMissed, skipped)
logger.Printf("[INFO] Deadline check: %d customers, %d backup missed, %d backup unknown (deferred), %d dbdump missed, %d skipped (down), %d unknown (no host ever bound)",
len(customerIDs), backupMissed, deferred, dbdumpMissed, skipped, unbound)
}