hub v0.93.0: the retention keeps the key it was built to keep (R-198) + three honesty fixes (R-197, R-192, R-196)
gates / gates (push) Successful in 7s

R-198 — host_escrow_superseded shipped with `blob` (the K-escrow / PBS datastore key) and
identity_blob was added to host_escrow LATER, never here. The offsite restic REPOSITORY
password lives in identity_blob. So demoteCurrentEscrowTx -- whose own comment calls it "THE
ONE escrow row-copy routine" -- retained the whole-guest key and silently dropped the off-site
data key, which is the secret the retention was built to preserve. And because the copy happens
as the new blob overwrites the old, the destroying act was the ESCROW CEREMONY: the exact thing
a rebuilt box tells its customer to run, on a card promising in Hungarian that the old backups
stay recoverable. Both demo boxes crossed that line on 2026-08-04.

  - identity_blob added to the table (CREATE + additive ALTER) and carried in the shared copy
    routine, so BOTH callers are fixed at once: re-escrow and host-delete demotion.
  - ListSupersededEscrow reads it back; store.HostEscrow gains IdentityBlob.
  - CountCurrentEscrowWithIdentity is the census of who the fix protects.
  - Nothing is backfillable: pre-v0.93.0 retained rows have no blob and their sources are gone.
  - Tests assert the CONSEQUENCE (a retained row can still yield a repo password), which is why
    the pre-existing retention test stayed green for two months asserting the mechanism.

R-197 — SaveHostEscrow returns the hash it replaced; the escrow PUT raises
offsite_repo_key_changed (warning, operator-only, edge-triggered) when both hashes are known and
differ. No hash value travels. Severity chosen for the world v0.93.0 creates: with the identity
blob retained, a changed key is "this history now depends on an older recovery code", not a loss.

R-192 (half) — the stuck alert now reports the two shapes it actually covers, burned and
regressed, each stating its own measurement; the regressed text withdraws the Re-issue
recommendation. Every self-heal refusal leaves a notification_log row with its reason. The
guard's logic is unchanged; its 500-oldest-reports scoping stays OPEN and the window is named in
the alert text so the limitation travels with the number. offsite_delivery_stuck and
offsite_credential_restaged are added to operatorOnlyEvents -- neither was registered and neither
has a customerMessages entry, which is not a block.

R-196 — five comments (not the three the spec expected) claimed ReissueCredentials rotates the
restic repo password. It resets the PROVIDER password and cannot touch the repo password, which
is generated on the box. All five corrected; the staleness mark documented as precautionary. The
BEHAVIOUR stays open.

Not in this release: R-199, R-200, R-201 remain open -- the chain that hands the key back is
still unassembled. Part 5 hit its gate; the orphan card is untouched (R-202).
This commit is contained in:
2026-08-04 12:56:58 +02:00
parent 3f2b7bc023
commit 91cabdde1b
28 changed files with 1122 additions and 140 deletions
+104 -16
View File
@@ -94,40 +94,112 @@ func (c *OffsiteDeliveryChecker) Check() {
if age < stuckAfter {
continue // normal convergence window
}
c.maybeEmitStuck(cfg.CustomerID, status, age)
c.maybeHeal(cfg.CustomerID, status)
emitted := c.maybeEmitStuck(cfg.CustomerID, status, age)
c.maybeHeal(cfg.CustomerID, status, emitted)
}
}
// maybeEmitStuck emits offsite_delivery_stuck (warning) once per stuckCooldown per customer.
func (c *OffsiteDeliveryChecker) maybeEmitStuck(customerID string, status offsite.DeliveryStatus, age time.Duration) {
// deliveryShape names the two situations the ONE stuck state actually covers. They need different
// text and different advice, and conflating them is R-192's defect (a).
type deliveryShape string
const (
// shapeBurned — NO report since the consume carried an offbox target: the apply never persisted.
// Re-issue is the indicated action.
shapeBurned deliveryShape = "burned"
// shapeRegressed — reports since the consume DID carry an offbox target and the latest does not:
// the credential worked and the target was later lost (a guest rebuild does exactly this, R-193).
// Re-issue is NOT indicated; it treats a symptom whose cause is elsewhere.
shapeRegressed deliveryShape = "regressed"
)
func shapeOf(status offsite.DeliveryStatus) deliveryShape {
if status.OffsiteReportsSinceConsume == 0 {
return shapeBurned
}
return shapeRegressed
}
// maybeEmitStuck emits offsite_delivery_stuck (warning) once per stuckCooldown per customer. Returns
// whether it emitted, so the heal's refusal record rides the same cadence rather than inventing one.
//
// R-192 defect (a), fixed here: the message used to interpolate ReportsSinceConsume (the TOTAL) into
// a hardcoded phrase "report(s) since carry no offbox target", and never consulted
// OffsiteReportsSinceConsume — the field that says the opposite. On demo-hp it stated, daily, that
// 500 reports carried no offbox target when all 500 of them did, and prescribed Re-issue for a
// failure mode that had not occurred. The message now STATES WHAT WAS MEASURED and lets the operator
// read it; the recommendation follows the shape rather than being hardcoded.
//
// THE WINDOW IS NAMED ON PURPOSE. CountReportsOffsiteSince reads `ORDER BY id LIMIT 500` — the OLDEST
// 500 reports after the consume, not the newest — so on a long-lived customer these counts describe
// the beginning of the window and not the present. That is a real scoping defect (R-192's other half)
// and it stays OPEN because its correct shape depends on the recovery chain that is not yet
// assembled (R-199/R-200/R-201). Naming the window in the text is how it stays visible instead of
// being laundered into a confident sentence — an instrument that can silently mis-scope its results
// must say so where it reports them.
func (c *OffsiteDeliveryChecker) maybeEmitStuck(customerID string, status offsite.DeliveryStatus, age time.Duration) bool {
last, err := c.store.LastEventAt(customerID, eventDeliveryStuck)
if err != nil {
c.logger.Printf("[WARN] offsite-delivery: %s: cooldown read: %v", customerID, err)
return
return false
}
if !last.IsZero() && c.now().Sub(last) < stuckCooldown {
return
return false
}
shape := shapeOf(status)
var msg string
switch shape {
case shapeBurned:
msg = fmt.Sprintf("Offsite delivery stuck (BURNED-credential shape): the one-time password was consumed %s ago; of the first %d report(s) after that consume, NONE carried an offbox target, and the latest report carries none either. The credential never reached a persisted apply. Re-issue delivers a fresh one.",
age.Round(time.Minute), status.ReportsSinceConsume)
default:
msg = fmt.Sprintf("Offsite delivery stuck (REGRESSED-apply shape): the one-time password was consumed %s ago; of the first %d report(s) after that consume, %d DID carry an offbox target — and the latest report carries none. The credential was applied and worked; the target was lost afterwards. Re-issue is NOT the indicated action: find what removed the offbox target (a guest rebuild does, R-193). Automatic restage is deliberately withheld for this shape. NOTE: the counts cover at most the first 500 reports after the consume, so on a long-lived box they describe the start of the window, not now (R-192, open).",
age.Round(time.Minute), status.ReportsSinceConsume, status.OffsiteReportsSinceConsume)
}
msg := fmt.Sprintf("Offsite delivery stuck: one-time password consumed %s ago and %d report(s) since carry no offbox target — the credential is likely burned (apply died between consume and persist). Re-issue delivers a fresh one.",
age.Round(time.Minute), status.ReportsSinceConsume)
details, _ := json.Marshal(map[string]any{
"state": string(status.State),
"consumed_at": status.Since.UTC().Format(time.RFC3339),
"reports_since_consume": status.ReportsSinceConsume,
"state": string(status.State),
"shape": string(shape),
"consumed_at": status.Since.UTC().Format(time.RFC3339),
"reports_since_consume": status.ReportsSinceConsume,
"offsite_reports_since_consume": status.OffsiteReportsSinceConsume,
"count_window": "oldest 500 reports after consumed_at (R-192, open)",
})
c.emit(customerID, eventDeliveryStuck, "warning", msg, string(details))
return true
}
// maybeHeal fires the R-71c self-heal when the burned-credential shape is unambiguous:
// consumed ≥ stuckAfter ago, ≥ healMinReports consecutive reports since with ZERO offbox evidence,
// one heal per healCooldown — and the R-39(a) guard holds at act time.
func (c *OffsiteDeliveryChecker) maybeHeal(customerID string, status offsite.DeliveryStatus) {
//
// R-192 defect (b), fixed here: every refusal above the "not configured" line now leaves a RECORD.
// The regressed-shape branch used to be a bare `return`, so the operator received a daily e-mail with
// the wrong story, no heal, and nothing anywhere saying why the heal declined — "we chose not to act"
// and "the heal never ran" looked identical. `offsite_credential_restaged` has never fired for any
// customer, and until now that fact was indistinguishable from the checker being dead.
//
// The record is a notification_log row (the dispatcher's suppressed-operator-e-mail precedent, R-182:
// a decision not to act is written down on the channel it would have used). It rides `recordRefusal`
// — true only when the stuck event was emitted this pass — so it appears once per stuckCooldown
// beside the e-mail it explains, rather than once per monitor tick. The GUARD ITSELF IS UNCHANGED:
// the set of situations in which the heal fires is byte-for-byte what it was; only the silence is
// gone. The two conditions are split into separate branches solely so each refusal can name its own
// reason.
func (c *OffsiteDeliveryChecker) maybeHeal(customerID string, status offsite.DeliveryStatus, recordRefusal bool) {
if c.reissuer == nil {
return // no provisioner configured: the heal does not exist on this hub, so there is nothing to explain
}
if status.OffsiteReportsSinceConsume != 0 {
c.recordHealRefusal(customerID, recordRefusal, fmt.Sprintf(
"regressed-apply shape: %d of the first %d report(s) after the consume DID carry an offbox target, so a burned credential is ruled out — a restage would treat a symptom whose cause is elsewhere. Operator's call (R-193).",
status.OffsiteReportsSinceConsume, status.ReportsSinceConsume))
return
}
if status.ReportsSinceConsume < healMinReports || status.OffsiteReportsSinceConsume != 0 {
return // box not reporting enough, or offbox evidence exists (regressed-apply shape) → operator's call
if status.ReportsSinceConsume < healMinReports {
c.recordHealRefusal(customerID, recordRefusal, fmt.Sprintf(
"only %d report(s) since the consume (need %d): the box has not reported enough for the burned shape to be unambiguous.",
status.ReportsSinceConsume, healMinReports))
return
}
last, err := c.store.LastEventAt(customerID, eventCredentialRestaged)
if err != nil {
@@ -146,8 +218,8 @@ func (c *OffsiteDeliveryChecker) maybeHeal(customerID string, status offsite.Del
return
}
if info == nil || info.ConsumedAt.IsZero() {
c.logger.Printf("[INFO] offsite-delivery: %s: heal refused — secret row is now %s (R-39(a) guard: never restage over an unconsumed secret)",
customerID, secretShape(info))
c.recordHealRefusal(customerID, recordRefusal, fmt.Sprintf(
"R-39(a) guard: the secret row is now %s — restaging over an unconsumed secret would clobber a password the box may be about to consume.", secretShape(info)))
return
}
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
@@ -167,6 +239,22 @@ func (c *OffsiteDeliveryChecker) maybeHeal(customerID string, status offsite.Del
string(details))
}
// recordHealRefusal makes a decision NOT to self-heal visible. Always logs; additionally writes a
// notification_log row on the "operator" channel with status "refused" when `record` is set (the
// stuck event was emitted this pass), so the refusal sits next to the e-mail that prompted the
// question. A LogNotification failure is logged, never swallowed, and never blocks the refusal — the
// refusal is the primary effect.
func (c *OffsiteDeliveryChecker) recordHealRefusal(customerID string, record bool, reason string) {
c.logger.Printf("[INFO] offsite-delivery: %s: self-heal REFUSED — %s", customerID, reason)
if !record {
return
}
if err := c.store.LogNotification(customerID, eventCredentialRestaged, "warning",
"Automatic offsite credential restage was NOT performed.", "refused", reason, "operator"); err != nil {
c.logger.Printf("[WARN] offsite-delivery: %s: could not record the heal refusal: %v", customerID, err)
}
}
func secretShape(info *store.OneTimeSecretInfo) string {
if info == nil {
return "absent"
@@ -0,0 +1,223 @@
package monitor
import (
"encoding/json"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// R-192 (v0.93.0) — the alert must describe what was MEASURED, and a refusal to self-heal must leave a
// trace.
//
// What was wrong, quoted from the live e-mail demo-hp sent daily from 2026-08-03: "one-time password
// consumed 284h19m0s ago and 500 report(s) since carry no offbox target — the credential is likely
// burned … Re-issue delivers a fresh one." OffsiteReportsSinceConsume for that same call was 500:
// every one of those reports carried a target. The message stated the precise negation of its own
// measurement and prescribed a remedy for a failure mode that had not occurred — while the self-heal
// correctly declined, in complete silence.
// regressedFixture puts c1 into demo-hp's real shape: consumed >1h ago, reports WITH an offbox target
// after the consume, then reports WITHOUT — the latest carrying none.
//
// The report timestamps get jitter deliberately: a perfectly regular series lands exactly on
// boundaries and survives mutations that a realistic one catches.
func (h dtHarness) regressedFixture(t *testing.T, withOffsite, without int) {
t.Helper()
if err := h.st.SaveOneTimeSecret("c1", "x"); err != nil {
t.Fatal(err)
}
consumed := time.Now().UTC().Add(-30 * time.Hour).Format("2006-01-02 15:04:05")
staged := time.Now().UTC().Add(-31 * time.Hour).Format("2006-01-02 15:04:05")
if err := h.st.SetOneTimeSecretTimesForTest("c1", staged, consumed); err != nil {
t.Fatal(err)
}
jitter := []int{0, 3, 1, 7, 2, 5, 11, 4}
for i := 0; i < withOffsite; i++ {
if err := h.st.SaveReport("c1", []byte(dtReportWithOffsite)); err != nil {
t.Fatal(err)
}
time.Sleep(time.Duration(jitter[i%len(jitter)]) * time.Millisecond)
}
for i := 0; i < without; i++ {
if err := h.st.SaveReport("c1", []byte(dtReportNoOffsite)); err != nil {
t.Fatal(err)
}
time.Sleep(time.Duration(jitter[(i+3)%len(jitter)]) * time.Millisecond)
}
}
func (h dtHarness) refusalRows(t *testing.T) []store.NotificationLogEntry {
t.Helper()
all, err := h.st.GetRecentNotifications("c1", 50)
if err != nil {
t.Fatal(err)
}
var out []store.NotificationLogEntry
for _, n := range all {
if n.Status == "refused" {
out = append(out, n)
}
}
return out
}
// Scenario F — the message states the measured state, not a denied cause.
// RED-PROOF: restore the old single hardcoded sentence (interpolate ReportsSinceConsume into
// "report(s) since carry no offbox target") → the regressed assertions FAIL.
func TestDeliveryChecker_RegressedShape_MessageMatchesMeasurement(t *testing.T) {
h := newDTHarness(t, true)
h.regressedFixture(t, 6, 2)
h.checker.Check()
saved := h.savedEvents(t, "offsite_delivery_stuck")
if len(saved) != 1 {
t.Fatalf("stuck events = %d, want 1", len(saved))
}
msg := saved[0].Message
// It must NOT assert the thing the data contradicts.
if strings.Contains(msg, "carry no offbox target") || strings.Contains(msg, "likely burned") {
t.Fatalf("R-192(a): the message still asserts a burned credential on a REGRESSED shape — this is "+
"the sentence that stated the negation of its own measurement:\n%s", msg)
}
// It must name the shape it detected and the measurement behind it.
if !strings.Contains(msg, "REGRESSED") {
t.Errorf("the message must name the detected shape, got:\n%s", msg)
}
if !strings.Contains(msg, "6 DID carry an offbox target") {
t.Errorf("the message must state the measured offbox-bearing count (6), got:\n%s", msg)
}
// It must not prescribe the action for a failure mode that did not occur.
if !strings.Contains(msg, "Re-issue is NOT the indicated action") {
t.Errorf("the message must withdraw the Re-issue recommendation for this shape, got:\n%s", msg)
}
// The counting window is named, because CountReportsOffsiteSince reads the OLDEST 500 — an
// instrument that can silently mis-scope must say so where it reports (R-192's other half, open).
if !strings.Contains(msg, "500") {
t.Errorf("the message must name its counting window, got:\n%s", msg)
}
var details map[string]any
if err := json.Unmarshal([]byte(saved[0].DetailsJSON), &details); err != nil {
t.Fatal(err)
}
if details["shape"] != "regressed" {
t.Errorf("details.shape = %v, want regressed", details["shape"])
}
// The field whose absence was defect (a) is now carried.
if got, ok := details["offsite_reports_since_consume"].(float64); !ok || int(got) != 6 {
t.Errorf("details must carry offsite_reports_since_consume=6, got %v", details["offsite_reports_since_consume"])
}
}
// The burned shape keeps its own (correct) text and its recommendation — the fix must not blunt the
// case that was always right.
func TestDeliveryChecker_BurnedShape_KeepsItsRecommendation(t *testing.T) {
h := newDTHarness(t, false)
h.burnedFixture(t, 5)
h.checker.Check()
saved := h.savedEvents(t, "offsite_delivery_stuck")
if len(saved) != 1 {
t.Fatalf("stuck events = %d, want 1", len(saved))
}
msg := saved[0].Message
if !strings.Contains(msg, "BURNED") || !strings.Contains(msg, "NONE carried an offbox target") {
t.Errorf("the burned shape must state its own measurement, got:\n%s", msg)
}
if !strings.Contains(msg, "Re-issue delivers a fresh one") {
t.Errorf("the burned shape must keep its recommendation, got:\n%s", msg)
}
var details map[string]any
_ = json.Unmarshal([]byte(saved[0].DetailsJSON), &details)
if details["shape"] != "burned" {
t.Errorf("details.shape = %v, want burned", details["shape"])
}
}
// Scenario G — a refusal to self-heal leaves a durable record naming its reason.
// RED-PROOF: replace the recordHealRefusal call in the regressed branch with a bare `return`
// (production behaviour up to v0.92.0) → no row → this FAILS.
func TestDeliveryChecker_HealRefusal_LeavesARecord(t *testing.T) {
h := newDTHarness(t, true)
h.regressedFixture(t, 6, 2)
h.checker.Check()
if h.reissuer.count() != 0 {
t.Fatal("the heal must NOT fire on a regressed shape — the guard is unchanged by this fix")
}
rows := h.refusalRows(t)
if len(rows) != 1 {
t.Fatalf("R-192(b): the heal declined and left %d record(s), want 1 — 'we chose not to act' and "+
"'the heal never ran' must not look identical", len(rows))
}
r := rows[0]
if r.Channel != "operator" {
t.Errorf("refusal channel = %q, want operator (the channel the heal's own event would use)", r.Channel)
}
if !strings.Contains(r.ErrorMessage, "regressed-apply shape") {
t.Errorf("the record must name WHY, got %q", r.ErrorMessage)
}
if !strings.Contains(r.ErrorMessage, "6 of the first 8") {
t.Errorf("the record must carry the measurement it refused on, got %q", r.ErrorMessage)
}
}
// The R-39(a) guard's refusal is recorded too — it was the one refusal that already logged, and it
// still must not be the only one that does.
//
// Reaching it needs the real TOCTOU, not a pre-staged row: staging BEFORE Check() makes the derive
// itself return staged_awaiting_consume and the pass never reaches the heal at all. Same technique as
// TestDeliveryChecker_R39aGuard_NeverRestagesOverUnconsumed — the operator stages mid-tick, on the
// stuck event's dispatch, which runs after the derive and before maybeHeal.
func TestDeliveryChecker_R39aRefusal_LeavesARecord(t *testing.T) {
h := newDTHarness(t, true)
h.burnedFixture(t, 5)
base := h.checker.onEvent
h.checker.onEvent = func(cid, et, sev, msg, det, src string) {
if et == "offsite_delivery_stuck" {
if err := h.st.SaveOneTimeSecret("c1", "operator-fresh"); err != nil {
t.Errorf("mid-tick stage: %v", err)
}
if err := h.st.SetOneTimeSecretTimesForTest("c1", "2026-07-23 12:00:00", ""); err != nil {
t.Errorf("mid-tick stamp: %v", err)
}
}
base(cid, et, sev, msg, det, src)
}
h.checker.Check()
if h.reissuer.count() != 0 {
t.Fatal("R-39(a): the heal must never restage over an unconsumed secret")
}
rows := h.refusalRows(t)
if len(rows) != 1 {
t.Fatalf("the R-39(a) refusal left %d record(s), want 1", len(rows))
}
if !strings.Contains(rows[0].ErrorMessage, "R-39(a)") {
t.Errorf("the record must name the guard, got %q", rows[0].ErrorMessage)
}
}
// The refusal record rides the stuck event's 24h cadence: it explains the e-mail the operator just
// received, and must not accumulate one row per monitor tick.
func TestDeliveryChecker_HealRefusal_RidesTheStuckCadence(t *testing.T) {
h := newDTHarness(t, true)
h.regressedFixture(t, 6, 2)
h.checker.Check()
h.checker.Check() // the stuck event is swallowed by its cooldown; so is the refusal record
h.checker.Check()
if n := len(h.refusalRows(t)); n != 1 {
t.Fatalf("refusal records = %d after three passes, want 1 (one per stuck e-mail, not per tick)", n)
}
}
@@ -102,7 +102,7 @@ func TestOffsiteStale_NeverRanEscrowAnchorWins(t *testing.T) {
t.Fatal(err)
}
// escrow blob stored now (SaveHostEscrow stamps updated_at with datetime('now'))
if _, err := st.SaveHostEscrow("h1", []byte("blob"), "fp", "zero_knowledge", time.Now().UTC().Format(time.RFC3339), "sha"); err != nil {
if _, _, err := st.SaveHostEscrow("h1", []byte("blob"), "fp", "zero_knowledge", time.Now().UTC().Format(time.RFC3339), "sha"); err != nil {
t.Fatal(err)
}
saveOffsiteReport(t, st, "c1", offsiteJSON(true, "escrowed", "", "", 0, 50))