hub (unreleased): review fix 2 — the notice's daily limit counts failed tries too, and the wait between tries doubles (a send that errors after it was accepted)

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
admin committed 2026-11-15 17:14:23 +01:00
1 parent 508f936ad4
commit b391a2322c
9 files changed
+146 -18

No files matched your search

+1 -1
View File
@@ -80,7 +80,7 @@
| `store.NoteAppList` / `AppPresences` + `offsitekeys.agreeEnded` (D72/D84, `07` §8.2f) | hub/internal/store/offsite_ended.go, hub/internal/offsitekeys/ended.go | `NoteAppList(customerID, deployed, claimed, now)`; `agreeEnded(now, presence, notices, listing, haveListing, claims)` | The hub's OWN record of when an app left a box's app list (fed by `handleReport`), and the decision which snapshot ids of ended apps a window may remove outside its cap | **The box's dates decide nothing.** An id is agreed only when its app has been absent 30 days by the hub's clock AND the hub's own listing shows the file older than that absence. `WindowCredit.CapExplains` is the cap's part, `EndedIDs` the agreed ids — a reader that explains ids by name must not also spend `Explains`. **After the review of 2026-11-15:** one window agrees to at most the cap's size from the hub's OWN count (`endedCap`/`boundEndedToCap`); a request for more than half of the store waits a week — its first sight is the hub's own row (`store.NoteLargeEndedRequest`, key `largeSetKey` = a hash of the whole app set), never a search of text a box could write; `store.NoteAppListClaims` returns never-seen apps once for the operator's warning; a claimed name that is not an app name is written nowhere (`ledgerName`). |
| `notify.householdReceiptEvents` + `Dispatcher.SendEndNotice` (D84; the notice lock, `09` §3 decision 315) | hub/internal/notify/end_notice.go | register; `SendEndNotice(customerID, app, ends) (EndNoticeOutcome, why)` | The mail that tells the household about its own press (one type: `offsite_end_scheduled`). **The box's event mails nobody**: the hub sends the notice from its own state (`api.endNotices`, at every report, until the mail service accepts it) | Bypasses the household's group switches; the hub writes it from a checked app name and its OWN day. `SendEndNotice` keeps no state and tries once: only `EndNoticeSent` tells the household; the daily limit (5 per household, counted in the sent log) answers `EndNoticeLimit` — the caller tries again, it never drops. **Every new entry needs its own ruling** — pinned by `TestD84_Receipt_OpensNothingElse`, `TestNotice_*`. |
| `store.NoteEndNotices` / `EndNotices` / `MarkEndNoticeTold` + `api.endNotices` (the notice lock, `07` §8.2f) | hub/internal/store/offsite_end_notice.go, hub/internal/api/end_notice.go | `NoteEndNotices(customerID, deployed, claims, tierOn, now)`; `EndNotices(customerID)`; `MarkEndNoticeTold(customerID, app, press, to, now)` | The hub's OWN record that a household (or the operator in its place) was told that an app's off-site copies will be deleted. `offsitekeys.agreeEnded` agrees only when it says „told” 30 days ago | **Written only by the hub, only after the mail service accepted the mail; never derived from the sent log or from a box's date.** `press` (the box's `removed_at`) is a key only: another value VOIDS the notice. An install, a take-back (a report that shows the tier ON and no record) and a household delete remove the row. A new condition of a deletion goes HERE, not into `notification_log`. |
| `Dispatcher.SendOperatorNow` | hub/internal/notify/end_notice.go | `SendOperatorNow(customerID, eventType, severity, message, detailsJSON) error` | An operator mail whose DELIVERY is a condition of something else: it answers only when the mail service has (nil = accepted) | Operator-only types only; no cooldown and no background retry — the caller keeps its own „told once” record and tries again, **and bounds itself** from the sent log (`store.CountOperatorSentSince`, `OperatorSentAboutSince`: per household a day, per thing a week — `api.endNoticeToOperator`). For every other operator mail use `ProcessEvent`. |
| `Dispatcher.SendOperatorNow` | hub/internal/notify/end_notice.go | `SendOperatorNow(customerID, eventType, severity, message, detailsJSON) error` | An operator mail whose DELIVERY is a condition of something else: it answers only when the mail service has (nil = accepted) | Operator-only types only; no cooldown and no background retry — the caller keeps its own „told once” record and tries again, **and bounds itself** from the sent log (`store.CountOperatorTriesSince` — sent AND failed tries — and `OperatorSentAboutSince`: per household a day, per thing a week — `api.endNoticeToOperator`). For every other operator mail use `ProcessEvent`. |
| `accesslog.New` / `Middleware` / `Mark` (v0.158.0, R-1140) | hub/internal/accesslog/accesslog.go | `Mark(ctx, kind)` | The hub's own access log (`<data_dir>/access-log/`, 90 days) and marking the identity KIND an auth check established | Never log a value: the path only (no query; `/bind/<token>` and 32+-char segments redacted). A new auth path calls `Mark` with its kind. A logging failure never fails a request. |
| the timed SQLite driver `sqlite-timed` + `SetSlowStatementThreshold` (v0.178.0, R-1346) | hub/internal/store/timeddriver.go | `store.New` opens it — nothing to call | Names a statement over 1 s: `[WARN] slow statement: <ms> (over 1s): <SQL>` | The SQL text only, never a bound value. Put a heavy statement's SQL in a const and pin its plan (`r1346_speed.go`, `TestR1346_HeavyStatementsUseTheirIndex`): on the hub's network disk a whole-table read is seconds to minutes. |
| `(*Store).SaveOneTimeSecret` / `OffsitePassword` / `SealLegacyOffsiteSecrets` | hub/internal/store/offsite_seal.go | — | Storing / reading the sub-account password | **Sealed AES-256-GCM; no key → refused (fail-closed).** Under `go test` every store gets a fixed key (`testing.Testing()`); production needs `OFFSITE_SECRET_KEY`. Never serve the value to a box. |
@@ -0,0 +1,36 @@
# Review finding 2 — a send that errors after it was accepted is mailed every hour, outside the daily limit. The test on the code BEFORE the fix:
--- FAIL: TestReview2_ASendThatErrorsAfterItWasAcceptedIsNotMailedEveryHour (0.11s)
--- FAIL: TestReview2_ASendThatErrorsAfterItWasAcceptedIsNotMailedEveryHour/the_household's_notice (0.06s)
end_notice_review_test.go:123: 12 hours of a send that errors after it was accepted: 12 mails, want at most 4 (tries at 0, 1, 3 and 7 hours)
--- FAIL: TestReview2_ASendThatErrorsAfterItWasAcceptedIsNotMailedEveryHour/the_operator's_in-place_notice (0.06s)
end_notice_review_test.go:123: 12 hours of a send that errors after it was accepted: 12 mails, want at most 4 (tries at 0, 1, 3 and 7 hours)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-hub/internal/api 0.134s
FAIL
# After the fix, by mutation:
== MUTATION: the wait does not grow with the tries
RED on the mutant, GREEN on the real file
--- FAIL: TestReview2_ASendThatErrorsAfterItWasAcceptedIsNotMailedEveryHour (0.10s)
end_notice_review_test.go:123: 12 hours of a send that errors after it was accepted: 5 mails, want at most 4 (tries at 0, 1, 3 and 7 hours)
end_notice_review_test.go:123: 12 hours of a send that errors after it was accepted: 5 mails, want at most 4 (tries at 0, 1, 3 and 7 hours)
== MUTATION: the operator's daily limit counts only sent mails
RED on the mutant, GREEN on the real file
--- FAIL: TestReview2_ASendThatErrorsAfterItWasAcceptedIsNotMailedEveryHour (0.12s)
end_notice_review_test.go:129: four days of it: 8 mails — the daily limit must count a try that failed (all of them are inside this test's real 24 hours)
--- FAIL: TestNotice_NoAddress_ABoxCannotFloodTheOperator (0.29s)
end_notice_test.go:543: 101 presses in a day mailed the operator 51 times, want the household's daily limit 5
2 mutations, 0 not a red-proof
== MUTATION: the household's daily limit counts only sent mails
RED on the mutant, GREEN on the real file
--- FAIL: TestReview2_ASendThatErrorsAfterItWasAcceptedIsNotMailedEveryHour (0.12s)
end_notice_review_test.go:129: four days of it: 8 mails — the daily limit must count a try that failed (all of them are inside this test's real 24 hours)
1 mutations, 0 not a red-proof
== MUTATION: the operator's daily limit counts only sent mails (the query)
RED on the mutant, GREEN on the real file
--- FAIL: TestReview2_ASendThatErrorsAfterItWasAcceptedIsNotMailedEveryHour (0.12s)
end_notice_review_test.go:129: four days of it: 8 mails — the daily limit must count a try that failed (all of them are inside this test's real 24 hours)
1 mutations, 0 not a red-proof
+11
View File
@@ -1,5 +1,16 @@
## Unreleased (after v0.182.0)
- **Review of the notice lock, finding 2 (2026-11-15 night): a send that errors AFTER the mail service accepted it is
not mailed every hour.** A client timeout is logged `failed` although the mail may have gone; the notice was tried
again every hour and the daily limit counted only `sent` rows — 12 household mails in 12 hours in the review's
test. Now (a) the daily limit counts every try that reached the mail service, `sent` and `failed`
(`store.CountCustomerTriesSince`, `CountOperatorTriesSince` — the latter replaces `CountOperatorSentSince`), for
the household's notice and for the operator's in-place mail; (b) the wait before the next try doubles with the
tries: 1 h, 2 h, 4 h, 8 h, 16 h, then 24 h (`api.endNoticeGap`). The „stuck” line after 3 days still comes. **Cost,
said plainly:** after a real outage of the mail service the notice goes up to a day later than before, and a
household that already had 5 failed tries in 24 hours waits for the limit. Test
`TestReview2_ASendThatErrorsAfterItWasAcceptedIsNotMailedEveryHour` (both paths: at most 4 mails in 12 hours, 5 in
four days, never „told”, one stuck line); seen red before the fix (12 mails), 4 mutations red.
- **Review of the notice lock, finding 4 (2026-11-15 night): „told” is the time AFTER the send returned.** One walk
used its start time for every row, so with many rows (or a slow mail service) the last row was „told” minutes
before its mail was accepted, and the 30 days counted from before the mail. `api.endNotices` now reads its clock
+21 -6
View File
@@ -24,15 +24,18 @@ import (
// (`offsite_end_notice_operator`), and that mail starts the 30 days. When an address appears later, the household
// gets its own mail and the 30 days start again from it.
// - the daily limit, the mail hold, a send that fails → nothing is told; the reason is written on the row (the
// window's answer names it) and the next try is at a report at least endNoticeRetryGap later.
// window's answer names it) and the next try is at a report at least endNoticeGap(tries) later (1 h, 2 h, 4 h …).
// - a row that has not been told for EndNoticeStuckAfter → one line to the operator (`offsite_end_notice_stuck`).
//
// Nothing here reads a date a box sent, and no event of a box starts or stops anything.
const (
// endNoticeRetryGap: after a try that told nobody, the next try waits this long (a report comes every 15 minutes;
// a mail service that is down is not asked 96 times a day per app).
endNoticeRetryGap = time.Hour
// endNoticeRetryGap: after the FIRST try that told nobody, the next try waits this long (a report comes every 15
// minutes; a mail service that is down is not asked 96 times a day per app). It doubles with every further try
// and stops at endNoticeRetryGapMax (endNoticeGap): a send that errors AFTER the mail service accepted it is a
// mail each time, so the tries must thin out (review of 2026-11-15, finding 2).
endNoticeRetryGap = time.Hour
endNoticeRetryGapMax = 24 * time.Hour
// EndNoticeStuckAfter: how long a notice may stay unsent before the operator gets one line. THREE DAYS: longer
// than any outage of the mail service the hub has had and than the hold on a restored copy is meant to last, and
@@ -42,6 +45,18 @@ const (
EndNoticeStuckAfter = 3 * 24 * time.Hour
)
// endNoticeGap is the wait after the tries-th try that told nobody: 1 h, 2 h, 4 h, 8 h, 16 h, then 24 h.
func endNoticeGap(tries int) time.Duration {
g := endNoticeRetryGap
for i := 1; i < tries && g < endNoticeRetryGapMax; i++ {
g *= 2
}
if g > endNoticeRetryGapMax {
g = endNoticeRetryGapMax
}
return g
}
// endNoticeSpawn runs the sends off the report's request (a send can wait 10 s on the mail service). A seam: tests
// run it in line.
var endNoticeSpawn = func(f func()) { go f() }
@@ -97,7 +112,7 @@ func (h *Handler) endNotices(customerID string, start time.Time) {
continue
}
now := clock()
if n.LastReason != "" && !n.LastTry.IsZero() && now.Sub(n.LastTry) < endNoticeRetryGap {
if n.LastReason != "" && !n.LastTry.IsZero() && now.Sub(n.LastTry) < endNoticeGap(n.Tries) {
h.endNoticeStuck(customerID, n, now)
continue
}
@@ -162,7 +177,7 @@ func (h *Handler) endNoticeToOperator(customerID string, n store.EndNotice, why
} else if again {
return time.Time{}, fmt.Errorf("the operator was told about this app of this household less than %d days ago — the notice for this press goes later", int(operatorNoticeSameAppGap.Hours()/24))
}
if sent, err := h.store.CountOperatorSentSince(customerID, offsitekeys.EventEndNoticeOperator, wall.Add(-24*time.Hour)); err != nil {
if sent, err := h.store.CountOperatorTriesSince(customerID, offsitekeys.EventEndNoticeOperator, wall.Add(-24*time.Hour)); err != nil {
return time.Time{}, fmt.Errorf("the hub's sent log could not be read")
} else if sent >= operatorNoticesPerDay {
return time.Time{}, fmt.Errorf("the operator already got %d of these notices about this household in the last 24 hours (the daily limit) — this one goes later", operatorNoticesPerDay)
@@ -8,6 +8,7 @@ import (
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/offsitekeys"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
@@ -85,3 +86,54 @@ func TestReview4_ToldIsTheTimeAfterTheSendReturned(t *testing.T) {
n.ToldAt.Format(time.RFC3339), start.UTC().Format(time.RFC3339))
}
}
// Finding 2. A send that ERRORS AFTER the mail service accepted it (a client timeout) was mailed again every hour,
// and it was outside the daily limit, because the limit counted only `sent` rows: 12 mails in 12 hours. Now the
// limit counts every try that reached the mail service (`sent` + `failed`), and the wait before the next try grows
// with the tries (1 h, 2 h, 4 h … at most 24 h). The operator's in-place mail has the same shape and the same
// fix. The „stuck” line after 3 days still comes.
func TestReview2_ASendThatErrorsAfterItWasAcceptedIsNotMailedEveryHour(t *testing.T) {
for _, c := range []struct{ name, email, to string }{
{"the household's notice", household, household},
{"the operator's in-place notice", "", operator},
} {
t.Run(c.name, func(t *testing.T) {
r := newNoticeRig(t, c.email)
r.d.SetSendFnForTest(func(to, subject, body string, _ map[string]string) error {
r.mu.Lock()
defer r.mu.Unlock()
r.mails = append(r.mails, noticeMail{to, subject, body}) // the mail WENT …
return errors.New("context deadline exceeded") // … and the client gave up waiting
})
count := func() int {
n := 0
for _, m := range r.to(c.to) {
if !strings.Contains(m.subject, "offsite_end_notice_stuck") && !strings.Contains(m.subject, "offsite_end_claim_unseen") {
n++
}
}
return n
}
r.report(repPressed)
now := time.Now()
for i := 1; i <= 11; i++ { // a report every hour for twelve hours
r.h.endNotices("c1", now.Add(time.Duration(i)*61*time.Minute))
}
if n := count(); n > 4 {
t.Fatalf("12 hours of a send that errors after it was accepted: %d mails, want at most 4 (tries at 0, 1, 3 and 7 hours)", n)
}
for i := 12; i <= 100; i++ { // … and four days of it
r.h.endNotices("c1", now.Add(time.Duration(i)*61*time.Minute))
}
if n := count(); n > 5 {
t.Fatalf("four days of it: %d mails — the daily limit must count a try that failed (all of them are inside this test's real 24 hours)", n)
}
if n, _ := r.notice("vikunja"); n.Told() {
t.Fatalf("a send that returned an error counts as told: %+v", n)
}
if e := r.events(offsitekeys.EventEndNoticeStuck); len(e) != 1 {
t.Fatalf("the operator's line after 3 days: %d, want 1", len(e))
}
})
}
}
+2 -2
View File
@@ -419,12 +419,12 @@ func TestNotice_StuckForThreeDays_TheOperatorGetsOneLine(t *testing.T) {
t.Fatalf("the line was filed while the hold is on: %+v", e)
}
_ = os.Remove(marker)
r.h.endNotices("c1", now.Add(EndNoticeStuckAfter+3*time.Hour))
r.h.endNotices("c1", now.Add(EndNoticeStuckAfter+6*time.Hour)) // past the wait after the third try (4 h)
e := stuck()
if len(e) != 1 || e[0].Severity != "warning" || !strings.Contains(e[0].Message, `"vikunja"`) || !strings.Contains(e[0].Message, "did not accept") || !strings.Contains(e[0].DetailsJSON, `"subject":"vikunja"`) {
t.Fatalf("want ONE warning naming the app and the reason: %+v", e)
}
for i := 5; i < 10; i++ {
for i := 7; i < 12; i++ {
r.h.endNotices("c1", now.Add(EndNoticeStuckAfter+time.Duration(i)*time.Hour))
}
if e := stuck(); len(e) != 1 {
+3 -2
View File
@@ -26,7 +26,8 @@ import (
// is built so that a box can do nothing else with it:
// - THE HUB WRITES IT. Subject and body come only from the hub's own bundle, with an app name the hub checked
// (the shape of an app name, not reserved) and a day the hub computed from its own clock. Nothing of the box's.
// - AT MOST receiptsPerDay PER HOUSEHOLD IN 24 HOURS, counted in the hub's own sent log (it survives a restart).
// - AT MOST receiptsPerDay PER HOUSEHOLD IN 24 HOURS, counted in the hub's own sent log (it survives a restart) —
// every try that reached the mail service, `sent` or `failed`: a send that timed out here may have gone.
// The limit DELAYS a notice — the record stays „not told” and the next report tries again — it never drops one.
// - to the household's own notification address; never to the operator.
//
@@ -103,7 +104,7 @@ func (d *Dispatcher) SendEndNotice(customerID, app string, ends time.Time) (EndN
return EndNoticeNoAddress, "the household has no notification address"
}
now := time.Now()
sentToday, cerr := d.store.CountCustomerSentSince(customerID, EndNoticeEvent, now.Add(-24*time.Hour))
sentToday, cerr := d.store.CountCustomerTriesSince(customerID, EndNoticeEvent, now.Add(-24*time.Hour))
if cerr != nil {
// A log that cannot be read is not „nothing sent”: no receipt goes, and the notice stays to be tried again.
return EndNoticeLimit, "the hub's sent log could not be read"
+5 -4
View File
@@ -299,11 +299,12 @@ func (s *Store) BackdateNotificationLogForTest(customerID string, by time.Durati
return err
}
// CountOperatorSentSince counts the operator mails of one type that the hub's own log says were SENT about a
// household since `since` (the daily limit of the operator's in-place notice).
func (s *Store) CountOperatorSentSince(customerID, eventType string, since time.Time) (int, error) {
// CountOperatorTriesSince counts the operator mails of one type about a household that REACHED the mail service
// since `since` — `sent` and `failed`, for the reason CountCustomerTriesSince gives (the daily limit of the
// operator's in-place notice).
func (s *Store) CountOperatorTriesSince(customerID, eventType string, since time.Time) (int, error) {
var n int
err := s.db.QueryRow(`SELECT COUNT(*) FROM notification_log WHERE customer_id = ? AND event_type = ? AND channel = 'operator' AND status = 'sent' AND created_at >= ?`,
err := s.db.QueryRow(`SELECT COUNT(*) FROM notification_log WHERE customer_id = ? AND event_type = ? AND channel = 'operator' AND status IN ('sent', 'failed') AND created_at >= ?`,
customerID, eventType, since.UTC().Format("2006-01-02 15:04:05")).Scan(&n)
return n, err
}
+15 -3
View File
@@ -466,11 +466,23 @@ func (s *Store) BackdateLargeEndedRequestsForTest(customerID string, by time.Dur
return nil
}
// CountCustomerSentSince counts the mails of one type the hub's own log says were SENT to a household since `since`
// (the receipt's daily limit — it survives a hub restart, which a counter in memory would not).
// CountCustomerSentSince counts the mails of one type the hub's own log says were SENT to a household since `since`.
func (s *Store) CountCustomerSentSince(customerID, eventType string, since time.Time) (int, error) {
return s.countCustomerLog(customerID, eventType, since, `status = 'sent'`)
}
// CountCustomerTriesSince counts every try of one type that REACHED the mail service for a household since `since`
// — `sent` and `failed` (the notice's daily limit; it survives a hub restart, which a counter in memory would not).
// A try that failed is counted because „failed” is what the hub saw, not what the mail service did: a send that
// timed out on the hub's side may have gone (review of 2026-11-15, finding 2). A held mail never reached the
// service and is not counted.
func (s *Store) CountCustomerTriesSince(customerID, eventType string, since time.Time) (int, error) {
return s.countCustomerLog(customerID, eventType, since, `status IN ('sent', 'failed')`)
}
func (s *Store) countCustomerLog(customerID, eventType string, since time.Time, status string) (int, error) {
var n int
err := s.db.QueryRow(`SELECT COUNT(*) FROM notification_log WHERE customer_id = ? AND event_type = ? AND channel = 'customer' AND status = 'sent' AND created_at >= ?`,
err := s.db.QueryRow(`SELECT COUNT(*) FROM notification_log WHERE customer_id = ? AND event_type = ? AND channel = 'customer' AND `+status+` AND created_at >= ?`,
customerID, eventType, since.UTC().Format("2006-01-02 15:04:05")).Scan(&n)
return n, err
}