hub (unreleased): R-922 option A — a household's clear deletes its notification address (email_cleared); MAIL-HOLD — a restored hub sends no mail until released; two log lines drop the address; runbooks: mail hold is restore step 1; 07 §6.4 R-921 pre-check; R-921/R-922 narrowed
gates / gates (push) Successful in 5m25s
gates / gates (push) Successful in 5m25s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
@@ -1,3 +1,38 @@
|
||||
## Unreleased — a household's cleared mail address is deleted (R-922) and a restored hub starts quiet (MAIL-HOLD) (2026-10-09)
|
||||
|
||||
Not released: ships with the next hub release. No version bump here.
|
||||
|
||||
- **Two log lines no longer print a household's mail address** (found while building R-922, fixed without a row): the
|
||||
preferences push logs `address set=<true|false>` and the dispatcher logs `Customer email sent for <customer>/<event>`.
|
||||
|
||||
- **R-922 — the household clears its mail address, the hub deletes it** (operator ruling 2026-10-09 11:19, option A).
|
||||
`POST /api/v1/preferences` decodes a new boolean `email_cleared`, which the controller sends true ONLY when the
|
||||
household deliberately cleared the address on its dashboard. `email_cleared: true` + empty `email` → the stored
|
||||
notification address is deleted (INFO `household cleared its mail address — deleted`). Every other empty push keeps
|
||||
today's F12 no-clobber guard (v0.71.0) unchanged; the flag beside a non-empty address is an ordinary update. Older
|
||||
controllers never send the flag, so nothing changes for them. `TestSavePreferences_DeliberateClearDeletesAddress`
|
||||
(red-proved: the address stayed), `_ClearFlagWithAddressIsAnUpdate`, `_ClearFlagFalseKeepsAddress`; the F12 test
|
||||
re-proved red. The operator-registered address in `customer_configs.email` is a separate copy and is NOT touched
|
||||
(the kernel notice, claim and self-bind mails read it). `scripts/wire_contract_gate.py` gains the root
|
||||
`controller -> hub (POST /preferences)` (seen convicting a planted controller `email_cleared` against a hub without
|
||||
it). Privacy-notice draft: one retention row for the notification address.
|
||||
- **MAIL-HOLD — a restored hub starts quiet.** Evidence: the 2026-10-09 restore drill (`documentation/audits/
|
||||
dooplex-survival-2026-10-09/partD-05-checks.txt`) — a hub started on a restored DB mailed households pending kernel
|
||||
notices within a minute; `notifications.operator_enabled: false` did not stop household mail. New package
|
||||
`internal/mailhold`: while `<data_dir>/MAIL-HOLD` exists the hub sends **no e-mail on any path** — the dispatcher
|
||||
(household + operator events, recovery, test mail, kernel notice, claim codes, self-bind link; all through ONE
|
||||
`deliver()` gate), the legacy `/api/v1/notify`, and the app-mail relay `/api/v1/mail` (503 while held). Each held mail
|
||||
is logged once, `[WARN] MAIL-HOLD: not sending <kind> to <customer|operator>` (no address, no body), and recorded
|
||||
`held` in the notification log. **Held mails are dropped, not re-sent** — a restored copy's pending mails were decided
|
||||
from a snapshot of the past; the cooldown a dropped mail armed is given back, so the first real alarm after the
|
||||
release is not silenced. A held kernel notice returns an error, so the kernel step does not run (no mail, no step).
|
||||
An unreadable marker state counts as held (fail-closed). The operator lifts it with `POST
|
||||
/configuration/mail-hold/release` (operator auth + the R-135 CSRF gate; a button on Configuration); every operator page
|
||||
shows an amber banner while held. Tests: `TestMailHold_*` (notify, api, web), `TestHold_*`,
|
||||
`TestMailHoldBanner_*` (both branches), `TestMailHoldRelease_*`; the route joined `r135PostRoutes`. Red-proved: gate
|
||||
skipped → 7 mails sent; cooldown not given back → 0 mails after the release; banner func false → no banner; release
|
||||
without `Release()` → marker stays; CSRF gate open → a Basic-only POST released the hold.
|
||||
|
||||
## v0.144.0 — the operator's decision sheet (D1, D2, D3, D6, D7) and the morning's R-243, R-901, R-304, R-415 (2026-10-09)
|
||||
|
||||
Release together with controller 0.304.0 (D1's operator actions need both). No wire change an older box cannot read.
|
||||
|
||||
@@ -21,6 +21,7 @@ import (
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/gitea"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/hetznerapi"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/intent"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/mailhold"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/mailrelay"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/monitor"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/notify"
|
||||
@@ -319,6 +320,17 @@ func main() {
|
||||
)
|
||||
apiHandler.SetDispatcher(dispatcher)
|
||||
|
||||
// MAIL-HOLD (internal/mailhold): while <data_dir>/MAIL-HOLD exists the hub sends NO e-mail on any path —
|
||||
// dispatcher, /notify, app-mail relay. A hub started on a restored database mailed households within a minute
|
||||
// (2026-10-09 restore drill); the marker makes a restored copy start quiet. Lifted by the operator with
|
||||
// POST /configuration/mail-hold/release; held mails are dropped, not re-sent.
|
||||
mailHold := mailhold.New(cfg.Server.DataDir, logger)
|
||||
if mailHold.Held() {
|
||||
logger.Printf("[WARN] MAIL-HOLD: marker present at start (%s) — the hub sends no e-mail until the operator releases it on the Configuration page", mailHold.Path)
|
||||
}
|
||||
dispatcher.SetMailHold(mailHold)
|
||||
apiHandler.SetMailHold(mailHold)
|
||||
|
||||
// Customer-claim password arc (v0.50.0, F-4): the code engine — the dispatcher delivers the
|
||||
// Hungarian emails, the store holds bcrypt(code) only. Wired into the API (Day-0 issue at
|
||||
// config retrieve, live-box issue + claimed ingest + ACK on report, reset endpoint) and the
|
||||
@@ -341,6 +353,8 @@ func main() {
|
||||
webServer := web.New(dataStore, cfg.Auth.PasswordHash, cfg.API.ReportAPIKey, Version, staleThreshold, logger)
|
||||
webServer.SetTemplateFetcher(templateFetcher)
|
||||
webServer.SetAssetManager(assetsMgr)
|
||||
// MAIL-HOLD banner on every operator page + the release button on Configuration.
|
||||
webServer.SetMailHold(mailHold)
|
||||
webServer.SetClaimEngine(claimEngine) // v0.50.0 — Setup-tab claim chip + resend button
|
||||
webServer.SetSelfBindMailer(dispatcher) // v0.66.0 (R-27) — customer self-bind link button (sibling of claim mailer)
|
||||
webServer.SetEventEmitter(dispatcher.ProcessEvent) // v0.135.0 (R-604) — the "floor raise skipped boxes" mail
|
||||
|
||||
@@ -20,6 +20,7 @@ import (
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/claim"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/configgen"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/intent"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/mailhold"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/mailrelay"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/notify"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
@@ -67,7 +68,9 @@ type Handler struct {
|
||||
floorNotes sync.Map
|
||||
|
||||
// App-email passthrough (POST /api/v1/mail). nil sender = endpoint returns 503.
|
||||
mailSender mailrelay.Sender
|
||||
mailSender mailrelay.Sender
|
||||
// mailHold is the hub-wide mail gate (MAIL-HOLD marker; internal/mailhold). nil never holds.
|
||||
mailHold *mailhold.Hold
|
||||
mailLimiter *mailRateLimiter
|
||||
mailFromAllow map[string]bool
|
||||
|
||||
@@ -156,6 +159,12 @@ func New(store *store.Store, apiKey, resendAPIKey, fromEmail string, templatePro
|
||||
}
|
||||
}
|
||||
|
||||
// SetMailHold wires the hub-wide mail gate: while the MAIL-HOLD marker exists, /notify and the app-mail relay send
|
||||
// nothing (internal/mailhold).
|
||||
func (h *Handler) SetMailHold(m *mailhold.Hold) {
|
||||
h.mailHold = m
|
||||
}
|
||||
|
||||
// SetDispatcher sets the notification dispatcher for event-triggered emails.
|
||||
func (h *Handler) SetDispatcher(d *notify.Dispatcher) {
|
||||
h.dispatcher = d
|
||||
@@ -2583,6 +2592,14 @@ func (h *Handler) handleNotify(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
|
||||
// MAIL-HOLD (internal/mailhold): a restored or quarantined hub sends nothing. The mail is dropped, not queued.
|
||||
if err := h.mailHold.Check("customer event "+payload.EventType, payload.CustomerID); err != nil {
|
||||
h.store.LogNotification(payload.CustomerID, payload.EventType, payload.Severity, payload.Message, "held", "MAIL-HOLD marker present — dropped", "customer")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
w.Write([]byte(`{"status":"ok","sent":false,"reason":"mail_hold"}`))
|
||||
return
|
||||
}
|
||||
|
||||
subject, emailBody := formatNotificationEmail(payload.CustomerID, payload.EventType, payload.Severity, payload.Message, payload.Details)
|
||||
sendErr := h.sendResendEmail(prefs.Email, subject, emailBody)
|
||||
if sendErr != nil {
|
||||
@@ -2617,6 +2634,10 @@ func (h *Handler) handleSavePreferences(w http.ResponseWriter, r *http.Request)
|
||||
Email string `json:"email"`
|
||||
EnabledEvents []string `json:"enabled_events"`
|
||||
CooldownHours int `json:"cooldown_hours"`
|
||||
// EmailCleared (R-922, operator ruling 2026-10-09 option A) is sent true by the controller ONLY when the
|
||||
// household deliberately cleared its address on the dashboard. Omitted (false) on every other push —
|
||||
// including every push from a controller older than the one that emits it.
|
||||
EmailCleared bool `json:"email_cleared"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &payload); err != nil || payload.CustomerID == "" {
|
||||
http.Error(w, "Invalid payload: customer_id required", http.StatusBadRequest)
|
||||
@@ -2627,8 +2648,17 @@ func (h *Handler) handleSavePreferences(w http.ResponseWriter, r *http.Request)
|
||||
// (e.g. an unconfigured box) must never wipe a stored non-empty address — the seeded/edited
|
||||
// email is the customer's alert lifeline. Events + cooldown from the push still apply; a push
|
||||
// with a non-empty email updates everything (customer edits keep working).
|
||||
//
|
||||
// R-922: the ONE exception is a deliberate clear — `email_cleared: true` with an empty address. The household
|
||||
// removed its address, so the hub deletes it (personal data the household withdrew does not stay on our side).
|
||||
// The guard cannot tell a clear from an unconfigured box by the address alone, which is why the clear travels
|
||||
// as its own flag. A flag beside a NON-empty address is an ordinary update (the address wins). Pinned by
|
||||
// TestSavePreferences_DeliberateClearDeletesAddress, TestSavePreferences_EmptyEmailCannotClobber and
|
||||
// TestSavePreferences_ClearFlagWithAddressIsAnUpdate.
|
||||
saveEmail := payload.Email
|
||||
if saveEmail == "" {
|
||||
if saveEmail == "" && payload.EmailCleared {
|
||||
h.logger.Printf("[INFO] Notification prefs push for %s: household cleared its mail address — deleted", payload.CustomerID)
|
||||
} else if saveEmail == "" {
|
||||
if existing, err := h.store.GetNotificationPrefs(payload.CustomerID); err == nil && existing != nil && existing.Email != "" {
|
||||
saveEmail = existing.Email
|
||||
h.logger.Printf("[INFO] Notification prefs push for %s had empty email — preserving stored address", payload.CustomerID)
|
||||
@@ -2641,7 +2671,7 @@ func (h *Handler) handleSavePreferences(w http.ResponseWriter, r *http.Request)
|
||||
return
|
||||
}
|
||||
|
||||
h.logger.Printf("[INFO] Notification preferences updated for %s: email=%s, events=%v", payload.CustomerID, saveEmail, payload.EnabledEvents)
|
||||
h.logger.Printf("[INFO] Notification preferences updated for %s: address set=%t, events=%v", payload.CustomerID, saveEmail != "", payload.EnabledEvents) // never the address (R-922)
|
||||
w.WriteHeader(http.StatusOK)
|
||||
w.Write([]byte(`{"status":"ok"}`))
|
||||
}
|
||||
|
||||
@@ -144,6 +144,17 @@ func (h *Handler) handleMail(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
|
||||
// MAIL-HOLD (internal/mailhold): the relay sends nothing while the hub holds mail. 503 — a temporary refusal the
|
||||
// box's shim surfaces to the app; the hub keeps no copy (held mail is dropped, never queued).
|
||||
holdWho := custID
|
||||
if holdWho == "" {
|
||||
holdWho = "a global-key caller"
|
||||
}
|
||||
if err := h.mailHold.Check("app mail", holdWho); err != nil {
|
||||
http.Error(w, "Mail is on hold at the hub", http.StatusServiceUnavailable)
|
||||
return
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 30*time.Second)
|
||||
defer cancel()
|
||||
if err := h.mailSender.Send(ctx, req.RawMIME, req.MailFrom, req.RcptTo); err != nil {
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/mailhold"
|
||||
)
|
||||
|
||||
// MAIL-HOLD on the two API send paths (the dispatcher's paths: internal/notify/mailhold_test.go).
|
||||
// COMPANION RED-PROOF (REPORT): remove the h.mailHold.Check block from handleNotify / handleMail → these fail with the
|
||||
// mail handed to Resend (the recorded transport / the fake SMTP sender).
|
||||
|
||||
func heldHold(t *testing.T) *mailhold.Hold {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(dir, mailhold.FileName), nil, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return mailhold.New(dir, nil)
|
||||
}
|
||||
|
||||
// recordingTransport stands in for api.resend.com: it records each request and never leaves the process.
|
||||
type recordingTransport struct {
|
||||
mu sync.Mutex
|
||||
calls int
|
||||
}
|
||||
|
||||
func (rt *recordingTransport) RoundTrip(r *http.Request) (*http.Response, error) {
|
||||
rt.mu.Lock()
|
||||
rt.calls++
|
||||
rt.mu.Unlock()
|
||||
return &http.Response{StatusCode: 200, Body: io.NopCloser(strings.NewReader(`{"id":"x"}`)), Header: http.Header{}}, nil
|
||||
}
|
||||
|
||||
func TestMailHold_NotifySendsNothingWhileHeld(t *testing.T) {
|
||||
h, st := newEventTestHandler(t)
|
||||
rt := &recordingTransport{}
|
||||
h.httpClient = &http.Client{Transport: rt}
|
||||
h.resendAPIKey = "test-key"
|
||||
if err := st.SaveNotificationPrefs("c1", "household@example.hu", []string{"backup_failed"}, 6); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h.SetMailHold(heldHold(t))
|
||||
|
||||
rr := do(h, http.MethodPost, "/notify", "ckey", `{"customer_id":"c1","event_type":"backup_failed","severity":"error","message":"m"}`)
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, body=%s", rr.Code, rr.Body.String())
|
||||
}
|
||||
if rt.calls != 0 {
|
||||
t.Fatalf("MAIL-HOLD present but /notify handed %d mail(s) to Resend", rt.calls)
|
||||
}
|
||||
if !strings.Contains(rr.Body.String(), `"reason":"mail_hold"`) {
|
||||
t.Fatalf("the answer must say the mail was held: %s", rr.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestMailHold_AppMailRelayRefusesWhileHeld(t *testing.T) {
|
||||
h, st, _ := newTestHandler(t)
|
||||
withCustomer(t, st, "c1", "ckey")
|
||||
fake := &fakeSender{}
|
||||
h.SetMailRelay(fake, 30, []string{"felhom.eu"})
|
||||
hold := heldHold(t)
|
||||
h.SetMailHold(hold)
|
||||
|
||||
rr := do(h, "POST", "/mail", "ckey", mailBody(t, "vaultwarden@felhom.eu"))
|
||||
if fake.callCount() != 0 {
|
||||
t.Fatalf("MAIL-HOLD present but the relay sent %d mail(s)", fake.callCount())
|
||||
}
|
||||
if rr.Code != http.StatusServiceUnavailable {
|
||||
t.Fatalf("held relay: status %d, want 503", rr.Code)
|
||||
}
|
||||
|
||||
// Released: the relay sends again.
|
||||
if err := hold.Release(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rr = do(h, "POST", "/mail", "ckey", mailBody(t, "vaultwarden@felhom.eu"))
|
||||
if rr.Code != http.StatusOK || fake.callCount() != 1 {
|
||||
t.Fatalf("after release: status %d, sends %d — want 200 and 1", rr.Code, fake.callCount())
|
||||
}
|
||||
}
|
||||
@@ -65,3 +65,64 @@ func TestSavePreferences_EmptyEmailNoStoredRow(t *testing.T) {
|
||||
t.Fatalf("all-off push must store the empty row unchanged, got %+v", prefs)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSavePreferences_DeliberateClearDeletesAddress (R-922, operator ruling 2026-10-09 option A): when the household
|
||||
// clears its address on the dashboard, the controller pushes `email_cleared: true` with an empty address, and the
|
||||
// hub DELETES the stored address. Seen on Tester 1 before the fix: the push answered 200 and the old address stayed.
|
||||
// Red-proof: with the `payload.EmailCleared` branch removed, this fails with email="seeded@example.hu".
|
||||
func TestSavePreferences_DeliberateClearDeletesAddress(t *testing.T) {
|
||||
h, st := newEventTestHandler(t)
|
||||
if err := st.SaveNotificationPrefs("c1", "seeded@example.hu", []string{"node_down"}, 6); err != nil {
|
||||
t.Fatalf("stored prefs: %v", err)
|
||||
}
|
||||
|
||||
rr := do(h, http.MethodPost, "/preferences", "ckey",
|
||||
`{"customer_id":"c1","email":"","email_cleared":true,"enabled_events":[],"cooldown_hours":6}`)
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, body=%s", rr.Code, rr.Body.String())
|
||||
}
|
||||
prefs, err := st.GetNotificationPrefs("c1")
|
||||
if err != nil || prefs == nil {
|
||||
t.Fatalf("prefs: %+v err=%v", prefs, err)
|
||||
}
|
||||
if prefs.Email != "" {
|
||||
t.Fatalf("a deliberate clear must delete the stored address, still stored: email=%q", prefs.Email)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSavePreferences_ClearFlagWithAddressIsAnUpdate (R-922): `email_cleared: true` beside a NON-empty address is an
|
||||
// ordinary update — the address in the push wins, it is never deleted.
|
||||
func TestSavePreferences_ClearFlagWithAddressIsAnUpdate(t *testing.T) {
|
||||
h, st := newEventTestHandler(t)
|
||||
if err := st.SaveNotificationPrefs("c1", "old@example.hu", []string{"node_down"}, 6); err != nil {
|
||||
t.Fatalf("stored prefs: %v", err)
|
||||
}
|
||||
|
||||
rr := do(h, http.MethodPost, "/preferences", "ckey",
|
||||
`{"customer_id":"c1","email":"new@example.hu","email_cleared":true,"enabled_events":["node_down"],"cooldown_hours":6}`)
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, body=%s", rr.Code, rr.Body.String())
|
||||
}
|
||||
prefs, _ := st.GetNotificationPrefs("c1")
|
||||
if prefs == nil || prefs.Email != "new@example.hu" {
|
||||
t.Fatalf("flag + non-empty address must store the pushed address, got %+v", prefs)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSavePreferences_ClearFlagFalseKeepsAddress (R-922): an explicit `email_cleared: false` with an empty address is
|
||||
// NOT a clear — the F12 no-clobber guard still holds.
|
||||
func TestSavePreferences_ClearFlagFalseKeepsAddress(t *testing.T) {
|
||||
h, st := newEventTestHandler(t)
|
||||
if err := st.SaveNotificationPrefs("c1", "seeded@example.hu", []string{"node_down"}, 6); err != nil {
|
||||
t.Fatalf("stored prefs: %v", err)
|
||||
}
|
||||
rr := do(h, http.MethodPost, "/preferences", "ckey",
|
||||
`{"customer_id":"c1","email":"","email_cleared":false,"enabled_events":["node_down"],"cooldown_hours":6}`)
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, body=%s", rr.Code, rr.Body.String())
|
||||
}
|
||||
prefs, _ := st.GetNotificationPrefs("c1")
|
||||
if prefs == nil || prefs.Email != "seeded@example.hu" {
|
||||
t.Fatalf("email_cleared:false with an empty address must keep the stored address, got %+v", prefs)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
// Package mailhold is the hub's one mail gate: while the marker file `<data_dir>/MAIL-HOLD` exists, the hub sends
|
||||
// NO e-mail at all — household or operator, on every path that reaches Resend (the dispatcher's HTTP-API sends, the
|
||||
// legacy /notify endpoint, the app-mail SMTP relay).
|
||||
//
|
||||
// WHY. The 2026-10-09 restore drill (documentation/audits/dooplex-survival-2026-10-09/partD-05-checks.txt) started a
|
||||
// hub on a restored database, and within a minute it mailed households the pending "kernel notice" mails.
|
||||
// `notifications.operator_enabled: false` did not stop it, because that switch covers the operator channel only. A
|
||||
// restored copy is a second hub speaking with the first one's voice; it must start quiet until the operator says so.
|
||||
// The marker is created (an empty file is enough) before a restored copy's first start.
|
||||
//
|
||||
// HELD MAILS ARE DROPPED, NOT QUEUED, and that is deliberate: every mail a restored copy wants to send was decided
|
||||
// from a snapshot of the past — a kernel notice for a step the real hub already ran or will run, an alarm about a
|
||||
// state that has since changed. Re-sending them after the hold is lifted would deliver stale news as if it were
|
||||
// current. After the release, the hub's checkers and boxes produce fresh mails from live state on their own. Each
|
||||
// dropped mail is logged once (`[WARN] MAIL-HOLD: not sending <kind> to <who>` — never the address, never the body).
|
||||
//
|
||||
// The marker is read on every send (one stat of a file in the data dir; no wait on a device the hub does not
|
||||
// already depend on for its database). A stat error other than "does not exist" counts as HELD: the gate fails
|
||||
// closed, and the operator pages show the banner, so the state is never invisible.
|
||||
package mailhold
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
)
|
||||
|
||||
// FileName is the marker's name inside the hub's data directory.
|
||||
const FileName = "MAIL-HOLD"
|
||||
|
||||
// ErrHeld is returned by Check for a mail the hold stopped. Callers treat it as "not sent" (never as sent).
|
||||
var ErrHeld = errors.New("mail hold: the hub is not sending e-mail (MAIL-HOLD marker present)")
|
||||
|
||||
// Hold reads and removes the marker. The zero value and a nil *Hold never hold (tests and wiring without a data dir).
|
||||
type Hold struct {
|
||||
Path string
|
||||
Logger *log.Logger
|
||||
}
|
||||
|
||||
// New returns the hold for a data directory.
|
||||
func New(dataDir string, logger *log.Logger) *Hold {
|
||||
return &Hold{Path: filepath.Join(dataDir, FileName), Logger: logger}
|
||||
}
|
||||
|
||||
// Held reports whether the marker is present (fail-closed on an unreadable state).
|
||||
func (h *Hold) Held() bool {
|
||||
if h == nil || h.Path == "" {
|
||||
return false
|
||||
}
|
||||
_, err := os.Stat(h.Path)
|
||||
if err == nil {
|
||||
return true
|
||||
}
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return false
|
||||
}
|
||||
h.logf("[WARN] MAIL-HOLD: cannot read the marker state (%v) — treating mail as HELD", err)
|
||||
return true
|
||||
}
|
||||
|
||||
// Check is the gate every sender calls immediately before handing a mail to Resend. When held it logs the one WARN
|
||||
// line for this mail and returns ErrHeld; otherwise nil. kind names the mail ("customer event backup_failed",
|
||||
// "kernel notice", …); who is a customer ID or "operator" — never an address.
|
||||
func (h *Hold) Check(kind, who string) error {
|
||||
if !h.Held() {
|
||||
return nil
|
||||
}
|
||||
h.logf("[WARN] MAIL-HOLD: not sending %s to %s", kind, who)
|
||||
return ErrHeld
|
||||
}
|
||||
|
||||
// Release removes the marker. Removing a marker that is not there is not an error (the release is idempotent).
|
||||
func (h *Hold) Release() error {
|
||||
if h == nil || h.Path == "" {
|
||||
return nil
|
||||
}
|
||||
if err := os.Remove(h.Path); err != nil && !errors.Is(err, os.ErrNotExist) {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (h *Hold) logf(format string, args ...interface{}) {
|
||||
if h != nil && h.Logger != nil {
|
||||
h.Logger.Printf(format, args...)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
package mailhold
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestHold_MarkerDecidesAndReleaseRemovesIt(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
buf := &bytes.Buffer{}
|
||||
h := New(dir, log.New(buf, "", 0))
|
||||
if h.Held() {
|
||||
t.Fatal("no marker: must not hold")
|
||||
}
|
||||
if err := h.Check("kernel notice", "c1"); err != nil {
|
||||
t.Fatalf("no marker: Check = %v", err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, FileName), nil, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := h.Check("kernel notice", "c1"); !errors.Is(err, ErrHeld) {
|
||||
t.Fatalf("marker present: Check = %v, want ErrHeld", err)
|
||||
}
|
||||
if got := buf.String(); !strings.Contains(got, "[WARN] MAIL-HOLD: not sending kernel notice to c1") {
|
||||
t.Fatalf("held mail log line missing: %q", got)
|
||||
}
|
||||
if err := h.Release(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(dir, FileName)); !os.IsNotExist(err) {
|
||||
t.Fatalf("Release left the marker: %v", err)
|
||||
}
|
||||
if err := h.Release(); err != nil {
|
||||
t.Fatalf("a second Release must be a no-op, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Fail-closed: a marker state that cannot be read (here: the data dir is a FILE, so stat says ENOTDIR) holds mail.
|
||||
func TestHold_UnreadableStateHolds(t *testing.T) {
|
||||
f := filepath.Join(t.TempDir(), "not-a-dir")
|
||||
if err := os.WriteFile(f, nil, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !New(f, nil).Held() {
|
||||
t.Fatal("an unreadable marker state must count as HELD")
|
||||
}
|
||||
}
|
||||
|
||||
func TestHold_NilNeverHolds(t *testing.T) {
|
||||
var h *Hold
|
||||
if h.Held() || h.Check("x", "y") != nil || h.Release() != nil {
|
||||
t.Fatal("a nil hold must never hold and never fail")
|
||||
}
|
||||
}
|
||||
@@ -3,6 +3,7 @@ package notify
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
@@ -11,6 +12,7 @@ import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/mailhold"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/i18n"
|
||||
@@ -40,8 +42,32 @@ type Dispatcher struct {
|
||||
// afterFn schedules a delayed call (time.AfterFunc; a seam so tests run the retries at once). Used by
|
||||
// retryOperatorEmail (the lost alarm of 2026-10-05).
|
||||
afterFn func(time.Duration, func())
|
||||
|
||||
// mailHold is the hub-wide mail gate (internal/mailhold): while its marker exists, deliver() hands nothing to
|
||||
// sendEmailFn. nil never holds.
|
||||
mailHold *mailhold.Hold
|
||||
}
|
||||
|
||||
// SetMailHold wires the hub-wide mail gate (the MAIL-HOLD marker in the data dir).
|
||||
func (d *Dispatcher) SetMailHold(m *mailhold.Hold) {
|
||||
d.mailHold = m
|
||||
}
|
||||
|
||||
// deliver is the ONE way the dispatcher sends a mail: the MAIL-HOLD gate, then sendEmailFn. Every send in this file
|
||||
// goes through it, so the hold cannot be bypassed by a path someone adds later without calling sendEmailFn directly
|
||||
// (pinned by TestMailHold_EverySendPathIsGated). kind names the mail; who is a customer ID or "operator" — the hold's
|
||||
// log line carries those two and never the address. A held mail returns mailhold.ErrHeld and is DROPPED, never
|
||||
// queued: see the package comment for why.
|
||||
func (d *Dispatcher) deliver(kind, who, to, subject, body string, headers map[string]string) error {
|
||||
if err := d.mailHold.Check(kind, who); err != nil {
|
||||
return err
|
||||
}
|
||||
return d.sendEmailFn(to, subject, body, headers)
|
||||
}
|
||||
|
||||
// isHeld reports whether a send error is the mail hold (not a delivery failure).
|
||||
func isHeld(err error) bool { return errors.Is(err, mailhold.ErrHeld) }
|
||||
|
||||
// operatorRetryDelays are the waits before each further try of an operator mail whose send FAILED (the lost alarm of 2026-10-05). Measured
|
||||
// 2026-10-05: demo-hp's whole_guest_backup_failed (error) hit a 10 s Resend client timeout, was logged `failed`, and was
|
||||
// never sent — the only operator mail of 692 that failed, and the one alarm the operator needed that night. The
|
||||
@@ -77,7 +103,12 @@ func (d *Dispatcher) retryOperatorEmail(customerID, eventType, severity, message
|
||||
}
|
||||
d.afterFn(operatorRetryDelays[attempt], func() {
|
||||
n := attempt + 1
|
||||
if err := d.sendEmailFn(d.operatorEmail, subject, body, headers); err != nil {
|
||||
if err := d.deliver("operator mail "+eventType+" (retry)", "operator", d.operatorEmail, subject, body, headers); err != nil {
|
||||
if isHeld(err) {
|
||||
// The hold began between tries: the mail is dropped like every other held mail, and the retries stop.
|
||||
d.store.LogNotification(customerID, eventType, severity, message, "held", "MAIL-HOLD marker present — dropped", "operator")
|
||||
return
|
||||
}
|
||||
d.logger.Printf("[ERROR] Operator email retry %d failed for %s/%s: %v", n, customerID, eventType, err)
|
||||
d.store.LogNotification(customerID, eventType, severity, message, "failed", fmt.Sprintf("retry %d: %v", n, err), "operator")
|
||||
d.retryOperatorEmail(customerID, eventType, severity, message, subject, body, headers, n)
|
||||
@@ -246,7 +277,9 @@ func (d *Dispatcher) sendTestEmail(customerID string) {
|
||||
subject := b.Msg(lang, "mail.test.subject")
|
||||
body := b.Msg(lang, "mail.test.body")
|
||||
|
||||
if err := d.sendEmailFn(prefs.Email, subject, body, nil); err != nil {
|
||||
if err := d.deliver("test mail", customerID, prefs.Email, subject, body, nil); isHeld(err) {
|
||||
d.store.LogNotification(customerID, "test", "info", "Teszt értesítés", "held", "MAIL-HOLD marker present — dropped", "customer")
|
||||
} else if err != nil {
|
||||
d.logger.Printf("[ERROR] Test email to %s failed: %v", prefs.Email, err)
|
||||
d.store.LogNotification(customerID, "test", "info", "Teszt értesítés", "failed", err.Error(), "customer")
|
||||
} else {
|
||||
@@ -268,7 +301,10 @@ headers render correctly. The customer test mail result is recorded in the
|
||||
notification log.
|
||||
|
||||
Dashboard: https://hub.felhom.eu/customers/%s`, customerID, customerID)
|
||||
if err := d.sendEmailFn(d.operatorEmail, opSubject, opBody, priorityHeaders("critical")); err != nil {
|
||||
if err := d.deliver("operator test mail", "operator", d.operatorEmail, opSubject, opBody, priorityHeaders("critical")); isHeld(err) {
|
||||
d.store.LogNotification(customerID, "test", "info", "operator test copy", "held", "MAIL-HOLD marker present — dropped", "operator")
|
||||
return
|
||||
} else if err != nil {
|
||||
d.logger.Printf("[ERROR] Operator test email failed for %s: %v", customerID, err)
|
||||
d.store.LogNotification(customerID, "test", "info", "operator test copy", "failed", err.Error(), "operator")
|
||||
return
|
||||
@@ -330,7 +366,11 @@ func (d *Dispatcher) processRecovery(customerID, eventType, severity, message, m
|
||||
|
||||
subject, body := FormatCustomerEmail(d.store.CustomerLanguage(customerID),
|
||||
customerID, eventType, severity, message, messageCustomer, detailsJSON)
|
||||
if err := d.sendEmailFn(prefs.Email, subject, body, priorityHeaders(severity)); err != nil {
|
||||
if err := d.deliver("customer recovery mail "+eventType, customerID, prefs.Email, subject, body, priorityHeaders(severity)); isHeld(err) {
|
||||
d.forgetCustomerCooldown(cooldownKey)
|
||||
d.store.LogNotification(customerID, eventType, severity, message, "held", "MAIL-HOLD marker present — dropped", "customer")
|
||||
return
|
||||
} else if err != nil {
|
||||
d.logger.Printf("[ERROR] Customer recovery email failed for %s/%s: %v", customerID, eventType, err)
|
||||
d.store.LogNotification(customerID, eventType, severity, message, "failed", err.Error(), "customer")
|
||||
return
|
||||
@@ -607,7 +647,15 @@ func (d *Dispatcher) processOperator(customerID, eventType, severity, message, d
|
||||
subject, body := FormatOperatorEmail(customerID, eventType, severity, message, detailsJSON)
|
||||
|
||||
hdrs := priorityHeaders(severity)
|
||||
if err := d.sendEmailFn(d.operatorEmail, subject, body, hdrs); err != nil {
|
||||
if err := d.deliver("operator mail "+eventType, "operator", d.operatorEmail, subject, body, hdrs); isHeld(err) {
|
||||
// Dropped, and the cooldown it armed is given back: the first real alarm after the release must not be
|
||||
// silenced by a mail that was never sent.
|
||||
d.mu.Lock()
|
||||
delete(d.opCooldowns, cooldownKey)
|
||||
d.mu.Unlock()
|
||||
d.store.LogNotification(customerID, eventType, severity, message, "held", "MAIL-HOLD marker present — dropped", "operator")
|
||||
return
|
||||
} else if err != nil {
|
||||
d.logger.Printf("[ERROR] Operator email failed for %s/%s: %v — retrying in %v", customerID, eventType, err, operatorRetryDelays[0])
|
||||
d.store.LogNotification(customerID, eventType, severity, message, "failed", err.Error(), "operator")
|
||||
d.retryOperatorEmail(customerID, eventType, severity, message, subject, body, hdrs, 0)
|
||||
@@ -883,12 +931,16 @@ func (d *Dispatcher) processCustomer(customerID, eventType, severity, message, m
|
||||
subject, body := FormatCustomerEmail(d.store.CustomerLanguage(customerID),
|
||||
customerID, eventType, severity, message, messageCustomer, detailsJSON)
|
||||
|
||||
if err := d.sendEmailFn(prefs.Email, subject, body, priorityHeaders(severity)); err != nil {
|
||||
if err := d.deliver("customer mail "+eventType, customerID, prefs.Email, subject, body, priorityHeaders(severity)); isHeld(err) {
|
||||
d.forgetCustomerCooldown(cooldownKey)
|
||||
d.store.LogNotification(customerID, eventType, severity, message, "held", "MAIL-HOLD marker present — dropped", "customer")
|
||||
return
|
||||
} else if err != nil {
|
||||
d.logger.Printf("[ERROR] Customer email failed for %s/%s: %v", customerID, eventType, err)
|
||||
d.store.LogNotification(customerID, eventType, severity, message, "failed", err.Error(), "customer")
|
||||
return
|
||||
}
|
||||
d.logger.Printf("[INFO] Customer email sent to %s for %s/%s", prefs.Email, customerID, eventType)
|
||||
d.logger.Printf("[INFO] Customer email sent for %s/%s", customerID, eventType) // never the address (R-922)
|
||||
d.store.LogNotification(customerID, eventType, severity, message, "sent", "", "customer")
|
||||
}
|
||||
|
||||
@@ -944,6 +996,13 @@ func (d *Dispatcher) replyToFor(to string) string {
|
||||
return op
|
||||
}
|
||||
|
||||
// forgetCustomerCooldown gives back a customer cooldown armed for a mail the hold dropped.
|
||||
func (d *Dispatcher) forgetCustomerCooldown(key string) {
|
||||
d.mu.Lock()
|
||||
delete(d.custCooldowns, key)
|
||||
d.mu.Unlock()
|
||||
}
|
||||
|
||||
func isEventEnabled(enabledEvents []string, eventType string) bool {
|
||||
for _, e := range enabledEvents {
|
||||
if e == eventType {
|
||||
@@ -964,7 +1023,10 @@ func (d *Dispatcher) SendClaimEmail(kind, customerID, email, domain, code string
|
||||
}
|
||||
subject, body := FormatClaimEmail(d.store.CustomerLanguage(customerID), kind, customerID, domain, code)
|
||||
eventType := "claim_" + kind
|
||||
if err := d.sendEmailFn(email, subject, body, nil); err != nil {
|
||||
if err := d.deliver("claim "+kind+" mail", customerID, email, subject, body, nil); isHeld(err) {
|
||||
d.store.LogNotification(customerID, eventType, "info", subject, "held", "MAIL-HOLD marker present — dropped", "customer")
|
||||
return err
|
||||
} else if err != nil {
|
||||
d.logger.Printf("[ERROR] claim %s email to customer %s failed: %v", kind, customerID, err)
|
||||
d.store.LogNotification(customerID, eventType, "info", subject, "failed", err.Error(), "customer")
|
||||
return err
|
||||
@@ -985,7 +1047,10 @@ func (d *Dispatcher) SendSelfBindEmail(customerID, email, link string) error {
|
||||
return fmt.Errorf("notify: no resend api key")
|
||||
}
|
||||
subject, body := FormatSelfBindEmail(d.store.CustomerLanguage(customerID), customerID, link)
|
||||
if err := d.sendEmailFn(email, subject, body, nil); err != nil {
|
||||
if err := d.deliver("self-bind link mail", customerID, email, subject, body, nil); isHeld(err) {
|
||||
d.store.LogNotification(customerID, "selfbind_link", "info", subject, "held", "MAIL-HOLD marker present — dropped", "customer")
|
||||
return err
|
||||
} else if err != nil {
|
||||
d.logger.Printf("[ERROR] self-bind link email to customer %s failed: %v", customerID, err)
|
||||
d.store.LogNotification(customerID, "selfbind_link", "info", subject, "failed", err.Error(), "customer")
|
||||
return err
|
||||
@@ -1018,7 +1083,11 @@ func (d *Dispatcher) SendKernelNotice(customerID, kver string) (string, error) {
|
||||
}
|
||||
lang := d.store.CustomerLanguage(customerID)
|
||||
subject, body := FormatKernelNoticeEmail(lang)
|
||||
if err := d.sendEmailFn(to, subject, body, nil); err != nil {
|
||||
if err := d.deliver("kernel notice", customerID, to, subject, body, nil); isHeld(err) {
|
||||
// The caller runs no kernel step on an error (no mail, no step) — exactly right for a held hub.
|
||||
d.store.LogNotification(customerID, "kernel_notice", "info", subject, "held", "MAIL-HOLD marker present — dropped", "customer")
|
||||
return lang, err
|
||||
} else if err != nil {
|
||||
d.logger.Printf("[ERROR] kernel notice (%s) to customer %s failed: %v", kver, customerID, err)
|
||||
d.store.LogNotification(customerID, "kernel_notice", "info", subject, "failed", err.Error(), "customer")
|
||||
return lang, err
|
||||
|
||||
@@ -0,0 +1,143 @@
|
||||
package notify
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/mailhold"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
// MAIL-HOLD — a restored hub starts quiet. Evidence: the 2026-10-09 restore drill
|
||||
// (documentation/audits/dooplex-survival-2026-10-09/partD-05-checks.txt): a hub started on a restored database mailed
|
||||
// households pending kernel notices within a minute, and `notifications.operator_enabled: false` did not stop it.
|
||||
//
|
||||
// COMPANION RED-PROOF (REPORT): make deliver() skip the mailHold.Check call (the pre-fix shape: sendEmailFn is reached
|
||||
// directly) → TestMailHold_DispatcherSendsNothingWhileHeld fails with every path's mail SENT.
|
||||
|
||||
type holdRecorder struct {
|
||||
mu sync.Mutex
|
||||
sent []string // subjects
|
||||
}
|
||||
|
||||
func (r *holdRecorder) fn(to, subject, body string, headers map[string]string) error {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.sent = append(r.sent, subject)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r *holdRecorder) count() int {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
return len(r.sent)
|
||||
}
|
||||
|
||||
// holdDispatcher returns a dispatcher whose household c1 has a registered address, notification prefs with
|
||||
// backup_failed on, the operator channel ON, the MAIL-HOLD marker present, and a log captured in buf.
|
||||
func holdDispatcher(t *testing.T) (*Dispatcher, *store.Store, *holdRecorder, *mailhold.Hold, *bytes.Buffer) {
|
||||
t.Helper()
|
||||
st := newDispStore(t)
|
||||
if err := st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c1", APIKey: "k", RetrievalPassword: "p", Email: "household@example.hu"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.SaveNotificationPrefs("c1", "household@example.hu", []string{"backup_failed"}, 6); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
buf := &bytes.Buffer{}
|
||||
logger := log.New(buf, "", 0)
|
||||
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, logger)
|
||||
rec := &holdRecorder{}
|
||||
d.sendEmailFn = rec.fn
|
||||
d.afterFn = func(time.Duration, func()) {} // a retry must never fire in these tests
|
||||
dir := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(dir, mailhold.FileName), nil, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
hold := mailhold.New(dir, logger)
|
||||
d.SetMailHold(hold)
|
||||
return d, st, rec, hold, buf
|
||||
}
|
||||
|
||||
// The consequence asserted: with the marker present, NOT ONE mail reaches the sender, on any dispatcher path —
|
||||
// household event, operator event, recovery, test mail, kernel notice, claim code, self-bind link.
|
||||
func TestMailHold_DispatcherSendsNothingWhileHeld(t *testing.T) {
|
||||
d, st, rec, _, buf := holdDispatcher(t)
|
||||
|
||||
d.ProcessEvent("c1", "backup_failed", "error", "Backup failed", "", "box") // operator + household
|
||||
d.ProcessEvent("c1", "test", "info", "test", "", "operator") // test mail, both channels
|
||||
if _, err := d.SendKernelNotice("c1", "6.8.12-1"); err == nil {
|
||||
t.Errorf("kernel notice: a held mail must report an error (the caller then runs no kernel step)")
|
||||
}
|
||||
if err := d.SendClaimEmail("claim", "c1", "household@example.hu", "c1.example", "CODE-1234"); err == nil {
|
||||
t.Errorf("claim mail: a held mail must report an error")
|
||||
}
|
||||
if err := d.SendSelfBindEmail("c1", "household@example.hu", "https://hub.example/bind/x"); err == nil {
|
||||
t.Errorf("self-bind mail: a held mail must report an error")
|
||||
}
|
||||
|
||||
if n := rec.count(); n != 0 {
|
||||
t.Fatalf("MAIL-HOLD present but %d mail(s) were SENT: %v", n, rec.sent)
|
||||
}
|
||||
out := buf.String()
|
||||
if got := strings.Count(out, "[WARN] MAIL-HOLD: not sending"); got < 7 {
|
||||
t.Errorf("each held mail is logged once: want >= 7 MAIL-HOLD lines, got %d\n%s", got, out)
|
||||
}
|
||||
if strings.Contains(out, "household@example.hu") || strings.Contains(out, "op@felhom.eu") {
|
||||
t.Errorf("the hold's log must carry no address:\n%s", out)
|
||||
}
|
||||
if strings.Contains(out, "CODE-1234") {
|
||||
t.Errorf("the hold's log must carry no mail body")
|
||||
}
|
||||
// The record says what happened: held, not sent and not failed.
|
||||
if _, ok, _ := st.LastCustomerSentAt("c1", []string{"backup_failed"}); ok {
|
||||
t.Errorf("a held household mail was recorded as SENT")
|
||||
}
|
||||
}
|
||||
|
||||
// After the release the hub mails again — and the very same event key mails at once: the cooldown a held mail armed is
|
||||
// given back, so the first real alarm after the release is not silenced by a mail that was never sent.
|
||||
func TestMailHold_ReleaseResumesSends(t *testing.T) {
|
||||
d, _, rec, hold, _ := holdDispatcher(t)
|
||||
|
||||
d.ProcessEvent("c1", "backup_failed", "error", "Backup failed", "", "box")
|
||||
if rec.count() != 0 {
|
||||
t.Fatalf("held: %d mail(s) sent", rec.count())
|
||||
}
|
||||
if err := hold.Release(); err != nil {
|
||||
t.Fatalf("release: %v", err)
|
||||
}
|
||||
if hold.Held() {
|
||||
t.Fatalf("the marker is still there after Release")
|
||||
}
|
||||
d.ProcessEvent("c1", "backup_failed", "error", "Backup failed", "", "box")
|
||||
if n := rec.count(); n != 2 {
|
||||
t.Fatalf("after the release the operator AND the household mail must go out: sent=%d %v", n, rec.sent)
|
||||
}
|
||||
// Held mails are DROPPED, not re-sent: exactly the two fresh ones, no backlog.
|
||||
}
|
||||
|
||||
// Belt for the "one gate" claim: in dispatcher.go the sender seam is called in exactly ONE place, inside deliver().
|
||||
// A new send path that calls sendEmailFn directly would bypass the hold; this fails when one appears.
|
||||
func TestMailHold_EverySendPathIsGated(t *testing.T) {
|
||||
src, err := os.ReadFile("dispatcher.go")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
calls := regexp.MustCompile(`d\.sendEmailFn\(`).FindAllIndex(src, -1)
|
||||
if len(calls) != 1 {
|
||||
t.Fatalf("dispatcher.go calls d.sendEmailFn( %d times; every send must go through deliver() (the MAIL-HOLD gate)", len(calls))
|
||||
}
|
||||
body := string(src)
|
||||
i := strings.Index(body, "func (d *Dispatcher) deliver(")
|
||||
if i < 0 || calls[0][0] < i || calls[0][0] > i+600 {
|
||||
t.Fatalf("the one d.sendEmailFn( call is not inside deliver()")
|
||||
}
|
||||
}
|
||||
@@ -50,6 +50,7 @@ func TestTemplatesParseWithFuncmap(t *testing.T) {
|
||||
"memoryColor": memoryColor,
|
||||
"accuracyClass": accuracyClass,
|
||||
"gt": func(a, b int) bool { return false },
|
||||
"mailHeld": func() bool { return false },
|
||||
}).ParseFS(templateFS, "templates/*.html"); err != nil {
|
||||
t.Fatalf("templates failed to parse: %v", err)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,135 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/mailhold"
|
||||
)
|
||||
|
||||
// MAIL-HOLD on the operator UI: the banner on every operator page while the marker exists, and the release endpoint
|
||||
// behind the operator's auth + the R-135 CSRF gate.
|
||||
|
||||
const mailHoldBannerText = "E-mail is on hold"
|
||||
|
||||
func holdIn(t *testing.T, held bool) (*mailhold.Hold, string) {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
marker := filepath.Join(dir, mailhold.FileName)
|
||||
if held {
|
||||
if err := os.WriteFile(marker, nil, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
return mailhold.New(dir, nil), marker
|
||||
}
|
||||
|
||||
func getPage(t *testing.T, s *Server, path string) string {
|
||||
t.Helper()
|
||||
r := httptest.NewRequest(http.MethodGet, path, nil)
|
||||
w := httptest.NewRecorder()
|
||||
s.ServeHTTP(w, r)
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("GET %s: %d", path, w.Code)
|
||||
}
|
||||
return w.Body.String()
|
||||
}
|
||||
|
||||
// Both branches of the {{if mailHeld}} gate, on the dashboard and on Configuration (which also carries the button).
|
||||
func TestMailHoldBanner_RendersOnlyWhileHeld(t *testing.T) {
|
||||
for _, held := range []bool{true, false} {
|
||||
s, _ := newTestServer(t)
|
||||
h, _ := holdIn(t, held)
|
||||
s.SetMailHold(h)
|
||||
for _, p := range []string{"/", "/configuration", "/hosts", "/configs"} {
|
||||
body := getPage(t, s, p)
|
||||
if got := strings.Contains(body, mailHoldBannerText); got != held {
|
||||
t.Errorf("held=%v GET %s: banner shown=%v", held, p, got)
|
||||
}
|
||||
}
|
||||
cfg := getPage(t, s, "/configuration")
|
||||
if got := strings.Contains(cfg, `action="/configuration/mail-hold/release"`); got != held {
|
||||
t.Errorf("held=%v: release button shown=%v", held, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// No hold wired at all (nil): no banner, pages render.
|
||||
func TestMailHoldBanner_NilHoldRendersNoBanner(t *testing.T) {
|
||||
s, _ := newTestServer(t)
|
||||
if strings.Contains(getPage(t, s, "/"), mailHoldBannerText) {
|
||||
t.Fatal("a nil hold must render no banner")
|
||||
}
|
||||
}
|
||||
|
||||
// The operator's release, through the production wiring (RequireAuth around ServeHTTP) with a session and its CSRF
|
||||
// token: the marker is gone, and the banner with it.
|
||||
func TestMailHoldRelease_RemovesMarker(t *testing.T) {
|
||||
s, h := r135Handler(t)
|
||||
hold, marker := holdIn(t, true)
|
||||
s.SetMailHold(hold)
|
||||
s.sessionsMu.Lock()
|
||||
s.sessions["sess1"] = &hubSession{expiresAt: time.Now().Add(time.Hour), csrfToken: "tok1"}
|
||||
s.sessionsMu.Unlock()
|
||||
|
||||
w := r135Post(h, "/configuration/mail-hold/release", func(r *http.Request) {
|
||||
r.AddCookie(&http.Cookie{Name: SessionCookieName, Value: "sess1"})
|
||||
r.Header.Set("X-CSRF-Token", "tok1")
|
||||
})
|
||||
if w.Code != http.StatusSeeOther || !strings.Contains(w.Header().Get("Location"), "flash=mail_hold_released") {
|
||||
t.Fatalf("release: %d Location=%q", w.Code, w.Header().Get("Location"))
|
||||
}
|
||||
if _, err := os.Stat(marker); !os.IsNotExist(err) {
|
||||
t.Fatalf("the marker is still there after the release: %v", err)
|
||||
}
|
||||
if hold.Held() {
|
||||
t.Fatal("still held after the release")
|
||||
}
|
||||
}
|
||||
|
||||
// Refusals: Basic auth without the operator header (the R-135 cross-site shape), a session without its token, and no
|
||||
// credentials at all. The marker stays in every case.
|
||||
func TestMailHoldRelease_RefusedWithoutOperatorAuthOrCSRF(t *testing.T) {
|
||||
s, h := r135Handler(t)
|
||||
hold, marker := holdIn(t, true)
|
||||
s.SetMailHold(hold)
|
||||
s.sessionsMu.Lock()
|
||||
s.sessions["sess1"] = &hubSession{expiresAt: time.Now().Add(time.Hour), csrfToken: "tok1"}
|
||||
s.sessionsMu.Unlock()
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
mut func(*http.Request)
|
||||
ok func(int) bool
|
||||
}{
|
||||
{"basic without operator header", func(r *http.Request) { r.SetBasicAuth("", "op-pass") }, func(c int) bool { return c == http.StatusForbidden }},
|
||||
{"session without token", func(r *http.Request) { r.AddCookie(&http.Cookie{Name: SessionCookieName, Value: "sess1"}) }, func(c int) bool { return c == http.StatusForbidden }},
|
||||
{"no credentials", func(r *http.Request) { r.Header.Set(OperatorCLIHeader, "cli") }, func(c int) bool { return c == http.StatusFound || c == http.StatusUnauthorized }},
|
||||
}
|
||||
for _, c := range cases {
|
||||
w := r135Post(h, "/configuration/mail-hold/release", c.mut)
|
||||
if !c.ok(w.Code) {
|
||||
t.Errorf("%s: status %d", c.name, w.Code)
|
||||
}
|
||||
if _, err := os.Stat(marker); err != nil {
|
||||
t.Fatalf("%s: the marker was removed by a refused request: %v", c.name, err)
|
||||
}
|
||||
}
|
||||
|
||||
// The CLI shape (Basic + the operator header) passes.
|
||||
w := r135Post(h, "/configuration/mail-hold/release", func(r *http.Request) {
|
||||
r.SetBasicAuth("", "op-pass")
|
||||
r.Header.Set(OperatorCLIHeader, "cli")
|
||||
})
|
||||
if w.Code != http.StatusSeeOther {
|
||||
t.Fatalf("operator CLI release: %d", w.Code)
|
||||
}
|
||||
if hold.Held() {
|
||||
t.Fatal("operator CLI release left the hold in place")
|
||||
}
|
||||
}
|
||||
@@ -42,6 +42,7 @@ var r135PostRoutes = []string{
|
||||
"/configuration/global-floor",
|
||||
"/configuration/artifacts",
|
||||
"/configuration/password",
|
||||
"/configuration/mail-hold/release",
|
||||
"/configs/c1/delete",
|
||||
"/configs/c1/edit",
|
||||
"/configs/c1/offsite-reissue",
|
||||
|
||||
@@ -20,6 +20,7 @@ import (
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/claim"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/gitea"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/intent"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/mailhold"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/monitor"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/offsite"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/poke"
|
||||
@@ -124,13 +125,20 @@ type Server struct {
|
||||
// beforeCreateSave is a TEST seam: called in handleConfigCreate after the duplicate check, before the
|
||||
// save. nil in production.
|
||||
beforeCreateSave func(customerID string)
|
||||
// mailHold is the hub-wide mail gate (internal/mailhold); nil never holds.
|
||||
mailHold *mailhold.Hold
|
||||
// cfAPIBase is a TEST seam for the Cloudflare token reach check (R-138 option C): "" = the real API.
|
||||
cfAPIBase string
|
||||
}
|
||||
|
||||
// New creates a new web server.
|
||||
func New(store *store.Store, passwordHash, apiKey, version string, staleThreshold time.Duration, logger *log.Logger) *Server {
|
||||
// srv is assigned below; template funcs that read server state close over it (they run at render time only).
|
||||
var srv *Server
|
||||
funcMap := template.FuncMap{
|
||||
// mailHeld drives the MAIL-HOLD banner on every operator page (internal/mailhold). Read at render time, so the
|
||||
// banner disappears the moment the marker is gone. Pinned by TestMailHoldBanner_*.
|
||||
"mailHeld": func() bool { return srv != nil && srv.mailHold.Held() },
|
||||
"timeAgo": timeAgo,
|
||||
"timeAgoPtr": func(t *time.Time) string {
|
||||
if t == nil {
|
||||
@@ -160,7 +168,7 @@ func New(store *store.Store, passwordHash, apiKey, version string, staleThreshol
|
||||
|
||||
tmpl := template.Must(template.New("").Funcs(funcMap).ParseFS(templateFS, "templates/*.html"))
|
||||
|
||||
return &Server{
|
||||
srv = &Server{
|
||||
store: store,
|
||||
configPasswordHash: passwordHash,
|
||||
apiKey: apiKey,
|
||||
@@ -171,6 +179,7 @@ func New(store *store.Store, passwordHash, apiKey, version string, staleThreshol
|
||||
sessions: make(map[string]*hubSession),
|
||||
bindLimiter: newBindRateLimiter(30), // public /bind/ surface: 30 req/min/IP burst (R-27)
|
||||
}
|
||||
return srv
|
||||
}
|
||||
|
||||
// effectivePasswordHash returns the operator login password bcrypt hash in force: the UI-set DB
|
||||
@@ -406,6 +415,29 @@ func (s *Server) artifactChoices(ctx context.Context, pkg, file string) []artifa
|
||||
}
|
||||
|
||||
// SetEventEmitter wires the notification dispatcher (R-604). INIT-ONLY.
|
||||
// SetMailHold wires the hub-wide mail gate (internal/mailhold): the banner on every operator page and the release
|
||||
// endpoint. nil = never held (no banner, the release is a no-op).
|
||||
func (s *Server) SetMailHold(m *mailhold.Hold) {
|
||||
s.mailHold = m
|
||||
}
|
||||
|
||||
// handleMailHoldRelease lifts the MAIL-HOLD: it removes the marker, and the hub sends e-mail again from the next mail
|
||||
// on. Mails dropped during the hold are NOT re-sent (internal/mailhold explains why). Operator-only: it sits behind
|
||||
// RequireAuth and the R-135 CSRF gate in ServeHTTP like every other state-changing route.
|
||||
func (s *Server) handleMailHoldRelease(w http.ResponseWriter, r *http.Request) {
|
||||
if s.mailHold == nil || !s.mailHold.Held() {
|
||||
http.Redirect(w, r, "/configuration?flash=mail_hold_not_held", http.StatusSeeOther)
|
||||
return
|
||||
}
|
||||
if err := s.mailHold.Release(); err != nil {
|
||||
s.logger.Printf("[ERROR] MAIL-HOLD: release failed — the marker is still in place: %v", err)
|
||||
http.Redirect(w, r, "/configuration?flash=mail_hold_release_failed", http.StatusSeeOther)
|
||||
return
|
||||
}
|
||||
s.logger.Printf("[INFO] MAIL-HOLD released by the operator from %s — the hub sends e-mail again; mail held until now was dropped", r.RemoteAddr)
|
||||
http.Redirect(w, r, "/configuration?flash=mail_hold_released", http.StatusSeeOther)
|
||||
}
|
||||
|
||||
func (s *Server) SetEventEmitter(f func(customerID, eventType, severity, message, detailsJSON, source string)) {
|
||||
s.emit = f
|
||||
}
|
||||
@@ -668,6 +700,12 @@ func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
} else {
|
||||
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
|
||||
}
|
||||
case path == "/configuration/mail-hold/release":
|
||||
if r.Method == http.MethodPost {
|
||||
s.handleMailHoldRelease(w, r)
|
||||
} else {
|
||||
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
|
||||
}
|
||||
case path == "/configuration/password":
|
||||
if r.Method == http.MethodPost {
|
||||
s.handleChangePassword(w, r)
|
||||
|
||||
@@ -23,6 +23,7 @@
|
||||
<a href="/configuration" class="nav-link">Configuration</a>
|
||||
</nav>
|
||||
</header>
|
||||
{{template "mail_hold_banner"}}
|
||||
|
||||
<a href="/apps{{if .Period}}?period={{.Period}}{{end}}" class="back-link">← Apps</a>
|
||||
|
||||
|
||||
@@ -21,6 +21,7 @@
|
||||
<a href="/configuration" class="nav-link">Configuration</a>
|
||||
</nav>
|
||||
</header>
|
||||
{{template "mail_hold_banner"}}
|
||||
|
||||
<h2 style="margin-bottom: 1rem;">App Telemetry</h2>
|
||||
|
||||
|
||||
@@ -22,6 +22,7 @@
|
||||
<a href="/configuration" class="nav-link">Configuration</a>
|
||||
</nav>
|
||||
</header>
|
||||
{{template "mail_hold_banner"}}
|
||||
|
||||
<a href="{{if .IsNew}}/configs{{else}}/customers/{{.Config.CustomerID}}{{end}}" class="back-link">← Back</a>
|
||||
<h2>{{if .IsNew}}Add Customer{{else}}Edit: {{.Config.CustomerID}}{{end}}</h2>
|
||||
|
||||
@@ -21,6 +21,7 @@
|
||||
<a href="/configuration" class="nav-link">Configuration</a>
|
||||
</nav>
|
||||
</header>
|
||||
{{template "mail_hold_banner"}}
|
||||
|
||||
{{if .Flash}}
|
||||
<div class="flash flash-success">
|
||||
|
||||
@@ -21,9 +21,26 @@
|
||||
<a href="/configuration" class="nav-link active">Configuration</a>
|
||||
</nav>
|
||||
</header>
|
||||
{{template "mail_hold_banner"}}
|
||||
|
||||
<h2 style="margin-bottom: 1rem;">Configuration</h2>
|
||||
|
||||
{{if mailHeld}}
|
||||
<form method="POST" action="/configuration/mail-hold/release" class="mail-hold-release">
|
||||
<input type="hidden" name="_csrf" value="{{.CSRFToken}}">
|
||||
<button type="submit" class="btn">Release the mail hold</button>
|
||||
<span class="text-muted">Removes the marker. The hub sends e-mail again from the next mail on; mail held until now is not re-sent.</span>
|
||||
</form>
|
||||
{{end}}
|
||||
{{if eq .Flash "mail_hold_released"}}
|
||||
<div class="flash flash-success">Mail hold released — the hub sends e-mail again. Mail held until now was not re-sent.</div>
|
||||
{{end}}
|
||||
{{if eq .Flash "mail_hold_release_failed"}}
|
||||
<div class="flash flash-error">The mail hold could not be released — the marker is still in place. Check the hub log.</div>
|
||||
{{end}}
|
||||
{{if eq .Flash "mail_hold_not_held"}}
|
||||
<div class="flash flash-success">The mail hold was not on — nothing to release.</div>
|
||||
{{end}}
|
||||
{{if eq .Flash "assets_refreshed"}}
|
||||
<div class="flash flash-success">Assets refreshed successfully from image seed.</div>
|
||||
{{end}}
|
||||
|
||||
@@ -42,6 +42,7 @@
|
||||
<p class="subtitle">No reports received yet</p>
|
||||
{{end}}
|
||||
</header>
|
||||
{{template "mail_hold_banner"}}
|
||||
|
||||
{{if .Flash}}
|
||||
<div class="flash flash-success">
|
||||
|
||||
@@ -22,6 +22,7 @@
|
||||
<a href="/configuration" class="nav-link">Configuration</a>
|
||||
</nav>
|
||||
</header>
|
||||
{{template "mail_hold_banner"}}
|
||||
|
||||
{{if or .OffsiteTile .PBSTile}}
|
||||
<div class="offsite-gauges">
|
||||
|
||||
@@ -21,6 +21,7 @@
|
||||
<a href="/configuration" class="nav-link">Configuration</a>
|
||||
</nav>
|
||||
</header>
|
||||
{{template "mail_hold_banner"}}
|
||||
|
||||
<a href="/hosts" class="back-link">← Hosts</a>
|
||||
|
||||
|
||||
@@ -22,6 +22,7 @@
|
||||
<a href="/configuration" class="nav-link">Configuration</a>
|
||||
</nav>
|
||||
</header>
|
||||
{{template "mail_hold_banner"}}
|
||||
|
||||
<h2 style="margin-bottom: 1rem;">Hosts</h2>
|
||||
|
||||
|
||||
@@ -21,6 +21,7 @@
|
||||
<a href="/configuration" class="nav-link">Configuration</a>
|
||||
</nav>
|
||||
</header>
|
||||
{{template "mail_hold_banner"}}
|
||||
|
||||
<a href="/customers/{{.CustomerID}}" class="back-link">← {{.CustomerID}}</a>
|
||||
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
{{define "mail_hold_banner"}}{{if mailHeld}}
|
||||
<div class="flash flash-warn" id="mail-hold" role="alert">
|
||||
<strong>E-mail is on hold</strong> — this hub sends no e-mail at all, to households or to you, while the
|
||||
<code>MAIL-HOLD</code> marker is in its data directory. Mail it would have sent is dropped, not queued.
|
||||
Release the hold on the <a href="/configuration#mail-hold">Configuration</a> page once this is the only hub
|
||||
that is running.
|
||||
</div>
|
||||
{{end}}{{end}}
|
||||
@@ -21,6 +21,7 @@
|
||||
<a href="/configuration" class="nav-link">Configuration</a>
|
||||
</nav>
|
||||
</header>
|
||||
{{template "mail_hold_banner"}}
|
||||
|
||||
<h2 style="margin-bottom: 0.75rem;">Offsite</h2>
|
||||
|
||||
|
||||
@@ -1020,3 +1020,14 @@ body.js-tabs .tab-panel:not(.tab-panel-active) { display: none; }
|
||||
@media (prefers-reduced-motion: reduce) {
|
||||
*, *::before, *::after { animation: none !important; transition: none !important; }
|
||||
}
|
||||
|
||||
/* MAIL-HOLD release (internal/mailhold): the button and its one-line explanation sit on one row under the banner. */
|
||||
.mail-hold-release {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 0.75rem;
|
||||
flex-wrap: wrap;
|
||||
margin-bottom: 1rem;
|
||||
}
|
||||
.mail-hold-release .text-muted { font-size: 0.8rem; }
|
||||
#mail-hold a { color: var(--text-1); }
|
||||
|
||||
@@ -30,6 +30,7 @@
|
||||
<a href="/configuration" class="nav-link">Configuration</a>
|
||||
</nav>
|
||||
</header>
|
||||
{{template "mail_hold_banner"}}
|
||||
|
||||
<h2 style="margin-bottom: 1rem;">System — versions and OS updates</h2>
|
||||
|
||||
|
||||
Reference in New Issue
Block a user