hub (unreleased): R-922 option A — a household's clear deletes its notification address (email_cleared); MAIL-HOLD — a restored hub sends no mail until released; two log lines drop the address; runbooks: mail hold is restore step 1; 07 §6.4 R-921 pre-check; R-921/R-922 narrowed
gates / gates (push) Successful in 5m25s

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
2026-10-09 12:37:28 +02:00
parent 2c48feb325
commit d55c590c5a
35 changed files with 861 additions and 23 deletions
+1
View File
@@ -31,6 +31,7 @@
| `severityNotifies` | hub/internal/notify/dispatcher.go (~L77) | `(severity string) bool` | Deciding whether a severity emails | warning/error/critical notify; info intentionally doesn't; anything else is logged as unrecognized (v0.24.0 fix — do not regress). Recovery mails exist DESPITE this gate (eventType branch), not through it. |
| `priorityHeaders` | hub/internal/notify/dispatcher.go (~L56) | `(severity string) map[string]string` | High-priority mail-client nudge (audit F14-light) | error/critical → `X-Priority: 1` + `Importance: high`; everything else nil — a warning/info mail must NOT masquerade as urgent (red-proofed). |
| `sendEmailFn` seam / `sendEmail` | hub/internal/notify/dispatcher.go (~L33 / ~L300) | `func(to, subject, textBody string, headers map[string]string) error` | Test seam for all sends; Resend POST | Signature grew a `headers` param in v0.71.0 — payload carries `"headers"` only when non-empty. Tests capture recipient+subject+headers through the seam. |
| `mailhold.Hold` (`Check` / `Held` / `Release`) / `(*Dispatcher).deliver` | hub/internal/mailhold/mailhold.go; hub/internal/notify/dispatcher.go | `Check(kind, who string) error` (ErrHeld when held) | THE mail gate: while `<data_dir>/MAIL-HOLD` exists the hub sends nothing | Every send goes through it — dispatcher sends ONLY via `deliver()` (pinned by `TestMailHold_EverySendPathIsGated`), `/notify` and the app-mail relay call `Check` first. Held mail is DROPPED, never queued. A new send path that skips it re-opens the restored-hub mail leak. |
| `FormatOperatorEmail` / `FormatCustomerEmail` | hub/internal/notify/templates.go (~L24 / ~L118) | `(...) (subject, body)` | Operator (English) / customer (Hungarian) email bodies | Customer messages come from the `customerMessages` map — add the Hungarian text when adding an event type. Budapest TZ via package `init()`. Operator icon is eventType-aware: `*_recovered` → ✅ (severity is the fallback). |
| `monitor.EventNotifyFunc` | hub/internal/monitor/staleness.go (~L14) | `func(customerID, eventType, severity, message, detailsJSON, source)` | Decoupling checkers from notify; wired to `dispatcher.ProcessEvent` in main | May be nil — always nil-check before calling (all checkers do). |
| `(*Store).LogNotification` | hub/internal/store/store.go (~L433) | `(customerID, eventType, severity, message, status, errorMsg, channel)` | Audit trail of every send attempt (sent/failed, per channel) | Log BOTH success and failure (dispatcher does). Since v0.71.0 these rows are also the recovery PAIRING evidence — never prune them casually. |
@@ -751,6 +751,12 @@ the apps' own stop and start (demo-hp 2026-10-05: stop 21 s, start 46 s). **Not
rule (the scratch guest has no agent connection; the demo boxes take only deliveries and read-backs) — the first
night run on the demo boxes is the proof owed (R-518). The page states about 1–1.5 minutes (an estimate from the
parts above, not a measured press).
**Check first, stop second (R-921, built 2026-10-09, ships with the next controller release).** Measured 2026-10-08 on
demo-hp: the off-site tier stopped every app for ~1 min and the agent then refused it (BUSY: the night OS step held the
agent's host-wide lock). Before stopping anything the controller now asks the agent which tiers have a job in flight
(`GET /backup/status`); another tier's job running or snapshotted → no stop, the tier stays due. A refusal after the
stop still resumes the apps at once (pinned by test). **Not covered:** the host-wide busy lock (OS step, restore test,
fstrim) — the agent serves it on no endpoint, so the measured case needs an agent change too (R-921, LEFT).
**Measured 2026-10-05 on demo-hp (9 apps, controller v0.295.0):** „Mentés most" stopped the apps at 09:19:08Z, the
local tier ran 09:19:29–09:24:09, the PBS tier was busy (the controller logged a retry in 15 min; no second stop was seen in the next 55 min), the last app was back at 09:24:55Z — the
longest stop **5 min 47 s**, for the local tier alone. The button text and its confirm (v0.296.0) give both
File diff suppressed because one or more lines are too long
@@ -113,6 +113,7 @@ A központi rendszer (`hub.felhom.eu`) a Felhom saját szerverén fut (k3s fürt
| Adat | Mire kell | Meddig marad meg |
|---|---|---|
| Ügyfél azonosítója, neve, domainje, e-mail-címe, nyelve | szerződés teljesítése, értesítések | az ügyfél törléséig <!-- source: hub/internal/store/store.go:161-171 (customer_configs: customer_id, customer_name, domain, email); :223 (language); store.go DELETE FROM customer_configs; documentation/architecture/00-capability-map.md:97 (Customer DELETE cascade) --> |
| Értesítési e-mail-cím (amelyre a háztartás a figyelmeztetéseket kéri) | értesítések | amíg a háztartás meg nem változtatja; ha a háztartás a dashboardon kitörli, a központi rendszer a következő szinkronizáláskor törli; az ügyfél törlésekor törlődik <!-- source: hub/internal/api/handler.go handleSavePreferences (R-922, email_cleared → address deleted; operator ruling 2026-10-09 option A, from the next hub + controller release); hub/internal/store/customer_delete.go:60 (customer_notifications purged at delete) --> |
| A szerver állapotjelentései (gépnév, processzor-, memória-, lemezhasználat, hőmérséklet, a telepített alkalmazások neve és állapota, mentések állapota, a dashboard nyelve) | felügyelet, hibajelzés | **90 nap** <!-- source: felhom-controller/controller/internal/report/types.go:13-56, :66-79 (report fields); hub/internal/store/store.go:1528-1540 (Prune: reports + host_reports); manifests/hub.yaml:78 (retention.max_days: 90); hub/cmd/hub/main.go:930 (default 90) --> |
| Események (pl. „mentés sikertelen", „lemez megtelt") | felügyelet, ügyfélnek látható napló | **90 nap** <!-- source: hub/internal/store/store.go:2695-2702 (PruneEvents); manifests/hub.yaml:78 --> |
| Alkalmazásonkénti erőforrás-statisztika | kapacitástervezés | **90 nap** <!-- source: hub/cmd/hub/main.go:1022 (PruneAppTelemetry 90 days) --> |
@@ -158,7 +158,7 @@ Run the unit by hand; read the snapshot on ep0 (`proxmox-backup-client snapshot
test by hand; stop the timer for a day on purpose and see `HubDBBackupStale` mail arrive (positive observable), then
start it again.
## 3. Bringing the hub back from this copy (the procedure the plan exists for) — TESTED 2026-10-05 (steps 1–3) and 2026-10-09 (steps 4–5)
## 3. Bringing the hub back from this copy (the procedure the plan exists for) — TESTED 2026-10-05 (steps 1–3) and 2026-10-09 (steps 4–5) — *steps renumbered 2026-10-09: the mail hold became step 1, so the old 1–3 are now 2–4 and the old 4–5 are 5–6*
Steps 1–3 were run on 2026-10-05 against the real copy on ep0 (`audits/hub-db-offsite-2026-10-05/partD/restore-procedure/drill.txt`):
4 hosts, **4 of 4 console passwords opened with the saved seal key, 0 of 4 with a random key**. **Steps 4–5 were run on
@@ -175,10 +175,17 @@ copy, the customer list and host list equal live (4/4, 4/4), all 4 console passw
What you need, from the break-glass sheet (`break-glass-sheet.md`; the password manager only if it survived or was restored first — `total-loss-of-dooplex.md` step 5): the seal key (`OFFSITE_SECRET_KEY`), the backup key's `data` field, and
the read-only token (or ep0 root to mint a new one: Step 2).
1. **The backup key file.** On the machine doing the restore, as root, `umask 077`, write
1. **Mail held — before the copy is ever started (from the next hub release, R-923 follow-up).** Create the empty
marker `MAIL-HOLD` in the hub's data directory (`/data/MAIL-HOLD`, in the PVC, by the helper pod of step 6) BEFORE
the restored hub first starts. While it exists the hub sends no e-mail at all (households and operator), logs each
dropped mail as `[WARN] MAIL-HOLD: not sending <kind> to <who>`, and every operator page shows a banner. Held mails
are dropped, not queued (they were decided from a snapshot). Release it on the Configuration page (or `POST
/configuration/mail-hold/release`) once the restored hub is the only hub and its pending notices are understood.
Until that release is deployed: start a test restore with no network at all.
2. **The backup key file.** On the machine doing the restore, as root, `umask 077`, write
`{"kdf": null, "created": "2026-01-01T00:00:00+00:00", "modified": "2026-01-01T00:00:00+00:00", "data": "<data>"}`
to `enc.key` (Step 0 note). A copy of DooPlex's `/etc/felhom-hub-backup/enc.key` works as is.
2. **Restore the newest copy** (from any machine that reaches ep0's PBS on 8007 — DooPlex uses the tunnel 127.0.0.1:18007):
3. **Restore the newest copy** (from any machine that reaches ep0's PBS on 8007 — DooPlex uses the tunnel 127.0.0.1:18007):
```bash
export PBS_PASSWORD_FILE=<token-restore file> PBS_FINGERPRINT=<ep0 cert fingerprint, /etc/felhom-hub-backup/env>
R='dooplex-hub@pbs!restore@<ep0>:8007:felhom-offsite'
@@ -187,14 +194,14 @@ the read-only token (or ep0 root to mint a new one: Step 2).
sqlite3 -readonly out/hub.db 'PRAGMA integrity_check' # must print: ok
```
If DooPlex's `ep0-copy` datastore survived, the same copy is there too (pulled nightly).
3. **Prove the seal key matches BEFORE putting the copy in place** — on a COPY of `out/hub.db` (the check migrates it):
4. **Prove the seal key matches BEFORE putting the copy in place** — on a COPY of `out/hub.db` (the check migrates it):
```bash
cd felhom.eu/hub && go build -o hubdb-check ./cmd/hubdb-check
printf '%s' "<OFFSITE_SECRET_KEY>" > k; chmod 600 k # from the password manager — not on a command line in a shared shell
./hubdb-check copy-of-hub.db k # want: hosts=N console_passwords_opened=N failed=0; exit 0
```
`failed>0` means the wrong seal key: the hub would start but could open no console password (`05` §16.2).
4. **A k3s with the `felhom` ArgoCD app**, and `Secret/offsite-secret-key` recreated with the SAME value:
5. **A k3s with the `felhom` ArgoCD app**, and `Secret/offsite-secret-key` recreated with the SAME value:
`kubectl -n felhom-system create secret generic offsite-secret-key --from-file=OFFSITE_SECRET_KEY=k`.
**Corrected 2026-10-09:** the Deployment also needs, NOT optional, `Secret/resend-api` (`RESEND_API_KEY`),
`Secret/report-api` (`REPORT_API_KEY`) and `Secret/gitea-creds` (`username`, `password`); without them the pod does
@@ -202,7 +209,7 @@ the read-only token (or ep0 root to mint a new one: Step 2).
off-site (`runbooks/gitea-restore.md`, `secrets/*.gpg`, opened with DooPlex's restic passphrase). A test uses dummies.
The hub's image is pulled from Gitea's registry — on a rebuild with no registry, `docker save` it from any machine
that has it, or build it from the restored code.
5. **Into the PVC:** scale `deploy/hub` to 0; put `out/hub.db` into the volume as `/data/hub.db` (a helper pod mounting
6. **Into the PVC:** scale `deploy/hub` to 0; put `out/hub.db` into the volume as `/data/hub.db` (a helper pod mounting
`hub-data`; delete any `hub.db-wal`/`-shm` there — the snapshot is a whole database); scale to 1. The start-up log
line `console passwords sealed at rest (0 legacy plaintext row(s) sealed now)` and one reveal on a host page confirm it.
As run 2026-10-09: `kubectl scale deploy/hub --replicas=0`; a `busybox` pod mounting `hub-data` at `/data`;
@@ -69,7 +69,7 @@ local restic repos; Prometheus history; DooPlex's other homelab apps (not Felhom
7. **Gitea:** `gitea-restore.md` (on a normal network this time). Then rebuild the images from the code into the new
registry: hub, controller, agent (`RUNBOOK-manual-build.md`).
8. **The hub:** k3s, then `RUNBOOK-hub-db-offsite-backup.md` §3 with S2 and S4 — **start it with mail held**
(`MAIL-HOLD`, from the next hub release; until then: no network until the pending notices are understood).
(§3 step 1: the `MAIL-HOLD` marker, from the next hub release; until then: no network until the pending notices are understood).
9. **DNS:** in Cloudflare (S11) point `hub.felhom.eu` (today a CNAME to `dooplex.hopto.org`) and `gitea.dooplex.hu` at
the new place. The boxes reconnect by themselves: their API keys are in the restored hub DB.
10. **Signing:** with S6 on paper, put the keys back and sign as before. Without it, no box accepts an agent update or a
+35
View File
@@ -1,3 +1,38 @@
## Unreleased — a household's cleared mail address is deleted (R-922) and a restored hub starts quiet (MAIL-HOLD) (2026-10-09)
Not released: ships with the next hub release. No version bump here.
- **Two log lines no longer print a household's mail address** (found while building R-922, fixed without a row): the
preferences push logs `address set=<true|false>` and the dispatcher logs `Customer email sent for <customer>/<event>`.
- **R-922 — the household clears its mail address, the hub deletes it** (operator ruling 2026-10-09 11:19, option A).
`POST /api/v1/preferences` decodes a new boolean `email_cleared`, which the controller sends true ONLY when the
household deliberately cleared the address on its dashboard. `email_cleared: true` + empty `email` → the stored
notification address is deleted (INFO `household cleared its mail address — deleted`). Every other empty push keeps
today's F12 no-clobber guard (v0.71.0) unchanged; the flag beside a non-empty address is an ordinary update. Older
controllers never send the flag, so nothing changes for them. `TestSavePreferences_DeliberateClearDeletesAddress`
(red-proved: the address stayed), `_ClearFlagWithAddressIsAnUpdate`, `_ClearFlagFalseKeepsAddress`; the F12 test
re-proved red. The operator-registered address in `customer_configs.email` is a separate copy and is NOT touched
(the kernel notice, claim and self-bind mails read it). `scripts/wire_contract_gate.py` gains the root
`controller -> hub (POST /preferences)` (seen convicting a planted controller `email_cleared` against a hub without
it). Privacy-notice draft: one retention row for the notification address.
- **MAIL-HOLD — a restored hub starts quiet.** Evidence: the 2026-10-09 restore drill (`documentation/audits/
dooplex-survival-2026-10-09/partD-05-checks.txt`) — a hub started on a restored DB mailed households pending kernel
notices within a minute; `notifications.operator_enabled: false` did not stop household mail. New package
`internal/mailhold`: while `<data_dir>/MAIL-HOLD` exists the hub sends **no e-mail on any path** — the dispatcher
(household + operator events, recovery, test mail, kernel notice, claim codes, self-bind link; all through ONE
`deliver()` gate), the legacy `/api/v1/notify`, and the app-mail relay `/api/v1/mail` (503 while held). Each held mail
is logged once, `[WARN] MAIL-HOLD: not sending <kind> to <customer|operator>` (no address, no body), and recorded
`held` in the notification log. **Held mails are dropped, not re-sent** — a restored copy's pending mails were decided
from a snapshot of the past; the cooldown a dropped mail armed is given back, so the first real alarm after the
release is not silenced. A held kernel notice returns an error, so the kernel step does not run (no mail, no step).
An unreadable marker state counts as held (fail-closed). The operator lifts it with `POST
/configuration/mail-hold/release` (operator auth + the R-135 CSRF gate; a button on Configuration); every operator page
shows an amber banner while held. Tests: `TestMailHold_*` (notify, api, web), `TestHold_*`,
`TestMailHoldBanner_*` (both branches), `TestMailHoldRelease_*`; the route joined `r135PostRoutes`. Red-proved: gate
skipped → 7 mails sent; cooldown not given back → 0 mails after the release; banner func false → no banner; release
without `Release()` → marker stays; CSRF gate open → a Basic-only POST released the hold.
## v0.144.0 — the operator's decision sheet (D1, D2, D3, D6, D7) and the morning's R-243, R-901, R-304, R-415 (2026-10-09)
Release together with controller 0.304.0 (D1's operator actions need both). No wire change an older box cannot read.
+14
View File
@@ -21,6 +21,7 @@ import (
"gitea.dooplex.hu/admin/felhom-hub/internal/gitea"
"gitea.dooplex.hu/admin/felhom-hub/internal/hetznerapi"
"gitea.dooplex.hu/admin/felhom-hub/internal/intent"
"gitea.dooplex.hu/admin/felhom-hub/internal/mailhold"
"gitea.dooplex.hu/admin/felhom-hub/internal/mailrelay"
"gitea.dooplex.hu/admin/felhom-hub/internal/monitor"
"gitea.dooplex.hu/admin/felhom-hub/internal/notify"
@@ -319,6 +320,17 @@ func main() {
)
apiHandler.SetDispatcher(dispatcher)
// MAIL-HOLD (internal/mailhold): while <data_dir>/MAIL-HOLD exists the hub sends NO e-mail on any path —
// dispatcher, /notify, app-mail relay. A hub started on a restored database mailed households within a minute
// (2026-10-09 restore drill); the marker makes a restored copy start quiet. Lifted by the operator with
// POST /configuration/mail-hold/release; held mails are dropped, not re-sent.
mailHold := mailhold.New(cfg.Server.DataDir, logger)
if mailHold.Held() {
logger.Printf("[WARN] MAIL-HOLD: marker present at start (%s) — the hub sends no e-mail until the operator releases it on the Configuration page", mailHold.Path)
}
dispatcher.SetMailHold(mailHold)
apiHandler.SetMailHold(mailHold)
// Customer-claim password arc (v0.50.0, F-4): the code engine — the dispatcher delivers the
// Hungarian emails, the store holds bcrypt(code) only. Wired into the API (Day-0 issue at
// config retrieve, live-box issue + claimed ingest + ACK on report, reset endpoint) and the
@@ -341,6 +353,8 @@ func main() {
webServer := web.New(dataStore, cfg.Auth.PasswordHash, cfg.API.ReportAPIKey, Version, staleThreshold, logger)
webServer.SetTemplateFetcher(templateFetcher)
webServer.SetAssetManager(assetsMgr)
// MAIL-HOLD banner on every operator page + the release button on Configuration.
webServer.SetMailHold(mailHold)
webServer.SetClaimEngine(claimEngine) // v0.50.0 — Setup-tab claim chip + resend button
webServer.SetSelfBindMailer(dispatcher) // v0.66.0 (R-27) — customer self-bind link button (sibling of claim mailer)
webServer.SetEventEmitter(dispatcher.ProcessEvent) // v0.135.0 (R-604) — the "floor raise skipped boxes" mail
+33 -3
View File
@@ -20,6 +20,7 @@ import (
"gitea.dooplex.hu/admin/felhom-hub/internal/claim"
"gitea.dooplex.hu/admin/felhom-hub/internal/configgen"
"gitea.dooplex.hu/admin/felhom-hub/internal/intent"
"gitea.dooplex.hu/admin/felhom-hub/internal/mailhold"
"gitea.dooplex.hu/admin/felhom-hub/internal/mailrelay"
"gitea.dooplex.hu/admin/felhom-hub/internal/notify"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
@@ -67,7 +68,9 @@ type Handler struct {
floorNotes sync.Map
// App-email passthrough (POST /api/v1/mail). nil sender = endpoint returns 503.
mailSender mailrelay.Sender
mailSender mailrelay.Sender
// mailHold is the hub-wide mail gate (MAIL-HOLD marker; internal/mailhold). nil never holds.
mailHold *mailhold.Hold
mailLimiter *mailRateLimiter
mailFromAllow map[string]bool
@@ -156,6 +159,12 @@ func New(store *store.Store, apiKey, resendAPIKey, fromEmail string, templatePro
}
}
// SetMailHold wires the hub-wide mail gate: while the MAIL-HOLD marker exists, /notify and the app-mail relay send
// nothing (internal/mailhold).
func (h *Handler) SetMailHold(m *mailhold.Hold) {
h.mailHold = m
}
// SetDispatcher sets the notification dispatcher for event-triggered emails.
func (h *Handler) SetDispatcher(d *notify.Dispatcher) {
h.dispatcher = d
@@ -2583,6 +2592,14 @@ func (h *Handler) handleNotify(w http.ResponseWriter, r *http.Request) {
return
}
// MAIL-HOLD (internal/mailhold): a restored or quarantined hub sends nothing. The mail is dropped, not queued.
if err := h.mailHold.Check("customer event "+payload.EventType, payload.CustomerID); err != nil {
h.store.LogNotification(payload.CustomerID, payload.EventType, payload.Severity, payload.Message, "held", "MAIL-HOLD marker present — dropped", "customer")
w.WriteHeader(http.StatusOK)
w.Write([]byte(`{"status":"ok","sent":false,"reason":"mail_hold"}`))
return
}
subject, emailBody := formatNotificationEmail(payload.CustomerID, payload.EventType, payload.Severity, payload.Message, payload.Details)
sendErr := h.sendResendEmail(prefs.Email, subject, emailBody)
if sendErr != nil {
@@ -2617,6 +2634,10 @@ func (h *Handler) handleSavePreferences(w http.ResponseWriter, r *http.Request)
Email string `json:"email"`
EnabledEvents []string `json:"enabled_events"`
CooldownHours int `json:"cooldown_hours"`
// EmailCleared (R-922, operator ruling 2026-10-09 option A) is sent true by the controller ONLY when the
// household deliberately cleared its address on the dashboard. Omitted (false) on every other push —
// including every push from a controller older than the one that emits it.
EmailCleared bool `json:"email_cleared"`
}
if err := json.Unmarshal(body, &payload); err != nil || payload.CustomerID == "" {
http.Error(w, "Invalid payload: customer_id required", http.StatusBadRequest)
@@ -2627,8 +2648,17 @@ func (h *Handler) handleSavePreferences(w http.ResponseWriter, r *http.Request)
// (e.g. an unconfigured box) must never wipe a stored non-empty address — the seeded/edited
// email is the customer's alert lifeline. Events + cooldown from the push still apply; a push
// with a non-empty email updates everything (customer edits keep working).
//
// R-922: the ONE exception is a deliberate clear — `email_cleared: true` with an empty address. The household
// removed its address, so the hub deletes it (personal data the household withdrew does not stay on our side).
// The guard cannot tell a clear from an unconfigured box by the address alone, which is why the clear travels
// as its own flag. A flag beside a NON-empty address is an ordinary update (the address wins). Pinned by
// TestSavePreferences_DeliberateClearDeletesAddress, TestSavePreferences_EmptyEmailCannotClobber and
// TestSavePreferences_ClearFlagWithAddressIsAnUpdate.
saveEmail := payload.Email
if saveEmail == "" {
if saveEmail == "" && payload.EmailCleared {
h.logger.Printf("[INFO] Notification prefs push for %s: household cleared its mail address — deleted", payload.CustomerID)
} else if saveEmail == "" {
if existing, err := h.store.GetNotificationPrefs(payload.CustomerID); err == nil && existing != nil && existing.Email != "" {
saveEmail = existing.Email
h.logger.Printf("[INFO] Notification prefs push for %s had empty email — preserving stored address", payload.CustomerID)
@@ -2641,7 +2671,7 @@ func (h *Handler) handleSavePreferences(w http.ResponseWriter, r *http.Request)
return
}
h.logger.Printf("[INFO] Notification preferences updated for %s: email=%s, events=%v", payload.CustomerID, saveEmail, payload.EnabledEvents)
h.logger.Printf("[INFO] Notification preferences updated for %s: address set=%t, events=%v", payload.CustomerID, saveEmail != "", payload.EnabledEvents) // never the address (R-922)
w.WriteHeader(http.StatusOK)
w.Write([]byte(`{"status":"ok"}`))
}
+11
View File
@@ -144,6 +144,17 @@ func (h *Handler) handleMail(w http.ResponseWriter, r *http.Request) {
return
}
// MAIL-HOLD (internal/mailhold): the relay sends nothing while the hub holds mail. 503 — a temporary refusal the
// box's shim surfaces to the app; the hub keeps no copy (held mail is dropped, never queued).
holdWho := custID
if holdWho == "" {
holdWho = "a global-key caller"
}
if err := h.mailHold.Check("app mail", holdWho); err != nil {
http.Error(w, "Mail is on hold at the hub", http.StatusServiceUnavailable)
return
}
ctx, cancel := context.WithTimeout(r.Context(), 30*time.Second)
defer cancel()
if err := h.mailSender.Send(ctx, req.RawMIME, req.MailFrom, req.RcptTo); err != nil {
+87
View File
@@ -0,0 +1,87 @@
package api
import (
"io"
"net/http"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"gitea.dooplex.hu/admin/felhom-hub/internal/mailhold"
)
// MAIL-HOLD on the two API send paths (the dispatcher's paths: internal/notify/mailhold_test.go).
// COMPANION RED-PROOF (REPORT): remove the h.mailHold.Check block from handleNotify / handleMail → these fail with the
// mail handed to Resend (the recorded transport / the fake SMTP sender).
func heldHold(t *testing.T) *mailhold.Hold {
t.Helper()
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(dir, mailhold.FileName), nil, 0o644); err != nil {
t.Fatal(err)
}
return mailhold.New(dir, nil)
}
// recordingTransport stands in for api.resend.com: it records each request and never leaves the process.
type recordingTransport struct {
mu sync.Mutex
calls int
}
func (rt *recordingTransport) RoundTrip(r *http.Request) (*http.Response, error) {
rt.mu.Lock()
rt.calls++
rt.mu.Unlock()
return &http.Response{StatusCode: 200, Body: io.NopCloser(strings.NewReader(`{"id":"x"}`)), Header: http.Header{}}, nil
}
func TestMailHold_NotifySendsNothingWhileHeld(t *testing.T) {
h, st := newEventTestHandler(t)
rt := &recordingTransport{}
h.httpClient = &http.Client{Transport: rt}
h.resendAPIKey = "test-key"
if err := st.SaveNotificationPrefs("c1", "household@example.hu", []string{"backup_failed"}, 6); err != nil {
t.Fatal(err)
}
h.SetMailHold(heldHold(t))
rr := do(h, http.MethodPost, "/notify", "ckey", `{"customer_id":"c1","event_type":"backup_failed","severity":"error","message":"m"}`)
if rr.Code != http.StatusOK {
t.Fatalf("status = %d, body=%s", rr.Code, rr.Body.String())
}
if rt.calls != 0 {
t.Fatalf("MAIL-HOLD present but /notify handed %d mail(s) to Resend", rt.calls)
}
if !strings.Contains(rr.Body.String(), `"reason":"mail_hold"`) {
t.Fatalf("the answer must say the mail was held: %s", rr.Body.String())
}
}
func TestMailHold_AppMailRelayRefusesWhileHeld(t *testing.T) {
h, st, _ := newTestHandler(t)
withCustomer(t, st, "c1", "ckey")
fake := &fakeSender{}
h.SetMailRelay(fake, 30, []string{"felhom.eu"})
hold := heldHold(t)
h.SetMailHold(hold)
rr := do(h, "POST", "/mail", "ckey", mailBody(t, "vaultwarden@felhom.eu"))
if fake.callCount() != 0 {
t.Fatalf("MAIL-HOLD present but the relay sent %d mail(s)", fake.callCount())
}
if rr.Code != http.StatusServiceUnavailable {
t.Fatalf("held relay: status %d, want 503", rr.Code)
}
// Released: the relay sends again.
if err := hold.Release(); err != nil {
t.Fatal(err)
}
rr = do(h, "POST", "/mail", "ckey", mailBody(t, "vaultwarden@felhom.eu"))
if rr.Code != http.StatusOK || fake.callCount() != 1 {
t.Fatalf("after release: status %d, sends %d — want 200 and 1", rr.Code, fake.callCount())
}
}
@@ -65,3 +65,64 @@ func TestSavePreferences_EmptyEmailNoStoredRow(t *testing.T) {
t.Fatalf("all-off push must store the empty row unchanged, got %+v", prefs)
}
}
// TestSavePreferences_DeliberateClearDeletesAddress (R-922, operator ruling 2026-10-09 option A): when the household
// clears its address on the dashboard, the controller pushes `email_cleared: true` with an empty address, and the
// hub DELETES the stored address. Seen on Tester 1 before the fix: the push answered 200 and the old address stayed.
// Red-proof: with the `payload.EmailCleared` branch removed, this fails with email="seeded@example.hu".
func TestSavePreferences_DeliberateClearDeletesAddress(t *testing.T) {
h, st := newEventTestHandler(t)
if err := st.SaveNotificationPrefs("c1", "seeded@example.hu", []string{"node_down"}, 6); err != nil {
t.Fatalf("stored prefs: %v", err)
}
rr := do(h, http.MethodPost, "/preferences", "ckey",
`{"customer_id":"c1","email":"","email_cleared":true,"enabled_events":[],"cooldown_hours":6}`)
if rr.Code != http.StatusOK {
t.Fatalf("status = %d, body=%s", rr.Code, rr.Body.String())
}
prefs, err := st.GetNotificationPrefs("c1")
if err != nil || prefs == nil {
t.Fatalf("prefs: %+v err=%v", prefs, err)
}
if prefs.Email != "" {
t.Fatalf("a deliberate clear must delete the stored address, still stored: email=%q", prefs.Email)
}
}
// TestSavePreferences_ClearFlagWithAddressIsAnUpdate (R-922): `email_cleared: true` beside a NON-empty address is an
// ordinary update — the address in the push wins, it is never deleted.
func TestSavePreferences_ClearFlagWithAddressIsAnUpdate(t *testing.T) {
h, st := newEventTestHandler(t)
if err := st.SaveNotificationPrefs("c1", "old@example.hu", []string{"node_down"}, 6); err != nil {
t.Fatalf("stored prefs: %v", err)
}
rr := do(h, http.MethodPost, "/preferences", "ckey",
`{"customer_id":"c1","email":"new@example.hu","email_cleared":true,"enabled_events":["node_down"],"cooldown_hours":6}`)
if rr.Code != http.StatusOK {
t.Fatalf("status = %d, body=%s", rr.Code, rr.Body.String())
}
prefs, _ := st.GetNotificationPrefs("c1")
if prefs == nil || prefs.Email != "new@example.hu" {
t.Fatalf("flag + non-empty address must store the pushed address, got %+v", prefs)
}
}
// TestSavePreferences_ClearFlagFalseKeepsAddress (R-922): an explicit `email_cleared: false` with an empty address is
// NOT a clear — the F12 no-clobber guard still holds.
func TestSavePreferences_ClearFlagFalseKeepsAddress(t *testing.T) {
h, st := newEventTestHandler(t)
if err := st.SaveNotificationPrefs("c1", "seeded@example.hu", []string{"node_down"}, 6); err != nil {
t.Fatalf("stored prefs: %v", err)
}
rr := do(h, http.MethodPost, "/preferences", "ckey",
`{"customer_id":"c1","email":"","email_cleared":false,"enabled_events":["node_down"],"cooldown_hours":6}`)
if rr.Code != http.StatusOK {
t.Fatalf("status = %d, body=%s", rr.Code, rr.Body.String())
}
prefs, _ := st.GetNotificationPrefs("c1")
if prefs == nil || prefs.Email != "seeded@example.hu" {
t.Fatalf("email_cleared:false with an empty address must keep the stored address, got %+v", prefs)
}
}
+88
View File
@@ -0,0 +1,88 @@
// Package mailhold is the hub's one mail gate: while the marker file `<data_dir>/MAIL-HOLD` exists, the hub sends
// NO e-mail at all — household or operator, on every path that reaches Resend (the dispatcher's HTTP-API sends, the
// legacy /notify endpoint, the app-mail SMTP relay).
//
// WHY. The 2026-10-09 restore drill (documentation/audits/dooplex-survival-2026-10-09/partD-05-checks.txt) started a
// hub on a restored database, and within a minute it mailed households the pending "kernel notice" mails.
// `notifications.operator_enabled: false` did not stop it, because that switch covers the operator channel only. A
// restored copy is a second hub speaking with the first one's voice; it must start quiet until the operator says so.
// The marker is created (an empty file is enough) before a restored copy's first start.
//
// HELD MAILS ARE DROPPED, NOT QUEUED, and that is deliberate: every mail a restored copy wants to send was decided
// from a snapshot of the past — a kernel notice for a step the real hub already ran or will run, an alarm about a
// state that has since changed. Re-sending them after the hold is lifted would deliver stale news as if it were
// current. After the release, the hub's checkers and boxes produce fresh mails from live state on their own. Each
// dropped mail is logged once (`[WARN] MAIL-HOLD: not sending <kind> to <who>` — never the address, never the body).
//
// The marker is read on every send (one stat of a file in the data dir; no wait on a device the hub does not
// already depend on for its database). A stat error other than "does not exist" counts as HELD: the gate fails
// closed, and the operator pages show the banner, so the state is never invisible.
package mailhold
import (
"errors"
"log"
"os"
"path/filepath"
)
// FileName is the marker's name inside the hub's data directory.
const FileName = "MAIL-HOLD"
// ErrHeld is returned by Check for a mail the hold stopped. Callers treat it as "not sent" (never as sent).
var ErrHeld = errors.New("mail hold: the hub is not sending e-mail (MAIL-HOLD marker present)")
// Hold reads and removes the marker. The zero value and a nil *Hold never hold (tests and wiring without a data dir).
type Hold struct {
Path string
Logger *log.Logger
}
// New returns the hold for a data directory.
func New(dataDir string, logger *log.Logger) *Hold {
return &Hold{Path: filepath.Join(dataDir, FileName), Logger: logger}
}
// Held reports whether the marker is present (fail-closed on an unreadable state).
func (h *Hold) Held() bool {
if h == nil || h.Path == "" {
return false
}
_, err := os.Stat(h.Path)
if err == nil {
return true
}
if errors.Is(err, os.ErrNotExist) {
return false
}
h.logf("[WARN] MAIL-HOLD: cannot read the marker state (%v) — treating mail as HELD", err)
return true
}
// Check is the gate every sender calls immediately before handing a mail to Resend. When held it logs the one WARN
// line for this mail and returns ErrHeld; otherwise nil. kind names the mail ("customer event backup_failed",
// "kernel notice", …); who is a customer ID or "operator" — never an address.
func (h *Hold) Check(kind, who string) error {
if !h.Held() {
return nil
}
h.logf("[WARN] MAIL-HOLD: not sending %s to %s", kind, who)
return ErrHeld
}
// Release removes the marker. Removing a marker that is not there is not an error (the release is idempotent).
func (h *Hold) Release() error {
if h == nil || h.Path == "" {
return nil
}
if err := os.Remove(h.Path); err != nil && !errors.Is(err, os.ErrNotExist) {
return err
}
return nil
}
func (h *Hold) logf(format string, args ...interface{}) {
if h != nil && h.Logger != nil {
h.Logger.Printf(format, args...)
}
}
+59
View File
@@ -0,0 +1,59 @@
package mailhold
import (
"bytes"
"errors"
"log"
"os"
"path/filepath"
"strings"
"testing"
)
func TestHold_MarkerDecidesAndReleaseRemovesIt(t *testing.T) {
dir := t.TempDir()
buf := &bytes.Buffer{}
h := New(dir, log.New(buf, "", 0))
if h.Held() {
t.Fatal("no marker: must not hold")
}
if err := h.Check("kernel notice", "c1"); err != nil {
t.Fatalf("no marker: Check = %v", err)
}
if err := os.WriteFile(filepath.Join(dir, FileName), nil, 0o644); err != nil {
t.Fatal(err)
}
if err := h.Check("kernel notice", "c1"); !errors.Is(err, ErrHeld) {
t.Fatalf("marker present: Check = %v, want ErrHeld", err)
}
if got := buf.String(); !strings.Contains(got, "[WARN] MAIL-HOLD: not sending kernel notice to c1") {
t.Fatalf("held mail log line missing: %q", got)
}
if err := h.Release(); err != nil {
t.Fatal(err)
}
if _, err := os.Stat(filepath.Join(dir, FileName)); !os.IsNotExist(err) {
t.Fatalf("Release left the marker: %v", err)
}
if err := h.Release(); err != nil {
t.Fatalf("a second Release must be a no-op, got %v", err)
}
}
// Fail-closed: a marker state that cannot be read (here: the data dir is a FILE, so stat says ENOTDIR) holds mail.
func TestHold_UnreadableStateHolds(t *testing.T) {
f := filepath.Join(t.TempDir(), "not-a-dir")
if err := os.WriteFile(f, nil, 0o644); err != nil {
t.Fatal(err)
}
if !New(f, nil).Held() {
t.Fatal("an unreadable marker state must count as HELD")
}
}
func TestHold_NilNeverHolds(t *testing.T) {
var h *Hold
if h.Held() || h.Check("x", "y") != nil || h.Release() != nil {
t.Fatal("a nil hold must never hold and never fail")
}
}
+79 -10
View File
@@ -3,6 +3,7 @@ package notify
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"io"
"log"
@@ -11,6 +12,7 @@ import (
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/mailhold"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
"gitea.dooplex.hu/admin/felhom-hub/internal/i18n"
@@ -40,8 +42,32 @@ type Dispatcher struct {
// afterFn schedules a delayed call (time.AfterFunc; a seam so tests run the retries at once). Used by
// retryOperatorEmail (the lost alarm of 2026-10-05).
afterFn func(time.Duration, func())
// mailHold is the hub-wide mail gate (internal/mailhold): while its marker exists, deliver() hands nothing to
// sendEmailFn. nil never holds.
mailHold *mailhold.Hold
}
// SetMailHold wires the hub-wide mail gate (the MAIL-HOLD marker in the data dir).
func (d *Dispatcher) SetMailHold(m *mailhold.Hold) {
d.mailHold = m
}
// deliver is the ONE way the dispatcher sends a mail: the MAIL-HOLD gate, then sendEmailFn. Every send in this file
// goes through it, so the hold cannot be bypassed by a path someone adds later without calling sendEmailFn directly
// (pinned by TestMailHold_EverySendPathIsGated). kind names the mail; who is a customer ID or "operator" — the hold's
// log line carries those two and never the address. A held mail returns mailhold.ErrHeld and is DROPPED, never
// queued: see the package comment for why.
func (d *Dispatcher) deliver(kind, who, to, subject, body string, headers map[string]string) error {
if err := d.mailHold.Check(kind, who); err != nil {
return err
}
return d.sendEmailFn(to, subject, body, headers)
}
// isHeld reports whether a send error is the mail hold (not a delivery failure).
func isHeld(err error) bool { return errors.Is(err, mailhold.ErrHeld) }
// operatorRetryDelays are the waits before each further try of an operator mail whose send FAILED (the lost alarm of 2026-10-05). Measured
// 2026-10-05: demo-hp's whole_guest_backup_failed (error) hit a 10 s Resend client timeout, was logged `failed`, and was
// never sent — the only operator mail of 692 that failed, and the one alarm the operator needed that night. The
@@ -77,7 +103,12 @@ func (d *Dispatcher) retryOperatorEmail(customerID, eventType, severity, message
}
d.afterFn(operatorRetryDelays[attempt], func() {
n := attempt + 1
if err := d.sendEmailFn(d.operatorEmail, subject, body, headers); err != nil {
if err := d.deliver("operator mail "+eventType+" (retry)", "operator", d.operatorEmail, subject, body, headers); err != nil {
if isHeld(err) {
// The hold began between tries: the mail is dropped like every other held mail, and the retries stop.
d.store.LogNotification(customerID, eventType, severity, message, "held", "MAIL-HOLD marker present — dropped", "operator")
return
}
d.logger.Printf("[ERROR] Operator email retry %d failed for %s/%s: %v", n, customerID, eventType, err)
d.store.LogNotification(customerID, eventType, severity, message, "failed", fmt.Sprintf("retry %d: %v", n, err), "operator")
d.retryOperatorEmail(customerID, eventType, severity, message, subject, body, headers, n)
@@ -246,7 +277,9 @@ func (d *Dispatcher) sendTestEmail(customerID string) {
subject := b.Msg(lang, "mail.test.subject")
body := b.Msg(lang, "mail.test.body")
if err := d.sendEmailFn(prefs.Email, subject, body, nil); err != nil {
if err := d.deliver("test mail", customerID, prefs.Email, subject, body, nil); isHeld(err) {
d.store.LogNotification(customerID, "test", "info", "Teszt értesítés", "held", "MAIL-HOLD marker present — dropped", "customer")
} else if err != nil {
d.logger.Printf("[ERROR] Test email to %s failed: %v", prefs.Email, err)
d.store.LogNotification(customerID, "test", "info", "Teszt értesítés", "failed", err.Error(), "customer")
} else {
@@ -268,7 +301,10 @@ headers render correctly. The customer test mail result is recorded in the
notification log.
Dashboard: https://hub.felhom.eu/customers/%s`, customerID, customerID)
if err := d.sendEmailFn(d.operatorEmail, opSubject, opBody, priorityHeaders("critical")); err != nil {
if err := d.deliver("operator test mail", "operator", d.operatorEmail, opSubject, opBody, priorityHeaders("critical")); isHeld(err) {
d.store.LogNotification(customerID, "test", "info", "operator test copy", "held", "MAIL-HOLD marker present — dropped", "operator")
return
} else if err != nil {
d.logger.Printf("[ERROR] Operator test email failed for %s: %v", customerID, err)
d.store.LogNotification(customerID, "test", "info", "operator test copy", "failed", err.Error(), "operator")
return
@@ -330,7 +366,11 @@ func (d *Dispatcher) processRecovery(customerID, eventType, severity, message, m
subject, body := FormatCustomerEmail(d.store.CustomerLanguage(customerID),
customerID, eventType, severity, message, messageCustomer, detailsJSON)
if err := d.sendEmailFn(prefs.Email, subject, body, priorityHeaders(severity)); err != nil {
if err := d.deliver("customer recovery mail "+eventType, customerID, prefs.Email, subject, body, priorityHeaders(severity)); isHeld(err) {
d.forgetCustomerCooldown(cooldownKey)
d.store.LogNotification(customerID, eventType, severity, message, "held", "MAIL-HOLD marker present — dropped", "customer")
return
} else if err != nil {
d.logger.Printf("[ERROR] Customer recovery email failed for %s/%s: %v", customerID, eventType, err)
d.store.LogNotification(customerID, eventType, severity, message, "failed", err.Error(), "customer")
return
@@ -607,7 +647,15 @@ func (d *Dispatcher) processOperator(customerID, eventType, severity, message, d
subject, body := FormatOperatorEmail(customerID, eventType, severity, message, detailsJSON)
hdrs := priorityHeaders(severity)
if err := d.sendEmailFn(d.operatorEmail, subject, body, hdrs); err != nil {
if err := d.deliver("operator mail "+eventType, "operator", d.operatorEmail, subject, body, hdrs); isHeld(err) {
// Dropped, and the cooldown it armed is given back: the first real alarm after the release must not be
// silenced by a mail that was never sent.
d.mu.Lock()
delete(d.opCooldowns, cooldownKey)
d.mu.Unlock()
d.store.LogNotification(customerID, eventType, severity, message, "held", "MAIL-HOLD marker present — dropped", "operator")
return
} else if err != nil {
d.logger.Printf("[ERROR] Operator email failed for %s/%s: %v — retrying in %v", customerID, eventType, err, operatorRetryDelays[0])
d.store.LogNotification(customerID, eventType, severity, message, "failed", err.Error(), "operator")
d.retryOperatorEmail(customerID, eventType, severity, message, subject, body, hdrs, 0)
@@ -883,12 +931,16 @@ func (d *Dispatcher) processCustomer(customerID, eventType, severity, message, m
subject, body := FormatCustomerEmail(d.store.CustomerLanguage(customerID),
customerID, eventType, severity, message, messageCustomer, detailsJSON)
if err := d.sendEmailFn(prefs.Email, subject, body, priorityHeaders(severity)); err != nil {
if err := d.deliver("customer mail "+eventType, customerID, prefs.Email, subject, body, priorityHeaders(severity)); isHeld(err) {
d.forgetCustomerCooldown(cooldownKey)
d.store.LogNotification(customerID, eventType, severity, message, "held", "MAIL-HOLD marker present — dropped", "customer")
return
} else if err != nil {
d.logger.Printf("[ERROR] Customer email failed for %s/%s: %v", customerID, eventType, err)
d.store.LogNotification(customerID, eventType, severity, message, "failed", err.Error(), "customer")
return
}
d.logger.Printf("[INFO] Customer email sent to %s for %s/%s", prefs.Email, customerID, eventType)
d.logger.Printf("[INFO] Customer email sent for %s/%s", customerID, eventType) // never the address (R-922)
d.store.LogNotification(customerID, eventType, severity, message, "sent", "", "customer")
}
@@ -944,6 +996,13 @@ func (d *Dispatcher) replyToFor(to string) string {
return op
}
// forgetCustomerCooldown gives back a customer cooldown armed for a mail the hold dropped.
func (d *Dispatcher) forgetCustomerCooldown(key string) {
d.mu.Lock()
delete(d.custCooldowns, key)
d.mu.Unlock()
}
func isEventEnabled(enabledEvents []string, eventType string) bool {
for _, e := range enabledEvents {
if e == eventType {
@@ -964,7 +1023,10 @@ func (d *Dispatcher) SendClaimEmail(kind, customerID, email, domain, code string
}
subject, body := FormatClaimEmail(d.store.CustomerLanguage(customerID), kind, customerID, domain, code)
eventType := "claim_" + kind
if err := d.sendEmailFn(email, subject, body, nil); err != nil {
if err := d.deliver("claim "+kind+" mail", customerID, email, subject, body, nil); isHeld(err) {
d.store.LogNotification(customerID, eventType, "info", subject, "held", "MAIL-HOLD marker present — dropped", "customer")
return err
} else if err != nil {
d.logger.Printf("[ERROR] claim %s email to customer %s failed: %v", kind, customerID, err)
d.store.LogNotification(customerID, eventType, "info", subject, "failed", err.Error(), "customer")
return err
@@ -985,7 +1047,10 @@ func (d *Dispatcher) SendSelfBindEmail(customerID, email, link string) error {
return fmt.Errorf("notify: no resend api key")
}
subject, body := FormatSelfBindEmail(d.store.CustomerLanguage(customerID), customerID, link)
if err := d.sendEmailFn(email, subject, body, nil); err != nil {
if err := d.deliver("self-bind link mail", customerID, email, subject, body, nil); isHeld(err) {
d.store.LogNotification(customerID, "selfbind_link", "info", subject, "held", "MAIL-HOLD marker present — dropped", "customer")
return err
} else if err != nil {
d.logger.Printf("[ERROR] self-bind link email to customer %s failed: %v", customerID, err)
d.store.LogNotification(customerID, "selfbind_link", "info", subject, "failed", err.Error(), "customer")
return err
@@ -1018,7 +1083,11 @@ func (d *Dispatcher) SendKernelNotice(customerID, kver string) (string, error) {
}
lang := d.store.CustomerLanguage(customerID)
subject, body := FormatKernelNoticeEmail(lang)
if err := d.sendEmailFn(to, subject, body, nil); err != nil {
if err := d.deliver("kernel notice", customerID, to, subject, body, nil); isHeld(err) {
// The caller runs no kernel step on an error (no mail, no step) — exactly right for a held hub.
d.store.LogNotification(customerID, "kernel_notice", "info", subject, "held", "MAIL-HOLD marker present — dropped", "customer")
return lang, err
} else if err != nil {
d.logger.Printf("[ERROR] kernel notice (%s) to customer %s failed: %v", kver, customerID, err)
d.store.LogNotification(customerID, "kernel_notice", "info", subject, "failed", err.Error(), "customer")
return lang, err
+143
View File
@@ -0,0 +1,143 @@
package notify
import (
"bytes"
"log"
"os"
"path/filepath"
"regexp"
"strings"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/mailhold"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// MAIL-HOLD — a restored hub starts quiet. Evidence: the 2026-10-09 restore drill
// (documentation/audits/dooplex-survival-2026-10-09/partD-05-checks.txt): a hub started on a restored database mailed
// households pending kernel notices within a minute, and `notifications.operator_enabled: false` did not stop it.
//
// COMPANION RED-PROOF (REPORT): make deliver() skip the mailHold.Check call (the pre-fix shape: sendEmailFn is reached
// directly) → TestMailHold_DispatcherSendsNothingWhileHeld fails with every path's mail SENT.
type holdRecorder struct {
mu sync.Mutex
sent []string // subjects
}
func (r *holdRecorder) fn(to, subject, body string, headers map[string]string) error {
r.mu.Lock()
defer r.mu.Unlock()
r.sent = append(r.sent, subject)
return nil
}
func (r *holdRecorder) count() int {
r.mu.Lock()
defer r.mu.Unlock()
return len(r.sent)
}
// holdDispatcher returns a dispatcher whose household c1 has a registered address, notification prefs with
// backup_failed on, the operator channel ON, the MAIL-HOLD marker present, and a log captured in buf.
func holdDispatcher(t *testing.T) (*Dispatcher, *store.Store, *holdRecorder, *mailhold.Hold, *bytes.Buffer) {
t.Helper()
st := newDispStore(t)
if err := st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c1", APIKey: "k", RetrievalPassword: "p", Email: "household@example.hu"}); err != nil {
t.Fatal(err)
}
if err := st.SaveNotificationPrefs("c1", "household@example.hu", []string{"backup_failed"}, 6); err != nil {
t.Fatal(err)
}
buf := &bytes.Buffer{}
logger := log.New(buf, "", 0)
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, logger)
rec := &holdRecorder{}
d.sendEmailFn = rec.fn
d.afterFn = func(time.Duration, func()) {} // a retry must never fire in these tests
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(dir, mailhold.FileName), nil, 0o644); err != nil {
t.Fatal(err)
}
hold := mailhold.New(dir, logger)
d.SetMailHold(hold)
return d, st, rec, hold, buf
}
// The consequence asserted: with the marker present, NOT ONE mail reaches the sender, on any dispatcher path —
// household event, operator event, recovery, test mail, kernel notice, claim code, self-bind link.
func TestMailHold_DispatcherSendsNothingWhileHeld(t *testing.T) {
d, st, rec, _, buf := holdDispatcher(t)
d.ProcessEvent("c1", "backup_failed", "error", "Backup failed", "", "box") // operator + household
d.ProcessEvent("c1", "test", "info", "test", "", "operator") // test mail, both channels
if _, err := d.SendKernelNotice("c1", "6.8.12-1"); err == nil {
t.Errorf("kernel notice: a held mail must report an error (the caller then runs no kernel step)")
}
if err := d.SendClaimEmail("claim", "c1", "household@example.hu", "c1.example", "CODE-1234"); err == nil {
t.Errorf("claim mail: a held mail must report an error")
}
if err := d.SendSelfBindEmail("c1", "household@example.hu", "https://hub.example/bind/x"); err == nil {
t.Errorf("self-bind mail: a held mail must report an error")
}
if n := rec.count(); n != 0 {
t.Fatalf("MAIL-HOLD present but %d mail(s) were SENT: %v", n, rec.sent)
}
out := buf.String()
if got := strings.Count(out, "[WARN] MAIL-HOLD: not sending"); got < 7 {
t.Errorf("each held mail is logged once: want >= 7 MAIL-HOLD lines, got %d\n%s", got, out)
}
if strings.Contains(out, "household@example.hu") || strings.Contains(out, "op@felhom.eu") {
t.Errorf("the hold's log must carry no address:\n%s", out)
}
if strings.Contains(out, "CODE-1234") {
t.Errorf("the hold's log must carry no mail body")
}
// The record says what happened: held, not sent and not failed.
if _, ok, _ := st.LastCustomerSentAt("c1", []string{"backup_failed"}); ok {
t.Errorf("a held household mail was recorded as SENT")
}
}
// After the release the hub mails again — and the very same event key mails at once: the cooldown a held mail armed is
// given back, so the first real alarm after the release is not silenced by a mail that was never sent.
func TestMailHold_ReleaseResumesSends(t *testing.T) {
d, _, rec, hold, _ := holdDispatcher(t)
d.ProcessEvent("c1", "backup_failed", "error", "Backup failed", "", "box")
if rec.count() != 0 {
t.Fatalf("held: %d mail(s) sent", rec.count())
}
if err := hold.Release(); err != nil {
t.Fatalf("release: %v", err)
}
if hold.Held() {
t.Fatalf("the marker is still there after Release")
}
d.ProcessEvent("c1", "backup_failed", "error", "Backup failed", "", "box")
if n := rec.count(); n != 2 {
t.Fatalf("after the release the operator AND the household mail must go out: sent=%d %v", n, rec.sent)
}
// Held mails are DROPPED, not re-sent: exactly the two fresh ones, no backlog.
}
// Belt for the "one gate" claim: in dispatcher.go the sender seam is called in exactly ONE place, inside deliver().
// A new send path that calls sendEmailFn directly would bypass the hold; this fails when one appears.
func TestMailHold_EverySendPathIsGated(t *testing.T) {
src, err := os.ReadFile("dispatcher.go")
if err != nil {
t.Fatal(err)
}
calls := regexp.MustCompile(`d\.sendEmailFn\(`).FindAllIndex(src, -1)
if len(calls) != 1 {
t.Fatalf("dispatcher.go calls d.sendEmailFn( %d times; every send must go through deliver() (the MAIL-HOLD gate)", len(calls))
}
body := string(src)
i := strings.Index(body, "func (d *Dispatcher) deliver(")
if i < 0 || calls[0][0] < i || calls[0][0] > i+600 {
t.Fatalf("the one d.sendEmailFn( call is not inside deliver()")
}
}
+1
View File
@@ -50,6 +50,7 @@ func TestTemplatesParseWithFuncmap(t *testing.T) {
"memoryColor": memoryColor,
"accuracyClass": accuracyClass,
"gt": func(a, b int) bool { return false },
"mailHeld": func() bool { return false },
}).ParseFS(templateFS, "templates/*.html"); err != nil {
t.Fatalf("templates failed to parse: %v", err)
}
+135
View File
@@ -0,0 +1,135 @@
package web
import (
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/mailhold"
)
// MAIL-HOLD on the operator UI: the banner on every operator page while the marker exists, and the release endpoint
// behind the operator's auth + the R-135 CSRF gate.
const mailHoldBannerText = "E-mail is on hold"
func holdIn(t *testing.T, held bool) (*mailhold.Hold, string) {
t.Helper()
dir := t.TempDir()
marker := filepath.Join(dir, mailhold.FileName)
if held {
if err := os.WriteFile(marker, nil, 0o644); err != nil {
t.Fatal(err)
}
}
return mailhold.New(dir, nil), marker
}
func getPage(t *testing.T, s *Server, path string) string {
t.Helper()
r := httptest.NewRequest(http.MethodGet, path, nil)
w := httptest.NewRecorder()
s.ServeHTTP(w, r)
if w.Code != http.StatusOK {
t.Fatalf("GET %s: %d", path, w.Code)
}
return w.Body.String()
}
// Both branches of the {{if mailHeld}} gate, on the dashboard and on Configuration (which also carries the button).
func TestMailHoldBanner_RendersOnlyWhileHeld(t *testing.T) {
for _, held := range []bool{true, false} {
s, _ := newTestServer(t)
h, _ := holdIn(t, held)
s.SetMailHold(h)
for _, p := range []string{"/", "/configuration", "/hosts", "/configs"} {
body := getPage(t, s, p)
if got := strings.Contains(body, mailHoldBannerText); got != held {
t.Errorf("held=%v GET %s: banner shown=%v", held, p, got)
}
}
cfg := getPage(t, s, "/configuration")
if got := strings.Contains(cfg, `action="/configuration/mail-hold/release"`); got != held {
t.Errorf("held=%v: release button shown=%v", held, got)
}
}
}
// No hold wired at all (nil): no banner, pages render.
func TestMailHoldBanner_NilHoldRendersNoBanner(t *testing.T) {
s, _ := newTestServer(t)
if strings.Contains(getPage(t, s, "/"), mailHoldBannerText) {
t.Fatal("a nil hold must render no banner")
}
}
// The operator's release, through the production wiring (RequireAuth around ServeHTTP) with a session and its CSRF
// token: the marker is gone, and the banner with it.
func TestMailHoldRelease_RemovesMarker(t *testing.T) {
s, h := r135Handler(t)
hold, marker := holdIn(t, true)
s.SetMailHold(hold)
s.sessionsMu.Lock()
s.sessions["sess1"] = &hubSession{expiresAt: time.Now().Add(time.Hour), csrfToken: "tok1"}
s.sessionsMu.Unlock()
w := r135Post(h, "/configuration/mail-hold/release", func(r *http.Request) {
r.AddCookie(&http.Cookie{Name: SessionCookieName, Value: "sess1"})
r.Header.Set("X-CSRF-Token", "tok1")
})
if w.Code != http.StatusSeeOther || !strings.Contains(w.Header().Get("Location"), "flash=mail_hold_released") {
t.Fatalf("release: %d Location=%q", w.Code, w.Header().Get("Location"))
}
if _, err := os.Stat(marker); !os.IsNotExist(err) {
t.Fatalf("the marker is still there after the release: %v", err)
}
if hold.Held() {
t.Fatal("still held after the release")
}
}
// Refusals: Basic auth without the operator header (the R-135 cross-site shape), a session without its token, and no
// credentials at all. The marker stays in every case.
func TestMailHoldRelease_RefusedWithoutOperatorAuthOrCSRF(t *testing.T) {
s, h := r135Handler(t)
hold, marker := holdIn(t, true)
s.SetMailHold(hold)
s.sessionsMu.Lock()
s.sessions["sess1"] = &hubSession{expiresAt: time.Now().Add(time.Hour), csrfToken: "tok1"}
s.sessionsMu.Unlock()
cases := []struct {
name string
mut func(*http.Request)
ok func(int) bool
}{
{"basic without operator header", func(r *http.Request) { r.SetBasicAuth("", "op-pass") }, func(c int) bool { return c == http.StatusForbidden }},
{"session without token", func(r *http.Request) { r.AddCookie(&http.Cookie{Name: SessionCookieName, Value: "sess1"}) }, func(c int) bool { return c == http.StatusForbidden }},
{"no credentials", func(r *http.Request) { r.Header.Set(OperatorCLIHeader, "cli") }, func(c int) bool { return c == http.StatusFound || c == http.StatusUnauthorized }},
}
for _, c := range cases {
w := r135Post(h, "/configuration/mail-hold/release", c.mut)
if !c.ok(w.Code) {
t.Errorf("%s: status %d", c.name, w.Code)
}
if _, err := os.Stat(marker); err != nil {
t.Fatalf("%s: the marker was removed by a refused request: %v", c.name, err)
}
}
// The CLI shape (Basic + the operator header) passes.
w := r135Post(h, "/configuration/mail-hold/release", func(r *http.Request) {
r.SetBasicAuth("", "op-pass")
r.Header.Set(OperatorCLIHeader, "cli")
})
if w.Code != http.StatusSeeOther {
t.Fatalf("operator CLI release: %d", w.Code)
}
if hold.Held() {
t.Fatal("operator CLI release left the hold in place")
}
}
+1
View File
@@ -42,6 +42,7 @@ var r135PostRoutes = []string{
"/configuration/global-floor",
"/configuration/artifacts",
"/configuration/password",
"/configuration/mail-hold/release",
"/configs/c1/delete",
"/configs/c1/edit",
"/configs/c1/offsite-reissue",
+39 -1
View File
@@ -20,6 +20,7 @@ import (
"gitea.dooplex.hu/admin/felhom-hub/internal/claim"
"gitea.dooplex.hu/admin/felhom-hub/internal/gitea"
"gitea.dooplex.hu/admin/felhom-hub/internal/intent"
"gitea.dooplex.hu/admin/felhom-hub/internal/mailhold"
"gitea.dooplex.hu/admin/felhom-hub/internal/monitor"
"gitea.dooplex.hu/admin/felhom-hub/internal/offsite"
"gitea.dooplex.hu/admin/felhom-hub/internal/poke"
@@ -124,13 +125,20 @@ type Server struct {
// beforeCreateSave is a TEST seam: called in handleConfigCreate after the duplicate check, before the
// save. nil in production.
beforeCreateSave func(customerID string)
// mailHold is the hub-wide mail gate (internal/mailhold); nil never holds.
mailHold *mailhold.Hold
// cfAPIBase is a TEST seam for the Cloudflare token reach check (R-138 option C): "" = the real API.
cfAPIBase string
}
// New creates a new web server.
func New(store *store.Store, passwordHash, apiKey, version string, staleThreshold time.Duration, logger *log.Logger) *Server {
// srv is assigned below; template funcs that read server state close over it (they run at render time only).
var srv *Server
funcMap := template.FuncMap{
// mailHeld drives the MAIL-HOLD banner on every operator page (internal/mailhold). Read at render time, so the
// banner disappears the moment the marker is gone. Pinned by TestMailHoldBanner_*.
"mailHeld": func() bool { return srv != nil && srv.mailHold.Held() },
"timeAgo": timeAgo,
"timeAgoPtr": func(t *time.Time) string {
if t == nil {
@@ -160,7 +168,7 @@ func New(store *store.Store, passwordHash, apiKey, version string, staleThreshol
tmpl := template.Must(template.New("").Funcs(funcMap).ParseFS(templateFS, "templates/*.html"))
return &Server{
srv = &Server{
store: store,
configPasswordHash: passwordHash,
apiKey: apiKey,
@@ -171,6 +179,7 @@ func New(store *store.Store, passwordHash, apiKey, version string, staleThreshol
sessions: make(map[string]*hubSession),
bindLimiter: newBindRateLimiter(30), // public /bind/ surface: 30 req/min/IP burst (R-27)
}
return srv
}
// effectivePasswordHash returns the operator login password bcrypt hash in force: the UI-set DB
@@ -406,6 +415,29 @@ func (s *Server) artifactChoices(ctx context.Context, pkg, file string) []artifa
}
// SetEventEmitter wires the notification dispatcher (R-604). INIT-ONLY.
// SetMailHold wires the hub-wide mail gate (internal/mailhold): the banner on every operator page and the release
// endpoint. nil = never held (no banner, the release is a no-op).
func (s *Server) SetMailHold(m *mailhold.Hold) {
s.mailHold = m
}
// handleMailHoldRelease lifts the MAIL-HOLD: it removes the marker, and the hub sends e-mail again from the next mail
// on. Mails dropped during the hold are NOT re-sent (internal/mailhold explains why). Operator-only: it sits behind
// RequireAuth and the R-135 CSRF gate in ServeHTTP like every other state-changing route.
func (s *Server) handleMailHoldRelease(w http.ResponseWriter, r *http.Request) {
if s.mailHold == nil || !s.mailHold.Held() {
http.Redirect(w, r, "/configuration?flash=mail_hold_not_held", http.StatusSeeOther)
return
}
if err := s.mailHold.Release(); err != nil {
s.logger.Printf("[ERROR] MAIL-HOLD: release failed — the marker is still in place: %v", err)
http.Redirect(w, r, "/configuration?flash=mail_hold_release_failed", http.StatusSeeOther)
return
}
s.logger.Printf("[INFO] MAIL-HOLD released by the operator from %s — the hub sends e-mail again; mail held until now was dropped", r.RemoteAddr)
http.Redirect(w, r, "/configuration?flash=mail_hold_released", http.StatusSeeOther)
}
func (s *Server) SetEventEmitter(f func(customerID, eventType, severity, message, detailsJSON, source string)) {
s.emit = f
}
@@ -668,6 +700,12 @@ func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
} else {
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
}
case path == "/configuration/mail-hold/release":
if r.Method == http.MethodPost {
s.handleMailHoldRelease(w, r)
} else {
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
}
case path == "/configuration/password":
if r.Method == http.MethodPost {
s.handleChangePassword(w, r)
@@ -23,6 +23,7 @@
<a href="/configuration" class="nav-link">Configuration</a>
</nav>
</header>
{{template "mail_hold_banner"}}
<a href="/apps{{if .Period}}?period={{.Period}}{{end}}" class="back-link">&larr; Apps</a>
+1
View File
@@ -21,6 +21,7 @@
<a href="/configuration" class="nav-link">Configuration</a>
</nav>
</header>
{{template "mail_hold_banner"}}
<h2 style="margin-bottom: 1rem;">App Telemetry</h2>
@@ -22,6 +22,7 @@
<a href="/configuration" class="nav-link">Configuration</a>
</nav>
</header>
{{template "mail_hold_banner"}}
<a href="{{if .IsNew}}/configs{{else}}/customers/{{.Config.CustomerID}}{{end}}" class="back-link">&larr; Back</a>
<h2>{{if .IsNew}}Add Customer{{else}}Edit: {{.Config.CustomerID}}{{end}}</h2>
+1
View File
@@ -21,6 +21,7 @@
<a href="/configuration" class="nav-link">Configuration</a>
</nav>
</header>
{{template "mail_hold_banner"}}
{{if .Flash}}
<div class="flash flash-success">
@@ -21,9 +21,26 @@
<a href="/configuration" class="nav-link active">Configuration</a>
</nav>
</header>
{{template "mail_hold_banner"}}
<h2 style="margin-bottom: 1rem;">Configuration</h2>
{{if mailHeld}}
<form method="POST" action="/configuration/mail-hold/release" class="mail-hold-release">
<input type="hidden" name="_csrf" value="{{.CSRFToken}}">
<button type="submit" class="btn">Release the mail hold</button>
<span class="text-muted">Removes the marker. The hub sends e-mail again from the next mail on; mail held until now is not re-sent.</span>
</form>
{{end}}
{{if eq .Flash "mail_hold_released"}}
<div class="flash flash-success">Mail hold released — the hub sends e-mail again. Mail held until now was not re-sent.</div>
{{end}}
{{if eq .Flash "mail_hold_release_failed"}}
<div class="flash flash-error">The mail hold could not be released — the marker is still in place. Check the hub log.</div>
{{end}}
{{if eq .Flash "mail_hold_not_held"}}
<div class="flash flash-success">The mail hold was not on — nothing to release.</div>
{{end}}
{{if eq .Flash "assets_refreshed"}}
<div class="flash flash-success">Assets refreshed successfully from image seed.</div>
{{end}}
@@ -42,6 +42,7 @@
<p class="subtitle">No reports received yet</p>
{{end}}
</header>
{{template "mail_hold_banner"}}
{{if .Flash}}
<div class="flash flash-success">
@@ -22,6 +22,7 @@
<a href="/configuration" class="nav-link">Configuration</a>
</nav>
</header>
{{template "mail_hold_banner"}}
{{if or .OffsiteTile .PBSTile}}
<div class="offsite-gauges">
@@ -21,6 +21,7 @@
<a href="/configuration" class="nav-link">Configuration</a>
</nav>
</header>
{{template "mail_hold_banner"}}
<a href="/hosts" class="back-link">&larr; Hosts</a>
+1
View File
@@ -22,6 +22,7 @@
<a href="/configuration" class="nav-link">Configuration</a>
</nav>
</header>
{{template "mail_hold_banner"}}
<h2 style="margin-bottom: 1rem;">Hosts</h2>
+1
View File
@@ -21,6 +21,7 @@
<a href="/configuration" class="nav-link">Configuration</a>
</nav>
</header>
{{template "mail_hold_banner"}}
<a href="/customers/{{.CustomerID}}" class="back-link">&larr; {{.CustomerID}}</a>
@@ -0,0 +1,8 @@
{{define "mail_hold_banner"}}{{if mailHeld}}
<div class="flash flash-warn" id="mail-hold" role="alert">
<strong>E-mail is on hold</strong> — this hub sends no e-mail at all, to households or to you, while the
<code>MAIL-HOLD</code> marker is in its data directory. Mail it would have sent is dropped, not queued.
Release the hold on the <a href="/configuration#mail-hold">Configuration</a> page once this is the only hub
that is running.
</div>
{{end}}{{end}}
+1
View File
@@ -21,6 +21,7 @@
<a href="/configuration" class="nav-link">Configuration</a>
</nav>
</header>
{{template "mail_hold_banner"}}
<h2 style="margin-bottom: 0.75rem;">Offsite</h2>
+11
View File
@@ -1020,3 +1020,14 @@ body.js-tabs .tab-panel:not(.tab-panel-active) { display: none; }
@media (prefers-reduced-motion: reduce) {
*, *::before, *::after { animation: none !important; transition: none !important; }
}
/* MAIL-HOLD release (internal/mailhold): the button and its one-line explanation sit on one row under the banner. */
.mail-hold-release {
display: flex;
align-items: center;
gap: 0.75rem;
flex-wrap: wrap;
margin-bottom: 1rem;
}
.mail-hold-release .text-muted { font-size: 0.8rem; }
#mail-hold a { color: var(--text-1); }
+1
View File
@@ -30,6 +30,7 @@
<a href="/configuration" class="nav-link">Configuration</a>
</nav>
</header>
{{template "mail_hold_banner"}}
<h2 style="margin-bottom: 1rem;">System — versions and OS updates</h2>
+4
View File
@@ -94,6 +94,10 @@ ROOTS = [
# ACK's operator_actions list. Named type for the same reason as R-311's root above.
("hub -> controller (report ACK, `operator_actions` entry)",
"hub", "internal/store", "OperatorActionDirective", "controller"),
# R-922 (2026-10-09): the notification-prefs push. Declared when the household's deliberate clear
# (`email_cleared`) joined it — a flag the hub cannot decode would leave the cleared address stored.
("controller -> hub (POST /preferences)",
"controller", "internal/notify", "preferencesRequest", "hub"),
]
# R-315: a root whose receiver decodes it into ONE NAMED MIRROR TYPE gets the stronger check —