hub v0.71.0: paired recovery mails (F11), prefs seeding at claim + empty-email no-clobber (F12), priority headers + operator test leg (F14-light)

This commit is contained in:
2026-07-22 20:57:29 +02:00
parent 5b35023574
commit c766c8af82
15 changed files with 832 additions and 30 deletions
+18
View File
@@ -3,6 +3,24 @@
> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
- **2026-07-22 — hub v0.71.0: the notification train (audit F11+F12+F14-light).** Four rulings now
standing: **(1) recovery pairing** — `*_recovered` notifies the operator always and the customer
**iff the customer was mailed the paired stale/down** (evidence = `notification_log` customer-
channel sent rows via `store.LastCustomerSentAt`; `enabled_events` deliberately ignored for
recovery; ties → no mail). Severity semantics FROZEN — recovery stays `info`, routed by an
explicit eventType branch before the severity gate. **(2) seed-at-claim** — `MarkClaimed` seeds
`customer_notifications` from the registered email, INSERT-if-absent only (never touches an
edited row; empty email = no-op; seed failure never fails the claim); default set is
critical-only: node_down, backup_failed, disk_critical, host_disk_critical,
storage_fill_critical, offbox_repo_orphaned. **(3) empty-email no-clobber** — a prefs push with
empty email preserves a stored address (hub-side belt; controller 0.160.0 already guards its two
push legs). **(4) priority headers** — error/critical mails carry `X-Priority: 1` +
`Importance: high` (Resend `headers`, live-probed); the `test` event now also mails the operator
with those headers (one click proves both channels + rendering). Latent nil-deref fixed:
`sendTestEmail` panicked on a customer with no prefs row. NOT yet live: a natural `*_recovered`
mail (next real staleness cycle / reboot drill — never fabricate one by blocking reports) and
seed-at-claim on a real claim (Peti's Friday reinstall is the natural candidate).
- **2026-07-22 — hub v0.70.1: the ghost customer's Delete button exists now.** The v0.70.0
ghost-delete path was fully implemented and fully unreachable — the **fourth inert-seam defect**,
this time a TEMPLATE GATE: the Danger-zone card (and the `customerDeleteOpen` script) sat inside
+9 -4
View File
@@ -26,11 +26,16 @@
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
| `(*Dispatcher).ProcessEvent` | hub/internal/notify/dispatcher.go (~L67) | `(customerID, eventType, severity, message, detailsJSON, source)` | THE notification pipeline (operator + customer channels, cooldowns, prefs) | Call in a goroutine (handlers do `go d.ProcessEvent(...)`). No Resend key → silent no-op. `eventType=="test"` bypasses prefs/cooldown. |
| `severityNotifies` | hub/internal/notify/dispatcher.go (~L56) | `(severity string) bool` | Deciding whether a severity emails | warning/error/critical notify; info intentionally doesn't; anything else is logged as unrecognized (v0.24.0 fix — do not regress). |
| `FormatOperatorEmail` / `FormatCustomerEmail` | hub/internal/notify/templates.go (~L24 / ~L118) | `(...) (subject, body)` | Operator (English) / customer (Hungarian) email bodies | Customer messages come from the `customerMessages` map — add the Hungarian text when adding an event type. Budapest TZ via package `init()`. |
| `(*Dispatcher).ProcessEvent` | hub/internal/notify/dispatcher.go (~L88) | `(customerID, eventType, severity, message, detailsJSON, source)` | THE notification pipeline (operator + customer channels, cooldowns, prefs) | Call in a goroutine (handlers do `go d.ProcessEvent(...)`). No Resend key → silent no-op. `eventType=="test"` bypasses prefs/cooldown (and since v0.71.0 also mails the operator). `*_recovered` routes via the explicit recovery branch BEFORE the severity gate (v0.71.0) — see `processRecovery`. |
| `(*Dispatcher).processRecovery` | hub/internal/notify/dispatcher.go (~L160) | internal | `node_recovered`/`host_recovered` routing (audit F11) | Operator always (1 h cooldown); customer iff PAIRED — a customer-channel `sent` stale/down row newer than the last sent recovery (`store.LastCustomerSentAt`); `enabled_events` deliberately ignored for recovery; timestamp ties → no mail (flap-safe). Severity stays `info` — never "fix" that by widening `severityNotifies`. |
| `severityNotifies` | hub/internal/notify/dispatcher.go (~L77) | `(severity string) bool` | Deciding whether a severity emails | warning/error/critical notify; info intentionally doesn't; anything else is logged as unrecognized (v0.24.0 fix — do not regress). Recovery mails exist DESPITE this gate (eventType branch), not through it. |
| `priorityHeaders` | hub/internal/notify/dispatcher.go (~L56) | `(severity string) map[string]string` | High-priority mail-client nudge (audit F14-light) | error/critical → `X-Priority: 1` + `Importance: high`; everything else nil — a warning/info mail must NOT masquerade as urgent (red-proofed). |
| `sendEmailFn` seam / `sendEmail` | hub/internal/notify/dispatcher.go (~L33 / ~L300) | `func(to, subject, textBody string, headers map[string]string) error` | Test seam for all sends; Resend POST | Signature grew a `headers` param in v0.71.0 — payload carries `"headers"` only when non-empty. Tests capture recipient+subject+headers through the seam. |
| `FormatOperatorEmail` / `FormatCustomerEmail` | hub/internal/notify/templates.go (~L24 / ~L118) | `(...) (subject, body)` | Operator (English) / customer (Hungarian) email bodies | Customer messages come from the `customerMessages` map — add the Hungarian text when adding an event type. Budapest TZ via package `init()`. Operator icon is eventType-aware: `*_recovered` → ✅ (severity is the fallback). |
| `monitor.EventNotifyFunc` | hub/internal/monitor/staleness.go (~L14) | `func(customerID, eventType, severity, message, detailsJSON, source)` | Decoupling checkers from notify; wired to `dispatcher.ProcessEvent` in main | May be nil — always nil-check before calling (all checkers do). |
| `(*Store).LogNotification` | hub/internal/store/store.go (~L433) | `(customerID, eventType, severity, message, status, errorMsg, channel)` | Audit trail of every send attempt (sent/failed, per channel) | Log BOTH success and failure (dispatcher does). |
| `(*Store).LogNotification` | hub/internal/store/store.go (~L433) | `(customerID, eventType, severity, message, status, errorMsg, channel)` | Audit trail of every send attempt (sent/failed, per channel) | Log BOTH success and failure (dispatcher does). Since v0.71.0 these rows are also the recovery PAIRING evidence — never prune them casually. |
| `(*Store).LastCustomerSentAt` | hub/internal/store/store.go (~L815) | `(customerID, eventTypes []string) (time.Time, bool, error)` | Pairing-evidence query (max customer-channel `sent` created_at over types) | Uses the `(customer_id, created_at DESC)` index. Empty type list → `(zero, false, nil)`. |
| `(*Store).SeedNotificationPrefs` | hub/internal/store/store.go (~L850) | `(customerID, email, enabledEvents) (seeded bool, err)` | Claim-time prefs seeding (audit F12) | INSERT OR IGNORE — never an upsert (red-proofed); empty email = no-op. Customer edits go through `SaveNotificationPrefs`, seeds NEVER do. |
### App-mail passthrough (hub/internal/mailrelay/)
@@ -104,7 +104,8 @@
|---|---|---|---|---|
| Health-degradation email (edge-triggered, cooldowns, Hungarian) via hub → Resend | controller, hub | **IMPLEMENTED** | delivery pipeline live-proven for the **enlarge-block** trigger (`CAMPAIGN-6D` P3-DELIVERY, op+customer "Kedves Ügyfél!"); `NotifyHealthChange` ok→warn/fail edge-trigger implemented | The **health-degradation** trigger specifically has never fired an email live in any doc. Demoted (pipeline proven for a different event). Deliverability to HU freemail → R-4 |
| Event catalog: app_start_failed, dead-app, offbox_enlarge_blocked, claim/reset codes, critical severity | controller, hub v0.31/48/50/55 | **PROVEN-LIVE** | live-delivered: `CAMPAIGN-6D` P3-DELIVERY (enlarge-block, op+customer); `DRILL-day0-vm` F-4 (claim code); `DRILL-day0-take2` F-15 (reset code) | `app_start_failed`/`dead-app` delivery is unit-only (6C inconclusive) — the pipeline + 3 event families are live, those two are not |
| Prefs safety: empty-email wipe guard | controller v0.137 | **IMPLEMENTED** | red-proofed 07-15 | Born from a live incident; guard itself unit-proven |
| Prefs safety: empty-email wipe guard | controller v0.137 + hub v0.71.0 | **IMPLEMENTED** | controller leg red-proofed 07-15; hub-side no-clobber belt (`handleSavePreferences` preserves a stored non-empty address on an empty-email push) red-proofed 07-22 | Born from a live incident; controller 0.160.0 guards both its push legs, so the hub belt covers older/rogue boxes |
| Paired recovery notifications + prefs seeding at claim + priority headers (power-outage audit F11/F12/F14-light) | hub v0.71.0 | **IMPLEMENTED** (recovery leg **PARTIAL** until a live staleness cycle fires it) | `hub/CHANGELOG.md` v0.71.0; 17 tests + 4 red-proofs (`REPORT.md` 2026-07-22); Resend `headers` mechanism probed live (HTTP 200) pre-implementation; operator+customer `test` rows live-fired via the controller's own test endpoint | Recovery = explicit eventType branch, severity semantics frozen; customer gate = PAIRING (`notification_log` evidence), not `enabled_events`. Live legs pending: a natural `*_recovered` mail (next real staleness cycle or the reboot-drill arc — never fabricated by blocking reports) and seed-at-claim on a real claim (Peti Friday reinstall). F14-full (operator push channel, ntfy/Telegram) stays open → R-69 |
| System + container metrics (SQLite, Chart.js, 30-day downsampling) | controller | **IMPLEMENTED** | metrics collection + `/monitoring` render present (page 200) | The cited `CAMPAIGN-2` T-RES-CPU/T-SOAK-LOOP are H1/H2 harness artifacts (auth-302), not metrics tests; SQLite/Chart.js/30-day downsampling validated in no campaign. Demoted |
| Always-on debug rings + on-demand log-bundle pulls with TTL/custody | controller v0.116, agent v0.83, hub v0.46 | **PROVEN-LIVE** | debug rings live-exercised `CAMPAIGN-3` fix-6 (1000-cap ring, ~55min horizon under load) | The **log-bundle-pull TTL/custody** half is changelog-only (no dedicated observability audit doc); ring persistence across restart is a known gap |
| Operator alerting (Healthchecks → monitoring@felhom.eu) | k3s, Resend | **IMPLEMENTED** | operator infra, stated in production since 02-04; no corpus validation doc | Per the status enum, no citation → not PROVEN-LIVE. Demoted pending an operator-cited live alert (candidate re-upgrade — see REPORT) |
+2
View File
@@ -79,6 +79,8 @@
| R-64 | **„Felhom↔Felhom media pairing blessed" — the two-box SMB pairing (one box shares, the other mounts it as NAS storage) becomes a supported, documented flow.** | XSS | idea (2026-07-22) | Origin: the operator ran the pairing drill on the live demo pair and it WORKS — the drill itself is the pending evidence leg (a written run-through with the R-66 surfaces in play). R-66 shipped the enabling visibility: the serving box's address is now on its own Beállítások → Rendszer „Hálózat" card, and the add form names the NetBIOS trap. Blessing = a short customer-facing recipe (`documentation/controller/network-storage-nas.md` naming-caveat paragraph is the seed) + one supported-path sentence in the capability map. Flips: would add a "Felhom↔Felhom media pairing" capability row (currently unlisted). Pairs with R-65 (same two-box topology, entirely different transport + guarantees) |
| R-66 | **The box's own address becomes visible — „Hálózat" card, Debug network dump, NetBIOS hint.** | XS | **SHIPPED (controller v0.159.0, 2026-07-22)** | Origin: the pairing drill — the serving box's IP was findable only as a hint buried on the OTHER box's Megosztás page, and the add form's failure for „FELHOM" taught nothing. Three legs: (A) „Hálózat" card on Beállítások → Rendszer (Helyi cím / Hálózati név only-while-sharing / Átjáró; live per render, stored nowhere — S-5; „—" when unavailable); (B) `network` section in the Debug dump (interfaces/route/DNS/lan_address, best-effort per item); (C) the NetBIOS trap named (Szerver helper text + a purely lexical hint on `unreachable` for single-label non-IP names). **Design decision recorded:** the controller is bridge-netns'd, so ALL guest-net reads go through the one netns door (docker exec into host-networked felhom-samba, `stacks/guestnet.go`) — with Megosztás off the card honestly shows „—" rather than the plausible-wrong 172.x answer. Deployed demo-felhom + demo-hp 2026-07-22; demo-hp live-shows the closed-door path (sharing off → dashes + in-place dump errors), demo-felhom the open one (real .104/.1/\\FELHOM values). Flips no capability-map row (diagnosability/UX polish); enables R-64 |
| R-67 | **The NAS share appears in FileBrowser — browse what you mounted.** | S | **SHIPPED (controller v0.160.0, 2026-07-22)** | Origin: the R-64 pairing drill — the share said „Elérhető" and the customer had no way to BROWSE it (FileBrowser synced drives only). **Couples to R-64: browsing was its missing UX half.** A registered network storage now binds its share ROOT into FileBrowser (`/mnt/felhom-drives/<name>:/srv/<name>:rslave`) with its display label as the sidebar source; NAS add/remove trigger the same debounced sync. Two classes, two gates: drives keep the drive-absent gate byte-identically (proven live: the drives-only box logged a no-op sync); network shares gate on the STUB classifier instead — idle autofs is HEALTHY and included (Phase-0 probe on demo-hp: an in-container access through an rslave bind WAKES the idle trigger), while a stub verdict excludes the share from mounts AND sources with a WARN (an exposed stub swallows uploads the real mount later shadows). Nothing is ever written toward the NAS (no skeleton — red-proven). Live leg: cross-box upload round-trip demo-hp → demo-felhom + dead-NAS check (`Host is down` in seconds, unaided recovery after samba restart). Operator residual: the FileBrowser UI click-through (its admin credential is customer-held by design). Evidence: `felhom-controller/REPORT.md` (2026-07-22) |
| R-68 | **Notification train: paired recovery mails + prefs seeding at claim + priority headers (power-outage audit F11+F12+F14-light).** The dead-man's-switch fired perfectly on 07-22 and the customer who got „A szerver nem elérhető!" was never told it recovered (F11); a customer without a `customer_notifications` row is silently unnotifiable (F12, demo-hp live); delivered ≠ noticed (F14). | M | **SHIPPED (hub v0.71.0, 2026-07-22)** | Origin: `AUDIT-power-outage-recovery-2026-07-22.md`. Recovery = explicit eventType branch (severity semantics frozen; `severityNotifies` untouched): operator always hears both edges, customer iff PAIRED (customer-channel `sent` stale/down row newer than the last sent recovery — `store.LastCustomerSentAt`; `enabled_events` deliberately ignored for recovery; ties → no mail, flap-safe). Seed-at-claim: `MarkClaimed``SeedNotificationPrefs` (INSERT-if-absent, never upsert — red-proofed; empty email no-op; never fails the claim; default critical-only set). Hub-side empty-email no-clobber belt in `handleSavePreferences` (controller 0.160.0 already guards its own two push legs — latent, not live). `X-Priority: 1` + `Importance: high` on error/critical via Resend `headers` (live-probed HTTP 200 before implementation); the `test` event now also mails the operator with those headers (one click proves both channels + rendering); latent `sendTestEmail` nil-prefs panic fixed. 17 tests + 4 red-proofs. **Live legs pending:** natural `*_recovered` mail on the next real staleness cycle (or the reboot-drill arc — NEVER fabricated by blocking reports, that is F9-bypass-shaped) and seed-at-claim on a real claim (Peti Friday reinstall is the natural candidate) |
| R-69 | **F14-full: an operator push channel that actually interrupts (ntfy / Telegram / similar), beyond mail-client priority flags.** F14-light (v0.71.0 headers + Gmail filter) nudges a mail client; a 15:29 node_down should reach the operator's pocket in seconds regardless of inbox hygiene. Needs: channel choice (self-hosted ntfy on k3s vs Telegram bot), dispatcher fan-out seam, per-severity routing, quiet hours. | M | idea | Origin: `AUDIT-power-outage-recovery-2026-07-22.md` F14. Deliberately NOT built in the v0.71.0 train (scope-forked per the task spec) |
| R-53 | **`app_export.html` substituted the CSRF token where the customer domain belongs** - the open-in-browser link was wrong for every app with a subdomain, and a session CSRF token landed in a URL. | XS | **SHIPPED (controller v0.150.0, 2026-07-20)** | One template token (`{{$.CSRFToken}}` -> `{{$.Domain}}`) plus the `Domain` key in `exportPageHandler`'s data map - that handler does not go through `baseData`, which is where every other page gets it, so the template had no domain to read. Render tests assert the joined `<sub>.<domain>` and that the token appears nowhere in that line; red-proofed against the pre-fix template. Origin: `audits/AUDIT-vacation-remote-ops-2026-07-20.md` (F7) |
## P3 — post-alpha
+38
View File
@@ -1,5 +1,43 @@
# Felhom Hub — Changelog
## v0.71.0 — paired recovery mails, prefs seeding at claim, priority headers, operator test leg (2026-07-22)
Origin: `documentation/audits/AUDIT-power-outage-recovery-2026-07-22.md` F11 (recovery is silent),
F12 (prefs row optional → customer never notified), F14-light (delivered ≠ noticed). Live proof of
the gap: the demo customer got „A szerver nem elérhető!" at 15:29 and was never told it recovered.
- **Paired recovery notifications (F11)**`dispatcher.go` `processRecovery`, an explicit
eventType branch in `ProcessEvent` BEFORE the severity gate (`severityNotifies` and the checkers'
`emitTransition` severities are byte-untouched; `*_recovered` stays `info`). Operator always gets
both edges (existing 1 h per-type cooldown); the customer gets recovery **iff the customer was
mailed the paired stale/down** — pairing evidence is `store.LastCustomerSentAt` over
`notification_log` (customer channel, status=sent, `node_recovered→{node_stale,node_down}`,
`host_recovered→{host_stale,host_down}`), ties resolve to no-mail (flap-safe). `enabled_events`
is deliberately NOT consulted for recovery. Suppressions log at INFO with the reason.
`FormatOperatorEmail` renders ✅ for `*_recovered`; `customerMessages` gains `host_recovered`.
- **Prefs seeding at claim (F12)**`claim.Engine.MarkClaimed` seeds `customer_notifications`
from the registered `customer_configs.email` on the unclaimed→claimed transition via new
`store.SeedNotificationPrefs` (INSERT OR IGNORE — never touches an existing row; empty email =
no-op; a seed failure never fails the claim). Default set (critical-only, Viktor may adjust):
node_down, backup_failed, disk_critical, host_disk_critical, storage_fill_critical,
offbox_repo_orphaned.
- **Empty-email no-clobber guard (F12)**`handleSavePreferences`: a push with an empty email
preserves a stored non-empty address (events + cooldown still apply); a non-empty push updates
everything. Phase-0a fact: controller 0.160.0 guards both push legs itself
(`cmd/controller/main.go:821` startup skips empty email; `web/handlers.go:1532` refuses
empty-with-events), so the clobber was latent — this is the hub-side belt for older/rogue boxes.
- **Priority headers (F14-light)**`sendEmailFn`/`sendEmail` gain a `headers` param; Resend
payload carries `"headers"` only when non-empty. `priorityHeaders(severity)`: error/critical →
`X-Priority: 1` + `Importance: high`; warning/info → none. Mechanism probed live pre-implementation
(Resend accepted, HTTP 200, mail id `34d3f7f3…`).
- **Operator test leg** — the `test` event now also mails the operator (`✅ <id>: teszt / operator
channel OK`, priority headers forced) — one click proves customer channel + operator channel +
header rendering. Fixed a latent nil-deref found here: `sendTestEmail` dereferenced
`prefs.Email` while `GetNotificationPrefs` returns `(nil, nil)` for a customer with no row — a
test event for such a customer (e.g. demo-hp) panicked the dispatcher goroutine.
- Tests: 17 new (449 → 466) across store/notify/claim/api; 4 red-proofs run + reverted (pairing
removed, upsert-seed, guard removed, unconditional headers) — see `REPORT.md`.
## v0.70.1 — the ghost customer's Delete button must exist (2026-07-22)
**The fourth inert-seam defect: v0.70.0's ghost-delete path was fully implemented and fully
+14 -2
View File
@@ -1904,13 +1904,25 @@ func (h *Handler) handleSavePreferences(w http.ResponseWriter, r *http.Request)
return
}
if err := h.store.SaveNotificationPrefs(payload.CustomerID, payload.Email, payload.EnabledEvents, payload.CooldownHours); err != nil {
// Empty-email no-clobber guard (v0.71.0, audit F12): a controller push with an empty email
// (e.g. an unconfigured box) must never wipe a stored non-empty address — the seeded/edited
// email is the customer's alert lifeline. Events + cooldown from the push still apply; a push
// with a non-empty email updates everything (customer edits keep working).
saveEmail := payload.Email
if saveEmail == "" {
if existing, err := h.store.GetNotificationPrefs(payload.CustomerID); err == nil && existing != nil && existing.Email != "" {
saveEmail = existing.Email
h.logger.Printf("[INFO] Notification prefs push for %s had empty email — preserving stored address", payload.CustomerID)
}
}
if err := h.store.SaveNotificationPrefs(payload.CustomerID, saveEmail, payload.EnabledEvents, payload.CooldownHours); err != nil {
h.logger.Printf("[ERROR] Failed to save notification prefs for %s: %v", payload.CustomerID, err)
http.Error(w, "Internal error", http.StatusInternalServerError)
return
}
h.logger.Printf("[INFO] Notification preferences updated for %s: email=%s, events=%v", payload.CustomerID, payload.Email, payload.EnabledEvents)
h.logger.Printf("[INFO] Notification preferences updated for %s: email=%s, events=%v", payload.CustomerID, saveEmail, payload.EnabledEvents)
w.WriteHeader(http.StatusOK)
w.Write([]byte(`{"status":"ok"}`))
}
@@ -0,0 +1,67 @@
package api
import (
"net/http"
"testing"
)
// TestSavePreferences_EmptyEmailCannotClobber (v0.71.0 F12, Scenario E): a controller push with an
// empty email must preserve a stored non-empty address while still applying events + cooldown.
// Companion red-proof: removing the guard in handleSavePreferences makes this fail.
func TestSavePreferences_EmptyEmailCannotClobber(t *testing.T) {
h, st := newEventTestHandler(t)
if err := st.SaveNotificationPrefs("c1", "seeded@example.hu", []string{"node_down", "backup_failed"}, 6); err != nil {
t.Fatalf("stored prefs: %v", err)
}
rr := do(h, http.MethodPost, "/preferences", "ckey",
`{"customer_id":"c1","email":"","enabled_events":["node_down"],"cooldown_hours":12}`)
if rr.Code != http.StatusOK {
t.Fatalf("status = %d, body=%s", rr.Code, rr.Body.String())
}
prefs, err := st.GetNotificationPrefs("c1")
if err != nil || prefs == nil {
t.Fatalf("prefs: %+v err=%v", prefs, err)
}
if prefs.Email != "seeded@example.hu" {
t.Fatalf("empty-email push CLOBBERED the stored address: email=%q", prefs.Email)
}
if len(prefs.EnabledEvents) != 1 || prefs.EnabledEvents[0] != "node_down" || prefs.CooldownHours != 12 {
t.Fatalf("events/cooldown from the push must still apply: %+v", prefs)
}
}
// TestSavePreferences_NonEmptyEmailStillUpdates: a push with a real email updates everything —
// customer edits keep working (the guard must not freeze the address forever).
func TestSavePreferences_NonEmptyEmailStillUpdates(t *testing.T) {
h, st := newEventTestHandler(t)
if err := st.SaveNotificationPrefs("c1", "old@example.hu", []string{"node_down"}, 6); err != nil {
t.Fatalf("stored prefs: %v", err)
}
rr := do(h, http.MethodPost, "/preferences", "ckey",
`{"customer_id":"c1","email":"new@example.hu","enabled_events":["backup_failed"],"cooldown_hours":3}`)
if rr.Code != http.StatusOK {
t.Fatalf("status = %d, body=%s", rr.Code, rr.Body.String())
}
prefs, _ := st.GetNotificationPrefs("c1")
if prefs.Email != "new@example.hu" || len(prefs.EnabledEvents) != 1 || prefs.EnabledEvents[0] != "backup_failed" || prefs.CooldownHours != 3 {
t.Fatalf("non-empty push must update everything: %+v", prefs)
}
}
// TestSavePreferences_EmptyEmailNoStoredRow: an empty-email push with no stored row behaves as
// before (row created with empty email — the all-off case stays legitimate).
func TestSavePreferences_EmptyEmailNoStoredRow(t *testing.T) {
h, st := newEventTestHandler(t)
rr := do(h, http.MethodPost, "/preferences", "ckey",
`{"customer_id":"c1","email":"","enabled_events":[],"cooldown_hours":6}`)
if rr.Code != http.StatusOK {
t.Fatalf("status = %d, body=%s", rr.Code, rr.Body.String())
}
prefs, _ := st.GetNotificationPrefs("c1")
if prefs == nil || prefs.Email != "" {
t.Fatalf("all-off push must store the empty row unchanged, got %+v", prefs)
}
}
+23 -1
View File
@@ -199,8 +199,23 @@ func (e *Engine) ResetToUnclaimed(cc *store.CustomerConfig) error {
return nil
}
// defaultSeedEvents is the enabled_events set seeded at claim (v0.71.0, audit F12) — critical-only:
// no node_stale (too chatty), no *_recovered (the recovery pairing gate handles those and never
// consults enabled_events). Viktor may adjust the list at review.
var defaultSeedEvents = []string{
"node_down",
"backup_failed",
"disk_critical",
"host_disk_critical",
"storage_fill_critical",
"offbox_repo_orphaned",
}
// MarkClaimed records a controller-reported successful claim and sends the one-time confirmation
// email on the unclaimed→claimed transition (idempotent — repeated reports are no-ops).
// email on the unclaimed→claimed transition (idempotent — repeated reports are no-ops). On the
// transition it also seeds notification prefs from the registered email (v0.71.0, audit F12) so a
// claimed customer can no longer be silently unnotifiable — insert-if-absent, never overwriting a
// customer-edited row, and never failing the claim (notification plumbing must not gate claiming).
func (e *Engine) MarkClaimed(cc *store.CustomerConfig) error {
transitioned, err := e.Store.MarkClaimed(cc.CustomerID)
if err != nil {
@@ -210,6 +225,13 @@ func (e *Engine) MarkClaimed(cc *store.CustomerConfig) error {
return nil
}
e.logf("[INFO] [claim] customer %s CLAIMED its dashboard (password set by the customer)", cc.CustomerID)
if seeded, err := e.Store.SeedNotificationPrefs(cc.CustomerID, cc.Email, defaultSeedEvents); err != nil {
e.logf("[WARN] [claim] notification-prefs seed for %s failed (claim unaffected): %v", cc.CustomerID, err)
} else if seeded {
e.logf("[INFO] [claim] notification prefs seeded for %s from the registered email (default critical set)", cc.CustomerID)
} else {
e.logf("[INFO] [claim] notification prefs for %s left untouched (row exists or no registered email)", cc.CustomerID)
}
if cc.Email != "" {
if err := e.Mailer.SendClaimEmail(string(EmailClaimed), cc.CustomerID, cc.Email, cc.Domain, ""); err != nil {
e.logf("[WARN] [claim] claimed-confirmation email to %s failed: %v", cc.CustomerID, err)
+78
View File
@@ -0,0 +1,78 @@
package claim
import (
"testing"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// TestMarkClaimed_SeedsNotificationPrefs (v0.71.0 F12, Scenario D): the unclaimed→claimed
// transition seeds a customer_notifications row from the registered email with the default
// critical-only event set; a pre-existing (customer-edited) row is NEVER modified.
// Companion red-proof: seeding via SaveNotificationPrefs (upsert) makes the second half fail.
func TestMarkClaimed_SeedsNotificationPrefs(t *testing.T) {
e, st, _ := newTestEngine(t)
if _, err := e.EnsureIssued(cust()); err != nil {
t.Fatalf("EnsureIssued: %v", err)
}
if err := e.MarkClaimed(cust()); err != nil {
t.Fatalf("MarkClaimed: %v", err)
}
prefs, err := st.GetNotificationPrefs("c1")
if err != nil || prefs == nil {
t.Fatalf("prefs must be seeded at claim, got %+v err=%v", prefs, err)
}
if prefs.Email != "owner@example.hu" {
t.Fatalf("seeded email = %q, want the registered address", prefs.Email)
}
if len(prefs.EnabledEvents) != len(defaultSeedEvents) {
t.Fatalf("seeded events = %v, want the default set %v", prefs.EnabledEvents, defaultSeedEvents)
}
for i, ev := range defaultSeedEvents {
if prefs.EnabledEvents[i] != ev {
t.Fatalf("seeded events = %v, want %v", prefs.EnabledEvents, defaultSeedEvents)
}
}
if prefs.CooldownHours != 6 {
t.Fatalf("seeded cooldown = %d, want 6", prefs.CooldownHours)
}
}
// TestMarkClaimed_NeverOverwritesExistingPrefs: a customer-edited row survives a claim
// byte-identical (re-claim after RESET is the natural trigger).
func TestMarkClaimed_NeverOverwritesExistingPrefs(t *testing.T) {
e, st, _ := newTestEngine(t)
if err := st.SaveNotificationPrefs("c1", "edited@example.hu", []string{"node_down"}, 12); err != nil {
t.Fatalf("pre-existing prefs: %v", err)
}
if _, err := e.EnsureIssued(cust()); err != nil {
t.Fatalf("EnsureIssued: %v", err)
}
if err := e.MarkClaimed(cust()); err != nil {
t.Fatalf("MarkClaimed: %v", err)
}
prefs, _ := st.GetNotificationPrefs("c1")
if prefs == nil || prefs.Email != "edited@example.hu" || len(prefs.EnabledEvents) != 1 || prefs.CooldownHours != 12 {
t.Fatalf("claim seed MODIFIED an existing row (upsert bug): %+v", prefs)
}
}
// TestMarkClaimed_EmptyEmail_NoRow_ClaimSucceeds: no registered email → no seed, and the claim
// itself still succeeds (notification plumbing must never gate claiming).
func TestMarkClaimed_EmptyEmail_NoRow_ClaimSucceeds(t *testing.T) {
e, st, _ := newTestEngine(t)
noMail := &store.CustomerConfig{CustomerID: "c1", Email: "", Domain: "example.hu"}
if _, err := e.EnsureIssued(cust()); err != nil { // issue needs an email to deliver the code
t.Fatalf("EnsureIssued: %v", err)
}
if err := e.MarkClaimed(noMail); err != nil {
t.Fatalf("MarkClaimed with empty email must succeed: %v", err)
}
if prefs, _ := st.GetNotificationPrefs("c1"); prefs != nil {
t.Fatalf("empty registered email must seed nothing, got %+v", prefs)
}
// The transition happened — a second call is a no-op (unchanged idempotency).
if err := e.MarkClaimed(noMail); err != nil {
t.Fatalf("idempotent re-claim: %v", err)
}
}
+133 -16
View File
@@ -29,8 +29,10 @@ type Dispatcher struct {
custCooldowns map[string]time.Time // "customerID:eventType" → last customer notify
// sendEmailFn is the email sender, seam-injected so tests exercise routing without real HTTP.
// Defaults to (*Dispatcher).sendEmail (Resend) in NewDispatcher.
sendEmailFn func(to, subject, textBody string) error
// Defaults to (*Dispatcher).sendEmail (Resend) in NewDispatcher. headers (nil = none) become
// Resend custom headers — used for the high-priority nudge on error/critical mails (v0.71.0,
// audit F14-light).
sendEmailFn func(to, subject, textBody string, headers map[string]string) error
}
// NewDispatcher creates a new notification dispatcher.
@@ -50,6 +52,26 @@ func NewDispatcher(s *store.Store, resendAPIKey, fromEmail, operatorEmail string
return d
}
// priorityHeaders returns the Resend custom headers that nudge mail clients toward attention for
// error/critical mails (X-Priority + Importance; v0.71.0, audit F14-light: delivered ≠ noticed).
// Everything else gets nil — a warning or info mail must NOT masquerade as urgent. Pure → tested.
func priorityHeaders(severity string) map[string]string {
switch severity {
case "error", "critical":
return map[string]string{"X-Priority": "1", "Importance": "high"}
default:
return nil
}
}
// recoveredPairedDownTypes maps a *_recovered eventType to the stale/down set whose customer-channel
// "sent" evidence licenses the customer recovery mail (v0.71.0, audit F11): recovery notifies
// exactly whoever the down notified.
var recoveredPairedDownTypes = map[string][]string{
"node_recovered": {"node_stale", "node_down"},
"host_recovered": {"host_stale", "host_down"},
}
// severityNotifies reports whether a severity triggers email notifications. warning / error / critical
// notify; everything else (info, recovery/status, or an unrecognized value) does not. Pure → unit-tested.
// (Before v0.24.0 a "critical" severity was silently dropped here — the host_disk-class bug.)
@@ -75,6 +97,14 @@ func (d *Dispatcher) ProcessEvent(customerID, eventType, severity, message, deta
return
}
// Recovery branch (v0.71.0, audit F11) — BEFORE the severity gate, as an explicit eventType
// branch: *_recovered stays severity "info" (semantics frozen), but is no longer silent.
// Operator always hears both edges; the customer hears recovery iff they heard the down.
if _, isRecovery := recoveredPairedDownTypes[eventType]; isRecovery {
d.processRecovery(customerID, eventType, severity, message, detailsJSON, source)
return
}
// warning / error / critical trigger notifications. "info" is an intentional non-notify (status/
// recovery events). Anything else is UNRECOGNIZED — log it (don't silently drop), so a bad severity
// surfaces instead of vanishing (the felhom-pve-class lesson: a critical event must never be lost).
@@ -93,22 +123,106 @@ func (d *Dispatcher) ProcessEvent(customerID, eventType, severity, message, deta
}
func (d *Dispatcher) sendTestEmail(customerID string) {
// nil-prefs guard (v0.71.0): GetNotificationPrefs returns (nil, nil) for a customer with no
// notification row — dereferencing prefs.Email here panicked the dispatcher goroutine for such
// a customer (latent since the test leg shipped; found while adding the operator copy).
prefs, err := d.store.GetNotificationPrefs(customerID)
if err != nil || prefs.Email == "" {
if err != nil || prefs == nil || prefs.Email == "" {
d.logger.Printf("[WARN] Test email: no email configured for %s", customerID)
} else {
subject := "[Felhom] Teszt értesítés"
body := "Kedves Ügyfél!\n\nEz egy teszt értesítés a Felhom monitoring rendszerből.\nAz értesítések megfelelően működnek.\n\nÜdvözlettel,\nFelhom.eu monitoring"
if err := d.sendEmailFn(prefs.Email, subject, body, nil); err != nil {
d.logger.Printf("[ERROR] Test email to %s failed: %v", prefs.Email, err)
d.store.LogNotification(customerID, "test", "info", "Teszt értesítés", "failed", err.Error(), "customer")
} else {
d.logger.Printf("[INFO] Test email sent to %s for %s", prefs.Email, customerID)
d.store.LogNotification(customerID, "test", "info", "Teszt értesítés", "sent", "", "customer")
}
}
// Operator copy (v0.71.0, audit F14-light): one test click proves the customer channel, the
// operator channel AND the high-priority header rendering in a single shot.
if !d.operatorOn || d.operatorEmail == "" {
return
}
opSubject := fmt.Sprintf("[Felhom] ✅ %s: teszt / operator channel OK", customerID)
opBody := fmt.Sprintf(`Operator copy of the customer notification test for %s.
If this mail shows as high priority in your client, the X-Priority/Importance
headers render correctly. The customer test mail result is recorded in the
notification log.
Dashboard: https://hub.felhom.eu/customers/%s`, customerID, customerID)
if err := d.sendEmailFn(d.operatorEmail, opSubject, opBody, priorityHeaders("critical")); err != nil {
d.logger.Printf("[ERROR] Operator test email failed for %s: %v", customerID, err)
d.store.LogNotification(customerID, "test", "info", "operator test copy", "failed", err.Error(), "operator")
return
}
d.logger.Printf("[INFO] Operator test email sent for %s", customerID)
d.store.LogNotification(customerID, "test", "info", "operator test copy", "sent", "", "operator")
}
// processRecovery routes a *_recovered event (v0.71.0, audit F11). Severity semantics stay frozen
// ("info" everywhere else remains non-notify) — this is an explicit eventType branch.
// - Operator leg: always wanted (both edges), gated only by operatorOn + the 1h per-type
// cooldown — exactly processOperator.
// - Customer leg: gated by the PAIRING rule, not enabled_events — "recovery notifies exactly
// whoever the down notified." Evidence = a customer-channel status=sent row for the paired
// stale/down set newer than the last customer-channel sent recovery of this type.
func (d *Dispatcher) processRecovery(customerID, eventType, severity, message, detailsJSON, source string) {
d.processOperator(customerID, eventType, severity, message, detailsJSON, source)
if d.store.IsCustomerBlocked(customerID) {
return
}
prefs, err := d.store.GetNotificationPrefs(customerID)
if err != nil || prefs == nil || prefs.Email == "" {
return
}
subject := "[Felhom] Teszt értesítés"
body := "Kedves Ügyfél!\n\nEz egy teszt értesítés a Felhom monitoring rendszerből.\nAz értesítések megfelelően működnek.\n\nÜdvözlettel,\nFelhom.eu monitoring"
if err := d.sendEmailFn(prefs.Email, subject, body); err != nil {
d.logger.Printf("[ERROR] Test email to %s failed: %v", prefs.Email, err)
d.store.LogNotification(customerID, "test", "info", "Teszt értesítés", "failed", err.Error(), "customer")
// Pairing check — the customer gate. enabled_events is deliberately ignored here: a customer
// who was told "down" must be told "recovered", and one who wasn't must not be.
lastDown, downOk, err := d.store.LastCustomerSentAt(customerID, recoveredPairedDownTypes[eventType])
if err != nil {
d.logger.Printf("[ERROR] Recovery pairing query failed for %s/%s: %v", customerID, eventType, err)
return
}
d.logger.Printf("[INFO] Test email sent to %s for %s", prefs.Email, customerID)
d.store.LogNotification(customerID, "test", "info", "Teszt értesítés", "sent", "", "customer")
lastRecovered, recOk, err := d.store.LastCustomerSentAt(customerID, []string{eventType})
if err != nil {
d.logger.Printf("[ERROR] Recovery pairing query failed for %s/%s: %v", customerID, eventType, err)
return
}
// Second-granularity ties resolve to NOT-after → no mail (flap-safe direction).
if !downOk || (recOk && !lastDown.After(lastRecovered)) {
d.logger.Printf("[INFO] Recovery %s for %s: customer mail skipped — no unanswered customer down mail (pairing miss)", eventType, customerID)
return
}
// Prefs cooldown keyed on the recovered eventType (belt over the pairing braces).
cooldownHours := prefs.CooldownHours
if cooldownHours <= 0 {
cooldownHours = 6
}
cooldownKey := customerID + ":" + eventType
d.mu.Lock()
if last, ok := d.custCooldowns[cooldownKey]; ok && time.Since(last) < time.Duration(cooldownHours)*time.Hour {
d.mu.Unlock()
d.logger.Printf("[INFO] Recovery %s for %s: customer mail skipped — cooldown", eventType, customerID)
return
}
d.custCooldowns[cooldownKey] = time.Now()
d.mu.Unlock()
subject, body := FormatCustomerEmail(customerID, eventType, severity, message, detailsJSON)
if err := d.sendEmailFn(prefs.Email, subject, body, priorityHeaders(severity)); err != nil {
d.logger.Printf("[ERROR] Customer recovery email failed for %s/%s: %v", customerID, eventType, err)
d.store.LogNotification(customerID, eventType, severity, message, "failed", err.Error(), "customer")
return
}
d.logger.Printf("[INFO] Customer recovery email sent to %s for %s/%s", prefs.Email, customerID, eventType)
d.store.LogNotification(customerID, eventType, severity, message, "sent", "", "customer")
}
func (d *Dispatcher) processOperator(customerID, eventType, severity, message, detailsJSON, source string) {
@@ -127,7 +241,7 @@ func (d *Dispatcher) processOperator(customerID, eventType, severity, message, d
subject, body := FormatOperatorEmail(customerID, eventType, severity, message, detailsJSON)
if err := d.sendEmailFn(d.operatorEmail, subject, body); err != nil {
if err := d.sendEmailFn(d.operatorEmail, subject, body, priorityHeaders(severity)); err != nil {
d.logger.Printf("[ERROR] Operator email failed for %s/%s: %v", customerID, eventType, err)
d.store.LogNotification(customerID, eventType, severity, message, "failed", err.Error(), "operator")
return
@@ -173,7 +287,7 @@ func (d *Dispatcher) processCustomer(customerID, eventType, severity, message, d
subject, body := FormatCustomerEmail(customerID, eventType, severity, message, detailsJSON)
if err := d.sendEmailFn(prefs.Email, subject, body); err != nil {
if err := d.sendEmailFn(prefs.Email, subject, body, priorityHeaders(severity)); err != nil {
d.logger.Printf("[ERROR] Customer email failed for %s/%s: %v", customerID, eventType, err)
d.store.LogNotification(customerID, eventType, severity, message, "failed", err.Error(), "customer")
return
@@ -182,13 +296,16 @@ func (d *Dispatcher) processCustomer(customerID, eventType, severity, message, d
d.store.LogNotification(customerID, eventType, severity, message, "sent", "", "customer")
}
func (d *Dispatcher) sendEmail(to, subject, textBody string) error {
func (d *Dispatcher) sendEmail(to, subject, textBody string, headers map[string]string) error {
payload := map[string]interface{}{
"from": d.fromEmail,
"to": []string{to},
"subject": subject,
"text": textBody,
}
if len(headers) > 0 {
payload["headers"] = headers
}
jsonData, err := json.Marshal(payload)
if err != nil {
@@ -236,7 +353,7 @@ func (d *Dispatcher) SendClaimEmail(kind, customerID, email, domain, code string
}
subject, body := FormatClaimEmail(kind, customerID, domain, code)
eventType := "claim_" + kind
if err := d.sendEmailFn(email, subject, body); err != nil {
if err := d.sendEmailFn(email, subject, body, nil); err != nil {
d.logger.Printf("[ERROR] claim %s email to customer %s failed: %v", kind, customerID, err)
d.store.LogNotification(customerID, eventType, "info", subject, "failed", err.Error(), "customer")
return err
@@ -257,7 +374,7 @@ func (d *Dispatcher) SendSelfBindEmail(customerID, email, link string) error {
return fmt.Errorf("notify: no resend api key")
}
subject, body := FormatSelfBindEmail(customerID, link)
if err := d.sendEmailFn(email, subject, body); err != nil {
if err := d.sendEmailFn(email, subject, body, nil); err != nil {
d.logger.Printf("[ERROR] self-bind link email to customer %s failed: %v", customerID, err)
d.store.LogNotification(customerID, "selfbind_link", "info", subject, "failed", err.Error(), "customer")
return err
@@ -0,0 +1,264 @@
package notify
import (
"io"
"log"
"strings"
"sync"
"testing"
)
// capturedMail is one seam-captured send: recipient, subject and headers.
type capturedMail struct {
to string
subject string
headers map[string]string
}
// captureSeam installs a capturing sendEmailFn and returns the capture slice pointer.
func captureSeam(d *Dispatcher) *[]capturedMail {
var mu sync.Mutex
sent := &[]capturedMail{}
d.sendEmailFn = func(to, subject, _ string, headers map[string]string) error {
mu.Lock()
defer mu.Unlock()
*sent = append(*sent, capturedMail{to: to, subject: subject, headers: headers})
return nil
}
return sent
}
func mailsFor(sent []capturedMail, to string) []capturedMail {
var out []capturedMail
for _, m := range sent {
if m.to == to {
out = append(out, m)
}
}
return out
}
// TestRecovery_PairedCustomerMail (Scenario A — the doodoo21 case, fixed): a customer who was
// mailed node_down gets the node_recovered mail; the operator gets the ✅ mail; both are logged.
func TestRecovery_PairedCustomerMail(t *testing.T) {
st := newDispStore(t)
if err := st.SaveNotificationPrefs("c1", "cust@example.com", []string{"node_down"}, 6); err != nil {
t.Fatal(err)
}
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
sent := captureSeam(d)
// The down edge: customer (enabled) + operator both mailed.
d.ProcessEvent("c1", "node_down", "error", "No report received for 1h", "{}", "hub")
if len(*sent) != 2 {
t.Fatalf("down edge must mail operator + customer, got %d sends", len(*sent))
}
// The recovery edge.
d.ProcessEvent("c1", "node_recovered", "info", "Reports resumed (was down for 3h48m)", "{}", "hub")
cust := mailsFor(*sent, "cust@example.com")
if len(cust) != 2 {
t.Fatalf("customer must get down + recovery, got %d customer mails", len(cust))
}
recMail := cust[1]
if !strings.Contains(recMail.subject, "Információ: A szerver újra elérhető.") {
t.Fatalf("customer recovery subject wrong: %q", recMail.subject)
}
op := mailsFor(*sent, "op@felhom.eu")
if len(op) != 2 {
t.Fatalf("operator must get down + recovery, got %d", len(op))
}
if !strings.Contains(op[1].subject, "✅") || !strings.Contains(op[1].subject, "node_recovered") {
t.Fatalf("operator recovery subject must carry ✅ + node_recovered: %q", op[1].subject)
}
// notification_log holds sent rows on both channels for node_recovered.
entries, err := st.GetRecentNotifications("c1", 10)
if err != nil {
t.Fatal(err)
}
var gotCust, gotOp bool
for _, e := range entries {
if e.EventType == "node_recovered" && e.Status == "sent" {
switch e.Channel {
case "customer":
gotCust = true
case "operator":
gotOp = true
}
}
}
if !gotCust || !gotOp {
t.Fatalf("node_recovered sent rows missing (customer=%v operator=%v)", gotCust, gotOp)
}
}
// TestRecovery_UnpairedStaysCustomerSilent (Scenario B): no customer-channel down evidence → the
// operator is mailed, the customer is NOT — even with node_recovered in enabled_events (the
// pairing rule is THE customer gate; enabled_events is deliberately ignored for recovery).
// Companion red-proof: removing the pairing check in processRecovery makes this fail.
func TestRecovery_UnpairedStaysCustomerSilent(t *testing.T) {
st := newDispStore(t)
if err := st.SaveNotificationPrefs("c1", "cust@example.com", []string{"node_down", "node_recovered"}, 6); err != nil {
t.Fatal(err)
}
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
sent := captureSeam(d)
d.ProcessEvent("c1", "node_recovered", "info", "Reports resumed", "{}", "hub")
if n := len(mailsFor(*sent, "cust@example.com")); n != 0 {
t.Fatalf("unpaired recovery must NOT mail the customer, got %d", n)
}
if n := len(mailsFor(*sent, "op@felhom.eu")); n != 1 {
t.Fatalf("operator must still get the recovery mail, got %d", n)
}
}
// TestRecovery_FlapDamping (Scenario C): down mailed → recovery mailed → second down SUPPRESSED by
// the customer cooldown → second recovery must NOT mail the customer (the suppressed down left no
// fresh pairing evidence), and the operator recovery mail obeys the 1h per-type cooldown.
func TestRecovery_FlapDamping(t *testing.T) {
st := newDispStore(t)
if err := st.SaveNotificationPrefs("c1", "cust@example.com", []string{"node_down"}, 6); err != nil {
t.Fatal(err)
}
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
sent := captureSeam(d)
d.ProcessEvent("c1", "node_down", "error", "down", "{}", "hub") // down #1: cust+op mailed
d.ProcessEvent("c1", "node_recovered", "info", "up", "{}", "hub") // recovery #1: cust+op mailed
d.ProcessEvent("c1", "node_down", "error", "down", "{}", "hub") // down #2 10min later: customer
// cooldown suppresses (op too, 1h)
// Clear the recovery CUSTOMER cooldown so the pairing gate — not the cooldown — decides #2:
d.mu.Lock()
delete(d.custCooldowns, "c1:node_recovered")
d.mu.Unlock()
d.ProcessEvent("c1", "node_recovered", "info", "up", "{}", "hub") // recovery #2
cust := mailsFor(*sent, "cust@example.com")
if len(cust) != 2 { // down #1 + recovery #1 only
t.Fatalf("second recovery must not mail the customer (no fresh pairing), customer mails=%d: %+v", len(cust), cust)
}
// Operator: down #1 + recovery #1; down #2 and recovery #2 suppressed by the 1h op cooldown.
if op := mailsFor(*sent, "op@felhom.eu"); len(op) != 2 {
t.Fatalf("operator recovery must obey the 1h cooldown, op mails=%d", len(op))
}
}
// TestTestEvent_BothChannels (Scenario F): a "test" event mails the customer (existing copy) AND
// the operator, the operator mail carrying both priority headers; both logged.
func TestTestEvent_BothChannels(t *testing.T) {
st := newDispStore(t)
if err := st.SaveNotificationPrefs("c1", "cust@example.com", nil, 6); err != nil {
t.Fatal(err)
}
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
sent := captureSeam(d)
d.ProcessEvent("c1", "test", "info", "", "{}", "hub")
cust := mailsFor(*sent, "cust@example.com")
if len(cust) != 1 || !strings.Contains(cust[0].subject, "Teszt értesítés") {
t.Fatalf("customer test mail wrong: %+v", cust)
}
op := mailsFor(*sent, "op@felhom.eu")
if len(op) != 1 || !strings.Contains(op[0].subject, "✅") || !strings.Contains(op[0].subject, "operator channel OK") {
t.Fatalf("operator test mail wrong: %+v", op)
}
if op[0].headers["X-Priority"] != "1" || op[0].headers["Importance"] != "high" {
t.Fatalf("operator test mail must carry priority headers, got %v", op[0].headers)
}
entries, _ := st.GetRecentNotifications("c1", 10)
var chans []string
for _, e := range entries {
if e.EventType == "test" && e.Status == "sent" {
chans = append(chans, e.Channel)
}
}
if len(chans) != 2 {
t.Fatalf("both test sends must be logged, got channels %v", chans)
}
}
// TestTestEvent_NoPrefsRow_NoPanic: a test event for a customer WITHOUT a notification row must
// not panic (the nil-prefs deref found in v0.71.0 — GetNotificationPrefs returns (nil, nil)) and
// the operator copy still proves the operator channel.
func TestTestEvent_NoPrefsRow_NoPanic(t *testing.T) {
st := newDispStore(t)
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
sent := captureSeam(d)
d.ProcessEvent("c1", "test", "info", "", "{}", "hub") // must not panic
if n := len(mailsFor(*sent, "op@felhom.eu")); n != 1 {
t.Fatalf("operator test copy must send even without customer prefs, got %d", n)
}
if n := len(*sent); n != 1 {
t.Fatalf("no customer mail possible without prefs, total sends=%d", n)
}
}
// TestPriorityHeaders (Scenario G): error/critical carry the two headers; warning/info/unknown nil.
// The negative half is the red-proof target: adding headers unconditionally fails the nil cases.
func TestPriorityHeaders(t *testing.T) {
for _, sev := range []string{"error", "critical"} {
h := priorityHeaders(sev)
if h["X-Priority"] != "1" || h["Importance"] != "high" || len(h) != 2 {
t.Errorf("%s: headers = %v", sev, h)
}
}
for _, sev := range []string{"warning", "info", "", "frobnicate"} {
if h := priorityHeaders(sev); h != nil {
t.Errorf("%s must have NO priority headers, got %v", sev, h)
}
}
}
// TestPriorityHeaders_EndToEnd (Scenario G, seam level): an error mail carries headers on both
// channels; a warning mail carries none.
func TestPriorityHeaders_EndToEnd(t *testing.T) {
st := newDispStore(t)
if err := st.SaveNotificationPrefs("c1", "cust@example.com", []string{"node_down", "node_stale"}, 6); err != nil {
t.Fatal(err)
}
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
sent := captureSeam(d)
d.ProcessEvent("c1", "node_down", "error", "down", "{}", "hub")
for _, m := range *sent {
if m.headers["X-Priority"] != "1" || m.headers["Importance"] != "high" {
t.Fatalf("error mail to %s must carry priority headers, got %v", m.to, m.headers)
}
}
*sent = (*sent)[:0]
d.ProcessEvent("c1", "node_stale", "warning", "stale", "{}", "hub")
if len(*sent) == 0 {
t.Fatal("warning must still send")
}
for _, m := range *sent {
if m.headers != nil {
t.Fatalf("warning mail to %s must carry NO headers, got %v", m.to, m.headers)
}
}
}
// TestRecovery_SeverityStaysInfo guards the frozen semantics: the recovery branch must not have
// widened severityNotifies — a non-recovery info event is still silent.
func TestRecovery_SeverityStaysInfo(t *testing.T) {
if severityNotifies("info") {
t.Fatal("info must remain non-notify — recovery is an eventType branch, not a severity change")
}
st := newDispStore(t)
if err := st.SaveNotificationPrefs("c1", "cust@example.com", []string{"controller_started"}, 6); err != nil {
t.Fatal(err)
}
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
sent := captureSeam(d)
d.ProcessEvent("c1", "controller_started", "info", "started", "{}", "hub")
if len(*sent) != 0 {
t.Fatalf("non-recovery info must stay silent, got %d sends", len(*sent))
}
}
+5 -5
View File
@@ -56,7 +56,7 @@ func TestProcessEvent_PoolBoxScopeOperatorOnly(t *testing.T) {
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
var mu sync.Mutex
var sent []string
d.sendEmailFn = func(to, _, _ string) error { mu.Lock(); defer mu.Unlock(); sent = append(sent, to); return nil }
d.sendEmailFn = func(to, _, _ string, _ map[string]string) error { mu.Lock(); defer mu.Unlock(); sent = append(sent, to); return nil }
d.ProcessEvent("pool-box", "offsite_box_fill", "warning", "Offsite pool box 82% full", `{"scope":"pool-box"}`, "hub")
if len(sent) != 1 || sent[0] != "op@felhom.eu" {
@@ -71,7 +71,7 @@ func TestProcessEvent_PBSDRBoxScopeOperatorOnly(t *testing.T) {
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
var mu sync.Mutex
var sent []string
d.sendEmailFn = func(to, _, _ string) error { mu.Lock(); defer mu.Unlock(); sent = append(sent, to); return nil }
d.sendEmailFn = func(to, _, _ string, _ map[string]string) error { mu.Lock(); defer mu.Unlock(); sent = append(sent, to); return nil }
d.ProcessEvent("pbsdr-box", "pbsdr_box_fill", "warning", "PBS DR datastore 82% full", `{"scope":"pbsdr-box"}`, "hub")
if len(sent) != 1 || sent[0] != "op@felhom.eu" {
@@ -85,7 +85,7 @@ func TestProcessEvent_CriticalRoutes(t *testing.T) {
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
var mu sync.Mutex
var sent []string
d.sendEmailFn = func(to, _, _ string) error { mu.Lock(); defer mu.Unlock(); sent = append(sent, to); return nil }
d.sendEmailFn = func(to, _, _ string, _ map[string]string) error { mu.Lock(); defer mu.Unlock(); sent = append(sent, to); return nil }
d.ProcessEvent("c1", "host_disk_critical", "critical", "root full", "{}", "hub")
if len(sent) != 1 || sent[0] != "op@felhom.eu" {
@@ -100,7 +100,7 @@ func TestProcessEvent_UnknownSeverityLogged(t *testing.T) {
var buf bytes.Buffer
d := NewDispatcher(st, "test-key", "from", "op@felhom.eu", true, log.New(&buf, "", 0))
sent := 0
d.sendEmailFn = func(_, _, _ string) error { sent++; return nil }
d.sendEmailFn = func(_, _, _ string, _ map[string]string) error { sent++; return nil }
d.ProcessEvent("c1", "weird_event", "frobnicate", "msg", "{}", "hub")
if sent != 0 {
@@ -118,7 +118,7 @@ func TestProcessEvent_InfoSilent(t *testing.T) {
var buf bytes.Buffer
d := NewDispatcher(st, "test-key", "from", "op@felhom.eu", true, log.New(&buf, "", 0))
sent := 0
d.sendEmailFn = func(_, _, _ string) error { sent++; return nil }
d.sendEmailFn = func(_, _, _ string, _ map[string]string) error { sent++; return nil }
d.ProcessEvent("c1", "controller_started", "info", "msg", "{}", "hub")
if sent != 0 {
+7 -1
View File
@@ -2,6 +2,7 @@ package notify
import (
"fmt"
"strings"
"time"
)
@@ -22,8 +23,12 @@ func init() {
// FormatOperatorEmail returns (subject, textBody) for the operator channel.
func FormatOperatorEmail(customerID, eventType, severity, message, detailsJSON string) (string, string) {
// Icon is eventType-aware for recovery (v0.71.0, audit F11); severity stays the fallback.
icon := "⚠️"
if severity == "error" || severity == "critical" {
switch {
case strings.HasSuffix(eventType, "_recovered"):
icon = "✅"
case severity == "error" || severity == "critical":
icon = "🔴"
}
@@ -86,6 +91,7 @@ var customerMessages = map[string]string{
"node_stale": "A szerver nem küldött jelentést az elmúlt időszakban.",
"node_down": "A szerver nem elérhető!",
"node_recovered": "A szerver újra elérhető.",
"host_recovered": "A házszerver alaprendszere (Proxmox-gazda) újra elérhető.",
// Health events
"health_degraded": "A rendszer állapota romlott.",
@@ -0,0 +1,115 @@
package store
import (
"testing"
"time"
)
// insertNotifLog inserts a notification_log row with an EXPLICIT created_at so pairing tests can
// order rows deterministically (datetime('now') is second-granularity — real calls tie).
func insertNotifLog(t *testing.T, s *Store, customerID, eventType, status, channel, createdAt string) {
t.Helper()
if _, err := s.db.Exec(`
INSERT INTO notification_log (customer_id, event_type, severity, message, status, error_message, channel, created_at)
VALUES (?, ?, 'error', 'm', ?, '', ?, ?)`,
customerID, eventType, status, channel, createdAt); err != nil {
t.Fatalf("insertNotifLog: %v", err)
}
}
// TestLastCustomerSentAt_PairingQuery (v0.71.0 F11): the pairing-evidence query returns the max
// created_at over customer-channel status=sent rows of the given types ONLY — operator rows,
// failed rows and other event types must not count.
func TestLastCustomerSentAt_PairingQuery(t *testing.T) {
s := newTestStore(t)
// Noise that must NOT count:
insertNotifLog(t, s, "c1", "node_down", "sent", "operator", "2026-07-22 15:00:00") // wrong channel
insertNotifLog(t, s, "c1", "node_down", "failed", "customer", "2026-07-22 16:00:00") // wrong status
insertNotifLog(t, s, "c1", "backup_failed", "sent", "customer", "2026-07-22 17:00:00") // wrong type
insertNotifLog(t, s, "c2", "node_down", "sent", "customer", "2026-07-22 18:00:00") // wrong customer
// No qualifying row yet:
_, ok, err := s.LastCustomerSentAt("c1", []string{"node_stale", "node_down"})
if err != nil {
t.Fatalf("LastCustomerSentAt: %v", err)
}
if ok {
t.Fatal("no qualifying row must yield ok=false")
}
// Two qualifying rows — max wins:
insertNotifLog(t, s, "c1", "node_stale", "sent", "customer", "2026-07-22 12:59:00")
insertNotifLog(t, s, "c1", "node_down", "sent", "customer", "2026-07-22 13:29:00")
got, ok, err := s.LastCustomerSentAt("c1", []string{"node_stale", "node_down"})
if err != nil {
t.Fatalf("LastCustomerSentAt: %v", err)
}
if !ok {
t.Fatal("qualifying rows exist — ok must be true")
}
want := time.Date(2026, 7, 22, 13, 29, 0, 0, time.UTC)
if !got.Equal(want) {
t.Fatalf("max created_at = %v, want %v", got, want)
}
// Empty type list is a defined no-op:
if _, ok, err := s.LastCustomerSentAt("c1", nil); err != nil || ok {
t.Fatalf("empty eventTypes must return (zero, false, nil), got ok=%v err=%v", ok, err)
}
}
// TestSeedNotificationPrefs_InsertIfAbsent (v0.71.0 F12, Scenario D): the seed creates a row when
// none exists, and NEVER modifies a pre-existing row (insert-if-absent, not upsert). Companion
// red-proof: replacing the INSERT OR IGNORE with SaveNotificationPrefs makes the second half fail.
func TestSeedNotificationPrefs_InsertIfAbsent(t *testing.T) {
s := newTestStore(t)
defaults := []string{"node_down", "backup_failed", "disk_critical"}
seeded, err := s.SeedNotificationPrefs("c1", "x@example.com", defaults)
if err != nil {
t.Fatalf("seed: %v", err)
}
if !seeded {
t.Fatal("first seed must create a row")
}
prefs, err := s.GetNotificationPrefs("c1")
if err != nil || prefs == nil {
t.Fatalf("prefs after seed: %v %v", prefs, err)
}
if prefs.Email != "x@example.com" || prefs.CooldownHours != 6 || len(prefs.EnabledEvents) != 3 {
t.Fatalf("seeded row wrong: %+v", prefs)
}
// A customer-edited row must survive a later seed byte-identical:
if err := s.SaveNotificationPrefs("c1", "edited@example.com", []string{"node_down"}, 12); err != nil {
t.Fatalf("customer edit: %v", err)
}
seeded, err = s.SeedNotificationPrefs("c1", "reinstall@example.com", defaults)
if err != nil {
t.Fatalf("re-seed: %v", err)
}
if seeded {
t.Fatal("seed over an existing row must report seeded=false")
}
prefs, _ = s.GetNotificationPrefs("c1")
if prefs.Email != "edited@example.com" || prefs.CooldownHours != 12 || len(prefs.EnabledEvents) != 1 {
t.Fatalf("seed MODIFIED an existing row (upsert bug): %+v", prefs)
}
}
// TestSeedNotificationPrefs_EmptyEmailNoop: an empty registered email must not seed an
// unnotifiable row.
func TestSeedNotificationPrefs_EmptyEmailNoop(t *testing.T) {
s := newTestStore(t)
seeded, err := s.SeedNotificationPrefs("c1", "", []string{"node_down"})
if err != nil {
t.Fatalf("seed: %v", err)
}
if seeded {
t.Fatal("empty email must be a no-op")
}
if prefs, _ := s.GetNotificationPrefs("c1"); prefs != nil {
t.Fatalf("no row must exist, got %+v", prefs)
}
}
+57
View File
@@ -7,6 +7,7 @@ import (
"fmt"
"log"
"strconv"
"strings"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/semver"
@@ -811,6 +812,62 @@ func (s *Store) GetRecentNotifications(customerID string, limit int) ([]Notifica
return entries, rows.Err()
}
// LastCustomerSentAt returns the most recent notification_log created_at over the given event
// types on the CUSTOMER channel with status='sent', and whether any such row exists. It is the
// pairing-evidence query for recovery notifications (v0.71.0, audit F11): "was the customer told
// about the down since they were last told about a recovery?" Uses the
// (customer_id, created_at DESC) index. An empty eventTypes slice returns (zero, false, nil).
func (s *Store) LastCustomerSentAt(customerID string, eventTypes []string) (time.Time, bool, error) {
if len(eventTypes) == 0 {
return time.Time{}, false, nil
}
placeholders := make([]string, len(eventTypes))
args := make([]interface{}, 0, len(eventTypes)+1)
args = append(args, customerID)
for i, et := range eventTypes {
placeholders[i] = "?"
args = append(args, et)
}
var createdAt sql.NullString
err := s.db.QueryRow(`
SELECT MAX(created_at) FROM notification_log
WHERE customer_id = ? AND channel = 'customer' AND status = 'sent'
AND event_type IN (`+strings.Join(placeholders, ",")+`)`,
args...,
).Scan(&createdAt)
if err != nil {
return time.Time{}, false, err
}
if !createdAt.Valid || createdAt.String == "" {
return time.Time{}, false, nil
}
return parseSQLiteTime(createdAt.String), true, nil
}
// SeedNotificationPrefs creates a customer_notifications row IF AND ONLY IF none exists —
// insert-if-absent, never an upsert (a customer-edited row must never be overwritten by a seed;
// audit F12). An empty email is a no-op: seeding an unnotifiable row would only mask the gap.
// Returns whether a row was created.
func (s *Store) SeedNotificationPrefs(customerID, email string, enabledEvents []string) (bool, error) {
if email == "" {
return false, nil
}
eventsJSON, _ := json.Marshal(enabledEvents)
res, err := s.db.Exec(`
INSERT OR IGNORE INTO customer_notifications (customer_id, email, enabled_events, cooldown_hours)
VALUES (?, ?, ?, 6)`,
customerID, email, string(eventsJSON),
)
if err != nil {
return false, err
}
n, err := res.RowsAffected()
if err != nil {
return false, err
}
return n > 0, nil
}
// SaveReport stores a new report. The reportJSON should be the raw JSON payload.
func (s *Store) SaveReport(customerID string, reportJSON []byte) error {
// Parse denormalized fields from the JSON