diff --git a/CONTEXT.md b/CONTEXT.md index f046214..85b260e 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,6 +3,24 @@ > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. +- **2026-07-22 — hub v0.71.0: the notification train (audit F11+F12+F14-light).** Four rulings now + standing: **(1) recovery pairing** — `*_recovered` notifies the operator always and the customer + **iff the customer was mailed the paired stale/down** (evidence = `notification_log` customer- + channel sent rows via `store.LastCustomerSentAt`; `enabled_events` deliberately ignored for + recovery; ties → no mail). Severity semantics FROZEN — recovery stays `info`, routed by an + explicit eventType branch before the severity gate. **(2) seed-at-claim** — `MarkClaimed` seeds + `customer_notifications` from the registered email, INSERT-if-absent only (never touches an + edited row; empty email = no-op; seed failure never fails the claim); default set is + critical-only: node_down, backup_failed, disk_critical, host_disk_critical, + storage_fill_critical, offbox_repo_orphaned. **(3) empty-email no-clobber** — a prefs push with + empty email preserves a stored address (hub-side belt; controller 0.160.0 already guards its two + push legs). **(4) priority headers** — error/critical mails carry `X-Priority: 1` + + `Importance: high` (Resend `headers`, live-probed); the `test` event now also mails the operator + with those headers (one click proves both channels + rendering). Latent nil-deref fixed: + `sendTestEmail` panicked on a customer with no prefs row. NOT yet live: a natural `*_recovered` + mail (next real staleness cycle / reboot drill — never fabricate one by blocking reports) and + seed-at-claim on a real claim (Peti's Friday reinstall is the natural candidate). + - **2026-07-22 — hub v0.70.1: the ghost customer's Delete button exists now.** The v0.70.0 ghost-delete path was fully implemented and fully unreachable — the **fourth inert-seam defect**, this time a TEMPLATE GATE: the Danger-zone card (and the `customerDeleteOpen` script) sat inside diff --git a/REUSE.md b/REUSE.md index eecb3ad..f9b63ff 100644 --- a/REUSE.md +++ b/REUSE.md @@ -26,11 +26,16 @@ | Symbol | File | Short signature | Use for | Gotchas | |---|---|---|---|---| -| `(*Dispatcher).ProcessEvent` | hub/internal/notify/dispatcher.go (~L67) | `(customerID, eventType, severity, message, detailsJSON, source)` | THE notification pipeline (operator + customer channels, cooldowns, prefs) | Call in a goroutine (handlers do `go d.ProcessEvent(...)`). No Resend key → silent no-op. `eventType=="test"` bypasses prefs/cooldown. | -| `severityNotifies` | hub/internal/notify/dispatcher.go (~L56) | `(severity string) bool` | Deciding whether a severity emails | warning/error/critical notify; info intentionally doesn't; anything else is logged as unrecognized (v0.24.0 fix — do not regress). | -| `FormatOperatorEmail` / `FormatCustomerEmail` | hub/internal/notify/templates.go (~L24 / ~L118) | `(...) (subject, body)` | Operator (English) / customer (Hungarian) email bodies | Customer messages come from the `customerMessages` map — add the Hungarian text when adding an event type. Budapest TZ via package `init()`. | +| `(*Dispatcher).ProcessEvent` | hub/internal/notify/dispatcher.go (~L88) | `(customerID, eventType, severity, message, detailsJSON, source)` | THE notification pipeline (operator + customer channels, cooldowns, prefs) | Call in a goroutine (handlers do `go d.ProcessEvent(...)`). No Resend key → silent no-op. `eventType=="test"` bypasses prefs/cooldown (and since v0.71.0 also mails the operator). `*_recovered` routes via the explicit recovery branch BEFORE the severity gate (v0.71.0) — see `processRecovery`. | +| `(*Dispatcher).processRecovery` | hub/internal/notify/dispatcher.go (~L160) | internal | `node_recovered`/`host_recovered` routing (audit F11) | Operator always (1 h cooldown); customer iff PAIRED — a customer-channel `sent` stale/down row newer than the last sent recovery (`store.LastCustomerSentAt`); `enabled_events` deliberately ignored for recovery; timestamp ties → no mail (flap-safe). Severity stays `info` — never "fix" that by widening `severityNotifies`. | +| `severityNotifies` | hub/internal/notify/dispatcher.go (~L77) | `(severity string) bool` | Deciding whether a severity emails | warning/error/critical notify; info intentionally doesn't; anything else is logged as unrecognized (v0.24.0 fix — do not regress). Recovery mails exist DESPITE this gate (eventType branch), not through it. | +| `priorityHeaders` | hub/internal/notify/dispatcher.go (~L56) | `(severity string) map[string]string` | High-priority mail-client nudge (audit F14-light) | error/critical → `X-Priority: 1` + `Importance: high`; everything else nil — a warning/info mail must NOT masquerade as urgent (red-proofed). | +| `sendEmailFn` seam / `sendEmail` | hub/internal/notify/dispatcher.go (~L33 / ~L300) | `func(to, subject, textBody string, headers map[string]string) error` | Test seam for all sends; Resend POST | Signature grew a `headers` param in v0.71.0 — payload carries `"headers"` only when non-empty. Tests capture recipient+subject+headers through the seam. | +| `FormatOperatorEmail` / `FormatCustomerEmail` | hub/internal/notify/templates.go (~L24 / ~L118) | `(...) (subject, body)` | Operator (English) / customer (Hungarian) email bodies | Customer messages come from the `customerMessages` map — add the Hungarian text when adding an event type. Budapest TZ via package `init()`. Operator icon is eventType-aware: `*_recovered` → ✅ (severity is the fallback). | | `monitor.EventNotifyFunc` | hub/internal/monitor/staleness.go (~L14) | `func(customerID, eventType, severity, message, detailsJSON, source)` | Decoupling checkers from notify; wired to `dispatcher.ProcessEvent` in main | May be nil — always nil-check before calling (all checkers do). | -| `(*Store).LogNotification` | hub/internal/store/store.go (~L433) | `(customerID, eventType, severity, message, status, errorMsg, channel)` | Audit trail of every send attempt (sent/failed, per channel) | Log BOTH success and failure (dispatcher does). | +| `(*Store).LogNotification` | hub/internal/store/store.go (~L433) | `(customerID, eventType, severity, message, status, errorMsg, channel)` | Audit trail of every send attempt (sent/failed, per channel) | Log BOTH success and failure (dispatcher does). Since v0.71.0 these rows are also the recovery PAIRING evidence — never prune them casually. | +| `(*Store).LastCustomerSentAt` | hub/internal/store/store.go (~L815) | `(customerID, eventTypes []string) (time.Time, bool, error)` | Pairing-evidence query (max customer-channel `sent` created_at over types) | Uses the `(customer_id, created_at DESC)` index. Empty type list → `(zero, false, nil)`. | +| `(*Store).SeedNotificationPrefs` | hub/internal/store/store.go (~L850) | `(customerID, email, enabledEvents) (seeded bool, err)` | Claim-time prefs seeding (audit F12) | INSERT OR IGNORE — never an upsert (red-proofed); empty email = no-op. Customer edits go through `SaveNotificationPrefs`, seeds NEVER do. | ### App-mail passthrough (hub/internal/mailrelay/) diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index 9321b77..f7089de 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -104,7 +104,8 @@ |---|---|---|---|---| | Health-degradation email (edge-triggered, cooldowns, Hungarian) via hub → Resend | controller, hub | **IMPLEMENTED** | delivery pipeline live-proven for the **enlarge-block** trigger (`CAMPAIGN-6D` P3-DELIVERY, op+customer "Kedves Ügyfél!"); `NotifyHealthChange` ok→warn/fail edge-trigger implemented | The **health-degradation** trigger specifically has never fired an email live in any doc. Demoted (pipeline proven for a different event). Deliverability to HU freemail → R-4 | | Event catalog: app_start_failed, dead-app, offbox_enlarge_blocked, claim/reset codes, critical severity | controller, hub v0.31/48/50/55 | **PROVEN-LIVE** | live-delivered: `CAMPAIGN-6D` P3-DELIVERY (enlarge-block, op+customer); `DRILL-day0-vm` F-4 (claim code); `DRILL-day0-take2` F-15 (reset code) | `app_start_failed`/`dead-app` delivery is unit-only (6C inconclusive) — the pipeline + 3 event families are live, those two are not | -| Prefs safety: empty-email wipe guard | controller v0.137 | **IMPLEMENTED** | red-proofed 07-15 | Born from a live incident; guard itself unit-proven | +| Prefs safety: empty-email wipe guard | controller v0.137 + hub v0.71.0 | **IMPLEMENTED** | controller leg red-proofed 07-15; hub-side no-clobber belt (`handleSavePreferences` preserves a stored non-empty address on an empty-email push) red-proofed 07-22 | Born from a live incident; controller 0.160.0 guards both its push legs, so the hub belt covers older/rogue boxes | +| Paired recovery notifications + prefs seeding at claim + priority headers (power-outage audit F11/F12/F14-light) | hub v0.71.0 | **IMPLEMENTED** (recovery leg **PARTIAL** until a live staleness cycle fires it) | `hub/CHANGELOG.md` v0.71.0; 17 tests + 4 red-proofs (`REPORT.md` 2026-07-22); Resend `headers` mechanism probed live (HTTP 200) pre-implementation; operator+customer `test` rows live-fired via the controller's own test endpoint | Recovery = explicit eventType branch, severity semantics frozen; customer gate = PAIRING (`notification_log` evidence), not `enabled_events`. Live legs pending: a natural `*_recovered` mail (next real staleness cycle or the reboot-drill arc — never fabricated by blocking reports) and seed-at-claim on a real claim (Peti Friday reinstall). F14-full (operator push channel, ntfy/Telegram) stays open → R-69 | | System + container metrics (SQLite, Chart.js, 30-day downsampling) | controller | **IMPLEMENTED** | metrics collection + `/monitoring` render present (page 200) | The cited `CAMPAIGN-2` T-RES-CPU/T-SOAK-LOOP are H1/H2 harness artifacts (auth-302), not metrics tests; SQLite/Chart.js/30-day downsampling validated in no campaign. Demoted | | Always-on debug rings + on-demand log-bundle pulls with TTL/custody | controller v0.116, agent v0.83, hub v0.46 | **PROVEN-LIVE** | debug rings live-exercised `CAMPAIGN-3` fix-6 (1000-cap ring, ~55min horizon under load) | The **log-bundle-pull TTL/custody** half is changelog-only (no dedicated observability audit doc); ring persistence across restart is a known gap | | Operator alerting (Healthchecks → monitoring@felhom.eu) | k3s, Resend | **IMPLEMENTED** | operator infra, stated in production since 02-04; no corpus validation doc | Per the status enum, no citation → not PROVEN-LIVE. Demoted pending an operator-cited live alert (candidate re-upgrade — see REPORT) | diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index f32606c..2e6696f 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -79,6 +79,8 @@ | R-64 | **„Felhom↔Felhom media pairing blessed" — the two-box SMB pairing (one box shares, the other mounts it as NAS storage) becomes a supported, documented flow.** | XS–S | idea (2026-07-22) | Origin: the operator ran the pairing drill on the live demo pair and it WORKS — the drill itself is the pending evidence leg (a written run-through with the R-66 surfaces in play). R-66 shipped the enabling visibility: the serving box's address is now on its own Beállítások → Rendszer „Hálózat" card, and the add form names the NetBIOS trap. Blessing = a short customer-facing recipe (`documentation/controller/network-storage-nas.md` naming-caveat paragraph is the seed) + one supported-path sentence in the capability map. Flips: would add a "Felhom↔Felhom media pairing" capability row (currently unlisted). Pairs with R-65 (same two-box topology, entirely different transport + guarantees) | | R-66 | **The box's own address becomes visible — „Hálózat" card, Debug network dump, NetBIOS hint.** | XS | **SHIPPED (controller v0.159.0, 2026-07-22)** | Origin: the pairing drill — the serving box's IP was findable only as a hint buried on the OTHER box's Megosztás page, and the add form's failure for „FELHOM" taught nothing. Three legs: (A) „Hálózat" card on Beállítások → Rendszer (Helyi cím / Hálózati név only-while-sharing / Átjáró; live per render, stored nowhere — S-5; „—" when unavailable); (B) `network` section in the Debug dump (interfaces/route/DNS/lan_address, best-effort per item); (C) the NetBIOS trap named (Szerver helper text + a purely lexical hint on `unreachable` for single-label non-IP names). **Design decision recorded:** the controller is bridge-netns'd, so ALL guest-net reads go through the one netns door (docker exec into host-networked felhom-samba, `stacks/guestnet.go`) — with Megosztás off the card honestly shows „—" rather than the plausible-wrong 172.x answer. Deployed demo-felhom + demo-hp 2026-07-22; demo-hp live-shows the closed-door path (sharing off → dashes + in-place dump errors), demo-felhom the open one (real .104/.1/\\FELHOM values). Flips no capability-map row (diagnosability/UX polish); enables R-64 | | R-67 | **The NAS share appears in FileBrowser — browse what you mounted.** | S | **SHIPPED (controller v0.160.0, 2026-07-22)** | Origin: the R-64 pairing drill — the share said „Elérhető" and the customer had no way to BROWSE it (FileBrowser synced drives only). **Couples to R-64: browsing was its missing UX half.** A registered network storage now binds its share ROOT into FileBrowser (`/mnt/felhom-drives/:/srv/:rslave`) with its display label as the sidebar source; NAS add/remove trigger the same debounced sync. Two classes, two gates: drives keep the drive-absent gate byte-identically (proven live: the drives-only box logged a no-op sync); network shares gate on the STUB classifier instead — idle autofs is HEALTHY and included (Phase-0 probe on demo-hp: an in-container access through an rslave bind WAKES the idle trigger), while a stub verdict excludes the share from mounts AND sources with a WARN (an exposed stub swallows uploads the real mount later shadows). Nothing is ever written toward the NAS (no skeleton — red-proven). Live leg: cross-box upload round-trip demo-hp → demo-felhom + dead-NAS check (`Host is down` in seconds, unaided recovery after samba restart). Operator residual: the FileBrowser UI click-through (its admin credential is customer-held by design). Evidence: `felhom-controller/REPORT.md` (2026-07-22) | +| R-68 | **Notification train: paired recovery mails + prefs seeding at claim + priority headers (power-outage audit F11+F12+F14-light).** The dead-man's-switch fired perfectly on 07-22 and the customer who got „A szerver nem elérhető!" was never told it recovered (F11); a customer without a `customer_notifications` row is silently unnotifiable (F12, demo-hp live); delivered ≠ noticed (F14). | M | **SHIPPED (hub v0.71.0, 2026-07-22)** | Origin: `AUDIT-power-outage-recovery-2026-07-22.md`. Recovery = explicit eventType branch (severity semantics frozen; `severityNotifies` untouched): operator always hears both edges, customer iff PAIRED (customer-channel `sent` stale/down row newer than the last sent recovery — `store.LastCustomerSentAt`; `enabled_events` deliberately ignored for recovery; ties → no mail, flap-safe). Seed-at-claim: `MarkClaimed` → `SeedNotificationPrefs` (INSERT-if-absent, never upsert — red-proofed; empty email no-op; never fails the claim; default critical-only set). Hub-side empty-email no-clobber belt in `handleSavePreferences` (controller 0.160.0 already guards its own two push legs — latent, not live). `X-Priority: 1` + `Importance: high` on error/critical via Resend `headers` (live-probed HTTP 200 before implementation); the `test` event now also mails the operator with those headers (one click proves both channels + rendering); latent `sendTestEmail` nil-prefs panic fixed. 17 tests + 4 red-proofs. **Live legs pending:** natural `*_recovered` mail on the next real staleness cycle (or the reboot-drill arc — NEVER fabricated by blocking reports, that is F9-bypass-shaped) and seed-at-claim on a real claim (Peti Friday reinstall is the natural candidate) | +| R-69 | **F14-full: an operator push channel that actually interrupts (ntfy / Telegram / similar), beyond mail-client priority flags.** F14-light (v0.71.0 headers + Gmail filter) nudges a mail client; a 15:29 node_down should reach the operator's pocket in seconds regardless of inbox hygiene. Needs: channel choice (self-hosted ntfy on k3s vs Telegram bot), dispatcher fan-out seam, per-severity routing, quiet hours. | M | idea | Origin: `AUDIT-power-outage-recovery-2026-07-22.md` F14. Deliberately NOT built in the v0.71.0 train (scope-forked per the task spec) | | R-53 | **`app_export.html` substituted the CSRF token where the customer domain belongs** - the open-in-browser link was wrong for every app with a subdomain, and a session CSRF token landed in a URL. | XS | **SHIPPED (controller v0.150.0, 2026-07-20)** | One template token (`{{$.CSRFToken}}` -> `{{$.Domain}}`) plus the `Domain` key in `exportPageHandler`'s data map - that handler does not go through `baseData`, which is where every other page gets it, so the template had no domain to read. Render tests assert the joined `.` and that the token appears nowhere in that line; red-proofed against the pre-fix template. Origin: `audits/AUDIT-vacation-remote-ops-2026-07-20.md` (F7) | ## P3 — post-alpha diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index 272915a..80e833c 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,5 +1,43 @@ # Felhom Hub — Changelog +## v0.71.0 — paired recovery mails, prefs seeding at claim, priority headers, operator test leg (2026-07-22) + +Origin: `documentation/audits/AUDIT-power-outage-recovery-2026-07-22.md` F11 (recovery is silent), +F12 (prefs row optional → customer never notified), F14-light (delivered ≠ noticed). Live proof of +the gap: the demo customer got „A szerver nem elérhető!" at 15:29 and was never told it recovered. + +- **Paired recovery notifications (F11)** — `dispatcher.go` `processRecovery`, an explicit + eventType branch in `ProcessEvent` BEFORE the severity gate (`severityNotifies` and the checkers' + `emitTransition` severities are byte-untouched; `*_recovered` stays `info`). Operator always gets + both edges (existing 1 h per-type cooldown); the customer gets recovery **iff the customer was + mailed the paired stale/down** — pairing evidence is `store.LastCustomerSentAt` over + `notification_log` (customer channel, status=sent, `node_recovered→{node_stale,node_down}`, + `host_recovered→{host_stale,host_down}`), ties resolve to no-mail (flap-safe). `enabled_events` + is deliberately NOT consulted for recovery. Suppressions log at INFO with the reason. + `FormatOperatorEmail` renders ✅ for `*_recovered`; `customerMessages` gains `host_recovered`. +- **Prefs seeding at claim (F12)** — `claim.Engine.MarkClaimed` seeds `customer_notifications` + from the registered `customer_configs.email` on the unclaimed→claimed transition via new + `store.SeedNotificationPrefs` (INSERT OR IGNORE — never touches an existing row; empty email = + no-op; a seed failure never fails the claim). Default set (critical-only, Viktor may adjust): + node_down, backup_failed, disk_critical, host_disk_critical, storage_fill_critical, + offbox_repo_orphaned. +- **Empty-email no-clobber guard (F12)** — `handleSavePreferences`: a push with an empty email + preserves a stored non-empty address (events + cooldown still apply); a non-empty push updates + everything. Phase-0a fact: controller 0.160.0 guards both push legs itself + (`cmd/controller/main.go:821` startup skips empty email; `web/handlers.go:1532` refuses + empty-with-events), so the clobber was latent — this is the hub-side belt for older/rogue boxes. +- **Priority headers (F14-light)** — `sendEmailFn`/`sendEmail` gain a `headers` param; Resend + payload carries `"headers"` only when non-empty. `priorityHeaders(severity)`: error/critical → + `X-Priority: 1` + `Importance: high`; warning/info → none. Mechanism probed live pre-implementation + (Resend accepted, HTTP 200, mail id `34d3f7f3…`). +- **Operator test leg** — the `test` event now also mails the operator (`✅ : teszt / operator + channel OK`, priority headers forced) — one click proves customer channel + operator channel + + header rendering. Fixed a latent nil-deref found here: `sendTestEmail` dereferenced + `prefs.Email` while `GetNotificationPrefs` returns `(nil, nil)` for a customer with no row — a + test event for such a customer (e.g. demo-hp) panicked the dispatcher goroutine. +- Tests: 17 new (449 → 466) across store/notify/claim/api; 4 red-proofs run + reverted (pairing + removed, upsert-seed, guard removed, unconditional headers) — see `REPORT.md`. + ## v0.70.1 — the ghost customer's Delete button must exist (2026-07-22) **The fourth inert-seam defect: v0.70.0's ghost-delete path was fully implemented and fully diff --git a/hub/internal/api/handler.go b/hub/internal/api/handler.go index 042d91d..598af5e 100644 --- a/hub/internal/api/handler.go +++ b/hub/internal/api/handler.go @@ -1904,13 +1904,25 @@ func (h *Handler) handleSavePreferences(w http.ResponseWriter, r *http.Request) return } - if err := h.store.SaveNotificationPrefs(payload.CustomerID, payload.Email, payload.EnabledEvents, payload.CooldownHours); err != nil { + // Empty-email no-clobber guard (v0.71.0, audit F12): a controller push with an empty email + // (e.g. an unconfigured box) must never wipe a stored non-empty address — the seeded/edited + // email is the customer's alert lifeline. Events + cooldown from the push still apply; a push + // with a non-empty email updates everything (customer edits keep working). + saveEmail := payload.Email + if saveEmail == "" { + if existing, err := h.store.GetNotificationPrefs(payload.CustomerID); err == nil && existing != nil && existing.Email != "" { + saveEmail = existing.Email + h.logger.Printf("[INFO] Notification prefs push for %s had empty email — preserving stored address", payload.CustomerID) + } + } + + if err := h.store.SaveNotificationPrefs(payload.CustomerID, saveEmail, payload.EnabledEvents, payload.CooldownHours); err != nil { h.logger.Printf("[ERROR] Failed to save notification prefs for %s: %v", payload.CustomerID, err) http.Error(w, "Internal error", http.StatusInternalServerError) return } - h.logger.Printf("[INFO] Notification preferences updated for %s: email=%s, events=%v", payload.CustomerID, payload.Email, payload.EnabledEvents) + h.logger.Printf("[INFO] Notification preferences updated for %s: email=%s, events=%v", payload.CustomerID, saveEmail, payload.EnabledEvents) w.WriteHeader(http.StatusOK) w.Write([]byte(`{"status":"ok"}`)) } diff --git a/hub/internal/api/preferences_guard_test.go b/hub/internal/api/preferences_guard_test.go new file mode 100644 index 0000000..e2dc078 --- /dev/null +++ b/hub/internal/api/preferences_guard_test.go @@ -0,0 +1,67 @@ +package api + +import ( + "net/http" + "testing" +) + +// TestSavePreferences_EmptyEmailCannotClobber (v0.71.0 F12, Scenario E): a controller push with an +// empty email must preserve a stored non-empty address while still applying events + cooldown. +// Companion red-proof: removing the guard in handleSavePreferences makes this fail. +func TestSavePreferences_EmptyEmailCannotClobber(t *testing.T) { + h, st := newEventTestHandler(t) + if err := st.SaveNotificationPrefs("c1", "seeded@example.hu", []string{"node_down", "backup_failed"}, 6); err != nil { + t.Fatalf("stored prefs: %v", err) + } + + rr := do(h, http.MethodPost, "/preferences", "ckey", + `{"customer_id":"c1","email":"","enabled_events":["node_down"],"cooldown_hours":12}`) + if rr.Code != http.StatusOK { + t.Fatalf("status = %d, body=%s", rr.Code, rr.Body.String()) + } + + prefs, err := st.GetNotificationPrefs("c1") + if err != nil || prefs == nil { + t.Fatalf("prefs: %+v err=%v", prefs, err) + } + if prefs.Email != "seeded@example.hu" { + t.Fatalf("empty-email push CLOBBERED the stored address: email=%q", prefs.Email) + } + if len(prefs.EnabledEvents) != 1 || prefs.EnabledEvents[0] != "node_down" || prefs.CooldownHours != 12 { + t.Fatalf("events/cooldown from the push must still apply: %+v", prefs) + } +} + +// TestSavePreferences_NonEmptyEmailStillUpdates: a push with a real email updates everything — +// customer edits keep working (the guard must not freeze the address forever). +func TestSavePreferences_NonEmptyEmailStillUpdates(t *testing.T) { + h, st := newEventTestHandler(t) + if err := st.SaveNotificationPrefs("c1", "old@example.hu", []string{"node_down"}, 6); err != nil { + t.Fatalf("stored prefs: %v", err) + } + + rr := do(h, http.MethodPost, "/preferences", "ckey", + `{"customer_id":"c1","email":"new@example.hu","enabled_events":["backup_failed"],"cooldown_hours":3}`) + if rr.Code != http.StatusOK { + t.Fatalf("status = %d, body=%s", rr.Code, rr.Body.String()) + } + prefs, _ := st.GetNotificationPrefs("c1") + if prefs.Email != "new@example.hu" || len(prefs.EnabledEvents) != 1 || prefs.EnabledEvents[0] != "backup_failed" || prefs.CooldownHours != 3 { + t.Fatalf("non-empty push must update everything: %+v", prefs) + } +} + +// TestSavePreferences_EmptyEmailNoStoredRow: an empty-email push with no stored row behaves as +// before (row created with empty email — the all-off case stays legitimate). +func TestSavePreferences_EmptyEmailNoStoredRow(t *testing.T) { + h, st := newEventTestHandler(t) + rr := do(h, http.MethodPost, "/preferences", "ckey", + `{"customer_id":"c1","email":"","enabled_events":[],"cooldown_hours":6}`) + if rr.Code != http.StatusOK { + t.Fatalf("status = %d, body=%s", rr.Code, rr.Body.String()) + } + prefs, _ := st.GetNotificationPrefs("c1") + if prefs == nil || prefs.Email != "" { + t.Fatalf("all-off push must store the empty row unchanged, got %+v", prefs) + } +} diff --git a/hub/internal/claim/engine.go b/hub/internal/claim/engine.go index f830bd2..5141e7d 100644 --- a/hub/internal/claim/engine.go +++ b/hub/internal/claim/engine.go @@ -199,8 +199,23 @@ func (e *Engine) ResetToUnclaimed(cc *store.CustomerConfig) error { return nil } +// defaultSeedEvents is the enabled_events set seeded at claim (v0.71.0, audit F12) — critical-only: +// no node_stale (too chatty), no *_recovered (the recovery pairing gate handles those and never +// consults enabled_events). Viktor may adjust the list at review. +var defaultSeedEvents = []string{ + "node_down", + "backup_failed", + "disk_critical", + "host_disk_critical", + "storage_fill_critical", + "offbox_repo_orphaned", +} + // MarkClaimed records a controller-reported successful claim and sends the one-time confirmation -// email on the unclaimed→claimed transition (idempotent — repeated reports are no-ops). +// email on the unclaimed→claimed transition (idempotent — repeated reports are no-ops). On the +// transition it also seeds notification prefs from the registered email (v0.71.0, audit F12) so a +// claimed customer can no longer be silently unnotifiable — insert-if-absent, never overwriting a +// customer-edited row, and never failing the claim (notification plumbing must not gate claiming). func (e *Engine) MarkClaimed(cc *store.CustomerConfig) error { transitioned, err := e.Store.MarkClaimed(cc.CustomerID) if err != nil { @@ -210,6 +225,13 @@ func (e *Engine) MarkClaimed(cc *store.CustomerConfig) error { return nil } e.logf("[INFO] [claim] customer %s CLAIMED its dashboard (password set by the customer)", cc.CustomerID) + if seeded, err := e.Store.SeedNotificationPrefs(cc.CustomerID, cc.Email, defaultSeedEvents); err != nil { + e.logf("[WARN] [claim] notification-prefs seed for %s failed (claim unaffected): %v", cc.CustomerID, err) + } else if seeded { + e.logf("[INFO] [claim] notification prefs seeded for %s from the registered email (default critical set)", cc.CustomerID) + } else { + e.logf("[INFO] [claim] notification prefs for %s left untouched (row exists or no registered email)", cc.CustomerID) + } if cc.Email != "" { if err := e.Mailer.SendClaimEmail(string(EmailClaimed), cc.CustomerID, cc.Email, cc.Domain, ""); err != nil { e.logf("[WARN] [claim] claimed-confirmation email to %s failed: %v", cc.CustomerID, err) diff --git a/hub/internal/claim/seed_test.go b/hub/internal/claim/seed_test.go new file mode 100644 index 0000000..5067a8a --- /dev/null +++ b/hub/internal/claim/seed_test.go @@ -0,0 +1,78 @@ +package claim + +import ( + "testing" + + "gitea.dooplex.hu/admin/felhom-hub/internal/store" +) + +// TestMarkClaimed_SeedsNotificationPrefs (v0.71.0 F12, Scenario D): the unclaimed→claimed +// transition seeds a customer_notifications row from the registered email with the default +// critical-only event set; a pre-existing (customer-edited) row is NEVER modified. +// Companion red-proof: seeding via SaveNotificationPrefs (upsert) makes the second half fail. +func TestMarkClaimed_SeedsNotificationPrefs(t *testing.T) { + e, st, _ := newTestEngine(t) + if _, err := e.EnsureIssued(cust()); err != nil { + t.Fatalf("EnsureIssued: %v", err) + } + if err := e.MarkClaimed(cust()); err != nil { + t.Fatalf("MarkClaimed: %v", err) + } + prefs, err := st.GetNotificationPrefs("c1") + if err != nil || prefs == nil { + t.Fatalf("prefs must be seeded at claim, got %+v err=%v", prefs, err) + } + if prefs.Email != "owner@example.hu" { + t.Fatalf("seeded email = %q, want the registered address", prefs.Email) + } + if len(prefs.EnabledEvents) != len(defaultSeedEvents) { + t.Fatalf("seeded events = %v, want the default set %v", prefs.EnabledEvents, defaultSeedEvents) + } + for i, ev := range defaultSeedEvents { + if prefs.EnabledEvents[i] != ev { + t.Fatalf("seeded events = %v, want %v", prefs.EnabledEvents, defaultSeedEvents) + } + } + if prefs.CooldownHours != 6 { + t.Fatalf("seeded cooldown = %d, want 6", prefs.CooldownHours) + } +} + +// TestMarkClaimed_NeverOverwritesExistingPrefs: a customer-edited row survives a claim +// byte-identical (re-claim after RESET is the natural trigger). +func TestMarkClaimed_NeverOverwritesExistingPrefs(t *testing.T) { + e, st, _ := newTestEngine(t) + if err := st.SaveNotificationPrefs("c1", "edited@example.hu", []string{"node_down"}, 12); err != nil { + t.Fatalf("pre-existing prefs: %v", err) + } + if _, err := e.EnsureIssued(cust()); err != nil { + t.Fatalf("EnsureIssued: %v", err) + } + if err := e.MarkClaimed(cust()); err != nil { + t.Fatalf("MarkClaimed: %v", err) + } + prefs, _ := st.GetNotificationPrefs("c1") + if prefs == nil || prefs.Email != "edited@example.hu" || len(prefs.EnabledEvents) != 1 || prefs.CooldownHours != 12 { + t.Fatalf("claim seed MODIFIED an existing row (upsert bug): %+v", prefs) + } +} + +// TestMarkClaimed_EmptyEmail_NoRow_ClaimSucceeds: no registered email → no seed, and the claim +// itself still succeeds (notification plumbing must never gate claiming). +func TestMarkClaimed_EmptyEmail_NoRow_ClaimSucceeds(t *testing.T) { + e, st, _ := newTestEngine(t) + noMail := &store.CustomerConfig{CustomerID: "c1", Email: "", Domain: "example.hu"} + if _, err := e.EnsureIssued(cust()); err != nil { // issue needs an email to deliver the code + t.Fatalf("EnsureIssued: %v", err) + } + if err := e.MarkClaimed(noMail); err != nil { + t.Fatalf("MarkClaimed with empty email must succeed: %v", err) + } + if prefs, _ := st.GetNotificationPrefs("c1"); prefs != nil { + t.Fatalf("empty registered email must seed nothing, got %+v", prefs) + } + // The transition happened — a second call is a no-op (unchanged idempotency). + if err := e.MarkClaimed(noMail); err != nil { + t.Fatalf("idempotent re-claim: %v", err) + } +} diff --git a/hub/internal/notify/dispatcher.go b/hub/internal/notify/dispatcher.go index a787215..f1e9f79 100644 --- a/hub/internal/notify/dispatcher.go +++ b/hub/internal/notify/dispatcher.go @@ -29,8 +29,10 @@ type Dispatcher struct { custCooldowns map[string]time.Time // "customerID:eventType" → last customer notify // sendEmailFn is the email sender, seam-injected so tests exercise routing without real HTTP. - // Defaults to (*Dispatcher).sendEmail (Resend) in NewDispatcher. - sendEmailFn func(to, subject, textBody string) error + // Defaults to (*Dispatcher).sendEmail (Resend) in NewDispatcher. headers (nil = none) become + // Resend custom headers — used for the high-priority nudge on error/critical mails (v0.71.0, + // audit F14-light). + sendEmailFn func(to, subject, textBody string, headers map[string]string) error } // NewDispatcher creates a new notification dispatcher. @@ -50,6 +52,26 @@ func NewDispatcher(s *store.Store, resendAPIKey, fromEmail, operatorEmail string return d } +// priorityHeaders returns the Resend custom headers that nudge mail clients toward attention for +// error/critical mails (X-Priority + Importance; v0.71.0, audit F14-light: delivered ≠ noticed). +// Everything else gets nil — a warning or info mail must NOT masquerade as urgent. Pure → tested. +func priorityHeaders(severity string) map[string]string { + switch severity { + case "error", "critical": + return map[string]string{"X-Priority": "1", "Importance": "high"} + default: + return nil + } +} + +// recoveredPairedDownTypes maps a *_recovered eventType to the stale/down set whose customer-channel +// "sent" evidence licenses the customer recovery mail (v0.71.0, audit F11): recovery notifies +// exactly whoever the down notified. +var recoveredPairedDownTypes = map[string][]string{ + "node_recovered": {"node_stale", "node_down"}, + "host_recovered": {"host_stale", "host_down"}, +} + // severityNotifies reports whether a severity triggers email notifications. warning / error / critical // notify; everything else (info, recovery/status, or an unrecognized value) does not. Pure → unit-tested. // (Before v0.24.0 a "critical" severity was silently dropped here — the host_disk-class bug.) @@ -75,6 +97,14 @@ func (d *Dispatcher) ProcessEvent(customerID, eventType, severity, message, deta return } + // Recovery branch (v0.71.0, audit F11) — BEFORE the severity gate, as an explicit eventType + // branch: *_recovered stays severity "info" (semantics frozen), but is no longer silent. + // Operator always hears both edges; the customer hears recovery iff they heard the down. + if _, isRecovery := recoveredPairedDownTypes[eventType]; isRecovery { + d.processRecovery(customerID, eventType, severity, message, detailsJSON, source) + return + } + // warning / error / critical trigger notifications. "info" is an intentional non-notify (status/ // recovery events). Anything else is UNRECOGNIZED — log it (don't silently drop), so a bad severity // surfaces instead of vanishing (the felhom-pve-class lesson: a critical event must never be lost). @@ -93,22 +123,106 @@ func (d *Dispatcher) ProcessEvent(customerID, eventType, severity, message, deta } func (d *Dispatcher) sendTestEmail(customerID string) { + // nil-prefs guard (v0.71.0): GetNotificationPrefs returns (nil, nil) for a customer with no + // notification row — dereferencing prefs.Email here panicked the dispatcher goroutine for such + // a customer (latent since the test leg shipped; found while adding the operator copy). prefs, err := d.store.GetNotificationPrefs(customerID) - if err != nil || prefs.Email == "" { + if err != nil || prefs == nil || prefs.Email == "" { d.logger.Printf("[WARN] Test email: no email configured for %s", customerID) + } else { + subject := "[Felhom] Teszt értesítés" + body := "Kedves Ügyfél!\n\nEz egy teszt értesítés a Felhom monitoring rendszerből.\nAz értesítések megfelelően működnek.\n\nÜdvözlettel,\nFelhom.eu monitoring" + + if err := d.sendEmailFn(prefs.Email, subject, body, nil); err != nil { + d.logger.Printf("[ERROR] Test email to %s failed: %v", prefs.Email, err) + d.store.LogNotification(customerID, "test", "info", "Teszt értesítés", "failed", err.Error(), "customer") + } else { + d.logger.Printf("[INFO] Test email sent to %s for %s", prefs.Email, customerID) + d.store.LogNotification(customerID, "test", "info", "Teszt értesítés", "sent", "", "customer") + } + } + + // Operator copy (v0.71.0, audit F14-light): one test click proves the customer channel, the + // operator channel AND the high-priority header rendering in a single shot. + if !d.operatorOn || d.operatorEmail == "" { + return + } + opSubject := fmt.Sprintf("[Felhom] ✅ %s: teszt / operator channel OK", customerID) + opBody := fmt.Sprintf(`Operator copy of the customer notification test for %s. + +If this mail shows as high priority in your client, the X-Priority/Importance +headers render correctly. The customer test mail result is recorded in the +notification log. + +Dashboard: https://hub.felhom.eu/customers/%s`, customerID, customerID) + if err := d.sendEmailFn(d.operatorEmail, opSubject, opBody, priorityHeaders("critical")); err != nil { + d.logger.Printf("[ERROR] Operator test email failed for %s: %v", customerID, err) + d.store.LogNotification(customerID, "test", "info", "operator test copy", "failed", err.Error(), "operator") + return + } + d.logger.Printf("[INFO] Operator test email sent for %s", customerID) + d.store.LogNotification(customerID, "test", "info", "operator test copy", "sent", "", "operator") +} + +// processRecovery routes a *_recovered event (v0.71.0, audit F11). Severity semantics stay frozen +// ("info" everywhere else remains non-notify) — this is an explicit eventType branch. +// - Operator leg: always wanted (both edges), gated only by operatorOn + the 1h per-type +// cooldown — exactly processOperator. +// - Customer leg: gated by the PAIRING rule, not enabled_events — "recovery notifies exactly +// whoever the down notified." Evidence = a customer-channel status=sent row for the paired +// stale/down set newer than the last customer-channel sent recovery of this type. +func (d *Dispatcher) processRecovery(customerID, eventType, severity, message, detailsJSON, source string) { + d.processOperator(customerID, eventType, severity, message, detailsJSON, source) + + if d.store.IsCustomerBlocked(customerID) { + return + } + prefs, err := d.store.GetNotificationPrefs(customerID) + if err != nil || prefs == nil || prefs.Email == "" { return } - subject := "[Felhom] Teszt értesítés" - body := "Kedves Ügyfél!\n\nEz egy teszt értesítés a Felhom monitoring rendszerből.\nAz értesítések megfelelően működnek.\n\nÜdvözlettel,\nFelhom.eu monitoring" - - if err := d.sendEmailFn(prefs.Email, subject, body); err != nil { - d.logger.Printf("[ERROR] Test email to %s failed: %v", prefs.Email, err) - d.store.LogNotification(customerID, "test", "info", "Teszt értesítés", "failed", err.Error(), "customer") + // Pairing check — the customer gate. enabled_events is deliberately ignored here: a customer + // who was told "down" must be told "recovered", and one who wasn't must not be. + lastDown, downOk, err := d.store.LastCustomerSentAt(customerID, recoveredPairedDownTypes[eventType]) + if err != nil { + d.logger.Printf("[ERROR] Recovery pairing query failed for %s/%s: %v", customerID, eventType, err) return } - d.logger.Printf("[INFO] Test email sent to %s for %s", prefs.Email, customerID) - d.store.LogNotification(customerID, "test", "info", "Teszt értesítés", "sent", "", "customer") + lastRecovered, recOk, err := d.store.LastCustomerSentAt(customerID, []string{eventType}) + if err != nil { + d.logger.Printf("[ERROR] Recovery pairing query failed for %s/%s: %v", customerID, eventType, err) + return + } + // Second-granularity ties resolve to NOT-after → no mail (flap-safe direction). + if !downOk || (recOk && !lastDown.After(lastRecovered)) { + d.logger.Printf("[INFO] Recovery %s for %s: customer mail skipped — no unanswered customer down mail (pairing miss)", eventType, customerID) + return + } + + // Prefs cooldown keyed on the recovered eventType (belt over the pairing braces). + cooldownHours := prefs.CooldownHours + if cooldownHours <= 0 { + cooldownHours = 6 + } + cooldownKey := customerID + ":" + eventType + d.mu.Lock() + if last, ok := d.custCooldowns[cooldownKey]; ok && time.Since(last) < time.Duration(cooldownHours)*time.Hour { + d.mu.Unlock() + d.logger.Printf("[INFO] Recovery %s for %s: customer mail skipped — cooldown", eventType, customerID) + return + } + d.custCooldowns[cooldownKey] = time.Now() + d.mu.Unlock() + + subject, body := FormatCustomerEmail(customerID, eventType, severity, message, detailsJSON) + if err := d.sendEmailFn(prefs.Email, subject, body, priorityHeaders(severity)); err != nil { + d.logger.Printf("[ERROR] Customer recovery email failed for %s/%s: %v", customerID, eventType, err) + d.store.LogNotification(customerID, eventType, severity, message, "failed", err.Error(), "customer") + return + } + d.logger.Printf("[INFO] Customer recovery email sent to %s for %s/%s", prefs.Email, customerID, eventType) + d.store.LogNotification(customerID, eventType, severity, message, "sent", "", "customer") } func (d *Dispatcher) processOperator(customerID, eventType, severity, message, detailsJSON, source string) { @@ -127,7 +241,7 @@ func (d *Dispatcher) processOperator(customerID, eventType, severity, message, d subject, body := FormatOperatorEmail(customerID, eventType, severity, message, detailsJSON) - if err := d.sendEmailFn(d.operatorEmail, subject, body); err != nil { + if err := d.sendEmailFn(d.operatorEmail, subject, body, priorityHeaders(severity)); err != nil { d.logger.Printf("[ERROR] Operator email failed for %s/%s: %v", customerID, eventType, err) d.store.LogNotification(customerID, eventType, severity, message, "failed", err.Error(), "operator") return @@ -173,7 +287,7 @@ func (d *Dispatcher) processCustomer(customerID, eventType, severity, message, d subject, body := FormatCustomerEmail(customerID, eventType, severity, message, detailsJSON) - if err := d.sendEmailFn(prefs.Email, subject, body); err != nil { + if err := d.sendEmailFn(prefs.Email, subject, body, priorityHeaders(severity)); err != nil { d.logger.Printf("[ERROR] Customer email failed for %s/%s: %v", customerID, eventType, err) d.store.LogNotification(customerID, eventType, severity, message, "failed", err.Error(), "customer") return @@ -182,13 +296,16 @@ func (d *Dispatcher) processCustomer(customerID, eventType, severity, message, d d.store.LogNotification(customerID, eventType, severity, message, "sent", "", "customer") } -func (d *Dispatcher) sendEmail(to, subject, textBody string) error { +func (d *Dispatcher) sendEmail(to, subject, textBody string, headers map[string]string) error { payload := map[string]interface{}{ "from": d.fromEmail, "to": []string{to}, "subject": subject, "text": textBody, } + if len(headers) > 0 { + payload["headers"] = headers + } jsonData, err := json.Marshal(payload) if err != nil { @@ -236,7 +353,7 @@ func (d *Dispatcher) SendClaimEmail(kind, customerID, email, domain, code string } subject, body := FormatClaimEmail(kind, customerID, domain, code) eventType := "claim_" + kind - if err := d.sendEmailFn(email, subject, body); err != nil { + if err := d.sendEmailFn(email, subject, body, nil); err != nil { d.logger.Printf("[ERROR] claim %s email to customer %s failed: %v", kind, customerID, err) d.store.LogNotification(customerID, eventType, "info", subject, "failed", err.Error(), "customer") return err @@ -257,7 +374,7 @@ func (d *Dispatcher) SendSelfBindEmail(customerID, email, link string) error { return fmt.Errorf("notify: no resend api key") } subject, body := FormatSelfBindEmail(customerID, link) - if err := d.sendEmailFn(email, subject, body); err != nil { + if err := d.sendEmailFn(email, subject, body, nil); err != nil { d.logger.Printf("[ERROR] self-bind link email to customer %s failed: %v", customerID, err) d.store.LogNotification(customerID, "selfbind_link", "info", subject, "failed", err.Error(), "customer") return err diff --git a/hub/internal/notify/dispatcher_recovery_test.go b/hub/internal/notify/dispatcher_recovery_test.go new file mode 100644 index 0000000..ada31be --- /dev/null +++ b/hub/internal/notify/dispatcher_recovery_test.go @@ -0,0 +1,264 @@ +package notify + +import ( + "io" + "log" + "strings" + "sync" + "testing" +) + +// capturedMail is one seam-captured send: recipient, subject and headers. +type capturedMail struct { + to string + subject string + headers map[string]string +} + +// captureSeam installs a capturing sendEmailFn and returns the capture slice pointer. +func captureSeam(d *Dispatcher) *[]capturedMail { + var mu sync.Mutex + sent := &[]capturedMail{} + d.sendEmailFn = func(to, subject, _ string, headers map[string]string) error { + mu.Lock() + defer mu.Unlock() + *sent = append(*sent, capturedMail{to: to, subject: subject, headers: headers}) + return nil + } + return sent +} + +func mailsFor(sent []capturedMail, to string) []capturedMail { + var out []capturedMail + for _, m := range sent { + if m.to == to { + out = append(out, m) + } + } + return out +} + +// TestRecovery_PairedCustomerMail (Scenario A — the doodoo21 case, fixed): a customer who was +// mailed node_down gets the node_recovered mail; the operator gets the ✅ mail; both are logged. +func TestRecovery_PairedCustomerMail(t *testing.T) { + st := newDispStore(t) + if err := st.SaveNotificationPrefs("c1", "cust@example.com", []string{"node_down"}, 6); err != nil { + t.Fatal(err) + } + d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0)) + sent := captureSeam(d) + + // The down edge: customer (enabled) + operator both mailed. + d.ProcessEvent("c1", "node_down", "error", "No report received for 1h", "{}", "hub") + if len(*sent) != 2 { + t.Fatalf("down edge must mail operator + customer, got %d sends", len(*sent)) + } + + // The recovery edge. + d.ProcessEvent("c1", "node_recovered", "info", "Reports resumed (was down for 3h48m)", "{}", "hub") + + cust := mailsFor(*sent, "cust@example.com") + if len(cust) != 2 { + t.Fatalf("customer must get down + recovery, got %d customer mails", len(cust)) + } + recMail := cust[1] + if !strings.Contains(recMail.subject, "Információ: A szerver újra elérhető.") { + t.Fatalf("customer recovery subject wrong: %q", recMail.subject) + } + op := mailsFor(*sent, "op@felhom.eu") + if len(op) != 2 { + t.Fatalf("operator must get down + recovery, got %d", len(op)) + } + if !strings.Contains(op[1].subject, "✅") || !strings.Contains(op[1].subject, "node_recovered") { + t.Fatalf("operator recovery subject must carry ✅ + node_recovered: %q", op[1].subject) + } + + // notification_log holds sent rows on both channels for node_recovered. + entries, err := st.GetRecentNotifications("c1", 10) + if err != nil { + t.Fatal(err) + } + var gotCust, gotOp bool + for _, e := range entries { + if e.EventType == "node_recovered" && e.Status == "sent" { + switch e.Channel { + case "customer": + gotCust = true + case "operator": + gotOp = true + } + } + } + if !gotCust || !gotOp { + t.Fatalf("node_recovered sent rows missing (customer=%v operator=%v)", gotCust, gotOp) + } +} + +// TestRecovery_UnpairedStaysCustomerSilent (Scenario B): no customer-channel down evidence → the +// operator is mailed, the customer is NOT — even with node_recovered in enabled_events (the +// pairing rule is THE customer gate; enabled_events is deliberately ignored for recovery). +// Companion red-proof: removing the pairing check in processRecovery makes this fail. +func TestRecovery_UnpairedStaysCustomerSilent(t *testing.T) { + st := newDispStore(t) + if err := st.SaveNotificationPrefs("c1", "cust@example.com", []string{"node_down", "node_recovered"}, 6); err != nil { + t.Fatal(err) + } + d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0)) + sent := captureSeam(d) + + d.ProcessEvent("c1", "node_recovered", "info", "Reports resumed", "{}", "hub") + + if n := len(mailsFor(*sent, "cust@example.com")); n != 0 { + t.Fatalf("unpaired recovery must NOT mail the customer, got %d", n) + } + if n := len(mailsFor(*sent, "op@felhom.eu")); n != 1 { + t.Fatalf("operator must still get the recovery mail, got %d", n) + } +} + +// TestRecovery_FlapDamping (Scenario C): down mailed → recovery mailed → second down SUPPRESSED by +// the customer cooldown → second recovery must NOT mail the customer (the suppressed down left no +// fresh pairing evidence), and the operator recovery mail obeys the 1h per-type cooldown. +func TestRecovery_FlapDamping(t *testing.T) { + st := newDispStore(t) + if err := st.SaveNotificationPrefs("c1", "cust@example.com", []string{"node_down"}, 6); err != nil { + t.Fatal(err) + } + d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0)) + sent := captureSeam(d) + + d.ProcessEvent("c1", "node_down", "error", "down", "{}", "hub") // down #1: cust+op mailed + d.ProcessEvent("c1", "node_recovered", "info", "up", "{}", "hub") // recovery #1: cust+op mailed + d.ProcessEvent("c1", "node_down", "error", "down", "{}", "hub") // down #2 10min later: customer + // cooldown suppresses (op too, 1h) + // Clear the recovery CUSTOMER cooldown so the pairing gate — not the cooldown — decides #2: + d.mu.Lock() + delete(d.custCooldowns, "c1:node_recovered") + d.mu.Unlock() + d.ProcessEvent("c1", "node_recovered", "info", "up", "{}", "hub") // recovery #2 + + cust := mailsFor(*sent, "cust@example.com") + if len(cust) != 2 { // down #1 + recovery #1 only + t.Fatalf("second recovery must not mail the customer (no fresh pairing), customer mails=%d: %+v", len(cust), cust) + } + // Operator: down #1 + recovery #1; down #2 and recovery #2 suppressed by the 1h op cooldown. + if op := mailsFor(*sent, "op@felhom.eu"); len(op) != 2 { + t.Fatalf("operator recovery must obey the 1h cooldown, op mails=%d", len(op)) + } +} + +// TestTestEvent_BothChannels (Scenario F): a "test" event mails the customer (existing copy) AND +// the operator, the operator mail carrying both priority headers; both logged. +func TestTestEvent_BothChannels(t *testing.T) { + st := newDispStore(t) + if err := st.SaveNotificationPrefs("c1", "cust@example.com", nil, 6); err != nil { + t.Fatal(err) + } + d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0)) + sent := captureSeam(d) + + d.ProcessEvent("c1", "test", "info", "", "{}", "hub") + + cust := mailsFor(*sent, "cust@example.com") + if len(cust) != 1 || !strings.Contains(cust[0].subject, "Teszt értesítés") { + t.Fatalf("customer test mail wrong: %+v", cust) + } + op := mailsFor(*sent, "op@felhom.eu") + if len(op) != 1 || !strings.Contains(op[0].subject, "✅") || !strings.Contains(op[0].subject, "operator channel OK") { + t.Fatalf("operator test mail wrong: %+v", op) + } + if op[0].headers["X-Priority"] != "1" || op[0].headers["Importance"] != "high" { + t.Fatalf("operator test mail must carry priority headers, got %v", op[0].headers) + } + entries, _ := st.GetRecentNotifications("c1", 10) + var chans []string + for _, e := range entries { + if e.EventType == "test" && e.Status == "sent" { + chans = append(chans, e.Channel) + } + } + if len(chans) != 2 { + t.Fatalf("both test sends must be logged, got channels %v", chans) + } +} + +// TestTestEvent_NoPrefsRow_NoPanic: a test event for a customer WITHOUT a notification row must +// not panic (the nil-prefs deref found in v0.71.0 — GetNotificationPrefs returns (nil, nil)) and +// the operator copy still proves the operator channel. +func TestTestEvent_NoPrefsRow_NoPanic(t *testing.T) { + st := newDispStore(t) + d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0)) + sent := captureSeam(d) + + d.ProcessEvent("c1", "test", "info", "", "{}", "hub") // must not panic + + if n := len(mailsFor(*sent, "op@felhom.eu")); n != 1 { + t.Fatalf("operator test copy must send even without customer prefs, got %d", n) + } + if n := len(*sent); n != 1 { + t.Fatalf("no customer mail possible without prefs, total sends=%d", n) + } +} + +// TestPriorityHeaders (Scenario G): error/critical carry the two headers; warning/info/unknown nil. +// The negative half is the red-proof target: adding headers unconditionally fails the nil cases. +func TestPriorityHeaders(t *testing.T) { + for _, sev := range []string{"error", "critical"} { + h := priorityHeaders(sev) + if h["X-Priority"] != "1" || h["Importance"] != "high" || len(h) != 2 { + t.Errorf("%s: headers = %v", sev, h) + } + } + for _, sev := range []string{"warning", "info", "", "frobnicate"} { + if h := priorityHeaders(sev); h != nil { + t.Errorf("%s must have NO priority headers, got %v", sev, h) + } + } +} + +// TestPriorityHeaders_EndToEnd (Scenario G, seam level): an error mail carries headers on both +// channels; a warning mail carries none. +func TestPriorityHeaders_EndToEnd(t *testing.T) { + st := newDispStore(t) + if err := st.SaveNotificationPrefs("c1", "cust@example.com", []string{"node_down", "node_stale"}, 6); err != nil { + t.Fatal(err) + } + d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0)) + sent := captureSeam(d) + + d.ProcessEvent("c1", "node_down", "error", "down", "{}", "hub") + for _, m := range *sent { + if m.headers["X-Priority"] != "1" || m.headers["Importance"] != "high" { + t.Fatalf("error mail to %s must carry priority headers, got %v", m.to, m.headers) + } + } + + *sent = (*sent)[:0] + d.ProcessEvent("c1", "node_stale", "warning", "stale", "{}", "hub") + if len(*sent) == 0 { + t.Fatal("warning must still send") + } + for _, m := range *sent { + if m.headers != nil { + t.Fatalf("warning mail to %s must carry NO headers, got %v", m.to, m.headers) + } + } +} + +// TestRecovery_SeverityStaysInfo guards the frozen semantics: the recovery branch must not have +// widened severityNotifies — a non-recovery info event is still silent. +func TestRecovery_SeverityStaysInfo(t *testing.T) { + if severityNotifies("info") { + t.Fatal("info must remain non-notify — recovery is an eventType branch, not a severity change") + } + st := newDispStore(t) + if err := st.SaveNotificationPrefs("c1", "cust@example.com", []string{"controller_started"}, 6); err != nil { + t.Fatal(err) + } + d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0)) + sent := captureSeam(d) + d.ProcessEvent("c1", "controller_started", "info", "started", "{}", "hub") + if len(*sent) != 0 { + t.Fatalf("non-recovery info must stay silent, got %d sends", len(*sent)) + } +} diff --git a/hub/internal/notify/dispatcher_test.go b/hub/internal/notify/dispatcher_test.go index 5ba82f0..ef07036 100644 --- a/hub/internal/notify/dispatcher_test.go +++ b/hub/internal/notify/dispatcher_test.go @@ -56,7 +56,7 @@ func TestProcessEvent_PoolBoxScopeOperatorOnly(t *testing.T) { d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0)) var mu sync.Mutex var sent []string - d.sendEmailFn = func(to, _, _ string) error { mu.Lock(); defer mu.Unlock(); sent = append(sent, to); return nil } + d.sendEmailFn = func(to, _, _ string, _ map[string]string) error { mu.Lock(); defer mu.Unlock(); sent = append(sent, to); return nil } d.ProcessEvent("pool-box", "offsite_box_fill", "warning", "Offsite pool box 82% full", `{"scope":"pool-box"}`, "hub") if len(sent) != 1 || sent[0] != "op@felhom.eu" { @@ -71,7 +71,7 @@ func TestProcessEvent_PBSDRBoxScopeOperatorOnly(t *testing.T) { d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0)) var mu sync.Mutex var sent []string - d.sendEmailFn = func(to, _, _ string) error { mu.Lock(); defer mu.Unlock(); sent = append(sent, to); return nil } + d.sendEmailFn = func(to, _, _ string, _ map[string]string) error { mu.Lock(); defer mu.Unlock(); sent = append(sent, to); return nil } d.ProcessEvent("pbsdr-box", "pbsdr_box_fill", "warning", "PBS DR datastore 82% full", `{"scope":"pbsdr-box"}`, "hub") if len(sent) != 1 || sent[0] != "op@felhom.eu" { @@ -85,7 +85,7 @@ func TestProcessEvent_CriticalRoutes(t *testing.T) { d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0)) var mu sync.Mutex var sent []string - d.sendEmailFn = func(to, _, _ string) error { mu.Lock(); defer mu.Unlock(); sent = append(sent, to); return nil } + d.sendEmailFn = func(to, _, _ string, _ map[string]string) error { mu.Lock(); defer mu.Unlock(); sent = append(sent, to); return nil } d.ProcessEvent("c1", "host_disk_critical", "critical", "root full", "{}", "hub") if len(sent) != 1 || sent[0] != "op@felhom.eu" { @@ -100,7 +100,7 @@ func TestProcessEvent_UnknownSeverityLogged(t *testing.T) { var buf bytes.Buffer d := NewDispatcher(st, "test-key", "from", "op@felhom.eu", true, log.New(&buf, "", 0)) sent := 0 - d.sendEmailFn = func(_, _, _ string) error { sent++; return nil } + d.sendEmailFn = func(_, _, _ string, _ map[string]string) error { sent++; return nil } d.ProcessEvent("c1", "weird_event", "frobnicate", "msg", "{}", "hub") if sent != 0 { @@ -118,7 +118,7 @@ func TestProcessEvent_InfoSilent(t *testing.T) { var buf bytes.Buffer d := NewDispatcher(st, "test-key", "from", "op@felhom.eu", true, log.New(&buf, "", 0)) sent := 0 - d.sendEmailFn = func(_, _, _ string) error { sent++; return nil } + d.sendEmailFn = func(_, _, _ string, _ map[string]string) error { sent++; return nil } d.ProcessEvent("c1", "controller_started", "info", "msg", "{}", "hub") if sent != 0 { diff --git a/hub/internal/notify/templates.go b/hub/internal/notify/templates.go index 473a732..9c225e1 100644 --- a/hub/internal/notify/templates.go +++ b/hub/internal/notify/templates.go @@ -2,6 +2,7 @@ package notify import ( "fmt" + "strings" "time" ) @@ -22,8 +23,12 @@ func init() { // FormatOperatorEmail returns (subject, textBody) for the operator channel. func FormatOperatorEmail(customerID, eventType, severity, message, detailsJSON string) (string, string) { + // Icon is eventType-aware for recovery (v0.71.0, audit F11); severity stays the fallback. icon := "⚠️" - if severity == "error" || severity == "critical" { + switch { + case strings.HasSuffix(eventType, "_recovered"): + icon = "✅" + case severity == "error" || severity == "critical": icon = "🔴" } @@ -86,6 +91,7 @@ var customerMessages = map[string]string{ "node_stale": "A szerver nem küldött jelentést az elmúlt időszakban.", "node_down": "A szerver nem elérhető!", "node_recovered": "A szerver újra elérhető.", + "host_recovered": "A házszerver alaprendszere (Proxmox-gazda) újra elérhető.", // Health events "health_degraded": "A rendszer állapota romlott.", diff --git a/hub/internal/store/notifications_seed_test.go b/hub/internal/store/notifications_seed_test.go new file mode 100644 index 0000000..0c2c14f --- /dev/null +++ b/hub/internal/store/notifications_seed_test.go @@ -0,0 +1,115 @@ +package store + +import ( + "testing" + "time" +) + +// insertNotifLog inserts a notification_log row with an EXPLICIT created_at so pairing tests can +// order rows deterministically (datetime('now') is second-granularity — real calls tie). +func insertNotifLog(t *testing.T, s *Store, customerID, eventType, status, channel, createdAt string) { + t.Helper() + if _, err := s.db.Exec(` + INSERT INTO notification_log (customer_id, event_type, severity, message, status, error_message, channel, created_at) + VALUES (?, ?, 'error', 'm', ?, '', ?, ?)`, + customerID, eventType, status, channel, createdAt); err != nil { + t.Fatalf("insertNotifLog: %v", err) + } +} + +// TestLastCustomerSentAt_PairingQuery (v0.71.0 F11): the pairing-evidence query returns the max +// created_at over customer-channel status=sent rows of the given types ONLY — operator rows, +// failed rows and other event types must not count. +func TestLastCustomerSentAt_PairingQuery(t *testing.T) { + s := newTestStore(t) + + // Noise that must NOT count: + insertNotifLog(t, s, "c1", "node_down", "sent", "operator", "2026-07-22 15:00:00") // wrong channel + insertNotifLog(t, s, "c1", "node_down", "failed", "customer", "2026-07-22 16:00:00") // wrong status + insertNotifLog(t, s, "c1", "backup_failed", "sent", "customer", "2026-07-22 17:00:00") // wrong type + insertNotifLog(t, s, "c2", "node_down", "sent", "customer", "2026-07-22 18:00:00") // wrong customer + + // No qualifying row yet: + _, ok, err := s.LastCustomerSentAt("c1", []string{"node_stale", "node_down"}) + if err != nil { + t.Fatalf("LastCustomerSentAt: %v", err) + } + if ok { + t.Fatal("no qualifying row must yield ok=false") + } + + // Two qualifying rows — max wins: + insertNotifLog(t, s, "c1", "node_stale", "sent", "customer", "2026-07-22 12:59:00") + insertNotifLog(t, s, "c1", "node_down", "sent", "customer", "2026-07-22 13:29:00") + got, ok, err := s.LastCustomerSentAt("c1", []string{"node_stale", "node_down"}) + if err != nil { + t.Fatalf("LastCustomerSentAt: %v", err) + } + if !ok { + t.Fatal("qualifying rows exist — ok must be true") + } + want := time.Date(2026, 7, 22, 13, 29, 0, 0, time.UTC) + if !got.Equal(want) { + t.Fatalf("max created_at = %v, want %v", got, want) + } + + // Empty type list is a defined no-op: + if _, ok, err := s.LastCustomerSentAt("c1", nil); err != nil || ok { + t.Fatalf("empty eventTypes must return (zero, false, nil), got ok=%v err=%v", ok, err) + } +} + +// TestSeedNotificationPrefs_InsertIfAbsent (v0.71.0 F12, Scenario D): the seed creates a row when +// none exists, and NEVER modifies a pre-existing row (insert-if-absent, not upsert). Companion +// red-proof: replacing the INSERT OR IGNORE with SaveNotificationPrefs makes the second half fail. +func TestSeedNotificationPrefs_InsertIfAbsent(t *testing.T) { + s := newTestStore(t) + defaults := []string{"node_down", "backup_failed", "disk_critical"} + + seeded, err := s.SeedNotificationPrefs("c1", "x@example.com", defaults) + if err != nil { + t.Fatalf("seed: %v", err) + } + if !seeded { + t.Fatal("first seed must create a row") + } + prefs, err := s.GetNotificationPrefs("c1") + if err != nil || prefs == nil { + t.Fatalf("prefs after seed: %v %v", prefs, err) + } + if prefs.Email != "x@example.com" || prefs.CooldownHours != 6 || len(prefs.EnabledEvents) != 3 { + t.Fatalf("seeded row wrong: %+v", prefs) + } + + // A customer-edited row must survive a later seed byte-identical: + if err := s.SaveNotificationPrefs("c1", "edited@example.com", []string{"node_down"}, 12); err != nil { + t.Fatalf("customer edit: %v", err) + } + seeded, err = s.SeedNotificationPrefs("c1", "reinstall@example.com", defaults) + if err != nil { + t.Fatalf("re-seed: %v", err) + } + if seeded { + t.Fatal("seed over an existing row must report seeded=false") + } + prefs, _ = s.GetNotificationPrefs("c1") + if prefs.Email != "edited@example.com" || prefs.CooldownHours != 12 || len(prefs.EnabledEvents) != 1 { + t.Fatalf("seed MODIFIED an existing row (upsert bug): %+v", prefs) + } +} + +// TestSeedNotificationPrefs_EmptyEmailNoop: an empty registered email must not seed an +// unnotifiable row. +func TestSeedNotificationPrefs_EmptyEmailNoop(t *testing.T) { + s := newTestStore(t) + seeded, err := s.SeedNotificationPrefs("c1", "", []string{"node_down"}) + if err != nil { + t.Fatalf("seed: %v", err) + } + if seeded { + t.Fatal("empty email must be a no-op") + } + if prefs, _ := s.GetNotificationPrefs("c1"); prefs != nil { + t.Fatalf("no row must exist, got %+v", prefs) + } +} diff --git a/hub/internal/store/store.go b/hub/internal/store/store.go index 7e5ebe0..9113c21 100644 --- a/hub/internal/store/store.go +++ b/hub/internal/store/store.go @@ -7,6 +7,7 @@ import ( "fmt" "log" "strconv" + "strings" "time" "gitea.dooplex.hu/admin/felhom-hub/internal/semver" @@ -811,6 +812,62 @@ func (s *Store) GetRecentNotifications(customerID string, limit int) ([]Notifica return entries, rows.Err() } +// LastCustomerSentAt returns the most recent notification_log created_at over the given event +// types on the CUSTOMER channel with status='sent', and whether any such row exists. It is the +// pairing-evidence query for recovery notifications (v0.71.0, audit F11): "was the customer told +// about the down since they were last told about a recovery?" Uses the +// (customer_id, created_at DESC) index. An empty eventTypes slice returns (zero, false, nil). +func (s *Store) LastCustomerSentAt(customerID string, eventTypes []string) (time.Time, bool, error) { + if len(eventTypes) == 0 { + return time.Time{}, false, nil + } + placeholders := make([]string, len(eventTypes)) + args := make([]interface{}, 0, len(eventTypes)+1) + args = append(args, customerID) + for i, et := range eventTypes { + placeholders[i] = "?" + args = append(args, et) + } + var createdAt sql.NullString + err := s.db.QueryRow(` + SELECT MAX(created_at) FROM notification_log + WHERE customer_id = ? AND channel = 'customer' AND status = 'sent' + AND event_type IN (`+strings.Join(placeholders, ",")+`)`, + args..., + ).Scan(&createdAt) + if err != nil { + return time.Time{}, false, err + } + if !createdAt.Valid || createdAt.String == "" { + return time.Time{}, false, nil + } + return parseSQLiteTime(createdAt.String), true, nil +} + +// SeedNotificationPrefs creates a customer_notifications row IF AND ONLY IF none exists — +// insert-if-absent, never an upsert (a customer-edited row must never be overwritten by a seed; +// audit F12). An empty email is a no-op: seeding an unnotifiable row would only mask the gap. +// Returns whether a row was created. +func (s *Store) SeedNotificationPrefs(customerID, email string, enabledEvents []string) (bool, error) { + if email == "" { + return false, nil + } + eventsJSON, _ := json.Marshal(enabledEvents) + res, err := s.db.Exec(` + INSERT OR IGNORE INTO customer_notifications (customer_id, email, enabled_events, cooldown_hours) + VALUES (?, ?, ?, 6)`, + customerID, email, string(eventsJSON), + ) + if err != nil { + return false, err + } + n, err := res.RowsAffected() + if err != nil { + return false, err + } + return n > 0, nil +} + // SaveReport stores a new report. The reportJSON should be the raw JSON payload. func (s *Store) SaveReport(customerID string, reportJSON []byte) error { // Parse denormalized fields from the JSON