feat: Hub monitoring takeover — event push system + config cleanup (v0.21.0)

Replace external Healthchecks.io with Hub-native event system. Controller
now pushes structured events via POST /api/v1/event with typed detail
structs. Hub handles dead man's switch, notification dispatch, and cooldowns.

Phase 5: PushEvent() core method, 21 event types, expanded notification
settings (11 toggles), Hub connection monitoring on dashboard, alerts.
Phase 6: Deprecation log for ping UUIDs, pinger kept for transition.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-02-20 18:53:21 +01:00
parent 55abe401ee
commit 8aebbb8902
13 changed files with 722 additions and 318 deletions
+47
View File
@@ -1,5 +1,52 @@
## Changelog ## Changelog
### What was just completed (2026-02-20 session 64)
- **v0.21.0 — Hub Monitoring Takeover (Controller-side, Phases 5+6):**
Replaces external Healthchecks.io dependency with Hub-native event system. The controller now pushes structured events directly to the Hub's `/api/v1/event` endpoint, and the Hub handles dead man's switch detection, notification dispatch, and cooldown management.
**Phase 5 — Event Push System (`internal/notify/notifier.go`):**
- New core method `PushEvent(eventType, severity, message, details)` — non-blocking goroutine, 2 retries with 3s backoff, POSTs to Hub `/api/v1/event`
- 8 typed detail structs: `BackupDetails`, `DBDumpDetails`, `DiskDetails`, `HealthDetails`, `StorageDetails`, `UpdateDetails`, `AppDetails`, `CrossDriveDetails`
- Replaced all old `Notify*` methods with event-based equivalents:
- `NotifyBackupCompleted/Failed``backup_completed`/`backup_failed` events
- `NotifyDBDumpCompleted/Failed``db_dump_completed`/`db_dump_failed` events
- `NotifyIntegrityOK/Failed``backup_integrity_ok`/`backup_integrity_failed` events
- `NotifyHealthChange` → detects transitions, pushes `health_degraded`/`health_critical`/`health_recovered`
- `NotifyStorageDisconnected/Reconnected``storage_disconnected`/`storage_reconnected` events
- `NotifyControllerStarted``controller_started` event on startup
- `NotifyControllerUpdated``controller_updated` event (replaces `NotifyUpdateSuccess/Failed`)
- `NotifyAppDeployed/Removed``app_deployed`/`app_removed` events
- `NotifyCrossDriveCompleted/Failed``crossdrive_completed`/`crossdrive_failed` events
- `NotifyDRStarted/Completed``disaster_recovery_started`/`disaster_recovery_completed` events
- Removed old `/api/v1/notify` relay, `classifyWarning()`, and client-side cooldown logic (Hub handles cooldowns now)
- `SendTest()` now pushes `test` event type via `PushEvent`
- `SyncPreferences` updated to include `cooldownHours` parameter
**Phase 5 — Event Wiring:**
- `main.go`: Wired success events for backup, db-dump, integrity check; startup event with 5s delay; update event after `VerifyStartup()`
- `router.go`: Added `NotifyAppDeployed`/`NotifyAppRemoved` after successful deploy/remove via API
- `handler_restore.go`: Added `NotifyDRStarted`/`NotifyDRCompleted` in DR restore flow
- `server.go`: New `HubPushStatusData` struct and `SetHubPushStatus` callback for monitoring page
**Phase 5 — Hub Connection Monitoring:**
- `pusher.go`: Added `PushStatus` tracking (LastAttempt, LastSuccess, LastError, Consecutive failures) to report Pusher
- `handlers.go`: Monitoring page now shows Hub connection status (connected/unreachable, URL, customer ID, last success, last error) instead of Healthchecks ping UUIDs
- `monitoring.html`: Replaced "Távoli monitoring" section with "Hub kapcsolat" section
- `alerts.go`: Replaced "Missing ping UUIDs" alert with Hub connection alerts (`hub-disabled` warning, `hub-unreachable` error)
**Phase 5 — Expanded Notification Settings:**
- `settings.html`: Expanded from 4 checkboxes to 11 grouped toggles in two categories:
- "Hibák és figyelmeztetések": backup_failed, db_dump_failed, backup_integrity_failed, crossdrive_failed, disk alerts, storage_disconnected, node_down, health_critical, expected missed
- "Tájékoztató": storage_reconnected, health_recovered
- Compound toggles: "Lemez figyelmeztetés" maps to `disk_warning` + `disk_critical`; "Elvárt mentés elmaradt" maps to `expected_backup_missed` + `expected_dbdump_missed`
- `settings.go`: Updated `DefaultEnabledEvents` to new Hub event types
- `handlers.go`: Updated settings POST handler for expanded event names and compound toggles
**Phase 6 — Config Cleanup:**
- `main.go`: Deprecation log on startup when ping UUIDs are configured: `[INFO] Healthchecks ping UUIDs configured but no longer used — monitoring is now handled by the Hub`
- Pinger still runs for transitional backward compatibility
### What was just completed (2026-02-20 session 63) ### What was just completed (2026-02-20 session 63)
- **v0.20.0 — Hub Config Management (Phase B):** - **v0.20.0 — Hub Config Management (Phase B):**
+48 -34
View File
@@ -4,7 +4,7 @@
A single, lightweight Go container that replaces Portainer + scattered systemd scripts with a unified, Hungarian-language web dashboard for managing Docker Compose stacks, backups, storage, monitoring, and notifications on customer hardware. A single, lightweight Go container that replaces Portainer + scattered systemd scripts with a unified, Hungarian-language web dashboard for managing Docker Compose stacks, backups, storage, monitoring, and notifications on customer hardware.
**Current version: v0.20.0** **Current version: v0.21.0**
--- ---
@@ -509,16 +509,9 @@ Backup destination validation (`CheckBackupDestination`) has tiered checks:
- Disk >95% full → critical/blocked - Disk >95% full → critical/blocked
- Disk >90% full → warning - Disk >90% full → warning
#### Healthchecks.io Integration (`internal/monitor/pinger.go`) #### Healthchecks.io Integration (deprecated)
Five ping UUIDs for external monitoring: Legacy pinger (`internal/monitor/pinger.go`) still runs for backward compatibility but is no longer the primary monitoring mechanism. Monitoring is now handled by the Hub event system (see [Notifications](#5-notifications)). A deprecation log is emitted on startup if ping UUIDs are configured.
- **Heartbeat**: every 5 min (simple "I'm alive")
- **System Health**: periodic health check results
- **DB Dump**: after nightly database dumps
- **Backup**: after nightly restic backup
- **Backup Integrity**: weekly `restic check` result
3-attempt retry with 2-second backoff. Pinger never fails the caller.
#### Metrics Store (`internal/metrics/`) #### Metrics Store (`internal/metrics/`)
@@ -535,48 +528,66 @@ Full-page system monitor at `/monitoring`:
- **System Metrics Charts**: 4 line charts (CPU, Memory, Temperature, Load) in 2x2 grid - **System Metrics Charts**: 4 line charts (CPU, Memory, Temperature, Load) in 2x2 grid
- **Container Resources**: horizontal bar charts (CPU% and Memory per container) - **Container Resources**: horizontal bar charts (CPU% and Memory per container)
- **Per-container Detail**: click-to-expand historical charts - **Per-container Detail**: click-to-expand historical charts
- **Remote Monitoring Status**: shows Healthchecks ping UUID configuration - **Hub Connection Status**: shows Hub URL, customer ID, connection state (connected/unreachable), last successful push, last error
Chart.js 4.4.7 embedded locally (works in offline environments), dark theme matching site design. Chart.js 4.4.7 embedded locally (works in offline environments), dark theme matching site design.
#### Alert System (`internal/web/alerts.go`) #### Alert System (`internal/web/alerts.go`)
State-based alerts displayed on all pages: State-based alerts displayed on all pages:
- Sources: health issues, missing ping UUIDs, backup disabled - Sources: health issues, Hub connection status, backup disabled, storage disconnected, update available
- Hub alerts: `hub-disabled` (warning) when Hub not enabled, `hub-unreachable` (error) when last push failed and no success in 30 min
- Sorted by severity (error > warning > info), capped at 5 visible - Sorted by severity (error > warning > info), capped at 5 visible
- Refreshed every 5 min + on startup - Refreshed every 5 min + on startup + on storage state changes
- Monitoring page suppresses ping-related alerts (shown in dedicated table instead)
--- ---
### 5. Notifications ### 5. Notifications
#### Email Delivery #### Hub Event System (`internal/notify/notifier.go`)
The controller relays notifications through the central hub, which sends emails via the Resend API: The controller pushes structured events to the Hub's `/api/v1/event` endpoint. The Hub handles notification dispatch, cooldown management, and dead man's switch detection.
1. Controller detects event (health degradation, backup failure, etc.)
2. Non-blocking POST to hub's `/api/v1/notify` with event details **Core method:** `PushEvent(eventType, severity, message, details)` — non-blocking goroutine, 2 retries with 3s backoff, never blocks the caller.
3. Hub checks customer notification preferences
4. Hub sends Hungarian-language email via Resend
#### Event Types #### Event Types
| Event | Trigger | | Event Type | Severity | Trigger |
|-------|---------| |------------|----------|---------|
| `disk_warning` | Disk usage crosses warning/critical threshold | | `backup_completed` | info | Nightly restic backup succeeds |
| `backup_failed` | Nightly backup or DB dump fails | | `backup_failed` | error | Nightly restic backup fails |
| `update_available` | New app version detected in catalog | | `db_dump_completed` | info | Nightly database dumps succeed |
| `security_update` | Critical security update available | | `db_dump_failed` | error | Nightly database dumps fail |
| `backup_integrity_ok` | info | Weekly `restic check` passes |
| `backup_integrity_failed` | error | Weekly `restic check` fails |
| `crossdrive_completed` | info | Cross-drive secondary backup succeeds |
| `crossdrive_failed` | error | Cross-drive secondary backup fails |
| `health_degraded` | warning | Health status degrades (ok→warn) |
| `health_critical` | error | Health status critical (any→fail) |
| `health_recovered` | info | Health status recovers (fail/warn→ok) |
| `disk_warning` | warning | Disk usage crosses 90% |
| `disk_critical` | error | Disk usage crosses 95% |
| `storage_disconnected` | error | Storage drive physically removed |
| `storage_reconnected` | info | Storage drive reconnected |
| `controller_started` | info | Controller process starts |
| `controller_updated` | info/error | Self-update success or failure |
| `app_deployed` | info | New app deployed via API |
| `app_removed` | info | App removed via API |
| `disaster_recovery_started` | warning | DR restore begins |
| `disaster_recovery_completed` | info/error | DR restore finishes (success/partial) |
#### Cooldown System Each event carries typed detail structs (e.g., `BackupDetails`, `DiskDetails`, `HealthDetails`) serialized as JSON.
Per-event-type cooldown (default 6 hours, configurable) prevents notification spam. Only notifies on **status degradation** (ok→warn, ok→fail, warn→fail), not on repeated same-status checks. #### Default Enabled Events
Events the customer receives notifications for (configurable in settings):
`backup_failed`, `db_dump_failed`, `disk_warning`, `disk_critical`, `storage_disconnected`, `node_down`, `health_critical`, `expected_backup_missed`, `expected_dbdump_missed`
#### Preference Sync #### Preference Sync
Notification preferences (email, enabled events, cooldown) are: Notification preferences (email, enabled events, cooldown hours) are:
- Stored locally in `settings.json` - Stored locally in `settings.json`
- Synced to hub on save and on controller startup - Synced to Hub on save and on controller startup via `POST /api/v1/preferences`
- Hub sync failure doesn't block local save - Hub sync failure doesn't block local save
--- ---
@@ -776,7 +787,7 @@ Periodic JSON push (default every 15 min) to the central felhom-hub service:
- Stacks: deployed apps with versions and states - Stacks: deployed apps with versions and states
- Config hash: SHA256 of `controller.yaml` for Hub-side config comparison - Config hash: SHA256 of `controller.yaml` for Hub-side config comparison
Bearer token authentication, 3-attempt retry with 5-second backoff. Bearer token authentication, 3-attempt retry with 5-second backoff. Push status tracked via `PushStatus` struct (LastAttempt, LastSuccess, LastError, consecutive failures) — used by the monitoring page and alert system to show Hub connection health.
#### Infrastructure Backup to Hub (`internal/report/infra_backup.go`) #### Infrastructure Backup to Hub (`internal/report/infra_backup.go`)
@@ -792,11 +803,14 @@ This enables fully automated recovery when the system drive is replaced — the
#### Hub Dashboard #### Hub Dashboard
The hub service (separate Go app in the `felhom.eu` repo) provides: The hub service (separate Go app in the `felhom.eu` repo) provides:
- Multi-customer overview table with status indicators - Multi-customer overview table with status indicators and event count badges
- Customer detail page with system/storage/containers/backup/health sections - Customer detail page with system/storage/containers/backup/health/events sections
- Event timeline: last 50 events with severity filter, colored badges, source tracking
- Dead man's switch: staleness detection (30min stale, 60min down), missed backup detection (daily at 05:00)
- Notification dispatch: operator (English) + customer (Hungarian) emails via Resend with per-event cooldowns
- Infra backup status per customer (last sync, stack count, disk count) - Infra backup status per customer (last sync, stack count, disk count)
- Color coding: green (<30min), yellow (30-60min), red (>60min since last report) - Color coding: green (<30min), yellow (30-60min), red (>60min since last report)
- 90-day report retention with daily prune - 90-day report + event retention with daily prune at 04:30 Budapest time
### 9. Disaster Recovery ### 9. Disaster Recovery
+40 -8
View File
@@ -197,9 +197,15 @@ func main() {
logger.Println("[INFO] Metrics collector started (60s interval)") logger.Println("[INFO] Metrics collector started (60s interval)")
} }
// --- Initialize health pinger --- // --- Initialize health pinger (legacy, will be removed) ---
pinger := monitor.NewPinger(&cfg.Monitoring, logger) pinger := monitor.NewPinger(&cfg.Monitoring, logger)
// Deprecation notice for ping UUIDs
uuids := cfg.Monitoring.PingUUIDs
if uuids.Heartbeat != "" || uuids.SystemHealth != "" || uuids.DBDump != "" || uuids.Backup != "" || uuids.BackupIntegrity != "" {
logger.Println("[INFO] Healthchecks ping UUIDs configured but no longer used — monitoring is now handled by the Hub")
}
// --- Initialize backup manager --- // --- Initialize backup manager ---
var backupMgr *backup.Manager var backupMgr *backup.Manager
stackProv := &stackAdapter{ stackProv := &stackAdapter{
@@ -241,11 +247,7 @@ func main() {
}) })
// Check for post-update state (did a previous update succeed or fail?) // Check for post-update state (did a previous update succeed or fail?)
if state := updater.VerifyStartup(); state != nil { if state := updater.VerifyStartup(); state != nil {
if state.Status == "success" { notifier.NotifyControllerUpdated(state.PreviousVersion, state.TargetVersion, state.Status == "success")
notifier.NotifyUpdateSuccess(state.PreviousVersion, state.TargetVersion)
} else if state.Status == "failed" {
notifier.NotifyUpdateFailed(state.TargetVersion, state.Error)
}
} }
logger.Printf("[INFO] Self-update enabled (check every %s, auto-update: %v, auto-update time: %s)", logger.Printf("[INFO] Self-update enabled (check every %s, auto-update: %v, auto-update time: %s)",
cfg.SelfUpdate.CheckInterval, cfg.SelfUpdate.AutoUpdate, cfg.SelfUpdate.AutoUpdateTime) cfg.SelfUpdate.CheckInterval, cfg.SelfUpdate.AutoUpdate, cfg.SelfUpdate.AutoUpdateTime)
@@ -302,6 +304,16 @@ func main() {
var hubPusher *report.Pusher var hubPusher *report.Pusher
if cfg.Hub.URL != "" && cfg.Hub.APIKey != "" { if cfg.Hub.URL != "" && cfg.Hub.APIKey != "" {
hubPusher = report.NewPusher(&cfg.Hub, logger) hubPusher = report.NewPusher(&cfg.Hub, logger)
// Wire hub push status into alert manager for dashboard alerts
alertMgr.SetHubPushStatus(func() web.HubPushStatusData {
s := hubPusher.GetStatus()
return web.HubPushStatusData{
LastAttempt: s.LastAttempt,
LastSuccess: s.LastSuccess,
LastError: s.LastError,
Consecutive: s.Consecutive,
}
})
} }
// Backup daily jobs // Backup daily jobs
@@ -310,6 +322,8 @@ func main() {
err := backupMgr.RunDBDumps(ctx) err := backupMgr.RunDBDumps(ctx)
if err != nil { if err != nil {
notifier.NotifyDBDumpFailed("Adatbázis mentés sikertelen", err.Error()) notifier.NotifyDBDumpFailed("Adatbázis mentés sikertelen", err.Error())
} else {
notifier.NotifyDBDumpCompleted(notify.DBDumpDetails{})
} }
return err return err
}) })
@@ -317,6 +331,8 @@ func main() {
err := backupMgr.RunBackup(ctx) err := backupMgr.RunBackup(ctx)
if err != nil { if err != nil {
notifier.NotifyBackupFailed("Biztonsági mentés sikertelen", err.Error()) notifier.NotifyBackupFailed("Biztonsági mentés sikertelen", err.Error())
} else {
notifier.NotifyBackupCompleted(notify.BackupDetails{})
} }
// Phase 3: Chain cross-drive backups immediately after restic (regardless of restic success) // Phase 3: Chain cross-drive backups immediately after restic (regardless of restic success)
// Daily jobs run every night; weekly jobs only on Sunday // Daily jobs run every night; weekly jobs only on Sunday
@@ -345,6 +361,8 @@ func main() {
err := backupMgr.RunIntegrityCheck(ctx) err := backupMgr.RunIntegrityCheck(ctx)
if err != nil { if err != nil {
notifier.NotifyIntegrityFailed("Mentés integritás ellenőrzés sikertelen", err.Error()) notifier.NotifyIntegrityFailed("Mentés integritás ellenőrzés sikertelen", err.Error())
} else {
notifier.NotifyIntegrityOK("Mentés integritás ellenőrzés sikeres")
} }
return err return err
}) })
@@ -454,6 +472,9 @@ func main() {
go func() { go func() {
time.Sleep(5 * time.Second) // Let all subsystems fully initialize time.Sleep(5 * time.Second) // Let all subsystems fully initialize
// Push controller startup event to Hub
notifier.NotifyControllerStarted(Version)
// Heartbeat ping // Heartbeat ping
pinger.Ping(cfg.Monitoring.PingUUIDs.Heartbeat, "startup") pinger.Ping(cfg.Monitoring.PingUUIDs.Heartbeat, "startup")
logger.Println("[INFO] Startup heartbeat ping sent") logger.Println("[INFO] Startup heartbeat ping sent")
@@ -533,7 +554,7 @@ func main() {
go func() { go func() {
prefs := sett.GetNotificationPrefs() prefs := sett.GetNotificationPrefs()
if prefs.Email != "" { if prefs.Email != "" {
if err := notifier.SyncPreferences(prefs.Email, prefs.EnabledEvents); err != nil { if err := notifier.SyncPreferences(prefs.Email, prefs.EnabledEvents, prefs.CooldownHours); err != nil {
logger.Printf("[WARN] Failed to sync notification preferences on startup: %v", err) logger.Printf("[WARN] Failed to sync notification preferences on startup: %v", err)
} }
} }
@@ -547,11 +568,22 @@ func main() {
}() }()
// --- Initialize API router --- // --- Initialize API router ---
apiRouter := api.NewRouter(cfg, *configPath, sett, stackMgr, syncer, cpuCollector, backupMgr, crossDriveRunner, metricsStore, updater, logger) apiRouter := api.NewRouter(cfg, *configPath, sett, stackMgr, syncer, cpuCollector, backupMgr, crossDriveRunner, metricsStore, updater, notifier, logger)
// --- Initialize web server --- // --- Initialize web server ---
webServer := web.NewServer(cfg, stackMgr, cpuCollector, backupMgr, crossDriveRunner, sched, sett, alertMgr, notifier, updater, logger, Version) webServer := web.NewServer(cfg, stackMgr, cpuCollector, backupMgr, crossDriveRunner, sched, sett, alertMgr, notifier, updater, logger, Version)
webServer.SetStorageWatchdog(storageWatchdog) webServer.SetStorageWatchdog(storageWatchdog)
if hubPusher != nil {
webServer.SetHubPushStatus(func() web.HubPushStatusData {
s := hubPusher.GetStatus()
return web.HubPushStatusData{
LastAttempt: s.LastAttempt,
LastSuccess: s.LastSuccess,
LastError: s.LastError,
Consecutive: s.Consecutive,
}
})
}
// --- Initialize drive migrator --- // --- Initialize drive migrator ---
driveMigrator := &storage.DriveMigrator{ driveMigrator := &storage.DriveMigrator{
+18 -2
View File
@@ -16,6 +16,7 @@ import (
"gitea.dooplex.hu/admin/felhom-controller/internal/backup" "gitea.dooplex.hu/admin/felhom-controller/internal/backup"
"gitea.dooplex.hu/admin/felhom-controller/internal/config" "gitea.dooplex.hu/admin/felhom-controller/internal/config"
"gitea.dooplex.hu/admin/felhom-controller/internal/metrics" "gitea.dooplex.hu/admin/felhom-controller/internal/metrics"
"gitea.dooplex.hu/admin/felhom-controller/internal/notify"
"gitea.dooplex.hu/admin/felhom-controller/internal/selfupdate" "gitea.dooplex.hu/admin/felhom-controller/internal/selfupdate"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings" "gitea.dooplex.hu/admin/felhom-controller/internal/settings"
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks" "gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
@@ -35,11 +36,12 @@ type Router struct {
crossDriveRunner *backup.CrossDriveRunner crossDriveRunner *backup.CrossDriveRunner
metricsStore *metrics.MetricsStore metricsStore *metrics.MetricsStore
updater *selfupdate.Updater updater *selfupdate.Updater
notifier *notify.Notifier
logger *log.Logger logger *log.Logger
} }
func NewRouter(cfg *config.Config, configPath string, sett *settings.Settings, stackMgr *stacks.Manager, syncer *catalogsync.Syncer, cpuCollector *system.CPUCollector, backupMgr *backup.Manager, crossDrive *backup.CrossDriveRunner, metricsStore *metrics.MetricsStore, updater *selfupdate.Updater, logger *log.Logger) *Router { func NewRouter(cfg *config.Config, configPath string, sett *settings.Settings, stackMgr *stacks.Manager, syncer *catalogsync.Syncer, cpuCollector *system.CPUCollector, backupMgr *backup.Manager, crossDrive *backup.CrossDriveRunner, metricsStore *metrics.MetricsStore, updater *selfupdate.Updater, notif *notify.Notifier, logger *log.Logger) *Router {
return &Router{cfg: cfg, configPath: configPath, sett: sett, stackMgr: stackMgr, syncer: syncer, cpuCollector: cpuCollector, backupMgr: backupMgr, crossDriveRunner: crossDrive, metricsStore: metricsStore, updater: updater, logger: logger} return &Router{cfg: cfg, configPath: configPath, sett: sett, stackMgr: stackMgr, syncer: syncer, cpuCollector: cpuCollector, backupMgr: backupMgr, crossDriveRunner: crossDrive, metricsStore: metricsStore, updater: updater, notifier: notif, logger: logger}
} }
type apiResponse struct { type apiResponse struct {
@@ -280,6 +282,15 @@ func (r *Router) deployStack(w http.ResponseWriter, req *http.Request, name stri
resp.Data = map[string]string{"warning": warning} resp.Data = map[string]string{"warning": warning}
} }
writeJSON(w, http.StatusOK, resp) writeJSON(w, http.StatusOK, resp)
// Push app deployed event to Hub
if r.notifier != nil {
displayName := name
if s, ok := r.stackMgr.GetStack(name); ok && s.Meta.DisplayName != "" {
displayName = s.Meta.DisplayName
}
r.notifier.NotifyAppDeployed(name, displayName)
}
} }
func (r *Router) actionStack(w http.ResponseWriter, action, name string) { func (r *Router) actionStack(w http.ResponseWriter, action, name string) {
@@ -438,6 +449,11 @@ func (r *Router) removeStack(w http.ResponseWriter, req *http.Request, name stri
} }
writeJSON(w, http.StatusOK, apiResponse{OK: true, Data: resp, Message: "Stack " + name + " removed"}) writeJSON(w, http.StatusOK, apiResponse{OK: true, Data: resp, Message: "Stack " + name + " removed"})
// Push app removed event to Hub
if r.notifier != nil {
r.notifier.NotifyAppRemoved(name, name)
}
} }
func (r *Router) deleteStack(w http.ResponseWriter, req *http.Request, name string) { func (r *Router) deleteStack(w http.ResponseWriter, req *http.Request, name string) {
+358 -207
View File
@@ -7,15 +7,15 @@ import (
"io" "io"
"log" "log"
"net/http" "net/http"
"strings"
"sync" "sync"
"time" "time"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings" "gitea.dooplex.hu/admin/felhom-controller/internal/settings"
) )
// Notifier sends notification events to the hub relay service. // Notifier sends structured events to the hub via /api/v1/event.
// Non-blocking: fires requests in goroutines, logs errors but doesn't retry aggressively. // Non-blocking: fires requests in goroutines, logs errors but doesn't retry aggressively.
// Cooldown logic is handled by the Hub — the controller sends all events unconditionally.
type Notifier struct { type Notifier struct {
hubURL string hubURL string
apiKey string apiKey string
@@ -26,11 +26,7 @@ type Notifier struct {
settings *settings.Settings settings *settings.Settings
mu sync.Mutex mu sync.Mutex
cooldowns map[string]time.Time // event_type -> last notification time prevHealthStatus string // tracks previous health check status for change detection
perEventCooldown map[string]time.Duration // per-event override cooldown durations
// prevHealthStatus tracks the previous health check status for change detection
prevHealthStatus string
} }
// New creates a new Notifier. Returns a no-op notifier if hub is not enabled. // New creates a new Notifier. Returns a no-op notifier if hub is not enabled.
@@ -50,11 +46,6 @@ func New(hubURL, apiKey, customerID string, sett *settings.Settings, logger *log
logger: logger, logger: logger,
enabled: enabled, enabled: enabled,
settings: sett, settings: sett,
cooldowns: make(map[string]time.Time),
perEventCooldown: map[string]time.Duration{
"storage_disconnected": 1 * time.Hour,
"storage_reconnected": 1 * time.Hour,
},
} }
} }
@@ -63,17 +54,310 @@ func (n *Notifier) IsEnabled() bool {
return n.enabled return n.enabled
} }
// preferencesRequest is the JSON payload sent to the hub preferences endpoint. // ── Detail structs ───────────────────────────────────────────────────
// BackupDetails holds structured data for backup events.
type BackupDetails struct {
DriveCount int `json:"drive_count,omitempty"`
SnapshotID string `json:"snapshot_id,omitempty"`
DurationSec int `json:"duration_sec,omitempty"`
DataAdded string `json:"data_added,omitempty"`
Error string `json:"error,omitempty"`
}
// DBDumpDetails holds structured data for DB dump events.
type DBDumpDetails struct {
DatabaseCount int `json:"database_count,omitempty"`
TotalSize string `json:"total_size,omitempty"`
DurationSec int `json:"duration_sec,omitempty"`
Error string `json:"error,omitempty"`
}
// DiskDetails holds structured data for disk warning/critical events.
type DiskDetails struct {
Mount string `json:"mount,omitempty"`
UsagePercent float64 `json:"usage_percent,omitempty"`
Label string `json:"label,omitempty"`
}
// HealthDetails holds structured data for health events.
type HealthDetails struct {
PreviousStatus string `json:"previous_status,omitempty"`
CurrentStatus string `json:"current_status,omitempty"`
Issues []string `json:"issues,omitempty"`
Warnings []string `json:"warnings,omitempty"`
}
// StorageDetails holds structured data for storage events.
type StorageDetails struct {
DrivePath string `json:"drive_path,omitempty"`
Label string `json:"label,omitempty"`
StoppedApps []string `json:"stopped_apps,omitempty"`
}
// UpdateDetails holds structured data for controller update events.
type UpdateDetails struct {
FromVersion string `json:"from_version,omitempty"`
ToVersion string `json:"to_version,omitempty"`
Error string `json:"error,omitempty"`
}
// AppDetails holds structured data for app lifecycle events.
type AppDetails struct {
StackName string `json:"stack_name,omitempty"`
DisplayName string `json:"display_name,omitempty"`
}
// CrossDriveDetails holds structured data for cross-drive backup events.
type CrossDriveDetails struct {
StackName string `json:"stack_name,omitempty"`
Method string `json:"method,omitempty"`
DestPath string `json:"dest_path,omitempty"`
Duration string `json:"duration,omitempty"`
Error string `json:"error,omitempty"`
}
// ── Core event push ──────────────────────────────────────────────────
// eventRequest is the JSON payload sent to /api/v1/event.
type eventRequest struct {
CustomerID string `json:"customer_id"`
EventType string `json:"event_type"`
Severity string `json:"severity"`
Message string `json:"message"`
Details json.RawMessage `json:"details,omitempty"`
}
// PushEvent sends a structured event to the hub's /api/v1/event endpoint.
// Non-blocking (goroutine). Retries twice with 3s backoff.
// details may be nil (omitted from JSON) or a struct that marshals to JSON.
func (n *Notifier) PushEvent(eventType, severity, message string, details interface{}) {
if !n.enabled {
return
}
var detailsJSON json.RawMessage
if details != nil {
b, err := json.Marshal(details)
if err != nil {
n.logger.Printf("[WARN] PushEvent: failed to marshal details for %s: %v", eventType, err)
} else {
detailsJSON = b
}
}
payload := eventRequest{
CustomerID: n.customerID,
EventType: eventType,
Severity: severity,
Message: message,
Details: detailsJSON,
}
jsonData, err := json.Marshal(payload)
if err != nil {
n.logger.Printf("[ERROR] PushEvent: marshal failed for %s: %v", eventType, err)
return
}
go func() {
url := n.hubURL + "/api/v1/event"
var lastErr error
for attempt := 0; attempt < 3; attempt++ {
if attempt > 0 {
time.Sleep(3 * time.Second)
}
req, err := http.NewRequest("POST", url, bytes.NewReader(jsonData))
if err != nil {
lastErr = err
continue
}
req.Header.Set("Authorization", "Bearer "+n.apiKey)
req.Header.Set("Content-Type", "application/json")
resp, err := n.httpClient.Do(req)
if err != nil {
lastErr = err
continue
}
io.Copy(io.Discard, resp.Body)
resp.Body.Close()
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
n.logger.Printf("[INFO] Event pushed: %s (%s) — %s", eventType, severity, message)
return
}
lastErr = fmt.Errorf("HTTP %d", resp.StatusCode)
}
n.logger.Printf("[WARN] Event push failed after 3 attempts (%s/%s): %v", eventType, severity, lastErr)
}()
}
// ── Convenience methods ──────────────────────────────────────────────
// NotifyHealthChange checks if health status changed and sends appropriate events.
// Detects both degradation (ok→warn, ok→fail, warn→fail) and recovery (fail→ok, warn→ok, fail→warn).
func (n *Notifier) NotifyHealthChange(status string, issues, warnings []string) {
if !n.enabled {
return
}
n.mu.Lock()
prev := n.prevHealthStatus
n.prevHealthStatus = status
n.mu.Unlock()
if prev == "" {
return // First run, just record status
}
if status == prev {
return
}
details := HealthDetails{
PreviousStatus: prev,
CurrentStatus: status,
Issues: issues,
Warnings: warnings,
}
prevRank := statusRank(prev)
newRank := statusRank(status)
if newRank > prevRank {
// Degradation
if status == "fail" {
n.PushEvent("health_critical", "error",
fmt.Sprintf("Rendszer állapot kritikus (volt: %s)", prev), details)
} else if status == "warn" {
n.PushEvent("health_degraded", "warning",
fmt.Sprintf("Rendszer állapot romlott (volt: %s)", prev), details)
}
} else {
// Recovery
n.PushEvent("health_recovered", "info",
fmt.Sprintf("Rendszer állapot helyreállt: %s (volt: %s)", status, prev), details)
}
}
// NotifyBackupFailed sends a backup failure event.
func (n *Notifier) NotifyBackupFailed(message, errMsg string) {
n.PushEvent("backup_failed", "error", message, BackupDetails{Error: errMsg})
}
// NotifyBackupCompleted sends a backup success event.
func (n *Notifier) NotifyBackupCompleted(details BackupDetails) {
n.PushEvent("backup_completed", "info", "Biztonsági mentés elkészült", details)
}
// NotifyDBDumpFailed sends a DB dump failure event.
func (n *Notifier) NotifyDBDumpFailed(message, errMsg string) {
n.PushEvent("db_dump_failed", "error", message, DBDumpDetails{Error: errMsg})
}
// NotifyDBDumpCompleted sends a DB dump success event.
func (n *Notifier) NotifyDBDumpCompleted(details DBDumpDetails) {
n.PushEvent("db_dump_completed", "info", "Adatbázis mentés elkészült", details)
}
// NotifyIntegrityFailed sends a backup integrity check failure event.
func (n *Notifier) NotifyIntegrityFailed(message, errMsg string) {
n.PushEvent("backup_integrity_failed", "error", message, &BackupDetails{Error: errMsg})
}
// NotifyIntegrityOK sends a backup integrity check success event.
func (n *Notifier) NotifyIntegrityOK(message string) {
n.PushEvent("backup_integrity_ok", "info", message, nil)
}
// NotifyControllerUpdated sends a controller update event.
func (n *Notifier) NotifyControllerUpdated(fromVer, toVer string, success bool) {
severity := "info"
msg := fmt.Sprintf("Controller frissítve: %s → %s", fromVer, toVer)
details := UpdateDetails{FromVersion: fromVer, ToVersion: toVer}
if !success {
severity = "error"
msg = fmt.Sprintf("Controller frissítés sikertelen: %s → %s", fromVer, toVer)
}
n.PushEvent("controller_updated", severity, msg, details)
}
// NotifyControllerStarted sends a controller startup event.
func (n *Notifier) NotifyControllerStarted(version string) {
n.PushEvent("controller_started", "info",
fmt.Sprintf("Controller elindult (v%s)", version), nil)
}
// NotifyStorageDisconnected sends a drive disconnection event.
func (n *Notifier) NotifyStorageDisconnected(label string, stoppedApps []string) {
msg := fmt.Sprintf("Meghajtó váratlanul leválasztva: %s", label)
n.PushEvent("storage_disconnected", "error", msg, StorageDetails{
Label: label,
StoppedApps: stoppedApps,
})
}
// NotifyStorageReconnected sends a drive reconnection event.
func (n *Notifier) NotifyStorageReconnected(label string) {
n.PushEvent("storage_reconnected", "info",
fmt.Sprintf("Meghajtó újra csatlakoztatva: %s", label), StorageDetails{Label: label})
}
// NotifyAppDeployed sends an app deployment event.
func (n *Notifier) NotifyAppDeployed(stackName, displayName string) {
n.PushEvent("app_deployed", "info",
fmt.Sprintf("Alkalmazás telepítve: %s", displayName),
AppDetails{StackName: stackName, DisplayName: displayName})
}
// NotifyAppRemoved sends an app removal event.
func (n *Notifier) NotifyAppRemoved(stackName, displayName string) {
n.PushEvent("app_removed", "info",
fmt.Sprintf("Alkalmazás eltávolítva: %s", displayName),
AppDetails{StackName: stackName, DisplayName: displayName})
}
// NotifyCrossDriveCompleted sends a cross-drive backup success event.
func (n *Notifier) NotifyCrossDriveCompleted(details CrossDriveDetails) {
n.PushEvent("crossdrive_completed", "info",
fmt.Sprintf("Másodlagos mentés elkészült: %s", details.StackName), details)
}
// NotifyCrossDriveFailed sends a cross-drive backup failure event.
func (n *Notifier) NotifyCrossDriveFailed(details CrossDriveDetails) {
n.PushEvent("crossdrive_failed", "error",
fmt.Sprintf("Másodlagos mentés sikertelen: %s", details.StackName), details)
}
// NotifyDRStarted sends a disaster recovery start event.
func (n *Notifier) NotifyDRStarted(appCount int) {
n.PushEvent("disaster_recovery_started", "warning",
fmt.Sprintf("Katasztrófa helyreállítás elindítva (%d alkalmazás)", appCount), nil)
}
// NotifyDRCompleted sends a disaster recovery completion event.
func (n *Notifier) NotifyDRCompleted(successCount, failCount int) {
severity := "info"
if failCount > 0 {
severity = "warning"
}
n.PushEvent("disaster_recovery_completed", severity,
fmt.Sprintf("Katasztrófa helyreállítás befejezve (%d sikeres, %d sikertelen)", successCount, failCount), nil)
}
// ── Preferences sync ─────────────────────────────────────────────────
type preferencesRequest struct { type preferencesRequest struct {
CustomerID string `json:"customer_id"` CustomerID string `json:"customer_id"`
Email string `json:"email"` Email string `json:"email"`
EnabledEvents []string `json:"enabled_events"` EnabledEvents []string `json:"enabled_events"`
CooldownHours int `json:"cooldown_hours,omitempty"`
} }
// SyncPreferences pushes the current notification preferences to the hub. // SyncPreferences pushes the current notification preferences to the hub.
// Called after the user saves notification settings on the settings page.
// Synchronous — returns error for the handler to display to the user. // Synchronous — returns error for the handler to display to the user.
func (n *Notifier) SyncPreferences(email string, enabledEvents []string) error { func (n *Notifier) SyncPreferences(email string, enabledEvents []string, cooldownHours int) error {
if !n.enabled { if !n.enabled {
return fmt.Errorf("hub nem konfigurált") return fmt.Errorf("hub nem konfigurált")
} }
@@ -82,6 +366,7 @@ func (n *Notifier) SyncPreferences(email string, enabledEvents []string) error {
CustomerID: n.customerID, CustomerID: n.customerID,
Email: email, Email: email,
EnabledEvents: enabledEvents, EnabledEvents: enabledEvents,
CooldownHours: cooldownHours,
} }
jsonData, err := json.Marshal(payload) jsonData, err := json.Marshal(payload)
@@ -108,172 +393,23 @@ func (n *Notifier) SyncPreferences(email string, enabledEvents []string) error {
return fmt.Errorf("hub hiba (%d): %s", resp.StatusCode, string(body)) return fmt.Errorf("hub hiba (%d): %s", resp.StatusCode, string(body))
} }
n.logger.Printf("[INFO] Notification preferences synced to hub: email=%s, events=%v", email, enabledEvents) n.logger.Printf("[INFO] Notification preferences synced to hub: email=%s, events=%v, cooldown=%dh", email, enabledEvents, cooldownHours)
return nil return nil
} }
// notifyRequest is the JSON payload sent to the hub. // ── Test notification ────────────────────────────────────────────────
type notifyRequest struct {
CustomerID string `json:"customer_id"`
EventType string `json:"event_type"`
Severity string `json:"severity"`
Message string `json:"message"`
Details string `json:"details,omitempty"`
}
// Notify sends a notification event to the hub relay. // SendTest sends a test event for verifying the notification flow (synchronous).
// Checks local cooldown and event preferences before sending.
// Non-blocking: fires the HTTP request in a goroutine.
func (n *Notifier) Notify(eventType, severity, message, details string) {
if !n.enabled {
return
}
prefs := n.settings.GetNotificationPrefs()
if prefs.Email == "" {
return // No email configured, skip
}
// Check if event is enabled in preferences
eventEnabled := false
for _, e := range prefs.EnabledEvents {
if e == eventType {
eventEnabled = true
break
}
}
if !eventEnabled {
return
}
// Check cooldown — per-event override takes priority over global
cooldownDuration := time.Duration(prefs.CooldownHours) * time.Hour
if cooldownDuration == 0 {
cooldownDuration = 6 * time.Hour
}
if override, ok := n.perEventCooldown[eventType]; ok {
cooldownDuration = override
}
n.mu.Lock()
lastSent, exists := n.cooldowns[eventType]
if exists && time.Since(lastSent) < cooldownDuration {
n.mu.Unlock()
n.logger.Printf("[DEBUG] Notification cooldown active for %s (sent %s ago)", eventType, time.Since(lastSent).Round(time.Minute))
return
}
n.cooldowns[eventType] = time.Now()
n.mu.Unlock()
// Fire the notification in a goroutine (non-blocking)
go func() {
payload := notifyRequest{
CustomerID: n.customerID,
EventType: eventType,
Severity: severity,
Message: message,
Details: details,
}
jsonData, err := json.Marshal(payload)
if err != nil {
n.logger.Printf("[ERROR] Failed to marshal notification: %v", err)
return
}
url := n.hubURL + "/api/v1/notify"
req, err := http.NewRequest("POST", url, bytes.NewReader(jsonData))
if err != nil {
n.logger.Printf("[ERROR] Failed to create notification request: %v", err)
return
}
req.Header.Set("Authorization", "Bearer "+n.apiKey)
req.Header.Set("Content-Type", "application/json")
resp, err := n.httpClient.Do(req)
if err != nil {
n.logger.Printf("[WARN] Failed to send notification to hub: %v", err)
return
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
n.logger.Printf("[WARN] Hub notification returned %d for %s/%s", resp.StatusCode, eventType, severity)
return
}
n.logger.Printf("[INFO] Notification sent: %s (%s) — %s", eventType, severity, message)
}()
}
// NotifyHealthChange checks if health status changed and sends appropriate notifications.
// Call this after each health check with the new report status/issues/warnings.
func (n *Notifier) NotifyHealthChange(status string, issues, warnings []string) {
if !n.enabled {
return
}
prev := n.prevHealthStatus
n.prevHealthStatus = status
// Only notify on status degradation (ok→warn, ok→fail, warn→fail)
if prev == "" {
return // First run, just record status
}
if statusRank(status) <= statusRank(prev) {
return // Status improved or stayed the same
}
// Notify about each issue/warning
for _, issue := range issues {
n.Notify("container_unhealthy", "critical", issue, "")
}
for _, w := range warnings {
// Determine specific event type from warning message
eventType := classifyWarning(w)
n.Notify(eventType, "warning", w, "")
}
}
// NotifyBackupFailed sends a notification about a backup failure.
func (n *Notifier) NotifyBackupFailed(message, details string) {
n.Notify("backup_failed", "critical", message, details)
}
// NotifyDBDumpFailed sends a notification about a database dump failure.
func (n *Notifier) NotifyDBDumpFailed(message, details string) {
n.Notify("db_dump_failed", "critical", message, details)
}
// NotifyIntegrityFailed sends a notification about a backup integrity check failure.
func (n *Notifier) NotifyIntegrityFailed(message, details string) {
n.Notify("integrity_failed", "warning", message, details)
}
// NotifyUpdateSuccess sends a notification about a successful controller update.
func (n *Notifier) NotifyUpdateSuccess(fromVer, toVer string) {
n.Notify("update_success", "info",
fmt.Sprintf("Controller frissítve: %s → %s", fromVer, toVer), "")
}
// NotifyUpdateFailed sends a notification about a failed controller update.
func (n *Notifier) NotifyUpdateFailed(targetVer, errMsg string) {
n.Notify("update_failed", "warning",
fmt.Sprintf("Controller frissítés sikertelen: %s — %s", targetVer, errMsg), "")
}
// SendTest sends a test notification for verifying the notification flow.
func (n *Notifier) SendTest() error { func (n *Notifier) SendTest() error {
if !n.enabled { if !n.enabled {
return fmt.Errorf("notifications not enabled (hub not configured)") return fmt.Errorf("notifications not enabled (hub not configured)")
} }
payload := notifyRequest{ payload := eventRequest{
CustomerID: n.customerID, CustomerID: n.customerID,
EventType: "test", EventType: "test",
Severity: "info", Severity: "info",
Message: "Teszt értesítés a Felhom rendszerből", Message: "Teszt értesítés a Felhom rendszerből",
Details: "Ha ezt az emailt megkapta, az értesítések megfelelően működnek.",
} }
jsonData, err := json.Marshal(payload) jsonData, err := json.Marshal(payload)
@@ -281,7 +417,7 @@ func (n *Notifier) SendTest() error {
return fmt.Errorf("marshal: %w", err) return fmt.Errorf("marshal: %w", err)
} }
url := n.hubURL + "/api/v1/notify" url := n.hubURL + "/api/v1/event"
req, err := http.NewRequest("POST", url, bytes.NewReader(jsonData)) req, err := http.NewRequest("POST", url, bytes.NewReader(jsonData))
if err != nil { if err != nil {
return fmt.Errorf("request: %w", err) return fmt.Errorf("request: %w", err)
@@ -302,6 +438,59 @@ func (n *Notifier) SendTest() error {
return nil return nil
} }
// ── Backward compatibility ───────────────────────────────────────────
// notifyRequest is the JSON payload for the legacy /api/v1/notify endpoint.
type notifyRequest struct {
CustomerID string `json:"customer_id"`
EventType string `json:"event_type"`
Severity string `json:"severity"`
Message string `json:"message"`
Details string `json:"details,omitempty"`
}
// Notify sends a legacy notification to /api/v1/notify (backward compat).
// Kept for old Hub instances that don't support /api/v1/event yet.
// No local cooldown — Hub handles cooldowns.
func (n *Notifier) Notify(eventType, severity, message, details string) {
if !n.enabled {
return
}
go func() {
payload := notifyRequest{
CustomerID: n.customerID,
EventType: eventType,
Severity: severity,
Message: message,
Details: details,
}
jsonData, err := json.Marshal(payload)
if err != nil {
n.logger.Printf("[ERROR] Failed to marshal notification: %v", err)
return
}
url := n.hubURL + "/api/v1/notify"
req, err := http.NewRequest("POST", url, bytes.NewReader(jsonData))
if err != nil {
return
}
req.Header.Set("Authorization", "Bearer "+n.apiKey)
req.Header.Set("Content-Type", "application/json")
resp, err := n.httpClient.Do(req)
if err != nil {
return
}
io.Copy(io.Discard, resp.Body)
resp.Body.Close()
}()
}
// ── Helpers ──────────────────────────────────────────────────────────
func statusRank(status string) int { func statusRank(status string) int {
switch status { switch status {
case "ok": case "ok":
@@ -315,41 +504,3 @@ func statusRank(status string) int {
} }
} }
func classifyWarning(message string) string {
// Try to classify the warning message into a specific event type
switch {
case contains(message, "disk") || contains(message, "Disk") || contains(message, "SSD") || contains(message, "HDD"):
if contains(message, "critical") || contains(message, "Critical") {
return "disk_critical"
}
return "disk_warning"
case contains(message, "Memory") || contains(message, "memory"):
return "disk_warning" // group memory under system warnings
case contains(message, "Temperature") || contains(message, "temperature"):
return "disk_warning" // group temp under system warnings
case contains(message, "container") || contains(message, "Container"):
return "container_unhealthy"
default:
return "disk_warning" // fallback to generic system warning
}
}
func contains(s, substr string) bool {
return strings.Contains(s, substr)
}
// NotifyStorageDisconnected sends a notification about a drive disconnection.
func (n *Notifier) NotifyStorageDisconnected(label string, stoppedApps []string) {
msg := fmt.Sprintf("Meghajtó váratlanul leválasztva: %s", label)
details := ""
if len(stoppedApps) > 0 {
details = fmt.Sprintf("Leállított alkalmazások: %s", strings.Join(stoppedApps, ", "))
}
n.Notify("storage_disconnected", "critical", msg, details)
}
// NotifyStorageReconnected sends a notification about a drive reconnection.
func (n *Notifier) NotifyStorageReconnected(label string) {
n.Notify("storage_reconnected", "info",
fmt.Sprintf("Meghajtó újra csatlakoztatva: %s. Az alkalmazások manuálisan indíthatók.", label), "")
}
+33
View File
@@ -8,11 +8,20 @@ import (
"log" "log"
"net/http" "net/http"
"strings" "strings"
"sync"
"time" "time"
"gitea.dooplex.hu/admin/felhom-controller/internal/config" "gitea.dooplex.hu/admin/felhom-controller/internal/config"
) )
// PushStatus tracks the last hub push attempt and result.
type PushStatus struct {
LastAttempt time.Time
LastSuccess time.Time
LastError string
Consecutive int // consecutive failures
}
// Pusher sends reports to the central hub. // Pusher sends reports to the central hub.
type Pusher struct { type Pusher struct {
hubURL string hubURL string
@@ -20,6 +29,9 @@ type Pusher struct {
httpClient *http.Client httpClient *http.Client
logger *log.Logger logger *log.Logger
enabled bool enabled bool
statusMu sync.RWMutex
status PushStatus
} }
// NewPusher creates a new report pusher from hub configuration. // NewPusher creates a new report pusher from hub configuration.
@@ -48,6 +60,10 @@ func (p *Pusher) Push(report *Report) error {
url := p.hubURL + "/api/v1/report" url := p.hubURL + "/api/v1/report"
p.statusMu.Lock()
p.status.LastAttempt = time.Now()
p.statusMu.Unlock()
var lastErr error var lastErr error
for attempt := 0; attempt < 3; attempt++ { for attempt := 0; attempt < 3; attempt++ {
if attempt > 0 { if attempt > 0 {
@@ -74,14 +90,31 @@ func (p *Pusher) Push(report *Report) error {
if resp.StatusCode >= 200 && resp.StatusCode < 300 { if resp.StatusCode >= 200 && resp.StatusCode < 300 {
p.logger.Printf("[INFO] Hub report pushed successfully (%d bytes)", len(data)) p.logger.Printf("[INFO] Hub report pushed successfully (%d bytes)", len(data))
p.statusMu.Lock()
p.status.LastSuccess = time.Now()
p.status.LastError = ""
p.status.Consecutive = 0
p.statusMu.Unlock()
return nil return nil
} }
lastErr = fmt.Errorf("HTTP %d", resp.StatusCode) lastErr = fmt.Errorf("HTTP %d", resp.StatusCode)
} }
p.statusMu.Lock()
p.status.LastError = lastErr.Error()
p.status.Consecutive++
p.statusMu.Unlock()
return fmt.Errorf("hub push failed after 3 attempts: %w", lastErr) return fmt.Errorf("hub push failed after 3 attempts: %w", lastErr)
} }
// GetStatus returns a snapshot of the current push status.
func (p *Pusher) GetStatus() PushStatus {
p.statusMu.RLock()
defer p.statusMu.RUnlock()
return p.status
}
// PushInfraBackup sends the infrastructure backup payload to the Hub. // PushInfraBackup sends the infrastructure backup payload to the Hub.
// Uses the same retry logic as Push. // Uses the same retry logic as Push.
func (p *Pusher) PushInfraBackup(data []byte) error { func (p *Pusher) PushInfraBackup(data []byte) error {
+7 -3
View File
@@ -85,11 +85,15 @@ type NotificationPrefs struct {
// DefaultEnabledEvents are the events enabled by default for new customers. // DefaultEnabledEvents are the events enabled by default for new customers.
var DefaultEnabledEvents = []string{ var DefaultEnabledEvents = []string{
"disk_warning",
"backup_failed", "backup_failed",
"update_available", "db_dump_failed",
"disk_warning",
"disk_critical",
"storage_disconnected", "storage_disconnected",
"storage_reconnected", "node_down",
"health_critical",
"expected_backup_missed",
"expected_dbdump_missed",
} }
// DBValidationCache holds cached DB dump validation results. // DBValidationCache holds cached DB dump validation results.
+27 -28
View File
@@ -5,6 +5,7 @@ import (
"log" "log"
"strings" "strings"
"sync" "sync"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/backup" "gitea.dooplex.hu/admin/felhom-controller/internal/backup"
"gitea.dooplex.hu/admin/felhom-controller/internal/config" "gitea.dooplex.hu/admin/felhom-controller/internal/config"
@@ -27,9 +28,10 @@ type Alert struct {
// Alerts are state-based (not event-based) — they reflect current system state // Alerts are state-based (not event-based) — they reflect current system state
// and are regenerated after each health check cycle. // and are regenerated after each health check cycle.
type AlertManager struct { type AlertManager struct {
mu sync.RWMutex mu sync.RWMutex
alerts []Alert alerts []Alert
logger *log.Logger logger *log.Logger
hubPushStatusFn func() HubPushStatusData
} }
// NewAlertManager creates a new AlertManager. // NewAlertManager creates a new AlertManager.
@@ -39,6 +41,13 @@ func NewAlertManager(logger *log.Logger) *AlertManager {
} }
} }
// SetHubPushStatus sets the hub push status callback for generating hub alerts.
func (am *AlertManager) SetHubPushStatus(fn func() HubPushStatusData) {
am.mu.Lock()
am.hubPushStatusFn = fn
am.mu.Unlock()
}
// Refresh regenerates alerts from the latest health check report and config state. // Refresh regenerates alerts from the latest health check report and config state.
// Called after each health check cycle (every 5 minutes) and on storage state changes. // Called after each health check cycle (every 5 minutes) and on storage state changes.
func (am *AlertManager) Refresh(report *monitor.HealthReport, cfg *config.Config, backupMgr *backup.Manager, updateAvailable bool, latestVersion string, storagePaths ...[]settings.StoragePath) { func (am *AlertManager) Refresh(report *monitor.HealthReport, cfg *config.Config, backupMgr *backup.Manager, updateAvailable bool, latestVersion string, storagePaths ...[]settings.StoragePath) {
@@ -92,14 +101,22 @@ func (am *AlertManager) Refresh(report *monitor.HealthReport, cfg *config.Config
alerts = append(alerts, alert) alerts = append(alerts, alert)
} }
// Missing ping UUIDs // Hub connection status
if cfg.Monitoring.Enabled { if !cfg.Hub.Enabled || cfg.Hub.URL == "" {
missing := countMissingPings(cfg) alerts = append(alerts, Alert{
if missing > 0 { ID: "hub-disabled",
Level: "warning",
Message: "Hub kapcsolat kikapcsolva — a központi monitoring nem aktív",
Link: "/monitoring",
LinkText: "Rendszermonitor",
})
} else if am.hubPushStatusFn != nil {
ps := am.hubPushStatusFn()
if ps.LastError != "" && (ps.LastSuccess.IsZero() || time.Since(ps.LastSuccess) > 30*time.Minute) {
alerts = append(alerts, Alert{ alerts = append(alerts, Alert{
ID: "pings-missing", ID: "hub-unreachable",
Level: "warning", Level: "error",
Message: fmt.Sprintf("%d monitoring ellenőrzés nincs beállítva", missing), Message: fmt.Sprintf("Hub nem elérhető — utolsó hiba: %s", ps.LastError),
Link: "/monitoring", Link: "/monitoring",
LinkText: "Rendszermonitor", LinkText: "Rendszermonitor",
}) })
@@ -200,24 +217,6 @@ func (am *AlertManager) GetInlineAlerts(page string) []Alert {
return result return result
} }
// countMissingPings counts how many ping UUIDs are not configured.
func countMissingPings(cfg *config.Config) int {
count := 0
uuids := []string{
cfg.Monitoring.PingUUIDs.Heartbeat,
cfg.Monitoring.PingUUIDs.SystemHealth,
cfg.Monitoring.PingUUIDs.DBDump,
cfg.Monitoring.PingUUIDs.Backup,
cfg.Monitoring.PingUUIDs.BackupIntegrity,
}
for _, uuid := range uuids {
if !isPingConfigured(uuid) {
count++
}
}
return count
}
// simpleHash returns a short deterministic hash for deduplication. // simpleHash returns a short deterministic hash for deduplication.
func simpleHash(s string) string { func simpleHash(s string) string {
h := uint32(0) h := uint32(0)
@@ -110,6 +110,18 @@ func (s *Server) executeAllRestores() {
return return
} }
// Count pending apps and push DR start event
pendingCount := 0
for _, app := range plan.Apps {
if app.Status == "pending" {
pendingCount++
}
}
if s.notifier != nil {
s.notifier.NotifyDRStarted(pendingCount)
}
successCount, failCount := 0, 0
for i := range plan.Apps { for i := range plan.Apps {
app := &plan.Apps[i] app := &plan.Apps[i]
if app.Status != "pending" { if app.Status != "pending" {
@@ -126,15 +138,22 @@ func (s *Server) executeAllRestores() {
if err != nil { if err != nil {
plan.UpdateApp(app.Name, "failed", err.Error()) plan.UpdateApp(app.Name, "failed", err.Error())
s.logger.Printf("[ERROR] Restore failed for %s: %v", app.Name, err) s.logger.Printf("[ERROR] Restore failed for %s: %v", app.Name, err)
failCount++
} else { } else {
plan.UpdateApp(app.Name, "done", "") plan.UpdateApp(app.Name, "done", "")
s.logger.Printf("[INFO] Restore completed for %s", app.Name) s.logger.Printf("[INFO] Restore completed for %s", app.Name)
successCount++
} }
} }
plan.SetStatus("done") plan.SetStatus("done")
s.logger.Println("[INFO] All app restores completed") s.logger.Println("[INFO] All app restores completed")
// Push DR completion event
if s.notifier != nil {
s.notifier.NotifyDRCompleted(successCount, failCount)
}
// Re-scan stacks so dashboard picks up restored apps // Re-scan stacks so dashboard picks up restored apps
if s.stackMgr != nil { if s.stackMgr != nil {
if err := s.stackMgr.ScanStacks(); err != nil { if err := s.stackMgr.ScanStacks(); err != nil {
+38 -10
View File
@@ -411,21 +411,36 @@ func (s *Server) monitoringHandler(w http.ResponseWriter, _ *http.Request) {
data["SystemInfo"] = system.GetInfo(s.primaryHDDPath(), s.cpuCollector) data["SystemInfo"] = system.GetInfo(s.primaryHDDPath(), s.cpuCollector)
data["StorageBars"] = s.buildStorageBars() data["StorageBars"] = s.buildStorageBars()
// On monitoring page, exclude the "pings-missing" alert since the detailed table is visible
if s.alertManager != nil { if s.alertManager != nil {
data["Alerts"] = s.alertManager.GetAlerts("pings-missing") data["Alerts"] = s.alertManager.GetAlerts()
data["DiskWarnings"] = s.alertManager.GetInlineAlerts("monitoring") data["DiskWarnings"] = s.alertManager.GetInlineAlerts("monitoring")
} }
// Ping status section // Hub connection status section
data["HubEnabled"] = s.cfg.Hub.Enabled && s.cfg.Hub.URL != ""
data["HubURL"] = s.cfg.Hub.URL
data["CustomerID"] = s.cfg.Customer.ID
if s.hubPushStatusFn != nil {
ps := s.hubPushStatusFn()
data["HubLastAttempt"] = ps.LastAttempt
data["HubLastSuccess"] = ps.LastSuccess
data["HubLastError"] = ps.LastError
data["HubConsecutiveFailures"] = ps.Consecutive
// Connected if last success was within 2x the push interval (or 30min default)
connected := !ps.LastSuccess.IsZero() && time.Since(ps.LastSuccess) < 30*time.Minute
data["HubConnected"] = connected
}
// Legacy ping status section (still shown for backward compat during transition)
data["MonitoringEnabled"] = s.cfg.Monitoring.Enabled data["MonitoringEnabled"] = s.cfg.Monitoring.Enabled
if s.cfg.Monitoring.Enabled { if s.cfg.Monitoring.Enabled {
pings := []map[string]interface{}{ pings := []map[string]interface{}{
{"Label": "Életjel (Heartbeat)", "Icon": "💓", "Configured": isPingConfigured(s.cfg.Monitoring.PingUUIDs.Heartbeat), "Schedule": "5 percenként"}, {"Label": "Eletjel (Heartbeat)", "Icon": "heartbeat", "Configured": isPingConfigured(s.cfg.Monitoring.PingUUIDs.Heartbeat), "Schedule": "5 percenkent"},
{"Label": "Rendszer állapot", "Icon": "🖥️", "Configured": isPingConfigured(s.cfg.Monitoring.PingUUIDs.SystemHealth), "Schedule": "5 percenként"}, {"Label": "Rendszer allapot", "Icon": "system", "Configured": isPingConfigured(s.cfg.Monitoring.PingUUIDs.SystemHealth), "Schedule": "5 percenkent"},
{"Label": "Adatbázis mentés", "Icon": "🗄️", "Configured": isPingConfigured(s.cfg.Monitoring.PingUUIDs.DBDump), "Schedule": "Naponta " + s.cfg.Backup.DBDumpSchedule}, {"Label": "Adatbazis mentes", "Icon": "db", "Configured": isPingConfigured(s.cfg.Monitoring.PingUUIDs.DBDump), "Schedule": "Naponta " + s.cfg.Backup.DBDumpSchedule},
{"Label": "Biztonsági mentés", "Icon": "💾", "Configured": isPingConfigured(s.cfg.Monitoring.PingUUIDs.Backup), "Schedule": "Naponta " + s.cfg.Backup.ResticSchedule}, {"Label": "Biztonsagi mentes", "Icon": "backup", "Configured": isPingConfigured(s.cfg.Monitoring.PingUUIDs.Backup), "Schedule": "Naponta " + s.cfg.Backup.ResticSchedule},
{"Label": "Mentés integritás", "Icon": "🔍", "Configured": isPingConfigured(s.cfg.Monitoring.PingUUIDs.BackupIntegrity), "Schedule": "Hetente (vasárnap)"}, {"Label": "Mentes integritas", "Icon": "integrity", "Configured": isPingConfigured(s.cfg.Monitoring.PingUUIDs.BackupIntegrity), "Schedule": "Hetente (vasarnap)"},
} }
allConfigured := true allConfigured := true
for _, p := range pings { for _, p := range pings {
@@ -1076,11 +1091,24 @@ func (s *Server) settingsNotificationsHandler(w http.ResponseWriter, r *http.Req
// Collect enabled events from checkboxes // Collect enabled events from checkboxes
var enabledEvents []string var enabledEvents []string
for _, evt := range []string{"disk_warning", "backup_failed", "update_available", "security_update"} { // Single-event checkboxes
for _, evt := range []string{
"backup_failed", "db_dump_failed", "backup_integrity_failed",
"crossdrive_failed", "storage_disconnected",
"node_down", "health_critical",
"storage_reconnected", "health_recovered",
} {
if r.FormValue("event_"+evt) == "on" { if r.FormValue("event_"+evt) == "on" {
enabledEvents = append(enabledEvents, evt) enabledEvents = append(enabledEvents, evt)
} }
} }
// Compound toggles: one checkbox → two event types
if r.FormValue("event_disk_alerts") == "on" {
enabledEvents = append(enabledEvents, "disk_warning", "disk_critical")
}
if r.FormValue("event_expected_missed") == "on" {
enabledEvents = append(enabledEvents, "expected_backup_missed", "expected_dbdump_missed")
}
prefs := &settings.NotificationPrefs{ prefs := &settings.NotificationPrefs{
Email: email, Email: email,
@@ -1101,7 +1129,7 @@ func (s *Server) settingsNotificationsHandler(w http.ResponseWriter, r *http.Req
// Sync preferences to hub // Sync preferences to hub
data := s.settingsData() data := s.settingsData()
if s.notifier != nil && s.notifier.IsEnabled() { if s.notifier != nil && s.notifier.IsEnabled() {
if err := s.notifier.SyncPreferences(email, enabledEvents); err != nil { if err := s.notifier.SyncPreferences(email, enabledEvents, cooldownHours); err != nil {
s.logger.Printf("[WARN] Failed to sync preferences to hub: %v", err) s.logger.Printf("[WARN] Failed to sync preferences to hub: %v", err)
data["NotificationSuccess"] = fmt.Sprintf("Értesítési beállítások mentve (helyi). A központi szinkronizálás sikertelen: %v", err) data["NotificationSuccess"] = fmt.Sprintf("Értesítési beállítások mentve (helyi). A központi szinkronizálás sikertelen: %v", err)
} else { } else {
+17
View File
@@ -9,6 +9,7 @@ import (
"path/filepath" "path/filepath"
"strings" "strings"
"sync" "sync"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/backup" "gitea.dooplex.hu/admin/felhom-controller/internal/backup"
"gitea.dooplex.hu/admin/felhom-controller/internal/config" "gitea.dooplex.hu/admin/felhom-controller/internal/config"
@@ -57,6 +58,9 @@ type Server struct {
// Storage watchdog (set after construction to break init ordering) // Storage watchdog (set after construction to break init ordering)
storageWatchdog *monitor.StorageWatchdog storageWatchdog *monitor.StorageWatchdog
// Hub push status callback — set via SetHubPushStatus for monitoring page
hubPushStatusFn func() HubPushStatusData
} }
func NewServer(cfg *config.Config, stackMgr *stacks.Manager, cpuCollector *system.CPUCollector, backupMgr *backup.Manager, crossDrive *backup.CrossDriveRunner, sched *scheduler.Scheduler, sett *settings.Settings, alertMgr *AlertManager, notif *notify.Notifier, updater *selfupdate.Updater, logger *log.Logger, version string) *Server { func NewServer(cfg *config.Config, stackMgr *stacks.Manager, cpuCollector *system.CPUCollector, backupMgr *backup.Manager, crossDrive *backup.CrossDriveRunner, sched *scheduler.Scheduler, sett *settings.Settings, alertMgr *AlertManager, notif *notify.Notifier, updater *selfupdate.Updater, logger *log.Logger, version string) *Server {
@@ -117,6 +121,19 @@ func (s *Server) SetDriveMigrator(dm *storage.DriveMigrator) {
s.driveMigrator = dm s.driveMigrator = dm
} }
// HubPushStatusData holds hub push status for the monitoring page.
type HubPushStatusData struct {
LastAttempt time.Time
LastSuccess time.Time
LastError string
Consecutive int
}
// SetHubPushStatus sets the hub push status callback for the monitoring page.
func (s *Server) SetHubPushStatus(fn func() HubPushStatusData) {
s.hubPushStatusFn = fn
}
// InRestoreMode returns true if the server is in DR restore mode. // InRestoreMode returns true if the server is in DR restore mode.
func (s *Server) InRestoreMode() bool { func (s *Server) InRestoreMode() bool {
s.restoreMu.RLock() s.restoreMu.RLock()
@@ -86,33 +86,44 @@
{{end}} {{end}}
</div> </div>
<!-- Section 2: Remote Monitoring Status --> <!-- Section 2: Hub Connection Status -->
<div class="monitor-card"> <div class="monitor-card">
<h3>Távoli monitoring</h3> <h3>Hub kapcsolat</h3>
{{if not .MonitoringEnabled}} {{if .HubEnabled}}
<div class="monitoring-banner monitoring-banner-red"> {{if .HubConnected}}
⚠️ A távoli monitoring ki van kapcsolva. Az üzemeltető nem kap értesítést hibák esetén.
</div>
{{else}}
{{if .AllPingsConfigured}}
<div class="monitoring-banner monitoring-banner-green"> <div class="monitoring-banner monitoring-banner-green">
✅ Minden távoli monitoring aktív — az üzemeltető értesítést kap hibák esetén. Kapcsolódva — a központi rendszer aktívan figyeli a szervert.
</div> </div>
{{else}} {{else}}
<div class="monitoring-banner monitoring-banner-yellow"> <div class="monitoring-banner monitoring-banner-red">
⚠️ Egyes monitoring ellenőrzések nincsenek beállítva. Kérd az üzemeltetőt a konfiguráláshoz. Nem elérhető — a központi rendszer nem kapott friss jelentést.
</div> </div>
{{end}} {{end}}
<div class="sysinfo-grid" style="margin-top: 0.75rem"> <div class="sysinfo-grid" style="margin-top: 0.75rem">
{{range .PingStatus}}
<div class="sysinfo-row"> <div class="sysinfo-row">
<span class="sysinfo-label">{{.Icon}} {{.Label}}</span> <span class="sysinfo-label">Hub URL</span>
<span class="sysinfo-value"> <span class="sysinfo-value"><code>{{.HubURL}}</code></span>
{{if .Configured}}<span class="ping-status-ok">✅ Beállítva</span>{{else}}<span class="ping-status-warn">⚠️ Nincs beállítva</span>{{end}} </div>
<span class="ping-schedule">{{.Schedule}}</span> <div class="sysinfo-row">
</span> <span class="sysinfo-label">Ügyfél azonosító</span>
<span class="sysinfo-value"><code>{{.CustomerID}}</code></span>
</div>
{{if not .HubLastSuccess.IsZero}}
<div class="sysinfo-row">
<span class="sysinfo-label">Utolsó sikeres jelentés</span>
<span class="sysinfo-value">{{.HubLastSuccess | timeAgo}}</span>
</div> </div>
{{end}} {{end}}
{{if .HubLastError}}
<div class="sysinfo-row">
<span class="sysinfo-label">Utolsó hiba</span>
<span class="sysinfo-value"><span class="text-error">{{.HubLastError}}</span></span>
</div>
{{end}}
</div>
{{else}}
<div class="monitoring-banner monitoring-banner-yellow">
A Hub kapcsolat nincs bekapcsolva — a központi monitoring nem aktív.
</div> </div>
{{end}} {{end}}
</div> </div>
@@ -413,23 +413,56 @@ function pollUntilBack() {
placeholder="pelda@email.hu" class="form-control"> placeholder="pelda@email.hu" class="form-control">
</div> </div>
<div class="form-group"> <div class="form-group">
<label>Az alábbi eseményekről kapjon értesítést:</label> <label>Hibák és figyelmeztetések:</label>
<div class="checkbox-group"> <div class="checkbox-group">
<label class="toggle">
<input type="checkbox" name="event_disk_warning" {{with .NotificationPrefs}}{{range .EnabledEvents}}{{if eq . "disk_warning"}}checked{{end}}{{end}}{{end}}>
<span class="toggle-label">Lemez figyelmeztetés (80%+)</span>
</label>
<label class="toggle"> <label class="toggle">
<input type="checkbox" name="event_backup_failed" {{with .NotificationPrefs}}{{range .EnabledEvents}}{{if eq . "backup_failed"}}checked{{end}}{{end}}{{end}}> <input type="checkbox" name="event_backup_failed" {{with .NotificationPrefs}}{{range .EnabledEvents}}{{if eq . "backup_failed"}}checked{{end}}{{end}}{{end}}>
<span class="toggle-label">Biztonsági mentés sikertelen</span> <span class="toggle-label">Biztonsági mentés sikertelen</span>
</label> </label>
<label class="toggle"> <label class="toggle">
<input type="checkbox" name="event_update_available" {{with .NotificationPrefs}}{{range .EnabledEvents}}{{if eq . "update_available"}}checked{{end}}{{end}}{{end}}> <input type="checkbox" name="event_db_dump_failed" {{with .NotificationPrefs}}{{range .EnabledEvents}}{{if eq . "db_dump_failed"}}checked{{end}}{{end}}{{end}}>
<span class="toggle-label">Frissítés elérhető</span> <span class="toggle-label">Adatbázis mentés sikertelen</span>
</label> </label>
<label class="toggle"> <label class="toggle">
<input type="checkbox" name="event_security_update" {{with .NotificationPrefs}}{{range .EnabledEvents}}{{if eq . "security_update"}}checked{{end}}{{end}}{{end}}> <input type="checkbox" name="event_backup_integrity_failed" {{with .NotificationPrefs}}{{range .EnabledEvents}}{{if eq . "backup_integrity_failed"}}checked{{end}}{{end}}{{end}}>
<span class="toggle-label">Biztonsági frissítés</span> <span class="toggle-label">Mentés sérülés észlelve</span>
</label>
<label class="toggle">
<input type="checkbox" name="event_crossdrive_failed" {{with .NotificationPrefs}}{{range .EnabledEvents}}{{if eq . "crossdrive_failed"}}checked{{end}}{{end}}{{end}}>
<span class="toggle-label">Másodlagos mentés sikertelen</span>
</label>
<label class="toggle">
<input type="checkbox" name="event_disk_alerts" {{with .NotificationPrefs}}{{range .EnabledEvents}}{{if eq . "disk_warning"}}checked{{end}}{{end}}{{end}}>
<span class="toggle-label">Lemez figyelmeztetés (90%+)</span>
</label>
<label class="toggle">
<input type="checkbox" name="event_storage_disconnected" {{with .NotificationPrefs}}{{range .EnabledEvents}}{{if eq . "storage_disconnected"}}checked{{end}}{{end}}{{end}}>
<span class="toggle-label">Meghajtó leválasztva</span>
</label>
<label class="toggle">
<input type="checkbox" name="event_node_down" {{with .NotificationPrefs}}{{range .EnabledEvents}}{{if eq . "node_down"}}checked{{end}}{{end}}{{end}}>
<span class="toggle-label">Szerver nem elérhető</span>
</label>
<label class="toggle">
<input type="checkbox" name="event_health_critical" {{with .NotificationPrefs}}{{range .EnabledEvents}}{{if eq . "health_critical"}}checked{{end}}{{end}}{{end}}>
<span class="toggle-label">Rendszer állapot kritikus</span>
</label>
<label class="toggle">
<input type="checkbox" name="event_expected_missed" {{with .NotificationPrefs}}{{range .EnabledEvents}}{{if eq . "expected_backup_missed"}}checked{{end}}{{end}}{{end}}>
<span class="toggle-label">Elvárt mentés elmaradt</span>
</label>
</div>
</div>
<div class="form-group">
<label>Tájékoztató:</label>
<div class="checkbox-group">
<label class="toggle">
<input type="checkbox" name="event_storage_reconnected" {{with .NotificationPrefs}}{{range .EnabledEvents}}{{if eq . "storage_reconnected"}}checked{{end}}{{end}}{{end}}>
<span class="toggle-label">Meghajtó újra csatlakoztatva</span>
</label>
<label class="toggle">
<input type="checkbox" name="event_health_recovered" {{with .NotificationPrefs}}{{range .EnabledEvents}}{{if eq . "health_recovered"}}checked{{end}}{{end}}{{end}}>
<span class="toggle-label">Rendszer állapot helyreállt</span>
</label> </label>
</div> </div>
</div> </div>