feat: storage watchdog — USB disconnect detection, auto-stop, safe eject, auto-reconnect (v0.17.0)

New storage watchdog monitors registered storage paths every 5s. On disconnect
(3 consecutive probe failures), auto-stops affected apps, lazy-unmounts stale
VFS entries, fires alerts/notifications/hub report. On reconnect (UUID detected),
auto-remounts via fstab, cleans stale restic locks, offers app restart.

Safe disconnect UI for USB drives: confirmation dialog, stop apps, sync, unmount.
Disconnected state visible across all pages (dashboard, settings, backups, monitoring)
with hatched red bars and badges. Backup guards skip disconnected drives.

22 files changed (1 new: monitor/watchdog.go), ~1500 lines added.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-02-19 19:42:26 +01:00
parent 276be5a88e
commit bdbe170a54
22 changed files with 1537 additions and 57 deletions
+48 -8
View File
@@ -85,7 +85,7 @@ A single, lightweight Go container that replaces Portainer + scattered systemd s
| **Backup** | `internal/backup/` | Per-drive 3-layer backup: DB dumps → restic snapshots → cross-drive copies, restore |
| **Storage** | `internal/storage/` | Disk scanning (`lsblk`), partitioning (`sfdisk`), formatting (`mkfs.ext4`), mounting, data migration (`rsync`) |
| **System** | `internal/system/` | System info (`/proc`), CPU collector, mount points, disk usage, FS info |
| **Monitor** | `internal/monitor/` | Healthchecks.io pinger, system health checks |
| **Monitor** | `internal/monitor/` | Healthchecks.io pinger, system health checks, storage watchdog |
| **Metrics** | `internal/metrics/` | SQLite time-series store, system + container metric collection |
| **Scheduler** | `internal/scheduler/` | Central job scheduler (periodic + daily, skip-if-running, panic recovery) |
| **SelfUpdate** | `internal/selfupdate/` | Version checking (registry), update trigger, state persistence, startup verification |
@@ -403,8 +403,9 @@ Multiple external storage paths supported with:
- **Label**: Human-readable name (editable inline)
- **Default flag**: New deploys use this path by default
- **Schedulable flag**: Path appears in deploy dropdown
- **Disconnected state**: `Disconnected`, `DisconnectedAt`, `StoppedStacks` — set by watchdog or safe-disconnect API, cleared on reconnect
- **Auto-discovery**: On startup, scans deployed apps' `HDD_PATH` values and registers unknown paths
- Thread-safe CRUD: Add, Remove, SetDefault, SetSchedulable, SetLabel
- Thread-safe CRUD: Add, Remove, SetDefault, SetSchedulable, SetLabel, SetDisconnected, ClearDisconnected
#### Data Migration (`internal/storage/migrate.go`)
@@ -431,6 +432,39 @@ After migration, the deploy page detects leftover data on previous storage paths
When storage paths are added or removed, `syncFileBrowserMounts()` auto-regenerates FileBrowser's `docker-compose.yml` with volume mounts for all registered paths, then recreates the container.
#### Storage Watchdog (`internal/monitor/watchdog.go`)
Continuously monitors registered storage paths for disconnection/reconnection (primarily USB drives):
- **Probe loop**: `ProbeStoragePath()` calls `syscall.Statfs()` with 3-second timeout in a goroutine. Runs every 5s per connected path, 30s per disconnected path.
- **Debouncing**: 3 consecutive probe failures required before declaring a drive disconnected (prevents false positives from transient I/O).
- **Disconnect reaction** (automatic, ~15s detection):
1. Stops all deployed stacks whose `HDD_PATH` is under the disconnected drive (skips protected stacks)
2. Persists `Disconnected`, `DisconnectedAt`, `StoppedStacks` to `settings.json`
3. Lazy-unmounts stale VFS entries (`umount -l`) — for attach-wizard drives, unmounts bind first, then raw
4. Fires alert refresh (red banner on all pages), notification (`storage_disconnected`), and immediate hub report push
- **Auto-reconnect** (for UUID-based fstab entries):
1. Checks `/host-dev/disk/by-uuid/<uuid>` for device reappearance
2. Cleans stale mounts, then `mount -T /host-fstab <path>` (raw + bind for attach-wizard drives)
3. Verifies with a post-mount probe
4. Runs `restic unlock` if stale lock files exist
5. Validates `StoppedStacks` (filters to actually-stopped stacks), clears `Disconnected` flag
6. Fires alert refresh, notification (`storage_reconnected`), hub report push
**Safe disconnect UI** (manual, Settings page):
- "Leválasztás" button shown for USB drives (detected via sysfs symlink path containing `/usb`)
- Confirmation dialog lists affected apps
- Flow: stop apps → `sync` → `umount` (fallback `umount -l`) → mark disconnected → notification
- Disconnected card: dashed border, red badge, timestamp, stopped apps list, "Csatlakoztatás" (reconnect) button
- After reconnect: "Alkalmazások indítása" button to restart auto-stopped stacks
**USB detection** (`system.IsUSBDevice`): Reads `/host/sys/block/<disk>` symlink — if target path contains `/usb`, it's a USB device. The `removable` sysfs flag is unreliable for USB HDDs (returns 0).
**Backup guards**: Nightly DB dumps, restic snapshots, and cross-drive backups all skip disconnected drives with WARN log (not treated as failures).
**UI integration**: Disconnected drives show with hatched red bars on dashboard, monitoring, and backup pages. Per-app backup rows show "Meghajtó leválasztva" badge. Health check emits warnings for disconnected paths.
---
### 4. Monitoring & Health
@@ -446,7 +480,7 @@ When storage paths are added or removed, `syncFileBrowserMounts()` auto-regenera
| CPU temperature | >= 75C | >= 85C |
| Docker daemon | — | unreachable |
| Protected containers | — | not running |
| Storage paths | not a mount point (data on SSD) | path inaccessible, disk >= 95% |
| Storage paths | not a mount point (data on SSD), drive disconnected | path inaccessible, disk >= 95% |
Backup destination validation (`CheckBackupDestination`) has tiered checks:
- Path doesn't exist → critical/blocked
@@ -694,7 +728,7 @@ Runtime-mutable settings in `settings.json` (separate from infrastructure config
| `notifications` | email, enabled events, cooldown hours |
| `db_validations` | per-DB dump validation results (survives restarts) |
| `app_backup` | per-app map: enabled flag, cross-drive config (method, dest, schedule, runtime status) |
| `storage_paths` | registered paths with label, default flag, schedulable flag |
| `storage_paths` | registered paths with label, default flag, schedulable flag, disconnected state |
| `cross_drive_restic_password` | auto-generated restic password for cross-drive repos |
All public methods use `sync.RWMutex`. File writes are atomic (`.tmp` + rename).
@@ -704,7 +738,7 @@ All public methods use `sync.RWMutex`. File writes are atomic (`.tmp` + rename).
Five sections:
1. **System config** — read-only display of `controller.yaml` values
2. **Version & update** — current/latest version, check/update buttons, auto-update status, last update result
3. **Storage paths** — add/remove, edit labels, set default, toggle schedulable, per-path app list with sizes
3. **Storage paths** — add/remove, edit labels, set default, toggle schedulable, per-path app list with sizes, safe disconnect/reconnect for USB drives
4. **Password change** — current + new + confirm, min 8 chars
5. **Notifications** — email, event checkboxes, cooldown hours, test email button
@@ -805,10 +839,11 @@ controller/
│ ├── system/
│ │ ├── info.go, info_linux.go # RAM, disk, CPU, temperature, load average
│ │ ├── cpu_linux.go # Background /proc/stat sampling
│ │ └── mounts_linux.go # Mount points, disk usage, FS info, backup dest checks
│ │ └── mounts_linux.go # Mount points, disk usage, FS info, backup dest checks, storage probing, USB detection
│ ├── monitor/
│ │ ├── pinger.go # Healthchecks.io HTTP ping client
│ │ ── healthcheck.go # System health checks (disk, mem, CPU, temp, Docker)
│ │ ── healthcheck.go # System health checks (disk, mem, CPU, temp, Docker)
│ │ └── watchdog.go # Storage watchdog (probe, disconnect/reconnect, safe eject)
│ ├── metrics/
│ │ ├── store.go # SQLite time-series (WAL mode, downsampled queries)
│ │ ├── collector.go # Background collector (60s, system + docker stats)
@@ -827,7 +862,7 @@ controller/
│ ├── auth.go # Session auth, login/logout, session cleanup
│ ├── handlers.go # Page handlers (dashboard, stacks, deploy, backups, etc.)
│ ├── handler_restore.go # DR: restore page handler + APIs (scan, restore all, skip)
│ ├── storage_handlers.go # Storage API handlers (scan, format, attach, migrate, cleanup)
│ ├── storage_handlers.go # Storage API handlers (scan, format, attach, migrate, cleanup, disconnect/reconnect)
│ ├── alerts.go # State-based alert generation
│ ├── funcmap.go # Template functions (state colors, Hungarian formatting)
│ ├── embed.go # go:embed for templates + Chart.js
@@ -973,6 +1008,10 @@ All daily jobs use Europe/Budapest timezone. Skip-if-running prevents concurrent
| POST | `/api/storage/attach/cancel` | Cleanup temp raw mount |
| POST | `/api/storage/migrate` | Start app data migration |
| GET | `/api/storage/migrate/status` | Migration progress |
| POST | `/api/storage/disconnect` | Safe disconnect (stop apps, unmount) |
| POST | `/api/storage/reconnect` | Reconnect disconnected drive |
| POST | `/api/storage/restart-apps` | Restart auto-stopped apps |
| GET | `/api/storage/status` | All storage paths with connection state |
### Self-Update
@@ -1060,6 +1099,7 @@ See `docker-compose.yml` for the full volume configuration.
- [x] Storage management (scan, format, mount, registry)
- [x] Attach existing drive wizard (v0.15.0) — bind-mount subfolder from pre-formatted drive, directory browser
- [x] App data migration between storage paths
- [x] Storage watchdog (v0.17.0) — USB disconnect detection (~15s), auto-stop apps, auto-remount on reconnect, safe eject UI
- [x] Central hub reporting
- [x] Email notifications via hub relay
- [x] Settings persistence and password management
+69 -1
View File
@@ -284,7 +284,7 @@ func main() {
latestVersion = status.LastCheck.LatestVersion
}
}
alertMgr.Refresh(healthReport, cfg, backupMgr, updateAvailable, latestVersion)
alertMgr.Refresh(healthReport, cfg, backupMgr, updateAvailable, latestVersion, sett.GetStoragePaths())
// Notify on health status changes
notifier.NotifyHealthChange(healthReport.Status, healthReport.Issues, healthReport.Warnings)
return nil
@@ -409,6 +409,36 @@ func main() {
}
}
// --- Storage watchdog ---
storageWatchdog := monitor.NewStorageWatchdog(sett, &watchdogStackAdapter{mgr: stackMgr}, notifier, cfg, logger)
storageWatchdog.SetAlertRefresh(func() {
healthReport := monitor.RunHealthCheck(cfg, cpuCollector, sett.GetStoragePaths())
updateAvailable := false
latestVersion := ""
if updater != nil {
status := updater.GetStatus()
if status.LastCheck != nil {
updateAvailable = status.LastCheck.UpdateAvailable
latestVersion = status.LastCheck.LatestVersion
}
}
alertMgr.Refresh(healthReport, cfg, backupMgr, updateAvailable, latestVersion, sett.GetStoragePaths())
})
if hubPusher != nil {
storageWatchdog.SetHubReportPusher(func() {
r := report.BuildReport(cfg, stackMgr, backupMgr, cpuCollector, metricsStore, Version, sett.GetStoragePaths())
hubPusher.Push(r)
})
}
if backupMgr != nil {
storageWatchdog.SetRepoUnlocker(func(ctx context.Context, repoPath string) error {
return backupMgr.UnlockRepo(ctx, repoPath)
})
}
sched.Every("storage-watchdog", 5*time.Second, func(ctx context.Context) error {
return storageWatchdog.Check(ctx)
})
sched.Start(ctx)
defer sched.Stop()
@@ -513,6 +543,7 @@ func main() {
// --- Initialize web server ---
webServer := web.NewServer(cfg, stackMgr, cpuCollector, backupMgr, crossDriveRunner, sched, sett, alertMgr, notifier, updater, logger, Version)
webServer.SetStorageWatchdog(storageWatchdog)
// Phase 3: Set DR restore mode if a restore plan was built
if restorePlan != nil && len(restorePlan.Apps) > 0 {
@@ -673,6 +704,43 @@ func (a *stackAdapter) GetStackHDDPath(name string) string {
return ""
}
// watchdogStackAdapter implements monitor.WatchdogStackProvider using stacks.Manager.
type watchdogStackAdapter struct {
mgr *stacks.Manager
}
func (a *watchdogStackAdapter) ListDeployedStacks() []monitor.WatchdogStackInfo {
var result []monitor.WatchdogStackInfo
for _, s := range a.mgr.GetStacks() {
if !s.Deployed {
continue
}
result = append(result, monitor.WatchdogStackInfo{Name: s.Name})
}
return result
}
func (a *watchdogStackAdapter) GetStackHDDPath(name string) string {
s, ok := a.mgr.GetStack(name)
if !ok {
return ""
}
stackDir := filepath.Dir(s.ComposePath)
appCfg := stacks.LoadAppConfig(stackDir)
if appCfg != nil && appCfg.Env["HDD_PATH"] != "" {
return filepath.Clean(appCfg.Env["HDD_PATH"])
}
return ""
}
func (a *watchdogStackAdapter) StopStack(name string) error {
return a.mgr.StopStack(name)
}
func (a *watchdogStackAdapter) StartStack(name string) error {
return a.mgr.StartStack(name)
}
// pushInfraBackup builds and sends the infrastructure snapshot to the Hub.
func pushInfraBackup(cfg *config.Config, sett *settings.Settings,
stackProv *stackAdapter, pusher *report.Pusher, logger *log.Logger) {
+28
View File
@@ -242,6 +242,14 @@ func (m *Manager) runDBDumpsInternal(ctx context.Context) error {
for _, db := range dbs {
drivePath := m.GetAppDrivePath(db.StackName)
// Skip if drive is disconnected
if m.settings != nil && m.settings.IsDisconnected(drivePath) {
m.logger.Printf("[WARN] Skipping DB dump for %s — drive disconnected: %s", db.StackName, drivePath)
summary = append(summary, fmt.Sprintf("SKIP %s (drive disconnected)", db.ContainerName))
continue
}
dumpDir := AppDBDumpPath(drivePath, db.StackName)
result := DumpOne(ctx, db, dumpDir, m.logger)
@@ -331,6 +339,12 @@ func (m *Manager) runBackupInternal(ctx context.Context) error {
driveCount := 0
for drivePath, stacks := range driveStacks {
// Skip disconnected drives
if m.settings != nil && m.settings.IsDisconnected(drivePath) {
m.logger.Printf("[WARN] Skipping backup for drive %s — disconnected", drivePath)
continue
}
repoPath := PrimaryResticRepoPath(drivePath)
// Ensure repo is initialized
@@ -650,6 +664,20 @@ func (m *Manager) SetStackProvider(provider StackDataProvider) {
m.mu.Unlock()
}
// UnlockRepo runs restic unlock on the given repo path.
func (m *Manager) UnlockRepo(ctx context.Context, repoPath string) error {
if !m.restic.RepoExists(repoPath) {
return nil // no repo to unlock
}
cmd := m.restic.UnlockCommand(ctx, repoPath)
out, err := cmd.CombinedOutput()
if err != nil {
return fmt.Errorf("restic unlock: %v (%s)", err, strings.TrimSpace(string(out)))
}
m.logger.Printf("[INFO] Restic repo unlocked: %s", repoPath)
return nil
}
// GetStackHDDMounts returns HDD mount paths for the named stack via the stack provider.
func (m *Manager) GetStackHDDMounts(name string) []string {
if m.stackProvider == nil {
+15
View File
@@ -81,6 +81,21 @@ func (r *CrossDriveRunner) RunAppBackup(ctx context.Context, stackName string) e
r.mu.Unlock()
}()
// Check if source or destination drive is disconnected
srcDrive := r.stackProvider.GetStackHDDPath(stackName)
if srcDrive != "" && r.sett.IsDisconnected(srcDrive) {
r.mu.Lock()
r.running[stackName] = false
r.mu.Unlock()
return fmt.Errorf("source drive disconnected: %s", srcDrive)
}
if r.sett.IsDisconnected(cfg.DestinationPath) {
r.mu.Lock()
r.running[stackName] = false
r.mu.Unlock()
return fmt.Errorf("destination drive disconnected: %s", cfg.DestinationPath)
}
// Mark as running in settings
_ = r.sett.UpdateCrossDriveStatus(stackName, func(c *settings.CrossDriveBackup) {
c.LastStatus = "running"
+5
View File
@@ -348,6 +348,11 @@ func (r *ResticManager) RepoExists(repoPath string) bool {
return err == nil
}
// UnlockCommand returns an exec.Cmd that runs restic unlock on the given repo.
func (r *ResticManager) UnlockCommand(ctx context.Context, repoPath string) *exec.Cmd {
return r.command(ctx, repoPath, "unlock")
}
func (r *ResticManager) command(ctx context.Context, repoPath string, args ...string) *exec.Cmd {
cmd := exec.CommandContext(ctx, "restic", args...)
cmd.Env = append(os.Environ(),
@@ -172,6 +172,12 @@ func checkProtectedContainers(protected []string) []string {
func checkStoragePaths(paths []settings.StoragePath) (issues, warnings []string) {
for _, sp := range paths {
// Skip disconnected paths — handled by the storage watchdog
if sp.Disconnected {
warnings = append(warnings, fmt.Sprintf("Meghajtó leválasztva: %s (%s)", sp.Label, sp.Path))
continue
}
// Path accessible?
if _, err := os.Stat(sp.Path); err != nil {
warnings = append(warnings, fmt.Sprintf("Adattároló nem elérhető: %s", sp.Path))
+612
View File
@@ -0,0 +1,612 @@
package monitor
import (
"context"
"fmt"
"log"
"os"
"os/exec"
"path/filepath"
"strings"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
"gitea.dooplex.hu/admin/felhom-controller/internal/notify"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
"gitea.dooplex.hu/admin/felhom-controller/internal/system"
)
const (
// probeThreshold is the number of consecutive probe failures before declaring disconnected.
probeThreshold = 3
// defaultProbeInterval is the normal probe interval for connected drives.
defaultProbeInterval = 5 * time.Second
// disconnectedProbeInterval is the slower probe interval for disconnected drives
// (checking for UUID reappearance, not I/O probing).
disconnectedProbeInterval = 30 * time.Second
// hostFstabPath is where the host's fstab is mounted inside the container.
hostFstabPath = "/host-fstab"
// hostDevUUIDPath is where the host's /dev/disk/by-uuid is accessible.
hostDevUUIDPath = "/host-dev/disk/by-uuid"
// primaryResticSubpath is the relative path to the primary restic repo under a drive.
primaryResticSubpath = "backups/primary/restic"
)
// WatchdogStackInfo holds minimal stack info for the watchdog.
type WatchdogStackInfo struct {
Name string
}
// WatchdogStackProvider provides stack operations needed by the watchdog.
// Defined here to avoid circular imports with the backup package.
type WatchdogStackProvider interface {
ListDeployedStacks() []WatchdogStackInfo
GetStackHDDPath(name string) string
StopStack(name string) error
StartStack(name string) error
}
// pathProbeState tracks in-memory probe state for a single storage path.
type pathProbeState struct {
consecutiveFailures int
lastStatus string // "connected", "disconnected"
lastProbeTime time.Time
probeInterval time.Duration
}
// StorageWatchdog monitors registered storage paths and reacts to disconnection/reconnection.
type StorageWatchdog struct {
settings *settings.Settings
stackProvider WatchdogStackProvider
notifier *notify.Notifier
cfg *config.Config
logger *log.Logger
// Callbacks to break import cycles — set via SetXxx methods after construction
alertRefresh func()
pushHubReport func()
unlockRepo func(ctx context.Context, repoPath string) error
mu sync.Mutex
pathState map[string]*pathProbeState
}
// NewStorageWatchdog creates a new storage watchdog.
func NewStorageWatchdog(
sett *settings.Settings,
stackProvider WatchdogStackProvider,
notifier *notify.Notifier,
cfg *config.Config,
logger *log.Logger,
) *StorageWatchdog {
return &StorageWatchdog{
settings: sett,
stackProvider: stackProvider,
notifier: notifier,
cfg: cfg,
logger: logger,
pathState: make(map[string]*pathProbeState),
}
}
// SetAlertRefresh sets the callback to trigger alert refresh.
func (w *StorageWatchdog) SetAlertRefresh(fn func()) {
w.alertRefresh = fn
}
// SetHubReportPusher sets the callback to push an immediate hub report.
func (w *StorageWatchdog) SetHubReportPusher(fn func()) {
w.pushHubReport = fn
}
// SetRepoUnlocker sets the callback to unlock a restic repo on reconnect.
func (w *StorageWatchdog) SetRepoUnlocker(fn func(ctx context.Context, repoPath string) error) {
w.unlockRepo = fn
}
// Check probes all registered storage paths and reacts to state changes.
// Called by the scheduler every 5 seconds.
func (w *StorageWatchdog) Check(ctx context.Context) error {
paths := w.settings.GetStoragePaths()
if len(paths) == 0 {
return nil
}
for _, sp := range paths {
select {
case <-ctx.Done():
return ctx.Err()
default:
}
state := w.getOrCreateState(sp.Path)
// Rate-limit per-path probes
if time.Since(state.lastProbeTime) < state.probeInterval {
continue
}
state.lastProbeTime = time.Now()
if sp.Disconnected {
w.handleReconnectCheck(ctx, sp)
} else {
w.handleConnectedProbe(sp, state)
}
}
return nil
}
// getOrCreateState returns the in-memory probe state for a path, creating if needed.
func (w *StorageWatchdog) getOrCreateState(path string) *pathProbeState {
w.mu.Lock()
defer w.mu.Unlock()
if s, ok := w.pathState[path]; ok {
return s
}
s := &pathProbeState{
lastStatus: "connected",
probeInterval: defaultProbeInterval,
}
w.pathState[path] = s
return s
}
// handleConnectedProbe probes a connected drive and triggers disconnect if needed.
func (w *StorageWatchdog) handleConnectedProbe(sp settings.StoragePath, state *pathProbeState) {
result := system.ProbeStoragePath(sp.Path)
if result.Status == system.ProbeConnected {
if state.consecutiveFailures > 0 {
w.logger.Printf("[DEBUG] [STORAGE] Probe recovered for %s after %d failures", sp.Path, state.consecutiveFailures)
}
state.consecutiveFailures = 0
state.lastStatus = "connected"
return
}
state.consecutiveFailures++
w.logger.Printf("[WARN] [STORAGE] Probe failed for %s (%d/%d): %v",
sp.Path, state.consecutiveFailures, probeThreshold, result.Err)
if state.consecutiveFailures >= probeThreshold {
w.handleDisconnect(sp, state, result)
}
}
// handleDisconnect reacts to a confirmed drive disconnection.
func (w *StorageWatchdog) handleDisconnect(sp settings.StoragePath, state *pathProbeState, probe system.ProbeResult) {
label := sp.Label
if label == "" {
label = sp.Path
}
w.logger.Printf("[ERROR] [STORAGE] Drive disconnected: %s (%s)", sp.Path, label)
// 1. Find and stop affected stacks
stoppedStacks := w.stopAffectedStacks(sp.Path)
// 2. Mark disconnected in settings (persists to settings.json)
if err := w.settings.SetDisconnected(sp.Path, true, stoppedStacks); err != nil {
w.logger.Printf("[ERROR] [STORAGE] Failed to mark disconnected: %v", err)
}
// 3. Lazy unmount stale mount (if probe timed out — mount is likely hanging)
if probe.Status == system.ProbeTimeout {
w.lazyUnmount(sp.Path)
}
// 4. Update in-memory state
state.lastStatus = "disconnected"
state.probeInterval = disconnectedProbeInterval
state.consecutiveFailures = 0
// 5. Trigger alert refresh
if w.alertRefresh != nil {
w.alertRefresh()
}
// 6. Send notification
w.notifier.NotifyStorageDisconnected(label, stoppedStacks)
// 7. Push immediate hub report
if w.pushHubReport != nil {
go w.pushHubReport()
}
}
// handleReconnectCheck checks if a disconnected drive has been reconnected.
func (w *StorageWatchdog) handleReconnectCheck(ctx context.Context, sp settings.StoragePath) {
// Find the UUID for this path from fstab
// For attach-wizard drives, the UUID is on the raw mount, not the bind mount
mountPath := sp.Path
rawPath, isAttachWizard := system.HasFelhomRawMount(hostFstabPath, sp.Path)
if isAttachWizard {
mountPath = rawPath
}
uuid := system.ParseFstabUUID(hostFstabPath, mountPath)
if uuid == "" {
// No UUID in fstab — can't detect reconnection automatically
return
}
// Check if the UUID block device is present
uuidPath := filepath.Join(hostDevUUIDPath, uuid)
if _, err := os.Stat(uuidPath); err != nil {
return // Drive not reconnected yet
}
label := sp.Label
if label == "" {
label = sp.Path
}
w.logger.Printf("[INFO] [STORAGE] Drive reconnected (UUID found), attempting remount: %s (%s)", sp.Path, label)
// Attempt remount
if err := w.remount(sp.Path, rawPath, isAttachWizard); err != nil {
w.logger.Printf("[ERROR] [STORAGE] Remount failed for %s: %v", sp.Path, err)
return // Try again next cycle
}
// Verify with a probe
verifyResult := system.ProbeStoragePath(sp.Path)
if verifyResult.Status != system.ProbeConnected {
w.logger.Printf("[ERROR] [STORAGE] Post-remount probe failed for %s: %v", sp.Path, verifyResult.Err)
return
}
w.logger.Printf("[INFO] [STORAGE] Drive successfully remounted: %s (%s)", sp.Path, label)
// Clean stale restic locks
w.cleanResticLocks(ctx, sp.Path)
// Validate stopped stacks — filter to only actually stopped ones
filteredStacks := w.filterStoppedStacks(sp.StoppedStacks)
// Clear disconnected but preserve StoppedStacks for the restart UI
if err := w.settings.SetDisconnected(sp.Path, false, filteredStacks); err != nil {
w.logger.Printf("[ERROR] [STORAGE] Failed to clear disconnected: %v", err)
}
// Update in-memory state
state := w.getOrCreateState(sp.Path)
state.lastStatus = "connected"
state.probeInterval = defaultProbeInterval
state.consecutiveFailures = 0
// Trigger alert refresh
if w.alertRefresh != nil {
w.alertRefresh()
}
// Send notification
w.notifier.NotifyStorageReconnected(label)
// Push immediate hub report
if w.pushHubReport != nil {
go w.pushHubReport()
}
}
// stopAffectedStacks stops all deployed stacks whose HDD_PATH matches the disconnected drive.
func (w *StorageWatchdog) stopAffectedStacks(drivePath string) []string {
if w.stackProvider == nil {
return nil
}
var stopped []string
cleanDrive := filepath.Clean(drivePath)
for _, stack := range w.stackProvider.ListDeployedStacks() {
hddPath := w.stackProvider.GetStackHDDPath(stack.Name)
if hddPath == "" {
continue
}
cleanHDD := filepath.Clean(hddPath)
if cleanHDD != cleanDrive && !strings.HasPrefix(cleanHDD, cleanDrive+"/") {
continue
}
// Don't stop protected stacks
if w.cfg.IsProtectedStack(stack.Name) {
w.logger.Printf("[WARN] [STORAGE] Skipping protected stack: %s", stack.Name)
continue
}
w.logger.Printf("[INFO] [STORAGE] Stopping stack %s (drive disconnected: %s)", stack.Name, drivePath)
if err := w.stackProvider.StopStack(stack.Name); err != nil {
w.logger.Printf("[ERROR] [STORAGE] Failed to stop stack %s: %v", stack.Name, err)
continue // Don't add to stopped list if stop failed
}
stopped = append(stopped, stack.Name)
}
if len(stopped) > 0 {
w.logger.Printf("[INFO] [STORAGE] Stopped %d stack(s) due to drive disconnect: %v", len(stopped), stopped)
}
return stopped
}
// lazyUnmount performs a lazy unmount of a path and its raw mount (if attach-wizard).
func (w *StorageWatchdog) lazyUnmount(path string) {
// For attach-wizard, unmount bind first, then raw
rawPath, isAttachWizard := system.HasFelhomRawMount(hostFstabPath, path)
// Unmount the bind/main path
cmd := exec.Command("umount", "-l", path)
if out, err := cmd.CombinedOutput(); err != nil {
w.logger.Printf("[WARN] [STORAGE] umount -l %s: %v (%s)", path, err, strings.TrimSpace(string(out)))
} else {
w.logger.Printf("[INFO] [STORAGE] Lazy unmounted: %s", path)
}
// Then unmount the raw path if it's an attach-wizard drive
if isAttachWizard && rawPath != "" {
cmd = exec.Command("umount", "-l", rawPath)
if out, err := cmd.CombinedOutput(); err != nil {
w.logger.Printf("[WARN] [STORAGE] umount -l %s: %v (%s)", rawPath, err, strings.TrimSpace(string(out)))
} else {
w.logger.Printf("[INFO] [STORAGE] Lazy unmounted raw: %s", rawPath)
}
}
}
// remount attempts to remount a storage path using fstab entries.
func (w *StorageWatchdog) remount(path, rawPath string, isAttachWizard bool) error {
// Clean any stale mount entries first
exec.Command("umount", "-l", path).Run()
if isAttachWizard && rawPath != "" {
exec.Command("umount", "-l", rawPath).Run()
}
if isAttachWizard && rawPath != "" {
// Mount raw first, then bind
cmd := exec.Command("mount", "-T", hostFstabPath, rawPath)
if out, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf("mount raw %s: %v (%s)", rawPath, err, strings.TrimSpace(string(out)))
}
w.logger.Printf("[INFO] [STORAGE] Mounted raw: %s", rawPath)
cmd = exec.Command("mount", "-T", hostFstabPath, path)
if out, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf("mount bind %s: %v (%s)", path, err, strings.TrimSpace(string(out)))
}
w.logger.Printf("[INFO] [STORAGE] Mounted bind: %s", path)
} else {
cmd := exec.Command("mount", "-T", hostFstabPath, path)
if out, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf("mount %s: %v (%s)", path, err, strings.TrimSpace(string(out)))
}
w.logger.Printf("[INFO] [STORAGE] Mounted: %s", path)
}
return nil
}
// cleanResticLocks runs restic unlock on the primary repo for a drive path.
func (w *StorageWatchdog) cleanResticLocks(ctx context.Context, drivePath string) {
repoPath := filepath.Join(drivePath, primaryResticSubpath)
locksDir := filepath.Join(repoPath, "locks")
entries, err := os.ReadDir(locksDir)
if err != nil || len(entries) == 0 {
return // No locks dir or no lock files
}
w.logger.Printf("[INFO] [STORAGE] Found %d restic lock file(s) in %s, running unlock", len(entries), repoPath)
if w.unlockRepo != nil {
if err := w.unlockRepo(ctx, repoPath); err != nil {
w.logger.Printf("[WARN] [STORAGE] Restic unlock failed for %s: %v", repoPath, err)
}
}
}
// filterStoppedStacks validates that stacks in the list still exist as deployed stacks.
func (w *StorageWatchdog) filterStoppedStacks(stackNames []string) []string {
if w.stackProvider == nil || len(stackNames) == 0 {
return nil
}
deployed := make(map[string]bool)
for _, s := range w.stackProvider.ListDeployedStacks() {
deployed[s.Name] = true
}
var result []string
for _, name := range stackNames {
if deployed[name] {
result = append(result, name)
}
}
return result
}
// SafeDisconnect performs a safe disconnect of a storage path.
// Stops affected apps, syncs filesystem, and unmounts the drive.
func (w *StorageWatchdog) SafeDisconnect(ctx context.Context, path string) (stoppedStacks []string, err error) {
sp := w.findStoragePath(path)
if sp == nil {
return nil, fmt.Errorf("storage path %q not found", path)
}
if sp.Disconnected {
return nil, fmt.Errorf("drive already disconnected")
}
label := sp.Label
if label == "" {
label = sp.Path
}
w.logger.Printf("[INFO] [STORAGE] Safe disconnect requested: %s (%s)", path, label)
// 1. Stop affected stacks
stoppedStacks = w.stopAffectedStacks(path)
// 2. Sync filesystem
exec.Command("sync").Run()
// 3. Unmount
rawPath, isAttachWizard := system.HasFelhomRawMount(hostFstabPath, path)
// Unmount bind/main
cmd := exec.Command("umount", path)
if out, umountErr := cmd.CombinedOutput(); umountErr != nil {
// Try lazy unmount as fallback
w.logger.Printf("[WARN] [STORAGE] umount %s failed, trying lazy: %v", path, umountErr)
cmd = exec.Command("umount", "-l", path)
if out, umountErr = cmd.CombinedOutput(); umountErr != nil {
return stoppedStacks, fmt.Errorf("umount %s failed: %v (%s)", path, umountErr, strings.TrimSpace(string(out)))
}
}
// Unmount raw if attach-wizard
if isAttachWizard && rawPath != "" {
cmd = exec.Command("umount", rawPath)
if out, umountErr := cmd.CombinedOutput(); umountErr != nil {
cmd = exec.Command("umount", "-l", rawPath)
if out, umountErr = cmd.CombinedOutput(); umountErr != nil {
w.logger.Printf("[WARN] [STORAGE] umount raw %s failed: %v (%s)", rawPath, umountErr, strings.TrimSpace(string(out)))
}
}
}
// 4. Mark disconnected
if setErr := w.settings.SetDisconnected(path, true, stoppedStacks); setErr != nil {
w.logger.Printf("[ERROR] [STORAGE] Failed to mark disconnected: %v", setErr)
}
// 5. Update in-memory state
state := w.getOrCreateState(path)
state.lastStatus = "disconnected"
state.probeInterval = disconnectedProbeInterval
state.consecutiveFailures = 0
// 6. Trigger alert refresh
if w.alertRefresh != nil {
w.alertRefresh()
}
// 7. Notify and push hub report
w.notifier.Notify("storage_safe_disconnect", "info",
fmt.Sprintf("Meghajtó biztonságosan leválasztva: %s", label), "")
if w.pushHubReport != nil {
go w.pushHubReport()
}
w.logger.Printf("[INFO] [STORAGE] Safe disconnect completed: %s — drive can be removed", path)
return stoppedStacks, nil
}
// Reconnect attempts to remount a disconnected storage path.
func (w *StorageWatchdog) Reconnect(ctx context.Context, path string) (stoppedStacks []string, err error) {
sp := w.findStoragePath(path)
if sp == nil {
return nil, fmt.Errorf("storage path %q not found", path)
}
if !sp.Disconnected {
return nil, fmt.Errorf("drive is not disconnected")
}
label := sp.Label
if label == "" {
label = sp.Path
}
// Check UUID availability
mountPath := sp.Path
rawPath, isAttachWizard := system.HasFelhomRawMount(hostFstabPath, sp.Path)
if isAttachWizard {
mountPath = rawPath
}
uuid := system.ParseFstabUUID(hostFstabPath, mountPath)
if uuid != "" {
uuidPath := filepath.Join(hostDevUUIDPath, uuid)
if _, statErr := os.Stat(uuidPath); statErr != nil {
return nil, fmt.Errorf("drive not detected (UUID %s not found) — ensure the drive is physically connected", uuid)
}
}
// Attempt remount
if mountErr := w.remount(path, rawPath, isAttachWizard); mountErr != nil {
return nil, fmt.Errorf("mount failed: %w", mountErr)
}
// Verify
verifyResult := system.ProbeStoragePath(path)
if verifyResult.Status != system.ProbeConnected {
return nil, fmt.Errorf("mount appeared to succeed but probe failed: %v", verifyResult.Err)
}
// Clean restic locks
w.cleanResticLocks(ctx, path)
// Validate stopped stacks
filteredStacks := w.filterStoppedStacks(sp.StoppedStacks)
// Clear disconnected, preserve stopped stacks for restart UI
if setErr := w.settings.SetDisconnected(path, false, filteredStacks); setErr != nil {
w.logger.Printf("[ERROR] [STORAGE] Failed to clear disconnected: %v", setErr)
}
// Update in-memory state
state := w.getOrCreateState(path)
state.lastStatus = "connected"
state.probeInterval = defaultProbeInterval
state.consecutiveFailures = 0
// Trigger alert refresh
if w.alertRefresh != nil {
w.alertRefresh()
}
// Notify
w.notifier.NotifyStorageReconnected(label)
if w.pushHubReport != nil {
go w.pushHubReport()
}
w.logger.Printf("[INFO] [STORAGE] Reconnect completed: %s", path)
return filteredStacks, nil
}
// RestartStoppedApps restarts apps that were auto-stopped due to a drive disconnect.
func (w *StorageWatchdog) RestartStoppedApps(path string) (started, failed []string) {
sp := w.findStoragePath(path)
if sp == nil || sp.Disconnected {
return nil, nil
}
stacks := w.settings.GetStoppedStacks(path)
if len(stacks) == 0 {
return nil, nil
}
for _, name := range stacks {
w.logger.Printf("[INFO] [STORAGE] Starting stack %s (drive reconnected: %s)", name, path)
if err := w.stackProvider.StartStack(name); err != nil {
w.logger.Printf("[ERROR] [STORAGE] Failed to start stack %s: %v", name, err)
failed = append(failed, name)
} else {
started = append(started, name)
}
}
// Clear stopped stacks list
if err := w.settings.ClearStoppedStacks(path); err != nil {
w.logger.Printf("[ERROR] [STORAGE] Failed to clear stopped stacks: %v", err)
}
return started, failed
}
// findStoragePath returns the storage path entry for a given path, or nil.
func (w *StorageWatchdog) findStoragePath(path string) *settings.StoragePath {
for _, sp := range w.settings.GetStoragePaths() {
if sp.Path == path {
return &sp
}
}
return nil
}
+25 -1
View File
@@ -27,6 +27,7 @@ type Notifier struct {
mu sync.Mutex
cooldowns map[string]time.Time // event_type -> last notification time
perEventCooldown map[string]time.Duration // per-event override cooldown durations
// prevHealthStatus tracks the previous health check status for change detection
prevHealthStatus string
@@ -50,6 +51,10 @@ func New(hubURL, apiKey, customerID string, sett *settings.Settings, logger *log
enabled: enabled,
settings: sett,
cooldowns: make(map[string]time.Time),
perEventCooldown: map[string]time.Duration{
"storage_disconnected": 1 * time.Hour,
"storage_reconnected": 1 * time.Hour,
},
}
}
@@ -141,11 +146,14 @@ func (n *Notifier) Notify(eventType, severity, message, details string) {
return
}
// Check cooldown
// Check cooldown — per-event override takes priority over global
cooldownDuration := time.Duration(prefs.CooldownHours) * time.Hour
if cooldownDuration == 0 {
cooldownDuration = 6 * time.Hour
}
if override, ok := n.perEventCooldown[eventType]; ok {
cooldownDuration = override
}
n.mu.Lock()
lastSent, exists := n.cooldowns[eventType]
@@ -329,3 +337,19 @@ func classifyWarning(message string) string {
func contains(s, substr string) bool {
return strings.Contains(s, substr)
}
// NotifyStorageDisconnected sends a notification about a drive disconnection.
func (n *Notifier) NotifyStorageDisconnected(label string, stoppedApps []string) {
msg := fmt.Sprintf("Meghajtó váratlanul leválasztva: %s", label)
details := ""
if len(stoppedApps) > 0 {
details = fmt.Sprintf("Leállított alkalmazások: %s", strings.Join(stoppedApps, ", "))
}
n.Notify("storage_disconnected", "critical", msg, details)
}
// NotifyStorageReconnected sends a notification about a drive reconnection.
func (n *Notifier) NotifyStorageReconnected(label string) {
n.Notify("storage_reconnected", "info",
fmt.Sprintf("Meghajtó újra csatlakoztatva: %s. Az alkalmazások manuálisan indíthatók.", label), "")
}
+8
View File
@@ -69,6 +69,14 @@ func BuildReport(
{Mount: "/", Label: "SSD", TotalGB: sysInfo.DiskTotalGB, UsedGB: sysInfo.DiskUsedGB, Percent: sysInfo.DiskPercent},
}
for _, sp := range storagePaths {
if sp.Disconnected {
r.Storage = append(r.Storage, StorageReport{
Mount: sp.Path,
Label: sp.Label,
Disconnected: true,
})
continue
}
di := system.GetDiskUsage(sp.Path)
if di == nil {
continue
+1
View File
@@ -44,6 +44,7 @@ type StorageReport struct {
TotalGB float64 `json:"total_gb"`
UsedGB float64 `json:"used_gb"`
Percent float64 `json:"percent"`
Disconnected bool `json:"disconnected,omitempty"`
}
// ContainerReport holds aggregate and per-container status.
+111
View File
@@ -73,6 +73,9 @@ type StoragePath struct {
IsDefault bool `json:"is_default,omitempty"` // new apps use this by default
Schedulable bool `json:"schedulable"` // whether new apps can be deployed here
AddedAt string `json:"added_at"` // RFC3339
Disconnected bool `json:"disconnected,omitempty"` // true when drive detected as disconnected
DisconnectedAt string `json:"disconnected_at,omitempty"` // RFC3339 timestamp of disconnect detection
StoppedStacks []string `json:"stopped_stacks,omitempty"` // stacks auto-stopped on disconnect
}
// NotificationPrefs holds customer notification preferences.
@@ -87,6 +90,8 @@ var DefaultEnabledEvents = []string{
"disk_warning",
"backup_failed",
"update_available",
"storage_disconnected",
"storage_reconnected",
}
// DBValidationCache holds cached DB dump validation results.
@@ -499,3 +504,109 @@ func InferStorageLabel(path string) string {
}
return fmt.Sprintf("Tárhely (%s)", base)
}
// SetDisconnected marks a storage path as disconnected (or connected) and records which stacks were stopped.
func (s *Settings) SetDisconnected(path string, disconnected bool, stoppedStacks []string) error {
s.mu.Lock()
defer s.mu.Unlock()
for i := range s.StoragePaths {
if s.StoragePaths[i].Path == path {
s.StoragePaths[i].Disconnected = disconnected
if disconnected {
s.StoragePaths[i].DisconnectedAt = time.Now().UTC().Format(time.RFC3339)
s.StoragePaths[i].StoppedStacks = stoppedStacks
} else {
s.StoragePaths[i].DisconnectedAt = ""
// Preserve StoppedStacks on reconnect so the UI can offer restart
if stoppedStacks != nil {
s.StoragePaths[i].StoppedStacks = stoppedStacks
}
}
return s.save()
}
}
return fmt.Errorf("storage path %q not found", path)
}
// ClearDisconnected marks a path as connected and clears all disconnect-related fields.
func (s *Settings) ClearDisconnected(path string) error {
s.mu.Lock()
defer s.mu.Unlock()
for i := range s.StoragePaths {
if s.StoragePaths[i].Path == path {
s.StoragePaths[i].Disconnected = false
s.StoragePaths[i].DisconnectedAt = ""
s.StoragePaths[i].StoppedStacks = nil
return s.save()
}
}
return fmt.Errorf("storage path %q not found", path)
}
// IsDisconnected returns whether a storage path is marked as disconnected.
func (s *Settings) IsDisconnected(path string) bool {
s.mu.RLock()
defer s.mu.RUnlock()
for _, sp := range s.StoragePaths {
if sp.Path == path {
return sp.Disconnected
}
}
return false
}
// GetDisconnectedPaths returns a copy of all storage paths that are marked disconnected.
func (s *Settings) GetDisconnectedPaths() []StoragePath {
s.mu.RLock()
defer s.mu.RUnlock()
var result []StoragePath
for _, sp := range s.StoragePaths {
if sp.Disconnected {
result = append(result, sp)
}
}
return result
}
// GetConnectedPaths returns a copy of all storage paths that are NOT disconnected.
func (s *Settings) GetConnectedPaths() []StoragePath {
s.mu.RLock()
defer s.mu.RUnlock()
var result []StoragePath
for _, sp := range s.StoragePaths {
if !sp.Disconnected {
result = append(result, sp)
}
}
return result
}
// GetStoppedStacks returns the list of stacks that were auto-stopped for a storage path.
func (s *Settings) GetStoppedStacks(path string) []string {
s.mu.RLock()
defer s.mu.RUnlock()
for _, sp := range s.StoragePaths {
if sp.Path == path {
if len(sp.StoppedStacks) == 0 {
return nil
}
result := make([]string, len(sp.StoppedStacks))
copy(result, sp.StoppedStacks)
return result
}
}
return nil
}
// ClearStoppedStacks removes the stopped stacks list for a storage path (e.g., after restart).
func (s *Settings) ClearStoppedStacks(path string) error {
s.mu.Lock()
defer s.mu.Unlock()
for i := range s.StoragePaths {
if s.StoragePaths[i].Path == path {
s.StoragePaths[i].StoppedStacks = nil
return s.save()
}
}
return fmt.Errorf("storage path %q not found", path)
}
+161 -10
View File
@@ -3,12 +3,14 @@
package system
import (
"bufio"
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
"syscall"
"time"
)
// IsMountPoint checks if a path is on a different device than its parent.
@@ -212,21 +214,22 @@ func isSameBlockDevice(pathA, pathB string) bool {
return statA.Dev == statB.Dev
}
// diskModel reads the disk model from /sys/block/<dev>/device/model.
func diskModel(device string) string {
// /dev/sda1 → sda, /dev/nvme0n1p1 → nvme0n1
base := filepath.Base(device)
// Strip partition number: sda1 → sda, nvme0n1p1 → nvme0n1
disk := base
// stripPartition strips the partition suffix from a device name.
// e.g., "sda1" → "sda", "nvme0n1p1" → "nvme0n1".
func stripPartition(base string) string {
if strings.HasPrefix(base, "nvme") {
// nvme0n1p1 → find last 'p' followed by digits
if idx := strings.LastIndex(base, "p"); idx > 4 {
disk = base[:idx]
return base[:idx]
}
} else {
// sda1 → sda: strip trailing digits
disk = strings.TrimRight(base, "0123456789")
return strings.TrimRight(base, "0123456789")
}
return base
}
// diskModel reads the disk model from /sys/block/<dev>/device/model.
func diskModel(device string) string {
disk := stripPartition(filepath.Base(device))
modelPath := "/sys/block/" + disk + "/device/model"
data, err := os.ReadFile(modelPath)
if err != nil {
@@ -234,3 +237,151 @@ func diskModel(device string) string {
}
return strings.TrimSpace(string(data))
}
// ProbeStatus represents the result of a storage path probe.
type ProbeStatus int
const (
ProbeConnected ProbeStatus = iota
ProbeDisconnected
ProbeTimeout
)
// ProbeResult holds the outcome of a storage path probe.
type ProbeResult struct {
Status ProbeStatus
Err error
}
// ProbeStoragePath checks if a storage path is responsive.
// Uses a goroutine with a 3-second timeout to avoid blocking on dead mounts.
func ProbeStoragePath(path string) ProbeResult {
// Quick check: does the path exist at all?
if _, err := os.Lstat(path); os.IsNotExist(err) {
return ProbeResult{Status: ProbeDisconnected, Err: err}
}
type statResult struct {
err error
}
ch := make(chan statResult, 1)
go func() {
var stat syscall.Statfs_t
err := syscall.Statfs(path, &stat)
ch <- statResult{err: err}
}()
select {
case res := <-ch:
if res.err == nil {
return ProbeResult{Status: ProbeConnected}
}
errStr := res.err.Error()
if strings.Contains(errStr, "transport endpoint") ||
strings.Contains(errStr, "input/output error") ||
strings.Contains(errStr, "no such device") {
return ProbeResult{Status: ProbeDisconnected, Err: res.err}
}
return ProbeResult{Status: ProbeDisconnected, Err: res.err}
case <-time.After(3 * time.Second):
return ProbeResult{Status: ProbeTimeout, Err: fmt.Errorf("stat timed out after 3s")}
}
}
// IsUSBDevice checks if a block device is connected via USB.
// devicePath should be like "/dev/sdb" or "/dev/sdb1".
// Checks the sysfs symlink for the disk — if the path contains "/usb", it's a USB device.
func IsUSBDevice(devicePath string) bool {
disk := stripPartition(filepath.Base(devicePath))
if disk == "" {
return false
}
// Try /host/sys first (Docker mount), then /sys (native)
for _, prefix := range []string{"/host/sys", "/sys"} {
link, err := os.Readlink(prefix + "/block/" + disk)
if err != nil {
continue
}
if strings.Contains(link, "/usb") {
return true
}
return false // found the sysfs entry, but not USB
}
return false
}
// ParseFstabUUID extracts the UUID for a given mount point from an fstab file.
// Returns empty string if the mount point is not found or has no UUID.
func ParseFstabUUID(fstabPath, mountPath string) string {
f, err := os.Open(fstabPath)
if err != nil {
return ""
}
defer f.Close()
cleanMount := filepath.Clean(mountPath)
scanner := bufio.NewScanner(f)
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
if line == "" || strings.HasPrefix(line, "#") {
continue
}
fields := strings.Fields(line)
if len(fields) < 2 {
continue
}
if filepath.Clean(fields[1]) != cleanMount {
continue
}
source := fields[0]
if strings.HasPrefix(source, "UUID=") {
return strings.TrimPrefix(source, "UUID=")
}
}
return ""
}
// HasFelhomRawMount checks if a mount path was set up via the attach wizard
// (which uses a raw mount at /mnt/.felhom-raw/<x> + a bind mount).
// Returns the raw mount path if found, e.g., "/mnt/.felhom-raw/hdd_1".
func HasFelhomRawMount(fstabPath, mountPath string) (rawPath string, ok bool) {
f, err := os.Open(fstabPath)
if err != nil {
return "", false
}
defer f.Close()
cleanMount := filepath.Clean(mountPath)
scanner := bufio.NewScanner(f)
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
if line == "" || strings.HasPrefix(line, "#") {
continue
}
fields := strings.Fields(line)
if len(fields) < 4 {
continue
}
// Look for a bind mount line targeting our mount path
// Format: /mnt/.felhom-raw/hdd_1/subfolder /mnt/hdd_1 none bind,...
target := filepath.Clean(fields[1])
if target != cleanMount {
continue
}
source := fields[0]
if !strings.Contains(source, ".felhom-raw") {
continue
}
// Extract the raw mount path: /mnt/.felhom-raw/hdd_1/subfolder → /mnt/.felhom-raw/hdd_1
// The raw mount is the first two components after /mnt/.felhom-raw/
parts := strings.Split(filepath.Clean(source), string(os.PathSeparator))
// parts: ["", "mnt", ".felhom-raw", "hdd_1", "subfolder"]
for i, p := range parts {
if p == ".felhom-raw" && i+1 < len(parts) {
rawPath = string(os.PathSeparator) + filepath.Join(parts[1:i+2]...)
return rawPath, true
}
}
}
return "", false
}
@@ -80,3 +80,30 @@ func CheckBackupDestination(path string) DestinationHealth {
Severity: "ok",
}
}
// ProbeStatus represents the result of a storage path probe.
type ProbeStatus int
const (
ProbeConnected ProbeStatus = iota
ProbeDisconnected
ProbeTimeout
)
// ProbeResult holds the outcome of a storage path probe.
type ProbeResult struct {
Status ProbeStatus
Err error
}
// ProbeStoragePath always returns connected on non-Linux.
func ProbeStoragePath(_ string) ProbeResult { return ProbeResult{Status: ProbeConnected} }
// IsUSBDevice always returns false on non-Linux.
func IsUSBDevice(_ string) bool { return false }
// ParseFstabUUID always returns empty on non-Linux.
func ParseFstabUUID(_, _ string) string { return "" }
// HasFelhomRawMount always returns false on non-Linux.
func HasFelhomRawMount(_, _ string) (string, bool) { return "", false }
+22 -2
View File
@@ -9,6 +9,7 @@ import (
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
"gitea.dooplex.hu/admin/felhom-controller/internal/monitor"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// Alert represents a persistent dashboard alert banner.
@@ -39,10 +40,29 @@ func NewAlertManager(logger *log.Logger) *AlertManager {
}
// Refresh regenerates alerts from the latest health check report and config state.
// Called after each health check cycle (every 5 minutes).
func (am *AlertManager) Refresh(report *monitor.HealthReport, cfg *config.Config, backupMgr *backup.Manager, updateAvailable bool, latestVersion string) {
// Called after each health check cycle (every 5 minutes) and on storage state changes.
func (am *AlertManager) Refresh(report *monitor.HealthReport, cfg *config.Config, backupMgr *backup.Manager, updateAvailable bool, latestVersion string, storagePaths ...[]settings.StoragePath) {
var alerts []Alert
// Disconnected storage alerts (top-level error banners on all pages)
if len(storagePaths) > 0 {
for _, sp := range storagePaths[0] {
if sp.Disconnected {
label := sp.Label
if label == "" {
label = sp.Path
}
alerts = append(alerts, Alert{
ID: "storage-disconnected-" + simpleHash(sp.Path),
Level: "error",
Message: fmt.Sprintf("Meghajtó leválasztva: %s (%s)", label, sp.Path),
Link: "/settings",
LinkText: "Beállítások",
})
}
}
}
// From health check issues (critical)
for _, issue := range report.Issues {
alerts = append(alerts, Alert{
+50 -3
View File
@@ -27,12 +27,21 @@ type StorageBarInfo struct {
TotalGB float64
UsedGB float64
Percent float64
Disconnected bool
}
// buildStorageBars returns usage bars for all registered storage paths.
func (s *Server) buildStorageBars() []StorageBarInfo {
var bars []StorageBarInfo
for _, sp := range s.settings.GetStoragePaths() {
if sp.Disconnected {
bars = append(bars, StorageBarInfo{
Label: sp.Label,
Path: sp.Path,
Disconnected: true,
})
continue
}
di := system.GetDiskUsage(sp.Path)
if di == nil {
continue
@@ -70,6 +79,8 @@ type StoragePathView struct {
IsMounted bool
AppDetails []StorageAppDetail
FSInfo *system.FSInfo
IsUSB bool // true if this is a USB-attached device (safe disconnect available)
StoppedApps []string // stacks auto-stopped due to disconnect (for restart UI)
}
func (s *Server) baseData(page, title string) map[string]interface{} {
@@ -659,6 +670,9 @@ type AppBackupRow struct {
Tier2SizeHuman string
Tier2Browsable bool // true for rsync (plain files), false for restic
// Drive disconnected — app's home drive is currently disconnected
DriveDisconnected bool
// Warnings accumulated for this app
Warnings []string
}
@@ -700,10 +714,32 @@ func (s *Server) buildAppBackupRows(
}
}
// Build disconnected paths set for drive-disconnected detection
disconnectedPaths := make(map[string]bool)
for _, dp := range s.settings.GetDisconnectedPaths() {
disconnectedPaths[dp.Path] = true
}
var rows []AppBackupRow
for _, app := range status.AppDataInfo {
hasDB := dbStacks[app.StackName] || app.HasDBDump
// Check if this app's home drive is disconnected
driveDisconnected := false
if app.HasHDDData && len(app.HDDPaths) > 0 {
for dp := range disconnectedPaths {
for _, hp := range app.HDDPaths {
if strings.HasPrefix(hp.HostPath, dp+"/") || hp.HostPath == dp {
driveDisconnected = true
break
}
}
if driveDisconnected {
break
}
}
}
// Build backup contents label
var parts []string
if hasDB {
@@ -720,6 +756,7 @@ func (s *Server) buildAppBackupRows(
DisplayName: app.DisplayName,
HasHDDData: app.HasHDDData,
HasDB: hasDB,
DriveDisconnected: driveDisconnected,
StorageLabel: app.StorageLabel,
HDDSizeHuman: app.HDDSizeHuman,
BackupContents: contents,
@@ -933,14 +970,24 @@ func (s *Server) settingsData() map[string]interface{} {
for _, sp := range storagePaths {
view := StoragePathView{
StoragePath: sp,
IsMounted: system.IsMountPoint(sp.Path),
AppDetails: s.appDetailsForPath(sp.Path),
FSInfo: system.GetFSInfo(sp.Path),
StoppedApps: sp.StoppedStacks,
}
if sp.Disconnected {
// Skip I/O calls on disconnected drives — they'd hang or fail
view.IsMounted = false
} else {
view.IsMounted = system.IsMountPoint(sp.Path)
view.AppDetails = s.appDetailsForPath(sp.Path)
view.FSInfo = system.GetFSInfo(sp.Path)
view.AppCount = len(view.AppDetails)
if di := system.GetDiskUsage(sp.Path); di != nil {
view.DiskInfo = di
}
// Detect USB for safe disconnect button
if view.FSInfo != nil && view.FSInfo.Device != "" {
view.IsUSB = system.IsUSBDevice(view.FSInfo.Device)
}
}
storageViews = append(storageViews, view)
}
data["StoragePaths"] = storageViews
+9
View File
@@ -12,6 +12,7 @@ import (
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
"gitea.dooplex.hu/admin/felhom-controller/internal/monitor"
"gitea.dooplex.hu/admin/felhom-controller/internal/notify"
"gitea.dooplex.hu/admin/felhom-controller/internal/scheduler"
"gitea.dooplex.hu/admin/felhom-controller/internal/selfupdate"
@@ -49,6 +50,9 @@ type Server struct {
// DR restore mode state
restoreMu sync.RWMutex
restorePlan *backup.RestorePlan
// Storage watchdog (set after construction to break init ordering)
storageWatchdog *monitor.StorageWatchdog
}
func NewServer(cfg *config.Config, stackMgr *stacks.Manager, cpuCollector *system.CPUCollector, backupMgr *backup.Manager, crossDrive *backup.CrossDriveRunner, sched *scheduler.Scheduler, sett *settings.Settings, alertMgr *AlertManager, notif *notify.Notifier, updater *selfupdate.Updater, logger *log.Logger, version string) *Server {
@@ -99,6 +103,11 @@ func (s *Server) SetRestoreState(plan *backup.RestorePlan) {
s.restorePlan = plan
}
// SetStorageWatchdog sets the storage watchdog for disconnect/reconnect operations.
func (s *Server) SetStorageWatchdog(w *monitor.StorageWatchdog) {
s.storageWatchdog = w
}
// InRestoreMode returns true if the server is in DR restore mode.
func (s *Server) InRestoreMode() bool {
s.restoreMu.RLock()
+160
View File
@@ -154,6 +154,14 @@ func (s *Server) storageAPIHandler(w http.ResponseWriter, r *http.Request) {
s.storageAttachStatusAPIHandler(w, r)
case path == "/api/storage/attach/cancel" && r.Method == http.MethodPost:
s.storageAttachCancelHandler(w, r)
case path == "/api/storage/disconnect" && r.Method == http.MethodPost:
s.storageDisconnectHandler(w, r)
case path == "/api/storage/reconnect" && r.Method == http.MethodPost:
s.storageReconnectHandler(w, r)
case path == "/api/storage/restart-apps" && r.Method == http.MethodPost:
s.storageRestartAppsHandler(w, r)
case path == "/api/storage/status" && r.Method == http.MethodGet:
s.storageStatusHandler(w, r)
default:
http.NotFound(w, r)
}
@@ -1091,3 +1099,155 @@ func (s *Server) storageAttachCancelHandler(w http.ResponseWriter, r *http.Reque
jsonResponse(w, map[string]interface{}{"ok": true})
}
// storageDisconnectHandler handles POST /api/storage/disconnect.
// Performs a safe disconnect: stops affected apps, syncs, unmounts.
func (s *Server) storageDisconnectHandler(w http.ResponseWriter, r *http.Request) {
var req struct {
Path string `json:"path"`
}
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
jsonError(w, "Érvénytelen kérés", http.StatusBadRequest)
return
}
if req.Path == "" {
jsonError(w, "Hiányzó útvonal", http.StatusBadRequest)
return
}
if s.storageWatchdog == nil {
jsonError(w, "Szolgáltatás nem elérhető", http.StatusServiceUnavailable)
return
}
// Check if USB device (only USB drives can be safely disconnected)
fsInfo := system.GetFSInfo(req.Path)
if fsInfo != nil && fsInfo.Device != "" && !system.IsUSBDevice(fsInfo.Device) {
jsonError(w, "Csak USB meghajtó választható le biztonságosan", http.StatusBadRequest)
return
}
stoppedStacks, err := s.storageWatchdog.SafeDisconnect(r.Context(), req.Path)
if err != nil {
s.logger.Printf("[ERROR] Safe disconnect %s: %v", req.Path, err)
jsonError(w, fmt.Sprintf("Leválasztás sikertelen: %v", err), http.StatusInternalServerError)
return
}
jsonResponse(w, map[string]interface{}{
"ok": true,
"message": "A meghajtó biztonságosan eltávolítható.",
"stopped_stacks": stoppedStacks,
})
}
// storageReconnectHandler handles POST /api/storage/reconnect.
// Attempts to remount a disconnected drive.
func (s *Server) storageReconnectHandler(w http.ResponseWriter, r *http.Request) {
var req struct {
Path string `json:"path"`
}
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
jsonError(w, "Érvénytelen kérés", http.StatusBadRequest)
return
}
if req.Path == "" {
jsonError(w, "Hiányzó útvonal", http.StatusBadRequest)
return
}
if s.storageWatchdog == nil {
jsonError(w, "Szolgáltatás nem elérhető", http.StatusServiceUnavailable)
return
}
stoppedStacks, err := s.storageWatchdog.Reconnect(r.Context(), req.Path)
if err != nil {
s.logger.Printf("[ERROR] Reconnect %s: %v", req.Path, err)
jsonError(w, fmt.Sprintf("Csatlakoztatás sikertelen: %v", err), http.StatusInternalServerError)
return
}
jsonResponse(w, map[string]interface{}{
"ok": true,
"message": "Meghajtó sikeresen csatlakoztatva.",
"stopped_stacks": stoppedStacks,
})
}
// storageRestartAppsHandler handles POST /api/storage/restart-apps.
// Restarts apps that were auto-stopped due to a drive disconnect.
func (s *Server) storageRestartAppsHandler(w http.ResponseWriter, r *http.Request) {
var req struct {
Path string `json:"path"`
}
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
jsonError(w, "Érvénytelen kérés", http.StatusBadRequest)
return
}
if req.Path == "" {
jsonError(w, "Hiányzó útvonal", http.StatusBadRequest)
return
}
if s.storageWatchdog == nil {
jsonError(w, "Szolgáltatás nem elérhető", http.StatusServiceUnavailable)
return
}
// Validate drive is connected
if s.settings.IsDisconnected(req.Path) {
jsonError(w, "A meghajtó jelenleg leválasztva — először csatlakoztassa", http.StatusBadRequest)
return
}
started, failed := s.storageWatchdog.RestartStoppedApps(req.Path)
jsonResponse(w, map[string]interface{}{
"ok": true,
"started": started,
"failed": failed,
})
}
// storageStatusHandler handles GET /api/storage/status.
// Returns status of all storage paths including connection state and USB detection.
func (s *Server) storageStatusHandler(w http.ResponseWriter, r *http.Request) {
paths := s.settings.GetStoragePaths()
type pathStatus struct {
Path string `json:"path"`
Label string `json:"label"`
Connected bool `json:"connected"`
IsUSB bool `json:"is_usb"`
DisconnectedAt string `json:"disconnected_at"`
StoppedStacks []string `json:"stopped_stacks"`
}
result := make([]pathStatus, 0, len(paths))
for _, sp := range paths {
ps := pathStatus{
Path: sp.Path,
Label: sp.Label,
Connected: !sp.Disconnected,
DisconnectedAt: sp.DisconnectedAt,
StoppedStacks: sp.StoppedStacks,
}
if ps.StoppedStacks == nil {
ps.StoppedStacks = []string{}
}
// Detect USB for connected drives
if !sp.Disconnected {
if fsInfo := system.GetFSInfo(sp.Path); fsInfo != nil && fsInfo.Device != "" {
ps.IsUSB = system.IsUSBDevice(fsInfo.Device)
}
}
result = append(result, ps)
}
jsonResponse(w, map[string]interface{}{
"ok": true,
"data": result,
})
}
+13 -1
View File
@@ -38,6 +38,15 @@
</div>
</div>
{{range $.StorageBars}}
{{if .Disconnected}}
<div class="storage-item storage-disconnected">
<div class="storage-header">
<span class="storage-label">{{.Label}}</span>
<span class="storage-value badge-error" style="font-size:.75rem">Leválasztva</span>
</div>
<div class="system-bar"><div class="system-bar-disconnected"></div></div>
</div>
{{else}}
<div class="storage-item">
<div class="storage-header">
<span class="storage-label">{{.Label}}</span>
@@ -49,6 +58,7 @@
</div>
{{end}}
{{end}}
{{end}}
</div>
<div class="storage-stats">
{{if .Backup.RepoStats}}
@@ -253,7 +263,9 @@
<span class="status-dot status-{{.Status}}" title="{{.StatusText}}"></span>
<span class="app-backup-row-name">{{.DisplayName}}</span>
<div class="app-backup-row-meta">
{{if .HasHDDData}}
{{if .DriveDisconnected}}
<span class="badge badge-error" style="font-size:.7rem">Meghajtó leválasztva</span>
{{else if .HasHDDData}}
{{if .StorageLabel}}<span class="meta-badge meta-badge-storage">{{.StorageLabel}}</span>{{end}}
<span class="mono app-backup-size" style="font-size:.8rem">{{.HDDSizeHuman}}</span>
{{else}}
@@ -66,6 +66,15 @@
</div>
</div>
{{range .StorageBars}}
{{if .Disconnected}}
<div class="system-info-item storage-disconnected">
<div class="system-info-header">
<span class="system-info-label">{{.Label}}</span>
<span class="system-info-value badge-error" style="font-size:.75rem">Leválasztva</span>
</div>
<div class="system-bar"><div class="system-bar-disconnected"></div></div>
</div>
{{else}}
<div class="system-info-item">
<div class="system-info-header">
<span class="system-info-label">{{.Label}}</span>
@@ -76,6 +85,7 @@
</div>
</div>
{{end}}
{{end}}
</div>
{{if .DiskWarnings}}
<div class="inline-warnings">
@@ -51,6 +51,15 @@
</div>
</div>
{{range $.StorageBars}}
{{if .Disconnected}}
<div class="storage-item storage-disconnected">
<div class="storage-header">
<span class="storage-label">{{.Label}}</span>
<span class="storage-value badge-error" style="font-size:.75rem">Leválasztva</span>
</div>
<div class="system-bar"><div class="system-bar-disconnected"></div></div>
</div>
{{else}}
<div class="storage-item">
<div class="storage-header">
<span class="storage-label">{{.Label}}</span>
@@ -62,6 +71,7 @@
</div>
{{end}}
{{end}}
{{end}}
</div>
{{if .DiskWarnings}}
<div class="inline-warnings">
@@ -205,21 +205,38 @@ function pollUntilBack() {
{{if .StoragePaths}}
<div class="storage-paths-list">
{{range .StoragePaths}}
<div class="storage-path-item">
<div class="storage-path-item{{if .Disconnected}} storage-disconnected{{end}}">
<div class="storage-path-header">
<div class="storage-path-info">
<div class="storage-path-label-wrap" id="label-wrap-{{.Path}}">
<span class="storage-path-label" id="label-display-{{.Path}}">{{.Label}}</span>
<button class="btn btn-xs btn-ghost" onclick="editStorageLabel('{{.Path}}', '{{.Label}}')" title="Átnevezés">✏️</button>
{{if not .Disconnected}}<button class="btn btn-xs btn-ghost" onclick="editStorageLabel('{{.Path}}', '{{.Label}}')" title="Átnevezés">✏️</button>{{end}}
</div>
<span class="storage-path-path mono">{{.Path}}</span>
</div>
<div class="storage-path-badges">
{{if .Disconnected}}
<span class="badge badge-error">Leválasztva</span>
{{else}}
{{if .IsDefault}}<span class="badge state-green">Alapértelmezett</span>{{end}}
{{if .Schedulable}}<span class="badge" style="background:rgba(0,136,204,0.15);color:var(--accent-light)">Aktív</span>{{else}}<span class="badge state-gray">Inaktív</span>{{end}}
{{if not .IsMounted}}<span class="badge badge-warn">Rendszermeghajtón</span>{{end}}
{{end}}
</div>
</div>
{{if .Disconnected}}
<div class="storage-path-details">
<div class="storage-disconnected-info">
{{if .DisconnectedAt}}<span class="form-hint">Leválasztva: {{.DisconnectedAt}}</span>{{end}}
{{if .StoppedApps}}
<span class="form-hint">Leállított alkalmazások: {{range $i, $name := .StoppedApps}}{{if $i}}, {{end}}{{$name}}{{end}}</span>
{{end}}
</div>
</div>
<div class="storage-path-actions" id="storage-actions-{{.Path}}">
<button class="btn btn-xs btn-primary" onclick="storageReconnect('{{.Path}}')">Csatlakoztatás</button>
</div>
{{else}}
<div class="storage-path-details">
{{if .DiskInfo}}
<div class="storage-path-disk">
@@ -237,6 +254,12 @@ function pollUntilBack() {
{{.FSInfo.FSType}} · {{.FSInfo.Device}}{{if .FSInfo.Model}} · {{.FSInfo.Model}}{{end}}
</div>
{{end}}
{{if .StoppedApps}}
<div class="storage-stopped-apps-info" id="storage-stopped-{{.Path}}">
<span class="form-hint" style="color:var(--accent-light)">Újraindításra váró alkalmazások: {{range $i, $name := .StoppedApps}}{{if $i}}, {{end}}{{$name}}{{end}}</span>
<button class="btn btn-xs btn-primary" onclick="storageRestartApps('{{.Path}}')" style="margin-left:.5rem">Alkalmazások indítása</button>
</div>
{{end}}
<div class="storage-path-meta">
{{if .AppDetails}}
<details class="storage-app-details">
@@ -278,6 +301,9 @@ function pollUntilBack() {
<button type="submit" class="btn btn-xs btn-outline">Engedélyezés</button>
</form>
{{end}}
{{if .IsUSB}}
<button class="btn btn-xs btn-danger-outline" onclick="storageDisconnect('{{.Path}}', '{{.Label}}', {{.AppCount}})">Leválasztás</button>
{{end}}
{{if and (not .IsDefault) (eq .AppCount 0)}}
<form method="POST" action="/settings/storage/remove" style="display:inline"
onsubmit="return confirm('Biztosan eltávolítja a(z) {{.Path}} adattárolót?')">
@@ -286,6 +312,7 @@ function pollUntilBack() {
</form>
{{end}}
</div>
{{end}}
</div>
{{end}}
</div>
@@ -420,6 +447,60 @@ function editStorageLabel(path, currentLabel) {
'</form>';
wrap.querySelector('input[name=storage_label]').focus();
}
function storageDisconnect(path, label, appCount) {
var msg = 'Biztos leválasztja a meghajtót: ' + label + '?';
if (appCount > 0) msg += '\n\nA rajta futó ' + appCount + ' alkalmazás le fog állni.';
msg += '\n\nA meghajtó ezután biztonságosan eltávolítható.';
if (!confirm(msg)) return;
fetch('/api/storage/disconnect', {
method: 'POST',
headers: {'Content-Type': 'application/json'},
body: JSON.stringify({path: path})
}).then(function(r) { return r.json(); }).then(function(data) {
if (data.ok) {
alert('A meghajtó biztonságosan eltávolítható.');
location.reload();
} else {
alert('Hiba: ' + (data.error || 'ismeretlen'));
}
}).catch(function(e) { alert('Hiba: ' + e); });
}
function storageReconnect(path) {
var actionsDiv = document.getElementById('storage-actions-' + path);
if (actionsDiv) actionsDiv.innerHTML = '<span class="form-hint">Csatlakoztatás...</span>';
fetch('/api/storage/reconnect', {
method: 'POST',
headers: {'Content-Type': 'application/json'},
body: JSON.stringify({path: path})
}).then(function(r) { return r.json(); }).then(function(data) {
if (data.ok) {
location.reload();
} else {
alert('Hiba: ' + (data.error || 'ismeretlen'));
if (actionsDiv) actionsDiv.innerHTML = '<button class="btn btn-xs btn-primary" onclick="storageReconnect(\'' + path + '\')">Csatlakoztatás</button>';
}
}).catch(function(e) {
alert('Hiba: ' + e);
if (actionsDiv) actionsDiv.innerHTML = '<button class="btn btn-xs btn-primary" onclick="storageReconnect(\'' + path + '\')">Csatlakoztatás</button>';
});
}
function storageRestartApps(path) {
fetch('/api/storage/restart-apps', {
method: 'POST',
headers: {'Content-Type': 'application/json'},
body: JSON.stringify({path: path})
}).then(function(r) { return r.json(); }).then(function(data) {
if (data.ok) {
var msg = '';
if (data.started && data.started.length) msg += 'Elindítva: ' + data.started.join(', ');
if (data.failed && data.failed.length) msg += (msg ? '\n' : '') + 'Sikertelen: ' + data.failed.join(', ');
if (msg) alert(msg);
location.reload();
} else {
alert('Hiba: ' + (data.error || 'ismeretlen'));
}
}).catch(function(e) { alert('Hiba: ' + e); });
}
function cancelEditLabel(path, label) {
var wrap = document.getElementById('label-wrap-' + path);
if (!wrap) return;
@@ -2254,6 +2254,41 @@ a.stat-card:hover {
background: rgba(250, 204, 21, 0.15);
color: #facc15;
}
.badge-error {
background: rgba(239, 68, 68, 0.15);
color: #ef4444;
}
/* Disconnected storage path card */
.storage-disconnected {
opacity: 0.75;
border-style: dashed;
}
.storage-disconnected .storage-disconnected-info {
display: flex;
flex-direction: column;
gap: .25rem;
}
.storage-stopped-apps-info {
display: flex;
align-items: center;
flex-wrap: wrap;
gap: .25rem;
margin-bottom: .5rem;
}
/* Disconnected bar on dashboard/monitoring */
.system-bar-disconnected {
background: repeating-linear-gradient(
-45deg,
rgba(239, 68, 68, 0.15),
rgba(239, 68, 68, 0.15) 5px,
transparent 5px,
transparent 10px
);
height: 100%;
border-radius: 4px;
}
/* Task progress bar (storage init — not disk usage zone gradient) */
.progress-bar-task {