diff --git a/CHANGELOG.md b/CHANGELOG.md index de862c0..60d41e7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,23 @@ ## Changelog +### v0.33.0 — Docker volume backup + Tier 2 restore + restore dropdown fixes (2026-02-27) + +#### Added +- **backup/backup.go**: `DumpAppVolumes()` exports Docker named volumes to tar files using `docker run alpine tar`; `runVolumeDumpsInternal()` runs volume dumps for all stacks in nightly schedule (Phase 1b between DB dumps and restic); volume dump dirs included in per-drive restic snapshots +- **backup/appdata.go**: `ResolveDockerVolumeNames()` resolves full Docker volume names with project prefix (e.g., `mealie_mealie_data` instead of `mealie_data`); `GetDockerVolumes()` added to `StackDataProvider` interface; `HasVolumeData` field on `AppBackupInfo`, `HasVolumes` on `StackSummary` +- **backup/paths.go**: `AppVolumeDumpPath()` returns `/felhom-data/backups/primary//volume-dumps/` +- **backup/restore.go**: `RestoreAppFromTier2()` restores from cross-drive rsync mirror (config, HDD data, DB dumps, Docker volumes via rsync); `restoreDockerVolumes()` populates Docker volumes from tar files after Tier 1 restore; `restoreDockerVolumesFromDir()` for Tier 2 volume restore +- **backup/crossdrive.go**: `VolumeDumper` interface + `SetVolumeDumper()` for pre-backup volume dumps; `copyStackVolumeDumps()` copies volume tars to `_volumes/` in rsync mirror +- **backup/backup.go**: `ListSnapshotsForApp()` returns snapshots only from the app's home drive primary repo +- **backup/restic.go**: `Source` field on `SnapshotInfo` ("restic" or "rsync") +- **api/router.go**: `backupSnapshots()` now accepts `?stack=` param to filter by app's home drive; appends synthetic Tier 2 entry from cross-drive config when backup succeeded +- **web/handlers.go**: `backupRestoreHandler()` routes `tier2-rsync` snapshot ID to `RestoreAppFromTier2()` +- **web/templates/backups.html**: Import from `.fab` bundle link in restore section; `data-has-volumes` attribute on restore app options; volume-aware restore type banners; "Konfig + Adatok" label for volume-backed apps + +#### Fixed +- **Volume name resolution bug**: `ParseComposeNamedVolumes()` returned short names but Docker Compose V2 uses `_` — fixed in both backup and export adapters via `ResolveDockerVolumeNames()` +- **Double Tier 1 in restore dropdown**: snapshots from non-home drives appeared because stacks dir is in every drive's primary repo — now filtered by app's home drive via `ListSnapshotsForApp()` + ### v0.32.8 — Move optional config to deploy/settings page (2026-02-27) #### Changed diff --git a/controller/README.md b/controller/README.md index 1aee87f..24d965d 100644 --- a/controller/README.md +++ b/controller/README.md @@ -4,7 +4,7 @@ A single, lightweight Go container that replaces Portainer + scattered systemd scripts with a unified, Hungarian-language web dashboard for managing Docker Compose stacks, backups, storage, monitoring, and notifications on customer hardware. -**Current version: v0.32.6** +**Current version: v0.33.0** --- @@ -282,7 +282,9 @@ self-sufficient backup** — any single tier can fully restore an app. |----------|----------------|---------| | HDD + DB | Config + DB + User data | Immich, Paperless-ngx | | HDD, no DB | Config + User data | — | -| DB, no HDD | Config + DB | Mealie, Vikunja | +| Docker volumes + DB | Config + DB + Volume data | Tandoor | +| Docker volumes, no DB | Config + Volume data | Mealie (SQLite) | +| DB, no HDD/volumes | Config + DB | Vikunja | | Config only | Config | Gokapi, Homepage | #### Tier 1: Nightly Backup (mandatory, same drive) @@ -296,8 +298,10 @@ The nightly backup has two phases that run sequentially. All paths are **per-dri │ ├── appdata// ← app user data │ └── backups/ │ ├── primary/ -│ │ ├── restic/ ← one restic repo per drive (all apps on this drive) -│ │ └── /db-dumps/ ← per-app DB dump files +│ │ ├── restic/ ← one restic repo per drive (all apps on this drive) +│ │ └── / +│ │ ├── db-dumps/ ← per-app DB dump files +│ │ └── volume-dumps/ ← per-app Docker volume tars (v0.33.0) │ └── secondary/ │ ├── restic/ ← secondary restic repo (cross-drive) │ ├── _infra/ ← infra config mirror @@ -313,6 +317,7 @@ The nightly backup has two phases that run sequentially. All paths are **per-dri Path computation is centralized in `backup/paths.go` via the `FelhomDataDir = "felhom-data"` constant: - `PrimaryResticRepoPath(drivePath)` → `/felhom-data/backups/primary/restic/` - `AppDBDumpPath(drivePath, stackName)` → `/felhom-data/backups/primary//db-dumps/` +- `AppVolumeDumpPath(drivePath, stackName)` → `/felhom-data/backups/primary//volume-dumps/` - `AppDataDir(drivePath, stackName)` → `/felhom-data/appdata//` - `SecondaryResticRepoPath(drivePath)` → `/felhom-data/backups/secondary/restic/` - `AppSecondaryRsyncPath(drivePath, stackName)` → `/felhom-data/backups/secondary//rsync/` @@ -328,6 +333,15 @@ Path computation is centralized in `backup/paths.go` via the `FelhomDataDir = "f - **Validation** after each dump: checks file size, header presence, counts `CREATE TABLE` - Results cached in `settings.json` surviving container restarts +**Phase 1b — Docker Volume Dumps** (`internal/backup/backup.go`, runs after DB dumps) + +- Iterates all deployed stacks that have Docker named volumes (`GetDockerVolumes()`) +- For each volume: `docker run --rm -v :/vol:ro -v :/out alpine tar cf /out/.tar -C /vol .` +- 10-minute timeout per volume; warnings on failure (non-fatal) +- Stale tars cleaned up (volumes that no longer exist) +- Volume names resolved with project prefix via `ResolveDockerVolumeNames()` (e.g., `mealie_mealie_data`) +- Dumps written to `AppVolumeDumpPath(appDrive, stackName)` + **Phase 2 — Restic Snapshot** (`internal/backup/restic.go`, scheduled 03:00) - Apps are **grouped by drive** via `groupStacksByDrive()` — each drive's apps are backed up to that drive's restic repo @@ -335,6 +349,7 @@ Path computation is centralized in `backup/paths.go` via the `FelhomDataDir = "f - Auto-generated repository password (32 random bytes, base64url), shared across all repos, synced to hub - **Paths included in every per-drive snapshot:** - Per-app DB dump dirs on that drive + - Per-app Docker volume dump dirs (`volume-dumps/*.tar`) - Per-app HDD mount paths (user data) - Stacks dir (compose.yml + app.yaml + .felhom.yml for all apps) - `controller.yaml` (controller config) @@ -348,7 +363,7 @@ Does NOT protect against drive failure (backup is on the same physical drive). #### Tier 2: Cross-Drive Backup (opt-in, different device) (`internal/backup/crossdrive.go`) **Complete backup** to a different physical drive. Available for **all apps** — apps with HDD -data back up config + DB + user data; apps without HDD back up config + DB dumps only. +data back up config + DB + user data + Docker volumes; apps without HDD back up config + DB dumps + Docker volumes. - **Auto-enable for small apps (v0.14.1):** Apps without HDD mounts (config-only, DB-only) are automatically configured for daily rsync Tier 2 when ≥2 storage paths are registered. @@ -362,6 +377,7 @@ data back up config + DB + user data; apps without HDD back up config + DB dumps - **restic** — Versioned, deduplicated, encrypted (shared repo across apps, not browsable) - Per-app configuration in settings.json: destination path, method, schedule (daily/weekly/manual) - **Pre-backup DB dump:** `DumpStackDB()` runs fresh pg_dump/mariadb-dump before each cross-drive backup; non-fatal on failure (wired via `DBDumper` interface to avoid circular imports) +- **Pre-backup volume dump (v0.33.0):** `DumpAppVolumes()` exports Docker named volumes to tar before each cross-drive backup (wired via `VolumeDumper` interface) - **Empty mounts allowed:** `RunAppBackup` accepts apps with no HDD mounts — the rsync mount loop simply doesn't execute, but DB + config copy still runs - **Drive-type-aware validation** (`ValidateDestination`): @@ -380,12 +396,13 @@ data back up config + DB + user data; apps without HDD back up config + DB dumps ├── /rsync/ ← per-app rsync mirror │ ├── _db/ ← DB dump files │ ├── _config/ ← compose.yml, app.yaml, .felhom.yml + │ ├── _volumes/ ← Docker volume tars (v0.33.0) │ └── ← HDD mount contents (if app has HDD data) └── restic/ ← shared restic repo (all cross-drive apps) ``` - DB dump files read from **per-app home drive** path (`AppDBDumpPath`) - `_` prefix directories prevent collision with user data - - For non-HDD apps, only `_db/` and `_config/` are present (no user data directory) + - For non-HDD apps, only `_db/`, `_config/`, and `_volumes/` (if applicable) are present (no user data directory) - **Restic backup paths:** includes HDD mounts (if any) + config dir + per-app DB dump dir from home drive + stacks dir + controller.yaml (infra, v0.14.1) - Safety guards: destination ≠ source, path-overlap check (HDD mounts only), writable check - **Chained execution:** runs immediately after nightly restic — daily apps every night, weekly apps on Sundays @@ -402,28 +419,40 @@ Placeholder shown in UI ("3. mentés — Hamarosan"). #### Restore (`internal/backup/restore.go`) -All deployed apps appear in the restore dropdown — every app has restic snapshot data -(stacks dir + DB dumps are always backed up). +Both **Tier 1** (restic) and **Tier 2** (rsync) restores are supported. All deployed apps +appear in the restore dropdown with per-app snapshot filtering. -| App type | Config restored | DB restored | User data restored | -|----------|----------------|------------|-------------------| -| Has HDD data | Yes | Yes | Yes (always — backup is mandatory) | -| DB only, no HDD | Yes | Yes | n/a | -| No DB, no HDD | Yes | — | n/a | +| App type | Config restored | DB restored | User data restored | Docker volumes restored | +|----------|----------------|------------|-------------------|------------------------| +| Has HDD data | Yes | Yes | Yes (always) | Yes (if present) | +| Docker volumes, no HDD | Yes | Yes | n/a | Yes | +| DB only, no HDD/volumes | Yes | Yes | n/a | n/a | +| Config only | Yes | — | n/a | n/a | -- **Snapshot API** returns ALL snapshots unfiltered — older snapshots still allow config+DB restore; `RestoreApp` extracts whatever paths are available -- **Restore type info** shown per-app when selected in dropdown (Hungarian banners): - - Has HDD: "Teljes visszaállitas: adatbazis + konfiguracio + felhasznaloi adatok" - - Has DB, no HDD: "Adatbazis es konfiguracio visszaallitasa" - - No DB, no HDD: "Csak konfiguracio visszaallitasa" -- **Execution flow:** stop app → resolve app's home drive → `restic restore --target / --include ...` from per-drive repo → restart app -- Restic repo resolved via `PrimaryResticRepoPath(appDrivePath)` -- DB dumps restored from `AppDBDumpPath(appDrivePath, stackName)` +**Snapshot API** (`/api/backup/snapshots?stack=`): +- Returns snapshots **only from the app's home drive** primary repo (prevents showing irrelevant snapshots from other drives) +- Appends a synthetic Tier 2 entry (ID `tier2-rsync`) from cross-drive config when last backup was successful +- Dropdown groups by tier: "1. szint — Helyi mentes" and "2. szint — Masodlagos masolat" + +**Restore type info** shown per-app when selected in dropdown (Hungarian banners): + - Has HDD or Docker volumes: "Teljes visszaallitas: adatbazis + konfiguracio + felhasznaloi adatok" + - Has DB, no user data: "Adatbazis es konfiguracio visszaallitasa" + - Config only: "Csak konfiguracio visszaallitasa" + +**Tier 1 restore** (`RestoreApp`): +- Stop app → resolve app's home drive → `restic restore --target / --include ...` → populate Docker volumes from restored tars → restart app +- Restore paths: config dir, DB dump dir, volume dump dir, HDD mounts +- Docker volumes restored via `restoreDockerVolumes()`: `docker volume rm -f` → `docker volume create` → `docker run alpine tar xf` + +**Tier 2 restore** (`RestoreAppFromTier2`): +- Stop app → rsync config from `_config/` → rsync HDD data (single/multi-mount) → copy DB dumps from `_db/` → restore Docker volumes from `_volumes/` tars → restart app +- Uses rsync `--delete` for config and HDD data to ensure exact mirror state +- Single-mount apps: data directly in rsync dir (excluding `_*`); multi-mount: per-leaf subdirectories + +**Common:** - Running flag prevents concurrent backup/restore operations -- Snapshot ID validated (8-64 lowercase hex) - -**Note:** Restore currently uses Tier 1 (primary restic repo on app's home drive) only. -Restoring from Tier 2 (cross-drive) is a future enhancement. +- Snapshot ID validated (8-64 lowercase hex, or special `tier2-rsync`) +- Import from `.fab` bundle link shown in restore section for cross-system migration #### Backup Page UI (`internal/web/templates/backups.html`) @@ -448,6 +477,7 @@ Every app starts as yellow (1 tier only). Green requires Tier 2 configured with **Backup contents per app** (shown per tier): - Apps with DB + HDD: "DB + Konfig + Adatok" +- Apps with Docker volumes (no HDD): "Konfig + DB + Adatok" or "Konfig + Adatok" - Apps with DB only: "DB + Konfig" - Apps with HDD, no DB: "Konfig + Adatok" - Apps with neither: "Konfig" @@ -459,7 +489,7 @@ not just those with HDD data. Non-HDD apps can configure destination, method, an - Schedule overview with next run times for DB dump, restic, prune - Snapshot history table (last 20 snapshots aggregated from all per-drive repos, sorted by time) - Storage overview card (total size across repos, snapshot count, DB dump count/size, encryption key with show/copy) -- Restore section: app dropdown → snapshot dropdown → restore type info → confirmation checkbox → execute +- Restore section: app dropdown → per-app snapshot dropdown (Tier 1 + Tier 2 grouped) → restore type info → confirmation checkbox → execute → import from `.fab` bundle link --- @@ -1773,7 +1803,7 @@ See `docker-compose.yml` for the full volume configuration. ### In Progress / Planned - [ ] Update classification and auto-apply (optional/required/security markers) -- [ ] Docker volume backup (`/var/lib/docker/volumes:ro`) +- [x] Docker volume backup + Tier 2 restore (v0.33.0) - [ ] Raspberry Pi testing (pi-customer-1) - [x] CSRF protection on POST endpoints (v0.23.0) - [x] Verbose debug logging across all modules (v0.24.0) diff --git a/controller/cmd/controller/main.go b/controller/cmd/controller/main.go index 6272bae..ea73549 100644 --- a/controller/cmd/controller/main.go +++ b/controller/cmd/controller/main.go @@ -198,6 +198,7 @@ func main() { // Wire cross-drive → backup manager for pre-backup DB dumps if backupMgr != nil { crossDriveRunner.SetDBDumper(backupMgr) + crossDriveRunner.SetVolumeDumper(backupMgr) } // --- Initialize alert manager --- @@ -891,6 +892,7 @@ func (a *stackAdapter) ListDeployedStacks() []backup.StackSummary { DisplayName: s.Meta.DisplayName, ComposePath: s.ComposePath, NeedsHDD: s.Meta.Resources.NeedsHDD, + HasVolumes: len(backup.ParseComposeNamedVolumes(s.ComposePath)) > 0, }) } return result @@ -932,6 +934,14 @@ func (a *stackAdapter) GetStackHDDMounts(name string) []string { return allMounts } +func (a *stackAdapter) GetDockerVolumes(name string) []string { + s, ok := a.mgr.GetStack(name) + if !ok { + return nil + } + return backup.ResolveDockerVolumeNames(s.ComposePath) +} + func (a *stackAdapter) GetStackHDDPath(name string) string { s, ok := a.mgr.GetStack(name) if !ok { @@ -1163,12 +1173,7 @@ func (a *exportAdapter) GetDockerVolumes(name string) []string { if !ok { return nil } - vols := backup.ParseComposeNamedVolumes(s.ComposePath) - var names []string - for _, v := range vols { - names = append(names, v.Name) - } - return names + return backup.ResolveDockerVolumeNames(s.ComposePath) } func (a *exportAdapter) IsStackDeployed(name string) bool { diff --git a/controller/internal/api/router.go b/controller/internal/api/router.go index ba631f1..7728e6f 100644 --- a/controller/internal/api/router.go +++ b/controller/internal/api/router.go @@ -795,7 +795,18 @@ func (r *Router) backupSnapshots(w http.ResponseWriter, req *http.Request) { return } - snapshots, err := r.backupMgr.ListAllSnapshots(50) + stackName := req.URL.Query().Get("stack") + + var snapshots []backup.SnapshotInfo + var err error + + if stackName != "" { + // Per-app: only snapshots from the app's home drive + snapshots, err = r.backupMgr.ListSnapshotsForApp(stackName, 20) + } else { + // Fallback: all snapshots (general use) + snapshots, err = r.backupMgr.ListAllSnapshots(50) + } if err != nil { r.logger.Printf("[ERROR] [api] Failed to list backup snapshots: %v", err) writeJSON(w, http.StatusInternalServerError, apiResponse{OK: false, Error: err.Error()}) @@ -814,6 +825,32 @@ func (r *Router) backupSnapshots(w http.ResponseWriter, req *http.Request) { } } } + + // Append Tier 2 (cross-drive rsync) entry if available for this app + if stackName != "" { + cdCfg := r.sett.GetCrossDriveConfig(stackName) + if cdCfg != nil && cdCfg.Enabled && cdCfg.LastStatus == "ok" && cdCfg.LastRun != "" { + lastRun, _ := time.Parse(time.RFC3339, cdCfg.LastRun) + if !lastRun.IsZero() { + // Resolve drive label for destination + var destLabel string + for _, sp := range storagePaths { + if sp.Path == cdCfg.DestinationPath { + destLabel = sp.Label + break + } + } + tier2 := backup.SnapshotInfo{ + ID: "tier2-rsync", + Time: lastRun, + Tier: 2, + Source: "rsync", + DriveLabel: destLabel, + } + snapshots = append(snapshots, tier2) + } + } + } } if snapshots == nil { diff --git a/controller/internal/backup/appdata.go b/controller/internal/backup/appdata.go index ee527ec..aee95a7 100644 --- a/controller/internal/backup/appdata.go +++ b/controller/internal/backup/appdata.go @@ -6,6 +6,7 @@ import ( "log" "os" "os/exec" + "path/filepath" "strings" "time" @@ -17,7 +18,8 @@ type StackDataProvider interface { GetStackComposePath(name string) (composePath string, ok bool) ListDeployedStacks() []StackSummary GetStackHDDMounts(name string) []string - GetStackHDDPath(name string) string // raw HDD_PATH from app.yaml (empty if no HDD) + GetStackHDDPath(name string) string // raw HDD_PATH from app.yaml (empty if no HDD) + GetDockerVolumes(name string) []string // full Docker volume names (project-prefixed) StopStack(name string) error StartStack(name string) error } @@ -28,6 +30,7 @@ type StackSummary struct { DisplayName string ComposePath string NeedsHDD bool + HasVolumes bool } // AppBackupInfo holds backup-relevant data paths for a deployed app. @@ -42,6 +45,7 @@ type AppBackupInfo struct { BackupEnabled bool HasHDDData bool HasDBDump bool + HasVolumeData bool StorageLabel string // resolved from registered storage paths } @@ -91,6 +95,7 @@ func DiscoverAppData(provider StackDataProvider, discoveredDBs []DiscoveredDB) [ // Discover Docker named volumes from compose info.DockerVolumes = ParseComposeNamedVolumes(stack.ComposePath) + info.HasVolumeData = len(info.DockerVolumes) > 0 // Check if app has a DB container (already backed up via DB dump) for _, db := range discoveredDBs { @@ -137,6 +142,21 @@ func ParseComposeNamedVolumes(composePath string) []AppDockerVolume { return volumes } +// ResolveDockerVolumeNames returns full Docker volume names with the compose project prefix. +// Docker Compose V2 creates volumes as _ where project = directory name. +func ResolveDockerVolumeNames(composePath string) []string { + vols := ParseComposeNamedVolumes(composePath) + if len(vols) == 0 { + return nil + } + project := filepath.Base(filepath.Dir(composePath)) + names := make([]string, 0, len(vols)) + for _, v := range vols { + names = append(names, project+"_"+v.Name) + } + return names +} + // appDirSize returns the total byte count and a human-readable string for a directory. // H2/H3: Single du invocation with 30s timeout replaces two separate calls. func appDirSize(path string) (int64, string) { diff --git a/controller/internal/backup/backup.go b/controller/internal/backup/backup.go index df9cc84..9b38891 100644 --- a/controller/internal/backup/backup.go +++ b/controller/internal/backup/backup.go @@ -6,6 +6,7 @@ import ( "fmt" "log" "os" + "os/exec" "path/filepath" "sort" "strings" @@ -493,6 +494,11 @@ func (m *Manager) backupDrive(ctx context.Context, drivePath string, stacks []St if _, err := os.Stat(dumpDir); err == nil { paths = append(paths, dumpDir) } + // Docker volume dumps for this stack + volDumpDir := AppVolumeDumpPath(drivePath, stack.Name) + if _, err := os.Stat(volDumpDir); err == nil { + paths = append(paths, volDumpDir) + } } // Deduplicate paths @@ -624,7 +630,107 @@ func (m *Manager) RunIntegrityCheck(ctx context.Context) error { return nil } -// RunFullBackup runs DB dumps followed by restic backup. +// DumpAppVolumes exports Docker named volumes to tar files for the given stack. +// Tars are written to AppVolumeDumpPath(drivePath, stackName)/. +// Uses "docker run alpine tar" (same pattern as appexport). +func (m *Manager) DumpAppVolumes(stackName string) error { + if m.stackProvider == nil { + return nil + } + + volumes := m.stackProvider.GetDockerVolumes(stackName) + if len(volumes) == 0 { + return nil + } + + drivePath := m.GetAppDrivePath(stackName) + if drivePath == "" { + return fmt.Errorf("cannot determine drive path for %s", stackName) + } + + dumpDir := AppVolumeDumpPath(drivePath, stackName) + if err := os.MkdirAll(dumpDir, 0755); err != nil { + return fmt.Errorf("creating volume dump dir: %w", err) + } + + var dumpErrors []string + for _, volName := range volumes { + tarPath := filepath.Join(dumpDir, volName+".tar") + if m.isDebug() { + m.logger.Printf("[DEBUG] [backup] Dumping volume %s for %s", volName, stackName) + } + + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute) + cmd := exec.CommandContext(ctx, "docker", "run", "--rm", + "-v", volName+":/vol:ro", + "-v", dumpDir+":/out", + "alpine", "tar", "cf", "/out/"+volName+".tar", "-C", "/vol", ".") + out, err := cmd.CombinedOutput() + cancel() + + if err != nil { + m.logger.Printf("[WARN] [backup] Volume dump failed for %s/%s: %s — %v", + stackName, volName, strings.TrimSpace(string(out)), err) + os.Remove(tarPath) + dumpErrors = append(dumpErrors, volName) + continue + } + + if info, _ := os.Stat(tarPath); info != nil { + m.logger.Printf("[INFO] [backup] Volume dump: %s/%s → %s", stackName, volName, humanizeBytes(info.Size())) + } + } + + // Clean up tars for volumes that no longer exist + entries, _ := os.ReadDir(dumpDir) + activeVols := make(map[string]bool) + for _, v := range volumes { + activeVols[v+".tar"] = true + } + for _, e := range entries { + if !activeVols[e.Name()] && strings.HasSuffix(e.Name(), ".tar") { + os.Remove(filepath.Join(dumpDir, e.Name())) + if m.isDebug() { + m.logger.Printf("[DEBUG] [backup] Removed stale volume dump: %s/%s", stackName, e.Name()) + } + } + } + + if len(dumpErrors) > 0 { + return fmt.Errorf("volume dump failed for: %s", strings.Join(dumpErrors, ", ")) + } + return nil +} + +// runVolumeDumpsInternal dumps Docker named volumes for all deployed apps. +func (m *Manager) runVolumeDumpsInternal(ctx context.Context) error { + if m.stackProvider == nil { + return nil + } + + stacks := m.stackProvider.ListDeployedStacks() + var dumped, failed int + for _, stack := range stacks { + if !stack.HasVolumes { + continue + } + if ctx.Err() != nil { + return ctx.Err() + } + if err := m.DumpAppVolumes(stack.Name); err != nil { + m.logger.Printf("[WARN] [backup] Volume dump error for %s: %v", stack.Name, err) + failed++ + } else { + dumped++ + } + } + if dumped > 0 || failed > 0 { + m.logger.Printf("[INFO] [backup] Volume dumps completed: %d ok, %d failed", dumped, failed) + } + return nil +} + +// RunFullBackup runs DB dumps, volume dumps, then restic backup. func (m *Manager) RunFullBackup(ctx context.Context) error { if err := m.acquireRunning(); err != nil { return err @@ -643,13 +749,21 @@ func (m *Manager) RunFullBackup(ctx context.Context) error { // Step 1: DB dumps if m.isDebug() { - m.logger.Printf("[DEBUG] RunFullBackup: phase 1 — database dumps") + m.logger.Printf("[DEBUG] RunFullBackup: phase 1a — database dumps") } if err := m.runDBDumpsInternal(ctx); err != nil { m.logger.Printf("[WARN] [backup] DB dump had errors, continuing with backup anyway") } - // Step 2: Restic backup + // Step 2: Volume dumps + if m.isDebug() { + m.logger.Printf("[DEBUG] RunFullBackup: phase 1b — Docker volume dumps") + } + if err := m.runVolumeDumpsInternal(ctx); err != nil { + m.logger.Printf("[WARN] [backup] Volume dump had errors, continuing with backup anyway") + } + + // Step 3: Restic backup if m.isDebug() { m.logger.Printf("[DEBUG] RunFullBackup: phase 2 — restic snapshots") } @@ -764,6 +878,41 @@ func (m *Manager) ListAllSnapshots(limit int) ([]SnapshotInfo, error) { return allSnapshots, nil } +// ListSnapshotsForApp returns snapshots only from the app's home drive primary repo. +// This prevents showing irrelevant snapshots from other drives (e.g. a 544 KB SYS_DRIVE +// snapshot appearing for Immich because it contains the shared stacks directory). +func (m *Manager) ListSnapshotsForApp(stackName string, limit int) ([]SnapshotInfo, error) { + drivePath := m.GetAppDrivePath(stackName) + if drivePath == "" { + return []SnapshotInfo{}, nil + } + repoPath := PrimaryResticRepoPath(drivePath) + + if !m.restic.RepoExists(repoPath) { + return []SnapshotInfo{}, nil + } + + snapshots, err := m.restic.ListSnapshots(repoPath, limit) + if err != nil { + return nil, err + } + + for i := range snapshots { + snapshots[i].RepoPath = repoPath + snapshots[i].Tier = 1 + snapshots[i].Source = "restic" + } + + // Sort newest first + sort.Slice(snapshots, func(i, j int) bool { + return snapshots[i].Time.After(snapshots[j].Time) + }) + if limit > 0 && len(snapshots) > limit { + snapshots = snapshots[:limit] + } + return snapshots, nil +} + // SetStackProvider sets the stack data provider for app data discovery. // C3: Write is protected by mutex since stackProvider is read by concurrent goroutines. func (m *Manager) SetStackProvider(provider StackDataProvider) { diff --git a/controller/internal/backup/crossdrive.go b/controller/internal/backup/crossdrive.go index 040d294..b1241a3 100644 --- a/controller/internal/backup/crossdrive.go +++ b/controller/internal/backup/crossdrive.go @@ -22,11 +22,17 @@ type DBDumper interface { DumpStackDB(ctx context.Context, stackName string) error } +// VolumeDumper can dump Docker named volumes for a specific stack. +type VolumeDumper interface { + DumpAppVolumes(stackName string) error +} + // CrossDriveRunner handles per-app backup to secondary storage. type CrossDriveRunner struct { sett *settings.Settings stackProvider StackDataProvider dbDumper DBDumper + volDumper VolumeDumper systemDataPath string // fallback drive for SSD-only apps stacksDir string // path to stacks dir (for infra backup) controllerYAMLPath string // path to controller.yaml (for infra backup) @@ -56,6 +62,11 @@ func (r *CrossDriveRunner) SetDBDumper(d DBDumper) { r.dbDumper = d } +// SetVolumeDumper sets the volume dumper for pre-backup Docker volume dumps. +func (r *CrossDriveRunner) SetVolumeDumper(d VolumeDumper) { + r.volDumper = d +} + // GetAppDrivePath returns the drive path for an app (HDD path or system data path fallback). func (r *CrossDriveRunner) GetAppDrivePath(stackName string) string { if hddPath := r.stackProvider.GetStackHDDPath(stackName); hddPath != "" { @@ -125,7 +136,16 @@ func (r *CrossDriveRunner) RunAppBackup(ctx context.Context, stackName string) e } if err := r.dbDumper.DumpStackDB(ctx, stackName); err != nil { r.logger.Printf("[WARN] [backup] Pre-backup DB dump failed for %s: %v — proceeding with user data backup", stackName, err) - // Non-fatal: user data backup is still valuable without fresh dump + } + } + + // Trigger fresh volume dump for this app before cross-drive backup + if r.volDumper != nil { + if r.debug { + r.logger.Printf("[DEBUG] RunAppBackup: triggering pre-backup volume dump for %s", stackName) + } + if err := r.volDumper.DumpAppVolumes(stackName); err != nil { + r.logger.Printf("[WARN] [backup] Pre-backup volume dump failed for %s: %v — proceeding with backup", stackName, err) } } @@ -441,6 +461,16 @@ func (r *CrossDriveRunner) runRsyncBackup(ctx context.Context, stackName, destBa // Non-fatal: user data is the primary concern } + // --- Copy volume dumps for this stack from its home drive --- + volDestDir := filepath.Join(destDir, "_volumes") + if err := os.MkdirAll(volDestDir, 0755); err != nil { + return fmt.Errorf("creating volume dump dest dir: %w", err) + } + if err := r.copyStackVolumeDumps(stackName, volDestDir); err != nil { + r.logger.Printf("[WARN] [backup] Cross-drive volume dump copy failed for %s: %v", stackName, err) + // Non-fatal: user data is the primary concern + } + // --- Rsync app config (compose dir) --- if composePath, ok := r.stackProvider.GetStackComposePath(stackName); ok { configSrcDir := filepath.Dir(composePath) @@ -494,6 +524,37 @@ func (r *CrossDriveRunner) copyStackDBDumps(stackName, destDir string) error { return nil } +// copyStackVolumeDumps copies Docker volume dump tars for the given stack from its home drive. +func (r *CrossDriveRunner) copyStackVolumeDumps(stackName, destDir string) error { + appDrive := r.GetAppDrivePath(stackName) + dumpDir := AppVolumeDumpPath(appDrive, stackName) + + entries, err := os.ReadDir(dumpDir) + if err != nil { + if os.IsNotExist(err) { + return nil + } + return fmt.Errorf("reading volume dump dir: %w", err) + } + + copied := 0 + for _, e := range entries { + if e.IsDir() || !strings.HasSuffix(e.Name(), ".tar") { + continue + } + src := filepath.Join(dumpDir, e.Name()) + dst := filepath.Join(destDir, e.Name()) + if err := copyFile(src, dst); err != nil { + return fmt.Errorf("copying %s: %w", e.Name(), err) + } + copied++ + } + if copied > 0 { + r.logger.Printf("[INFO] [backup] Copied %d volume dump(s) for %s", copied, stackName) + } + return nil +} + // --- infra backup --- // syncInfraConfig rsyncs infrastructure config (stacks dir + controller.yaml) to all diff --git a/controller/internal/backup/paths.go b/controller/internal/backup/paths.go index 2bd1fd9..d0a0235 100644 --- a/controller/internal/backup/paths.go +++ b/controller/internal/backup/paths.go @@ -20,6 +20,11 @@ func AppDBDumpPath(drivePath, stackName string) string { return filepath.Join(drivePath, FelhomDataDir, "backups", "primary", stackName, "db-dumps") } +// AppVolumeDumpPath returns the directory for Docker volume dump tars on an app's home drive. +func AppVolumeDumpPath(drivePath, stackName string) string { + return filepath.Join(drivePath, FelhomDataDir, "backups", "primary", stackName, "volume-dumps") +} + // SecondaryBackupPath returns the root secondary backup directory for a drive. func SecondaryBackupPath(drivePath string) string { return filepath.Join(drivePath, FelhomDataDir, "backups", "secondary") diff --git a/controller/internal/backup/restic.go b/controller/internal/backup/restic.go index d1d9d3b..7ca515e 100644 --- a/controller/internal/backup/restic.go +++ b/controller/internal/backup/restic.go @@ -44,6 +44,7 @@ type SnapshotInfo struct { RepoPath string `json:"-"` // set by caller for multi-repo aggregation Tier int `json:"tier"` // 1 = primary, 2 = secondary DriveLabel string `json:"drive_label"` // filled by caller from settings + Source string `json:"source"` // "restic" or "rsync" } // RepoStats holds repository statistics. diff --git a/controller/internal/backup/restore.go b/controller/internal/backup/restore.go index d15d3ad..244a51d 100644 --- a/controller/internal/backup/restore.go +++ b/controller/internal/backup/restore.go @@ -1,9 +1,14 @@ package backup import ( + "context" "fmt" + "os" + "os/exec" "path/filepath" "regexp" + "strings" + "time" ) // snapshotIDRe validates restic snapshot IDs: 8-64 lowercase hex characters. @@ -76,6 +81,10 @@ func (m *Manager) RestoreApp(stackName, snapshotID string) error { } } + // Restore Docker volume dumps (if present in snapshot) + volDumpDir := AppVolumeDumpPath(drivePath, stackName) + restorePaths = append(restorePaths, volDumpDir) + if len(restorePaths) == 0 { return fmt.Errorf("no restorable paths found for %s", stackName) } @@ -113,21 +122,284 @@ func (m *Manager) RestoreApp(stackName, snapshotID string) error { return err } + // Populate Docker volumes from restored tars + if m.isDebug() { + m.logger.Printf("[DEBUG] RestoreApp: step 3/5 — restoring Docker volumes for %s", stackName) + } + if err := m.restoreDockerVolumes(stackName, drivePath); err != nil { + m.logger.Printf("[WARN] RESTORE volume restore failed for %s: %v (continuing)", stackName, err) + } + // Restart the app if m.isDebug() { - m.logger.Printf("[DEBUG] RestoreApp: step 3/4 — restarting app %s after successful restore", stackName) + m.logger.Printf("[DEBUG] RestoreApp: step 4/5 — restarting app %s after successful restore", stackName) } if err := m.stackProvider.StartStack(stackName); err != nil { m.logger.Printf("[WARN] RESTORE could not restart %s after restore: %v", stackName, err) } + hasVolumes := len(m.stackProvider.GetDockerVolumes(stackName)) > 0 restoreType := "config+DB" - if hasHDD { + if hasHDD || hasVolumes { restoreType = "full (config+DB+userdata)" } if m.isDebug() { - m.logger.Printf("[DEBUG] RestoreApp: step 4/4 — restore completed, type=%s", restoreType) + m.logger.Printf("[DEBUG] RestoreApp: step 5/5 — restore completed, type=%s", restoreType) } m.logger.Printf("[INFO] RESTORE completed: stack=%s, snapshot=%s, type=%s", stackName, snapshotID, restoreType) return nil } + +// RestoreAppFromTier2 restores an app from its cross-drive rsync backup mirror. +func (m *Manager) RestoreAppFromTier2(stackName string) error { + if m.stackProvider == nil { + return fmt.Errorf("stack provider not configured") + } + if m.settings == nil { + return fmt.Errorf("settings not available") + } + + cdCfg := m.settings.GetCrossDriveConfig(stackName) + if cdCfg == nil || !cdCfg.Enabled { + return fmt.Errorf("cross-drive backup not configured for %s", stackName) + } + + rsyncDir := AppSecondaryRsyncPath(cdCfg.DestinationPath, stackName) + if _, err := os.Stat(rsyncDir); os.IsNotExist(err) { + return fmt.Errorf("Tier 2 backup directory not found: %s", rsyncDir) + } + + if m.isDebug() { + m.logger.Printf("[DEBUG] RestoreAppFromTier2: stack=%s, rsyncDir=%s", stackName, rsyncDir) + } + + // Prevent concurrent operations + m.mu.Lock() + if m.running { + m.mu.Unlock() + return fmt.Errorf("backup or restore already in progress") + } + m.running = true + m.mu.Unlock() + defer func() { + m.mu.Lock() + m.running = false + m.mu.Unlock() + }() + + hddMounts := m.stackProvider.GetStackHDDMounts(stackName) + hasHDD := len(hddMounts) > 0 + drivePath := m.GetAppDrivePath(stackName) + + m.logger.Printf("[INFO] [backup] Starting Tier 2 restore for %s from %s", stackName, rsyncDir) + + // Step 1: Stop the app + if err := m.stackProvider.StopStack(stackName); err != nil { + m.logger.Printf("[WARN] RESTORE could not stop %s: %v (proceeding anyway)", stackName, err) + } + + // Step 2: Restore config from _config/ + configSrc := filepath.Join(rsyncDir, "_config") + "/" + if _, err := os.Stat(filepath.Join(rsyncDir, "_config")); err == nil { + if composePath, ok := m.stackProvider.GetStackComposePath(stackName); ok { + configDst := filepath.Dir(composePath) + "/" + if m.isDebug() { + m.logger.Printf("[DEBUG] RestoreAppFromTier2: rsync config %s → %s", configSrc, configDst) + } + cmd := exec.Command("rsync", "-a", "--delete", configSrc, configDst) + if out, err := cmd.CombinedOutput(); err != nil { + m.logger.Printf("[ERROR] [backup] Tier 2 config restore failed for %s: %v (%s)", stackName, err, strings.TrimSpace(string(out))) + // Try to restart and return error + m.stackProvider.StartStack(stackName) + return fmt.Errorf("config restore failed: %w", err) + } + } + } + + // Step 3: Restore HDD data + if hasHDD { + // Check for data directory structure — single mount vs multi-mount + if len(hddMounts) == 1 { + // Single mount: data is directly in rsyncDir (excluding _* dirs) + src := strings.TrimRight(rsyncDir, "/") + "/" + dst := strings.TrimRight(hddMounts[0], "/") + "/" + if m.isDebug() { + m.logger.Printf("[DEBUG] RestoreAppFromTier2: rsync HDD data %s → %s", src, dst) + } + cmd := exec.Command("rsync", "-a", "--delete", + "--exclude", "_*", + src, dst) + if out, err := cmd.CombinedOutput(); err != nil { + m.logger.Printf("[ERROR] [backup] Tier 2 HDD data restore failed for %s: %v (%s)", stackName, err, strings.TrimSpace(string(out))) + m.stackProvider.StartStack(stackName) + return fmt.Errorf("HDD data restore failed: %w", err) + } + } else { + // Multiple mounts: each has a subdirectory named by leaf + for _, mount := range hddMounts { + leaf := filepath.Base(mount) + src := filepath.Join(rsyncDir, leaf) + "/" + dst := strings.TrimRight(mount, "/") + "/" + if _, err := os.Stat(filepath.Join(rsyncDir, leaf)); os.IsNotExist(err) { + m.logger.Printf("[WARN] [backup] Tier 2 restore: no backup data for mount %s", mount) + continue + } + if m.isDebug() { + m.logger.Printf("[DEBUG] RestoreAppFromTier2: rsync HDD mount %s → %s", src, dst) + } + cmd := exec.Command("rsync", "-a", "--delete", src, dst) + if out, err := cmd.CombinedOutput(); err != nil { + m.logger.Printf("[ERROR] [backup] Tier 2 HDD restore failed for mount %s: %v (%s)", mount, err, strings.TrimSpace(string(out))) + m.stackProvider.StartStack(stackName) + return fmt.Errorf("HDD restore failed for %s: %w", mount, err) + } + } + } + } + + // Step 4: Restore DB dumps from _db/ + dbSrc := filepath.Join(rsyncDir, "_db") + if _, err := os.Stat(dbSrc); err == nil { + dbDst := AppDBDumpPath(drivePath, stackName) + if err := os.MkdirAll(dbDst, 0755); err == nil { + entries, _ := os.ReadDir(dbSrc) + for _, e := range entries { + if !e.IsDir() { + src := filepath.Join(dbSrc, e.Name()) + dst := filepath.Join(dbDst, e.Name()) + if data, err := os.ReadFile(src); err == nil { + os.WriteFile(dst, data, 0644) + } + } + } + if m.isDebug() { + m.logger.Printf("[DEBUG] RestoreAppFromTier2: restored DB dumps from %s", dbSrc) + } + } + } + + // Step 5: Restore Docker volumes from _volumes/ + volSrc := filepath.Join(rsyncDir, "_volumes") + if _, err := os.Stat(volSrc); err == nil { + if err := m.restoreDockerVolumesFromDir(stackName, volSrc); err != nil { + m.logger.Printf("[WARN] [backup] Tier 2 volume restore failed for %s: %v (continuing)", stackName, err) + } + } + + // Step 6: Restart the app + if err := m.stackProvider.StartStack(stackName); err != nil { + m.logger.Printf("[WARN] RESTORE could not restart %s after Tier 2 restore: %v", stackName, err) + } + + hasVolumes := len(m.stackProvider.GetDockerVolumes(stackName)) > 0 + restoreType := "config+DB" + if hasHDD || hasVolumes { + restoreType = "full (config+DB+userdata)" + } + m.logger.Printf("[INFO] RESTORE (Tier 2) completed: stack=%s, type=%s", stackName, restoreType) + return nil +} + +// restoreDockerVolumesFromDir populates Docker volumes from tar files in an arbitrary directory. +// Used by Tier 2 restore where volume tars are in the rsync mirror's _volumes/ dir. +func (m *Manager) restoreDockerVolumesFromDir(stackName, dumpDir string) error { + entries, err := os.ReadDir(dumpDir) + if err != nil { + if os.IsNotExist(err) { + return nil + } + return fmt.Errorf("reading volume dump dir: %w", err) + } + + var restored int + for _, entry := range entries { + if entry.IsDir() || !strings.HasSuffix(entry.Name(), ".tar") { + continue + } + volName := strings.TrimSuffix(entry.Name(), ".tar") + + m.logger.Printf("[INFO] [backup] Restoring Docker volume %s for %s (Tier 2)", volName, stackName) + + exec.Command("docker", "volume", "rm", "-f", volName).Run() + + if out, err := exec.Command("docker", "volume", "create", volName).CombinedOutput(); err != nil { + m.logger.Printf("[WARN] [backup] Failed to create volume %s: %s — %v", volName, strings.TrimSpace(string(out)), err) + continue + } + + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute) + cmd := exec.CommandContext(ctx, "docker", "run", "--rm", + "-v", volName+":/vol", + "-v", dumpDir+":/in:ro", + "alpine", "tar", "xf", "/in/"+entry.Name(), "-C", "/vol") + out, err := cmd.CombinedOutput() + cancel() + + if err != nil { + m.logger.Printf("[WARN] [backup] Failed to populate volume %s: %s — %v", volName, strings.TrimSpace(string(out)), err) + continue + } + + restored++ + } + + if restored > 0 { + m.logger.Printf("[INFO] [backup] Restored %d Docker volume(s) for %s (Tier 2)", restored, stackName) + } + return nil +} + +// restoreDockerVolumes populates Docker volumes from tar files in the volume dump directory. +func (m *Manager) restoreDockerVolumes(stackName, drivePath string) error { + dumpDir := AppVolumeDumpPath(drivePath, stackName) + entries, err := os.ReadDir(dumpDir) + if err != nil { + if os.IsNotExist(err) { + return nil // No volume dumps to restore + } + return fmt.Errorf("reading volume dump dir: %w", err) + } + + var restored int + for _, entry := range entries { + if entry.IsDir() || !strings.HasSuffix(entry.Name(), ".tar") { + continue + } + volName := strings.TrimSuffix(entry.Name(), ".tar") + + m.logger.Printf("[INFO] [backup] Restoring Docker volume %s for %s", volName, stackName) + + // Remove existing volume (ignore errors — may not exist) + exec.Command("docker", "volume", "rm", "-f", volName).Run() + + // Create fresh volume + if out, err := exec.Command("docker", "volume", "create", volName).CombinedOutput(); err != nil { + m.logger.Printf("[WARN] [backup] Failed to create volume %s: %s — %v", volName, strings.TrimSpace(string(out)), err) + continue + } + + // Populate from tar + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute) + cmd := exec.CommandContext(ctx, "docker", "run", "--rm", + "-v", volName+":/vol", + "-v", dumpDir+":/in:ro", + "alpine", "tar", "xf", "/in/"+entry.Name(), "-C", "/vol") + out, err := cmd.CombinedOutput() + cancel() + + if err != nil { + m.logger.Printf("[WARN] [backup] Failed to populate volume %s: %s — %v", volName, strings.TrimSpace(string(out)), err) + continue + } + + restored++ + if m.isDebug() { + m.logger.Printf("[DEBUG] [backup] Volume %s restored successfully", volName) + } + } + + if restored > 0 { + m.logger.Printf("[INFO] [backup] Restored %d Docker volume(s) for %s", restored, stackName) + } + return nil +} diff --git a/controller/internal/web/handlers.go b/controller/internal/web/handlers.go index e46df6f..b4e5ec1 100644 --- a/controller/internal/web/handlers.go +++ b/controller/internal/web/handlers.go @@ -1094,7 +1094,13 @@ func (s *Server) backupRestoreHandler(w http.ResponseWriter, r *http.Request) { s.logger.Printf("[WARN] [web] Restore requested: stack=%s, snapshot=%s from %s", stackName, snapshotID, r.RemoteAddr) start := time.Now() - if err := s.backupMgr.RestoreApp(stackName, snapshotID); err != nil { + var err error + if snapshotID == "tier2-rsync" { + err = s.backupMgr.RestoreAppFromTier2(stackName) + } else { + err = s.backupMgr.RestoreApp(stackName, snapshotID) + } + if err != nil { s.logger.Printf("[ERROR] [web] Restore failed: %v", err) if s.isDebug() { s.logger.Printf("[DEBUG] [web] backupRestoreHandler: stack=%s failed after %s", stackName, time.Since(start)) diff --git a/controller/internal/web/templates/backups.html b/controller/internal/web/templates/backups.html index 0d0b2dd..67fa9e7 100644 --- a/controller/internal/web/templates/backups.html +++ b/controller/internal/web/templates/backups.html @@ -268,6 +268,8 @@ {{else if .HasHDDData}} {{if .StorageLabel}}{{.StorageLabel}}{{end}} {{.HDDSizeHuman}} + {{else if .HasVolumeData}} + Konfig{{if .HasDB}} + DB{{end}} + Adatok {{else}} Konfig{{if .HasDB}} + DB{{end}} {{end}} @@ -540,7 +542,7 @@ @@ -568,6 +570,9 @@
+ {{end}} @@ -731,8 +736,9 @@ function onRestoreAppChange() { var opt = sel.options[sel.selectedIndex]; var hasHDD = opt.getAttribute('data-has-hdd') === 'true'; var hasDB = opt.getAttribute('data-has-db') === 'true'; + var hasVolumes = opt.getAttribute('data-has-volumes') === 'true'; - if (hasHDD) { + if (hasHDD || hasVolumes) { typeInfo.innerHTML = '🔄 Teljes visszaállítás: adatbázis + konfiguráció + felhasználói adatok a kiválasztott pillanatképből.'; typeInfo.className = 'restore-info'; } else if (hasDB) {