v0.12.6: fix rsync destination nesting and exclude app-internal DB dumps
Two rsync bugs found during Immich cross-drive backup testing: Fix 3: Simplified destination path structure in runRsyncBackup. Old SplitN logic kept "storage/immich" as a subpath, creating redundant nesting: backups/rsync/immich/storage/immich/<data>. New logic: single mount → rsync directly into the stack folder; multiple mounts → use each mount's leaf dir name as subfolder (disambiguated by _N suffix). Fix 4: Exclude app-internal DB dump files from rsync. Apps like Immich store periodic pg_dumps in <data>/backups/*.sql.gz. The controller already handles DB backups via pg_dump — copying them again wastes space. Added --exclude backups/*.sql.gz/sql/dump to the rsync command. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -1,5 +1,25 @@
|
|||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
|
### What was just completed (2026-02-18 session 42)
|
||||||
|
- **v0.12.6 — Cross-Drive Backup Rsync Fixes:**
|
||||||
|
|
||||||
|
**Context:** After fixing mount-point validation and system-drive thresholds (v0.12.5), testing revealed two more rsync issues for Immich.
|
||||||
|
|
||||||
|
**Fix 3: Simplified rsync destination path structure (`internal/backup/crossdrive.go` `runRsyncBackup`)**
|
||||||
|
- Old logic stripped only the first 2 path segments and kept the rest as a subpath, producing redundant nesting: `backups/rsync/immich/storage/immich/<data>` instead of `backups/rsync/immich/<data>`
|
||||||
|
- New logic: if app has a single mount, rsync directly into the stack folder (`backups/rsync/immich/`); if multiple mounts, use each mount's leaf directory name as subfolder
|
||||||
|
- Duplicate leaf names disambiguated by appending `_N` index suffix
|
||||||
|
- Loop variable changed from `_, srcMount` to `i, srcMount` to support the index-based disambiguation
|
||||||
|
- Old nested `storage/immich/` folder will remain orphaned after first run (no data loss; `--delete` only affects the target subtree)
|
||||||
|
|
||||||
|
**Fix 4: Exclude app-internal DB dump files from rsync (`internal/backup/crossdrive.go` `runRsyncBackup`)**
|
||||||
|
- Apps like Immich store their own periodic DB dumps in `<data>/backups/*.sql.gz` (~16 MB/day)
|
||||||
|
- The controller already handles DB backups via `pg_dump` separately — copying these again via rsync is redundant and wastes space
|
||||||
|
- Added `--exclude backups/*.sql.gz`, `--exclude backups/*.sql`, `--exclude backups/*.dump` to rsync command
|
||||||
|
- The `backups/` directory itself and non-dump files within it are preserved
|
||||||
|
|
||||||
|
**Files modified (1):** `internal/backup/crossdrive.go`
|
||||||
|
|
||||||
### What was just completed (2026-02-18 session 41)
|
### What was just completed (2026-02-18 session 41)
|
||||||
- **v0.12.5 — Cross-Drive Backup Validation Fix:**
|
- **v0.12.5 — Cross-Drive Backup Validation Fix:**
|
||||||
|
|
||||||
|
|||||||
@@ -4,7 +4,7 @@
|
|||||||
|
|
||||||
A single, lightweight Go container that replaces Portainer + scattered systemd scripts with a unified, Hungarian-language web dashboard for managing Docker Compose stacks, backups, storage, monitoring, and notifications on customer hardware.
|
A single, lightweight Go container that replaces Portainer + scattered systemd scripts with a unified, Hungarian-language web dashboard for managing Docker Compose stacks, backups, storage, monitoring, and notifications on customer hardware.
|
||||||
|
|
||||||
**Current version: v0.12.5**
|
**Current version: v0.12.6**
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -196,6 +196,10 @@ Implements the 3-2-1 backup rule by copying data to a different physical drive.
|
|||||||
- External mount: block if <100 MB free; warn/block at 90%/95% usage
|
- External mount: block if <100 MB free; warn/block at 90%/95% usage
|
||||||
- System drive (same block device as `/`): require ≥10 GB free AND <90% usage to protect OS stability; allowed with a logged warning (no hard block for non-mount-point destinations)
|
- System drive (same block device as `/`): require ≥10 GB free AND <90% usage to protect OS stability; allowed with a logged warning (no hard block for non-mount-point destinations)
|
||||||
- Web UI `CheckBackupDestination` matches runner thresholds — no surprise divergence between UI and actual enforcement
|
- Web UI `CheckBackupDestination` matches runner thresholds — no surprise divergence between UI and actual enforcement
|
||||||
|
- **Rsync destination layout** (`runRsyncBackup`):
|
||||||
|
- Single mount: data goes directly into `backups/rsync/<app>/` (no extra nesting)
|
||||||
|
- Multiple mounts: each gets a `backups/rsync/<app>/<leaf>/` subfolder named after the mount's base directory; duplicate leaf names disambiguated with `_N` suffix
|
||||||
|
- DB dump files excluded: `--exclude backups/*.sql.gz`, `--exclude backups/*.sql`, `--exclude backups/*.dump` — avoids duplicating data already managed by the pg_dump layer
|
||||||
- Safety guards: destination ≠ source, path-overlap check, writable check
|
- Safety guards: destination ≠ source, path-overlap check, writable check
|
||||||
- **Chained execution**: cross-drive runs immediately after nightly restic backup (daily apps every night, weekly apps on Sundays) for DB/file consistency
|
- **Chained execution**: cross-drive runs immediately after nightly restic backup (daily apps every night, weekly apps on Sundays) for DB/file consistency
|
||||||
- Per-app concurrency lock prevents overlapping runs
|
- Per-app concurrency lock prevents overlapping runs
|
||||||
|
|||||||
@@ -203,20 +203,22 @@ func (r *CrossDriveRunner) runRsyncBackup(ctx context.Context, stackName, destBa
|
|||||||
return fmt.Errorf("creating rsync dest dir: %w", err)
|
return fmt.Errorf("creating rsync dest dir: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, srcMount := range mounts {
|
for i, srcMount := range mounts {
|
||||||
// Preserve directory structure: strip the storage path prefix to get relative subpath
|
var dstPath string
|
||||||
// e.g., /mnt/hdd_placeholder/storage/immich/ → storage/immich/
|
if len(mounts) == 1 {
|
||||||
rel := srcMount
|
// Single mount: rsync directly into the stack folder (no extra nesting)
|
||||||
// Find the topmost non-root segment of the mount path (after the mount point itself)
|
dstPath = destDir
|
||||||
// Use a simple approach: keep everything from the first significant segment after /mnt/...
|
|
||||||
parts := strings.SplitN(strings.TrimPrefix(srcMount, "/"), "/", 3)
|
|
||||||
if len(parts) >= 3 {
|
|
||||||
rel = parts[2] // e.g., "storage/immich"
|
|
||||||
} else {
|
} else {
|
||||||
rel = filepath.Base(srcMount)
|
// Multiple mounts: use the leaf directory name as subfolder
|
||||||
|
leaf := filepath.Base(srcMount)
|
||||||
|
dstPath = filepath.Join(destDir, leaf)
|
||||||
|
// Disambiguate duplicate leaf names (e.g. two mounts both named "data")
|
||||||
|
if i > 0 {
|
||||||
|
if _, err := os.Stat(dstPath); err == nil {
|
||||||
|
dstPath = filepath.Join(destDir, fmt.Sprintf("%s_%d", leaf, i))
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
dstPath := filepath.Join(destDir, rel)
|
|
||||||
if err := os.MkdirAll(dstPath, 0755); err != nil {
|
if err := os.MkdirAll(dstPath, 0755); err != nil {
|
||||||
return fmt.Errorf("creating rsync destination: %w", err)
|
return fmt.Errorf("creating rsync destination: %w", err)
|
||||||
}
|
}
|
||||||
@@ -225,7 +227,12 @@ func (r *CrossDriveRunner) runRsyncBackup(ctx context.Context, stackName, destBa
|
|||||||
src := strings.TrimRight(srcMount, "/") + "/"
|
src := strings.TrimRight(srcMount, "/") + "/"
|
||||||
dst := strings.TrimRight(dstPath, "/") + "/"
|
dst := strings.TrimRight(dstPath, "/") + "/"
|
||||||
|
|
||||||
cmd := exec.CommandContext(ctx, "rsync", "-a", "--delete", src, dst)
|
// Exclude app-internal DB dump files — the controller handles DB backups via pg_dump separately.
|
||||||
|
cmd := exec.CommandContext(ctx, "rsync", "-a", "--delete",
|
||||||
|
"--exclude", "backups/*.sql.gz",
|
||||||
|
"--exclude", "backups/*.sql",
|
||||||
|
"--exclude", "backups/*.dump",
|
||||||
|
src, dst)
|
||||||
r.logger.Printf("[DEBUG] rsync: %s → %s", src, dst)
|
r.logger.Printf("[DEBUG] rsync: %s → %s", src, dst)
|
||||||
if out, err := cmd.CombinedOutput(); err != nil {
|
if out, err := cmd.CombinedOutput(); err != nil {
|
||||||
return fmt.Errorf("rsync failed for %s: %v (%s)", srcMount, err, strings.TrimSpace(string(out)))
|
return fmt.Errorf("rsync failed for %s: %v (%s)", srcMount, err, strings.TrimSpace(string(out)))
|
||||||
|
|||||||
Reference in New Issue
Block a user