# TASK: Phase A — Storage Paths Foundation & Backup Toggle Fix **Version target:** controller 0.9.0 **Repos:** `deploy-felhom-compose` (controller), `app-catalog-felhom.eu` (compose mount change) ## Overview Per-app backup toggles (implemented in v0.8.0) don't appear on the backup page. Root cause is two-layered: 1. **`controller.yaml` has no `paths.hdd_path`** → `m.cfg.Paths.HDDPath` is `""` → `ParseComposeHDDMounts` returns nil → `DiscoverAppData` finds zero HDD data → no backup toggles shown 2. **Even with a global hdd_path, the parser uses ONE path** → apps deployed with a different `HDD_PATH` (e.g., `/mnt/hdd_placeholder` vs `/mnt/hdd_1`) won't match The fix: stop relying on a single global HDD path. Instead, read each app's **own** `HDD_PATH` from its `app.yaml` env section, and introduce a storage paths registry in `settings.json` that auto-discovers paths from deployed apps. This phase also lays the foundation for multi-storage management (Phase B: UI polish, Phase C: migration wizard). --- ## 1. Root Cause Analysis ### Current broken flow ``` controller.yaml → paths.hdd_path = "" (missing) ↓ backup.Manager.stackProvider.GetStackHDDMounts(name) → stacks.ParseComposeHDDMounts(composePath, hddPath="") → hddPath == "" → return nil ← BUG: exits early ↓ DiscoverAppData → info.HasHDDData = false for ALL apps ↓ backups.html → {{if .HasHDDData}} → false → no checkbox rendered ``` ### Second layer (would hit even with hdd_path set) ``` controller.yaml → paths.hdd_path = "/mnt/hdd_1" ↓ ParseComposeHDDMounts(composePath, "/mnt/hdd_1") → reads compose: "- ${HDD_PATH}/storage/immich:/usr/src/app/upload" → resolves ${HDD_PATH} → "/mnt/hdd_1/storage/immich" → BUT app.yaml has HDD_PATH="/mnt/hdd_placeholder" → actual data is at /mnt/hdd_placeholder/storage/immich → os.Stat("/mnt/hdd_1/storage/immich") → not found → path.Exists = false, SizeBytes = 0 ``` ### Correct approach Read each app's actual `HDD_PATH` from its `app.yaml` env section. The compose template uses `${HDD_PATH}` which gets resolved at `docker compose up` time from the environment. We need to resolve it the same way during discovery. --- ## 2. Storage Paths Registry ### 2.1 New struct in `settings.go` ```go // StoragePath represents a registered external storage location. type StoragePath struct { Path string `json:"path"` // e.g., "/mnt/hdd_1" Label string `json:"label,omitempty"` // e.g., "Külső HDD 1TB" IsDefault bool `json:"is_default,omitempty"` // new apps use this by default Schedulable bool `json:"schedulable"` // whether new apps can be deployed here AddedAt string `json:"added_at"` // RFC3339 } ``` ### 2.2 Add to Settings struct ```go type Settings struct { // ... existing fields ... // Storage paths registry StoragePaths []StoragePath `json:"storage_paths,omitempty"` } ``` ### 2.3 Helper methods on Settings ```go // GetStoragePaths returns all registered storage paths. func (s *Settings) GetStoragePaths() []StoragePath // GetDefaultStoragePath returns the default storage path, or "" if none. func (s *Settings) GetDefaultStoragePath() string // GetSchedulableStoragePaths returns paths where new apps can be deployed. func (s *Settings) GetSchedulableStoragePaths() []StoragePath // AddStoragePath registers a new storage path after validation. func (s *Settings) AddStoragePath(path, label string, isDefault bool) error // RemoveStoragePath removes a path only if no apps reference it. // Returns error with list of apps using this path if removal is blocked. func (s *Settings) RemoveStoragePath(path string, appChecker func(string) []string) error // SetDefaultStoragePath changes which path is the default. func (s *Settings) SetDefaultStoragePath(path string) error // SetSchedulable enables/disables a path for new deployments. func (s *Settings) SetSchedulable(path string, schedulable bool) error ``` ### 2.4 Validation rules (in AddStoragePath) 1. **Must be a real mount point** — compare device ID of path vs parent using `syscall.Stat`. Reject paths that are just directories on the boot SSD. Error message: "Ez az útvonal nem külön csatlakoztatott meghajtó. Adatok az SSD-re kerülnének." 2. **Must exist and be a directory** — `os.Stat` check 3. **Must be writable** — attempt to create + remove a temp file 4. **No overlapping paths** — reject if new path is a parent or child of an existing path (e.g., `/mnt/hdd_1` and `/mnt/hdd_1/subdir`) 5. **No duplicates** — reject if path already registered (normalized with `filepath.Clean`) 6. **Must be under `/mnt/`** — soft warning, not hard block (log a WARN for edge cases) ### 2.5 Auto-discovery on startup On controller startup, if `StoragePaths` is empty in settings.json, auto-discover from deployed apps: ```go func (s *Settings) AutoDiscoverStoragePaths(stacksDir string, fallbackHDDPath string, logger *log.Logger) { if len(s.StoragePaths) > 0 { return // already configured, don't override } // Scan all deployed apps' app.yaml for HDD_PATH values seen := map[string]bool{} entries, _ := os.ReadDir(stacksDir) for _, e := range entries { if !e.IsDir() { continue } appCfg := LoadAppConfig(filepath.Join(stacksDir, e.Name())) if appCfg == nil || !appCfg.Deployed { continue } if hddPath, ok := appCfg.Env["HDD_PATH"]; ok && hddPath != "" { cleaned := filepath.Clean(hddPath) if !seen[cleaned] { seen[cleaned] = true } } } // Also use controller.yaml paths.hdd_path as fallback seed if fallbackHDDPath != "" { cleaned := filepath.Clean(fallbackHDDPath) if !seen[cleaned] { seen[cleaned] = true } } // Register discovered paths isFirst := true for path := range seen { sp := StoragePath{ Path: path, Label: inferStorageLabel(path), IsDefault: isFirst, Schedulable: true, AddedAt: time.Now().UTC().Format(time.RFC3339), } s.StoragePaths = append(s.StoragePaths, sp) isFirst = false } if len(s.StoragePaths) > 0 { s.Save() logger.Printf("[INFO] Auto-discovered %d storage path(s)", len(s.StoragePaths)) for _, sp := range s.StoragePaths { logger.Printf("[INFO] %s (%s) default=%v", sp.Path, sp.Label, sp.IsDefault) } } } ``` Helper `inferStorageLabel(path)`: - `/mnt/hdd_1` → "Külső tárhely (hdd_1)" - `/mnt/hdd_placeholder` → "Külső tárhely (hdd_placeholder)" - Anything else → path basename Note: `LoadAppConfig` is in the `stacks` package. To avoid circular imports, either pass a scanner function or do the scanning in `main.go` and pass the discovered paths in. Use whichever approach avoids import issues. --- ## 3. Fix `GetStackHDDMounts` — Per-App HDD_PATH Resolution ### 3.1 Change the adapter Currently: ```go func (a *stackAdapter) GetStackHDDMounts(name string) []string { s, ok := a.mgr.GetStack(name) if !ok { return nil } return stacks.ParseComposeHDDMounts(s.ComposePath, a.hddPath) // ← uses global path } ``` Change to: ```go func (a *stackAdapter) GetStackHDDMounts(name string) []string { s, ok := a.mgr.GetStack(name) if !ok { return nil } // Priority 1: Read the app's own HDD_PATH from its app.yaml stackDir := filepath.Dir(s.ComposePath) appCfg := stacks.LoadAppConfig(stackDir) if appCfg != nil && appCfg.Env["HDD_PATH"] != "" { return stacks.ParseComposeHDDMounts(s.ComposePath, appCfg.Env["HDD_PATH"]) } // Priority 2: Try all registered storage paths (fallback) var allMounts []string seen := map[string]bool{} for _, sp := range a.getStoragePaths() { mounts := stacks.ParseComposeHDDMounts(s.ComposePath, sp.Path) for _, m := range mounts { if !seen[m] { seen[m] = true allMounts = append(allMounts, m) } } } return allMounts } ``` ### 3.2 Update adapter struct ```go type stackAdapter struct { mgr *stacks.Manager getStoragePaths func() []settings.StoragePath // getter from settings } ``` Wire in `main.go`: ```go adapter := &stackAdapter{ mgr: stackMgr, getStoragePaths: func() []settings.StoragePath { return settingsMgr.GetStoragePaths() }, } ``` Remove the old `hddPath string` field from the adapter. ### 3.3 `resolveAppBackupPaths()` in backup Manager No changes needed — it already calls `m.stackProvider.GetStackHDDMounts(stackName)` which we're fixing above. The fix propagates automatically. --- ## 4. Controller Docker Compose Mount Change ### 4.1 Change in `controller/docker-compose.yml` Replace: ```yaml - ${HDD_PATH:-/mnt/hdd_placeholder}:${HDD_PATH:-/mnt/hdd_placeholder}:ro ``` With: ```yaml # All external storage — covers /mnt/* paths for multi-storage support + restore - /mnt:/mnt:rw ``` **Why `:rw`:** Required for restore feature (already implemented). Also needed for mount-point validation (write test in AddStoragePath). **Why `/mnt:/mnt`:** All standard storage mounts are under `/mnt`. Covers current and future drives with one entry. No controller restart when adding new drives. ### 4.2 Also update `docker-setup.sh` If `docker-setup.sh` generates the controller compose, update it to emit `/mnt:/mnt:rw` instead of the old `${HDD_PATH}` line. --- ## 5. Deploy Page: Storage Path Dropdown ### 5.1 When app has a `path` type deploy field (HDD_PATH) Currently renders as a plain text input. Change to dropdown when storage paths exist: ```html {{if eq .Type "path"}} {{if $.StoragePaths}} {{else}} Nincs regisztrált adattároló. Adjon hozzá egyet a Beállítások oldalon. {{end}} {{end}} ``` ### 5.2 Pass storage paths to deploy page template data In the deploy handler: ```go data["StoragePaths"] = settingsMgr.GetSchedulableStoragePaths() ``` ### 5.3 Already-deployed: show current path read-only When `AlreadyDeployed == true`, the field is disabled. Show the actual `HDD_PATH` from `app.yaml` as the displayed value (existing behavior works as-is with a disabled select or text input). ### 5.4 Edge case: no schedulable paths but app needs HDD If `resources.needs_hdd: true` and zero schedulable paths: - Show warning: "Nincs elérhető adattároló. Csatlakoztasson külső meghajtót és adja hozzá a Beállítások oldalon." - Disable deploy button (add `data-needs-storage="true"` to form, JS disables submit) --- ## 6. Storage Path Monitoring Integration ### 6.1 Periodic mount-point check Add to existing system health check (runs every `system_health_interval`, default 5m): ```go func checkStoragePaths(paths []settings.StoragePath, logger *log.Logger) []string { var warnings []string for _, sp := range paths { // Check 1: path exists and is directory fi, err := os.Stat(sp.Path) if err != nil { warnings = append(warnings, fmt.Sprintf("Adattároló nem elérhető: %s", sp.Path)) continue } if !fi.IsDir() { warnings = append(warnings, fmt.Sprintf("Adattároló nem mappa: %s", sp.Path)) continue } // Check 2: still a real mount point (not writing to SSD) if !isMountPoint(sp.Path) { warnings = append(warnings, fmt.Sprintf("FIGYELEM: %s nincs felcsatolva (leválasztva?) — "+ "az adatok az SSD-re íródnának!", sp.Path)) } // Check 3: disk space (reuse existing threshold config) usage := getDiskUsage(sp.Path) if usage.UsedPercent >= 90 { warnings = append(warnings, fmt.Sprintf("Adattároló majdnem tele: %s (%.0f%%)", sp.Path, usage.UsedPercent)) } } return warnings } ``` ### 6.2 `isMountPoint()` implementation New file: `controller/internal/system/mounts.go` ```go // IsMountPoint checks if a path is a mount point by comparing device IDs. // Returns true if path is on a different device than its parent. func IsMountPoint(path string) bool { var pathStat, parentStat syscall.Stat_t if err := syscall.Stat(path, &pathStat); err != nil { return false } parent := filepath.Dir(path) if err := syscall.Stat(parent, &parentStat); err != nil { return false } return pathStat.Dev != parentStat.Dev } ``` Platform-specific: needs build tag `//go:build linux` (consistent with existing system package). Add stub for non-Linux that always returns true. ### 6.3 Surface warnings - Add storage warnings to monitoring page warnings section (existing infrastructure) - Include in hub report `SystemWarnings` field (already exists) - Fire `disk_warning` notification event for enabled customers --- ## 7. Beállítások Page: Storage Section Add between "Rendszer konfiguráció" and "Jelszó módosítás" sections. ### 7.1 Template: "Adattárolók" section ```html

Adattárolók

Külső meghajtók, ahol az alkalmazások felhasználói adatai tárolódnak.

{{if .StorageError}}
{{.StorageError}}
{{end}} {{if .StorageSuccess}}
{{.StorageSuccess}}
{{end}} {{if .StoragePaths}}
{{range .StoragePaths}}
{{.Label}} {{.Path}}
{{if .IsDefault}}alapértelmezett{{end}} {{if not .Schedulable}}nem ütemezhet{{end}} {{if .DiskInfo}} {{.DiskInfo.UsedHuman}} / {{.DiskInfo.TotalHuman}} ({{.DiskInfo.UsedPercent}}%) {{end}}
{{if .AppCount}}
{{.AppCount}} alkalmazás használja
{{end}}
{{if not .IsDefault}}
{{end}}
{{if and (not .IsDefault) (eq .AppCount 0)}}
{{end}}
{{end}}
{{else}}
Nincs regisztrált adattároló. Csatlakoztasson külső meghajtót és adja hozzá alább.
{{end}}
+ Új adattároló hozzáadása
A felcsatolt meghajtó elérési útja (pl. /mnt/hdd_1)
``` ### 7.2 Routes | Method | Path | Auth? | Handler | |--------|------|-------|---------| | POST | `/settings/storage/add` | Yes | Add storage path with validation | | POST | `/settings/storage/remove` | Yes | Remove (blocked if apps use it) | | POST | `/settings/storage/default` | Yes | Set default storage path | | POST | `/settings/storage/schedulable` | Yes | Toggle schedulable on/off | ### 7.3 Handler: `/settings/storage/add` 1. Parse form: `storage_path`, `storage_label`, `storage_default` 2. Clean path with `filepath.Clean` 3. Validate (Section 2.4): - Exists + is directory → error: "Az útvonal nem létezik vagy nem mappa." - Is mount point → error: "Ez az útvonal nem külön csatlakoztatott meghajtó. Adatok az SSD-re kerülnének!" - Writable → error: "Az útvonal nem írható." - No overlap → error: "Az útvonal átfedi a már regisztrált XYZ útvonalat." - No duplicate → error: "Ez az útvonal már regisztrálva van." 4. If `storage_default`, unset previous default 5. Add to settings.json via `AddStoragePath()` 6. Redirect to `/settings` with flash: "Adattároló sikeresen hozzáadva: /mnt/hdd_1" 7. On error: redirect with error flash ### 7.4 Handler: `/settings/storage/remove` 1. Parse form: `storage_path` 2. Count apps using this path (scan app.yaml files for `HDD_PATH`) 3. If apps found → error: "Nem törölhető: az alábbi alkalmazások használják: Immich, Paperless-ngx" 4. Cannot be the default path → error: "Az alapértelmezett adattároló nem törölhető." 5. If only one path left → error: "Az utolsó adattároló nem törölhető." 6. Remove from settings.json 7. Redirect with success flash ### 7.5 Template data for settings page Add to the settings handler: ```go type StoragePathView struct { settings.StoragePath DiskInfo *DiskUsageInfo // total, used, percent AppCount int // number of deployed apps with HDD_PATH matching this IsMounted bool // mount-point check result } ``` Populate by scanning `app.yaml` files for `HDD_PATH` and matching against registered paths. --- ## 8. Deprecate `paths.hdd_path` from controller.yaml ### 8.1 Backward compatibility Keep reading `paths.hdd_path` — use it only as fallback seed for auto-discovery on first run. Startup order in `main.go`: ```go // 1. Load settings.json settingsMgr, _ := settings.Load(...) // 2. Auto-discover storage paths (if settings.json has none) settingsMgr.AutoDiscoverStoragePaths(cfg.Paths.StacksDir, cfg.Paths.HDDPath, logger) // 3. Wire adapter with storage paths getter adapter := &stackAdapter{ mgr: stackMgr, getStoragePaths: func() []settings.StoragePath { return settingsMgr.GetStoragePaths() }, } ``` ### 8.2 Update configs/controller.yaml.example ```yaml paths: stacks_dir: "/opt/docker/stacks" # hdd_path is DEPRECATED — storage paths are managed via web UI (Beállítások > Adattárolók) # Existing value is auto-migrated to settings.json on first startup # hdd_path: "/mnt/hdd_1" ``` --- ## 9. Implementation Steps ### Step 1: Storage paths in settings.json - Add `StoragePath` struct and `StoragePaths` field to `Settings` - Add all getter/setter methods - Add `AutoDiscoverStoragePaths()` — note: may need to pass a scanner function or do scanning in main.go to avoid circular import with stacks package - Add `inferStorageLabel()` helper - **Test:** Manually create a `settings.json` with `storage_paths` → Load → verify GetStoragePaths returns them ### Step 2: Mount-point validation utilities - Create `controller/internal/system/mounts.go` with `IsMountPoint()` (Linux + stub) - Add overlap check helper: `PathsOverlap(a, b string) bool` - Add writable check helper: `IsWritable(path string) bool` - **Test:** `IsMountPoint("/mnt/hdd_1")` → true, `IsMountPoint("/tmp")` → false ### Step 3: Wire auto-discovery on startup - In `main.go`: call `AutoDiscoverStoragePaths` after loading settings - Pass `cfg.Paths.HDDPath` as fallback - **Test:** Start controller on demo-felhom with empty settings.json → should auto-discover `/mnt/hdd_placeholder` from deployed apps' app.yaml ### Step 4: Fix per-app HDD_PATH resolution (THE CORE FIX) - Update `stackAdapter` struct: replace `hddPath string` with `getStoragePaths func()` - Update `GetStackHDDMounts()` to read per-app HDD_PATH from app.yaml first, fallback to registered paths - Wire updated adapter in `main.go` - **Test:** Backup page → "Alkalmazás adatok" section now shows with correct paths (e.g., `/mnt/hdd_placeholder/storage/immich`). Backup toggles are visible and functional. ### Step 5: Controller compose mount change - Change `controller/docker-compose.yml`: `/mnt:/mnt:rw` - Update `docker-setup.sh` controller compose generation - **Test:** Recreate controller container → verify both `/mnt/hdd_1` and `/mnt/hdd_placeholder` accessible inside container ### Step 6: Beállítások storage section - Add "Adattárolók" section to `settings.html` - Add handlers: `POST /settings/storage/add`, `/remove`, `/default`, `/schedulable` - Add `StoragePathView` type for template data with disk info + app count - Pass storage data to settings page handler - Add CSS for storage path items - **Test:** Add path via UI → validation catches non-mount-point. Remove path with apps → blocked with app list. Set default → badge updates. ### Step 7: Deploy page dropdown - Modify `deploy.html`: path field becomes dropdown when storage paths exist - Pass `StoragePaths` to deploy page template data - Fall back to text input if no paths registered - Block deploy if app needs HDD but no schedulable paths - **Test:** Deploy page for Immich → dropdown shows registered paths. Already-deployed shows current path read-only. ### Step 8: Storage monitoring integration - Add `checkStoragePaths()` to system health check - Surface warnings on monitoring page - Include in hub report - Fire `disk_warning` notification for unmounted drives - **Test:** (Simulated) If path doesn't exist → warning appears on monitoring page within one health check cycle ### Step 9: Cleanup & version bump - Deprecate `paths.hdd_path` in controller.yaml.example - Create new CHANGELOG.md, changelogs will be updated there, not in CONTEXT.md. CONTEXT.md will only have information about architecture decisions, roadmap, information about the project - Regenerate CONTEXT.md with current architecture, deparating different sections, with detailed descriptions how it should work, what is planned, what is the architecture. Sections like: Storage management, Notifications, Backup management, App management, Monitoring, Infra, Settings management, etc-etc.. - Update CONTEXT.md / CHANGELOG.md / CLAUDE.md - Bump to 0.9.0 - Build + deploy - **Test end-to-end:** Auto-discover → backup toggles visible → enable Immich backup → manual backup → verify HDD data in restic snapshot → storage management in Beállítások works --- ## 10. Files to Create / Modify ### New files: - `controller/internal/system/mounts.go` — `IsMountPoint()`, `IsWritable()`, `PathsOverlap()` + non-Linux stub - (No new packages — StoragePath goes in existing `settings` package) ### Modified files: - `controller/internal/settings/settings.go` — `StoragePath` struct, `StoragePaths` field, all getter/setter methods, `AutoDiscoverStoragePaths()`, `inferStorageLabel()`, validation logic - `controller/cmd/controller/main.go` — Wire auto-discovery, update stackAdapter struct (replace `hddPath` with `getStoragePaths`), update adapter constructor - `controller/internal/web/handlers.go` — Storage management handlers (`/settings/storage/*`), pass StoragePaths to deploy + settings templates - `controller/internal/web/server.go` — Register new storage routes - `controller/internal/web/templates/settings.html` — New "Adattárolók" section - `controller/internal/web/templates/deploy.html` — Path field → dropdown, edge case warnings - `controller/internal/web/templates/style.css` — Storage path item styles, badges - `controller/internal/monitoring/health.go` (or wherever health checks live) — Add `checkStoragePaths()` - `controller/docker-compose.yml` — `/mnt:/mnt:rw` - `controller/configs/controller.yaml.example` — Deprecation comment - `docker-setup.sh` — Update controller compose generation --- ## 11. Design Decisions & Notes ### Why settings.json, not controller.yaml? controller.yaml is operator-controlled (read-only from customer perspective). User-configurable state lives in settings.json. Clear separation: operator configures infrastructure, customer configures preferences. ### Why auto-discover from app.yaml? Existing deployments already have apps with HDD_PATH set. Auto-discovery makes the upgrade seamless — backup toggles appear without manual intervention after the update. ### Why `/mnt:/mnt:rw` instead of individual mounts? Single mount covers all current and future storage devices. No controller restart when adding new drives. Required for restore + validation. Trade-off: broader access, but controller already has Docker socket access and is a trusted component. ### Why validate mount points? If a USB drive is disconnected but the empty directory remains (created by Docker or `nofail` fstab), data silently writes to the SSD boot drive. Mount-point check (device ID comparison) catches this. This is the #1 gotcha for home server setups. ### Why no overlapping paths? If `/mnt/hdd_1` and `/mnt/hdd_1/storage` are both registered, discovery counts files twice and backup may include data twice. One entry per physical mount point keeps it simple. ### Why fallback chain in GetStackHDDMounts? 1. App's own HDD_PATH from app.yaml (most accurate — the actual deployed value) 2. Try all registered storage paths (handles edge cases: missing app.yaml, new template before deploy) Ensures backup discovery works even in degraded states. ### Why `Schedulable` toggle? An "unschedulable" path means existing apps stay put but no new apps can use it. Useful for drives that are filling up or being deprecated — prevents new deployments while allowing existing apps to continue operating. ### Future phases (NOT in scope) - **Phase B:** Storage management UI polish — disk usage bars per path, per-app storage column on apps page - **Phase C:** Migration wizard — "Mozgatás" button per app, rsync with progress reporting, automatic app.yaml HDD_PATH update + restart, old data cleanup option