BUGFIX: Storage Scan — System Disk Detection & FSType in Container

This commit is contained in:
2026-02-17 10:41:05 +01:00
parent 2fb2c6e1ae
commit 0b23834e3c
+213 -666
View File
@@ -1,705 +1,252 @@
# TASK: Phase C — Storage Initialization, Data Migration & Startup Fixes # BUGFIX: Storage Scan — System Disk Detection & FSType in Container
**Version target:** controller 0.11.0 **Affects:** v0.11.0, `internal/storage/scan_linux.go`
**Repo:** `deploy-felhom-compose` (controller) **Root cause:** Controller runs in a Docker container. Even with `--privileged`, `lsblk` reports mount points from the container's mount namespace (not host), and often can't probe filesystem types due to missing udev/blkid cache.
## Overview ## Bug 1: System disk (sda) shows as available
Three features in this phase: ### Current broken logic
```go
if part.MountPoint == "/" || part.MountPoint == "/boot" || part.MountPoint == "/boot/efi" {
isSystem = true
}
```
Inside the container, sda2 (host's `/`) shows mounted at `/opt/docker/felhom-controller/data` (bind mount), not `/`. So `isSystem` stays false → sda appears in AvailableDisks.
1. **Startup ping + hub report** — Controller should announce itself immediately on start, not wait 515 minutes for the first scheduler tick ### Fix: Parse host's fstab + blkid to detect system disk
2. **Storage initialization** — Detect unformatted/unmounted disks, format (ext4), mount, and register as storage path — all from the web UI
3. **Data migration** — Per-app "Mozgatás" button to move app data between storage paths with rsync + progress
--- The host's fstab is mounted at `/host-fstab` inside the container. Parse it to find which devices/UUIDs are used for `/`, `/boot`, `/boot/efi`, and `swap`. Then resolve UUIDs to device paths via `blkid`, and mark their parent disks as system disks.
## 0. Startup Ping & Hub Report (Quick Fix)
### 0.1 Problem
After controller starts (e.g., after `docker restart felhom-controller` or system reboot), the first heartbeat fires after 5 minutes, first system_health after 5 minutes, and first hub report after 15 minutes. During this gap, Healthchecks shows stale "Last Ping: X minutes ago" and hub has no fresh data.
### 0.2 Fix: Fire initial pings + report immediately after scheduler starts
Add to `main.go` after `sched.Start(ctx)`, inside a goroutine (non-blocking):
```go ```go
// Fire startup pings + hub report immediately (don't wait for first scheduler tick) // getSystemDiskNames returns the set of parent disk names (e.g., "sda")
go func() { // that contain system partitions (/, /boot, /boot/efi, swap).
time.Sleep(5 * time.Second) // Let all subsystems fully initialize func getSystemDiskNames() map[string]bool {
systemDisks := map[string]bool{}
// Heartbeat ping
pinger.Ping(cfg.Monitoring.PingUUIDs.Heartbeat, "startup") // Step 1: Parse /host-fstab for system mount points
logger.Println("[INFO] Startup heartbeat ping sent") fstabPath := "/host-fstab"
if _, err := os.Stat(fstabPath); err != nil {
// System health ping // Fallback: try /etc/fstab (if not containerized or different mount)
healthReport := monitor.RunHealthCheck(cfg, cpuCollector, sett.GetStoragePaths()) fstabPath = "/etc/fstab"
body := healthReport.FormatMessage()
healthUUID := cfg.Monitoring.PingUUIDs.SystemHealth
if healthReport.Status == "fail" {
pinger.Fail(healthUUID, body)
} else {
pinger.Ping(healthUUID, body)
} }
logger.Printf("[INFO] Startup health ping sent (status: %s)", healthReport.Status)
data, err := os.ReadFile(fstabPath)
// Hub report if err != nil {
if cfg.Hub.Enabled && cfg.Hub.URL != "" { return systemDisks // Can't read fstab, return empty (safe default: nothing excluded)
pusher := report.NewPusher(&cfg.Hub, logger) }
r := report.BuildReport(cfg, stackMgr, backupMgr, cpuCollector, metricsStore, Version, sett.GetStoragePaths())
if err := pusher.Push(r); err != nil { // System mount points we care about
logger.Printf("[WARN] Startup hub report failed: %v", err) systemMounts := map[string]bool{"/": true, "/boot": true, "/boot/efi": true}
} else {
logger.Println("[INFO] Startup hub report sent") var systemUUIDs []string
var systemDevices []string
for _, line := range strings.Split(string(data), "\n") {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
continue
}
fields := strings.Fields(line)
if len(fields) < 3 {
continue
}
source := fields[0]
mountPoint := fields[1]
fsType := fields[2]
isSystemEntry := systemMounts[mountPoint] || fsType == "swap"
if !isSystemEntry {
continue
}
if strings.HasPrefix(source, "UUID=") {
systemUUIDs = append(systemUUIDs, strings.TrimPrefix(source, "UUID="))
} else if strings.HasPrefix(source, "/dev/") {
systemDevices = append(systemDevices, source)
} }
} }
}()
``` // Step 2: Resolve UUIDs to device paths via blkid
for _, uuid := range systemUUIDs {
**Note:** The existing `go func()` that runs `alertMgr.Refresh()` at startup already does a health check but doesn't ping Healthchecks or hub. Merge this logic or add alongside it. The 5-second delay gives Docker, metrics, and backup subsystems time to initialize. out, err := exec.Command("blkid", "-U", uuid).Output()
if err == nil {
### 0.3 Reuse existing pusher instance devPath := strings.TrimSpace(string(out)) // e.g., "/dev/sda2"
systemDevices = append(systemDevices, devPath)
The hub-report scheduler task already creates a `pusher` — consider creating it once at init and reusing: }
}
```go
var pusher *report.Pusher // Step 3: Extract parent disk names from device paths
if cfg.Hub.Enabled && cfg.Hub.URL != "" { for _, devPath := range systemDevices {
pusher = report.NewPusher(&cfg.Hub, logger) diskName := partitionToParentDisk(devPath)
sched.Every("hub-report", pushInterval, func(ctx context.Context) error { if diskName != "" {
r := report.BuildReport(...) systemDisks[diskName] = true
return pusher.Push(r) }
}) }
return systemDisks
} }
// ... then in startup goroutine:
if pusher != nil { // partitionToParentDisk extracts parent disk name from a partition device path.
// use same pusher // "/dev/sda2" → "sda", "/dev/nvme0n1p2" → "nvme0n1"
func partitionToParentDisk(devPath string) string {
name := filepath.Base(devPath) // "sda2"
// NVMe: nvme0n1p2 → nvme0n1
if strings.Contains(name, "nvme") {
if idx := strings.LastIndex(name, "p"); idx > 0 {
candidate := name[:idx]
// Verify it's actually a partition number after 'p'
if _, err := strconv.Atoi(name[idx+1:]); err == nil {
return candidate
}
}
return name
}
// Standard: sda2 → sda, sdb1 → sdb
return strings.TrimRight(name, "0123456789")
} }
``` ```
--- Then in `ScanDisks()`, replace the mount-point-based detection:
## 1. Storage Initialization Feature
### 1.1 Concept
The controller UI provides a guided wizard to:
1. **Scan** for block devices that are not mounted (like `sdb` in the demo system)
2. **Show** disk info: size, model, existing partitions/filesystems
3. **Format** with ext4 after explicit confirmation with safety warnings
4. **Mount** at a user-specified path under `/mnt/`
5. **Register** automatically as a storage path in settings.json
This replaces the need to SSH in and run `hdd-setup.sh` manually. The existing `scripts/hdd-setup.sh` in the repo provides the proven logic — the controller wraps this as a web-based flow.
### 1.2 Architecture: Go package, not bash wrapper
Implement in Go directly (not shelling out to `hdd-setup.sh`) for:
- Structured error handling and progress feedback
- Testability
- No dependency on bash script being present in the container
Use `exec.Command` for: `lsblk`, `blkid`, `mkfs.ext4`, `mount`, and fstab editing.
### 1.3 New package: `internal/storage/`
```go ```go
package storage
// BlockDevice represents a detected physical disk.
type BlockDevice struct {
Name string // "sdb"
Path string // "/dev/sdb"
Size string // "931.5G"
SizeBytes int64
Model string // "WD Elements 25A2"
Serial string // "WX..." (if available)
Type string // "disk"
Removable bool // true for USB
Partitions []Partition
Mounted bool // any partition mounted
}
// Partition represents a partition on a block device.
type Partition struct {
Name string // "sdb1"
Path string // "/dev/sdb1"
Size string // "931.5G"
FSType string // "ext4", "" (no filesystem)
Label string // filesystem label
UUID string
MountPoint string // "" if not mounted
}
// ScanResult from disk detection.
type ScanResult struct {
AvailableDisks []BlockDevice // Unmounted disks/partitions
SystemDisks []BlockDevice // Mounted/system disks (for display only, not selectable)
}
// FormatRequest parameters for formatting a disk.
type FormatRequest struct {
DevicePath string // "/dev/sdb" (whole disk) or "/dev/sdb1" (partition)
MountName string // "hdd_1" → will mount at /mnt/hdd_1
Label string // Display label for storage path registry
CreatePartition bool // If true, wipe and create single partition first
}
// FormatProgress tracks the formatting/mounting progress.
type FormatProgress struct {
Step string // "partitioning", "formatting", "mounting", "fstab", "registering", "done", "error"
Message string // Human-readable status
Error string // Error message if Step == "error"
Percent int // 0-100
}
```
### 1.4 Core functions
```go
// ScanDisks detects all block devices and classifies them.
func ScanDisks() (*ScanResult, error) { func ScanDisks() (*ScanResult, error) {
// Run: lsblk -J -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT,MODEL,SERIAL,RM,PKNAME -b // ... lsblk parsing as before ...
// Parse JSON output
// Classify: system disk (has / or /boot mount) vs available // Get system disk names from host fstab
// A disk is "available" if NO partition is mounted AND it's not the system disk systemDiskNames := getSystemDiskNames()
}
for _, dev := range parsed.BlockDevices {
// FormatAndMount formats a partition and mounts it. if dev.Type != "disk" { continue }
// This is a long-running operation — use a channel for progress updates.
func FormatAndMount(req FormatRequest, progress chan<- FormatProgress) error { // ... build BlockDevice as before ...
// Step 1: Validate
// - Device exists // Check if this is a system disk (from fstab analysis)
// - Not already mounted isSystem := systemDiskNames[dev.Name]
// - Not system disk (compare with / device)
// - Mount name valid (alphanumeric + underscore, no spaces) // Also check if any partition is currently mounted (fallback safety)
// - Mount path doesn't already exist OR is empty dir anyMounted := false
for _, child := range dev.Children {
// Step 2: Partition (if CreatePartition) // ... as before ...
// - sfdisk: create single Linux partition filling whole disk if part.MountPoint != "" {
// - Wait for kernel to re-read partition table (partprobe) anyMounted = true
// - Update device path to new partition (e.g., /dev/sdb1) }
}
// Step 3: Format
// - mkfs.ext4 -L <label> <partition> bd.Mounted = anyMounted || isSystem
// - This is the slow step for large disks
if isSystem || anyMounted {
// Step 4: Mount result.SystemDisks = append(result.SystemDisks, bd)
// - mkdir -p /mnt/<mount_name> } else {
// - Get UUID via blkid result.AvailableDisks = append(result.AvailableDisks, bd)
// - Backup /etc/fstab to /etc/fstab.bak.YYYYMMDD }
// - Append fstab entry: UUID=<uuid> /mnt/<name> ext4 defaults,nofail,noatime 0 2 }
// - mount /mnt/<name> return result, nil
// - Verify mount succeeded
// Step 5: Set permissions
// - chown 1000:1000 /mnt/<name>
// - Create standard Felhom subdirs: storage/, Dokumentumok/
// Step 6: Register
// - Return mount path for caller to register in settings.json
} }
``` ```
### 1.5 UI: Wizard on Settings Page ## Bug 2: "nincs fájlrendszer" for all partitions
Add a new section on the Beállítások page (or as a separate page `/settings/storage/init`): ### Current broken logic
`lsblk` inside a container often returns `null` for `fstype` because it relies on udev/blkid cache that's incomplete in the container's environment.
**Step 1: Scan** ### Fix: Enrich with blkid
```
┌─────────────────────────────────────────────┐
│ Új meghajtó inicializálása │
│ │
│ [🔍 Meghajtók keresése] │
│ │
│ Talált meghajtók: │
│ ┌──────────────────────────────────────┐ │
│ │ ○ /dev/sdb — 931.5 GB │ │
│ │ WD Elements 25A2 │ │
│ │ 1 partíció (sdb1) — nincs fájlrendszer│ │
│ └──────────────────────────────────────┘ │
│ │
│ A rendszermeghajtó (sda) nem választható. │
└─────────────────────────────────────────────┘
```
**Step 2: Configure** After lsblk parsing, run `blkid` to get filesystem types for all partitions. `blkid` directly probes the device (works in privileged containers):
```
┌─────────────────────────────────────────────┐
│ Meghajtó: /dev/sdb (931.5 GB, WD Elements) │
│ │
│ Csatlakoztatási név: [hdd_1 ] │
│ (A meghajtó a /mnt/hdd_1 útvonalra kerül) │
│ │
│ Megnevezés: [Külső HDD 1TB ] │
│ │
│ ☑ Beállítás alapértelmezett adattárolóként │
│ │
│ ⚠️ FIGYELEM: A meghajtó ÖSSZES adata │
│ törlődik! Ez a művelet NEM vonható vissza. │
│ │
│ A folytatáshoz írja be: FORMÁZÁS │
│ [ ] │
│ │
│ [Inicializálás indítása] [Mégsem] │
└─────────────────────────────────────────────┘
```
**Step 3: Progress (live updates via polling)**
```
┌─────────────────────────────────────────────┐
│ Meghajtó inicializálás folyamatban... │
│ │
│ ✅ Partíció létrehozása │
│ ✅ Fájlrendszer formázása (ext4) │
│ ⏳ Csatlakoztatás (/mnt/hdd_1)... │
│ ○ Mappák létrehozása │
│ ○ Regisztráció │
│ │
│ ████████████████░░░░░░░░ 60% │
└─────────────────────────────────────────────┘
```
**Step 4: Done**
```
┌─────────────────────────────────────────────┐
│ ✅ Meghajtó sikeresen inicializálva! │
│ │
│ Útvonal: /mnt/hdd_1 │
│ Méret: 931.5 GB │
│ Fájlrendszer: ext4 │
│ Állapot: Alapértelmezett adattároló │
│ │
│ [Vissza a Beállításokhoz] │
└─────────────────────────────────────────────┘
```
### 1.6 Routes
| Method | Path | Auth? | Description |
|--------|------|-------|-------------|
| GET | `/settings/storage/init` | Yes | Storage init wizard page |
| POST | `/api/storage/scan` | Yes | Scan for available disks (JSON response) |
| POST | `/api/storage/init` | Yes | Start format+mount (returns job ID) |
| GET | `/api/storage/init/status` | Yes | Poll format progress (JSON) |
### 1.7 Safety guardrails
1. **Never touch system disk** — compare block device of target with block device of `/`. If same parent disk, hard reject.
2. **Confirmation typing** — user must type "FORMÁZÁS" (Hungarian for "FORMAT") before proceeding. Not a simple checkbox.
3. **Pre-flight checks:**
- Device still exists when init starts (not unplugged between scan and confirm)
- Device still not mounted (not mounted by something else in between)
- Mount path doesn't conflict with existing registered storage paths
- No duplicate mount names
4. **fstab safety:**
- Backup fstab before editing
- Use UUID-based mount (not device path — device names can change)
- Use `nofail` option (system boots even if drive missing)
- Use `noatime` option (reduces write wear)
5. **Container requirements:** The controller container needs:
- `/dev:/dev:ro` mount (to see block devices) — actually needs `:rw` for mkfs
- `/mnt:/mnt:rw` (already have this from v0.9.0)
- `--privileged` or specific `--cap-add` for disk operations
- Access to host's `/etc/fstab` — mount as `/host-fstab:/etc/fstab:rw` or similar
### 1.8 Container privilege escalation
**This is the biggest architectural question.** Currently the controller runs as a regular container with Docker socket and `/mnt:/mnt:rw`. Disk formatting requires:
- Access to `/dev/sdb*` (block devices)
- Permission to run `mkfs.ext4`, `sfdisk`, `mount`, `blkid`
- Write access to host's `/etc/fstab`
**Options:**
**Option A: `--privileged` controller container**
- Simplest. Full access to everything.
- Already has Docker socket (full host control anyway).
- Risk: Broader attack surface if container is compromised.
- For a managed home server where the controller IS the management layer, this is pragmatic.
**Option B: Specific capabilities**
- `--cap-add SYS_ADMIN --cap-add MKNOD`
- Mount `/dev:/dev:rw`, host's `/etc/fstab`
- More surgical but still significant privilege.
**Option C: Helper script on host**
- Controller calls a script on the host via Docker exec into a sidecar or host process.
- Complex, fragile. Not recommended.
**Recommended: Option A (`--privileged`)** — the controller already has Docker socket which is equivalent to root anyway. Adding `--privileged` just makes the disk operations possible without workaround complexity. Update the controller's docker-compose.yml.
### 1.9 Required container compose changes
```yaml
services:
felhom-controller:
# ... existing ...
privileged: true # NEW: for disk operations
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /opt/docker:/opt/docker
- /mnt:/mnt:rw
- /dev:/dev # NEW: block device access
- /etc/fstab:/host-fstab # NEW: fstab read/write
- /run/udev:/run/udev:ro # NEW: for blkid/lsblk metadata
```
Note: Mount host fstab at `/host-fstab` inside the container to avoid shadowing the container's own `/etc/fstab`. The Go code reads/writes `/host-fstab`.
### 1.10 Prerequisite packages in controller Docker image
The controller's Dockerfile needs: `lsblk`, `blkid`, `mkfs.ext4`, `sfdisk`, `mount`, `partprobe`
These come from: `util-linux`, `e2fsprogs`, `parted` (or `fdisk`). Add to the Dockerfile's `apt-get install` if not already present.
Alternatively, the controller can shell out to the HOST's tools via `nsenter` if running with `--pid=host`. This avoids adding packages to the container image. But `--privileged` + installing the tools is simpler.
### 1.11 After successful init: auto-register
After format+mount succeeds:
```go
sett.AddStoragePath(settings.StoragePath{
Path: "/mnt/" + req.MountName,
Label: req.Label,
IsDefault: req.SetDefault,
Schedulable: true,
AddedAt: time.Now().UTC().Format(time.RFC3339),
})
```
This makes the new drive immediately available for app deployment.
---
## 2. Data Migration Feature
### 2.1 Concept
Per-app "Mozgatás" button to move an app's user data from one storage path to another. This is essential when:
- Adding a larger drive and wanting to move photos/documents to it
- A drive is filling up
- Reorganizing storage
### 2.2 Migration flow
```
User clicks "Mozgatás" on app detail or settings page
Select target storage path from dropdown
Pre-flight: check free space, estimate size, show warnings
Confirm (app will be stopped during migration)
Stop app containers
rsync data (with progress reporting)
Update app.yaml HDD_PATH to new path
Start app containers
Success: old data kept as safety net (user can delete later)
```
### 2.3 UI: Migration dialog
Accessible from two places:
- **Settings page**: per-app row in storage path details → "Mozgatás" button
- **App detail page** (deploy page in read-only mode): storage info section → "Mozgatás" button
Dialog/page:
```
┌─────────────────────────────────────────────────┐
│ Immich adatok áthelyezése │
│ │
│ Jelenlegi tárhely: Külső tárhely (hdd_placeholder) │
│ Adatméret: 92 MB (/mnt/hdd_placeholder/storage/immich) │
│ │
│ Cél tárhely: │
│ [▼ Külső HDD 1TB (/mnt/hdd_1) — 800 GB szabad] │
│ │
│ ⚠️ Figyelmeztetések: │
│ • Az alkalmazás a mozgatás idejére leáll │
│ • Nagy adatmennyiségnél ez percekig tarthat │
│ • A régi adatok megmaradnak biztonsági másolatként │
│ │
│ [Mozgatás indítása] [Mégsem] │
└─────────────────────────────────────────────────┘
```
Progress:
```
┌─────────────────────────────────────────────────┐
│ Adatok áthelyezése... │
│ │
│ ✅ Alkalmazás leállítva │
│ ⏳ Adatok másolása... 45 MB / 92 MB │
│ ○ Konfiguráció frissítése │
│ ○ Alkalmazás indítása │
│ │
│ ████████████████░░░░░░░░ 49% │
│ ~30 másodperc hátravan │
└─────────────────────────────────────────────────┘
```
### 2.4 Implementation: `internal/storage/migrate.go`
```go ```go
// MigrateRequest parameters. // enrichWithBlkid fills in missing FSType, UUID, and Label from blkid.
type MigrateRequest struct { func enrichWithBlkid(disks []BlockDevice) {
StackName string // e.g., "immich" // Run blkid once for all devices
TargetPath string // e.g., "/mnt/hdd_1" out, err := exec.Command("blkid", "-o", "export").Output()
if err != nil {
return // Best-effort; lsblk data still usable
}
// Parse blkid output — blocks separated by blank lines:
// DEVNAME=/dev/sda1
// UUID=XXXX-YYYY
// TYPE=vfat
// ...
blkidMap := parseBlkidExport(out)
for i := range disks {
for j := range disks[i].Partitions {
p := &disks[i].Partitions[j]
if info, ok := blkidMap[p.Path]; ok {
if p.FSType == "" {
p.FSType = info.FSType
}
if p.UUID == "" {
p.UUID = info.UUID
}
if p.Label == "" {
p.Label = info.Label
}
}
}
}
} }
// MigrateProgress tracks migration state. type blkidInfo struct {
type MigrateProgress struct { FSType string
Step string // "stopping", "copying", "updating", "starting", "done", "error", "rolling_back" UUID string
Message string Label string
BytesCopied int64
BytesTotal int64
Percent int
Error string
ElapsedSeconds int
} }
// MigrateAppData moves an app's data from current to target storage path. func parseBlkidExport(data []byte) map[string]blkidInfo {
func MigrateAppData(req MigrateRequest, stackMgr *stacks.Manager, sett *settings.Settings, progress chan<- MigrateProgress) error { result := map[string]blkidInfo{}
// 1. Load app config, get current HDD_PATH
// 2. Parse compose to find HDD mount mappings (e.g., ${HDD_PATH}/storage/immich) blocks := strings.Split(string(data), "\n\n")
// 3. Validate target path: exists, mounted, sufficient space, different from current for _, block := range blocks {
// 4. Stop app: docker compose -f ... stop var devName string
// 5. rsync each HDD mount: info := blkidInfo{}
// rsync -avP /mnt/hdd_placeholder/storage/immich/ /mnt/hdd_1/storage/immich/ for _, line := range strings.Split(strings.TrimSpace(block), "\n") {
// Parse rsync output for progress (--info=progress2) parts := strings.SplitN(line, "=", 2)
// 6. Update app.yaml: change HDD_PATH to target path if len(parts) != 2 { continue }
// 7. Start app: docker compose -f ... up -d key, val := parts[0], parts[1]
// 8. Verify containers running switch key {
// 9. Log migration: "Moved immich from /mnt/hdd_placeholder to /mnt/hdd_1" case "DEVNAME":
// 10. Keep old data (don't delete — user cleanup later) devName = val
case "TYPE":
info.FSType = val
case "UUID":
info.UUID = val
case "LABEL":
info.Label = val
}
}
if devName != "" {
result[devName] = info
}
}
return result
} }
``` ```
### 2.5 rsync progress parsing Call `enrichWithBlkid()` at the end of `ScanDisks()` on both `AvailableDisks` and `SystemDisks`.
Use `rsync --info=progress2` which outputs lines like: ## UI impact
```
45,678,901 49% 12.34MB/s 0:00:30
```
Parse with regex: After these fixes:
```go - sda will appear in `SystemDisks` (shown grayed out with "Rendszermeghajtó" label, or hidden entirely)
progressRe := regexp.MustCompile(`([\d,]+)\s+(\d+)%\s+([\d.]+\w+/s)\s+(\d+:\d+:\d+)`) - sdb will be the only entry in `AvailableDisks`
``` - sda partitions will show: sda1 (vfat, /boot/efi), sda2 (ext4, /), sda3 (swap)
- sdb1 will correctly show "(nincs fájlrendszer)" since it genuinely has none
Feed into `MigrateProgress.Percent` and `MigrateProgress.BytesCopied`. ## Template update
### 2.6 Rollback on failure If SystemDisks are currently shown alongside AvailableDisks (both selectable), the template should either:
- **Option A:** Hide system disks entirely — simpler, less confusion
- **Option B:** Show them grayed out with a "Rendszermeghajtó — nem választható" badge
If rsync fails or app doesn't start after migration: Recommended: **Option A** — only show AvailableDisks. The user doesn't need to see sda at all.
1. Revert `app.yaml` HDD_PATH to original value
2. Start app with original config
3. Log error but keep partially copied data (don't delete)
4. Show error to user: "Áthelyezés sikertelen. Az alkalmazás az eredeti tárolóról fut. A részlegesen másolt adatok: /mnt/hdd_1/storage/immich"
### 2.7 Routes ## Files modified
- `controller/internal/storage/scan_linux.go``getSystemDiskNames()`, `partitionToParentDisk()`, `enrichWithBlkid()`, `parseBlkidExport()`, updated `ScanDisks()`
| Method | Path | Auth? | Description | ## Testing
|--------|------|-------|-------------| 1. After fix: scan page shows only sdb (931.5 GB, HD710 PRO, sdb1 no filesystem)
| GET | `/stacks/{name}/migrate` | Yes | Migration page for app | 2. sda is no longer listed
| POST | `/api/storage/migrate` | Yes | Start migration (returns job ID) | 3. If you temporarily disconnect the USB HDD: scan shows "Nem található inicializálható meghajtó"
| GET | `/api/storage/migrate/status` | Yes | Poll migration progress (JSON) |
### 2.8 Mutex / concurrency
Only one migration or format operation at a time. Use a global mutex:
```go
var diskOpMutex sync.Mutex
var diskOpActive atomic.Bool
```
Check `diskOpActive` before starting. Return error if another operation is running. Show "Egy másik lemezművelet folyamatban van" message.
### 2.9 Old data cleanup
After successful migration, the old data remains. Options for the user:
- **Manual:** User can SSH in and delete, or use FileBrowser
- **UI button (future):** "Régi adatok törlése" button with confirmation — Phase D
For now, just leave old data and show a note on the app detail page: "Korábbi adatok a(z) /mnt/hdd_placeholder/storage/immich útvonalon továbbra is elérhetők."
---
## 3. Per-App Storage Display Enhancements
### 3.1 App detail page (deploy page read-only mode)
Currently shows deployed config fields as disabled inputs. Add a storage info section for deployed apps:
```html
{{if and .AlreadyDeployed .StorageInfo}}
<div class="deploy-storage-info">
<h4>Adattárolás</h4>
<div class="settings-grid">
<div class="settings-row">
<span class="settings-label">Tárhely</span>
<span class="settings-value">{{.StorageInfo.Label}} ({{.StorageInfo.Path}})</span>
</div>
<div class="settings-row">
<span class="settings-label">Adatméret</span>
<span class="settings-value mono">{{.StorageInfo.DataSizeHuman}}</span>
</div>
<div class="settings-row">
<span class="settings-label">Szabad hely</span>
<span class="settings-value mono">{{.StorageInfo.FreeHuman}} ({{printf "%.0f" .StorageInfo.FreePercent}}%)</span>
</div>
</div>
{{if gt (len $.OtherStoragePaths) 0}}
<a href="/stacks/{{.Meta.Slug}}/migrate" class="btn btn-sm btn-outline" style="margin-top:0.5rem">
📦 Mozgatás másik tárolóra
</a>
{{end}}
</div>
{{end}}
```
### 3.2 Data struct
```go
type DeployStorageInfo struct {
Path string
Label string
DataSizeHuman string
FreeHuman string
FreePercent float64
}
```
---
## 4. Implementation Steps
### Step 0: Startup ping + hub report
- Add startup goroutine in `main.go` after `sched.Start(ctx)`
- Fire heartbeat ping, system_health ping, and hub report with 5s delay
- Reuse existing `pinger` and create `pusher` once (not per-task)
- **Test:** Restart controller → Healthchecks shows fresh ping within 10 seconds. Hub shows fresh report.
### Step 1: Storage initialization — backend
- Create `internal/storage/` package with `BlockDevice`, `Partition`, `ScanResult` structs
- Implement `ScanDisks()` using `lsblk -J`
- Implement `FormatAndMount()` with progress channel
- Implement safety checks: system disk detection, device validation, mount conflict check
- Implement fstab editing (read from `/host-fstab`, backup, append, write)
- **Test:** Call `ScanDisks()` → returns sdb as available. FormatAndMount (dry-run or test partition).
### Step 2: Storage initialization — container setup
- Update controller's `docker-compose.yml`:
- Add `privileged: true`
- Add `/dev:/dev` volume
- Add `/etc/fstab:/host-fstab` volume
- Add `/run/udev:/run/udev:ro` volume
- Update controller Dockerfile: install `util-linux`, `e2fsprogs` (for `mkfs.ext4`, `lsblk`, `blkid`, `sfdisk`, `mount`)
- Update `docker-setup.sh` to generate compose with new mounts
- **Test:** `docker exec felhom-controller lsblk` shows host block devices. `mkfs.ext4 --version` works.
### Step 3: Storage initialization — UI
- Create `/settings/storage/init` page with wizard flow
- Implement `/api/storage/scan` endpoint (JSON)
- Implement `/api/storage/init` endpoint (start format, return job ID)
- Implement `/api/storage/init/status` endpoint (poll progress)
- JS: scan button → display disks → configure form → confirmation typing → progress polling → done
- Auto-register new storage path in settings.json on completion
- **Test:** Full wizard: scan → select sdb → name "hdd_1" → type "FORMÁZÁS" → format → mount → registered. `lsblk` shows `/mnt/hdd_1` mounted.
### Step 4: Data migration — backend
- Create `internal/storage/migrate.go`
- Implement `MigrateAppData()` with progress channel
- rsync with `--info=progress2` output parsing
- Rollback on failure (revert app.yaml, restart with original config)
- Global mutex for disk operations
- **Test:** Migrate Paperless-ngx from `/mnt/hdd_placeholder` to `/mnt/hdd_1` → data appears at new location, app restarts.
### Step 5: Data migration — UI
- Create `/stacks/{name}/migrate` page
- Target storage dropdown (exclude current path)
- Pre-flight: size estimate, free space check
- Progress polling via `/api/storage/migrate/status`
- Add "Mozgatás" button to app detail page (deploy page read-only mode) and settings storage details
- **Test:** Navigate to Immich detail → "Mozgatás" → select new drive → confirm → progress → done → Immich running from new path.
### Step 6: Per-app storage display on deploy page
- Add `DeployStorageInfo` to deploy page template data for deployed apps
- Show current storage path, data size, free space
- Show "Mozgatás" link if other storage paths exist
- **Test:** Visit deployed Immich page → shows "Adattárolás" section with path, size, free space, migrate button.
### Step 7: Version bump & cleanup
- Update CHANGELOG.md / CONTEXT.md / CLAUDE.md / README.md for controller
- Bump to 0.11.0
- Build + deploy
- **Full test cycle:**
1. Controller starts → Healthchecks shows ping within 10s ✓
2. Navigate to settings → "Új meghajtó inicializálása" → scan → sdb visible ✓
3. Configure hdd_1, type FORMÁZÁS → format + mount succeeds ✓
4. Settings shows /mnt/hdd_1 as registered storage path ✓
5. Deploy something (or use existing app) → migrate to /mnt/hdd_1 ✓
6. App running from new storage path ✓
---
## 5. Files to Create / Modify
### New files:
- `controller/internal/storage/scan.go``ScanDisks()`, `BlockDevice`, `Partition` structs
- `controller/internal/storage/format.go``FormatAndMount()`, `FormatRequest`, `FormatProgress`
- `controller/internal/storage/migrate.go``MigrateAppData()`, `MigrateRequest`, `MigrateProgress`
- `controller/internal/storage/safety.go` — System disk detection, mount conflict check, fstab helpers
- `controller/internal/web/templates/storage_init.html` — Storage init wizard page
- `controller/internal/web/templates/migrate.html` — Migration page
### Modified files:
- `controller/cmd/controller/main.go` — Startup pings + hub report, reuse pusher instance
- `controller/internal/web/handlers.go` — Storage init handlers, migration handlers, deploy page storage info
- `controller/internal/web/server.go` — Register new routes
- `controller/internal/web/templates/deploy.html` — Storage info section for deployed apps, "Mozgatás" link
- `controller/internal/web/templates/settings.html` — "Új meghajtó inicializálása" link/button, "Mozgatás" buttons per app
- `controller/internal/web/templates/style.css` — Wizard styles, progress bars, migration UI
- `controller/docker-compose.yml``privileged: true`, new volume mounts
- `controller/Dockerfile` — Install `util-linux`, `e2fsprogs`
- `scripts/docker-setup.sh` — Updated compose generation
---
## 6. Design Decisions
### Why Go package instead of wrapping hdd-setup.sh?
The bash script has interactive prompts, color codes, and reads stdin — none of which work from a web handler. Reimplementing the core operations in Go gives structured errors, progress channels, and clean API integration. The bash script remains useful for manual SSH access.
### Why `--privileged` instead of fine-grained capabilities?
The controller already has Docker socket access (root-equivalent). Adding `--privileged` adds no meaningful additional risk but greatly simplifies disk operations. Fine-grained caps (`SYS_ADMIN`, `MKNOD`, device cgroup rules) are complex to get right and test.
### Why keep old data after migration?
Deleting data is irreversible. The old copy serves as an implicit backup during the transition period. Users can clean up manually when they're confident the migration worked. A future "cleanup" button can be added.
### Why rsync instead of cp/mv?
rsync has progress reporting (`--info=progress2`), handles interruptions gracefully (can resume), preserves permissions/timestamps, and works across filesystems. `mv` can't cross filesystem boundaries (would fall back to copy+delete). `cp -a` works but has no progress output.
### Why a separate `/settings/storage/init` page instead of inline?
The format wizard has multiple steps and safety confirmations that don't fit well inside the existing settings page layout. A dedicated page with clear back-navigation keeps the settings page clean.
### Why UUID-based fstab instead of device path?
Device paths (`/dev/sdb1`) can change between boots (especially with USB devices). UUID-based mounting is stable regardless of device enumeration order. This is standard best practice for Linux.
### Why mount under `/mnt/`?
Standard convention for additional mount points on Linux. The controller already mounts `/mnt:/mnt:rw`. Keeps all storage in a predictable, manageable location.
### Future (NOT in scope):
- **Old data cleanup button** — "Régi adatok törlése" after successful migration
- **SMART health monitoring** — smartmontools integration
- **Auto-detect new drives** — udev event watching
- **Batch migration** — move multiple apps at once
- **Filesystem check**`fsck` integration for maintenance