# felhom-controller **Central management container for Felhom home servers.** The **in-guest controller**: one per customer LXC, Docker-only, Hungarian web dashboard for managing the customer's app stacks, app-data backups, monitoring and notifications. All Proxmox/disk operations are delegated to the host agent (`felhom-agent`). **Current version: v0.59.0** > ## Documentation has moved (and is now code-verified) > > The **authoritative architecture & feature documentation** now lives in the central docs home: > **[`felhom.eu/documentation/controller/`](../../felhom.eu/documentation/controller/README.md)** — > module map, deploy & stack lifecycle, backup architecture, storage/monitoring/metrics, and > auth/hub/sync/integrations. Those docs are grounded in current source (v0.59.0). > > **Quick build & deploy** is in the "Build & Deploy" section below and in the repo `CLAUDE.md` > (authoritative for the workflow). NOTE the demo controller runs in an LXC guest (9201) under the > **bootstrap-managed** mechanism (`/etc/felhom-controller-image` + `felhom-controller-bootstrap.service`), > not the bare-metal `/opt/docker` compose path some sections below still describe. > > The prose below this banner is **retained legacy reference** and may lag the central docs — when they > disagree, the central docs win. (Some sections still mention restic / pre-8C disk handling that has > since moved to the host agent.) --- ## Table of Contents - [Architecture](#architecture) - [Features](#features) - [App Management](#1-app-management) - [App Export/Import](#2-app-exportimport-fab-bundles) - [Backup System](#3-backup-system) - [Storage Management](#4-storage-management) - [Monitoring & Health](#5-monitoring--health) - [Notifications](#6-notifications) - [Update Management](#7-update-management) - [Authentication & Settings](#8-authentication--settings) - [Central Hub](#9-central-hub-reporting) - [Setup Wizard](#10-first-run-setup-wizard) - [Disaster Recovery](#11-disaster-recovery) - [Asset Sync](#12-asset-sync) - [Debug Mode](#13-debug-mode) - [Geo-Restriction](#14-geo-restriction) - [App-to-App Integrations](#15-app-to-app-integrations) - [Repository Layout](#repository-layout) - [Configuration](#configuration) - [REST API](#rest-api) - [Build & Deploy](#build--deploy) - [Roadmap](#roadmap) --- ## Architecture ``` ┌─────────────────────────────────────────────────────────────────┐ │ Customer Hardware (N100 mini PC / Raspberry Pi) │ │ │ │ ┌──────────┐ ┌────────────────────────────────────────────┐ │ │ │ Traefik │ │ felhom-controller (privileged container) │ │ │ │ (reverse │──▶│ │ │ │ │ proxy) │ │ ┌──────────┐ ┌─────────────────────────┐│ │ │ └──────────┘ │ │ Web UI │ │ Stack Manager ││ │ │ │ │ (HU dash │ │ (compose ops, git sync, ││ │ │ ┌──────────┐ │ │ board) │ │ deploy, delete, update) ││ │ │ │cloudflared│ │ └──────────┘ └─────────────────────────┘│ │ │ │ (tunnel) │ │ ┌──────────┐ ┌─────────────────────────┐│ │ │ └──────────┘ │ │ Backup │ │ Storage Manager ││ │ │ │ │ (3-layer │ │ (disk scan, format, ││ │ │ ┌──────────┐ │ │ restic) │ │ mount, migrate) ││ │ │ │ App │ │ └──────────┘ └─────────────────────────┘│ │ │ │ stacks │ │ ┌──────────┐ ┌─────────────────────────┐│ │ │ │ (docker │ │ │Scheduler │ │ Monitor & Metrics ││ │ │ │ compose) │ │ │(cron-like│ │ (health, SQLite ││ │ │ └──────────┘ │ │ jobs) │ │ time-series, Chart.js) ││ │ │ │ └──────────┘ └─────────────────────────┘│ │ │ │ ┌──────────┐ ┌─────────────────────────┐│ │ │ │ │ Notify │ │ REST API + Hub Reporter ││ │ │ │ │ (events) │ │ (JSON push + events) ││ │ │ │ └──────────┘ └─────────────────────────┘│ │ │ │ ┌──────────┐ │ │ │ │ │ Assets │ │ │ │ │ │ (Hub │ │ │ │ │ │ sync) │ │ │ │ │ └──────────┘ │ │ │ └────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────┘ │ events + reports │ git pull │ asset sync ▼ ▼ ▼ hub.felhom.eu gitea.dooplex.hu hub.felhom.eu (central dashboard) (stack definitions) (logos, screenshots) ``` ### Key Architecture Decisions - **Pure Go, no frameworks** — stdlib `net/http` + `html/template`. Only external deps: `bcrypt`, `yaml.v3`, `modernc.org/sqlite` (pure Go, no CGO). - **Privileged container** — Required for disk operations (format, mount, fstab), `/dev` access, and Docker socket control. - **`/host-dev` indirection** — Docker overrides `/dev` with a tmpfs. The host's `/dev` is mounted at `/host-dev` to access block devices. - **`StackDataProvider` interface** — Breaks circular import between the backup packages and stacks. Defined in `internal/appbackup` (and re-exposed via a type alias in `internal/backup`). Implemented by `stackAdapter` in `main.go`. Provides `GetStackHDDPath()` for per-drive backup routing. - **Atomic file writes** — All persistent state (`settings.json`, `app.yaml`) written to `.tmp` then `os.Rename` for crash safety. - **`go:embed` templates** — All HTML/CSS/JS compiled into the binary. No runtime file dependencies. - **Europe/Budapest timezone** — All scheduled jobs, timestamps, and UI labels use Hungarian timezone. ### Module Map | Module | Path | Responsibility | |--------|------|----------------| | **Config** | `internal/config/` | YAML loader, validation, `FELHOM_*` env overrides | | **Settings** | `internal/settings/` | Runtime-mutable `settings.json` (passwords, backup prefs, storage paths, notifications) | | **Stacks** | `internal/stacks/` | Compose operations, scanning, `.felhom.yml` metadata, deploy/delete flow; **base-infra bring-up** (`infra.go` — `EnsureBaseStack`) | | **Infra** | `internal/infra/` | Pure renderers (embedded `text/template`) for the base-infra stacks (traefik/cloudflared/filebrowser); **pinned image tags as the single source of truth** (web filebrowser sync delegates here) | | **Crypto** | `internal/crypto/` | AES-256-GCM encryption for sensitive app.yaml values (passwords, secrets), key management | | **Sync** | `internal/sync/` | Git-based app catalog sync (clone/pull, content-hash copy) | | **AppBackup** | `internal/appbackup/` | Self-contained app-data backup primitives: DB dump discovery/execution (`DiscoverDatabases`, `DumpOne`), Docker-volume/app-data discovery (`StackDataProvider`, `DiscoverAppData`), keep-side path helpers (`AppDBDumpPath`, `AppVolumeDumpPath`, `AppDataDir`). `DiscoverDatabases` takes the deployed-stack set so a DB container maps to the right stack even when a slug ends in a DB-role token (M19, v0.62.0). `ListDumpFiles` takes an optional `cached(name,size,mod)` lookup so an unchanged dump isn't re-validated (line-scan) every ~5-min cycle (M18, v0.62.0). No dependency on restic/cross-drive/drive-mount. Imported directly by `appexport` and `storage`. | | **Backup** | `internal/backup/` | Per-drive 3-layer backup: DB dumps → restic snapshots → cross-drive copies, restore. Re-exposes the `appbackup` primitives via aliases/forwarders (`appbackup_bridge.go`) for the disk/host-side code and the web/api/report consumers. | | **Storage** | `internal/storage/` | Disk scanning (`lsblk`), partitioning (`sfdisk`), formatting (`mkfs.ext4`), mounting, data migration (`rsync`) | | **System** | `internal/system/` | System info (`/proc`), CPU collector, mount points, disk usage, FS info | | **Monitor** | `internal/monitor/` | System health checks, storage watchdog, legacy Healthchecks pinger (deprecated) | | **Metrics** | `internal/metrics/` | SQLite time-series store, system + container metric collection | | **Scheduler** | `internal/scheduler/` | Central job scheduler (periodic + daily, skip-if-running, panic recovery) | | **SelfUpdate** | `internal/selfupdate/` | Version checking (registry), update trigger, state persistence, startup verification | | **Notify** | `internal/notify/` | Email notifications via hub relay, preference sync, per-event cooldowns | | **Report** | `internal/report/` | Hub report builder + HTTP pusher (system, stacks, backup, health) | | **Assets** | `internal/assets/` | Hub-managed asset syncer: downloads logos/screenshots with SHA-256 change detection | | **SelfTest** | `internal/selftest/` | Startup self-test: 9 diagnostic checks (Docker, dirs, storage, hub, restic, metrics) | | **Util** | `internal/util/` | Shared utilities: `TruncateStr` for debug log output truncation | | **AppExport** | `internal/appexport/` | Per-app export/import via `.fab` bundles (config + DB + user data), optional AES-256 encryption | | **API** | `internal/api/` | REST JSON endpoints, diagnostic dump (`/api/debug/dump`) | | **Web** | `internal/web/` | Hungarian dashboard, auth, page handlers, template functions, alerts | --- ## Features ### 1. App Management The controller manages Docker Compose stacks through a complete lifecycle: catalog sync, first-time deployment, runtime operations, and deletion. #### Git Sync (`internal/sync/`) The app catalog lives in a separate Git repository. The controller: - Shallow-clones the catalog on startup - Periodically fetches updates (configurable, default 15 min) - Copies only `docker-compose.yml` and `.felhom.yml` to the stacks directory - **Never overwrites** `app.yaml` (user secrets are safe) - Uses SHA-256 content hashing — only writes files that actually changed - Triggers stack rescan after sync so the dashboard updates immediately - **Post-sync hook**: auto-injects missing deploy fields (new secrets, domains) into existing `app.yaml` for stacks whose templates were updated (see Missing Field Injection below) - Manual sync via "Sablonok frissitese" button or `POST /api/sync` #### First-Time Deploy Flow 1. Customer sees app card with "Telepites" button 2. Deploy page pre-generates and **displays** all auto-values before the user clicks deploy: - `domain` fields: shown as readonly text input with the customer's configured base domain - `subdomain` fields: editable text input pre-filled with the default from `.felhom.yml`, shown with `.base-domain` suffix. Validated for DNS-safe format, reserved names, and uniqueness across deployed stacks. Locked after deploy — changing requires Remove + Redeploy - `secret` fields: pre-generated and shown as masked password inputs with a "Megjelenítés" reveal button — user can see/copy all DB passwords and keys before deploying - User-configurable inputs (admin password, language, storage path) remain editable - Section header prompts the user to note down any passwords they need 3. `checkBeforeDeploy()` JS guard fetches live state first (prevents double-deploy from another tab) 4. **Memory validation** (F1, v0.61.0): the controller runs as a Docker container inside an LXC, where `/proc/meminfo` shows the **Proxmox host's** RAM (no lxcfs in the container) and the container's own cgroup is unlimited (the guest cap lives on the LXC ancestor). So the guest cap is read from the **Docker daemon** (`system.GuestMemTotalMB()` → `docker info` MemTotal — the daemon runs in the LXC and reports the guest's lxcfs-backed RAM; the cgroup limit is preferred when present, e.g. non-nested): - `usable_memory = guest_cap - reserved_memory_mb` (default 384MB reserved) - Hard block if `committed_used + new_request > usable_memory`, where `committed_used` = `CommittedMemory()` (sum of running apps' mem requests) — the guest-wide RSS is not observable from the container, so the controller's own committed accounting is the accurate, cheap "used". - `/api/system/info` reports the guest cap as total and committed memory as used. 4b. **Docker-data volume reserved-buffer gate (v0.58.0, storage-split prevention layer):** the OS rootfs and Docker data are split onto separate volumes; infra (controller/traefik/cloudflared/filebrowser) shares the one Docker data-root (`/var/lib/docker`) and is protected by **prevention, not placement**. `system.GetDockerVolumeHeadroom()` measures the Docker-data volume via `statfs("/")` (the controller container's overlay root is the upperdir on that volume — true with the golden's **overlay2** driver) and reserves `max(5 GB, 10%)`. `deployStack` **refuses a new deploy (HTTP 507)** when free space is at/under the buffer; the deploy page shows the warning + disables the button. Fail-open on a statfs error. The runtime disk monitor (`healthcheck.go`, warn 80% / crit 90%) watches the same volume and trips above the buffer. (Assumes the split guest's large data volume; the golden bakes overlay2 + log rotation so images+volumes live on the data volume, not `/var/lib/containerd`.) 5. Pre-generated secret values are submitted as hidden form inputs so the **same values** the user saw are saved to `app.yaml` (no silent re-generation on submit). Controller saves `app.yaml`, sets in-memory `Deployed` + `Deploying` flags, then runs `docker compose up -d` **asynchronously** in a goroutine — API returns immediately so the UI switches to the progress panel without waiting for image pulls. On failure the goroutine reverts both disk and in-memory state and sets `DeployError`. 6. 3-step progress panel polls `GET /api/stacks/{name}` every 3s: config saved → `deploying` (pulling images) → containers starting → health check passed. New `StateDeploying` state shown while compose-up is in progress (no containers yet). 7. Post-deploy: locked fields (DB_PASSWORD, etc.) become read-only; the "Automatikusan generált értékek" section continues to show the saved values on the settings page 8. The deploy/settings page includes **start/stop/restart** buttons for deployed apps, plus a "Megnyitás ↗" link to the app's subdomain URL (only visible when running) #### Catch-All Page for Stopped Apps When a user visits a stopped or undeployed app's subdomain (e.g., `travel.demo-felhom.eu`), the controller serves a branded error page instead of Traefik's raw 404: - **Traefik catch-all router**: The controller's `docker-compose.yml` registers a second router (`catchall`) with `priority=1` (lowest) and `HostRegexp(.+)`. Running apps always win; only requests with no matching container reach the controller. - **`CatchAllMiddleware`** in `server.go` intercepts requests where `Host` ≠ `felhom.DOMAIN`, serves the catch-all page **without auth** (user has no session on the app subdomain). - **`findStackBySubdomain()`** identifies the app by matching the subdomain against deployed `app.yaml` `SUBDOMAIN` env or metadata fallback. - **`catchall.html`** — standalone template (no layout, inline CSS) showing the app name, status ("leállítva" / "nincs telepítve" / "nem található"), and links to the controller dashboard or the app's detail page. - **Subdomain links** on the Alkalmazások page are only shown for deployed apps (non-deployed apps have no guaranteed subdomain yet). #### Dashboard "Megnyitás" Button Running apps on the Vezérlőpult now show a "Megnyitás ↗" button that opens the app's subdomain in a new tab. The `Subdomains` map is built in `dashboardHandler` from `app.yaml` env or metadata fallback. #### App Info Pages Each app can define rich metadata in `.felhom.yml`: - `app_info`: tagline, use_cases, first_steps, prerequisites, default_creds, docs_url - `optional_config`: groups of post-deploy configurable env vars (e.g., API keys for metadata providers) - `resources`: mem_request, mem_limit, pi_compatible, needs_hdd, hungarian_ui The `/apps/{slug}` page renders hero section, screenshots, setup guide, and optional config form. #### Stack Operations | Operation | What it does | |-----------|-------------| | Start | `docker compose up -d` — pre-start memory check rejects with 409 if insufficient RAM | | Stop | `docker compose stop` (blocked for protected stacks) | | Restart | `docker compose restart` | | Update | `docker compose pull` + `docker compose up -d` | | Remove | `docker compose down --volumes` + remove `app.yaml` + optional HDD/backup cleanup; template preserved for redeploy | | Delete | `docker compose down --rmi local --volumes` + optional HDD data cleanup (orphaned stacks only) | **Remove vs Delete**: "Eltávolítás" (Remove) is for deployed catalog stacks — it reverts the stack to "Nincs telepítve" state while keeping the template for easy redeployment. "Törlés" (Delete) is for orphaned stacks — it removes the entire stack directory including templates. Both require stopping the stack first. **Remove modal** shows three sections: (1) always-removed items (Docker volumes, app.yaml, cross-drive schedule), (2) optional HDD data deletion with reimport warning, (3) optional backup data deletion (DB dumps + cross-drive rsync) with restic retention note. **Protected stacks** (traefik, cloudflared, felhom-controller) cannot be stopped, removed, or deleted from the UI. Restart is allowed. **Orphan detection**: Deployed stacks with no matching catalog template are marked as orphaned with an "Elavult" badge and can be safely deleted. #### Base-infrastructure bring-up (`stacks/infra.go` + `internal/infra/`, v0.41.0) The controller stands up its own base stack — **traefik** (reverse proxy), **cloudflared** (external tunnel), **filebrowser** — instead of relying on the bare-metal `scripts/docker-setup.sh` (which a Proxmox-provisioned guest never runs). `internal/infra` renders the compose + config files from `controller.yaml` via embedded `text/template`s (lifted from `docker-setup.sh`); image tags are **pinned constants there** (`TraefikImage`/`CloudflaredImage`/`FileBrowserImage`) and the web FileBrowser sync path delegates to the same renderers, so the pinned versions can never diverge. `Manager.EnsureBaseStack()` creates the `traefik-public` network, then deploys traefik → cloudflared → filebrowser under `${stacks_dir}/`. It is: - **single-flight** (a `TryLock` guard — it's called from both first boot and every health tick, so overlapping runs must not race on the same stack dir), - **idempotent** (skips a stack whose container is already running; never overwrites an existing filebrowser compose, preserving the storage mounts `SyncFileBrowserMounts` manages), - **non-fatal** (logs, never crashes the controller). cloudflared is only deployed when a tunnel token is configured. **Triggers**: a first-boot goroutine (after stack init) and an unconditional call on every `system-health` tick (self-heal — cheap when healthy thanks to the idempotency). `monitor.EffectiveProtected` mirrors the cloudflared condition so a LAN-only node (no tunnel token) doesn't report a perpetual "protected container not running" FAIL. > **Mount prerequisite (Section-G):** the controller writes these stacks under `/opt/docker/stacks` *inside its container*, but `docker compose up` runs on the **guest** Docker daemon. The golden's controller-bootstrap (`felhom-agent` `build-golden.sh`) therefore bind-mounts that path **same-path** (`-v /opt/docker/stacks:/opt/docker/stacks`) so the daemon resolves every relative bind source — without it, all bind-mounted stacks (base infra and customer apps) silently break. **Controller routing + the wildcard cert anchor (`wireController` → `RenderControllerRoute`, v0.41.1 / v0.42.1).** filebrowser self-registers with traefik via Docker labels + `traefik-public` membership baked into its compose; the controller can't (it's started by the golden bootstrap *before* `traefik-public` exists, and the v2 `bootstrap.json` carries no domain — that comes from the hub pull). So `EnsureBaseStack` wires the controller **post-pull**: it `docker network connect traefik-public felhom-controller` and writes a traefik file-provider route `dynamic/controller.yml` (`Host(felhom.) → http://felhom-controller:8080`, write-if-changed). When DNS-01 ACME is configured, that route is **also the wildcard-cert anchor**: its router-level `tls.domains: *.` makes traefik **proactively obtain the wildcard `*.` + apex via Cloudflare DNS-01 at startup** (an entrypoint-level `http.tls.domains` does *not* trigger issuance in traefik v3 — only a router-level `tls.domains` does). Every other router then serves that one real wildcard cert by SNI — no per-app `certresolver` labels. This is what lets a LAN client reach the box directly at `*.` with the real cert (the `felhom-agent` split-horizon resolver depends on it). #### Missing Field Injection (`deploy.go`) When app templates are updated (e.g., a new `APP_KEY` secret is added to `.felhom.yml`), existing deployed apps need the new field in their `app.yaml`. The controller handles this automatically: - **On startup**: `InjectMissingFields()` runs for all deployed stacks - **After sync**: the post-sync hook runs for stacks whose templates were updated - For each deployed stack, compares `.felhom.yml` `deploy_fields` against `app.yaml` env vars - Missing `secret` fields: auto-generated using the field's generator spec (`password:N`, `hex:N`, `base64key:N`) - Missing `domain` fields: filled with the customer's configured domain - Missing `subdomain` fields: filled with the field's default value or the `.felhom.yml` `subdomain:` metadata - Other field types (e.g., `text`, `select`): logged as warning for manual configuration - Locked fields are added to the locked list automatically **Generator types**: `password:N` (alphanumeric), `hex:N` (hex-encoded random bytes), `base64key:N` (`base64:` + N random bytes base64-encoded, for Laravel APP_KEY etc.), `static:VALUE` (literal value). #### Container State Display | State | Color | Label | Meaning | |-------|-------|-------|---------| | Running + healthy | Green | "Fut" | All containers running and healthy | | Running + starting | Orange | "Indulas..." | Healthcheck not yet passed | | Deploying | Orange | "Telepítés..." | Compose up in progress (image pull, container creation) | | Running + unhealthy | Yellow | "Nem egeszseges" | Docker or controller-side healthcheck failing | | Stopped/exited | Red | "Leallitva" | All containers stopped | | Restarting | Yellow | "Ujrainditas..." | Restart loop | | Not deployed | Gray | "Nincs telepitve" | Compose file exists, not deployed | **Route-unpublished indicator (F5, v0.61.0).** Traefik's Docker provider only publishes a route to a container that is healthy (or has no healthcheck), so an `unhealthy`/`restarting` deployed app returns a hard **404** at its URL even though the container is running. The `routeUnpublished` template helper (`funcmap.go`) drives a distinct "URL nem elérhető – útvonal nincs publikálva" indicator on the dashboard and stacks cards for such apps, so a dead URL isn't mistaken for a merely-degraded-but-reachable one. #### Controller-side Health Probes (`internal/stacks/healthprobe.go`) For apps that declare a `healthcheck:` section in `.felhom.yml`, the controller probes the container directly over the Docker network (both are on `traefik-public`). This complements Docker-level healthchecks and is the **only** health mechanism for distroless/scratch images that lack shell utilities. Three probe types are supported: - **`http`** — Any HTTP response (even 4xx/5xx) = service is alive. Only connection refused/timeout = unhealthy. - **`api`** — HTTP request with response validation (expected status code, body content). Fails if expectations aren't met. - **`tcp`** — Simple port reachability check via `net.Dial`. Multiple checks per app are supported (all must pass). The probe scheduler runs every 10 seconds; per-app intervals default to 5 minutes and are configurable via `healthcheck.interval` in `.felhom.yml`. Probe results are stored in `Stack.HealthProbe` and exposed via the API. Failed probes override the stack state to `StateUnhealthy`; the override clears automatically when the next probe passes. **Fast initial probing:** On start/restart, stale health probe results are cleared (so the stack doesn't immediately appear "unhealthy" from a previous result). Until the first healthy probe, the controller checks every 10 seconds instead of the normal 5-minute interval, giving fast feedback on whether the app came up successfully. --- ### 2. App Export/Import (.fab bundles) Per-app export creates a self-contained `.fab` file (tar.gz, optionally encrypted) that can be stored externally or used to restore the app on the same server. Distinct from the automatic backup system — user-initiated, per-app, produces a single portable file. **Bundle contents:** `manifest.json` + `config/` (compose, .felhom.yml, app.yaml with plaintext secrets) + `database/` (gzipped SQL dump) + `data/` (HDD bind mount tars or Docker named volume tars). **Encryption:** Optional AES-256-CTR + HMAC-SHA256 with scrypt key derivation (N=32768). Format: `"FABE"` magic header + salt + IV + encrypted tar.gz + HMAC tag. Streaming for multi-GB files. **Export flow:** Estimate size → check free space → optionally stop app → copy config → dump DB → tar user data → create tar.gz → optionally encrypt → atomic rename. App restarts automatically after export if it was stopped. **Import flow:** Decrypt if needed → extract → prepare stack dir (create new or `compose down --volumes` for existing) → restore config (re-encrypt app.yaml with current server key) → restore user data (HDD or volumes) → restore DB (start DB service, wait for ready, import dump) → start full stack → refresh UI. **Architecture:** `internal/appexport/` package with `ExportStackProvider` adapter interface (same pattern as `backup.StackDataProvider`). `exportAdapter` in `main.go` bridges `stacks.Manager` to the provider. **API endpoints:** `/api/export/estimate`, `/api/export/start`, `/api/export/status`, `/api/export/bundles`, `/api/export/manifest`, `/api/export/import`, `/api/export/import/status`. **UI:** Export button on app info page, standalone import page at `/import` accessible from the stacks page header. --- ### 3. Backup System The backup system implements a **3-2-1 backup architecture**. Each tier is a **complete, self-sufficient backup** — any single tier can fully restore an app. | Tier | Contents | Location | Can fully restore? | |------|----------|----------|--------------------| | **1. Nightly restic** | DB + Config + User data | Same drive as app | Yes (not against drive failure) | | **2. Cross-drive** | DB + Config + User data | Different physical device | Yes | | **3. Remote** | Everything | Cloud / remote server | Future | **Key principles:** - User data backup is **mandatory** — every app with HDD bind mounts is included automatically. There is no per-app toggle. - Each tier includes **everything** needed to restore: DB dumps, config, and user data. No tier depends on another tier's data. - **Tier 2 is configurable for ALL apps** — not just apps with HDD data. Non-HDD apps back up config + DB dumps to the secondary drive (small but protects against drive failure). - The `AppBackupPrefs.Enabled` field in settings.json is legacy and not read by any code. **Per-app Tier 2 contents by app type:** | App type | Tier 2 contents | Example | |----------|----------------|---------| | HDD + DB | Config + DB + User data | Immich, Paperless-ngx | | HDD, no DB | Config + User data | — | | Docker volumes + DB | Config + DB + Volume data | Tandoor | | Docker volumes, no DB | Config + Volume data | Mealie (SQLite) | | DB, no HDD/volumes | Config + DB | Vikunja | | Config only | Config | Gokapi, Homepage | #### Tier 1: Nightly Backup (mandatory, same drive) The nightly backup has two phases that run sequentially. All paths are **per-drive** — each physical drive gets its own restic repo and per-app DB dump directories. **Drive layout (v0.26.0):** ``` / ├── felhom-data/ ← all controller-managed data (namespace, v0.26.0+) │ ├── appdata// ← app user data │ └── backups/ │ ├── primary/ │ │ ├── restic/ ← one restic repo per drive (all apps on this drive) │ │ └── / │ │ ├── db-dumps/ ← per-app DB dump files │ │ └── volume-dumps/ ← per-app Docker volume tars (v0.33.0) │ └── secondary/ │ ├── restic/ ← secondary restic repo (cross-drive) │ ├── _infra/ ← infra config mirror │ └── /rsync/ ← per-app rsync data ├── .felhom-infra-backup/ ← DR marker (stays at drive root for scanner) ├── Dokumentumok/ ← user files (not controller-managed) └── media/ ← user files (not controller-managed) ``` > **Note (Model A — corrected in v0.52.0):** `HDD_PATH` in `app.yaml` is the **in-guest mount point** > (e.g., `/mnt/felhom-usb`). Under slice-10 Model A the host agent binds `/felhom-data` directly > onto that mount, so the in-guest mount **already is** the `felhom-data` namespace root. Neither the > compose templates nor the path helpers add a `felhom-data` segment for a drive-resident app: app data > is `${HDD_PATH}/appdata/` and backups `${HDD_PATH}/backups/...`, **single-nested**. Only the > SSD-only system-data fallback (a bare root, `inGuestDrive=false`) appends `felhom-data`. See > `NamespaceRoot(drivePath, inGuestDrive)` in `internal/appbackup/paths.go`. > Earlier catalog templates used `${HDD_PATH}/felhom-data/appdata/`, which double-nested to > `.../felhom-data/felhom-data/...` on a Model-A drive; v0.52.0 dropped that segment in the catalog and > locks deploy↔backup path agreement with `internal/stacks/hddpath_agreement_test.go`. Path computation is centralized in `backup/paths.go` via the `FelhomDataDir = "felhom-data"` constant: - `PrimaryResticRepoPath(drivePath)` → `/felhom-data/backups/primary/restic/` - `AppDBDumpPath(drivePath, stackName)` → `/felhom-data/backups/primary//db-dumps/` - `AppVolumeDumpPath(drivePath, stackName)` → `/felhom-data/backups/primary//volume-dumps/` - `AppDataDir(drivePath, stackName)` → `/felhom-data/appdata//` - `SecondaryResticRepoPath(drivePath)` → `/felhom-data/backups/secondary/restic/` - `AppSecondaryRsyncPath(drivePath, stackName)` → `/felhom-data/backups/secondary//rsync/` - `SecondaryInfraPath(drivePath)` → `/felhom-data/backups/secondary/_infra/` - `InfraBackupDir(mountPath)` → `/.felhom-infra-backup/` (**unchanged** — stays at drive root for DR scanner) > **⚠️ Stale:** the restic/secondary helpers above (`PrimaryResticRepoPath`, `SecondaryResticRepoPath`, > `AppSecondaryRsyncPath`, `SecondaryInfraPath`) describe the pre-strip layout — restic/cross-drive was > removed in slice 8C. This section is rewritten when Tier 2 (Phase 3) lands. #### Per-app recovery unit (Phase 2, v0.53.x) — SECRET-FREE Each app's `backups/primary//` is a self-contained, recreatable **recovery unit**: ``` backups/primary// ├── compose/ docker-compose.yml + .felhom.yml + a SECRET-STRIPPED app.yaml ├── db-dumps/ app-consistent DB dump(s) ├── volume-dumps/ named-volume tars └── manifest.json image pins, secret env-var NAMES, data_key names, checksums, secret_source ``` - **Secret-free by design.** The unit stores **no secret value, no data-encrypting key, and not the Docker image** — only the pinned image tag(s) (re-pulled on restore) and the *names* of the secret / `data_key` env vars. Rationale: app.yaml + the encryption key live on the guest rootfs → already in the PBS whole-guest snapshot, and the hub is deliberately zero-knowledge. Restore recovers the original secrets from the guest's own app.yaml (live, or via PBS) and **regenerates nothing**; for a `data_key` app it **fails closed** (refuse + warn) if the key can't be recovered. - Helpers: `RecoveryUnitPath` / `RecoveryUnitComposePath` / `RecoveryUnitManifestPath` (`internal/appbackup/paths.go`). Capture: `Manager.CaptureRecoveryUnit` (`internal/backup/recovery_unit.go`), run from the daily DB dump and the periodic `RefreshCache` (idempotent checksum-skip). The non-secret env comes from `StackDataProvider.GetStackRecoveryInfo` (excludes secret-named + encrypted values, so the capture never touches a secret). `data_key` fields are marked in `.felhom.yml` (`DeployField.DataKey`). - **Restore replays the DB dump (F17, v0.61.0).** `RestoreFromRecoveryUnit` (and the `RestoreApp` fallback) stops the app → restores named-volume tars → recreates the compose definition + redeploys with the recovered env → **replays each `db-dumps/*.sql` into the now-running DB** via `backup.reimportDBDumps` → `appbackup.ImportDump` (psql / mariadb client, using the live container's own discovered credentials). The DB replay runs AFTER the volume restore, so the **logical SQL dump wins** over any volume-tar copy of the database (the dumps use DROP/CREATE — `pg_dump --clean --if-exists`, `mariadb-dump` default `--add-drop-table` — so replay is idempotent). Volume-restore and DB-import failures now **surface** (restore returns an error) instead of a swallowed WARN. Prior to v0.61.0 the per-app restore never replayed the `.sql`, so DB-resident data did not come back. #### Tier 2 — off-drive copy (Phase 3, v0.55.x) For every HDD app, Tier 2 (`internal/backup/tier2.go`) rsync-mirrors the recovery unit (`backups/primary//`) + the app's `appdata//` to `/backups/secondary//` on a **different physical disk** — the only off-drive protection bind-mounted HDD userdata can get (PBS can't reach bind mounts). Auto-targeted: **prefer another registered user-data drive** (off-disk via `system.SamePhysicalDevice`); else the **internal SSD for small units only**, behind a size-aware **rootfs-headroom guard** (`tier2FitsHeadroom`) that **refuses rather than fills** the ~8 GB guest rootfs (reserve = `max(2 GB, 20%)`), recording an honest "needs a 2nd HDD" status. Status persists via `settings.CrossDriveBackup` and drives the "2. mentés" card. Runs daily (`tier2-backup`, 03:30) or via `POST /api/backup/tier2`. restic is **not** used — a plain browsable mirror. **Per-app Tier-2 config panel (v0.57.0)** — `GET/POST /stacks/{name}/backup` (`internal/web/tier2_config_handler.go` + `templates/tier2_config.html`). The "2. mentés" row's **Beállítás** button links here (was the dead-end deploy page). Shows the effective off-drive target (pinned or auto), whether it's the size-limited internal SSD, the last-run reason, and lets the customer **pin a registered drive** (off physical disk) or **toggle Tier 2 off**. Always visible — single-SSD apps get the "csak DB/konfiguráció" note, non-HDD apps the "already in the PBS whole-guest snapshot" context. Two preference fields on `CrossDriveBackup` — `UserDisabled` + `PreferredTarget` (set via `Settings.SetTier2Preference`) — are **preserved across the runner's status writes** (`withTier2Prefs`): `selectTier2Target` honors a valid pin before auto-picking; `RunTier2` skips a disabled app. The runner re-validates the pin off-disk at run time. `Manager.Tier2Info(stackName)` is the read-only panel view (effective target + eligible alternative drives). **Phase 1 — Database Dumps** (`internal/backup/dbdump.go`, scheduled 02:30) - **Auto-discovery** of PostgreSQL and MariaDB containers via `docker ps` + `docker inspect` - Dumps via `docker exec pg_dump` / `docker exec mariadb-dump` with 5-minute timeout - Dumps are written to the app's **home drive**: `AppDBDumpPath(appDrive, stackName)` - Atomic writes (`.tmp` → `.sql`) to prevent corruption - **Validation** after each dump: checks file size, header presence, counts `CREATE TABLE` - Results cached in `settings.json` surviving container restarts **Phase 1b — Docker Volume Dumps** (`internal/backup/backup.go`, runs after DB dumps) - Iterates all deployed stacks that have Docker named volumes (`GetDockerVolumes()`) - **v0.34.0:** Each stack is stopped before dump, restarted after (`DumpAppVolumesSafe()`) — prevents inconsistent tars of live databases. Protected stacks (traefik, etc.) that reject StopStack are skipped with a warning. - For each volume: `docker run --rm -v :/vol:ro -v :/out alpine tar cf /out/.tar -C /vol .` - 10-minute timeout per volume; warnings on failure (non-fatal) - Stale tars cleaned up (volumes that no longer exist) - Volume names resolved with project prefix via `ResolveDockerVolumeNames()` (e.g., `mealie_mealie_data`) - Dumps written to `AppVolumeDumpPath(appDrive, stackName)` **Phase 2 — Restic Snapshot** (`internal/backup/restic.go`, scheduled 03:00) - Apps are **grouped by drive** via `groupStacksByDrive()` — each drive's apps are backed up to that drive's restic repo - App drive resolution: `GetStackHDDPath()` (from `StackDataProvider`) → falls back to `SystemDataPath` - Auto-generated repository password (32 random bytes, base64url), shared across all repos, synced to hub - **Paths included in each per-drive snapshot (v0.34.0: per-app scoped):** - Per-app DB dump dirs on that drive - Per-app Docker volume dump dirs (`volume-dumps/*.tar`) - Per-app HDD mount paths (user data) - Per-app stack config dir (`//` — only for stacks on this drive) - `controller.yaml` — only on the system drive (not duplicated across all drives) - Auto-detects and unlocks stale locks (restic repo lock) - Weekly prune on Sundays with configurable retention (keep-daily, keep-weekly, keep-monthly) - Weekly integrity check (`restic check`) on Sunday 04:00 — checks **all** primary repos **Protects against:** accidental deletion, data corruption, point-in-time rollback. Does NOT protect against drive failure (backup is on the same physical drive). #### Tier 2: Cross-Drive Backup (opt-in, different device) (`internal/backup/crossdrive.go`) **Complete backup** to a different physical drive. Available for **all apps** — apps with HDD data back up config + DB + user data + Docker volumes; apps without HDD back up config + DB dumps + Docker volumes. - **Auto-enable for small apps (v0.14.1):** Apps without HDD mounts (config-only, DB-only) are automatically configured for daily rsync Tier 2 when ≥2 storage paths are registered. `AutoEnableSmallApps()` runs at the start of each nightly backup cycle. Never overwrites existing user-configured cross-drive settings (even disabled ones). - **Infrastructure config backup (v0.14.1):** `syncInfraConfig()` rsyncs the stacks directory and `controller.yaml` to `/backups/secondary/_infra/` on every secondary destination drive. Runs before per-app backups. Cross-drive restic also includes infra paths. - **Two methods:** - **rsync** — Simple mirror with `--delete` (fast, no versioning, **browsable** on disk) - **restic** — Versioned, deduplicated, encrypted (shared repo across apps, not browsable) - Per-app configuration in settings.json: destination path, method, schedule (daily/weekly/manual) - **Pre-backup DB dump:** `DumpStackDB()` runs fresh pg_dump/mariadb-dump before each cross-drive backup; non-fatal on failure (wired via `DBDumper` interface to avoid circular imports) - **Pre-backup volume dump (v0.33.0, safe stop/start v0.34.0):** `DumpAppVolumesSafe()` stops the stack, exports Docker named volumes to tar, restarts — wired via `VolumeDumper` interface - **Empty mounts allowed:** `RunAppBackup` accepts apps with no HDD mounts — the rsync mount loop simply doesn't execute, but DB + config copy still runs - **Drive-type-aware validation** (`ValidateDestination`): | Destination type | Space checks | |-----------------|--------------| | External mount (different device than `/`) | Block if <100 MB free | | System drive (same device as `/`) | Require ≥10 GB free AND <90% used; logged warning | - **Secondary drive layout (v0.14.1):** ``` /backups/secondary/ ├── _infra/ ← infrastructure config mirror (v0.14.1) │ ├── controller.yaml │ └── stacks/ ← full stacks dir (all app configs) ├── /rsync/ ← per-app rsync mirror │ ├── _db/ ← DB dump files │ ├── _config/ ← compose.yml, app.yaml, .felhom.yml │ ├── _volumes/ ← Docker volume tars (v0.33.0) │ └── ← HDD mount contents (if app has HDD data) └── restic/ ← shared restic repo (all cross-drive apps) ``` - DB dump files read from **per-app home drive** path (`AppDBDumpPath`) - `_` prefix directories prevent collision with user data - For non-HDD apps, only `_db/`, `_config/`, and `_volumes/` (if applicable) are present (no user data directory) - **Restic backup paths:** includes HDD mounts (if any) + config dir + per-app DB dump dir from home drive + stacks dir + controller.yaml (infra, v0.14.1) - Safety guards: destination ≠ source, path-overlap check (HDD mounts only), writable check - **Chained execution:** runs immediately after nightly restic — daily apps every night, weekly apps on Sundays - **Hub reporting after manual triggers (v0.27.2):** `OnCrossDriveComplete` callback on Router pushes infra backup snapshot to Hub + writes local infra backup after both single-app and run-all manual triggers complete (previously only automatic scheduled runs reported) - Per-app concurrency lock prevents overlapping runs - Status (last_run, duration, size, error) persisted to settings.json **Protects against:** primary drive failure, drive theft/damage. #### Tier 3: Remote Backup (future) Complete offsite backup for disaster recovery. Not yet implemented. Placeholder shown in UI ("3. mentés — Hamarosan"). #### Restore (`internal/backup/restore.go`) Both **Tier 1** (restic) and **Tier 2** (rsync) restores are supported. All deployed apps appear in the restore dropdown with per-app snapshot filtering. | App type | Config restored | DB restored | User data restored | Docker volumes restored | |----------|----------------|------------|-------------------|------------------------| | Has HDD data | Yes | Yes | Yes (always) | Yes (if present) | | Docker volumes, no HDD | Yes | Yes | n/a | Yes | | DB only, no HDD/volumes | Yes | Yes | n/a | n/a | | Config only | Yes | — | n/a | n/a | **Snapshot API** (`/api/backup/snapshots?stack=`): - Returns snapshots **only from the app's home drive** primary repo (prevents showing irrelevant snapshots from other drives) - Appends a synthetic Tier 2 entry (ID `tier2-rsync`) from cross-drive config when last backup was successful - Dropdown groups by tier: "1. szint — Helyi mentes" and "2. szint — Masodlagos masolat" **Restore type info** shown per-app when selected in dropdown (Hungarian banners): - Has HDD or Docker volumes: "Teljes visszaallitas: adatbazis + konfiguracio + felhasznaloi adatok" - Has DB, no user data: "Adatbazis es konfiguracio visszaallitasa" - Config only: "Csak konfiguracio visszaallitasa" **Tier 1 restore** (`RestoreApp`): - Stop app → resolve app's home drive → `restic restore --target / --include ...` → populate Docker volumes from restored tars → restart app → health check - Restore paths: config dir, DB dump dir, volume dump dir, HDD mounts - Docker volumes restored via `restoreDockerVolumes()`: `docker volume rm -f` → `docker volume create` → `docker run alpine tar xf` **Tier 2 restore** (`RestoreAppFromTier2`): - Stop app → rsync config from `_config/` → rsync HDD data (single/multi-mount) → copy DB dumps from `_db/` (streaming `copyFile`) → restore Docker volumes from `_volumes/` tars → restart app → health check - Uses rsync `--delete` for config and HDD data to ensure exact mirror state - Single-mount apps: data directly in rsync dir (excluding `_*`); multi-mount: per-leaf subdirectories **Common:** - **v0.34.0:** Post-restore health check (`waitForHealthy`) polls container state with `docker ps` refresh every 5s for up to 90s. Warning logged if app doesn't reach running state; restore still returns success (data is restored regardless). - Running flag prevents concurrent backup/restore operations - Snapshot ID validated (8-64 lowercase hex, or special `tier2-rsync`) - Import from `.fab` bundle link shown in restore section for cross-system migration #### Backup Page UI (`internal/web/templates/backups.html`) Unified per-app status table with expandable rows showing **per-tier** backup status: **Status dot per app:** | Dot color | Meaning | |-----------|---------| | Green | 2+ tiers configured with successful backups + destination healthy | | Yellow | Only 1 tier, or Tier 2 failing, or Tier 2 configured but never run, or destination disconnected/inactive | | Red | Tier 2 destination blocked or inaccessible | Every app starts as yellow (1 tier only). Green requires Tier 2 configured with successful backup. **Per-app backup tiers (3 rows per app):** - **1. mentes** (Tier 1, always present) — Auto badge + "helyi" + last run + contents (e.g., "DB + Konfig + Adatok") - **2. mentes** (Tier 2, configurable for ALL apps) — one of: - Configured: method (rsync/restic) + destination + schedule + last run + status + contents + browsable indicator (folder icon for rsync) + action buttons - Not configured: "1. mentes auto" + "Nincs 2. masolat" + settings link - **3. mentes** (Tier 3, placeholder) — grayed out "Hamarosan" + "tavoli (offsite)" + future note **Backup contents per app** (shown per tier): - Apps with DB + HDD: "DB + Konfig + Adatok" - Apps with Docker volumes (no HDD): "Konfig + DB + Adatok" or "Konfig + Adatok" - Apps with DB only: "DB + Konfig" - Apps with HDD, no DB: "Konfig + Adatok" - Apps with neither: "Konfig" **Deploy page** shows cross-drive (Tier 2) configuration form for **all deployed apps**, not just those with HDD data. Non-HDD apps can configure destination, method, and schedule. **Other sections:** - Schedule overview with next run times for DB dump, restic, prune - Snapshot history table (last 20 snapshots aggregated from all per-drive repos, sorted by time) - Storage overview card (total size across repos, snapshot count, DB dump count/size, encryption key with show/copy) - Restore section: app dropdown → per-app snapshot dropdown (Tier 1 + Tier 2 grouped) → restore type info → confirmation checkbox → execute → import from `.fab` bundle link --- ### 4. Storage Management > **⚠️ INTERMEDIARY-MOUNT model (v0.67.x, pairs with agent v0.35.x).** External data drives are now > visible in the guest at a STABLE path **`/mnt/felhom-drives/`**, NOT the raw `/mnt/`. The > agent keeps a single permanent parent bind `/mnt/felhom-drives` in the guest and swaps each drive's > `felhom-data` namespace **underneath it host-side** (`mount --bind`), so the guest sees attach/detach > **live with no reboot** (mount propagation), the bind source never disappears (C1-immune), and only > `felhom-data` crosses in (confinement). The per-drive `pct set -mpN` bind is **deprecated**. > - The registered storage path + every app's **`HDD_PATH`** + the FileBrowser source = the stable > `/mnt/felhom-drives/`. The controller maps it back to the raw `/mnt/` (`agentWhere()`) > only for agent calls (assign/attach/eject/decommission). `GET /api/disks` carries `guest_path` + > `bound_under_parent` (the guest-visible signal). > - **Drive-absent gate** (`internal/web/intermediary.go`, `driveGateLoop` 30s): an absent > `/mnt/felhom-drives/` drive stops + blocks its apps (`StoppedStacks` = the gate-stopped set); > a returned drive re-attaches + auto-restarts them; `actionStack` refuses to start an app whose drive > is absent ("tárhely nem elérhető"). SSD/system paths are never gated. > - **H1 endpoints** `POST /api/storage/{disconnect,reconnect,restart-apps}` drive the host-side > eject/reconnect (no guest reboot). > - **Lifecycle (v0.68.x):** a "Leszerelés" button decommissions a drive (migrate-then-decommission OR > decommission-anyway type-to-confirm — non-destructive, never touches the parent mp); a > "Visszacsatlakoztatás" button one-click re-enrolls a decommissioned/ejected drive (clears the marker, > re-binds under the parent, restarts gate-stopped apps). Decommissioning the DEFAULT auto-promotes > another schedulable drive (`defaultPromotionTarget`), or BLOCKS if it's the only one. Eject and > decommission keep the RAW drive mounted (logical retire) so re-enroll re-binds it. > - **Guest-reboot convergence is DETERMINISTIC** via the agent's `guest_boot_id`: the controller persists > `LastGuestBootID` and, when it changes, recreates EVERY deployed drive-backed app onto the > re-propagated drive (`processGuestBootChange` — no fragile container-uptime sampling). > **v0.71.0 — boot-race fix:** on a guest `pct reboot`, in-guest dockerd auto-starts the apps ~18s > BEFORE the agent re-binds the drive, so their volume bind fails at create-time > (`mkdir …/userdata: permission denied`, `RestartCount=0` → never retried → stuck `Exited`). The old > recovery sampled the agent's `BoundUnderParent` ONCE, raced that rebind, recreated nothing, and burned > its boot-id one-shot. `processGuestBootChange` now **gates on the REAL live in-guest bind** > (`driveBindLive`: is `/mnt/felhom-drives/` an actual mountpoint in the controller's own `/mnt` > rslave `/proc/self/mountinfo`?) and **waits** for it (`pollLiveBinds`, bounded ~120s) before recreating > — including apps stuck `Exited` with a create-time mount failure (`shouldRecreateOnBoot` is > state-independent). The **guest-only reboot path** (which the host-reboot sweep never exercised) is now > covered; drives that never go live in the window are left to the drive-absent gate. `processGuestBootChange` > also runs on every periodic `driveGateLoop` tick now (idempotent, boot-id gated) so a momentarily-unreachable > agent right after a guest reboot no longer permanently strands recovery. > **Agent-path prerequisite (also v0.71.0):** the whole drive gate needs `cfg.LocalAPI.Endpoint` (the > per-guest agent local API). `bootstrap.MaybeIngest` now calls `ensureLocalAPI` on the already-configured > path — merging `local_api` from `bootstrap.json` into an existing controller.yaml that lacks it (seeded > before `local_api` existed) — because without it `agentClient()` returns "agent not configured" and the > entire gate + boot recovery silently die. > **v0.72.0 — FileBrowser convergence on boot-recreate:** FileBrowser is base-infra (it binds each > drive's `userdata` but has no `HDD_PATH`, so it is NOT in the drive-backed recreate set) — after a > host reboot its mounts could be stale (the early first-boot bring-up bound them before the drives went > live). `processGuestBootChange` now, **after** `pollLiveBinds` confirms the binds and the apps are > recreated, triggers `go s.SyncFileBrowserMounts()` so FileBrowser converges against the now-live drives. > The sync runs unconditionally (FileBrowser reflects the current bind state even if no app needed > recreating). The recreate loop is a pure `recreateDriveBackedApps(stacks, present, recreate, syncFB)` > that calls `syncFB` exactly once, after every recreate. Live-accepted over two real `felhom-pve` > reboots (FileBrowser non-stale, all drive-backed apps recovered, agent tolerated a `/dev/sdX` swap by UUID). > > **⚠️ Rebuilt on the agent-delegated disk model (v0.43.0), made ROLE-AWARE in v0.44.0, UX-polished in > v0.45.0.** After the 8C > de-privileging, the controller holds **no Proxmox/disk credentials and no destructive authority** — disk > execution + the gate live entirely in the **host agent**. The drive UI is driven by the agent's > authoritative **role** (`system` | `backup` | `user-data`, from `GET /api/disks`): the appliance's own > system storage and the backup safety-net are visibly **protected** (lock badge, NO destructive controls); > the customer manages their own **user-data** drives with informed consent. The agent re-enforces role at > wipe time — the UI lockout is defense-in-depth, not the gate. > - **Overview** (`settings.html` ← `GET /api/disks`): styled **cards** (not a table) — name, mono > device/mount, badges for class (gyors/lassú), data (`Adatot tartalmaz`), **role** (🔒 Rendszer / 🔒 > Biztonsági mentés — védett / Felhasználói adat) and registered state, plus a **capacity bar** (the > monitoring `system-bar`, from the agent's `total_bytes`/`used_bytes`). Eject/Wipe render **only** for > user-data drives mounted under `/mnt`. > - **(v0.45.0) Deterministic order** — `agentDisksListHandler` sorts the list server-side > (`sortDisksForView`): **user-data → system → backup** (then unrecognized), alpha by name within a > tier, so it no longer reorders on each reload (the agent's view iterates an unordered Go map). > - **(v0.45.0) Purpose + app-backing clarity (B4)** — `local` and `local-lvm` are both shown (not > collapsed); each card carries a plain-Hungarian **purpose description** keyed on the agent's > role/type, the app-backing storages are tagged (`local-lvm` → "Alkalmazás-rendszer"; user-data → > "Alkalmazás-adatok"), and a one-line tiering note above the list answers "which storage do the > apps use?". Pure presentation — role/type stay authoritative from the agent. > - **(v0.45.0) Register shortcut (B3)** — a mounted, **unregistered** user-data drive offers > **Regisztrálás** as its PRIMARY action: `POST /api/storage/register` → `registerStoragePath` records > the existing mount (no format, no eject) + FileBrowser-syncs. Leválasztás/Törlés stay secondary. > - **Customer wipe/eject** — a **type-to-confirm** modal that names the deployed apps that break > (`GET /api/storage/impact` → `appsUsingPath`) and disables the destructive button until the **mount > name is typed exactly**. Wipe (`POST /api/storage/wipe`): eject (unmount + deregister) → server-side > two-step customer-confirmed format (learn the agent's durable id, then re-submit `confirmed:true` bound > to it). The agent refuses a protected device regardless of what the controller sends. > - **Guided init** (`/settings/storage/init`, `POST /api/storage/init`, `web/storage_handlers.go`): format > → resolve the new fs UUID → `assign` → register. The selector lists **only user-data** targets. A > data-bearing user-data device now uses the **customer-confirmation** flow (type-to-confirm → re-submit > `confirmed:true` + durable id), NOT the `felhom-opsign` command. The opsign surface remains a fallback > only if a protected device somehow reaches init. > - **Guided attach** (`/settings/storage/attach`, `POST /api/storage/attach`): non-destructive — resolve > the existing fs UUID → `assign` → register. Selector restyled to cards (user-data only). > - **Eject** (`POST /api/storage/eject`): benign unmount + deregister, with the agent's dependent-guest > warning + the affected-app list (parity with wipe). **The eject is ROLE-GATED at the agent** (felhom- > agent v0.24.0): `POST /disks/eject` refuses to unmount a system/backup mount — the UI hiding the button > is defense-in-depth, not the control. Only user-data mounts are ejectable. > - **`agentapi`** (`internal/agentapi`) is the pinned client to the agent local API: `Disks`/`AssignDisk`/ > `EjectDisk`/`FormatDisk(…, confirmed, durableID)`; `DiskInfo.role`+capacity; > `FormatResult.{role,needs_confirmation,durable_id}`; `ErrNeedsConfirmation` (user-data) vs > `ErrFormatRefused` (system/backup). `FormatResult.PendingOp.OpsignCommand()` for the operator path. > - **(v0.74.0) Client lifecycle — ONE shared client, reused.** `Server.agentClient()` builds the > `agentapi.Client` once (memoized via `sync.Once`) and returns the same instance to all ~19 call > sites; the `http.Transport` is bounded + expiring (`MaxIdleConnsPerHost:2`, `IdleConnTimeout:90s`). > This replaced a per-call `agentapi.New(...)` that leaked one idle ESTABLISHED socket per call and > exhausted the ephemeral source-port range to the agent's `:8443` after ~5 days (EADDRNOTAVAIL). > Safe because `cfg.LocalAPI` is static per process (a config-apply triggers a graceful self-restart). > - The **`StoragePath` registry** (`settings.go`: `AddStoragePath`/default/schedulable/label) is unchanged. > - **(v0.64.0) `AutoDiscoverStoragePaths` is now ADDITIVE** — it no longer bails when the registry is > non-empty; instead it registers only deployed-app paths missing from the registry. It never removes > or mutates an existing entry, never re-adds or reactivates a path already present in ANY state > (incl. `Decommissioned`), and never flips `IsDefault` (a new path defaults only if the registry has > no default at all). This is NOT auto-register-on-attach (that recommendation was rejected — manual > enrollment is by design); it only picks up drives that deployed apps already reference. > - **(v0.64.0) `InferStorageLabel` disambiguates the internal SSD** — a path whose basename is the > `felhom-data` namespace dir (the internal system volume, e.g. `/mnt/sys_drive/felhom-data`) now reads > **"Belső SSD (rendszer)"** instead of the colliding "Tárhely (felhom-data)". Model-A user drives > register their MOUNT ROOT (e.g. `/mnt/felhom-usb`), never `.../felhom-data`, so this can't mislabel a > user drive. Still overridable via `SetStorageLabel`. > - **(v0.65.0) Data migration** (`internal/stacks/migrate.go`) — move app data between drives, in-process > over the controller's `/mnt:/mnt:rslave` RW mount; crash-safe + resumable via a journal > (`/migration.json`). `MigrateAll(source,target)` moves the whole felhom-data namespace (every > app + a conflict-merge walk for non-app/customer content); `MigrateApp(app,target)` moves one app's > subtree (drive→drive AND SSD→drive). Pipeline: validate → stop → copy (`rsync -a --checksum`, additive, > **no `--delete`**) → verify (`rsync -ani --checksum`, zero pending) → flip+redeploy (`RedeployFromEnv`) > → cleanup. **CLEANUP — the only destructive step — is gated on every unit verified AND every app > redeployed.** Conflict-merge: skip-identical (checksum vs target + its `(N)` siblings), rename-on-differ > to lowest-free `(N)`, never overwrite; idempotent. Single-flight + mutually exclusive with the > backup orchestrator. UI: `POST /api/storage/migrate{,-app}` + `GET /api/storage/migrate/status` (poll); > migrate-all on the settings page, per-app on the app-info page, shared Hungarian progress panel. > - **(v0.65.0) Self-serve decommission** (`handleStorageDecommission`, `POST /api/storage/decommission`) — > retire a drive, non-destructive (never formats). Two choices (no partial): migrate-all-then-decommission > (runs `MigrateAll`; the migration done-hook soft-marks the source + calls the agent on success), or > decommission-anyway (type-to-confirm; stops the apps, keeps their `HDD_PATH` so they show "Hiányzó > tárhely"). Both end at `settings.SetDecommissioned` (soft marker retained — blocks A1 resurrection) + > `agentapi.Decommission` (agent v0.32.0: `POST /disks/decommission`, role-gated user-data, intent + > bind-prune + unmount). Re-enrolling a decommissioned drive (`registerStoragePath`) clears the marker + > restores `Schedulable`. A deployed app whose drive is decommissioned/disconnected/absent shows the > **"Hiányzó tárhely"** warning badge on the dashboard/stacks/app card. > > - **(v0.66.0) Userdata layout + shared-storage ownership convention** (`internal/appbackup/userdata.go`). > Each drive's felhom-data namespace gains a customer-facing **`userdata/`** tree (sibling of > `appdata/`/`backups/`) — the ONLY thing FileBrowser mounts. Skeleton: > `media/{movies,tv,music,audiobooks,books,comics,photos}`, `downloads`, `import/{paperless,calibre}`, > `roms`, `documents`. **Convention:** every userdata dir is group-owned by `SharedContentGID` (1000), > mode **2775** (setgid + group-rwx) — `EnsureUserdataDir` = MkdirAll → explicit `Chmod(ModeSetgid|0775)` > (MkdirAll's mode is umask-masked AND drops setgid) → chown group 1000. Setgid makes new files inherit > group 1000 so FileBrowser (uid 1000) + the content apps collaborate without permission collisions. > - **`USERDATA_PATH`** = `/userdata` is injected into the compose env (the shared > `withUserdataPath`, used by BOTH `stackEnv` and `composeExecWithEnv` — the initial deploy builds env > from values, not app.yaml). Catalog media mounts use `${USERDATA_PATH}/...`. > - **Pre-create**: the full skeleton is created with the convention on `registerStoragePath` + > `syncFileBrowserMounts` (system + additional drives); a **deploy belt** in `composeExecCustomEnv` > (on `up`) pre-creates every `${USERDATA_PATH}/...` bind source the app declares, so Docker never > auto-creates a userdata dir as guest-root (covers apps not in the skeleton). > - **FileBrowser** mounts `/userdata` (was `appdata`) and runs as uid 1000 → it can create > folders + upload into the 2775 setgid tree (fixes the old permission-denied); app internals > (`appdata/`) are no longer browsable. **(v0.66.2)** its entrypoint is wrapped > `["sh","-c","umask 002; exec /home/filebrowser/filebrowser"]` so folders the customer creates come > out **2775 (group-writable)** — the gtstef image is a single Go binary that ignores a `UMASK` env > (verified), so the wrapper is the mechanism. Without it, customer folders were 2755 (group-read > only) and apps in group 1000 couldn't write into them. > - **Run-identity**: PUID/PGID-1000 apps (radarr/sonarr/calibre with `UMASK=002`) write **group- > writable**, so FileBrowser can fully manage their output. Root-only apps (jellyfin :ro; komga + > audiobookshelf RW after the `user:1000` fallback) write **group-readable** via setgid (FileBrowser > browses/reads, the app manages its own files). > - **Migration-safe**: `migrate.go`'s merge walk preserves the source dir's setgid + group (and > `copyFile` the full file mode + group), so the convention survives a whole-drive `MigrateAll`. > > The privileged controller-side disk subsections **below are historical** (the `internal/storage/*` scan/ > format code was removed in 8C — execution is the agent's now). The storage subsystem handles the full lifecycle of external storage: detection, initialization, path registration, and data migration. #### Disk Scanning (`internal/storage/scan.go`) - `ScanDisks()` uses `lsblk -J -b` for block device enumeration - System disk detection via host fstab parsing (`/host-fstab`) + UUID resolution via `blkid` - Partitions enriched with filesystem type, UUID, and label from direct `blkid` probing (Docker containers have incomplete udev cache) - Returns `AvailableDisks` (non-system, non-loop, non-CDROM), `SystemDisks`, and `FormatablePartitions` (empty partitions on system disks that are safe to format) - Handles NVMe (`nvme0n1p1`), SCSI (`sdb1`), and eMMC (`mmcblk0p1`) naming #### Disk Initialization Wizard (`internal/storage/format.go`) A step-by-step UI at `/settings/storage/init`: 1. **Scan** — Lists available disks with model, size, partition info 2. **Select** — User picks a disk and enters a mount name (e.g., `hdd_1`) 3. **Confirm** — User types "FORMAZAS" to confirm destructive operation 4. **Format pipeline**: `wipefs` → `sfdisk` (GPT) → `mkfs.ext4` → `blkid` UUID → backup fstab → append UUID-based fstab entry → mount → `findmnt` verification → `chown 1000:1000` → create `felhom-data/` and `Dokumentumok/` subdirectories 5. Auto-registers new storage path in settings.json 6. Smart partition detection: skips repartitioning for existing empty partitions Safety guards: system disk detection, mount path conflict check, confirmation required, progress channel for real-time UI feedback. **System-disk partition formatting:** When the system disk has an empty partition (no filesystem, not mounted, not used for /, /boot, /boot/efi, or swap), the init wizard detects it via `FormatablePartitions` in the scan result and offers to format just that partition. Uses `IsSystemPartition()` (granular per-partition check via fstab) instead of `IsSystemDisk()` (whole-disk block), so sda1 can be formatted while sda3 (root) remains protected. #### Attach Existing Drive Wizard (`internal/storage/attach.go`) A step-by-step UI at `/settings/storage/attach` for drives that already have a filesystem (e.g., a previously used ext4 drive). Unlike the init wizard, this does **not** format the drive — existing data is preserved. **Problem solved:** Mounting a whole drive at `/mnt/` would mix existing user data with the controller's directory structure (`felhom-data/`, `Dokumentumok/`, etc.). The bind-mount approach isolates the controller's working directory from other data on the drive. 1. **Scan** — Lists available disks, filtered to partitions that have an existing filesystem (FSType != "") 2. **Mount raw** — Partition is mounted read-only at a hidden staging path (`/mnt/.felhom-raw/