# felhom-controller **Central management container for Felhom home servers.** The **in-guest controller**: one per customer LXC, Docker-only, Hungarian web dashboard for managing the customer's app stacks, app-data backups, monitoring and notifications. All Proxmox/disk operations are delegated to the host agent (`felhom-agent`). **Current version: v0.59.0** > ## Documentation has moved (and is now code-verified) > > The **authoritative architecture & feature documentation** now lives in the central docs home: > **[`felhom.eu/documentation/controller/`](../../felhom.eu/documentation/controller/README.md)** — > module map, deploy & stack lifecycle, backup architecture, storage/monitoring/metrics, and > auth/hub/sync/integrations. Those docs are grounded in current source (v0.59.0). > > **Quick build & deploy** is in the "Build & Deploy" section below and in the repo `CLAUDE.md` > (authoritative for the workflow). NOTE the demo controller runs in an LXC guest (9201) under the > **bootstrap-managed** mechanism (`/etc/felhom-controller-image` + `felhom-controller-bootstrap.service`), > not the bare-metal `/opt/docker` compose path some sections below still describe. > > The prose below this banner is **retained legacy reference** and may lag the central docs — when they > disagree, the central docs win. (Some sections still mention restic / pre-8C disk handling that has > since moved to the host agent.) --- ## Table of Contents - [Architecture](#architecture) - [Features](#features) - [App Management](#1-app-management) - [App Export/Import](#2-app-exportimport-fab-bundles) - [Backup System](#3-backup-system) - [Storage Management](#4-storage-management) - [Monitoring & Health](#5-monitoring--health) - [Notifications](#6-notifications) - [Update Management](#7-update-management) - [Authentication & Settings](#8-authentication--settings) - [Central Hub](#9-central-hub-reporting) - [Setup Wizard](#10-first-run-setup-wizard) - [Disaster Recovery](#11-disaster-recovery) - [Asset Sync](#12-asset-sync) - [Debug Mode](#13-debug-mode) - [Geo-Restriction](#14-geo-restriction) - [App-to-App Integrations](#15-app-to-app-integrations) - [Repository Layout](#repository-layout) - [Configuration](#configuration) - [REST API](#rest-api) - [Build & Deploy](#build--deploy) - [Roadmap](#roadmap) --- ## Architecture ``` ┌─────────────────────────────────────────────────────────────────┐ │ Customer Hardware (N100 mini PC / Raspberry Pi) │ │ │ │ ┌──────────┐ ┌────────────────────────────────────────────┐ │ │ │ Traefik │ │ felhom-controller (privileged container) │ │ │ │ (reverse │──▶│ │ │ │ │ proxy) │ │ ┌──────────┐ ┌─────────────────────────┐│ │ │ └──────────┘ │ │ Web UI │ │ Stack Manager ││ │ │ │ │ (HU dash │ │ (compose ops, git sync, ││ │ │ ┌──────────┐ │ │ board) │ │ deploy, delete, update) ││ │ │ │cloudflared│ │ └──────────┘ └─────────────────────────┘│ │ │ │ (tunnel) │ │ ┌──────────┐ ┌─────────────────────────┐│ │ │ └──────────┘ │ │ Backup │ │ Storage Manager ││ │ │ │ │ (3-layer │ │ (disk scan, format, ││ │ │ ┌──────────┐ │ │ restic) │ │ mount, migrate) ││ │ │ │ App │ │ └──────────┘ └─────────────────────────┘│ │ │ │ stacks │ │ ┌──────────┐ ┌─────────────────────────┐│ │ │ │ (docker │ │ │Scheduler │ │ Monitor & Metrics ││ │ │ │ compose) │ │ │(cron-like│ │ (health, SQLite ││ │ │ └──────────┘ │ │ jobs) │ │ time-series, Chart.js) ││ │ │ │ └──────────┘ └─────────────────────────┘│ │ │ │ ┌──────────┐ ┌─────────────────────────┐│ │ │ │ │ Notify │ │ REST API + Hub Reporter ││ │ │ │ │ (events) │ │ (JSON push + events) ││ │ │ │ └──────────┘ └─────────────────────────┘│ │ │ │ ┌──────────┐ │ │ │ │ │ Assets │ │ │ │ │ │ (Hub │ │ │ │ │ │ sync) │ │ │ │ │ └──────────┘ │ │ │ └────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────┘ │ events + reports │ git pull │ asset sync ▼ ▼ ▼ hub.felhom.eu gitea.dooplex.hu hub.felhom.eu (central dashboard) (stack definitions) (logos, screenshots) ``` ### Key Architecture Decisions - **Pure Go, no frameworks** — stdlib `net/http` + `html/template`. Only external deps: `bcrypt`, `yaml.v3`, `modernc.org/sqlite` (pure Go, no CGO). - **Privileged container** — Required for disk operations (format, mount, fstab), `/dev` access, and Docker socket control. - **`/host-dev` indirection** — Docker overrides `/dev` with a tmpfs. The host's `/dev` is mounted at `/host-dev` to access block devices. - **`StackDataProvider` interface** — Breaks circular import between the backup packages and stacks. Defined in `internal/appbackup` (and re-exposed via a type alias in `internal/backup`). Implemented by `stackAdapter` in `main.go`. Provides `GetStackHDDPath()` for per-drive backup routing. - **Atomic file writes** — All persistent state (`settings.json`, `app.yaml`) written to `.tmp` then `os.Rename` for crash safety. - **`go:embed` templates** — All HTML/CSS/JS compiled into the binary. No runtime file dependencies. - **Design system v2 (v0.96.0, TASK-D0)** — The whole UI renders in the Felhom v2 language (canonical reference: `felhom.eu/documentation/design/design-system.md`): navy token palette, single 2px radius, no shadows, exception-based status color (nominal = blue/neutral; amber/red only on deviation — `stateColor` emits `run/progress/warn/neutral/off`, `usageColor`/`tempColor` emit `nominal/warn/crit`; a stopped app is neutral, NOT red). Capacity bars are the hairline `.meter` component; state chips are `.tag`; informational pills are `.metarow`. **Fonts (Plus Jakarta Sans + JetBrains Mono, variable woff2, latin+latin-ext) and a 30-icon Lucide sprite are vendored in the binary** (`internal/web/static/fonts/` served at `/static/fonts/`; `templates/icons.html`) — no CDN, no emoji. The setup wizard serves the same embedded stylesheet via `web.StyleCSS()`. - **Page IA (v0.97.0, TASK-D1)** — the settings monolith is split into four pages, each with its own data builder (`systemPageData`/`storagePageData`/`notificationsPageData`/`securityPageData`, sharing `settingsBaseData`) and template. Routes: `/settings` (Rendszer), `/settings/notifications` (GET→page, POST→save on the same path), `/settings/security`, `/storage` (main-nav **Tárhely** — Meghajtók), and `/storage/network` (Hálózati tárhely / NAS; v0.98.0 split — nested sub-links under Tárhely). The enrollment wizards live at `/storage/{init,attach}`; `/settings/storage/{init,attach}` **301** to them; all storage action successes redirect to `/storage?storage_msg=…`. `/storage` shows a **unified drive view**: server-rendered registry cards enriched in place from the agent `/api/disks` (role/durable-id/actions, joined on mount path), plus read-only **Rendszermeghajtók** and **Nem regisztrált meghajtók** groups; agent-down degrades to a single warn note. Consequential actions use an in-page `.confirm-overlay` (`openDialog`), never native `confirm()`/`prompt()`. **Enforcement:** `scripts/template_id_gate.py` (JS element-ID integrity) + `scripts/emoji_gate.py` (no emoji — run after any template change; the Go `TestNoEmojiInTemplates` mirrors it). - **Europe/Budapest timezone** — All scheduled jobs, timestamps, and UI labels use Hungarian timezone. ### Module Map | Module | Path | Responsibility | |--------|------|----------------| | **Config** | `internal/config/` | YAML loader, validation, `FELHOM_*` env overrides | | **Settings** | `internal/settings/` | Runtime-mutable `settings.json` (passwords, backup prefs, storage paths, notifications) | | **Stacks** | `internal/stacks/` | Compose operations, scanning, `.felhom.yml` metadata, deploy/delete flow; **base-infra bring-up** (`infra.go` — `EnsureBaseStack`) | | **Infra** | `internal/infra/` | Pure renderers (embedded `text/template`) for the base-infra stacks (traefik/cloudflared/filebrowser); **pinned image tags as the single source of truth** (web filebrowser sync delegates here) | | **Crypto** | `internal/crypto/` | AES-256-GCM encryption for sensitive app.yaml values (passwords, secrets), key management | | **Sync** | `internal/sync/` | Git-based app catalog sync (clone/pull, content-hash copy) | | **AppBackup** | `internal/appbackup/` | Self-contained app-data backup primitives: DB dump discovery/execution (`DiscoverDatabases`, `DumpOne`), Docker-volume/app-data discovery (`StackDataProvider`, `DiscoverAppData`), keep-side path helpers (`AppDBDumpPath`, `AppVolumeDumpPath`, `AppDataDir`). `DiscoverDatabases` takes the deployed-stack set so a DB container maps to the right stack even when a slug ends in a DB-role token (M19, v0.62.0). `ListDumpFiles` takes an optional `cached(name,size,mod)` lookup so an unchanged dump isn't re-validated (line-scan) every ~5-min cycle (M18, v0.62.0). No dependency on restic/cross-drive/drive-mount. Imported directly by `appexport` and `storage`. | | **Backup** | `internal/backup/` | Per-drive 3-layer backup: DB dumps → restic snapshots → cross-drive copies, restore. Re-exposes the `appbackup` primitives via aliases/forwarders (`appbackup_bridge.go`) for the disk/host-side code and the web/api/report consumers. | | **Storage** | `internal/storage/` | Disk scanning (`lsblk`), partitioning (`sfdisk`), formatting (`mkfs.ext4`), mounting, data migration (`rsync`) | | **System** | `internal/system/` | System info (`/proc`), CPU collector, mount points, disk usage, FS info | | **Monitor** | `internal/monitor/` | System health checks, storage watchdog, legacy Healthchecks pinger (deprecated) | | **Metrics** | `internal/metrics/` | SQLite time-series store, system + container metric collection | | **Scheduler** | `internal/scheduler/` | Central job scheduler (periodic + daily, skip-if-running, panic recovery) | | **SelfUpdate** | `internal/selfupdate/` | Version checking (registry), update trigger, state persistence, startup verification | | **Notify** | `internal/notify/` | Email notifications via hub relay, preference sync, per-event cooldowns | | **Report** | `internal/report/` | Hub report builder + HTTP pusher (system, stacks, backup, health) | | **Assets** | `internal/assets/` | Hub-managed asset syncer: downloads logos/screenshots with SHA-256 change detection | | **SelfTest** | `internal/selftest/` | Startup self-test: 9 diagnostic checks (Docker, dirs, storage, hub, restic, metrics) | | **Util** | `internal/util/` | Shared utilities: `TruncateStr` for debug log output truncation | | **AppExport** | `internal/appexport/` | Per-app export/import via `.fab` bundles (config + DB + user data), optional AES-256 encryption | | **API** | `internal/api/` | REST JSON endpoints, diagnostic dump (`/api/debug/dump`) | | **Web** | `internal/web/` | Hungarian dashboard, auth, page handlers, template functions, alerts | --- ## Features ### 1. App Management The controller manages Docker Compose stacks through a complete lifecycle: catalog sync, first-time deployment, runtime operations, and deletion. #### Git Sync (`internal/sync/`) The app catalog lives in a separate Git repository. The controller: - Shallow-clones the catalog on startup - Periodically fetches updates (configurable, default 15 min) - Copies only `docker-compose.yml` and `.felhom.yml` to the stacks directory - **Never overwrites** `app.yaml` (user secrets are safe) - Uses SHA-256 content hashing — only writes files that actually changed - Triggers stack rescan after sync so the dashboard updates immediately - **Post-sync hook**: auto-injects missing deploy fields (new secrets, domains) into existing `app.yaml` for stacks whose templates were updated (see Missing Field Injection below) - Manual sync via "Sablonok frissitese" button or `POST /api/sync` #### First-Time Deploy Flow 1. Customer sees app card with "Telepites" button 2. Deploy page pre-generates and **displays** all auto-values before the user clicks deploy: - `domain` fields: shown as readonly text input with the customer's configured base domain - `subdomain` fields: editable text input pre-filled with the default from `.felhom.yml`, shown with `.base-domain` suffix. Validated for DNS-safe format, reserved names, and uniqueness across deployed stacks. Locked after deploy — changing requires Remove + Redeploy - `secret` fields: pre-generated and shown as masked password inputs with a "Megjelenítés" reveal button — user can see/copy all DB passwords and keys before deploying - User-configurable inputs (admin password, language, storage path) remain editable - Section header prompts the user to note down any passwords they need 3. `checkBeforeDeploy()` JS guard fetches live state first (prevents double-deploy from another tab) 4. **Memory validation** (F1, v0.61.0): the controller runs as a Docker container inside an LXC, where `/proc/meminfo` shows the **Proxmox host's** RAM (no lxcfs in the container) and the container's own cgroup is unlimited (the guest cap lives on the LXC ancestor). So the guest cap is read from the **Docker daemon** (`system.GuestMemTotalMB()` → `docker info` MemTotal — the daemon runs in the LXC and reports the guest's lxcfs-backed RAM; the cgroup limit is preferred when present, e.g. non-nested): - `usable_memory = guest_cap - reserved_memory_mb` (default 384MB reserved) - Hard block if `committed_used + new_request > usable_memory`, where `committed_used` = `CommittedMemory()` (sum of running apps' mem requests) — the guest-wide RSS is not observable from the container, so the controller's own committed accounting is the accurate, cheap "used". - `/api/system/info` reports the guest cap as total and committed memory as used. 4b. **Docker-data volume reserved-buffer gate (v0.58.0, storage-split prevention layer):** the OS rootfs and Docker data are split onto separate volumes; infra (controller/traefik/cloudflared/filebrowser) shares the one Docker data-root (`/var/lib/docker`) and is protected by **prevention, not placement**. `system.GetDockerVolumeHeadroom()` measures the Docker-data volume via `statfs("/")` (the controller container's overlay root is the upperdir on that volume — true with the golden's **overlay2** driver) and reserves `max(5 GB, 10%)`. `deployStack` **refuses a new deploy (HTTP 507)** when free space is at/under the buffer; the deploy page shows the warning + disables the button. Fail-open on a statfs error. The runtime disk monitor (`healthcheck.go`, warn 80% / crit 90%) watches the same volume and trips above the buffer. (Assumes the split guest's large data volume; the golden bakes overlay2 + log rotation so images+volumes live on the data volume, not `/var/lib/containerd`.) 5. Pre-generated secret values are submitted as hidden form inputs so the **same values** the user saw are saved to `app.yaml` (no silent re-generation on submit). Controller saves `app.yaml`, sets in-memory `Deployed` + `Deploying` flags, then runs `docker compose up -d` **asynchronously** in a goroutine — API returns immediately so the UI switches to the progress panel without waiting for image pulls. On failure the goroutine reverts both disk and in-memory state and sets `DeployError`. 6. 3-step progress panel polls `GET /api/stacks/{name}` every 3s: config saved → `deploying` (pulling images) → containers starting → health check passed. New `StateDeploying` state shown while compose-up is in progress (no containers yet). 7. Post-deploy: locked fields (DB_PASSWORD, etc.) become read-only; the "Automatikusan generált értékek" section continues to show the saved values on the settings page 8. The deploy/settings page includes **start/stop/restart** buttons for deployed apps, plus a "Megnyitás ↗" link to the app's subdomain URL (only visible when running) #### Catch-All Page for Stopped Apps When a user visits a stopped or undeployed app's subdomain (e.g., `travel.demo-felhom.eu`), the controller serves a branded error page instead of Traefik's raw 404: - **Traefik catch-all router**: The controller's `docker-compose.yml` registers a second router (`catchall`) with `priority=1` (lowest) and `HostRegexp(.+)`. Running apps always win; only requests with no matching container reach the controller. - **`CatchAllMiddleware`** in `server.go` intercepts requests where `Host` ≠ `felhom.DOMAIN`, serves the catch-all page **without auth** (user has no session on the app subdomain). - **`findStackBySubdomain()`** identifies the app by matching the subdomain against deployed `app.yaml` `SUBDOMAIN` env or metadata fallback. - **`catchall.html`** — standalone template (no layout, inline CSS) showing the app name, status ("leállítva" / "nincs telepítve" / "nem található"), and links to the controller dashboard or the app's detail page. - **Subdomain links** on the Alkalmazások page are only shown for deployed apps (non-deployed apps have no guaranteed subdomain yet). #### Dashboard "Megnyitás" Button Running apps on the Vezérlőpult now show a "Megnyitás ↗" button that opens the app's subdomain in a new tab. The `Subdomains` map is built in `dashboardHandler` from `app.yaml` env or metadata fallback. The optional `.felhom.yml` **`open_path`** field (`Metadata.OpenPath`) is appended to that URL for apps whose UI isn't at `/` — e.g. `gokapi` → `/admin`, `ghost` → `/ghost/`. Empty = bare root. Rendered via `.Meta.OpenPath` in `dashboard.html`, `deploy.html`, and `app_info.html` (the same field flows through all three open-link sites; no handler change needed). Must start with `/`; cosmetic only (does not affect routing). #### App Info Pages Each app can define rich metadata in `.felhom.yml`: - `app_info`: tagline, use_cases, first_steps, prerequisites, default_creds, docs_url - `optional_config`: groups of post-deploy configurable env vars (e.g., API keys for metadata providers) - `resources`: mem_request, mem_limit, pi_compatible, needs_hdd, hungarian_ui - `initial_credentials` (v0.84.0): for apps that **auto-generate a first-login password into a file** at first boot (vs taking it from a deploy field) — e.g. Crafty → `/crafty/app/config/default-creds.txt`. Fields: `file` (path inside the container), `format` (`json`|`regex`|`plain`), optional `container` (defaults to the stack's main container), `username_key`/`password_key` (json), `username_pattern`/ `password_pattern` (regex, first capture group), `note`. `ReadInitialCredentials` (`internal/stacks/initialcreds.go`) reads the file **live** via `docker exec … cat` and parses it with the pure `parseInitialCreds`; the value is **never persisted** to `app.yaml`. `appDetailHandler` surfaces it on `/apps/{slug}` as a "Kezdeti belépési adatok" card (masked password + reveal/copy), labelled as the *initial* credential (stays valid only until the customer changes it in-app). Hidden when the container is down / file missing / parse fails. Reuse for any future self-seeding app. The `/apps/{slug}` page renders hero section, screenshots, setup guide, and optional config form. #### Stack Operations | Operation | What it does | |-----------|-------------| | Start | `docker compose up -d` — pre-start memory check rejects with 409 if insufficient RAM | | Stop | `docker compose stop` (blocked for protected stacks) | | Restart | `docker compose restart` | | Update | `docker compose pull` + `docker compose up -d` | | Remove | `docker compose down --volumes` + remove `app.yaml` + optional HDD/backup cleanup; template preserved for redeploy | | Delete | `docker compose down --rmi local --volumes` + optional HDD data cleanup (orphaned stacks only) | **Remove vs Delete**: "Eltávolítás" (Remove) is for deployed catalog stacks — it reverts the stack to "Nincs telepítve" state while keeping the template for easy redeployment. "Törlés" (Delete) is for orphaned stacks — it removes the entire stack directory including templates. Both require stopping the stack first. **Remove modal** shows three sections: (1) always-removed items (Docker volumes, app.yaml, cross-drive schedule), (2) optional HDD data deletion with reimport warning, (3) optional backup data deletion (DB dumps + cross-drive rsync) with restic retention note. **Protected stacks** (traefik, cloudflared, felhom-controller) cannot be stopped, removed, or deleted from the UI. Restart is allowed. **Orphan detection**: Deployed stacks with no matching catalog template are marked as orphaned with an "Elavult" badge and can be safely deleted. #### Base-infrastructure bring-up (`stacks/infra.go` + `internal/infra/`, v0.41.0) The controller stands up its own base stack — **traefik** (reverse proxy), **cloudflared** (external tunnel), **filebrowser** — instead of relying on the bare-metal `scripts/docker-setup.sh` (which a Proxmox-provisioned guest never runs). `internal/infra` renders the compose + config files from `controller.yaml` via embedded `text/template`s (lifted from `docker-setup.sh`); image tags are **pinned constants there** (`TraefikImage`/`CloudflaredImage`/`FileBrowserImage`) and the web FileBrowser sync path delegates to the same renderers, so the pinned versions can never diverge. `Manager.EnsureBaseStack()` creates the `traefik-public` network, then deploys traefik → cloudflared → filebrowser under `${stacks_dir}/`. It is: - **single-flight** (a `TryLock` guard — it's called from both first boot and every health tick, so overlapping runs must not race on the same stack dir), - **idempotent** (skips a stack whose container is already running; never overwrites an existing filebrowser compose, preserving the storage mounts `SyncFileBrowserMounts` manages), - **non-fatal** (logs, never crashes the controller). cloudflared is only deployed when a tunnel token is configured. **Triggers**: a first-boot goroutine (after stack init) and an unconditional call on every `system-health` tick (self-heal — cheap when healthy thanks to the idempotency). `monitor.EffectiveProtected` mirrors the cloudflared condition so a LAN-only node (no tunnel token) doesn't report a perpetual "protected container not running" FAIL. **Backend transports — self-signed HTTPS backends (`ensureServersTransports` → `RenderServersTransports`, v0.83.0).** Traefik talks **HTTP** to app backends by default, which is correct for every catalog app that serves plain HTTP. The exception is an app that serves its **own self-signed TLS** on the internal docker bridge (the first is Crafty, HTTPS-only on `:8443`): Traefik must speak `https` to it *and* skip verifying a per-container self-signed cert (no CA to verify against; the hop never leaves the host). `insecureSkipVerify` is **not settable via Docker labels** in traefik v3 — it must live in static/file config — so `EnsureBaseStack` writes a file-provider dynamic file `dynamic/serverstransports.yml` defining a **named** transport `insecure-skip-verify` (write-if-changed; hot-loaded by the file watcher). An app **opts in per-service** via two catalog labels — `loadbalancer.server.scheme=https` + `loadbalancer.serverstransport=insecure-skip-verify@file` (the `@file` suffix is the cross-provider reference). Backend verification stays the default (ON) for every other service — there is deliberately **no global `insecureSkipVerify`** in `traefik.yml`. This write runs **outside** `ensureTraefik` (which early-returns when traefik is already up) so an established node still materializes the file on a self-heal tick. > **Mount prerequisite (Section-G):** the controller writes these stacks under `/opt/docker/stacks` *inside its container*, but `docker compose up` runs on the **guest** Docker daemon. The golden's controller-bootstrap (`felhom-agent` `build-golden.sh`) therefore bind-mounts that path **same-path** (`-v /opt/docker/stacks:/opt/docker/stacks`) so the daemon resolves every relative bind source — without it, all bind-mounted stacks (base infra and customer apps) silently break. **Controller routing + the wildcard cert anchor (`wireController` → `RenderControllerRoute`, v0.41.1 / v0.42.1).** filebrowser self-registers with traefik via Docker labels + `traefik-public` membership baked into its compose; the controller can't (it's started by the golden bootstrap *before* `traefik-public` exists, and the v2 `bootstrap.json` carries no domain — that comes from the hub pull). So `EnsureBaseStack` wires the controller **post-pull**: it `docker network connect traefik-public felhom-controller` and writes a traefik file-provider route `dynamic/controller.yml` (`Host(felhom.) → http://felhom-controller:8080`, write-if-changed). When DNS-01 ACME is configured, that route is **also the wildcard-cert anchor**: its router-level `tls.domains: *.` makes traefik **proactively obtain the wildcard `*.` + apex via Cloudflare DNS-01 at startup** (an entrypoint-level `http.tls.domains` does *not* trigger issuance in traefik v3 — only a router-level `tls.domains` does). Every other router then serves that one real wildcard cert by SNI — no per-app `certresolver` labels. This is what lets a LAN client reach the box directly at `*.` with the real cert (the `felhom-agent` split-horizon resolver depends on it). #### Missing Field Injection (`deploy.go`) When app templates are updated (e.g., a new `APP_KEY` secret is added to `.felhom.yml`), existing deployed apps need the new field in their `app.yaml`. The controller handles this automatically: - **On startup**: `InjectMissingFields()` runs for all deployed stacks - **After sync**: the post-sync hook runs for stacks whose templates were updated - For each deployed stack, compares `.felhom.yml` `deploy_fields` against `app.yaml` env vars - Missing `secret` fields: auto-generated using the field's generator spec (`password:N`, `hex:N`, `base64key:N`) - Missing `domain` fields: filled with the customer's configured domain - Missing `subdomain` fields: filled with the field's default value or the `.felhom.yml` `subdomain:` metadata - Other field types (e.g., `text`, `select`): logged as warning for manual configuration - Locked fields are added to the locked list automatically **Generator types**: `password:N` (alphanumeric), `hex:N` (hex-encoded random bytes), `base64key:N` (`base64:` + N random bytes base64-encoded, for Laravel APP_KEY etc.), `static:VALUE` (literal value). #### Container State Display | State | Color | Label | Meaning | |-------|-------|-------|---------| | Running + healthy | Green | "Fut" | All containers running and healthy | | Running + starting | Orange | "Indulas..." | Healthcheck not yet passed | | Deploying | Orange | "Telepítés..." | Compose up in progress (image pull, container creation) | | Running + unhealthy | Yellow | "Nem egeszseges" | Docker or controller-side healthcheck failing | | Stopped/exited | Red | "Leallitva" | All containers stopped | | Restarting | Yellow | "Ujrainditas..." | Restart loop | | Not deployed | Gray | "Nincs telepitve" | Compose file exists, not deployed | **Route-unpublished indicator (F5, v0.61.0).** Traefik's Docker provider only publishes a route to a container that is healthy (or has no healthcheck), so an `unhealthy`/`restarting` deployed app returns a hard **404** at its URL even though the container is running. The `routeUnpublished` template helper (`funcmap.go`) drives a distinct "URL nem elérhető – útvonal nincs publikálva" indicator on the dashboard and stacks cards for such apps, so a dead URL isn't mistaken for a merely-degraded-but-reachable one. #### Controller-side Health Probes (`internal/stacks/healthprobe.go`) For apps that declare a `healthcheck:` section in `.felhom.yml`, the controller probes the container directly over the Docker network (both are on `traefik-public`). This complements Docker-level healthchecks and is the **only** health mechanism for distroless/scratch images that lack shell utilities. Three probe types are supported: - **`http`** — Any HTTP response (even 4xx/5xx) = service is alive. Only connection refused/timeout = unhealthy. - **`api`** — HTTP request with response validation (expected status code, body content). Fails if expectations aren't met. - **`tcp`** — Simple port reachability check via `net.Dial`. Multiple checks per app are supported (all must pass). The probe scheduler runs every 10 seconds; per-app intervals default to 5 minutes and are configurable via `healthcheck.interval` in `.felhom.yml`. Probe results are stored in `Stack.HealthProbe` and exposed via the API. Failed probes override the stack state to `StateUnhealthy`; the override clears automatically when the next probe passes. **Fast initial probing:** On start/restart, stale health probe results are cleared (so the stack doesn't immediately appear "unhealthy" from a previous result). Until the first healthy probe, the controller checks every 10 seconds instead of the normal 5-minute interval, giving fast feedback on whether the app came up successfully. --- ### 2. App Export/Import (.fab bundles) Per-app export creates a self-contained `.fab` file (tar.gz, optionally encrypted) that can be stored externally or used to restore the app on the same server. Distinct from the automatic backup system — user-initiated, per-app, produces a single portable file. **Bundle contents:** `manifest.json` + `config/` (compose, .felhom.yml, app.yaml with plaintext secrets) + `database/` (gzipped SQL dump) + `data/` (HDD bind mount tars or Docker named volume tars). **Encryption:** Optional AES-256-CTR + HMAC-SHA256 with scrypt key derivation (N=32768). Format: `"FABE"` magic header + salt + IV + encrypted tar.gz + HMAC tag. Streaming for multi-GB files. **Export flow:** Estimate size → check free space → optionally stop app → copy config → dump DB → tar user data → create tar.gz → optionally encrypt → atomic rename. App restarts automatically after export if it was stopped. **Import flow:** Decrypt if needed → extract → prepare stack dir (create new or `compose down --volumes` for existing) → restore config (re-encrypt app.yaml with current server key) → restore user data (HDD or volumes) → restore DB (start DB service, wait for ready, import dump) → start full stack → refresh UI. **Architecture:** `internal/appexport/` package with `ExportStackProvider` adapter interface (same pattern as `backup.StackDataProvider`). `exportAdapter` in `main.go` bridges `stacks.Manager` to the provider. **API endpoints:** `/api/export/estimate`, `/api/export/start`, `/api/export/status`, `/api/export/bundles`, `/api/export/manifest`, `/api/export/import`, `/api/export/import/status`. **UI:** Export button on app info page, standalone import page at `/import` accessible from the stacks page header. --- ### 3. Backup System The backup system implements a **3-2-1 backup architecture**. Each tier is a **complete, self-sufficient backup** — any single tier can fully restore an app. | Tier | Contents | Location | Can fully restore? | |------|----------|----------|--------------------| | **1. Nightly restic** | DB + Config + User data | Same drive as app | Yes (not against drive failure) | | **2. Cross-drive** | DB + Config + User data | Different physical device | Yes | | **3. Remote** | Everything | Cloud / remote server | Future | **Key principles:** - User data backup is **mandatory** — every app with HDD bind mounts is included automatically. There is no per-app toggle. - Each tier includes **everything** needed to restore: DB dumps, config, and user data. No tier depends on another tier's data. - **Tier 2 is configurable for ALL apps** — not just apps with HDD data. Non-HDD apps back up config + DB dumps to the secondary drive (small but protects against drive failure). - The `AppBackupPrefs.Enabled` field in settings.json is legacy and not read by any code. **Per-app Tier 2 contents by app type:** | App type | Tier 2 contents | Example | |----------|----------------|---------| | HDD + DB | Config + DB + User data | Immich, Paperless-ngx | | HDD, no DB | Config + User data | — | | Docker volumes + DB | Config + DB + Volume data | Tandoor | | Docker volumes, no DB | Config + Volume data | Mealie (SQLite) | | DB, no HDD/volumes | Config + DB | Vikunja | | Config only | Config | Gokapi, Homepage | #### Tier 1: Nightly Backup (mandatory, same drive) The nightly backup has two phases that run sequentially. All paths are **per-drive** — each physical drive gets its own restic repo and per-app DB dump directories. **Drive layout (v0.26.0):** ``` / ├── felhom-data/ ← all controller-managed data (namespace, v0.26.0+) │ ├── appdata// ← app user data │ └── backups/ │ ├── primary/ │ │ ├── restic/ ← one restic repo per drive (all apps on this drive) │ │ └── / │ │ ├── db-dumps/ ← per-app DB dump files │ │ └── volume-dumps/ ← per-app Docker volume tars (v0.33.0) │ └── secondary/ │ ├── restic/ ← secondary restic repo (cross-drive) │ ├── _infra/ ← infra config mirror │ └── /rsync/ ← per-app rsync data ├── .felhom-infra-backup/ ← DR marker (stays at drive root for scanner) ├── Dokumentumok/ ← user files (not controller-managed) └── media/ ← user files (not controller-managed) ``` > **Note (Model A — corrected in v0.52.0):** `HDD_PATH` in `app.yaml` is the **in-guest mount point** > (e.g., `/mnt/felhom-usb`). Under slice-10 Model A the host agent binds `/felhom-data` directly > onto that mount, so the in-guest mount **already is** the `felhom-data` namespace root. Neither the > compose templates nor the path helpers add a `felhom-data` segment for a drive-resident app: app data > is `${HDD_PATH}/appdata/` and backups `${HDD_PATH}/backups/...`, **single-nested**. Only the > SSD-only system-data fallback (a bare root, `inGuestDrive=false`) appends `felhom-data`. See > `NamespaceRoot(drivePath, inGuestDrive)` in `internal/appbackup/paths.go`. > Earlier catalog templates used `${HDD_PATH}/felhom-data/appdata/`, which double-nested to > `.../felhom-data/felhom-data/...` on a Model-A drive; v0.52.0 dropped that segment in the catalog and > locks deploy↔backup path agreement with `internal/stacks/hddpath_agreement_test.go`. Path computation is centralized in `backup/paths.go` via the `FelhomDataDir = "felhom-data"` constant: - `PrimaryResticRepoPath(drivePath)` → `/felhom-data/backups/primary/restic/` - `AppDBDumpPath(drivePath, stackName)` → `/felhom-data/backups/primary//db-dumps/` - `AppVolumeDumpPath(drivePath, stackName)` → `/felhom-data/backups/primary//volume-dumps/` - `AppDataDir(drivePath, stackName)` → `/felhom-data/appdata//` - `SecondaryResticRepoPath(drivePath)` → `/felhom-data/backups/secondary/restic/` - `AppSecondaryRsyncPath(drivePath, stackName)` → `/felhom-data/backups/secondary//rsync/` - `SecondaryInfraPath(drivePath)` → `/felhom-data/backups/secondary/_infra/` - `InfraBackupDir(mountPath)` → `/.felhom-infra-backup/` (**unchanged** — stays at drive root for DR scanner) > **⚠️ Stale:** the restic/secondary helpers above (`PrimaryResticRepoPath`, `SecondaryResticRepoPath`, > `AppSecondaryRsyncPath`, `SecondaryInfraPath`) describe the pre-strip layout — restic/cross-drive was > removed in slice 8C. This section is rewritten when Tier 2 (Phase 3) lands. #### Per-app recovery unit (Phase 2, v0.53.x) — SECRET-FREE Each app's `backups/primary//` is a self-contained, recreatable **recovery unit**: ``` backups/primary// ├── compose/ docker-compose.yml + .felhom.yml + a SECRET-STRIPPED app.yaml ├── db-dumps/ app-consistent DB dump(s) ├── volume-dumps/ named-volume tars └── manifest.json image pins, secret env-var NAMES, data_key names, checksums, secret_source ``` - **Secret-free by design.** The unit stores **no secret value, no data-encrypting key, and not the Docker image** — only the pinned image tag(s) (re-pulled on restore) and the *names* of the secret / `data_key` env vars. Rationale: app.yaml + the encryption key live on the guest rootfs → already in the PBS whole-guest snapshot, and the hub is deliberately zero-knowledge. Restore recovers the original secrets from the guest's own app.yaml (live, or via PBS); for a `data_key` app it **fails closed** (refuse + warn) if the key can't be recovered — data-keys are NEVER generated. **Resettable secrets (O4, v0.99.0):** an unrecoverable resettable secret (DB password etc.) gets a **generated replacement** from its catalog `generate` spec (`stacks.GenerateSecretForField` via the `backup.SetSecretGenerator` seam) instead of redeploying blank (which failed compose-up); the new value persists encrypted through the normal `RecreateStackFromUnit` → `SaveAppConfig` path. Fields with no `generate` spec still proceed with a loud "may fail to start" WARN. Residual case: a restored volume tar carrying the OLD internal credential hash may still need a manual in-DB reset. - Helpers: `RecoveryUnitPath` / `RecoveryUnitComposePath` / `RecoveryUnitManifestPath` (`internal/appbackup/paths.go`). Capture: `Manager.CaptureRecoveryUnit` (`internal/backup/recovery_unit.go`), run from the daily DB dump and the periodic `RefreshCache` (idempotent checksum-skip). The non-secret env comes from `StackDataProvider.GetStackRecoveryInfo` (excludes secret-named + encrypted values, so the capture never touches a secret). `data_key` fields are marked in `.felhom.yml` (`DeployField.DataKey`). - **Restore replays the DB dump (F17, v0.61.0).** `RestoreFromRecoveryUnit` (and the `RestoreApp` fallback) stops the app → restores named-volume tars → recreates the compose definition + redeploys with the recovered env → **replays each `db-dumps/*.sql` into the now-running DB** via `backup.reimportDBDumps` → `appbackup.ImportDump` (psql / mariadb client, using the live container's own discovered credentials). The DB replay runs AFTER the volume restore, so the **logical SQL dump wins** over any volume-tar copy of the database (the dumps use DROP/CREATE — `pg_dump --clean --if-exists`, `mariadb-dump` default `--add-drop-table` — so replay is idempotent). Volume-restore and DB-import failures now **surface** (restore returns an error) instead of a swallowed WARN. Prior to v0.61.0 the per-app restore never replayed the `.sql`, so DB-resident data did not come back. #### Tier 2 — off-drive copy (Phase 3, v0.55.x) For every HDD app, Tier 2 (`internal/backup/tier2.go`) rsync-mirrors the recovery unit (`backups/primary//`) + the app's `appdata//` to `/backups/secondary//` on a **different physical disk** — the only off-drive protection bind-mounted HDD userdata can get (PBS can't reach bind mounts). Auto-targeted: **prefer another registered user-data drive** (off-disk via `system.SamePhysicalDevice`); else the **internal SSD for small units only**, behind a size-aware **rootfs-headroom guard** (`tier2FitsHeadroom`) that **refuses rather than fills** the ~8 GB guest rootfs (reserve = `max(2 GB, 20%)`), recording an honest "needs a 2nd HDD" status. Status persists via `settings.CrossDriveBackup` and drives the "2. mentés" card. Runs daily (`tier2-backup`, 03:30) or via `POST /api/backup/tier2`. restic is **not** used — a plain browsable mirror. **In-place file restore from the Tier-2 copy (C2, v0.100.0 — closes drill finding F2)** — `POST /backup/tier2/restore` (`backup.RestoreTier2Files`, `internal/backup/tier2_restore.go`) + the **"Fájlok visszaállítása"** button on the healthy Tier-2 layer row. **Additive-only** semantics (`rsyncRestoreMissing`: `rsync -a --ignore-existing`): files missing from the live `appdata/` are copied back from the RECORDED Tier-2 copy; existing live files are **never overwritten** (a customer edit after the last copy wins) and **nothing is ever deleted** — this exactly serves the "I deleted my files" scenario with zero risk to newer data. Source = the recorded `CrossDriveBackup.DestinationPath` (never a fresh target selection). Single-flight with backup/restore; refusals (no copy / never ran / copy dir gone / either drive disconnected / decommissioned) happen before the app is stopped, with customer-readable Hungarian reasons; stop → copy → start → health-wait; zero files copied is a success ("Nincs hiányzó fájl…"). Out of scope by design: overwrite/point-in-time restore (offbox + operator paths), per-file selection, `recovery-unit/`. Apps that index their data dir (e.g. Nextcloud) may need a rescan (occ files:scan) before restored files appear in their own UI. **Per-app Tier-2 config panel (v0.57.0)** — `GET/POST /stacks/{name}/backup` (`internal/web/tier2_config_handler.go` + `templates/tier2_config.html`). The "2. mentés" row's **Beállítás** button links here (was the dead-end deploy page). Shows the effective off-drive target (pinned or auto), whether it's the size-limited internal SSD, the last-run reason, and lets the customer **pin a registered drive** (off physical disk) or **toggle Tier 2 off**. Always visible — single-SSD apps get the "csak DB/konfiguráció" note, non-HDD apps the "already in the PBS whole-guest snapshot" context. Two preference fields on `CrossDriveBackup` — `UserDisabled` + `PreferredTarget` (set via `Settings.SetTier2Preference`) — are **preserved across the runner's status writes** (`withTier2Prefs`): `selectTier2Target` honors a valid pin before auto-picking; `RunTier2` skips a disabled app. The runner re-validates the pin off-disk at run time. `Manager.Tier2Info(stackName)` is the read-only panel view (effective target + eligible alternative drives). **Phase 1 — Database Dumps** (`internal/backup/dbdump.go`, scheduled 02:30) - **Auto-discovery** of PostgreSQL and MariaDB containers via `docker ps` + `docker inspect` - Dumps via `docker exec pg_dump` / `docker exec mariadb-dump` with 5-minute timeout - Dumps are written to the app's **home drive**: `AppDBDumpPath(appDrive, stackName)` - Atomic writes (`.tmp` → `.sql`) to prevent corruption - **Validation** after each dump: checks file size, header presence, counts `CREATE TABLE` - Results cached in `settings.json` surviving container restarts **Phase 1b — Docker Volume Dumps** (`internal/backup/backup.go` `runVolumeDumps`, part of the same run) - **F3 (v0.99.0):** re-wired into the nightly/manual app-data backup run (`runDBDumpsInternal`) — after the restic removal `DumpAppVolumesSafe` had no caller, so `volume-dumps/` was never produced. Runs AFTER the DB dumps and BEFORE `captureAllRecoveryUnits` so the manifests enumerate fresh tars. - Gate order (load-bearing): protected-stack (`cfg.IsProtectedStack`) and has-volumes (`GetDockerVolumes()`) checks come BEFORE `DumpAppVolumesSafe` — the Safe variant stops the stack before its own volume check, so unconditional calls would bounce every volume-less app nightly. Disconnected/decommissioned drives skip with the same summary style as the DB loop. - Each volume-bearing stack is stopped before dump, restarted after (`DumpAppVolumesSafe()`) — prevents inconsistent tars of live databases. - For each volume: `docker run --rm -v :/vol:ro -v :/out alpine tar cf /out/.tar -C /vol .` - 10-minute timeout per volume; a per-stack failure lands in the run summary as `FAIL volumes:`, flips the run's Success flag and fails the run (no silent partial) — other stacks still proceed - Stale tars cleaned up (volumes that no longer exist) - Volume names resolved with project prefix via `ResolveDockerVolumeNames()` (e.g., `mealie_mealie_data`) - Dumps written to `AppVolumeDumpPath(nsRoot, stackName)` **Phase 2 — Restic Snapshot** (`internal/backup/restic.go`, scheduled 03:00) - Apps are **grouped by drive** via `groupStacksByDrive()` — each drive's apps are backed up to that drive's restic repo - App drive resolution: `GetStackHDDPath()` (from `StackDataProvider`) → falls back to `SystemDataPath` - Auto-generated repository password (32 random bytes, base64url), shared across all repos, synced to hub - **Paths included in each per-drive snapshot (v0.34.0: per-app scoped):** - Per-app DB dump dirs on that drive - Per-app Docker volume dump dirs (`volume-dumps/*.tar`) - Per-app HDD mount paths (user data) - Per-app stack config dir (`//` — only for stacks on this drive) - `controller.yaml` — only on the system drive (not duplicated across all drives) - Auto-detects and unlocks stale locks (restic repo lock) - Weekly prune on Sundays with configurable retention (keep-daily, keep-weekly, keep-monthly) - Weekly integrity check (`restic check`) on Sunday 04:00 — checks **all** primary repos **Protects against:** accidental deletion, data corruption, point-in-time rollback. Does NOT protect against drive failure (backup is on the same physical drive). #### Tier 2: Cross-Drive Backup (opt-in, different device) (`internal/backup/crossdrive.go`) **Complete backup** to a different physical drive. Available for **all apps** — apps with HDD data back up config + DB + user data + Docker volumes; apps without HDD back up config + DB dumps + Docker volumes. - **Auto-enable for small apps (v0.14.1):** Apps without HDD mounts (config-only, DB-only) are automatically configured for daily rsync Tier 2 when ≥2 storage paths are registered. `AutoEnableSmallApps()` runs at the start of each nightly backup cycle. Never overwrites existing user-configured cross-drive settings (even disabled ones). - **Infrastructure config backup (v0.14.1):** `syncInfraConfig()` rsyncs the stacks directory and `controller.yaml` to `/backups/secondary/_infra/` on every secondary destination drive. Runs before per-app backups. Cross-drive restic also includes infra paths. - **Two methods:** - **rsync** — Simple mirror with `--delete` (fast, no versioning, **browsable** on disk) - **restic** — Versioned, deduplicated, encrypted (shared repo across apps, not browsable) - Per-app configuration in settings.json: destination path, method, schedule (daily/weekly/manual) - **Pre-backup DB dump:** `DumpStackDB()` runs fresh pg_dump/mariadb-dump before each cross-drive backup; non-fatal on failure (wired via `DBDumper` interface to avoid circular imports) - **Pre-backup volume dump (v0.33.0, safe stop/start v0.34.0):** `DumpAppVolumesSafe()` stops the stack, exports Docker named volumes to tar, restarts — wired via `VolumeDumper` interface - **Empty mounts allowed:** `RunAppBackup` accepts apps with no HDD mounts — the rsync mount loop simply doesn't execute, but DB + config copy still runs - **Drive-type-aware validation** (`ValidateDestination`): | Destination type | Space checks | |-----------------|--------------| | External mount (different device than `/`) | Block if <100 MB free | | System drive (same device as `/`) | Require ≥10 GB free AND <90% used; logged warning | - **Secondary drive layout (v0.14.1):** ``` /backups/secondary/ ├── _infra/ ← infrastructure config mirror (v0.14.1) │ ├── controller.yaml │ └── stacks/ ← full stacks dir (all app configs) ├── /rsync/ ← per-app rsync mirror │ ├── _db/ ← DB dump files │ ├── _config/ ← compose.yml, app.yaml, .felhom.yml │ ├── _volumes/ ← Docker volume tars (v0.33.0) │ └── ← HDD mount contents (if app has HDD data) └── restic/ ← shared restic repo (all cross-drive apps) ``` - DB dump files read from **per-app home drive** path (`AppDBDumpPath`) - `_` prefix directories prevent collision with user data - For non-HDD apps, only `_db/`, `_config/`, and `_volumes/` (if applicable) are present (no user data directory) - **Restic backup paths:** includes HDD mounts (if any) + config dir + per-app DB dump dir from home drive + stacks dir + controller.yaml (infra, v0.14.1) - Safety guards: destination ≠ source, path-overlap check (HDD mounts only), writable check - **Chained execution:** runs immediately after nightly restic — daily apps every night, weekly apps on Sundays - **Hub reporting after manual triggers (v0.27.2):** `OnCrossDriveComplete` callback on Router pushes infra backup snapshot to Hub + writes local infra backup after both single-app and run-all manual triggers complete (previously only automatic scheduled runs reported) - Per-app concurrency lock prevents overlapping runs - Status (last_run, duration, size, error) persisted to settings.json **Protects against:** primary drive failure, drive theft/damage. #### Tier 3: Remote Backup (future) Complete offsite backup for disaster recovery. Not yet implemented. Placeholder shown in UI ("3. mentés — Hamarosan"). #### Restore (`internal/backup/restore.go`) Both **Tier 1** (restic) and **Tier 2** (rsync) restores are supported. All deployed apps appear in the restore dropdown with per-app snapshot filtering. | App type | Config restored | DB restored | User data restored | Docker volumes restored | |----------|----------------|------------|-------------------|------------------------| | Has HDD data | Yes | Yes | Yes (always) | Yes (if present) | | Docker volumes, no HDD | Yes | Yes | n/a | Yes | | DB only, no HDD/volumes | Yes | Yes | n/a | n/a | | Config only | Yes | — | n/a | n/a | **Snapshot API** (`GET /api/backup/snapshots?stack=` — F1, v0.99.0): - Backed by `backup.Manager.ListRestorePoints` (`internal/backup/restore_points.go`). The keep-side restore has exactly **one** restore point per app — the current recovery unit — so the endpoint returns at most one entry: `time` = newest artifact mtime (manifest / db-dumps / volume-dumps), `short_id:"helyi"`, `tier:1`, `drive_label` from the storage registry (empty on the SSD fallback) - **Never emits tier-2 entries**: Tier-2 copies are not restorable via `POST /backup/restore` (it only reads the primary unit) — listing them would silently restore tier-1 data while claiming tier-2 - Guards: empty/traversal stack name → 400 (`validStackParam`), unknown stack → 404, known stack with no unit yet → `ok:true, data:[]` (the UI shows "Nincs elérhető mentés") - History: the route was a restic-era leftover fetched by the template but unregistered — the dropdown could never populate and the restore button never enabled (drill finding F1) **Restore type info** shown per-app when selected in dropdown (Hungarian banners): - Has HDD or Docker volumes: "Teljes visszaallitas: adatbazis + konfiguracio + felhasznaloi adatok" - Has DB, no user data: "Adatbazis es konfiguracio visszaallitasa" - Config only: "Csak konfiguracio visszaallitasa" **Tier 1 restore** (`RestoreApp`): - Stop app → resolve app's home drive → `restic restore --target / --include ...` → populate Docker volumes from restored tars → restart app → health check - Restore paths: config dir, DB dump dir, volume dump dir, HDD mounts - Docker volumes restored via `restoreDockerVolumes()`: `docker volume rm -f` → `docker volume create` → `docker run alpine tar xf` **Tier 2 restore** (`RestoreAppFromTier2`): - Stop app → rsync config from `_config/` → rsync HDD data (single/multi-mount) → copy DB dumps from `_db/` (streaming `copyFile`) → restore Docker volumes from `_volumes/` tars → restart app → health check - Uses rsync `--delete` for config and HDD data to ensure exact mirror state - Single-mount apps: data directly in rsync dir (excluding `_*`); multi-mount: per-leaf subdirectories **Common:** - **v0.34.0:** Post-restore health check (`waitForHealthy`) polls container state with `docker ps` refresh every 5s for up to 90s. Warning logged if app doesn't reach running state; restore still returns success (data is restored regardless). - Running flag prevents concurrent backup/restore operations - Snapshot ID validated (8-64 lowercase hex, or special `tier2-rsync`) - Import from `.fab` bundle link shown in restore section for cross-system migration #### Backup Page UI (`internal/web/templates/backups.html`) Unified per-app status table with expandable rows showing **per-tier** backup status: **Status dot per app:** | Dot color | Meaning | |-----------|---------| | Green | 2+ tiers configured with successful backups + destination healthy | | Yellow | Only 1 tier, or Tier 2 failing, or Tier 2 configured but never run, or destination disconnected/inactive | | Red | Tier 2 destination blocked or inaccessible | Every app starts as yellow (1 tier only). Green requires Tier 2 configured with successful backup. **Per-app backup tiers (3 rows per app):** - **1. mentes** (Tier 1, always present) — Auto badge + "helyi" + last run + contents (e.g., "DB + Konfig + Adatok") - **2. mentes** (Tier 2, configurable for ALL apps) — one of: - Configured: method (rsync/restic) + destination + schedule + last run + status + contents + browsable indicator (folder icon for rsync) + action buttons - Not configured: "1. mentes auto" + "Nincs 2. masolat" + settings link - **3. mentes** (Tier 3, placeholder) — grayed out "Hamarosan" + "tavoli (offsite)" + future note **Backup contents per app** (shown per tier): - Apps with DB + HDD: "DB + Konfig + Adatok" - Apps with Docker volumes (no HDD): "Konfig + DB + Adatok" or "Konfig + Adatok" - Apps with DB only: "DB + Konfig" - Apps with HDD, no DB: "Konfig + Adatok" - Apps with neither: "Konfig" **Deploy page** shows cross-drive (Tier 2) configuration form for **all deployed apps**, not just those with HDD data. Non-HDD apps can configure destination, method, and schedule. **Other sections:** - Schedule overview with next run times for DB dump, restic, prune - Snapshot history table (last 20 snapshots aggregated from all per-drive repos, sorted by time) - Storage overview card (total size across repos, snapshot count, DB dump count/size, encryption key with show/copy) - Restore section: app dropdown → per-app snapshot dropdown (Tier 1 + Tier 2 grouped) → restore type info → confirmation checkbox → execute → import from `.fab` bundle link --- ### 4. Storage Management > **Off-box (NAS) backup — restic-over-SFTP (v0.93.0, Part B).** The "1 off-site" leg of 3-2-1 for the > app-data tier: each off-box-toggled app's recovery unit + DB dumps + volume tars are backed up to the > customer's NAS as an **encrypted restic repo over SFTP** (`internal/backup/offbox.go`). No kernel mount — > restic talks SFTP directly; the NAS sees only ciphertext. Distinct from the local cross-drive rsync copy > and the agent's PBS whole-CT DR. > - **Fail-fast (load-bearing):** every restic call uses `-o sftp.command="ssh … -oConnectTimeout=10 … -s > sftp"` so a dead NAS errors in ~10 s, never hangs the backup runner; a failure raises the `backup_failed` > operator alert. `-oStrictHostKeyChecking=yes` + a pinned `known_hosts` (no blind TOFU). > - init-if-absent (idempotent), `forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune`, > single-flight (shares `m.running`) + migration-guard, restic's own exit code checked, restore to a > scratch dir (non-destructive). > - **Unit discovery (v0.104.0) — durable, deployment-independent.** Each toggled app's recovery unit is > **discovered** by scanning the durable storage registry — every registered *schedulable* storage path > (`GetSchedulableStoragePaths`) ∪ the system-data fallback drive — for `backups/primary/`, rather > than *inferring* the drive from the app's live `app.yaml` `HDD_PATH` (which silently fell back to > `systemDataPath` for a toggled-but-undeployed app → offbox looked on the wrong drive, backed up > nothing, and reported `ok`/0). If the same app's unit exists on two drives (drive churn), the **newest > by manifest `CreatedAt`** is used and the stale one is WARN-logged. The WRITE side (`CaptureRecoveryUnit` > / dumps) is unchanged — this only changes offbox's read/discovery path. Boundary: decommissioned or > non-schedulable drives are not searched. > - **No silent success (v0.104.0):** a run where ≥1 app is toggled but **0** were backed up (no unit found > anywhere) is a **hard error** → `LastStatus="error"` + operator alert (was a misleading `ok`/0 snapshots). > A *partial* run (some units missing) stays `ok` but sets a Hungarian **`LastWarning`** naming the skipped > apps, shown on `/backups`. > - **Secrets** (SSH key + auto-gen repo password) are **0600 files in the data dir** — never logged/committed. > - **Password custody + atomicity (v0.105.0, fork-4; pairs with agent v0.77.0).** The repo password is the > irreplaceable DATA key for the offsite tier, so it rides the **customer-recovery-code (R) escrow** > (age-under-R, in the agent's `IdentityBundle` — custody spike validated a recovered password opens the > real repo). **Atomicity gate:** enabling offsite pushes the password to the agent (`StageEscrowSecret` → > `POST /escrow/stage-secret`) and marks `EscrowState="pending"`; **no offsite backup runs until the escrow > is confirmed** (`OffboxRunnable()`), so an un-recoverable offsite copy can never exist. > **Operator ceremony (pilot, supervised):** enable offsite (→ pending) → run > `felhom-agent --selftest=escrow-create --identity-bundle --upload` (the bundle now carries > the staged restic password, auto-injected, then wiped) → hand the customer the fresh **R** (once; > supersedes any prior code) → `POST /backup/offbox/confirm-escrow` (or the "Letét megerősítése" button). > **DR:** recover R → the escrow yields the password → `POST /backup/offbox/inject-password {password}` > pre-places it 0600 → configure offbox → restore. The SFTP access key is **regenerated** at DR (a fresh > sub-account key), NOT escrowed; the DR recipe carries only the non-secret `offsite_restic` > coordinates (`DRResticCoord`). *(A hub-verified auto-confirm + customer-self-serve enable + fresh-R UX > are the provisioning task, deferred.)* > - **Injection guard** (`ValidateOffboxTarget`): host/user/repo must not start with `-` (ssh > option-injection) or carry metacharacters/traversal; `OffboxConfigured` fails closed on an invalid > target. Image: `restic` + `openssh-client` (re-added; restic's sftp backend shells out to `ssh`). > - UI: the "Külső (NAS) mentés" section on the backups page (configure target, per-app toggles, run-now, > restore, status). Config: `settings.OffboxTarget` + per-app `AppBackupPrefs.Offbox`. Daily at 04:15. > **NAS network storage (v0.92.0, Part A2; pairs with agent v0.50.0).** A customer NAS share (NFS or SMB) > is a **distinct storage KIND** from a physical drive (`StoragePath.Kind == "network"`), for **bulk media**. > The controller is a thin proxy over the agent's `/netstorage/*` (A1) + the local registry — it holds **no > mount authority** and **never persists the SMB password** (it passes the credential to the agent's add > request, which writes the 0600 file). Endpoints `POST /api/storage/netstorage/{add,remove}` + > `GET /api/storage/netstorage` (merges the agent's per-share liveness: `ok` | `idle` (benign idle-unmount) > | `unreachable` | `unknown`). A network share registers as `/mnt/felhom-drives/`, is **selectable as > a media app's `HDD_PATH`** (Schedulable), and shows in the "Hálózati tárhely (NAS)" settings section. > - **NOT a drive:** the drive lifecycle (eject/decommission/migrate/wipe/SMART) is **refused** on a network > path server-side (`refuseNetworkLifecycle`). The **drive-absent gate** (`planDriveGates`) and the > **missing-storage** surface **skip network paths**, so an `unreachable` NAS is a **recoverable warning** > (`networkStorageWarnings` → a distinct app-card badge), **never** the drive "missing → stopped" cascade. > - **Limits (v1):** a share's `+100000` uid mapping is fixed at add-time (one app / same-uid apps); for > write apps on a soft NFS mount, an in-flight file can truncate if the NAS vanishes mid-write (prefer > atomic-write apps / SSD-staging). > **⚠️ INTERMEDIARY-MOUNT model (v0.67.x, pairs with agent v0.35.x).** External data drives are now > visible in the guest at a STABLE path **`/mnt/felhom-drives/`**, NOT the raw `/mnt/`. The > agent keeps a single permanent parent bind `/mnt/felhom-drives` in the guest and swaps each drive's > `felhom-data` namespace **underneath it host-side** (`mount --bind`), so the guest sees attach/detach > **live with no reboot** (mount propagation), the bind source never disappears (C1-immune), and only > `felhom-data` crosses in (confinement). The per-drive `pct set -mpN` bind is **deprecated**. > - The registered storage path + every app's **`HDD_PATH`** + the FileBrowser source = the stable > `/mnt/felhom-drives/`. The controller maps it back to the raw `/mnt/` (`agentWhere()`) > only for agent calls (assign/attach/eject/decommission). `GET /api/disks` carries `guest_path` + > `bound_under_parent` (the guest-visible signal). > - **Drive-absent gate** (`internal/web/intermediary.go`, `driveGateLoop` 30s): an absent > `/mnt/felhom-drives/` drive stops + blocks its apps (`StoppedStacks` = the gate-stopped set); > a returned drive re-attaches + auto-restarts them; `actionStack` refuses to start an app whose drive > is absent ("tárhely nem elérhető"). SSD/system paths are never gated. > - **H1 endpoints** `POST /api/storage/{disconnect,reconnect,restart-apps}` drive the host-side > eject/reconnect (no guest reboot). > - **Lifecycle (v0.68.x):** a "Leszerelés" button decommissions a drive (migrate-then-decommission OR > decommission-anyway type-to-confirm — non-destructive, never touches the parent mp); a > "Visszacsatlakoztatás" button one-click re-enrolls a decommissioned/ejected drive (clears the marker, > re-binds under the parent, restarts gate-stopped apps). Decommissioning the DEFAULT auto-promotes > another schedulable drive (`defaultPromotionTarget`), or BLOCKS if it's the only one. Eject and > decommission keep the RAW drive mounted (logical retire) so re-enroll re-binds it. > - **Guest-reboot convergence is DETERMINISTIC** via the agent's `guest_boot_id`: the controller persists > `LastGuestBootID` and, when it changes, recreates EVERY deployed drive-backed app onto the > re-propagated drive (`processGuestBootChange` — no fragile container-uptime sampling). > **v0.71.0 — boot-race fix:** on a guest `pct reboot`, in-guest dockerd auto-starts the apps ~18s > BEFORE the agent re-binds the drive, so their volume bind fails at create-time > (`mkdir …/userdata: permission denied`, `RestartCount=0` → never retried → stuck `Exited`). The old > recovery sampled the agent's `BoundUnderParent` ONCE, raced that rebind, recreated nothing, and burned > its boot-id one-shot. `processGuestBootChange` now **gates on the REAL live in-guest bind** > (`driveBindLive`: is `/mnt/felhom-drives/` an actual mountpoint in the controller's own `/mnt` > rslave `/proc/self/mountinfo`?) and **waits** for it (`pollLiveBinds`, bounded ~120s) before recreating > — including apps stuck `Exited` with a create-time mount failure (`shouldRecreateOnBoot` is > state-independent). The **guest-only reboot path** (which the host-reboot sweep never exercised) is now > covered; drives that never go live in the window are left to the drive-absent gate. `processGuestBootChange` > also runs on every periodic `driveGateLoop` tick now (idempotent, boot-id gated) so a momentarily-unreachable > agent right after a guest reboot no longer permanently strands recovery. > **Agent-path prerequisite (also v0.71.0):** the whole drive gate needs `cfg.LocalAPI.Endpoint` (the > per-guest agent local API). `bootstrap.MaybeIngest` now calls `ensureLocalAPI` on the already-configured > path — merging `local_api` from `bootstrap.json` into an existing controller.yaml that lacks it (seeded > before `local_api` existed) — because without it `agentClient()` returns "agent not configured" and the > entire gate + boot recovery silently die. > **v0.72.0 — FileBrowser convergence on boot-recreate:** FileBrowser is base-infra (it binds each > drive's `userdata` but has no `HDD_PATH`, so it is NOT in the drive-backed recreate set) — after a > host reboot its mounts could be stale (the early first-boot bring-up bound them before the drives went > live). `processGuestBootChange` now, **after** `pollLiveBinds` confirms the binds and the apps are > recreated, triggers `go s.SyncFileBrowserMounts()` so FileBrowser converges against the now-live drives. > The sync runs unconditionally (FileBrowser reflects the current bind state even if no app needed > recreating). The recreate loop is a pure `recreateDriveBackedApps(stacks, present, recreate, syncFB)` > that calls `syncFB` exactly once, after every recreate. Live-accepted over two real `felhom-pve` > reboots (FileBrowser non-stale, all drive-backed apps recovered, agent tolerated a `/dev/sdX` swap by UUID). > > **⚠️ Rebuilt on the agent-delegated disk model (v0.43.0), made ROLE-AWARE in v0.44.0, UX-polished in > v0.45.0.** After the 8C > de-privileging, the controller holds **no Proxmox/disk credentials and no destructive authority** — disk > execution + the gate live entirely in the **host agent**. The drive UI is driven by the agent's > authoritative **role** (`system` | `backup` | `user-data`, from `GET /api/disks`): the appliance's own > system storage and the backup safety-net are visibly **protected** (lock badge, NO destructive controls); > the customer manages their own **user-data** drives with informed consent. The agent re-enforces role at > wipe time — the UI lockout is defense-in-depth, not the gate. > - **Overview** (`settings.html` ← `GET /api/disks`): styled **cards** (not a table) — name, mono > device/mount, badges for class (gyors/lassú), data (`Adatot tartalmaz`), **role** (🔒 Rendszer / 🔒 > Biztonsági mentés — védett / Felhasználói adat) and registered state, plus a **capacity bar** (the > monitoring `system-bar`, from the agent's `total_bytes`/`used_bytes`). Eject/Wipe render **only** for > user-data drives mounted under `/mnt`. > - **(v0.45.0) Deterministic order** — `agentDisksListHandler` sorts the list server-side > (`sortDisksForView`): **user-data → system → backup** (then unrecognized), alpha by name within a > tier, so it no longer reorders on each reload (the agent's view iterates an unordered Go map). > - **(v0.45.0) Purpose + app-backing clarity (B4)** — `local` and `local-lvm` are both shown (not > collapsed); each card carries a plain-Hungarian **purpose description** keyed on the agent's > role/type, the app-backing storages are tagged (`local-lvm` → "Alkalmazás-rendszer"; user-data → > "Alkalmazás-adatok"), and a one-line tiering note above the list answers "which storage do the > apps use?". Pure presentation — role/type stay authoritative from the agent. > - **(v0.45.0) Register shortcut (B3)** — a mounted, **unregistered** user-data drive offers > **Regisztrálás** as its PRIMARY action: `POST /api/storage/register` → `registerStoragePath` records > the existing mount (no format, no eject) + FileBrowser-syncs. Leválasztás/Törlés stay secondary. > - **Customer wipe/eject** — a **type-to-confirm** modal that names the deployed apps that break > (`GET /api/storage/impact` → `appsUsingPath`) and disables the destructive button until the **mount > name is typed exactly**. Wipe (`POST /api/storage/wipe`): eject (unmount + deregister) → server-side > two-step customer-confirmed format (learn the agent's durable id, then re-submit `confirmed:true` bound > to it). The agent refuses a protected device regardless of what the controller sends. > - **Guided init** (`/settings/storage/init`, `POST /api/storage/init`, `web/storage_handlers.go`): format > → resolve the new fs UUID → `assign` → register. The selector lists **only user-data** targets. A > data-bearing user-data device now uses the **customer-confirmation** flow (type-to-confirm → re-submit > `confirmed:true` + durable id), NOT the `felhom-opsign` command. The opsign surface remains a fallback > only if a protected device somehow reaches init. > - **Guided attach** (`/settings/storage/attach`, `POST /api/storage/attach`): non-destructive — resolve > the existing fs UUID → `assign` → register. Selector restyled to cards (user-data only). > - **Eject** (`POST /api/storage/eject`): benign unmount + deregister, with the agent's dependent-guest > warning + the affected-app list (parity with wipe). **The eject is ROLE-GATED at the agent** (felhom- > agent v0.24.0): `POST /disks/eject` refuses to unmount a system/backup mount — the UI hiding the button > is defense-in-depth, not the control. Only user-data mounts are ejectable. > - **`agentapi`** (`internal/agentapi`) is the pinned client to the agent local API: `Disks`/`AssignDisk`/ > `EjectDisk`/`FormatDisk(…, confirmed, durableID)`; `DiskInfo.role`+capacity; > `FormatResult.{role,needs_confirmation,durable_id}`; `ErrNeedsConfirmation` (user-data) vs > `ErrFormatRefused` (system/backup). `FormatResult.PendingOp.OpsignCommand()` for the operator path. > - **(v0.74.0) Client lifecycle — ONE shared client, reused.** `Server.agentClient()` builds the > `agentapi.Client` once (memoized via `sync.Once`) and returns the same instance to all ~19 call > sites; the `http.Transport` is bounded + expiring (`MaxIdleConnsPerHost:2`, `IdleConnTimeout:90s`). > This replaced a per-call `agentapi.New(...)` that leaked one idle ESTABLISHED socket per call and > exhausted the ephemeral source-port range to the agent's `:8443` after ~5 days (EADDRNOTAVAIL). > Safe because `cfg.LocalAPI` is static per process (a config-apply triggers a graceful self-restart). > - **(v0.90.0) Channel health-check — `internal/channelhealth`.** A ~60s scheduler job > (`agent-channel-health`) probes the channel via `Server.ProbeAgentChannel` (the SAME memoized > client + `GET /storage` — not a fresh client) and classifies failures into > `up | down:` (pin_mismatch / unauthorized / unreachable / timeout / misconfigured / > construction_error). Transient reasons (refused/timeout) are **debounced** (N≥2 consecutive) > so a clean agent restart's ~1s blip doesn't page; pin/401/DNS/construction alert on the first > down. On a transition it fires an English **operator-only** event > (`Notifier.NotifyAgentChannelDown/Recovered`, hub 1h cooldown) and sets a Hungarian dashboard > banner (`AlertManager.SetAgentChannelAlert`). The first observation seeds state silently. This is > the controller half of the self-health story (the agent watches its own privileged capabilities; > the controller watches its link to the agent). A construction error (`agentClient()` can't build > — a latching config fault) is surfaced distinctly. Detection/surfacing only — it never touches the > pin, transport, or gate. > - The **`StoragePath` registry** (`settings.go`: `AddStoragePath`/default/schedulable/label) is unchanged. > - **(v0.64.0) `AutoDiscoverStoragePaths` is now ADDITIVE** — it no longer bails when the registry is > non-empty; instead it registers only deployed-app paths missing from the registry. It never removes > or mutates an existing entry, never re-adds or reactivates a path already present in ANY state > (incl. `Decommissioned`), and never flips `IsDefault` (a new path defaults only if the registry has > no default at all). This is NOT auto-register-on-attach (that recommendation was rejected — manual > enrollment is by design); it only picks up drives that deployed apps already reference. > - **(v0.64.0) `InferStorageLabel` disambiguates the internal SSD** — a path whose basename is the > `felhom-data` namespace dir (the internal system volume, e.g. `/mnt/sys_drive/felhom-data`) now reads > **"Belső SSD (rendszer)"** instead of the colliding "Tárhely (felhom-data)". Model-A user drives > register their MOUNT ROOT (e.g. `/mnt/felhom-usb`), never `.../felhom-data`, so this can't mislabel a > user drive. Still overridable via `SetStorageLabel`. > - **(v0.65.0) Data migration** (`internal/stacks/migrate.go`) — move app data between drives, in-process > over the controller's `/mnt:/mnt:rslave` RW mount; crash-safe + resumable via a journal > (`/migration.json`). `MigrateAll(source,target)` moves the whole felhom-data namespace (every > app + a conflict-merge walk for non-app/customer content); `MigrateApp(app,target)` moves one app's > subtree (drive→drive AND SSD→drive). Pipeline: validate → stop → copy (`rsync -a --checksum`, additive, > **no `--delete`**) → verify (`rsync -ani --checksum`, zero pending) → flip+redeploy (`RedeployFromEnv`) > → cleanup. **CLEANUP — the only destructive step — is gated on every unit verified AND every app > redeployed.** Conflict-merge: skip-identical (checksum vs target + its `(N)` siblings), rename-on-differ > to lowest-free `(N)`, never overwrite; idempotent. Single-flight + mutually exclusive with the > backup orchestrator. UI: `POST /api/storage/migrate{,-app}` + `GET /api/storage/migrate/status` (poll); > migrate-all on the settings page, per-app on the app-info page, shared Hungarian progress panel. > - **(v0.65.0) Self-serve decommission** (`handleStorageDecommission`, `POST /api/storage/decommission`) — > retire a drive, non-destructive (never formats). Two choices (no partial): migrate-all-then-decommission > (runs `MigrateAll`; the migration done-hook soft-marks the source + calls the agent on success), or > decommission-anyway (type-to-confirm; stops the apps, keeps their `HDD_PATH` so they show "Hiányzó > tárhely"). Both end at `settings.SetDecommissioned` (soft marker retained — blocks A1 resurrection) + > `agentapi.Decommission` (agent v0.32.0: `POST /disks/decommission`, role-gated user-data, intent + > bind-prune + unmount). Re-enrolling a decommissioned drive (`registerStoragePath`) clears the marker + > restores `Schedulable`. A deployed app whose drive is decommissioned/disconnected/absent shows the > **"Hiányzó tárhely"** warning badge on the dashboard/stacks/app card. > > - **(v0.66.0) Userdata layout + shared-storage ownership convention** (`internal/appbackup/userdata.go`). > Each drive's felhom-data namespace gains a customer-facing **`userdata/`** tree (sibling of > `appdata/`/`backups/`) — the ONLY thing FileBrowser mounts. Skeleton: > `media/{movies,tv,music,audiobooks,books,comics,photos}`, `downloads`, `import/{paperless,calibre}`, > `roms`, `documents`. **Convention:** every userdata dir is group-owned by `SharedContentGID` (1000), > mode **2775** (setgid + group-rwx) — `EnsureUserdataDir` = MkdirAll → explicit `Chmod(ModeSetgid|0775)` > (MkdirAll's mode is umask-masked AND drops setgid) → chown group 1000. Setgid makes new files inherit > group 1000 so FileBrowser (uid 1000) + the content apps collaborate without permission collisions. > - **`USERDATA_PATH`** = `/userdata` is injected into the compose env (the shared > `withUserdataPath`, used by BOTH `stackEnv` and `composeExecWithEnv` — the initial deploy builds env > from values, not app.yaml). Catalog media mounts use `${USERDATA_PATH}/...`. > - **Pre-create**: the full skeleton is created with the convention on `registerStoragePath` + > `syncFileBrowserMounts` (system + additional drives); a **deploy belt** in `composeExecCustomEnv` > (on `up`) pre-creates every `${USERDATA_PATH}/...` bind source the app declares, so Docker never > auto-creates a userdata dir as guest-root (covers apps not in the skeleton). > - **FileBrowser** mounts `/userdata` (was `appdata`) and runs as uid 1000 → it can create > folders + upload into the 2775 setgid tree (fixes the old permission-denied); app internals > (`appdata/`) are no longer browsable. **(v0.66.2)** its entrypoint is wrapped > `["sh","-c","umask 002; exec /home/filebrowser/filebrowser"]` so folders the customer creates come > out **2775 (group-writable)** — the gtstef image is a single Go binary that ignores a `UMASK` env > (verified), so the wrapper is the mechanism. Without it, customer folders were 2755 (group-read > only) and apps in group 1000 couldn't write into them. > - **Run-identity**: PUID/PGID-1000 apps (radarr/sonarr/calibre with `UMASK=002`) write **group- > writable**, so FileBrowser can fully manage their output. Root-only apps (jellyfin :ro; komga + > audiobookshelf RW after the `user:1000` fallback) write **group-readable** via setgid (FileBrowser > browses/reads, the app manages its own files). > - **Migration-safe**: `migrate.go`'s merge walk preserves the source dir's setgid + group (and > `copyFile` the full file mode + group), so the convention survives a whole-drive `MigrateAll`. > > The privileged controller-side disk subsections **below are historical** (the `internal/storage/*` scan/ > format code was removed in 8C — execution is the agent's now). The storage subsystem handles the full lifecycle of external storage: detection, initialization, path registration, and data migration. > **CURRENT (post-de-privileging + Impl-2b, v0.95.0):** the in-guest storage code below > (`internal/storage/scan.go`, `format.go`, `attach.go`) is **retired** — all disk ops are delegated to > the host agent via `internal/agentapi`. The two enrollment wizards (`/settings/storage/init`, > `/settings/storage/attach`) now populate candidates from the agent's **raw-device scan** > (`GET /api/disks/candidates` → agent Impl-2a, proxied by `agentDiskCandidatesHandler`): `initialize` = > every unclaimed disk (blank or data-bearing), `attach` = the mountable-FS subset. The agent's > unclaimed-disk filter (Impl-1 `claim.go`) is authoritative + fail-safe (never offers OS/enrolled/claimed > disks), so the controller does NO client- or server-side filtering. Enrollment posts to the unchanged > `/api/storage/init` (format via the agent's Impl-1 guarded `mkfs` → mount → bind → intent) or > `/api/storage/attach` (mount + bind, no format). The legacy text below is kept for historical context. #### Disk Scanning (`internal/storage/scan.go`) - `ScanDisks()` uses `lsblk -J -b` for block device enumeration - System disk detection via host fstab parsing (`/host-fstab`) + UUID resolution via `blkid` - Partitions enriched with filesystem type, UUID, and label from direct `blkid` probing (Docker containers have incomplete udev cache) - Returns `AvailableDisks` (non-system, non-loop, non-CDROM), `SystemDisks`, and `FormatablePartitions` (empty partitions on system disks that are safe to format) - Handles NVMe (`nvme0n1p1`), SCSI (`sdb1`), and eMMC (`mmcblk0p1`) naming #### Disk Initialization Wizard (`internal/storage/format.go`) A step-by-step UI at `/settings/storage/init`: 1. **Scan** — Lists available disks with model, size, partition info 2. **Select** — User picks a disk and enters a mount name (e.g., `hdd_1`) 3. **Confirm** — User types "FORMAZAS" to confirm destructive operation 4. **Format pipeline**: `wipefs` → `sfdisk` (GPT) → `mkfs.ext4` → `blkid` UUID → backup fstab → append UUID-based fstab entry → mount → `findmnt` verification → `chown 1000:1000` → create `felhom-data/` and `Dokumentumok/` subdirectories 5. Auto-registers new storage path in settings.json 6. Smart partition detection: skips repartitioning for existing empty partitions Safety guards: system disk detection, mount path conflict check, confirmation required, progress channel for real-time UI feedback. **System-disk partition formatting:** When the system disk has an empty partition (no filesystem, not mounted, not used for /, /boot, /boot/efi, or swap), the init wizard detects it via `FormatablePartitions` in the scan result and offers to format just that partition. Uses `IsSystemPartition()` (granular per-partition check via fstab) instead of `IsSystemDisk()` (whole-disk block), so sda1 can be formatted while sda3 (root) remains protected. #### Attach Existing Drive Wizard (`internal/storage/attach.go`) A step-by-step UI at `/settings/storage/attach` for drives that already have a filesystem (e.g., a previously used ext4 drive). Unlike the init wizard, this does **not** format the drive — existing data is preserved. **Problem solved:** Mounting a whole drive at `/mnt/` would mix existing user data with the controller's directory structure (`felhom-data/`, `Dokumentumok/`, etc.). The bind-mount approach isolates the controller's working directory from other data on the drive. 1. **Scan** — Lists available disks, filtered to partitions that have an existing filesystem (FSType != "") 2. **Mount raw** — Partition is mounted read-only at a hidden staging path (`/mnt/.felhom-raw/