# felhom-controller **Central management container for Felhom home servers.** The **in-guest controller**: one per customer LXC, Docker-only, Hungarian web dashboard for managing the customer's app stacks, app-data backups, monitoring and notifications. All Proxmox/disk operations are delegated to the host agent (`felhom-agent`). **Current version: v0.59.0** > ## Documentation has moved (and is now code-verified) > > The **authoritative architecture & feature documentation** now lives in the central docs home: > **[`felhom.eu/documentation/controller/`](../../felhom.eu/documentation/controller/README.md)** — > module map, deploy & stack lifecycle, backup architecture, storage/monitoring/metrics, and > auth/hub/sync/integrations. Those docs are grounded in current source (v0.59.0). > > **Quick build & deploy** is in the "Build & Deploy" section below and in the repo `CLAUDE.md` > (authoritative for the workflow). NOTE the demo controller runs in an LXC guest (9201) under the > **bootstrap-managed** mechanism (`/etc/felhom-controller-image` + `felhom-controller-bootstrap.service`), > not the bare-metal `/opt/docker` compose path some sections below still describe. > > The prose below this banner is **retained legacy reference** and may lag the central docs — when they > disagree, the central docs win. (Some sections still mention restic / pre-8C disk handling that has > since moved to the host agent.) --- ## Table of Contents - [Architecture](#architecture) - [Features](#features) - [App Management](#1-app-management) - [App Export/Import](#2-app-exportimport-fab-bundles) - [Backup System](#3-backup-system) - [Storage Management](#4-storage-management) - [Monitoring & Health](#5-monitoring--health) - [Notifications](#6-notifications) - [Update Management](#7-update-management) - [Authentication & Settings](#8-authentication--settings) - [Central Hub](#9-central-hub-reporting) - [Setup Wizard](#10-first-run-setup-wizard) - [Disaster Recovery](#11-disaster-recovery) - [Asset Sync](#12-asset-sync) - [Debug Mode](#13-debug-mode) - [Geo-Restriction](#14-geo-restriction) - [App-to-App Integrations](#15-app-to-app-integrations) - [Network File Sharing — SMB](#16-network-file-sharing--smb-megosztás-v01440-r-7-slice-1) - [Async-job feedback](#17-async-job-feedback-v0147x-feedback-slice-1) - [Repository Layout](#repository-layout) - [Configuration](#configuration) - [REST API](#rest-api) - [Build & Deploy](#build--deploy) - [Roadmap](#roadmap) --- ## Architecture ``` ┌─────────────────────────────────────────────────────────────────┐ │ Customer Hardware (N100 mini PC / Raspberry Pi) │ │ │ │ ┌──────────┐ ┌────────────────────────────────────────────┐ │ │ │ Traefik │ │ felhom-controller (privileged container) │ │ │ │ (reverse │──▶│ │ │ │ │ proxy) │ │ ┌──────────┐ ┌─────────────────────────┐│ │ │ └──────────┘ │ │ Web UI │ │ Stack Manager ││ │ │ │ │ (HU dash │ │ (compose ops, git sync, ││ │ │ ┌──────────┐ │ │ board) │ │ deploy, delete, update) ││ │ │ │cloudflared│ │ └──────────┘ └─────────────────────────┘│ │ │ │ (tunnel) │ │ ┌──────────┐ ┌─────────────────────────┐│ │ │ └──────────┘ │ │ Backup │ │ Storage Manager ││ │ │ │ │ (3-layer │ │ (disk scan, format, ││ │ │ ┌──────────┐ │ │ restic) │ │ mount, migrate) ││ │ │ │ App │ │ └──────────┘ └─────────────────────────┘│ │ │ │ stacks │ │ ┌──────────┐ ┌─────────────────────────┐│ │ │ │ (docker │ │ │Scheduler │ │ Monitor & Metrics ││ │ │ │ compose) │ │ │(cron-like│ │ (health, SQLite ││ │ │ └──────────┘ │ │ jobs) │ │ time-series, Chart.js) ││ │ │ │ └──────────┘ └─────────────────────────┘│ │ │ │ ┌──────────┐ ┌─────────────────────────┐│ │ │ │ │ Notify │ │ REST API + Hub Reporter ││ │ │ │ │ (events) │ │ (JSON push + events) ││ │ │ │ └──────────┘ └─────────────────────────┘│ │ │ │ ┌──────────┐ │ │ │ │ │ Assets │ │ │ │ │ │ (Hub │ │ │ │ │ │ sync) │ │ │ │ │ └──────────┘ │ │ │ └────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────┘ │ events + reports │ git pull │ asset sync ▼ ▼ ▼ hub.felhom.eu gitea.dooplex.hu hub.felhom.eu (central dashboard) (stack definitions) (logos, screenshots) ``` ### Key Architecture Decisions - **Pure Go, no frameworks** — stdlib `net/http` + `html/template`. Only external deps: `bcrypt`, `yaml.v3`, `modernc.org/sqlite` (pure Go, no CGO). - **Privileged container** — Required for disk operations (format, mount, fstab), `/dev` access, and Docker socket control. - **`/host-dev` indirection** — Docker overrides `/dev` with a tmpfs. The host's `/dev` is mounted at `/host-dev` to access block devices. - **`StackDataProvider` interface** — Breaks circular import between the backup packages and stacks. Defined in `internal/appbackup` (and re-exposed via a type alias in `internal/backup`). Implemented by `stackAdapter` in `main.go`. Provides `GetStackHDDPath()` for per-drive backup routing. - **Atomic file writes** — All persistent state (`settings.json`, `app.yaml`) written to `.tmp` then `os.Rename` for crash safety. - **`go:embed` templates** — All HTML/CSS/JS compiled into the binary. No runtime file dependencies. - **Design system v2 (v0.96.0, TASK-D0)** — The whole UI renders in the Felhom v2 language (canonical reference: `felhom.eu/documentation/design/design-system.md`): navy token palette, single 2px radius, no shadows, exception-based status color (nominal = blue/neutral; amber/red only on deviation — `stateColor` emits `run/progress/warn/neutral/off`, `usageColor`/`tempColor` emit `nominal/warn/crit`; a stopped app is neutral, NOT red). Capacity bars are the hairline `.meter` component; state chips are `.tag`; informational pills are `.metarow`. **Fonts (Plus Jakarta Sans + JetBrains Mono, variable woff2, latin+latin-ext) and a 30-icon Lucide sprite are vendored in the binary** (`internal/web/static/fonts/` served at `/static/fonts/`; `templates/icons.html`) — no CDN, no emoji. The setup wizard serves the same embedded stylesheet via `web.StyleCSS()`. - **Guest RAM resize (v0.143.0, R-24; MinAgent 0.90.0)** — the Rendszer page's "Szerver memória (RAM)" card shows the guest's current/used memory + the allowed range and lets the customer resize it. The controller only proxies + maps the agent's machine `code` to Hungarian; **the agent (felhom-agent v0.90.0) enforces every bound and applies the change live via PVE SetConfig — no reboot** (min 2048 / max host_total−2048 / a shrink is refused below max(2048, usage+512)). `systemPageData` calls `memoryCardData` (a 2 s agent GET behind the `FeatureGuestMemoryResize` gate); `POST /api/system/memory/resize` (`ServeSystemAPI`, `internal/web/system_memory_handlers.go`) → `agentapi.ResizeMemory` → the code→Hungarian map. A JS confirm fires only on a shrink; an outdated agent hides the control; an unreachable agent falls back to the guest's own `/proc/meminfo`. Memory only (cores stay observation). The lxcfs ripple means the deploy-page memory math follows a resize for free. - **„Hálózat" card (v0.159.0, R-66)** — Beállítások → Rendszer, between „Verzió és frissítés" and „Szerver memória": **Helyi cím (LAN)**, **Hálózati név** (`\\` — rendered ONLY while Megosztás is enabled; the NetBIOS name exists only while samba runs), **Átjáró**, plus a muted footer ("read this page aloud during remote troubleshooting"). Every value is live-computed per render and stored NOWHERE (S-5): the reads go through the samba-container netns door (`internal/stacks/guestnet.go` — the controller's own netns is the docker bridge, so in-process answers like `/proc/net/route` would report 172.x, the S-2 trap); with Megosztás off the door is closed and rows render „—" (an address-less row beats a wrong address). Companions: the Debug system dump gains a `network` section (interfaces without veth*/docker*/br-* plumbing, default route + gateway + source interface, DNS from the guest's resolv.conf, and the SAME `lan_address` the card shows for cross-checking), and the NAS add form names the NetBIOS trap — helper text under Szerver, plus a purely lexical hint appended to an `unreachable` failure when the submitted server is a single-label non-IP name (`looksLikeFlatNetworkName`; no NetBIOS/mDNS resolution is ever attempted). - **Page IA (v0.97.0, TASK-D1)** — the settings monolith is split into four pages, each with its own data builder (`systemPageData`/`storagePageData`/`notificationsPageData`/`securityPageData`, sharing `settingsBaseData`) and template. Routes: `/settings` (Rendszer), `/settings/notifications` (GET→page, POST→save on the same path), `/settings/security`, `/storage` (main-nav **Tárhely** — Meghajtók), and `/storage/network` (Hálózati tárhely / NAS; v0.98.0 split — nested sub-links under Tárhely). The enrollment wizards live at `/storage/{init,attach}`; `/settings/storage/{init,attach}` **301** to them; all storage action successes redirect to `/storage?storage_msg=…`. `/storage` shows a **unified drive view**: server-rendered registry cards enriched in place from the agent `/api/disks` (role/durable-id/actions, joined on mount path), plus read-only **Rendszermeghajtók** and **Nem regisztrált meghajtók** groups; agent-down degrades to a single warn note. Consequential actions use an in-page `.confirm-overlay` (`openDialog`) or the LIGHT inline two-step (`felhomConfirm` in layout.html, v0.123.0: the trigger swaps in place to "kérdés + Igen/Mégse"; form buttons opt in via `data-confirm="…"`) — never native `confirm()`/`prompt()` (OS-modals freeze browser automation; drill F-11). **Shared app-list row (v0.126.0):** `templates/app_row.html` (`app_list_row`/`app_list_row_end`) is the ONE row grammar for app lists — icon + name (+ optional secondary line) left, caller action block right; used by the dashboard installed-apps list, the Távoli mentés toggle list and the Visszaállítás restore-to-verify/.fab lists; the backups-apps expander header is ALIGNED to the same grammar (own markup — it carries the toggle). Protected infra stacks (traefik/cloudflared/filebrowser) get curated Hungarian display identity from the `inframeta.go` map (name + description + generic `/static/infra-logo.svg` fallback icon); filebrowser is the only infra stack with a customer link (`files.`). **Enforcement:** `scripts/template_id_gate.py` (JS element-ID integrity) + `scripts/emoji_gate.py` (no emoji) + `scripts/native_confirm_gate.py` (zero native confirm/prompt) + `scripts/app_row_dedup_gate.py` (row markup single-sourced) + `scripts/mojibake_gate.py` (no double-encoded UTF-8 in templates/Go sources) — run after any template change; the Go `TestNoEmojiInTemplates` mirrors the emoji gate. - **Europe/Budapest timezone** — All scheduled jobs, timestamps, and UI labels use Hungarian timezone. ### Module Map | Module | Path | Responsibility | |--------|------|----------------| | **Config** | `internal/config/` | YAML loader, validation, `FELHOM_*` env overrides | | **Settings** | `internal/settings/` | Runtime-mutable `settings.json` (passwords, backup prefs, storage paths, notifications) | | **Stacks** | `internal/stacks/` | Compose operations, scanning, `.felhom.yml` metadata, deploy/delete flow; **base-infra bring-up** (`infra.go` — `EnsureBaseStack`) | | **Infra** | `internal/infra/` | Pure renderers (embedded `text/template`) for the base-infra stacks (traefik/cloudflared/filebrowser); **pinned image tags as the single source of truth** (web filebrowser sync delegates here) | | **Crypto** | `internal/crypto/` | AES-256-GCM encryption for sensitive app.yaml values (passwords, secrets), key management | | **Sync** | `internal/sync/` | Git-based app catalog sync (clone/pull, content-hash copy) | | **AppBackup** | `internal/appbackup/` | Self-contained app-data backup primitives: DB dump discovery/execution (`DiscoverDatabases`, `DumpOne`), Docker-volume/app-data discovery (`StackDataProvider`, `DiscoverAppData`), keep-side path helpers (`AppDBDumpPath`, `AppVolumeDumpPath`, `AppDataDir`). `DiscoverDatabases` takes the deployed-stack set so a DB container maps to the right stack even when a slug ends in a DB-role token (M19, v0.62.0). `ListDumpFiles` takes an optional `cached(name,size,mod)` lookup so an unchanged dump isn't re-validated (line-scan) every ~5-min cycle (M18, v0.62.0). No dependency on restic/cross-drive/drive-mount. Imported directly by `appexport` and `storage`. | | **Backup** | `internal/backup/` | Per-drive 3-layer backup: DB dumps → restic snapshots → cross-drive copies, restore. Re-exposes the `appbackup` primitives via aliases/forwarders (`appbackup_bridge.go`) for the disk/host-side code and the web/api/report consumers. | | **Storage** | `internal/storage/` | Disk scanning (`lsblk`), partitioning (`sfdisk`), formatting (`mkfs.ext4`), mounting, data migration (`rsync`) | | **System** | `internal/system/` | System info (`/proc`), CPU collector, mount points, disk usage, FS info | | **Monitor** | `internal/monitor/` | System health checks, storage watchdog, legacy Healthchecks pinger (deprecated) | | **Metrics** | `internal/metrics/` | SQLite time-series store, system + container metric collection | | **Scheduler** | `internal/scheduler/` | Central job scheduler (periodic + daily, skip-if-running, panic recovery) | | **SelfUpdate** | `internal/selfupdate/` | Version checking (registry), update trigger, state persistence, startup verification | | **Notify** | `internal/notify/` | Email notifications via hub relay, preference sync, per-event cooldowns | | **Report** | `internal/report/` | Hub report builder + HTTP pusher (system, stacks, backup, health) | | **Assets** | `internal/assets/` | Hub-managed asset syncer: downloads logos/screenshots with SHA-256 change detection | | **SelfTest** | `internal/selftest/` | Startup self-test: 9 diagnostic checks (Docker, dirs, storage, hub, restic, metrics) | | **Util** | `internal/util/` | Shared utilities: `TruncateStr` for debug log output truncation | | **AppExport** | `internal/appexport/` | Per-app export/import via `.fab` bundles (config + DB + user data), optional AES-256 encryption | | **API** | `internal/api/` | REST JSON endpoints, diagnostic dump (`/api/debug/dump`) | | **Web** | `internal/web/` | Hungarian dashboard, auth, page handlers, template functions, alerts | --- ## Features ### 1. App Management The controller manages Docker Compose stacks through a complete lifecycle: catalog sync, first-time deployment, runtime operations, and deletion. #### Git Sync (`internal/sync/`) The app catalog lives in a separate Git repository. The controller: - Shallow-clones the catalog on startup - Periodically fetches updates (configurable, default 15 min) - Copies only `docker-compose.yml` and `.felhom.yml` to the stacks directory - **Never overwrites** `app.yaml` (user secrets are safe) - Uses SHA-256 content hashing — only writes files that actually changed - Triggers stack rescan after sync so the dashboard updates immediately - **Post-sync hook**: auto-injects missing deploy fields (new secrets, domains) into existing `app.yaml` for stacks whose templates were updated (see Missing Field Injection below) - Manual sync via "Sablonok frissitese" button or `POST /api/sync` #### First-Time Deploy Flow 1. Customer sees app card with "Telepites" button 2. Deploy page pre-generates and **displays** all auto-values before the user clicks deploy: - `domain` fields: shown as readonly text input with the customer's configured base domain - `subdomain` fields: editable text input pre-filled with the default from `.felhom.yml`, shown with `.base-domain` suffix. Validated for DNS-safe format, reserved names, and uniqueness across deployed stacks. Locked after deploy — changing requires Remove + Redeploy - `secret` fields: pre-generated and shown as masked password inputs with a "Megjelenítés" reveal button — user can see/copy all DB passwords and keys before deploying - User-configurable inputs (admin password, language, storage path) remain editable - Section header prompts the user to note down any passwords they need 3. `checkBeforeDeploy()` JS guard fetches live state first (prevents double-deploy from another tab) 4. **Memory validation** (F1, v0.61.0): the controller runs as a Docker container inside an LXC, where `/proc/meminfo` shows the **Proxmox host's** RAM (no lxcfs in the container) and the container's own cgroup is unlimited (the guest cap lives on the LXC ancestor). So the guest cap is read from the **Docker daemon** (`system.GuestMemTotalMB()` → `docker info` MemTotal — the daemon runs in the LXC and reports the guest's lxcfs-backed RAM; the cgroup limit is preferred when present, e.g. non-nested): - `usable_memory = guest_cap - reserved_memory_mb` (default 384MB reserved) - Hard block if `committed_used + new_request > usable_memory`, where `committed_used` = `CommittedMemory()` (sum of running apps' mem requests) — the guest-wide RSS is not observable from the container, so the controller's own committed accounting is the accurate, cheap "used". - `/api/system/info` reports the guest cap as total and committed memory as used. 4b. **Docker-data volume reserved-buffer gate (v0.58.0, storage-split prevention layer):** the OS rootfs and Docker data are split onto separate volumes; infra (controller/traefik/cloudflared/filebrowser) shares the one Docker data-root (`/var/lib/docker`) and is protected by **prevention, not placement**. `system.GetDockerVolumeHeadroom()` measures the Docker-data volume via `statfs("/")` (the controller container's overlay root is the upperdir on that volume — true with the golden's **overlay2** driver) and reserves `max(5 GB, 10%)`. `deployStack` **refuses a new deploy (HTTP 507)** when free space is at/under the buffer; the deploy page shows the warning + disables the button. Fail-open on a statfs error. The runtime disk monitor (`healthcheck.go`, warn 80% / crit 90%) watches the same volume and trips above the buffer. (Assumes the split guest's large data volume; the golden bakes overlay2 + log rotation so images+volumes live on the data volume, not `/var/lib/containerd`.) 5. Pre-generated secret values are submitted as hidden form inputs so the **same values** the user saw are saved to `app.yaml` (no silent re-generation on submit). Controller saves `app.yaml`, sets in-memory `Deployed` + `Deploying` flags, then runs `docker compose up -d` **asynchronously** in a goroutine — API returns immediately so the UI switches to the progress panel without waiting for image pulls. On failure the goroutine reverts both disk and in-memory state and sets `DeployError`. 6. 3-step progress panel polls `GET /api/stacks/{name}` every 3s: config saved → `deploying` (pulling images) → containers starting → health check passed. New `StateDeploying` state shown while compose-up is in progress (no containers yet). 7. Post-deploy: locked fields (DB_PASSWORD, etc.) become read-only; the "Automatikusan generált értékek" section continues to show the saved values on the settings page 8. The deploy/settings page includes **start/stop/restart** buttons for deployed apps, plus a "Megnyitás ↗" link to the app's subdomain URL (only visible when running) #### Catch-All Page for Stopped Apps When a user visits a stopped or undeployed app's subdomain (e.g., `travel.demo-felhom.eu`), the controller serves a branded error page instead of Traefik's raw 404: - **Traefik catch-all router**: The controller's `docker-compose.yml` registers a second router (`catchall`) with `priority=1` (lowest) and `HostRegexp(.+)`. Running apps always win; only requests with no matching container reach the controller. - **`CatchAllMiddleware`** in `server.go` intercepts requests where `Host` ≠ `felhom.DOMAIN`, serves the catch-all page **without auth** (user has no session on the app subdomain). - **`findStackBySubdomain()`** identifies the app by matching the subdomain against deployed `app.yaml` `SUBDOMAIN` env or metadata fallback. - **`catchall.html`** — standalone template (no layout, inline CSS) showing the app name, status ("leállítva" / "nincs telepítve" / "nem található"), and links to the controller dashboard or the app's detail page. - **Subdomain links** on the Alkalmazások page are only shown for deployed apps (non-deployed apps have no guaranteed subdomain yet). #### Dashboard "Megnyitás" Button Running apps on the Vezérlőpult now show a "Megnyitás ↗" button that opens the app's subdomain in a new tab. The `Subdomains` map is built in `dashboardHandler` from `app.yaml` env or metadata fallback. The optional `.felhom.yml` **`open_path`** field (`Metadata.OpenPath`) is appended to that URL for apps whose UI isn't at `/` — e.g. `gokapi` → `/admin`, `ghost` → `/ghost/`. Empty = bare root. Rendered via `.Meta.OpenPath` in `dashboard.html`, `deploy.html`, and `app_info.html` (the same field flows through all three open-link sites; no handler change needed). Must start with `/`; cosmetic only (does not affect routing). #### App lifecycle — withdrawing an app without orphaning anyone (v0.158.0) `.felhom.yml` carries an optional top-level **`lifecycle:`** (`Metadata.Lifecycle`), the catalog's answer to "stop offering this app" that does not punish the customers already running it. | value | offered for new installs? | shown to someone already running it | |---|---|---| | `available` (default; absent/empty ≡ this) | yes | nothing special | | `hidden` | **no** | nothing — "we stopped offering this" is not their problem | | `abandoned` | **no** | „Nem karbantartott" badge + a notice on the app page that updates and security fixes will no longer arrive | **A deployed instance keeps full function in every state.** Lifecycle governs what is OFFERED, never what runs — deleting a template instead would mark every deployed instance `Elavult` and offer a Törlés button for working software. Three predicates on `Metadata` are the single interpretation of the field — every surface goes through them: `EffectiveLifecycle()`, `CanInstall()`, `IsAbandoned()`. - **Listing** — `web.visibleCatalogStacks` drops a template that is not installable AND not deployed here (`Deployed || Protected || CanInstall()`). - **Deploy gate** — `api.deployStack` refuses server-side before any mutation with „Ez az alkalmazás jelenleg nem telepíthető." (409). `stacks.DeployStack` repeats the check for any caller that does not route through the API. Hiding the button is not a gate. - **Unknown values fail OPEN** (→ `available` + one WARN), deliberately opposite to the gate's fail-closed posture: a typo, or a state from a newer catalog than this controller, must never pull a working app out of every customer's catalog. Both read the same `EffectiveLifecycle`, so they cannot disagree. - **Orphan detection must never see this field.** `getCatalogTemplateSlugs` keys on directory + compose presence only; withdrawn templates stay in the catalog tree. Asserted by `TestCatalogTemplateSlugs_IgnoresLifecycle` with a red-proof. - **Badges are generic plumbing**: `web.MetaBadge` + the `meta_badge` template partial + the `lifecycleBadge` funcmap entry. R-56's difficulty labels are intended as a sibling funcmap function returning the same `*MetaBadge` — no new markup or CSS. #### App Info Pages Each app can define rich metadata in `.felhom.yml`: - `app_info`: tagline, use_cases, first_steps, prerequisites, default_creds, docs_url - `optional_config`: groups of post-deploy configurable env vars (e.g., API keys for metadata providers) - `resources`: mem_request, mem_limit, pi_compatible, needs_hdd, hungarian_ui - `initial_credentials` (v0.84.0): for apps that **auto-generate a first-login password into a file** at first boot (vs taking it from a deploy field) — e.g. Crafty → `/crafty/app/config/default-creds.txt`. Fields: `file` (path inside the container), `format` (`json`|`regex`|`plain`), optional `container` (defaults to the stack's main container), `username_key`/`password_key` (json), `username_pattern`/ `password_pattern` (regex, first capture group), `note`. `ReadInitialCredentials` (`internal/stacks/initialcreds.go`) reads the file **live** via `docker exec … cat` and parses it with the pure `parseInitialCreds`; the value is **never persisted** to `app.yaml`. `appDetailHandler` surfaces it on `/apps/{slug}` as a "Kezdeti belépési adatok" card (masked password + reveal/copy), labelled as the *initial* credential (stays valid only until the customer changes it in-app). Hidden when the container is down / file missing / parse fails. Reuse for any future self-seeding app. - `backup` (v0.132.0): the **referential-coupling classification** block (Task 2). Two optional lists, `userdata:` (relative to `${USERDATA_PATH}`) and `hdd:` (relative to `${HDD_PATH}`), each of `{path, class}` where `class ∈ {mandatory, optional, excluded}` (COUPLED / DECOUPLED-precious / DECOUPLED-bulk). `LoadMetadata` validates the block against the app's compose binds and **rejects the WHOLE block** (→ nil + one `[ERROR]`, app behaves as legacy) on any defect. Semantics (`appbackup.ClassifyBinds`): an explicit entry wins over the `:ro` default; an unlisted writable bind defaults `mandatory`, an unlisted `:ro` bind defaults `excluded`; **no block at all = legacy behavior per tier**. **INERT** as of v0.132.0 — the schema/parser/classifier + the `Manager.ClassifiedBinds` / `StackDataProvider.GetStackClassifiedBinds` seam exist, but no backup tier consumes them yet (Task 3 = tier policy engine, Task 4 = manual `.fab` UI). See `felhom.eu/documentation/audits/SPIKE-backup-classification-2026-07-14.md`. - **Capture-set computation (v0.133.0, Task 3-core, INERT):** `appbackup.ComputeCaptureSet(binds, hasClassification, tier, hddPath)` (`internal/appbackup/captureset.go`) is the pure path-algebra that turns classified binds into a tier-filtered, structurally-guarded, containment-deduped absolute `CaptureSet{HasClassification, Paths, Skipped}` (slash algebra, no `filepath`/FS/log). `TierOffsite` = mandatory only; `TierSecondary` = mandatory + optional; excluded dropped; legacy short-circuits to unit-only. Structural guards (traversal / bare HDD drive-root / reserved `backups/` zone) move refused would-be captures into `Skipped` for the engines to log. Companion `CrossAppOverlaps` (pure; WARN wiring deferred to 3a/3b). See `felhom.eu/documentation/architecture/07-backup-architecture.md` §3. - **Offsite tier engine (v0.134.0, Task 3a — consumes the above):** `internal/backup/offbox.go` + `offbox_capture.go` + `offbox_restore.go`. Each toggled app's push is ONE multi-path restic snapshot = recovery unit + its `TierOffsite` mandatory set (`offboxCaptureSet`); legacy/undeployed stay unit-only. Skipped/missing mandatory paths are loud gaps (English log + Hungarian `LastWarning`) because restic 0.14.0 silently skips a missing source path (SP-3.4). Quota reads `stats --mode raw-data` (real repo bytes, SP-1); a pre-push gate blocks an ENLARGEMENT that would cross the soft quota (unit-only push continues; `OffboxTarget.EnlargedBlocked`; edge-triggered notify). Retention `forget --group-by host,tags` (SP-2). Restore (`RestoreOffboxScratch`) scratches to a data drive off the rootfs (F-A1) behind a headroom gate; unit-only default via `--include` the absolute unit path; `PlaceOffsiteRestore` merges a full scratch into live via `rsync --ignore-existing` (never `--delete`), refusing on the pure `mapOffsiteRestorePaths` guards. It restores FILES ONLY — no database, no stack restart — and the UI now says so. - **Coherent snapshot pairs (v0.148.0, R-44):** every offsite run — manual AND nightly — refreshes the DB/volume dumps and recovery units (`offsitePreDump` → `runDBDumpsInternal`) BEFORE the restic capture, so each snapshot is an internally coherent `{DB@T, files@T}` bundle and retention is a history of restorable points. Order is the mechanism: the gap can only add files the DB does not reference yet, never remove one it does. Each manifest carries `offsite_run_id` + `dumps_at`; a manifest without them is a pre-v0.148 pair of unknown skew, surfaced at restore time. The periodic refresh carries the prior stamp forward and never invents one. A dump-leg failure is a loud WARN that does NOT abort the push (data-first: a degraded backup beats none). - **Offsite reconstitution (v0.148.0, R-43 — `offbox_reconstitute.go`):** the leg that was missing. `ReconstituteFromOffsite` (`/backup/offbox/reconstitute`, „Teljes visszaállítás (fájlok + adatbázis)") makes the live app equal to the chosen snapshot: **safety dump → stop → files overwritten (`rsyncRestoreOverwrite`: no `--ignore-existing`, no `--delete`) → the DATABASE SERVICE ONLY started (`StartStackServices`, v0.153.0) → the snapshot's dump replayed (`reimportDBDumpsFrom`, reading the SCRATCH unit) → the full stack started → health wait**. Two invariants: nothing is ever deleted (post-snapshot files survive as extras), and the `pre-restore-` safety dump is verified on disk BEFORE anything is stopped or overwritten — if it cannot be taken the operation refuses with zero changes. Safety dumps appear in `ListDumpFiles` (they are the undo). The live recovery unit is still never overwritten, which is why the replay source is the scratch. Honesty surfaces (`OffsiteScratchPair`): dump age, an unstamped-pair warning, and the R-44 empty-dump sniff — all warn-level, none of them gates. - **The restore wizard (v0.154.0, R-48 — `web/restore_wizard.go`).** The offsite restore controls used to render as up to five inline forms per app row, two of which — the missing-only merge and the true reconstitution — were sibling buttons whose difference is whether the data comes back. That mis-selection caused the round-2 incident. Each row now carries ONE entry linking to a per-app wizard; the three intents are cards with consequence sentences, and the dangerous one keeps the R-43 double-confirm verbatim. `deriveWizardStep` is pure — (op running, size-gate flash, scratch ready) → step + which intents unlock — and a running op outranks a stale `?full_prep=`, so no commit button survives into a restore. While ANY op runs, every mutation form is suppressed server-side rather than offered and refused. No new endpoint, no job registry (that stays R-45), and the page works with JavaScript disabled. **v0.155.0 fix:** the "is an op running" read must come from `RestoreStatus()` (the `opRunning` display flag, set synchronously by `BeginRestoreOp`), NOT `Manager.IsRunning()` (the concurrency single-flight, which `RestoreOffboxScratch` never acquires — so v0.154.0's execution step was unreachable for the verification restore). The strip's highlight is its own derived `Phase`, so a finished restore reads „Eredmény" while the intent step is available again; the outcome card is window-bounded and app-bound. - **The DB-only replay window (v0.153.0, R-47).** Until v0.153.0 the whole stack was started before the replay, so the application's own schema management raced the dump: measured live on 2026-07-19 (H4), immich-server rebuilt `clip_index` two seconds before the dump's `CREATE INDEX` and the replay aborted `already exists` under `ON_ERROR_STOP=1`. The DB service is now brought up alone (`appbackup.DBServiceNames` reads the LIVE compose's `services:` map to name it), the dump is replayed with the app still down, and only then does the full start run. **Fail-closed:** a dump with no identifiable DB service refuses before the first mutation. Every exit from the window — replay failure, DB-only start failure — still does a best-effort full start, so a failed restore never leaves the box with a database and no application. The `/apps/{slug}` page renders hero section, screenshots, setup guide, and optional config form. #### Stack Operations | Operation | What it does | |-----------|-------------| | Start | `docker compose up -d` — pre-start memory check rejects with 409 if insufficient RAM | | Stop | `docker compose stop` (blocked for protected stacks) | | Restart | `docker compose restart` | | Update | `docker compose pull` + `docker compose up -d` | | Remove | `docker compose down --volumes` + remove `app.yaml` + optional HDD/backup cleanup; template preserved for redeploy | | Delete | `docker compose down --rmi local --volumes` + optional HDD data cleanup (orphaned stacks only) | **Remove vs Delete**: "Eltávolítás" (Remove) is for deployed catalog stacks — it reverts the stack to "Nincs telepítve" state while keeping the template for easy redeployment. "Törlés" (Delete) is for orphaned stacks — it removes the entire stack directory including templates. Both require stopping the stack first. **Remove modal** shows three sections: (1) always-removed items (Docker volumes, app.yaml, cross-drive schedule), (2) optional HDD data deletion with reimport warning, (3) optional backup data deletion (DB dumps + cross-drive rsync) with restic retention note. **Protected stacks** (traefik, cloudflared, felhom-controller) cannot be stopped, removed, or deleted from the UI. Restart is allowed. **Orphan detection**: Deployed stacks with no matching catalog template are marked as orphaned with an "Elavult" badge and can be safely deleted. #### Base-infrastructure bring-up (`stacks/infra.go` + `internal/infra/`, v0.41.0) The controller stands up its own base stack — **traefik** (reverse proxy), **cloudflared** (external tunnel), **filebrowser** — instead of relying on the bare-metal `scripts/docker-setup.sh` (which a Proxmox-provisioned guest never runs). `internal/infra` renders the compose + config files from `controller.yaml` via embedded `text/template`s (lifted from `docker-setup.sh`); image tags are **pinned constants there** (`TraefikImage`/`CloudflaredImage`/`FileBrowserImage`) and the web FileBrowser sync path delegates to the same renderers, so the pinned versions can never diverge. `Manager.EnsureBaseStack()` creates the `traefik-public` network, then deploys traefik → cloudflared → filebrowser under `${stacks_dir}/`. It is: - **single-flight** (a `TryLock` guard — it's called from both first boot and every health tick, so overlapping runs must not race on the same stack dir), - **idempotent** (skips a stack whose container is already running; never overwrites an existing filebrowser compose, preserving the storage mounts `SyncFileBrowserMounts` manages), - **non-fatal** (logs, never crashes the controller). cloudflared is only deployed when a tunnel token is configured. **Triggers**: a first-boot goroutine (after stack init) and an unconditional call on every `system-health` tick (self-heal — cheap when healthy thanks to the idempotency). `monitor.EffectiveProtected` mirrors the cloudflared condition so a LAN-only node (no tunnel token) doesn't report a perpetual "protected container not running" FAIL. **Backend transports — self-signed HTTPS backends (`ensureServersTransports` → `RenderServersTransports`, v0.83.0).** Traefik talks **HTTP** to app backends by default, which is correct for every catalog app that serves plain HTTP. The exception is an app that serves its **own self-signed TLS** on the internal docker bridge (the first is Crafty, HTTPS-only on `:8443`): Traefik must speak `https` to it *and* skip verifying a per-container self-signed cert (no CA to verify against; the hop never leaves the host). `insecureSkipVerify` is **not settable via Docker labels** in traefik v3 — it must live in static/file config — so `EnsureBaseStack` writes a file-provider dynamic file `dynamic/serverstransports.yml` defining a **named** transport `insecure-skip-verify` (write-if-changed; hot-loaded by the file watcher). An app **opts in per-service** via two catalog labels — `loadbalancer.server.scheme=https` + `loadbalancer.serverstransport=insecure-skip-verify@file` (the `@file` suffix is the cross-provider reference). Backend verification stays the default (ON) for every other service — there is deliberately **no global `insecureSkipVerify`** in `traefik.yml`. This write runs **outside** `ensureTraefik` (which early-returns when traefik is already up) so an established node still materializes the file on a self-heal tick. > **Mount prerequisite (Section-G):** the controller writes these stacks under `/opt/docker/stacks` *inside its container*, but `docker compose up` runs on the **guest** Docker daemon. The golden's controller-bootstrap (`felhom-agent` `build-golden.sh`) therefore bind-mounts that path **same-path** (`-v /opt/docker/stacks:/opt/docker/stacks`) so the daemon resolves every relative bind source — without it, all bind-mounted stacks (base infra and customer apps) silently break. **Controller routing + the wildcard cert anchor (`wireController` → `RenderControllerRoute`, v0.41.1 / v0.42.1).** filebrowser self-registers with traefik via Docker labels + `traefik-public` membership baked into its compose; the controller can't (it's started by the golden bootstrap *before* `traefik-public` exists, and the v2 `bootstrap.json` carries no domain — that comes from the hub pull). So `EnsureBaseStack` wires the controller **post-pull**: it `docker network connect traefik-public felhom-controller` and writes a traefik file-provider route `dynamic/controller.yml` (`Host(felhom.) → http://felhom-controller:8080`, write-if-changed). When DNS-01 ACME is configured, that route is **also the wildcard-cert anchor**: its router-level `tls.domains: *.` makes traefik **proactively obtain the wildcard `*.` + apex via Cloudflare DNS-01 at startup** (an entrypoint-level `http.tls.domains` does *not* trigger issuance in traefik v3 — only a router-level `tls.domains` does). Every other router then serves that one real wildcard cert by SNI — no per-app `certresolver` labels. This is what lets a LAN client reach the box directly at `*.` with the real cert (the `felhom-agent` split-horizon resolver depends on it). #### Missing Field Injection (`deploy.go`) When app templates are updated (e.g., a new `APP_KEY` secret is added to `.felhom.yml`), existing deployed apps need the new field in their `app.yaml`. The controller handles this automatically: - **On startup**: `InjectMissingFields()` runs for all deployed stacks - **After sync**: the post-sync hook runs for stacks whose templates were updated - For each deployed stack, compares `.felhom.yml` `deploy_fields` against `app.yaml` env vars - Missing `secret` fields: auto-generated using the field's generator spec (`password:N`, `hex:N`, `base64key:N`) - Missing `domain` fields: filled with the customer's configured domain - Missing `subdomain` fields: filled with the field's default value or the `.felhom.yml` `subdomain:` metadata - Other field types (e.g., `text`, `select`): logged as warning for manual configuration - Locked fields are added to the locked list automatically **Generator types**: `password:N` (alphanumeric), `hex:N` (hex-encoded random bytes), `base64key:N` (`base64:` + N random bytes base64-encoded, for Laravel APP_KEY etc.), `static:VALUE` (literal value). #### Container State Display | State | Color | Label | Meaning | |-------|-------|-------|---------| | Running + healthy | Green | "Fut" | All containers running and healthy | | Running + starting | Orange | "Indulas..." | Healthcheck not yet passed | | Deploying | Orange | "Telepítés..." | Compose up in progress (image pull, container creation) | | Running + unhealthy | Yellow | "Nem egeszseges" | Docker or controller-side healthcheck failing | | Stopped/exited | Red | "Leallitva" | All containers stopped | | Restarting | Yellow | "Ujrainditas..." | Restart loop | | Not deployed | Gray | "Nincs telepitve" | Compose file exists, not deployed | **Route-unpublished indicator (F5, v0.61.0).** Traefik's Docker provider only publishes a route to a container that is healthy (or has no healthcheck), so an `unhealthy`/`restarting` deployed app returns a hard **404** at its URL even though the container is running. The `routeUnpublished` template helper (`funcmap.go`) drives a distinct "URL nem elérhető – útvonal nincs publikálva" indicator on the dashboard and stacks cards for such apps, so a dead URL isn't mistaken for a merely-degraded-but-reachable one. #### Controller-side Health Probes (`internal/stacks/healthprobe.go`) For apps that declare a `healthcheck:` section in `.felhom.yml`, the controller probes the container directly over the Docker network (both are on `traefik-public`). This complements Docker-level healthchecks and is the **only** health mechanism for distroless/scratch images that lack shell utilities. Three probe types are supported: - **`http`** — Any HTTP response (even 4xx/5xx) = service is alive. Only connection refused/timeout = unhealthy. - **`api`** — HTTP request with response validation (expected status code, body content). Fails if expectations aren't met. - **`tcp`** — Simple port reachability check via `net.Dial`. Multiple checks per app are supported (all must pass). The probe scheduler runs every 10 seconds; per-app intervals default to 5 minutes and are configurable via `healthcheck.interval` in `.felhom.yml`. Probe results are stored in `Stack.HealthProbe` and exposed via the API. Failed probes override the stack state to `StateUnhealthy`; the override clears automatically when the next probe passes. **Fast initial probing:** On start/restart, stale health probe results are cleared (so the stack doesn't immediately appear "unhealthy" from a previous result). Until the first healthy probe, the controller checks every 10 seconds instead of the normal 5-minute interval, giving fast feedback on whether the app came up successfully. --- ### 2. App Export/Import (.fab bundles) Per-app export creates a self-contained `.fab` file (tar.gz, optionally encrypted) that can be stored externally or used to restore the app on the same server. Distinct from the automatic backup system — user-initiated, per-app, produces a single portable file. **Bundle contents:** `manifest.json` + `config/` (compose, .felhom.yml, app.yaml with plaintext secrets) + `database/` (gzipped SQL dump) + `data/` (HDD bind mount tars AND Docker named volume tars — **additive since v0.130.0**; a `needs_hdd` app bundles both). **Encryption:** Optional AES-256-CTR + HMAC-SHA256 with scrypt key derivation (N=32768). Format: `"FABE"` magic header + salt + IV + encrypted tar.gz + HMAC tag. Streaming for multi-GB files. **Export flow:** Estimate size → check free space → optionally stop app → copy config → dump DB → tar user data (docker volumes stream via `docker cp` through a stopped helper container — v0.125.0; never a `docker run -v` host mount, which strands data on containerized controllers) → **fail-loud bundle assertion** (every manifest-claimed tar must exist non-empty, AND a `needs_hdd` bundle claiming NO data at all is refused — v0.130.0 C6B-F1) → create tar.gz → optionally encrypt → atomic rename. App restarts automatically after export if it was stopped. **Mount discovery (v0.130.0, C6B-F1):** `stacks.ExportDataMounts` — the `${HDD_PATH}` binds unioned with the `${USERDATA_PATH}` **root** (single `userdata` entry) when the compose uses the standard userdata convention; pre-fix the adapter was `${HDD_PATH}`-only, so 12/13 `needs_hdd` catalog apps exported hollow (config-only) bundles. The root (not per-bind) keying is what round-trips through the import's basename→`/` mapping. A basename collision between mounts fails the export loudly. The share-removal endpoint also refuses while a deployed app's HDD_PATH is on the share (C6B-F2 guard). **Class-scoped export (v0.136.0, Task 4 — the SQ6 fix):** for a **classified** app the userdata root tar is **exclude-scoped** — it keeps only dirs that are an ancestor-or-descendant of a SELECTED bind relpath (mandatory ∪ checked-optional ∪ opted-in-excluded; R1-C, the `tier2Reconcile` keep-rule), so sibling apps' content no longer rides along. No selected userdata bind ⇒ the root tar is **skipped entirely** (radarr → state-only). Non-selected HDD bind mounts are skipped; a mount matching no classified bind is kept (fail toward capture). Mechanics unchanged: **ONE** userdata tar, per-mount skip, **manifest v1 + import untouched**. The plan is pure (`appexport/fabplan.go` `computeFabPlan` over `appbackup.ComputeFabBuckets`); `tarDirectoryExcluding` prunes excluded subtrees in the walk. **Legacy (no-block) apps export byte-identically to v0.130.0.** Mandatory paths are a **server-side floor** (a client cannot deselect them). The export page shows the class selection UI (locked mandatory, pre-selected optional checkboxes, opt-in excluded behind the two-number warning + FileBrowser pointer); the estimate carries an additive class split (`ExportEstimate.MandatoryItems/OptionalItems/ExcludedItems` + `BaseBytes`). **Import flow:** Decrypt if needed → extract → **validate-before-destroy** (v0.125.0: every manifest-claimed data tar must be present non-empty BEFORE the app is stopped or any volume removed — hollow bundles from containerized ≤0.124.0 exporters are refused with the app untouched) → prepare stack dir (create new or `compose down --volumes` for existing) → restore config (re-encrypt app.yaml with current server key) → restore user data (HDD tars; volumes populate via `docker cp` streaming) → restore DB (start DB service, wait for ready, import dump) → start full stack → refresh UI. **Architecture:** `internal/appexport/` package with `ExportStackProvider` adapter interface (same pattern as `backup.StackDataProvider`). `exportAdapter` in `main.go` bridges `stacks.Manager` to the provider. **API endpoints:** `/api/export/estimate`, `/api/export/start`, `/api/export/status`, `/api/export/bundles`, `/api/export/manifest`, `/api/export/import`, `/api/export/import/status`; **browser download (v0.124.0):** `/api/export/download/estimate`, `/api/export/download/start`, `GET /api/export/download?file=`; **browser upload (v0.128.0):** `POST /api/export/upload/{init,chunk,finalize,abort}`. **Browser download (v0.124.0 — portability, NOT a backup tier):** the same export pipeline runs with dest = `/fab-downloads/` (same producer → byte-identical bundle), then streams via a guarded endpoint (basename-shape + dir-containment guard; `Content-Disposition: attachment`; `io.Copy`; the staged bundle is removed after the stream and a 1h TTL sweep runs on startup + each start). Estimate is shown BEFORE starting; the batch UI downloads apps one at a time (no combined archive). `handler_export_download.go`. **Browser upload (v0.128.0 — the download's return leg):** the `/import` page uploads a `.fab` straight from the browser. Chunked because the Cloudflare tunnel caps request bodies at ~100 MB (probed live 2026-07-13: 120 MiB → edge 413, 80 MiB → origin): JS slices the file into 64 MiB chunks (strictly sequential offsets; one retry per chunk re-synced from the 409 `received_bytes` echo), the server streams each to a `.part-` file in the DEFAULT drive's exports dir (`io.Copy`, 96 MiB per-request cap, free-space gate = size + 1 GiB), finalize checks the exact declared size, fsyncs and atomically renames (collision → lowest-free `"name (N).fab"`). Single-flight; no client-side hash (the .fab format self-validates at import); in-memory state — startup GC sweeps `*.part-*`, 15-min idle timeout aborts server-side. The scan + validation + import pipeline are untouched. `handler_export_upload.go`. **UI:** Export button on app info page, the "Hordozható mentéscsomag (.fab)" section on `/backups/restore` (per-app download + batch), standalone import page at `/import` (upload zone + drive-scanned bundle list). --- ### 3. Backup System The backup system implements a **3-2-1 backup architecture**. Each tier is a **complete, self-sufficient backup** — any single tier can fully restore an app. **Sidebar behaviour (v0.146.0).** Groups that own sub-pages — Tárhely, Biztonsági mentés, Megosztás — render as **accordions**: the header is a real `