# felhom-controller

**Central management container for Felhom home servers.** The **in-guest controller**: one per
customer LXC, Docker-only, Hungarian web dashboard for managing the customer's app stacks, app-data
backups, monitoring and notifications. All Proxmox/disk operations are delegated to the host agent
(`felhom-agent`).

**Current version: v0.59.0**

> ## Documentation has moved (and is now code-verified)
>
> The **authoritative architecture & feature documentation** now lives in the central docs home:
> **[`felhom.eu/documentation/controller/`](../../felhom.eu/documentation/controller/README.md)** —
> module map, deploy & stack lifecycle, backup architecture, storage/monitoring/metrics, and
> auth/hub/sync/integrations. Those docs are grounded in current source (v0.59.0).
>
> **Quick build & deploy** is in the "Build & Deploy" section below and in the repo `CLAUDE.md`
> (authoritative for the workflow). NOTE the demo controller runs in an LXC guest (9201) under the
> **bootstrap-managed** mechanism (`/etc/felhom-controller-image` + `felhom-controller-bootstrap.service`),
> not the bare-metal `/opt/docker` compose path some sections below still describe.
>
> The prose below this banner is **retained legacy reference** and may lag the central docs — when they
> disagree, the central docs win. (Some sections still mention restic / pre-8C disk handling that has
> since moved to the host agent.)

---

## Table of Contents

- [Architecture](#architecture)
- [Features](#features)
  - [App Management](#1-app-management)
  - [App Export/Import](#2-app-exportimport-fab-bundles)
  - [Backup System](#3-backup-system)
  - [Storage Management](#4-storage-management)
  - [Monitoring & Health](#5-monitoring--health)
  - [Notifications](#6-notifications)
  - [Update Management](#7-update-management)
  - [Authentication & Settings](#8-authentication--settings)
  - [Central Hub](#9-central-hub-reporting)
  - [Setup Wizard](#10-first-run-setup-wizard)
  - [Disaster Recovery](#11-disaster-recovery)
  - [Asset Sync](#12-asset-sync)
  - [Debug Mode](#13-debug-mode)
  - [Geo-Restriction](#14-geo-restriction)
  - [App-to-App Integrations](#15-app-to-app-integrations)
- [Repository Layout](#repository-layout)
- [Configuration](#configuration)
- [REST API](#rest-api)
- [Build & Deploy](#build--deploy)
- [Roadmap](#roadmap)

---

## Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│  Customer Hardware (N100 mini PC / Raspberry Pi)                │
│                                                                 │
│  ┌──────────┐   ┌────────────────────────────────────────────┐  │
│  │ Traefik  │   │  felhom-controller (privileged container)  │  │
│  │ (reverse │──▶│                                            │  │
│  │  proxy)  │   │  ┌──────────┐  ┌─────────────────────────┐│  │
│  └──────────┘   │  │ Web UI   │  │ Stack Manager           ││  │
│                 │  │ (HU dash │  │ (compose ops, git sync,  ││  │
│  ┌──────────┐   │  │  board)  │  │  deploy, delete, update) ││  │
│  │cloudflared│   │  └──────────┘  └─────────────────────────┘│  │
│  │ (tunnel) │   │  ┌──────────┐  ┌─────────────────────────┐│  │
│  └──────────┘   │  │ Backup   │  │ Storage Manager         ││  │
│                 │  │ (3-layer │  │ (disk scan, format,     ││  │
│  ┌──────────┐   │  │  restic) │  │  mount, migrate)        ││  │
│  │ App      │   │  └──────────┘  └─────────────────────────┘│  │
│  │ stacks   │   │  ┌──────────┐  ┌─────────────────────────┐│  │
│  │ (docker  │   │  │Scheduler │  │ Monitor & Metrics       ││  │
│  │ compose) │   │  │(cron-like│  │ (health, SQLite         ││  │
│  └──────────┘   │  │  jobs)   │  │  time-series, Chart.js) ││  │
│                 │  └──────────┘  └─────────────────────────┘│  │
│                 │  ┌──────────┐  ┌─────────────────────────┐│  │
│                 │  │ Notify   │  │ REST API + Hub Reporter ││  │
│                 │  │ (events) │  │ (JSON push + events)    ││  │
│                 │  └──────────┘  └─────────────────────────┘│  │
│                 │  ┌──────────┐                              │  │
│                 │  │ Assets   │                              │  │
│                 │  │ (Hub     │                              │  │
│                 │  │  sync)   │                              │  │
│                 │  └──────────┘                              │  │
│                 └────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘
         │ events + reports      │ git pull       │ asset sync
         ▼                       ▼                ▼
   hub.felhom.eu           gitea.dooplex.hu  hub.felhom.eu
   (central dashboard)     (stack definitions) (logos, screenshots)
```

### Key Architecture Decisions

- **Pure Go, no frameworks** — stdlib `net/http` + `html/template`. Only external deps: `bcrypt`, `yaml.v3`, `modernc.org/sqlite` (pure Go, no CGO).
- **Privileged container** — Required for disk operations (format, mount, fstab), `/dev` access, and Docker socket control.
- **`/host-dev` indirection** — Docker overrides `/dev` with a tmpfs. The host's `/dev` is mounted at `/host-dev` to access block devices.
- **`StackDataProvider` interface** — Breaks circular import between the backup packages and stacks. Defined in `internal/appbackup` (and re-exposed via a type alias in `internal/backup`). Implemented by `stackAdapter` in `main.go`. Provides `GetStackHDDPath()` for per-drive backup routing.
- **Atomic file writes** — All persistent state (`settings.json`, `app.yaml`) written to `.tmp` then `os.Rename` for crash safety.
- **`go:embed` templates** — All HTML/CSS/JS compiled into the binary. No runtime file dependencies.
- **Design system v2 (v0.96.0, TASK-D0)** — The whole UI renders in the Felhom v2 language (canonical
  reference: `felhom.eu/documentation/design/design-system.md`): navy token palette, single 2px radius,
  no shadows, exception-based status color (nominal = blue/neutral; amber/red only on deviation —
  `stateColor` emits `run/progress/warn/neutral/off`, `usageColor`/`tempColor` emit
  `nominal/warn/crit`; a stopped app is neutral, NOT red). Capacity bars are the hairline `.meter`
  component; state chips are `.tag`; informational pills are `.metarow`. **Fonts (Plus Jakarta Sans +
  JetBrains Mono, variable woff2, latin+latin-ext) and a 30-icon Lucide sprite are vendored in the
  binary** (`internal/web/static/fonts/` served at `/static/fonts/`; `templates/icons.html`) — no CDN,
  no emoji. The setup wizard serves the same embedded stylesheet via `web.StyleCSS()`.
- **Page IA (v0.97.0, TASK-D1)** — the settings monolith is split into four pages, each with its own
  data builder (`systemPageData`/`storagePageData`/`notificationsPageData`/`securityPageData`, sharing
  `settingsBaseData`) and template. Routes: `/settings` (Rendszer), `/settings/notifications`
  (GET→page, POST→save on the same path), `/settings/security`, `/storage` (main-nav **Tárhely** — Meghajtók), and `/storage/network` (Hálózati tárhely / NAS; v0.98.0 split — nested sub-links under Tárhely).
  The enrollment wizards live at `/storage/{init,attach}`; `/settings/storage/{init,attach}` **301**
  to them; all storage action successes redirect to `/storage?storage_msg=…`. `/storage` shows a
  **unified drive view**: server-rendered registry cards enriched in place from the agent `/api/disks`
  (role/durable-id/actions, joined on mount path), plus read-only **Rendszermeghajtók** and
  **Nem regisztrált meghajtók** groups; agent-down degrades to a single warn note. Consequential
  actions use an in-page `.confirm-overlay` (`openDialog`), never native `confirm()`/`prompt()`.
  **Enforcement:** `scripts/template_id_gate.py` (JS element-ID integrity) + `scripts/emoji_gate.py`
  (no emoji — run after any template change; the Go `TestNoEmojiInTemplates` mirrors it).
- **Europe/Budapest timezone** — All scheduled jobs, timestamps, and UI labels use Hungarian timezone.

### Module Map

| Module | Path | Responsibility |
|--------|------|----------------|
| **Config** | `internal/config/` | YAML loader, validation, `FELHOM_*` env overrides |
| **Settings** | `internal/settings/` | Runtime-mutable `settings.json` (passwords, backup prefs, storage paths, notifications) |
| **Stacks** | `internal/stacks/` | Compose operations, scanning, `.felhom.yml` metadata, deploy/delete flow; **base-infra bring-up** (`infra.go` — `EnsureBaseStack`) |
| **Infra** | `internal/infra/` | Pure renderers (embedded `text/template`) for the base-infra stacks (traefik/cloudflared/filebrowser); **pinned image tags as the single source of truth** (web filebrowser sync delegates here) |
| **Crypto** | `internal/crypto/` | AES-256-GCM encryption for sensitive app.yaml values (passwords, secrets), key management |
| **Sync** | `internal/sync/` | Git-based app catalog sync (clone/pull, content-hash copy) |
| **AppBackup** | `internal/appbackup/` | Self-contained app-data backup primitives: DB dump discovery/execution (`DiscoverDatabases`, `DumpOne`), Docker-volume/app-data discovery (`StackDataProvider`, `DiscoverAppData`), keep-side path helpers (`AppDBDumpPath`, `AppVolumeDumpPath`, `AppDataDir`). `DiscoverDatabases` takes the deployed-stack set so a DB container maps to the right stack even when a slug ends in a DB-role token (M19, v0.62.0). `ListDumpFiles` takes an optional `cached(name,size,mod)` lookup so an unchanged dump isn't re-validated (line-scan) every ~5-min cycle (M18, v0.62.0). No dependency on restic/cross-drive/drive-mount. Imported directly by `appexport` and `storage`. |
| **Backup** | `internal/backup/` | Per-drive 3-layer backup: DB dumps → restic snapshots → cross-drive copies, restore. Re-exposes the `appbackup` primitives via aliases/forwarders (`appbackup_bridge.go`) for the disk/host-side code and the web/api/report consumers. |
| **Storage** | `internal/storage/` | Disk scanning (`lsblk`), partitioning (`sfdisk`), formatting (`mkfs.ext4`), mounting, data migration (`rsync`) |
| **System** | `internal/system/` | System info (`/proc`), CPU collector, mount points, disk usage, FS info |
| **Monitor** | `internal/monitor/` | System health checks, storage watchdog, legacy Healthchecks pinger (deprecated) |
| **Metrics** | `internal/metrics/` | SQLite time-series store, system + container metric collection |
| **Scheduler** | `internal/scheduler/` | Central job scheduler (periodic + daily, skip-if-running, panic recovery) |
| **SelfUpdate** | `internal/selfupdate/` | Version checking (registry), update trigger, state persistence, startup verification |
| **Notify** | `internal/notify/` | Email notifications via hub relay, preference sync, per-event cooldowns |
| **Report** | `internal/report/` | Hub report builder + HTTP pusher (system, stacks, backup, health) |
| **Assets** | `internal/assets/` | Hub-managed asset syncer: downloads logos/screenshots with SHA-256 change detection |
| **SelfTest** | `internal/selftest/` | Startup self-test: 9 diagnostic checks (Docker, dirs, storage, hub, restic, metrics) |
| **Util** | `internal/util/` | Shared utilities: `TruncateStr` for debug log output truncation |
| **AppExport** | `internal/appexport/` | Per-app export/import via `.fab` bundles (config + DB + user data), optional AES-256 encryption |
| **API** | `internal/api/` | REST JSON endpoints, diagnostic dump (`/api/debug/dump`) |
| **Web** | `internal/web/` | Hungarian dashboard, auth, page handlers, template functions, alerts |

---

## Features

### 1. App Management

The controller manages Docker Compose stacks through a complete lifecycle: catalog sync, first-time deployment, runtime operations, and deletion.

#### Git Sync (`internal/sync/`)

The app catalog lives in a separate Git repository. The controller:
- Shallow-clones the catalog on startup
- Periodically fetches updates (configurable, default 15 min)
- Copies only `docker-compose.yml` and `.felhom.yml` to the stacks directory
- **Never overwrites** `app.yaml` (user secrets are safe)
- Uses SHA-256 content hashing — only writes files that actually changed
- Triggers stack rescan after sync so the dashboard updates immediately
- **Post-sync hook**: auto-injects missing deploy fields (new secrets, domains) into existing `app.yaml` for stacks whose templates were updated (see Missing Field Injection below)
- Manual sync via "Sablonok frissitese" button or `POST /api/sync`

#### First-Time Deploy Flow

1. Customer sees app card with "Telepites" button
2. Deploy page pre-generates and **displays** all auto-values before the user clicks deploy:
   - `domain` fields: shown as readonly text input with the customer's configured base domain
   - `subdomain` fields: editable text input pre-filled with the default from `.felhom.yml`, shown with `.base-domain` suffix. Validated for DNS-safe format, reserved names, and uniqueness across deployed stacks. Locked after deploy — changing requires Remove + Redeploy
   - `secret` fields: pre-generated and shown as masked password inputs with a "Megjelenítés" reveal button — user can see/copy all DB passwords and keys before deploying
   - User-configurable inputs (admin password, language, storage path) remain editable
   - Section header prompts the user to note down any passwords they need
3. `checkBeforeDeploy()` JS guard fetches live state first (prevents double-deploy from another tab)
4. **Memory validation** (F1, v0.61.0): the controller runs as a Docker container inside an LXC, where
   `/proc/meminfo` shows the **Proxmox host's** RAM (no lxcfs in the container) and the container's own
   cgroup is unlimited (the guest cap lives on the LXC ancestor). So the guest cap is read from the
   **Docker daemon** (`system.GuestMemTotalMB()` → `docker info` MemTotal — the daemon runs in the LXC and
   reports the guest's lxcfs-backed RAM; the cgroup limit is preferred when present, e.g. non-nested):
   - `usable_memory = guest_cap - reserved_memory_mb` (default 384MB reserved)
   - Hard block if `committed_used + new_request > usable_memory`, where `committed_used` =
     `CommittedMemory()` (sum of running apps' mem requests) — the guest-wide RSS is not observable from
     the container, so the controller's own committed accounting is the accurate, cheap "used".
   - `/api/system/info` reports the guest cap as total and committed memory as used.
4b. **Docker-data volume reserved-buffer gate (v0.58.0, storage-split prevention layer):** the OS rootfs
   and Docker data are split onto separate volumes; infra (controller/traefik/cloudflared/filebrowser)
   shares the one Docker data-root (`/var/lib/docker`) and is protected by **prevention, not placement**.
   `system.GetDockerVolumeHeadroom()` measures the Docker-data volume via `statfs("/")` (the controller
   container's overlay root is the upperdir on that volume — true with the golden's **overlay2** driver)
   and reserves `max(5 GB, 10%)`. `deployStack` **refuses a new deploy (HTTP 507)** when free space is
   at/under the buffer; the deploy page shows the warning + disables the button. Fail-open on a statfs
   error. The runtime disk monitor (`healthcheck.go`, warn 80% / crit 90%) watches the same volume and
   trips above the buffer. (Assumes the split guest's large data volume; the golden bakes overlay2 + log
   rotation so images+volumes live on the data volume, not `/var/lib/containerd`.)
5. Pre-generated secret values are submitted as hidden form inputs so the **same values** the user saw are saved to `app.yaml` (no silent re-generation on submit). Controller saves `app.yaml`, sets in-memory `Deployed` + `Deploying` flags, then runs `docker compose up -d` **asynchronously** in a goroutine — API returns immediately so the UI switches to the progress panel without waiting for image pulls. On failure the goroutine reverts both disk and in-memory state and sets `DeployError`.
6. 3-step progress panel polls `GET /api/stacks/{name}` every 3s: config saved → `deploying` (pulling images) → containers starting → health check passed. New `StateDeploying` state shown while compose-up is in progress (no containers yet).
7. Post-deploy: locked fields (DB_PASSWORD, etc.) become read-only; the "Automatikusan generált értékek" section continues to show the saved values on the settings page
8. The deploy/settings page includes **start/stop/restart** buttons for deployed apps, plus a "Megnyitás ↗" link to the app's subdomain URL (only visible when running)

#### Catch-All Page for Stopped Apps

When a user visits a stopped or undeployed app's subdomain (e.g., `travel.demo-felhom.eu`), the controller serves a branded error page instead of Traefik's raw 404:

- **Traefik catch-all router**: The controller's `docker-compose.yml` registers a second router (`catchall`) with `priority=1` (lowest) and `HostRegexp(.+)`. Running apps always win; only requests with no matching container reach the controller.
- **`CatchAllMiddleware`** in `server.go` intercepts requests where `Host` ≠ `felhom.DOMAIN`, serves the catch-all page **without auth** (user has no session on the app subdomain).
- **`findStackBySubdomain()`** identifies the app by matching the subdomain against deployed `app.yaml` `SUBDOMAIN` env or metadata fallback.
- **`catchall.html`** — standalone template (no layout, inline CSS) showing the app name, status ("leállítva" / "nincs telepítve" / "nem található"), and links to the controller dashboard or the app's detail page.
- **Subdomain links** on the Alkalmazások page are only shown for deployed apps (non-deployed apps have no guaranteed subdomain yet).

#### Dashboard "Megnyitás" Button

Running apps on the Vezérlőpult now show a "Megnyitás ↗" button that opens the app's subdomain in a new tab. The `Subdomains` map is built in `dashboardHandler` from `app.yaml` env or metadata fallback.

The optional `.felhom.yml` **`open_path`** field (`Metadata.OpenPath`) is appended to that URL for apps whose UI isn't at `/` — e.g. `gokapi` → `/admin`, `ghost` → `/ghost/`. Empty = bare root. Rendered via `.Meta.OpenPath` in `dashboard.html`, `deploy.html`, and `app_info.html` (the same field flows through all three open-link sites; no handler change needed). Must start with `/`; cosmetic only (does not affect routing).

#### App Info Pages

Each app can define rich metadata in `.felhom.yml`:
- `app_info`: tagline, use_cases, first_steps, prerequisites, default_creds, docs_url
- `optional_config`: groups of post-deploy configurable env vars (e.g., API keys for metadata providers)
- `resources`: mem_request, mem_limit, pi_compatible, needs_hdd, hungarian_ui
- `initial_credentials` (v0.84.0): for apps that **auto-generate a first-login password into a file** at
  first boot (vs taking it from a deploy field) — e.g. Crafty → `/crafty/app/config/default-creds.txt`.
  Fields: `file` (path inside the container), `format` (`json`|`regex`|`plain`), optional `container`
  (defaults to the stack's main container), `username_key`/`password_key` (json), `username_pattern`/
  `password_pattern` (regex, first capture group), `note`. `ReadInitialCredentials`
  (`internal/stacks/initialcreds.go`) reads the file **live** via `docker exec … cat` and parses it with
  the pure `parseInitialCreds`; the value is **never persisted** to `app.yaml`. `appDetailHandler`
  surfaces it on `/apps/{slug}` as a "Kezdeti belépési adatok" card (masked password + reveal/copy),
  labelled as the *initial* credential (stays valid only until the customer changes it in-app). Hidden
  when the container is down / file missing / parse fails. Reuse for any future self-seeding app.

The `/apps/{slug}` page renders hero section, screenshots, setup guide, and optional config form.

#### Stack Operations

| Operation | What it does |
|-----------|-------------|
| Start | `docker compose up -d` — pre-start memory check rejects with 409 if insufficient RAM |
| Stop | `docker compose stop` (blocked for protected stacks) |
| Restart | `docker compose restart` |
| Update | `docker compose pull` + `docker compose up -d` |
| Remove | `docker compose down --volumes` + remove `app.yaml` + optional HDD/backup cleanup; template preserved for redeploy |
| Delete | `docker compose down --rmi local --volumes` + optional HDD data cleanup (orphaned stacks only) |

**Remove vs Delete**: "Eltávolítás" (Remove) is for deployed catalog stacks — it reverts the stack to "Nincs telepítve" state while keeping the template for easy redeployment. "Törlés" (Delete) is for orphaned stacks — it removes the entire stack directory including templates. Both require stopping the stack first.

**Remove modal** shows three sections: (1) always-removed items (Docker volumes, app.yaml, cross-drive schedule), (2) optional HDD data deletion with reimport warning, (3) optional backup data deletion (DB dumps + cross-drive rsync) with restic retention note.

**Protected stacks** (traefik, cloudflared, felhom-controller) cannot be stopped, removed, or deleted from the UI. Restart is allowed.

**Orphan detection**: Deployed stacks with no matching catalog template are marked as orphaned with an "Elavult" badge and can be safely deleted.

#### Base-infrastructure bring-up (`stacks/infra.go` + `internal/infra/`, v0.41.0)

The controller stands up its own base stack — **traefik** (reverse proxy), **cloudflared** (external tunnel), **filebrowser** — instead of relying on the bare-metal `scripts/docker-setup.sh` (which a Proxmox-provisioned guest never runs). `internal/infra` renders the compose + config files from `controller.yaml` via embedded `text/template`s (lifted from `docker-setup.sh`); image tags are **pinned constants there** (`TraefikImage`/`CloudflaredImage`/`FileBrowserImage`) and the web FileBrowser sync path delegates to the same renderers, so the pinned versions can never diverge.

`Manager.EnsureBaseStack()` creates the `traefik-public` network, then deploys traefik → cloudflared → filebrowser under `${stacks_dir}/<name>`. It is:
- **single-flight** (a `TryLock` guard — it's called from both first boot and every health tick, so overlapping runs must not race on the same stack dir),
- **idempotent** (skips a stack whose container is already running; never overwrites an existing filebrowser compose, preserving the storage mounts `SyncFileBrowserMounts` manages),
- **non-fatal** (logs, never crashes the controller).

cloudflared is only deployed when a tunnel token is configured. **Triggers**: a first-boot goroutine (after stack init) and an unconditional call on every `system-health` tick (self-heal — cheap when healthy thanks to the idempotency). `monitor.EffectiveProtected` mirrors the cloudflared condition so a LAN-only node (no tunnel token) doesn't report a perpetual "protected container not running" FAIL.

**Backend transports — self-signed HTTPS backends (`ensureServersTransports` → `RenderServersTransports`, v0.83.0).** Traefik talks **HTTP** to app backends by default, which is correct for every catalog app that serves plain HTTP. The exception is an app that serves its **own self-signed TLS** on the internal docker bridge (the first is Crafty, HTTPS-only on `:8443`): Traefik must speak `https` to it *and* skip verifying a per-container self-signed cert (no CA to verify against; the hop never leaves the host). `insecureSkipVerify` is **not settable via Docker labels** in traefik v3 — it must live in static/file config — so `EnsureBaseStack` writes a file-provider dynamic file `dynamic/serverstransports.yml` defining a **named** transport `insecure-skip-verify` (write-if-changed; hot-loaded by the file watcher). An app **opts in per-service** via two catalog labels — `loadbalancer.server.scheme=https` + `loadbalancer.serverstransport=insecure-skip-verify@file` (the `@file` suffix is the cross-provider reference). Backend verification stays the default (ON) for every other service — there is deliberately **no global `insecureSkipVerify`** in `traefik.yml`. This write runs **outside** `ensureTraefik` (which early-returns when traefik is already up) so an established node still materializes the file on a self-heal tick.

> **Mount prerequisite (Section-G):** the controller writes these stacks under `/opt/docker/stacks` *inside its container*, but `docker compose up` runs on the **guest** Docker daemon. The golden's controller-bootstrap (`felhom-agent` `build-golden.sh`) therefore bind-mounts that path **same-path** (`-v /opt/docker/stacks:/opt/docker/stacks`) so the daemon resolves every relative bind source — without it, all bind-mounted stacks (base infra and customer apps) silently break.

**Controller routing + the wildcard cert anchor (`wireController` → `RenderControllerRoute`, v0.41.1 / v0.42.1).** filebrowser self-registers with traefik via Docker labels + `traefik-public` membership baked into its compose; the controller can't (it's started by the golden bootstrap *before* `traefik-public` exists, and the v2 `bootstrap.json` carries no domain — that comes from the hub pull). So `EnsureBaseStack` wires the controller **post-pull**: it `docker network connect traefik-public felhom-controller` and writes a traefik file-provider route `dynamic/controller.yml` (`Host(felhom.<domain>) → http://felhom-controller:8080`, write-if-changed). When DNS-01 ACME is configured, that route is **also the wildcard-cert anchor**: its router-level `tls.domains: *.<domain>` makes traefik **proactively obtain the wildcard `*.<domain>` + apex via Cloudflare DNS-01 at startup** (an entrypoint-level `http.tls.domains` does *not* trigger issuance in traefik v3 — only a router-level `tls.domains` does). Every other router then serves that one real wildcard cert by SNI — no per-app `certresolver` labels. This is what lets a LAN client reach the box directly at `*.<domain>` with the real cert (the `felhom-agent` split-horizon resolver depends on it).

#### Missing Field Injection (`deploy.go`)

When app templates are updated (e.g., a new `APP_KEY` secret is added to `.felhom.yml`), existing deployed apps need the new field in their `app.yaml`. The controller handles this automatically:

- **On startup**: `InjectMissingFields()` runs for all deployed stacks
- **After sync**: the post-sync hook runs for stacks whose templates were updated
- For each deployed stack, compares `.felhom.yml` `deploy_fields` against `app.yaml` env vars
- Missing `secret` fields: auto-generated using the field's generator spec (`password:N`, `hex:N`, `base64key:N`)
- Missing `domain` fields: filled with the customer's configured domain
- Missing `subdomain` fields: filled with the field's default value or the `.felhom.yml` `subdomain:` metadata
- Other field types (e.g., `text`, `select`): logged as warning for manual configuration
- Locked fields are added to the locked list automatically

**Generator types**: `password:N` (alphanumeric), `hex:N` (hex-encoded random bytes), `base64key:N` (`base64:` + N random bytes base64-encoded, for Laravel APP_KEY etc.), `static:VALUE` (literal value).

#### Container State Display

| State | Color | Label | Meaning |
|-------|-------|-------|---------|
| Running + healthy | Green | "Fut" | All containers running and healthy |
| Running + starting | Orange | "Indulas..." | Healthcheck not yet passed |
| Deploying | Orange | "Telepítés..." | Compose up in progress (image pull, container creation) |
| Running + unhealthy | Yellow | "Nem egeszseges" | Docker or controller-side healthcheck failing |
| Stopped/exited | Red | "Leallitva" | All containers stopped |
| Restarting | Yellow | "Ujrainditas..." | Restart loop |
| Not deployed | Gray | "Nincs telepitve" | Compose file exists, not deployed |

**Route-unpublished indicator (F5, v0.61.0).** Traefik's Docker provider only publishes a route to a
container that is healthy (or has no healthcheck), so an `unhealthy`/`restarting` deployed app returns a
hard **404** at its URL even though the container is running. The `routeUnpublished` template helper
(`funcmap.go`) drives a distinct "URL nem elérhető – útvonal nincs publikálva" indicator on the dashboard
and stacks cards for such apps, so a dead URL isn't mistaken for a merely-degraded-but-reachable one.

#### Controller-side Health Probes (`internal/stacks/healthprobe.go`)

For apps that declare a `healthcheck:` section in `.felhom.yml`, the controller probes the container directly over the Docker network (both are on `traefik-public`). This complements Docker-level healthchecks and is the **only** health mechanism for distroless/scratch images that lack shell utilities.

Three probe types are supported:
- **`http`** — Any HTTP response (even 4xx/5xx) = service is alive. Only connection refused/timeout = unhealthy.
- **`api`** — HTTP request with response validation (expected status code, body content). Fails if expectations aren't met.
- **`tcp`** — Simple port reachability check via `net.Dial`.

Multiple checks per app are supported (all must pass). The probe scheduler runs every 10 seconds; per-app intervals default to 5 minutes and are configurable via `healthcheck.interval` in `.felhom.yml`. Probe results are stored in `Stack.HealthProbe` and exposed via the API. Failed probes override the stack state to `StateUnhealthy`; the override clears automatically when the next probe passes.

**Fast initial probing:** On start/restart, stale health probe results are cleared (so the stack doesn't immediately appear "unhealthy" from a previous result). Until the first healthy probe, the controller checks every 10 seconds instead of the normal 5-minute interval, giving fast feedback on whether the app came up successfully.

---

### 2. App Export/Import (.fab bundles)

Per-app export creates a self-contained `.fab` file (tar.gz, optionally encrypted) that can be stored externally or used to restore the app on the same server. Distinct from the automatic backup system — user-initiated, per-app, produces a single portable file.

**Bundle contents:** `manifest.json` + `config/` (compose, .felhom.yml, app.yaml with plaintext secrets) + `database/` (gzipped SQL dump) + `data/` (HDD bind mount tars or Docker named volume tars).

**Encryption:** Optional AES-256-CTR + HMAC-SHA256 with scrypt key derivation (N=32768). Format: `"FABE"` magic header + salt + IV + encrypted tar.gz + HMAC tag. Streaming for multi-GB files.

**Export flow:** Estimate size → check free space → optionally stop app → copy config → dump DB → tar user data → create tar.gz → optionally encrypt → atomic rename. App restarts automatically after export if it was stopped.

**Import flow:** Decrypt if needed → extract → prepare stack dir (create new or `compose down --volumes` for existing) → restore config (re-encrypt app.yaml with current server key) → restore user data (HDD or volumes) → restore DB (start DB service, wait for ready, import dump) → start full stack → refresh UI.

**Architecture:** `internal/appexport/` package with `ExportStackProvider` adapter interface (same pattern as `backup.StackDataProvider`). `exportAdapter` in `main.go` bridges `stacks.Manager` to the provider.

**API endpoints:** `/api/export/estimate`, `/api/export/start`, `/api/export/status`, `/api/export/bundles`, `/api/export/manifest`, `/api/export/import`, `/api/export/import/status`.

**UI:** Export button on app info page, standalone import page at `/import` accessible from the stacks page header.

---

### 3. Backup System

The backup system implements a **3-2-1 backup architecture**. Each tier is a **complete,
self-sufficient backup** — any single tier can fully restore an app.

| Tier | Contents | Location | Can fully restore? |
|------|----------|----------|--------------------|
| **1. Nightly restic** | DB + Config + User data | Same drive as app | Yes (not against drive failure) |
| **2. Cross-drive** | DB + Config + User data | Different physical device | Yes |
| **3. Remote** | Everything | Cloud / remote server | Future |

**Key principles:**
- User data backup is **mandatory** — every app with HDD bind mounts is included
  automatically. There is no per-app toggle.
- Each tier includes **everything** needed to restore: DB dumps, config, and user data.
  No tier depends on another tier's data.
- **Tier 2 is configurable for ALL apps** — not just apps with HDD data. Non-HDD apps
  back up config + DB dumps to the secondary drive (small but protects against drive failure).
  **(v0.118.0, F6):** the nightly tier-2 RUN (`RunAllTier2`) now actually includes volume-only apps
  too — it used to skip every non-HDD app, leaving them a single controller-level copy on `sys_drive`.
  A sys_drive app's restore-point drive label is now clear ("Belső SSD (rendszer)"), never blank.
- **3-2-1 on single vs multi drive (v0.118.0, F6):** on a box with a second physical drive, tier-2 is
  the off-drive copy. On a **single-drive** box (no off-drive target at all) there is genuinely only
  ONE local copy — `FullBackupStatus.SingleCopyWarning` surfaces an honest Hungarian notice on the
  backup page ("Csak egy másolat készül…") instead of implying a 3-2-1 guarantee the box cannot keep.
- The `AppBackupPrefs.Enabled` field in settings.json is legacy and not read by any code.

**Per-app Tier 2 contents by app type:**

| App type | Tier 2 contents | Example |
|----------|----------------|---------|
| HDD + DB | Config + DB + User data | Immich, Paperless-ngx |
| HDD, no DB | Config + User data | — |
| Docker volumes + DB | Config + DB + Volume data | Tandoor |
| Docker volumes, no DB | Config + Volume data | Mealie (SQLite) |
| DB, no HDD/volumes | Config + DB | Vikunja |
| Config only | Config | Gokapi, Homepage |

#### Tier 1: Nightly Backup (mandatory, same drive)

The nightly backup has two phases that run sequentially. All paths are **per-drive** — each physical drive gets its own restic repo and per-app DB dump directories.

> **Atomic dump writes (v0.118.0, CAMPAIGN-3 F7).** BOTH dump paths are crash-safe: the DB dump
> (`dbdump.go` DumpOne) and the Docker-volume dump (`DumpAppVolumes`) write to a `.tmp` sibling, fsync,
> then `os.Rename` over the restore point ONLY on success. A mid-write failure (a NFS cut mid-tar, an
> EIO, a timeout) removes only the `.tmp` and leaves the last good `.sql`/`.tar` **byte-untouched** —
> a tier-1 restore is replace-semantics, so an in-place write that got truncated to 0 bytes used to
> destroy the only restore point. `.tar.tmp` files are invisible to the restore-point/stale scans and
> orphans are swept on the next run.

> **Stale-primary sweep (v0.118.0, F5).** After each cycle, `pruneStalePrimaryDirs` removes an
> orphaned `backups/primary/<app>` dir left on an OLD drive when an app's `HDD_PATH` moved to another
> drive. Guarded: only for a DEPLOYED app whose CURRENT drive differs from the dir's drive; never the
> current-drive dir (the live restore point) or an undeployed app's dir; strictly under `backups/primary/`.

> **NAS backup locality (v0.118.0, CAMPAIGN-3 Part 4 — decision A).** A NAS-resident app's tier-1
> artifacts live **on the NAS itself** (`nas-media/backups/primary/<app>`), beside the data. During a
> NAS outage both the app data AND its freshest tier-1 dump are on the dead device — the **tier-2**
> cross-drive copy to a local drive is the off-NAS leg that saves them (and only after it has run).
> This locality is deliberate (kept in the fork over retargeting tier-1 to a local drive); the tier-2
> copy is the mitigation. Stated here so the outage window is never a surprise.

**Drive layout (v0.26.0):**
```
<drive>/
├── felhom-data/                ← all controller-managed data (namespace, v0.26.0+)
│   ├── appdata/<app>/          ← app user data
│   └── backups/
│       ├── primary/
│       │   ├── restic/              ← one restic repo per drive (all apps on this drive)
│       │   └── <app>/
│       │       ├── db-dumps/       ← per-app DB dump files
│       │       └── volume-dumps/   ← per-app Docker volume tars (v0.33.0)
│       └── secondary/
│           ├── restic/         ← secondary restic repo (cross-drive)
│           ├── _infra/         ← infra config mirror
│           └── <app>/rsync/    ← per-app rsync data
├── .felhom-infra-backup/       ← DR marker (stays at drive root for scanner)
├── Dokumentumok/               ← user files (not controller-managed)
└── media/                      ← user files (not controller-managed)
```

> **Note (Model A — corrected in v0.52.0):** `HDD_PATH` in `app.yaml` is the **in-guest mount point**
> (e.g., `/mnt/felhom-usb`). Under slice-10 Model A the host agent binds `<drive>/felhom-data` directly
> onto that mount, so the in-guest mount **already is** the `felhom-data` namespace root. Neither the
> compose templates nor the path helpers add a `felhom-data` segment for a drive-resident app: app data
> is `${HDD_PATH}/appdata/<app>` and backups `${HDD_PATH}/backups/...`, **single-nested**. Only the
> SSD-only system-data fallback (a bare root, `inGuestDrive=false`) appends `felhom-data`. See
> `NamespaceRoot(drivePath, inGuestDrive)` in `internal/appbackup/paths.go`.
> Earlier catalog templates used `${HDD_PATH}/felhom-data/appdata/<app>`, which double-nested to
> `.../felhom-data/felhom-data/...` on a Model-A drive; v0.52.0 dropped that segment in the catalog and
> locks deploy↔backup path agreement with `internal/stacks/hddpath_agreement_test.go`.

Path computation is centralized in `backup/paths.go` via the `FelhomDataDir = "felhom-data"` constant:
- `PrimaryResticRepoPath(drivePath)` → `<drive>/felhom-data/backups/primary/restic/`
- `AppDBDumpPath(drivePath, stackName)` → `<drive>/felhom-data/backups/primary/<stack>/db-dumps/`
- `AppVolumeDumpPath(drivePath, stackName)` → `<drive>/felhom-data/backups/primary/<stack>/volume-dumps/`
- `AppDataDir(drivePath, stackName)` → `<drive>/felhom-data/appdata/<stack>/`
- `SecondaryResticRepoPath(drivePath)` → `<drive>/felhom-data/backups/secondary/restic/`
- `AppSecondaryRsyncPath(drivePath, stackName)` → `<drive>/felhom-data/backups/secondary/<stack>/rsync/`
- `SecondaryInfraPath(drivePath)` → `<drive>/felhom-data/backups/secondary/_infra/`
- `InfraBackupDir(mountPath)` → `<drive>/.felhom-infra-backup/` (**unchanged** — stays at drive root for DR scanner)

> **⚠️ Stale:** the restic/secondary helpers above (`PrimaryResticRepoPath`, `SecondaryResticRepoPath`,
> `AppSecondaryRsyncPath`, `SecondaryInfraPath`) describe the pre-strip layout — restic/cross-drive was
> removed in slice 8C. This section is rewritten when Tier 2 (Phase 3) lands.

#### Per-app recovery unit (Phase 2, v0.53.x) — SECRET-FREE

Each app's `backups/primary/<app>/` is a self-contained, recreatable **recovery unit**:

```
backups/primary/<app>/
├── compose/        docker-compose.yml + .felhom.yml + a SECRET-STRIPPED app.yaml
├── db-dumps/       app-consistent DB dump(s)
├── volume-dumps/   named-volume tars
└── manifest.json   image pins, secret env-var NAMES, data_key names, checksums, secret_source
```

- **Secret-free by design.** The unit stores **no secret value, no data-encrypting key, and not the
  Docker image** — only the pinned image tag(s) (re-pulled on restore) and the *names* of the secret /
  `data_key` env vars. Rationale: app.yaml + the encryption key live on the guest rootfs → already in
  the PBS whole-guest snapshot, and the hub is deliberately zero-knowledge. Restore recovers the
  original secrets from the guest's own app.yaml (live, or via PBS); for a `data_key` app it
  **fails closed** (refuse + warn) if the key can't be recovered — data-keys are NEVER generated.
  **Resettable secrets (O4, v0.99.0):** an unrecoverable resettable secret (DB password etc.) gets a
  **generated replacement** from its catalog `generate` spec (`stacks.GenerateSecretForField` via the
  `backup.SetSecretGenerator` seam) instead of redeploying blank (which failed compose-up); the new
  value persists encrypted through the normal `RecreateStackFromUnit` → `SaveAppConfig` path. Fields
  with no `generate` spec still proceed with a loud "may fail to start" WARN. Residual case: a restored
  volume tar carrying the OLD internal credential hash may still need a manual in-DB reset.
- Helpers: `RecoveryUnitPath` / `RecoveryUnitComposePath` / `RecoveryUnitManifestPath`
  (`internal/appbackup/paths.go`). Capture: `Manager.CaptureRecoveryUnit` (`internal/backup/recovery_unit.go`),
  run from the daily DB dump and the periodic `RefreshCache` (idempotent checksum-skip). The non-secret
  env comes from `StackDataProvider.GetStackRecoveryInfo` (excludes secret-named + encrypted values, so
  the capture never touches a secret). `data_key` fields are marked in `.felhom.yml`
  (`DeployField.DataKey`).
- **Restore replays the DB dump (F17, v0.61.0).** `RestoreFromRecoveryUnit` (and the `RestoreApp`
  fallback) stops the app → restores named-volume tars → recreates the compose definition + redeploys
  with the recovered env → **replays each `db-dumps/*.sql` into the now-running DB** via
  `backup.reimportDBDumps` → `appbackup.ImportDump` (psql / mariadb client, using the live container's own
  discovered credentials). The DB replay runs AFTER the volume restore, so the **logical SQL dump wins**
  over any volume-tar copy of the database (the dumps use DROP/CREATE — `pg_dump --clean --if-exists`,
  `mariadb-dump` default `--add-drop-table` — so replay is idempotent). Volume-restore and DB-import
  failures now **surface** (restore returns an error) instead of a swallowed WARN. Prior to v0.61.0 the
  per-app restore never replayed the `.sql`, so DB-resident data did not come back.

#### Tier 2 — off-drive copy (Phase 3, v0.55.x)

For every HDD app, Tier 2 (`internal/backup/tier2.go`) rsync-mirrors the recovery unit
(`backups/primary/<app>/`) + the app's `appdata/<app>/` to `<target>/backups/secondary/<app>/` on a
**different physical disk** — the only off-drive protection bind-mounted HDD userdata can get (PBS can't
reach bind mounts). Auto-targeted: **prefer another registered user-data drive** (off-disk via
`system.SamePhysicalDevice`); else the **internal SSD for small units only**, behind a size-aware
**rootfs-headroom guard** (`tier2FitsHeadroom`) that **refuses rather than fills** the ~8 GB guest rootfs
(reserve = `max(2 GB, 20%)`), recording an honest "needs a 2nd HDD" status. Status persists via
`settings.CrossDriveBackup` and drives the "2. mentés" card. Runs daily (`tier2-backup`, 03:30) or via
`POST /api/backup/tier2`. restic is **not** used — a plain browsable mirror.

**In-place file restore from the Tier-2 copy (C2, v0.100.0 — closes drill finding F2)** —
`POST /backup/tier2/restore` (`backup.RestoreTier2Files`, `internal/backup/tier2_restore.go`) + the
**"Fájlok visszaállítása"** button on the healthy Tier-2 layer row. **Additive-only** semantics
(`rsyncRestoreMissing`: `rsync -a --ignore-existing`): files missing from the live `appdata/<stack>`
are copied back from the RECORDED Tier-2 copy; existing live files are **never overwritten** (a
customer edit after the last copy wins) and **nothing is ever deleted** — this exactly serves the
"I deleted my files" scenario with zero risk to newer data. Source = the recorded
`CrossDriveBackup.DestinationPath` (never a fresh target selection). Single-flight with
backup/restore; refusals (no copy / never ran / copy dir gone / either drive disconnected /
decommissioned) happen before the app is stopped, with customer-readable Hungarian reasons;
stop → copy → start → health-wait; zero files copied is a success ("Nincs hiányzó fájl…"). Out of
scope by design: overwrite/point-in-time restore (offbox + operator paths), per-file selection,
`recovery-unit/`. Apps that index their data dir (e.g. Nextcloud) may need a rescan (occ
files:scan) before restored files appear in their own UI.

**Per-app Tier-2 config panel (v0.57.0)** — `GET/POST /stacks/{name}/backup`
(`internal/web/tier2_config_handler.go` + `templates/tier2_config.html`). The "2. mentés" row's
**Beállítás** button links here (was the dead-end deploy page). Shows the effective off-drive target
(pinned or auto), whether it's the size-limited internal SSD, the last-run reason, and lets the customer
**pin a registered drive** (off physical disk) or **toggle Tier 2 off**. Always visible — single-SSD apps
get the "csak DB/konfiguráció" note, non-HDD apps the "already in the PBS whole-guest snapshot" context.
Two preference fields on `CrossDriveBackup` — `UserDisabled` + `PreferredTarget` (set via
`Settings.SetTier2Preference`) — are **preserved across the runner's status writes** (`withTier2Prefs`):
`selectTier2Target` honors a valid pin before auto-picking; `RunTier2` skips a disabled app. The runner
re-validates the pin off-disk at run time. `Manager.Tier2Info(stackName)` is the read-only panel view
(effective target + eligible alternative drives).

**Phase 1 — Database Dumps** (`internal/backup/dbdump.go`, scheduled 02:30)

- **Auto-discovery** of PostgreSQL and MariaDB containers via `docker ps` + `docker inspect`
- Dumps via `docker exec pg_dump` / `docker exec mariadb-dump` with 5-minute timeout
- Dumps are written to the app's **home drive**: `AppDBDumpPath(appDrive, stackName)`
- Atomic writes (`.tmp` → `.sql`) to prevent corruption
- **Validation** after each dump: checks file size, header presence, counts `CREATE TABLE`
- Results cached in `settings.json` surviving container restarts

**Phase 1b — Docker Volume Dumps** (`internal/backup/backup.go` `runVolumeDumps`, part of the same run)

- **F3 (v0.99.0):** re-wired into the nightly/manual app-data backup run (`runDBDumpsInternal`) —
  after the restic removal `DumpAppVolumesSafe` had no caller, so `volume-dumps/` was never produced.
  Runs AFTER the DB dumps and BEFORE `captureAllRecoveryUnits` so the manifests enumerate fresh tars.
- Gate order (load-bearing): protected-stack (`cfg.IsProtectedStack`) and has-volumes
  (`GetDockerVolumes()`) checks come BEFORE `DumpAppVolumesSafe` — the Safe variant stops the stack
  before its own volume check, so unconditional calls would bounce every volume-less app nightly.
  Disconnected/decommissioned drives skip with the same summary style as the DB loop.
- Each volume-bearing stack is stopped before dump, restarted after (`DumpAppVolumesSafe()`) —
  prevents inconsistent tars of live databases.
- For each volume: `docker run --rm -v <vol>:/vol:ro -v <dumpDir>:/out alpine tar cf /out/<vol>.tar -C /vol .`
- 10-minute timeout per volume; a per-stack failure lands in the run summary as `FAIL <app> volumes:`,
  flips the run's Success flag and fails the run (no silent partial) — other stacks still proceed
- Stale tars cleaned up (volumes that no longer exist)
- Volume names resolved with project prefix via `ResolveDockerVolumeNames()` (e.g., `mealie_mealie_data`)
- Dumps written to `AppVolumeDumpPath(nsRoot, stackName)`

**Phase 2 — Restic Snapshot** (`internal/backup/restic.go`, scheduled 03:00)

- Apps are **grouped by drive** via `groupStacksByDrive()` — each drive's apps are backed up to that drive's restic repo
- App drive resolution: `GetStackHDDPath()` (from `StackDataProvider`) → falls back to `SystemDataPath`
- Auto-generated repository password (32 random bytes, base64url), shared across all repos, synced to hub
- **Paths included in each per-drive snapshot (v0.34.0: per-app scoped):**
  - Per-app DB dump dirs on that drive
  - Per-app Docker volume dump dirs (`volume-dumps/*.tar`)
  - Per-app HDD mount paths (user data)
  - Per-app stack config dir (`<StacksDir>/<stackName>/` — only for stacks on this drive)
  - `controller.yaml` — only on the system drive (not duplicated across all drives)
- Auto-detects and unlocks stale locks (restic repo lock)
- Weekly prune on Sundays with configurable retention (keep-daily, keep-weekly, keep-monthly)
- Weekly integrity check (`restic check`) on Sunday 04:00 — checks **all** primary repos

**Protects against:** accidental deletion, data corruption, point-in-time rollback.
Does NOT protect against drive failure (backup is on the same physical drive).

#### Tier 2: Cross-Drive Backup (opt-in, different device) (`internal/backup/crossdrive.go`)

**Complete backup** to a different physical drive. Available for **all apps** — apps with HDD
data back up config + DB + user data + Docker volumes; apps without HDD back up config + DB dumps + Docker volumes.

- **Auto-enable for small apps (v0.14.1):** Apps without HDD mounts (config-only, DB-only) are
  automatically configured for daily rsync Tier 2 when ≥2 storage paths are registered.
  `AutoEnableSmallApps()` runs at the start of each nightly backup cycle. Never overwrites
  existing user-configured cross-drive settings (even disabled ones).
- **Infrastructure config backup (v0.14.1):** `syncInfraConfig()` rsyncs the stacks directory
  and `controller.yaml` to `<dest>/backups/secondary/_infra/` on every secondary destination
  drive. Runs before per-app backups. Cross-drive restic also includes infra paths.
- **Two methods:**
  - **rsync** — Simple mirror with `--delete` (fast, no versioning, **browsable** on disk)
  - **restic** — Versioned, deduplicated, encrypted (shared repo across apps, not browsable)
- Per-app configuration in settings.json: destination path, method, schedule (daily/weekly/manual)
- **Pre-backup DB dump:** `DumpStackDB()` runs fresh pg_dump/mariadb-dump before each cross-drive backup; non-fatal on failure (wired via `DBDumper` interface to avoid circular imports)
- **Pre-backup volume dump (v0.33.0, safe stop/start v0.34.0):** `DumpAppVolumesSafe()` stops the stack, exports Docker named volumes to tar, restarts — wired via `VolumeDumper` interface
- **Empty mounts allowed:** `RunAppBackup` accepts apps with no HDD mounts — the rsync
  mount loop simply doesn't execute, but DB + config copy still runs
- **Drive-type-aware validation** (`ValidateDestination`):

  | Destination type | Space checks |
  |-----------------|--------------|
  | External mount (different device than `/`) | Block if <100 MB free |
  | System drive (same device as `/`) | Require ≥10 GB free AND <90% used; logged warning |

- **Secondary drive layout (v0.14.1):**
  ```
  <dest-drive>/backups/secondary/
  ├── _infra/              ← infrastructure config mirror (v0.14.1)
  │   ├── controller.yaml
  │   └── stacks/          ← full stacks dir (all app configs)
  ├── <app>/rsync/         ← per-app rsync mirror
  │   ├── _db/             ← DB dump files
  │   ├── _config/         ← compose.yml, app.yaml, .felhom.yml
  │   ├── _volumes/        ← Docker volume tars (v0.33.0)
  │   └── <user data>      ← HDD mount contents (if app has HDD data)
  └── restic/              ← shared restic repo (all cross-drive apps)
  ```
  - DB dump files read from **per-app home drive** path (`AppDBDumpPath`)
  - `_` prefix directories prevent collision with user data
  - For non-HDD apps, only `_db/`, `_config/`, and `_volumes/` (if applicable) are present (no user data directory)
- **Restic backup paths:** includes HDD mounts (if any) + config dir + per-app DB dump dir from home drive + stacks dir + controller.yaml (infra, v0.14.1)
- Safety guards: destination ≠ source, path-overlap check (HDD mounts only), writable check
- **Chained execution:** runs immediately after nightly restic — daily apps every night, weekly apps on Sundays
- **Hub reporting after manual triggers (v0.27.2):** `OnCrossDriveComplete` callback on Router pushes infra backup snapshot to Hub + writes local infra backup after both single-app and run-all manual triggers complete (previously only automatic scheduled runs reported)
- Per-app concurrency lock prevents overlapping runs
- Status (last_run, duration, size, error) persisted to settings.json

**Protects against:** primary drive failure, drive theft/damage.

#### Tier 3: Off-box (NAS) Backup — live

The off-site "1" of 3-2-1: each off-box-toggled app's recovery unit + DB dumps + volume tars are
backed up to the customer's NAS / Felhom offsite as an encrypted restic repo over SFTP (see the
off-box section below and `internal/backup/offbox.go`). The per-app "3. mentés" row on the backups
page renders one of four real states via the pure `tier3State` helper (`internal/web/backup_page_state.go`):
`unconfigured` (no target) / `off` (app not toggled) / `escrow_pending` (fork-4 key-escrow gate holds —
never a false success) / `active` (status badge + `restic → <host>` + relative last-run). Off-box run
status is repo-global (one `LastRun`); no per-app run time is fabricated.

#### Restore (`internal/backup/restore.go`)

Both **Tier 1** (restic) and **Tier 2** (rsync) restores are supported. All deployed apps
appear in the restore dropdown with per-app snapshot filtering.

| App type | Config restored | DB restored | User data restored | Docker volumes restored |
|----------|----------------|------------|-------------------|------------------------|
| Has HDD data | Yes | Yes | Yes (always) | Yes (if present) |
| Docker volumes, no HDD | Yes | Yes | n/a | Yes |
| DB only, no HDD/volumes | Yes | Yes | n/a | n/a |
| Config only | Yes | — | n/a | n/a |

**Snapshot API** (`GET /api/backup/snapshots?stack=<name>` — F1, v0.99.0):
- Backed by `backup.Manager.ListRestorePoints` (`internal/backup/restore_points.go`). The keep-side
  restore has exactly **one** restore point per app — the current recovery unit — so the endpoint
  returns at most one entry: `time` = newest artifact mtime (manifest / db-dumps / volume-dumps),
  `short_id:"helyi"`, `tier:1`, `drive_label` from the storage registry (empty on the SSD fallback)
- **Never emits tier-2 entries**: Tier-2 copies are not restorable via `POST /backup/restore` (it
  only reads the primary unit) — listing them would silently restore tier-1 data while claiming tier-2
- Guards: empty/traversal stack name → 400 (`validStackParam`), unknown stack → 404, known stack
  with no unit yet → `ok:true, data:[]` (the UI shows "Nincs elérhető mentés")
- History: the route was a restic-era leftover fetched by the template but unregistered — the
  dropdown could never populate and the restore button never enabled (drill finding F1)

**Restore type info** shown per-app when selected in dropdown (Hungarian banners):
  - Has HDD or Docker volumes: "Teljes visszaallitas: adatbazis + konfiguracio + felhasznaloi adatok"
  - Has DB, no user data: "Adatbazis es konfiguracio visszaallitasa"
  - Config only: "Csak konfiguracio visszaallitasa"

**Tier 1 restore** (`RestoreApp`):
- Stop app → resolve app's home drive → `restic restore <id> --target / --include <path>...` → populate Docker volumes from restored tars → restart app → health check
- Restore paths: config dir, DB dump dir, volume dump dir, HDD mounts
- Docker volumes restored via `restoreDockerVolumes()`: `docker volume rm -f` → `docker volume create` → `docker run alpine tar xf`

**Tier 2 restore** (`RestoreAppFromTier2`):
- Stop app → rsync config from `_config/` → rsync HDD data (single/multi-mount) → copy DB dumps from `_db/` (streaming `copyFile`) → restore Docker volumes from `_volumes/` tars → restart app → health check
- Uses rsync `--delete` for config and HDD data to ensure exact mirror state
- Single-mount apps: data directly in rsync dir (excluding `_*`); multi-mount: per-leaf subdirectories

**Common:**
- **v0.34.0:** Post-restore health check (`waitForHealthy`) polls container state with `docker ps` refresh every 5s for up to 90s. Warning logged if app doesn't reach running state; restore still returns success (data is restored regardless).
- Running flag prevents concurrent backup/restore operations
- Snapshot ID validated (8-64 lowercase hex, or special `tier2-rsync`)
- Import from `.fab` bundle link shown in restore section for cross-system migration

#### Backup Page UI (`internal/web/templates/backups.html`)

Unified per-app status table with expandable rows showing **per-tier** backup status:

**Status dot per app:**

| Dot color | Meaning |
|-----------|---------|
| Green | 2+ tiers configured with successful backups + destination healthy |
| Yellow | Only 1 tier, or Tier 2 failing, or Tier 2 configured but never run, or destination disconnected/inactive |
| Red | Tier 2 destination blocked or inaccessible |

Every app starts as yellow (1 tier only). Green requires Tier 2 configured with successful backup.

**Per-app backup tiers (3 rows per app):**
- **1. mentes** (Tier 1, always present) — Auto badge + "helyi" + last run + contents (e.g., "DB + Konfig + Adatok")
- **2. mentes** (Tier 2, configurable for ALL apps) — one of:
  - Configured: method (rsync/restic) + destination + schedule + last run + status + contents + browsable indicator (folder icon for rsync) + action buttons
  - Not configured: "1. mentes auto" + "Nincs 2. masolat" + settings link
- **3. mentes** (Tier 3, off-box/NAS — live) — one of four `tier3State` states: `unconfigured`
  ("Nincs beallitva" + Beallitas link), `off` ("Kikapcsolva" + Bekapcsolas link), `escrow_pending`
  ("Kulcsletetre var"), `active` (status badge + "restic -> <host>" + relative last-run)

**Backup contents per app** (shown per tier):
- Apps with DB + HDD: "DB + Konfig + Adatok"
- Apps with Docker volumes (no HDD): "Konfig + DB + Adatok" or "Konfig + Adatok"
- Apps with DB only: "DB + Konfig"
- Apps with HDD, no DB: "Konfig + Adatok"
- Apps with neither: "Konfig"

**Deploy page** shows cross-drive (Tier 2) configuration form for **all deployed apps**,
not just those with HDD data. Non-HDD apps can configure destination, method, and schedule.

**Other sections:**
- Schedule overview with next run times for DB dump, restic, prune
- Snapshot history table (last 20 snapshots aggregated from all per-drive repos, sorted by time)
- Storage overview card (total size across repos, snapshot count, DB dump count/size, encryption key with show/copy)
- Restore section: app dropdown → per-app snapshot dropdown (Tier 1 + Tier 2 grouped) → restore type info → confirmation checkbox → execute → import from `.fab` bundle link

---

### 4. Storage Management

> **Off-box (NAS) backup — restic-over-SFTP (v0.93.0, Part B).** The "1 off-site" leg of 3-2-1 for the
> app-data tier: each off-box-toggled app's recovery unit + DB dumps + volume tars are backed up to the
> customer's NAS as an **encrypted restic repo over SFTP** (`internal/backup/offbox.go`). No kernel mount —
> restic talks SFTP directly; the NAS sees only ciphertext. Distinct from the local cross-drive rsync copy
> and the agent's PBS whole-CT DR.
> - **Fail-fast (load-bearing):** every restic call uses `-o sftp.command="ssh … -oConnectTimeout=10 … -s
>   sftp"` so a dead NAS errors in ~10 s, never hangs the backup runner; a failure raises the `backup_failed`
>   operator alert. `-oStrictHostKeyChecking=yes` + a pinned `known_hosts` (no blind TOFU).
> - init-if-absent (idempotent), `forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune`,
>   single-flight (shares `m.running`) + migration-guard, restic's own exit code checked, restore to a
>   scratch dir (non-destructive).
> - **Unit discovery (v0.104.0) — durable, deployment-independent.** Each toggled app's recovery unit is
>   **discovered** by scanning the durable storage registry — every registered *schedulable* storage path
>   (`GetSchedulableStoragePaths`) ∪ the system-data fallback drive — for `backups/primary/<app>`, rather
>   than *inferring* the drive from the app's live `app.yaml` `HDD_PATH` (which silently fell back to
>   `systemDataPath` for a toggled-but-undeployed app → offbox looked on the wrong drive, backed up
>   nothing, and reported `ok`/0). If the same app's unit exists on two drives (drive churn), the **newest
>   by manifest `CreatedAt`** is used and the stale one is WARN-logged. The WRITE side (`CaptureRecoveryUnit`
>   / dumps) is unchanged — this only changes offbox's read/discovery path. Boundary: decommissioned or
>   non-schedulable drives are not searched.
> - **No silent success (v0.104.0):** a run where ≥1 app is toggled but **0** were backed up (no unit found
>   anywhere) is a **hard error** → `LastStatus="error"` + operator alert (was a misleading `ok`/0 snapshots).
>   A *partial* run (some units missing) stays `ok` but sets a Hungarian **`LastWarning`** naming the skipped
>   apps, shown on `/backups`.
> - **Crash-lock self-heal (v0.110.0).** A crash mid-prune leaves a restic EXCLUSIVE lock that plain
>   `restic unlock` can't clear (the recreated container's new hostname stops restic proving the dead PID
>   stale for ~30 min). Every backup/prune/restore step runs through `resticStep`, which on a lock error
>   escalates to `unlock --remove-all` + one retry — safe because the repo has a SINGLE legitimate writer
>   (per-customer sub-account isolation + the in-process single-flight mutex). **Boundary:** a DR-cloned
>   SECOND controller writing the same repo would defeat this premise — operator-supervised territory, out of
>   scope for the auto-heal. A crash mid-run also flips a persisted `LastStatus="running"` to a truthful
>   error on the next startup (self-corrects on the next successful run).
> - **Secrets** (SSH key + auto-gen repo password) are **0600 files in the data dir** — never logged/committed.
> - **Password custody + atomicity (v0.105.0, fork-4; pairs with agent v0.77.0).** The repo password is the
>   irreplaceable DATA key for the offsite tier, so it rides the **customer-recovery-code (R) escrow**
>   (age-under-R, in the agent's `IdentityBundle` — custody spike validated a recovered password opens the
>   real repo). **Atomicity gate:** enabling offsite pushes the password to the agent (`StageEscrowSecret` →
>   `POST /escrow/stage-secret`) and marks `EscrowState="pending"`; **no offsite backup runs until the escrow
>   is confirmed** (`OffboxRunnable()`), so an un-recoverable offsite copy can never exist.
>   **Operator ceremony (pilot, supervised):** enable offsite (→ pending) → run
>   `felhom-agent --selftest=escrow-create --identity-bundle <assembled> --upload` (the bundle now carries
>   the staged restic password, auto-injected, then wiped) → hand the customer the fresh **R** (once;
>   supersedes any prior code) → `POST /backup/offbox/confirm-escrow` (or the "Letét megerősítése" button).
>   **DR:** recover R → the escrow yields the password → `POST /backup/offbox/inject-password {password}`
>   pre-places it 0600 → configure offbox → restore. The SFTP access key is **regenerated** at DR (a fresh
>   sub-account key), NOT escrowed; the DR recipe carries only the non-secret `offsite_restic`
>   coordinates (`DRResticCoord`).
> - **Hub-verified auto-confirm (v0.108.0, SLICE 3; pairs with agent v0.79.0 + hub v0.40.0).** The report
>   ACK carries `escrow:{identity_blob_present, restic_pw_sha256, created_at}` (the hash is recorded at
>   ceremony time — non-reversible sha256 of the sealed password, safe to serve). `report.EscrowAutoConfirmer`
>   flips `EscrowState` pending→escrowed ONLY when `sha256(local repo_password)` **matches** — i.e. the
>   stored escrow provably covers the CURRENT key; blob-presence alone never confirms (a stale blob would
>   re-open the fork-4 gap). Mismatch → stays pending + a loud warn naming the ceremony (deduped per hash);
>   no hash / no row / no local file → silently pending; **never un-confirms**. On flip it wipes the
>   agent-staged secret. The canonical hasher (`backup.HashResticPassword`, trimmed-string sha256) is pinned
>   by a cross-repo test vector against the agent's. The manual `POST /backup/offbox/confirm-escrow` is a
>   **deprecated fallback** for legacy hash-less blobs (e.g. the demo's).
> - **Injection guard** (`ValidateOffboxTarget`): host/user/repo must not start with `-` (ssh
>   option-injection) or carry metacharacters/traversal; `OffboxConfigured` fails closed on an invalid
>   target. Image: `restic` + `openssh-client` (re-added; restic's sftp backend shells out to `ssh`).
> - UI: the "Külső (NAS) mentés" section on the backups page (configure target, per-app toggles, run-now,
>   restore, status). Config: `settings.OffboxTarget` + per-app `AppBackupPrefs.Offbox`. Daily at 04:15.
> - **Soft quota (v0.109.x, SLICE 4; pairs with hub v0.41.0).** The shared-model quota (`quota_gb`) rides
>   the descriptor into `OffboxTarget.QuotaGB` (0 = no limit — dedicated boxes are Hetzner-enforced; the
>   hash includes it, so a hub-side quota change re-applies via key-auth-first, no password consumed).
>   `RepoSizeBytes` persists from `restic stats` (last-known on failure — stale-but-safe). Pre-run gate:
>   **≥100% refuses NEW backup runs** (Hungarian error + operator alert) but the **prune step still runs**
>   (the only way back under quota) and **restore is never gated**; ≥80% sets a Hungarian usage warning.
>   `/backups` shows a usage bar when quota>0. The hub report carries
>   `offsite:{enabled, escrow_state, last_run, last_status, snapshot_count, repo_size_bytes, quota_gb}` —
>   the hub's OffsiteChecker alerts on fill (90/95%) and staleness (escrowed + no run >48h); the Hetzner
>   `readonly` freeze is an OPERATOR lever on the hub (never automatic). A RE-apply preserves the existing
>   target's EscrowState + runtime status (v0.109.1 — custody tracks the preserved repo password).
> - **Hub-driven provisioning apply-bridge (v0.106.x, SLICE 2; pairs with hub v0.38.x — validated live
>   2026-07-09).** When the hub provisions the offsite tier (a Hetzner Storage Box sub-account or dedicated
>   box), the served `controller.yaml` gains an `offsite:` section (host/user/port/repo_path/quota_gb +
>   **`host_fingerprint`**) and `internal/offsiteapply.Bridge` reconciles it at startup: scan + **verify the
>   box host key against `host_fingerprint` (no blind TOFU)** → generate an ed25519 keypair → **consume the
>   hub's one-time transient password** (`POST /api/v1/offsite/consume-password/{id}`, single-use) → install
>   the pubkey (`sshpass -e ssh-copy-id -p 23 -s -f`, **pinned known_hosts, `StrictHostKeyChecking=yes`**;
>   the installer ensures `~/.ssh` exists — ssh-copy-id's SFTP mode needs it) → verify key-only auth →
>   configure the offbox target → `EscrowState="pending"` (the fork-4 gate above still holds) → persist a
>   descriptor-hash marker LAST. **Idempotent** (marker → no re-consume) and **fail-safe** (any failure →
>   nothing persisted, retried on the next config refresh/restart; a consumed-but-failed install logs a loud
>   "password is spent — reset on the hub"). The 15-min hub-report → config-refresh cycle is the trigger, so
>   descriptor-to-applied latency is ≤ ~15 min.
> - **Key-auth-first (v0.107.0):** on a descriptor change, if the ALREADY-INSTALLED key still authenticates
>   (`SFTPKeyAuthProber`, pinned to the freshly-verified host key — the probe never bypasses the fingerprint
>   verify), the bridge re-pins + reconfigures WITHOUT consuming a one-time password. Fresh guests fall
>   through to the full consume+install path. The hub's "Re-issue offsite credentials" (v0.39.0) is the
>   recovery for a genuinely-spent password on a fresh guest.
> - **Staged-secret wipe (v0.107.0):** confirm-escrow calls the agent's `DELETE /escrow/stage-secret`
>   (agent ≥ v0.78.0) whenever `EscrowState` flips to `escrowed` — best-effort, loud-logged on failure.

> **NAS network storage (v0.92.0, Part A2; pairs with agent v0.50.0).** A customer NAS share (NFS or SMB)
> is a **distinct storage KIND** from a physical drive (`StoragePath.Kind == "network"`), for **bulk media**.
> The controller is a thin proxy over the agent's `/netstorage/*` (A1) + the local registry — it holds **no
> mount authority** and **never persists the SMB password** (it passes the credential to the agent's add
> request, which writes the 0600 file). Endpoints `POST /api/storage/netstorage/{add,remove}` +
> `GET /api/storage/netstorage` (merges the agent's per-share liveness: `ok` | `idle` (benign idle-unmount)
> | `unreachable` | `unknown`). A network share registers as `/mnt/felhom-drives/<name>`, is **selectable as
> a media app's `HDD_PATH`** (Schedulable), and shows in the "Hálózati tárhely (NAS)" settings section.
> - **NOT a drive:** the drive lifecycle (eject/decommission/migrate/wipe/SMART) is **refused** on a network
>   path server-side (`refuseNetworkLifecycle`). The **drive-absent gate** (`planDriveGates`) and the
>   **missing-storage** surface **skip network paths**, so an `unreachable` NAS is a **recoverable warning**
>   (`networkStorageWarnings` → a distinct app-card badge), **never** the drive "missing → stopped" cascade.
> - **Verify-before-commit (v0.113.0, pairs with agent v0.81.0; SPIKE-nas-verify-2026-07-11):** add no
>   longer registers blind. `POST /api/storage/netstorage/add` returns `{started:true}` and a DETACHED
>   single-flight orchestration job (`internal/web/netstorage_job.go`, migrate.go shape; polled on
>   `GET /api/storage/netstorage/add/status`) drives `agent_add → verifying → probing → registering`:
>   the agent installs the units + runs its own detached mount-verify (journal-classified, agent-side
>   auto-rollback), then the controller re-execs itself as **`--netprobe <dir>` at uid/gid 1000**
>   (`SysProcAttr.Credential`, `netprobe*.go`) to prove a media app can WRITE through the share (the
>   squash trap), and only then registers. ANY failure = full rollback (nothing registered, nothing
>   installed, no creds file); categorized Hungarian errors (`netAddMessage`, §3.2 map — note
>   `nfs_export` merges not-found/not-permitted: NFSv4 returns identical strings). An agent restart
>   mid-verify (`verify-status` phase `none`) ⇒ controller rollback; any agent-configured share NOT in
>   the registry surfaces as a remove-only **"Árva megosztás"** orphan row. The page
>   (`storage_network.html`) is on the canonical form pattern with staged poll progress and the
>   protocol-honest NAS guidance (SMB-first; NFS map-all-users vs full-fidelity `anonuid=<uid+100000>`).
>   Authoritative doc: `felhom.eu/documentation/controller/network-storage-nas.md`.
> - **Agent-capability gate (v0.114.0):** the verify-before-commit add is COUPLED to agent ≥ v0.81.0,
>   so the add entry point now probes the agent first (`internal/agentapi/features.go`,
>   `Supports(FeatureNetstorageVerify)` — a route probe on `GET /netstorage/verify-status`: 2xx ⇒
>   supported, typed 404 ⇒ older agent, transport/5xx ⇒ indeterminate, cached 5 min both polarities).
>   On an older agent the add is refused SYNCHRONOUSLY (HTTP 412, machine code `agent_outdated`,
>   honest Hungarian message) BEFORE the single-flight claim — never a misleading mid-pipeline
>   rollback; indeterminate NEVER refuses (a down agent speaks through the existing error paths). The
>   settings page swaps the add form for a banner on `SupportNo` (share list + remove stay usable in
>   every state). Convention: every future coupled feature adds a `featureProbes` row + a gate call at
>   its entry point and declares `MinAgent` in its CHANGELOG header — see
>   `felhom.eu/documentation/runbooks/publish-train-rules.md`.
> - **Consuming-namespace verification (v0.117.0, RCA AUDIT-nas-cwa-rca-2026-07-11 fix 2):** a guest
>   reboot silently replaces an idle NAS trigger with a plain local STUB dir in the app namespace while
>   host-side (agent) health stays green — so the controller now verifies where the apps consume:
>   `internal/system/fsclass*.go` classifies a path by **statfs f_type** in THIS process's namespace
>   (`network` nfs/cifs/smb2 | `autofs` idle trigger = HEALTHY, never force-mounted | `stub` | `unknown`
>   = fail-open). Three consumers: (1) the `--netprobe` child now REQUIRES a mounted network fs after
>   its create (exit 5 → category `not_network_fs`, full rollback — a writable stub can never verify);
>   (2) `POST /api/stacks/{name}/deploy` refuses (409) a registered network `HDD_PATH` that classifies
>   as a stub (`refuseNetworkStubDeploy`; idle autofs deploys fine — first app access mounts it);
>   (3) the dashboard/stacks app cards gain a distinct **stub badge** ("Hálózati tárhely hibás — az
>   alkalmazás nem a NAS-t látja", `NetworkStubs`) that WINS over the recoverable `unreachable` badge.
>   The agent pair (v0.84.0 `ReassertNetworkMounts`) re-arms triggers on guest start; this layer is the
>   detection net beneath it.
> - **One classification, two surfaces (v0.119.0, CAMPAIGN-3 F8):** the **share row** on the
>   `/storage/network` page now reads the SAME classifier. Its health used to come only from the agent's
>   server-level TCP dial (`server:2049/445`), which stays green when a *single* export is `exportfs -u`'d
>   — so the row showed benign "Készenlét" while the stacks cards showed the stub. `fuseNetHealth`
>   (`netstorage_handlers.go`) fuses the agent view with `classifyFSPath(Where)`: a new **`stub`** health
>   (badge "Hibás — az alkalmazások nem a NAS-t látják") overrides idle/ok when the namespace sees local
>   disk; a whole-server `unreachable` still wins over stub; autofs-healthy / network / `unknown` leave
>   the agent health intact (never force-mount an idle trigger). The row and the stacks badge derive from
>   ONE classifier and can never contradict.
> - **mapped_uid validated at the door (v0.119.0, F8's sibling F4):** `handleNetStorageAdd` range-checks
>   the container uid/gid (**1..65533**; the guest maps `<uid>`→`<uid>+100000`, so 65534=nobody and a
>   host-side value like 101000 must not be entered) after the `<=0` default — out of range → a friendly
>   Hungarian 400, nothing installed (previously a raw `agent_error` from the agent).
> - **Deploy view truth (v0.117.0, RCA fix 4):** a deployed app's read-only storage select now marks
>   `selected` by the app's STORED `HDD_PATH` (extra disabled `<path> (nem elérhető)` option when the
>   stored path left the schedulable list) — `IsDefault` selects only for NEW deploys. Pre-fix the view
>   showed the default drive regardless of app.yaml (the RCA's S-C symptom).
> - **Limits (v1):** a share's `+100000` uid mapping is fixed at add-time (one app / same-uid apps); for
>   write apps on a soft NFS mount, an in-flight file can truncate if the NAS vanishes mid-write (prefer
>   atomic-write apps / SSD-staging).

> **⚠️ INTERMEDIARY-MOUNT model (v0.67.x, pairs with agent v0.35.x).** External data drives are now
> visible in the guest at a STABLE path **`/mnt/felhom-drives/<name>`**, NOT the raw `/mnt/<name>`. The
> agent keeps a single permanent parent bind `/mnt/felhom-drives` in the guest and swaps each drive's
> `felhom-data` namespace **underneath it host-side** (`mount --bind`), so the guest sees attach/detach
> **live with no reboot** (mount propagation), the bind source never disappears (C1-immune), and only
> `felhom-data` crosses in (confinement). The per-drive `pct set -mpN` bind is **deprecated**.
> - The registered storage path + every app's **`HDD_PATH`** + the FileBrowser source = the stable
>   `/mnt/felhom-drives/<name>`. The controller maps it back to the raw `/mnt/<name>` (`agentWhere()`)
>   only for agent calls (assign/attach/eject/decommission). `GET /api/disks` carries `guest_path` +
>   `bound_under_parent` (the guest-visible signal).
> - **Drive-absent gate** (`internal/web/intermediary.go`, `driveGateLoop` 30s): an absent
>   `/mnt/felhom-drives/<name>` drive stops + blocks its apps (`StoppedStacks` = the gate-stopped set);
>   a returned drive re-attaches + auto-restarts them; `actionStack` refuses to start an app whose drive
>   is absent ("tárhely nem elérhető"). SSD/system paths are never gated.
> - **H1 endpoints** `POST /api/storage/{disconnect,reconnect,restart-apps}` drive the host-side
>   eject/reconnect (no guest reboot).
> - **Lifecycle (v0.68.x):** a "Leszerelés" button decommissions a drive (migrate-then-decommission OR
>   decommission-anyway type-to-confirm — non-destructive, never touches the parent mp); a
>   "Visszacsatlakoztatás" button one-click re-enrolls a decommissioned/ejected drive (clears the marker,
>   re-binds under the parent, restarts gate-stopped apps). Decommissioning the DEFAULT auto-promotes
>   another schedulable drive (`defaultPromotionTarget`), or BLOCKS if it's the only one. Eject and
>   decommission keep the RAW drive mounted (logical retire) so re-enroll re-binds it.
> - **Guest-reboot convergence is DETERMINISTIC** via the agent's `guest_boot_id`: the controller persists
>   `LastGuestBootID` and, when it changes, recreates EVERY deployed drive-backed app onto the
>   re-propagated drive (`processGuestBootChange` — no fragile container-uptime sampling).
>   **v0.71.0 — boot-race fix:** on a guest `pct reboot`, in-guest dockerd auto-starts the apps ~18s
>   BEFORE the agent re-binds the drive, so their volume bind fails at create-time
>   (`mkdir …/userdata: permission denied`, `RestartCount=0` → never retried → stuck `Exited`). The old
>   recovery sampled the agent's `BoundUnderParent` ONCE, raced that rebind, recreated nothing, and burned
>   its boot-id one-shot. `processGuestBootChange` now **gates on the REAL live in-guest bind**
>   (`driveBindLive`: is `/mnt/felhom-drives/<drive>` an actual mountpoint in the controller's own `/mnt`
>   rslave `/proc/self/mountinfo`?) and **waits** for it (`pollLiveBinds`, bounded ~120s) before recreating
>   — including apps stuck `Exited` with a create-time mount failure (`shouldRecreateOnBoot` is
>   state-independent). The **guest-only reboot path** (which the host-reboot sweep never exercised) is now
>   covered; drives that never go live in the window are left to the drive-absent gate. `processGuestBootChange`
>   also runs on every periodic `driveGateLoop` tick now (idempotent, boot-id gated) so a momentarily-unreachable
>   agent right after a guest reboot no longer permanently strands recovery.
>   **Agent-path prerequisite (also v0.71.0):** the whole drive gate needs `cfg.LocalAPI.Endpoint` (the
>   per-guest agent local API). `bootstrap.MaybeIngest` now calls `ensureLocalAPI` on the already-configured
>   path — merging `local_api` from `bootstrap.json` into an existing controller.yaml that lacks it (seeded
>   before `local_api` existed) — because without it `agentClient()` returns "agent not configured" and the
>   entire gate + boot recovery silently die.
>   **v0.72.0 — FileBrowser convergence on boot-recreate:** FileBrowser is base-infra (it binds each
>   drive's `userdata` but has no `HDD_PATH`, so it is NOT in the drive-backed recreate set) — after a
>   host reboot its mounts could be stale (the early first-boot bring-up bound them before the drives went
>   live). `processGuestBootChange` now, **after** `pollLiveBinds` confirms the binds and the apps are
>   recreated, triggers `go s.SyncFileBrowserMounts()` so FileBrowser converges against the now-live drives.
>   The sync runs unconditionally (FileBrowser reflects the current bind state even if no app needed
>   recreating). The recreate loop is a pure `recreateDriveBackedApps(stacks, present, recreate, syncFB)`
>   that calls `syncFB` exactly once, after every recreate. Live-accepted over two real `felhom-pve`
>   reboots (FileBrowser non-stale, all drive-backed apps recovered, agent tolerated a `/dev/sdX` swap by UUID).
>
> **⚠️ Rebuilt on the agent-delegated disk model (v0.43.0), made ROLE-AWARE in v0.44.0, UX-polished in
> v0.45.0.** After the 8C
> de-privileging, the controller holds **no Proxmox/disk credentials and no destructive authority** — disk
> execution + the gate live entirely in the **host agent**. The drive UI is driven by the agent's
> authoritative **role** (`system` | `backup` | `user-data`, from `GET /api/disks`): the appliance's own
> system storage and the backup safety-net are visibly **protected** (lock badge, NO destructive controls);
> the customer manages their own **user-data** drives with informed consent. The agent re-enforces role at
> wipe time — the UI lockout is defense-in-depth, not the gate.
> - **Overview** (`settings.html` ← `GET /api/disks`): styled **cards** (not a table) — name, mono
>   device/mount, badges for class (gyors/lassú), data (`Adatot tartalmaz`), **role** (🔒 Rendszer / 🔒
>   Biztonsági mentés — védett / Felhasználói adat) and registered state, plus a **capacity bar** (the
>   monitoring `system-bar`, from the agent's `total_bytes`/`used_bytes`). Eject/Wipe render **only** for
>   user-data drives mounted under `/mnt`.
>   - **(v0.45.0) Deterministic order** — `agentDisksListHandler` sorts the list server-side
>     (`sortDisksForView`): **user-data → system → backup** (then unrecognized), alpha by name within a
>     tier, so it no longer reorders on each reload (the agent's view iterates an unordered Go map).
>   - **(v0.45.0) Purpose + app-backing clarity (B4)** — `local` and `local-lvm` are both shown (not
>     collapsed); each card carries a plain-Hungarian **purpose description** keyed on the agent's
>     role/type, the app-backing storages are tagged (`local-lvm` → "Alkalmazás-rendszer"; user-data →
>     "Alkalmazás-adatok"), and a one-line tiering note above the list answers "which storage do the
>     apps use?". Pure presentation — role/type stay authoritative from the agent.
>   - **(v0.45.0) Register shortcut (B3)** — a mounted, **unregistered** user-data drive offers
>     **Regisztrálás** as its PRIMARY action: `POST /api/storage/register` → `registerStoragePath` records
>     the existing mount (no format, no eject) + FileBrowser-syncs. Leválasztás/Törlés stay secondary.
> - **Customer wipe/eject** — a **type-to-confirm** modal that names the deployed apps that break
>   (`GET /api/storage/impact` → `appsUsingPath`) and disables the destructive button until the **mount
>   name is typed exactly**. Wipe (`POST /api/storage/wipe`): eject (unmount + deregister) → server-side
>   two-step customer-confirmed format (learn the agent's durable id, then re-submit `confirmed:true` bound
>   to it). The agent refuses a protected device regardless of what the controller sends.
> - **Guided init** (`/settings/storage/init`, `POST /api/storage/init`, `web/storage_handlers.go`): format
>   → resolve the new fs UUID → `assign` → register. The selector lists **only user-data** targets. A
>   data-bearing user-data device now uses the **customer-confirmation** flow (type-to-confirm → re-submit
>   `confirmed:true` + durable id), NOT the `felhom-opsign` command. The opsign surface remains a fallback
>   only if a protected device somehow reaches init.
> - **Guided attach** (`/settings/storage/attach`, `POST /api/storage/attach`): non-destructive — resolve
>   the existing fs UUID → `assign` → register. Selector restyled to cards (user-data only).
> - **Eject** (`POST /api/storage/eject`): benign unmount + deregister, with the agent's dependent-guest
>   warning + the affected-app list (parity with wipe). **The eject is ROLE-GATED at the agent** (felhom-
>   agent v0.24.0): `POST /disks/eject` refuses to unmount a system/backup mount — the UI hiding the button
>   is defense-in-depth, not the control. Only user-data mounts are ejectable.
> - **`agentapi`** (`internal/agentapi`) is the pinned client to the agent local API: `Disks`/`AssignDisk`/
>   `EjectDisk`/`FormatDisk(…, confirmed, durableID)`; `DiskInfo.role`+capacity;
>   `FormatResult.{role,needs_confirmation,durable_id}`; `ErrNeedsConfirmation` (user-data) vs
>   `ErrFormatRefused` (system/backup). `FormatResult.PendingOp.OpsignCommand()` for the operator path.
>   - **(v0.74.0) Client lifecycle — ONE shared client, reused.** `Server.agentClient()` builds the
>     `agentapi.Client` once (memoized via `sync.Once`) and returns the same instance to all ~19 call
>     sites; the `http.Transport` is bounded + expiring (`MaxIdleConnsPerHost:2`, `IdleConnTimeout:90s`).
>     This replaced a per-call `agentapi.New(...)` that leaked one idle ESTABLISHED socket per call and
>     exhausted the ephemeral source-port range to the agent's `:8443` after ~5 days (EADDRNOTAVAIL).
>     Safe because `cfg.LocalAPI` is static per process (a config-apply triggers a graceful self-restart).
>   - **(v0.90.0) Channel health-check — `internal/channelhealth`.** A ~60s scheduler job
>     (`agent-channel-health`) probes the channel via `Server.ProbeAgentChannel` (the SAME memoized
>     client + `GET /storage` — not a fresh client) and classifies failures into
>     `up | down:<reason>` (pin_mismatch / unauthorized / unreachable / timeout / misconfigured /
>     construction_error). Transient reasons (refused/timeout) are **debounced** (N≥2 consecutive)
>     so a clean agent restart's ~1s blip doesn't page; pin/401/DNS/construction alert on the first
>     down. On a transition it fires an English **operator-only** event
>     (`Notifier.NotifyAgentChannelDown/Recovered`, hub 1h cooldown) and sets a Hungarian dashboard
>     banner (`AlertManager.SetAgentChannelAlert`). The first observation seeds state silently. This is
>     the controller half of the self-health story (the agent watches its own privileged capabilities;
>     the controller watches its link to the agent). A construction error (`agentClient()` can't build
>     — a latching config fault) is surfaced distinctly. Detection/surfacing only — it never touches the
>     pin, transport, or gate.
> - The **`StoragePath` registry** (`settings.go`: `AddStoragePath`/default/schedulable/label) is unchanged.
>   - **(v0.64.0) `AutoDiscoverStoragePaths` is now ADDITIVE** — it no longer bails when the registry is
>     non-empty; instead it registers only deployed-app paths missing from the registry. It never removes
>     or mutates an existing entry, never re-adds or reactivates a path already present in ANY state
>     (incl. `Decommissioned`), and never flips `IsDefault` (a new path defaults only if the registry has
>     no default at all). This is NOT auto-register-on-attach (that recommendation was rejected — manual
>     enrollment is by design); it only picks up drives that deployed apps already reference.
>   - **(v0.64.0) `InferStorageLabel` disambiguates the internal SSD** — a path whose basename is the
>     `felhom-data` namespace dir (the internal system volume, e.g. `/mnt/sys_drive/felhom-data`) now reads
>     **"Belső SSD (rendszer)"** instead of the colliding "Tárhely (felhom-data)". Model-A user drives
>     register their MOUNT ROOT (e.g. `/mnt/felhom-usb`), never `.../felhom-data`, so this can't mislabel a
>     user drive. Still overridable via `SetStorageLabel`.
> - **(v0.65.0) Data migration** (`internal/stacks/migrate.go`) — move app data between drives, in-process
>   over the controller's `/mnt:/mnt:rslave` RW mount; crash-safe + resumable via a journal
>   (`<dataDir>/migration.json`). `MigrateAll(source,target)` moves the whole felhom-data namespace (every
>   app + a conflict-merge walk for non-app/customer content); `MigrateApp(app,target)` moves one app's
>   subtree (drive→drive AND SSD→drive). Pipeline: validate → stop → copy (`rsync -a --checksum`, additive,
>   **no `--delete`**) → verify (`rsync -ani --checksum`, zero pending) → flip+redeploy (`RedeployFromEnv`)
>   → cleanup. **CLEANUP — the only destructive step — is gated on every unit verified AND every app
>   redeployed.** Conflict-merge: skip-identical (checksum vs target + its `(N)` siblings), rename-on-differ
>   to lowest-free `<base>(N)<ext>`, never overwrite; idempotent. Single-flight + mutually exclusive with the
>   backup orchestrator. UI: `POST /api/storage/migrate{,-app}` + `GET /api/storage/migrate/status` (poll);
>   migrate-all on the settings page, per-app on the app-info page, shared Hungarian progress panel.
> - **(v0.65.0) Self-serve decommission** (`handleStorageDecommission`, `POST /api/storage/decommission`) —
>   retire a drive, non-destructive (never formats). Two choices (no partial): migrate-all-then-decommission
>   (runs `MigrateAll`; the migration done-hook soft-marks the source + calls the agent on success), or
>   decommission-anyway (type-to-confirm; stops the apps, keeps their `HDD_PATH` so they show "Hiányzó
>   tárhely"). Both end at `settings.SetDecommissioned` (soft marker retained — blocks A1 resurrection) +
>   `agentapi.Decommission` (agent v0.32.0: `POST /disks/decommission`, role-gated user-data, intent +
>   bind-prune + unmount). Re-enrolling a decommissioned drive (`registerStoragePath`) clears the marker +
>   restores `Schedulable`. A deployed app whose drive is decommissioned/disconnected/absent shows the
>   **"Hiányzó tárhely"** warning badge on the dashboard/stacks/app card.
>
> - **(v0.66.0) Userdata layout + shared-storage ownership convention** (`internal/appbackup/userdata.go`).
>   Each drive's felhom-data namespace gains a customer-facing **`userdata/`** tree (sibling of
>   `appdata/`/`backups/`) — the ONLY thing FileBrowser mounts. Skeleton:
>   `media/{movies,tv,music,audiobooks,books,comics,photos}`, `downloads`, `import/{paperless,calibre}`,
>   `roms`, `documents`. **Convention:** every userdata dir is group-owned by `SharedContentGID` (1000),
>   mode **2775** (setgid + group-rwx) — `EnsureUserdataDir` = MkdirAll → explicit `Chmod(ModeSetgid|0775)`
>   (MkdirAll's mode is umask-masked AND drops setgid) → chown group 1000. Setgid makes new files inherit
>   group 1000 so FileBrowser (uid 1000) + the content apps collaborate without permission collisions.
>   - **`USERDATA_PATH`** = `<HDD_PATH>/userdata` is injected into the compose env (the shared
>     `withUserdataPath`, used by BOTH `stackEnv` and `composeExecWithEnv` — the initial deploy builds env
>     from values, not app.yaml). Catalog media mounts use `${USERDATA_PATH}/...`.
>   - **Pre-create**: the full skeleton is created with the convention on `registerStoragePath` +
>     `syncFileBrowserMounts` (system + additional drives); a **deploy belt** in `composeExecCustomEnv`
>     (on `up`) pre-creates every `${USERDATA_PATH}/...` bind source the app declares, so Docker never
>     auto-creates a userdata dir as guest-root (covers apps not in the skeleton).
>   - **FileBrowser** mounts `<drive>/userdata` (was `appdata`) and runs as uid 1000 → it can create
>     folders + upload into the 2775 setgid tree (fixes the old permission-denied); app internals
>     (`appdata/`) are no longer browsable. **(v0.66.2)** its entrypoint is wrapped
>     `["sh","-c","umask 002; exec /home/filebrowser/filebrowser"]` so folders the customer creates come
>     out **2775 (group-writable)** — the gtstef image is a single Go binary that ignores a `UMASK` env
>     (verified), so the wrapper is the mechanism. Without it, customer folders were 2755 (group-read
>     only) and apps in group 1000 couldn't write into them.
>   - **Run-identity**: PUID/PGID-1000 apps (radarr/sonarr/calibre with `UMASK=002`) write **group-
>     writable**, so FileBrowser can fully manage their output. Root-only apps (jellyfin :ro; komga +
>     audiobookshelf RW after the `user:1000` fallback) write **group-readable** via setgid (FileBrowser
>     browses/reads, the app manages its own files).
>   - **Migration-safe**: `migrate.go`'s merge walk preserves the source dir's setgid + group (and
>     `copyFile` the full file mode + group), so the convention survives a whole-drive `MigrateAll`.
>
> The privileged controller-side disk subsections **below are historical** (the `internal/storage/*` scan/
> format code was removed in 8C — execution is the agent's now).

The storage subsystem handles the full lifecycle of external storage: detection, initialization, path registration, and data migration.

> **CURRENT (post-de-privileging + Impl-2b, v0.95.0):** the in-guest storage code below
> (`internal/storage/scan.go`, `format.go`, `attach.go`) is **retired** — all disk ops are delegated to
> the host agent via `internal/agentapi`. The two enrollment wizards (`/settings/storage/init`,
> `/settings/storage/attach`) now populate candidates from the agent's **raw-device scan**
> (`GET /api/disks/candidates` → agent Impl-2a, proxied by `agentDiskCandidatesHandler`): `initialize` =
> every unclaimed disk (blank or data-bearing), `attach` = the mountable-FS subset. The agent's
> unclaimed-disk filter (Impl-1 `claim.go`) is authoritative + fail-safe (never offers OS/enrolled/claimed
> disks), so the controller does NO client- or server-side filtering. Enrollment posts to the unchanged
> `/api/storage/init` (format via the agent's Impl-1 guarded `mkfs` → mount → bind → intent) or
> `/api/storage/attach` (mount + bind, no format). The legacy text below is kept for historical context.

#### Disk Scanning (`internal/storage/scan.go`)

- `ScanDisks()` uses `lsblk -J -b` for block device enumeration
- System disk detection via host fstab parsing (`/host-fstab`) + UUID resolution via `blkid`
- Partitions enriched with filesystem type, UUID, and label from direct `blkid` probing (Docker containers have incomplete udev cache)
- Returns `AvailableDisks` (non-system, non-loop, non-CDROM), `SystemDisks`, and `FormatablePartitions` (empty partitions on system disks that are safe to format)
- Handles NVMe (`nvme0n1p1`), SCSI (`sdb1`), and eMMC (`mmcblk0p1`) naming

#### Disk Initialization Wizard (`internal/storage/format.go`)

A step-by-step UI at `/settings/storage/init`:

1. **Scan** — Lists available disks with model, size, partition info
2. **Select** — User picks a disk and enters a mount name (e.g., `hdd_1`)
3. **Confirm** — User types "FORMAZAS" to confirm destructive operation
4. **Format pipeline**: `wipefs` → `sfdisk` (GPT) → `mkfs.ext4` → `blkid` UUID → backup fstab → append UUID-based fstab entry → mount → `findmnt` verification → `chown 1000:1000` → create `felhom-data/` and `Dokumentumok/` subdirectories
5. Auto-registers new storage path in settings.json
6. Smart partition detection: skips repartitioning for existing empty partitions

Safety guards: system disk detection, mount path conflict check, confirmation required, progress channel for real-time UI feedback.

**System-disk partition formatting:** When the system disk has an empty partition (no filesystem, not mounted, not used for /, /boot, /boot/efi, or swap), the init wizard detects it via `FormatablePartitions` in the scan result and offers to format just that partition. Uses `IsSystemPartition()` (granular per-partition check via fstab) instead of `IsSystemDisk()` (whole-disk block), so sda1 can be formatted while sda3 (root) remains protected.

#### Attach Existing Drive Wizard (`internal/storage/attach.go`)

A step-by-step UI at `/settings/storage/attach` for drives that already have a filesystem (e.g., a previously used ext4 drive). Unlike the init wizard, this does **not** format the drive — existing data is preserved.

**Problem solved:** Mounting a whole drive at `/mnt/<name>` would mix existing user data with the controller's directory structure (`felhom-data/`, `Dokumentumok/`, etc.). The bind-mount approach isolates the controller's working directory from other data on the drive.

1. **Scan** — Lists available disks, filtered to partitions that have an existing filesystem (FSType != "")
2. **Mount raw** — Partition is mounted read-only at a hidden staging path (`/mnt/.felhom-raw/<label>`)
3. **Browse** — Directory browser shows the drive's contents. User can navigate and create a new folder (e.g., `felhom_data`)
4. **Configure** — User enters a mount name and display label. Warning: mount path is immutable until detached
5. **Finalize** — Bind-mounts the selected subfolder at `/mnt/<name>`. Two fstab entries are created (both with `nofail`):
   - Raw mount: `UUID=<uuid> /mnt/.felhom-raw/<x> <fstype> defaults,nofail,noatime 0 2`
   - Bind mount: `/mnt/.felhom-raw/<x>/<subfolder> /mnt/<name> none bind,nofail 0 0`
6. Sets permissions (`chown 1000:1000`), creates `felhom-data/` and `Dokumentumok/` subdirectories
7. Auto-registers the storage path in settings.json + syncs FileBrowser mounts

Cancel at any point cleans up the temporary raw mount. The bind mount path (`/mnt/<name>`) is a real mount point, so all existing code (disk usage, IsMountPoint checks, etc.) works unchanged.

#### Storage Path Registry (`internal/settings/settings.go`)

Multiple external storage paths supported with:
- **Label**: Human-readable name (editable inline)
- **Default flag**: New deploys use this path by default
- **Schedulable flag**: Path appears in deploy dropdown
- **Disconnected state**: `Disconnected`, `DisconnectedAt`, `StoppedStacks` — set by watchdog or safe-disconnect API, cleared on reconnect
- **Auto-discovery**: On startup, scans deployed apps' `HDD_PATH` values and registers unknown paths
- Thread-safe CRUD: Add, Remove, SetDefault, SetSchedulable, SetLabel, SetDisconnected, ClearDisconnected

#### Data Migration (`internal/storage/migrate.go`)

Move app data between storage paths (e.g., SSD → HDD, HDD → new HDD):

1. Validate: stack exists, deployed, has HDD data, target differs from source
2. Estimate total size, check free space on target
3. Stop the application
4. `rsync -a --info=progress2` per mount path with real-time progress parsing
5. Update `app.yaml` HDD_PATH to new location
6. Start the application
7. **Rollback on failure**: reverts config, restarts on old storage

Progress UI at `/stacks/{name}/migrate` with byte counter and percentage.

#### Stale Data Cleanup

After migration, the deploy page detects leftover data on previous storage paths:
- Shows path, size, and a delete button
- Two-step confirmation required
- Protected paths (`felhom-data/`, `felhom-data/appdata/`, `felhom-data/backups/`, `media/`, `Dokumentumok/`) cannot be deleted

#### FileBrowser Mount Sync

When storage paths are added or removed, `syncFileBrowserMounts()` auto-regenerates FileBrowser's `docker-compose.yml` with volume mounts for all registered paths. It then **recreates the container only when the generated `config.yaml` or compose actually changed** (v0.82.0, F2) — gated by the pure helper `fbNeedsRecreate(oldCfg,newCfg,oldCompose,newCompose)`, which compares the on-disk content captured before the writes against the final content read after them (so the integrations' `ReapplyConfigForTarget` edits count). When nothing changed (a controller restart, a no-op sync) it issues a plain `up -d --remove-orphans` that does **not** bounce the running FileBrowser. The restore-mode DB reset (`down -v`) still forces a recreate.

#### Storage Watchdog (`internal/monitor/watchdog.go`)

Continuously monitors registered storage paths for disconnection/reconnection (primarily USB drives):

- **Probe loop**: `ProbeStoragePath()` calls `syscall.Statfs()` with 3-second timeout in a goroutine. Runs every 5s per connected path, 30s per disconnected path.
- **Debouncing**: 3 consecutive probe failures required before declaring a drive disconnected (prevents false positives from transient I/O).
- **Disconnect reaction** (automatic, ~15s detection):
  1. Stops all deployed stacks whose `HDD_PATH` is under the disconnected drive (skips protected stacks)
  2. Persists `Disconnected`, `DisconnectedAt`, `StoppedStacks` to `settings.json`
  3. Lazy-unmounts stale VFS entries (`umount -l`) — for attach-wizard drives, unmounts bind first, then raw
  4. Fires alert refresh (red banner on all pages), notification (`storage_disconnected`), and immediate hub report push
- **Auto-reconnect** (for UUID-based fstab entries):
  1. Checks `/host-dev/disk/by-uuid/<uuid>` for device reappearance
  2. Cleans stale mounts, then `mount -T /host-fstab <path>` (raw + bind for attach-wizard drives)
  3. Verifies with a post-mount probe
  4. Runs `restic unlock` if stale lock files exist
  5. Validates `StoppedStacks` (filters to actually-stopped stacks), clears `Disconnected` flag
  6. Fires alert refresh, notification (`storage_reconnected`), hub report push

**Safe disconnect UI** (manual, Settings page):

- "Leválasztás" button shown for USB drives (detected via sysfs symlink path containing `/usb`)
- Confirmation dialog lists affected apps
- Flow: stop apps → `sync` → `umount` (fallback `umount -l`) → mark disconnected → notification
- Disconnected card: dashed border, red badge, timestamp, stopped apps list, "Csatlakoztatás" (reconnect) button
- After reconnect: "Alkalmazások indítása" button to restart auto-stopped stacks

**USB detection** (`system.IsUSBDevice`): Reads `/host/sys/block/<disk>` symlink — if target path contains `/usb`, it's a USB device. The `removable` sysfs flag is unreliable for USB HDDs (returns 0). USB drives show an orange "USB" badge on their storage card alongside Aktív/Alapértelmezett badges (v0.27.2). Handles findmnt bind-mount suffix stripping (`/dev/sdb1[/subdir]` → `/dev/sdb1`) for attach-wizard drives (v0.32.5).

**Backup guards**: Nightly DB dumps, restic snapshots, and cross-drive backups all skip disconnected, removed, and inactive drives with WARN log (not treated as failures). Cross-drive `RunAppBackup()` returns nil (not error) for unavailable destinations — prevents noisy error aggregation in scheduled runs (v0.32.5).

**Tier2 destination unavailable (v0.32.5)**: When a Tier2 backup destination drive is disconnected, removed from storage, or deactivated (Inaktív), the backup page shows:
- Yellow status dot with "2. mentés szünetel" tooltip (not red)
- Warning badge: "Cél meghajtó leválasztva" (disconnected/removed) or "Cél meghajtó inaktív" (deactivated)
- Grayed-out last-run info and backup contents
- Hidden "Futtatás most" button (prevents futile manual triggers)
- "Beállítás" link preserved for reconfiguration
- Tier2 config persists — backups auto-resume when drive returns/reactivates
- Detection: `IsStoragePathKnown()` catches removed paths, `IsStoragePathSchedulable()` catches inactive/disconnected/decommissioned

**UI integration**: Disconnected drives show with hatched red bars on dashboard, monitoring, and backup pages. Per-app backup rows show "Meghajtó leválasztva" badge. Health check emits warnings for disconnected paths.

---

### 5. Monitoring & Health

#### System Health Checks (`internal/monitor/healthcheck.go`)

`RunHealthCheck()` evaluates multiple subsystems and returns a `HealthReport` with status (`ok`/`warn`/`fail`):

| Check | Warning | Critical |
|-------|---------|----------|
| Disk usage (SSD/HDD) | >= 90% | >= 95% |
| Memory | available < 512MB | available < 256MB |
| CPU temperature | >= 75C | >= 85C |
| Docker daemon | — | unreachable |
| Protected containers | — | not running |
| Storage paths | not a mount point (data on SSD), drive disconnected | path inaccessible, disk >= 95% |

Backup destination validation (`CheckBackupDestination`) has tiered checks:
- Path doesn't exist → critical/blocked
- Not writable → critical/blocked
- Same block device as root → warning (data on system drive)
- Disk >95% full → critical/blocked
- Disk >90% full → warning

#### Healthchecks.io Integration (deprecated)

Legacy pinger (`internal/monitor/pinger.go`) still runs for backward compatibility but is no longer the primary monitoring mechanism. Monitoring is now handled by the Hub event system (see [Notifications](#5-notifications)). A deprecation log is emitted on startup if ping UUIDs are configured.

#### Metrics Store (`internal/metrics/`)

- **SQLite with WAL mode** for concurrent reads during collection
- **System metrics**: CPU%, memory (total/used/available), temperature, load average — collected every 60 seconds
- **Container metrics**: CPU%, memory, network I/O, block I/O per container
- Downsampled queries for chart time ranges (1h, 6h, 24h, 7d, 30d)
- 30-day auto-prune via daily scheduler job

#### Monitoring Page

Full-page system monitor at `/monitoring`:
- **System Overview**: hostname, OS, kernel, CPU model/cores, uptime
- **System Metrics Charts**: 4 line charts (CPU, Memory, Temperature, Load) in 2x2 grid
- **Memory Distribution Bar**: stacked bar showing per-container memory usage, OS/system overhead, and free memory (real-time from `/proc/meminfo` + container stats)
- **Container Resources**: horizontal bar charts (CPU% and Memory per container)
- **Per-container Detail**: click-to-expand historical charts
- **Hub Connection Status**: shows Hub URL, customer ID, connection state (connected/unreachable), last successful push, last error

Chart.js 4.4.7 embedded locally (works in offline environments), dark theme matching site design.

#### Host (Proxmox box) Health — agent-proxied (slice 9, `internal/agentapi` + `agent_host_metrics_handler.go`)

The de-privileged controller (slice 8C) sees only its own cgroup and cannot read the host. The top card of `/monitoring` ("Szerver állapota (gazdagép)") instead shows the **real Proxmox box**, proxied from the host agent's `GET /host/metrics`:
- **Host block**: CPU% + load average, memory used/total, **CPU/chassis temperature** (or "n/a" when the hardware exposes no sensor — graceful-null), uptime.
- **Per-storage capacity**: a used/total bar per host storage target, with thin-pool fill (a full lvmthin pool corrupts every guest on it) and disk SMART temperature/wear.

Path: `GET /api/host-metrics` → `Client.HostMetrics()` (leaf-pinned, per-guest-token agentapi client) → agent `GET /host/metrics`. Host-wide and token-authed (assumption: **one customer per host** — the home-server model). It is a **live** fetch (a fresh agent collect, not the 15-minute hub snapshot), so the page polls it every **8 s** while open. When the agent is unconfigured/unreachable the card shows a "nem elérhető" banner; the controller's own metric charts are unaffected.

**Storage-bar ordering + labels (v0.57.0):** the agent enumerates storages via `pvesm` in a
non-deterministic order, so the per-storage capacity list (`#host-storage-bars`) reordered on every poll.
`enrichHostStorageTargets` (`agent_host_metrics_handler.go`) sorts the response **server-side** —
user-data (`usb`/`local-dir`) → system+apps (`lvmthin`/`lvm`) → builtin `local` → backup
(`pbs`/`nfs`/`cifs`) → other, alphabetical by id within a tier — and attaches a friendly Hungarian
`label` + one-line `purpose` per entry (rendered by `monitoring.html`, with the raw PVE id shown muted).
**Display labels only — the PVE storage ids are never renamed** (vzdump/PBS configs reference them by
name). This is distinct from the server-rendered, user-data-only `buildStorageBars` "Tárhely" list.

#### Alert System (`internal/web/alerts.go`)

State-based alerts displayed on all pages:
- Sources: health issues, Hub connection status, backup disabled, storage disconnected, update available
- Hub alerts: `hub-disabled` (warning) when Hub not enabled, `hub-unreachable` (error) when last push failed and no success in 30 min
- Sorted by severity (error > warning > info), capped at 5 visible
- Refreshed every 5 min + on startup + on storage state changes

---

### 6. Notifications

#### Hub Event System (`internal/notify/notifier.go`)

The controller pushes structured events to the Hub's `/api/v1/event` endpoint. The Hub handles notification dispatch, cooldown management, and dead man's switch detection.

**Core method:** `PushEvent(eventType, severity, message, details)` — non-blocking goroutine, 2 retries with 3s backoff, never blocks the caller.

#### Event Types

| Event Type | Severity | Trigger |
|------------|----------|---------|
| `backup_failed` | error | Nightly restic backup fails |
| `db_dump_completed` | info | Nightly database dumps succeed |
| `db_dump_failed` | error | Nightly database dumps fail |
| `backup_integrity_ok` | info | Weekly `restic check` passes |
| `backup_integrity_failed` | error | Weekly `restic check` fails |
| `crossdrive_completed` | info | Cross-drive secondary backup succeeds |
| `crossdrive_failed` | error | Cross-drive secondary backup fails |
| `health_degraded` | warning | Health status degrades (ok→warn) |
| `health_critical` | error | Health status critical (any→fail) |
| `health_recovered` | info | Health status recovers (fail/warn→ok) |
| `disk_warning` | warning | Disk usage crosses 90% |
| `disk_critical` | error | Disk usage crosses 95% |
| `storage_disconnected` | error | Storage drive physically removed |
| `storage_reconnected` | info | Storage drive reconnected |
| `controller_started` | info | Controller process starts |
| `controller_updated` | info/error | Self-update success or failure |
| `app_deployed` | info | New app deployed via API |
| `app_removed` | info | App removed via API |
| `app_start_failed` | warn | A DEPLOYED app is not running (fix-3) — fired ONCE per running→down transition |
| `disaster_recovery_started` | warning | DR restore begins |
| `disaster_recovery_completed` | info/error | DR restore finishes (success/partial) |

Each event carries typed detail structs (e.g., `BackupDetails`, `DiskDetails`, `HealthDetails`) serialized as JSON.

**Deployed-app-down alerting (fix-3, v0.120.0, CAMPAIGN-3).** A `deadapp-check` scheduler job (every
30 s, after a 90 s boot grace) scans `stackMgr.GetStacks()`: a DEPLOYED app whose containers are
`stopped`/`exited` (`stacks.IsDownState` — a Docker `created`/`dead` container, the F11 dead-at-boot
case, resolves to `stopped`) gets a **state-based WARN dashboard banner** ("Telepített alkalmazás nem
fut: <app>", grouped above 3 so a reboot storm doesn't wall the dashboard) that self-clears when the
app runs again, AND an `app_start_failed` hub event fired **once per running→down transition**
(`Notifier.NotifyAppStartFailures` tracks per-app state; down→down cycles are silent — the hub owns
the real cooldown, the controller adds no timer). The boot grace prevents false alarms during the
controller's own startup while STILL firing for an app that never came up. This closes the campaign's
4-hour silent CWA death.

#### Default Enabled Events

Events the customer receives notifications for (configurable in settings):
`backup_failed`, `db_dump_failed`, `disk_warning`, `disk_critical`, `storage_disconnected`, `node_down`, `health_critical`, `expected_backup_missed`, `expected_dbdump_missed`

#### Preference Sync

Notification preferences (email, enabled events, cooldown hours) are:
- Stored locally in `settings.json`
- Synced to Hub on save and on controller startup via `POST /api/v1/preferences`
- Hub sync failure doesn't block local save

---

### 7. Update Management

#### App Catalog Sync

- Periodic `git fetch` + `git reset --hard` of the app catalog repo
- Content-hash comparison prevents unnecessary file writes
- Post-sync stack rescan detects new/changed apps immediately
- **Stale lock recovery**: automatically removes `.git/index.lock`, `.git/shallow.lock`, and `.git/HEAD.lock` before each fetch — prevents permanent sync failures after interrupted operations (e.g. container restart mid-sync)

#### Planned Update Classifications

| Marker | Behavior |
|--------|----------|
| No marker | Optional — shown on dashboard, customer clicks "Update" |
| `UPDATE_REQUIRED=true` | Mandatory — auto-applied during next update window |
| `UPDATE_SECURITY=true` | Critical — applied immediately |

#### Controller Self-Update (`internal/selfupdate/`)

The controller can update itself to the latest registry version with a one-click Settings button. In the
LXC architecture there is **no in-container compose** to drive (the old `docker compose -f
/opt/docker/felhom-controller/docker-compose.yml up -d` path does not exist in the guest — it produced
"docker-compose.yml nem elérhető"). Instead (Phase 1, v0.85.0) the controller **pulls** the target image
in-guest and delegates the container **swap to the host agent**, which owns the restart + health-verify +
**rollback**.

##### How It Works

```
1. Check Gitea Docker Registry V2 API for the latest semver tag (queryRegistry, BasicAuth).
2. If newer than current (ldflags Version): docker login --password-stdin → docker pull <image>
   → docker logout — IN-GUEST over the shared docker socket (token via stdin, never argv).
3. Delegate to the host agent: agentapi.SwapController → POST /controller/swap {image} (202). The agent
   (external to this container) rewrites /etc/felhom-controller-image, restarts
   felhom-controller-bootstrap.service, polls the new controller to healthy, and ROLLS BACK to the
   previous image if it doesn't come up. The controller never docker-rm/recreates itself.
4. On startup the new container reads update-state.json → VerifyStartup marks success (current==target)
   or failure (rollback → version mismatch). The Settings button polls /api/health and reloads.
```

The button is latest-only (no version picker) and opt-in. No host agent wired (un-provisioned guest) →
self-update unavailable.

##### Phase 2 — managed updates: the version FLOOR (v0.86.0)

On top of the opt-in button, the controller now honors an operator-enforced **minimum version (FLOOR)**.
The hub returns the customer's *effective* floor (per-customer override else a global default) on the
**report ACK** (`min_controller_version`, alongside `latest_version`). The controller's report pusher
(`internal/report/pusher.go`, `OnPushResponse`) hands the floor to the updater (`SetFloor`) and calls
`MaybeAutoUpdate()` — **on the existing report cycle, no new timer/endpoint**:

- If the box is **below** the floor it **auto-updates to the floor** (not latest) by reusing the Phase 1
  flow above (`performUpdate`, `initiatedBy="auto-floor"`) — same pull → agent swap → rollback. No
  customer click.
- **At/above** the floor: nothing (it does **not** chase latest — that's the button's job).
- Guards: dev build / no agent / backup running → skip; floor must be **pullable** (floor ≤ latest
  available; floor > latest → warn + do nothing); one attempt per below-floor condition (in-memory flag
  + persisted `update-state.json`) → **no flapping/storm**.
- Settings UI shows "Minimális verzió (üzemeltető): X" and, during an auto-update, the same restart-poll
  panel as the button.

The floor is the **auto-target** (the operator raises it for a controlled fleet rollout); latest stays
the customer's manual opt-in. Floor source + operator UI are hub-side (felhom-hub v0.15.0). **No agent
change — Phase 2 reuses the Phase 1 `POST /controller/swap`.**

##### Pull-based config-refresh (v0.94.0)

The same **report ACK** also carries a per-customer **`config_version`** (a hub-side stored counter,
bumped on every config save; felhom-hub v0.26.0). This is how an operator config edit reaches a running
box — **the hub never connects into the box** (it replaced the retired inbound "Push Config"). Wired in
`OnPushResponse` beside the floor reconcile (`internal/report/config_refresh.go`, `ConfigRefresher.Reconcile`):

- **Changed** vs. the last-applied version (`settings.applied_config_version`) → `bootstrap.RefreshConfig`
  re-pulls `controller.yaml` from the hub and rewrites it, **re-merging the per-guest `local_api`** from
  `bootstrap.json` (reuses the first-boot pull machinery; overwrites `controller.yaml` since the hub is its
  source of truth; **never touches `settings.json`**) → record the new version → **graceful self-restart**
  (`api.GracefulSelfRestart` → `os.Exit(0)` → Docker `restart: unless-stopped` re-reads the new config).
- **First-ever** ACK (nothing recorded) → record the baseline **without restarting** (the first-boot pull
  already has the current config).
- **Unchanged** version → no-op (so no restart storm — after a refresh `applied == ACK`).
- **Failed pull** → keep the current config, do **not** record/restart, retry next report cycle.
- Record-before-restart so the restarted process sees the version applied and doesn't loop. Only the
  `felhom-controller` container restarts; customer app stacks are untouched. (A config apply rotates
  `web.session_secret`, so dashboard sessions are invalidated — same as the old Push-Config path.)

##### Design Philosophy

- **No automatic rollback** — follows the Watchtower pattern (24k+ GitHub stars, no rollback). Docker's `restart: unless-stopped` policy is the crash safety net. The Hub's dead man's switch detects when the controller goes down.
- **Audit state file** — `update-state.json` in the data volume records every update attempt (previous version, target version, initiator, result). Operators can SSH in and revert using `PreviousImage` from this file.
- **Backup-aware** — refuses to start an update while a backup is in progress (`backupRunning()` guard).

##### Package Structure

| File | Purpose |
|------|---------|
| `version.go` | `ParseVersion("X.Y.Z")` → `Version{Major,Minor,Patch}`, `Compare()` returns -1/0/1. Hand-rolled, no external deps. Rejects "dev" and "latest". |
| `state.go` | `UpdateState` struct persisted as JSON. `LoadState()`, `SaveState()` (atomic: `.tmp` + rename), `ClearState()`. Status values: `"pending"`, `"success"`, `"failed"`. |
| `updater.go` | Core `Updater` struct. Registry check via HTTP GET to `gitea.dooplex.hu/v2/admin/felhom-controller/tags/list` with Basic Auth (git username/token). Update trigger: `docker pull` → compose file regex replace → `docker compose up -d`. Thread-safe with `sync.Mutex`. |

##### Update Trigger Flow

1. **Guard checks:** concurrent update lock, dev version check, backup running check, compose file accessible
2. Write `update-state.json` with status `"pending"` (audit trail)
3. `docker pull <image>:<targetVersion>`
4. Read compose file → replace image tag via regexp → atomic write (`.tmp` + rename)
5. `docker compose -f /opt/docker/felhom-controller/docker-compose.yml -p felhom-controller up -d`
6. Docker kills the current container, starts the new one

##### Startup Verification

Called once from `main.go` before the scheduler starts:
1. Load `update-state.json` — if missing or status != `"pending"`, nothing to do
2. Compare running `Version` with `state.TargetVersion`
3. **Match** → mark `"success"`, notify via hub
4. **Mismatch** → mark `"failed"`, notify via hub
5. No rollback attempt — operator reverts manually if needed

##### Auto-Update Scheduling

Two separate scheduler jobs prevent interference with backups:

| Job | Type | Default | Purpose |
|-----|------|---------|---------|
| `selfupdate-check` | `sched.Every` | 6h | Check registry, cache result (for UI). Never triggers update. |
| `selfupdate-auto` | `sched.Daily` | 04:30 | If auto-update enabled + update available + backup not running → trigger. |

The auto-update time (`config.SelfUpdate.AutoUpdateTime`, default `"04:30"`) is deliberately separate from the backup window (02:30-~04:00) to avoid collisions. The `backupRunning()` guard is the hard safety check — if backups run long past 04:30, the update is skipped and retried the next day.

An initial version check fires 30s after startup so the Settings page shows version info quickly.

##### Compose File Access

The controller needs write access to its own `docker-compose.yml`. This is achieved via Docker volume mount ordering:

```yaml
volumes:
  # 1. Directory mount — gives access to compose file + config
  - /opt/docker/felhom-controller:/opt/docker/felhom-controller
  # 2. Read-only override — prevents accidental config writes
  - /opt/docker/felhom-controller/controller.yaml:/opt/docker/felhom-controller/controller.yaml:ro
  # 3. Named volume override — persistent data in Docker-managed volume
  - controller-data:/opt/docker/felhom-controller/data
```

##### API Endpoints

| Method | Path | Auth | Description |
|--------|------|------|-------------|
| GET | `/api/selfupdate/status` | Session or API key | Current status (cached, no network call) |
| POST | `/api/selfupdate/check` | Session or API key | Force registry check, return result |
| POST | `/api/selfupdate/update` | Session or API key | Trigger update (async, returns immediately) |

Self-update endpoints accept either session auth (for UI) or hub API key as bearer token (for external triggering from build scripts or hub). This enables the post-v0.16.0 deploy workflow:

```bash
# After building + pushing new image:
curl -s -X POST https://felhom.demo-felhom.eu/api/selfupdate/update \
  -H "Authorization: Bearer <HUB_API_KEY>"
```

##### Settings Page UI

The "Verzió és frissítés" card on the Settings page (`/settings`) shows:
- Current version and latest available version
- "Frissítés elérhető" (update available) badge
- Last check time and any errors
- Registry mode line (v0.112.0): "Registry: nyilvános (hitelesítés nélkül)" vs "Registry: hitelesített" — credential-less is a supported mode, not an error state
- Auto-update status with configured time
- Last update result (success/failed/pending)
- **Buttons:** "Frissítés keresése" (check) + "Frissítés telepítése" (apply)

##### Registry access modes (v0.112.0)

Git Sync credentials (`git.username`/`git.token`) are **optional — for private catalogs only**; version
discovery and self-update work without them:
- **Anonymous (both empty):** `queryRegistry` performs the Docker Registry v2 anonymous token dance —
  plain GET → 401 with `WWW-Authenticate` → token fetched from the ADVERTISED realm (parsed from the
  header, never hardcoded — registry-agnostic) with no credentials → Bearer retry. `pullImage` skips
  `docker login` entirely (docker's native anonymous flow covers public packages).
- **Authenticated (both set):** the previous BasicAuth + login/pull/logout path, unchanged.
- **Half-configured (only one set):** loud incomplete-credentials error — never a silent anonymous
  downgrade.
- A registry that genuinely denies anonymous access surfaces
  "registry denied anonymous access — a private registry requires Git Sync credentials".

After triggering an update, the page polls `/api/health` every 3s and reloads when the new container responds.

A global info-level alert ("Új controller verzió elérhető") appears on all pages when an update is available, linking to the Settings page.

##### Configuration

```yaml
self_update:
  enabled: true
  check_interval: "6h"          # How often to check registry
  image: "gitea.dooplex.hu/admin/felhom-controller"  # Default
  auto_update: false             # Set true for unattended updates
  auto_update_time: "04:30"     # When to auto-apply (after backups)
  health_timeout_seconds: 60    # Reserved for future use
```

##### Edge Cases

| Scenario | Behavior |
|----------|----------|
| `Version == "dev"` | `ParseVersion` returns error → no updates reported, trigger refused |
| Registry unreachable | Log warning, return error in check result. No crash. |
| No registry credentials | Return error "Registry hitelesítő adatok hiányoznak" |
| Compose file not writable | Refuse update before doing anything |
| Backup running | Refuse with "Mentés fut, próbálja később" |
| Concurrent update | Mutex prevents duplicates: "Frissítés már folyamatban" |
| Bad update (crash loop) | Docker restarts container. State file stays "pending". Operator SSH-reverts using `PreviousImage`. |
| Corrupt state file | Treated as "no pending update", logged, deleted |

---

### 8. Authentication & Settings

#### Customer-claim gate (`internal/web/claim.go`, v0.122.0 — closes DRILL-day0-vm F-4/F-5)

The dashboard password is **customer-owned**, set through a one-time claim code the hub emails to
the registered address (no operator-set path, no open-until-set window). This closes the fresh-box
race where a new `felhom.<domain>` cert appears in CT logs minutes before any password exists.

- **States** (precedence): a SET password (settings→config) always wins — the gate never shows.
  Else a delivered **claim-code hash + not-yet-claimed** → GATED: every route serves the claim
  page (`302 → /claim`) or `401` JSON (API); only `/claim*`, `/static/*`, `/api/health` pass. Else
  (no password, no hash) → **legacy-open** with a red transition banner until the hub delivers a
  hash (transitional only, never the fresh-box state).
- **Claim/reset flow**: `GET /claim` (code + new password ×2, min 12) → `POST /claim` verifies the
  code (bcrypt match AND generation not yet consumed AND ≤ 72 h old), sets the customer's password
  via `settings.SetPasswordHash`, marks `Claimed` (set-only), consumes the generation (single-use),
  invalidates sessions, issues a fresh one. `POST /claim/request-new-code` (the "Új kód kérése" /
  login-page "Elfelejtett jelszó") forwards to the hub, which emails a fresh code to the
  registered address only. Reset rides the same page (a claimed box reaches `/claim` pre-auth).
- **Anti-brute-force**: per-source + global counter, 5 failures → 15-minute lockout (both scopes),
  raising the allowlisted `claim_lockout` event. Pre-auth CSRF is an HMAC over `web.session_secret`
  (fixes the CTRL-007 bare-double-submit weakness).
- **Delivery**: the hub bakes `web.claim_code_{hash,generation,issued_at}` into the Day-0
  controller.yaml (gate-from-first-boot) and serves the freshest state in the report ACK
  (`report/claim_sync.go` caches it idempotently by generation — newer advances, same/older/nil
  never rewrites, a hub outage never clears). The report carries `claimed` (hub ingests set-only).
- **Escape hatch**: `felhom-controller --print-reset-code` prints a one-time local code (generation
  above cached/baked/consumed); the same gate consumes it. Root-gated by `docker exec` reachability.

#### Session Auth (`internal/web/auth.go`)

- bcrypt password verification with configurable source priority: `settings.json` → `controller.yaml` → no auth (open access)
- 7-day session duration with random 32-byte hex tokens
- `?next=` redirect after login preserves the page the user was visiting
- Session cleanup every 15 minutes
- All sessions invalidated on password change
- Conditional logout link (hidden when auth is disabled)
- Each session stores a dedicated CSRF token (separate 32-byte random value) alongside the session token

#### CSRF Protection (`internal/web/csrf.go`)

Synchronizer-token CSRF protection on all browser-facing state-mutating endpoints.

**How it works:**
- `CsrfProtect` middleware wraps all route handlers in `main.go`
- Safe methods (GET, HEAD, OPTIONS) pass through without validation
- For POST/DELETE/PATCH: reads token from `_csrf` form field or `X-CSRF-Token` request header; constant-time compares against the session's stored CSRF token
- On rejection: JSON `{"ok":false,"error":"CSRF token missing or invalid"}` for `/api/` paths; HTTP 403 text page for UI routes
- Logs: `[WARN] CSRF rejected: METHOD /path from addr (reason)`

**Exempt paths (no CSRF check):**
- Requests with `Authorization: Bearer ...` header — hub→controller API calls (selfupdate, config/apply). Browsers cannot auto-send Bearer headers, so cross-site requests are impossible on these endpoints.
- Auth-disabled mode (`authEnabled() == false`) — CSRF is meaningless when there is no session.

**Token delivery to templates:**
- `executeTemplate(w, r, name, data)` wrapper in `server.go` auto-injects `CSRFField` (`template.HTML` hidden `<input>`) and `CSRFToken` (raw string) into every page's data map
- `layout.html` emits `<meta name="csrf-token" content="{{.CSRFToken}}">` and defines `csrfHeaders()` JS function in `<head>` (before page scripts)
- Forms: `{{.CSRFField}}` (or `{{$.CSRFField}}` inside `{{range}}` loops — outer scope required)
- JS `fetch()` calls: `headers: csrfHeaders()` — returns `{'X-CSRF-Token': metaContent}`
- Dynamically-created JS forms: read token from `document.querySelector('meta[name="csrf-token"]').content`
- `navigator.sendBeacon()` replaced with `fetch(..., {keepalive: true})` where used — `sendBeacon` cannot send custom headers

#### Settings Persistence (`internal/settings/settings.go`)

Runtime-mutable settings in `settings.json` (separate from infrastructure config):

| Section | Contents |
|---------|----------|
| `password_hash` | bcrypt hash override |
| `notifications` | email, enabled events, cooldown hours |
| `db_validations` | per-DB dump validation results (survives restarts) |
| `app_backup` | per-app map: enabled flag, cross-drive config (method, dest, schedule, runtime status) |
| `storage_paths` | registered paths with label, default flag, schedulable flag, disconnected state |
| `cross_drive_restic_password` | auto-generated restic password for cross-drive repos |

All public methods use `sync.RWMutex`. File writes are atomic (`.tmp` + rename).

#### Settings Page (`/settings`)

Five sections:
1. **System config** — read-only display of `controller.yaml` values
2. **Version & update** — current/latest version, check/update buttons, auto-update status, last update result
3. **Storage paths** — add/remove, edit labels, set default, toggle schedulable, per-path app list with sizes, safe disconnect/reconnect for USB drives
4. **Password change** — current + new + confirm, min 8 chars
5. **Notifications** — email, event checkboxes, cooldown hours, test email button

---

### 9. Central Hub Reporting

#### Report Push (`internal/report/`)

Periodic JSON push (default every 15 min) to the central felhom-hub service:
- System: hostname, OS, CPU, memory, disk usage, uptime
- Containers: running/stopped counts, per-container CPU/memory
- Backup: last DB-dump run, success (disk-tier backup is the host agent's; restic password is no longer reported — removed v0.69.0)
- Health: current status, issues, warnings
- Stacks: deployed apps with versions and states
- Config hash: SHA256 of `controller.yaml` for Hub-side config comparison
- **Geo-restriction (always present, v0.70.0):** `geo_restriction` is always populated — `Enabled=false` with an empty country list when never configured — so the Hub always renders the geo section ("Inaktív" when off) instead of hiding it. `buildGeoRestrictionReport` in `internal/report/builder.go`.
- **App telemetry** (v0.28.0+): Per-stack memory (current/avg/peak) and CPU averages from the last 15 minutes of metrics data, plus log scan results (error/warning counts with deduplicated issues). Only non-protected, deployed stacks are included. Backward-compatible: old Hub versions silently ignore this field.
- **Controller telemetry** (v0.32.4+): The controller's own container (`felhom-controller`) is included as a special entry in the `app_telemetry` array. Its memory/CPU metrics come from the same metrics collector, and its log warnings/errors are scanned via `docker logs` using the same pipeline as app containers. This reuses all existing Hub telemetry infrastructure (memory trend charts, known issues, fleet aggregation) with zero Hub-side changes.
- **DR recipe — customer + apps half (v0.73.0):** `dr_recipe` is the controller half of the secret-free reconstruction recipe (`SPIKE-dr-recipe-2026-06-16.md`) that complements escrow (keys) + PBS/restic (bytes). `BuildDRRecipeAppHalf` (`internal/report/dr_recipe.go`) emits `{recipe_version, customer{id,display,domain}, apps[]}`; each deployed, non-protected app contributes `AppRecipe{catalog_ref, enabled, storage_bindings}` where bindings are parsed from the compose (`${HDD_PATH}`/`${USERDATA_PATH}` volume binds → `{container_path, drive, subpath}`, e.g. romm → `felhom-flash:userdata/roms`). **THE BOUNDARY:** the emitter is the enforcement point — it ships an explicit allowlist of those three fields and reads NOTHING from `AppConfig.Env`, so no `ENC:`/token/password can leak (allowlist, not denylist → new fields excluded by default). The load-bearing `TestBuildAppRecipe_NoSecrets` + its red-proof companion live here. The hub assembles this half with the agent's storage/guest/PBS half into one customer recipe. `recipe_version=1`, ignore-unknown on read.

Bearer token authentication, 3-attempt retry with 5-second backoff. Push status tracked via `PushStatus` struct (LastAttempt, LastSuccess, LastError, consecutive failures) — used by the monitoring page and alert system to show Hub connection health.

**Immediate report push on geo change (v0.70.0):** besides the periodic cycle, a successful geo settings save and a successful manual geo sync fire an **out-of-band, non-blocking** report push (`Router.reportPushNow`, wired in `main.go` to `BuildReport`+`Push` in a goroutine), so the Hub reflects the new geo state / clears a stale `last_sync_error` within seconds instead of after the next ~15-min cycle. Currently scoped to the geo handlers; the same seam can be reused for other settings later.

#### Config apply + self-restart (`internal/api/router.go`, `internal/api/selfrestart.go`)

`POST /api/config/apply` (Hub-authed) writes a new `controller.yaml`, but the new config only takes effect on **restart** — singletons such as the Cloudflare client are built once at startup (so a rotated CF API token would otherwise keep failing). Behaviour (v0.70.0):

- **No-op guard:** if the pushed body is byte-identical to the current file, do nothing — no rewrite, no restart (the Hub may re-push idempotently; never flap on a no-op).
- Otherwise: write `0600`, respond `200` (flushed), then **gracefully self-restart** — `gracefulSelfRestart` waits ~500 ms for the response to flush, then `os.Exit(0)`. The container runs `restart: unless-stopped`, so Docker brings it back with the fresh config, and a startup report is pushed. The exit sits behind an injectable seam (`Router.restart` / `SetRestarter`) for unit testing.

**Manual restart (v0.70.0):** `POST /api/selfrestart` (session auth + CSRF via the `/api/` mount) runs the same helper — surfaced as the **"Vezérlő újraindítása"** button on the settings page (confirm → POST → poll `GET /` every 2 s → reload), so a customer can recover the controller without rebooting the whole guest.

**Full-server restart (v0.81.0):** `POST /api/server/reboot` (session auth + CSRF) reboots the **whole guest** via the host agent's `GuestReboot` primitive (`HandleServerReboot` in `internal/web/storage_handlers.go`, delegating to the testable `serverReboot` core; agent reboots detached + returns 202). Surfaced as a separate **"Kiszolgáló újraindítása"** settings card alongside the controller-only restart, reusing the same `pollRestart()` reload loop. It replaces the retired drive-activation banner (v0.81.0): in the intermediary-mount model an enrolled drive binds **live** into the running guest, so storage no longer needs a reboot to activate — this button is purely a deliberate full-system restart.

#### App Telemetry (`internal/metrics/telemetry.go`, `internal/metrics/logscanner.go`, `internal/report/telemetry.go`)

Each report push now includes per-app telemetry data:

**Metrics collection** (`telemetry.go`):
- `MetricsStore.GetContainerTelemetry(since)` aggregates container-level memory (avg, peak, current) and CPU averages from the `container_metrics` SQLite table for the last 15 minutes.

**Log scanning** (`logscanner.go`):
- `ScanContainerLogs(containerNames, since, logger)` runs `docker logs --since=15m --tail=1000` sequentially on all non-protected deployed containers.
- Classifies lines by keyword match (errors: `error`, `fatal`, `panic`, `crit`, `oom`, `killed`, `exception`, `traceback`; warnings: `warn`, `warning`) on the first 5 words (case-insensitive).
- Deduplicates via fingerprinting: strips ANSI escape codes, ISO timestamps (with timezone offsets), and syslog timestamps (including mid-line); replaces 6+ digit numbers with `<N>`, 8+ char hex with `<HEX>`, UUIDs with `<UUID>`. Groups identical fingerprints, keeps top 10 per container.
- Returns `[]ContainerLogSummary` with `ErrorCount`, `WarnCount`, `RecentIssues []LogIssue`.
- **Error context** (v0.111.0): each error-severity issue carries `Context []string` — up to ±5 raw lines around its FIRST occurrence in the scrape window (≤11 lines, ≤400 chars/line with `…`, ANSI-stripped, redacted). Warns never carry context. The classification loop is the pure `analyzeLogLines()` (unit-tested with synthetic windows).
- **Redaction** (`redact.go`, v0.111.0): `RedactLine()` masks `password|passwd|secret|token|api[_-]?key|authorization|bearer` values (`[REDACTED]`, incl. `Authorization: Bearer <tok>`) and 64-hex strings (`[REDACTED-HEX64]`) on every context and log-tail line before it leaves the box — controller-side, authoritative.

**Report integration** (`report/telemetry.go`):
- `buildAppTelemetrySection()` calls both, then `buildAppTelemetry()` aggregates by stack — summing container metrics, merging issues, capping at 10 per app. Additionally, `buildControllerTelemetry()` creates a special entry for the controller container itself (`app_name: "felhom-controller"`).
- Results stored as `[]AppTelemetry` in the `Report` struct field `app_telemetry`.
- **Context budget** (v0.111.0): `enforceContextBudget()` caps the per-report total of all issue-context bytes at 16KB, dropping context from the lowest-count issues first.

**On-demand log tails** (`report/logtail.go`, v0.111.0 — pull-based, same ACK-flag pattern as escrow/config-refresh):
- The report ACK (`PushResponse`) gains `log_tail_requests: [app…]` — apps the operator requested logs for on the hub. `main.go`'s `OnPushResponse` hands them to `report.SetPendingLogTails()`; the NEXT `BuildReport` drains them (consume-once) and ships `log_tails: [{app, collected_at, lines[]}]`.
- Collection: `stacks.GetLogs(app, 200)` (compose logs, ordered as emitted) for stacks; `metrics.FetchContainerLogTail()` (`docker logs --tail=200`) for the `felhom-controller` container. Caps: ≤400 chars/line, ≤64KB/app head-truncated (newest lines kept), every line through `RedactLine()`.
- Fail-safe: a failed fetch or failed push leaves the hub's request pending — the next ACK re-arms it. The hub clears the request when a tail arrives. NO hub→controller push channel exists; the guest listens to no one.

#### Infrastructure Backup to Hub — RETIRED (2026-06-16)

> **Removed.** The controller no longer pushes any infra-backup to the Hub, and the Hub no longer
> accepts or stores one (hub v0.12.0). The builder (`internal/report/infra_backup.go`) and local
> mirror (`internal/backup/local_infra.go`) were deleted back in slice 8C; the last caller-less stub
> (`Pusher.PushInfraBackup`) and the `backup_completed` event were removed in controller v0.69.0.
> DR now rests on the agent's PBS whole-CT snapshot + the Hub-generated controller.yaml. The text
> below is **historical** and describes the removed mechanism — much of this section (and the
> `local_infra.go` / `setup/scanner.go` / `PullRecovery` / `restore_drives` references elsewhere in
> this README) is stale slice-8C debt. See `felhom.eu/documentation/audits/SPIKE-infra-backup-2026-06-15.md`.

After each backup cycle (including manual Tier 2 triggers via `OnCrossDriveComplete` callback), the controller pushed a full infrastructure snapshot to the Hub for disaster recovery. This snapshot included:
- `controller.yaml` (base64-encoded, full config including secrets)
- `settings.json` (base64-encoded, backup prefs, storage paths, cross-drive configs)
- Disk layout (UUIDs, labels, mount points, fstab options, bind-mount topology)
- Deployed stacks manifest (app names, HDD paths) with actual config files: `docker-compose.yml`, `app.yaml`, `.felhom.yml` (base64-encoded per stack, v0.34.0)
- Restic passwords (primary + cross-drive, base64-encoded)

This enables fully automated recovery when the system drive is replaced — the new controller pulls the snapshot from the Hub, auto-mounts surviving drives by UUID, and restores all applications.

#### Hub Dashboard

The hub service (separate Go app in the `felhom.eu` repo) provides:
- Multi-customer overview table with status indicators and event count badges
- Customer detail page with system/storage/containers/backup/health/events sections
- Event timeline: last 50 events with severity filter, colored badges, source tracking
- Dead man's switch: staleness detection (30min stale, 60min down), missed backup detection (daily at 05:00)
- Notification dispatch: operator (English) + customer (Hungarian) emails via Resend with per-event cooldowns
- Infra backup status per customer (last sync, stack count, disk count)
- Color coding: green (<30min), yellow (30-60min), red (>60min since last report)
- 90-day report + event retention with daily prune at 04:30 Budapest time

### 10. First-Run Setup Wizard

When the controller starts with no valid customer configuration (`customer.id` empty), it enters **setup mode** — a web-based wizard that handles all initial configuration. This replaces the old interactive shell wizard in `docker-setup.sh`.

#### Setup Mode Detection (`internal/setup/setup.go`)

`NeedsSetup(cfg)` returns true when `customer.id` is empty or a `.needs-setup` marker file exists. In setup mode, the controller skips normal startup (no scheduler, no backup, no stacks) and serves only the wizard UI on two listeners:
- `:8080` — behind Traefik (accessible via domain, e.g. `https://felhom.example.com`)
- `:8081` — direct HTTP (accessible via LAN IP, e.g. `http://192.168.0.100:8081`)

#### Wizard Flow

```
┌──────────────────────────────────┐
│  1. Welcome                      │
│  Choose: Restore / Fresh install │
└─────────┬───────────┬────────────┘
          │           │
    ┌─────▼─────┐  ┌──▼───────────────┐
    │ 2a. Scan  │  │ 2b. Hub download  │
    │ drives for│  │ (customer ID +    │
    │ local     │  │  password)        │
    │ backups   │  │                   │
    └─────┬─────┘  └──────┬────────────┘
          │               │
    ┌─────▼─────┐         │
    │ 2a.2 Hub  │         │
    │ recovery  │         │
    │ (fallback)│         │
    └─────┬─────┘         │
          │               │
    ┌─────▼─────┐  ┌──────▼───────────┐
    │ Execute   │  │ Execute fresh    │
    │ restore   │  │ install          │
    └─────┬─────┘  └──────┬───────────┘
          │               │
          └───────┬───────┘
                  ▼
          os.Exit(0) → Docker restarts
          → normal mode
```

#### Hub Pre-Seeding

When `docker-setup.sh` is run with `--hub-customer` / `--hub-password`, the controller receives
pre-seeded credentials via environment variables:

| Env var | Purpose |
|---------|---------|
| `FELHOM_SETUP_CUSTOMER_ID` | Pre-fills customer ID in wizard forms |
| `FELHOM_SETUP_PASSWORD` | Pre-fills retrieval password for auto-processing |

In hub mode, the welcome page shows three cards instead of two:
1. **"Visszaállítás a Hub-ról"** — auto-calls `PullRecovery()`, shows infra backup details
2. **"Visszaállítás helyi meghajtóról"** — standard drive scan
3. **"Friss telepítés"** — auto-calls `PullConfig()`, downloads config only

Both hub paths auto-process when credentials are pre-seeded (no form entry needed).
On error, the wizard falls back to the manual form with the error displayed.

#### Key Components

| File | Purpose |
|------|---------|
| `setup/setup.go` | `NeedsSetup()` detection, `SetupState` persistence to `setup-state.json` |
| `setup/handlers.go` | HTTP handlers for each wizard step (welcome, scan, hub-restore, fresh, manual) |
| `setup/scanner.go` | Scans all block devices for `.felhom-infra-backup/` directories (current + `history/`) via `lsblk` + temp mounts; returns rich info (app names, disk count) |
| `setup/hub.go` | Hub recovery pull (`GET /api/v1/recovery/{id}`) and config download |
| `setup/csrf.go` | Lightweight CSRF protection (cookie + hidden field, `SameSite=Strict`) |
| `setup/network.go` | Detects local IPs for LAN access URL display |
| `setup/templates/` | 8 embedded HTML templates (Hungarian, dark theme matching main UI) — includes `setup_hub_versions.html` for Hub backup version picker |

#### Local Infra Backup (`internal/backup/local_infra.go`)

The controller writes infrastructure snapshots to **every connected drive** after each backup cycle and on startup. Location: `<drive>/.felhom-infra-backup/`. Files:
- `backup.json` — full infra backup (config, settings, disk layout, passwords, stacks)
- `metadata.json` — schema version, timestamp, customer ID, controller version, SHA256 checksum
- `history/` — previous backup versions (last 5), rotated automatically before each write
  - `{timestamp}-backup.json` + `{timestamp}-metadata.json` pairs (timestamp format: `20060102T150405Z`)
  - Oldest entries pruned when count exceeds 5

During setup wizard drive scan, both current and historical backups are discovered, integrity-verified, and offered for one-click restore. The scan results table shows app names/count, disk count, and a "korábbi" badge for historical versions.

#### Recovery Info (`internal/recovery/info.go`)

Generates `recovery-info.txt` on the system data partition with customer ID, Hub URL, retrieval password, and recovery instructions in Hungarian. Updated on startup and after config changes. Also displayed on the Settings page in a "Vészhelyzeti információk" section.

### 11. Disaster Recovery

When a system drive fails and is replaced, the recovery flow uses the setup wizard:

```
1. docker-setup.sh deploys fresh controller with minimal config
   - With --hub-customer: credentials pre-seeded via env vars
   - Without: user enters credentials manually in wizard
2. Controller detects empty customer.id → enters setup mode
3. User opens wizard at http://<LAN-IP>:8081
4. Hub mode: welcome page shows Hub restore / local scan / fresh install
   Non-hub mode: welcome page shows restore / fresh install
5. Hub restore: auto-connects to Hub, shows version picker if multiple versions
   Local restore: scans all drives for .felhom-infra-backup/ directories (current + history/)
6. User selects backup version → restore: config, settings, passwords, disk layout
7. Controller restarts into normal mode with full config
8. Controller auto-mounts surviving drives by UUID from disk layout
9. Dashboard shows "Visszaállítás" (Restore) page for app-level recovery
10. User confirms → sequential restore: rsync first, restic fallback, DB import
```

**Backup sources (priority order):**
1. **Local infra backup** (`.felhom-infra-backup/` on surviving drives) — fastest, no network needed
2. **Hub recovery endpoint** (`GET /api/v1/recovery/{id}`) — requires retrieval password, supports `?version=ID` for specific versions; Hub retains ~14 versions via GFS pruning (7 daily / 4 weekly / 3 monthly)
3. **Manual config** (wizard form) — enter all details manually as last resort

**Hub verification:** After setup, the controller periodically verifies customer standing via the Hub report push response (`customer_blocked` field). If blocked or Hub unreachable for >7 days, the controller enters limited mode (no new deployments).

---

### 12. Asset Sync

App assets (logos, screenshots) are managed centrally by the Hub and downloaded to each controller via a daily sync process. This decouples asset updates from controller image rebuilds — new app icons only require a Hub redeploy.

#### How It Works (`internal/assets/syncer.go`)

```
1. Fetch manifest from Hub: GET /api/v1/assets/manifest (Bearer auth)
2. Compare SHA-256 checksums with local cache (<dataDir>/assets/)
3. Download changed/new files: GET /api/v1/assets/file/{filename}
4. Remove local files not in Hub manifest (stale cleanup)
5. Save local manifest copy for next comparison
```

#### Asset Resolution (two-tier)

| Priority | Path | Source |
|----------|------|--------|
| 1 | `<dataDir>/assets/` | Downloaded from Hub (synced cache) |
| 2 | `/usr/share/felhom/assets/` | Baked into Docker image (fallback) |

The `Resolve(filename)` method checks the synced cache first, then falls back to the baked-in directory. This ensures assets are always available even before the first sync.

The Felhom logo (`/static/felhom-logo.svg`) also uses this two-tier resolution: the logo handler checks synced assets first, then falls back to the embedded SVG constant. This allows logo updates via Hub without a controller rebuild. The logo is also used as an SVG favicon.

#### Configuration

```yaml
assets:
  sync_enabled: true       # Opt-in: download assets from Hub API
  sync_schedule: "05:00"   # Daily sync time (HH:MM, Budapest timezone)
```

Asset sync requires `hub.enabled: true` with valid `hub.url` and `hub.api_key`. The initial sync runs 10 seconds after startup (to let subsystems initialize), then daily at the configured time.

#### Sync Status

The syncer tracks status (last sync time, result, file count, total bytes) accessible via `GET /api/assets/status`. On-demand sync can be triggered via `POST /api/assets/sync`.

#### File Types

The Hub serves three asset types per app:
- `{slug}-logo.svg` — primary SVG logo
- `{slug}-logo.png` — PNG fallback
- `{slug}-screenshot-{N}.webp` — app screenshots

#### Key Design Decisions

- **Opt-in via `sync_enabled`** — backward compatible, baked-in assets still work without Hub
- **SHA-256 change detection** — only downloads files that actually changed (bandwidth efficient)
- **Atomic file writes** — downloads to `.tmp` then `os.Rename` for crash safety
- **Stale file cleanup** — removes local files not in the Hub manifest (e.g., deleted apps)
- **Non-blocking initial sync** — runs in a goroutine with 10s delay, doesn't block startup

---

### 13. Debug Mode

When `logging.level: "debug"` is set in `controller.yaml`, the controller exposes a full diagnostic dashboard at `/debug` with 9 testing sections. All debug endpoints are gated — at `info` level, the sidebar link disappears and all `/api/debug/*` routes return 404.

#### Debug Page Sections

| # | Section | Endpoints | Description |
|---|---------|-----------|-------------|
| 1 | Rendszer diagnosztika | `GET /api/debug/dump` | Full state dump: controller info, storage, stacks, scheduler, health, alerts. JSON download. |
| 2 | Értesítés teszt | `POST /api/debug/event/test`, `GET /api/debug/event/history` | Send test events with configurable type/severity, view event history ring buffer. |
| 3 | Mentés teszt | `POST /api/debug/backup/{dbdump,crossdrive,integrity,infra}` | Trigger individual backup phases independently. |
| 4 | Tárhely teszt | `POST /api/debug/storage/simulate-{disconnect,reconnect}`, `GET /api/debug/storage/watchdog-status` | Simulate drive disconnect/reconnect without unmounting. Per-path probe state with 5s auto-refresh. |
| 5 | Hub & Kapcsolatok | `POST /api/debug/hub/{push,infra-push,test-connectivity,preferences-sync}`, `POST /api/debug/gitea/test-connectivity` | Test Hub/Gitea connectivity with latency. Push reports and sync preferences. |
| — | Telemetria teszt | `GET /api/debug/telemetry` | Run the full telemetry collection pipeline on-demand (metrics query + log scan). Returns per-app table: container list, memory current/avg/peak, CPU avg, catalog limit, log error/warning counts, and top issues. Useful for verifying container→stack mapping and testing log scanner patterns without waiting for the 15-minute report cycle. |
| 6 | Önfrissítés teszt | `POST /api/debug/selfupdate/dry-run` | Dry-run update check: current vs new image lines, compose writability, backup state. |
| 7 | DR / Telepítő varázsló | `POST /api/debug/dr/trigger-setup`, `GET /api/debug/dr/infra-status` | Infra backup status per drive. Trigger setup mode via marker file (requires "RESET" + infra backup pre-check). |
| 8 | Naplóviewer | `GET /api/debug/logs?level=&limit=&after=`, `GET /api/debug/agent-logs` | In-memory log viewer (last 5000 entries, spill-persisted across restart — fix-6), level filter, 2s auto-refresh, color-coded entries. Two tabs (v0.116.0): **Vezérlő** (own ring) and **Ügynök** (the agent's always-DEBUG ring proxied over the local API; a pre-0.83 agent renders the "available after the agent's next update" notice). |

#### Key Implementation Details

- **Log buffer** (`internal/web/logbuffer.go`): Ring buffer implementing `io.Writer`. Since v0.116.0 it ALWAYS exists (any `logging.level`) and captures every line INCLUDING `[DEBUG]`: the logger is `io.MultiWriter(LevelFilterWriter(os.Stdout, logging.level), logBuffer)` — stdout/docker-logs keep respecting `logging.level`, the ring holds the full detail for remote diagnostics. `logBuffer.Lines(maxBytes)` renders the newest-kept plain-text tail (the report `controller_log_tail` source). New leveled lines use `internal/logx` (`Debugf/Infof/Warnf/Errorf`); legacy `isDebug()` call sites are unchanged.
  - **Ring sizing, spill persistence, periodic-noise policy (fix-6, v0.120.0, CAMPAIGN-3).** The
    campaign measured the 1000-entry ring wrapping in ~6.5 min under load and dying on every restart —
    the exact post-incident window was the first thing lost. Three changes: **(a) cap 1000→5000**
    (`Entries`/the debug handler display cap raised to match — a larger ring is useless if the viewer
    can't request more than 1000 of it; the Naplóviewer default pull is 1000). **(b) periodic-noise
    policy:** a periodic job's ROUTINE success is not ring-worthy — the every-cycle scheduler "job
    finished" line and `refreshStatusLocked` per-cycle enumeration are logged at a new `[TRACE]` level
    that the ring DROPS at write-time (`levelPriority("TRACE") < DEBUG`). Failures and state changes are
    never TRACE, so nothing load-bearing is lost; this was the biggest ring filler. **(c) spill
    persistence:** `LogBuffer.SpillTo`/`LoadFrom` atomically (tmp+rename, JSON-lines) spill the ring to
    `<DataDir>/debug-ring.log` on the **SSD state dir** (the persistent data volume that survives
    container recreation — NEVER a NAS/HDD path) every 30 s and on clean shutdown, and load it back on
    boot so a restart / recreate preserves the pre-restart window. Corruption-safe: a truncated/partial
    line is skipped on load, never fatal.
- **Controller self-log pull** (`internal/report/selftail.go`): the hub's report ACK may carry `controller_log_requested` — the NEXT report ships `controller_log_tail` (ring, 128 KB cap, consume-once, the v0.111.0 app-tail pattern; additive fields, app-tail wire unchanged). Serving a pull logs the customer-visible `operator log pull served` INFO line.
- **Storage simulation**: `simulatedPaths` map in watchdog prevents the watchdog from re-probing simulated-disconnected paths. Disconnect runs all real steps except `lazyUnmount` (drive stays physically mounted).
- **DR trigger safety**: Uses marker file (`data/.needs-setup`) instead of modifying controller.yaml. Pre-checks that infra backup exists on at least one drive.
- **Routing**: `/api/debug/` carved out in HTTP mux (same pattern as `/api/storage/`), routed to web server with auth + CSRF.
- **DebugCallbacks**: 7 closures wired from main.go for operations needing modules not on Server struct (hub push, infra backup, connectivity tests, telemetry preview).
- **Telemetry debug**: `GetTelemetryPreview` callback calls `report.BuildAppTelemetryForDebug()` (exported wrapper around the private `buildAppTelemetrySection()`). Result renders as a table with collapsible raw JSON. Available regardless of hub configuration.

#### Per-Module Logging

All modules emit structured log lines at `[INFO]`, `[WARN]`, and `[ERROR]` levels for operational events (state changes, completions, failures). When `logging.level: "debug"`, additional detailed `[DEBUG] [module]` prefixed log lines are emitted. Each module with stateful debug (struct-based) exposes a `SetDebug(bool)` method, wired from `main.go`. Modules without a struct use package-level `DebugLogger` variables (e.g., `system.DebugLogger`).

**Standard-level logging** (always active):
- `[INFO]` — Operational events: stack deploy/start/stop, backup completion, config changes, disk operations, sync results
- `[WARN]` — Degraded states: health threshold breaches, unsafe backup destinations, retryable failures, best-effort operation failures
- `[ERROR]` — Hard failures: data restore errors, integration apply failures, compose file update errors, disk format failures

| Module | Debug Field | Prefix | Key Areas |
|--------|------------|--------|-----------|
| `stacks` | `cfg.Logging.Level` | `[DEBUG] [stacks]` | Stack CRUD, compose commands, env vars, HDD mounts, encryption migration, health probes |
| `backup` | `ResticManager.debug` | `[DEBUG] [restic]` / `[DEBUG] [backup]` | Restic commands, snapshot operations, restore scanning, drive mounting |
| `cloudflare` | `Client.debug` + `GeoSyncManager.debug` | `[CF-DEBUG]` / `[DEBUG] [cloudflare]` | API requests/responses, WAF rule CRUD, zone resolution, geo sync diff |
| `integrations` | `Manager.debug` | `[DEBUG] [integrations]` | Toggle apply/revoke timing, lifecycle hooks, config reapply |
| `system` | `DebugLogger` | `[DEBUG] [system]` | Memory/disk/CPU/load/temp collection, mount probing, USB detection |
| `monitor` | `Pinger.debug` | `[DEBUG] [pinger]` | Health ping URLs, retry attempts, response codes |
| `settings` | `Settings.debug` | `[DEBUG] [settings]` | Load/save sizes, storage path ops, geo/integration state changes |
| `scheduler` | `Scheduler.debug` | `[DEBUG] [sched]` | Job registration, execution timing, daily schedule calculations |
| `web` | `cfg.Logging.Level` | `[DEBUG] [web]` | HTTP requests, auth decisions, session management, storage API ops |
| `api` | `Router.debug` | `[DEBUG] [api]` | API routing, handler entry points, request details |
| `selfupdate` | `Updater.debug` | `[DEBUG] [selfupdate]` | Version checks, update preconditions, docker pull timing |
| `assets` | `Syncer.debug` | `[DEBUG] [assets]` | Manifest fetch, hash comparison, file download timing |
| `storage` | logger-based | `[DEBUG] [storage]` | Disk scanning, formatting, attach, drive migration |
| `metrics` | logger-based | `[DEBUG] [metrics]` | Per-container log scanning, error/warning counts |
| `appexport` | `Exporter.debug` | `[DEBUG] [appexport]` | Export/import steps, crypto operations, bundle scanning |

---

### 14. Geo-Restriction

Country-based access control via **Cloudflare WAF Custom Rules**. The controller manages WAF rules in the `http_request_firewall_custom` phase to block requests from non-allowed countries. Rules are identified by a `[felhom-geo]` description prefix — other WAF rules are never touched.

#### Prerequisites

The existing `cf_api_token` (used for DNS-01 ACME) needs **Zone WAF:Edit** permission added. No new token is needed — just expanded permissions on the same token. The settings UI only appears when a CF API token is configured.

#### Architecture

```
┌─────────────┐     ┌──────────────────┐     ┌──────────────────────┐
│  Settings UI │────▶│  GeoSyncManager  │────▶│  Cloudflare WAF API  │
│ (settings.   │     │  (geosync.go)    │     │  /zones/{id}/        │
│  html)       │     │  diff & apply    │     │  rulesets/{id}/rules │
└─────────────┘     └──────────────────┘     └──────────────────────┘
       │                     ▲
       │  POST /api/geo/*    │  Scheduler (6h)
       ▼                     │  + deploy/remove hooks
┌─────────────┐              │
│  API layer  │──────────────┘
│  (geo.go)   │
└─────────────┘
```

**Rule structure:**
- **Global rule**: `(not ip.src.country in {"HU"})` → block (with `http.host ne` exclusions for apps that have per-app overrides)
- **Per-app rule**: `(http.host eq "app.example.com" and not ip.src.country in {"HU" "US"})` → block
- **Block response**: HTTP 403 with Hungarian message

**Local network access** is inherently unaffected — traffic from the LAN goes directly to the server, bypassing Cloudflare entirely.

#### Cloudflare API Client (`internal/cloudflare/`)

| File | Purpose |
|------|---------|
| `client.go` | HTTP client with Bearer token auth, 15s timeout, generic `do()` helper |
| `zone.go` | Zone ID resolution — tries exact domain, then parent domains progressively |
| `waf.go` | WAF rule CRUD, expression builders (`BuildGlobalExpression`, `BuildAppExpression`) |
| `countries.go` | ~250 ISO 3166-1 alpha-2 codes with Hungarian names |
| `geosync.go` | Sync orchestrator — diffs desired vs existing rules, creates/updates/deletes |

**GeoSyncManager** uses a `StackLister` interface (implemented by `geoStackAdapter` in main.go) to get deployed app hostnames without circular imports.

#### Settings Model

Stored in `settings.json` (runtime-modifiable):

```go
type GeoRestriction struct {
    Enabled          bool                      `json:"enabled"`
    AllowedCountries []string                  `json:"allowed_countries"`
    AppOverrides     map[string]AppGeoOverride `json:"app_overrides,omitempty"`
    LastSync         string                    `json:"last_sync,omitempty"`
    LastSyncError    string                    `json:"last_sync_error,omitempty"`
    ZoneID           string                    `json:"zone_id,omitempty"`
    RulesetID        string                    `json:"ruleset_id,omitempty"`
}
```

Thread-safe access via `GetGeoRestriction()`, `SetGeoRestriction()`, `SetGeoAppOverride()`, `RemoveGeoAppOverride()`, `SetGeoSyncState()`.

#### API Endpoints

| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/geo/status` | Current geo settings + sync state |
| POST | `/api/geo/settings` | Update global settings (enable/disable, countries) |
| POST | `/api/geo/sync` | Trigger manual sync |
| GET | `/api/geo/countries` | Full country list for search UI |
| POST | `/api/stacks/{name}/geo/override` | Set per-app country override |
| DELETE | `/api/stacks/{name}/geo/override` | Remove per-app override |

All mutating endpoints trigger an async Cloudflare sync. The `/api/geo/` path accepts both session auth and Hub Bearer token auth (via `selfUpdateAuthMiddleware`), enabling Hub-side geo-disable for lockout recovery.

#### Sync Triggers

1. **Settings change** — user saves geo settings or per-app override
2. **Deploy/remove** — app deployment or removal changes the hostname list
3. **Scheduler** — periodic verification every 6 hours
4. **Startup** — delayed initial sync 15s after boot
5. **Manual** — "Szinkronizálás" button on settings page

#### UI

**Settings page** ("Beállítások" → "Földrajzi korlátozás"):
- Enable/disable toggle
- Searchable country autocomplete with tag-based selection
- Hungary pinned with `confirm()` warning on removal
- Per-app overrides summary with add/edit/remove
- Sync status display (last sync time, errors)

**App detail page** (per-app override, shown when geo is globally enabled):
- Toggle for custom country restriction
- Independent country selector

---

### 15. App-to-App Integrations

Generic framework for connecting deployed applications to each other. Provider apps declare available integrations in `.felhom.yml`, and users enable/disable them via toggle switches on the provider's deploy/settings page ("Beállítások").

#### Architecture (`internal/integrations/`)

- **`integrations.go`** — Core types: `Handler` interface (`Apply`/`Revoke`), `ApplyContext` (carries domain, decrypted env vars, provider metadata, stacks dir, logger, restart func), `StatusInfo` (UI data), `IntegrationKey()`/`ParseIntegrationKey()` key helpers
- **`manager.go`** — `Manager` coordinates toggle operations, builds apply contexts from decrypted app.yaml env vars. Uses `StackProvider` interface (GetStack, GetStacks, RestartStack) to break circular imports with stacks package — adapted via `integrationStackAdapter` in main.go. Key methods:
  - `Toggle(ctx, provider, target, enable)` — Validates both apps deployed+running, calls Apply/Revoke, persists state
  - `ListForProvider(slug)` — Returns `[]StatusInfo` for UI with target deployment/running status
  - `ReapplyConfigForTarget(name)` — Re-applies all active integrations targeting a stack (config-only, no restart). Used by `SyncFileBrowserMounts` after config regeneration
- **`lifecycle.go`** — Lifecycle hooks called from API router goroutines:
  - `OnStackStop` — Revokes active integrations, sets `"provider_stopped"`/`"target_unavailable"` (keeps `enabled=true`)
  - `OnStackStart` — Re-applies enabled integrations after 5s delay (waits for stack state refresh). Accepts both `StateRunning` and `StateStarting` via `isStackUp()` helper
  - `OnStackRemove` — Revokes and permanently deletes integration state
- **Handler implementations** — One file per integration pair (e.g. `onlyoffice_filebrowser.go`, `onlyoffice_nextcloud.go`)

#### Integration State

Stored in `settings.json` under `integrations` map (key: `"provider:target"`):
- `enabled` — User intent (survives stop/restart)
- `status` — Current state: `"active"`, `"error"`, `"disabled"`, `"provider_stopped"`, `"target_unavailable"`
- `last_error` — Most recent error message
- `enabled_at` — RFC3339 timestamp

CRUD methods in settings.go: `GetIntegrationState`, `SetIntegrationState`, `RemoveIntegrationState`, `GetIntegrationsForProvider`, `GetIntegrationsForTarget` (all use existing RWMutex + atomic write pattern).

#### Lifecycle

1. **Enable**: User toggles on → validates both apps deployed+running → calls `Handler.Apply()` → persists state as `"active"`
2. **Disable**: User toggles off → calls `Handler.Revoke()` → persists state as `"disabled"`
3. **Provider/target stops**: `OnStackStop` → calls `Handler.Revoke()` → sets status to `"provider_stopped"` or `"target_unavailable"` (keeps `enabled=true`)
4. **Provider/target starts**: `OnStackStart` (5s delay) → finds enabled integrations with non-active status → re-applies if both sides running/starting
5. **Provider/target removed**: `OnStackRemove` → revokes and deletes integration state permanently
6. **FileBrowser config regen**: `SyncFileBrowserMounts` regenerates `config.yaml` from scratch → `ReapplyConfigForTarget("filebrowser")` patches integration config synchronously → recreates the container **only when the final `config.yaml`/compose differ from the pre-sync content** (`fbNeedsRecreate` gate, v0.82.0)

**Important**: `SyncFileBrowserMounts` uses `--force-recreate` (rather than a plain `up -d`) **when something changed**, because `config.yaml` is a bind mount — without `--force-recreate`, `docker compose up -d` won't recreate the container when only the config file changes (compose only detects compose-file changes). The recreate is now **gated on an actual change** (v0.82.0, F2): a controller restart or no-op sync where the generated config+compose are byte-identical issues a plain `up -d --remove-orphans` and does **not** bounce the customer's file UI. `ReapplyConfigForTarget` calls each handler's `Apply` with a no-op `RestartStack` since the caller handles the restart.

#### Built-in Handlers

**OnlyOffice → FileBrowser** (`onlyoffice_filebrowser.go`):
- Apply: Reads `JWT_SECRET` + `SUBDOMAIN` from OnlyOffice app.yaml (decrypted), strips any existing `integrations:` block from FileBrowser `config.yaml` via `removeIntegrationsSection()`, appends new block with `url` (public HTTPS), `internalUrl` (`http://onlyoffice:80`), `secret`, `viewOnly: false`. Atomic write (`.tmp` + rename). Restarts FileBrowser
- Revoke: Strips `integrations:` block from config.yaml, restarts FileBrowser

**OnlyOffice → Nextcloud** (`onlyoffice_nextcloud.go`):
- Apply: Runs `docker exec -u www-data nextcloud php occ` commands:
  1. `app:install onlyoffice` (tolerates "already installed")
  2. `app:enable onlyoffice`
  3. `config:app:set onlyoffice DocumentServerUrl --value=https://{subdomain}.{domain}`
  4. `config:app:set onlyoffice DocumentServerInternalUrl --value=http://onlyoffice:80`
  5. `config:app:set onlyoffice jwt_secret --value={JWT_SECRET}`
  6. `config:app:set onlyoffice StorageUrl --value=http://nextcloud` (internal callback URL)
- Revoke: Runs `occ app:disable onlyoffice` (tolerates container not running / app not enabled)

**OnlyOffice compose template notes**: Requires Traefik middleware `X-Forwarded-Proto=https` in labels so the Document Server generates HTTPS URLs for editor resources (prevents mixed content errors in browser).

#### Metadata (`.felhom.yml`)

Provider apps declare integrations in their `.felhom.yml`. Parsed into `IntegrationDef` struct in `metadata.go`, with `HasIntegrations()` helper.

```yaml
integrations:
  - target: filebrowser
    label: "FileBrowser integráció"
    description: "Dokumentumok szerkesztése a fájlkezelőben"
  - target: nextcloud
    label: "Nextcloud integráció"
    description: "Dokumentumok szerkesztése a Nextcloudban"
```

#### API Endpoints

| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/api/integrations/{provider}` | List integrations for a provider app (status, target availability) |
| POST | `/api/integrations/{provider}/{target}` | Enable/disable integration (`{"enabled": true/false}`) |

Routes registered **before** `hasSuffix`-based stack routes in router.go (see router bug pattern).

#### UI

Toggle switches on the provider's deploy/settings page ("Integrációk" section, within `deploy.html`). Data wired in `deployHandler()` for deployed apps only. Each integration shows:
- Label and description from `.felhom.yml` metadata
- Status badge: "Aktív", "Nincs telepítve", "Célalkalmazás leállítva", "Hiba"
- Toggle checkbox (disabled when target not deployed/running)
- JS `toggleIntegration()` → POST to API → reload on success

#### Wiring (main.go)

- `integrationStackAdapter` type implements `integrations.StackProvider` (same pattern as `stackAdapter`, `geoStackAdapter`)
- `integrations.NewManager(sett, adapter, domain, stacksDir, encKey, logger)` — registers built-in handlers
- Wired into API router via `SetIntegrationManager()` and web server via `SetIntegrationManager()`

---

## Repository Layout

```
controller/
├── cmd/controller/main.go           # Entry point, wires all 17 modules (setup mode branch + normal startup)
├── internal/
│   ├── config/config.go             # YAML loader, validation, env overrides
│   ├── crypto/crypto.go             # AES-256-GCM encryption for app.yaml secrets, key management
│   ├── settings/settings.go         # Runtime settings (JSON, atomic writes, RWMutex)
│   ├── stacks/
│   │   ├── manager.go               # Stack scanning, compose ops, container status
│   │   ├── metadata.go              # Parse .felhom.yml app metadata
│   │   ├── deploy.go                # First-deploy: secret gen, app.yaml, compose up; missing field injection
│   │   └── delete.go                # Stack deletion/removal + HDD/backup data cleanup
│   ├── sync/sync.go                 # Git sync: clone/pull app catalog, content-hash copy
│   ├── storage/
│   │   ├── scan.go, scan_linux.go   # Disk detection via lsblk + blkid
│   │   ├── format.go, format_linux.go  # Partition, format, mount pipeline
│   │   ├── attach.go, attach_linux.go  # Attach existing FS drive (raw mount + bind mount)
│   │   ├── safety.go, safety_linux.go  # System disk detection, mount guards, fstab ops
│   │   ├── migrate.go              # App data migration (rsync with progress)
│   │   └── *_other.go              # Non-Linux stubs for cross-compilation
│   ├── backup/
│   │   ├── backup.go               # Orchestrator (per-drive dumps + restic + cross-drive chain)
│   │   ├── paths.go                # Per-drive path helpers (FelhomDataDir constant, PrimaryResticRepoPath, AppDataDir, InfraBackupDir, etc.)
│   │   ├── local_infra.go          # Local infra backup to all drives (.felhom-infra-backup/)
│   │   ├── dbdump.go               # DB auto-discovery + dump (pg_dump, mariadb-dump)
│   │   ├── restic.go               # Restic operations (init, snapshot, prune, check) — repoPath as param
│   │   ├── appdata.go              # StackDataProvider interface, app data discovery
│   │   ├── crossdrive.go           # Per-app backup to secondary storage (rsync/restic)
│   │   ├── restore.go              # Per-app restore from per-drive repo
│   │   ├── restore_scan.go         # DR: scan drives for backup data, build restore plan
│   │   ├── restore_app_linux.go    # DR: per-app restore (rsync config/data + docker compose up)
│   │   └── restore_drives_linux.go # DR: auto-mount drives by UUID from Hub infra backup
│   ├── cloudflare/
│   │   ├── client.go               # CF API client (Bearer auth, generic JSON helper)
│   │   ├── zone.go                 # Zone ID resolution (domain → zone)
│   │   ├── waf.go                  # WAF rule CRUD + expression builders
│   │   ├── countries.go            # ISO 3166-1 country codes + Hungarian names
│   │   └── geosync.go              # Geo sync orchestrator (diff & apply rules)
│   ├── integrations/
│   │   ├── integrations.go          # Core types: Handler interface, ApplyContext, StatusInfo
│   │   ├── manager.go               # Manager: Toggle, ListForProvider, StackProvider interface
│   │   ├── lifecycle.go             # OnStackStop, OnStackStart, OnStackRemove hooks
│   │   ├── onlyoffice_filebrowser.go # OnlyOffice → FileBrowser handler (config.yaml patch)
│   │   └── onlyoffice_nextcloud.go  # OnlyOffice → Nextcloud handler (occ commands)
│   ├── assets/syncer.go             # Hub asset sync (download, SHA-256 compare, resolve)
│   ├── api/
│   │   ├── router.go               # REST API endpoints (~36 routes)
│   │   └── geo.go                  # Geo-restriction API handlers
│   ├── scheduler/scheduler.go      # Central job scheduler (Every, Daily)
│   ├── system/
│   │   ├── info.go, info_linux.go  # RAM, disk, CPU, temperature, load average
│   │   ├── cpu_linux.go            # Background /proc/stat sampling
│   │   └── mounts_linux.go         # Mount points, disk usage, FS info, backup dest checks, storage probing, USB detection
│   ├── monitor/
│   │   ├── pinger.go               # Healthchecks.io HTTP ping client
│   │   ├── healthcheck.go          # System health checks (disk, mem, CPU, temp, Docker)
│   │   └── watchdog.go             # Storage watchdog (probe, disconnect/reconnect, safe eject)
│   ├── metrics/
│   │   ├── store.go                # SQLite time-series (WAL mode, downsampled queries)
│   │   ├── collector.go            # Background collector (60s, system + docker stats)
│   │   └── sysinfo.go              # Static system info (/proc, /etc)
│   ├── selfupdate/
│   │   ├── version.go              # Semver parsing + comparison (hand-rolled)
│   │   ├── state.go                # Update audit state (JSON, atomic writes)
│   │   └── updater.go              # Registry check, update trigger, startup verify
│   ├── notify/notifier.go          # Email relay to hub, preference sync, cooldowns
│   ├── report/
│   │   ├── builder.go              # Hub report builder (all subsystems → JSON)
│   │   ├── pusher.go               # HTTP POST to hub (retry, Bearer auth, parses customer_blocked)
│   │   └── infra_pull.go           # DR: pull recovery/config from Hub (retrieval password auth)
│   ├── setup/                      # First-run setup wizard (web-based, replaces docker-setup.sh wizard)
│   │   ├── setup.go                # NeedsSetup() detection, state persistence
│   │   ├── handlers.go             # HTTP handlers for all wizard steps
│   │   ├── scanner.go              # Drive scanner for local infra backups
│   │   ├── csrf.go                 # Lightweight CSRF (cookie + hidden field)
│   │   ├── network.go              # Local IP detection for LAN access URLs
│   │   └── templates/              # 7 wizard HTML templates (Hungarian)
│   ├── recovery/info.go            # Recovery info file generator (recovery-info.txt)
│   └── web/
│       ├── server.go               # HTTP server, routing, static files, catch-all middleware, executeTemplate wrapper
│       ├── auth.go                 # Session auth + per-session CSRF token, login/logout, session cleanup
│       ├── csrf.go                 # CsrfProtect middleware, csrfToken/csrfField helpers
│       ├── handlers.go             # Page handlers (dashboard, stacks, deploy, backups, etc.)
│       ├── handler_restore.go      # DR: restore page handler + APIs (scan, restore all, skip)
│       ├── handler_debug.go        # Debug page handler + 20 debug API endpoints (debug-mode only)
│       ├── logbuffer.go            # Ring buffer (io.Writer) for in-memory log capture
│       ├── storage_handlers.go     # Storage API handlers (scan, format, attach, migrate, cleanup, disconnect/reconnect)
│       ├── alerts.go               # State-based alert generation
│       ├── funcmap.go              # Template functions (state colors, Hungarian formatting)
│       ├── embed.go                # go:embed for templates + Chart.js
│       └── templates/              # 21 HTML files (4-page settings split) + style.css + icons.html sprite (Hungarian UI, design system v2)
├── configs/
│   ├── controller.yaml.example     # Full config reference
│   └── example-felhom-metadata.yml # .felhom.yml format reference
├── Dockerfile                      # Multi-stage: Go 1.24 builder + debian-slim runtime
├── docker-compose.yml              # Controller's own compose (privileged, /mnt rshared)
└── go.mod                          # Go 1.24, deps: bcrypt, yaml.v3, modernc.org/sqlite
```

---

## Configuration

### Controller config (`controller.yaml`)

Single YAML file per customer, infrastructure-only. Does **not** contain app-specific config.

Key sections:
```yaml
customer:
  name: "Demo Felhom"
  id: "demo-felhom"

paths:
  stacks_dir: "/opt/docker/stacks"
  data_dir: "/opt/docker/felhom-controller/data"
  system_data_path: "/mnt/sys_drive"   # NVMe/system drive — fallback for apps without HDD

git:
  repo_url: "https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git"
  sync_interval: "15m"

# Per-drive backup paths are computed automatically:
#   <drive>/backups/primary/restic/          — restic repo per drive
#   <drive>/backups/primary/<app>/db-dumps/  — DB dumps per app
#   <drive>/backups/secondary/               — cross-drive rsync + restic
backup:
  enabled: true
  restic_password_file: "/opt/docker/felhom-controller/data/restic-password"
  db_dump_schedule: "02:30"
  restic_schedule: "03:00"
  retention: { keep_daily: 7, keep_weekly: 4, keep_monthly: 6 }

monitoring:
  health_interval: "5m"
  ping_uuids:
    heartbeat: "uuid-here"
    system_health: "uuid-here"
    db_dump: "uuid-here"
    backup: "uuid-here"
    backup_integrity: "uuid-here"

web:
  listen: ":8080"
  setup_listen: ":8081"   # Plain HTTP for setup wizard LAN access

hub:
  enabled: true
  url: "https://hub.felhom.eu"
  api_key: "bearer-token-here"

assets:
  sync_enabled: true       # Download app assets (logos, screenshots) from Hub API
  sync_schedule: "05:00"   # Daily sync time (HH:MM, Budapest timezone)

system:
  reserved_memory_mb: 384  # RAM reserved for OS + controller
```

Environment variable overrides: `FELHOM_LOGGING_LEVEL=debug`, `FELHOM_HUB_ENABLED=false`, etc.

### Runtime settings (`settings.json`)

Auto-managed by the controller. Contains password hash overrides, notification preferences, per-app backup configs, storage path registry, DB validation cache, Hub verification state (`hub_verified`, `hub_verified_at`), retrieval password for disaster recovery, and pending event queue. All writes are atomic (write `.tmp`, rename).

### Per-app config (`app.yaml`)

Auto-generated during deployment. Contains env vars, locked fields list, deploy timestamp. Secret fields are locked (read-only after first deploy). Missing fields from updated templates are auto-injected on startup and after sync (see Missing Field Injection).

**Encryption at rest**: Sensitive env values (`type: password` and `type: secret` from `.felhom.yml` metadata) are stored encrypted as `ENC:base64(nonce+ciphertext)` using AES-256-GCM. The 32-byte encryption key is stored at `{dataDir}/encryption.key` (generated on first run, 0600 permissions). Values are decrypted transparently when passed to docker-compose or displayed in the UI. The key is included in infra backups (Hub + local drives) and restored during disaster recovery. On upgrade, existing plaintext values are migrated automatically on startup.

---

## Scheduler Jobs

| Job | Type | When | Purpose |
|-----|------|------|---------|
| status-refresh | periodic | 30s | Refresh container states |
| stack-scan | periodic | 2m | Rescan stacks directory |
| heartbeat | periodic | 5m | Legacy Healthchecks ping (deprecated — Hub handles via event system) |
| system-health | periodic | configurable | Health checks + alert refresh |
| backup-cache | periodic | 5m | Refresh backup status cache |
| hub-report | periodic | 15m | Push report to central hub |
| db-dump | daily | 02:30 | Database dumps |
| backup | daily | 03:00 | Restic backup → cross-drive chain |
| backup-integrity | daily | Sun 04:00 | Restic check |
| metrics-prune | daily | 04:00 | Delete metrics older than 30 days |
| selfupdate-check | periodic | 6h | Check registry for new version (cache for UI) |
| selfupdate-auto | daily | 04:30 | Auto-update if enabled + backup not running |
| asset-sync | daily | 05:00 | Download changed app assets from Hub |

All daily jobs use Europe/Budapest timezone. Skip-if-running prevents concurrent execution. Panic recovery in all jobs.

---

## REST API

### Stack Operations

| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/api/health` | Health check (no auth) |
| GET | `/api/stacks` | List all stacks |
| GET | `/api/stacks/{name}` | Stack details |
| POST | `/api/stacks/{name}/deploy` | First-time deploy |
| POST | `/api/stacks/{name}/start` | Start stack (409 if insufficient memory) |
| POST | `/api/stacks/{name}/stop` | Stop stack |
| POST | `/api/stacks/{name}/restart` | Restart stack |
| POST | `/api/stacks/{name}/update` | Pull + recreate |
| POST | `/api/stacks/{name}/optional-config` | Update optional env vars |
| GET | `/api/stacks/{name}/logs` | Container logs (`?raw=1` for plain text) |
| GET | `/api/stacks/{name}/hdd-data` | HDD data paths + sizes |
| GET | `/api/stacks/{name}/backup-data` | Backup data paths + sizes (DB dumps, cross-drive rsync) |
| POST | `/api/stacks/{name}/remove` | Remove deployed stack (revert to "not deployed") |
| DELETE | `/api/stacks/{name}` | Delete orphaned stack |
| POST | `/api/sync` | Trigger catalog sync |
| GET | `/api/system/info` | System info + sync status |

### Backup & Restore

| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/api/backup/status` | Full backup status |
| POST | `/api/backup/run` | Trigger manual backup |
| GET | `/api/backup/snapshots` | List snapshots (`?stack={name}` for filtering) |
| POST | `/api/stacks/{name}/cross-backup` | Save cross-drive config |
| POST | `/api/stacks/{name}/cross-backup/run` | Trigger cross-drive backup |
| GET | `/api/stacks/{name}/cross-backup/status` | Cross-drive status |
| POST | `/api/backup/cross-drive/run-all` | Run all scheduled cross-drive backups |

### Storage

| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/api/storage/scan` | Scan available disks |
| POST | `/api/storage/init` | Format and mount a disk |
| GET | `/api/storage/init/status` | Format progress |
| POST | `/api/storage/attach/mount-raw` | Temp-mount partition for browsing |
| GET | `/api/storage/attach/browse?path=` | List directories on raw mount |
| POST | `/api/storage/attach/mkdir` | Create folder on raw mount |
| POST | `/api/storage/attach` | Finalize attach (bind mount + fstab) |
| GET | `/api/storage/attach/status` | Attach progress |
| POST | `/api/storage/attach/cancel` | Cleanup temp raw mount |
| POST | `/api/storage/migrate` | Start app data migration |
| GET | `/api/storage/migrate/status` | Migration progress |
| POST | `/api/storage/disconnect` | Safe disconnect (stop apps, unmount) |
| POST | `/api/storage/reconnect` | Reconnect disconnected drive |
| POST | `/api/storage/restart-apps` | Restart auto-stopped apps |
| GET | `/api/storage/status` | All storage paths with connection state |

### Self-Update

| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/api/selfupdate/status` | Update status (cached check result + last state) |
| POST | `/api/selfupdate/check` | Force registry check |
| POST | `/api/selfupdate/update` | Trigger self-update (async) |

Self-update endpoints accept session auth OR `Authorization: Bearer <hub_api_key>` for external triggering.

### Config Management

| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/api/config/apply` | Apply new controller.yaml from Hub (atomic write) |
| GET | `/api/config/hash` | Get SHA256 hash of current controller.yaml |
| GET | `/api/config` | Get raw controller.yaml content (text/yaml) for live diff and pull |

Config endpoints accept session auth OR `Authorization: Bearer <hub_api_key>` (same as self-update). The `/api/config/apply` endpoint:
- Accepts raw YAML body (the generated config from Hub)
- Validates YAML is parseable before writing
- Atomic write: writes to `.tmp` then `os.Rename` for crash safety
- Does NOT reload config — restart required to apply changes
- Returns `{"ok": true, "message": "Config applied. Restart controller to apply changes."}`

### Metrics

| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/api/metrics/system` | System metrics time-series (`?range=1h|6h|24h|7d|30d`) |
| GET | `/api/metrics/containers/summary` | Current container stats |
| GET | `/api/metrics/containers/{name}` | Per-container time-series |
| GET | `/api/metrics/sysinfo` | Static system info |

### Assets

| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/api/assets/sync` | Trigger on-demand asset sync from Hub (async) |
| GET | `/api/assets/status` | Asset sync status (last sync, file count, total bytes) |

### Integrations

| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/api/integrations/{provider}` | List integrations for provider app (status, target availability) |
| POST | `/api/integrations/{provider}/{target}` | Enable/disable integration (`{"enabled": true/false}`) |

### Debug (debug mode only)

| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/api/debug/dump` | Full diagnostic JSON dump (controller state, storage, stacks, backup, hub, scheduler, health, alerts). Returns 404 when `logging.level` is not `"debug"`. |
| GET | `/api/debug/telemetry` | Run telemetry collection on-demand; returns per-app metrics + log summary with latency. Response: `{latency_ms, app_count, total_errors, total_warnings, app_telemetry[]}`. |

Response format: `{"ok": true/false, "data": ..., "error": "...", "message": "..."}`

---

## App-email relay (`internal/mailrelay/`)

Gives deployed apps outbound email (password resets, invites, confirmations) through one managed path —
**app → in-process SMTP shim → hub → Resend** — with the Resend key staying **hub-side** (never on the box).
Architecture **Shape 1**: the shim runs in-process inside the controller, reusing the existing hub client.
Implements `felhom.eu/documentation/audits/SPIKE-smtp-app-relay-2026-06-28.md`.

- **The shim (`internal/mailrelay/`)** is a `go-smtp` server with two listeners — `:2525` plaintext+STARTTLS
  and `:2465` implicit-TLS (self-signed cert at boot). It advertises AUTH PLAIN+LOGIN and **accepts any
  credentials, ignoring them** (apps send none; some require the offer). `Data` reads the raw message, enforces
  the **From-header** domain allowlist (reject `5xx` **before** any hub call), then forwards the **raw MIME** to
  the hub `POST /api/v1/mail` with the controller's hub Bearer key — **single-shot** (no retry, no spool in v1).
  The hub HTTP status maps to an SMTP reply (2xx→250, 4xx→451, 5xx→554). Listeners bind to the app Docker
  network only (the controller container joins `traefik-public`); apps reach the shim by `felhom-controller`.
- **Lifecycle** (`lifecycle.go`): the shim starts/stops at runtime to match the global app-email toggle (no
  controller restart). Wired in `main.go`, gated on a configured hub + the `mail_relay` kill-switch.
- **Toggles + injection:** a **global** toggle (`settings.AppEmail`, Settings page) and a **per-app** toggle
  (`AppConfig.EmailEnabled`, on the app's config page, shown only for apps with an `smtp_mapping`). When both are
  on and the app declares `.felhom.yml` **`smtp_mapping`**, `stackEnv` injects the relay env at compose time
  (host=shim, port=2525, security/from per the mapping, From=`<app>@felhom.eu`) — derived each compose, never
  persisted to `app.yaml`. Config knobs: `mail_relay` (listeners, `shim_host`, `from_domains`, kill-switch).
- **What the box never holds:** the Resend key, or any durable mail queue. v2 (deferred) = a separate
  `felhom-smtp-shim` container + accept-and-spool retry + a `Resend-Idempotency-Key`.

---

## Build & Deploy

### Build

```bash
# On build server (192.168.0.180)
cd ~/build/felhom-controller
git -C ~/git/felhom-controller pull
./build.sh v0.20.0 --push
```

### Deploy on customer node

**Option A: Self-Update API (v0.16.0+)**

After building and pushing the new image, trigger the controller's self-update endpoint:

```bash
curl -s -X POST https://felhom.demo-felhom.eu/api/selfupdate/update \
  -H "Authorization: Bearer <HUB_API_KEY>"
```

The controller pulls the new image, updates its own compose file, and runs `docker compose up -d` to replace itself. The Settings page also has a "Frissítés telepítése" button for manual triggering.

**Option B: Manual SSH (pre-v0.16.0 or fallback)**

```bash
# On customer node (e.g., 192.168.0.162)
cd /opt/docker/felhom-controller
sudo docker pull gitea.dooplex.hu/admin/felhom-controller:<VERSION>
sudo sed -i 's|image: gitea.dooplex.hu/admin/felhom-controller:.*|image: gitea.dooplex.hu/admin/felhom-controller:<VERSION>|' docker-compose.yml
sudo docker compose up -d
```

**Important:** Always use `docker compose up -d`, NOT `docker compose restart` — restart doesn't pick up new images.

### Docker Requirements

The controller container needs:
- `privileged: true` (disk operations)
- Docker socket mount (`/var/run/docker.sock`)
- `/mnt` mount with `propagation: rshared` (container mounts visible to host)
- `/dev` mounted as `/host-dev` (block device access)
- `/etc/fstab` mounted as `/host-fstab` (persistent mount config)

See `docker-compose.yml` for the full volume configuration.

---

## Roadmap

### Completed

- [x] Stack management with deploy flow and memory validation
- [x] Git-based app catalog sync
- [x] Central job scheduler
- [x] System monitoring with SQLite metrics and Chart.js charts
- [x] Healthchecks.io integration (5 ping types)
- [x] 3-layer backup system (DB dumps + restic + cross-drive)
- [x] Per-app backup restore with auto stop/restart
- [x] Storage management (scan, format, mount, registry)
- [x] Attach existing drive wizard (v0.15.0) — bind-mount subfolder from pre-formatted drive, directory browser
- [x] App data migration between storage paths
- [x] Storage watchdog (v0.17.0) — USB disconnect detection (~15s), auto-stop apps, auto-remount on reconnect, safe eject UI
- [x] Central hub reporting
- [x] Email notifications via hub relay
- [x] Settings persistence and password management
- [x] Dashboard alert system
- [x] Per-drive backup architecture (v0.14.0) — per-drive restic repos, per-app DB dumps, path helpers
- [x] Cross-drive restic pruning (v0.14.0)
- [x] Auto Tier 2 for small apps (v0.14.1) — auto-enable daily rsync for non-HDD apps when ≥2 drives
- [x] Infrastructure config in cross-drive backup (v0.14.1) — stacks dir + controller.yaml in `_infra/` + restic
- [x] Disaster recovery (v0.15.5) — Hub-based infra backup, auto-mount by UUID, restore UI with full-page takeover
- [x] Controller self-update (v0.16.0) — Watchtower-style pull + restart, Settings page UI, API key auth, auto-update scheduling
- [x] Hub-managed config (v0.20.0) — Config apply endpoint (`POST /api/config/apply`), config hash in reports for sync comparison
- [x] Config content endpoint (v0.21.1) — `GET /api/config` returns raw YAML for Hub live diff and pull operations
- [x] First-run setup wizard (v0.22.0) — Web-based wizard replaces shell scripts, drive scan for local backups, Hub recovery, fresh install flow
- [x] Setup wizard logo fix (v0.22.2) — Use embedded SVG instead of filesystem path
- [x] Hub-managed asset sync (v0.22.3) — Download app logos/screenshots from Hub API with SHA-256 change detection, daily sync schedule

### In Progress / Planned

- [ ] Update classification and auto-apply (optional/required/security markers)
- [x] Docker volume backup + Tier 2 restore (v0.33.0)
- [ ] Raspberry Pi testing (pi-customer-1)
- [x] CSRF protection on POST endpoints (v0.23.0)
- [x] Verbose debug logging across all modules (v0.24.0)
- [x] Diagnostic dump endpoint `/api/debug/dump` (v0.24.0)
- [x] Startup self-test with 9 subsystem checks (v0.24.0)
- [ ] Login rate limiting

---

## Test Environments

| Node | Hardware | Domain | Status |
|------|----------|--------|--------|
| demo-felhom | Acemagic GK3PLUS N100, 16G RAM, 512G SSD + 1TB HDD | demo-felhom.eu | Active |
| felhotest | Proxmox VM (4-16G RAM, 8 vCPU, 200G + 100G SCSI) | router.abonet.hu:33022 | Active |
| pi-customer-1 | Raspberry Pi 3B+, 1G RAM, 32G SD | pi-customer-1.local | Not yet tested |

## Related Repositories

| Repository | Purpose |
|------------|---------|
| [felhom-controller](https://gitea.dooplex.hu/admin/felhom-controller) | This repo — controller + deploy scripts |
| [app-catalog-felhom.eu](https://gitea.dooplex.hu/admin/app-catalog-felhom.eu) | Docker Compose templates + .felhom.yml metadata |
| [felhom.eu](https://gitea.dooplex.hu/admin/felhom.eu) | Website + app assets + felhom-hub service |
