# felhom-controller

**Central management container for Felhom home servers.**

Replaces Portainer + scattered systemd scripts with a single, lightweight container that provides:
- Hungarian-language web dashboard for customers
- Docker Compose stack management (start/stop/update)
- Interactive first-deployment flow with auto-generated secrets
- Health-aware container state monitoring (starting/unhealthy/running)
- Backup orchestration (DB dumps + restic snapshots) — Phase 3
- System health monitoring with Healthchecks pings — Phase 2
- Git-based stack synchronization with update management — Phase 4
- Self-update with automatic rollback on failure — Phase 5

## Current Status

**Phase 1 — Stack Manager + Deploy Flow: ✅ COMPLETE**

The controller is built, deployed, and running on the N100 test node (demo-felhom.eu).
First application (Paperless-ngx) successfully deployed end-to-end through the dashboard on 2026-02-13.

**Milestone achieved:** Full deploy cycle works — customer clicks "Telepítés", fills in fields,
controller generates secrets, saves app.yaml, runs `docker compose up -d`, and the app comes up
with Traefik routing and health checks. The dashboard correctly shows real-time container states
including health substatus (starting → healthy → running).

Current version: **v0.7.2**

### What works
- Dashboard with live container state (green/orange/yellow/red)
- Deploy form with password validation, auto-generation, and field locking
- Stack operations: start, stop, restart, update (pull + recreate)
- Live-scrolling log viewer with auto-refresh (3s polling), pause/resume, and scroll position tracking
- Deploy page doubles as config viewer (read-only mode for deployed apps)
- App detail/info pages with use cases, setup guide, screenshots, and optional config
- Optional config saves to `app.yaml` and restarts deployed apps (e.g., metadata provider API keys)
- Periodic stack rescanning (every 2 minutes)
- Manual rescan endpoint (`POST /api/stacks/rescan`)
- Alphabetically sorted stack display (consistent card ordering)
- Protected stacks (traefik, cloudflared, felhom-controller) can't be stopped
- System info bar on dashboard: RAM, SSD, and HDD usage with progress bars
- Docker Compose memory limits enforced via `deploy.resources.limits.memory`
- Pre-deploy memory validation (hard block on `mem_request` overcommit, soft warning on `mem_limit` overcommit)
- Memory summary bar shown on deploy page before deployment
- Felhom.eu logo SVG in sidebar and login page
- Verbose debug logging with operation timing, post-start container state checks, and image pull detection
- Clickable app cards on dashboard and applications pages (navigate to info page)
- Memory bar with two-segment visualization on deploy page (committed vs new app allocation)
- Deployment progress UI: 3-step progress panel with real-time health polling (config → containers → health check)
- CPU usage bar with load average display (1/5/15 min)
- Temperature display with colored indicator dot (thermal zone reading)
- Central job scheduler replacing ad-hoc goroutines (periodic + daily jobs)
- Healthchecks.io-compatible system health pings with retry logic
- Database auto-discovery and dump (PostgreSQL/MariaDB via docker exec)
- Restic backup with auto-password generation, snapshot, prune, stats
- Backup status card on dashboard with manual "Mentés most" trigger button
- Backup API endpoints: status query and manual trigger
- SQLite metrics store (system + container metrics, 60s collection, 30-day retention)
- Heartbeat ping (5-minute "I'm alive" signal to Healthchecks)
- Weekly backup integrity check (restic check, Sunday 04:00)
- Central hub reporting (periodic JSON push to felhom-hub service)
- Notification preferences sync to hub (controller → hub on save + startup)
- Notification system with email delivery via Resend API (hub relay)

### Known issues / next priorities
- Cloudflare Tunnel + Traefik TLS: paperless.demo-felhom.eu works locally but shows "Not secure" (certificate chain not fully validated through tunnel)
- No undo/delete for deployed apps yet
- Dashboard theme doesn't yet match felhom.eu dark theme

## Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│  Customer Hardware (N100 mini PC / Raspberry Pi)                │
│                                                                 │
│  ┌──────────┐   ┌────────────────────────────────────────────┐  │
│  │ Traefik  │   │  felhom-controller                         │  │
│  │ (reverse │──▶│                                            │  │
│  │  proxy)  │   │  ┌──────────┐  ┌─────────────────────────┐│  │
│  └──────────┘   │  │ Web UI   │  │ Stack Manager           ││  │
│                 │  │ (HU dash │  │ (compose up/down/pull,  ││  │
│  ┌──────────┐   │  │  board)  │  │  git sync, update mgmt) ││  │
│  │cloudflared│   │  └──────────┘  └─────────────────────────┘│  │
│  │ (tunnel) │   │  ┌──────────┐  ┌─────────────────────────┐│  │
│  └──────────┘   │  │ Backup   │  │ Monitor & Pinger        ││  │
│                 │  │ (db dump │  │ (healthchecks pings,    ││  │
│  ┌──────────┐   │  │  restic) │  │  system metrics)        ││  │
│  │ App      │   │  └──────────┘  └─────────────────────────┘│  │
│  │ stacks   │   │  ┌──────────┐  ┌─────────────────────────┐│  │
│  │ (docker  │   │  │Scheduler │  │ REST API                ││  │
│  │ compose) │   │  │(cron-like│  │ (for UI + remote mgmt)  ││  │
│  └──────────┘   │  │  jobs)   │  └─────────────────────────┘│  │
│                 │  └──────────┘                              │  │
│                 └────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘
         │ pings              │ JSON push          │ git pull
         ▼                    ▼                    ▼
  status.felhom.eu      hub.felhom.eu       gitea.dooplex.hu
  (Healthchecks)        (central dashboard)  (stack definitions)
```

## Repository Layout

This is the `controller/` subfolder of the `deploy-felhom-compose` repository.

```
controller/
├── cmd/controller/main.go           # Entry point, wires all modules
├── internal/
│   ├── config/config.go             # YAML loader, validation, env overrides
│   ├── stacks/
│   │   ├── manager.go               # Stack scanning, compose ops, container status, debug logging
│   │   ├── metadata.go              # Parse .felhom.yml app metadata
│   │   └── deploy.go                # First-deploy flow: secret gen, app.yaml, compose up
│   ├── sync/
│   │   └── sync.go                  # Git sync: clone/pull app catalog, content-hash copy
│   ├── api/router.go                # REST API endpoints
│   ├── scheduler/
│   │   └── scheduler.go             # Central job scheduler (Every, Daily, skip-if-running)
│   ├── system/
│   │   ├── info.go                  # SystemInfo struct
│   │   ├── info_linux.go            # Linux: /proc/meminfo + statfs + loadavg + temperature
│   │   ├── info_other.go            # Non-Linux stub
│   │   ├── cpu_linux.go             # CPU collector (background /proc/stat sampling)
│   │   └── cpu_other.go             # CPU collector stub (non-Linux)
│   ├── monitor/
│   │   ├── pinger.go                # Healthchecks.io HTTP ping client
│   │   └── healthcheck.go           # System health checks (disk, mem, CPU, temp, Docker)
│   ├── backup/
│   │   ├── backup.go                # Backup orchestrator (DB dumps + restic + prune + integrity)
│   │   ├── dbdump.go                # Database auto-discovery + dump (pg_dump, mariadb-dump)
│   │   └── restic.go                # Restic operations (init, snapshot, prune, check, stats)
│   ├── metrics/
│   │   ├── store.go                 # SQLite metrics storage (system + container, downsampled queries)
│   │   ├── collector.go             # Background collector (60s interval, system + docker stats)
│   │   ├── types.go                 # SystemSample, ContainerSample, StaticSystemInfo structs
│   │   └── sysinfo.go              # Host-level static info (/proc, /etc)
│   ├── notify/
│   │   └── notifier.go              # Notification relay (hub → email), preferences sync, cooldowns
│   ├── report/
│   │   ├── types.go                 # Hub report JSON payload definitions
│   │   ├── builder.go               # Builds report from system/stacks/backup/metrics state
│   │   └── pusher.go                # HTTP POST to central hub (retry, Bearer auth)
│   └── web/
│       ├── server.go                # HTTP server, routing, static file serving
│       ├── auth.go                  # Session auth, login/logout handlers
│       ├── handlers.go              # Page handlers (dashboard, stacks, deploy, etc.)
│       ├── funcmap.go               # Template function map (state colors, formatting)
│       ├── embed.go                 # go:embed directive for templates
│       ├── templates.go             # Felhom logo SVG constant
│       └── templates/               # go:embed HTML/CSS files (Hungarian UI)
│           ├── layout.html, dashboard.html, stacks.html, login.html
│           ├── logs.html, deploy.html, app_info.html
│           └── style.css
├── configs/
│   ├── controller.yaml.example      # Full config reference (infrastructure only)
│   └── example-felhom-metadata.yml  # .felhom.yml format reference
├── scripts/hashpass.go              # Bcrypt password hash generator
├── assets/                          # App logos + screenshots (gitignored, synced at build)
├── docs/BUILDING.md                 # Container image build & registry guide
├── Dockerfile                       # Multi-stage build (Go 1.22 + debian-slim)
├── docker-compose.yml               # Controller's own compose definition
├── Makefile                         # Build targets + asset sync
└── go.mod
```

## Module Overview

| Module | Path | Status | Responsibility |
|--------|------|--------|----------------|
| **Config** | `internal/config/` | ✅ Done | Load & validate controller.yaml, env overrides |
| **Stacks** | `internal/stacks/` | ✅ Done | Compose operations, scanning, metadata, deploy flow |
| **API** | `internal/api/` | ✅ Done | REST endpoints (stacks, deploy, rescan, system info, health) |
| **System** | `internal/system/` | ✅ Done | System resource info (RAM, disk, CPU, temperature, load) |
| **Web** | `internal/web/` | ✅ Done | Hungarian dashboard, auth, deploy pages, asset serving |
| **Sync** | `internal/sync/` | ✅ Done | Git-based app catalog sync (clone/pull, content-hash copy) |
| **Scheduler** | `internal/scheduler/` | ✅ Done | Central job scheduler (periodic + daily, skip-if-running) |
| **Monitor** | `internal/monitor/` | ✅ Done | Healthchecks.io pings, system health checks |
| **Metrics** | `internal/metrics/` | ✅ Done | SQLite time-series store, system + container collection |
| **Notify** | `internal/notify/` | ✅ Done | Notification relay to hub, preferences sync, cooldown tracking |
| **Report** | `internal/report/` | ✅ Done | Central hub push (JSON report builder + HTTP pusher) |
| **Backup** | `internal/backup/` | ✅ Done | DB auto-discovery + dump, restic snapshots, prune, manual trigger |

## Stack Management

### How stacks get onto the machine

1. During initial setup, `deploy-felhom-compose.sh` clones the app catalog
2. Compose files + `.felhom.yml` metadata land in `/opt/docker/stacks/<app>/`
3. The controller periodically pulls from Git to detect changes (Phase 4)

### First deployment flow (via dashboard)

1. Customer sees app card with "🚀 Telepítés" (Deploy) button
2. Clicks → deploy page shows:
   - **Auto-filled**: DOMAIN (from controller config), read-only
   - **Auto-generated**: DB passwords, secret keys (shown as "✓ Generated")
   - **User input**: HDD path, admin password, language, etc.
   - **"🎲 Generálás"** button next to password fields
3. Clicks "Telepítés" → `checkBeforeDeploy()` JS guard fetches live state from API first (prevents deploying if already deployed from another tab). Then controller:
   - **Memory validation**: checks `mem_request` against available system RAM (see below)
   - Validates all required fields (password fields must be explicitly filled or generated)
   - Generates auto-secrets (DB passwords, hex keys)
   - Saves `app.yaml` (env vars + locked fields list)
   - **Updates in-memory state immediately** (so UI shows "deployed" during slow compose ops)
   - Runs `docker compose up -d` with env vars injected
   - On failure: reverts both in-memory state and disk (app.yaml `deployed: false`)
4. **Progress UI** replaces the form with a 3-step progress panel:
   - ✅ "Konfiguráció mentve" — shown immediately after API success
   - ⏳ "Konténer(ek) indítása..." → ✅ when containers are up
   - ⏳ "Alkalmazás inicializálása..." → ✅ when state = `running` (healthy)
   - Polls `GET /api/stacks/{name}` every 3 seconds for real-time health status
   - Handles: `running` (auto-redirect), `starting` (keep polling), `unhealthy` (warning), `exited` (error)
   - Timeout after 120 seconds with informational message
5. Post-deploy: locked fields (DB_PASSWORD, etc.) become read-only
6. "Részletek" button opens deploy page in read-only mode showing current config

### Memory validation during deploy

Before deploying an app, the controller checks if there's enough RAM. This uses the Kubernetes-inspired `mem_request` / `mem_limit` model:

| Field | In `.felhom.yml` | Purpose | Validation |
|-------|-------------------|---------|------------|
| `mem_request` | `resources.mem_request: "500M"` | Expected memory usage during normal operation | **Hard block** — sum of requests must not exceed usable RAM |
| `mem_limit` | `resources.mem_limit: "1152M"` | Docker `deploy.resources.limits.memory` total across all containers | **Soft warning** — overcommit is allowed for limits |
| `pi_compatible` | `resources.pi_compatible: true` | Whether the app can run on Raspberry Pi | Display-only hint |
| `needs_hdd` | `resources.needs_hdd: true` | Whether the app needs external storage | Display-only hint |

**How it works:**
- `usable_memory = total_ram - reserved_memory_mb` (default: 384MB reserved for OS + controller)
- If `sum(deployed mem_requests) + new_mem_request > usable_memory` → deploy is **blocked** with error
- If `sum(deployed mem_limits) + new_mem_limit > total_ram` → deploy proceeds with **warning**
- Apps without `mem_request` set are treated as 0MB (never blocked)
- The deploy page shows a memory summary bar before the user clicks deploy

Configure the reserved memory via `controller.yaml`:
```yaml
system:
  reserved_memory_mb: 384  # default
```

### Container state display

The dashboard shows health-aware container states with distinct colors:

| State | Color | Label | Meaning |
|-------|-------|-------|---------|
| Running + healthy | 🟢 Green | "Fut" | All containers running and healthy |
| Running + health: starting | 🟠 Orange | "Indulás..." | Container up but healthcheck not yet passed |
| Running + unhealthy | 🟡 Yellow | "Nem egészséges" | Container running but healthcheck failing |
| Stopped/exited | 🔴 Red | "Leállítva" | All containers stopped |
| Restarting | 🟡 Yellow | "Újraindítás..." | Container in restart loop |
| Not deployed | ⚪ Gray | "Nincs telepítve" | Compose file exists but not yet deployed |

Action buttons adapt: "operational" states (running/starting/unhealthy/restarting) show restart/stop,
while stopped states show a start button.

### Update strategy (Phase 4)

Stack updates are classified in the Git repository via markers:

| Marker | Behavior |
|--------|----------|
| No marker | Optional update — shown on dashboard, customer clicks "Update" |
| `UPDATE_REQUIRED=true` | Mandatory — auto-applied during next update window |
| `UPDATE_SECURITY=true` | Critical — applied immediately (within minutes) |

### Protected stacks

The following stacks cannot be stopped from the customer UI:
- `traefik` (reverse proxy)
- `cloudflared` (tunnel)
- `felhom-controller` (this container)

## Logging

The controller uses two-tier logging controlled by `logging.level` in `controller.yaml`:

```yaml
logging:
  level: debug    # debug | info | warn | error (default: info)
  file: ""        # optional log file path
  max_size_mb: 10
  max_files: 3
```

Can also be set via environment variable: `FELHOM_LOGGING_LEVEL=debug`

### What gets logged at each level

| Level | What's logged |
|-------|--------------|
| **info** | Operation success/failure with elapsed time, post-start container states, stack scan counts, deploy memory checks |
| **debug** | All of above + env var keys per compose command, local image availability checks, compose command timing, log fetch byte counts |

### Example output (debug level)

```
[INFO] Starting stack: immich
[DEBUG] Env vars for compose: [DOMAIN, DB_PASSWORD, HDD_PATH] (3 app + 42 system)
[DEBUG] Running: docker compose up -d (in /opt/docker/stacks/immich)
[DEBUG] Command completed: docker compose up -d (took 12.3s)
[INFO] Stack immich started successfully (took 12.3s)
[INFO] Stack immich post-start status:
[INFO]   immich-server   ghcr.io/immich-app/immich-server:release   running   Up 3 seconds (health: starting)
[INFO]   immich-postgres docker.io/tensorchord/pgvecto-rs:pg16...   running   Up 3 seconds (healthy)
[INFO]   immich-redis    docker.io/library/redis:7-alpine            running   Up 3 seconds (healthy)
```

On failure:
```
[ERROR] Command failed: docker compose up -d (in /opt/docker/stacks/immich) — exit code 1 (took 2.1s)
[ERROR] stderr: Error response from daemon: pull access denied...
[ERROR] Stack immich start failed after 2.1s: exit code 1
```

### Security

- Env var **values** are never logged — only keys appear in debug output
- stdout/stderr in error logs are truncated to 500 characters to prevent log spam

## Configuration

### Controller config (infrastructure only)

Single YAML file per customer: `/opt/docker/felhom-controller/controller.yaml`

Contains customer identity, infrastructure secrets, backup/monitoring settings.
Does **not** contain app-specific config (HDD paths, DB passwords, etc.).

See `configs/controller.yaml.example` for the full reference.

### Per-app config (created during deployment)

Each deployed app gets an `app.yaml` in its stack directory:

```yaml
# /opt/docker/stacks/paperless-ngx/app.yaml
# Auto-generated by felhom-controller — do not edit locked fields manually
deployed: true
deployed_at: "2026-02-13T21:10:00Z"
env:
  DOMAIN: "demo-felhom.eu"
  DB_PASSWORD: "a7f2b9c1e4d..."       # locked
  PAPERLESS_SECRET_KEY: "8b3e..."      # locked
  PAPERLESS_ADMIN_USER: "admin"        # editable
  PAPERLESS_OCR_LANGUAGE: "hun+eng"    # editable
  HDD_PATH: "/mnt/hdd_placeholder"    # locked
locked_fields:
  - DB_PASSWORD
  - PAPERLESS_SECRET_KEY
  - DOMAIN
  - HDD_PATH
```

### App assets (logos, screenshots)

Baked into the container image at build time — no external dependencies at runtime.
Synced from the felhom.eu website repo before building.

Served locally at `/static/assets/`. Logos try SVG first, fall back to PNG.

## Build & Deploy

Source: `https://gitea.dooplex.hu/admin/deploy-felhom-compose` → `controller/` subfolder.

Build happens outside the repo in `~/build/felhom-controller/` to keep the repo clean.
See `docs/BUILDING.md` for the full guide.

```bash
# Quick build (current platform only)
cd ~/build/felhom-controller
./build.sh 0.2.11

# Build + push to Gitea registry
./build.sh 0.2.11 --push
```

### Deploy on customer node

```bash
# Pull new image
docker pull gitea.dooplex.hu/admin/felhom-controller:0.2.11

# IMPORTANT: use 'up -d', NOT 'restart' — restart doesn't pick up new images
cd /opt/docker/felhom-controller
docker compose up -d
```

## Test Environments

| Node | Hardware | Domain | IP | Status |
|------|----------|--------|----|--------|
| demo-felhom | Acemagic GK3PLUS N100, 16G RAM, 512G SSD + 1TB HDD | demo-felhom.eu | 192.168.0.162 | ✅ Controller v0.6.0 + Paperless-ngx running |
| pi-customer-1 | Raspberry Pi 3B+, 1G RAM, 32G SD | pi-customer-1.local | — | 📲 Not yet tested |

### First deployment log (Paperless-ngx on demo-felhom)

- **Date:** 2026-02-13
- **App:** Paperless-ngx (document management)
- **Deploy method:** Dashboard UI → "Telepítés" button
- **Issues encountered & resolved:**
  1. Password fields accepted empty values → Added server-side + client-side validation
  2. "Telepítés" button appeared for already-deployed apps → Fixed in-memory Deployed flag update
  3. Green status shown for `(health: starting)` containers → Added health-aware state parsing
  4. Stack cards switched positions on refresh → Added alphabetical sorting in GetStacks()
  5. "Részletek" button did nothing for deployed apps → Redirects to deploy page (read-only)
  6. OCR crash: `PAPERLESS_OCR_LANGUAGE=hun` not installed → Added `PAPERLESS_OCR_LANGUAGES` (plural) to docker-compose
  7. Container restart vs recreate: `docker compose restart` doesn't pick up new images → Documented: always use `docker compose up -d`

## REST API

| Method | Endpoint | Auth | Description |
|--------|----------|------|-------------|
| GET | `/api/health` | No | Health check (for monitoring) |
| GET | `/api/stacks` | Yes | List all stacks |
| GET | `/api/stacks/{name}` | Yes | Stack details |
| GET | `/api/stacks/{name}/deploy-fields` | Yes | Get deploy form fields |
| POST | `/api/stacks/{name}/deploy` | Yes | First-time deploy with config |
| POST | `/api/stacks/{name}/start` | Yes | Start stack |
| POST | `/api/stacks/{name}/stop` | Yes | Stop stack (not protected) |
| POST | `/api/stacks/{name}/restart` | Yes | Restart stack |
| POST | `/api/stacks/{name}/update` | Yes | Pull images + recreate |
| POST | `/api/stacks/{name}/optional-config` | Yes | Update optional config env vars |
| GET | `/api/stacks/{name}/logs` | Yes | Container logs (add `?raw=1` for plain text) |
| POST | `/api/stacks/rescan` | Yes | Trigger manual stack discovery |
| GET | `/api/system/info` | Yes | System resource usage (RAM, disk, CPU, temp, load) |
| GET | `/api/backup/status` | Yes | Backup status (last run, DB dump count, repo stats) |
| POST | `/api/backup/run` | Yes | Trigger manual backup (DB dumps + restic snapshot) |

## Status & Roadmap

### Phase 1 — Stack Manager + Deploy Flow ✅ COMPLETE
- [x] Project skeleton & config format
- [x] .felhom.yml app metadata format with deploy fields
- [x] Per-app config persistence (app.yaml)
- [x] Secret generation engine (password, hex, static)
- [x] Stack catalog (read compose files + metadata from disk)
- [x] Docker Compose operations (up/down/pull/ps/logs)
- [x] Deploy flow with interactive field input
- [x] Password validation (server-side + client-side, no empty passwords)
- [x] Basic web dashboard with start/stop/deploy buttons
- [x] Health-aware container states (starting/unhealthy/running)
- [x] REST API for stack + deploy operations
- [x] Simple web authentication (bcrypt sessions)
- [x] App assets baked into container (SVG/PNG logos, webp screenshots)
- [x] Container image build pipeline (Dockerfile + build.sh)
- [x] Build + push to Gitea container registry
- [x] Deploy on N100 test node — dashboard accessible
- [x] Stack scanning + display working
- [x] **First app deployed: Paperless-ngx via dashboard** (2026-02-13)
- [x] Periodic stack rescanning (every 2 minutes)
- [x] Alphabetically sorted stack display
- [x] Deploy page doubles as read-only config viewer for deployed apps

### Phase 2 — Monitoring & Health ✅ COMPLETE
- [x] System metrics on dashboard (RAM, SSD, HDD usage bars)
- [x] `/api/system/info` endpoint with live resource data
- [x] Pre-deploy memory validation (mem_request hard block, mem_limit soft warning)
- [x] Memory summary bar on deploy page
- [x] CPU usage collector (background /proc/stat sampling, 5s interval)
- [x] CPU usage bar on dashboard with load average display
- [x] Temperature reading from /sys/class/thermal (with /host/sys Docker mount)
- [x] Temperature display with colored indicator dot (green/yellow/red)
- [x] Central job scheduler (replaces ad-hoc goroutines)
- [x] Healthchecks.io-compatible HTTP pinger with retry logic
- [x] System health checks (disk, memory, CPU, temp, Docker, protected containers)
- [x] Heartbeat ping (5-minute "I'm alive" signal)
- [x] SQLite metrics store (system + container metrics, 60s collection, 30-day prune)
- [x] Backup integrity check (weekly restic check with Healthchecks ping)
- [x] Customer notifications (email via hub relay + Resend API)
- [x] Notification preferences sync (controller → hub on save + startup)

### Phase 3 — Backups ✅ COMPLETE
- [x] DB auto-discovery (PostgreSQL/MariaDB containers via docker inspect)
- [x] DB dump engine (pg_dump/mariadb-dump via docker exec, atomic writes)
- [x] Restic integration (auto-init, snapshot, prune, check, stats)
- [x] Restic password auto-generation (no manual setup needed)
- [x] Backup orchestrator (DB dumps + restic + weekly prune)
- [x] Backup status on dashboard (last run, DB count, repo stats)
- [x] Manual backup trigger from UI ("Mentés most" button)
- [x] `GET /api/backup/status` and `POST /api/backup/run` endpoints
- [ ] Restore workflow

### Phase 4 — Git Sync & Updates
- [x] Periodic git pull for stack definitions (git sync module)
- [x] Manual sync button on Alkalmazások page ("Sablonok frissítése")
- [x] Sync status in `/api/system/info`
- [ ] Update classification (optional/required/security)
- [ ] Update window enforcement
- [ ] Dashboard update notifications with "Update" button

### Phase 5 — Self-Update & Resilience
- [ ] Self-update check & execution
- [ ] Pre-update config backup
- [ ] Health-based rollback mechanism
- [ ] Config export/import

### Phase 6 — Central Management (in progress)
- [x] Central hub reporting (controller → hub JSON push with Bearer auth)
- [x] Hub report builder (system, stacks, backup, health, containers, metrics)
- [x] Hub service (felhom-hub: REST API + SQLite + dark-theme dashboard)
- [x] K8s manifests for hub deployment on k3s
- [ ] Fleet-wide update management
- [x] Customer notifications (email via hub relay)
- [x] Notification preferences sync (controller → hub)
- [x] Hub customer detail page: notification preferences + log display

## Related Repositories

| Repository | Purpose |
|------------|---------|
| [deploy-felhom-compose](https://gitea.dooplex.hu/admin/deploy-felhom-compose) | This repo — controller + deploy scripts |
| [app-catalog-felhom.eu](https://gitea.dooplex.hu/admin/app-catalog-felhom.eu) | Docker Compose templates + .felhom.yml metadata |
| [felhom.eu](https://gitea.dooplex.hu/admin/felhom.eu) | Website + app assets + felhom infra manifests (incl. felhom-hub) |
| [homelab-manifests](https://gitea.dooplex.hu/admin/homelab-manifests) | k3s cluster manifests (dooplex.hu) |