3494 lines
272 KiB
Markdown
3494 lines
272 KiB
Markdown
# felhom-controller
|
||
|
||
**Central management container for Felhom home servers.** The **in-guest controller**: one per
|
||
customer LXC, Docker-only, Hungarian web dashboard for managing the customer's app stacks, app-data
|
||
backups, monitoring and notifications. All Proxmox/disk operations are delegated to the host agent
|
||
(`felhom-agent`).
|
||
|
||
**Current version: v0.59.0**
|
||
|
||
> ## Documentation has moved (and is now code-verified)
|
||
>
|
||
> The **authoritative architecture & feature documentation** now lives in the central docs home:
|
||
> **[`felhom.eu/documentation/controller/`](../../felhom.eu/documentation/controller/README.md)** —
|
||
> module map, deploy & stack lifecycle, backup architecture, storage/monitoring/metrics, and
|
||
> auth/hub/sync/integrations. Those docs are grounded in current source (v0.59.0).
|
||
>
|
||
> **Quick build & deploy** is in the "Build & Deploy" section below and in the repo `CLAUDE.md`
|
||
> (authoritative for the workflow). NOTE the demo controller runs in an LXC guest (9201) under the
|
||
> **bootstrap-managed** mechanism (`/etc/felhom-controller-image` + `felhom-controller-bootstrap.service`),
|
||
> not the bare-metal `/opt/docker` compose path some sections below still describe.
|
||
>
|
||
> The prose below this banner is **retained legacy reference** and may lag the central docs — when they
|
||
> disagree, the central docs win. (Some sections still mention restic / pre-8C disk handling that has
|
||
> since moved to the host agent.)
|
||
|
||
---
|
||
|
||
## Table of Contents
|
||
|
||
- [Architecture](#architecture)
|
||
- [Features](#features)
|
||
- [App Management](#1-app-management)
|
||
- [App Export/Import](#2-app-exportimport-fab-bundles)
|
||
- [Backup System](#3-backup-system)
|
||
- [Storage Management](#4-storage-management)
|
||
- [Monitoring & Health](#5-monitoring--health)
|
||
- [Notifications](#6-notifications)
|
||
- [Update Management](#7-update-management)
|
||
- [Authentication & Settings](#8-authentication--settings)
|
||
- [Central Hub](#9-central-hub-reporting)
|
||
- [Setup Wizard](#10-first-run-setup-wizard)
|
||
- [Disaster Recovery](#11-disaster-recovery)
|
||
- [Asset Sync](#12-asset-sync)
|
||
- [Debug Mode](#13-debug-mode)
|
||
- [Geo-Restriction](#14-geo-restriction)
|
||
- [App-to-App Integrations](#15-app-to-app-integrations)
|
||
- [Network File Sharing — SMB](#16-network-file-sharing--smb-megosztás-v01440-r-7-slice-1)
|
||
- [Async-job feedback](#17-async-job-feedback-v0147x-feedback-slice-1)
|
||
- [Repository Layout](#repository-layout)
|
||
- [Configuration](#configuration)
|
||
- [REST API](#rest-api)
|
||
- [Build & Deploy](#build--deploy)
|
||
- [Roadmap](#roadmap)
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────┐
|
||
│ Customer Hardware (N100 mini PC / Raspberry Pi) │
|
||
│ │
|
||
│ ┌──────────┐ ┌────────────────────────────────────────────┐ │
|
||
│ │ Traefik │ │ felhom-controller (privileged container) │ │
|
||
│ │ (reverse │──▶│ │ │
|
||
│ │ proxy) │ │ ┌──────────┐ ┌─────────────────────────┐│ │
|
||
│ └──────────┘ │ │ Web UI │ │ Stack Manager ││ │
|
||
│ │ │ (HU dash │ │ (compose ops, git sync, ││ │
|
||
│ ┌──────────┐ │ │ board) │ │ deploy, delete, update) ││ │
|
||
│ │cloudflared│ │ └──────────┘ └─────────────────────────┘│ │
|
||
│ │ (tunnel) │ │ ┌──────────┐ ┌─────────────────────────┐│ │
|
||
│ └──────────┘ │ │ Backup │ │ Storage Manager ││ │
|
||
│ │ │ (3-layer │ │ (disk scan, format, ││ │
|
||
│ ┌──────────┐ │ │ restic) │ │ mount, migrate) ││ │
|
||
│ │ App │ │ └──────────┘ └─────────────────────────┘│ │
|
||
│ │ stacks │ │ ┌──────────┐ ┌─────────────────────────┐│ │
|
||
│ │ (docker │ │ │Scheduler │ │ Monitor & Metrics ││ │
|
||
│ │ compose) │ │ │(cron-like│ │ (health, SQLite ││ │
|
||
│ └──────────┘ │ │ jobs) │ │ time-series, Chart.js) ││ │
|
||
│ │ └──────────┘ └─────────────────────────┘│ │
|
||
│ │ ┌──────────┐ ┌─────────────────────────┐│ │
|
||
│ │ │ Notify │ │ REST API + Hub Reporter ││ │
|
||
│ │ │ (events) │ │ (JSON push + events) ││ │
|
||
│ │ └──────────┘ └─────────────────────────┘│ │
|
||
│ │ ┌──────────┐ │ │
|
||
│ │ │ Assets │ │ │
|
||
│ │ │ (Hub │ │ │
|
||
│ │ │ sync) │ │ │
|
||
│ │ └──────────┘ │ │
|
||
│ └────────────────────────────────────────────┘ │
|
||
└─────────────────────────────────────────────────────────────────┘
|
||
│ events + reports │ git pull │ asset sync
|
||
▼ ▼ ▼
|
||
hub.felhom.eu gitea.dooplex.hu hub.felhom.eu
|
||
(central dashboard) (stack definitions) (logos, screenshots)
|
||
```
|
||
|
||
### Key Architecture Decisions
|
||
|
||
- **Pure Go, no frameworks** — stdlib `net/http` + `html/template`. Only external deps: `bcrypt`, `yaml.v3`, `modernc.org/sqlite` (pure Go, no CGO).
|
||
- **Privileged container** — Required for disk operations (format, mount, fstab), `/dev` access, and Docker socket control.
|
||
- **`/host-dev` indirection** — Docker overrides `/dev` with a tmpfs. The host's `/dev` is mounted at `/host-dev` to access block devices.
|
||
- **`StackDataProvider` interface** — Breaks circular import between the backup packages and stacks. Defined in `internal/appbackup` (and re-exposed via a type alias in `internal/backup`). Implemented by `stackAdapter` in `main.go`. Provides `GetStackHDDPath()` for per-drive backup routing.
|
||
- **Atomic file writes** — All persistent state (`settings.json`, `app.yaml`) written to `.tmp` then `os.Rename` for crash safety.
|
||
- **`go:embed` templates** — All HTML/CSS/JS compiled into the binary. No runtime file dependencies.
|
||
- **Design system v2 (v0.96.0, TASK-D0)** — The whole UI renders in the Felhom v2 language (canonical
|
||
reference: `felhom.eu/documentation/design/design-system.md`): navy token palette, single 2px radius,
|
||
no shadows, exception-based status color (nominal = blue/neutral; amber/red only on deviation —
|
||
`stateColor` emits `run/progress/warn/neutral/off`, `usageColor`/`tempColor` emit
|
||
`nominal/warn/crit`; a stopped app is neutral, NOT red). Capacity bars are the hairline `.meter`
|
||
component; state chips are `.tag`; informational pills are `.metarow`. **Fonts (Plus Jakarta Sans +
|
||
JetBrains Mono, variable woff2, latin+latin-ext) and a 30-icon Lucide sprite are vendored in the
|
||
binary** (`internal/web/static/fonts/` served at `/static/fonts/`; `templates/icons.html`) — no CDN,
|
||
no emoji. The setup wizard serves the same embedded stylesheet via `web.StyleCSS()`.
|
||
- **Guest RAM resize (v0.143.0, R-24; MinAgent 0.90.0)** — the Rendszer page's "Szerver memória (RAM)"
|
||
card shows the guest's current/used memory + the allowed range and lets the customer resize it. The
|
||
controller only proxies + maps the agent's machine `code` to Hungarian; **the agent (felhom-agent
|
||
v0.90.0) enforces every bound and applies the change live via PVE SetConfig — no reboot** (min 2048 /
|
||
max host_total−2048 / a shrink is refused below max(2048, usage+512)). `systemPageData` calls
|
||
`memoryCardData` (a 2 s agent GET behind the `FeatureGuestMemoryResize` gate); `POST
|
||
/api/system/memory/resize` (`ServeSystemAPI`, `internal/web/system_memory_handlers.go`) →
|
||
`agentapi.ResizeMemory` → the code→Hungarian map. A JS confirm fires only on a shrink; an outdated agent
|
||
hides the control; an unreachable agent falls back to the guest's own `/proc/meminfo`. Memory only
|
||
(cores stay observation). The lxcfs ripple means the deploy-page memory math follows a resize for free.
|
||
- **„Hálózat" card (v0.159.0, R-66)** — Beállítások → Rendszer, between „Verzió és frissítés" and
|
||
„Szerver memória": **Helyi cím (LAN)**, **Hálózati név** (`\\<SMBServerName>` — rendered ONLY while
|
||
Megosztás is enabled; the NetBIOS name exists only while samba runs), **Átjáró**, plus a muted
|
||
footer ("read this page aloud during remote troubleshooting"). Every value is live-computed per
|
||
render and stored NOWHERE (S-5): the reads go through the samba-container netns door
|
||
(`internal/stacks/guestnet.go` — the controller's own netns is the docker bridge, so in-process
|
||
answers like `/proc/net/route` would report 172.x, the S-2 trap); with Megosztás off the door is
|
||
closed and rows render „—" (an address-less row beats a wrong address). Companions: the Debug
|
||
system dump gains a `network` section (interfaces without veth*/docker*/br-* plumbing, default
|
||
route + gateway + source interface, DNS from the guest's resolv.conf, and the SAME `lan_address`
|
||
the card shows for cross-checking), and the NAS add form names the NetBIOS trap — helper text
|
||
under Szerver, plus a purely lexical hint appended to an `unreachable` failure when the submitted
|
||
server is a single-label non-IP name (`looksLikeFlatNetworkName`; no NetBIOS/mDNS resolution is
|
||
ever attempted).
|
||
- **Page IA (v0.97.0, TASK-D1)** — the settings monolith is split into four pages, each with its own
|
||
data builder (`systemPageData`/`storagePageData`/`notificationsPageData`/`securityPageData`, sharing
|
||
`settingsBaseData`) and template. Routes: `/settings` (Rendszer), `/settings/notifications`
|
||
(GET→page, POST→save on the same path), `/settings/security`, `/storage` (main-nav **Tárhely** — Meghajtók), and `/storage/network` (Hálózati tárhely / NAS; v0.98.0 split — nested sub-links under Tárhely).
|
||
The enrollment wizards live at `/storage/{init,attach}`; `/settings/storage/{init,attach}` **301**
|
||
to them; all storage action successes redirect to `/storage?storage_msg=…`. `/storage` shows a
|
||
**unified drive view**: server-rendered registry cards enriched in place from the agent `/api/disks`
|
||
(role/durable-id/actions, joined on mount path), plus read-only **Rendszermeghajtók** and
|
||
**Nem regisztrált meghajtók** groups; agent-down degrades to a single warn note. Consequential
|
||
actions use an in-page `.confirm-overlay` (`openDialog`) or the LIGHT inline two-step
|
||
(`felhomConfirm` in layout.html, v0.123.0: the trigger swaps in place to "kérdés + Igen/Mégse";
|
||
form buttons opt in via `data-confirm="…"`) — never native `confirm()`/`prompt()` (OS-modals
|
||
freeze browser automation; drill F-11).
|
||
**Shared app-list row (v0.126.0):** `templates/app_row.html` (`app_list_row`/`app_list_row_end`)
|
||
is the ONE row grammar for app lists — icon + name (+ optional secondary line) left, caller
|
||
action block right; used by the dashboard installed-apps list, the Távoli mentés toggle list
|
||
and the Visszaállítás restore-to-verify/.fab lists; the backups-apps expander header is
|
||
ALIGNED to the same grammar (own markup — it carries the toggle). Protected infra stacks
|
||
(traefik/cloudflared/filebrowser) get curated Hungarian display identity from the
|
||
`inframeta.go` map (name + description + generic `/static/infra-logo.svg` fallback icon);
|
||
filebrowser is the only infra stack with a customer link (`files.<domain>`).
|
||
**Universal app placeholder (v0.163.0):** `app_list_row` now DEFAULTS its fallback icon to the
|
||
embedded `AppPlaceholderSVG` (a 2×2 app-grid glyph, served at `/static/app-placeholder.svg`), so a
|
||
logo-less app shows a placeholder on every list surface instead of a hidden icon; infra rows still
|
||
override with the server glyph. The felhom brand mark is never an app placeholder (brand = platform
|
||
identity only).
|
||
**Enforcement — one entry point (2026-08-02):** run `python3 scripts/controller_gates.py` from
|
||
`controller/` after any template change. It is THE runner and invokes every gate:
|
||
`template_id_gate.py` (JS element-ID integrity), `emoji_gate.py` (no emoji),
|
||
`native_confirm_gate.py` (zero native confirm/prompt), `app_row_dedup_gate.py` (row markup
|
||
single-sourced), `mojibake_gate.py` (no double-encoded UTF-8 in templates/Go sources),
|
||
`docker_run_volume_path_gate.py` (every `docker … -v` mount reviewed), and `reuse_refs_check.py`
|
||
on the repo root. It exits non-zero if any gate does, and a missing gate script is a FAILURE, not
|
||
a skip. `--fast` (what `.githooks/pre-push` runs) selects the gates that touch no network and no
|
||
container runtime — today all of them. The Go `TestNoEmojiInTemplates` mirrors the emoji gate.
|
||
Why a runner: of this project's gates, only the ones named by a `CLAUDE.md` entry point ever got
|
||
run — the 2026-08-02 census found the two unnamed ones red, one for nineteen days.
|
||
- **Europe/Budapest timezone** — All scheduled jobs, timestamps, and UI labels use Hungarian timezone.
|
||
|
||
### Module Map
|
||
|
||
| Module | Path | Responsibility |
|
||
|--------|------|----------------|
|
||
| **Config** | `internal/config/` | YAML loader, validation, `FELHOM_*` env overrides |
|
||
| **Settings** | `internal/settings/` | Runtime-mutable `settings.json` (passwords, backup prefs, storage paths, notifications) |
|
||
| **Stacks** | `internal/stacks/` | Compose operations, scanning, `.felhom.yml` metadata, deploy/delete flow; **base-infra bring-up** (`infra.go` — `EnsureBaseStack`) |
|
||
| **Infra** | `internal/infra/` | Pure renderers (embedded `text/template`) for the base-infra stacks (traefik/cloudflared/filebrowser); **pinned image tags as the single source of truth** (web filebrowser sync delegates here) |
|
||
| **Crypto** | `internal/crypto/` | AES-256-GCM encryption for sensitive app.yaml values (passwords, secrets), key management |
|
||
| **Sync** | `internal/sync/` | Git-based app catalog sync (clone/pull, content-hash copy) |
|
||
| **AppBackup** | `internal/appbackup/` | Self-contained app-data backup primitives: DB dump discovery/execution (`DiscoverDatabases`, `DumpOne`), Docker-volume/app-data discovery (`StackDataProvider`, `DiscoverAppData`), keep-side path helpers (`AppDBDumpPath`, `AppVolumeDumpPath`, `AppDataDir`). `DiscoverDatabases` takes the deployed-stack set so a DB container maps to the right stack even when a slug ends in a DB-role token (M19, v0.62.0). `ListDumpFiles` takes an optional `cached(name,size,mod)` lookup so an unchanged dump isn't re-validated (line-scan) every ~5-min cycle (M18, v0.62.0). No dependency on restic/cross-drive/drive-mount. Imported directly by `appexport` and `storage`. |
|
||
| **Backup** | `internal/backup/` | Per-drive 3-layer backup: DB dumps → restic snapshots → cross-drive copies, restore. Re-exposes the `appbackup` primitives via aliases/forwarders (`appbackup_bridge.go`) for the disk/host-side code and the web/api/report consumers. |
|
||
| **Storage** | `internal/storage/` | Disk scanning (`lsblk`), partitioning (`sfdisk`), formatting (`mkfs.ext4`), mounting, data migration (`rsync`) |
|
||
| **System** | `internal/system/` | System info (`/proc`), CPU collector, mount points, disk usage, FS info |
|
||
| **Monitor** | `internal/monitor/` | System health checks, storage watchdog, legacy Healthchecks pinger (deprecated) |
|
||
| **Metrics** | `internal/metrics/` | SQLite time-series store, system + container metric collection |
|
||
| **Scheduler** | `internal/scheduler/` | Central job scheduler (periodic + daily, skip-if-running, panic recovery). `UpdateDaily` reschedules a daily job at runtime (no restart) via a per-job reschedule signal (v0.168.0). |
|
||
| **Backupwindow** | `internal/backupwindow/` | Pure time math for the customer-configurable backup window (v0.168.0): `ParseHHMM`/`FmtHHMM`, `LegTimes` (W / W+60m / W+105m, wrap-safe), `GateWindow` ([W+2h, W+6h)), `EffectiveWindow` (settings > yaml > "02:30"). Offsets are constants — derived, never stored. |
|
||
| **SelfUpdate** | `internal/selfupdate/` | Version checking (registry), update trigger, state persistence, startup verification |
|
||
| **Notify** | `internal/notify/` | Email notifications via hub relay, preference sync, per-event cooldowns |
|
||
| **Report** | `internal/report/` | Hub report builder + HTTP pusher (system, stacks, backup, health) |
|
||
| **Assets** | `internal/assets/` | Hub-managed asset syncer: downloads logos/screenshots with SHA-256 change detection |
|
||
| **SelfTest** | `internal/selftest/` | Startup self-test: 9 diagnostic checks (Docker, dirs, storage, hub, restic, metrics) |
|
||
| **Util** | `internal/util/` | Shared utilities: `TruncateStr` for debug log output truncation |
|
||
| **AppExport** | `internal/appexport/` | Per-app export/import via `.fab` bundles (config + DB + user data), optional AES-256 encryption |
|
||
| **API** | `internal/api/` | REST JSON endpoints, diagnostic dump (`/api/debug/dump`) |
|
||
| **Web** | `internal/web/` | Hungarian dashboard, auth, page handlers, template functions, alerts |
|
||
|
||
---
|
||
|
||
## Features
|
||
|
||
### Disk health — "Lemezek állapota" card + degradation alert (v0.169.0)
|
||
|
||
The dashboard shows a per-physical-disk health card driven by the agent's SMART summary (serialized
|
||
into `/disks` from agent v0.94.0 — no new smartctl load; the controller only reads it). One pure
|
||
verdict function (`agentapi.DiskVerdictFor`) is the shared truth for the card chip and the check:
|
||
|
||
- **Rendben** (PASSED, clean) · **Figyelmeztetés** (PASSED but reallocated/pending/offline-uncorrectable
|
||
or NVMe critical/media/percentage-used ≥ 90) · **Hiba** (FAILING) · **Nincs adat** (nil/UNKNOWN/old
|
||
agent — never alarms).
|
||
- The card fetches `/disks` through a **60 s TTL cache** (dashboard refresh-spam can't smartctl-storm
|
||
the host); an unreachable agent renders "Nincs adat" and the page still loads.
|
||
- A **6-hourly `disk-health-check`** emits `disk_health_degraded` (warn/critical) **only on a
|
||
degradation** vs an in-memory baseline — first run baselines silently, recovery/UNKNOWN never notify,
|
||
and a controller restart re-baselines silently. **No global banner** (deliberate): the card + email
|
||
carry it. The hub allowlist must include `disk_health_degraded`.
|
||
|
||
### 1. App Management
|
||
|
||
The controller manages Docker Compose stacks through a complete lifecycle: catalog sync, first-time deployment, runtime operations, and deletion.
|
||
|
||
#### Git Sync (`internal/sync/`)
|
||
|
||
The app catalog lives in a separate Git repository. The controller:
|
||
- Shallow-clones the catalog on startup
|
||
- Periodically fetches updates (configurable, default 15 min)
|
||
- Copies only `docker-compose.yml` and `.felhom.yml` to the stacks directory
|
||
- **Never overwrites** `app.yaml` (user secrets are safe)
|
||
- Uses SHA-256 content hashing — only writes files that actually changed
|
||
- Triggers stack rescan after sync so the dashboard updates immediately
|
||
- **Post-sync hook**: auto-injects missing deploy fields (new secrets, domains) into existing `app.yaml` for stacks whose templates were updated (see Missing Field Injection below)
|
||
- Manual sync via "Sablonok frissitese" button or `POST /api/sync`
|
||
|
||
#### First-Time Deploy Flow
|
||
|
||
1. Customer sees app card with "Telepites" button
|
||
2. Deploy page pre-generates and **displays** all auto-values before the user clicks deploy:
|
||
- `domain` fields: shown as readonly text input with the customer's configured base domain
|
||
- `subdomain` fields: editable text input pre-filled with the default from `.felhom.yml`, shown with `.base-domain` suffix. Validated for DNS-safe format, reserved names, and uniqueness across deployed stacks. Locked after deploy — changing requires Remove + Redeploy
|
||
- `secret` fields: pre-generated and shown as masked password inputs with a "Megjelenítés" reveal button — user can see/copy all DB passwords and keys before deploying
|
||
- User-configurable inputs (admin password, language, storage path) remain editable
|
||
- Section header prompts the user to note down any passwords they need
|
||
3. `checkBeforeDeploy()` JS guard fetches live state first (prevents double-deploy from another tab)
|
||
4. **Memory validation** (F1, v0.61.0): the controller runs as a Docker container inside an LXC, where
|
||
`/proc/meminfo` shows the **Proxmox host's** RAM (no lxcfs in the container) and the container's own
|
||
cgroup is unlimited (the guest cap lives on the LXC ancestor). So the guest cap is read from the
|
||
**Docker daemon** (`system.GuestMemTotalMB()` → `docker info` MemTotal — the daemon runs in the LXC and
|
||
reports the guest's lxcfs-backed RAM; the cgroup limit is preferred when present, e.g. non-nested):
|
||
- `usable_memory = guest_cap - reserved_memory_mb` (default 384MB reserved)
|
||
- Hard block if `committed_used + new_request > usable_memory`, where `committed_used` =
|
||
`CommittedMemory()` (sum of running apps' mem requests) — the guest-wide RSS is not observable from
|
||
the container, so the controller's own committed accounting is the accurate, cheap "used".
|
||
- `/api/system/info` reports the guest cap as total and committed memory as used.
|
||
4b. **Docker-data volume reserved-buffer gate (v0.58.0, storage-split prevention layer):** the OS rootfs
|
||
and Docker data are split onto separate volumes; infra (controller/traefik/cloudflared/filebrowser)
|
||
shares the one Docker data-root (`/var/lib/docker`) and is protected by **prevention, not placement**.
|
||
`system.GetDockerVolumeHeadroom()` measures the Docker-data volume via `statfs("/")` (the controller
|
||
container's overlay root is the upperdir on that volume — true with the golden's **overlay2** driver)
|
||
and reserves `max(5 GB, 10%)`. `deployStack` **refuses a new deploy (HTTP 507)** when free space is
|
||
at/under the buffer; the deploy page shows the warning + disables the button. Fail-open on a statfs
|
||
error. The runtime disk monitor (`healthcheck.go`, warn 80% / crit 90%) watches the same volume and
|
||
trips above the buffer. (Assumes the split guest's large data volume; the golden bakes overlay2 + log
|
||
rotation so images+volumes live on the data volume, not `/var/lib/containerd`.)
|
||
5. Pre-generated secret values are submitted as hidden form inputs so the **same values** the user saw are saved to `app.yaml` (no silent re-generation on submit). Controller saves `app.yaml`, sets in-memory `Deployed` + `Deploying` flags, then runs `docker compose up -d` **asynchronously** in a goroutine — API returns immediately so the UI switches to the progress panel without waiting for image pulls. On failure the goroutine reverts both disk and in-memory state and sets `DeployError`.
|
||
6. 3-step progress panel polls `GET /api/stacks/{name}` every 3s: config saved → `deploying` (pulling images) → containers starting → health check passed. New `StateDeploying` state shown while compose-up is in progress (no containers yet).
|
||
7. Post-deploy: locked fields (DB_PASSWORD, etc.) become read-only; the "Automatikusan generált értékek" section continues to show the saved values on the settings page
|
||
8. The deploy/settings page includes **start/stop/restart** buttons for deployed apps, plus a "Megnyitás ↗" link to the app's subdomain URL (only visible when running)
|
||
|
||
#### Catch-All Page for Stopped Apps
|
||
|
||
When a user visits a stopped or undeployed app's subdomain (e.g., `travel.demo-felhom.eu`), the controller serves a branded error page instead of Traefik's raw 404:
|
||
|
||
- **Traefik catch-all router**: The controller's `docker-compose.yml` registers a second router (`catchall`) with `priority=1` (lowest) and `HostRegexp(.+)`. Running apps always win; only requests with no matching container reach the controller.
|
||
- **`CatchAllMiddleware`** in `server.go` intercepts requests where `Host` ≠ `felhom.DOMAIN`, serves the catch-all page **without auth** (user has no session on the app subdomain).
|
||
- **`findStackBySubdomain()`** identifies the app by matching the subdomain against deployed `app.yaml` `SUBDOMAIN` env or metadata fallback.
|
||
- **`catchall.html`** — standalone template (no layout, inline CSS) showing the app name, status ("leállítva" / "nincs telepítve" / "nem található"), and links to the controller dashboard or the app's detail page.
|
||
- **Subdomain links** on the Alkalmazások page are only shown for deployed apps (non-deployed apps have no guaranteed subdomain yet).
|
||
|
||
#### Indítópult (app launcher page) (v0.163.0)
|
||
|
||
`/launcher` (`launcherHandler` + `templates/launcher.html`) is the **first** sidebar item, above
|
||
Vezérlőpult; `/` still lands on the Vezérlőpult. It renders a grid of large tappable tiles — one per
|
||
**openable** deployed app. Openability has a single criterion, shared with the „Megnyitás" button: the
|
||
stack has a subdomain (env `SUBDOMAIN` > `.felhom.yml` subdomain > `protectedStackSubdomains`),
|
||
resolved through the extracted `Server.subdomainMap` helper (the dashboard and Alkalmazások pages use
|
||
the same helper). The controller's own stack is excluded by name.
|
||
|
||
Each tile is a colored rounded square: `funcmap.tileColor(slug, brand)` returns a validated
|
||
`.felhom.yml` **`brand_color`** (`#rgb`/`#rrggbb`, `Metadata.BrandColor`) or, when absent/invalid, a
|
||
deterministic FNV-1a-of-slug → HSL color (fixed S/L, hue varies per app). The white monochrome logo
|
||
renders on top of a **monogram** initial (`funcmap.initial`, multibyte-safe); if the logo fails to
|
||
load the monogram shows through (the launcher does NOT use the app-placeholder here). Operational apps
|
||
are `<a target="_blank" rel="noopener">` links; stopped/degraded apps render greyed + unclickable with
|
||
the Hungarian state badge. Empty state links to `/stacks`.
|
||
|
||
#### Indítópult megosztása — guest launcher via capability URL (v0.165.0)
|
||
|
||
The admin launcher's **"Indítópult megosztása"** button mints a **capability URL** —
|
||
`https://<host>/s/<token>`, where `token` is a 160-bit `crypto/rand` value
|
||
(`newShareToken`, base64.RawURLEncoding, 27 chars) — that serves a **standalone, read-only guest
|
||
launcher** with **no account and no admin session**. The link grants **information only, zero
|
||
control**: app names + public URLs; every privilege stays behind each app's own auth and the
|
||
controller admin password. The tile visual is shared with the admin launcher via the `launch_tile`
|
||
template partial; the app slice comes from the extracted `Server.launcherApps()` helper.
|
||
|
||
- **Routing** (`internal/web/share.go`, `share_handlers.go`): `/s/<token>` joins the RequireAuth
|
||
pre-auth allowlist **after** the claim-gate block (an unclaimed box never serves the guest page —
|
||
the claim gate stays supreme) and is exempted from session CSRF (the guest password POST carries a
|
||
pre-auth HMAC CSRF, `validShareCSRF`, mirroring the claim POST). Token match is
|
||
`subtle.ConstantTimeCompare`; an empty stored token (= sharing OFF, there is no separate flag)
|
||
matches nothing, so a wrong/disabled token returns a **byte-identical mux-default 404** (`share404`).
|
||
Guest responses set `X-Robots-Tag: noindex, nofollow` / `Referrer-Policy: no-referrer` /
|
||
`Cache-Control: no-store`. The token is a secret: the ServeHTTP debug line and the 404 WARN redact
|
||
`/s/` paths to `/s/<redacted>`.
|
||
- **Optional per-share password** (`settings.LauncherSharePasswordHash`): a SEPARATE bcrypt credential
|
||
(never the admin `PasswordHash`), guarded by its OWN per-IP 5/1-min attempt map (`shareAttempts`,
|
||
never the admin `loginAttempts`). A correct password mints a signed gate cookie =
|
||
HMAC-SHA256(`token|passwordHash`) keyed with the persisted, box-scoped `web.session_secret` — so
|
||
rotating the token OR changing the password invalidates every outstanding cookie with no bookkeeping.
|
||
- **Guest state labels** ride the v0.164.0 ruling and never expose internal vocabulary: clickable ⇔
|
||
`isOperationalState && !routeUnpublished` (operational AND route actually published, so a tap never
|
||
dead-ends); `StateStopped` ⇒ "A tulajdonos leállította"; any other non-clickable state ⇒
|
||
"Átmenetileg nem elérhető". Empty ⇒ "Jelenleg nincs elérhető alkalmazás." (`buildGuestApps` is the
|
||
pure, tested mapping; templates `launcher_shared.html` + `launcher_share_password.html`).
|
||
- **Admin modal** (in `launcher.html`): current link + copy button, QR code
|
||
(`GET /launcher/share/qr.png`, ~256px PNG via `github.com/skip2/go-qrcode`, admin-authed, `no-store`),
|
||
set/clear share password, "Új link készítése" (rotate), "Megosztás kikapcsolása" (clears token AND
|
||
password). The management POSTs live under `/launcher/share/*` and ride the normal admin session +
|
||
session CSRF; rotate/disable use the inline `data-confirm` (felhomConfirm) affordance. A
|
||
feature-detected **"Megosztás…"** button (v0.165.1) opens the OS share sheet via `navigator.share`
|
||
(title + text + URL only — never the QR as a `files:` attachment); hidden unless the browser
|
||
supports it, with "Link másolása" as the universal fallback (the non-cancel rejection path falls
|
||
back to it too).
|
||
|
||
Design ruling: member accounts are superseded by this capability-URL model; per-member tile
|
||
visibility is parked under the SSO arc.
|
||
|
||
#### Dashboard "Megnyitás" Button
|
||
|
||
Running apps on the Vezérlőpult now show a "Megnyitás ↗" button that opens the app's subdomain in a new tab. The `Subdomains` map is built in `dashboardHandler` from `app.yaml` env or metadata fallback.
|
||
|
||
The optional `.felhom.yml` **`open_path`** field (`Metadata.OpenPath`) is appended to that URL for apps whose UI isn't at `/` — e.g. `gokapi` → `/admin`, `ghost` → `/ghost/`. Empty = bare root. Rendered via `.Meta.OpenPath` in `dashboard.html`, `deploy.html`, and `app_info.html` (the same field flows through all three open-link sites; no handler change needed). Must start with `/`; cosmetic only (does not affect routing).
|
||
|
||
#### App lifecycle — withdrawing an app without orphaning anyone (v0.158.0)
|
||
|
||
`.felhom.yml` carries an optional top-level **`lifecycle:`** (`Metadata.Lifecycle`), the catalog's
|
||
answer to "stop offering this app" that does not punish the customers already running it.
|
||
|
||
| value | offered for new installs? | shown to someone already running it |
|
||
|---|---|---|
|
||
| `available` (default; absent/empty ≡ this) | yes | nothing special |
|
||
| `hidden` | **no** | nothing — "we stopped offering this" is not their problem |
|
||
| `abandoned` | **no** | „Nem karbantartott" badge + a notice on the app page that updates and security fixes will no longer arrive |
|
||
|
||
**A deployed instance keeps full function in every state.** Lifecycle governs what is OFFERED, never
|
||
what runs — deleting a template instead would mark every deployed instance `Elavult` and offer a
|
||
Törlés button for working software.
|
||
|
||
Three predicates on `Metadata` are the single interpretation of the field — every surface goes
|
||
through them: `EffectiveLifecycle()`, `CanInstall()`, `IsAbandoned()`.
|
||
|
||
- **Listing** — `web.visibleCatalogStacks` drops a template that is not installable AND not deployed
|
||
here (`Deployed || Protected || CanInstall()`).
|
||
- **Deploy gate** — `api.deployStack` refuses server-side before any mutation with
|
||
„Ez az alkalmazás jelenleg nem telepíthető." (409). `stacks.DeployStack` repeats the check for any
|
||
caller that does not route through the API. Hiding the button is not a gate.
|
||
- **Unknown values fail OPEN** (→ `available` + one WARN), deliberately opposite to the gate's
|
||
fail-closed posture: a typo, or a state from a newer catalog than this controller, must never pull
|
||
a working app out of every customer's catalog. Both read the same `EffectiveLifecycle`, so they
|
||
cannot disagree.
|
||
- **Orphan detection must never see this field.** `getCatalogTemplateSlugs` keys on directory +
|
||
compose presence only; withdrawn templates stay in the catalog tree. Asserted by
|
||
`TestCatalogTemplateSlugs_IgnoresLifecycle` with a red-proof.
|
||
- **Badges are generic plumbing**: `web.MetaBadge` + the `meta_badge` template partial + the
|
||
`lifecycleBadge` funcmap entry. R-56's difficulty labels are intended as a sibling funcmap
|
||
function returning the same `*MetaBadge` — no new markup or CSS.
|
||
|
||
#### App Info Pages
|
||
|
||
Each app can define rich metadata in `.felhom.yml`:
|
||
- `app_info`: tagline, use_cases, first_steps, prerequisites, default_creds, docs_url
|
||
- `optional_config`: groups of post-deploy configurable env vars (e.g., API keys for metadata providers)
|
||
- `resources`: mem_request, mem_limit, pi_compatible, needs_hdd, hungarian_ui
|
||
- `initial_credentials` (v0.84.0): for apps that **auto-generate a first-login password into a file** at
|
||
first boot (vs taking it from a deploy field) — e.g. Crafty → `/crafty/app/config/default-creds.txt`.
|
||
Fields: `file` (path inside the container), `format` (`json`|`regex`|`plain`), optional `container`
|
||
(defaults to the stack's main container), `username_key`/`password_key` (json), `username_pattern`/
|
||
`password_pattern` (regex, first capture group), `note`. `ReadInitialCredentials`
|
||
(`internal/stacks/initialcreds.go`) reads the file **live** via `docker exec … cat` and parses it with
|
||
the pure `parseInitialCreds`; the value is **never persisted** to `app.yaml`. `appDetailHandler`
|
||
surfaces it on `/apps/{slug}` as a "Kezdeti belépési adatok" card (masked password + reveal/copy),
|
||
labelled as the *initial* credential (stays valid only until the customer changes it in-app). Hidden
|
||
when the container is down / file missing / parse fails. Reuse for any future self-seeding app.
|
||
- `data_paths` (v0.172.0, R-75): the **customer-facing folder annotation** — `{path, root, role, label}`
|
||
where `root ∈ {import, userdata, hdd}` and `role ∈ {import, library, export}`. It ANNOTATES paths that
|
||
must already exist as compose binds and can never declare one (so no new filesystem-write primitive
|
||
comes from catalog data). Validation is deliberately ASYMMETRIC: a malformed **path** is a whole-block
|
||
reject (reusing `appbackup.ValidateRelPath`, the same refusal set as `backup:`), an unknown **role**
|
||
fails OPEN with one WARN (the `Lifecycle` precedent — presentation, not data handling). Rendered on
|
||
`/apps/{slug}` as „Hova tegyem a fájlokat?" for DEPLOYED apps only, each row a FileBrowser deep link
|
||
plus a **class-driven** consequence line. Full contract:
|
||
`felhom.eu/documentation/controller/import-and-data-paths.md`.
|
||
- `backup` (v0.132.0; **three** lists since v0.172.0): the **referential-coupling classification** block
|
||
(Task 2). Optional lists `userdata:` (relative to `${USERDATA_PATH}`), `hdd:` (relative to
|
||
`${HDD_PATH}`) and `import:` (relative to `${IMPORT_PATH}`, R-75), each of
|
||
`{path, class}` where `class ∈ {mandatory, optional, excluded}` (COUPLED / DECOUPLED-precious /
|
||
DECOUPLED-bulk). `LoadMetadata` validates the block against the app's compose binds and **rejects the
|
||
WHOLE block** (→ nil + one `[ERROR]`, app behaves as legacy) on any defect. Semantics
|
||
(`appbackup.ClassifyBinds`): an explicit entry wins over the `:ro` default; an unlisted writable bind
|
||
defaults `mandatory`, an unlisted `:ro` bind defaults `excluded`; **no block at all = legacy behavior
|
||
per tier**. **INERT** as of v0.132.0 — the schema/parser/classifier + the
|
||
`Manager.ClassifiedBinds` / `StackDataProvider.GetStackClassifiedBinds` seam exist, but no backup
|
||
tier consumes them yet (Task 3 = tier policy engine, Task 4 = manual `.fab` UI). See
|
||
`felhom.eu/documentation/audits/SPIKE-backup-classification-2026-07-14.md`.
|
||
- **Capture-set computation (v0.133.0, Task 3-core, INERT):** `appbackup.ComputeCaptureSet(binds,
|
||
hasClassification, tier, hddPath)` (`internal/appbackup/captureset.go`) is the pure path-algebra
|
||
that turns classified binds into a tier-filtered, structurally-guarded, containment-deduped
|
||
absolute `CaptureSet{HasClassification, Paths, Skipped}` (slash algebra, no `filepath`/FS/log).
|
||
`TierOffsite` = mandatory only; `TierSecondary` = mandatory + optional; excluded dropped;
|
||
legacy short-circuits to unit-only. Structural guards (traversal / bare HDD drive-root / reserved
|
||
`backups/` zone) move refused would-be captures into `Skipped` for the engines to log. Companion
|
||
`CrossAppOverlaps` (pure; WARN wiring deferred to 3a/3b). See
|
||
`felhom.eu/documentation/architecture/07-backup-architecture.md` §3.
|
||
- **Offsite tier engine (v0.134.0, Task 3a — consumes the above):** `internal/backup/offbox.go`
|
||
+ `offbox_capture.go` + `offbox_restore.go`. Each toggled app's push is ONE multi-path restic
|
||
snapshot = recovery unit + its `TierOffsite` mandatory set (`offboxCaptureSet`); legacy/undeployed
|
||
stay unit-only. Skipped/missing mandatory paths are loud gaps (English log + Hungarian
|
||
`LastWarning`) because restic 0.14.0 silently skips a missing source path (SP-3.4). Quota reads
|
||
`stats --mode raw-data` (real repo bytes, SP-1); a pre-push gate blocks an ENLARGEMENT that would
|
||
cross the soft quota (unit-only push continues; `OffboxTarget.EnlargedBlocked`; edge-triggered
|
||
notify). Retention `forget --group-by host,tags` (SP-2). Restore (`RestoreOffboxScratch`) scratches
|
||
to a data drive off the rootfs (F-A1) behind a headroom gate; unit-only default via `--include`
|
||
the absolute unit path; `PlaceOffsiteRestore` merges a full scratch into live via
|
||
`rsync --ignore-existing` (never `--delete`), refusing on the pure `mapOffsiteRestorePaths` guards.
|
||
It restores FILES ONLY — no database, no stack restart — and the UI now says so.
|
||
- **Truthful hub-managed empty state (v0.161.0, R-70):** when `controller.yaml`'s `offsite.enabled`
|
||
is true but no `offbox` target exists yet (the pre-apply window — or a burned one-time credential,
|
||
DIAG-f10), the Távoli mentés status card AND the target empty-state line say „Felhom offsite
|
||
tárhely kiépítve — a beállítás automatikus, folyamatban…" instead of „igényelhető szolgáltatás" /
|
||
„Még nincs beállítva…". Data key `OffsiteHubEnabled` from `backupsOffboxData`; own-NAS setup form
|
||
unchanged. The hub side (v0.72.0) watches the same window from its end (delivery-state detector,
|
||
stuck event, R-71c credential self-heal).
|
||
- **Coherent snapshot pairs (v0.148.0, R-44):** every offsite run — manual AND nightly — refreshes
|
||
the DB/volume dumps and recovery units (`offsitePreDump` → `runDBDumpsInternal`) BEFORE the
|
||
restic capture, so each snapshot is an internally coherent `{DB@T, files@T}` bundle and
|
||
retention is a history of restorable points. Order is the mechanism: the gap can only add files
|
||
the DB does not reference yet, never remove one it does. Each manifest carries `offsite_run_id`
|
||
+ `dumps_at`; a manifest without them is a pre-v0.148 pair of unknown skew, surfaced at restore
|
||
time. The periodic refresh carries the prior stamp forward and never invents one. A dump-leg
|
||
failure is a loud WARN that does NOT abort the push (data-first: a degraded backup beats none).
|
||
- **A scratch restore NAMES ITS SCOPE (v0.198.0, R-204 item 3 — `restoreScratchOutcomeMsg`).**
|
||
`mode=unit` (the default) restores the recovery unit only — the app's definition, configuration
|
||
and DB dumps — because `RestoreOffboxScratch` passes `--include <unit path>`; the userdata that is
|
||
in the SAME snapshot is excluded by it. The outcome flash used to be one sentence for both modes
|
||
and named neither scope, so on the last step of a disaster recovery the customer was told
|
||
„visszaállítva" after the thing they were looking for had not been. The unit outcome now states
|
||
what came back, that the customer's own files did NOT, and the step that gets them; the full
|
||
outcome states that the files came with it (an absence is not a statement). The wizard's intent
|
||
card 1 carries the same scope BEFORE the choice. **The `mode=full` two-step size gate is
|
||
untouched**, and the default stays `unit` — all three wizard forms set `mode` explicitly, so
|
||
changing it would alter nothing the customer sees while silently changing a mode-less POST.
|
||
- **Offsite reconstitution (v0.148.0, R-43 — `offbox_reconstitute.go`):** the leg that was missing.
|
||
`ReconstituteFromOffsite` (`/backup/offbox/reconstitute`, „Teljes visszaállítás (fájlok +
|
||
adatbázis)") makes the live app equal to the chosen snapshot: **safety dump → stop → files
|
||
overwritten (`rsyncRestoreOverwrite`: no `--ignore-existing`, no `--delete`) → the DATABASE
|
||
SERVICE ONLY started (`StartStackServices`, v0.153.0) → the snapshot's dump replayed
|
||
(`reimportDBDumpsFrom`, reading the SCRATCH unit) → the full stack started → health wait**.
|
||
Two invariants: nothing is ever deleted (post-snapshot files survive as extras), and the
|
||
`pre-restore-` safety dump is verified on disk BEFORE anything is stopped or overwritten — if it
|
||
cannot be taken the operation refuses with zero changes. Safety dumps appear in `ListDumpFiles`
|
||
(they are the undo). The live recovery unit is still never overwritten, which is why the replay
|
||
source is the scratch. Honesty surfaces (`OffsiteScratchPair`): dump age, an unstamped-pair
|
||
warning, and the R-44 empty-dump sniff — all warn-level, none of them gates.
|
||
- **The restore wizard (v0.154.0, R-48 — `web/restore_wizard.go`).** The offsite restore controls
|
||
used to render as up to five inline forms per app row, two of which — the missing-only merge and
|
||
the true reconstitution — were sibling buttons whose difference is whether the data comes back.
|
||
That mis-selection caused the round-2 incident. Each row now carries ONE entry linking to a
|
||
per-app wizard; the three intents are cards with consequence sentences, and the dangerous one
|
||
keeps the R-43 double-confirm verbatim. `deriveWizardStep` is pure — (op running, size-gate
|
||
flash, scratch ready) → step + which intents unlock — and a running op outranks a stale
|
||
`?full_prep=`, so no commit button survives into a restore. While ANY op runs, every mutation
|
||
form is suppressed server-side rather than offered and refused. No new endpoint, no job registry
|
||
(that stays R-45), and the page works with JavaScript disabled.
|
||
**v0.155.0 fix:** the "is an op running" read must come from `RestoreStatus()` (the `opRunning`
|
||
display flag, set synchronously by `BeginRestoreOp`), NOT `Manager.IsRunning()` (the concurrency
|
||
single-flight, which `RestoreOffboxScratch` never acquires — so v0.154.0's execution step was
|
||
unreachable for the verification restore). The strip's highlight is its own derived `Phase`, so a
|
||
finished restore reads „Eredmény" while the intent step is available again; the outcome card is
|
||
window-bounded and app-bound.
|
||
- **The DB-only replay window (v0.153.0, R-47).** Until v0.153.0 the whole stack was started before
|
||
the replay, so the application's own schema management raced the dump: measured live on
|
||
2026-07-19 (H4), immich-server rebuilt `clip_index` two seconds before the dump's `CREATE INDEX`
|
||
and the replay aborted `already exists` under `ON_ERROR_STOP=1`. The DB service is now brought up
|
||
alone (`appbackup.DBServiceNames` reads the LIVE compose's `services:` map to name it), the dump
|
||
is replayed with the app still down, and only then does the full start run. **Fail-closed:** a
|
||
dump with no identifiable DB service refuses before the first mutation. Every exit from the window
|
||
— replay failure, DB-only start failure — still does a best-effort full start, so a failed restore
|
||
never leaves the box with a database and no application.
|
||
|
||
The `/apps/{slug}` page renders hero section, screenshots, setup guide, and optional config form.
|
||
|
||
#### Stack Operations
|
||
|
||
| Operation | What it does |
|
||
|-----------|-------------|
|
||
| Start | `docker compose up -d` — pre-start memory check rejects with 409 if insufficient RAM |
|
||
| Stop | `docker compose stop` (blocked for protected stacks) |
|
||
| Restart | `docker compose restart` |
|
||
| Update | `docker compose pull` + `docker compose up -d` |
|
||
| Remove | `docker compose down --volumes` + remove `app.yaml` + optional HDD/backup cleanup; template preserved for redeploy |
|
||
| Delete | `docker compose down --rmi local --volumes` + optional HDD data cleanup (orphaned stacks only) |
|
||
|
||
**Remove vs Delete**: "Eltávolítás" (Remove) is for deployed catalog stacks — it reverts the stack to "Nincs telepítve" state while keeping the template for easy redeployment. "Törlés" (Delete) is for orphaned stacks — it removes the entire stack directory including templates. Both require stopping the stack first.
|
||
|
||
**Remove modal** shows three sections: (1) always-removed items (Docker volumes, app.yaml, cross-drive schedule), (2) optional HDD data deletion with reimport warning, (3) optional backup data deletion (DB dumps + cross-drive rsync) with restic retention note.
|
||
|
||
**Protected stacks** (traefik, cloudflared, felhom-controller) cannot be stopped, removed, or deleted from the UI. Restart is allowed.
|
||
|
||
**Orphan detection**: Deployed stacks with no matching catalog template are marked as orphaned with an "Elavult" badge and can be safely deleted.
|
||
|
||
#### Base-infrastructure bring-up (`stacks/infra.go` + `internal/infra/`, v0.41.0)
|
||
|
||
**Canonical import root (v0.172.0, R-75).** `${IMPORT_PATH}` = `<system namespace root>/userdata/import`
|
||
— ONE drop-zone per box, on the always-available system drive, never per data drive (each drop-zone app
|
||
has exactly one ingest bind, so a per-drive `import/` would put a dead lookalike on every other drive,
|
||
and `import/*` is `class: excluded` so files stranded there are unbacked too). Injected at BOTH
|
||
compose-env builders; **no per-drive fallback** — unresolvable leaves it unset so compose fails loudly.
|
||
The system drive is deliberately NOT a registered `StoragePath`, so the FileBrowser bind
|
||
(`/srv/beolvasas`, sidebar „Beolvasás"), the skeleton and the system-owned `beolvasas` SMB share each
|
||
reach it explicitly. The **userdata skeleton is catalog-derived** (`DeriveUserdataDirs` +
|
||
`UserdataSkeletonCarry`, sorted — the sort is load-bearing, see REUSE.md) and can only ever ADD.
|
||
|
||
The controller stands up its own base stack — **traefik** (reverse proxy), **cloudflared** (external tunnel), **filebrowser** — instead of relying on the bare-metal `scripts/docker-setup.sh` (which a Proxmox-provisioned guest never runs). `internal/infra` renders the compose + config files from `controller.yaml` via embedded `text/template`s (lifted from `docker-setup.sh`); image tags are **pinned constants there** (`TraefikImage`/`CloudflaredImage`/`FileBrowserImage`) and the web FileBrowser sync path delegates to the same renderers, so the pinned versions can never diverge.
|
||
|
||
`Manager.EnsureBaseStack()` creates the `traefik-public` network, then deploys traefik → cloudflared → filebrowser under `${stacks_dir}/<name>`. It is:
|
||
- **single-flight** (a `TryLock` guard — it's called from both first boot and every health tick, so overlapping runs must not race on the same stack dir),
|
||
- **idempotent** (skips a stack whose container is already running; never overwrites an existing filebrowser compose, preserving the storage mounts `SyncFileBrowserMounts` manages),
|
||
- **non-fatal** (logs, never crashes the controller).
|
||
|
||
cloudflared is only deployed when a tunnel token is configured. **Triggers**: a first-boot goroutine (after stack init) and an unconditional call on every `system-health` tick (self-heal — cheap when healthy thanks to the idempotency). `monitor.EffectiveProtected` mirrors the cloudflared condition so a LAN-only node (no tunnel token) doesn't report a perpetual "protected container not running" FAIL.
|
||
|
||
**Backend transports — self-signed HTTPS backends (`ensureServersTransports` → `RenderServersTransports`, v0.83.0).** Traefik talks **HTTP** to app backends by default, which is correct for every catalog app that serves plain HTTP. The exception is an app that serves its **own self-signed TLS** on the internal docker bridge (the first is Crafty, HTTPS-only on `:8443`): Traefik must speak `https` to it *and* skip verifying a per-container self-signed cert (no CA to verify against; the hop never leaves the host). `insecureSkipVerify` is **not settable via Docker labels** in traefik v3 — it must live in static/file config — so `EnsureBaseStack` writes a file-provider dynamic file `dynamic/serverstransports.yml` defining a **named** transport `insecure-skip-verify` (write-if-changed; hot-loaded by the file watcher). An app **opts in per-service** via two catalog labels — `loadbalancer.server.scheme=https` + `loadbalancer.serverstransport=insecure-skip-verify@file` (the `@file` suffix is the cross-provider reference). Backend verification stays the default (ON) for every other service — there is deliberately **no global `insecureSkipVerify`** in `traefik.yml`. This write runs **outside** `ensureTraefik` (which early-returns when traefik is already up) so an established node still materializes the file on a self-heal tick.
|
||
|
||
> **Mount prerequisite (Section-G):** the controller writes these stacks under `/opt/docker/stacks` *inside its container*, but `docker compose up` runs on the **guest** Docker daemon. The golden's controller-bootstrap (`felhom-agent` `build-golden.sh`) therefore bind-mounts that path **same-path** (`-v /opt/docker/stacks:/opt/docker/stacks`) so the daemon resolves every relative bind source — without it, all bind-mounted stacks (base infra and customer apps) silently break.
|
||
|
||
**Controller routing + the wildcard cert anchor (`wireController` → `RenderControllerRoute`, v0.41.1 / v0.42.1).** filebrowser self-registers with traefik via Docker labels + `traefik-public` membership baked into its compose; the controller can't (it's started by the golden bootstrap *before* `traefik-public` exists, and the v2 `bootstrap.json` carries no domain — that comes from the hub pull). So `EnsureBaseStack` wires the controller **post-pull**: it `docker network connect traefik-public felhom-controller` and writes a traefik file-provider route `dynamic/controller.yml` (`Host(felhom.<domain>) → http://felhom-controller:8080`, write-if-changed). When DNS-01 ACME is configured, that route is **also the wildcard-cert anchor**: its router-level `tls.domains: *.<domain>` makes traefik **proactively obtain the wildcard `*.<domain>` + apex via Cloudflare DNS-01 at startup** (an entrypoint-level `http.tls.domains` does *not* trigger issuance in traefik v3 — only a router-level `tls.domains` does). Every other router then serves that one real wildcard cert by SNI — no per-app `certresolver` labels. This is what lets a LAN client reach the box directly at `*.<domain>` with the real cert (the `felhom-agent` split-horizon resolver depends on it).
|
||
|
||
#### Missing Field Injection (`deploy.go`)
|
||
|
||
When app templates are updated (e.g., a new `APP_KEY` secret is added to `.felhom.yml`), existing deployed apps need the new field in their `app.yaml`. The controller handles this automatically:
|
||
|
||
- **On startup**: `InjectMissingFields()` runs for all deployed stacks
|
||
- **After sync**: the post-sync hook runs for stacks whose templates were updated
|
||
- For each deployed stack, compares `.felhom.yml` `deploy_fields` against `app.yaml` env vars
|
||
- Missing `secret` fields: auto-generated using the field's generator spec (`password:N`, `hex:N`, `base64key:N`)
|
||
- Missing `domain` fields: filled with the customer's configured domain
|
||
- Missing `subdomain` fields: filled with the field's default value or the `.felhom.yml` `subdomain:` metadata
|
||
- Other field types (e.g., `text`, `select`): logged as warning for manual configuration
|
||
- Locked fields are added to the locked list automatically
|
||
|
||
**Generator types**: `password:N` (alphanumeric), `hex:N` (hex-encoded random bytes), `base64key:N` (`base64:` + N random bytes base64-encoded, for Laravel APP_KEY etc.), `static:VALUE` (literal value).
|
||
|
||
#### Container State Display
|
||
|
||
| State | Color | Label | Meaning |
|
||
|-------|-------|-------|---------|
|
||
| Running + healthy | Green | "Fut" | All containers running and healthy |
|
||
| Running + starting | Orange | "Indulas..." | Healthcheck not yet passed |
|
||
| Deploying | Orange | "Telepítés..." | Compose up in progress (image pull, container creation) |
|
||
| Running + unhealthy | Yellow | "Nem egeszseges" | Docker or controller-side healthcheck failing |
|
||
| Stopped/exited | Red | "Leallitva" | All containers stopped |
|
||
| Restarting | Yellow | "Ujrainditas..." | Restart loop |
|
||
| Not deployed | Gray | "Nincs telepitve" | Compose file exists, not deployed |
|
||
|
||
**Route-unpublished indicator (F5, v0.61.0).** Traefik's Docker provider only publishes a route to a
|
||
container that is healthy (or has no healthcheck), so an `unhealthy`/`restarting` deployed app returns a
|
||
hard **404** at its URL even though the container is running. The `routeUnpublished` template helper
|
||
(`funcmap.go`) drives a distinct "URL nem elérhető – útvonal nincs publikálva" indicator on the dashboard
|
||
and stacks cards for such apps, so a dead URL isn't mistaken for a merely-degraded-but-reachable one.
|
||
|
||
#### Controller-side Health Probes (`internal/stacks/healthprobe.go`)
|
||
|
||
For apps that declare a `healthcheck:` section in `.felhom.yml`, the controller probes the container directly over the Docker network (both are on `traefik-public`). This complements Docker-level healthchecks and is the **only** health mechanism for distroless/scratch images that lack shell utilities.
|
||
|
||
Three probe types are supported:
|
||
- **`http`** — Any HTTP response (even 4xx/5xx) = service is alive. Only connection refused/timeout = unhealthy.
|
||
- **`api`** — HTTP request with response validation (expected status code, body content). Fails if expectations aren't met.
|
||
- **`tcp`** — Simple port reachability check via `net.Dial`.
|
||
|
||
Multiple checks per app are supported (all must pass). The probe scheduler runs every 10 seconds; per-app intervals default to 5 minutes and are configurable via `healthcheck.interval` in `.felhom.yml`. Probe results are stored in `Stack.HealthProbe` and exposed via the API. Failed probes override the stack state to `StateUnhealthy`; the override clears automatically when the next probe passes.
|
||
|
||
**Fast initial probing:** On start/restart, stale health probe results are cleared (so the stack doesn't immediately appear "unhealthy" from a previous result). Until the first healthy probe, the controller checks every 10 seconds instead of the normal 5-minute interval, giving fast feedback on whether the app came up successfully.
|
||
|
||
---
|
||
|
||
### 2. App Export/Import (.fab bundles)
|
||
|
||
Per-app export creates a self-contained `.fab` file (tar.gz, optionally encrypted) that can be stored externally or used to restore the app on the same server. Distinct from the automatic backup system — user-initiated, per-app, produces a single portable file.
|
||
|
||
**Bundle contents:** `manifest.json` + `config/` (compose, .felhom.yml, app.yaml with plaintext secrets) + `database/` (gzipped SQL dump) + `data/` (HDD bind mount tars AND Docker named volume tars — **additive since v0.130.0**; a `needs_hdd` app bundles both).
|
||
|
||
**Encryption:** Optional AES-256-CTR + HMAC-SHA256 with scrypt key derivation (N=32768). Format: `"FABE"` magic header + salt + IV + encrypted tar.gz + HMAC tag. Streaming for multi-GB files.
|
||
|
||
**Export flow:** Estimate size → check free space → optionally stop app → copy config → dump DB → tar user data (docker volumes stream via `docker cp` through a stopped helper container — v0.125.0; never a `docker run -v` host mount, which strands data on containerized controllers) → **fail-loud bundle assertion** (every manifest-claimed tar must exist non-empty, AND a `needs_hdd` bundle claiming NO data at all is refused — v0.130.0 C6B-F1) → create tar.gz → optionally encrypt → atomic rename. App restarts automatically after export if it was stopped.
|
||
|
||
**Mount discovery (v0.130.0, C6B-F1):** `stacks.ExportDataMounts` — the `${HDD_PATH}` binds unioned with the `${USERDATA_PATH}` **root** (single `userdata` entry) when the compose uses the standard userdata convention; pre-fix the adapter was `${HDD_PATH}`-only, so 12/13 `needs_hdd` catalog apps exported hollow (config-only) bundles. The root (not per-bind) keying is what round-trips through the import's basename→`<HDD_PATH>/<subdir>` mapping. A basename collision between mounts fails the export loudly. The share-removal endpoint also refuses while a deployed app's HDD_PATH is on the share (C6B-F2 guard).
|
||
|
||
**Class-scoped export (v0.136.0, Task 4 — the SQ6 fix):** for a **classified** app the userdata root tar is **exclude-scoped** — it keeps only dirs that are an ancestor-or-descendant of a SELECTED bind relpath (mandatory ∪ checked-optional ∪ opted-in-excluded; R1-C, the `tier2Reconcile` keep-rule), so sibling apps' content no longer rides along. No selected userdata bind ⇒ the root tar is **skipped entirely** (radarr → state-only). Non-selected HDD bind mounts are skipped; a mount matching no classified bind is kept (fail toward capture). Mechanics unchanged: **ONE** userdata tar, per-mount skip, **manifest v1 + import untouched**. The plan is pure (`appexport/fabplan.go` `computeFabPlan` over `appbackup.ComputeFabBuckets`); `tarDirectoryExcluding` prunes excluded subtrees in the walk. **Legacy (no-block) apps export byte-identically to v0.130.0.** Mandatory paths are a **server-side floor** (a client cannot deselect them). The export page shows the class selection UI (locked mandatory, pre-selected optional checkboxes, opt-in excluded behind the two-number warning + FileBrowser pointer); the estimate carries an additive class split (`ExportEstimate.MandatoryItems/OptionalItems/ExcludedItems` + `BaseBytes`).
|
||
|
||
**Import flow:** Decrypt if needed → extract → **validate-before-destroy** (v0.125.0: every manifest-claimed data tar must be present non-empty BEFORE the app is stopped or any volume removed — hollow bundles from containerized ≤0.124.0 exporters are refused with the app untouched) → prepare stack dir (create new or `compose down --volumes` for existing) → restore config (re-encrypt app.yaml with current server key) → restore user data (HDD tars; volumes populate via `docker cp` streaming) → restore DB (start DB service, wait for ready, import dump) → start full stack → refresh UI.
|
||
|
||
**Architecture:** `internal/appexport/` package with `ExportStackProvider` adapter interface (same pattern as `backup.StackDataProvider`). `exportAdapter` in `main.go` bridges `stacks.Manager` to the provider.
|
||
|
||
**API endpoints:** `/api/export/estimate`, `/api/export/start`, `/api/export/status`, `/api/export/bundles`, `/api/export/manifest`, `/api/export/import`, `/api/export/import/status`; **browser download (v0.124.0):** `/api/export/download/estimate`, `/api/export/download/start`, `GET /api/export/download?file=<basename>`; **browser upload (v0.128.0):** `POST /api/export/upload/{init,chunk,finalize,abort}`.
|
||
|
||
**Browser download (v0.124.0 — portability, NOT a backup tier):** the same export pipeline runs with dest = `<DataDir>/fab-downloads/` (same producer → byte-identical bundle), then streams via a guarded endpoint (basename-shape + dir-containment guard; `Content-Disposition: attachment`; `io.Copy`; the staged bundle is removed after the stream and a 1h TTL sweep runs on startup + each start). Estimate is shown BEFORE starting; the batch UI downloads apps one at a time (no combined archive). `handler_export_download.go`.
|
||
|
||
**Browser upload (v0.128.0 — the download's return leg):** the `/import` page uploads a `.fab` straight from the browser. Chunked because the Cloudflare tunnel caps request bodies at ~100 MB (probed live 2026-07-13: 120 MiB → edge 413, 80 MiB → origin): JS slices the file into 64 MiB chunks (strictly sequential offsets; one retry per chunk re-synced from the 409 `received_bytes` echo), the server streams each to a `.part-<random>` file in the DEFAULT drive's exports dir (`io.Copy`, 96 MiB per-request cap, free-space gate = size + 1 GiB), finalize checks the exact declared size, fsyncs and atomically renames (collision → lowest-free `"name (N).fab"`). Single-flight; no client-side hash (the .fab format self-validates at import); in-memory state — startup GC sweeps `*.part-*`, 15-min idle timeout aborts server-side. The scan + validation + import pipeline are untouched. `handler_export_upload.go`.
|
||
|
||
**UI:** Export button on app info page, the "Hordozható mentéscsomag (.fab)" section on `/backups/restore` (per-app download + batch), standalone import page at `/import` (upload zone + drive-scanned bundle list).
|
||
|
||
---
|
||
|
||
### 3. Backup System
|
||
|
||
The backup system implements a **3-2-1 backup architecture**. Each tier is a **complete,
|
||
self-sufficient backup** — any single tier can fully restore an app.
|
||
|
||
**The reserve — per-app backup admission (v0.192.0 decision B2, widened by v0.193.0 / R-181).**
|
||
`internal/backup/admission.go`. Since the `mp1`→`mp0` merge (R-165) local backups and Docker's
|
||
data-root share one filesystem, so an unbounded backup write is a stopped box rather than a slow one.
|
||
Before **any** of an app's three local write legs runs — DB dump, volume dump, recovery-unit capture —
|
||
`admitApp` takes **one verdict for that app for that run** and the other two legs reuse it. A refused
|
||
app writes nothing at all, is **not stopped**, keeps its previous unit byte-identical, and produces
|
||
**exactly one** operator alert (`recovery_unit_capture_failed`, operator-tier).
|
||
|
||
- **The verdict is lazy, not run-wide.** It is taken at the app's first write, because app A's dump
|
||
can put app B under the reserve; a verdict taken at run start would read a disk that no longer
|
||
exists by the time B writes.
|
||
- **It is never re-decided between an app's own legs**, and the memo is reset per run.
|
||
- **Two questions, both against two thresholds (97% used / 1 GiB free).** *Headroom*: is the
|
||
filesystem already below the reserve? *Size*: would this app's own write take it below? The size
|
||
estimate is the app's **previous** `.sql` + `.tar` already on disk. **No history → headroom-only**,
|
||
deliberately — otherwise the first backup is the one that can never happen — and the alert says so.
|
||
- **The thresholds sit beyond `fillwatch`'s critical band (95% / 2 GiB)**, so the customer is always
|
||
warned before a refusal is possible.
|
||
- **It refuses; it never deletes.** Nothing here is generational — one unit per app at one fixed path
|
||
— so "prune the oldest" could only destroy a different app's only local copy.
|
||
|
||
**Sidebar behaviour (v0.146.0).** Groups that own sub-pages — Tárhely, Biztonsági mentés,
|
||
Megosztás — render as **accordions**: the header is a real `<button class="nav-group-toggle">`
|
||
(keyboard- and AT-reachable for free) carrying a chevron, and **exactly one group is open at a
|
||
time**. The group containing the active page is rendered open **server-side** (`.is-open` in
|
||
`layout.html`), so the right group is already open before any JS runs and stays open if JS never
|
||
does; the vanilla listener in `layout.html` only handles clicks. Groups without sub-items
|
||
(Vezérlőpult, Alkalmazások, Rendszermonitor, Debug) are plain links, unchanged. Making the header a
|
||
button cost no reachability because **every group's own landing page is also its first sub-item**
|
||
(`/storage` → Meghajtók, `/backups` → Áttekintés, `/sharing` → Hálózati megosztás). Collapse uses
|
||
`grid-template-rows: 0fr → 1fr` rather than `max-height`, so it animates to the content's real
|
||
height with no magic number to drift as item counts change.
|
||
|
||
**Page map (v0.124.0 IA split — sidebar children under Biztonsági mentés):**
|
||
|
||
| Route | Page | Sections |
|
||
|-------|------|----------|
|
||
| `/backups` | Áttekintés | storage overview, whole-guest Rendszermentés, status stat cards, single-copy warning, **backup-target banner + offer (v0.186.0)** |
|
||
| `/backups/remote` | Távoli mentés | Felhom-offsite status card (3 states, display-only), tier-3 status block + quota, participation toggles (+ zero-toggle hint; the persisted zero-toggle run-warning is DISPLAY-replaced by a "kijelölés módosult" note once ≥1 app is toggled — `offboxWarningDisplay`, v0.126.0), manual-target form (`#offbox-section`) |
|
||
| `/backups/apps` | Alkalmazások | schedule, Adatbázisok table, per-app 1./2./3. tier rows (tier-2 config entry; tier-3 actions deep-link to `/backups/remote#offbox-section`) |
|
||
| `/backups/restore` | Visszaállítás | restore panel, offsite restore list (**one „Visszaállítás…" entry per app** since v0.154.0), existing verification copies, .fab download/import loop |
|
||
| `/backups/restore/app?name=<app>` | Visszaállítás — <app> | **R-48 per-app offsite restore wizard** (v0.154.0). GET-only; three described intent cards (ellenőrzés / hiányzó fájlok / teljes visszaállítás), a visible phase strip, and a server-derived step. Adds NO mutation endpoint — every card posts to the pre-existing `/backup/offbox/{restore,place,reconstitute}` |
|
||
|
||
**Backup-target banner + offer (E-2 · v0.186.0, R-114 + R-112).** The `/backups` page renders the
|
||
whole-system backup-target state server-side, from the AGENT's view (never from our own intent flag).
|
||
Four outcomes, three of which the customer sees nothing for or one thing for:
|
||
|
||
| State | Renders |
|
||
|---|---|
|
||
| healthy — a real drive holds the target | **nothing** (no badge, no reassurance: a working box must look normal) |
|
||
| degraded, never configured (`local`/unset) | the system-disk copy **+ an offer control** that POSTs `/api/storage/backup-target/assign` |
|
||
| **configured, drive absent** (`TargetAbsent`) | the absent-drive copy, **no offer** — the remedy is to reconnect *that* drive |
|
||
| unknown (agent unreachable / pre-R-82) | **nothing** — absence of an answer is not degradation |
|
||
|
||
`degradedMessageFor` is the single decision point for customer copy; `backupTargetView` returns nil
|
||
for the two silent states. The absent copy is verbatim the hub's `backup_target_absent` email so the
|
||
banner and the mail agree. The offer never auto-submits, and `restart_required` from assign is shown
|
||
rather than papered over with a self-restart (the agent deliberately does not restart itself).
|
||
|
||
Shared data builders: `backupsCommonData` (chrome + full-status + flash) + `backupsOffboxData`
|
||
(offbox target/toggles) in `handlers.go`; shared partials in `templates/backups_shared.html`.
|
||
(The v0.124.0 split was MOVE-only, gated one-shot by `backups_split_move_check.py`; the gate was
|
||
retired in v0.126.0 when the moved blocks were legitimately rewritten onto the shared row partial.)
|
||
|
||
| Tier | Contents | Location | Can fully restore? |
|
||
|------|----------|----------|--------------------|
|
||
| **1. Nightly restic** | DB + Config + User data | Same drive as app | Yes (not against drive failure) |
|
||
| **2. Cross-drive** | DB + Config + User data | Different physical device | Yes |
|
||
| **3. Remote** | Everything | Cloud / remote server | Future |
|
||
|
||
**Key principles:**
|
||
- User data backup is **mandatory** — every app with HDD bind mounts is included
|
||
automatically. There is no per-app toggle.
|
||
- Each tier includes **everything** needed to restore: DB dumps, config, and user data.
|
||
No tier depends on another tier's data.
|
||
- **Tier 2 is configurable for ALL apps** — not just apps with HDD data. Non-HDD apps
|
||
back up config + DB dumps to the secondary drive (small but protects against drive failure).
|
||
**(v0.118.0, F6):** the nightly tier-2 RUN (`RunAllTier2`) now actually includes volume-only apps
|
||
too — it used to skip every non-HDD app, leaving them a single controller-level copy on `sys_drive`.
|
||
A sys_drive app's restore-point drive label is now clear ("Belső SSD (rendszer)"), never blank.
|
||
- **3-2-1 on single vs multi drive (v0.118.0, F6):** on a box with a second physical drive, tier-2 is
|
||
the off-drive copy. On a **single-drive** box (no off-drive target at all) there is genuinely only
|
||
ONE local copy — `FullBackupStatus.SingleCopyWarning` surfaces an honest Hungarian notice on the
|
||
backup page ("Csak egy másolat készül…") instead of implying a 3-2-1 guarantee the box cannot keep.
|
||
- The `AppBackupPrefs.Enabled` field in settings.json is legacy and not read by any code.
|
||
- **v2 layout + class-driven legs (v0.135.0, Task 3b):** `backups/secondary/<stack>/` is the v2
|
||
relpath-mirroring layout — `.felhom-tier2-layout` marker (written LAST) + `recovery-unit/` +
|
||
`hdd/<relpath>/` + `userdata/<relpath>/`. For a **classified** app the appdata leg is the
|
||
`TierSecondary` capture set (per-bind mandatory + optional; excluded drops out — `tier2_capture.go`);
|
||
**legacy** apps keep a byte-identical resolver set in the same layout. N>1 appdata dirs + nested
|
||
binds are native (the old flat-appdata N>1 refusal is gone). First v2 run per app = delete-and-rebuild
|
||
of the old flat `appdata/` + a reconcile pass that prunes dest dirs a bind no longer covers; all
|
||
removals go through `tier2SafeRemove` (refuses anything outside `backups/secondary/`). The **SSD**
|
||
fallback is a state-only tier (unit + mandatory; optional skipped, honest reason). **NETWORK (NAS)
|
||
storage is never a tier-2 target** — pinned or auto (F-6C-1: rsync `-og` under root_squash →
|
||
wrong-owner restore). Restore reads v2 behind the marker gate; a pre-v2 copy is refused.
|
||
|
||
**Per-app Tier 2 contents by app type:**
|
||
|
||
| App type | Tier 2 contents | Example |
|
||
|----------|----------------|---------|
|
||
| HDD + DB | Config + DB + User data | Immich, Paperless-ngx |
|
||
| HDD, no DB | Config + User data | — |
|
||
| Docker volumes + DB | Config + DB + Volume data | Tandoor |
|
||
| Docker volumes, no DB | Config + Volume data | Mealie (SQLite) |
|
||
| DB, no HDD/volumes | Config + DB | Vikunja |
|
||
| Config only | Config | Gokapi, Homepage |
|
||
|
||
#### Tier 1: Nightly Backup (mandatory, same drive)
|
||
|
||
The nightly backup has two phases that run sequentially. All paths are **per-drive** — each physical drive gets its own restic repo and per-app DB dump directories.
|
||
|
||
> **Customer-configurable backup window (v0.168.0).** ONE setting on the backup page — **"Mentési
|
||
> időablak kezdete"** (start W, default "02:30") — drives every leg at FIXED, never-stored offsets so
|
||
> they can never be misordered: DB dump at **W**, tier-2 mirror at **W+60m**, off-box at **W+105m**
|
||
> (wrap-safe). The whole-guest (agent PBS/vzdump) cycle is gated to **[W+2h, W+6h)** with a safety valve
|
||
> (runs regardless once the last successful backup is older than cadence+24h, so a box only ever on
|
||
> outside its window never starves); **manual "Mentés most" is never gated**. A saved window fans out to
|
||
> the three daily legs via `scheduler.UpdateDaily` and takes effect **without a restart**. Precedence:
|
||
> settings > controller.yaml `db_dump_schedule` > "02:30". See `internal/backupwindow`.
|
||
|
||
> **Multi-tier whole-guest backup (v0.174.0, R-82 Slice B).** The agent can serve SEVERAL whole-guest
|
||
> backup tiers with independent cadences — "local daily + PBS weekly" (agent >= v0.97.0,
|
||
> `GET /backup/tiers`). The controller owns quiescing, so it reconciles them: it collects EVERY due
|
||
> tier up front and runs them inside **ONE quiesce window** — one stop, N sequential backups (vzdump
|
||
> holds a guest lock), one resume. Two cycles on the weekly night would mean two app outages for one
|
||
> night's work. The app stays quiesced until the **LAST** tier snapshots, so every tier is
|
||
> app-consistent; the consequence is that both-due-night downtime is *(first tier's full backup)* +
|
||
> *(last tier's snapshot)*, which is why tiers run fast-first (the agent advertises primary/local
|
||
> first). A manual **"Mentés most"** covers every tier, due-ness ignored. The window gate's safety
|
||
> valve evaluates the OLDEST due tier, so a stale DR tier cannot be starved by a fresher local one.
|
||
> Against a **pre-R-82 agent** (`/backup/tiers` 404s) the loop degrades to the single untargeted
|
||
> tier, logs it once, and still takes the backup — MinAgent is unchanged. See `internal/quiesce`
|
||
> (`tiers.go`) and `internal/agentapi/backup_tiers.go`.
|
||
|
||
> **Atomic dump writes (v0.118.0, CAMPAIGN-3 F7).** BOTH dump paths are crash-safe: the DB dump
|
||
> (`dbdump.go` DumpOne) and the Docker-volume dump (`DumpAppVolumes`) write to a `.tmp` sibling, fsync,
|
||
> then `os.Rename` over the restore point ONLY on success. A mid-write failure (a NFS cut mid-tar, an
|
||
> EIO, a timeout) removes only the `.tmp` and leaves the last good `.sql`/`.tar` **byte-untouched** —
|
||
> a tier-1 restore is replace-semantics, so an in-place write that got truncated to 0 bytes used to
|
||
> destroy the only restore point. `.tar.tmp` files are invisible to the restore-point/stale scans and
|
||
> orphans are swept on the next run.
|
||
|
||
> **Stale-primary sweep (v0.118.0, F5).** After each cycle, `pruneStalePrimaryDirs` removes an
|
||
> orphaned `backups/primary/<app>` dir left on an OLD drive when an app's `HDD_PATH` moved to another
|
||
> drive. Guarded: only for a DEPLOYED app whose CURRENT drive differs from the dir's drive; never the
|
||
> current-drive dir (the live restore point) or an undeployed app's dir; strictly under `backups/primary/`.
|
||
|
||
> **~~NAS backup locality (v0.118.0, CAMPAIGN-3 Part 4 — decision A)~~ — SUPERSEDED by R-108
|
||
> (v0.187.0, 2026-07-30).** Decision A said a NAS-resident app's tier-1 artifacts live **on the NAS
|
||
> itself** (`nas-media/backups/primary/<app>`), beside the data, with the tier-2 cross-drive copy as the
|
||
> off-NAS mitigation. That locality is exactly what made a `backups/` tree reachable through
|
||
> FileBrowser's share-ROOT bind (`download: true`), and it is why architectural target **D5** — app
|
||
> secrets in the local recovery unit — could not be adopted.
|
||
>
|
||
> **An app's data namespace may no longer live on network storage at all** (operator ruling
|
||
> 2026-07-30), so the case decision A described can no longer arise: no app on a NAS ⇒ no
|
||
> `backups/primary/` on a NAS. `settings.RefuseAsAppNamespace` is the single predicate; every
|
||
> placement surface consults it (deploy POST, per-app migrate, decommission-with-migrate). The NAS
|
||
> keeps its **browse** capability unchanged — the share-root `:rslave` bind is load-bearing for
|
||
> automount wake (R-67) and was deliberately NOT narrowed; scoping it is undefinable anyway, since
|
||
> apps on a share store at `<share>/<app>` and creating a `userdata/` layer would write Felhom
|
||
> convention onto a customer's own NAS.
|
||
>
|
||
> The NAS-outage window decision A warned about is therefore also gone: an app's tier-1 artifacts are
|
||
> always on a local drive now, because the app itself always is.
|
||
|
||
**Drive layout (v0.26.0):**
|
||
```
|
||
<drive>/
|
||
├── felhom-data/ ← all controller-managed data (namespace, v0.26.0+)
|
||
│ ├── appdata/<app>/ ← app user data
|
||
│ └── backups/
|
||
│ ├── primary/
|
||
│ │ ├── restic/ ← one restic repo per drive (all apps on this drive)
|
||
│ │ └── <app>/
|
||
│ │ ├── db-dumps/ ← per-app DB dump files
|
||
│ │ └── volume-dumps/ ← per-app Docker volume tars (v0.33.0)
|
||
│ └── secondary/
|
||
│ ├── restic/ ← secondary restic repo (cross-drive)
|
||
│ ├── _infra/ ← infra config mirror
|
||
│ └── <app>/rsync/ ← per-app rsync data
|
||
├── .felhom-infra-backup/ ← DR marker (stays at drive root for scanner)
|
||
├── Dokumentumok/ ← user files (not controller-managed)
|
||
└── media/ ← user files (not controller-managed)
|
||
```
|
||
|
||
> **Note (Model A — corrected in v0.52.0):** `HDD_PATH` in `app.yaml` is the **in-guest mount point**
|
||
> (e.g., `/mnt/felhom-usb`). Under slice-10 Model A the host agent binds `<drive>/felhom-data` directly
|
||
> onto that mount, so the in-guest mount **already is** the `felhom-data` namespace root. Neither the
|
||
> compose templates nor the path helpers add a `felhom-data` segment for a drive-resident app: app data
|
||
> is `${HDD_PATH}/appdata/<app>` and backups `${HDD_PATH}/backups/...`, **single-nested**. Only the
|
||
> SSD-only system-data fallback (a bare root, `inGuestDrive=false`) appends `felhom-data`. See
|
||
> `NamespaceRoot(drivePath, inGuestDrive)` in `internal/appbackup/paths.go`.
|
||
> Earlier catalog templates used `${HDD_PATH}/felhom-data/appdata/<app>`, which double-nested to
|
||
> `.../felhom-data/felhom-data/...` on a Model-A drive; v0.52.0 dropped that segment in the catalog and
|
||
> locks deploy↔backup path agreement with `internal/stacks/hddpath_agreement_test.go`.
|
||
|
||
Path computation is centralized in `backup/paths.go` via the `FelhomDataDir = "felhom-data"` constant:
|
||
- `PrimaryResticRepoPath(drivePath)` → `<drive>/felhom-data/backups/primary/restic/`
|
||
- `AppDBDumpPath(drivePath, stackName)` → `<drive>/felhom-data/backups/primary/<stack>/db-dumps/`
|
||
- `AppVolumeDumpPath(drivePath, stackName)` → `<drive>/felhom-data/backups/primary/<stack>/volume-dumps/`
|
||
- `AppDataDir(drivePath, name)` → `<drive>/felhom-data/appdata/<name>/` (final segment is the app's real appdata dir NAME, resolved via `AppDataDirNames` from compose binds — NOT always the stack name; F-S2)
|
||
- `SecondaryResticRepoPath(drivePath)` → `<drive>/felhom-data/backups/secondary/restic/`
|
||
- `AppSecondaryRsyncPath(drivePath, stackName)` → `<drive>/felhom-data/backups/secondary/<stack>/rsync/`
|
||
- `SecondaryInfraPath(drivePath)` → `<drive>/felhom-data/backups/secondary/_infra/`
|
||
- `InfraBackupDir(mountPath)` → `<drive>/.felhom-infra-backup/` (**unchanged** — stays at drive root for DR scanner)
|
||
|
||
> **⚠️ Stale:** the restic/secondary helpers above (`PrimaryResticRepoPath`, `SecondaryResticRepoPath`,
|
||
> `AppSecondaryRsyncPath`, `SecondaryInfraPath`) describe the pre-strip layout — restic/cross-drive was
|
||
> removed in slice 8C. This section is rewritten when Tier 2 (Phase 3) lands.
|
||
|
||
#### Per-app recovery unit (Phase 2, v0.53.x; secret model rewritten by **D5**, v0.188.0)
|
||
|
||
Each app's `backups/primary/<app>/` is a self-contained, recreatable **recovery unit**:
|
||
|
||
```
|
||
backups/primary/<app>/
|
||
├── compose/ docker-compose.yml + .felhom.yml + app.yaml (0600 — CARRIES the portable secrets)
|
||
├── db-dumps/ app-consistent DB dump(s)
|
||
├── volume-dumps/ named-volume tars
|
||
└── manifest.json image pins, secret NAMES, data_key names, portable NAMES, checksums, secret_source
|
||
```
|
||
|
||
- **The secret split (D5, schema 2, operator ruling 2026-07-30).** The unit was secret-free until
|
||
v0.188.0, and that made "restore from the drive alone" false: the fast, local, customer-doable
|
||
Tier-1/2 restore secretly depended on the slow, operator-driven whole-guest restore, because a
|
||
data-encrypting key or a DB password absent from the guest cannot be regenerated without leaving the
|
||
restored data unreachable. **Tier-1/2 now needs the drive and nothing else.** What travels is decided
|
||
in ONE place, `stacks.PortableSecretEnvVars`:
|
||
- **TRAVELS — every `type: secret` field** (45 of 53 across the catalog): the declared `data_key`s,
|
||
the 18 DB/root passwords, and the internal signing/encryption secrets. Each of these either
|
||
decrypts data sitting on the SAME drive or authenticates to a container on an internal compose
|
||
network with no external listener, so possessing it adds nothing to possessing the drive — which is
|
||
exactly D2's argument for keeping the DATA plaintext. Written into the unit's app.yaml at **0600**,
|
||
plaintext, like the data beside it.
|
||
- **WITHHELD — every `type: password` field** (7 admin/UI logins) **plus the `nonPortableSecrets`
|
||
register** (`vaultwarden/ADMIN_TOKEN`, whose `/admin` panel is on the app's public web port).
|
||
These authenticate against published services, so their blast radius is NOT bounded by the drive.
|
||
They stay in the guest and are regenerated on restore (O4). **Excluding this class is what licenses
|
||
the plaintext ruling — the two are coupled and must not be relaxed independently.**
|
||
- The register is **code, not a catalog flag**, deliberately: a security boundary a catalog push can
|
||
silently move is not a boundary (cf. R-97a). Adding an app whose `type: secret` field gates an
|
||
internet-reachable login means adding a row there.
|
||
- **Fail-closed is unchanged.** A `data_key` missing from **both** the unit and the guest still refuses
|
||
the restore outright (never generated). D5 makes the key normally present; "normally" is not a reason
|
||
to soften the gate.
|
||
- **Precedence: the UNIT WINS** over the guest when both hold a value. Not "newest wins" — the unit's
|
||
secrets are captured in the same run as the dumps beside them, so the unit's value is the one that
|
||
MATCHES THE DATA BEING RESTORED, while the guest's is merely the most recent. A rotated data key does
|
||
not decrypt data encrypted with the old one, and a rotated DB password does not match the hash inside
|
||
the restored data directory. Pinned in both directions.
|
||
- **Resettable secrets (O4, v0.99.0)** — now the rare path, since the portable class comes from the
|
||
unit. An unrecoverable withheld secret gets a **generated replacement** from its catalog `generate`
|
||
spec (`stacks.GenerateSecretForField` via the `backup.SetSecretGenerator` seam) rather than redeploying
|
||
blank; the value persists encrypted through `RecreateStackDefinitionFromUnit` → `SaveAppConfig`.
|
||
⚠️ **R-127:** a regenerated **database** password is NOT harmless — `POSTGRES_PASSWORD` is ignored once
|
||
PGDATA is non-empty, so the restored data dir keeps the old role hash and the app cannot authenticate
|
||
against its own rows, while the dump replay (local trust socket) still reports success. The WARN says so.
|
||
- Helpers: `RecoveryUnitPath` / `RecoveryUnitComposePath` / `RecoveryUnitManifestPath`
|
||
(`internal/appbackup/paths.go`). Capture: `Manager.CaptureRecoveryUnit` (`internal/backup/recovery_unit.go`),
|
||
run from the daily DB dump and the periodic `RefreshCache` (idempotent checksum-skip); the split itself
|
||
is in `buildUnitAppYaml`. The env + portable values come from `StackDataProvider.GetStackRecoveryInfo`,
|
||
which keeps `NonSecretEnv` and the secret set disjoint by construction. `data_key` fields are marked in
|
||
`.felhom.yml` (`DeployField.DataKey`).
|
||
- **A schema-1 (pre-D5) unit carries no secrets** and still restores from the guest — the restore
|
||
degrades rather than failing, and the next capture rewrites the unit (the app.yaml checksum changes).
|
||
- **Consequence for the other tiers:** the unit is copied by Tier 2 (another customer drive, plaintext,
|
||
same reasoning) and pushed offsite by restic (`offbox.go` — encrypted at rest under the customer-owned
|
||
repo password). Neither tier's code changed; the secrets simply travel with the unit they already carried.
|
||
- **Restore replays the DB dump (F17, v0.61.0; re-sequenced v0.153.0, R-47).** `RestoreFromRecoveryUnit`
|
||
(and the `RestoreApp` fallback) stops the app → restores named-volume tars → recreates the compose
|
||
definition and persists the recovered env (`RecreateStackDefinitionFromUnit` — **starts nothing**)
|
||
→ starts the DATABASE SERVICE ONLY (`StartStackServices`, named from the unit's compose) →
|
||
**replays each `db-dumps/*.sql` into that DB** → starts the full stack → health wait. Before
|
||
v0.153.0 `RecreateStackFromUnit` ended in a full `compose up -d`, so this path carried the same
|
||
H4 race as the offsite one (see the reconstitution section above), with the same fail-closed rule
|
||
and the same guarantee that every exit still brings the app back up. The replay itself uses
|
||
`backup.reimportDBDumps` → `appbackup.ImportDump` (psql / mariadb client, using the live container's own
|
||
discovered credentials). The DB replay runs AFTER the volume restore, so the **logical SQL dump wins**
|
||
over any volume-tar copy of the database (the dumps use DROP/CREATE — `pg_dump --clean --if-exists`,
|
||
`mariadb-dump` default `--add-drop-table` — so replay is idempotent). Volume-restore and DB-import
|
||
failures now **surface** (restore returns an error) instead of a swallowed WARN. Prior to v0.61.0 the
|
||
per-app restore never replayed the `.sql`, so DB-resident data did not come back.
|
||
|
||
#### Tier 2 — off-drive copy (Phase 3, v0.55.x)
|
||
|
||
For every HDD app, Tier 2 (`internal/backup/tier2.go`) rsync-mirrors the recovery unit
|
||
(`backups/primary/<app>/`) + the app's resolved `appdata/<name>/` to `<target>/backups/secondary/<app>/`
|
||
on a **different physical disk** — the only off-drive protection bind-mounted HDD app data can get (PBS
|
||
can't reach bind mounts). **The appdata dir NAME is derived from the app's compose `${HDD_PATH}` binds,
|
||
not assumed to be the stack name** (F-S2, v0.131.0: paperless-ngx writes `appdata/paperless`;
|
||
`tier2AppDataName` → `appbackup.AppDataDirNames`); an app resolving to >1 distinct appdata dir is
|
||
refused loudly. This copies the recovery unit + `appdata/<name>` ONLY — **not** the browsable
|
||
`userdata/` tree (F-S1, owned by the backup-classification redesign) and **not** the namespace
|
||
wholesale. Auto-targeted: **prefer another registered user-data drive** (off-disk via
|
||
`system.SamePhysicalDevice`); else the **internal SSD for small units only**, behind a size-aware
|
||
**rootfs-headroom guard** (`tier2FitsHeadroom`) that **refuses rather than fills** the ~8 GB guest rootfs
|
||
(reserve = `max(2 GB, 20%)`), recording an honest "needs a 2nd HDD" status. Status persists via
|
||
`settings.CrossDriveBackup` and drives the "2. mentés" card. Runs daily (`tier2-backup`, 03:30) or via
|
||
`POST /api/backup/tier2`. restic is **not** used — a plain browsable mirror.
|
||
|
||
**In-place file restore from the Tier-2 copy (C2, v0.100.0 — closes drill finding F2)** —
|
||
`POST /backup/tier2/restore` (`backup.RestoreTier2Files`, `internal/backup/tier2_restore.go`) + the
|
||
**"Fájlok visszaállítása"** button on the healthy Tier-2 layer row. **Additive-only** semantics
|
||
(`rsyncRestoreMissing`: `rsync -a --ignore-existing`): files missing from the live resolved
|
||
`appdata/<name>` dir (F-S2 — compose-derived, not the stack name)
|
||
are copied back from the RECORDED Tier-2 copy; existing live files are **never overwritten** (a
|
||
customer edit after the last copy wins) and **nothing is ever deleted** — this exactly serves the
|
||
"I deleted my files" scenario with zero risk to newer data. Source = the recorded
|
||
`CrossDriveBackup.DestinationPath` (never a fresh target selection). Single-flight with
|
||
backup/restore; refusals (no copy / never ran / copy dir gone / either drive disconnected /
|
||
decommissioned) happen before the app is stopped, with customer-readable Hungarian reasons;
|
||
stop → copy → start → health-wait. Out of scope by design: overwrite/point-in-time restore (offbox +
|
||
operator paths) and per-file selection. Apps that index their data dir (e.g. Nextcloud) may need a
|
||
rescan (occ files:scan) before restored files appear in their own UI.
|
||
|
||
> **COVERAGE — read this before assuming an app is protected by this button (C9-F1, v0.183.0).**
|
||
> This restore reads `hdd/` and `userdata/` **only**. It has never read `recovery-unit/`, which every
|
||
> Tier-2 run also writes and which holds the app's DB dumps and named-volume tarballs. Enumerated
|
||
> across all 53 catalog templates: **43 apps have no readable subtree at all** (their data is entirely
|
||
> in named volumes — BookStack, Docmost, Vaultwarden, Gitea, …), **9** have file legs but never their
|
||
> database or volumes, 1 is stateless. So the button is a guaranteed no-op for 81% of the catalog and
|
||
> only ever partial for the rest.
|
||
>
|
||
> Since v0.183.0 it is HONEST about that instead of silently reporting success:
|
||
> `Tier2RestoreCoverage` is consulted **before** anything starts, an app with no readable subtree is
|
||
> refused **without being stopped** and told which action does work („…Használd a Visszaállítás
|
||
> indítása gombot a Biztonsági mentés → Visszaállítás oldalon."), and a run that does proceed claims
|
||
> only what it **examined** („Minden vizsgált fájl megvan a helyén.") plus a disclosure that the
|
||
> database and internal volumes are not part of this restore.
|
||
>
|
||
> The action that DOES cover those apps is the keep-side recovery-unit restore
|
||
> (`POST /backup/restore` → `RestoreFromRecoveryUnit`), which replays volume tarballs and DB dumps.
|
||
> Routing customers there from the Tier-2 card is filed as **C9-F1b** — it puts a destructive
|
||
> operation behind a button reached via a non-destructive one, so the confirm copy must carry that
|
||
> difference. **C9-F4** is filed separately: nothing reads the Tier-2 copy's `recovery-unit/` mirror,
|
||
> so the second local copy that exists precisely for drive loss is unreachable by any customer action.
|
||
|
||
**Per-app Tier-2 config panel (v0.57.0)** — `GET/POST /stacks/{name}/backup`
|
||
(`internal/web/tier2_config_handler.go` + `templates/tier2_config.html`). The "2. mentés" row's
|
||
**Beállítás** button links here (was the dead-end deploy page). Shows the effective off-drive target
|
||
(pinned or auto), whether it's the size-limited internal SSD, the last-run reason, and lets the customer
|
||
**pin a registered drive** (off physical disk) or **toggle Tier 2 off**. Always visible — single-SSD apps
|
||
get the "csak DB/konfiguráció" note, non-HDD apps the "already in the PBS whole-guest snapshot" context.
|
||
Two preference fields on `CrossDriveBackup` — `UserDisabled` + `PreferredTarget` (set via
|
||
`Settings.SetTier2Preference`) — are **preserved across the runner's status writes** (`withTier2Prefs`):
|
||
`selectTier2Target` honors a valid pin before auto-picking; `RunTier2` skips a disabled app. The runner
|
||
re-validates the pin off-disk at run time. `Manager.Tier2Info(stackName)` is the read-only panel view
|
||
(effective target + eligible alternative drives).
|
||
|
||
**Phase 1 — Database Dumps** (`internal/backup/dbdump.go`, scheduled 02:30)
|
||
|
||
- **Auto-discovery** of PostgreSQL and MariaDB containers via `docker ps` + `docker inspect`
|
||
- Dumps via `docker exec pg_dump` / `docker exec mariadb-dump` with 5-minute timeout
|
||
- Dumps are written to the app's **home drive**: `AppDBDumpPath(appDrive, stackName)`
|
||
- Atomic writes (`.tmp` → `.sql`) to prevent corruption
|
||
- **Validation** after each dump: checks file size, header presence, counts `CREATE TABLE`
|
||
- Results cached in `settings.json` surviving container restarts
|
||
|
||
**Phase 1b — Docker Volume Dumps** (`internal/backup/backup.go` `runVolumeDumps`, part of the same run)
|
||
|
||
- **F3 (v0.99.0):** re-wired into the nightly/manual app-data backup run (`runDBDumpsInternal`) —
|
||
after the restic removal `DumpAppVolumesSafe` had no caller, so `volume-dumps/` was never produced.
|
||
Runs AFTER the DB dumps and BEFORE `captureAllRecoveryUnits` so the manifests enumerate fresh tars.
|
||
- Gate order (load-bearing): protected-stack (`cfg.IsProtectedStack`) and has-volumes
|
||
(`GetDockerVolumes()`) checks come BEFORE `DumpAppVolumesSafe` — the Safe variant stops the stack
|
||
before its own volume check, so unconditional calls would bounce every volume-less app nightly.
|
||
Disconnected/decommissioned drives skip with the same summary style as the DB loop.
|
||
- Each volume-bearing stack is stopped before dump, restarted after (`DumpAppVolumesSafe()`) —
|
||
prevents inconsistent tars of live databases.
|
||
- For each volume: `docker run --rm -v <vol>:/vol:ro -v <dumpDir>:/out alpine tar cf /out/<vol>.tar -C /vol .`
|
||
- 10-minute timeout per volume; a per-stack failure lands in the run summary as `FAIL <app> volumes:`,
|
||
flips the run's Success flag and fails the run (no silent partial) — other stacks still proceed
|
||
- Stale tars cleaned up (volumes that no longer exist)
|
||
- Volume names resolved with project prefix via `ResolveDockerVolumeNames()` (e.g., `mealie_mealie_data`)
|
||
- Dumps written to `AppVolumeDumpPath(nsRoot, stackName)`
|
||
|
||
**Phase 2 — Restic Snapshot** (`internal/backup/restic.go`, scheduled 03:00)
|
||
|
||
- Apps are **grouped by drive** via `groupStacksByDrive()` — each drive's apps are backed up to that drive's restic repo
|
||
- App drive resolution: `GetStackHDDPath()` (from `StackDataProvider`) → falls back to `SystemDataPath`
|
||
- Auto-generated repository password (32 random bytes, base64url), shared across all repos, synced to hub
|
||
- **Paths included in each per-drive snapshot (v0.34.0: per-app scoped):**
|
||
- Per-app DB dump dirs on that drive
|
||
- Per-app Docker volume dump dirs (`volume-dumps/*.tar`)
|
||
- Per-app HDD mount paths (user data)
|
||
- Per-app stack config dir (`<StacksDir>/<stackName>/` — only for stacks on this drive)
|
||
- `controller.yaml` — only on the system drive (not duplicated across all drives)
|
||
- Auto-detects and unlocks stale locks (restic repo lock)
|
||
- Weekly prune on Sundays with configurable retention (keep-daily, keep-weekly, keep-monthly)
|
||
- Weekly integrity check (`restic check`) on Sunday 04:00 — checks **all** primary repos
|
||
|
||
**Protects against:** accidental deletion, data corruption, point-in-time rollback.
|
||
Does NOT protect against drive failure (backup is on the same physical drive).
|
||
|
||
#### Tier 2: Cross-Drive Backup (opt-in, different device) (`internal/backup/crossdrive.go`)
|
||
|
||
**Complete backup** to a different physical drive. Available for **all apps** — apps with HDD
|
||
data back up config + DB + user data + Docker volumes; apps without HDD back up config + DB dumps + Docker volumes.
|
||
|
||
- **Auto-enable for small apps (v0.14.1):** Apps without HDD mounts (config-only, DB-only) are
|
||
automatically configured for daily rsync Tier 2 when ≥2 storage paths are registered.
|
||
`AutoEnableSmallApps()` runs at the start of each nightly backup cycle. Never overwrites
|
||
existing user-configured cross-drive settings (even disabled ones).
|
||
- **Infrastructure config backup (v0.14.1):** `syncInfraConfig()` rsyncs the stacks directory
|
||
and `controller.yaml` to `<dest>/backups/secondary/_infra/` on every secondary destination
|
||
drive. Runs before per-app backups. Cross-drive restic also includes infra paths.
|
||
- **Two methods:**
|
||
- **rsync** — Simple mirror with `--delete` (fast, no versioning, **browsable** on disk)
|
||
- **restic** — Versioned, deduplicated, encrypted (shared repo across apps, not browsable)
|
||
- Per-app configuration in settings.json: destination path, method, schedule (daily/weekly/manual)
|
||
- **Pre-backup DB dump:** `DumpStackDB()` runs fresh pg_dump/mariadb-dump before each cross-drive backup; non-fatal on failure (wired via `DBDumper` interface to avoid circular imports)
|
||
- **Pre-backup volume dump (v0.33.0, safe stop/start v0.34.0):** `DumpAppVolumesSafe()` stops the stack, exports Docker named volumes to tar, restarts — wired via `VolumeDumper` interface
|
||
- **Empty mounts allowed:** `RunAppBackup` accepts apps with no HDD mounts — the rsync
|
||
mount loop simply doesn't execute, but DB + config copy still runs
|
||
- **Drive-type-aware validation** (`ValidateDestination`):
|
||
|
||
| Destination type | Space checks |
|
||
|-----------------|--------------|
|
||
| External mount (different device than `/`) | Block if <100 MB free |
|
||
| System drive (same device as `/`) | Require ≥10 GB free AND <90% used; logged warning |
|
||
|
||
- **Secondary drive layout (v0.14.1):**
|
||
```
|
||
<dest-drive>/backups/secondary/
|
||
├── _infra/ ← infrastructure config mirror (v0.14.1)
|
||
│ ├── controller.yaml
|
||
│ └── stacks/ ← full stacks dir (all app configs)
|
||
├── <app>/rsync/ ← per-app rsync mirror
|
||
│ ├── _db/ ← DB dump files
|
||
│ ├── _config/ ← compose.yml, app.yaml, .felhom.yml
|
||
│ ├── _volumes/ ← Docker volume tars (v0.33.0)
|
||
│ └── <user data> ← HDD mount contents (if app has HDD data)
|
||
└── restic/ ← shared restic repo (all cross-drive apps)
|
||
```
|
||
- DB dump files read from **per-app home drive** path (`AppDBDumpPath`)
|
||
- `_` prefix directories prevent collision with user data
|
||
- For non-HDD apps, only `_db/`, `_config/`, and `_volumes/` (if applicable) are present (no user data directory)
|
||
- **Restic backup paths:** includes HDD mounts (if any) + config dir + per-app DB dump dir from home drive + stacks dir + controller.yaml (infra, v0.14.1)
|
||
- Safety guards: destination ≠ source, path-overlap check (HDD mounts only), writable check
|
||
- **Chained execution:** runs immediately after nightly restic — daily apps every night, weekly apps on Sundays
|
||
- **Hub reporting after manual triggers (v0.27.2):** `OnCrossDriveComplete` callback on Router pushes infra backup snapshot to Hub + writes local infra backup after both single-app and run-all manual triggers complete (previously only automatic scheduled runs reported)
|
||
- Per-app concurrency lock prevents overlapping runs
|
||
- Status (last_run, duration, size, error) persisted to settings.json
|
||
|
||
**Protects against:** primary drive failure, drive theft/damage.
|
||
|
||
#### Tier 3: Off-box (NAS) Backup — live
|
||
|
||
The off-site "1" of 3-2-1: each off-box-toggled app's recovery unit + DB dumps + volume tars are
|
||
backed up to the customer's NAS / Felhom offsite as an encrypted restic repo over SFTP (see the
|
||
off-box section below and `internal/backup/offbox.go`). The per-app "3. mentés" row on the backups
|
||
page renders one of four real states via the pure `tier3State` helper (`internal/web/backup_page_state.go`):
|
||
`unconfigured` (no target) / `off` (app not toggled) / `escrow_pending` (fork-4 key-escrow gate holds —
|
||
never a false success) / `active` (status badge + `restic → <host>` + relative last-run). Off-box run
|
||
status is repo-global (one `LastRun`); no per-app run time is fabricated.
|
||
|
||
#### Restore (`internal/backup/restore.go`)
|
||
|
||
Both **Tier 1** (restic) and **Tier 2** (rsync) restores are supported. All deployed apps
|
||
appear in the restore dropdown with per-app snapshot filtering.
|
||
|
||
| App type | Config restored | DB restored | User data restored | Docker volumes restored |
|
||
|----------|----------------|------------|-------------------|------------------------|
|
||
| Has HDD data | Yes | Yes | Yes (always) | Yes (if present) |
|
||
| Docker volumes, no HDD | Yes | Yes | n/a | Yes |
|
||
| DB only, no HDD/volumes | Yes | Yes | n/a | n/a |
|
||
| Config only | Yes | — | n/a | n/a |
|
||
|
||
**Snapshot API** (`GET /api/backup/snapshots?stack=<name>` — F1, v0.99.0):
|
||
- Backed by `backup.Manager.ListRestorePoints` (`internal/backup/restore_points.go`). The keep-side
|
||
restore has exactly **one** restore point per app — the current recovery unit — so the endpoint
|
||
returns at most one entry: `time` = newest artifact mtime (manifest / db-dumps / volume-dumps),
|
||
`short_id:"helyi"`, `tier:1`, `drive_label` from the storage registry (empty on the SSD fallback)
|
||
- **Never emits tier-2 entries**: Tier-2 copies are not restorable via `POST /backup/restore` (it
|
||
only reads the primary unit) — listing them would silently restore tier-1 data while claiming tier-2
|
||
- Guards: empty/traversal stack name → 400 (`validStackParam`), unknown stack → 404, known stack
|
||
with no unit yet → `ok:true, data:[]` (the UI shows "Nincs elérhető mentés")
|
||
- History: the route was a restic-era leftover fetched by the template but unregistered — the
|
||
dropdown could never populate and the restore button never enabled (drill finding F1)
|
||
|
||
**Restore type info** shown per-app when selected in dropdown (Hungarian banners):
|
||
- Has HDD or Docker volumes: "Teljes visszaallitas: adatbazis + konfiguracio + felhasznaloi adatok"
|
||
- Has DB, no user data: "Adatbazis es konfiguracio visszaallitasa"
|
||
- Config only: "Csak konfiguracio visszaallitasa"
|
||
|
||
**Tier 1 restore** (`RestoreApp`):
|
||
- Stop app → resolve app's home drive → `restic restore <id> --target / --include <path>...` → populate Docker volumes from restored tars → restart app → health check
|
||
- Restore paths: config dir, DB dump dir, volume dump dir, HDD mounts
|
||
- Docker volumes restored via `restoreDockerVolumes()`: `docker volume rm -f` → `docker volume create` → `docker run alpine tar xf`
|
||
|
||
**Tier 2 restore** (`RestoreAppFromTier2`):
|
||
- Stop app → rsync config from `_config/` → rsync HDD data (single/multi-mount) → copy DB dumps from `_db/` (streaming `copyFile`) → restore Docker volumes from `_volumes/` tars → restart app → health check
|
||
- Uses rsync `--delete` for config and HDD data to ensure exact mirror state
|
||
- Single-mount apps: data directly in rsync dir (excluding `_*`); multi-mount: per-leaf subdirectories
|
||
|
||
**Common:**
|
||
- **v0.34.0:** Post-restore health check (`waitForHealthy`) polls container state with `docker ps` refresh every 5s for up to 90s. Warning logged if app doesn't reach running state; restore still returns success (data is restored regardless).
|
||
- Running flag prevents concurrent backup/restore operations
|
||
- Snapshot ID validated (8-64 lowercase hex, or special `tier2-rsync`)
|
||
- Import from `.fab` bundle link shown in restore section for cross-system migration
|
||
|
||
#### Backup Page UI (`internal/web/templates/backups.html`)
|
||
|
||
Unified per-app status table with expandable rows showing **per-tier** backup status:
|
||
|
||
**Status dot per app:**
|
||
|
||
| Dot color | Meaning |
|
||
|-----------|---------|
|
||
| Green | 2+ tiers configured with successful backups + destination healthy |
|
||
| Yellow | Only 1 tier, or Tier 2 failing, or Tier 2 configured but never run, or destination disconnected/inactive |
|
||
| Red | Tier 2 destination blocked or inaccessible |
|
||
|
||
Every app starts as yellow (1 tier only). Green requires Tier 2 configured with successful backup.
|
||
|
||
**Per-app backup tiers (3 rows per app):**
|
||
- **1. mentes** (Tier 1, always present) — Auto badge + "helyi" + last run + contents (e.g., "DB + Konfig + Adatok")
|
||
- **2. mentes** (Tier 2, configurable for ALL apps) — one of:
|
||
- Configured: method (rsync/restic) + destination + schedule + last run + status + contents + browsable indicator (folder icon for rsync) + action buttons
|
||
- Not configured: "1. mentes auto" + "Nincs 2. masolat" + settings link
|
||
- **3. mentes** (Tier 3, off-box/NAS — live) — one of four `tier3State` states: `unconfigured`
|
||
("Nincs beallitva" + Beallitas link), `off` ("Kikapcsolva" + Bekapcsolas link), `escrow_pending`
|
||
("Kulcsletetre var"), `active` (status badge + "restic -> <host>" + relative last-run)
|
||
|
||
**Backup contents per app** (shown per tier):
|
||
- Apps with DB + HDD: "DB + Konfig + Adatok"
|
||
- Apps with Docker volumes (no HDD): "Konfig + DB + Adatok" or "Konfig + Adatok"
|
||
- Apps with DB only: "DB + Konfig"
|
||
- Apps with HDD, no DB: "Konfig + Adatok"
|
||
- Apps with neither: "Konfig"
|
||
|
||
**Deploy page** shows cross-drive (Tier 2) configuration form for **all deployed apps**,
|
||
not just those with HDD data. Non-HDD apps can configure destination, method, and schedule.
|
||
|
||
**Other sections:**
|
||
- Schedule overview with next run times for DB dump, restic, prune
|
||
- Snapshot history table (last 20 snapshots aggregated from all per-drive repos, sorted by time)
|
||
- Storage overview card (total size across repos, snapshot count, DB dump count/size, encryption key with show/copy)
|
||
- Restore section: app dropdown → per-app snapshot dropdown (Tier 1 + Tier 2 grouped) → restore type info → confirmation checkbox → execute → import from `.fab` bundle link
|
||
|
||
---
|
||
|
||
### 4. Storage Management
|
||
|
||
> **Off-box (NAS) backup — restic-over-SFTP (v0.93.0, Part B).** The "1 off-site" leg of 3-2-1 for the
|
||
> app-data tier: each off-box-toggled app's recovery unit + DB dumps + volume tars are backed up to the
|
||
> customer's NAS as an **encrypted restic repo over SFTP** (`internal/backup/offbox.go`). No kernel mount —
|
||
> restic talks SFTP directly; the NAS sees only ciphertext. Distinct from the local cross-drive rsync copy
|
||
> and the agent's PBS whole-CT DR.
|
||
> - **Fail-fast (load-bearing):** every restic call uses `-o sftp.command="ssh … -oConnectTimeout=10 … -s
|
||
> sftp"` so a dead NAS errors in ~10 s, never hangs the backup runner; a failure raises the `backup_failed`
|
||
> operator alert. `-oStrictHostKeyChecking=yes` + a pinned `known_hosts` (no blind TOFU).
|
||
> - init-if-absent (idempotent), `forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune`,
|
||
> single-flight (shares `m.running`) + migration-guard, restic's own exit code checked, restore to a
|
||
> scratch dir (non-destructive).
|
||
> - **Unit discovery (v0.104.0) — durable, deployment-independent.** Each toggled app's recovery unit is
|
||
> **discovered** by scanning the durable storage registry — every registered *schedulable* storage path
|
||
> (`GetSchedulableStoragePaths`) ∪ the system-data fallback drive — for `backups/primary/<app>`, rather
|
||
> than *inferring* the drive from the app's live `app.yaml` `HDD_PATH` (which silently fell back to
|
||
> `systemDataPath` for a toggled-but-undeployed app → offbox looked on the wrong drive, backed up
|
||
> nothing, and reported `ok`/0). If the same app's unit exists on two drives (drive churn), the **newest
|
||
> by manifest `CreatedAt`** is used and the stale one is WARN-logged. The WRITE side (`CaptureRecoveryUnit`
|
||
> / dumps) is unchanged — this only changes offbox's read/discovery path. Boundary: decommissioned or
|
||
> non-schedulable drives are not searched.
|
||
> - **No silent success (v0.104.0):** a run where ≥1 app is toggled but **0** were backed up (no unit found
|
||
> anywhere) is a **hard error** → `LastStatus="error"` + operator alert (was a misleading `ok`/0 snapshots).
|
||
> A *partial* run (some units missing) stays `ok` but sets a Hungarian **`LastWarning`** naming the skipped
|
||
> apps, shown on `/backups`.
|
||
> - **Crash-lock self-heal (v0.110.0).** A crash mid-prune leaves a restic EXCLUSIVE lock that plain
|
||
> `restic unlock` can't clear (the recreated container's new hostname stops restic proving the dead PID
|
||
> stale for ~30 min). Every backup/prune/restore step runs through `resticStep`, which on a lock error
|
||
> escalates to `unlock --remove-all` + one retry — safe because the repo has a SINGLE legitimate writer
|
||
> (per-customer sub-account isolation + the in-process single-flight mutex). **Boundary:** a DR-cloned
|
||
> SECOND controller writing the same repo would defeat this premise — operator-supervised territory, out of
|
||
> scope for the auto-heal. A crash mid-run also flips a persisted `LastStatus="running"` to a truthful
|
||
> error on the next startup (self-corrects on the next successful run).
|
||
> - **Secrets** (SSH key + auto-gen repo password) are **0600 files in the data dir** — never logged/committed.
|
||
> - **Password custody + atomicity (v0.105.0, fork-4; pairs with agent v0.77.0).** The repo password is the
|
||
> irreplaceable DATA key for the offsite tier, so it rides the **customer-recovery-code (R) escrow**
|
||
> (age-under-R, in the agent's `IdentityBundle` — custody spike validated a recovered password opens the
|
||
> real repo). **Atomicity gate:** enabling offsite pushes the password to the agent (`StageEscrowSecret` →
|
||
> `POST /escrow/stage-secret`) and marks `EscrowState="pending"`; **no offsite backup runs until the escrow
|
||
> is confirmed** (`OffboxRunnable()`), so an un-recoverable offsite copy can never exist.
|
||
> **Ceremony — PRIMARY path (v0.127.0, agent ≥ v0.88.0): the customer wizard** at `/backup/escrow`
|
||
> (`web/escrow_handlers.go` + `templates/backups_escrow.html`): preflight (agent
|
||
> `GET /escrow/preflight` + version gate) → warnings (re-ceremony adds the supersede copy) →
|
||
> password re-auth (login rate limiter) → **re-stage-first** (offbox configured →
|
||
> `PushOffboxPasswordForEscrow`; a staging failure ABORTS the start — no hash-less blob, ever) →
|
||
> agent job (`POST /escrow/ceremony`, poll `GET /escrow/ceremony/status` @2 s) → **one-shot R
|
||
> reveal** (`POST /escrow/ceremony/claim`, `Cache-Control: no-store`; R exists only in the page's
|
||
> JS scope; 10-min unclaimed TTL → void, re-run supersedes) → typed-back (two random words,
|
||
> client-side) → finish. R is never templated/logged/persisted on either side.
|
||
> **Operator fallback (CLI, unchanged text mode):** enable offsite (→ pending) → run
|
||
> `felhom-agent --selftest=escrow-create --upload` (the staged restic password auto-injected,
|
||
> then wiped) → hand the customer the fresh **R** (once; supersedes any prior code) → the
|
||
> auto-confirm flips escrowed hands-free. (The manual "Letét megerősítése" button is GONE from
|
||
> the card; the deprecated endpoint remains for legacy hash-less blobs.)
|
||
> **DR:** recover R → the escrow yields the password → `POST /backup/offbox/inject-password {password}`
|
||
> pre-places it 0600 → configure offbox → restore. The SFTP access key is **regenerated** at DR (a fresh
|
||
> sub-account key), NOT escrowed; the DR recipe carries only the non-secret `offsite_restic`
|
||
> coordinates (`DRResticCoord`).
|
||
> - **Hub-verified auto-confirm (v0.108.0, SLICE 3; pairs with agent v0.79.0 + hub v0.40.0).** The report
|
||
> ACK carries `escrow:{identity_blob_present, restic_pw_sha256, created_at}` (the hash is recorded at
|
||
> ceremony time — non-reversible sha256 of the sealed password, safe to serve). `report.EscrowAutoConfirmer`
|
||
> flips `EscrowState` pending→escrowed ONLY when `sha256(local repo_password)` **matches** — i.e. the
|
||
> stored escrow provably covers the CURRENT key; blob-presence alone never confirms (a stale blob would
|
||
> re-open the fork-4 gap). Mismatch → stays pending + a loud warn naming the ceremony (deduped per hash);
|
||
> no hash / no row / no local file → silently pending; **never un-confirms**. On flip it wipes the
|
||
> agent-staged secret. The canonical hasher (`backup.HashResticPassword`, trimmed-string sha256) is pinned
|
||
> by a cross-repo test vector against the agent's. The manual `POST /backup/offbox/confirm-escrow` is a
|
||
> **deprecated fallback** for legacy hash-less blobs (e.g. the demo's).
|
||
> **Stale-blob re-check (v0.127.0, Scenario F):** an ESCROWED box re-compares the ACK hash every
|
||
> cycle — mismatch OR a present blob with an EMPTY hash (a superseding ceremony that missed the
|
||
> staged secret) sets an in-memory stale flag (`EscrowAutoConfirmer.StaleBlob` → the Távoli
|
||
> mentés card's warning + re-ceremony CTA) + one warn per distinct hub hash. State never flips,
|
||
> runs never block; a covering blob (or a fresh auto-confirm) clears it.
|
||
> **Awaiting-confirmation card (v0.138.0):** the flip above lands on the *next* report ACK, so a
|
||
> just-finished ceremony sits pending for up to ~15 min. To avoid re-showing the yellow
|
||
> "Helyreállítási kód szükséges" card during that gap, a successful recovery-code **claim** stamps
|
||
> `OffboxTarget.CeremonyCompletedAt` (RFC3339, persisted; zeroed on the flip). While stamped and
|
||
> pending, `/backups/remote` shows an **info** "megerősítésre vár, legfeljebb 15 perc" card
|
||
> (`offboxCeremonyWaitState`); past `escrowCeremonyGraceWindow` (35 min = 2 cycles + slack) it
|
||
> degrades to a **warn** "a megerősítés nem érkezett meg" + re-ceremony CTA — never an indefinite
|
||
> wait. The wizard's final step shows a matching "Mi történik ezután?" note. Display-only: no state
|
||
> change, no run-gate effect. (Phase-0 A: the wait itself is correct; only the feedback was missing.)
|
||
> - **Injection guard** (`ValidateOffboxTarget`): host/user/repo must not start with `-` (ssh
|
||
> option-injection) or carry metacharacters/traversal; `OffboxConfigured` fails closed on an invalid
|
||
> target. Image: `restic` + `openssh-client` (re-added; restic's sftp backend shells out to `ssh`).
|
||
> - UI: the "Külső (NAS) mentés" section on the backups page (configure target, per-app toggles, run-now,
|
||
> restore, status). Config: `settings.OffboxTarget` + per-app `AppBackupPrefs.Offbox`. Daily at 04:15.
|
||
> - **Soft quota (v0.109.x, SLICE 4; pairs with hub v0.41.0).** The shared-model quota (`quota_gb`) rides
|
||
> the descriptor into `OffboxTarget.QuotaGB` (0 = no limit — dedicated boxes are Hetzner-enforced; the
|
||
> hash includes it, so a hub-side quota change re-applies via key-auth-first, no password consumed).
|
||
> `RepoSizeBytes` persists from `restic stats` (last-known on failure — stale-but-safe). Pre-run gate:
|
||
> **≥100% refuses NEW backup runs** (Hungarian error + operator alert) but the **prune step still runs**
|
||
> (the only way back under quota) and **restore is never gated**; ≥80% sets a Hungarian usage warning.
|
||
> `/backups` shows a usage bar when quota>0. The hub report carries
|
||
> `offsite:{enabled, escrow_state, last_run, last_status, snapshot_count, repo_size_bytes, quota_gb}` —
|
||
> the hub's OffsiteChecker alerts on fill (90/95%) and staleness (escrowed + no run >48h); the Hetzner
|
||
> `readonly` freeze is an OPERATOR lever on the hub (never automatic). A RE-apply preserves the existing
|
||
> target's EscrowState + runtime status (v0.109.1 — custody tracks the preserved repo password).
|
||
> - **Hub-driven provisioning apply-bridge (v0.106.x, SLICE 2; pairs with hub v0.38.x — validated live
|
||
> 2026-07-09).** When the hub provisions the offsite tier (a Hetzner Storage Box sub-account or dedicated
|
||
> box), the served `controller.yaml` gains an `offsite:` section (host/user/port/repo_path/quota_gb +
|
||
> **`host_fingerprint`**) and `internal/offsiteapply.Bridge` reconciles it at startup: scan + **verify the
|
||
> box host key against `host_fingerprint` (no blind TOFU)** → generate an ed25519 keypair → **consume the
|
||
> hub's one-time transient password** (`POST /api/v1/offsite/consume-password/{id}`, single-use) → install
|
||
> the pubkey (`sshpass -e ssh-copy-id -p 23 -s -f`, **pinned known_hosts, `StrictHostKeyChecking=yes`**;
|
||
> the installer ensures `~/.ssh` exists — ssh-copy-id's SFTP mode needs it) → verify key-only auth →
|
||
> configure the offbox target → `EscrowState="pending"` (the fork-4 gate above still holds) → persist a
|
||
> descriptor-hash marker LAST. **Idempotent** (marker → no re-consume) and **fail-safe** (any failure →
|
||
> nothing persisted, retried on the next config refresh/restart; a consumed-but-failed install logs a loud
|
||
> "password is spent — reset on the hub"). The 15-min hub-report → config-refresh cycle is the trigger, so
|
||
> descriptor-to-applied latency is ≤ ~15 min.
|
||
> - **Key-auth-first (v0.107.0):** on a descriptor change, if the ALREADY-INSTALLED key still authenticates
|
||
> (`SFTPKeyAuthProber`, pinned to the freshly-verified host key — the probe never bypasses the fingerprint
|
||
> verify), the bridge re-pins + reconfigures WITHOUT consuming a one-time password. Fresh guests fall
|
||
> through to the full consume+install path. The hub's "Re-issue offsite credentials" (v0.39.0) is the
|
||
> recovery for a genuinely-spent password on a fresh guest.
|
||
> - **Staged-secret wipe (v0.107.0):** confirm-escrow calls the agent's `DELETE /escrow/stage-secret`
|
||
> (agent ≥ v0.78.0) whenever `EscrowState` flips to `escrowed` — best-effort, loud-logged on failure.
|
||
> - **Settle-gate (R-71a, v0.162.0) — the day-0 race removed.** The apply-bridge runs BEHIND a
|
||
> settle-gate (`Bridge.AwaitSettle` → `ReconcileWhenSettled`): before the consume/install path it
|
||
> polls the self-updater's own state via the `SettleProvider` seam (a `SettleFunc` adapter over
|
||
> `updater.GetFloor()`/`IsUpdateRunning()` in main.go — no second floor-fetch path). While a managed
|
||
> update is running OR the box is below the operator floor (an auto-floor update is imminent), the
|
||
> gate WAITS rather than consume the single-use password — the update's restart would otherwise kill
|
||
> the bridge mid-install and burn it (the F10 day-0 shape). At/above floor with no update in flight,
|
||
> it GOes on the first poll with zero added latency (B′). Bounds: 10 s poll, 90 s floor-knowledge
|
||
> sub-bound (sized to the ~5–10 s report-ACK floor latency; the floor is in-memory, not persisted,
|
||
> so it is unknown until the first ACK on any restart), 5 min overall — both bounds GO+WARN and lean
|
||
> on the R-71c hub self-heal as the belt (a hub that cannot serve a floor cannot serve a consume, so
|
||
> proceeding never burns a password). The gate is wired only when a self-updater exists (no updater
|
||
> → no floor-update to race → reconcile immediately). Ordering-only: the consume/install/persist
|
||
> internals and the 404-no-oracle contract are untouched. Three-layer defense: the v1.25.0
|
||
> golden≥floor build gate PREVENTS the trigger, (a) DEFERS it, R-71c HEALS a burn.
|
||
|
||
> **NAS network storage (v0.92.0, Part A2; pairs with agent v0.50.0).** A customer NAS share (NFS or SMB)
|
||
> is a **distinct storage KIND** from a physical drive (`StoragePath.Kind == "network"`), for **bulk media**.
|
||
> The controller is a thin proxy over the agent's `/netstorage/*` (A1) + the local registry — it holds **no
|
||
> mount authority** and **never persists the SMB password** (it passes the credential to the agent's add
|
||
> request, which writes the 0600 file). Endpoints `POST /api/storage/netstorage/{add,remove}` +
|
||
> `GET /api/storage/netstorage` (merges the agent's per-share liveness: `ok` | `idle` (benign idle-unmount)
|
||
> | `unreachable` | `unknown`). A network share registers as `/mnt/felhom-drives/<name>`, is **selectable as
|
||
> a media app's `HDD_PATH`** (Schedulable), and shows in the "Hálózati tárhely (NAS)" settings section.
|
||
> - **NOT a drive:** the drive lifecycle (eject/decommission/migrate/wipe/SMART) is **refused** on a network
|
||
> path server-side (`refuseNetworkLifecycle`). The **drive-absent gate** (`planDriveGates`) and the
|
||
> **missing-storage** surface **skip network paths**, so an `unreachable` NAS is a **recoverable warning**
|
||
> (`networkStorageWarnings` → a distinct app-card badge), **never** the drive "missing → stopped" cascade.
|
||
> - **Verify-before-commit (v0.113.0, pairs with agent v0.81.0; SPIKE-nas-verify-2026-07-11):** add no
|
||
> longer registers blind. `POST /api/storage/netstorage/add` returns `{started:true}` and a DETACHED
|
||
> single-flight orchestration job (`internal/web/netstorage_job.go`, migrate.go shape; polled on
|
||
> `GET /api/storage/netstorage/add/status`) drives `agent_add → verifying → probing → registering`:
|
||
> the agent installs the units + runs its own detached mount-verify (journal-classified, agent-side
|
||
> auto-rollback), then the controller re-execs itself as **`--netprobe <dir>` at uid/gid 1000**
|
||
> (`SysProcAttr.Credential`, `netprobe*.go`) to prove a media app can WRITE through the share (the
|
||
> squash trap), and only then registers. ANY failure = full rollback (nothing registered, nothing
|
||
> installed, no creds file); categorized Hungarian errors (`netAddMessage`, §3.2 map — note
|
||
> `nfs_export` merges not-found/not-permitted: NFSv4 returns identical strings). An agent restart
|
||
> mid-verify (`verify-status` phase `none`) ⇒ controller rollback; any agent-configured share NOT in
|
||
> the registry surfaces as a remove-only **"Árva megosztás"** orphan row. The page
|
||
> (`storage_network.html`) is on the canonical form pattern with staged poll progress and the
|
||
> protocol-honest NAS guidance (SMB-first; NFS map-all-users vs full-fidelity `anonuid=<uid+100000>`).
|
||
> Authoritative doc: `felhom.eu/documentation/controller/network-storage-nas.md`.
|
||
> - **Agent-capability gate (v0.114.0):** the verify-before-commit add is COUPLED to agent ≥ v0.81.0,
|
||
> so the add entry point now probes the agent first (`internal/agentapi/features.go`,
|
||
> `Supports(FeatureNetstorageVerify)` — a route probe on `GET /netstorage/verify-status`: 2xx ⇒
|
||
> supported, typed 404 ⇒ older agent, transport/5xx ⇒ indeterminate, cached 5 min both polarities).
|
||
> On an older agent the add is refused SYNCHRONOUSLY (HTTP 412, machine code `agent_outdated`,
|
||
> honest Hungarian message) BEFORE the single-flight claim — never a misleading mid-pipeline
|
||
> rollback; indeterminate NEVER refuses (a down agent speaks through the existing error paths). The
|
||
> settings page swaps the add form for a banner on `SupportNo` (share list + remove stay usable in
|
||
> every state). Convention: every future coupled feature adds a `featureProbes` row + a gate call at
|
||
> its entry point and declares `MinAgent` in its CHANGELOG header — see
|
||
> `felhom.eu/documentation/runbooks/publish-train-rules.md`.
|
||
> - **Consuming-namespace verification (v0.117.0, RCA AUDIT-nas-cwa-rca-2026-07-11 fix 2):** a guest
|
||
> reboot silently replaces an idle NAS trigger with a plain local STUB dir in the app namespace while
|
||
> host-side (agent) health stays green — so the controller now verifies where the apps consume:
|
||
> `internal/system/fsclass*.go` classifies a path by **statfs f_type** in THIS process's namespace
|
||
> (`network` nfs/cifs/smb2 | `autofs` idle trigger = HEALTHY, never force-mounted | `stub` | `unknown`
|
||
> = fail-open). Three consumers: (1) the `--netprobe` child now REQUIRES a mounted network fs after
|
||
> its create (exit 5 → category `not_network_fs`, full rollback — a writable stub can never verify);
|
||
> (2) `POST /api/stacks/{name}/deploy` refuses (409) a registered network `HDD_PATH` that classifies
|
||
> as a stub (`refuseNetworkStubDeploy`; idle autofs deploys fine — first app access mounts it);
|
||
> (3) the dashboard/stacks app cards gain a distinct **stub badge** ("Hálózati tárhely hibás — az
|
||
> alkalmazás nem a NAS-t látja", `NetworkStubs`) that WINS over the recoverable `unreachable` badge.
|
||
> The agent pair (v0.84.0 `ReassertNetworkMounts`) re-arms triggers on guest start; this layer is the
|
||
> detection net beneath it.
|
||
> - **One classification, two surfaces (v0.119.0, CAMPAIGN-3 F8):** the **share row** on the
|
||
> `/storage/network` page now reads the SAME classifier. Its health used to come only from the agent's
|
||
> server-level TCP dial (`server:2049/445`), which stays green when a *single* export is `exportfs -u`'d
|
||
> — so the row showed benign "Készenlét" while the stacks cards showed the stub. `fuseNetHealth`
|
||
> (`netstorage_handlers.go`) fuses the agent view with `classifyFSPath(Where)`: a new **`stub`** health
|
||
> (badge "Hibás — az alkalmazások nem a NAS-t látják") overrides idle/ok when the namespace sees local
|
||
> disk; a whole-server `unreachable` still wins over stub; autofs-healthy / network / `unknown` leave
|
||
> the agent health intact (never force-mount an idle trigger). The row and the stacks badge derive from
|
||
> ONE classifier and can never contradict.
|
||
> - **mapped_uid validated at the door (v0.119.0, F8's sibling F4):** `handleNetStorageAdd` range-checks
|
||
> the container uid/gid (**1..65533**; the guest maps `<uid>`→`<uid>+100000`, so 65534=nobody and a
|
||
> host-side value like 101000 must not be entered) after the `<=0` default — out of range → a friendly
|
||
> Hungarian 400, nothing installed (previously a raw `agent_error` from the agent).
|
||
> - **Deploy view truth (v0.117.0, RCA fix 4):** a deployed app's read-only storage select now marks
|
||
> `selected` by the app's STORED `HDD_PATH` (extra disabled `<path> (nem elérhető)` option when the
|
||
> stored path left the schedulable list) — `IsDefault` selects only for NEW deploys. Pre-fix the view
|
||
> showed the default drive regardless of app.yaml (the RCA's S-C symptom).
|
||
> - **Limits (v1):** a share's `+100000` uid mapping is fixed at add-time (one app / same-uid apps); for
|
||
> write apps on a soft NFS mount, an in-flight file can truncate if the NAS vanishes mid-write (prefer
|
||
> atomic-write apps / SSD-staging).
|
||
|
||
> **⚠️ INTERMEDIARY-MOUNT model (v0.67.x, pairs with agent v0.35.x).** External data drives are now
|
||
> visible in the guest at a STABLE path **`/mnt/felhom-drives/<name>`**, NOT the raw `/mnt/<name>`. The
|
||
> agent keeps a single permanent parent bind `/mnt/felhom-drives` in the guest and swaps each drive's
|
||
> `felhom-data` namespace **underneath it host-side** (`mount --bind`), so the guest sees attach/detach
|
||
> **live with no reboot** (mount propagation), the bind source never disappears (C1-immune), and only
|
||
> `felhom-data` crosses in (confinement). The per-drive `pct set -mpN` bind is **deprecated**.
|
||
> - The registered storage path + every app's **`HDD_PATH`** + the FileBrowser source = the stable
|
||
> `/mnt/felhom-drives/<name>`. The controller maps it back to the raw `/mnt/<name>` (`agentWhere()`)
|
||
> only for agent calls (assign/attach/eject/decommission). `GET /api/disks` carries `guest_path` +
|
||
> `bound_under_parent` (the guest-visible signal).
|
||
> - **Drive-absent gate** (`internal/web/intermediary.go`, `driveGateLoop` 30s): an absent
|
||
> `/mnt/felhom-drives/<name>` drive stops + blocks its apps (`StoppedStacks` = the gate-stopped set);
|
||
> a returned drive re-attaches + auto-restarts them; `actionStack` refuses to start an app whose drive
|
||
> is absent ("tárhely nem elérhető"). SSD/system paths are never gated.
|
||
> - **H1 endpoints** `POST /api/storage/{disconnect,reconnect,restart-apps}` drive the host-side
|
||
> eject/reconnect (no guest reboot).
|
||
> - **Lifecycle (v0.68.x):** a "Leszerelés" button decommissions a drive (migrate-then-decommission OR
|
||
> decommission-anyway type-to-confirm — non-destructive, never touches the parent mp); a
|
||
> "Visszacsatlakoztatás" button one-click re-enrolls a decommissioned/ejected drive (clears the marker,
|
||
> re-binds under the parent, restarts gate-stopped apps). Decommissioning the DEFAULT auto-promotes
|
||
> another schedulable drive (`defaultPromotionTarget`), or BLOCKS if it's the only one. Eject and
|
||
> decommission keep the RAW drive mounted (logical retire) so re-enroll re-binds it.
|
||
> - **Guest-reboot convergence is DETERMINISTIC** via the agent's `guest_boot_id`: the controller persists
|
||
> `LastGuestBootID` and, when it changes, recreates EVERY deployed drive-backed app onto the
|
||
> re-propagated drive (`processGuestBootChange` — no fragile container-uptime sampling).
|
||
> **v0.71.0 — boot-race fix:** on a guest `pct reboot`, in-guest dockerd auto-starts the apps ~18s
|
||
> BEFORE the agent re-binds the drive, so their volume bind fails at create-time
|
||
> (`mkdir …/userdata: permission denied`, `RestartCount=0` → never retried → stuck `Exited`). The old
|
||
> recovery sampled the agent's `BoundUnderParent` ONCE, raced that rebind, recreated nothing, and burned
|
||
> its boot-id one-shot. `processGuestBootChange` now **gates on the REAL live in-guest bind**
|
||
> (`driveBindLive`: is `/mnt/felhom-drives/<drive>` an actual mountpoint in the controller's own `/mnt`
|
||
> rslave `/proc/self/mountinfo`?) and **waits** for it (`pollLiveBinds`, bounded ~120s) before recreating
|
||
> — including apps stuck `Exited` with a create-time mount failure (`shouldRecreateOnBoot` is
|
||
> state-independent). The **guest-only reboot path** (which the host-reboot sweep never exercised) is now
|
||
> covered; drives that never go live in the window are left to the drive-absent gate. `processGuestBootChange`
|
||
> also runs on every periodic `driveGateLoop` tick now (idempotent, boot-id gated) so a momentarily-unreachable
|
||
> agent right after a guest reboot no longer permanently strands recovery.
|
||
> **v0.157.0 — R-55, the gate now honours a customer's Stop.** Until this version the recreate keyed on
|
||
> `Deployed && HDD_PATH && drive-present` alone, so a drive-backed app the customer had deliberately
|
||
> Stopped was silently restarted on every guest reboot (proven live: immich, stopped from the UI seconds
|
||
> earlier, came back running). `shouldRecreateOnBoot` now also requires the app to still HAVE containers
|
||
> (`len(Stack.Containers) > 0`, from `docker ps -a`, so `Exited` ones count) — R-52's `existing-Exited vs
|
||
> absent` distinction (`bootrecon.isBootOrphan`) translated to this gate. A UI Stop is `compose down`,
|
||
> which REMOVES the containers; a guest that went down under a running app leaves them present. **State is
|
||
> still NOT a filter** — that part of the original design is load-bearing and unchanged; `hasContainers`
|
||
> answers a different question ("does docker still have records of it") which, unlike liveness, survives a
|
||
> reboot as a statement of intent. The evidence is sampled BEFORE any recreate, because recreate's own
|
||
> `StopStack` erases it. Apps stopped by the drive-absent gate are also at zero containers and are likewise
|
||
> left alone here — they are restored by `ReconcileDriveGates`' `Return` branch from
|
||
> `StoragePath.StoppedStacks`, on the same loop tick. Honoured Stops are logged at INFO (`left stopped …`),
|
||
> counted separately from the "no live bind" skips so an intended outcome never fires a WARN.
|
||
> **Agent-path prerequisite (also v0.71.0):** the whole drive gate needs `cfg.LocalAPI.Endpoint` (the
|
||
> per-guest agent local API). `bootstrap.MaybeIngest` now calls `ensureLocalAPI` on the already-configured
|
||
> path — merging `local_api` from `bootstrap.json` into an existing controller.yaml that lacks it (seeded
|
||
> before `local_api` existed) — because without it `agentClient()` returns "agent not configured" and the
|
||
> entire gate + boot recovery silently die.
|
||
> **v0.72.0 — FileBrowser convergence on boot-recreate:** FileBrowser is base-infra (it binds each
|
||
> drive's `userdata` but has no `HDD_PATH`, so it is NOT in the drive-backed recreate set) — after a
|
||
> host reboot its mounts could be stale (the early first-boot bring-up bound them before the drives went
|
||
> live). `processGuestBootChange` now, **after** `pollLiveBinds` confirms the binds and the apps are
|
||
> recreated, triggers `go s.SyncFileBrowserMounts()` so FileBrowser converges against the now-live drives.
|
||
> The sync runs unconditionally (FileBrowser reflects the current bind state even if no app needed
|
||
> recreating). The recreate loop is a pure `recreateDriveBackedApps(stacks, present, recreate, syncFB)`
|
||
> that calls `syncFB` exactly once, after every recreate. Live-accepted over two real `felhom-pve`
|
||
> reboots (FileBrowser non-stale, all drive-backed apps recovered, agent tolerated a `/dev/sdX` swap by UUID).
|
||
>
|
||
> **⚠️ Rebuilt on the agent-delegated disk model (v0.43.0), made ROLE-AWARE in v0.44.0, UX-polished in
|
||
> v0.45.0.** After the 8C
|
||
> de-privileging, the controller holds **no Proxmox/disk credentials and no destructive authority** — disk
|
||
> execution + the gate live entirely in the **host agent**. The drive UI is driven by the agent's
|
||
> authoritative **role** (`system` | `backup` | `user-data`, from `GET /api/disks`): the appliance's own
|
||
> system storage and the backup safety-net are visibly **protected** (lock badge, NO destructive controls);
|
||
> the customer manages their own **user-data** drives with informed consent. The agent re-enforces role at
|
||
> wipe time — the UI lockout is defense-in-depth, not the gate.
|
||
> - **Overview** (`settings.html` ← `GET /api/disks`): styled **cards** (not a table) — name, mono
|
||
> device/mount, badges for class (gyors/lassú), data (`Adatot tartalmaz`), **role** (🔒 Rendszer / 🔒
|
||
> Biztonsági mentés — védett / Felhasználói adat) and registered state, plus a **capacity bar** (the
|
||
> monitoring `system-bar`, from the agent's `total_bytes`/`used_bytes`). Eject/Wipe render **only** for
|
||
> user-data drives mounted under `/mnt`.
|
||
> - **(v0.45.0) Deterministic order** — `agentDisksListHandler` sorts the list server-side
|
||
> (`sortDisksForView`): **user-data → system → backup** (then unrecognized), alpha by name within a
|
||
> tier, so it no longer reorders on each reload (the agent's view iterates an unordered Go map).
|
||
> - **(v0.45.0) Purpose + app-backing clarity (B4)** — `local` and `local-lvm` are both shown (not
|
||
> collapsed); each card carries a plain-Hungarian **purpose description** keyed on the agent's
|
||
> role/type, the app-backing storages are tagged (`local-lvm` → "Alkalmazás-rendszer"; user-data →
|
||
> "Alkalmazás-adatok"), and a one-line tiering note above the list answers "which storage do the
|
||
> apps use?". Pure presentation — role/type stay authoritative from the agent.
|
||
> - **(v0.45.0) Register shortcut (B3)** — a mounted, **unregistered** user-data drive offers
|
||
> **Regisztrálás** as its PRIMARY action: `POST /api/storage/register` → `registerStoragePath` records
|
||
> the existing mount (no format, no eject) + FileBrowser-syncs. Leválasztás/Törlés stay secondary.
|
||
> - **Customer wipe/eject** — a **type-to-confirm** modal that names the deployed apps that break
|
||
> (`GET /api/storage/impact` → `appsUsingPath`) and disables the destructive button until the **mount
|
||
> name is typed exactly**. Wipe (`POST /api/storage/wipe`): eject (unmount + deregister) → server-side
|
||
> two-step customer-confirmed format (learn the agent's durable id, then re-submit `confirmed:true` bound
|
||
> to it). The agent refuses a protected device regardless of what the controller sends.
|
||
> - **Guided init** (`/storage/init`, `POST /api/storage/init`, `web/storage_handlers.go`): format
|
||
> → resolve the new fs UUID → `assign` → register. The selector lists **only user-data** targets. A
|
||
> data-bearing user-data device now uses the **customer-confirmation** flow (type-to-confirm → re-submit
|
||
> `confirmed:true` + durable id), NOT the `felhom-opsign` command. The opsign surface remains a fallback
|
||
> only if a protected device somehow reaches init.
|
||
> **F6 (v0.141.0):** the chain runs as a DETACHED single-flight job (`web/storage_init_job.go`,
|
||
> `context.Background()`) the wizard polls via `GET /api/storage/init/status` (3-step progress:
|
||
> formatting → mounting → registering) — a closed tab / lost connection no longer aborts the
|
||
> post-mkfs mount+register. `register` is the LAST step (marker-last crash-safety). A slow mkfs that
|
||
> outruns the agentapi client's 15 s timeout is followed by polling the agent's
|
||
> `GET /disks/format/status` (`agentapi.Client.FormatStatus` → `awaitAgentFormat`) before
|
||
> continuing. Live-validated on a 64 GB USB (mkfs ~27 s → done, mounted+registered at
|
||
> `/mnt/felhom-drives/scratch1`).
|
||
> - **Guided attach** (`/settings/storage/attach`, `POST /api/storage/attach`): non-destructive — resolve
|
||
> the existing fs UUID → `assign` → register. Selector restyled to cards (user-data only).
|
||
> - **Eject** (`POST /api/storage/eject`): benign unmount + deregister, with the agent's dependent-guest
|
||
> warning + the affected-app list (parity with wipe). **The eject is ROLE-GATED at the agent** (felhom-
|
||
> agent v0.24.0): `POST /disks/eject` refuses to unmount a system/backup mount — the UI hiding the button
|
||
> is defense-in-depth, not the control. Only user-data mounts are ejectable.
|
||
> - **`agentapi`** (`internal/agentapi`) is the pinned client to the agent local API: `Disks`/`AssignDisk`/
|
||
> `EjectDisk`/`FormatDisk(…, confirmed, durableID)`; `DiskInfo.role`+capacity;
|
||
> `FormatResult.{role,needs_confirmation,durable_id}`; `ErrNeedsConfirmation` (user-data) vs
|
||
> `ErrFormatRefused` (system/backup). `FormatResult.PendingOp.OpsignCommand()` for the operator path.
|
||
> - **(v0.74.0) Client lifecycle — ONE shared client, reused.** `Server.agentClient()` builds the
|
||
> `agentapi.Client` once (memoized via `sync.Once`) and returns the same instance to all ~19 call
|
||
> sites; the `http.Transport` is bounded + expiring (`MaxIdleConnsPerHost:2`, `IdleConnTimeout:90s`).
|
||
> This replaced a per-call `agentapi.New(...)` that leaked one idle ESTABLISHED socket per call and
|
||
> exhausted the ephemeral source-port range to the agent's `:8443` after ~5 days (EADDRNOTAVAIL).
|
||
> Safe because `cfg.LocalAPI` is static per process (a config-apply triggers a graceful self-restart).
|
||
> - **(v0.90.0) Channel health-check — `internal/channelhealth`.** A ~60s scheduler job
|
||
> (`agent-channel-health`) probes the channel via `Server.ProbeAgentChannel` (the SAME memoized
|
||
> client + `GET /storage` — not a fresh client) and classifies failures into
|
||
> `up | down:<reason>` (pin_mismatch / unauthorized / unreachable / timeout / misconfigured /
|
||
> construction_error). Transient reasons (refused/timeout) are **debounced** (N≥2 consecutive)
|
||
> so a clean agent restart's ~1s blip doesn't page; pin/401/DNS/construction alert on the first
|
||
> down. On a transition it fires an English **operator-only** event
|
||
> (`Notifier.NotifyAgentChannelDown/Recovered`, hub 1h cooldown) and sets a Hungarian dashboard
|
||
> banner (`AlertManager.SetAgentChannelAlert`). The first observation seeds state silently. This is
|
||
> the controller half of the self-health story (the agent watches its own privileged capabilities;
|
||
> the controller watches its link to the agent). A construction error (`agentClient()` can't build
|
||
> — a latching config fault) is surfaced distinctly. Detection/surfacing only — it never touches the
|
||
> pin, transport, or gate.
|
||
> - The **`StoragePath` registry** (`settings.go`: `AddStoragePath`/default/schedulable/label) is unchanged.
|
||
> - **(v0.64.0) `AutoDiscoverStoragePaths` is now ADDITIVE** — it no longer bails when the registry is
|
||
> non-empty; instead it registers only deployed-app paths missing from the registry. It never removes
|
||
> or mutates an existing entry, never re-adds or reactivates a path already present in ANY state
|
||
> (incl. `Decommissioned`), and never flips `IsDefault` (a new path defaults only if the registry has
|
||
> no default at all). This is NOT auto-register-on-attach (that recommendation was rejected — manual
|
||
> enrollment is by design); it only picks up drives that deployed apps already reference.
|
||
> - **(v0.64.0) `InferStorageLabel` disambiguates the internal SSD** — a path whose basename is the
|
||
> `felhom-data` namespace dir (the internal system volume, e.g. `/mnt/sys_drive/felhom-data`) now reads
|
||
> **"Belső SSD (rendszer)"** instead of the colliding "Tárhely (felhom-data)". Model-A user drives
|
||
> register their MOUNT ROOT (e.g. `/mnt/felhom-usb`), never `.../felhom-data`, so this can't mislabel a
|
||
> user drive. Still overridable via `SetStorageLabel`.
|
||
> - **(v0.65.0) Data migration** (`internal/stacks/migrate.go`) — move app data between drives, in-process
|
||
> over the controller's `/mnt:/mnt:rslave` RW mount; crash-safe + resumable via a journal
|
||
> (`<dataDir>/migration.json`). `MigrateAll(source,target)` moves the whole felhom-data namespace (every
|
||
> app + a conflict-merge walk for non-app/customer content); `MigrateApp(app,target)` moves one app's
|
||
> subtree (drive→drive AND SSD→drive). Pipeline: validate → stop → copy (`rsync -a --checksum`, additive,
|
||
> **no `--delete`**) → verify (`rsync -ani --checksum`, zero pending) → flip+redeploy (`RedeployFromEnv`)
|
||
> → cleanup. **CLEANUP — the only destructive step — is gated on every unit verified AND every app
|
||
> redeployed.** Conflict-merge: skip-identical (checksum vs target + its `(N)` siblings), rename-on-differ
|
||
> to lowest-free `<base>(N)<ext>`, never overwrite; idempotent. Single-flight + mutually exclusive with the
|
||
> backup orchestrator. UI: `POST /api/storage/migrate{,-app}` + `GET /api/storage/migrate/status` (poll);
|
||
> migrate-all on the settings page, per-app on the app-info page, shared Hungarian progress panel.
|
||
> - **(v0.65.0) Self-serve decommission** (`handleStorageDecommission`, `POST /api/storage/decommission`) —
|
||
> retire a drive, non-destructive (never formats). Two choices (no partial): migrate-all-then-decommission
|
||
> (runs `MigrateAll`; the migration done-hook soft-marks the source + calls the agent on success), or
|
||
> decommission-anyway (type-to-confirm; stops the apps, keeps their `HDD_PATH` so they show "Hiányzó
|
||
> tárhely"). Both end at `settings.SetDecommissioned` (soft marker retained — blocks A1 resurrection) +
|
||
> `agentapi.Decommission` (agent v0.32.0: `POST /disks/decommission`, role-gated user-data, intent +
|
||
> bind-prune + unmount). Re-enrolling a decommissioned drive (`registerStoragePath`) clears the marker +
|
||
> restores `Schedulable`. A deployed app whose drive is decommissioned/disconnected/absent shows the
|
||
> **"Hiányzó tárhely"** warning badge on the dashboard/stacks/app card.
|
||
>
|
||
> - **(v0.66.0) Userdata layout + shared-storage ownership convention** (`internal/appbackup/userdata.go`).
|
||
> Each drive's felhom-data namespace gains a customer-facing **`userdata/`** tree (sibling of
|
||
> `appdata/`/`backups/`) — the ONLY thing FileBrowser mounts. Skeleton:
|
||
> `media/{movies,tv,music,audiobooks,books,comics,photos}`, `downloads`, `import/{paperless,calibre}`,
|
||
> `roms`, `documents`. **Convention:** every userdata dir is group-owned by `SharedContentGID` (1000),
|
||
> mode **2775** (setgid + group-rwx) — `EnsureUserdataDir` = MkdirAll → explicit `Chmod(ModeSetgid|0775)`
|
||
> (MkdirAll's mode is umask-masked AND drops setgid) → chown group 1000. Setgid makes new files inherit
|
||
> group 1000 so FileBrowser (uid 1000) + the content apps collaborate without permission collisions.
|
||
> - **`USERDATA_PATH`** = `<HDD_PATH>/userdata` is injected into the compose env (the shared
|
||
> `withUserdataPath`, used by BOTH `stackEnv` and `composeExecWithEnv` — the initial deploy builds env
|
||
> from values, not app.yaml). Catalog media mounts use `${USERDATA_PATH}/...`.
|
||
> - **Pre-create**: the full skeleton is created with the convention on `registerStoragePath` +
|
||
> `syncFileBrowserMounts` (system + additional drives); a **deploy belt** in `composeExecCustomEnv`
|
||
> (on `up`) pre-creates every `${USERDATA_PATH}/...` bind source the app declares, so Docker never
|
||
> auto-creates a userdata dir as guest-root (covers apps not in the skeleton).
|
||
> - **NAS shares browse too (v0.160.0, R-67):** a registered network storage binds its share ROOT
|
||
> into FileBrowser (`/mnt/felhom-drives/<name>:/srv/<name>:rslave`) — no userdata scoping and NO
|
||
> skeleton (Felhom convention dirs are never written onto a customer's own NAS; the sync is
|
||
> read-only toward the share). The gate differs from drives: an idle autofs trigger is HEALTHY
|
||
> and included (first access wakes it — Phase-0-probed through an rslave bind on demo-hp), while
|
||
> a `stub` classifier verdict (this namespace sees a local dir, not the NAS) EXCLUDES the share
|
||
> from both the mounts and the source list that pass — uploading into a stub would be silently
|
||
> shadowed by the real mount later. `unknown` fails open. NAS add-success and remove trigger the
|
||
> same debounced `SyncFileBrowserMounts`; pure assembly lives in `buildFileBrowserPaths`
|
||
> (handlers.go) with every edge seamed.
|
||
> - **FileBrowser** mounts `<drive>/userdata` (was `appdata`) and runs as uid 1000 → it can create
|
||
> folders + upload into the 2775 setgid tree (fixes the old permission-denied); app internals
|
||
> (`appdata/`) are no longer browsable. **(v0.66.2)** its entrypoint is wrapped
|
||
> `["sh","-c","umask 002; exec /home/filebrowser/filebrowser"]` so folders the customer creates come
|
||
> out **2775 (group-writable)** — the gtstef image is a single Go binary that ignores a `UMASK` env
|
||
> (verified), so the wrapper is the mechanism. Without it, customer folders were 2755 (group-read
|
||
> only) and apps in group 1000 couldn't write into them.
|
||
> - **Run-identity**: PUID/PGID-1000 apps (radarr/sonarr/calibre with `UMASK=002`) write **group-
|
||
> writable**, so FileBrowser can fully manage their output. Root-only apps (jellyfin :ro; komga +
|
||
> audiobookshelf RW after the `user:1000` fallback) write **group-readable** via setgid (FileBrowser
|
||
> browses/reads, the app manages its own files).
|
||
> - **Migration-safe**: `migrate.go`'s merge walk preserves the source dir's setgid + group (and
|
||
> `copyFile` the full file mode + group), so the convention survives a whole-drive `MigrateAll`.
|
||
>
|
||
> The privileged controller-side disk subsections **below are historical** (the `internal/storage/*` scan/
|
||
> format code was removed in 8C — execution is the agent's now).
|
||
|
||
The storage subsystem handles the full lifecycle of external storage: detection, initialization, path registration, and data migration.
|
||
|
||
> **CURRENT (post-de-privileging + Impl-2b, v0.95.0):** the in-guest storage code below
|
||
> (`internal/storage/scan.go`, `format.go`, `attach.go`) is **retired** — all disk ops are delegated to
|
||
> the host agent via `internal/agentapi`. The two enrollment wizards (`/settings/storage/init`,
|
||
> `/settings/storage/attach`) now populate candidates from the agent's **raw-device scan**
|
||
> (`GET /api/disks/candidates` → agent Impl-2a, proxied by `agentDiskCandidatesHandler`): `initialize` =
|
||
> every unclaimed disk (blank or data-bearing), `attach` = the mountable-FS subset. The agent's
|
||
> unclaimed-disk filter (Impl-1 `claim.go`) is authoritative + fail-safe (never offers OS/enrolled/claimed
|
||
> disks), so the controller does NO client- or server-side filtering. Enrollment posts to the unchanged
|
||
> `/api/storage/init` (format via the agent's Impl-1 guarded `mkfs` → mount → bind → intent) or
|
||
> `/api/storage/attach` (mount + bind, no format). The legacy text below is kept for historical context.
|
||
|
||
#### Disk Scanning (`internal/storage/scan.go`)
|
||
|
||
- `ScanDisks()` uses `lsblk -J -b` for block device enumeration
|
||
- System disk detection via host fstab parsing (`/host-fstab`) + UUID resolution via `blkid`
|
||
- Partitions enriched with filesystem type, UUID, and label from direct `blkid` probing (Docker containers have incomplete udev cache)
|
||
- Returns `AvailableDisks` (non-system, non-loop, non-CDROM), `SystemDisks`, and `FormatablePartitions` (empty partitions on system disks that are safe to format)
|
||
- Handles NVMe (`nvme0n1p1`), SCSI (`sdb1`), and eMMC (`mmcblk0p1`) naming
|
||
|
||
#### Disk Initialization Wizard (`internal/storage/format.go`)
|
||
|
||
A step-by-step UI at `/settings/storage/init`:
|
||
|
||
1. **Scan** — Lists available disks with model, size, partition info
|
||
2. **Select** — User picks a disk and enters a mount name (e.g., `hdd_1`)
|
||
3. **Confirm** — User types "FORMAZAS" to confirm destructive operation
|
||
4. **Format pipeline**: `wipefs` → `sfdisk` (GPT) → `mkfs.ext4` → `blkid` UUID → backup fstab → append UUID-based fstab entry → mount → `findmnt` verification → `chown 1000:1000` → create `felhom-data/` and `Dokumentumok/` subdirectories
|
||
5. Auto-registers new storage path in settings.json
|
||
6. Smart partition detection: skips repartitioning for existing empty partitions
|
||
|
||
Safety guards: system disk detection, mount path conflict check, confirmation required, progress channel for real-time UI feedback.
|
||
|
||
**System-disk partition formatting:** When the system disk has an empty partition (no filesystem, not mounted, not used for /, /boot, /boot/efi, or swap), the init wizard detects it via `FormatablePartitions` in the scan result and offers to format just that partition. Uses `IsSystemPartition()` (granular per-partition check via fstab) instead of `IsSystemDisk()` (whole-disk block), so sda1 can be formatted while sda3 (root) remains protected.
|
||
|
||
#### Attach Existing Drive Wizard (`internal/storage/attach.go`)
|
||
|
||
A step-by-step UI at `/settings/storage/attach` for drives that already have a filesystem (e.g., a previously used ext4 drive). Unlike the init wizard, this does **not** format the drive — existing data is preserved.
|
||
|
||
**Problem solved:** Mounting a whole drive at `/mnt/<name>` would mix existing user data with the controller's directory structure (`felhom-data/`, `Dokumentumok/`, etc.). The bind-mount approach isolates the controller's working directory from other data on the drive.
|
||
|
||
1. **Scan** — Lists available disks, filtered to partitions that have an existing filesystem (FSType != "")
|
||
2. **Mount raw** — Partition is mounted read-only at a hidden staging path (`/mnt/.felhom-raw/<label>`)
|
||
3. **Browse** — Directory browser shows the drive's contents. User can navigate and create a new folder (e.g., `felhom_data`)
|
||
4. **Configure** — User enters a mount name and display label. Warning: mount path is immutable until detached
|
||
5. **Finalize** — Bind-mounts the selected subfolder at `/mnt/<name>`. Two fstab entries are created (both with `nofail`):
|
||
- Raw mount: `UUID=<uuid> /mnt/.felhom-raw/<x> <fstype> defaults,nofail,noatime 0 2`
|
||
- Bind mount: `/mnt/.felhom-raw/<x>/<subfolder> /mnt/<name> none bind,nofail 0 0`
|
||
6. Sets permissions (`chown 1000:1000`), creates `felhom-data/` and `Dokumentumok/` subdirectories
|
||
7. Auto-registers the storage path in settings.json + syncs FileBrowser mounts
|
||
|
||
Cancel at any point cleans up the temporary raw mount. The bind mount path (`/mnt/<name>`) is a real mount point, so all existing code (disk usage, IsMountPoint checks, etc.) works unchanged.
|
||
|
||
#### Storage Path Registry (`internal/settings/settings.go`)
|
||
|
||
Multiple external storage paths supported with:
|
||
- **Label**: Human-readable name (editable inline)
|
||
- **Default flag**: New deploys use this path by default
|
||
- **Schedulable flag**: Path appears in deploy dropdown
|
||
- **Disconnected state**: `Disconnected`, `DisconnectedAt`, `StoppedStacks` — set by watchdog or safe-disconnect API, cleared on reconnect
|
||
- **Auto-discovery**: On startup, scans deployed apps' `HDD_PATH` values and registers unknown paths
|
||
- Thread-safe CRUD: Add, Remove, SetDefault, SetSchedulable, SetLabel, SetDisconnected, ClearDisconnected
|
||
|
||
#### Data Migration (`internal/storage/migrate.go`)
|
||
|
||
Move app data between storage paths (e.g., SSD → HDD, HDD → new HDD):
|
||
|
||
1. Validate: stack exists, deployed, has HDD data, target differs from source
|
||
2. Estimate total size, check free space on target
|
||
3. Stop the application
|
||
4. `rsync -a --info=progress2` per mount path with real-time progress parsing
|
||
5. Update `app.yaml` HDD_PATH to new location
|
||
6. Start the application
|
||
7. **Rollback on failure**: reverts config, restarts on old storage
|
||
|
||
Progress UI at `/stacks/{name}/migrate` with byte counter and percentage.
|
||
|
||
#### Stale Data Cleanup
|
||
|
||
After migration, the deploy page detects leftover data on previous storage paths:
|
||
- Shows path, size, and a delete button
|
||
- Two-step confirmation required
|
||
- Protected paths (`felhom-data/`, `felhom-data/appdata/`, `felhom-data/backups/`, `media/`, `Dokumentumok/`) cannot be deleted
|
||
|
||
#### FileBrowser Mount Sync
|
||
|
||
When storage paths are added or removed, `syncFileBrowserMounts()` auto-regenerates FileBrowser's `docker-compose.yml` with volume mounts for all registered paths. It then **recreates the container only when the generated `config.yaml` or compose actually changed** (v0.82.0, F2) — gated by the pure helper `fbNeedsRecreate(oldCfg,newCfg,oldCompose,newCompose)`, which compares the on-disk content captured before the writes against the final content read after them (so the integrations' `ReapplyConfigForTarget` edits count). When nothing changed (a controller restart, a no-op sync) it issues a plain `up -d --remove-orphans` that does **not** bounce the running FileBrowser. The restore-mode DB reset (`down -v`) still forces a recreate.
|
||
|
||
#### Storage Watchdog (`internal/monitor/watchdog.go`)
|
||
|
||
Continuously monitors registered storage paths for disconnection/reconnection (primarily USB drives):
|
||
|
||
- **Probe loop**: `ProbeStoragePath()` calls `syscall.Statfs()` with 3-second timeout in a goroutine. Runs every 5s per connected path, 30s per disconnected path.
|
||
- **Debouncing**: 3 consecutive probe failures required before declaring a drive disconnected (prevents false positives from transient I/O).
|
||
- **Disconnect reaction** (automatic, ~15s detection):
|
||
1. Stops all deployed stacks whose `HDD_PATH` is under the disconnected drive (skips protected stacks)
|
||
2. Persists `Disconnected`, `DisconnectedAt`, `StoppedStacks` to `settings.json`
|
||
3. Lazy-unmounts stale VFS entries (`umount -l`) — for attach-wizard drives, unmounts bind first, then raw
|
||
4. Fires alert refresh (red banner on all pages), notification (`storage_disconnected`), and immediate hub report push
|
||
- **Auto-reconnect** (for UUID-based fstab entries):
|
||
1. Checks `/host-dev/disk/by-uuid/<uuid>` for device reappearance
|
||
2. Cleans stale mounts, then `mount -T /host-fstab <path>` (raw + bind for attach-wizard drives)
|
||
3. Verifies with a post-mount probe
|
||
4. Runs `restic unlock` if stale lock files exist
|
||
5. Validates `StoppedStacks` (filters to actually-stopped stacks), clears `Disconnected` flag
|
||
6. Fires alert refresh, notification (`storage_reconnected`), hub report push
|
||
|
||
**Safe disconnect UI** (manual, Settings page):
|
||
|
||
- "Leválasztás" button shown for USB drives (detected via sysfs symlink path containing `/usb`)
|
||
- Confirmation dialog lists affected apps
|
||
- Flow: stop apps → `sync` → `umount` (fallback `umount -l`) → mark disconnected → notification
|
||
- Disconnected card: dashed border, red badge, timestamp, stopped apps list, "Csatlakoztatás" (reconnect) button
|
||
- After reconnect: "Alkalmazások indítása" button to restart auto-stopped stacks
|
||
|
||
**USB detection** (`system.IsUSBDevice`): Reads `/host/sys/block/<disk>` symlink — if target path contains `/usb`, it's a USB device. The `removable` sysfs flag is unreliable for USB HDDs (returns 0). USB drives show an orange "USB" badge on their storage card alongside Aktív/Alapértelmezett badges (v0.27.2). Handles findmnt bind-mount suffix stripping (`/dev/sdb1[/subdir]` → `/dev/sdb1`) for attach-wizard drives (v0.32.5).
|
||
|
||
**Backup guards**: Nightly DB dumps, restic snapshots, and cross-drive backups all skip disconnected, removed, and inactive drives with WARN log (not treated as failures). Cross-drive `RunAppBackup()` returns nil (not error) for unavailable destinations — prevents noisy error aggregation in scheduled runs (v0.32.5).
|
||
|
||
**Tier2 destination unavailable (v0.32.5)**: When a Tier2 backup destination drive is disconnected, removed from storage, or deactivated (Inaktív), the backup page shows:
|
||
- Yellow status dot with "2. mentés szünetel" tooltip (not red)
|
||
- Warning badge: "Cél meghajtó leválasztva" (disconnected/removed) or "Cél meghajtó inaktív" (deactivated)
|
||
- Grayed-out last-run info and backup contents
|
||
- Hidden "Futtatás most" button (prevents futile manual triggers)
|
||
- "Beállítás" link preserved for reconfiguration
|
||
- Tier2 config persists — backups auto-resume when drive returns/reactivates
|
||
- Detection: `IsStoragePathKnown()` catches removed paths, `IsStoragePathSchedulable()` catches inactive/disconnected/decommissioned
|
||
|
||
**UI integration**: Disconnected drives show with hatched red bars on dashboard, monitoring, and backup pages. Per-app backup rows show "Meghajtó leválasztva" badge. Health check emits warnings for disconnected paths.
|
||
|
||
---
|
||
|
||
### 5. Monitoring & Health
|
||
|
||
#### System Health Checks (`internal/monitor/healthcheck.go`)
|
||
|
||
`RunHealthCheck()` evaluates multiple subsystems and returns a `HealthReport` with status (`ok`/`warn`/`fail`):
|
||
|
||
| Check | Warning | Critical |
|
||
|-------|---------|----------|
|
||
| Disk usage (SSD/HDD) | >= 90% | >= 95% |
|
||
| Memory | available < 512MB | available < 256MB |
|
||
| CPU temperature | >= 75C | >= 85C |
|
||
| Docker daemon | — | unreachable |
|
||
| Protected containers | — | not running |
|
||
| Storage paths | not a mount point (data on SSD), drive disconnected | path inaccessible, disk >= 95% |
|
||
|
||
Backup destination validation (`CheckBackupDestination`) has tiered checks:
|
||
- Path doesn't exist → critical/blocked
|
||
- Not writable → critical/blocked
|
||
- Same block device as root → warning (data on system drive)
|
||
- Disk >95% full → critical/blocked
|
||
- Disk >90% full → warning
|
||
|
||
#### Healthchecks.io Integration (deprecated)
|
||
|
||
Legacy pinger (`internal/monitor/pinger.go`) still runs for backward compatibility but is no longer the primary monitoring mechanism. Monitoring is now handled by the Hub event system (see [Notifications](#5-notifications)). A deprecation log is emitted on startup if ping UUIDs are configured.
|
||
|
||
#### Metrics Store (`internal/metrics/`)
|
||
|
||
- **SQLite with WAL mode** for concurrent reads during collection
|
||
- **System metrics**: CPU%, memory (total/used/available), temperature, load average — collected every 60 seconds
|
||
- **Container metrics**: CPU%, memory, network I/O, block I/O per container
|
||
- Downsampled queries for chart time ranges (1h, 6h, 24h, 7d, 30d)
|
||
- 30-day auto-prune via daily scheduler job
|
||
|
||
#### Monitoring Page
|
||
|
||
Full-page system monitor at `/monitoring`:
|
||
- **System Overview**: hostname, OS, kernel, CPU model/cores, uptime
|
||
- **System Metrics Charts**: 4 line charts (CPU, Memory, Temperature, Load) in 2x2 grid
|
||
- **Memory Distribution Bar**: stacked bar showing per-container memory usage, OS/system overhead, and free memory (real-time from `/proc/meminfo` + container stats)
|
||
- **Container Resources**: horizontal bar charts (CPU% and Memory per container)
|
||
- **Per-container Detail**: click-to-expand historical charts
|
||
- **Hub Connection Status**: shows Hub URL, customer ID, connection state (connected/unreachable), last successful push, last error
|
||
|
||
Chart.js 4.4.7 embedded locally (works in offline environments), dark theme matching site design.
|
||
|
||
#### Host (Proxmox box) Health — agent-proxied (slice 9, `internal/agentapi` + `agent_host_metrics_handler.go`)
|
||
|
||
The de-privileged controller (slice 8C) sees only its own cgroup and cannot read the host. The top card of `/monitoring` ("Szerver állapota (gazdagép)") instead shows the **real Proxmox box**, proxied from the host agent's `GET /host/metrics`:
|
||
- **Host block**: CPU% + load average, memory used/total, **CPU/chassis temperature** (or "n/a" when the hardware exposes no sensor — graceful-null), uptime.
|
||
- **Per-storage capacity**: a used/total bar per host storage target, with thin-pool fill (a full lvmthin pool corrupts every guest on it) and disk SMART temperature/wear.
|
||
|
||
Path: `GET /api/host-metrics` → `Client.HostMetrics()` (leaf-pinned, per-guest-token agentapi client) → agent `GET /host/metrics`. Host-wide and token-authed (assumption: **one customer per host** — the home-server model). It is a **live** fetch (a fresh agent collect, not the 15-minute hub snapshot), so the page polls it every **8 s** while open. When the agent is unconfigured/unreachable the card shows a "nem elérhető" banner; the controller's own metric charts are unaffected.
|
||
|
||
**Storage-bar ordering + labels (v0.57.0):** the agent enumerates storages via `pvesm` in a
|
||
non-deterministic order, so the per-storage capacity list (`#host-storage-bars`) reordered on every poll.
|
||
`enrichHostStorageTargets` (`agent_host_metrics_handler.go`) sorts the response **server-side** —
|
||
user-data (`usb`/`local-dir`) → system+apps (`lvmthin`/`lvm`) → builtin `local` → backup
|
||
(`pbs`/`nfs`/`cifs`) → other, alphabetical by id within a tier — and attaches a friendly Hungarian
|
||
`label` + one-line `purpose` per entry (rendered by `monitoring.html`, with the raw PVE id shown muted).
|
||
**Display labels only — the PVE storage ids are never renamed** (vzdump/PBS configs reference them by
|
||
name). This is distinct from the server-rendered, user-data-only `buildStorageBars` "Tárhely" list.
|
||
|
||
#### Alert System (`internal/web/alerts.go`)
|
||
|
||
State-based alerts displayed on all pages:
|
||
- Sources: health issues, Hub connection status, backup disabled, storage disconnected, update available
|
||
- Hub alerts: `hub-disabled` (warning) when Hub not enabled, `hub-unreachable` (error) when last push failed and no success in 30 min
|
||
- Sorted by severity (error > warning > info), capped at 5 visible
|
||
- Refreshed every 5 min + on startup + on storage state changes
|
||
|
||
---
|
||
|
||
### 6. Notifications
|
||
|
||
#### Hub Event System (`internal/notify/notifier.go`)
|
||
|
||
The controller pushes structured events to the Hub's `/api/v1/event` endpoint. The Hub handles notification dispatch, cooldown management, and dead man's switch detection.
|
||
|
||
**Core method:** `PushEvent(eventType, severity, message, details)` — non-blocking goroutine, 2 retries with 3s backoff, never blocks the caller.
|
||
|
||
#### Event Types
|
||
|
||
| Event Type | Severity | Trigger |
|
||
|------------|----------|---------|
|
||
| `backup_failed` | error | Nightly restic backup fails |
|
||
| `db_dump_completed` | info | Nightly database dumps succeed |
|
||
| `db_dump_failed` | error | Nightly database dumps fail |
|
||
| `backup_integrity_ok` | info | Weekly `restic check` passes |
|
||
| `backup_integrity_failed` | error | Weekly `restic check` fails |
|
||
| `crossdrive_completed` | info | Cross-drive secondary backup succeeds |
|
||
| `crossdrive_failed` | error | Cross-drive secondary backup fails |
|
||
| `health_degraded` | warning | Health status degrades (ok→warn) |
|
||
| `health_critical` | error | Health status critical (any→fail) |
|
||
| `health_recovered` | info | Health status recovers (fail/warn→ok) |
|
||
| `disk_warning` | warning | Disk usage crosses 90% |
|
||
| `disk_critical` | error | Disk usage crosses 95% |
|
||
| `storage_disconnected` | error | Storage drive physically removed |
|
||
| `storage_reconnected` | info | Storage drive reconnected |
|
||
| `controller_started` | info | Controller process starts |
|
||
| `controller_updated` | info/error | Self-update success or failure |
|
||
| `app_deployed` | info | New app deployed via API |
|
||
| `app_removed` | info | App removed via API |
|
||
| `app_start_failed` | warn | A DEPLOYED app is not running (fix-3) — fired ONCE per running→down transition |
|
||
| `disaster_recovery_started` | warning | DR restore begins |
|
||
| `disaster_recovery_completed` | info/error | DR restore finishes (success/partial) |
|
||
|
||
Each event carries typed detail structs (e.g., `BackupDetails`, `DiskDetails`, `HealthDetails`) serialized as JSON.
|
||
|
||
**Deployed-app-down alerting (fix-3, v0.120.0, CAMPAIGN-3).** A `deadapp-check` scheduler job (every
|
||
30 s, after a 90 s boot grace) scans `stackMgr.GetStacks()`: a DEPLOYED app whose containers are
|
||
`exited`/`degraded` (`stacks.IsDownState` minus the `stopped` exclusion added in v0.164.0 — see below;
|
||
a Docker `created`/`dead` container, the F11 dead-at-boot case, resolves to `exited`) gets a
|
||
**state-based WARN dashboard banner** ("Telepített alkalmazás nem
|
||
fut: <app>", grouped above 3 so a reboot storm doesn't wall the dashboard) that self-clears when the
|
||
app runs again, AND an `app_start_failed` hub event fired **once per running→down transition**
|
||
(`Notifier.NotifyAppStartFailures` tracks per-app state; down→down cycles are silent — the hub owns
|
||
the real cooldown, the controller adds no timer). The boot grace prevents false alarms during the
|
||
controller's own startup while STILL firing for an app that never came up. This closes the campaign's
|
||
4-hour silent CWA death.
|
||
|
||
**Dead-primary alerting (R-51, v0.156.0).** fix-3 above only ever saw stacks that were *entirely*
|
||
down. A multi-container app whose MAIN container died while its helpers kept running aggregated to
|
||
`StateRunning` ("partial") and therefore alerted on nothing — `immich-server` was `Exited` for 18 h,
|
||
the app 100 % unreachable, with no banner and no event (F4, AUDIT-vacation-remote-ops-2026-07-20).
|
||
`aggregateState`'s mixed branch now inspects each DOWN member's docker restart policy: `always` /
|
||
`unless-stopped` means docker was supposed to be keeping it up, so the stack becomes
|
||
**`StateDegraded`** — a down state, so the existing banner and the existing `app_start_failed` event
|
||
fire unchanged. `no` / `on-failure` is a finished one-shot init/migrate container and stays benign.
|
||
An unreadable policy counts as supervised (fail-closed: a member is known dead, only the excuse is
|
||
missing). The `unhealthy` / `restarting` / `paused` / `unknown` exclusions are untouched — folding
|
||
`unhealthy` into down is precisely the flapping fix-3 avoided. UI: „Részlegesen leállt", warn
|
||
colour, counted with the stopped apps, URL flagged unpublished (Traefik withholds the route when the
|
||
routed member is the dead one). Policy reads are one `docker inspect` per down member of a mixed
|
||
stack, cached per container+state.
|
||
|
||
**Deliberate stops are silent (v0.164.0).** Stopping an app from the UI (Leállítás → `StopStack` →
|
||
`docker compose down` → zero containers → the deployed stack aggregates to `StateStopped`) is the
|
||
user's own action, not a fault, and must not raise the banner OR the `app_start_failed` email. The
|
||
scan's pure core was extracted to `classifyRunStates([]stacks.Stack)` and its down predicate is now
|
||
`stacks.IsDownState(st.State) && st.State != stacks.StateStopped` — the SINGLE fix-3 derivation point,
|
||
so `StateStopped` is dropped from both the banner dead-list and the notifier Down-set at once (the
|
||
launcher tile still shows greyed „Leállítva"; the monitoring page and dashboard counters are factual
|
||
display, not alarms, and are unchanged). This rests on two invariants: **I1** — a UI stop always ends
|
||
at `StateStopped` (compose down removes the containers); **I2** — the P2 restart-policy census
|
||
(53 templates / 78 services, all `unless-stopped`) means a crashing app never comes to rest at
|
||
`stopped`, so faults still surface as `exited`/`degraded`/`restarting`/`unhealthy`. If either
|
||
invariant changes, revisit the suppression.
|
||
|
||
> **C9-F2 (v0.183.0) — the `restarting` half of that sentence was a wish, not a fact.** `restarting`
|
||
> was named above as a state through which faults "still surface", but it was in no down set at all:
|
||
> `IsDownState` excludes it, so a crash-looping app raised no banner, no `app_start_failed`, no email
|
||
> and no hub event — and `unless-stopped` means Docker retries forever, so the silence was permanent.
|
||
> Campaign 9 watched docmost loop for nine minutes while the F-OBS heartbeat printed
|
||
> „180 scans since boot, 4 deployed app(s) evaluated, **0 currently down**".
|
||
>
|
||
> The fix does **not** add `StateRestarting` to `IsDownState` — that alarms on every deploy and update
|
||
> fleet-wide. A SUSTAINED restarting run becomes down after `stacks.crashLoopAfter` (**5 min**), chosen
|
||
> above the deploy flow's 120 s health timeout, Mealie's 60 s `start_period` and R-97b's 180 s quiesce
|
||
> grace, so the suppression windows compose into one bounded delay rather than leaving a gap. Carried
|
||
> by `Stack.RestartingSince` (stamped in `refreshStatusLocked`, cleared on any other state, not
|
||
> persisted) and read via `Stack.CrashLooping(now)` — used by BOTH the alarm and the dashboard
|
||
> "how many of my apps work" counter, which previously counted `restarting` as running and so
|
||
> contradicted the alarm on the same screen. Pinned by `crashloop_classify_test.go`; the test that a
|
||
> brief restart stays silent is the one that fails against the naive fix.
|
||
|
||
`IsDownState` itself is deliberately UNCHANGED (other
|
||
callers rely on stopped counting as down). An out-of-band `docker compose stop` leaves the containers
|
||
present → `StateExited` → still alerts, which is correct (out-of-band tampering is reportable).
|
||
|
||
**Boot desired-state reconciliation (R-52, v0.156.0, `internal/bootrecon`; rebuilt on recorded intent
|
||
in R-166, v0.189.0).** A `deployed: true` app that missed its boot start used to stay down until a
|
||
human noticed — the same shutdown that produced F4 left immich and calibre-web `Exited` while ten
|
||
sibling containers came back, and they were still down 18 h later (F5). At startup (5 s after the
|
||
quiesce and app-stop recoveries, so the three never race) the controller performs **one bounded
|
||
sweep**: every deployed, non-protected, not-mid-deploy stack that is down gets `StartStack`, at most
|
||
**2 attempts 30 s apart**, then it stops and the alarm owns the problem. Never a restart loop. The
|
||
whole sweep fits inside the 90 s boot grace, so a successful recovery is silent and a failed one
|
||
still alerts. Outcome is logged per attempt at INFO; no new hub event (the existing alarm is the
|
||
escalation).
|
||
|
||
**What "down" means here changed in v0.189.0.** Until then the sweep required the stack to still HAVE
|
||
containers, because the UI's Stop is `compose down` (which removes them) and "zero containers" was
|
||
read as a deliberate stop. That inference was wrong in two silent ways: a **power cut mid-compose**
|
||
and an **interrupted deploy** also leave zero containers, and both were skipped as "the customer
|
||
stopped it" and left down indefinitely. Since R-166 the sweep reads the customer's **recorded
|
||
intent** (`desired_state` in `app.yaml`) instead:
|
||
|
||
| `desired_state` | containers | result |
|
||
|---|---|---|
|
||
| `stopped` | any | **never** started — the customer said so, and no observation overrides it |
|
||
| `running` | 0 | **recovered** — the power-cut / interrupted-deploy case, invisible before v0.189.0 |
|
||
| `running` | >0 and down | **recovered** (unchanged) |
|
||
| `running` | >0 and up | left alone |
|
||
| absent (legacy) | 0 | **not** started — byte-identical to the pre-v0.189.0 behaviour |
|
||
| absent (legacy) | >0 and down | **recovered** — byte-identical to the pre-v0.189.0 behaviour |
|
||
|
||
**Absent means UNKNOWN, never "running".** Every `app.yaml` written before v0.189.0 lacks the field,
|
||
so absent is what an upgraded box reads for every app that has not been started or stopped since;
|
||
reading it as "running" would start every deliberately-stopped app on the first boot after the
|
||
upgrade. Where intent is unknown the sweep falls back to the old inference rather than inventing an
|
||
answer, and a running-only startup **backfill** converges the unambiguous cases (deployed and
|
||
observed up) without waiting for a button press. `stopped` is never backfilled from any signal.
|
||
|
||
**The sweep no longer looks only once (R-157 mechanism A, v0.190.0).** It used to sample the fleet at
|
||
T+5 s and return; at that moment docker is still restoring containers after a hard reset, so an app
|
||
that had not yet settled into a down state was never a candidate — **measured failing on three of six
|
||
hard resets**. It is now a **settle-then-sweep window**: the fleet (name, state, container count) is
|
||
sampled every 5 s, called settled after 3 identical samples, and swept **once**, at the end, on a
|
||
settled fleet. The window ends on whichever comes first — settled, or a **50 s budget** — and the log
|
||
says which. Sampling is read-only and there is still exactly one sweep with its per-app attempt bound
|
||
intact: this widens a bounded window, it does not remove the bound. `settle (5 s) + budget (50 s) +
|
||
one 30 s retry = 85 s` stays inside the 90 s `deadAppBootGrace`, which is what keeps a successful
|
||
recovery silent; a window that genuinely overruns emits a `LATE RECOVERY` WARN naming the apps rather
|
||
than the grace being widened to hide it.
|
||
|
||
**The sweep asks before it starts (R-171, v0.190.0).** Three things legitimately hold an app down,
|
||
and starting it would be the wrong repair for all three: its **data drive is absent** (compose would
|
||
create the bind sources on the guest rootfs — the hazard the drive gate exists to prevent), a
|
||
**quiesce** is holding it for a whole-guest backup, or an **app-data operation** (volume dump, offsite
|
||
restore, `.fab` export) is holding it. All three are refused through one seam, reusing the signals
|
||
their owners already publish. **Fail-safe: a drive whose liveness cannot be determined is treated as
|
||
absent.** Held apps are reported separately from `StillDown` — they are not a fault the sweep failed
|
||
to fix, and reporting them as one is a false alarm. This closed a regression v0.189.0 introduced:
|
||
before it, a drive-gate-stopped app read as `running` + zero containers, so the sweep started it,
|
||
burned both attempts and handed it to the dead-app alarm.
|
||
|
||
**Both boot gates read intent (R-170, v0.190.0).** There are two: the R-52 sweep above, and the
|
||
drive-backed **boot recreate gate** (`shouldRecreateOnBoot`, `internal/web/intermediary.go`), which
|
||
re-creates a drive-backed app onto its re-propagated drive after a guest reboot. Until v0.190.0 the
|
||
second still ended in `&& hasContainers`, so the two disagreed about the same question. It now uses
|
||
the identical three-way table — `stopped` → never, `running` → recreate whatever the container count,
|
||
absent → exactly the old `hasContainers` behaviour. Its drive-presence term is untouched and still
|
||
load-bearing: an app whose drive is absent is never recreated there either.
|
||
|
||
**Desired state — who owns it (R-166, v0.189.0).** `app.yaml` gains `desired_state`, a tri-state
|
||
`""` / `running` / `stopped`. It is written by **the customer's own action and nothing else**: the
|
||
`/api/stacks/{name}/{action}` switch (`start`/`restart`/`update` → running, `stop` → stopped),
|
||
`DeployStack`, `UpdateOptionalConfig`'s redeploy branch, and the `.fab` import. `StartStack` and
|
||
`StopStack` are deliberately **not** writers — a census found 14 callers of which only 2 are the
|
||
customer, and recording intent in the primitive would make a nightly backup indistinguishable from
|
||
the customer pressing Stop, which is the confusion the feature exists to end. Intent is written
|
||
**before** the act, and an action whose intent cannot be recorded is **refused**.
|
||
|
||
**Interrupted app-data operations (R-166, v0.189.0, `backup.AppStopGuard`).** A volume dump, an
|
||
off-site reconstitution and a `.fab` export all stop an app, work on its data, and start it again.
|
||
A controller killed inside that window left the app down with nothing on disk recording why or that
|
||
it was owed a restart. A persisted marker (`<data_dir>/appstop-state.json` — its **own** file, never
|
||
quiesce's, so one file has one writer) is now written **before** the stop and cleared only after a
|
||
restart that succeeded; a failed restart deliberately keeps it. At startup `Recover()` restarts the
|
||
recorded apps, clears the marker, and its outcome is reported to the operator on the existing
|
||
`backup_failed` event — an interrupted operation means the backup did not complete. **The `defer` in
|
||
those functions is not the mechanism**: a SIGKILL runs no deferred function (Campaign 8 fault 10, on
|
||
live hardware), which is exactly what the marker covers.
|
||
|
||
#### Default Enabled Events
|
||
|
||
Events the customer receives notifications for (configurable in settings):
|
||
`backup_failed`, `db_dump_failed`, `disk_warning`, `disk_critical`, `storage_disconnected`, `node_down`, `health_critical`, `expected_backup_missed`, `expected_dbdump_missed`
|
||
|
||
#### Preference Sync
|
||
|
||
Notification preferences (email, enabled events, cooldown hours) are:
|
||
- Stored locally in `settings.json`
|
||
- Synced to Hub on save and on controller startup via `POST /api/v1/preferences`
|
||
- Hub sync failure doesn't block local save
|
||
|
||
**Empty-email save guard (v0.137.0):** the save handler REFUSES a submit with a blank e-mail box
|
||
while any event is still enabled (it would store an empty address AND push it to the hub, wiping the
|
||
provisioning-seeded alert delivery). The form re-renders with a Hungarian error and the customer's
|
||
ticked events preserved; no save, no sync. Clearing the e-mail with **zero** events enabled is
|
||
allowed (an intentional turn-everything-off).
|
||
|
||
---
|
||
|
||
### 7. Update Management
|
||
|
||
#### App Catalog Sync
|
||
|
||
- Periodic `git fetch` + `git reset --hard` of the app catalog repo
|
||
- Content-hash comparison prevents unnecessary file writes
|
||
- Post-sync stack rescan detects new/changed apps immediately
|
||
- **Stale lock recovery**: automatically removes `.git/index.lock`, `.git/shallow.lock`, and `.git/HEAD.lock` before each fetch — prevents permanent sync failures after interrupted operations (e.g. container restart mid-sync)
|
||
|
||
#### Planned Update Classifications
|
||
|
||
| Marker | Behavior |
|
||
|--------|----------|
|
||
| No marker | Optional — shown on dashboard, customer clicks "Update" |
|
||
| `UPDATE_REQUIRED=true` | Mandatory — auto-applied during next update window |
|
||
| `UPDATE_SECURITY=true` | Critical — applied immediately |
|
||
|
||
#### Controller Self-Update (`internal/selfupdate/`)
|
||
|
||
The controller can update itself to the latest registry version with a one-click Settings button. In the
|
||
LXC architecture there is **no in-container compose** to drive (the old `docker compose -f
|
||
/opt/docker/felhom-controller/docker-compose.yml up -d` path does not exist in the guest — it produced
|
||
"docker-compose.yml nem elérhető"). Instead (Phase 1, v0.85.0) the controller **pulls** the target image
|
||
in-guest and delegates the container **swap to the host agent**, which owns the restart + health-verify +
|
||
**rollback**.
|
||
|
||
##### How It Works
|
||
|
||
```
|
||
1. Check Gitea Docker Registry V2 API for the latest semver tag (queryRegistry, BasicAuth).
|
||
2. If newer than current (ldflags Version): docker login --password-stdin → docker pull <image>
|
||
→ docker logout — IN-GUEST over the shared docker socket (token via stdin, never argv).
|
||
3. Delegate to the host agent: agentapi.SwapController → POST /controller/swap {image} (202). The agent
|
||
(external to this container) rewrites /etc/felhom-controller-image, restarts
|
||
felhom-controller-bootstrap.service, polls the new controller to healthy, and ROLLS BACK to the
|
||
previous image if it doesn't come up. The controller never docker-rm/recreates itself.
|
||
4. On startup the new container reads update-state.json → VerifyStartup marks success (current==target)
|
||
or failure (rollback → version mismatch). The Settings button polls /api/health and reloads.
|
||
```
|
||
|
||
The button is latest-only (no version picker) and opt-in. No host agent wired (un-provisioned guest) →
|
||
self-update unavailable.
|
||
|
||
##### Phase 2 — managed updates: the version FLOOR (v0.86.0)
|
||
|
||
On top of the opt-in button, the controller now honors an operator-enforced **minimum version (FLOOR)**.
|
||
The hub returns the customer's *effective* floor (per-customer override else a global default) on the
|
||
**report ACK** (`min_controller_version`, alongside `latest_version`). The controller's report pusher
|
||
(`internal/report/pusher.go`, `OnPushResponse`) hands the floor to the updater (`SetFloor`) and calls
|
||
`MaybeAutoUpdate()` — **on the existing report cycle, no new timer/endpoint**:
|
||
|
||
- If the box is **below** the floor it **auto-updates to the floor** (not latest) by reusing the Phase 1
|
||
flow above (`performUpdate`, `initiatedBy="auto-floor"`) — same pull → agent swap → rollback. No
|
||
customer click.
|
||
- **At/above** the floor: nothing (it does **not** chase latest — that's the button's job).
|
||
- Guards: dev build / no agent / backup running → skip; floor must be **pullable** (floor ≤ latest
|
||
available; floor > latest → warn + do nothing); one attempt per below-floor condition (in-memory flag
|
||
+ persisted `update-state.json`) → **no flapping/storm**.
|
||
- Settings UI shows "Minimális verzió (üzemeltető): X" and, during an auto-update, the same restart-poll
|
||
panel as the button.
|
||
|
||
The floor is the **auto-target** (the operator raises it for a controlled fleet rollout); latest stays
|
||
the customer's manual opt-in. Floor source + operator UI are hub-side (felhom-hub v0.15.0). **No agent
|
||
change — Phase 2 reuses the Phase 1 `POST /controller/swap`.**
|
||
|
||
##### Pull-based config-refresh (v0.94.0)
|
||
|
||
The same **report ACK** also carries a per-customer **`config_version`** (a hub-side stored counter,
|
||
bumped on every config save; felhom-hub v0.26.0). This is how an operator config edit reaches a running
|
||
box — **the hub never connects into the box** (it replaced the retired inbound "Push Config"). Wired in
|
||
`OnPushResponse` beside the floor reconcile (`internal/report/config_refresh.go`, `ConfigRefresher.Reconcile`):
|
||
|
||
- **Changed** vs. the last-applied version (`settings.applied_config_version`) → `bootstrap.RefreshConfig`
|
||
re-pulls `controller.yaml` from the hub and rewrites it, **re-merging the per-guest `local_api`** from
|
||
`bootstrap.json` (reuses the first-boot pull machinery; overwrites `controller.yaml` since the hub is its
|
||
source of truth; **never touches `settings.json`**) → record the new version → **graceful self-restart**
|
||
(`api.GracefulSelfRestart` → `os.Exit(0)` → Docker `restart: unless-stopped` re-reads the new config).
|
||
- **First-ever** ACK (nothing recorded) → record the baseline **without restarting** (the first-boot pull
|
||
already has the current config).
|
||
- **Unchanged** version → no-op (so no restart storm — after a refresh `applied == ACK`).
|
||
- **Failed pull** → keep the current config, do **not** record/restart, retry next report cycle.
|
||
- Record-before-restart so the restarted process sees the version applied and doesn't loop. Only the
|
||
`felhom-controller` container restarts; customer app stacks are untouched. (A config apply rotates
|
||
`web.session_secret`, so dashboard sessions are invalidated — same as the old Push-Config path.)
|
||
|
||
##### Design Philosophy
|
||
|
||
- **No automatic rollback** — follows the Watchtower pattern (24k+ GitHub stars, no rollback). Docker's `restart: unless-stopped` policy is the crash safety net. The Hub's dead man's switch detects when the controller goes down.
|
||
- **Audit state file** — `update-state.json` in the data volume records every update attempt (previous version, target version, initiator, result). Operators can SSH in and revert using `PreviousImage` from this file.
|
||
- **Backup-aware** — refuses to start an update while a backup is in progress (`backupRunning()` guard).
|
||
|
||
##### Package Structure
|
||
|
||
| File | Purpose |
|
||
|------|---------|
|
||
| `version.go` | `ParseVersion("X.Y.Z")` → `Version{Major,Minor,Patch}`, `Compare()` returns -1/0/1. Hand-rolled, no external deps. Rejects "dev" and "latest". |
|
||
| `state.go` | `UpdateState` struct persisted as JSON. `LoadState()`, `SaveState()` (atomic: `.tmp` + rename), `ClearState()`. Status values: `"pending"`, `"success"`, `"failed"`. |
|
||
| `updater.go` | Core `Updater` struct. Registry check via HTTP GET to `gitea.dooplex.hu/v2/admin/felhom-controller/tags/list` with Basic Auth (git username/token). Update trigger: `docker pull` → compose file regex replace → `docker compose up -d`. Thread-safe with `sync.Mutex`. |
|
||
|
||
##### Update Trigger Flow
|
||
|
||
1. **Guard checks:** concurrent update lock, dev version check, backup running check, compose file accessible
|
||
2. Write `update-state.json` with status `"pending"` (audit trail)
|
||
3. `docker pull <image>:<targetVersion>`
|
||
4. Read compose file → replace image tag via regexp → atomic write (`.tmp` + rename)
|
||
5. `docker compose -f /opt/docker/felhom-controller/docker-compose.yml -p felhom-controller up -d`
|
||
6. Docker kills the current container, starts the new one
|
||
|
||
##### Startup Verification
|
||
|
||
Called once from `main.go` before the scheduler starts:
|
||
1. Load `update-state.json` — if missing or status != `"pending"`, nothing to do
|
||
2. Compare running `Version` with `state.TargetVersion`
|
||
3. **Match** → mark `"success"`, notify via hub
|
||
4. **Mismatch** → mark `"failed"`, notify via hub
|
||
5. No rollback attempt — operator reverts manually if needed
|
||
|
||
##### Auto-Update Scheduling
|
||
|
||
Two separate scheduler jobs prevent interference with backups:
|
||
|
||
| Job | Type | Default | Purpose |
|
||
|-----|------|---------|---------|
|
||
| `selfupdate-check` | `sched.Every` | 6h | Check registry, cache result (for UI). Never triggers update. |
|
||
| `selfupdate-auto` | `sched.Daily` | 04:30 | If auto-update enabled + update available + backup not running → trigger. |
|
||
|
||
The auto-update time (`config.SelfUpdate.AutoUpdateTime`, default `"04:30"`) is deliberately separate from the backup window (02:30-~04:00) to avoid collisions. The `backupRunning()` guard is the hard safety check — if backups run long past 04:30, the update is skipped and retried the next day.
|
||
|
||
An initial version check fires 30s after startup so the Settings page shows version info quickly.
|
||
|
||
##### Compose File Access
|
||
|
||
The controller needs write access to its own `docker-compose.yml`. This is achieved via Docker volume mount ordering:
|
||
|
||
```yaml
|
||
volumes:
|
||
# 1. Directory mount — gives access to compose file + config
|
||
- /opt/docker/felhom-controller:/opt/docker/felhom-controller
|
||
# 2. Read-only override — prevents accidental config writes
|
||
- /opt/docker/felhom-controller/controller.yaml:/opt/docker/felhom-controller/controller.yaml:ro
|
||
# 3. Named volume override — persistent data in Docker-managed volume
|
||
- controller-data:/opt/docker/felhom-controller/data
|
||
```
|
||
|
||
##### API Endpoints
|
||
|
||
| Method | Path | Auth | Description |
|
||
|--------|------|------|-------------|
|
||
| GET | `/api/selfupdate/status` | Session or API key | Current status (cached, no network call) |
|
||
| POST | `/api/selfupdate/check` | Session or API key | Force registry check, return result |
|
||
| POST | `/api/selfupdate/update` | Session or API key | Trigger update (async, returns immediately) |
|
||
|
||
Self-update endpoints accept either session auth (for UI) or hub API key as bearer token (for external triggering from build scripts or hub). This enables the post-v0.16.0 deploy workflow:
|
||
|
||
```bash
|
||
# After building + pushing new image:
|
||
curl -s -X POST https://felhom.demo-felhom.eu/api/selfupdate/update \
|
||
-H "Authorization: Bearer <HUB_API_KEY>"
|
||
```
|
||
|
||
##### Settings Page UI
|
||
|
||
The "Verzió és frissítés" card on the Settings page (`/settings`) shows:
|
||
- Current version and latest available version
|
||
- "Frissítés elérhető" (update available) badge
|
||
- Last check time and any errors
|
||
- Registry mode line (v0.112.0): "Registry: nyilvános (hitelesítés nélkül)" vs "Registry: hitelesített" — credential-less is a supported mode, not an error state
|
||
- Auto-update status with configured time
|
||
- Last update result (success/failed/pending)
|
||
- **Buttons:** "Frissítés keresése" (check) + "Frissítés telepítése" (apply)
|
||
|
||
##### Registry access modes (v0.112.0)
|
||
|
||
Git Sync credentials (`git.username`/`git.token`) are **optional — for private catalogs only**; version
|
||
discovery and self-update work without them:
|
||
- **Anonymous (both empty):** `queryRegistry` performs the Docker Registry v2 anonymous token dance —
|
||
plain GET → 401 with `WWW-Authenticate` → token fetched from the ADVERTISED realm (parsed from the
|
||
header, never hardcoded — registry-agnostic) with no credentials → Bearer retry. `pullImage` skips
|
||
`docker login` entirely (docker's native anonymous flow covers public packages).
|
||
- **Authenticated (both set):** the previous BasicAuth + login/pull/logout path, unchanged.
|
||
- **Half-configured (only one set):** loud incomplete-credentials error — never a silent anonymous
|
||
downgrade.
|
||
- A registry that genuinely denies anonymous access surfaces
|
||
"registry denied anonymous access — a private registry requires Git Sync credentials".
|
||
|
||
After triggering an update, the page polls `/api/health` every 3s and reloads when the new container responds.
|
||
|
||
A global info-level alert ("Új controller verzió elérhető") appears on all pages when an update is available, linking to the Settings page.
|
||
|
||
##### Configuration
|
||
|
||
```yaml
|
||
self_update:
|
||
enabled: true
|
||
check_interval: "6h" # How often to check registry
|
||
image: "gitea.dooplex.hu/admin/felhom-controller" # Default
|
||
auto_update: false # Set true for unattended updates
|
||
auto_update_time: "04:30" # When to auto-apply (after backups)
|
||
health_timeout_seconds: 60 # Reserved for future use
|
||
```
|
||
|
||
##### Edge Cases
|
||
|
||
| Scenario | Behavior |
|
||
|----------|----------|
|
||
| `Version == "dev"` | `ParseVersion` returns error → no updates reported, trigger refused |
|
||
| Registry unreachable | Log warning, return error in check result. No crash. |
|
||
| No registry credentials | Return error "Registry hitelesítő adatok hiányoznak" |
|
||
| Compose file not writable | Refuse update before doing anything |
|
||
| Backup running | Refuse with "Mentés fut, próbálja később" |
|
||
| Concurrent update | Mutex prevents duplicates: "Frissítés már folyamatban" |
|
||
| Bad update (crash loop) | Docker restarts container. State file stays "pending". Operator SSH-reverts using `PreviousImage`. |
|
||
| Corrupt state file | Treated as "no pending update", logged, deleted |
|
||
|
||
---
|
||
|
||
### 8. Authentication & Settings
|
||
|
||
#### Customer-claim gate (`internal/web/claim.go`, v0.122.0 — closes DRILL-day0-vm F-4/F-5)
|
||
|
||
The dashboard password is **customer-owned**, set through a one-time claim code the hub emails to
|
||
the registered address (no operator-set path, no open-until-set window). This closes the fresh-box
|
||
race where a new `felhom.<domain>` cert appears in CT logs minutes before any password exists.
|
||
|
||
- **States** (precedence): a SET password (settings→config) always wins — the gate never shows.
|
||
Else a delivered **claim-code hash + not-yet-claimed** → GATED: every route serves the claim
|
||
page (`302 → /claim`) or `401` JSON (API); only `/claim*`, `/static/*`, `/api/health` pass. Else
|
||
(no password, no hash) → **legacy-open** with a red transition banner until the hub delivers a
|
||
hash (transitional only, never the fresh-box state).
|
||
- **Claim/reset flow**: `GET /claim` (code + new password ×2, min 12) → `POST /claim` verifies the
|
||
code (bcrypt match AND generation not yet consumed AND ≤ 72 h old), sets the customer's password
|
||
via `settings.SetPasswordHash`, marks `Claimed` (set-only), consumes the generation (single-use),
|
||
invalidates sessions, issues a fresh one. `POST /claim/request-new-code` (the "Új kód kérése" /
|
||
login-page "Elfelejtett jelszó") forwards to the hub, which emails a fresh code to the
|
||
registered address only. Reset rides the same page (a claimed box reaches `/claim` pre-auth).
|
||
- **Anti-brute-force**: per-source + global counter, 5 failures → 15-minute lockout (both scopes),
|
||
raising the allowlisted `claim_lockout` event. Pre-auth CSRF is an HMAC over `web.session_secret`
|
||
(fixes the CTRL-007 bare-double-submit weakness). The per-source key is the client IP resolved by
|
||
the shared `clientIP(r)` helper — XFF first-hop, else `RemoteAddr` with the ephemeral **port
|
||
stripped** (v0.129.0 F-B; keying on the raw `RemoteAddr` let distinct direct connections evade the
|
||
counter). The login form and the escrow wizard re-auth share the same helper/key.
|
||
- **Delivery**: the hub bakes `web.claim_code_{hash,generation,issued_at}` into the Day-0
|
||
controller.yaml (gate-from-first-boot) and serves the freshest state in the report ACK
|
||
(`report/claim_sync.go` caches it idempotently by generation — newer advances, same/older/nil
|
||
never rewrites, a hub outage never clears). The report carries `claimed` (hub ingests set-only).
|
||
- **Escape hatch**: `felhom-controller --print-reset-code` prints a one-time local code (generation
|
||
above cached/baked/consumed); the same gate consumes it. Root-gated by `docker exec` reachability.
|
||
**It runs as a SEPARATE PROCESS, which is why `effectiveClaimCode` READS THROUGH to the persisted
|
||
state (v0.198.0, R-204 item 1).** Until then the hatch persisted a new code while the running
|
||
server kept the old one cached, so the minted code was refused until the controller was restarted —
|
||
and nothing said so. The read-through (`settings.ReloadClaimCode`) is on the claim path only, and
|
||
only while the box carries no password (`claimGateActive` returns on `authEnabled()` first).
|
||
**Deliberately not a watcher, a signal handler or a TTL:** a TTL would leave a window in which a
|
||
SUPERSEDED code still works, which is worse than the bug. Refreshes hash/generation/issuedAt only —
|
||
never `ClaimConsumedGeneration`, which this process alone writes and must stay monotonic.
|
||
`effectiveClaimCode` returns an error and **every caller fails closed** (the gate stays UP, the claim
|
||
is refused); an ABSENT settings file is not an error, since a pre-first-save box falls back to the
|
||
controller.yaml bake.
|
||
|
||
#### Session Auth (`internal/web/auth.go`)
|
||
|
||
- bcrypt password verification with configurable source priority: `settings.json` → `controller.yaml` → no auth (open access)
|
||
- 7-day session duration with random 32-byte hex tokens
|
||
- `?next=` redirect after login preserves the page the user was visiting
|
||
- Session cleanup every 15 minutes
|
||
- All sessions invalidated on password change
|
||
- Conditional logout link (hidden when auth is disabled)
|
||
- Each session stores a dedicated CSRF token (separate 32-byte random value) alongside the session token
|
||
|
||
#### CSRF Protection (`internal/web/csrf.go`)
|
||
|
||
Synchronizer-token CSRF protection on all browser-facing state-mutating endpoints.
|
||
|
||
**How it works:**
|
||
- `CsrfProtect` middleware wraps all route handlers in `main.go`
|
||
- Safe methods (GET, HEAD, OPTIONS) pass through without validation
|
||
- For POST/DELETE/PATCH: reads token from `_csrf` form field or `X-CSRF-Token` request header; constant-time compares against the session's stored CSRF token
|
||
- On rejection: JSON `{"ok":false,"error":"CSRF token missing or invalid"}` for `/api/` paths; HTTP 403 text page for UI routes
|
||
- Logs: `[WARN] CSRF rejected: METHOD /path from addr (reason)`
|
||
|
||
**Exempt paths (no CSRF check):**
|
||
- Requests with `Authorization: Bearer ...` header — hub→controller API calls (selfupdate, config/apply). Browsers cannot auto-send Bearer headers, so cross-site requests are impossible on these endpoints.
|
||
- Auth-disabled mode (`authEnabled() == false`) — CSRF is meaningless when there is no session.
|
||
|
||
**Token delivery to templates:**
|
||
- `executeTemplate(w, r, name, data)` wrapper in `server.go` auto-injects `CSRFField` (`template.HTML` hidden `<input>`) and `CSRFToken` (raw string) into every page's data map
|
||
- `layout.html` emits `<meta name="csrf-token" content="{{.CSRFToken}}">` and defines `csrfHeaders()` JS function in `<head>` (before page scripts)
|
||
- Forms: `{{.CSRFField}}` (or `{{$.CSRFField}}` inside `{{range}}` loops — outer scope required)
|
||
- JS `fetch()` calls: `headers: csrfHeaders()` — returns `{'X-CSRF-Token': metaContent}`
|
||
- Dynamically-created JS forms: read token from `document.querySelector('meta[name="csrf-token"]').content`
|
||
- `navigator.sendBeacon()` replaced with `fetch(..., {keepalive: true})` where used — `sendBeacon` cannot send custom headers
|
||
|
||
#### Settings Persistence (`internal/settings/settings.go`)
|
||
|
||
Runtime-mutable settings in `settings.json` (separate from infrastructure config):
|
||
|
||
| Section | Contents |
|
||
|---------|----------|
|
||
| `password_hash` | bcrypt hash override |
|
||
| `notifications` | email, enabled events, cooldown hours |
|
||
| `db_validations` | per-DB dump validation results (survives restarts) |
|
||
| `app_backup` | per-app map: enabled flag, cross-drive config (method, dest, schedule, runtime status) |
|
||
| `storage_paths` | registered paths with label, default flag, schedulable flag, disconnected state |
|
||
| `cross_drive_restic_password` | auto-generated restic password for cross-drive repos |
|
||
|
||
All public methods use `sync.RWMutex`. File writes are atomic (`.tmp` + rename).
|
||
|
||
#### Settings Page (`/settings`)
|
||
|
||
Five sections:
|
||
1. **System config** — read-only display of `controller.yaml` values
|
||
2. **Version & update** — current/latest version, check/update buttons, auto-update status, last update result
|
||
3. **Storage paths** — add/remove, edit labels, set default, toggle schedulable, per-path app list with sizes, safe disconnect/reconnect for USB drives
|
||
4. **Password change** — current + new + confirm, min 8 chars
|
||
5. **Notifications** — email, event checkboxes, cooldown hours, test email button
|
||
|
||
---
|
||
|
||
### 9. Central Hub Reporting
|
||
|
||
#### Report Push (`internal/report/`)
|
||
|
||
Periodic JSON push (default every 15 min) to the central felhom-hub service:
|
||
- System: hostname, OS, CPU, memory, disk usage, uptime
|
||
- Containers: running/stopped counts, per-container CPU/memory
|
||
- Backup: last DB-dump run, success (disk-tier backup is the host agent's; restic password is no longer reported — removed v0.69.0)
|
||
- Health: current status, issues, warnings
|
||
- Stacks: deployed apps with versions and states
|
||
- Config hash: SHA256 of `controller.yaml` for Hub-side config comparison
|
||
- **Geo-restriction (always present, v0.70.0):** `geo_restriction` is always populated — `Enabled=false` with an empty country list when never configured — so the Hub always renders the geo section ("Inaktív" when off) instead of hiding it. `buildGeoRestrictionReport` in `internal/report/builder.go`.
|
||
- **App telemetry** (v0.28.0+): Per-stack memory (current/avg/peak) and CPU averages from the last 15 minutes of metrics data, plus log scan results (error/warning counts with deduplicated issues). Only non-protected, deployed stacks are included. Backward-compatible: old Hub versions silently ignore this field.
|
||
- **Controller telemetry** (v0.32.4+): The controller's own container (`felhom-controller`) is included as a special entry in the `app_telemetry` array. Its memory/CPU metrics come from the same metrics collector, and its log warnings/errors are scanned via `docker logs` using the same pipeline as app containers. This reuses all existing Hub telemetry infrastructure (memory trend charts, known issues, fleet aggregation) with zero Hub-side changes.
|
||
- **DR recipe — customer + apps half (v0.73.0):** `dr_recipe` is the controller half of the secret-free reconstruction recipe (`SPIKE-dr-recipe-2026-06-16.md`) that complements escrow (keys) + PBS/restic (bytes). `BuildDRRecipeAppHalf` (`internal/report/dr_recipe.go`) emits `{recipe_version, customer{id,display,domain}, apps[]}`; each deployed, non-protected app contributes `AppRecipe{catalog_ref, enabled, storage_bindings}` where bindings are parsed from the compose (`${HDD_PATH}`/`${USERDATA_PATH}` volume binds → `{container_path, drive, subpath}`, e.g. romm → `felhom-flash:userdata/roms`). **THE BOUNDARY:** the emitter is the enforcement point — it ships an explicit allowlist of those three fields and reads NOTHING from `AppConfig.Env`, so no `ENC:`/token/password can leak (allowlist, not denylist → new fields excluded by default). The load-bearing `TestBuildAppRecipe_NoSecrets` + its red-proof companion live here. The hub assembles this half with the agent's storage/guest/PBS half into one customer recipe. `recipe_version=1`, ignore-unknown on read.
|
||
|
||
Bearer token authentication, 3-attempt retry with 5-second backoff. Push status tracked via `PushStatus` struct (LastAttempt, LastSuccess, LastError, consecutive failures) — used by the monitoring page and alert system to show Hub connection health.
|
||
|
||
**Immediate out-of-cycle report on user actions (v0.139.0, generalizing the v0.70.0 geo push):** besides the periodic cycle, user actions with hub-side effects fire a **debounced, coalescing out-of-cycle report push** (`report.Trigger` in `internal/report/trigger.go`: buffered-1 signal channel + single worker; quiet window 2 s, min spacing 15 s, trailing-edge — a burst coalesces to ≤ 1 + ceil(burst/15 s) pushes and the LAST state always reaches the Hub). One canonical fire closure in `main.go` does the full `BuildReport`+`Claimed`+`Push`; the trigger adds NO retry of its own (the Pusher owns retries) and every failure degrades to the 15-min cycle, which stays the reconciliation backbone. Wired call sites: geo settings save/manual sync + app deploy/remove/delete (`api.Router.reportPushNow`), and via the `web.Server.SetReportTrigger` seam (`reportTriggerNow`, fired only AFTER a successful local commit): escrow recovery-code claim (the ACK hash-match flips pending→escrowed in seconds), notification-prefs save, app-email toggle, offsite target config + per-app offsite toggle, customer claim completion. `hub.enabled: false` → the seams stay nil (strict no-op).
|
||
|
||
**Direction 2 — hub→box wait channel (v0.140.0):** the reverse immediacy path, so an OPERATOR action on the hub reaches the box in seconds. `report.Waiter` (`internal/report/waiter.go`) holds a hanging authenticated `GET {hub}/api/v1/wait?gen=N` (same hub URL + key as the pusher; no new config keys) against hub ≥ v0.58.0's in-memory operator-intent generation counter. The hub completes the hold the instant any operator intent bumps that customer's generation (config save/delete, claim resend, offsite re-issue/freeze, floor, block/unblock, log-pull); on a generation **change** the Waiter fires the same Direction-1 `report.Trigger` — and nothing else, so the immediate report's ACK delivers everything through the unchanged config-refresh/escrow/claim/floor machinery (the box pulls even the wake-up; the hub never connects inbound). Its `http.Client` has **no overall timeout** (a held GET must stay open for the hub's ~240 s hold, which streams a 25 s heartbeat newline to defeat the nginx 60 s read-timeout — no ingress change needed); a per-request context bounds a dead connection. First-observation records-not-fires (no restart echo); a same-generation timeout fires nothing; any error (transport, a 404 from a pre-v0.58.0 hub, malformed body) backs off 5 s→5 min and the 15-min cycle keeps reconciling. Constructed beside the trigger under the same `hubPusher != nil && cfg.Hub.Enabled` gate. Grounding: `felhom.eu/documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md`.
|
||
|
||
#### Config apply + self-restart (`internal/api/router.go`, `internal/api/selfrestart.go`)
|
||
|
||
`POST /api/config/apply` (Hub-authed) writes a new `controller.yaml`, but the new config only takes effect on **restart** — singletons such as the Cloudflare client are built once at startup (so a rotated CF API token would otherwise keep failing). Behaviour (v0.70.0):
|
||
|
||
- **No-op guard:** if the pushed body is byte-identical to the current file, do nothing — no rewrite, no restart (the Hub may re-push idempotently; never flap on a no-op).
|
||
- Otherwise: write `0600`, respond `200` (flushed), then **gracefully self-restart** — `gracefulSelfRestart` waits ~500 ms for the response to flush, then `os.Exit(0)`. The container runs `restart: unless-stopped`, so Docker brings it back with the fresh config, and a startup report is pushed. The exit sits behind an injectable seam (`Router.restart` / `SetRestarter`) for unit testing.
|
||
|
||
**Manual restart (v0.70.0):** `POST /api/selfrestart` (session auth + CSRF via the `/api/` mount) runs the same helper — surfaced as the **"Vezérlő újraindítása"** button on the settings page (confirm → POST → poll `GET /` every 2 s → reload), so a customer can recover the controller without rebooting the whole guest.
|
||
|
||
**Full-server restart (v0.81.0):** `POST /api/server/reboot` (session auth + CSRF) reboots the **whole guest** via the host agent's `GuestReboot` primitive (`HandleServerReboot` in `internal/web/storage_handlers.go`, delegating to the testable `serverReboot` core; agent reboots detached + returns 202). Surfaced as a separate **"Kiszolgáló újraindítása"** settings card alongside the controller-only restart, reusing the same `pollRestart()` reload loop. It replaces the retired drive-activation banner (v0.81.0): in the intermediary-mount model an enrolled drive binds **live** into the running guest, so storage no longer needs a reboot to activate — this button is purely a deliberate full-system restart.
|
||
|
||
#### App Telemetry (`internal/metrics/telemetry.go`, `internal/metrics/logscanner.go`, `internal/report/telemetry.go`)
|
||
|
||
Each report push now includes per-app telemetry data:
|
||
|
||
**Metrics collection** (`telemetry.go`):
|
||
- `MetricsStore.GetContainerTelemetry(since)` aggregates container-level memory (avg, peak, current) and CPU averages from the `container_metrics` SQLite table for the last 15 minutes.
|
||
|
||
**Log scanning** (`logscanner.go`):
|
||
- `ScanContainerLogs(containerNames, since, logger)` runs `docker logs --since=15m --tail=1000` sequentially on all non-protected deployed containers.
|
||
- Classifies lines by keyword match (errors: `error`, `fatal`, `panic`, `crit`, `oom`, `killed`, `exception`, `traceback`; warnings: `warn`, `warning`) on the first 5 words (case-insensitive).
|
||
- Deduplicates via fingerprinting: strips ANSI escape codes, ISO timestamps (with timezone offsets), and syslog timestamps (including mid-line); replaces 6+ digit numbers with `<N>`, 8+ char hex with `<HEX>`, UUIDs with `<UUID>`. Groups identical fingerprints, keeps top 10 per container.
|
||
- Returns `[]ContainerLogSummary` with `ErrorCount`, `WarnCount`, `RecentIssues []LogIssue`.
|
||
- **Error context** (v0.111.0): each error-severity issue carries `Context []string` — up to ±5 raw lines around its FIRST occurrence in the scrape window (≤11 lines, ≤400 chars/line with `…`, ANSI-stripped, redacted). Warns never carry context. The classification loop is the pure `analyzeLogLines()` (unit-tested with synthetic windows).
|
||
- **Redaction** (`redact.go`, v0.111.0): `RedactLine()` masks `password|passwd|secret|token|api[_-]?key|authorization|bearer` values (`[REDACTED]`, incl. `Authorization: Bearer <tok>`) and 64-hex strings (`[REDACTED-HEX64]`) on every context and log-tail line before it leaves the box — controller-side, authoritative.
|
||
|
||
**Report integration** (`report/telemetry.go`):
|
||
- `buildAppTelemetrySection()` calls both, then `buildAppTelemetry()` aggregates by stack — summing container metrics, merging issues, capping at 10 per app. Additionally, `buildControllerTelemetry()` creates a special entry for the controller container itself (`app_name: "felhom-controller"`).
|
||
- Results stored as `[]AppTelemetry` in the `Report` struct field `app_telemetry`.
|
||
- **Context budget** (v0.111.0): `enforceContextBudget()` caps the per-report total of all issue-context bytes at 16KB, dropping context from the lowest-count issues first.
|
||
|
||
**On-demand log tails** (`report/logtail.go`, v0.111.0 — pull-based, same ACK-flag pattern as escrow/config-refresh):
|
||
- The report ACK (`PushResponse`) gains `log_tail_requests: [app…]` — apps the operator requested logs for on the hub. `main.go`'s `OnPushResponse` hands them to `report.SetPendingLogTails()`; the NEXT `BuildReport` drains them (consume-once) and ships `log_tails: [{app, collected_at, lines[]}]`.
|
||
- Collection: `stacks.GetLogs(app, 200)` (compose logs, ordered as emitted) for stacks; `metrics.FetchContainerLogTail()` (`docker logs --tail=200`) for the `felhom-controller` container. Caps: ≤400 chars/line, ≤64KB/app head-truncated (newest lines kept), every line through `RedactLine()`.
|
||
- Fail-safe: a failed fetch or failed push leaves the hub's request pending — the next ACK re-arms it. The hub clears the request when a tail arrives. NO hub→controller push channel exists; the guest listens to no one.
|
||
|
||
#### Infrastructure Backup to Hub — RETIRED (2026-06-16)
|
||
|
||
> **Removed.** The controller no longer pushes any infra-backup to the Hub, and the Hub no longer
|
||
> accepts or stores one (hub v0.12.0). The builder (`internal/report/infra_backup.go`) and local
|
||
> mirror (`internal/backup/local_infra.go`) were deleted back in slice 8C; the last caller-less stub
|
||
> (`Pusher.PushInfraBackup`) and the `backup_completed` event were removed in controller v0.69.0.
|
||
> DR now rests on the agent's PBS whole-CT snapshot + the Hub-generated controller.yaml. The text
|
||
> below is **historical** and describes the removed mechanism — much of this section (and the
|
||
> `local_infra.go` / `setup/scanner.go` / `PullRecovery` / `restore_drives` references elsewhere in
|
||
> this README) is stale slice-8C debt. See `felhom.eu/documentation/audits/SPIKE-infra-backup-2026-06-15.md`.
|
||
|
||
After each backup cycle (including manual Tier 2 triggers via `OnCrossDriveComplete` callback), the controller pushed a full infrastructure snapshot to the Hub for disaster recovery. This snapshot included:
|
||
- `controller.yaml` (base64-encoded, full config including secrets)
|
||
- `settings.json` (base64-encoded, backup prefs, storage paths, cross-drive configs)
|
||
- Disk layout (UUIDs, labels, mount points, fstab options, bind-mount topology)
|
||
- Deployed stacks manifest (app names, HDD paths) with actual config files: `docker-compose.yml`, `app.yaml`, `.felhom.yml` (base64-encoded per stack, v0.34.0)
|
||
- Restic passwords (primary + cross-drive, base64-encoded)
|
||
|
||
This enables fully automated recovery when the system drive is replaced — the new controller pulls the snapshot from the Hub, auto-mounts surviving drives by UUID, and restores all applications.
|
||
|
||
#### Hub Dashboard
|
||
|
||
The hub service (separate Go app in the `felhom.eu` repo) provides:
|
||
- Multi-customer overview table with status indicators and event count badges
|
||
- Customer detail page with system/storage/containers/backup/health/events sections
|
||
- Event timeline: last 50 events with severity filter, colored badges, source tracking
|
||
- Dead man's switch: staleness detection (30min stale, 60min down), missed backup detection (daily at 05:00)
|
||
- Notification dispatch: operator (English) + customer (Hungarian) emails via Resend with per-event cooldowns
|
||
- Infra backup status per customer (last sync, stack count, disk count)
|
||
- Color coding: green (<30min), yellow (30-60min), red (>60min since last report)
|
||
- 90-day report + event retention with daily prune at 04:30 Budapest time
|
||
|
||
### 10. First-Run Setup Wizard
|
||
|
||
When the controller starts with no valid customer configuration (`customer.id` empty), it enters **setup mode** — a web-based wizard that handles all initial configuration. This replaces the old interactive shell wizard in `docker-setup.sh`.
|
||
|
||
#### Setup Mode Detection (`internal/setup/setup.go`)
|
||
|
||
`NeedsSetup(cfg)` returns true when `customer.id` is empty or a `.needs-setup` marker file exists. In setup mode, the controller skips normal startup (no scheduler, no backup, no stacks) and serves only the wizard UI on two listeners:
|
||
- `:8080` — behind Traefik (accessible via domain, e.g. `https://felhom.example.com`)
|
||
- `:8081` — direct HTTP (accessible via LAN IP, e.g. `http://192.168.0.100:8081`)
|
||
|
||
#### Wizard Flow
|
||
|
||
```
|
||
┌──────────────────────────────────┐
|
||
│ 1. Welcome │
|
||
│ Choose: Restore / Fresh install │
|
||
└─────────┬───────────┬────────────┘
|
||
│ │
|
||
┌─────▼─────┐ ┌──▼───────────────┐
|
||
│ 2a. Scan │ │ 2b. Hub download │
|
||
│ drives for│ │ (customer ID + │
|
||
│ local │ │ password) │
|
||
│ backups │ │ │
|
||
└─────┬─────┘ └──────┬────────────┘
|
||
│ │
|
||
┌─────▼─────┐ │
|
||
│ 2a.2 Hub │ │
|
||
│ recovery │ │
|
||
│ (fallback)│ │
|
||
└─────┬─────┘ │
|
||
│ │
|
||
┌─────▼─────┐ ┌──────▼───────────┐
|
||
│ Execute │ │ Execute fresh │
|
||
│ restore │ │ install │
|
||
└─────┬─────┘ └──────┬───────────┘
|
||
│ │
|
||
└───────┬───────┘
|
||
▼
|
||
os.Exit(0) → Docker restarts
|
||
→ normal mode
|
||
```
|
||
|
||
#### Hub Pre-Seeding
|
||
|
||
When `docker-setup.sh` is run with `--hub-customer` / `--hub-password`, the controller receives
|
||
pre-seeded credentials via environment variables:
|
||
|
||
| Env var | Purpose |
|
||
|---------|---------|
|
||
| `FELHOM_SETUP_CUSTOMER_ID` | Pre-fills customer ID in wizard forms |
|
||
| `FELHOM_SETUP_PASSWORD` | Pre-fills retrieval password for auto-processing |
|
||
|
||
In hub mode, the welcome page shows three cards instead of two:
|
||
1. **"Visszaállítás a Hub-ról"** — auto-calls `PullRecovery()`, shows infra backup details
|
||
2. **"Visszaállítás helyi meghajtóról"** — standard drive scan
|
||
3. **"Friss telepítés"** — auto-calls `PullConfig()`, downloads config only
|
||
|
||
Both hub paths auto-process when credentials are pre-seeded (no form entry needed).
|
||
On error, the wizard falls back to the manual form with the error displayed.
|
||
|
||
#### Key Components
|
||
|
||
| File | Purpose |
|
||
|------|---------|
|
||
| `setup/setup.go` | `NeedsSetup()` detection, `SetupState` persistence to `setup-state.json` |
|
||
| `setup/handlers.go` | HTTP handlers for each wizard step (welcome, scan, hub-restore, fresh, manual) |
|
||
| `setup/scanner.go` | Scans all block devices for `.felhom-infra-backup/` directories (current + `history/`) via `lsblk` + temp mounts; returns rich info (app names, disk count) |
|
||
| `setup/hub.go` | Hub recovery pull (`GET /api/v1/recovery/{id}`) and config download |
|
||
| `setup/csrf.go` | Lightweight CSRF protection (cookie + hidden field, `SameSite=Strict`) |
|
||
| `setup/network.go` | Detects local IPs for LAN access URL display |
|
||
| `setup/templates/` | 8 embedded HTML templates (Hungarian, dark theme matching main UI) — includes `setup_hub_versions.html` for Hub backup version picker |
|
||
|
||
#### Local Infra Backup (`internal/backup/local_infra.go`)
|
||
|
||
The controller writes infrastructure snapshots to **every connected drive** after each backup cycle and on startup. Location: `<drive>/.felhom-infra-backup/`. Files:
|
||
- `backup.json` — full infra backup (config, settings, disk layout, passwords, stacks)
|
||
- `metadata.json` — schema version, timestamp, customer ID, controller version, SHA256 checksum
|
||
- `history/` — previous backup versions (last 5), rotated automatically before each write
|
||
- `{timestamp}-backup.json` + `{timestamp}-metadata.json` pairs (timestamp format: `20060102T150405Z`)
|
||
- Oldest entries pruned when count exceeds 5
|
||
|
||
During setup wizard drive scan, both current and historical backups are discovered, integrity-verified, and offered for one-click restore. The scan results table shows app names/count, disk count, and a "korábbi" badge for historical versions.
|
||
|
||
#### Recovery Info (`internal/recovery/info.go`)
|
||
|
||
Generates `recovery-info.txt` on the system data partition with customer ID, Hub URL, retrieval password, and recovery instructions in Hungarian. Updated on startup and after config changes. Also displayed on the Settings page in a "Vészhelyzeti információk" section.
|
||
|
||
### 11. Disaster Recovery
|
||
|
||
When a system drive fails and is replaced, the recovery flow uses the setup wizard:
|
||
|
||
```
|
||
1. docker-setup.sh deploys fresh controller with minimal config
|
||
- With --hub-customer: credentials pre-seeded via env vars
|
||
- Without: user enters credentials manually in wizard
|
||
2. Controller detects empty customer.id → enters setup mode
|
||
3. User opens wizard at http://<LAN-IP>:8081
|
||
4. Hub mode: welcome page shows Hub restore / local scan / fresh install
|
||
Non-hub mode: welcome page shows restore / fresh install
|
||
5. Hub restore: auto-connects to Hub, shows version picker if multiple versions
|
||
Local restore: scans all drives for .felhom-infra-backup/ directories (current + history/)
|
||
6. User selects backup version → restore: config, settings, passwords, disk layout
|
||
7. Controller restarts into normal mode with full config
|
||
8. Controller auto-mounts surviving drives by UUID from disk layout
|
||
9. Dashboard shows "Visszaállítás" (Restore) page for app-level recovery
|
||
10. User confirms → sequential restore: rsync first, restic fallback, DB import
|
||
```
|
||
|
||
**Backup sources (priority order):**
|
||
1. **Local infra backup** (`.felhom-infra-backup/` on surviving drives) — fastest, no network needed
|
||
2. **Hub recovery endpoint** (`GET /api/v1/recovery/{id}`) — requires retrieval password, supports `?version=ID` for specific versions; Hub retains ~14 versions via GFS pruning (7 daily / 4 weekly / 3 monthly)
|
||
3. **Manual config** (wizard form) — enter all details manually as last resort
|
||
|
||
**Hub verification:** After setup, the controller periodically verifies customer standing via the Hub report push response (`customer_blocked` field). If blocked or Hub unreachable for >7 days, the controller enters limited mode (no new deployments).
|
||
|
||
---
|
||
|
||
### 12. Asset Sync
|
||
|
||
App assets (logos, screenshots) are managed centrally by the Hub and downloaded to each controller via a daily sync process. This decouples asset updates from controller image rebuilds — new app icons only require a Hub redeploy.
|
||
|
||
#### How It Works (`internal/assets/syncer.go`)
|
||
|
||
```
|
||
1. Fetch manifest from Hub: GET /api/v1/assets/manifest (Bearer auth)
|
||
2. Compare SHA-256 checksums with local cache (<dataDir>/assets/)
|
||
3. Download changed/new files: GET /api/v1/assets/file/{filename}
|
||
4. Remove local files not in Hub manifest (stale cleanup)
|
||
5. Save local manifest copy for next comparison
|
||
```
|
||
|
||
#### Asset Resolution (two-tier)
|
||
|
||
| Priority | Path | Source |
|
||
|----------|------|--------|
|
||
| 1 | `<dataDir>/assets/` | Downloaded from Hub (synced cache) |
|
||
| 2 | `/usr/share/felhom/assets/` | Baked into Docker image (fallback) |
|
||
|
||
The `Resolve(filename)` method checks the synced cache first, then falls back to the baked-in directory. This ensures assets are always available even before the first sync.
|
||
|
||
The Felhom logo (`/static/felhom-logo.svg`) also uses this two-tier resolution: the logo handler checks synced assets first, then falls back to the embedded SVG constant. This allows logo updates via Hub without a controller rebuild. The logo is also used as an SVG favicon.
|
||
|
||
#### Configuration
|
||
|
||
```yaml
|
||
assets:
|
||
sync_enabled: true # Opt-in: download assets from Hub API
|
||
sync_schedule: "05:00" # Daily sync time (HH:MM, Budapest timezone)
|
||
```
|
||
|
||
Asset sync requires `hub.enabled: true` with valid `hub.url` and `hub.api_key`. The initial sync runs 10 seconds after startup (to let subsystems initialize), then daily at the configured time.
|
||
|
||
#### Sync Status
|
||
|
||
The syncer tracks status (last sync time, result, file count, total bytes) accessible via `GET /api/assets/status`. On-demand sync can be triggered via `POST /api/assets/sync`.
|
||
|
||
#### File Types
|
||
|
||
The Hub serves three asset types per app:
|
||
- `{slug}-logo.svg` — primary SVG logo
|
||
- `{slug}-logo.png` — PNG fallback
|
||
- `{slug}-screenshot-{N}.webp` — app screenshots
|
||
|
||
#### Key Design Decisions
|
||
|
||
- **Opt-in via `sync_enabled`** — backward compatible, baked-in assets still work without Hub
|
||
- **SHA-256 change detection** — only downloads files that actually changed (bandwidth efficient)
|
||
- **Atomic file writes** — downloads to `.tmp` then `os.Rename` for crash safety
|
||
- **Stale file cleanup** — removes local files not in the Hub manifest (e.g., deleted apps)
|
||
- **Non-blocking initial sync** — runs in a goroutine with 10s delay, doesn't block startup
|
||
|
||
---
|
||
|
||
### 13. Debug Mode
|
||
|
||
When `logging.level: "debug"` is set in `controller.yaml`, the controller exposes a full diagnostic dashboard at `/debug` with 9 testing sections. All debug endpoints are gated — at `info` level, the sidebar link disappears and all `/api/debug/*` routes return 404.
|
||
|
||
#### Debug Page Sections
|
||
|
||
| # | Section | Endpoints | Description |
|
||
|---|---------|-----------|-------------|
|
||
| 1 | Rendszer diagnosztika | `GET /api/debug/dump` | Full state dump: controller info, storage, stacks, network (guest-netns interfaces/route/DNS via the samba door, R-66; best-effort per item), scheduler, health, alerts. JSON download. |
|
||
| 2 | Értesítés teszt | `POST /api/debug/event/test`, `GET /api/debug/event/history` | Send test events with configurable type/severity, view event history ring buffer. |
|
||
| 3 | Mentés teszt | `POST /api/debug/backup/{dbdump,crossdrive,integrity,infra}` | Trigger individual backup phases independently. |
|
||
| 4 | Tárhely teszt | `POST /api/debug/storage/simulate-{disconnect,reconnect}`, `GET /api/debug/storage/watchdog-status` | Simulate drive disconnect/reconnect without unmounting. Per-path probe state with 5s auto-refresh. |
|
||
| 5 | Hub & Kapcsolatok | `POST /api/debug/hub/{push,infra-push,test-connectivity,preferences-sync}`, `POST /api/debug/gitea/test-connectivity` | Test Hub/Gitea connectivity with latency. Push reports and sync preferences. |
|
||
| — | Telemetria teszt | `GET /api/debug/telemetry` | Run the full telemetry collection pipeline on-demand (metrics query + log scan). Returns per-app table: container list, memory current/avg/peak, CPU avg, catalog limit, log error/warning counts, and top issues. Useful for verifying container→stack mapping and testing log scanner patterns without waiting for the 15-minute report cycle. |
|
||
| 6 | Önfrissítés teszt | `POST /api/debug/selfupdate/dry-run` | Dry-run update check: current vs new image lines, compose writability, backup state. |
|
||
| 7 | DR / Telepítő varázsló | `POST /api/debug/dr/trigger-setup`, `GET /api/debug/dr/infra-status` | Infra backup status per drive. Trigger setup mode via marker file (requires "RESET" + infra backup pre-check). |
|
||
| 8 | Naplóviewer | `GET /api/debug/logs?level=&limit=&after=`, `GET /api/debug/agent-logs` | In-memory log viewer (last 5000 entries, spill-persisted across restart — fix-6), level filter, 2s auto-refresh, color-coded entries. Two tabs (v0.116.0): **Vezérlő** (own ring) and **Ügynök** (the agent's always-DEBUG ring proxied over the local API; a pre-0.83 agent renders the "available after the agent's next update" notice). |
|
||
|
||
#### Key Implementation Details
|
||
|
||
- **Log buffer** (`internal/web/logbuffer.go`): Ring buffer implementing `io.Writer`. Since v0.116.0 it ALWAYS exists (any `logging.level`) and captures every line INCLUDING `[DEBUG]`: the logger is `io.MultiWriter(LevelFilterWriter(os.Stdout, logging.level), logBuffer)` — stdout/docker-logs keep respecting `logging.level`, the ring holds the full detail for remote diagnostics. `logBuffer.Lines(maxBytes)` renders the newest-kept plain-text tail (the report `controller_log_tail` source). New leveled lines use `internal/logx` (`Debugf/Infof/Warnf/Errorf`); legacy `isDebug()` call sites are unchanged.
|
||
- **Ring sizing, spill persistence, periodic-noise policy (fix-6, v0.120.0, CAMPAIGN-3).** The
|
||
campaign measured the 1000-entry ring wrapping in ~6.5 min under load and dying on every restart —
|
||
the exact post-incident window was the first thing lost. Three changes: **(a) cap 1000→5000**
|
||
(`Entries`/the debug handler display cap raised to match — a larger ring is useless if the viewer
|
||
can't request more than 1000 of it; the Naplóviewer default pull is 1000). **(b) periodic-noise
|
||
policy:** a periodic job's ROUTINE success is not ring-worthy — the every-cycle scheduler "job
|
||
finished" line and `refreshStatusLocked` per-cycle enumeration are logged at a new `[TRACE]` level
|
||
that the ring DROPS at write-time (`levelPriority("TRACE") < DEBUG`). Failures and state changes are
|
||
never TRACE, so nothing load-bearing is lost; this was the biggest ring filler. **(c) spill
|
||
persistence:** `LogBuffer.SpillTo`/`LoadFrom` atomically (tmp+rename, JSON-lines) spill the ring to
|
||
`<DataDir>/debug-ring.log` on the **SSD state dir** (the persistent data volume that survives
|
||
container recreation — NEVER a NAS/HDD path) every 30 s and on clean shutdown, and load it back on
|
||
boot so a restart / recreate preserves the pre-restart window. Corruption-safe: a truncated/partial
|
||
line is skipped on load, never fatal.
|
||
- **Controller self-log pull** (`internal/report/selftail.go`): the hub's report ACK may carry `controller_log_requested` — the NEXT report ships `controller_log_tail` (ring, 128 KB cap, consume-once, the v0.111.0 app-tail pattern; additive fields, app-tail wire unchanged). Serving a pull logs the customer-visible `operator log pull served` INFO line.
|
||
- **Storage simulation**: `simulatedPaths` map in watchdog prevents the watchdog from re-probing simulated-disconnected paths. Disconnect runs all real steps except `lazyUnmount` (drive stays physically mounted).
|
||
- **DR trigger safety**: Uses marker file (`data/.needs-setup`) instead of modifying controller.yaml. Pre-checks that infra backup exists on at least one drive.
|
||
- **Routing**: `/api/debug/` carved out in HTTP mux (same pattern as `/api/storage/`), routed to web server with auth + CSRF.
|
||
- **DebugCallbacks**: 7 closures wired from main.go for operations needing modules not on Server struct (hub push, infra backup, connectivity tests, telemetry preview).
|
||
- **Telemetry debug**: `GetTelemetryPreview` callback calls `report.BuildAppTelemetryForDebug()` (exported wrapper around the private `buildAppTelemetrySection()`). Result renders as a table with collapsible raw JSON. Available regardless of hub configuration.
|
||
|
||
#### Per-Module Logging
|
||
|
||
All modules emit structured log lines at `[INFO]`, `[WARN]`, and `[ERROR]` levels for operational events (state changes, completions, failures). When `logging.level: "debug"`, additional detailed `[DEBUG] [module]` prefixed log lines are emitted. Each module with stateful debug (struct-based) exposes a `SetDebug(bool)` method, wired from `main.go`. Modules without a struct use package-level `DebugLogger` variables (e.g., `system.DebugLogger`).
|
||
|
||
**Standard-level logging** (always active):
|
||
- `[INFO]` — Operational events: stack deploy/start/stop, backup completion, config changes, disk operations, sync results
|
||
- `[WARN]` — Degraded states: health threshold breaches, unsafe backup destinations, retryable failures, best-effort operation failures
|
||
- `[ERROR]` — Hard failures: data restore errors, integration apply failures, compose file update errors, disk format failures
|
||
|
||
| Module | Debug Field | Prefix | Key Areas |
|
||
|--------|------------|--------|-----------|
|
||
| `stacks` | `cfg.Logging.Level` | `[DEBUG] [stacks]` | Stack CRUD, compose commands, env vars, HDD mounts, encryption migration, health probes |
|
||
| `backup` | `ResticManager.debug` | `[DEBUG] [restic]` / `[DEBUG] [backup]` | Restic commands, snapshot operations, restore scanning, drive mounting |
|
||
| `cloudflare` | `Client.debug` + `GeoSyncManager.debug` | `[CF-DEBUG]` / `[DEBUG] [cloudflare]` | API requests/responses, WAF rule CRUD, zone resolution, geo sync diff |
|
||
| `integrations` | `Manager.debug` | `[DEBUG] [integrations]` | Toggle apply/revoke timing, lifecycle hooks, config reapply |
|
||
| `system` | `DebugLogger` | `[DEBUG] [system]` | Memory/disk/CPU/load/temp collection, mount probing, USB detection |
|
||
| `monitor` | `Pinger.debug` | `[DEBUG] [pinger]` | Health ping URLs, retry attempts, response codes |
|
||
| `settings` | `Settings.debug` | `[DEBUG] [settings]` | Load/save sizes, storage path ops, geo/integration state changes |
|
||
| `scheduler` | `Scheduler.debug` | `[DEBUG] [sched]` | Job registration, execution timing, daily schedule calculations |
|
||
| `web` | `cfg.Logging.Level` | `[DEBUG] [web]` | HTTP requests, auth decisions, session management, storage API ops |
|
||
| `api` | `Router.debug` | `[DEBUG] [api]` | API routing, handler entry points, request details |
|
||
| `selfupdate` | `Updater.debug` | `[DEBUG] [selfupdate]` | Version checks, update preconditions, docker pull timing |
|
||
| `assets` | `Syncer.debug` | `[DEBUG] [assets]` | Manifest fetch, hash comparison, file download timing |
|
||
| `storage` | logger-based | `[DEBUG] [storage]` | Disk scanning, formatting, attach, drive migration |
|
||
| `metrics` | logger-based | `[DEBUG] [metrics]` | Per-container log scanning, error/warning counts |
|
||
| `appexport` | `Exporter.debug` | `[DEBUG] [appexport]` | Export/import steps, crypto operations, bundle scanning |
|
||
|
||
---
|
||
|
||
### 14. Geo-Restriction
|
||
|
||
Country-based access control via **Cloudflare WAF Custom Rules**. The controller manages WAF rules in the `http_request_firewall_custom` phase to block requests from non-allowed countries. Rules are identified by a `[felhom-geo]` description prefix — other WAF rules are never touched.
|
||
|
||
#### Prerequisites
|
||
|
||
The existing `cf_api_token` (used for DNS-01 ACME) needs **Zone WAF:Edit** permission added. No new token is needed — just expanded permissions on the same token. The settings UI only appears when a CF API token is configured.
|
||
|
||
#### Architecture
|
||
|
||
```
|
||
┌─────────────┐ ┌──────────────────┐ ┌──────────────────────┐
|
||
│ Settings UI │────▶│ GeoSyncManager │────▶│ Cloudflare WAF API │
|
||
│ (settings. │ │ (geosync.go) │ │ /zones/{id}/ │
|
||
│ html) │ │ diff & apply │ │ rulesets/{id}/rules │
|
||
└─────────────┘ └──────────────────┘ └──────────────────────┘
|
||
│ ▲
|
||
│ POST /api/geo/* │ Scheduler (6h)
|
||
▼ │ + deploy/remove hooks
|
||
┌─────────────┐ │
|
||
│ API layer │──────────────┘
|
||
│ (geo.go) │
|
||
└─────────────┘
|
||
```
|
||
|
||
**Rule structure:**
|
||
- **Global rule**: `(not ip.src.country in {"HU"})` → block (with `http.host ne` exclusions for apps that have per-app overrides)
|
||
- **Per-app rule**: `(http.host eq "app.example.com" and not ip.src.country in {"HU" "US"})` → block
|
||
- **Block response**: HTTP 403 with Hungarian message
|
||
|
||
**Local network access** is inherently unaffected — traffic from the LAN goes directly to the server, bypassing Cloudflare entirely.
|
||
|
||
#### Cloudflare API Client (`internal/cloudflare/`)
|
||
|
||
| File | Purpose |
|
||
|------|---------|
|
||
| `client.go` | HTTP client with Bearer token auth, 15s timeout, generic `do()` helper |
|
||
| `zone.go` | Zone ID resolution — tries exact domain, then parent domains progressively |
|
||
| `waf.go` | WAF rule CRUD, expression builders (`BuildGlobalExpression`, `BuildAppExpression`) |
|
||
| `countries.go` | ~250 ISO 3166-1 alpha-2 codes with Hungarian names |
|
||
| `geosync.go` | Sync orchestrator — diffs desired vs existing rules, creates/updates/deletes |
|
||
|
||
**GeoSyncManager** uses a `StackLister` interface (implemented by `geoStackAdapter` in main.go) to get deployed app hostnames without circular imports.
|
||
|
||
#### Settings Model
|
||
|
||
Stored in `settings.json` (runtime-modifiable):
|
||
|
||
```go
|
||
type GeoRestriction struct {
|
||
Enabled bool `json:"enabled"`
|
||
AllowedCountries []string `json:"allowed_countries"`
|
||
AppOverrides map[string]AppGeoOverride `json:"app_overrides,omitempty"`
|
||
LastSync string `json:"last_sync,omitempty"`
|
||
LastSyncError string `json:"last_sync_error,omitempty"`
|
||
ZoneID string `json:"zone_id,omitempty"`
|
||
RulesetID string `json:"ruleset_id,omitempty"`
|
||
}
|
||
```
|
||
|
||
Thread-safe access via `GetGeoRestriction()`, `SetGeoRestriction()`, `SetGeoAppOverride()`, `RemoveGeoAppOverride()`, `SetGeoSyncState()`.
|
||
|
||
#### API Endpoints
|
||
|
||
| Method | Path | Description |
|
||
|--------|------|-------------|
|
||
| GET | `/api/geo/status` | Current geo settings + sync state |
|
||
| POST | `/api/geo/settings` | Update global settings (enable/disable, countries) |
|
||
| POST | `/api/geo/sync` | Trigger manual sync |
|
||
| GET | `/api/geo/countries` | Full country list for search UI |
|
||
| POST | `/api/stacks/{name}/geo/override` | Set per-app country override |
|
||
| DELETE | `/api/stacks/{name}/geo/override` | Remove per-app override |
|
||
|
||
All mutating endpoints trigger an async Cloudflare sync. The `/api/geo/` path accepts both session auth and Hub Bearer token auth (via `selfUpdateAuthMiddleware`), enabling Hub-side geo-disable for lockout recovery.
|
||
|
||
#### Sync Triggers
|
||
|
||
1. **Settings change** — user saves geo settings or per-app override
|
||
2. **Deploy/remove** — app deployment or removal changes the hostname list
|
||
3. **Scheduler** — periodic verification every 6 hours
|
||
4. **Startup** — delayed initial sync 15s after boot
|
||
5. **Manual** — "Szinkronizálás" button on settings page
|
||
|
||
#### UI
|
||
|
||
**Settings page** ("Beállítások" → "Földrajzi korlátozás"):
|
||
- Enable/disable toggle
|
||
- Searchable country autocomplete with tag-based selection
|
||
- Hungary pinned with `confirm()` warning on removal
|
||
- Per-app overrides summary with add/edit/remove
|
||
- Sync status display (last sync time, errors)
|
||
|
||
**App detail page** (per-app override, shown when geo is globally enabled):
|
||
- Toggle for custom country restriction
|
||
- Independent country selector
|
||
|
||
---
|
||
|
||
### 15. App-to-App Integrations
|
||
|
||
Generic framework for connecting deployed applications to each other. Provider apps declare available integrations in `.felhom.yml`, and users enable/disable them via toggle switches on the provider's deploy/settings page ("Beállítások").
|
||
|
||
#### Architecture (`internal/integrations/`)
|
||
|
||
- **`integrations.go`** — Core types: `Handler` interface (`Apply`/`Revoke`), `ApplyContext` (carries domain, decrypted env vars, provider metadata, stacks dir, logger, restart func), `StatusInfo` (UI data), `IntegrationKey()`/`ParseIntegrationKey()` key helpers
|
||
- **`manager.go`** — `Manager` coordinates toggle operations, builds apply contexts from decrypted app.yaml env vars. Uses `StackProvider` interface (GetStack, GetStacks, RestartStack) to break circular imports with stacks package — adapted via `integrationStackAdapter` in main.go. Key methods:
|
||
- `Toggle(ctx, provider, target, enable)` — Validates both apps deployed+running, calls Apply/Revoke, persists state
|
||
- `ListForProvider(slug)` — Returns `[]StatusInfo` for UI with target deployment/running status
|
||
- `ReapplyConfigForTarget(name)` — Re-applies all active integrations targeting a stack (config-only, no restart). Used by `SyncFileBrowserMounts` after config regeneration
|
||
- **`lifecycle.go`** — Lifecycle hooks called from API router goroutines:
|
||
- `OnStackStop` — Revokes active integrations, sets `"provider_stopped"`/`"target_unavailable"` (keeps `enabled=true`)
|
||
- `OnStackStart` — Re-applies enabled integrations after 5s delay (waits for stack state refresh). Accepts both `StateRunning` and `StateStarting` via `isStackUp()` helper
|
||
- `OnStackRemove` — Revokes and permanently deletes integration state
|
||
- **Handler implementations** — One file per integration pair (e.g. `onlyoffice_filebrowser.go`, `onlyoffice_nextcloud.go`)
|
||
|
||
#### Integration State
|
||
|
||
Stored in `settings.json` under `integrations` map (key: `"provider:target"`):
|
||
- `enabled` — User intent (survives stop/restart)
|
||
- `status` — Current state: `"active"`, `"error"`, `"disabled"`, `"provider_stopped"`, `"target_unavailable"`
|
||
- `last_error` — Most recent error message
|
||
- `enabled_at` — RFC3339 timestamp
|
||
|
||
CRUD methods in settings.go: `GetIntegrationState`, `SetIntegrationState`, `RemoveIntegrationState`, `GetIntegrationsForProvider`, `GetIntegrationsForTarget` (all use existing RWMutex + atomic write pattern).
|
||
|
||
#### Lifecycle
|
||
|
||
1. **Enable**: User toggles on → validates both apps deployed+running → calls `Handler.Apply()` → persists state as `"active"`
|
||
2. **Disable**: User toggles off → calls `Handler.Revoke()` → persists state as `"disabled"`
|
||
3. **Provider/target stops**: `OnStackStop` → calls `Handler.Revoke()` → sets status to `"provider_stopped"` or `"target_unavailable"` (keeps `enabled=true`)
|
||
4. **Provider/target starts**: `OnStackStart` (5s delay) → finds enabled integrations with non-active status → re-applies if both sides running/starting
|
||
5. **Provider/target removed**: `OnStackRemove` → revokes and deletes integration state permanently
|
||
6. **FileBrowser config regen**: `SyncFileBrowserMounts` regenerates `config.yaml` from scratch → `ReapplyConfigForTarget("filebrowser")` patches integration config synchronously → recreates the container **only when the final `config.yaml`/compose differ from the pre-sync content** (`fbNeedsRecreate` gate, v0.82.0)
|
||
|
||
**Important**: `SyncFileBrowserMounts` uses `--force-recreate` (rather than a plain `up -d`) **when something changed**, because `config.yaml` is a bind mount — without `--force-recreate`, `docker compose up -d` won't recreate the container when only the config file changes (compose only detects compose-file changes). The recreate is now **gated on an actual change** (v0.82.0, F2): a controller restart or no-op sync where the generated config+compose are byte-identical issues a plain `up -d --remove-orphans` and does **not** bounce the customer's file UI. `ReapplyConfigForTarget` calls each handler's `Apply` with a no-op `RestartStack` since the caller handles the restart.
|
||
|
||
#### Built-in Handlers
|
||
|
||
**OnlyOffice → FileBrowser** (`onlyoffice_filebrowser.go`):
|
||
- Apply: Reads `JWT_SECRET` + `SUBDOMAIN` from OnlyOffice app.yaml (decrypted), strips any existing `integrations:` block from FileBrowser `config.yaml` via `removeIntegrationsSection()`, appends new block with `url` (public HTTPS), `internalUrl` (`http://onlyoffice:80`), `secret`, `viewOnly: false`. Atomic write (`.tmp` + rename). Restarts FileBrowser
|
||
- Revoke: Strips `integrations:` block from config.yaml, restarts FileBrowser
|
||
|
||
**OnlyOffice → Nextcloud** (`onlyoffice_nextcloud.go`):
|
||
- Apply: Runs `docker exec -u www-data nextcloud php occ` commands:
|
||
1. `app:install onlyoffice` (tolerates "already installed")
|
||
2. `app:enable onlyoffice`
|
||
3. `config:app:set onlyoffice DocumentServerUrl --value=https://{subdomain}.{domain}`
|
||
4. `config:app:set onlyoffice DocumentServerInternalUrl --value=http://onlyoffice:80`
|
||
5. `config:app:set onlyoffice jwt_secret --value={JWT_SECRET}`
|
||
6. `config:app:set onlyoffice StorageUrl --value=http://nextcloud` (internal callback URL)
|
||
- Revoke: Runs `occ app:disable onlyoffice` (tolerates container not running / app not enabled)
|
||
|
||
**OnlyOffice compose template notes**: Requires Traefik middleware `X-Forwarded-Proto=https` in labels so the Document Server generates HTTPS URLs for editor resources (prevents mixed content errors in browser).
|
||
|
||
#### Metadata (`.felhom.yml`)
|
||
|
||
Provider apps declare integrations in their `.felhom.yml`. Parsed into `IntegrationDef` struct in `metadata.go`, with `HasIntegrations()` helper.
|
||
|
||
```yaml
|
||
integrations:
|
||
- target: filebrowser
|
||
label: "FileBrowser integráció"
|
||
description: "Dokumentumok szerkesztése a fájlkezelőben"
|
||
- target: nextcloud
|
||
label: "Nextcloud integráció"
|
||
description: "Dokumentumok szerkesztése a Nextcloudban"
|
||
```
|
||
|
||
#### API Endpoints
|
||
|
||
| Method | Endpoint | Description |
|
||
|--------|----------|-------------|
|
||
| GET | `/api/integrations/{provider}` | List integrations for a provider app (status, target availability) |
|
||
| POST | `/api/integrations/{provider}/{target}` | Enable/disable integration (`{"enabled": true/false}`) |
|
||
|
||
Routes registered **before** `hasSuffix`-based stack routes in router.go (see router bug pattern).
|
||
|
||
#### UI
|
||
|
||
Toggle switches on the provider's deploy/settings page ("Integrációk" section, within `deploy.html`). Data wired in `deployHandler()` for deployed apps only. Each integration shows:
|
||
- Label and description from `.felhom.yml` metadata
|
||
- Status badge: "Aktív", "Nincs telepítve", "Célalkalmazás leállítva", "Hiba"
|
||
- Toggle checkbox (disabled when target not deployed/running)
|
||
- JS `toggleIntegration()` → POST to API → reload on success
|
||
|
||
#### Wiring (main.go)
|
||
|
||
- `integrationStackAdapter` type implements `integrations.StackProvider` (same pattern as `stackAdapter`, `geoStackAdapter`)
|
||
- `integrations.NewManager(sett, adapter, domain, stacksDir, encKey, logger)` — registers built-in handlers
|
||
- Wired into API router via `SetIntegrationManager()` and web server via `SetIntegrationManager()`
|
||
|
||
---
|
||
|
||
### 16. Network File Sharing — SMB („Megosztás") (v0.144.0, R-7 slice 1; connect card + status contract v0.151.0)
|
||
|
||
LAN file sharing so the box behaves like a NAS: the customer enables sharing, sets ONE household SMB
|
||
password, and exports folders that appear in Windows Explorer / Mac Finder as `\\FELHOM\<share>`.
|
||
|
||
**Architecturally this is an EMBEDDED CONTROLLER FEATURE, not a catalog app** — three reasons:
|
||
it requires `network_mode: host` (the R-6 spike proved the default docker bridge is deaf to the LAN
|
||
multicast that WSD/mDNS discovery needs), its configuration is a dynamic share list rendered into
|
||
`smb.conf` (not env vars), and its share roots must ride the backup classification. It is therefore
|
||
the **fourth protected infra stack**: traefik / cloudflared / filebrowser / **samba**.
|
||
|
||
#### The image (`controller/infra-images/samba/`, `felhom-samba:1.1.0`)
|
||
|
||
Our own pinned image (alpine 3.21 by digest) running **four** daemons (v1.1.0) — `smbd` (445), **`nmbd`**
|
||
(NetBIOS flat-name resolution), `wsdd` (WS-Discovery), under `tini`. nmbd is not optional: the R-6
|
||
spike proved wsdd alone makes the box *visible* in Explorer while the double-click still fails
|
||
`0x80070035`, because WSD supplies an icon, not a name→IP mapping. The image is deliberately dumb —
|
||
`/etc/samba/smb.conf` is bind-mounted READ-ONLY by the controller, nothing is templated inside, no
|
||
name or password is baked, and the passdb lives on a named volume so the household password survives
|
||
container recreation. Built by `controller/scripts/build-samba-image.sh` (never `:latest`).
|
||
|
||
#### Data model (`internal/settings/smb.go`)
|
||
|
||
- `SMBSettings{Enabled, ServerName, UserSet}` — `ServerName` is the NetBIOS name (≤15, NetBIOS-safe).
|
||
- `SMBShare{Name, Path, ReadOnly, Offsite, CreatedAt}` — the share registry.
|
||
- **The SMB password is NEVER persisted.** Only `UserSet` (a boolean) is stored; the secret lives in
|
||
the container's passdb, applied via `smbpasswd` on STDIN.
|
||
|
||
#### Rendering + lifecycle (`internal/infra/samba.go`, `internal/stacks/samba.go`)
|
||
|
||
Pure renderers produce a hardened `smb.conf` (`server min protocol = SMB2`, `bind interfaces only`
|
||
on `lo eth0`, `disable netbios = no`, `map to guest = never`, per-share `force user/group = felhom`
|
||
so every SMB write lands as uid:gid 1000) and a compose file (`network_mode: host`, pinned image,
|
||
config `:ro`, passdb volume, one bind per share — `:ro` for read-only shares as defence in depth).
|
||
|
||
`ensureSamba` joins `EnsureBaseStack` after filebrowser, gated on `SMB.Enabled` (the cloudflared
|
||
conditional-deploy precedent); `ReconcileSamba` runs after every mutation. Both are idempotent —
|
||
unchanged config plus a running container performs **zero** compose calls. Config writes are atomic
|
||
(tmp+fsync+rename). A share whose drive is disconnected/decommissioned is rendered ABSENT from
|
||
smb.conf (never export a dead mountpoint) while its configuration is retained.
|
||
|
||
**Nothing in this feature deletes or moves customer files.** Disabling sharing is `compose down`
|
||
(passdb volume kept); deleting a share is a config-only edit. The only `os.Mkdir*` is the guarded
|
||
new-share-folder create.
|
||
|
||
#### UI + the picker guard (`internal/web/sharing_handlers.go`, `templates/sharing.html`)
|
||
|
||
Top-nav category **„Megosztás"** → **„Hálózati megosztás"**: enable/server-name card, household
|
||
password, shares table (Név · Mappa · Írásvédett · Felhőmentés · Törlés), and a create flow — either a
|
||
NEW folder under `<storage>/shares/` or an EXISTING folder chosen in a browse modal.
|
||
|
||
`sharingResolvePath` is the security gate for every customer-supplied path: absolute →
|
||
`EvalSymlinks` (before containment, so a planted symlink cannot escape) → must live inside a
|
||
registered, live storage root → must not be in a deny-listed system subtree → must be a directory.
|
||
**Refusals are uniform** (`Ez a mappa nem osztható meg.`) so the picker can never act as a filesystem
|
||
oracle. The deny-list is DERIVED from `stacks.SharingDeniedRoots`, itself provably a subset of
|
||
`ProtectedHDDPaths` — it can only shrink relative to the delete guard, never drift into a stale second
|
||
list. The drive root is an exact-match denial (a whole drive is never shareable) while user-data
|
||
folders under it stay shareable. `sharingResolveStorageRoot` is a separate, strictly tighter check
|
||
used only as the new-folder parent. The picker endpoint is `/api/sharing/browse`, registered on the
|
||
main mux behind `RequireAuth`+`CsrfProtect` (the `/api/` subtree is routed there, not in the web
|
||
`ServeHTTP` switch).
|
||
|
||
#### Discovery is per-platform, and the two halves do not overlap (v1.1.0)
|
||
|
||
| Client | Working form | Served by |
|
||
|---|---|---|
|
||
| Windows | `\\<NÉV>` | **nmbd** (flat-name resolution) + wsdd (Network view) |
|
||
| macOS | `smb://<NÉV>.local` | **avahi/mDNS** |
|
||
| Any | `smb://<IP>` | direct — always works |
|
||
|
||
**A Mac cannot use the bare `smb://<NÉV>`, and nothing we ship can change that.** Captured live
|
||
2026-07-20: macOS broadcasts a correct NBNS query for `<NÉV><20>`, nmbd answers in 140 µs with a
|
||
textbook positive response (flags `0x8580`, RCODE=0, right address), and macOS never opens a TCP
|
||
connection — NetBIOS there feeds legacy browsing, not `smb://` URL resolution. avahi templates its
|
||
config and `_smb._tcp` service file from `FELHOM_SERVER_NAME` at entrypoint, so a rename
|
||
re-advertises; both it and dbus are non-fatal on failure, because a discovery gap must never become
|
||
a sharing outage. **Automatic Finder-sidebar appearance is NOT claimed** — the record is published
|
||
and answers browse queries, but was not observed working on the test Mac.
|
||
|
||
**„Csatlakozás a megosztáshoz" card (v0.151.0).** Shown only while sharing is enabled: the Windows
|
||
form (`\\<NÉV>`), the Mac form (`smb://<NÉV>`), and — when derivable — the direct address
|
||
`smb://<IP>` as the fallback for networks that do not resolve the name. The address comes from
|
||
`stacks.SambaLANAddress()`, which reads the guest's netns through the **samba** container
|
||
(`network_mode: host`); the controller itself is on a docker bridge and would answer `172.x`. It is
|
||
**derived per render and stored nowhere** — the guest holds the address by DHCP, so a persisted copy
|
||
eventually misdirects customers — and an underivable address simply omits the line, because a page
|
||
without an address beats a page with a wrong one.
|
||
|
||
#### `/sharing/status` — two channels, one envelope (v0.147.0 card, v0.151.0 contract)
|
||
|
||
The bring-up poll target reports `phase` (the ensure JOB, which the page answers with a one-shot
|
||
`location.reload()` when it turns terminal-`running`) and `running` (the service LEVEL, straight from
|
||
the liveness probe). Keeping them apart is load-bearing: v0.147.0 coerced `idle`→`running` on the
|
||
PHASE channel, so every steady-state page load saw a fresh success edge and reloaded, forever. Since
|
||
v0.151.0 the level never reaches the phase channel, and a terminal `running` is **served exactly
|
||
once** (`consumeIfRunning`) so a real bring-up cannot re-arm the reload on the page it just caused.
|
||
`failed`, `needs_password` and in-flight phases are never consumed — their client path shows a card
|
||
and stops, with no reload.
|
||
|
||
#### Backup classification (`internal/stacks/samba_classify.go`)
|
||
|
||
`ClassifiedBinds("samba")` resolves from the shares registry rather than catalog metadata (samba has
|
||
no `.felhom.yml` and its binds are absolute share paths). Per-share `Felhőmentés` ON → `mandatory`
|
||
(offsite + tier-2); OFF → `optional` (tier-2 only). smb.conf/passdb are never classified.
|
||
|
||
#### Share backup EXECUTION — the sibling shares source (R-7b, v0.145.0)
|
||
|
||
The earlier KNOWN GAP is **closed**: share data is in both live tiers. It did not get there through
|
||
`GetStackClassifiedBinds` — the engines are recovery-unit shaped and Model B′ deliberately left every
|
||
per-app path byte-identical. Instead `internal/backup` runs a **sibling shares source** off the same
|
||
registry, applying the same per-share class rule:
|
||
|
||
| Tier | Entry point | Shape |
|
||
|---|---|---|
|
||
| 2 (cross-drive) | `RunSharesTier2` — after the per-stack loop in `RunAllTier2` | legs grouped by SOURCE DRIVE → `backups/secondary/_shares/<driveKey>/<share>` + `_payload/`, layout marker LAST |
|
||
| 3 (offsite) | `runOffboxSharesLeg` — after the per-app loop, before retention | ONE `restic backup --tag felhom-offbox --tag _shares` = manifest staging dir + every MANDATORY share |
|
||
| restore | `RestoreSharesScratch` → `PlaceSharesRestore` | scratch first, then a missing-only merge, each destination prefix-asserted against LIVE storage roots |
|
||
|
||
The **payload** (`shares_payload.go`) is what makes a restore give back a working feature rather than
|
||
loose files: a byte-deterministic `_shares-manifest.json` of the definitions plus a best-effort,
|
||
secret-bearing `passdb.tar`. Definitions protection is the floor — a quota-blocked offsite push
|
||
degrades to the manifest alone, never to nothing.
|
||
|
||
`_shares` is a **reserved key** (restic tag, dest root, status record). `ValidateSMBShareName` refuses
|
||
a leading underscore, and both run loops skip a `_shares` stack loudly. It never reaches a customer
|
||
surface: `backup.DisplayStackName` maps it to „Megosztások" at the notification and prose boundaries,
|
||
while the persisted set, the tag and the paths keep the raw key.
|
||
|
||
**Liveness:** `monitor.EffectiveProtected` adds `infra.SambaContainerName` exactly while sharing is
|
||
on, so a dead sharing service raises the standard protected-container issue → alert → degradation
|
||
e-mail. Note the container name is NOT the stack name (`samba` vs `felhom-samba`).
|
||
|
||
---
|
||
|
||
### 17. Async-job feedback (v0.147.x, feedback slice 1)
|
||
|
||
Three long operations that used to be silent now report what they are doing. **These are three
|
||
targeted cards on the two existing patterns** (the deploy 3-step panel and the storage-init status
|
||
poll), NOT a framework — a unified async-job layer is ROADMAP R-45.
|
||
|
||
**Verification-restore visibility** (`backup/offbox_verify_copies.go`, `web/offbox_handlers.go`,
|
||
`templates/backups_restore.html`). The offsite verification-restore flash now names the **full path**
|
||
it wrote to, and `/backups/restore` lists existing verification copies (app · size · date · path)
|
||
with a per-copy delete.
|
||
|
||
- `offsiteRestoreRootFor(drivePath)` is **THE** place `backups/offsite-restore` is spelled;
|
||
`offboxRestoreScratchDir` builds on it so listing and delete resolve byte-identical paths to what
|
||
the restore wrote.
|
||
- `ListOffsiteRestoreCopies()` sweeps every candidate drive in the same preference order the restore
|
||
path uses to *choose* one, so it can never miss a copy the restore was capable of creating.
|
||
- `DeleteOffsiteRestoreCopy(stack)` takes a **stack name, never a path** — the customer cannot hand
|
||
the controller a directory to remove. Guarded by `isSafeStackName` plus a containment assertion on
|
||
the resolved path. `POST /backup/offbox/verify-copy/delete` additionally requires `confirm=1`, is
|
||
double-confirmed in the UI, and refuses while any backup/restore op is running.
|
||
|
||
**SMB bring-up progress** (`web/samba_ensure_job.go`, `templates/sharing.html`). `/sharing/enable`
|
||
and `/sharing/password` no longer run `ReconcileSamba()` inside the POST — they start a detached
|
||
single-flight job (the `storage_init_job.go` shape) and the page polls `GET /sharing/status`.
|
||
|
||
- The opening phase is decided **before** the work starts, from `stacks.SambaImagePresent()`:
|
||
`pulling` („képfájl letöltése") when the pinned image is not in local Docker storage, else
|
||
`starting`. Afterwards the image is always present, so the distinction is unrecoverable later.
|
||
- Terminal success is **probed** via `stacks.SambaRunning()` — `compose up -d` exits 0 on a
|
||
crash-loop. A `nil` reconcile with `UserSet == false` reports `needs_password`, not `running`.
|
||
- `/sharing/status` lets live container state win over a stale/absent job, so a page loaded after a
|
||
restart still tells the truth.
|
||
|
||
**Offsite backup progress** (`backup/offbox_progress.go`, `templates/backups_remote.html`).
|
||
`RunOffboxBackupWithProgress` (the **manual** trigger only; the nightly `RunOffboxBackup` is
|
||
unchanged and stays silent) installs a progress sink, adds `--json` to the app-backup leg and scans
|
||
restic's stdout line-by-line through `offboxStreamRunner` — a streaming sibling of the existing
|
||
`offboxRunner` seam, injectable for tests. Published on the existing `GET /backup/offbox/status`
|
||
under `progress`.
|
||
|
||
- Restic's status object is only partly usable in practice, and the fallbacks matter more than the
|
||
percentage: on an **incremental** run restic transfers no bytes (`bytes_done` is `omitempty`, so
|
||
absent) and `percent_done` stays 0 for the whole run; and because restic 0.14 counts a file only
|
||
when it **completes**, an app dominated by one large archive freezes the file counters too. The
|
||
card therefore degrades: bytes -> files -> current file + elapsed seconds.
|
||
- A run is **not** only the per-app loop. `Phase` (`shares`, `retention`) names the post-app stages
|
||
and clears app-scoped counters, so the card never shows the last app's finished numbers against
|
||
work that is no longer about that app.
|
||
|
||
**Infra image pins** (`internal/infra`). `infra.Images()` returns every controller-managed infra
|
||
image, derived from the existing pinned consts. `felhom-controller --print-infra-images` prints it
|
||
(config-free by design — no `controller.yaml`, data dir or settings are touched) so the golden bake
|
||
(`felhom-agent configs/build-golden.sh`) can ask the controller image it is about to bake instead of
|
||
keeping its own list, which had already drifted. A `go/ast` test fails if a `*Image` const is added
|
||
without reaching `Images()`.
|
||
|
||
---
|
||
|
||
## Repository Layout
|
||
|
||
```
|
||
controller/
|
||
├── cmd/controller/main.go # Entry point, wires all 17 modules (setup mode branch + normal startup)
|
||
├── internal/
|
||
│ ├── config/config.go # YAML loader, validation, env overrides
|
||
│ ├── crypto/crypto.go # AES-256-GCM encryption for app.yaml secrets, key management
|
||
│ ├── settings/settings.go # Runtime settings (JSON, atomic writes, RWMutex)
|
||
│ ├── stacks/
|
||
│ │ ├── manager.go # Stack scanning, compose ops, container status
|
||
│ │ ├── metadata.go # Parse .felhom.yml app metadata
|
||
│ │ ├── deploy.go # First-deploy: secret gen, app.yaml, compose up; missing field injection
|
||
│ │ └── delete.go # Stack deletion/removal + HDD/backup data cleanup
|
||
│ ├── sync/sync.go # Git sync: clone/pull app catalog, content-hash copy
|
||
│ ├── storage/
|
||
│ │ ├── scan.go, scan_linux.go # Disk detection via lsblk + blkid
|
||
│ │ ├── format.go, format_linux.go # Partition, format, mount pipeline
|
||
│ │ ├── attach.go, attach_linux.go # Attach existing FS drive (raw mount + bind mount)
|
||
│ │ ├── safety.go, safety_linux.go # System disk detection, mount guards, fstab ops
|
||
│ │ ├── migrate.go # App data migration (rsync with progress)
|
||
│ │ └── *_other.go # Non-Linux stubs for cross-compilation
|
||
│ ├── backup/
|
||
│ │ ├── backup.go # Orchestrator (per-drive dumps + restic + cross-drive chain)
|
||
│ │ ├── paths.go # Per-drive path helpers (FelhomDataDir constant, PrimaryResticRepoPath, AppDataDir, InfraBackupDir, etc.)
|
||
│ │ ├── local_infra.go # Local infra backup to all drives (.felhom-infra-backup/)
|
||
│ │ ├── dbdump.go # DB auto-discovery + dump (pg_dump, mariadb-dump)
|
||
│ │ ├── restic.go # Restic operations (init, snapshot, prune, check) — repoPath as param
|
||
│ │ ├── appdata.go # StackDataProvider interface, app data discovery
|
||
│ │ ├── crossdrive.go # Per-app backup to secondary storage (rsync/restic)
|
||
│ │ ├── restore.go # Per-app restore from per-drive repo
|
||
│ │ ├── restore_scan.go # DR: scan drives for backup data, build restore plan
|
||
│ │ ├── restore_app_linux.go # DR: per-app restore (rsync config/data + docker compose up)
|
||
│ │ └── restore_drives_linux.go # DR: auto-mount drives by UUID from Hub infra backup
|
||
│ ├── cloudflare/
|
||
│ │ ├── client.go # CF API client (Bearer auth, generic JSON helper)
|
||
│ │ ├── zone.go # Zone ID resolution (domain → zone)
|
||
│ │ ├── waf.go # WAF rule CRUD + expression builders
|
||
│ │ ├── countries.go # ISO 3166-1 country codes + Hungarian names
|
||
│ │ └── geosync.go # Geo sync orchestrator (diff & apply rules)
|
||
│ ├── integrations/
|
||
│ │ ├── integrations.go # Core types: Handler interface, ApplyContext, StatusInfo
|
||
│ │ ├── manager.go # Manager: Toggle, ListForProvider, StackProvider interface
|
||
│ │ ├── lifecycle.go # OnStackStop, OnStackStart, OnStackRemove hooks
|
||
│ │ ├── onlyoffice_filebrowser.go # OnlyOffice → FileBrowser handler (config.yaml patch)
|
||
│ │ └── onlyoffice_nextcloud.go # OnlyOffice → Nextcloud handler (occ commands)
|
||
│ ├── assets/syncer.go # Hub asset sync (download, SHA-256 compare, resolve)
|
||
│ ├── api/
|
||
│ │ ├── router.go # REST API endpoints (~36 routes)
|
||
│ │ └── geo.go # Geo-restriction API handlers
|
||
│ ├── scheduler/scheduler.go # Central job scheduler (Every, Daily)
|
||
│ ├── system/
|
||
│ │ ├── info.go, info_linux.go # RAM, disk, CPU, temperature, load average
|
||
│ │ ├── cpu_linux.go # Background /proc/stat sampling
|
||
│ │ └── mounts_linux.go # Mount points, disk usage, FS info, backup dest checks, storage probing, USB detection
|
||
│ ├── monitor/
|
||
│ │ ├── pinger.go # Healthchecks.io HTTP ping client
|
||
│ │ ├── healthcheck.go # System health checks (disk, mem, CPU, temp, Docker)
|
||
│ │ └── watchdog.go # Storage watchdog (probe, disconnect/reconnect, safe eject)
|
||
│ ├── metrics/
|
||
│ │ ├── store.go # SQLite time-series (WAL mode, downsampled queries)
|
||
│ │ ├── collector.go # Background collector (60s, system + docker stats)
|
||
│ │ └── sysinfo.go # Static system info (/proc, /etc)
|
||
│ ├── selfupdate/
|
||
│ │ ├── version.go # Semver parsing + comparison (hand-rolled)
|
||
│ │ ├── state.go # Update audit state (JSON, atomic writes)
|
||
│ │ └── updater.go # Registry check, update trigger, startup verify
|
||
│ ├── notify/notifier.go # Email relay to hub, preference sync, cooldowns
|
||
│ ├── report/
|
||
│ │ ├── builder.go # Hub report builder (all subsystems → JSON)
|
||
│ │ ├── pusher.go # HTTP POST to hub (retry, Bearer auth, parses customer_blocked)
|
||
│ │ └── infra_pull.go # DR: pull recovery/config from Hub (retrieval password auth)
|
||
│ ├── setup/ # First-run setup wizard (web-based, replaces docker-setup.sh wizard)
|
||
│ │ ├── setup.go # NeedsSetup() detection, state persistence
|
||
│ │ ├── handlers.go # HTTP handlers for all wizard steps
|
||
│ │ ├── scanner.go # Drive scanner for local infra backups
|
||
│ │ ├── csrf.go # Lightweight CSRF (cookie + hidden field)
|
||
│ │ ├── network.go # Local IP detection for LAN access URLs
|
||
│ │ └── templates/ # 7 wizard HTML templates (Hungarian)
|
||
│ ├── recovery/info.go # Recovery info file generator (recovery-info.txt)
|
||
│ └── web/
|
||
│ ├── server.go # HTTP server, routing, static files, catch-all middleware, executeTemplate wrapper
|
||
│ ├── auth.go # Session auth + per-session CSRF token, login/logout, session cleanup
|
||
│ ├── csrf.go # CsrfProtect middleware, csrfToken/csrfField helpers
|
||
│ ├── handlers.go # Page handlers (dashboard, stacks, deploy, backups, etc.)
|
||
│ ├── handler_restore.go # DR: restore page handler + APIs (scan, restore all, skip)
|
||
│ ├── handler_debug.go # Debug page handler + 20 debug API endpoints (debug-mode only)
|
||
│ ├── logbuffer.go # Ring buffer (io.Writer) for in-memory log capture
|
||
│ ├── storage_handlers.go # Storage API handlers (scan, format, attach, migrate, cleanup, disconnect/reconnect)
|
||
│ ├── alerts.go # State-based alert generation
|
||
│ ├── funcmap.go # Template functions (state colors, Hungarian formatting)
|
||
│ ├── embed.go # go:embed for templates + Chart.js
|
||
│ └── templates/ # 21 HTML files (4-page settings split) + style.css + icons.html sprite (Hungarian UI, design system v2)
|
||
├── configs/
|
||
│ ├── controller.yaml.example # Full config reference
|
||
│ └── example-felhom-metadata.yml # .felhom.yml format reference
|
||
├── Dockerfile # Multi-stage: Go 1.24 builder + debian-slim runtime
|
||
├── docker-compose.yml # Controller's own compose (privileged, /mnt rshared)
|
||
└── go.mod # Go 1.24, deps: bcrypt, yaml.v3, modernc.org/sqlite
|
||
```
|
||
|
||
---
|
||
|
||
## Configuration
|
||
|
||
### Controller config (`controller.yaml`)
|
||
|
||
Single YAML file per customer, infrastructure-only. Does **not** contain app-specific config.
|
||
|
||
Key sections:
|
||
```yaml
|
||
customer:
|
||
name: "Demo Felhom"
|
||
id: "demo-felhom"
|
||
|
||
paths:
|
||
stacks_dir: "/opt/docker/stacks"
|
||
data_dir: "/opt/docker/felhom-controller/data"
|
||
system_data_path: "/mnt/sys_drive" # NVMe/system drive — fallback for apps without HDD
|
||
|
||
git:
|
||
repo_url: "https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git"
|
||
sync_interval: "15m"
|
||
|
||
# Per-drive backup paths are computed automatically:
|
||
# <drive>/backups/primary/restic/ — restic repo per drive
|
||
# <drive>/backups/primary/<app>/db-dumps/ — DB dumps per app
|
||
# <drive>/backups/secondary/ — cross-drive rsync + restic
|
||
backup:
|
||
enabled: true
|
||
restic_password_file: "/opt/docker/felhom-controller/data/restic-password"
|
||
db_dump_schedule: "02:30"
|
||
restic_schedule: "03:00"
|
||
retention: { keep_daily: 7, keep_weekly: 4, keep_monthly: 6 }
|
||
|
||
monitoring:
|
||
health_interval: "5m"
|
||
ping_uuids:
|
||
heartbeat: "uuid-here"
|
||
system_health: "uuid-here"
|
||
db_dump: "uuid-here"
|
||
backup: "uuid-here"
|
||
backup_integrity: "uuid-here"
|
||
|
||
web:
|
||
listen: ":8080"
|
||
setup_listen: ":8081" # Plain HTTP for setup wizard LAN access
|
||
|
||
hub:
|
||
enabled: true
|
||
url: "https://hub.felhom.eu"
|
||
api_key: "bearer-token-here"
|
||
|
||
assets:
|
||
sync_enabled: true # Download app assets (logos, screenshots) from Hub API
|
||
sync_schedule: "05:00" # Daily sync time (HH:MM, Budapest timezone)
|
||
|
||
system:
|
||
reserved_memory_mb: 384 # RAM reserved for OS + controller
|
||
```
|
||
|
||
Environment variable overrides: `FELHOM_LOGGING_LEVEL=debug`, `FELHOM_HUB_ENABLED=false`, etc.
|
||
|
||
### Runtime settings (`settings.json`)
|
||
|
||
Auto-managed by the controller. Contains password hash overrides, notification preferences, per-app backup configs, storage path registry, DB validation cache, Hub verification state (`hub_verified`, `hub_verified_at`), retrieval password for disaster recovery, and pending event queue. All writes are atomic (write `.tmp`, rename).
|
||
|
||
### Per-app config (`app.yaml`)
|
||
|
||
Auto-generated during deployment. Contains env vars, locked fields list, deploy timestamp. Secret fields are locked (read-only after first deploy). Missing fields from updated templates are auto-injected on startup and after sync (see Missing Field Injection).
|
||
|
||
**Encryption at rest**: Sensitive env values (`type: password` and `type: secret` from `.felhom.yml` metadata) are stored encrypted as `ENC:base64(nonce+ciphertext)` using AES-256-GCM. The 32-byte encryption key is stored at `{dataDir}/encryption.key` (generated on first run, 0600 permissions). Values are decrypted transparently when passed to docker-compose or displayed in the UI. The key is included in infra backups (Hub + local drives) and restored during disaster recovery. On upgrade, existing plaintext values are migrated automatically on startup.
|
||
|
||
---
|
||
|
||
## Scheduler Jobs
|
||
|
||
| Job | Type | When | Purpose |
|
||
|-----|------|------|---------|
|
||
| status-refresh | periodic | 30s | Refresh container states |
|
||
| stack-scan | periodic | 2m | Rescan stacks directory |
|
||
| heartbeat | periodic | 5m | Legacy Healthchecks ping (deprecated — Hub handles via event system) |
|
||
| system-health | periodic | configurable | Health checks + alert refresh |
|
||
| backup-cache | periodic | 5m | Refresh backup status cache |
|
||
| hub-report | periodic | 15m | Push report to central hub |
|
||
| db-dump | daily | 02:30 | Database dumps |
|
||
| backup | daily | 03:00 | Restic backup → cross-drive chain |
|
||
| backup-integrity | daily | Sun 04:00 | Restic check |
|
||
| metrics-prune | daily | 04:00 | Delete metrics older than 30 days |
|
||
| selfupdate-check | periodic | 6h | Check registry for new version (cache for UI) |
|
||
| selfupdate-auto | daily | 04:30 | Auto-update if enabled + backup not running |
|
||
| asset-sync | daily | 05:00 | Download changed app assets from Hub |
|
||
|
||
All daily jobs use Europe/Budapest timezone. Skip-if-running prevents concurrent execution. Panic recovery in all jobs.
|
||
|
||
---
|
||
|
||
## REST API
|
||
|
||
### Stack Operations
|
||
|
||
| Method | Endpoint | Description |
|
||
|--------|----------|-------------|
|
||
| GET | `/api/health` | Health check (no auth) |
|
||
| GET | `/api/stacks` | List all stacks |
|
||
| GET | `/api/stacks/{name}` | Stack details |
|
||
| POST | `/api/stacks/{name}/deploy` | First-time deploy |
|
||
| POST | `/api/stacks/{name}/start` | Start stack (409 if insufficient memory) |
|
||
| POST | `/api/stacks/{name}/stop` | Stop stack |
|
||
| POST | `/api/stacks/{name}/restart` | Restart stack |
|
||
| POST | `/api/stacks/{name}/update` | Pull + recreate |
|
||
| POST | `/api/stacks/{name}/optional-config` | Update optional env vars |
|
||
| GET | `/api/stacks/{name}/logs` | Container logs (`?raw=1` for plain text) |
|
||
| GET | `/api/stacks/{name}/hdd-data` | HDD data paths + sizes |
|
||
| GET | `/api/stacks/{name}/backup-data` | Backup data paths + sizes (DB dumps, cross-drive rsync) |
|
||
| POST | `/api/stacks/{name}/remove` | Remove deployed stack (revert to "not deployed") |
|
||
| DELETE | `/api/stacks/{name}` | Delete orphaned stack |
|
||
| POST | `/api/sync` | Trigger catalog sync |
|
||
| GET | `/api/system/info` | System info + sync status |
|
||
|
||
### Backup & Restore
|
||
|
||
| Method | Endpoint | Description |
|
||
|--------|----------|-------------|
|
||
| GET | `/api/backup/status` | Full backup status |
|
||
| POST | `/api/backup/run` | Trigger manual backup |
|
||
| GET | `/api/backup/snapshots` | List snapshots (`?stack={name}` for filtering) |
|
||
| POST | `/api/stacks/{name}/cross-backup` | Save cross-drive config |
|
||
| POST | `/api/stacks/{name}/cross-backup/run` | Trigger cross-drive backup |
|
||
| GET | `/api/stacks/{name}/cross-backup/status` | Cross-drive status |
|
||
| POST | `/api/backup/cross-drive/run-all` | Run all scheduled cross-drive backups |
|
||
| GET | `/backup/offbox/status` | Offsite run status **+ live `progress` for a manual run** (v0.147.x) |
|
||
| POST | `/backup/offbox/verify-copy/delete` | Delete ONE verification copy (stack name + `confirm=1`; v0.147.0) |
|
||
| GET | `/sharing/status` | SMB bring-up phase + live container state (v0.147.0) |
|
||
|
||
### Storage
|
||
|
||
| Method | Endpoint | Description |
|
||
|--------|----------|-------------|
|
||
| GET | `/api/storage/scan` | Scan available disks |
|
||
| POST | `/api/storage/init` | Format and mount a disk |
|
||
| GET | `/api/storage/init/status` | Format progress |
|
||
| POST | `/api/storage/attach/mount-raw` | Temp-mount partition for browsing |
|
||
| GET | `/api/storage/attach/browse?path=` | List directories on raw mount |
|
||
| POST | `/api/storage/attach/mkdir` | Create folder on raw mount |
|
||
| POST | `/api/storage/attach` | Finalize attach (bind mount + fstab) |
|
||
| GET | `/api/storage/attach/status` | Attach progress |
|
||
| POST | `/api/storage/attach/cancel` | Cleanup temp raw mount |
|
||
| POST | `/api/storage/migrate` | Start app data migration |
|
||
| GET | `/api/storage/migrate/status` | Migration progress |
|
||
| POST | `/api/storage/disconnect` | Safe disconnect (stop apps, unmount) |
|
||
| POST | `/api/storage/reconnect` | Reconnect disconnected drive |
|
||
| POST | `/api/storage/restart-apps` | Restart auto-stopped apps |
|
||
| GET | `/api/storage/status` | All storage paths with connection state |
|
||
|
||
### Self-Update
|
||
|
||
| Method | Endpoint | Description |
|
||
|--------|----------|-------------|
|
||
| GET | `/api/selfupdate/status` | Update status (cached check result + last state) |
|
||
| POST | `/api/selfupdate/check` | Force registry check |
|
||
| POST | `/api/selfupdate/update` | Trigger self-update (async) |
|
||
|
||
Self-update endpoints accept session auth OR `Authorization: Bearer <hub_api_key>` for external triggering.
|
||
|
||
### Config Management
|
||
|
||
| Method | Endpoint | Description |
|
||
|--------|----------|-------------|
|
||
| POST | `/api/config/apply` | Apply new controller.yaml from Hub (atomic write) |
|
||
| GET | `/api/config/hash` | Get SHA256 hash of current controller.yaml |
|
||
| GET | `/api/config` | Get raw controller.yaml content (text/yaml) for live diff and pull |
|
||
|
||
Config endpoints accept session auth OR `Authorization: Bearer <hub_api_key>` (same as self-update). The `/api/config/apply` endpoint:
|
||
- Accepts raw YAML body (the generated config from Hub)
|
||
- Validates YAML is parseable before writing
|
||
- Atomic write: writes to `.tmp` then `os.Rename` for crash safety
|
||
- Does NOT reload config — restart required to apply changes
|
||
- Returns `{"ok": true, "message": "Config applied. Restart controller to apply changes."}`
|
||
|
||
### Metrics
|
||
|
||
| Method | Endpoint | Description |
|
||
|--------|----------|-------------|
|
||
| GET | `/api/metrics/system` | System metrics time-series (`?range=1h|6h|24h|7d|30d`) |
|
||
| GET | `/api/metrics/containers/summary` | Current container stats |
|
||
| GET | `/api/metrics/containers/{name}` | Per-container time-series |
|
||
| GET | `/api/metrics/sysinfo` | Static system info |
|
||
|
||
### Assets
|
||
|
||
| Method | Endpoint | Description |
|
||
|--------|----------|-------------|
|
||
| POST | `/api/assets/sync` | Trigger on-demand asset sync from Hub (async) |
|
||
| GET | `/api/assets/status` | Asset sync status (last sync, file count, total bytes) |
|
||
|
||
### Integrations
|
||
|
||
| Method | Endpoint | Description |
|
||
|--------|----------|-------------|
|
||
| GET | `/api/integrations/{provider}` | List integrations for provider app (status, target availability) |
|
||
| POST | `/api/integrations/{provider}/{target}` | Enable/disable integration (`{"enabled": true/false}`) |
|
||
|
||
### Debug (debug mode only)
|
||
|
||
| Method | Endpoint | Description |
|
||
|--------|----------|-------------|
|
||
| GET | `/api/debug/dump` | Full diagnostic JSON dump (controller state, storage, stacks, backup, hub, scheduler, health, alerts). Returns 404 when `logging.level` is not `"debug"`. |
|
||
| GET | `/api/debug/telemetry` | Run telemetry collection on-demand; returns per-app metrics + log summary with latency. Response: `{latency_ms, app_count, total_errors, total_warnings, app_telemetry[]}`. |
|
||
|
||
Response format: `{"ok": true/false, "data": ..., "error": "...", "message": "..."}`
|
||
|
||
---
|
||
|
||
## App-email relay (`internal/mailrelay/`)
|
||
|
||
Gives deployed apps outbound email (password resets, invites, confirmations) through one managed path —
|
||
**app → in-process SMTP shim → hub → Resend** — with the Resend key staying **hub-side** (never on the box).
|
||
Architecture **Shape 1**: the shim runs in-process inside the controller, reusing the existing hub client.
|
||
Implements `felhom.eu/documentation/audits/SPIKE-smtp-app-relay-2026-06-28.md`.
|
||
|
||
- **The shim (`internal/mailrelay/`)** is a `go-smtp` server with two listeners — `:2525` plaintext+STARTTLS
|
||
and `:2465` implicit-TLS (self-signed cert at boot). It advertises AUTH PLAIN+LOGIN and **accepts any
|
||
credentials, ignoring them** (apps send none; some require the offer). `Data` reads the raw message, enforces
|
||
the **From-header** domain allowlist (reject `5xx` **before** any hub call), then forwards the **raw MIME** to
|
||
the hub `POST /api/v1/mail` with the controller's hub Bearer key — **single-shot** (no retry, no spool in v1).
|
||
The hub HTTP status maps to an SMTP reply (2xx→250, 4xx→451, 5xx→554). Listeners bind to the app Docker
|
||
network only (the controller container joins `traefik-public`); apps reach the shim by `felhom-controller`.
|
||
- **Lifecycle** (`lifecycle.go`): the shim starts/stops at runtime to match the global app-email toggle (no
|
||
controller restart). Wired in `main.go`, gated on a configured hub + the `mail_relay` kill-switch.
|
||
- **Toggles + injection:** a **global** toggle (`settings.AppEmail`, Settings page) and a **per-app** toggle
|
||
(`AppConfig.EmailEnabled`, on the app's config page, shown only for apps with an `smtp_mapping`). When both are
|
||
on and the app declares `.felhom.yml` **`smtp_mapping`**, `stackEnv` injects the relay env at compose time
|
||
(host=shim, port=2525, security/from per the mapping, From=`<app>@felhom.eu`) — derived each compose, never
|
||
persisted to `app.yaml`. Config knobs: `mail_relay` (listeners, `shim_host`, `from_domains`, kill-switch).
|
||
- **What the box never holds:** the Resend key, or any durable mail queue. v2 (deferred) = a separate
|
||
`felhom-smtp-shim` container + accept-and-spool retry + a `Resend-Idempotency-Key`.
|
||
|
||
---
|
||
|
||
## Build & Deploy
|
||
|
||
### Build
|
||
|
||
```bash
|
||
# On build server (192.168.0.180)
|
||
cd ~/build/felhom-controller
|
||
git -C ~/git/felhom-controller pull
|
||
./build.sh v0.20.0 --push
|
||
```
|
||
|
||
### Deploy on customer node
|
||
|
||
**Option A: Self-Update API (v0.16.0+)**
|
||
|
||
After building and pushing the new image, trigger the controller's self-update endpoint:
|
||
|
||
```bash
|
||
curl -s -X POST https://felhom.demo-felhom.eu/api/selfupdate/update \
|
||
-H "Authorization: Bearer <HUB_API_KEY>"
|
||
```
|
||
|
||
The controller pulls the new image, updates its own compose file, and runs `docker compose up -d` to replace itself. The Settings page also has a "Frissítés telepítése" button for manual triggering.
|
||
|
||
**Option B: Manual SSH (pre-v0.16.0 or fallback)**
|
||
|
||
```bash
|
||
# On customer node (e.g., 192.168.0.162)
|
||
cd /opt/docker/felhom-controller
|
||
sudo docker pull gitea.dooplex.hu/admin/felhom-controller:<VERSION>
|
||
sudo sed -i 's|image: gitea.dooplex.hu/admin/felhom-controller:.*|image: gitea.dooplex.hu/admin/felhom-controller:<VERSION>|' docker-compose.yml
|
||
sudo docker compose up -d
|
||
```
|
||
|
||
**Important:** Always use `docker compose up -d`, NOT `docker compose restart` — restart doesn't pick up new images.
|
||
|
||
### Docker Requirements
|
||
|
||
The controller container needs:
|
||
- `privileged: true` (disk operations)
|
||
- Docker socket mount (`/var/run/docker.sock`)
|
||
- `/mnt` mount with `propagation: rshared` (container mounts visible to host)
|
||
- `/dev` mounted as `/host-dev` (block device access)
|
||
- `/etc/fstab` mounted as `/host-fstab` (persistent mount config)
|
||
|
||
See `docker-compose.yml` for the full volume configuration.
|
||
|
||
---
|
||
|
||
## Roadmap
|
||
|
||
### Completed
|
||
|
||
- [x] Stack management with deploy flow and memory validation
|
||
- [x] Git-based app catalog sync
|
||
- [x] Central job scheduler
|
||
- [x] System monitoring with SQLite metrics and Chart.js charts
|
||
- [x] Healthchecks.io integration (5 ping types)
|
||
- [x] 3-layer backup system (DB dumps + restic + cross-drive)
|
||
- [x] Per-app backup restore with auto stop/restart
|
||
- [x] Storage management (scan, format, mount, registry)
|
||
- [x] Attach existing drive wizard (v0.15.0) — bind-mount subfolder from pre-formatted drive, directory browser
|
||
- [x] App data migration between storage paths
|
||
- [x] Storage watchdog (v0.17.0) — USB disconnect detection (~15s), auto-stop apps, auto-remount on reconnect, safe eject UI
|
||
- [x] Central hub reporting
|
||
- [x] Email notifications via hub relay
|
||
- [x] Settings persistence and password management
|
||
- [x] Dashboard alert system
|
||
- [x] Per-drive backup architecture (v0.14.0) — per-drive restic repos, per-app DB dumps, path helpers
|
||
- [x] Cross-drive restic pruning (v0.14.0)
|
||
- [x] Auto Tier 2 for small apps (v0.14.1) — auto-enable daily rsync for non-HDD apps when ≥2 drives
|
||
- [x] Infrastructure config in cross-drive backup (v0.14.1) — stacks dir + controller.yaml in `_infra/` + restic
|
||
- [x] Disaster recovery (v0.15.5) — Hub-based infra backup, auto-mount by UUID, restore UI with full-page takeover
|
||
- [x] Controller self-update (v0.16.0) — Watchtower-style pull + restart, Settings page UI, API key auth, auto-update scheduling
|
||
- [x] Hub-managed config (v0.20.0) — Config apply endpoint (`POST /api/config/apply`), config hash in reports for sync comparison
|
||
- [x] Config content endpoint (v0.21.1) — `GET /api/config` returns raw YAML for Hub live diff and pull operations
|
||
- [x] First-run setup wizard (v0.22.0) — Web-based wizard replaces shell scripts, drive scan for local backups, Hub recovery, fresh install flow
|
||
- [x] Setup wizard logo fix (v0.22.2) — Use embedded SVG instead of filesystem path
|
||
- [x] Hub-managed asset sync (v0.22.3) — Download app logos/screenshots from Hub API with SHA-256 change detection, daily sync schedule
|
||
|
||
### In Progress / Planned
|
||
|
||
- [ ] Update classification and auto-apply (optional/required/security markers)
|
||
- [x] Docker volume backup + Tier 2 restore (v0.33.0)
|
||
- [ ] Raspberry Pi testing (pi-customer-1)
|
||
- [x] CSRF protection on POST endpoints (v0.23.0)
|
||
- [x] Verbose debug logging across all modules (v0.24.0)
|
||
- [x] Diagnostic dump endpoint `/api/debug/dump` (v0.24.0)
|
||
- [x] Startup self-test with 9 subsystem checks (v0.24.0)
|
||
- [ ] Login rate limiting
|
||
|
||
---
|
||
|
||
## Test Environments
|
||
|
||
| Node | Hardware | Domain | Status |
|
||
|------|----------|--------|--------|
|
||
| demo-felhom | Acemagic GK3PLUS N100, 16G RAM, 512G SSD + 1TB HDD | demo-felhom.eu | Active |
|
||
| felhotest | Proxmox VM (4-16G RAM, 8 vCPU, 200G + 100G SCSI) | router.abonet.hu:33022 | Active |
|
||
| pi-customer-1 | Raspberry Pi 3B+, 1G RAM, 32G SD | pi-customer-1.local | Not yet tested |
|
||
|
||
## Related Repositories
|
||
|
||
| Repository | Purpose |
|
||
|------------|---------|
|
||
| [felhom-controller](https://gitea.dooplex.hu/admin/felhom-controller) | This repo — controller + deploy scripts |
|
||
| [app-catalog-felhom.eu](https://gitea.dooplex.hu/admin/app-catalog-felhom.eu) | Docker Compose templates + .felhom.yml metadata |
|
||
| [felhom.eu](https://gitea.dooplex.hu/admin/felhom.eu) | Website + app assets + felhom-hub service |
|