diff --git a/documentation/README.md b/documentation/README.md new file mode 100644 index 0000000..7f8f782 --- /dev/null +++ b/documentation/README.md @@ -0,0 +1,44 @@ +# Felhom — Documentation + +Felhom is a managed home-server service for Hungarian households, built on a **three-component model** +over Proxmox: + +- **Hub** — operator backend on k3s (`hub.felhom.eu`). Repo: `felhom.eu/hub/`. +- **Host agent** — one per Proxmox host; operator-tier; owns all Proxmox interaction. Repo: `felhom-agent/`. +- **In-guest controller** — one per customer LXC; Docker-only; manages the customer's apps. Repo: `felhom-controller/`. + +This directory is the central, code-verified documentation home for all three components plus the +platform and the security-audit record. + +## Sections + +### Controller (in-guest) — `controller/` +The Docker-only app-domain controller. Full per-area docs grounded in current source (v0.59.0). +→ [`controller/README.md`](controller/README.md): module map, deploy & stack lifecycle, backup +architecture, storage/monitoring/metrics, auth/hub/sync/integrations. + +### Host agent & platform — `architecture/`, `proxmox-platform.md` +The operator-tier agent and the Proxmox platform. +- [`architecture/01-topology-and-trust.md`](architecture/01-topology-and-trust.md) — topology & trust model +- [`architecture/03-host-agent.md`](architecture/03-host-agent.md) — the host agent (Go; v0.29.1) +- [`architecture/04-control-plane-authorization.md`](architecture/04-control-plane-authorization.md) — signing, escrow, authz +- [`architecture/02-controller-module-map.md`](architecture/02-controller-module-map.md) — **historical** v0.33 planning map; the live map is [`controller/module-map.md`](controller/module-map.md) +- [`proxmox-platform.md`](proxmox-platform.md) — Proxmox platform reference + +### Hub (operator backend) — `architecture/05` +- [`architecture/05-hub-architecture.md`](architecture/05-hub-architecture.md) — hub architecture (v0.11.0) + +### Security audits & remediation — `audits/` +- [`audits/deep-sweep-2026-06-13.md`](audits/deep-sweep-2026-06-13.md) — cross-repo deep audit (controller + agent) with remediation status +- [`audits/bughunt-reconcile-2026-06-13.md`](audits/bughunt-reconcile-2026-06-13.md) — reconciliation of the v0.30.3 BUGHUNT against current code + merged fix list + +### Spike & test findings — `tests/` +Per-slice spike/validation findings (phases 0–5, slices 7–10). See [`tests/`](tests/). + +## Conventions + +- **Code-verified, not memory-derived.** Architectural claims here are checked against the actual + current source; if a claim can't be verified it is omitted and flagged, not guessed. +- Per-repo operational working files (`CLAUDE.md`, `CONTEXT.md`, `CHANGELOG.md`, `BUGHUNT.md`, + `REPORT.md`, `TASK.md`) live in their own repos — they are operational, not published docs. +- Authoritative versions at last refresh: controller **v0.59.0**, agent **v0.29.1**, hub **v0.11.0**. diff --git a/documentation/REORG-NOTES-2026-06-13.md b/documentation/REORG-NOTES-2026-06-13.md new file mode 100644 index 0000000..79ebb0b --- /dev/null +++ b/documentation/REORG-NOTES-2026-06-13.md @@ -0,0 +1,105 @@ +# Documentation reorg + refresh — 2026-06-13 + +Rationale, inventory of changes, what was verified, and the doc-gaps left for review. Part of the +audit-fix + docs-centralize session (controller v0.59.0; agent v0.29.1; hub v0.11.0). + +## Goal + +Give the **controller** a documentation home under `felhom.eu/documentation/` as thorough as the +agent/platform/hub already have, and ground everything in **current code** (not memory or stale +repo-local docs — the controller's own `controller/README.md` banner referenced v0.45 while live was +v0.58). Centralize so there is one published-docs home; keep per-repo operational working files in their +repos. + +## Structure chosen (and why) + +``` +documentation/ + README.md # NEW — top-level index across controller/agent/platform/hub/audits/tests + controller/ # NEW subtree — the controller, mirroring the agent's depth + README.md # controller docs index + overview + module-map.md # current per-package map (supersedes architecture/02 planning map) + deploy-and-stack-lifecycle.md + backup-architecture.md + storage-monitoring-metrics.md + auth-hub-sync-integrations.md + audits/ # NEW — authoritative remediation record + deep-sweep-2026-06-13.md # (from controller audit/2026-06-13-deep-sweep, + status header) + bughunt-reconcile-2026-06-13.md + architecture/ 01..05, _reviews # unchanged (agent/platform/hub design docs) + proxmox-platform.md # unchanged + tests/ # unchanged (spike findings) +``` + +Why this shape: the agent/platform/hub already live under `architecture/` as numbered design docs; the +controller had **no** central home (its prose lived in the repo README, stale). Rather than renumber the +existing set, I added a sibling `controller/` subtree (feature-doc style, like the agent's areas) and a +top-level `README.md` index that ties all four components together. The four controller feature docs are +split by **area** (deploy, backup, storage/monitoring, auth/hub-glue) so each maps to a coherent set of +packages and can be maintained independently. + +## What was created / changed + +- **Created** `documentation/README.md` (top index), `documentation/controller/` (5 files), + `documentation/audits/` (2 records). All additive. +- **Superseded, not deleted:** `architecture/02-controller-module-map.md` (v0.33 planning map) is kept + for history; `controller/module-map.md` is the live map and links back to it. Used additive + cross-links rather than `git mv` because the old doc is a *planning* artifact with distinct value + (the KEEP/PORT/DELETE rationale), not a stale copy of the new one. +- **`controller/README.md` (controller repo):** slimmed to a build/dev quickstart that points here; the + architecture/feature prose now lives in `documentation/controller/`. Build instructions preserved. +- **`BUGHUNT.md` (controller repo):** added a reconciliation banner (H10 fixed, pointer to the + reconcile record); original content preserved. +- **`CONTEXT.md` (controller repo):** banner-and-below refreshed to v0.59.0 state. +- **Removed** the accidental committed artifact `controller/mnt/user-data/outputs/felhom-controller/` + (a stale duplicate of the README under a Claude-sandbox output path; see "Tidy" below). + +## What each major controller claim was verified against + +Each of the four feature docs was drafted by reading the **actual current source** and carries an inline +content grounded in these packages (the drafting captured a per-doc verification ledger; representative +anchors): +- **deploy-and-stack-lifecycle:** `internal/stacks/deploy.go` (AppConfig:98, H1 check-and-set:122-139, + CTRL-T2-1 transitional `deployed:false`:~295-336 + success flip in `runComposeDeploy`, H10 fail-closed + in `SaveAppConfig`), `manager.go` (Stack/ContainerState, up-d-not-restart), `delete.go` (H2 guards, + protected enforcement), `infra.go` (`EnsureBaseStack` TryLock + filebrowser-preserve), `healthprobe.go`. +- **backup-architecture:** `internal/appbackup/dbdump.go` (docker-ps discovery, H8 Sync+Close-before- + rename), `paths.go`/`backup.go` (`GetAppDrivePath` systemDataPath fallback — C3 moot), `recovery_unit.go` + (secret-free unit), `restore_unit.go` (fail-closed data-key gate), `tier2.go` (off-drive rsync + + rootfs-headroom guard), `quiesce/quiesce.go`, `agentapi/client.go` (whole-guest backup is the agent's). +- **storage-monitoring-metrics:** `internal/agentapi/client.go` (DiskInfo, leaf-pin), `web/agent_disk_handlers.go` + + `storage_handlers.go` (thin proxies, 409 on agent refusal), `system/dockervol.go` (reserve floor + max(5GB,10%), HTTP 507 deploy gate), `metrics/store.go` (WAL verified), `metrics/collector.go` + (cancellable ctx), `monitor/healthcheck.go` (watchdog/pinger gone). +- **auth-hub-sync-integrations:** `web/auth.go`+`csrf.go` (bcrypt, cookie flags, demo open-mode, XFF + rate-limit), `setup/`+`bootstrap/` (NeedsSetup, bootstrap.json v2), `report/`+`notify/` (zero-knowledge + hub push), `sync/sync.go` (never overwrites app.yaml), `integrations/`, `cloudflare/waf.go` + (`[felhom-geo]`), `selfupdate/updater.go` (pinned tags, never `:latest`), `assets/syncer.go`. + +## Doc-gaps / uncertainties left for review (per the cardinal rule — flagged, not guessed) + +1. **Daily volume-dump consistency:** `DumpAppVolumesSafe` (stop→dump→restart) exists, but whether the + daily scheduled path calls it or the unsafe `DumpAppVolumes`, and the exact volume-tar trigger on the + daily cadence, was not fully traced from `main.go`. Backup doc describes volume tars without asserting + live-dump consistency. (Trace `cmd/controller/main.go` schedule wiring next.) +2. **`metricsDBPath` hardcoded** to `/opt/docker/felhom-controller/data/metrics.db` in `main.go` even on + the bootstrap-managed guest (no compose). Documented as-is; whether that path is correct under the + golden/bootstrap data volume was not assessed. +3. **`onlyoffice:nextcloud` integration internals** not opened (only confirmed registered + occ-based); + documented at one-line depth. +4. **`selfUpdateAuthMiddleware`** (API auth wrapper for the hub-callback `/api/selfupdate|config|geo` + routes) referenced but its implementation not read in depth. +5. **Agent/platform/hub design docs (`architecture/01,03,04,05`, `proxmox-platform.md`)** are explicit + *design drafts*. I did NOT rewrite them — a half-verified refresh risks regressing decision content. + Known drift to fix in a dedicated pass: `05-hub-architecture.md` prose still says "felhom-hub v0.6.3" + (live is v0.11.0); `02`/`03` reference v0.33 controller precedent. The top index records the current + versions; full grounding of these against agent v0.29.1 / hub v0.11.0 is the recommended next docs slice. +6. **Front-end (templates) deploy progress markup** lives in `web/templates`, documented only via the + in-memory state it polls, not the template/endpoint markup. + +## Tidy + +- `controller/mnt/user-data/outputs/felhom-controller/{README.md,assets/README.md}` was a tracked + accidental commit (a stale 283-line duplicate README under a Claude sandbox output path, from the + initial `added controller` commit). Removed via `git rm` — clearly safe (stale duplicate, obviously + accidental path). Noted here for the record. diff --git a/documentation/controller/README.md b/documentation/controller/README.md new file mode 100644 index 0000000..061a5f4 --- /dev/null +++ b/documentation/controller/README.md @@ -0,0 +1,35 @@ +# Felhom Controller — Documentation + +The **in-guest controller** (`felhom-controller`): one per customer LXC, Docker-only, Hungarian web +dashboard. It manages the customer's app stacks and app-data backups, reports to the hub, and delegates +all Proxmox/disk operations to the host agent. Current version **v0.59.0**. + +These docs are **code-verified against current source** (`felhom-controller/controller/internal/`) and +are the authoritative architecture reference. The repo-local `controller/README.md` is a build/dev +quickstart that points here; operational working files (`CLAUDE.md`, `CONTEXT.md`, `CHANGELOG.md`, +`BUGHUNT.md`) stay in the controller repo. + +## Index + +- [module-map.md](module-map.md) — per-package map of the current controller (supersedes the v0.33 + planning map in `../architecture/02-controller-module-map.md`). +- [deploy-and-stack-lifecycle.md](deploy-and-stack-lifecycle.md) — stack model, the deploy flow + (incl. the v0.59 crash-safe `deployed` persistence + fail-closed secret encryption), protected + stacks, base-infra bring-up (`EnsureBaseStack`). +- [backup-architecture.md](backup-architecture.md) — app-data backup: DB dumps, per-app recovery + units, restore + the fail-closed data-key gate, Tier-2 off-drive copies, and the controller↔agent/PBS + split for whole-guest backup. +- [storage-monitoring-metrics.md](storage-monitoring-metrics.md) — storage registry, disk topology & + operations delegated to the agent, the v0.58 Docker-data headroom prevention layer, host metrics, + the SQLite metrics store, and health monitoring. +- [auth-hub-sync-integrations.md](auth-hub-sync-integrations.md) — auth/CSRF/sessions, the pre-auth + setup wizard + bootstrap ingestion, hub reporting & notifications, catalog sync, app-to-app + integrations, geo-restriction, self-update, and asset sync. + +## Cross-references + +- Host agent architecture: [`../architecture/03-host-agent.md`](../architecture/03-host-agent.md) +- Control-plane authorization (signing/escrow): [`../architecture/04-control-plane-authorization.md`](../architecture/04-control-plane-authorization.md) +- Hub: [`../architecture/05-hub-architecture.md`](../architecture/05-hub-architecture.md) +- Topology & trust: [`../architecture/01-topology-and-trust.md`](../architecture/01-topology-and-trust.md) +- Security audits & remediation: [`../audits/`](../audits/) diff --git a/documentation/controller/auth-hub-sync-integrations.md b/documentation/controller/auth-hub-sync-integrations.md new file mode 100644 index 0000000..95db49c --- /dev/null +++ b/documentation/controller/auth-hub-sync-integrations.md @@ -0,0 +1,121 @@ +# Controller control-plane & app-domain glue + +Source of truth: felhom-controller `internal/web` (auth/csrf/setup), `internal/report`, `internal/notify`, `internal/sync`, `internal/integrations`, `internal/cloudflare`, `internal/selfupdate`, `internal/assets` at v0.59.0. + +This document covers the in-guest controller's link to the operator control plane (the hub), its first-run onboarding, and the app-domain glue (catalog sync, app-to-app integrations, geo-restriction, self-update, asset sync). Every claim below is verified against current source; file:line citations are in the verification ledger of the implementing task. + +--- + +## 1. Auth & sessions + +The web tier is gated by two middlewares, `RequireAuth` and `CsrfProtect`, both methods on `web.Server`. + +**Password & hash priority.** `effectivePasswordHash()` resolves the active hash in priority order: (1) `settings.json` `password_hash` (customer-changed), then (2) `controller.yaml` `web.password_hash` (operator-provisioned). Empty from both sources means no auth. Hashes are bcrypt (`bcrypt.DefaultCost`); login compares with `bcrypt.CompareHashAndPassword`. + +**Sessions.** On successful login `createSession()` mints a 256-bit random hex session token AND a separate 256-bit hex CSRF token, stored server-side in an in-memory map (`s.sessions`). The session cookie is `felhom_session`, `MaxAge` 7 days, `HttpOnly`, `SameSite=Lax`, and `Secure` only when the request arrived over TLS or carried `X-Forwarded-Proto: https`. A background goroutine sweeps expired sessions every 15 minutes; `invalidateAllSessions()` clears them all after a password change. There is no server-side persistence — a controller restart drops all sessions (re-login required). + +**Demo / no-password open mode (verified).** When `authEnabled()` is false (no hash from either source), `RequireAuth` passes every request through, and `CsrfProtect` returns before any token check. So a controller with no dashboard password is fully open — both auth and CSRF are skipped. This is the demo posture (guest 9201). `/api/health` is always exempt from auth even when enabled. + +**CSRF.** Double-submit using the per-session CSRF token. Safe methods (GET/HEAD/OPTIONS) pass through. For unsafe methods, the submitted token is read from the `_csrf` form field first, then the `X-CSRF-Token` header, and compared to the session's stored token with `subtle.ConstantTimeCompare`. Two exemptions: auth disabled (above), and a valid `Authorization: Bearer ` header — the bearer is constant-time-compared against `cfg.Hub.APIKey` and, if it matches, CSRF is skipped (this is how the hub's signed callbacks and API-key clients reach mutating endpoints). An invalid bearer falls through to normal CSRF validation. Rejections return JSON 403 for `/api/` paths, plain text 403 otherwise. + +**Login rate-limit.** Per-IP: max 5 failed attempts in a 1-minute sliding window; the 6th within the window is refused with a Hungarian "too many attempts" message. The window resets after a minute of no failures, and a successful login clears the counter for that IP. **XFF caveat:** the rate-limit key (and login-failure logging) takes the client IP from the first `X-Forwarded-For` entry when present, falling back to `RemoteAddr`. Behind the per-guest reverse proxy this is the real client; a spoofable upstream XFF would let an attacker rotate the key, so the limit assumes a trusted proxy sets XFF. + +--- + +## 2. Setup wizard + first-run bootstrap + +Both run **pre-auth** — there is no password yet. They are mutually exclusive paths to a configured `controller.yaml`. + +**Gating.** `setup.NeedsSetup(cfg)` is true when `cfg.Customer.ID == ""`, or when a `.needs-setup` marker file exists in the data dir (a debug re-trigger). `main.go` calls `bootstrap.MaybeIngest(...)` first, then checks `NeedsSetup`; if still unconfigured it launches the setup server instead of the normal app. + +**Bootstrap (`internal/bootstrap`, schema `felhom.bootstrap/v2`).** The host agent's provisioning back-half writes a read-only `bootstrap.json` (default mount `/etc/felhom-bootstrap/bootstrap.json`, overridable via `FELHOM_BOOTSTRAP_PATH`). It carries ONLY: `customer.id`, `hub.url`, a per-customer `hub.retrieval_password` (secret), and the `local_api` block (`endpoint`, `fingerprint`, `token`; token secret). On an unconfigured controller, `MaybeIngest`: + +1. Is idempotent — if `cfg.Customer.ID` is already set, it never reads the file, never pulls, never clobbers `controller.yaml`. +2. Fail-safe — an absent/malformed file, a non-v2 schema, a missing required field, a missing pull function, or a pull that ultimately fails all leave cfg unchanged and log a WARN; the controller falls back to the setup wizard. A hub outage at first boot must not brick the guest. +3. On success, **pulls** the full `controller.yaml` from the hub (`PullFunc`, wired to `report.PullConfig`) authenticated by the retrieval passphrase, **merges** the per-guest `local_api` block into the pulled YAML at the map level (the hub never knows per-guest Proxmox internals), writes `controller.yaml` atomically at 0600, reloads it, and comes up configured — skipping the wizard. + +Transient pull failures (`ErrPullTransient`, wrapping `report.ErrHubUnreachable`) are retried with a 2s/4s/8s backoff (4 attempts total); permanent failures (auth/not-found) fail fast. + +**Setup wizard (`internal/setup`, pre-auth HTTP server).** Offered when bootstrap did not configure the guest. The wizard server (`setup.NewServer`) serves routes with no auth middleware (`/setup`, `/setup/fresh`, `/setup/manual`, `/setup/failed`). Disk-recovery wizardry was removed in slice 8C; today there are two modes: + +- **Fresh-from-hub** (`processFreshHub` / `autoProcessFreshHub`): the customer (or pre-seeded env vars `FELHOM_SETUP_CUSTOMER_ID` / `FELHOM_SETUP_PASSWORD` from `docker-setup.sh --hub-customer`) supplies customer-id + retrieval passphrase; `report.PullConfig` downloads the generated `controller.yaml`; `writeFreshConfig` writes it. +- **Manual** (`processManual`): `generateManualConfig()` builds a `controller.yaml` from form fields locally (no hub call). + +Both end in `writeFreshConfig`, which writes `/opt/docker/felhom-controller/controller.yaml` atomically at 0600, and — into `settings.json` — bcrypt-hashes the dashboard password (if the form supplied one) via `SetPasswordHash`, and stores the retrieval passphrase via `SetRetrievalPassword`. Wizard progress is persisted to `setup-state.json` (atomic write) so a browser crash mid-wizard survives. + +--- + +## 3. Hub reporting & notifications + +The controller reports **state**, never app secrets — consistent with the zero-knowledge posture (the hub holds only non-secret config plus a wrapped PBS key it can't decrypt; app secrets live encrypted on the guest rootfs, never sent to the hub). + +**Report (`report.BuildReport` → `report.Pusher`).** `BuildReport` assembles a snapshot: customer id/name, controller version, a SHA-256 hash of the on-disk `controller.yaml` (for hub drift comparison, not the content), controller URL (`https://felhom.`), system info (host/OS/kernel/CPU/mem/temp/load), storage (root fs + each registered user-data path, with decommissioned/disconnected markers), container counts + per-container cpu/mem, app-data backup status (DB-dump last-run only — disk-tier restic/snapshot fields are zero since that moved to the agent in slice 8C), health (status/issues/warnings), deployed/available stack lists, per-app telemetry, and geo-restriction status. **No app secrets, env, or volume contents are in the report.** + +`Pusher.Push` POSTs the JSON to `/api/v1/report` with `Authorization: Bearer `, retrying 3 times with 5s backoff; it tracks last-attempt/last-success/consecutive-failures and parses the response for a `customer_blocked` flag (relayed via `OnPushResponse`). Cadence: `sched.Every("hub-report", cfg.Hub.PushInterval)`, default 15m when unparseable, only when `cfg.Hub.Enabled`. A separate `PushInfraBackup` POSTs to `/api/v1/infra-backup`; `PushOnce` sends a single report regardless of the enabled flag (e.g. a reporting-disabled startup notice). + +**Notifications (`notify.Notifier`).** Structured events POSTed to `/api/v1/event` with the bearer key. Enabled only when both hub URL and API key are set. Sends are non-blocking (goroutine), 3 attempts with 3s backoff; **cooldown/dedup is the hub's job** — the controller sends unconditionally. Event types include `controller_started`, `controller_updated`, `backup_completed`/`backup_failed`, `db_dump_completed`/`db_dump_failed`, `backup_integrity_ok`/`backup_integrity_failed`, `crossdrive_completed`/`crossdrive_failed`, `health_critical`/`health_degraded`/`health_recovered` (driven by status-rank change detection), `storage_disconnected`/`storage_reconnected`, `app_deployed`/`app_removed`, `disaster_recovery_started`/`disaster_recovery_completed`, plus `test`. Each event carries `customer_id`, severity, a Hungarian message, and an optional typed `details` object. A 50-entry ring buffer (`GetEventHistory`) records each send (type/severity/message/hub-status/error) for the debug page. `SyncPreferences` POSTs email + enabled-event list + cooldown to `/api/v1/preferences` (synchronous). A legacy `Notify` → `/api/v1/notify` path is retained for old hubs. + +--- + +## 4. Catalog sync (`internal/sync`) + +A periodic git pull of the app catalog into the local stacks dir. `Syncer.Start` clones (`--depth 1 --branch `) into `/catalog-cache` on first run, else `fetch --depth 1` + `reset --hard origin/`; it runs an initial sync on startup and then on `cfg.Git.SyncInterval` (default 15m). Disabled (manual mode) when `cfg.Git.RepoURL` is empty. `TriggerSync` debounces to once per 30s. Credentials, if configured, are injected into the HTTPS URL and **masked in all logs**. + +**Safety — never overwrites app.yaml.** `copyTemplates` syncs only `docker-compose.yml` and `.felhom.yml` per app directory; `app.yaml` (the per-guest deployed config carrying encrypted secrets) is never in the sync set and is never touched. Copies are **content-hash gated** (`copyIfChanged` / `copyTemplates` compare SHA-256 of src vs dst and skip identical files), so an unchanged catalog produces no writes and no USB thrash. New app dirs are reported as `new_apps`, changed ones as `updated`; a rescan + a post-sync hook (missing-deploy-field injection) fire only when something changed. + +--- + +## 5. App-to-app integrations (`internal/integrations`) + +Lets one app configure another (e.g. enable OnlyOffice editing inside FileBrowser). + +**Adapter pattern.** `integrations.Manager` depends on a `StackProvider` interface (`GetStack`/`GetStacks`/`RestartStack`); `integrationStackAdapter` in `main.go` bridges `stacks.Manager` to it, breaking the import cycle. Handlers are keyed `"provider:target"` and registered at construction. Integration enable/disable state lives in `settings.json` (`SetIntegrationState`/`GetIntegrationState`); apply/revoke are serialized under a mutex. + +**Concrete handlers (registered in `NewManager`):** +- `onlyoffice:filebrowser` — `Apply` reads OnlyOffice's `JWT_SECRET` and subdomain from the provider's decrypted `app.yaml` env, then patches FileBrowser's `config.yaml` with an `integrations.office` block (public office URL, internal `http://onlyoffice:80`, the shared JWT secret) and restarts FileBrowser. `Revoke` strips that block and restarts. Writes are atomic (tmp+rename). FileBrowser is treated as always-present infrastructure (not gated on "deployed"). +- `onlyoffice:nextcloud` — patches Nextcloud via occ commands (separate handler). + +**Lifecycle hooks.** `Toggle` validates that provider (and target, unless it's filebrowser) are deployed and running before applying. The manager exposes: +- `OnStackStop` — revokes active integrations where the stack is provider or target, but keeps `enabled=true` and marks status `provider_stopped`/`target_unavailable` so they can come back. +- `OnStackStart` — after a 5s delay (so the 30s stack-state refresh catches up), re-applies previously-enabled integrations once both sides are running/starting. +- `OnStackRemove` — permanently revokes (best-effort) and deletes integration state for the removed stack. + +`ReapplyConfigForTarget` re-applies config without restarting (the caller, e.g. FileBrowser mount sync, handles restart). + +--- + +## 6. Geo-restriction (`internal/cloudflare`) + +The controller enforces country-based access via Cloudflare custom-WAF rules using a CF API token; this is the controller's own CF-API geo layer (distinct from any hub-side config). + +**Rule identity.** All managed rules carry the `[felhom-geo]` description prefix — the global rule is exactly `[felhom-geo] Global`, per-app rules are `[felhom-geo] app:`. `GetFelhomRules` filters the zone's `http_request_firewall_custom` ruleset to that prefix, so the controller only ever touches its own rules. Rules use the `block` action (plain 403; custom response bodies need a paid plan). Expressions are built from allowed ISO country codes: global = `(not ip.src.country in {...})` with per-app hostnames excluded; per-app = `(http.host eq "" and not ip.src.country in {...})`. Zero allowed countries means block-all. + +**Sync.** `GeoSyncManager.Sync` resolves zone-id and ruleset-id (creating the custom ruleset if absent, caching ids in settings), lists existing felhom rules, builds the desired set from `settings.GeoRestriction` + deployed hostnames (via the `StackLister`/`geoStackAdapter`), and diffs: create new, update changed-expression, delete obsolete. When geo is nil or disabled, `deleteAllRules` removes every felhom rule. Sync state (zone/ruleset ids, last error) is saved back to settings. + +**Cadence & hooks.** A delayed (15s) non-blocking initial sync at startup; periodic `geo-verify` every 6h; and `OnGeoRelevantChange` (wired to the API router) re-syncs on app deploy/remove — each guarded so it only runs when geo is enabled. + +--- + +## 7. Self-update (`internal/selfupdate`) + +**Version check.** `CheckForUpdate` queries the Gitea Docker Registry V2 tags API (`/v2///tags/list`) with basic auth from the git credentials, parses tags as semver (skipping `latest`/`dev`/non-semver), and reports the highest. A `dev` running version can't check. Scheduled as `selfupdate-check` on `cfg.SelfUpdate.CheckInterval` (default 6h) — **check only, never triggers an update**. The running version is injected at build time via ldflags (`-X main.Version`); the source default is `"dev"`. + +**Update mechanism (compose-managed path).** `TriggerUpdate` guards against concurrent updates, dev versions, an in-progress backup (`backupRunning` callback), and an inaccessible compose file, then runs `performUpdate` in a goroutine: write `pending` state → `docker pull :` → rewrite the `image:` line in `docker-compose.yml` (atomic tmp+rename, regex-pinned to `gitea.dooplex.hu/admin/felhom-controller:`) → `docker compose up -d` (which replaces the running container). On next boot, `VerifyStartup` reads the pending state and marks it `success` if the running version equals the target, else `failed`. Auto-update is a separate daily job (`selfupdate-auto` at `cfg.SelfUpdate.AutoUpdateTime`) gated on `cfg.SelfUpdate.AutoUpdate`. + +**Never `:latest`.** The image reference is always `:` with a concrete semver tag — both the registry query and the compose rewrite pin an explicit version; `latest`/`dev` tags are explicitly skipped during version selection. + +> Deployment note: on the bootstrap-managed golden guest (e.g. 9201) the controller has no `docker-compose.yml` — it is started by `felhom-controller-bootstrap.service` from `/etc/felhom-controller-image`. There the compose-rewrite self-update path does not apply; version drift is handled by the bootstrap service pulling the pinned tag. The compose path above is the `/opt/docker/felhom-controller` deployment model. + +--- + +## 8. Asset sync (`internal/assets`) + +Branding/app assets are sourced from the hub with a baked-in fallback. `Syncer.Sync` GETs `/api/v1/assets/manifest` (bearer key), compares each manifest entry's SHA-256 against the local cache (`/assets`), downloads only new/changed files (atomic tmp+rename), and removes local files absent from the manifest; it saves a local copy of the manifest. Cadence: an initial sync on startup plus a daily `asset-sync` job at `cfg.Assets.SyncSchedule`. `Resolve(filename)` returns the cached path if present, else the baked-in fallback (`/usr/share/felhom/assets`), else the cached path (so the caller emits a clean 404). Filenames are base-sanitized to prevent path traversal. + +--- + +## Cross-references + +- Hub API contract and zero-knowledge config generation: hub `internal/api`, `internal/configgen` (felhom.eu repo). +- Agent-side bootstrap emission and provisioning: `felhom-agent` `internal/provision` / `internal/localapi`. +- Backup/recovery-unit and Tier-2 mechanics: see the controller backup documentation. diff --git a/documentation/controller/backup-architecture.md b/documentation/controller/backup-architecture.md new file mode 100644 index 0000000..c2c15ea --- /dev/null +++ b/documentation/controller/backup-architecture.md @@ -0,0 +1,109 @@ +# Controller backup architecture + +Source of truth: felhom-controller `internal/appbackup/`, `internal/backup/`, `internal/quiesce/` at v0.59.0; whole-guest backup is the agent's (`felhom-agent`). + +This document describes what the in-guest controller actually does for backup and restore at v0.59.0. The controller was de-privileged in slice 8C: restic, cross-drive-to-arbitrary-disks execution, drive-recovery and infra-backup were removed from the controller and now live in the host agent + PBS. The `restic` string still appears in the backup package only inside comments that record its removal (`backup.go:20`, `restore.go:14`, `restore.go:19`); there is no restic execution in the controller. + +## 1. The split: controller vs agent/PBS + +The controller owns the **app-data domain** only. The agent owns the **whole-LXC domain**. + +| Domain | Owner | What | +|---|---|---| +| Per-app DB dumps | controller | `pg_dump` / `mariadb-dump` via `docker exec` | +| Per-app Docker-volume tars | controller | `docker run alpine tar` of named volumes | +| Per-app secret-free recovery unit | controller | compose + dumps + `manifest.json` on the app's drive | +| Tier-2 off-drive copy | controller | `rsync` mirror of an HDD app's unit + userdata to a different physical disk | +| Whole-LXC vzdump snapshot | agent | `POST /backup`, crash- or stop-consistent vzdump to local / PBS | +| Offsite / encrypted backup, verify, restore-test | agent + PBS | reported read-only to the controller | +| Whole-guest restore (rootfs, secrets, keys) | agent + PBS | the controller refuses any restore that needs this | + +The boundary is precise: app data that lives on an **external user-data drive** (an HDD bind mount) is NOT inside the PBS whole-guest snapshot — PBS cannot reach bind mounts. That data is protected only by the controller's recovery unit + Tier-2 copy. App data that lives on the **rootfs** (non-HDD apps) IS inside the PBS whole-guest snapshot, so it gets no Tier-2 copy (`tier2.go:197-218`). The encrypted `app.yaml` and the controller's encryption key live on the rootfs, so they are inside PBS too — which is exactly why the restore path recovers secrets from the guest rather than storing them off-rootfs (see §3, §4). + +The agent surface the controller talks to is `internal/agentapi/client.go`: a TLS client pinning the agent's self-signed leaf by SHA-256 (`client.go:69-82`) and authenticating with a per-guest bearer token. Backup-relevant calls: `BackupDue` (`GET /backup/due`), `StartBackup` (`POST /backup`), `BackupStatus` (`GET /backup/status`), `RestoreTestStatus` (`GET /restore-test/status`). The whole-guest backup record (`BackupRecord`) and restore-test record are rendered read-only — the comment at `client.go:135-136` states the controller does not own whole-guest backup. + +## 2. Database dumps (`internal/appbackup/dbdump.go`) + +**Discovery** is `docker ps`-driven. `DiscoverDatabases` (`dbdump.go:66`) runs `docker ps --format {{.ID}}\t{{.Names}}\t{{.Image}} --filter status=running` and classifies each container by image substring: `postgres` → Postgres, `mariadb`/`mysql` → MariaDB; everything else is skipped. The stack name is derived by stripping a known DB suffix from the container name (`deriveStackName`, `dbdump.go:536` — `postgres`/`db`/`mariadb`/`mysql`/`database`/`redis`/`cache`). Connection details come from the container's own env (`populateDBEnv`, `dbdump.go:477`): `POSTGRES_USER`/`POSTGRES_DB` (defaults `postgres`), or `MYSQL_DATABASE`/`MARIADB_DATABASE` with root. + +**Per-DB dump** is `DumpOne` (`dbdump.go:162`), one container at a time, 5-minute timeout each. It re-checks the container is still running, then: + +- Postgres: `docker exec pg_dump -U -d --clean --if-exists --no-owner --no-privileges` (`dbdump.go:202`). +- MariaDB: `docker exec mariadb-dump -u root -p --single-transaction --routines --triggers ` (`dbdump.go:228`). The root password is read from the container env (`MYSQL_ROOT_PASSWORD`/`MARIADB_ROOT_PASSWORD`, `dbdump.go:516`); the password is never logged (a redacted `-p***` form is built only for the debug log line). + +**The tmpfile safety (H8).** The dump streams to `.sql.tmp`, then before the rename to the final `.sql`: `tmpFile.Sync()` then `tmpFile.Close()` are called explicitly, each removing the tmp and failing the dump on error (`dbdump.go:266-278`). Only after a non-empty stat (`dbdump.go:281`) is the tmp atomically `os.Rename`d to the final path (`dbdump.go:293`). This guarantees the data is flushed to disk before the rename makes the dump "visible", so a crash never leaves a half-written `.sql`. Stale `.tmp` files older than one hour are reaped at the start of each run (`cleanupTmpFiles`, `dbdump.go:553`). Each finished dump is structurally validated (`ValidateDump`, `dbdump.go:320`): header line + at least one `CREATE TABLE`, scanned line-by-line with a bounded `bufio.Reader` so large data lines do not allocate (H1). + +**Where dumps are stored.** `Manager.GetAppDrivePath(stack)` (`backup.go:88`) returns the app's `HDD_PATH` if it has one, else falls back to the configured `systemDataPath` (the internal SSD) for SSD-only apps. That drive path is mapped to its felhom-data namespace root by `namespaceRoot` (`backup.go:105` — Model A: an in-guest drive mount IS the namespace root, so it is used as-is; only the system-data fallback gets the `felhom-data` subdir appended). Dumps land in `AppDBDumpPath(nsRoot, stack)` = `/backups/primary//db-dumps/` (`paths.go:58`). The SSD-only fallback is the case that the C3 DR finding was about — and it is handled here: SSD-only apps get a correct path via `systemDataPath`, so there is no path gap. (The fallback only warns when `systemDataPath` itself is unconfigured, `backup.go:75`/`backup.go:94`.) + +`RunDBDumps` (`backup.go:143`) acquires the running flag, discovers, dumps each DB to its app's path (skipping drives marked disconnected/decommissioned), persists each validation result to `settings.json`, and finally refreshes every recovery unit (`captureAllRecoveryUnits`, `backup.go:247`) — even on partial DB failure, so units never go stale. + +## 3. Recovery units (`internal/backup/recovery_unit.go`) + +A recovery unit is a per-app, **secret-free**, self-contained directory at `/backups/primary//` (`paths.go:42`). It contains: + +- `compose/` — `docker-compose.yml` + `.felhom.yml` + a **secret-stripped** `app.yaml`. +- `db-dumps/` — the `.sql` dumps from §2. +- `volume-dumps/` — named-volume `.tar` archives. +- `manifest.json` — the `RecoveryManifest` (`recovery_unit.go:31`). + +What the unit **excludes**: it holds no secret values, no data-encrypting keys, and not the Docker image. The manifest stores only the pinned image tag(s) (`ImagePins` — re-pulled on restore), the **names** of the secret env vars (`SecretEnvVars`), and the names of the data-key env vars (`DataKeyEnvVars`); `SecretSource` records in plain text that the values come from "guest app.yaml (live rootfs) or PBS whole-guest snapshot — never stored in this unit" (`recovery_unit.go:141`). The stripped `app.yaml` carries only non-secret env, with a header naming the omitted secrets (`buildStrippedAppYaml`, `recovery_unit.go:188`). + +`CaptureRecoveryUnit` (`recovery_unit.go:68`) pulls the app's `RecoveryInfo` from the stack provider (`GetStackRecoveryInfo`), builds the captured content in memory, and is **idempotent**: it skips all drive writes when the existing manifest matches the current controller version, config checksums (sha256 of each captured file), and the DB/volume dump set (`recovery_unit.go:112-118`). This is what lets it run on the 5-minute status refresh without thrashing a spinning USB drive. Writes are atomic (`atomicWrite`, `recovery_unit.go:270` — tmp + rename). `DataKeyEnvVars` is a fail-closed restore annotation only (see §4); it does not affect capture. + +The `.fab` portable export/import path (`internal/appexport/`) is a separate, operator-driven mechanism documented elsewhere — cross-reference that doc; it is not the periodic recovery unit. + +## 4. Restore + +Two keep-side restore entry points exist; neither does a whole-guest restore (that is the agent's). + +**`RestoreApp(stack, snapshotID)`** (`restore.go:21`) — the legacy volume-only path. Stops the stack, re-imports the named-volume `.tar` dumps (`restoreDockerVolumes`, `restore.go:85`: `docker volume rm -f` + `create` + `docker run alpine tar xf`), restarts, and health-checks. `snapshotID` is retained only for signature/logging compatibility now that restic is gone (`restore.go:19-20`). + +**`RestoreFromRecoveryUnit(stack)`** (`restore_unit.go:74`) — the recovery-unit path. It reads the unit manifest (falling back to `RestoreApp` if no unit exists, `restore_unit.go:99`), recovers the secret values from the **guest's own live `app.yaml`** via `stackProvider.RecoverStackSecrets` (never from the unit), reconciles them, restores the named-volume data, then `RecreateStackFromUnit` rebuilds the app's definition from `compose/` and redeploys with the reconstructed env (re-pulling the pinned image). Nothing is regenerated; no secret is read from the unit. + +**The fail-closed data-key gate** is `reconcileRestoreSecrets` (`restore_unit.go:22`) — a pure, unit-tested function. It merges non-secret env with recovered secrets, then: + +- A missing **resettable** secret (DB password, admin password) is non-fatal: returned in `missing`, the caller warns and proceeds (`restore_unit.go:117`). +- A missing **data-encrypting key** (`DataKeyEnvVars`) is **fatal**: the restore is refused with an explicit error directing the operator to do a PBS whole-guest restore first, because regenerating the key would render the stored data unreadable (`restore_unit.go:45-50`). This is the safety centerpiece: the controller never silently recreates an app whose data it can no longer decrypt. + +### 4b. Security note — `.fab` import path validation (CTRL-001, v0.59.0) + +The portable `.fab` import (`internal/appexport/`) validates every manifest path segment before it reaches a `filepath.Join` against a trusted base (`appexport.ValidateSegment`, `validate.go:28`; `validateManifestPaths`, `validate.go:51`, called from `UnmarshalManifest`). The attacker-controllable `AppName` / `HDDSubdirs` / `VolumeNames` are rejected on any `..`, path separator, or absolute path, closing the v0.59.0 path-traversal finding. This is cross-referenced here; the detail lives in the appexport doc and the v0.59.0 audit record. + +## 5. Tier-2 cross-drive (`internal/backup/tier2.go`) + +Tier 2 is the only off-drive protection browsable HDD userdata can get (PBS cannot reach bind mounts, `tier2.go:17-23`). It is an `rsync -a --delete` **mirror** (`rsyncMirror`, `tier2.go:358`) of an HDD app's recovery unit + bulk `appdata/` to `/backups/secondary//{recovery-unit,appdata}/` on a **different physical disk**. + +**Auto-target selection** (`selectTier2Target`, `tier2.go:54`), in order: + +1. A customer-pinned target (`PreferredTarget` from the config panel) if it is still registered, schedulable, and off-disk (`tier2.go:62-83`). +2. Another registered user-data drive on a different physical disk — can hold bulk userdata (`tier2.go:86-102`). +3. The internal SSD (system data path) — **small units only**, headroom-guarded. + +Off-disk-ness is decided by `system.SamePhysicalDevice` (a `Stat_t.Dev` compare). If the only candidate is the same physical disk, `errNoOffDiskTarget` is returned. + +**The rootfs-headroom guard** is the key safety. The internal SSD is the ~8 GB guest rootfs, so option 3 refuses rather than fills: `tier2FitsSystemDrive` (`tier2.go:121`) → `tier2FitsHeadroom` (`tier2.go:43`) requires the copy to leave a reserve of `max(2 GB, 20% of total)` free, else returns `errSSDNoHeadroom`. A non-fitting or single-drive case is recorded as an honest `no_target` status (`recordTier2NoTarget`, `tier2.go:331`) with a Hungarian "needs a 2nd HDD" reason — the rootfs is never filled. When the SSD target is used, it is labelled DB/config-only (`tier2.go:116`, log suffix `[SSD: DB/config only]`). + +`RunAllTier2` (`tier2.go:199`) iterates deployed stacks, processes only those with an `HDD_PATH` (non-HDD apps are skipped — they are already in PBS), and skips disconnected/decommissioned drives. Status is persisted into `settings.CrossDriveBackup` (method `rsync`), with the customer-preference fields (`UserDisabled`, `PreferredTarget`) preserved across runner writes by `withTier2Prefs` (`tier2.go:290`). The config-panel view is `Tier2Info` (`tier2.go:245`, read-only). + +## 6. Concurrency, scheduling, and the quiesce loop + +**Single-flight / running mutex.** `Manager` guards a `running` flag with a mutex; `acquireRunning`/`releaseRunning` (`backup.go:373`) reject a second backup or restore with "already in progress". `RestoreApp` and `RestoreFromRecoveryUnit` take the same flag (`restore.go:31`, `restore_unit.go:79`). + +**Scheduling** (wired in `cmd/controller/main.go`, Europe/Budapest): + +- `db-dump` daily at `cfg.Backup.DBDumpSchedule` (default `02:30`, `config.go:271`) → `RunDBDumps` (which also refreshes recovery units) (`main.go:328`). +- `tier2-backup` daily at `03:30` → `RunAllTier2` (`main.go:360`). +- `RefreshCache` runs on the 5-minute status refresh, re-scanning dump files and (idempotently) re-capturing recovery units (`backup.go:477`). + +Manual triggers exist via the API router (`RunDBDumps`, `RunAllTier2` launched as goroutines, `router.go:764`/`781`). + +**The quiesce loop** (`internal/quiesce/quiesce.go`) drives the whole-guest backup app-consistently. The agent's vzdump is crash-consistent only (an LXC has no fsfreeze), so the controller stops the app stacks first. `Loop.Run` polls `GET /backup/due`; when due, `quiesceAndPoll` (`quiesce.go:205`): + +1. Writes a persisted marker (atomic, `0600`) listing the stacks it is about to stop — **before** stopping anything (`quiesce.go:207`). +2. Stops the running app stacks. +3. `POST /backup`, records the job id, polls `GET /backup/status`. +4. Resumes early at the `snapshotted` phase (8B.2 downtime optimization — the storage snapshot has captured the stopped state, so the app may come back up; the loop keeps polling to `done`/`failed`), or at `done` in stop/downgraded mode (`quiesce.go:256-279`). + +Unquiesce is **guaranteed**: a deferred closure restarts exactly the stopped stacks and clears the marker on every exit path — backup error, status-poll error, the `MaxQuiesce` bound (default 30 min, restarts the app while the backup continues on the agent), or context cancellation (`quiesce.go:213-225`, `quiesce.go:243-249`). On startup, `Recover` (`quiesce.go:113`) restarts any stacks left stopped by a mid-quiesce crash, then clears the marker. Single-flight is enforced both within the process (a `TryLock` mutex shared by the scheduled loop and the manual `TriggerNow`, `quiesce.go:149`/`quiesce.go:181`) and across restarts (the active marker — a cycle refuses to start on top of one, `quiesce.go:157`). `TriggerNow` runs the same flow asynchronously for the manual "Mentés most" action, returning `ErrBackupInProgress` if a cycle is already running. + +**Restic is gone from the controller.** All disk-tier backup (restic snapshots, cross-drive-to-other-disks, drive recovery, infra backup) is the agent's; the controller's only remaining off-drive copy is the Tier-2 rsync mirror in §5. diff --git a/documentation/controller/deploy-and-stack-lifecycle.md b/documentation/controller/deploy-and-stack-lifecycle.md new file mode 100644 index 0000000..a074f7c --- /dev/null +++ b/documentation/controller/deploy-and-stack-lifecycle.md @@ -0,0 +1,256 @@ +# Controller: deploy & stack lifecycle + +> Source of truth: felhom-controller `internal/stacks/`, `internal/infra`, `internal/sync` at v0.59.0. + +This document describes how the in-guest controller represents application stacks, deploys +them, runs their lifecycle (update/stop/start/restart/remove/delete), and brings up the base +infrastructure (traefik / cloudflared / filebrowser) that everything else depends on. Every +claim below is verified against the current source. + +## 1. The stack model + +A *stack* is one customer app, represented on disk as a directory under +`/opt/docker/stacks//` (config key `paths.stacks_dir`, default `/opt/docker/stacks`) +containing: + +- `docker-compose.yml` — the compose template (synced from the catalog, see §7). +- `.felhom.yml` — app metadata (`Metadata`): display name, deploy fields, resource + requests/limits, optional config, healthcheck spec, integrations. +- `app.yaml` — the per-app deployment record (`AppConfig`), written **only after** the app + is first deployed. + +In memory each stack is a `Stack` struct (`internal/stacks/manager.go:62`): + +| Field | Meaning | +|---|---| +| `State` (`ContainerState`) | aggregated container state | +| `Deployed` | has an `app.yaml` with `deployed: true` | +| `Deploying` | a `compose up` is in progress (image pull, etc.) | +| `Protected` | a system stack that must never be stopped/removed (§4) | +| `Orphaned` | deployed but no longer present in the synced catalog | +| `DeployError` | last async deploy error, surfaced to the UI | +| `HealthProbe` | latest controller-side probe result (§6) | + +`ContainerState` values (`manager.go:23`): `running`, `starting` (`health: starting`), +`unhealthy`, `stopped`, `restarting`, `exited`, `paused`, `unknown`, `not_deployed`, +`deploying`, `orphaned`. The effective state is derived by `resolveContainerState` +(combines Docker `State` + the health hint in `Status`) and then `aggregateState` across +all of a stack's containers, with priority `unhealthy > starting > restarting > all-running +> stopped` (`manager.go:437`, `:468`). + +`ScanStacks` (`manager.go:250`) discovers stacks by walking `stacks_dir`, reading metadata +and `app.yaml`, marking `Protected` via `cfg.IsProtectedStack`, and computing `Orphaned` +against the synced catalog cache (`getCatalogTemplateSlugs`, `manager.go:1078`). While an +async deploy is in flight it deliberately does **not** overwrite `Deployed`/`AppConfig` — +the deploy goroutine owns those (H3 fix, `manager.go:293`). + +### `AppConfig` (`app.yaml`) + +```go +type AppConfig struct { + Deployed bool `yaml:"deployed"` + DeployedAt string `yaml:"deployed_at"` + Env map[string]string `yaml:"env"` + LockedFields []string `yaml:"locked_fields"` +} +``` +(`deploy.go:98`). `Env` holds the resolved deploy-field values; sensitive ones are stored +**encrypted** (§3). `LockedFields` lists env vars marked `locked_after_deploy` that +`UpdateStackConfig` refuses to change (`deploy.go:434`). + +## 2. Deploy flow + +Entry point: `DeployStack(req)` (`deploy.go:121`), driven by the deploy form ("Telepítés"). + +1. **Atomic check-and-set of `Deploying` (H1).** Under the manager lock, the stack must + exist, must not already be `Deploying`, and must not already be `Deployed`; then + `Deploying` is set true (`deploy.go:122`). This single critical section prevents two + concurrent deploys of the same app. Any later validation failure clears the flag via + `clearDeploying()`. + +2. **Memory gate.** `system.GetMemoryMB()` vs the app's `mem_request` and the configured + `reserved_memory_mb`. A hard block (real used + new request > usable) aborts with a + Hungarian error; a soft over-commit case returns a non-fatal warning string + (`deploy.go:159`). + +3. **Field resolution.** For each `DeployField`: `domain` is auto-filled from + `customer.domain`; `subdomain` is validated (DNS-safe, not reserved, not already in + use); `secret` uses the value previewed to the user or generates one; `password` is + **required** (never silently generated — the user must know it); `path` must exist on + the host. Required-but-empty fails (`deploy.go:206`–`292`). + +4. **`app.yaml` is persisted with `deployed: false` first (CTRL-T2-1).** The in-memory + `AppConfig` carries `Deployed: true` (for UX), but a clone with `Deployed: false` is what + `SaveAppConfig` writes to disk at this point (`deploy.go:302`–`314`). The durable record + is intentionally *not* marked deployed yet. + +5. **In-memory state flips to deployed** (`s.Deployed = true`, `s.AppConfig = appCfg`), + so the UI immediately stops showing a stale "Telepítés" button during the pull + (`deploy.go:331`). + +6. **Async compose.** `go m.runComposeDeploy(...)` runs `docker compose up -d` off the + request path, so the HTTP handler returns at once and the UI polls for progress (an + image pull can take 30–60s). The compose command injects the resolved env plus `DOMAIN` + (`deploy.go:338`, `composeExecWithEnv` `manager.go:505`). + +### CTRL-T2-1 crash-safety (the key current behaviour) + +The on-disk `app.yaml` records `deployed: true` **only after** `docker compose up -d` +actually succeeds. `runComposeDeploy` (`deploy.go:346`): + +- On compose **failure**: reverts in-memory state (`Deployed=false`, `Deploying=false`, + records `DeployError`, `AppConfig=nil`) and re-saves the reverted `AppConfig` + (still `deployed:false`) to disk (`deploy.go:350`). +- On compose **success**: re-runs `SaveAppConfig` with `appCfg.Deployed == true`, flipping + the durable record to deployed (`deploy.go:371`). If *that* save fails, it reverts in + memory so the customer can cleanly redeploy rather than be stuck half-recorded + (`deploy.go:375`). +- Only then is `Deploying` cleared and `RefreshStatus()` run. + +Why it matters: a crash or power loss **during the image-pull window** leaves `app.yaml` +at `deployed: false`. On restart, `ScanStacks` therefore sees the app as *not deployed* +(no ghost-deployed stack with no containers), and `DeployStack` will cleanly redeploy +instead of refusing with "already deployed". + +The UI's three-step progress (form submit → "pulling/deploying" via the `deploying` +state and `DeployError` → settled `running`/`unhealthy`) is driven entirely by this +in-memory state, which the front-end polls. + +## 3. `SaveAppConfig` and secret encryption (fail-closed, H10) + +`SaveAppConfig(stackDir, cfg, encKey, sensitiveVars)` (`deploy.go:669`) clones the env and, +for each var named in `sensitiveVars` (the `secret`/`password` deploy fields, computed by +`SensitiveEnvVars`, `deploy.go:741`), encrypts the value with the AES-256 key +(`crypto.Encrypt`) unless it is empty or already encrypted. + +**Fail-closed (H10, v0.59.0):** if encryption of a sensitive value errors, `SaveAppConfig` +**returns the error and writes nothing** — it never falls through to a plaintext write +(`deploy.go:683`–`691`). Earlier code logged a WARN and persisted plaintext, which leaked +the secret to disk; that path is gone. Callers (`DeployStack`, `UpdateStackConfig`, etc.) +already propagate this error, so the deploy fails cleanly with no secret on disk. + +The write itself is atomic: write to `app.yaml.tmp` then `rename` (H04, `deploy.go:711`), +mode `0600`. Decryption for compose happens lazily via `LoadAppConfigDecrypted` / +`stackEnv` (`deploy.go:727`, `manager.go:809`), which injects `DOMAIN` plus the decrypted +env into the compose process environment. A one-time startup `MigrateEncryption` +(`manager.go:128`) re-saves any deployed app still holding plaintext sensitive values. + +## 4. Lifecycle: update / stop / start / restart / remove / delete + +All operations resolve the stack dir, build env via `stackEnv` (decrypted), and shell out +to compose. Notably **start/restart use `up -d`, not bare `restart`**, so env changes and +template changes (new images, healthchecks) are picked up (`manager.go:709`, `:643`). + +- **Start** (`StartStack`, `manager.go:643`): `compose up -d`; clears the stale health + probe so the next probe runs fresh. +- **Stop** (`StopStack`, `manager.go:682`): refuses protected stacks, then `compose down`. +- **Restart** (`RestartStack`, `manager.go:709`): `compose up -d`; clears health probe. +- **Update** (`UpdateStack`, `manager.go:746`): `compose pull` then + `compose up -d --remove-orphans`. +- **Config update** (`UpdateStackConfig`, `deploy.go:405`): rejects locked fields, re-saves + `app.yaml` (encrypting secrets), then `up -d` with the decrypted env. +- **Remove** (`RemoveStack`, `delete.go:283`): for a *deployed* (non-orphaned) stack — + `compose down --volumes` (keeps images for redeploy), optional HDD-data and backup-path + cleanup, then deletes **only** `app.yaml`, reverting the stack to "not deployed". Compose + template files are preserved so the user can redeploy. +- **Delete** (`DeleteStack`, `delete.go:80`): for an *orphaned* stack only — + `compose down --rmi local --volumes`, optional HDD-data removal, then removes the whole + stack directory. + +### Protected-stack enforcement (server-side, defence in depth) + +The protected set is configured under `stacks.protected` (default +`traefik`, `cloudflared`, `felhom-controller`, `filebrowser` — see +`configs/controller.yaml.example:55` and the setup writer `internal/setup/handlers.go:476`). +`config.IsProtectedStack` matches case-insensitively (`config.go:347`). Enforcement exists +at three layers: + +- **Router/API:** `actionStack` blocks any action other than `restart` on a protected + stack with HTTP 403 (`internal/api/router.go:412`). +- **Manager:** `StopStack` (`manager.go:683`), `RemoveStack` (`delete.go:289`) and + `DeleteStack` (`delete.go:86`) each independently refuse protected stacks, so the guard + holds even if a caller bypasses the router. +- The quiesce loop's `RunningAppStacks` (`manager.go:232`) also excludes protected stacks, + so the controller never stops its own tunnel/proxy or itself for a backup. + +### Deploying-guard on remove/delete (H2) + +Both `RemoveStack` (`delete.go:309`) and `DeleteStack` (`delete.go:106`) refuse while the +stack is `Deploying`, and both also require the stack to be stopped (not +running/starting/restarting). HDD-path deletion is gated by `ProtectedHDDPaths` +(`delete.go:58`), which refuses to wipe top-level namespace directories (`appdata/`, +`backups/`, `media/`, etc.), and `ParseComposeHDDMounts` cleans paths before the prefix +check to block traversal (C10, `delete.go:555`). + +## 5. Base-infra bring-up: `EnsureBaseStack` + +`EnsureBaseStack` (`internal/stacks/infra.go:27`) renders and deploys the routing/access +infrastructure. Properties (verified): + +- **Single-flight:** guarded by `infraMu.TryLock()` (`infra.go:28`). It is fired both at + first boot and on every `system-health` tick (self-heal); a second concurrent invocation + returns immediately rather than racing a `compose up` on the same dir. +- **Idempotent:** each component is skipped when its container is already running + (`containerRunning`, `infra.go:256`), so the healthy-state re-run is a cheap trio of + `docker inspect` calls. +- **Non-fatal by contract:** per-component failures are collected into a joined error for + the caller to *log* — it must never crash the controller (`infra.go:67`). + +Deploy order is load-bearing (the composes declare `traefik-public` as +`external: true`, so it must exist first): + +1. `ensureTraefikNetwork` — creates the external `traefik-public` docker network if absent, + tolerating a create/inspect race (`infra.go:211`). A failure here is fatal to the run + (every stack `up` would fail without it). +2. `ensureTraefik` — prepares `dynamic/`, `certs/`, a `0600` `acme.json`, renders the + traefik config from `customer.email` + `infrastructure.cf_api_token`, writes it, and + `compose up -d` (`infra.go:73`). +3. `wireController` — writes the file-provider route `Host(felhom.) → + http://felhom-controller:8080` and joins the `felhom-controller` container to + `traefik-public`. Both steps are idempotent (route rewritten only when content changed, + so the traefik file watcher doesn't reload every tick; network-connect skipped when + already attached). A missing domain is a logged no-op, not an error (`infra.go:160`). +4. `ensureCloudflared` — **only when `infrastructure.cf_tunnel_token` is set**; a LAN-only + node legitimately runs without it, and `monitor.EffectiveProtected` mirrors this + condition so such a node doesn't report cloudflared as a perpetually-missing protected + container (`infra.go:55`, `monitor/healthcheck.go:252`). +5. `ensureFileBrowser` — **preserves an existing compose.** If + `filebrowser/docker-compose.yml` already exists, it `compose up -d` *without + regenerating*, so the storage mounts that `web.SyncFileBrowserMounts` manages are kept + intact. Only on first provision does it render the initial compose + `config.yaml` with + no mounts (`infra.go:122`). + +## 6. Health probes + +`RunHealthProbes` (`internal/stacks/healthprobe.go:23`) is a scheduler-driven, controller-side +probe (independent of Docker's own healthcheck). It only probes stacks in `running`/`unhealthy` +state that declare a `healthcheck` in metadata and whose interval has elapsed. The interval is +adaptive: a fast **10s** retry while unhealthy, otherwise the configured interval (default 5m); +when `HealthProbe` is nil (just started) it probes immediately (`healthprobe.go:41`). + +Targets are collected under a read-lock, probed **concurrently** with the lock released +(`healthprobe.go:91`), and results applied under the write-lock. Check types: `tcp` +(dial), `http` (any response = healthy), and `api` (validate expected status / body +substring) — `runSingleCheck`, `probeTCP`, `probeHTTP` (`healthprobe.go:169`+). A failed +probe overrides a Docker-`running` stack to `unhealthy`; a passing probe clears that +override back to `running`. `refreshStatusLocked` re-applies the last probe result so a +status refresh never resurrects a stale healthy state (`manager.go:418`). + +## 7. How stacks get their templates (catalog git-sync) + +The compose templates and metadata come from the app catalog via `internal/sync` +(`Syncer`, `sync.go:22`). On a configurable interval (`git.sync_interval`, default 15m, +with an initial sync at startup) it shallow-clones/pulls `git.repo_url` (branch +`git.branch`) into `/catalog-cache` via `git fetch --depth 1` + +`reset --hard origin/` (`sync.go:247`). It then copies **only** `docker-compose.yml` +and `.felhom.yml` from each `templates//` into `stacks_dir//`, hashing content +to skip unchanged files and **never** overwriting `app.yaml` (`copyTemplates`, +`sync.go:311`; `copyIfChanged`, `sync.go:395`). + +After a sync that changed anything, it triggers a rescan (`ScanStacks`) and, for *updated* +stacks, a post-sync hook that runs `InjectMissingFields` — auto-generating values for any +new `secret`/`domain`/`subdomain` deploy fields that aren't yet in a deployed app's +`app.yaml` (`sync.go:207`, `deploy.go:801`). `TriggerSync` (manual) debounces to one run +per 30s. Sync is disabled (manual mode) when `git.repo_url` is empty (`sync.go:79`). The +catalog cache also feeds orphan detection (`getCatalogTemplateSlugs`): a deployed stack +whose slug is no longer in the cache is flagged `Orphaned`. diff --git a/documentation/controller/module-map.md b/documentation/controller/module-map.md new file mode 100644 index 0000000..a276d40 --- /dev/null +++ b/documentation/controller/module-map.md @@ -0,0 +1,65 @@ +# Felhom Controller — Module Map (current) + +**Source of truth:** `felhom-controller/controller/internal/` at v0.59.0 (2026-06-13). +**Supersedes** the planning map `../architecture/02-controller-module-map.md` (a v0.33 KEEP/PORT/DELETE +plan written before slice 8C). That plan is now **executed**: the disk/storage/restic/watchdog +subsystems were deleted or moved to the host agent; the controller is Docker-only and holds no Proxmox +credentials. The old doc is kept for history; this is the live map. + +## What the controller is (and is not) + +The in-guest controller is **one per customer LXC, Docker-only**. It owns the **app domain**: stack +lifecycle, the Hungarian web UI, app-data backup (DB dumps + volume tars + recovery units + Tier-2 +off-drive copies), metrics/telemetry, hub reporting, catalog sync, integrations, geo-restriction, and +self-update. It does **not** touch Proxmox or raw disks: whole-guest backup (PBS vzdump), disk +classification, and destructive storage ops live in the **host agent** (`felhom-agent`), reached over a +TLS-leaf-pinned local API (`internal/agentapi`). + +## Package map + +| Package | Role | Notes / doc | +|---|---|---| +| `cmd/controller` | entry point, wiring, schedulers | `Version` via ldflags | +| `internal/stacks` | stack model + deploy/start/stop/update/remove lifecycle, protected-stack enforcement | [deploy-and-stack-lifecycle](deploy-and-stack-lifecycle.md) | +| `internal/infra` | base-infra bring-up (`EnsureBaseStack`: traefik/cloudflared/filebrowser), self-heal | [deploy-and-stack-lifecycle](deploy-and-stack-lifecycle.md) | +| `internal/sync` | catalog git-sync (compose/.felhom.yml; never overwrites app.yaml) | [auth-hub-sync-integrations](auth-hub-sync-integrations.md) | +| `internal/appbackup` | app-data backup primitives: DB dumps, recovery units, restore | [backup-architecture](backup-architecture.md) | +| `internal/backup` | backup orchestration: scheduling, Tier-2 cross-drive, running mutex | [backup-architecture](backup-architecture.md) | +| `internal/appexport` | `.fab` encrypted app export/import (AES-CTR+HMAC); path-segment validation (CTRL-001) | [backup-architecture](backup-architecture.md) §export | +| `internal/quiesce` | app-consistent quiesce loop (stop→backup→restart, crash-safe) | [backup-architecture](backup-architecture.md) | +| `internal/agentapi` | TLS-leaf-pinned client to the host agent (disks, host-metrics, whole-guest backup) | [storage-monitoring-metrics](storage-monitoring-metrics.md) | +| `internal/settings` | `settings.json` persistence (password hash, storage registry, caches) | RWMutex; atomic writes | +| `internal/config` | `controller.yaml` loading + defaults; `IsProtectedStack` | — | +| `internal/crypto` | AES-256-GCM at-rest secret encryption | — | +| `internal/web` | dashboard UI + JSON API: auth, CSRF, storage/disk handlers, host-metrics, templates | [auth-hub-sync-integrations](auth-hub-sync-integrations.md), [storage-monitoring-metrics](storage-monitoring-metrics.md) | +| `internal/api` | REST router (`/api/*`) | route→CSRF/auth coverage | +| `internal/setup` | first-run setup wizard (pre-auth) | [auth-hub-sync-integrations](auth-hub-sync-integrations.md) | +| `internal/bootstrap` | first-run `bootstrap.json` ingestion (seed config, skip setup) | [auth-hub-sync-integrations](auth-hub-sync-integrations.md) | +| `internal/report` | hub report builder + pusher (zero-knowledge; no app secrets) | [auth-hub-sync-integrations](auth-hub-sync-integrations.md) | +| `internal/notify` | event notifications + history | [auth-hub-sync-integrations](auth-hub-sync-integrations.md) | +| `internal/integrations` | app-to-app integrations (adapter pattern; OnlyOffice↔FileBrowser/Nextcloud) | [auth-hub-sync-integrations](auth-hub-sync-integrations.md) | +| `internal/cloudflare` | geo-restriction via CF WAF (`[felhom-geo]` rules, geosync) | [auth-hub-sync-integrations](auth-hub-sync-integrations.md) | +| `internal/selfupdate` | self-update (version check; pinned tags, never `:latest`) | [auth-hub-sync-integrations](auth-hub-sync-integrations.md) | +| `internal/assets` | hub-sourced asset sync (cache + baked-in fallback) | [auth-hub-sync-integrations](auth-hub-sync-integrations.md) | +| `internal/metrics` | SQLite (WAL) metrics store, docker-stats collector, log scanner/telemetry | [storage-monitoring-metrics](storage-monitoring-metrics.md) | +| `internal/monitor` | health checks (`healthcheck.go`). **watchdog/pinger deleted in 8C** | [storage-monitoring-metrics](storage-monitoring-metrics.md) | +| `internal/system` | system info; `dockervol.go` = OS/Docker-data-split prevention layer (v0.58) | [storage-monitoring-metrics](storage-monitoring-metrics.md) | +| `internal/scheduler` | cron-like job scheduler (Europe/Budapest); late-registration safe; per-job `recover()` | — | +| `internal/recovery` | recovery-file generation | — | +| `internal/selftest` | startup self-test (hub reachability, metrics DB, etc.) | — | +| `internal/util` | shared helpers | — | + +## Cross-repo boundary + +- **Host agent (`felhom-agent`):** owns Proxmox + all destructive storage. The controller calls it via + `internal/agentapi`; the agent enforces the data-bearing gate + operator signatures. See + `../architecture/03-host-agent.md` and `../audits/deep-sweep-2026-06-13.md` (agent destructive-path). +- **Hub (`felhom.eu/hub/`):** operator backend; the controller pushes zero-knowledge reports/events to + it. See `../architecture/05-hub-architecture.md`. + +## De-privileging — what is GONE from the controller (executed in 8C) + +`internal/storage/*` (scan/format/migrate), restic, cross-drive-to-other-disks execution, +`monitor/watchdog.go` + `pinger.go`, drive-restore, infra-backup, the storage UI's raw-disk operations. +The controller now **delegates** all of these to the agent. Do not document the controller as performing +them. diff --git a/documentation/controller/storage-monitoring-metrics.md b/documentation/controller/storage-monitoring-metrics.md new file mode 100644 index 0000000..1bc0ac8 --- /dev/null +++ b/documentation/controller/storage-monitoring-metrics.md @@ -0,0 +1,256 @@ +# Controller — storage, monitoring and metrics + +Source of truth: felhom-controller `internal/agentapi/`, `internal/web` storage handlers, +`internal/metrics/`, `internal/monitor/`, `internal/system/dockervol.go` at v0.59.0; disk +classification/execution is the host agent's. + +This document describes what the **in-guest controller** actually does with storage, host +health and metrics. The controller is **de-privileged** (slice 8C): it is Docker-only, holds +no Proxmox/disk credentials, and does not scan, format, mount or classify disks itself. Disk +topology and classification come from the host **agent**; the controller calls the agent over a +pinned local API, displays the agent's view, and keeps a small registry of the user-data drives +it cares about. + +--- + +## 1. Storage model in the controller + +There are two distinct storage views in the controller, and they must not be conflated: + +### 1a. The registered user-data drive registry (`settings.StoragePath`) + +The controller persists a list of user-data drives it manages, in `settings.json` as +`StoragePaths` (`internal/settings/settings.go:97`, the `StoragePath` struct). Each entry is a +mount the customer's apps deploy their large files onto: + +| Field | Meaning | +|---|---| +| `Path` | the in-guest mount, e.g. `/mnt/felhom-usb` | +| `Label` | friendly Hungarian name (`Külső HDD 1TB`) | +| `IsDefault` | new apps default here | +| `Schedulable` | new apps may be deployed here | +| `Disconnected` / `Decommissioned` | lifecycle flags | +| `MigratedTo` | target path after a decommission | + +`GetStoragePaths()` (`settings.go:439`) returns a defensive copy under `RWMutex`. This registry +is **only the external/user-data drives** — the agent's system disk (local-lvm), `local` +templates dir and PBS targets are **never** in it (they live host-side and are surfaced via the +agent's host-metrics view, §4). All public settings methods take the mutex; writes are atomic +(write `.tmp`, rename). + +### 1b. The agent's authoritative disk topology + +A "storage target" / "disk" in the operational sense is whatever the **agent** reports. The +controller learns topology by *asking the agent*, never by inspecting devices: + +- `Client.Disks()` → agent `GET /disks` → `[]DiskInfo` (`internal/agentapi/client.go:227`, + `:306`). Each `DiskInfo` carries the agent's **authoritative** `Role` + (`system | backup | user-data`), `DataBearing`, `DataReason`, capacity, and a `DurableID` + (`uuid:` for usb/local-dir). `DiskInfo.FSUUID()` (`client.go:252`) strips the + `uuid:` prefix — this is the only way the de-privileged controller learns a mount key it + cannot read off the device itself. +- `Client.HostMetrics()` → agent `GET /host/metrics` → host health + `[]StorageTarget` + (`client.go:459`, `:492`). + +Classification is the agent's; the controller **displays** it. The UI is driven from +`DiskInfo.Role`: `system`/`backup` get a lock badge and no destructive controls; `user-data` is +customer-manageable (`client.go:235-237`). + +--- + +## 2. Disk operations are delegated to the agent + +All disk execution (list/assign/eject/format) goes through `agentapi.Client` to the agent's +`/disks` endpoints. The controller's web handlers are **thin proxies** +(`internal/web/agent_disk_handlers.go`), wired behind `RequireAuth + CsrfProtect`: + +| Route | Proxies to | Notes | +|---|---|---| +| `GET /api/disks` | agent `GET /disks` | list; sorted Go-side for stable order | +| `POST /api/disks/assign` | agent `POST /disks/assign` | benign mount of an existing fs | +| `POST /api/disks/eject` | agent `POST /disks/eject` | safe-unmount (data preserved) | +| `POST /api/disks/format` | agent `POST /disks/format` | data-bearing gated agent-side | + +`agentClient()` (`agent_disk_handlers.go:43`) builds a pinned client from `cfg.LocalAPI` +(endpoint/token/fingerprint); it returns "agent not configured" on an unprovisioned guest. + +**The data-bearing gate is enforced on the agent, not the controller.** `FormatDisk()` +(`client.go:372`) sends the caller's `device/fstype/confirmed/durable_id`; the agent inspects +the device itself, tiers it by role (its own classification — the controller's claim is +ignored), and: + +- blank device → formatted; +- **user-data**, data-bearing, not confirmed → `ErrNeedsConfirmation` with the durable id to + type-to-confirm against (a *customer* confirmation, not an operator signature); +- **system/backup**, data-bearing → `ErrFormatRefused` with a `PendingOp` carrying the exact + offline `felhom-opsign` command (`PendingOp.OpsignCommand()`, `client.go:292`). + +The controller holds **no destructive authority** — there is no force-format path. The format +handler surfaces both refusals as HTTP 409 (`agent_disk_handlers.go:192-200`); the deep gate +mechanics live on the agent side (cross-ref the agent's destructive-path / AGENT-001 docs). + +### Eject and the guided init/wipe flows + +`internal/web/storage_handlers.go` orchestrates the guided init/attach/wipe over the same agent +endpoints plus the local registry: + +- **Init** (`runStorageInit`, `storage_handlers.go:83`): format → (confirm/refuse?) → resolve + the *new* fs UUID by re-listing disks → benign `assign` → register in the StoragePath + registry → `guest-attach` into this guest. On any refusal it performs no further destructive + or mount action. +- **Wipe** (`handleStorageWipe`, `:357`): customer-confirmed wipe of a **user-data** drive only; + server-side type-to-confirm (the typed name must equal the mount basename). A + system/backup-protected device is refused by the agent even though the controller sends + `confirmed:true` (`:402-406`). +- **Eject** (`handleStorageEject`, `:506`): benign unmount via the agent (data preserved) + + deregister the StoragePath + resync FileBrowser mounts. The eject **role-gate** is implicit: + eject is non-destructive, but the destructive wipe behind it is the agent's role-tiered gate. + The agent's `EjectResult` returns `DependentGuests` so the UI can warn about other guests + bound to the drive (`client.go:343`). +- Restricted to `/mnt/` (validated by `mountNameRe`, `storage_handlers.go:39`); only + `ext4`/`xfs` offered (the agent re-validates). + +--- + +## 3. The v0.58 infra-protection prevention layer (`internal/system/dockervol.go`) + +After the OS/Docker-data split, `/var/lib/docker` is a dedicated volume holding **all** images, +overlay and named volumes — both infra (controller/traefik/cloudflared/filebrowser) and customer +apps. Infra is protected by **prevention, not placement**: a reserved buffer the controller +refuses to deploy into. + +- `DockerVolumePath = "/"` (`dockervol.go:16`). The controller container's root is an overlay + whose upperdir lives on the guest's `/var/lib/docker` volume, so `statfs("/")` reports **that + volume's** capacity/free (true with the golden's overlay2 driver; pre-split it was the rootfs — + correct either way). +- `DockerVolumeReserveGB(totalGB)` (`dockervol.go:23`) = `max(5 GB, 10% of total)`. (10% rather + than the Tier-2 guard's 20%: 20% of a large data volume would reserve an absurd amount.) +- `GetDockerVolumeHeadroom()` (`dockervol.go:43`) measures the volume and returns + `DockerVolumeHeadroom{TotalGB, AvailGB, ReserveGB, BelowReserve, OK}`. `BelowReserve` is + `AvailGB <= ReserveGB` (`:53`). **`OK=false` when stats are unreadable** — callers MUST + fail-open (a transient measurement error must not block all deploys; the buffer is a safety + net, not a security control). + +**Deploy gate** (`internal/api/router.go:353`): `deployStack` refuses a new deploy with +**HTTP 507** + a Hungarian message when `hr.OK && hr.BelowReserve`. The fail-open is explicit — +the gate only engages when `hr.OK` is true. + +**UI pre-warning** (`internal/web/handlers.go:339`): for a NEW deploy only (an existing app's +config save consumes no fresh image space), the deploy page sets `DockerBelowReserve` + +human-readable free/reserve, and `deploy.html` shows a warning banner and disables the +"Telepítés indítása" button when below reserve. + +--- + +## 4. Host metrics (agent-sourced host health view) + +The de-privileged controller sees only its own cgroup, so it cannot read host health itself. +`ServeHostMetricsAPI` (`internal/web/agent_host_metrics_handler.go:20`, behind `RequireAuth`, +read-only) proxies `GET /api/host-metrics` → agent `GET /host/metrics` and returns the host-wide +view: cpu%/mem/load/uptime/cpu-temp (`HostMetrics`, `client.go:431`) plus per-storage capacity + +SMART/thin-pool health (`StorageTarget`, `client.go:459`). + +**v0.57 server-side enrichment** (`enrichHostStorageTargets`, `agent_host_metrics_handler.go:51`): +the agent enumerates storages via `pvesm` in non-deterministic order, so the +`#host-storage-bars` list reordered on every poll. The handler now: + +1. Sorts `StorageTargets` stably by `storageTypeRank` (`:67`): user-data drives (usb/local-dir) + → internal SSD (lvmthin/lvm) → `local` templates+backups → backup targets (pbs/nfs/cifs) → + other; alphabetical by id within a tier. +2. Attaches a friendly Hungarian `Label` + one-line `Purpose` per entry + (`storageLabelAndPurpose`, `:84`). These are **display-only** controller-side fields + (`StorageTarget.Label/Purpose`, `client.go:476-480`) — the raw PVE storage id stays in `Name` + and is never renamed. + +The disk-overview list (`GET /api/disks`) is similarly sorted Go-side by role +(`sortDisksForView`, `agent_disk_handlers.go:87`) for a stable, user-data-first order. + +**`buildStorageBars`** (`internal/web/handlers.go:50`) is the separate monitoring-page +"Tárolók kapacitása" list. It iterates the **registered user-data StoragePaths** (not the agent +view), skips decommissioned drives, reads local `system.GetDiskUsage` per path, and sorts by +`Path`. Every bar carries the same `storageBarPurpose` text (`:46`) because this list is +all-user-data by construction — the agent's system/PBS storage is not in this registry (it is on +the storage-management page via the agent host-metrics view). + +--- + +## 5. Metrics subsystem (`internal/metrics/`) + +### SQLite store (`store.go`) + +`NewMetricsStore(dbPath)` opens `modernc.org/sqlite` at +`/opt/docker/felhom-controller/data/metrics.db` and **verifies WAL mode took effect** — it +errors out if `PRAGMA journal_mode=WAL` does not return `"wal"` (`store.go:27-34`), then sets +`synchronous=NORMAL` and `busy_timeout=5000`. Two tables: `system_metrics` and +`container_metrics`, with ts indices (`:48-77`). Queries downsample into time buckets +(`QuerySystemMetrics`/`QueryContainerMetrics`, default resolution 200 points). `Prune` deletes +rows older than a cutoff. + +### Collector (`collector.go`) + +`MetricsCollector.Start(ctx)` runs a **60-second** loop guarded by `sync.Once` +(`collector.go:38,53`), wired in `cmd/controller/main.go:203` and stopped on shutdown. Each tick: + +- **System sample**: `system.GetInfo(hddPath, cpuCollector)` → cpu%, mem, temp, loadavg, SSD + + HDD usage (`sampleSystem`, `:81`). +- **Container sample** (`sampleContainers`, `:99`): runs `docker stats --no-stream` under a + **cancellable 30s timeout context** derived from the loop ctx (`:100`), parses the + tab-separated cpu/mem/net/block columns, and batch-inserts. + +### Telemetry + log scanner + +- `GetContainerTelemetry(since)` (`telemetry.go:20`) aggregates per-container avg/peak memory + + avg cpu over a window from the DB, then patches in the most-recent memory per container. +- `ScanContainerLogs(names, since, logger)` (`logscanner.go:52`) scans each container's + `docker logs --since= --tail=1000` **sequentially** (to avoid load spikes), under a + per-container 10s timeout (`:87`). It classifies lines on the first 5 words against + error/warn keyword sets (`error|fatal|panic|crit|oom|killed|exception|traceback`; + `warn|warning`), deduplicates by a normalized fingerprint (ANSI/timestamp stripped; UUIDs, + long hex and 6+ digit runs collapsed), and caps each container at its 10 most-frequent issues. + +These two feed the **hub report**, not a live UI: `internal/report/telemetry.go:18` +(`buildAppTelemetrySection`) collects 15-minute telemetry + a log scan for every non-protected, +deployed, running stack **plus the controller container itself** (`controllerContainerName = +"felhom-controller"`), merges per-app issues (capped at 10), and emits `[]AppTelemetry` pushed to +the hub. + +--- + +## 6. Monitoring (`internal/monitor/healthcheck.go`) + +`RunHealthCheck(cfg, cpuCollector, storagePaths, logger)` (`healthcheck.go:26`) produces a +`HealthReport{Status: ok|warn|fail, Issues, Warnings, Info}`. It is invoked at startup, on a +schedule, and from the debug endpoint (`cmd/controller/main.go:273,558`; +`internal/web/handler_debug.go:255`). Checks: + +1. **SSD disk usage** — `sysInfo.DiskPercent` is `statfs("/")`, i.e. the same Docker-data volume + the prevention layer guards; warn at `DiskWarnPercent` / crit at `DiskCritPercent`. These trip + **above** the 10%-free reserved buffer, so the customer is warned before the deploy gate + engages (`:47-76`). +2. **HDD usage** (when `HDDConfigured`), **memory**, **CPU**, **temperature** — each against its + configured threshold (`:79-145`). +3. **Docker** reachable (`docker info`, `:148`). +4. **Protected containers** running — using `EffectiveProtected(cfg)` (`:252`), which drops + `cloudflared` when no tunnel token is configured so a LAN-only node is not perpetually FAIL. +5. **Storage paths** (`checkStoragePaths`, `:279`): per registered StoragePath, warn on + disconnected/inaccessible, warn (not fail) when not on a separate mount point, warn ≥90% / + issue ≥95% usage. All messages Hungarian. + +Status rolls up to `fail` if any issue, else `warn` if any warning. + +### Removed: watchdog and pinger (slice 8C) + +The old storage **watchdog** (disk disconnect/reconnect detection) and the **pinger** were +deleted when the controller was de-privileged — disk detection moved to the host agent +(`cmd/controller/main.go:435` notes the watchdog moved host-side; `monitor/` now contains only +`healthcheck.go` + tests). Healthchecks pinging is likewise retired: ping UUIDs in config are +logged as no-ops and the hub now owns monitoring (`cmd/controller/main.go:209-214`). + +--- + +## Cross-references + +- Agent disk classification, the data-bearing gate and the signed destructive path: the host + agent's storage / AGENT-001 docs. +- Backup capture/restore and Tier-2 off-drive copies: `controller/backup-architecture.md`. +- Deploy flow and the headroom gate in context: `controller/deploy-and-stack-lifecycle.md`.