400 lines
30 KiB
Markdown
400 lines
30 KiB
Markdown
# felhom-hub
|
||
|
||
**Central operator dashboard for monitoring and managing Felhom customer deployments.**
|
||
|
||
A lightweight Go service that receives periodic reports and structured events from felhom-controller instances, stores them in SQLite, and provides a web dashboard for fleet monitoring. Also serves as the infrastructure backup store for disaster recovery, event-based dead man's switch monitoring, and notification dispatch.
|
||
|
||
**Current version: v0.6.3**
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
```
|
||
Customer nodes Central Hub (k3s)
|
||
┌─────────────────┐ ┌────────────────────────┐
|
||
│ felhom-controller│──── JSON push ────▶│ felhom-hub │
|
||
│ (every 15 min) │ (Bearer auth) │ │
|
||
│ │ │ ┌─────────────────┐ │
|
||
│ POST /api/v1/ │ │ │ API Handler │ │
|
||
│ report │ │ │ (ingest reports, │ │
|
||
│ host-report │◀── config push ────│ │ host reports, │ │
|
||
│ notify │ (YAML body) │ │ config push, │ │
|
||
│ │ │ │ asset serving) │ │
|
||
│ GET /api/v1/ │ │ └────────┬────────┘ │
|
||
│ assets/* │◀── asset download ─│ │ │
|
||
└─────────────────┘ (Bearer auth) │ ┌────────▼────────┐ │
|
||
│ │ SQLite Store │ │
|
||
Operator browser │ │ (reports, │ │
|
||
┌─────────────────┐ │ │ assets, │ │
|
||
│ Web Dashboard │◀── HTML pages ──────│ │ host_reports, │ │
|
||
│ (hub.felhom.eu) │ (bcrypt auth) │ │ configs, │ │
|
||
└─────────────────┘ │ │ notifications) │ │
|
||
│ └─────────────────┘ │
|
||
│ │
|
||
│ ┌─────────────────┐ │
|
||
│ │ Asset Manager │ │
|
||
│ │ (PVC storage, │ │
|
||
│ │ SHA-256 manifest│ │
|
||
│ │ file serving) │ │
|
||
│ └─────────────────┘ │
|
||
│ │
|
||
│ ┌─────────────────┐ │
|
||
│ │ Web Dashboard │ │
|
||
│ │ (unified customer│ │
|
||
│ │ management) │ │
|
||
│ └─────────────────┘ │
|
||
└────────────────────────┘
|
||
```
|
||
|
||
## API Endpoints
|
||
|
||
All API endpoints require `Authorization: Bearer <api_key>` (except `/healthz` and `/api/v1/config/{id}`). Auth accepts both the global `report_api_key` and per-customer API keys (generated when creating customer configs).
|
||
|
||
### Report Ingest
|
||
|
||
| Method | Path | Description |
|
||
|--------|------|-------------|
|
||
| `POST` | `/api/v1/report` | Controller pushes periodic status report (v0.28.0+ includes `app_telemetry` field) |
|
||
| `GET` | `/api/v1/customers` | List all customers with latest report summary |
|
||
| `GET` | `/api/v1/customers/{id}` | Get latest full report for a customer |
|
||
| `GET` | `/api/v1/customers/{id}/history?period=7d` | Get report history |
|
||
|
||
The `POST /api/v1/report` handler (v0.4.0+) automatically parses the optional `app_telemetry` JSON array from the request body and stores it in `app_telemetry` / `app_log_issues` tables. Old controllers (no `app_telemetry` key) continue to work unchanged.
|
||
|
||
### DR Recipe — secret-free reconstruction recipe (hub v0.13.0)
|
||
|
||
The hub assembles + stores + serves the **secret-free DR recipe** (`documentation/audits/SPIKE-dr-recipe-2026-06-16.md`) — the non-secret re-provision plan that complements escrow (keys) + PBS/restic (bytes). It arrives as two additive `dr_recipe` halves on the existing report paths: the **agent's** storage/guest/PBS half on `POST /api/v1/host-report` and the **controller's** customer/apps half on `POST /api/v1/report` (both backward-compatible, ignore-unknown). The hub stores them PLAINTEXT in a dedicated `dr_recipe` table keyed by `customer_id` (`SaveDRRecipeHostHalf` / `SaveDRRecipeAppHalf`, each preserving the other half), and `AssembleDRRecipe` stitches them into one operator-readable recipe (`{recipe_version, customer, guests, pbs, drives, pve_storage, apps}`; sub-sections pass through verbatim, version = max of the two halves). An operator views the panel on the customer page and downloads the assembled JSON at `GET /customers/{id}/dr-recipe.json`. **Plaintext-at-rest is correct here** — the recipe carries only identifiers/intents/sizes/coordinates, never a key/password/token (the boundary is enforced at the controller emitter). This is the clean inverse of the retired infra-backup.
|
||
|
||
### Offsite provisioning (SLICE 1, hub v0.37.0)
|
||
|
||
On operator enable (config form → **Offsite backup**), the hub provisions the customer's offsite tier against
|
||
the **Hetzner storage-box API at `https://api.hetzner.com/v1`** (NOT `api.hetzner.cloud` — the classic Cloud
|
||
API 404s for storage boxes; shapes measured in `documentation/audits/SPIKE-hetzner-api-provisioning-2026-07-09.md`).
|
||
Two models: **shared** (a sub-account on the pool box) or **dedicated** (its own box). Flow (`internal/offsite`):
|
||
idempotent lookup by label `felhom-customer=<id>` → create (password generated hub-side) → poll the action to
|
||
`success` → **`SaveOneTimeSecret`** (the transient password, single-use) → merge the **non-secret descriptor**
|
||
(`{enabled,type,host,user,port:23,repo_path,quota_gb|box_type}`) into `ConfigJSON` → `ConfigVersion` bump →
|
||
the controller re-pulls. The controller consumes the one-time password at `POST /offsite/consume-password/{id}`
|
||
(customer-API-key auth, single-use) — then installs its key and the hub resets the box password (SLICE 2+).
|
||
**Fail-closed:** a provisioning error returns 502 and saves nothing. **Secrets:** the Hetzner token
|
||
(`HETZNER_TOKEN`, out-of-band) and every generated password are NEVER logged / committed / in `ConfigJSON`.
|
||
**PREREQUISITE for live use:** the token MUST be scoped to a **dedicated Hetzner project** (the shared
|
||
project token can delete ep0 — spike §6); check the scope by listing boxes with it. Absent token → the UI
|
||
still renders; saving with offsite enabled returns "not configured". **Live-validated end-to-end 2026-07-09**
|
||
(`documentation/audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md`): SLICE 2 (controller apply-bridge)
|
||
is shipped and applied.
|
||
**Hardening (v0.38.1–v0.39.0):** provisioning runs on a client-disconnect-proof detached context (F1); the
|
||
host-key scan retries through fresh-subaccount DNS lag (~60s ladder, F2); the save button disables with an
|
||
in-flight notice (F5); and **"Re-issue offsite credentials"** (F4, `POST /configs/{id}/offsite-reissue`) is
|
||
the explicit recovery for a consumed-password dead-end — resets ONLY the resource labelled for that customer
|
||
(refuses unless exactly 1), stores a fresh one-time secret, bumps `ConfigVersion`. Never implicit rotation.
|
||
**SLICE 3 (v0.40.0) — hub-verified escrow auto-confirm:** the agent's ceremony upload carries
|
||
`restic_pw_sha256` (non-reversible hash of the sealed offsite repo password — safe to store/serve); the hub
|
||
stores it on `host_escrow` (NULL on legacy rows) and serves
|
||
`escrow:{identity_blob_present, restic_pw_sha256, created_at}` in the **report ACK**; the controller
|
||
auto-confirms its pending offbox escrow ONLY on a hash match with its current repo password (blob-presence
|
||
alone never confirms).
|
||
**SLICE 4 (v0.41.0) — offsite health monitoring + freeze lever:** `monitor.OffsiteChecker` reads the
|
||
controller report's `offsite` object — fill alerts at 90/95% of `quota_gb` (quota 0 = dedicated, silent)
|
||
and an `offsite_stale` warning when an enabled+escrowed target has no run in >48h (the silently-stuck
|
||
detector; run FAILURES are `backup_failed`'s job; pending/disabled never alert). Nil-safe on pre-v0.109
|
||
reports. The operator **Freeze/Unfreeze offsite** buttons (shared model only, next to Re-issue) flip ONLY
|
||
`readonly` on the exactly-1 labelled sub-account — MANUAL only, never automatic (freezing also blocks
|
||
prune, the customer's only way down from over-quota).
|
||
|
||
### Infrastructure Backup — RETIRED (Phase-1, 2026-06-16, hub v0.12.0)
|
||
|
||
The Infra Backup mechanism (`POST/GET /api/v1/infra-backup`, the operator panel, the
|
||
`infra_backup_versions` / `infra_backups` tables) has been **removed**. It had been dead since
|
||
slice 8C (the disk-tier backup moved to the host agent), and it stored each version as a **plaintext
|
||
JSON blob** holding the customer's app-secret encryption key, restic password, and Cloudflare tokens —
|
||
a zero-knowledge violation. The retirement migration `DROP`s both tables and `VACUUM`s the DB to
|
||
physically reclaim the plaintext pages. Disaster recovery now rests on the agent's **PBS whole-CT
|
||
snapshot** (the data bytes) plus the generated `controller.yaml` from the recovery endpoint (the
|
||
config); a secret-free **DR recipe** is the later DR slice's job. See
|
||
`documentation/audits/SPIKE-infra-backup-2026-06-15.md`.
|
||
|
||
### Recovery (Disaster Recovery)
|
||
|
||
| Method | Path | Description |
|
||
|--------|------|-------------|
|
||
| `GET` | `/api/v1/recovery/{customer_id}` | Returns the generated controller.yaml for a customer |
|
||
|
||
Auth: `X-Retrieval-Password` header (same per-customer password as config retrieval). Response:
|
||
```json
|
||
{
|
||
"customer_id": "example",
|
||
"config_yaml": "customer:\n id: example\n ...",
|
||
"has_infra_backup": false
|
||
}
|
||
```
|
||
The `has_infra_backup` field is retained as `false` so any old client degrades gracefully to the
|
||
config-only path.
|
||
|
||
### Report Response
|
||
|
||
The `POST /api/v1/report` response now includes `customer_blocked: true` when the customer's status is "blocked". Controllers use this to detect their standing and enter limited mode after a grace period.
|
||
|
||
### Events
|
||
|
||
| Method | Path | Description |
|
||
|--------|------|-------------|
|
||
| `POST` | `/api/v1/event` | Controller pushes structured event (27 allowed types, severity: info/warning/error) |
|
||
|
||
Events are the primary monitoring mechanism. Each event has: customer_id, event_type, severity, message, details_json, source. Per-customer API keys are validated against the customer_id in the payload. Stored in the `events` table with automatic pruning.
|
||
|
||
**Hub-generated events** (source="hub"):
|
||
- `node_stale` / `node_down` / `node_recovered` — dead man's switch from staleness checker (every 60s)
|
||
- `host_stale` / `host_down` / `host_recovered` — host-domain dead man's switch (agent host reports)
|
||
- `agent_capability_degraded` / `agent_capability_recovered` — `HostCapabilityChecker`: an agent's
|
||
required `sudo -n` grant went missing (the non-root cutover class)
|
||
- `host_leaf_changed` — `HostLeafChecker` (`monitor/host_leaf.go`, every 60s): an agent's reported
|
||
local-API **leaf fingerprint** changed (an agent re-key) — proactive, fleet-wide, independent of any
|
||
controller's channel-health check. Trust-on-first-report baseline; empty fp = unknown (never alerts).
|
||
Reads the fp from the latest host-report (`store.GetHostLeafFingerprints`, no schema migration).
|
||
- `expected_backup_missed` / `expected_dbdump_missed` — backup deadline checker (daily at 05:00 Budapest)
|
||
|
||
### Notifications
|
||
|
||
| Method | Path | Description |
|
||
|--------|------|-------------|
|
||
| `POST` | `/api/v1/notify` | Legacy notification relay (kept for backward compatibility) |
|
||
| `POST` | `/api/v1/preferences` | Controller syncs customer notification preferences (email, enabled_events, cooldown_hours) |
|
||
|
||
Notifications are dispatched automatically when events are processed:
|
||
- **Operator channel**: English emails for warning/error events, 1h cooldown per customer:eventType
|
||
- **Customer channel**: Hungarian emails per event type, respects customer preferences and cooldown (default 6h)
|
||
- Email delivery via Resend.com API
|
||
|
||
### Customer Config Retrieval
|
||
|
||
| Method | Path | Description |
|
||
|--------|------|-------------|
|
||
| `GET` | `/api/v1/config/{customer_id}` | Download generated controller.yaml (auth: `X-Retrieval-Password` header) |
|
||
|
||
Config retrieval uses a separate per-customer retrieval password (not the API key). Retrieval passwords are auto-generated as **Hungarian word passphrases** (e.g., `alma-kerék-madár-felhő`) for easy phone-based entry during disaster recovery. The Hub generates a complete `controller.yaml` by deep-merging `controller.yaml.example` (periodically fetched from the Gitea repo) with customer-specific overrides (identity, infrastructure tokens, hub API key, session secret).
|
||
|
||
### Host Enrollment (Day-0, option C)
|
||
|
||
| Method | Path | Description |
|
||
|--------|------|-------------|
|
||
| `POST` | `/api/v1/host-enroll` | Mint-or-reuse a host credential, body `{customer_id}` (auth: `X-Retrieval-Password` header) → `{host_id, api_key}` |
|
||
|
||
The Day-0 first-boot handshake (`documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md`) enrolls a Proxmox host's agent **using the customer's retrieval passphrase only** — the operator-tier global key never enters the field deploy path. `host-enroll` mints the per-host credential on the first call and **reuses it byte-for-byte** on every later call (idempotent — re-running the host-bootstrap never orphans a running agent's key), backed by `Store.GetHostByCustomer`. Auth is checked **before** any mint, so a wrong passphrase never creates a host row. Behaviour: first → `201`, reuse → `200`, wrong/missing passphrase → `401`, unknown customer → `404`, missing `customer_id` → `400`.
|
||
|
||
The global-key `POST /api/v1/admin/hosts` (operator/HQ pre-mint) remains as the escape hatch pending the enrollment-cutover lock-down; it and `GET /config/{id}` are unchanged by this endpoint.
|
||
|
||
### Assets
|
||
|
||
| Method | Path | Description |
|
||
|--------|------|-------------|
|
||
| `GET` | `/api/v1/assets/manifest` | JSON manifest of all assets with SHA-256 checksums |
|
||
| `GET` | `/api/v1/assets/file/{filename}` | Download a single asset file (logo, screenshot) |
|
||
|
||
Assets are stored on the Hub PVC at `<dataDir>/assets/`. On first run, assets are seeded from the Docker image (`/usr/share/felhom/assets-seed/`). The manifest includes filename, size, and SHA-256 hash for each file — controllers use this for efficient change detection.
|
||
|
||
**Asset types served:** `{slug}-logo.svg`, `{slug}-logo.png`, `{slug}-screenshot-{N}.webp`
|
||
|
||
The asset manager (`internal/assets/`) scans the assets directory on startup, builds an in-memory manifest, and serves files with appropriate Content-Type and cache headers. Both endpoints require Bearer token auth (global or per-customer API key).
|
||
|
||
### Health
|
||
|
||
| Method | Path | Description |
|
||
|--------|------|-------------|
|
||
| `GET` | `/healthz` | Health check (no auth required, returns 503 if SQLite ping fails) |
|
||
|
||
## Web Dashboard
|
||
|
||
Protected by bcrypt password + session cookie (7-day expiry).
|
||
|
||
### Authentication & Session Model (`internal/web/server.go`)
|
||
|
||
- Login generates a **cryptographically random 64-char hex session token** stored server-side in a `map[string]*hubSession` (+ `sync.RWMutex`). The old literal `hub_session=authenticated` cookie is gone.
|
||
- Each session also stores a **per-session CSRF token** (separate 64-char hex random value).
|
||
- Cookie attributes: `SameSite=Lax`, `Secure` (when TLS), `HttpOnly`, 7-day `Max-Age`.
|
||
- `RequireAuth` middleware validates the session token with `subtle.ConstantTimeCompare` and redirects to `/login` on failure.
|
||
- `CleanupSessions(ctx)` goroutine runs hourly to purge expired sessions.
|
||
- **Login password source (v0.54.0)** — the bcrypt hash checked at login comes from `effectivePasswordHash()`: a `hub_settings.operator_password_hash` DB override (set via **Configuration → Login password**) wins, otherwise the `auth.password_hash` seed from `hub.yaml`. The ConfigMap value is the break-glass reset path (blank the DB row / edit the manifest + redeploy). Changing the password (`POST /configuration/password`) requires the current password and leaves existing sessions valid.
|
||
|
||
### CSRF Protection (`internal/web/server.go`)
|
||
|
||
Synchronizer-token CSRF protection on all browser POST/DELETE/PATCH operations:
|
||
|
||
- CSRF validation block runs at the top of `ServeHTTP` before routing.
|
||
- Skipped when: no session cookie present (API/Basic-Auth path); or safe methods (GET/HEAD/OPTIONS).
|
||
- Token read from `_csrf` form field or `X-CSRF-Token` request header.
|
||
- On failure: JSON `{"ok":false,"error":"CSRF token missing or invalid"}` for `/api/` paths; HTTP 403 text otherwise.
|
||
- Template delivery: `csrfToken(r)` and `csrfField(r)` helpers inject `CSRFToken` and `CSRFField` into every render data struct via `configs.go`. Templates use `{{.CSRFField}}` in forms and `csrfHeaders()` JS helper for fetch calls.
|
||
|
||
### Pages
|
||
|
||
- **Dashboard (`/`)** — Fleet overview table showing all customers with live status and event count badges (error+warning in last 24h). Config-only customers (no reports yet) appear as "PENDING" with gray badge. Blocked customers are hidden. Auto-refreshes every 60 seconds.
|
||
- **Customers (`/configs`)** — Customer management list. Shows all customers (both managed and manual), their status, controller version, and config type (MANAGED/MANUAL). Blocked customers shown grayed-out with BLOCKED badge.
|
||
- **Fleet App Analytics (`/apps`)** — Fleet-wide app telemetry overview (v0.4.0+). Shows all deployed apps across all customers with deployment count, avg/P95 memory, catalog estimate/limit accuracy indicators, and 24h error/warning badge counts. Sortable columns (deployments/memory/errors), 24h/7d/30d time period selector.
|
||
- **App Detail (`/apps/{name}`)** — Per-app drill-down page with Chart.js memory trend (avg + peak lines, catalog limit dashed line), per-customer breakdown table, and known log issues table (severity, message, occurrence count, affected customers, first/last seen). Shows suggested mem_limit from P95×1.2 rounded to 32 MB.
|
||
- **Unified Customer Detail (`/customers/{id}`)** — Single page per customer combining config management and live monitoring. Auto-refresh toggle (localStorage-persisted, enabled by default) replaces the previous hardcoded 60s meta-refresh. Since v0.47.0 the sections are organized into **8 client-side hash tabs** (`#tab=<name>`, deep-linkable, survives auto-refresh) under a sticky summary strip (name, status, controller version, last report, containers). Panels hide only via a JS-added body class — with JS off every section renders stacked (graceful degradation). Tab map:
|
||
- **Overview** (default) — customer info, health/issues/warnings, system metrics, storage, backup
|
||
- **Applications** — containers, app telemetry, received app log tails
|
||
- **Setup** — credentials, setup command generator, YAML preview
|
||
- **Settings** — controller update + version floor, geo-restriction
|
||
- **Backup & DR** — DR recipe panel + download
|
||
- **Events** — events timeline (severity filter) + report history; the tab label carries a red count badge when error events exist (last 24h)
|
||
- **Notifications** — prefs + recent notification log
|
||
- **Host** — the customer's enrolled host(s), rendered via the shared `host_detail_body` sub-template (a list by design: 1 today, N for a future HA cluster) + cross-link to `/hosts/{id}`
|
||
- **Config Form (`/configs/new`, `/configs/{id}/edit`)** — Create/edit customer configurations with identity, infrastructure tokens, and monitoring overrides. Legacy Monitoring UUIDs section collapsed by default with deprecation notice. CF API token requires **Zone DNS:Edit** (ACME) and **Zone WAF:Edit** (geo-restriction) permissions.
|
||
- **Hosts (`/hosts`, `/hosts/{id}`)** — fleet list (read-only, zero buttons — pinned by test) + per-host detail (identity, vitals, guests, storage targets, log-bundle diagnostics, DR/escrow presence). The detail body is the shared `host_detail_body` sub-template also rendered on the customer page's Host tab.
|
||
- **Offsite (`/offsite`)** — WireGuard endpoint cards + peer registry, with endpoint management since v0.47.0 (see below).
|
||
|
||
### Host lifecycle — stale host removal (v0.47.0)
|
||
|
||
Enrollment mints the host's API key exactly once (Day-0 passphrase flow); host reports authenticate via that key, so deleting a host row permanently bricks its heartbeat channel. The delete flow is therefore gated:
|
||
|
||
- The **danger-zone card** renders only for non-online hosts (stale / down / no-report). An ONLINE host is never deletable — `POST /hosts/{id}/delete` returns 409 unconditionally; no override exists.
|
||
- `GET /hosts/{id}/delete-impact` returns the blast radius as counts/booleans only (guests, reports, host-scoped log bundles, escrow/wg-peer/PBS-secret/recovery presence) — never a secret or blob.
|
||
- The dialog requires **retyping the host id**; when a key escrow exists, an unchecked-by-default checkbox ("also delete the key escrow + DR bundle") must be ticked — otherwise 409 and the transaction never starts (`store.ErrHostEscrowPresent`).
|
||
- `store.DeleteHost` cascades in ONE transaction: guests, host_reports, signed_jobs, host_recovery, host_pbs_secrets, log_bundle_requests/log_bundles with `scope_id == host_id` (customer-scoped bundles are untouched), the bound wg_peers row, host_escrow (only when acked), then the hosts row. The wgsync reconciler's 5-minute declarative push converges the endpoint after the peer row disappears.
|
||
|
||
### Offsite endpoint management (v0.47.0)
|
||
|
||
`/offsite` lists **all** `wg_endpoints` rows as cards and can add/edit/delete them (`POST /offsite/endpoints`, `POST /offsite/endpoints/{id}/delete`). Guards: full field validation (CIDR subnet, PBS IP inside subnet, port 1–65535, pubkey non-empty, id `[a-z0-9-]+`) → 400 stores nothing; changing an endpoint's `tunnel_subnet` or deleting it is refused with 409 while any peer's /32 lies inside the (current) subnet; a server-pubkey change requires a type-to-confirm (peers converge on their next desired-state pull). The peer table shows each peer's containing endpoint. **Scope guard:** peer allocation, the wgsync reconciler push, and the desired-state merge still use the lowest `endpoint_id` only (`GetWGEndpoint`) — per-endpoint allocation (`wg_peers.endpoint_id`) is a deferred future arc.
|
||
|
||
### Customer States
|
||
|
||
| State | Dashboard | Customers List | Detail Page |
|
||
|-------|-----------|----------------|-------------|
|
||
| **Active + reporting** | Shown with live status | MANAGED + status badge | Full unified view |
|
||
| **Active + no reports** | Shown as PENDING (gray) | MANAGED + no status | Config + "waiting for report" |
|
||
| **Manual (report-only)** | Shown with live status | MANUAL + status badge | Reports + "Create Config" button |
|
||
| **Blocked** | Hidden | Shown grayed-out, BLOCKED badge | Blocked banner + Unblock button |
|
||
|
||
### Customer Actions
|
||
|
||
| Action | Description |
|
||
|--------|-------------|
|
||
| **Block/Unblock** | Toggle blocked status — blocked customers are hidden from dashboard and notifications are suppressed, but reports are still accepted and stored |
|
||
| **Push Config** | Generate YAML from Hub config and POST it to the controller's `/api/config/apply` endpoint (requires controller URL from reports) |
|
||
| **Pull Config** | Import controller's current config into Hub — fetches live YAML via `GET /api/config`, extracts identity and override fields, updates Hub's stored config |
|
||
| **Show Diff** | Compare Hub-generated config with controller's live config — shows per-key differences in a color-coded table (value-based comparison, ignores key ordering and volatile fields) |
|
||
| **Create Config** | Auto-create a managed config from a manual customer's report data, then redirect to edit form |
|
||
| **Trigger Update** | Instruct controller to self-update to the latest version |
|
||
| **Delete** | Remove customer config (customer reappears as manual if reports continue) |
|
||
|
||
### Status Logic
|
||
|
||
- **OK (green):** report < 30 min old, health = ok
|
||
- **WARN (yellow):** 30-60 min stale or health = warn
|
||
- **DOWN (red):** > 60 min stale or health = fail
|
||
- **DISABLED (gray):** controller monitoring paused
|
||
- **PENDING (gray):** config exists but no reports received yet
|
||
- **BLOCKED (gray):** customer blocked by operator
|
||
|
||
## Data Storage
|
||
|
||
SQLite with WAL mode. Tables:
|
||
|
||
| Table | Purpose |
|
||
|-------|---------|
|
||
| `reports` | Full JSON reports with denormalized fields for dashboard queries |
|
||
| `events` | Structured events from controllers and Hub (type, severity, message, details, source) |
|
||
| `customer_notifications` | Email, enabled event types, cooldown hours per customer |
|
||
| `notification_log` | Send/skip/fail history for notifications with channel (operator/customer) |
|
||
| `customer_configs` | Pre-configured customer settings, retrieval passwords, per-customer API keys, status (active/blocked) |
|
||
|
||
Retention: configurable (default 90 days), daily prune at 04:30 Budapest time.
|
||
|
||
### PVC Asset Storage
|
||
|
||
App assets (logos, screenshots, branding) are stored on the PVC at `<dataDir>/assets/`. On every startup, the Hub compares SHA-256 checksums between the image seed (`/usr/share/felhom/assets-seed/`) and the PVC, updating any changed files. This means redeploying the Hub image with updated assets automatically propagates changes without PVC deletion.
|
||
|
||
A manual "Refresh Assets from Image" button is available on the **Configuration** page (`/configuration`) for triggering a re-seed + manifest rebuild on demand.
|
||
|
||
## Configuration
|
||
|
||
```yaml
|
||
# hub.yaml
|
||
auth:
|
||
password_hash: "" # bcrypt SEED for dashboard login (empty = no auth). Since v0.54.0 this
|
||
# is only the fallback: a hub_settings DB override set via
|
||
# Configuration → Login password wins. This value stays the break-glass
|
||
# reset path (blank the DB row / edit here + redeploy to recover a lost pw).
|
||
|
||
api:
|
||
report_api_key: "" # Bearer token for API auth
|
||
|
||
notifications:
|
||
resend_api_key: "" # Resend.com API key for email
|
||
from_email: "monitoring@felhom.eu"
|
||
operator_email: "" # Operator alert recipient
|
||
operator_enabled: true # Enable operator email notifications
|
||
|
||
retention:
|
||
max_days: 90
|
||
prune_schedule: "04:30"
|
||
|
||
alerting:
|
||
stale_threshold: "30m" # Customer considered stale after this duration
|
||
|
||
registry:
|
||
image: "gitea.dooplex.hu/admin/felhom-controller"
|
||
username: "" # Gitea registry credentials
|
||
token: ""
|
||
check_interval: "30m" # How often to check for new controller versions
|
||
template_interval: "1h" # How often to refresh controller.yaml.example
|
||
|
||
server:
|
||
listen: ":8080"
|
||
data_dir: "/data" # SQLite database location
|
||
```
|
||
|
||
## Deployment
|
||
|
||
Runs on k3s (Kubernetes) in the `felhom-system` namespace:
|
||
- **PVC:** 1GB Longhorn volume for SQLite database + app assets
|
||
- **Resources:** 64Mi-256Mi memory, 50m-500m CPU
|
||
- **Ingress:** `hub.felhom.eu` with TLS (cert-manager)
|
||
- **Geo-restriction:** Hungary only (nginx annotation)
|
||
|
||
```bash
|
||
# Build and push (on 192.168.0.180; felhom build dirs moved to /mnt/5_hdd/felhom.eu/ off the SSD 2026-07-18)
|
||
cd /mnt/5_hdd/felhom.eu/build/felhom-hub
|
||
./build.sh v0.3.8 --push
|
||
# Build script auto-syncs app assets from website/assets/ into the image
|
||
|
||
# Deploy (ArgoCD managed — update manifests/hub.yaml image tag, commit+push)
|
||
git pull && kubectl apply -f manifests/hub.yaml
|
||
|
||
# Check
|
||
kubectl logs -n felhom-system -l app=hub --tail 20
|
||
```
|
||
|
||
**Note:** `kubectl set image` alone does NOT persist — ArgoCD reverts it. Always update `manifests/hub.yaml` and apply.
|
||
|
||
The Dockerfile includes `COPY assets/ /usr/share/felhom/assets-seed/` which bakes app assets into the image as a seed for the PVC. The build script copies `*-logo.svg`, `*-logo.png`, and `*-screenshot-*.webp` from the website repo's `assets/` directory.
|
||
|
||
## Background Services
|
||
|
||
| Service | Schedule | Description |
|
||
|---------|----------|-------------|
|
||
| **Staleness checker** | Every 60s | Detects controllers that stopped reporting. Generates `node_stale` (>30min), `node_down` (>60min), `node_recovered` events |
|
||
| **Backup deadline checker** | Daily 05:00 Budapest | Detects missing backup/db-dump events since midnight. Generates `expected_backup_missed`, `expected_dbdump_missed` events |
|
||
| **Report/event prune** | Daily 04:30 Budapest | Deletes reports and events older than retention period (default 90 days) |
|
||
| **Registry version check** | Every 30min | Checks Gitea registry for new controller image tags |
|
||
| **Template refresh** | Every 1h | Fetches latest `controller.yaml.example` from Gitea |
|
||
| **Asset seeding** | On startup | Compares SHA-256 checksums and updates changed assets from Docker image seed |
|
||
|
||
## Internal Packages
|
||
|
||
| Package | Purpose |
|
||
|---------|---------|
|
||
| `internal/api` | REST API handler (report ingest, config, events, assets, notifications) |
|
||
| `internal/web` | Web dashboard (session auth, customer management, fleet overview) |
|
||
| `internal/assets` | PVC asset manager (manifest generation, SHA-256 checksums, file serving, image seed) |
|
||
| `internal/configgen` | Shared YAML config generation (deep-merge template + customer overrides) |
|
||
|
||
## Dependencies
|
||
|
||
- `golang.org/x/crypto` — bcrypt for password hashing
|
||
- `gopkg.in/yaml.v3` — YAML config parsing
|
||
- `modernc.org/sqlite` — Pure Go SQLite (no CGo)
|