Files
felhom.eu/hub
Claude Code f5a5e2b911 hub v0.75.0: R-81 — "no signal" is not "bad signal" (anchor the backup deadline check)
Third instance of one class (hub v0.12.0, v0.73.0, this), fixed as a class.
On 2026-07-26 03:00 UTC expected_backup_missed fired on demo-felhom, demo-hp
and drill-r50 at once; the demo-felhom one reached the CUSTOMER channel
claiming "newest backup is 176h0m0s old". Nothing was wrong — three vzdump
archives were on disk. Cause: the agent backup store is in-memory, so the
R-50 fleet restart emptied `backups` until the next run, and the hub read
empty as "no backup exists".

- assessBackupFreshness returns OK/UNKNOWN/MISSED instead of `missed bool`;
  absence is UNKNOWN until it outlives an anchored window. Still pure.
- store.GetHostReportsSince + monitor.newestBackupEvidence read the hubs own
  retained history (bounded 7-day lookback, early-exit on fresh evidence) —
  "when did I last SEE evidence of a backup?" The anchor was free: the hub
  already retains 90 days. No agent change, no new persisted state.
- store.GetFirstHostReportAt anchors absence at first contact, reusing the
  existing 26h threshold as the grace (no new knob, the v0.73.0 shape).
- Deferrals logged + counted; reason strings kept distinct.
- backupStaleAfter untouched; landmine recorded (a weekly PBS snapshot would
  alarm six days in seven) and owned by R-82.

Tests 493->508. Red-proofs A/B/C observed and restored; A reproduces the live
message verbatim. Replayed the real 03:00 reports (600/417/77 rows): all
three now silent.

Source: documentation/audits/DIAG-backup-missed-2026-07-26.md
2026-07-26 11:44:15 +02:00
..
2026-02-16 14:16:11 +01:00

felhom-hub

Central operator dashboard for monitoring and managing Felhom customer deployments.

A lightweight Go service that receives periodic reports and structured events from felhom-controller instances, stores them in SQLite, and provides a web dashboard for fleet monitoring. Also serves as the infrastructure backup store for disaster recovery, event-based dead man's switch monitoring, and notification dispatch.

Current version: v0.6.3


Architecture

   Customer nodes                             Central Hub (k3s)
┌─────────────────┐                     ┌────────────────────────┐
│ felhom-controller│──── JSON push ────▶│  felhom-hub            │
│ (every 15 min)   │    (Bearer auth)   │                        │
│                  │                     │  ┌─────────────────┐   │
│ POST /api/v1/    │                     │  │ API Handler     │   │
│   report         │                     │  │ (ingest reports, │   │
│   host-report    │◀── config push ────│  │  host reports,   │   │
│   notify         │    (YAML body)     │  │  config push,    │   │
│                  │                     │  │  asset serving)  │   │
│ GET /api/v1/     │                     │  └────────┬────────┘   │
│   assets/*       │◀── asset download ─│           │             │
└─────────────────┘    (Bearer auth)    │  ┌────────▼────────┐   │
                                        │  │ SQLite Store    │   │
   Operator browser                     │  │ (reports,       │   │
┌─────────────────┐                     │  │  assets,        │   │
│ Web Dashboard   │◀── HTML pages ──────│  │  host_reports,  │   │
│ (hub.felhom.eu) │    (bcrypt auth)    │  │  configs,       │   │
└─────────────────┘                     │  │  notifications) │   │
                                        │  └─────────────────┘   │
                                        │                        │
                                        │  ┌─────────────────┐   │
                                        │  │ Asset Manager   │   │
                                        │  │ (PVC storage,   │   │
                                        │  │  SHA-256 manifest│   │
                                        │  │  file serving)  │   │
                                        │  └─────────────────┘   │
                                        │                        │
                                        │  ┌─────────────────┐   │
                                        │  │ Web Dashboard   │   │
                                        │  │ (unified customer│   │
                                        │  │  management)     │   │
                                        │  └─────────────────┘   │
                                        └────────────────────────┘

API Endpoints

All API endpoints require Authorization: Bearer <api_key> (except /healthz and /api/v1/config/{id}). Auth accepts both the global report_api_key and per-customer API keys (generated when creating customer configs).

Report Ingest

Method Path Description
POST /api/v1/report Controller pushes periodic status report (v0.28.0+ includes app_telemetry field)
GET /api/v1/customers List all customers with latest report summary
GET /api/v1/customers/{id} Get latest full report for a customer
GET /api/v1/customers/{id}/history?period=7d Get report history

The POST /api/v1/report handler (v0.4.0+) automatically parses the optional app_telemetry JSON array from the request body and stores it in app_telemetry / app_log_issues tables. Old controllers (no app_telemetry key) continue to work unchanged.

DR Recipe — secret-free reconstruction recipe (hub v0.13.0)

The hub assembles + stores + serves the secret-free DR recipe (documentation/audits/SPIKE-dr-recipe-2026-06-16.md) — the non-secret re-provision plan that complements escrow (keys) + PBS/restic (bytes). It arrives as two additive dr_recipe halves on the existing report paths: the agent's storage/guest/PBS half on POST /api/v1/host-report and the controller's customer/apps half on POST /api/v1/report (both backward-compatible, ignore-unknown). The hub stores them PLAINTEXT in a dedicated dr_recipe table keyed by customer_id (SaveDRRecipeHostHalf / SaveDRRecipeAppHalf, each preserving the other half), and AssembleDRRecipe stitches them into one operator-readable recipe ({recipe_version, customer, guests, pbs, drives, pve_storage, apps}; sub-sections pass through verbatim, version = max of the two halves). An operator views the panel on the customer page and downloads the assembled JSON at GET /customers/{id}/dr-recipe.json. Plaintext-at-rest is correct here — the recipe carries only identifiers/intents/sizes/coordinates, never a key/password/token (the boundary is enforced at the controller emitter). This is the clean inverse of the retired infra-backup.

Offsite provisioning (SLICE 1, hub v0.37.0)

On operator enable (config form → Offsite backup), the hub provisions the customer's offsite tier against the Hetzner storage-box API at https://api.hetzner.com/v1 (NOT api.hetzner.cloud — the classic Cloud API 404s for storage boxes; shapes measured in documentation/audits/SPIKE-hetzner-api-provisioning-2026-07-09.md). Two models: shared (a sub-account on the pool box) or dedicated (its own box). Flow (internal/offsite): idempotent lookup by label felhom-customer=<id> → create (password generated hub-side) → poll the action to successSaveOneTimeSecret (the transient password, single-use) → merge the non-secret descriptor ({enabled,type,host,user,port:23,repo_path,quota_gb|box_type}) into ConfigJSONConfigVersion bump → the controller re-pulls. The controller consumes the one-time password at POST /offsite/consume-password/{id} (customer-API-key auth, single-use) — then installs its key and the hub resets the box password (SLICE 2+). Fail-closed: a provisioning error returns 502 and saves nothing. Secrets: the Hetzner token (HETZNER_TOKEN, out-of-band) and every generated password are NEVER logged / committed / in ConfigJSON. PREREQUISITE for live use: the token MUST be scoped to a dedicated Hetzner project (the shared project token can delete ep0 — spike §6); check the scope by listing boxes with it. Absent token → the UI still renders; saving with offsite enabled returns "not configured". Live-validated end-to-end 2026-07-09 (documentation/audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md): SLICE 2 (controller apply-bridge) is shipped and applied. Hardening (v0.38.1v0.39.0): provisioning runs on a client-disconnect-proof detached context (F1); the host-key scan retries through fresh-subaccount DNS lag (~60s ladder, F2); the save button disables with an in-flight notice (F5); and "Re-issue offsite credentials" (F4, POST /configs/{id}/offsite-reissue) is the explicit recovery for a consumed-password dead-end — resets ONLY the resource labelled for that customer (refuses unless exactly 1), stores a fresh one-time secret, bumps ConfigVersion. Never implicit rotation. SLICE 3 (v0.40.0) — hub-verified escrow auto-confirm: the agent's ceremony upload carries restic_pw_sha256 (non-reversible hash of the sealed offsite repo password — safe to store/serve); the hub stores it on host_escrow (NULL on legacy rows) and serves escrow:{identity_blob_present, restic_pw_sha256, created_at} in the report ACK; the controller auto-confirms its pending offbox escrow ONLY on a hash match with its current repo password (blob-presence alone never confirms). SLICE 4 (v0.41.0) — offsite health monitoring + freeze lever: monitor.OffsiteChecker reads the controller report's offsite object — fill alerts at 90/95% of quota_gb (quota 0 = dedicated, silent) and an offsite_stale warning when an enabled+escrowed target has no run in >48h (the silently-stuck detector; run FAILURES are backup_failed's job; pending/disabled never alert). Nil-safe on pre-v0.109 reports. The operator Freeze/Unfreeze offsite buttons (shared model only, next to Re-issue) flip ONLY readonly on the exactly-1 labelled sub-account — MANUAL only, never automatic (freezing also blocks prune, the customer's only way down from over-quota).

Infrastructure Backup — RETIRED (Phase-1, 2026-06-16, hub v0.12.0)

The Infra Backup mechanism (POST/GET /api/v1/infra-backup, the operator panel, the infra_backup_versions / infra_backups tables) has been removed. It had been dead since slice 8C (the disk-tier backup moved to the host agent), and it stored each version as a plaintext JSON blob holding the customer's app-secret encryption key, restic password, and Cloudflare tokens — a zero-knowledge violation. The retirement migration DROPs both tables and VACUUMs the DB to physically reclaim the plaintext pages. Disaster recovery now rests on the agent's PBS whole-CT snapshot (the data bytes) plus the generated controller.yaml from the recovery endpoint (the config); a secret-free DR recipe is the later DR slice's job. See documentation/audits/SPIKE-infra-backup-2026-06-15.md.

Recovery (Disaster Recovery)

Method Path Description
GET /api/v1/recovery/{customer_id} Returns the generated controller.yaml for a customer

Auth: X-Retrieval-Password header (same per-customer password as config retrieval). Response:

{
  "customer_id": "example",
  "config_yaml": "customer:\n  id: example\n  ...",
  "has_infra_backup": false
}

The has_infra_backup field is retained as false so any old client degrades gracefully to the config-only path.

Report Response

The POST /api/v1/report response now includes customer_blocked: true when the customer's status is "blocked". Controllers use this to detect their standing and enter limited mode after a grace period.

Events

Method Path Description
POST /api/v1/event Controller pushes structured event (27 allowed types, severity: info/warning/error)

Events are the primary monitoring mechanism. Each event has: customer_id, event_type, severity, message, details_json, source. Per-customer API keys are validated against the customer_id in the payload. Stored in the events table with automatic pruning.

Hub-generated events (source="hub"):

  • node_stale / node_down / node_recovered — dead man's switch from staleness checker (every 60s)
  • host_stale / host_down / host_recovered — host-domain dead man's switch (agent host reports)
  • agent_capability_degraded / agent_capability_recoveredHostCapabilityChecker: an agent's required sudo -n grant went missing (the non-root cutover class)
  • host_leaf_changedHostLeafChecker (monitor/host_leaf.go, every 60s): an agent's reported local-API leaf fingerprint changed (an agent re-key) — proactive, fleet-wide, independent of any controller's channel-health check. Trust-on-first-report baseline; empty fp = unknown (never alerts). Reads the fp from the latest host-report (store.GetHostLeafFingerprints, no schema migration).
  • expected_backup_missed / expected_dbdump_missed — backup deadline checker (daily at 05:00 Budapest)

Notifications

Method Path Description
POST /api/v1/notify Legacy notification relay (kept for backward compatibility)
POST /api/v1/preferences Controller syncs customer notification preferences (email, enabled_events, cooldown_hours)

Notifications are dispatched automatically when events are processed:

  • Operator channel: English emails for warning/error events, 1h cooldown per customer:eventType
  • Customer channel: Hungarian emails per event type, respects customer preferences and cooldown (default 6h)
  • Email delivery via Resend.com API

Customer Config Retrieval

Method Path Description
GET /api/v1/config/{customer_id} Download generated controller.yaml (auth: X-Retrieval-Password header)

Config retrieval uses a separate per-customer retrieval password (not the API key). Retrieval passwords are auto-generated as Hungarian word passphrases (e.g., alma-kerék-madár-felhő) for easy phone-based entry during disaster recovery. The Hub generates a complete controller.yaml by deep-merging controller.yaml.example (periodically fetched from the Gitea repo) with customer-specific overrides (identity, infrastructure tokens, hub API key, session secret).

Host Enrollment (Day-0, option C)

Method Path Description
POST /api/v1/host-enroll Mint-or-reuse a host credential, body {customer_id} (auth: X-Retrieval-Password header) → {host_id, api_key}

The Day-0 first-boot handshake (documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md) enrolls a Proxmox host's agent using the customer's retrieval passphrase only — the operator-tier global key never enters the field deploy path. host-enroll mints the per-host credential on the first call and reuses it byte-for-byte on every later call (idempotent — re-running the host-bootstrap never orphans a running agent's key), backed by Store.GetHostByCustomer. Auth is checked before any mint, so a wrong passphrase never creates a host row. Behaviour: first → 201, reuse → 200, wrong/missing passphrase → 401, unknown customer → 404, missing customer_id400.

The global-key POST /api/v1/admin/hosts (operator/HQ pre-mint) remains as the escape hatch pending the enrollment-cutover lock-down; it and GET /config/{id} are unchanged by this endpoint.

Assets

Method Path Description
GET /api/v1/assets/manifest JSON manifest of all assets with SHA-256 checksums
GET /api/v1/assets/file/{filename} Download a single asset file (logo, screenshot)

Assets are stored on the Hub PVC at <dataDir>/assets/. On first run, assets are seeded from the Docker image (/usr/share/felhom/assets-seed/). The manifest includes filename, size, and SHA-256 hash for each file — controllers use this for efficient change detection.

Asset types served: {slug}-logo.svg, {slug}-logo.png, {slug}-screenshot-{N}.webp

The asset manager (internal/assets/) scans the assets directory on startup, builds an in-memory manifest, and serves files with appropriate Content-Type and cache headers. Both endpoints require Bearer token auth (global or per-customer API key).

Health

Method Path Description
GET /healthz Health check (no auth required, returns 503 if SQLite ping fails)

Web Dashboard

Protected by bcrypt password + session cookie (7-day expiry).

Authentication & Session Model (internal/web/server.go)

  • Login generates a cryptographically random 64-char hex session token stored server-side in a map[string]*hubSession (+ sync.RWMutex). The old literal hub_session=authenticated cookie is gone.
  • Each session also stores a per-session CSRF token (separate 64-char hex random value).
  • Cookie attributes: SameSite=Lax, Secure (when TLS), HttpOnly, 7-day Max-Age.
  • RequireAuth middleware validates the session token with subtle.ConstantTimeCompare and redirects to /login on failure.
  • CleanupSessions(ctx) goroutine runs hourly to purge expired sessions.
  • Login password source (v0.54.0) — the bcrypt hash checked at login comes from effectivePasswordHash(): a hub_settings.operator_password_hash DB override (set via Configuration → Login password) wins, otherwise the auth.password_hash seed from hub.yaml. The ConfigMap value is the break-glass reset path (blank the DB row / edit the manifest + redeploy). Changing the password (POST /configuration/password) requires the current password and leaves existing sessions valid.

CSRF Protection (internal/web/server.go)

Synchronizer-token CSRF protection on all browser POST/DELETE/PATCH operations:

  • CSRF validation block runs at the top of ServeHTTP before routing.
  • Skipped when: no session cookie present (API/Basic-Auth path); or safe methods (GET/HEAD/OPTIONS).
  • Token read from _csrf form field or X-CSRF-Token request header.
  • On failure: JSON {"ok":false,"error":"CSRF token missing or invalid"} for /api/ paths; HTTP 403 text otherwise.
  • Template delivery: csrfToken(r) and csrfField(r) helpers inject CSRFToken and CSRFField into every render data struct via configs.go. Templates use {{.CSRFField}} in forms and csrfHeaders() JS helper for fetch calls.

Pages

  • Dashboard (/) — Fleet overview table showing all customers with live status and event count badges (error+warning in last 24h). Config-only customers (no reports yet) appear as "PENDING" with gray badge. Blocked customers are hidden. Auto-refreshes every 60 seconds.
  • Customers (/configs) — Customer management list. Shows all customers (both managed and manual), their status, controller version, and config type (MANAGED/MANUAL). Blocked customers shown grayed-out with BLOCKED badge.
  • Fleet App Analytics (/apps) — Fleet-wide app telemetry overview (v0.4.0+). Shows all deployed apps across all customers with deployment count, avg/P95 memory, catalog estimate/limit accuracy indicators, and 24h error/warning badge counts. Sortable columns (deployments/memory/errors), 24h/7d/30d time period selector.
  • App Detail (/apps/{name}) — Per-app drill-down page with Chart.js memory trend (avg + peak lines, catalog limit dashed line), per-customer breakdown table, and known log issues table (severity, message, occurrence count, affected customers, first/last seen). Shows suggested mem_limit from P95×1.2 rounded to 32 MB.
  • Unified Customer Detail (/customers/{id}) — Single page per customer combining config management and live monitoring. Auto-refresh toggle (localStorage-persisted, enabled by default) replaces the previous hardcoded 60s meta-refresh. Since v0.47.0 the sections are organized into 8 client-side hash tabs (#tab=<name>, deep-linkable, survives auto-refresh) under a sticky summary strip (name, status, controller version, last report, containers). Panels hide only via a JS-added body class — with JS off every section renders stacked (graceful degradation). Tab map:
    • Overview (default) — customer info, health/issues/warnings, system metrics, storage, backup
    • Applications — containers, app telemetry, received app log tails
    • Setup — credentials, setup command generator, YAML preview
    • Settings — controller update + version floor, geo-restriction
    • Backup & DR — DR recipe panel + download
    • Events — events timeline (severity filter) + report history; the tab label carries a red count badge when error events exist (last 24h)
    • Notifications — prefs + recent notification log
    • Host — the customer's enrolled host(s), rendered via the shared host_detail_body sub-template (a list by design: 1 today, N for a future HA cluster) + cross-link to /hosts/{id}
  • Config Form (/configs/new, /configs/{id}/edit) — Create/edit customer configurations with identity, infrastructure tokens, and monitoring overrides. Legacy Monitoring UUIDs section collapsed by default with deprecation notice. CF API token requires Zone DNS:Edit (ACME) and Zone WAF:Edit (geo-restriction) permissions.
  • Hosts (/hosts, /hosts/{id}) — fleet list (read-only, zero buttons — pinned by test) + per-host detail (identity, vitals, guests, storage targets, log-bundle diagnostics, DR/escrow presence). The detail body is the shared host_detail_body sub-template also rendered on the customer page's Host tab.
  • Offsite (/offsite) — WireGuard endpoint cards + peer registry, with endpoint management since v0.47.0 (see below).

Host lifecycle — stale host removal (v0.47.0)

Enrollment mints the host's API key exactly once (Day-0 passphrase flow); host reports authenticate via that key, so deleting a host row permanently bricks its heartbeat channel. The delete flow is therefore gated:

  • The danger-zone card renders only for non-online hosts (stale / down / no-report). An ONLINE host is never deletable — POST /hosts/{id}/delete returns 409 unconditionally; no override exists.
  • GET /hosts/{id}/delete-impact returns the blast radius as counts/booleans only (guests, reports, host-scoped log bundles, escrow/wg-peer/PBS-secret/recovery presence) — never a secret or blob.
  • The dialog requires retyping the host id; when a key escrow exists, an unchecked-by-default checkbox ("also delete the key escrow + DR bundle") must be ticked — otherwise 409 and the transaction never starts (store.ErrHostEscrowPresent).
  • store.DeleteHost cascades in ONE transaction: guests, host_reports, signed_jobs, host_recovery, host_pbs_secrets, log_bundle_requests/log_bundles with scope_id == host_id (customer-scoped bundles are untouched), the bound wg_peers row, host_escrow (only when acked), then the hosts row. The wgsync reconciler's 5-minute declarative push converges the endpoint after the peer row disappears.

Offsite endpoint management (v0.47.0)

/offsite lists all wg_endpoints rows as cards and can add/edit/delete them (POST /offsite/endpoints, POST /offsite/endpoints/{id}/delete). Guards: full field validation (CIDR subnet, PBS IP inside subnet, port 165535, pubkey non-empty, id [a-z0-9-]+) → 400 stores nothing; changing an endpoint's tunnel_subnet or deleting it is refused with 409 while any peer's /32 lies inside the (current) subnet; a server-pubkey change requires a type-to-confirm (peers converge on their next desired-state pull). The peer table shows each peer's containing endpoint. Scope guard: peer allocation, the wgsync reconciler push, and the desired-state merge still use the lowest endpoint_id only (GetWGEndpoint) — per-endpoint allocation (wg_peers.endpoint_id) is a deferred future arc.

Customer States

State Dashboard Customers List Detail Page
Active + reporting Shown with live status MANAGED + status badge Full unified view
Active + no reports Shown as PENDING (gray) MANAGED + no status Config + "waiting for report"
Manual (report-only) Shown with live status MANUAL + status badge Reports + "Create Config" button
Blocked Hidden Shown grayed-out, BLOCKED badge Blocked banner + Unblock button

Customer Actions

Action Description
Block/Unblock Toggle blocked status — blocked customers are hidden from dashboard and notifications are suppressed, but reports are still accepted and stored
Push Config Generate YAML from Hub config and POST it to the controller's /api/config/apply endpoint (requires controller URL from reports)
Pull Config Import controller's current config into Hub — fetches live YAML via GET /api/config, extracts identity and override fields, updates Hub's stored config
Show Diff Compare Hub-generated config with controller's live config — shows per-key differences in a color-coded table (value-based comparison, ignores key ordering and volatile fields)
Create Config Auto-create a managed config from a manual customer's report data, then redirect to edit form
Trigger Update Instruct controller to self-update to the latest version
Delete Remove customer config (customer reappears as manual if reports continue)

Status Logic

  • OK (green): report < 30 min old, health = ok
  • WARN (yellow): 30-60 min stale or health = warn
  • DOWN (red): > 60 min stale or health = fail
  • DISABLED (gray): controller monitoring paused
  • PENDING (gray): config exists but no reports received yet
  • BLOCKED (gray): customer blocked by operator

Data Storage

SQLite with WAL mode. Tables:

Table Purpose
reports Full JSON reports with denormalized fields for dashboard queries
events Structured events from controllers and Hub (type, severity, message, details, source)
customer_notifications Email, enabled event types, cooldown hours per customer
notification_log Send/skip/fail history for notifications with channel (operator/customer)
customer_configs Pre-configured customer settings, retrieval passwords, per-customer API keys, status (active/blocked)

Retention: configurable (default 90 days), daily prune at 04:30 Budapest time.

PVC Asset Storage

App assets (logos, screenshots, branding) are stored on the PVC at <dataDir>/assets/. On every startup, the Hub compares SHA-256 checksums between the image seed (/usr/share/felhom/assets-seed/) and the PVC, updating any changed files. This means redeploying the Hub image with updated assets automatically propagates changes without PVC deletion.

A manual "Refresh Assets from Image" button is available on the Configuration page (/configuration) for triggering a re-seed + manifest rebuild on demand.

Configuration

# hub.yaml
auth:
  password_hash: ""           # bcrypt SEED for dashboard login (empty = no auth). Since v0.54.0 this
                              # is only the fallback: a hub_settings DB override set via
                              # Configuration → Login password wins. This value stays the break-glass
                              # reset path (blank the DB row / edit here + redeploy to recover a lost pw).

api:
  report_api_key: ""          # Bearer token for API auth

notifications:
  resend_api_key: ""          # Resend.com API key for email
  from_email: "monitoring@felhom.eu"
  operator_email: ""          # Operator alert recipient
  operator_enabled: true      # Enable operator email notifications

retention:
  max_days: 90
  prune_schedule: "04:30"

alerting:
  stale_threshold: "30m"      # Customer considered stale after this duration

registry:
  image: "gitea.dooplex.hu/admin/felhom-controller"
  username: ""                # Gitea registry credentials
  token: ""
  check_interval: "30m"      # How often to check for new controller versions
  template_interval: "1h"    # How often to refresh controller.yaml.example

server:
  listen: ":8080"
  data_dir: "/data"           # SQLite database location

Deployment

Runs on k3s (Kubernetes) in the felhom-system namespace:

  • PVC: 1GB Longhorn volume for SQLite database + app assets
  • Resources: 64Mi-256Mi memory, 50m-500m CPU
  • Ingress: hub.felhom.eu with TLS (cert-manager)
  • Geo-restriction: Hungary only (nginx annotation)
# Build and push (on 192.168.0.180; felhom build dirs moved to /mnt/5_hdd/felhom.eu/ off the SSD 2026-07-18)
cd /mnt/5_hdd/felhom.eu/build/felhom-hub
./build.sh v0.3.8 --push
# Build script auto-syncs app assets from website/assets/ into the image

# Deploy (ArgoCD managed — update manifests/hub.yaml image tag, commit+push)
git pull && kubectl apply -f manifests/hub.yaml

# Check
kubectl logs -n felhom-system -l app=hub --tail 20

Note: kubectl set image alone does NOT persist — ArgoCD reverts it. Always update manifests/hub.yaml and apply.

The Dockerfile includes COPY assets/ /usr/share/felhom/assets-seed/ which bakes app assets into the image as a seed for the PVC. The build script copies *-logo.svg, *-logo.png, and *-screenshot-*.webp from the website repo's assets/ directory.

Background Services

Service Schedule Description
Staleness checker Every 60s Detects controllers that stopped reporting. Generates node_stale (>30min), node_down (>60min), node_recovered events
Backup deadline checker Daily 05:00 Budapest Detects missing backup/db-dump events since midnight. Generates expected_backup_missed, expected_dbdump_missed events
Report/event prune Daily 04:30 Budapest Deletes reports and events older than retention period (default 90 days)
Registry version check Every 30min Checks Gitea registry for new controller image tags
Template refresh Every 1h Fetches latest controller.yaml.example from Gitea
Asset seeding On startup Compares SHA-256 checksums and updates changed assets from Docker image seed

Internal Packages

Package Purpose
internal/api REST API handler (report ingest, config, events, assets, notifications)
internal/web Web dashboard (session auth, customer management, fleet overview)
internal/assets PVC asset manager (manifest generation, SHA-256 checksums, file serving, image seed)
internal/configgen Shared YAML config generation (deep-merge template + customer overrides)

Dependencies

  • golang.org/x/crypto — bcrypt for password hashing
  • gopkg.in/yaml.v3 — YAML config parsing
  • modernc.org/sqlite — Pure Go SQLite (no CGo)