F2 claim re-issue on clean-slate re-enroll (ReissueForReenroll, host-enroll mint path, single-bump, reset code; hub never stores the password so fork B). F3 offsite re-issue on re-enroll (ReissueOffsiteForCustomer, same machinery as the manual button). 2.3 escrow honesty (red-proofed): re-issuing offsite marks the escrow stale (MarkEscrowStale), withholds the mismatched restic hash from auto-confirm, DR checklist shows stale not done. Events: claim_reissued_reenroll / offsite_reissued / escrow_stale. Controller + scripts unchanged (source contradicted both premises): the controller reads escrow prereqs live from the agent; the installer can't know the descriptor-provisioned storage id. F4 root fix is agent-side -> ROADMAP R-22; demo unblocked live (Part 0 ACL grant). VALIDATION doc F2 erratum + F3/F4 dispositions. Green gate + Scenario-C red-proof pass.
felhom-hub
Central operator dashboard for monitoring and managing Felhom customer deployments.
A lightweight Go service that receives periodic reports and structured events from felhom-controller instances, stores them in SQLite, and provides a web dashboard for fleet monitoring. Also serves as the infrastructure backup store for disaster recovery, event-based dead man's switch monitoring, and notification dispatch.
Current version: v0.6.3
Architecture
Customer nodes Central Hub (k3s)
┌─────────────────┐ ┌────────────────────────┐
│ felhom-controller│──── JSON push ────▶│ felhom-hub │
│ (every 15 min) │ (Bearer auth) │ │
│ │ │ ┌─────────────────┐ │
│ POST /api/v1/ │ │ │ API Handler │ │
│ report │ │ │ (ingest reports, │ │
│ host-report │◀── config push ────│ │ host reports, │ │
│ notify │ (YAML body) │ │ config push, │ │
│ │ │ │ asset serving) │ │
│ GET /api/v1/ │ │ └────────┬────────┘ │
│ assets/* │◀── asset download ─│ │ │
└─────────────────┘ (Bearer auth) │ ┌────────▼────────┐ │
│ │ SQLite Store │ │
Operator browser │ │ (reports, │ │
┌─────────────────┐ │ │ assets, │ │
│ Web Dashboard │◀── HTML pages ──────│ │ host_reports, │ │
│ (hub.felhom.eu) │ (bcrypt auth) │ │ configs, │ │
└─────────────────┘ │ │ notifications) │ │
│ └─────────────────┘ │
│ │
│ ┌─────────────────┐ │
│ │ Asset Manager │ │
│ │ (PVC storage, │ │
│ │ SHA-256 manifest│ │
│ │ file serving) │ │
│ └─────────────────┘ │
│ │
│ ┌─────────────────┐ │
│ │ Web Dashboard │ │
│ │ (unified customer│ │
│ │ management) │ │
│ └─────────────────┘ │
└────────────────────────┘
API Endpoints
All API endpoints require Authorization: Bearer <api_key> (except /healthz and /api/v1/config/{id}). Auth accepts both the global report_api_key and per-customer API keys (generated when creating customer configs).
Report Ingest
| Method | Path | Description |
|---|---|---|
POST |
/api/v1/report |
Controller pushes periodic status report (v0.28.0+ includes app_telemetry field) |
GET |
/api/v1/customers |
List all customers with latest report summary |
GET |
/api/v1/customers/{id} |
Get latest full report for a customer |
GET |
/api/v1/customers/{id}/history?period=7d |
Get report history |
The POST /api/v1/report handler (v0.4.0+) automatically parses the optional app_telemetry JSON array from the request body and stores it in app_telemetry / app_log_issues tables. Old controllers (no app_telemetry key) continue to work unchanged.
DR Recipe — secret-free reconstruction recipe (hub v0.13.0)
The hub assembles + stores + serves the secret-free DR recipe (documentation/audits/SPIKE-dr-recipe-2026-06-16.md) — the non-secret re-provision plan that complements escrow (keys) + PBS/restic (bytes). It arrives as two additive dr_recipe halves on the existing report paths: the agent's storage/guest/PBS half on POST /api/v1/host-report and the controller's customer/apps half on POST /api/v1/report (both backward-compatible, ignore-unknown). The hub stores them PLAINTEXT in a dedicated dr_recipe table keyed by customer_id (SaveDRRecipeHostHalf / SaveDRRecipeAppHalf, each preserving the other half), and AssembleDRRecipe stitches them into one operator-readable recipe ({recipe_version, customer, guests, pbs, drives, pve_storage, apps}; sub-sections pass through verbatim, version = max of the two halves). An operator views the panel on the customer page and downloads the assembled JSON at GET /customers/{id}/dr-recipe.json. Plaintext-at-rest is correct here — the recipe carries only identifiers/intents/sizes/coordinates, never a key/password/token (the boundary is enforced at the controller emitter). This is the clean inverse of the retired infra-backup.
Offsite provisioning (SLICE 1, hub v0.37.0)
On operator enable (config form → Offsite backup), the hub provisions the customer's offsite tier against
the Hetzner storage-box API at https://api.hetzner.com/v1 (NOT api.hetzner.cloud — the classic Cloud
API 404s for storage boxes; shapes measured in documentation/audits/SPIKE-hetzner-api-provisioning-2026-07-09.md).
Two models: shared (a sub-account on the pool box) or dedicated (its own box). Flow (internal/offsite):
idempotent lookup by label felhom-customer=<id> → create (password generated hub-side) → poll the action to
success → SaveOneTimeSecret (the transient password, single-use) → merge the non-secret descriptor
({enabled,type,host,user,port:23,repo_path,quota_gb|box_type}) into ConfigJSON → ConfigVersion bump →
the controller re-pulls. The controller consumes the one-time password at POST /offsite/consume-password/{id}
(customer-API-key auth, single-use) — then installs its key and the hub resets the box password (SLICE 2+).
Fail-closed: a provisioning error returns 502 and saves nothing. Secrets: the Hetzner token
(HETZNER_TOKEN, out-of-band) and every generated password are NEVER logged / committed / in ConfigJSON.
PREREQUISITE for live use: the token MUST be scoped to a dedicated Hetzner project (the shared
project token can delete ep0 — spike §6); check the scope by listing boxes with it. Absent token → the UI
still renders; saving with offsite enabled returns "not configured". Live-validated end-to-end 2026-07-09
(documentation/audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md): SLICE 2 (controller apply-bridge)
is shipped and applied.
Hardening (v0.38.1–v0.39.0): provisioning runs on a client-disconnect-proof detached context (F1); the
host-key scan retries through fresh-subaccount DNS lag (~60s ladder, F2); the save button disables with an
in-flight notice (F5); and "Re-issue offsite credentials" (F4, POST /configs/{id}/offsite-reissue) is
the explicit recovery for a consumed-password dead-end — resets ONLY the resource labelled for that customer
(refuses unless exactly 1), stores a fresh one-time secret, bumps ConfigVersion. Never implicit rotation.
SLICE 3 (v0.40.0) — hub-verified escrow auto-confirm: the agent's ceremony upload carries
restic_pw_sha256 (non-reversible hash of the sealed offsite repo password — safe to store/serve); the hub
stores it on host_escrow (NULL on legacy rows) and serves
escrow:{identity_blob_present, restic_pw_sha256, created_at} in the report ACK; the controller
auto-confirms its pending offbox escrow ONLY on a hash match with its current repo password (blob-presence
alone never confirms).
SLICE 4 (v0.41.0) — offsite health monitoring + freeze lever: monitor.OffsiteChecker reads the
controller report's offsite object — fill alerts at 90/95% of quota_gb (quota 0 = dedicated, silent)
and an offsite_stale warning when an enabled+escrowed target has no run in >48h (the silently-stuck
detector; run FAILURES are backup_failed's job; pending/disabled never alert). Nil-safe on pre-v0.109
reports. The operator Freeze/Unfreeze offsite buttons (shared model only, next to Re-issue) flip ONLY
readonly on the exactly-1 labelled sub-account — MANUAL only, never automatic (freezing also blocks
prune, the customer's only way down from over-quota).
Infrastructure Backup — RETIRED (Phase-1, 2026-06-16, hub v0.12.0)
The Infra Backup mechanism (POST/GET /api/v1/infra-backup, the operator panel, the
infra_backup_versions / infra_backups tables) has been removed. It had been dead since
slice 8C (the disk-tier backup moved to the host agent), and it stored each version as a plaintext
JSON blob holding the customer's app-secret encryption key, restic password, and Cloudflare tokens —
a zero-knowledge violation. The retirement migration DROPs both tables and VACUUMs the DB to
physically reclaim the plaintext pages. Disaster recovery now rests on the agent's PBS whole-CT
snapshot (the data bytes) plus the generated controller.yaml from the recovery endpoint (the
config); a secret-free DR recipe is the later DR slice's job. See
documentation/audits/SPIKE-infra-backup-2026-06-15.md.
Recovery (Disaster Recovery)
| Method | Path | Description |
|---|---|---|
GET |
/api/v1/recovery/{customer_id} |
Returns the generated controller.yaml for a customer |
Auth: X-Retrieval-Password header (same per-customer password as config retrieval). Response:
{
"customer_id": "example",
"config_yaml": "customer:\n id: example\n ...",
"has_infra_backup": false
}
The has_infra_backup field is retained as false so any old client degrades gracefully to the
config-only path.
Report Response
The POST /api/v1/report response now includes customer_blocked: true when the customer's status is "blocked". Controllers use this to detect their standing and enter limited mode after a grace period.
Events
| Method | Path | Description |
|---|---|---|
POST |
/api/v1/event |
Controller pushes structured event (27 allowed types, severity: info/warning/error) |
Events are the primary monitoring mechanism. Each event has: customer_id, event_type, severity, message, details_json, source. Per-customer API keys are validated against the customer_id in the payload. Stored in the events table with automatic pruning.
Hub-generated events (source="hub"):
node_stale/node_down/node_recovered— dead man's switch from staleness checker (every 60s)host_stale/host_down/host_recovered— host-domain dead man's switch (agent host reports)agent_capability_degraded/agent_capability_recovered—HostCapabilityChecker: an agent's requiredsudo -ngrant went missing (the non-root cutover class)host_leaf_changed—HostLeafChecker(monitor/host_leaf.go, every 60s): an agent's reported local-API leaf fingerprint changed (an agent re-key) — proactive, fleet-wide, independent of any controller's channel-health check. Trust-on-first-report baseline; empty fp = unknown (never alerts). Reads the fp from the latest host-report (store.GetHostLeafFingerprints, no schema migration).expected_backup_missed/expected_dbdump_missed— backup deadline checker (daily at 05:00 Budapest)
Notifications
| Method | Path | Description |
|---|---|---|
POST |
/api/v1/notify |
Legacy notification relay (kept for backward compatibility) |
POST |
/api/v1/preferences |
Controller syncs customer notification preferences (email, enabled_events, cooldown_hours) |
Notifications are dispatched automatically when events are processed:
- Operator channel: English emails for warning/error events, 1h cooldown per customer:eventType
- Customer channel: Hungarian emails per event type, respects customer preferences and cooldown (default 6h)
- Email delivery via Resend.com API
Customer Config Retrieval
| Method | Path | Description |
|---|---|---|
GET |
/api/v1/config/{customer_id} |
Download generated controller.yaml (auth: X-Retrieval-Password header) |
Config retrieval uses a separate per-customer retrieval password (not the API key). Retrieval passwords are auto-generated as Hungarian word passphrases (e.g., alma-kerék-madár-felhő) for easy phone-based entry during disaster recovery. The Hub generates a complete controller.yaml by deep-merging controller.yaml.example (periodically fetched from the Gitea repo) with customer-specific overrides (identity, infrastructure tokens, hub API key, session secret).
Host Enrollment (Day-0, option C)
| Method | Path | Description |
|---|---|---|
POST |
/api/v1/host-enroll |
Mint-or-reuse a host credential, body {customer_id} (auth: X-Retrieval-Password header) → {host_id, api_key} |
The Day-0 first-boot handshake (documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md) enrolls a Proxmox host's agent using the customer's retrieval passphrase only — the operator-tier global key never enters the field deploy path. host-enroll mints the per-host credential on the first call and reuses it byte-for-byte on every later call (idempotent — re-running the host-bootstrap never orphans a running agent's key), backed by Store.GetHostByCustomer. Auth is checked before any mint, so a wrong passphrase never creates a host row. Behaviour: first → 201, reuse → 200, wrong/missing passphrase → 401, unknown customer → 404, missing customer_id → 400.
The global-key POST /api/v1/admin/hosts (operator/HQ pre-mint) remains as the escape hatch pending the enrollment-cutover lock-down; it and GET /config/{id} are unchanged by this endpoint.
Assets
| Method | Path | Description |
|---|---|---|
GET |
/api/v1/assets/manifest |
JSON manifest of all assets with SHA-256 checksums |
GET |
/api/v1/assets/file/{filename} |
Download a single asset file (logo, screenshot) |
Assets are stored on the Hub PVC at <dataDir>/assets/. On first run, assets are seeded from the Docker image (/usr/share/felhom/assets-seed/). The manifest includes filename, size, and SHA-256 hash for each file — controllers use this for efficient change detection.
Asset types served: {slug}-logo.svg, {slug}-logo.png, {slug}-screenshot-{N}.webp
The asset manager (internal/assets/) scans the assets directory on startup, builds an in-memory manifest, and serves files with appropriate Content-Type and cache headers. Both endpoints require Bearer token auth (global or per-customer API key).
Health
| Method | Path | Description |
|---|---|---|
GET |
/healthz |
Health check (no auth required, returns 503 if SQLite ping fails) |
Web Dashboard
Protected by bcrypt password + session cookie (7-day expiry).
Authentication & Session Model (internal/web/server.go)
- Login generates a cryptographically random 64-char hex session token stored server-side in a
map[string]*hubSession(+sync.RWMutex). The old literalhub_session=authenticatedcookie is gone. - Each session also stores a per-session CSRF token (separate 64-char hex random value).
- Cookie attributes:
SameSite=Lax,Secure(when TLS),HttpOnly, 7-dayMax-Age. RequireAuthmiddleware validates the session token withsubtle.ConstantTimeCompareand redirects to/loginon failure.CleanupSessions(ctx)goroutine runs hourly to purge expired sessions.- Login password source (v0.54.0) — the bcrypt hash checked at login comes from
effectivePasswordHash(): ahub_settings.operator_password_hashDB override (set via Configuration → Login password) wins, otherwise theauth.password_hashseed fromhub.yaml. The ConfigMap value is the break-glass reset path (blank the DB row / edit the manifest + redeploy). Changing the password (POST /configuration/password) requires the current password and leaves existing sessions valid.
CSRF Protection (internal/web/server.go)
Synchronizer-token CSRF protection on all browser POST/DELETE/PATCH operations:
- CSRF validation block runs at the top of
ServeHTTPbefore routing. - Skipped when: no session cookie present (API/Basic-Auth path); or safe methods (GET/HEAD/OPTIONS).
- Token read from
_csrfform field orX-CSRF-Tokenrequest header. - On failure: JSON
{"ok":false,"error":"CSRF token missing or invalid"}for/api/paths; HTTP 403 text otherwise. - Template delivery:
csrfToken(r)andcsrfField(r)helpers injectCSRFTokenandCSRFFieldinto every render data struct viaconfigs.go. Templates use{{.CSRFField}}in forms andcsrfHeaders()JS helper for fetch calls.
Pages
- Dashboard (
/) — Fleet overview table showing all customers with live status and event count badges (error+warning in last 24h). Config-only customers (no reports yet) appear as "PENDING" with gray badge. Blocked customers are hidden. Auto-refreshes every 60 seconds. - Customers (
/configs) — Customer management list. Shows all customers (both managed and manual), their status, controller version, and config type (MANAGED/MANUAL). Blocked customers shown grayed-out with BLOCKED badge. - Fleet App Analytics (
/apps) — Fleet-wide app telemetry overview (v0.4.0+). Shows all deployed apps across all customers with deployment count, avg/P95 memory, catalog estimate/limit accuracy indicators, and 24h error/warning badge counts. Sortable columns (deployments/memory/errors), 24h/7d/30d time period selector. - App Detail (
/apps/{name}) — Per-app drill-down page with Chart.js memory trend (avg + peak lines, catalog limit dashed line), per-customer breakdown table, and known log issues table (severity, message, occurrence count, affected customers, first/last seen). Shows suggested mem_limit from P95×1.2 rounded to 32 MB. - Unified Customer Detail (
/customers/{id}) — Single page per customer combining config management and live monitoring. Auto-refresh toggle (localStorage-persisted, enabled by default) replaces the previous hardcoded 60s meta-refresh. Since v0.47.0 the sections are organized into 8 client-side hash tabs (#tab=<name>, deep-linkable, survives auto-refresh) under a sticky summary strip (name, status, controller version, last report, containers). Panels hide only via a JS-added body class — with JS off every section renders stacked (graceful degradation). Tab map:- Overview (default) — customer info, health/issues/warnings, system metrics, storage, backup
- Applications — containers, app telemetry, received app log tails
- Setup — credentials, setup command generator, YAML preview
- Settings — controller update + version floor, geo-restriction
- Backup & DR — DR recipe panel + download
- Events — events timeline (severity filter) + report history; the tab label carries a red count badge when error events exist (last 24h)
- Notifications — prefs + recent notification log
- Host — the customer's enrolled host(s), rendered via the shared
host_detail_bodysub-template (a list by design: 1 today, N for a future HA cluster) + cross-link to/hosts/{id}
- Config Form (
/configs/new,/configs/{id}/edit) — Create/edit customer configurations with identity, infrastructure tokens, and monitoring overrides. Legacy Monitoring UUIDs section collapsed by default with deprecation notice. CF API token requires Zone DNS:Edit (ACME) and Zone WAF:Edit (geo-restriction) permissions. - Hosts (
/hosts,/hosts/{id}) — fleet list (read-only, zero buttons — pinned by test) + per-host detail (identity, vitals, guests, storage targets, log-bundle diagnostics, DR/escrow presence). The detail body is the sharedhost_detail_bodysub-template also rendered on the customer page's Host tab. - Offsite (
/offsite) — WireGuard endpoint cards + peer registry, with endpoint management since v0.47.0 (see below).
Host lifecycle — stale host removal (v0.47.0)
Enrollment mints the host's API key exactly once (Day-0 passphrase flow); host reports authenticate via that key, so deleting a host row permanently bricks its heartbeat channel. The delete flow is therefore gated:
- The danger-zone card renders only for non-online hosts (stale / down / no-report). An ONLINE host is never deletable —
POST /hosts/{id}/deletereturns 409 unconditionally; no override exists. GET /hosts/{id}/delete-impactreturns the blast radius as counts/booleans only (guests, reports, host-scoped log bundles, escrow/wg-peer/PBS-secret/recovery presence) — never a secret or blob.- The dialog requires retyping the host id; when a key escrow exists, an unchecked-by-default checkbox ("also delete the key escrow + DR bundle") must be ticked — otherwise 409 and the transaction never starts (
store.ErrHostEscrowPresent). store.DeleteHostcascades in ONE transaction: guests, host_reports, signed_jobs, host_recovery, host_pbs_secrets, log_bundle_requests/log_bundles withscope_id == host_id(customer-scoped bundles are untouched), the bound wg_peers row, host_escrow (only when acked), then the hosts row. The wgsync reconciler's 5-minute declarative push converges the endpoint after the peer row disappears.
Offsite endpoint management (v0.47.0)
/offsite lists all wg_endpoints rows as cards and can add/edit/delete them (POST /offsite/endpoints, POST /offsite/endpoints/{id}/delete). Guards: full field validation (CIDR subnet, PBS IP inside subnet, port 1–65535, pubkey non-empty, id [a-z0-9-]+) → 400 stores nothing; changing an endpoint's tunnel_subnet or deleting it is refused with 409 while any peer's /32 lies inside the (current) subnet; a server-pubkey change requires a type-to-confirm (peers converge on their next desired-state pull). The peer table shows each peer's containing endpoint. Scope guard: peer allocation, the wgsync reconciler push, and the desired-state merge still use the lowest endpoint_id only (GetWGEndpoint) — per-endpoint allocation (wg_peers.endpoint_id) is a deferred future arc.
Customer States
| State | Dashboard | Customers List | Detail Page |
|---|---|---|---|
| Active + reporting | Shown with live status | MANAGED + status badge | Full unified view |
| Active + no reports | Shown as PENDING (gray) | MANAGED + no status | Config + "waiting for report" |
| Manual (report-only) | Shown with live status | MANUAL + status badge | Reports + "Create Config" button |
| Blocked | Hidden | Shown grayed-out, BLOCKED badge | Blocked banner + Unblock button |
Customer Actions
| Action | Description |
|---|---|
| Block/Unblock | Toggle blocked status — blocked customers are hidden from dashboard and notifications are suppressed, but reports are still accepted and stored |
| Push Config | Generate YAML from Hub config and POST it to the controller's /api/config/apply endpoint (requires controller URL from reports) |
| Pull Config | Import controller's current config into Hub — fetches live YAML via GET /api/config, extracts identity and override fields, updates Hub's stored config |
| Show Diff | Compare Hub-generated config with controller's live config — shows per-key differences in a color-coded table (value-based comparison, ignores key ordering and volatile fields) |
| Create Config | Auto-create a managed config from a manual customer's report data, then redirect to edit form |
| Trigger Update | Instruct controller to self-update to the latest version |
| Delete | Remove customer config (customer reappears as manual if reports continue) |
Status Logic
- OK (green): report < 30 min old, health = ok
- WARN (yellow): 30-60 min stale or health = warn
- DOWN (red): > 60 min stale or health = fail
- DISABLED (gray): controller monitoring paused
- PENDING (gray): config exists but no reports received yet
- BLOCKED (gray): customer blocked by operator
Data Storage
SQLite with WAL mode. Tables:
| Table | Purpose |
|---|---|
reports |
Full JSON reports with denormalized fields for dashboard queries |
events |
Structured events from controllers and Hub (type, severity, message, details, source) |
customer_notifications |
Email, enabled event types, cooldown hours per customer |
notification_log |
Send/skip/fail history for notifications with channel (operator/customer) |
customer_configs |
Pre-configured customer settings, retrieval passwords, per-customer API keys, status (active/blocked) |
Retention: configurable (default 90 days), daily prune at 04:30 Budapest time.
PVC Asset Storage
App assets (logos, screenshots, branding) are stored on the PVC at <dataDir>/assets/. On every startup, the Hub compares SHA-256 checksums between the image seed (/usr/share/felhom/assets-seed/) and the PVC, updating any changed files. This means redeploying the Hub image with updated assets automatically propagates changes without PVC deletion.
A manual "Refresh Assets from Image" button is available on the Configuration page (/configuration) for triggering a re-seed + manifest rebuild on demand.
Configuration
# hub.yaml
auth:
password_hash: "" # bcrypt SEED for dashboard login (empty = no auth). Since v0.54.0 this
# is only the fallback: a hub_settings DB override set via
# Configuration → Login password wins. This value stays the break-glass
# reset path (blank the DB row / edit here + redeploy to recover a lost pw).
api:
report_api_key: "" # Bearer token for API auth
notifications:
resend_api_key: "" # Resend.com API key for email
from_email: "monitoring@felhom.eu"
operator_email: "" # Operator alert recipient
operator_enabled: true # Enable operator email notifications
retention:
max_days: 90
prune_schedule: "04:30"
alerting:
stale_threshold: "30m" # Customer considered stale after this duration
registry:
image: "gitea.dooplex.hu/admin/felhom-controller"
username: "" # Gitea registry credentials
token: ""
check_interval: "30m" # How often to check for new controller versions
template_interval: "1h" # How often to refresh controller.yaml.example
server:
listen: ":8080"
data_dir: "/data" # SQLite database location
Deployment
Runs on k3s (Kubernetes) in the felhom-system namespace:
- PVC: 1GB Longhorn volume for SQLite database + app assets
- Resources: 64Mi-256Mi memory, 50m-500m CPU
- Ingress:
hub.felhom.euwith TLS (cert-manager) - Geo-restriction: Hungary only (nginx annotation)
# Build and push (on 192.168.0.180)
cd ~/build/felhom-hub
./build.sh v0.3.8 --push
# Build script auto-syncs app assets from website/assets/ into the image
# Deploy (ArgoCD managed — update manifests/hub.yaml image tag, commit+push)
git pull && kubectl apply -f manifests/hub.yaml
# Check
kubectl logs -n felhom-system -l app=hub --tail 20
Note: kubectl set image alone does NOT persist — ArgoCD reverts it. Always update manifests/hub.yaml and apply.
The Dockerfile includes COPY assets/ /usr/share/felhom/assets-seed/ which bakes app assets into the image as a seed for the PVC. The build script copies *-logo.svg, *-logo.png, and *-screenshot-*.webp from the website repo's assets/ directory.
Background Services
| Service | Schedule | Description |
|---|---|---|
| Staleness checker | Every 60s | Detects controllers that stopped reporting. Generates node_stale (>30min), node_down (>60min), node_recovered events |
| Backup deadline checker | Daily 05:00 Budapest | Detects missing backup/db-dump events since midnight. Generates expected_backup_missed, expected_dbdump_missed events |
| Report/event prune | Daily 04:30 Budapest | Deletes reports and events older than retention period (default 90 days) |
| Registry version check | Every 30min | Checks Gitea registry for new controller image tags |
| Template refresh | Every 1h | Fetches latest controller.yaml.example from Gitea |
| Asset seeding | On startup | Compares SHA-256 checksums and updates changed assets from Docker image seed |
Internal Packages
| Package | Purpose |
|---|---|
internal/api |
REST API handler (report ingest, config, events, assets, notifications) |
internal/web |
Web dashboard (session auth, customer management, fleet overview) |
internal/assets |
PVC asset manager (manifest generation, SHA-256 checksums, file serving, image seed) |
internal/configgen |
Shared YAML config generation (deep-merge template + customer overrides) |
Dependencies
golang.org/x/crypto— bcrypt for password hashinggopkg.in/yaml.v3— YAML config parsingmodernc.org/sqlite— Pure Go SQLite (no CGo)