felhom.eu/documentation/controller/storage-monitoring-metrics.md

Controller — storage, monitoring and metrics

Source of truth: felhom-controller internal/agentapi/, internal/web storage handlers, internal/metrics/, internal/monitor/, internal/system/dockervol.go at v0.59.0; disk classification/execution is the host agent's.

This document describes what the in-guest controller actually does with storage, host health and metrics. The controller is de-privileged (slice 8C): it is Docker-only, holds no Proxmox/disk credentials, and does not scan, format, mount or classify disks itself. Disk topology and classification come from the host agent; the controller calls the agent over a pinned local API, displays the agent's view, and keeps a small registry of the user-data drives it cares about.

---

1. Storage model in the controller

There are two distinct storage views in the controller, and they must not be conflated:

1a. The registered user-data drive registry (settings.StoragePath)

The controller persists a list of user-data drives it manages, in settings.json as StoragePaths (internal/settings/settings.go:97, the StoragePath struct). Each entry is a mount the customer's apps deploy their large files onto:

Field	Meaning
Path	the in-guest mount, e.g. /mnt/felhom-usb
Label	friendly Hungarian name (Külső HDD 1TB)
IsDefault	new apps default here
Schedulable	new apps may be deployed here
Disconnected / Decommissioned	lifecycle flags
MigratedTo	target path after a decommission

GetStoragePaths() (settings.go:439) returns a defensive copy under RWMutex. This registry is only the external/user-data drives — the agent's system disk (local-lvm), local templates dir and PBS targets are never in it (they live host-side and are surfaced via the agent's host-metrics view, §4). All public settings methods take the mutex; writes are atomic (write .tmp, rename).

1b. The agent's authoritative disk topology

A "storage target" / "disk" in the operational sense is whatever the agent reports. The controller learns topology by asking the agent, never by inspecting devices:

• Client.Disks() → agent GET /disks → []DiskInfo (internal/agentapi/client.go:227, :306). Each DiskInfo carries the agent's authoritative Role (system | backup | user-data), DataBearing, DataReason, capacity, and a DurableID (uuid:<fs-uuid> for usb/local-dir). DiskInfo.FSUUID() (client.go:252) strips the uuid: prefix — this is the only way the de-privileged controller learns a mount key it cannot read off the device itself.
• Client.HostMetrics() → agent GET /host/metrics → host health + []StorageTarget (client.go:459, :492).

Classification is the agent's; the controller displays it. The UI is driven from DiskInfo.Role: system/backup get a lock badge and no destructive controls; user-data is customer-manageable (client.go:235-237).

1c. Drive-absent gate: userdata MkdirAll only on a live mountpoint (v0.75.0)

The runtime drive gate planDriveGates (web/intermediary.go) stops apps when an external drive (under StableParentDir = /mnt/felhom-drives) is absent and re-attaches them on return (system/local paths excluded). Two MkdirAll-into-<drive>/userdata sites are now consistent with that gate so they never write into an unmounted mountpoint (which would land app data on the guest rootfs, shadowed when the drive returns):

• Deploy belt — stacks.ensureUserdataMounts skips when HDD_PATH is an external root (!= sysDataPath) that is not a live mountpoint (Manager.isMountPoint, defaults to system.IsMountPoint). The app is held by planDriveGates.
• FileBrowser sync — web.syncFileBrowserMounts skips (and does not mount) a registered path under StableParentDir that is not a live mountpoint (pure helper web.skipFileBrowserPath).

The system/local path is never gated (it is legitimately not a mountpoint, so EnsureUserdataDir must always run there). This eliminated the campaign-#2 mkdir …/userdata: permission denied + transient Created flapping during a drive-absent window.

Boot-time caveat (separate cause): a mkdir … permission denied at daemon boot is NOT the belt — docker's boot-restore auto-starts drive-backed containers (restart: unless-stopped) before the agent mounts the drives; planDriveGates recovers them after mount convergence. Decision deferred (see the tests/ finding-1 diagnosis doc).

---

2. Disk operations are delegated to the agent

All disk execution (list/assign/eject/format) goes through agentapi.Client to the agent's /disks endpoints. The controller's web handlers are thin proxies (internal/web/agent_disk_handlers.go), wired behind RequireAuth + CsrfProtect:

Route	Proxies to	Notes
GET /api/disks	agent GET /disks	list; sorted Go-side for stable order
POST /api/disks/assign	agent POST /disks/assign	benign mount of an existing fs
POST /api/disks/eject	agent POST /disks/eject	safe-unmount (data preserved)
POST /api/disks/format	agent POST /disks/format	data-bearing gated agent-side

agentClient() (agent_disk_handlers.go:43) builds a pinned client from cfg.LocalAPI (endpoint/token/fingerprint); it returns "agent not configured" on an unprovisioned guest.

The data-bearing gate is enforced on the agent, not the controller. FormatDisk() (client.go:372) sends the caller's device/fstype/confirmed/durable_id; the agent inspects the device itself, tiers it by role (its own classification — the controller's claim is ignored), and:

• blank device → formatted;
• user-data, data-bearing, not confirmed → ErrNeedsConfirmation with the durable id to type-to-confirm against (a customer confirmation, not an operator signature);
• system/backup, data-bearing → ErrFormatRefused with a PendingOp carrying the exact offline felhom-opsign command (PendingOp.OpsignCommand(), client.go:292).

The controller holds no destructive authority — there is no force-format path. The format handler surfaces both refusals as HTTP 409 (agent_disk_handlers.go:192-200); the deep gate mechanics live on the agent side (cross-ref the agent's destructive-path / AGENT-001 docs).

Eject and the guided init/wipe flows

internal/web/storage_handlers.go orchestrates the guided init/attach/wipe over the same agent endpoints plus the local registry:

• Init (runStorageInit, storage_handlers.go:83): format → (confirm/refuse?) → resolve the new fs UUID by re-listing disks → benign assign → register in the StoragePath registry → guest-attach into this guest. On any refusal it performs no further destructive or mount action.
• Wipe (handleStorageWipe, :357): customer-confirmed wipe of a user-data drive only; server-side type-to-confirm (the typed name must equal the mount basename). A system/backup-protected device is refused by the agent even though the controller sends confirmed:true (:402-406).
• Eject (handleStorageEject, :506): benign unmount via the agent (data preserved) + deregister the StoragePath + resync FileBrowser mounts. The eject role-gate is implicit: eject is non-destructive, but the destructive wipe behind it is the agent's role-tiered gate. The agent's EjectResult returns DependentGuests so the UI can warn about other guests bound to the drive (client.go:343).
• Restricted to /mnt/<name> (validated by mountNameRe, storage_handlers.go:39); only ext4/xfs offered (the agent re-validates).

---

3. The v0.58 infra-protection prevention layer (internal/system/dockervol.go)

After the OS/Docker-data split, /var/lib/docker is a dedicated volume holding all images, overlay and named volumes — both infra (controller/traefik/cloudflared/filebrowser) and customer apps. Infra is protected by prevention, not placement: a reserved buffer the controller refuses to deploy into.

• DockerVolumePath = "/" (dockervol.go:16). The controller container's root is an overlay whose upperdir lives on the guest's /var/lib/docker volume, so statfs("/") reports that volume's capacity/free (true with the golden's overlay2 driver; pre-split it was the rootfs — correct either way).
• DockerVolumeReserveGB(totalGB) (dockervol.go:23) = max(5 GB, 10% of total). (10% rather than the Tier-2 guard's 20%: 20% of a large data volume would reserve an absurd amount.)
• GetDockerVolumeHeadroom() (dockervol.go:43) measures the volume and returns DockerVolumeHeadroom{TotalGB, AvailGB, ReserveGB, BelowReserve, OK}. BelowReserve is AvailGB <= ReserveGB (:53). OK=false when stats are unreadable — callers MUST fail-open (a transient measurement error must not block all deploys; the buffer is a safety net, not a security control).

Deploy gate (internal/api/router.go:353): deployStack refuses a new deploy with HTTP 507 + a Hungarian message when hr.OK && hr.BelowReserve. The fail-open is explicit — the gate only engages when hr.OK is true.

UI pre-warning (internal/web/handlers.go:339): for a NEW deploy only (an existing app's config save consumes no fresh image space), the deploy page sets DockerBelowReserve + human-readable free/reserve, and deploy.html shows a warning banner and disables the "Telepítés indítása" button when below reserve.

---

4. Host metrics (agent-sourced host health view)

The de-privileged controller sees only its own cgroup, so it cannot read host health itself. ServeHostMetricsAPI (internal/web/agent_host_metrics_handler.go:20, behind RequireAuth, read-only) proxies GET /api/host-metrics → agent GET /host/metrics and returns the host-wide view: cpu%/mem/load/uptime/cpu-temp (HostMetrics, client.go:431) plus per-storage capacity + SMART/thin-pool health (StorageTarget, client.go:459).

v0.57 server-side enrichment (enrichHostStorageTargets, agent_host_metrics_handler.go:51): the agent enumerates storages via pvesm in non-deterministic order, so the #host-storage-bars list reordered on every poll. The handler now:

1. Sorts StorageTargets stably by storageTypeRank (:67): user-data drives (usb/local-dir) → internal SSD (lvmthin/lvm) → local templates+backups → backup targets (pbs/nfs/cifs) → other; alphabetical by id within a tier.
2. Attaches a friendly Hungarian Label + one-line Purpose per entry (storageLabelAndPurpose, :84). These are display-only controller-side fields (StorageTarget.Label/Purpose, client.go:476-480) — the raw PVE storage id stays in Name and is never renamed.

The disk-overview list (GET /api/disks) is similarly sorted Go-side by role (sortDisksForView, agent_disk_handlers.go:87) for a stable, user-data-first order.

buildStorageBars (internal/web/handlers.go:50) is the separate monitoring-page "Tárolók kapacitása" list. It iterates the registered user-data StoragePaths (not the agent view), skips decommissioned drives, reads local system.GetDiskUsage per path, and sorts by Path. Every bar carries the same storageBarPurpose text (:46) because this list is all-user-data by construction — the agent's system/PBS storage is not in this registry (it is on the storage-management page via the agent host-metrics view).

---

5. Metrics subsystem (internal/metrics/)

SQLite store (store.go)

NewMetricsStore(dbPath) opens modernc.org/sqlite at metricsDBPath, hardcoded in main.go:189 to /opt/docker/felhom-controller/data/metrics.db. On the live bootstrap-managed guest this coincides with the volume-backed data_dir (the felhom-controller-data volume mounts at /opt/docker/felhom-controller; data_dir: /opt/docker/felhom-controller/data), so metrics persist across restarts. The path is hardcoded rather than derived from cfg.Paths.DataDir — a latent fragility (would diverge if data_dir were reconfigured), tracked as a low-priority backlog item, not a live bug. The store verifies WAL mode took effect — it errors out if PRAGMA journal_mode=WAL does not return "wal" (store.go:27-34), then sets synchronous=NORMAL and busy_timeout=5000. Two tables: system_metrics and container_metrics, with ts indices (:48-77). Queries downsample into time buckets (QuerySystemMetrics/QueryContainerMetrics, default resolution 200 points). Prune deletes rows older than a cutoff.

Collector (collector.go)

MetricsCollector.Start(ctx) runs a 60-second loop guarded by sync.Once (collector.go:38,53), wired in cmd/controller/main.go:203 and stopped on shutdown. Each tick:

• System sample: system.GetInfo(hddPath, cpuCollector) → cpu%, mem, temp, loadavg, SSD + HDD usage (sampleSystem, :81).
• Container sample (sampleContainers, :99): runs docker stats --no-stream under a cancellable 30s timeout context derived from the loop ctx (:100), parses the tab-separated cpu/mem/net/block columns, and batch-inserts.

Telemetry + log scanner

• GetContainerTelemetry(since) (telemetry.go:20) aggregates per-container avg/peak memory + avg cpu over a window from the DB, then patches in the most-recent memory per container.
• ScanContainerLogs(names, since, logger) (logscanner.go:52) scans each container's docker logs --since=<m> --tail=1000 sequentially (to avoid load spikes), under a per-container 10s timeout (:87). It classifies lines on the first 5 words against error/warn keyword sets (error|fatal|panic|crit|oom|killed|exception|traceback; warn|warning), deduplicates by a normalized fingerprint (ANSI/timestamp stripped; UUIDs, long hex and 6+ digit runs collapsed), and caps each container at its 10 most-frequent issues.

These two feed the hub report, not a live UI: internal/report/telemetry.go:18 (buildAppTelemetrySection) collects 15-minute telemetry + a log scan for every non-protected, deployed, running stack plus the controller container itself (controllerContainerName = "felhom-controller"), merges per-app issues (capped at 10), and emits []AppTelemetry pushed to the hub.

---

6. Monitoring (internal/monitor/healthcheck.go)

RunHealthCheck(cfg, cpuCollector, storagePaths, logger) (healthcheck.go:26) produces a HealthReport{Status: ok|warn|fail, Issues, Warnings, Info}. It is invoked at startup, on a schedule, and from the debug endpoint (cmd/controller/main.go:273,558; internal/web/handler_debug.go:255). Checks:

1. SSD disk usage — sysInfo.DiskPercent is statfs("/"), i.e. the same Docker-data volume the prevention layer guards; warn at DiskWarnPercent / crit at DiskCritPercent. These trip above the 10%-free reserved buffer, so the customer is warned before the deploy gate engages (:47-76).
2. HDD usage (when HDDConfigured), memory, CPU, temperature — each against its configured threshold (:79-145).
3. Docker reachable (docker info, :148).
4. Protected containers running — using EffectiveProtected(cfg) (:252), which drops cloudflared when no tunnel token is configured so a LAN-only node is not perpetually FAIL.
5. Storage paths (checkStoragePaths, :279): per registered StoragePath, warn on disconnected/inaccessible, warn (not fail) when not on a separate mount point, warn ≥90% / issue ≥95% usage. All messages Hungarian.

Status rolls up to fail if any issue, else warn if any warning.

Removed: watchdog and pinger (slice 8C)

The old storage watchdog (disk disconnect/reconnect detection) and the pinger were deleted when the controller was de-privileged — disk detection moved to the host agent (cmd/controller/main.go:435 notes the watchdog moved host-side; monitor/ now contains only healthcheck.go + tests). Healthchecks pinging is likewise retired: ping UUIDs in config are logged as no-ops and the hub now owns monitoring (cmd/controller/main.go:209-214).

---

Cross-references

• Agent disk classification, the data-bearing gate and the signed destructive path: the host agent's storage / AGENT-001 docs.
• Backup capture/restore and Tier-2 off-drive copies: controller/backup-architecture.md.
• Deploy flow and the headroom gate in context: controller/deploy-and-stack-lifecycle.md.