# Controller — storage, monitoring and metrics

Source of truth: felhom-controller `internal/agentapi/`, `internal/web` storage handlers,
`internal/metrics/`, `internal/monitor/`, `internal/system/dockervol.go` at v0.59.0; disk
classification/execution is the host agent's.

This document describes what the **in-guest controller** actually does with storage, host
health and metrics. The controller is **de-privileged** (slice 8C): it is Docker-only, holds
no Proxmox/disk credentials, and does not scan, format, mount or classify disks itself. Disk
topology and classification come from the host **agent**; the controller calls the agent over a
pinned local API, displays the agent's view, and keeps a small registry of the user-data drives
it cares about.

---

## 1. Storage model in the controller

There are two distinct storage views in the controller, and they must not be conflated:

### 1a. The registered user-data drive registry (`settings.StoragePath`)

The controller persists a list of user-data drives it manages, in `settings.json` as
`StoragePaths` (`internal/settings/settings.go:97`, the `StoragePath` struct). Each entry is a
mount the customer's apps deploy their large files onto:

| Field | Meaning |
|---|---|
| `Path` | the in-guest mount, e.g. `/mnt/felhom-usb` |
| `Label` | friendly Hungarian name (`Külső HDD 1TB`) |
| `IsDefault` | new apps default here |
| `Schedulable` | new apps may be deployed here |
| `Disconnected` / `Decommissioned` | lifecycle flags |
| `MigratedTo` | target path after a decommission |

`GetStoragePaths()` (`settings.go:439`) returns a defensive copy under `RWMutex`. This registry
is **only the external/user-data drives** — the agent's system disk (local-lvm), `local`
templates dir and PBS targets are **never** in it (they live host-side and are surfaced via the
agent's host-metrics view, §4). All public settings methods take the mutex; writes are atomic
(write `.tmp`, rename).

### 1b. The agent's authoritative disk topology

A "storage target" / "disk" in the operational sense is whatever the **agent** reports. The
controller learns topology by *asking the agent*, never by inspecting devices:

- `Client.Disks()` → agent `GET /disks` → `[]DiskInfo` (`internal/agentapi/client.go:227`,
  `:306`). Each `DiskInfo` carries the agent's **authoritative** `Role`
  (`system | backup | user-data`), `DataBearing`, `DataReason`, capacity, and a `DurableID`
  (`uuid:<fs-uuid>` for usb/local-dir). `DiskInfo.FSUUID()` (`client.go:252`) strips the
  `uuid:` prefix — this is the only way the de-privileged controller learns a mount key it
  cannot read off the device itself.
- `Client.HostMetrics()` → agent `GET /host/metrics` → host health + `[]StorageTarget`
  (`client.go:459`, `:492`).

Classification is the agent's; the controller **displays** it. The UI is driven from
`DiskInfo.Role`: `system`/`backup` get a lock badge and no destructive controls; `user-data` is
customer-manageable (`client.go:235-237`).

### 1c. Drive-absent gate: `userdata` MkdirAll only on a live mountpoint (v0.75.0)

The runtime drive gate `planDriveGates` (`web/intermediary.go`) stops apps when an external drive
(under `StableParentDir = /mnt/felhom-drives`) is absent and re-attaches them on return (system/local
paths excluded). **Two `MkdirAll`-into-`<drive>/userdata` sites are now consistent with that gate** so
they never write into an unmounted mountpoint (which would land app data on the guest **rootfs**,
shadowed when the drive returns):

- **Deploy belt** — `stacks.ensureUserdataMounts` skips when `HDD_PATH` is an external root
  (`!= sysDataPath`) that is **not** a live mountpoint (`Manager.isMountPoint`, defaults to
  `system.IsMountPoint`). The app is held by `planDriveGates`.
- **FileBrowser sync** — `web.syncFileBrowserMounts` skips (and does not mount) a registered path under
  `StableParentDir` that is not a live mountpoint (pure helper `web.skipFileBrowserPath`).

The system/local path is **never** gated (it is legitimately not a mountpoint, so `EnsureUserdataDir`
must always run there). This eliminated the campaign-#2 `mkdir …/userdata: permission denied` +
transient `Created` flapping during a drive-absent window.

> **Boot-time caveat (separate cause):** a `mkdir … permission denied` at daemon boot is NOT the belt —
> docker's boot-restore auto-starts drive-backed containers (`restart: unless-stopped`) before the agent
> mounts the drives; `planDriveGates` recovers them after mount convergence. Decision deferred (see the
> tests/ finding-1 diagnosis doc).

---

## 2. Disk operations are delegated to the agent

All disk execution (list/assign/eject/format) goes through `agentapi.Client` to the agent's
`/disks` endpoints. The controller's web handlers are **thin proxies**
(`internal/web/agent_disk_handlers.go`), wired behind `RequireAuth + CsrfProtect`:

| Route | Proxies to | Notes |
|---|---|---|
| `GET /api/disks` | agent `GET /disks` | list; sorted Go-side for stable order |
| `POST /api/disks/assign` | agent `POST /disks/assign` | benign mount of an existing fs |
| `POST /api/disks/eject` | agent `POST /disks/eject` | safe-unmount (data preserved) |
| `POST /api/disks/format` | agent `POST /disks/format` | data-bearing gated agent-side |

`agentClient()` (`agent_disk_handlers.go:43`) builds a pinned client from `cfg.LocalAPI`
(endpoint/token/fingerprint); it returns "agent not configured" on an unprovisioned guest.

**The data-bearing gate is enforced on the agent, not the controller.** `FormatDisk()`
(`client.go:372`) sends the caller's `device/fstype/confirmed/durable_id`; the agent inspects
the device itself, tiers it by role (its own classification — the controller's claim is
ignored), and:

- blank device → formatted;
- **user-data**, data-bearing, not confirmed → `ErrNeedsConfirmation` with the durable id to
  type-to-confirm against (a *customer* confirmation, not an operator signature);
- **system/backup**, data-bearing → `ErrFormatRefused` with a `PendingOp` carrying the exact
  offline `felhom-opsign` command (`PendingOp.OpsignCommand()`, `client.go:292`).

The controller holds **no destructive authority** — there is no force-format path. The format
handler surfaces both refusals as HTTP 409 (`agent_disk_handlers.go:192-200`); the deep gate
mechanics live on the agent side (cross-ref the agent's destructive-path / AGENT-001 docs).

### Eject and the guided init/wipe flows

`internal/web/storage_handlers.go` orchestrates the guided init/attach/wipe over the same agent
endpoints plus the local registry:

- **Init** (`runStorageInit`, `storage_handlers.go:83`): format → (confirm/refuse?) → resolve
  the *new* fs UUID by re-listing disks → benign `assign` → register in the StoragePath
  registry → `guest-attach` into this guest. On any refusal it performs no further destructive
  or mount action.
- **Wipe** (`handleStorageWipe`, `:357`): customer-confirmed wipe of a **user-data** drive only;
  server-side type-to-confirm (the typed name must equal the mount basename). A
  system/backup-protected device is refused by the agent even though the controller sends
  `confirmed:true` (`:402-406`).
- **Eject** (`handleStorageEject`, `:506`): benign unmount via the agent (data preserved) +
  deregister the StoragePath + resync FileBrowser mounts. The eject **role-gate** is implicit:
  eject is non-destructive, but the destructive wipe behind it is the agent's role-tiered gate.
  The agent's `EjectResult` returns `DependentGuests` so the UI can warn about other guests
  bound to the drive (`client.go:343`).
- Restricted to `/mnt/<name>` (validated by `mountNameRe`, `storage_handlers.go:39`); only
  `ext4`/`xfs` offered (the agent re-validates).

---

## 3. The v0.58 infra-protection prevention layer (`internal/system/dockervol.go`)

After the OS/Docker-data split, `/var/lib/docker` is a dedicated volume holding **all** images,
overlay and named volumes — both infra (controller/traefik/cloudflared/filebrowser) and customer
apps. Infra is protected by **prevention, not placement**: a reserved buffer the controller
refuses to deploy into.

- `DockerVolumePath = "/"` (`dockervol.go:16`). The controller container's root is an overlay
  whose upperdir lives on the guest's `/var/lib/docker` volume, so `statfs("/")` reports **that
  volume's** capacity/free (true with the golden's overlay2 driver; pre-split it was the rootfs —
  correct either way).
- `DockerVolumeReserveGB(totalGB)` (`dockervol.go:23`) = `max(5 GB, 10% of total)`. (10% rather
  than the Tier-2 guard's 20%: 20% of a large data volume would reserve an absurd amount.)
- `GetDockerVolumeHeadroom()` (`dockervol.go:43`) measures the volume and returns
  `DockerVolumeHeadroom{TotalGB, AvailGB, ReserveGB, BelowReserve, OK}`. `BelowReserve` is
  `AvailGB <= ReserveGB` (`:53`). **`OK=false` when stats are unreadable** — callers MUST
  fail-open (a transient measurement error must not block all deploys; the buffer is a safety
  net, not a security control).

**Deploy gate** (`internal/api/router.go:353`): `deployStack` refuses a new deploy with
**HTTP 507** + a Hungarian message when `hr.OK && hr.BelowReserve`. The fail-open is explicit —
the gate only engages when `hr.OK` is true.

**UI pre-warning** (`internal/web/handlers.go:339`): for a NEW deploy only (an existing app's
config save consumes no fresh image space), the deploy page sets `DockerBelowReserve` +
human-readable free/reserve, and `deploy.html` shows a warning banner and disables the
"Telepítés indítása" button when below reserve.

---

## 4. Host metrics (agent-sourced host health view)

The de-privileged controller sees only its own cgroup, so it cannot read host health itself.
`ServeHostMetricsAPI` (`internal/web/agent_host_metrics_handler.go:20`, behind `RequireAuth`,
read-only) proxies `GET /api/host-metrics` → agent `GET /host/metrics` and returns the host-wide
view: cpu%/mem/load/uptime/cpu-temp (`HostMetrics`, `client.go:431`) plus per-storage capacity +
SMART/thin-pool health (`StorageTarget`, `client.go:459`).

**v0.57 server-side enrichment** (`enrichHostStorageTargets`, `agent_host_metrics_handler.go:51`):
the agent enumerates storages via `pvesm` in non-deterministic order, so the
`#host-storage-bars` list reordered on every poll. The handler now:

1. Sorts `StorageTargets` stably by `storageTypeRank` (`:67`): user-data drives (usb/local-dir)
   → internal SSD (lvmthin/lvm) → `local` templates+backups → backup targets (pbs/nfs/cifs) →
   other; alphabetical by id within a tier.
2. Attaches a friendly Hungarian `Label` + one-line `Purpose` per entry
   (`storageLabelAndPurpose`, `:84`). These are **display-only** controller-side fields
   (`StorageTarget.Label/Purpose`, `client.go:476-480`) — the raw PVE storage id stays in `Name`
   and is never renamed.

The disk-overview list (`GET /api/disks`) is similarly sorted Go-side by role
(`sortDisksForView`, `agent_disk_handlers.go:87`) for a stable, user-data-first order.

**`buildStorageBars`** (`internal/web/handlers.go:50`) is the separate monitoring-page
"Tárolók kapacitása" list. It iterates the **registered user-data StoragePaths** (not the agent
view), skips decommissioned drives, reads local `system.GetDiskUsage` per path, and sorts by
`Path`. Every bar carries the same `storageBarPurpose` text (`:46`) because this list is
all-user-data by construction — the agent's system/PBS storage is not in this registry (it is on
the storage-management page via the agent host-metrics view).

---

## 5. Metrics subsystem (`internal/metrics/`)

### SQLite store (`store.go`)

`NewMetricsStore(dbPath)` opens `modernc.org/sqlite` at `metricsDBPath`, hardcoded in `main.go:189` to
`/opt/docker/felhom-controller/data/metrics.db`. On the live bootstrap-managed guest this **coincides
with the volume-backed `data_dir`** (the `felhom-controller-data` volume mounts at
`/opt/docker/felhom-controller`; `data_dir: /opt/docker/felhom-controller/data`), so metrics persist
across restarts. The path is hardcoded rather than derived from `cfg.Paths.DataDir` — a latent fragility
(would diverge if `data_dir` were reconfigured), tracked as a low-priority backlog item, not a live bug.
The store **verifies WAL mode took effect** — it
errors out if `PRAGMA journal_mode=WAL` does not return `"wal"` (`store.go:27-34`), then sets
`synchronous=NORMAL` and `busy_timeout=5000`. Two tables: `system_metrics` and
`container_metrics`, with ts indices (`:48-77`). Queries downsample into time buckets
(`QuerySystemMetrics`/`QueryContainerMetrics`, default resolution 200 points). `Prune` deletes
rows older than a cutoff.

### Collector (`collector.go`)

`MetricsCollector.Start(ctx)` runs a **60-second** loop guarded by `sync.Once`
(`collector.go:38,53`), wired in `cmd/controller/main.go:203` and stopped on shutdown. Each tick:

- **System sample**: `system.GetInfo(hddPath, cpuCollector)` → cpu%, mem, temp, loadavg, SSD +
  HDD usage (`sampleSystem`, `:81`).
- **Container sample** (`sampleContainers`, `:99`): runs `docker stats --no-stream` under a
  **cancellable 30s timeout context** derived from the loop ctx (`:100`), parses the
  tab-separated cpu/mem/net/block columns, and batch-inserts.

### Telemetry + log scanner

- `GetContainerTelemetry(since)` (`telemetry.go:20`) aggregates per-container avg/peak memory +
  avg cpu over a window from the DB, then patches in the most-recent memory per container.
- `ScanContainerLogs(names, since, logger)` (`logscanner.go:52`) scans each container's
  `docker logs --since=<m> --tail=1000` **sequentially** (to avoid load spikes), under a
  per-container 10s timeout (`:87`). It classifies lines on the first 5 words against
  error/warn keyword sets (`error|fatal|panic|crit|oom|killed|exception|traceback`;
  `warn|warning`), deduplicates by a normalized fingerprint (ANSI/timestamp stripped; UUIDs,
  long hex and 6+ digit runs collapsed), and caps each container at its 10 most-frequent issues.

These two feed the **hub report**, not a live UI: `internal/report/telemetry.go:18`
(`buildAppTelemetrySection`) collects 15-minute telemetry + a log scan for every non-protected,
deployed, running stack **plus the controller container itself** (`controllerContainerName =
"felhom-controller"`), merges per-app issues (capped at 10), and emits `[]AppTelemetry` pushed to
the hub.

---

## 6. Monitoring (`internal/monitor/healthcheck.go`)

`RunHealthCheck(cfg, cpuCollector, storagePaths, logger)` (`healthcheck.go:26`) produces a
`HealthReport{Status: ok|warn|fail, Issues, Warnings, Info}`. It is invoked at startup, on a
schedule, and from the debug endpoint (`cmd/controller/main.go:273,558`;
`internal/web/handler_debug.go:255`). Checks:

1. **SSD disk usage** — `sysInfo.DiskPercent` is `statfs("/")`, i.e. the same Docker-data volume
   the prevention layer guards; warn at `DiskWarnPercent` / crit at `DiskCritPercent`. These trip
   **above** the 10%-free reserved buffer, so the customer is warned before the deploy gate
   engages (`:47-76`).
2. **HDD usage** (when `HDDConfigured`), **memory**, **CPU**, **temperature** — each against its
   configured threshold (`:79-145`).
3. **Docker** reachable (`docker info`, `:148`).
4. **Protected containers** running — using `EffectiveProtected(cfg)` (`:252`), which drops
   `cloudflared` when no tunnel token is configured so a LAN-only node is not perpetually FAIL.
5. **Storage paths** (`checkStoragePaths`, `:279`): per registered StoragePath, warn on
   disconnected/inaccessible, warn (not fail) when not on a separate mount point, warn ≥90% /
   issue ≥95% usage. All messages Hungarian.

Status rolls up to `fail` if any issue, else `warn` if any warning.

### Removed: watchdog and pinger (slice 8C)

The old storage **watchdog** (disk disconnect/reconnect detection) and the **pinger** were
deleted when the controller was de-privileged — disk detection moved to the host agent
(`cmd/controller/main.go:435` notes the watchdog moved host-side; `monitor/` now contains only
`healthcheck.go` + tests). Healthchecks pinging is likewise retired: ping UUIDs in config are
logged as no-ops and the hub now owns monitoring (`cmd/controller/main.go:209-214`).

---

## Cross-references

- Agent disk classification, the data-bearing gate and the signed destructive path: the host
  agent's storage / AGENT-001 docs.
- Backup capture/restore and Tier-2 off-drive copies: `controller/backup-architecture.md`.
- Deploy flow and the headroom gate in context: `controller/deploy-and-stack-lifecycle.md`.