285 lines
16 KiB
Markdown
285 lines
16 KiB
Markdown
# Controller — storage, monitoring and metrics
|
|
|
|
Source of truth: felhom-controller `internal/agentapi/`, `internal/web` storage handlers,
|
|
`internal/metrics/`, `internal/monitor/`, `internal/system/dockervol.go` at v0.59.0; disk
|
|
classification/execution is the host agent's.
|
|
|
|
This document describes what the **in-guest controller** actually does with storage, host
|
|
health and metrics. The controller is **de-privileged** (slice 8C): it is Docker-only, holds
|
|
no Proxmox/disk credentials, and does not scan, format, mount or classify disks itself. Disk
|
|
topology and classification come from the host **agent**; the controller calls the agent over a
|
|
pinned local API, displays the agent's view, and keeps a small registry of the user-data drives
|
|
it cares about.
|
|
|
|
---
|
|
|
|
## 1. Storage model in the controller
|
|
|
|
There are two distinct storage views in the controller, and they must not be conflated:
|
|
|
|
### 1a. The registered user-data drive registry (`settings.StoragePath`)
|
|
|
|
The controller persists a list of user-data drives it manages, in `settings.json` as
|
|
`StoragePaths` (`internal/settings/settings.go:97`, the `StoragePath` struct). Each entry is a
|
|
mount the customer's apps deploy their large files onto:
|
|
|
|
| Field | Meaning |
|
|
|---|---|
|
|
| `Path` | the in-guest mount, e.g. `/mnt/felhom-usb` |
|
|
| `Label` | friendly Hungarian name (`Külső HDD 1TB`) |
|
|
| `IsDefault` | new apps default here |
|
|
| `Schedulable` | new apps may be deployed here |
|
|
| `Disconnected` / `Decommissioned` | lifecycle flags |
|
|
| `MigratedTo` | target path after a decommission |
|
|
|
|
`GetStoragePaths()` (`settings.go:439`) returns a defensive copy under `RWMutex`. This registry
|
|
is **only the external/user-data drives** — the agent's system disk (local-lvm), `local`
|
|
templates dir and PBS targets are **never** in it (they live host-side and are surfaced via the
|
|
agent's host-metrics view, §4). All public settings methods take the mutex; writes are atomic
|
|
(write `.tmp`, rename).
|
|
|
|
### 1b. The agent's authoritative disk topology
|
|
|
|
A "storage target" / "disk" in the operational sense is whatever the **agent** reports. The
|
|
controller learns topology by *asking the agent*, never by inspecting devices:
|
|
|
|
- `Client.Disks()` → agent `GET /disks` → `[]DiskInfo` (`internal/agentapi/client.go:227`,
|
|
`:306`). Each `DiskInfo` carries the agent's **authoritative** `Role`
|
|
(`system | backup | user-data`), `DataBearing`, `DataReason`, capacity, and a `DurableID`
|
|
(`uuid:<fs-uuid>` for usb/local-dir). `DiskInfo.FSUUID()` (`client.go:252`) strips the
|
|
`uuid:` prefix — this is the only way the de-privileged controller learns a mount key it
|
|
cannot read off the device itself.
|
|
- `Client.HostMetrics()` → agent `GET /host/metrics` → host health + `[]StorageTarget`
|
|
(`client.go:459`, `:492`).
|
|
|
|
Classification is the agent's; the controller **displays** it. The UI is driven from
|
|
`DiskInfo.Role`: `system`/`backup` get a lock badge and no destructive controls; `user-data` is
|
|
customer-manageable (`client.go:235-237`).
|
|
|
|
### 1c. Drive-absent gate: `userdata` MkdirAll only on a live mountpoint (v0.75.0)
|
|
|
|
The runtime drive gate `planDriveGates` (`web/intermediary.go`) stops apps when an external drive
|
|
(under `StableParentDir = /mnt/felhom-drives`) is absent and re-attaches them on return (system/local
|
|
paths excluded). **Two `MkdirAll`-into-`<drive>/userdata` sites are now consistent with that gate** so
|
|
they never write into an unmounted mountpoint (which would land app data on the guest **rootfs**,
|
|
shadowed when the drive returns):
|
|
|
|
- **Deploy belt** — `stacks.ensureUserdataMounts` skips when `HDD_PATH` is an external root
|
|
(`!= sysDataPath`) that is **not** a live mountpoint (`Manager.isMountPoint`, defaults to
|
|
`system.IsMountPoint`). The app is held by `planDriveGates`.
|
|
- **FileBrowser sync** — `web.syncFileBrowserMounts` skips (and does not mount) a registered path under
|
|
`StableParentDir` that is not a live mountpoint (pure helper `web.skipFileBrowserPath`).
|
|
|
|
The system/local path is **never** gated (it is legitimately not a mountpoint, so `EnsureUserdataDir`
|
|
must always run there). This eliminated the campaign-#2 `mkdir …/userdata: permission denied` +
|
|
transient `Created` flapping during a drive-absent window.
|
|
|
|
> **Boot-time caveat (separate cause):** a `mkdir … permission denied` at daemon boot is NOT the belt —
|
|
> docker's boot-restore auto-starts drive-backed containers (`restart: unless-stopped`) before the agent
|
|
> mounts the drives; `planDriveGates` recovers them after mount convergence. Decision deferred (see the
|
|
> tests/ finding-1 diagnosis doc).
|
|
|
|
---
|
|
|
|
## 2. Disk operations are delegated to the agent
|
|
|
|
All disk execution (list/assign/eject/format) goes through `agentapi.Client` to the agent's
|
|
`/disks` endpoints. The controller's web handlers are **thin proxies**
|
|
(`internal/web/agent_disk_handlers.go`), wired behind `RequireAuth + CsrfProtect`:
|
|
|
|
| Route | Proxies to | Notes |
|
|
|---|---|---|
|
|
| `GET /api/disks` | agent `GET /disks` | list; sorted Go-side for stable order |
|
|
| `POST /api/disks/assign` | agent `POST /disks/assign` | benign mount of an existing fs |
|
|
| `POST /api/disks/eject` | agent `POST /disks/eject` | safe-unmount (data preserved) |
|
|
| `POST /api/disks/format` | agent `POST /disks/format` | data-bearing gated agent-side |
|
|
|
|
`agentClient()` (`agent_disk_handlers.go:43`) builds a pinned client from `cfg.LocalAPI`
|
|
(endpoint/token/fingerprint); it returns "agent not configured" on an unprovisioned guest.
|
|
|
|
**The data-bearing gate is enforced on the agent, not the controller.** `FormatDisk()`
|
|
(`client.go:372`) sends the caller's `device/fstype/confirmed/durable_id`; the agent inspects
|
|
the device itself, tiers it by role (its own classification — the controller's claim is
|
|
ignored), and:
|
|
|
|
- blank device → formatted;
|
|
- **user-data**, data-bearing, not confirmed → `ErrNeedsConfirmation` with the durable id to
|
|
type-to-confirm against (a *customer* confirmation, not an operator signature);
|
|
- **system/backup**, data-bearing → `ErrFormatRefused` with a `PendingOp` carrying the exact
|
|
offline `felhom-opsign` command (`PendingOp.OpsignCommand()`, `client.go:292`).
|
|
|
|
The controller holds **no destructive authority** — there is no force-format path. The format
|
|
handler surfaces both refusals as HTTP 409 (`agent_disk_handlers.go:192-200`); the deep gate
|
|
mechanics live on the agent side (cross-ref the agent's destructive-path / AGENT-001 docs).
|
|
|
|
### Eject and the guided init/wipe flows
|
|
|
|
`internal/web/storage_handlers.go` orchestrates the guided init/attach/wipe over the same agent
|
|
endpoints plus the local registry:
|
|
|
|
- **Init** (`runStorageInit`, `storage_handlers.go:83`): format → (confirm/refuse?) → resolve
|
|
the *new* fs UUID by re-listing disks → benign `assign` → register in the StoragePath
|
|
registry → `guest-attach` into this guest. On any refusal it performs no further destructive
|
|
or mount action.
|
|
- **Wipe** (`handleStorageWipe`, `:357`): customer-confirmed wipe of a **user-data** drive only;
|
|
server-side type-to-confirm (the typed name must equal the mount basename). A
|
|
system/backup-protected device is refused by the agent even though the controller sends
|
|
`confirmed:true` (`:402-406`).
|
|
- **Eject** (`handleStorageEject`, `:506`): benign unmount via the agent (data preserved) +
|
|
deregister the StoragePath + resync FileBrowser mounts. The eject **role-gate** is implicit:
|
|
eject is non-destructive, but the destructive wipe behind it is the agent's role-tiered gate.
|
|
The agent's `EjectResult` returns `DependentGuests` so the UI can warn about other guests
|
|
bound to the drive (`client.go:343`).
|
|
- Restricted to `/mnt/<name>` (validated by `mountNameRe`, `storage_handlers.go:39`); only
|
|
`ext4`/`xfs` offered (the agent re-validates).
|
|
|
|
---
|
|
|
|
## 3. The v0.58 infra-protection prevention layer (`internal/system/dockervol.go`)
|
|
|
|
After the OS/Docker-data split, `/var/lib/docker` is a dedicated volume holding **all** images,
|
|
overlay and named volumes — both infra (controller/traefik/cloudflared/filebrowser) and customer
|
|
apps. Infra is protected by **prevention, not placement**: a reserved buffer the controller
|
|
refuses to deploy into.
|
|
|
|
- `DockerVolumePath = "/"` (`dockervol.go:16`). The controller container's root is an overlay
|
|
whose upperdir lives on the guest's `/var/lib/docker` volume, so `statfs("/")` reports **that
|
|
volume's** capacity/free (true with the golden's overlay2 driver; pre-split it was the rootfs —
|
|
correct either way).
|
|
- `DockerVolumeReserveGB(totalGB)` (`dockervol.go:23`) = `max(5 GB, 10% of total)`. (10% rather
|
|
than the Tier-2 guard's 20%: 20% of a large data volume would reserve an absurd amount.)
|
|
- `GetDockerVolumeHeadroom()` (`dockervol.go:43`) measures the volume and returns
|
|
`DockerVolumeHeadroom{TotalGB, AvailGB, ReserveGB, BelowReserve, OK}`. `BelowReserve` is
|
|
`AvailGB <= ReserveGB` (`:53`). **`OK=false` when stats are unreadable** — callers MUST
|
|
fail-open (a transient measurement error must not block all deploys; the buffer is a safety
|
|
net, not a security control).
|
|
|
|
**Deploy gate** (`internal/api/router.go:353`): `deployStack` refuses a new deploy with
|
|
**HTTP 507** + a Hungarian message when `hr.OK && hr.BelowReserve`. The fail-open is explicit —
|
|
the gate only engages when `hr.OK` is true.
|
|
|
|
**UI pre-warning** (`internal/web/handlers.go:339`): for a NEW deploy only (an existing app's
|
|
config save consumes no fresh image space), the deploy page sets `DockerBelowReserve` +
|
|
human-readable free/reserve, and `deploy.html` shows a warning banner and disables the
|
|
"Telepítés indítása" button when below reserve.
|
|
|
|
---
|
|
|
|
## 4. Host metrics (agent-sourced host health view)
|
|
|
|
The de-privileged controller sees only its own cgroup, so it cannot read host health itself.
|
|
`ServeHostMetricsAPI` (`internal/web/agent_host_metrics_handler.go:20`, behind `RequireAuth`,
|
|
read-only) proxies `GET /api/host-metrics` → agent `GET /host/metrics` and returns the host-wide
|
|
view: cpu%/mem/load/uptime/cpu-temp (`HostMetrics`, `client.go:431`) plus per-storage capacity +
|
|
SMART/thin-pool health (`StorageTarget`, `client.go:459`).
|
|
|
|
**v0.57 server-side enrichment** (`enrichHostStorageTargets`, `agent_host_metrics_handler.go:51`):
|
|
the agent enumerates storages via `pvesm` in non-deterministic order, so the
|
|
`#host-storage-bars` list reordered on every poll. The handler now:
|
|
|
|
1. Sorts `StorageTargets` stably by `storageTypeRank` (`:67`): user-data drives (usb/local-dir)
|
|
→ internal SSD (lvmthin/lvm) → `local` templates+backups → backup targets (pbs/nfs/cifs) →
|
|
other; alphabetical by id within a tier.
|
|
2. Attaches a friendly Hungarian `Label` + one-line `Purpose` per entry
|
|
(`storageLabelAndPurpose`, `:84`). These are **display-only** controller-side fields
|
|
(`StorageTarget.Label/Purpose`, `client.go:476-480`) — the raw PVE storage id stays in `Name`
|
|
and is never renamed.
|
|
|
|
The disk-overview list (`GET /api/disks`) is similarly sorted Go-side by role
|
|
(`sortDisksForView`, `agent_disk_handlers.go:87`) for a stable, user-data-first order.
|
|
|
|
**`buildStorageBars`** (`internal/web/handlers.go:50`) is the separate monitoring-page
|
|
"Tárolók kapacitása" list. It iterates the **registered user-data StoragePaths** (not the agent
|
|
view), skips decommissioned drives, reads local `system.GetDiskUsage` per path, and sorts by
|
|
`Path`. Every bar carries the same `storageBarPurpose` text (`:46`) because this list is
|
|
all-user-data by construction — the agent's system/PBS storage is not in this registry (it is on
|
|
the storage-management page via the agent host-metrics view).
|
|
|
|
---
|
|
|
|
## 5. Metrics subsystem (`internal/metrics/`)
|
|
|
|
### SQLite store (`store.go`)
|
|
|
|
`NewMetricsStore(dbPath)` opens `modernc.org/sqlite` at `metricsDBPath`, hardcoded in `main.go:189` to
|
|
`/opt/docker/felhom-controller/data/metrics.db`. On the live bootstrap-managed guest this **coincides
|
|
with the volume-backed `data_dir`** (the `felhom-controller-data` volume mounts at
|
|
`/opt/docker/felhom-controller`; `data_dir: /opt/docker/felhom-controller/data`), so metrics persist
|
|
across restarts. The path is hardcoded rather than derived from `cfg.Paths.DataDir` — a latent fragility
|
|
(would diverge if `data_dir` were reconfigured), tracked as a low-priority backlog item, not a live bug.
|
|
The store **verifies WAL mode took effect** — it
|
|
errors out if `PRAGMA journal_mode=WAL` does not return `"wal"` (`store.go:27-34`), then sets
|
|
`synchronous=NORMAL` and `busy_timeout=5000`. Two tables: `system_metrics` and
|
|
`container_metrics`, with ts indices (`:48-77`). Queries downsample into time buckets
|
|
(`QuerySystemMetrics`/`QueryContainerMetrics`, default resolution 200 points). `Prune` deletes
|
|
rows older than a cutoff.
|
|
|
|
### Collector (`collector.go`)
|
|
|
|
`MetricsCollector.Start(ctx)` runs a **60-second** loop guarded by `sync.Once`
|
|
(`collector.go:38,53`), wired in `cmd/controller/main.go:203` and stopped on shutdown. Each tick:
|
|
|
|
- **System sample**: `system.GetInfo(hddPath, cpuCollector)` → cpu%, mem, temp, loadavg, SSD +
|
|
HDD usage (`sampleSystem`, `:81`).
|
|
- **Container sample** (`sampleContainers`, `:99`): runs `docker stats --no-stream` under a
|
|
**cancellable 30s timeout context** derived from the loop ctx (`:100`), parses the
|
|
tab-separated cpu/mem/net/block columns, and batch-inserts.
|
|
|
|
### Telemetry + log scanner
|
|
|
|
- `GetContainerTelemetry(since)` (`telemetry.go:20`) aggregates per-container avg/peak memory +
|
|
avg cpu over a window from the DB, then patches in the most-recent memory per container.
|
|
- `ScanContainerLogs(names, since, logger)` (`logscanner.go:52`) scans each container's
|
|
`docker logs --since=<m> --tail=1000` **sequentially** (to avoid load spikes), under a
|
|
per-container 10s timeout (`:87`). It classifies lines on the first 5 words against
|
|
error/warn keyword sets (`error|fatal|panic|crit|oom|killed|exception|traceback`;
|
|
`warn|warning`), deduplicates by a normalized fingerprint (ANSI/timestamp stripped; UUIDs,
|
|
long hex and 6+ digit runs collapsed), and caps each container at its 10 most-frequent issues.
|
|
|
|
These two feed the **hub report**, not a live UI: `internal/report/telemetry.go:18`
|
|
(`buildAppTelemetrySection`) collects 15-minute telemetry + a log scan for every non-protected,
|
|
deployed, running stack **plus the controller container itself** (`controllerContainerName =
|
|
"felhom-controller"`), merges per-app issues (capped at 10), and emits `[]AppTelemetry` pushed to
|
|
the hub.
|
|
|
|
---
|
|
|
|
## 6. Monitoring (`internal/monitor/healthcheck.go`)
|
|
|
|
`RunHealthCheck(cfg, cpuCollector, storagePaths, logger)` (`healthcheck.go:26`) produces a
|
|
`HealthReport{Status: ok|warn|fail, Issues, Warnings, Info}`. It is invoked at startup, on a
|
|
schedule, and from the debug endpoint (`cmd/controller/main.go:273,558`;
|
|
`internal/web/handler_debug.go:255`). Checks:
|
|
|
|
1. **SSD disk usage** — `sysInfo.DiskPercent` is `statfs("/")`, i.e. the same Docker-data volume
|
|
the prevention layer guards; warn at `DiskWarnPercent` / crit at `DiskCritPercent`. These trip
|
|
**above** the 10%-free reserved buffer, so the customer is warned before the deploy gate
|
|
engages (`:47-76`).
|
|
2. **HDD usage** (when `HDDConfigured`), **memory**, **CPU**, **temperature** — each against its
|
|
configured threshold (`:79-145`).
|
|
3. **Docker** reachable (`docker info`, `:148`).
|
|
4. **Protected containers** running — using `EffectiveProtected(cfg)` (`:252`), which drops
|
|
`cloudflared` when no tunnel token is configured so a LAN-only node is not perpetually FAIL.
|
|
5. **Storage paths** (`checkStoragePaths`, `:279`): per registered StoragePath, warn on
|
|
disconnected/inaccessible, warn (not fail) when not on a separate mount point, warn ≥90% /
|
|
issue ≥95% usage. All messages Hungarian.
|
|
|
|
Status rolls up to `fail` if any issue, else `warn` if any warning.
|
|
|
|
### Removed: watchdog and pinger (slice 8C)
|
|
|
|
The old storage **watchdog** (disk disconnect/reconnect detection) and the **pinger** were
|
|
deleted when the controller was de-privileged — disk detection moved to the host agent
|
|
(`cmd/controller/main.go:435` notes the watchdog moved host-side; `monitor/` now contains only
|
|
`healthcheck.go` + tests). Healthchecks pinging is likewise retired: ping UUIDs in config are
|
|
logged as no-ops and the hub now owns monitoring (`cmd/controller/main.go:209-214`).
|
|
|
|
---
|
|
|
|
## Cross-references
|
|
|
|
- Agent disk classification, the data-bearing gate and the signed destructive path: the host
|
|
agent's storage / AGENT-001 docs.
|
|
- Backup capture/restore and Tier-2 off-drive copies: `controller/backup-architecture.md`.
|
|
- Deploy flow and the headroom gate in context: `controller/deploy-and-stack-lifecycle.md`.
|