hub v0.23.0: host root-disk pressure monitoring + alert

New HostDiskChecker on the 60s sweep alerts the operator when a Proxmox host root
filesystem crosses warn (90%) / crit (95%). Born/persistent (a disk already full at
hub restart alerts on cycle 1); distinct host_disk_* event types from the guest disk_*;
critical band maps to severity error (the dispatcher only routes warning/error).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
This commit is contained in:
2026-06-30 13:48:55 +02:00
parent f133355e34
commit 897997c164
7 changed files with 510 additions and 1 deletions
+39
View File
@@ -1,5 +1,44 @@
# Felhom Hub — Changelog
## v0.23.0 — host root-disk pressure monitoring + alert (2026-06-30)
Closes the silent-failure gap behind the felhom-pve incident: a Proxmox host root fs filling up (vzdump
piling under `/var/lib/vz/dump`) went unnoticed because nothing alerted on the HOST root `disk_percent` the
agent already reports. New hub-side checker on the existing 60s sweep.
- **`internal/monitor/host_disk.go` (NEW) — `HostDiskChecker`.** A sibling of `HostCapabilityChecker` /
`HostLeafChecker`: reads each host's latest root-fs `disk_percent` (`store.GetHostDiskUsage`) and emits an
operator alert on a warning (default **90%**) or critical (default **95%**) crossing. Rank-based bands
(ok→warning→critical) so an escalation always alerts and a de-escalation/recovery re-arms silently.
- **Born/persistent (the F2 lesson):** a disk ALREADY over threshold when the hub/checker (re)starts
alerts on **cycle 1** — seeding leaves already-breached hosts UNSEEDED so the first `Check` emits (a
transition-only design would stay silent forever on a persistently-full disk). The dispatcher's 1h
operator cooldown dedups re-emits across a hub restart.
- **Distinct event types** `host_disk_warning` / `host_disk_critical` — NOT the controller's GUEST
`disk_warning`/`disk_critical` (the guest cgroup view), so the host and guest alerts never dedup or mask
each other.
- **Severity:** warning band → `warning`; **critical band → `error`** (NOT `"critical"`). The dispatcher
only routes `warning`/`error` severities — a `"critical"` severity would be silently dropped — so the
critical band maps to `error` (and the operator email's 🔴). (Deviation from the task's stated
"critical → critical", made to match the live dispatcher.)
- **Thresholds** are hub-config overridable (`alerting.host_disk_warn_percent` /
`host_disk_crit_percent`, seed-only); an unset/invalid/misordered config falls back to 90/95
(`normalizeDiskThresholds`) so a typo can never silence or invert the alert.
- **`internal/store/store.go`:** `GetHostDiskUsage()` + `HostDiskRow` — latest report per host (MAX(id)),
`disk_percent` from the denorm column + total/used bytes parsed from `report_json` (event detail). No
schema migration.
- **`internal/notify/templates.go`:** Hungarian customer templates for `host_disk_warning`/`_critical`
(customer delivery still requires per-customer opt-in via enabled events; operator alert is the headline).
- **`internal/api/handler.go`:** `host_disk_warning`/`host_disk_critical` added to `allowedEventTypes`.
- **`cmd/hub/main.go`:** register `hostDiskChecker` on the shared 60s tick.
- Tests: band transitions (seed/escalate/steady/recover/re-arm), severity mapping, threshold defaults, and
the **born/persistent companion red-proof** (a seed-all/transition-only model stays silent on a
born-breach; the real unseeded design emits). `go build/vet/test ./...` green.
- **Follow-ups (noted, not built):** per-storage `StorageTargets` worst-fill alerting (a dedicated
dump/backup storage filling — host root `disk_percent` already covers the observed case); and the
provisioning-side `prune-backups` retention default so a box can't refill its own root (operational fix,
separate from this detector).
## hub-config — enable operator email alerts (config-only, no image change) (2026-06-30)
`manifests/hub.yaml` `hub-config` ConfigMap: set `notifications.operator_email: admin@felhom.eu` +