hub v0.23.0: host root-disk pressure monitoring + alert
New HostDiskChecker on the 60s sweep alerts the operator when a Proxmox host root filesystem crosses warn (90%) / crit (95%). Born/persistent (a disk already full at hub restart alerts on cycle 1); distinct host_disk_* event types from the guest disk_*; critical band maps to severity error (the dispatcher only routes warning/error). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
This commit is contained in:
@@ -1,5 +1,44 @@
|
||||
# Felhom Hub — Changelog
|
||||
|
||||
## v0.23.0 — host root-disk pressure monitoring + alert (2026-06-30)
|
||||
|
||||
Closes the silent-failure gap behind the felhom-pve incident: a Proxmox host root fs filling up (vzdump
|
||||
piling under `/var/lib/vz/dump`) went unnoticed because nothing alerted on the HOST root `disk_percent` the
|
||||
agent already reports. New hub-side checker on the existing 60s sweep.
|
||||
|
||||
- **`internal/monitor/host_disk.go` (NEW) — `HostDiskChecker`.** A sibling of `HostCapabilityChecker` /
|
||||
`HostLeafChecker`: reads each host's latest root-fs `disk_percent` (`store.GetHostDiskUsage`) and emits an
|
||||
operator alert on a warning (default **90%**) or critical (default **95%**) crossing. Rank-based bands
|
||||
(ok→warning→critical) so an escalation always alerts and a de-escalation/recovery re-arms silently.
|
||||
- **Born/persistent (the F2 lesson):** a disk ALREADY over threshold when the hub/checker (re)starts
|
||||
alerts on **cycle 1** — seeding leaves already-breached hosts UNSEEDED so the first `Check` emits (a
|
||||
transition-only design would stay silent forever on a persistently-full disk). The dispatcher's 1h
|
||||
operator cooldown dedups re-emits across a hub restart.
|
||||
- **Distinct event types** `host_disk_warning` / `host_disk_critical` — NOT the controller's GUEST
|
||||
`disk_warning`/`disk_critical` (the guest cgroup view), so the host and guest alerts never dedup or mask
|
||||
each other.
|
||||
- **Severity:** warning band → `warning`; **critical band → `error`** (NOT `"critical"`). The dispatcher
|
||||
only routes `warning`/`error` severities — a `"critical"` severity would be silently dropped — so the
|
||||
critical band maps to `error` (and the operator email's 🔴). (Deviation from the task's stated
|
||||
"critical → critical", made to match the live dispatcher.)
|
||||
- **Thresholds** are hub-config overridable (`alerting.host_disk_warn_percent` /
|
||||
`host_disk_crit_percent`, seed-only); an unset/invalid/misordered config falls back to 90/95
|
||||
(`normalizeDiskThresholds`) so a typo can never silence or invert the alert.
|
||||
- **`internal/store/store.go`:** `GetHostDiskUsage()` + `HostDiskRow` — latest report per host (MAX(id)),
|
||||
`disk_percent` from the denorm column + total/used bytes parsed from `report_json` (event detail). No
|
||||
schema migration.
|
||||
- **`internal/notify/templates.go`:** Hungarian customer templates for `host_disk_warning`/`_critical`
|
||||
(customer delivery still requires per-customer opt-in via enabled events; operator alert is the headline).
|
||||
- **`internal/api/handler.go`:** `host_disk_warning`/`host_disk_critical` added to `allowedEventTypes`.
|
||||
- **`cmd/hub/main.go`:** register `hostDiskChecker` on the shared 60s tick.
|
||||
- Tests: band transitions (seed/escalate/steady/recover/re-arm), severity mapping, threshold defaults, and
|
||||
the **born/persistent companion red-proof** (a seed-all/transition-only model stays silent on a
|
||||
born-breach; the real unseeded design emits). `go build/vet/test ./...` green.
|
||||
- **Follow-ups (noted, not built):** per-storage `StorageTargets` worst-fill alerting (a dedicated
|
||||
dump/backup storage filling — host root `disk_percent` already covers the observed case); and the
|
||||
provisioning-side `prune-backups` retention default so a box can't refill its own root (operational fix,
|
||||
separate from this detector).
|
||||
|
||||
## hub-config — enable operator email alerts (config-only, no image change) (2026-06-30)
|
||||
|
||||
`manifests/hub.yaml` `hub-config` ConfigMap: set `notifications.operator_email: admin@felhom.eu` +
|
||||
|
||||
Reference in New Issue
Block a user