hub v0.23.0: host root-disk pressure monitoring + alert
New HostDiskChecker on the 60s sweep alerts the operator when a Proxmox host root filesystem crosses warn (90%) / crit (95%). Born/persistent (a disk already full at hub restart alerts on cycle 1); distinct host_disk_* event types from the guest disk_*; critical band maps to severity error (the dispatcher only routes warning/error). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
This commit is contained in:
@@ -1,5 +1,44 @@
|
||||
# Felhom Hub — Changelog
|
||||
|
||||
## v0.23.0 — host root-disk pressure monitoring + alert (2026-06-30)
|
||||
|
||||
Closes the silent-failure gap behind the felhom-pve incident: a Proxmox host root fs filling up (vzdump
|
||||
piling under `/var/lib/vz/dump`) went unnoticed because nothing alerted on the HOST root `disk_percent` the
|
||||
agent already reports. New hub-side checker on the existing 60s sweep.
|
||||
|
||||
- **`internal/monitor/host_disk.go` (NEW) — `HostDiskChecker`.** A sibling of `HostCapabilityChecker` /
|
||||
`HostLeafChecker`: reads each host's latest root-fs `disk_percent` (`store.GetHostDiskUsage`) and emits an
|
||||
operator alert on a warning (default **90%**) or critical (default **95%**) crossing. Rank-based bands
|
||||
(ok→warning→critical) so an escalation always alerts and a de-escalation/recovery re-arms silently.
|
||||
- **Born/persistent (the F2 lesson):** a disk ALREADY over threshold when the hub/checker (re)starts
|
||||
alerts on **cycle 1** — seeding leaves already-breached hosts UNSEEDED so the first `Check` emits (a
|
||||
transition-only design would stay silent forever on a persistently-full disk). The dispatcher's 1h
|
||||
operator cooldown dedups re-emits across a hub restart.
|
||||
- **Distinct event types** `host_disk_warning` / `host_disk_critical` — NOT the controller's GUEST
|
||||
`disk_warning`/`disk_critical` (the guest cgroup view), so the host and guest alerts never dedup or mask
|
||||
each other.
|
||||
- **Severity:** warning band → `warning`; **critical band → `error`** (NOT `"critical"`). The dispatcher
|
||||
only routes `warning`/`error` severities — a `"critical"` severity would be silently dropped — so the
|
||||
critical band maps to `error` (and the operator email's 🔴). (Deviation from the task's stated
|
||||
"critical → critical", made to match the live dispatcher.)
|
||||
- **Thresholds** are hub-config overridable (`alerting.host_disk_warn_percent` /
|
||||
`host_disk_crit_percent`, seed-only); an unset/invalid/misordered config falls back to 90/95
|
||||
(`normalizeDiskThresholds`) so a typo can never silence or invert the alert.
|
||||
- **`internal/store/store.go`:** `GetHostDiskUsage()` + `HostDiskRow` — latest report per host (MAX(id)),
|
||||
`disk_percent` from the denorm column + total/used bytes parsed from `report_json` (event detail). No
|
||||
schema migration.
|
||||
- **`internal/notify/templates.go`:** Hungarian customer templates for `host_disk_warning`/`_critical`
|
||||
(customer delivery still requires per-customer opt-in via enabled events; operator alert is the headline).
|
||||
- **`internal/api/handler.go`:** `host_disk_warning`/`host_disk_critical` added to `allowedEventTypes`.
|
||||
- **`cmd/hub/main.go`:** register `hostDiskChecker` on the shared 60s tick.
|
||||
- Tests: band transitions (seed/escalate/steady/recover/re-arm), severity mapping, threshold defaults, and
|
||||
the **born/persistent companion red-proof** (a seed-all/transition-only model stays silent on a
|
||||
born-breach; the real unseeded design emits). `go build/vet/test ./...` green.
|
||||
- **Follow-ups (noted, not built):** per-storage `StorageTargets` worst-fill alerting (a dedicated
|
||||
dump/backup storage filling — host root `disk_percent` already covers the observed case); and the
|
||||
provisioning-side `prune-backups` retention default so a box can't refill its own root (operational fix,
|
||||
separate from this detector).
|
||||
|
||||
## hub-config — enable operator email alerts (config-only, no image change) (2026-06-30)
|
||||
|
||||
`manifests/hub.yaml` `hub-config` ConfigMap: set `notifications.operator_email: admin@felhom.eu` +
|
||||
|
||||
@@ -47,6 +47,10 @@ type Config struct {
|
||||
} `yaml:"retention"`
|
||||
Alerting struct {
|
||||
StaleThreshold string `yaml:"stale_threshold"`
|
||||
// Host root-fs disk-pressure thresholds (percent). Empty/0/invalid → defaults 90/95
|
||||
// (normalizeDiskThresholds). Seed-only config, like stale_threshold.
|
||||
HostDiskWarnPercent float64 `yaml:"host_disk_warn_percent"`
|
||||
HostDiskCritPercent float64 `yaml:"host_disk_crit_percent"`
|
||||
} `yaml:"alerting"`
|
||||
Registry struct {
|
||||
Image string `yaml:"image"`
|
||||
@@ -322,6 +326,10 @@ func main() {
|
||||
// agent-v0.48.0: proactive fleet-wide agent-re-key detection — alert when a host's reported
|
||||
// local-API leaf fingerprint changes (independent of the controller channel-check). Same 60s sweep.
|
||||
hostLeafChecker := monitor.NewHostLeafChecker(dataStore, dispatcher.ProcessEvent, logger)
|
||||
// v0.23.0: HOST root-fs disk-pressure alert (the silent vzdump-on-root-fills class). Reads the host
|
||||
// root disk_percent the agent reports; born/persistent (a disk already full at hub restart alerts on
|
||||
// cycle 1). Distinct event types from the controller's GUEST disk_warning/disk_critical. Same 60s sweep.
|
||||
hostDiskChecker := monitor.NewHostDiskChecker(dataStore, cfg.Alerting.HostDiskWarnPercent, cfg.Alerting.HostDiskCritPercent, dispatcher.ProcessEvent, logger)
|
||||
go func() {
|
||||
ticker := time.NewTicker(60 * time.Second)
|
||||
defer ticker.Stop()
|
||||
@@ -334,6 +342,7 @@ func main() {
|
||||
hostStalenessChecker.Check()
|
||||
hostCapabilityChecker.Check()
|
||||
hostLeafChecker.Check()
|
||||
hostDiskChecker.Check()
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
@@ -1095,6 +1095,9 @@ var allowedEventTypes = map[string]bool{
|
||||
"host_stale": true,
|
||||
"host_down": true,
|
||||
"host_recovered": true,
|
||||
// Hub-generated host root-fs disk-pressure (v0.23.0) — distinct from the controller's GUEST disk_*
|
||||
"host_disk_warning": true,
|
||||
"host_disk_critical": true,
|
||||
"expected_backup_missed": true,
|
||||
"expected_dbdump_missed": true,
|
||||
// Special
|
||||
|
||||
@@ -0,0 +1,224 @@
|
||||
package monitor
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"sync"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
// HostDiskChecker raises an operator alert when a Proxmox HOST's root filesystem crosses a warning or
|
||||
// critical fill threshold — the silent-failure class observed on felhom-pve (vzdump piling under
|
||||
// /var/lib/vz/dump until PVE/sqlite/logging and even the agent's own writes start failing). It reads the
|
||||
// HOST root `disk_percent` the agent rides in its host-report (store.GetHostDiskUsage), so it is a
|
||||
// deliberate SIBLING of HostCapabilityChecker / HostLeafChecker on the same 60s sweep, with the same
|
||||
// SaveEvent+onEvent plumbing and the dispatcher's 1h operator cooldown.
|
||||
//
|
||||
// DISTINCT from the controller's GUEST-disk events (disk_warning/disk_critical): those are the guest's own
|
||||
// cgroup view; this is the HOST root fs, with its own event types (host_disk_warning/host_disk_critical)
|
||||
// so the two never dedup or mask each other.
|
||||
//
|
||||
// BORN/PERSISTENT (the F2 lesson, §7-B): a disk ALREADY over threshold when the hub/checker (re)starts
|
||||
// must alert on cycle 1 — it never observes a "crossing". This is achieved by seeding ONLY ok-band hosts
|
||||
// at init and leaving already-breached hosts UNSEEDED, so their first Check sees oldBand=="" (rank 0) and
|
||||
// emits. The dispatcher's 1h cooldown dedups the re-emit across a hub restart. A transition-only design
|
||||
// would stay silent forever on a persistently-full disk — exactly the wrong outcome.
|
||||
type HostDiskChecker struct {
|
||||
store *store.Store
|
||||
warn float64 // warning threshold (percent)
|
||||
crit float64 // critical threshold (percent)
|
||||
logger *log.Logger
|
||||
onEvent EventNotifyFunc
|
||||
|
||||
mu sync.Mutex
|
||||
states map[string]string // hostID → band (bandOK|bandWarning|bandCritical); breached unseeded at init
|
||||
customerOf map[string]string // hostID → customerID (event attribution)
|
||||
}
|
||||
|
||||
const (
|
||||
defaultHostDiskWarnPercent = 90.0
|
||||
defaultHostDiskCritPercent = 95.0
|
||||
)
|
||||
|
||||
// Disk fill bands, ordered by severity (bandRank).
|
||||
const (
|
||||
bandOK = "ok"
|
||||
bandWarning = "warning"
|
||||
bandCritical = "critical"
|
||||
)
|
||||
|
||||
// NewHostDiskChecker creates the checker with the configured thresholds (defaults 90/95 when unset or
|
||||
// invalid) and seeds state from the latest reports. NO events on init — except that already-breached
|
||||
// hosts are left UNSEEDED so their first Check emits (born/persistent).
|
||||
func NewHostDiskChecker(s *store.Store, warnPercent, critPercent float64, onEvent EventNotifyFunc, logger *log.Logger) *HostDiskChecker {
|
||||
warn, crit := normalizeDiskThresholds(warnPercent, critPercent)
|
||||
dc := &HostDiskChecker{
|
||||
store: s,
|
||||
warn: warn,
|
||||
crit: crit,
|
||||
logger: logger,
|
||||
onEvent: onEvent,
|
||||
states: make(map[string]string),
|
||||
customerOf: make(map[string]string),
|
||||
}
|
||||
rows, err := s.GetHostDiskUsage()
|
||||
if err != nil {
|
||||
logger.Printf("[WARN] Host disk checker: failed to seed states: %v", err)
|
||||
return dc
|
||||
}
|
||||
var okCount, breachedCount int
|
||||
for _, row := range rows {
|
||||
if s.IsCustomerBlocked(row.CustomerID) {
|
||||
continue
|
||||
}
|
||||
dc.customerOf[row.HostID] = row.CustomerID
|
||||
band := dc.band(row.DiskPercent)
|
||||
// F2: seed only OK hosts. A host already over threshold at (re)start is left UNSEEDED so the first
|
||||
// Check observes oldBand=="" and emits once — otherwise a box already full when the hub restarts
|
||||
// would stay silent forever. The dispatcher's 1h cooldown dedups the re-emit.
|
||||
if band != bandOK {
|
||||
breachedCount++
|
||||
continue
|
||||
}
|
||||
dc.states[row.HostID] = bandOK
|
||||
okCount++
|
||||
}
|
||||
logger.Printf("[INFO] Host disk checker initialized: warn=%.0f%% crit=%.0f%%, %d ok seeded, %d already-breached left unseeded (first Check emits)", warn, crit, okCount, breachedCount)
|
||||
return dc
|
||||
}
|
||||
|
||||
// Check evaluates all hosts and emits on each escalation to a more severe band (ok→warning, ok/warning→
|
||||
// critical), including the first-observation born-breach. De-escalation and recovery update state
|
||||
// silently (re-arming a future breach). Call on the same 60s sweep as the staleness checker.
|
||||
func (dc *HostDiskChecker) Check() {
|
||||
rows, err := dc.store.GetHostDiskUsage()
|
||||
if err != nil {
|
||||
dc.logger.Printf("[WARN] Host disk check failed: %v", err)
|
||||
return
|
||||
}
|
||||
dc.mu.Lock()
|
||||
defer dc.mu.Unlock()
|
||||
|
||||
seen := make(map[string]bool, len(rows))
|
||||
for _, row := range rows {
|
||||
seen[row.HostID] = true
|
||||
if dc.store.IsCustomerBlocked(row.CustomerID) {
|
||||
delete(dc.states, row.HostID)
|
||||
continue
|
||||
}
|
||||
dc.customerOf[row.HostID] = row.CustomerID
|
||||
|
||||
newBand := dc.band(row.DiskPercent)
|
||||
oldBand := dc.states[row.HostID] // "" (rank 0) for an unseen / breached-at-init host
|
||||
// Emit only when moving to a MORE severe band — this covers the born-breach (oldBand=="" → rank 0)
|
||||
// and any escalation, while de-escalation/recovery just re-arm (no noise, no recovery event).
|
||||
if bandRank(newBand) > bandRank(oldBand) {
|
||||
dc.emit(row, oldBand, newBand)
|
||||
}
|
||||
dc.states[row.HostID] = newBand
|
||||
}
|
||||
|
||||
for id := range dc.states {
|
||||
if !seen[id] {
|
||||
delete(dc.states, id)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// GetState returns the current band for a host ("unknown" if unseen).
|
||||
func (dc *HostDiskChecker) GetState(hostID string) string {
|
||||
dc.mu.Lock()
|
||||
defer dc.mu.Unlock()
|
||||
s := dc.states[hostID]
|
||||
if s == "" {
|
||||
return "unknown"
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// band maps a fill percentage to its severity band.
|
||||
func (dc *HostDiskChecker) band(pct float64) string {
|
||||
switch {
|
||||
case pct >= dc.crit:
|
||||
return bandCritical
|
||||
case pct >= dc.warn:
|
||||
return bandWarning
|
||||
default:
|
||||
return bandOK
|
||||
}
|
||||
}
|
||||
|
||||
// bandRank orders the bands so an escalation is a strictly increasing rank ("" / unseen = ok = 0).
|
||||
func bandRank(b string) int {
|
||||
switch b {
|
||||
case bandCritical:
|
||||
return 2
|
||||
case bandWarning:
|
||||
return 1
|
||||
default:
|
||||
return 0
|
||||
}
|
||||
}
|
||||
|
||||
func (dc *HostDiskChecker) emit(row store.HostDiskRow, oldBand, newBand string) {
|
||||
var eventType, severity, message string
|
||||
switch newBand {
|
||||
case bandCritical:
|
||||
eventType = "host_disk_critical"
|
||||
// NB: the dispatcher only routes severity "warning"/"error" — a "critical" severity would be
|
||||
// silently dropped — so the critical BAND maps to severity "error" (and the operator email's 🔴).
|
||||
severity = "error"
|
||||
message = fmt.Sprintf("Host %s: root filesystem CRITICALLY full at %.0f%% (threshold %.0f%%) — PVE/logging/agent writes may start failing; free space (e.g. old vzdump backups) immediately", row.HostID, row.DiskPercent, dc.crit)
|
||||
case bandWarning:
|
||||
eventType = "host_disk_warning"
|
||||
severity = "warning"
|
||||
message = fmt.Sprintf("Host %s: root filesystem high at %.0f%% (threshold %.0f%%) — free space (e.g. old backups) before it fills", row.HostID, row.DiskPercent, dc.warn)
|
||||
default:
|
||||
return // never emit for ok
|
||||
}
|
||||
|
||||
details, _ := json.Marshal(map[string]any{
|
||||
"host_id": row.HostID,
|
||||
"disk_percent": row.DiskPercent,
|
||||
"disk_total_bytes": row.DiskTotalBytes,
|
||||
"disk_used_bytes": row.DiskUsedBytes,
|
||||
"warn_percent": dc.warn,
|
||||
"crit_percent": dc.crit,
|
||||
})
|
||||
|
||||
dc.logger.Printf("[INFO] Host disk: %s root %.0f%% %s→%s (%s)", row.HostID, row.DiskPercent, bandLabel(oldBand), newBand, eventType)
|
||||
|
||||
if _, err := dc.store.SaveEvent(row.CustomerID, eventType, severity, message, string(details), "hub"); err != nil {
|
||||
dc.logger.Printf("[WARN] Failed to save host disk event for %s: %v", row.HostID, err)
|
||||
return
|
||||
}
|
||||
if dc.onEvent != nil {
|
||||
dc.onEvent(row.CustomerID, eventType, severity, message, string(details), "hub")
|
||||
}
|
||||
}
|
||||
|
||||
// bandLabel renders the previous band for the log line ("unknown" for an unseen/breached-at-init host).
|
||||
func bandLabel(b string) string {
|
||||
if b == "" {
|
||||
return "unknown"
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// normalizeDiskThresholds applies the 90/95 defaults and guards against an invalid/misordered config
|
||||
// (warn/crit out of (0,100), or crit ≤ warn) by falling back to the defaults — a config typo can never
|
||||
// silence the alert or invert the bands.
|
||||
func normalizeDiskThresholds(warn, crit float64) (float64, float64) {
|
||||
if warn <= 0 || warn >= 100 {
|
||||
warn = defaultHostDiskWarnPercent
|
||||
}
|
||||
if crit <= 0 || crit >= 100 {
|
||||
crit = defaultHostDiskCritPercent
|
||||
}
|
||||
if crit <= warn {
|
||||
return defaultHostDiskWarnPercent, defaultHostDiskCritPercent
|
||||
}
|
||||
return warn, crit
|
||||
}
|
||||
@@ -0,0 +1,182 @@
|
||||
package monitor
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
_ "modernc.org/sqlite"
|
||||
)
|
||||
|
||||
func newDiskStore(t *testing.T) *store.Store {
|
||||
t.Helper()
|
||||
st, err := store.New(filepath.Join(t.TempDir(), "test.db"), log.New(io.Discard, "", 0))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { st.Close() })
|
||||
st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c1", APIKey: "ck", RetrievalPassword: "p"})
|
||||
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"})
|
||||
return st
|
||||
}
|
||||
|
||||
// saveDiskReport records a host-report carrying the host root disk_percent (denorm column) + total/used.
|
||||
func saveDiskReport(t *testing.T, st *store.Store, pct float64) {
|
||||
t.Helper()
|
||||
body := fmt.Sprintf(`{"host_id":"h1","host":{"disk_total_bytes":100000000000,"disk_used_bytes":%d}}`, int64(pct*1e9))
|
||||
if err := st.SaveHostReport("h1", "c1", []byte(body), store.HostReportDenorm{DiskPercent: pct}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func quietLog() *log.Logger { return log.New(io.Discard, "", 0) }
|
||||
|
||||
// TestHostDiskChecker_Bands covers the band contract (§7-A/C, §10): seed ok (no event), ok→warning,
|
||||
// warning→critical (escalation), steady (no re-emit), recovery clears + re-arms, and the emitted type is
|
||||
// host_disk_* (distinct from the guest disk_*).
|
||||
func TestHostDiskChecker_Bands(t *testing.T) {
|
||||
st := newDiskStore(t)
|
||||
saveDiskReport(t, st, 80) // ok at init
|
||||
|
||||
var events []string
|
||||
onEvent := func(_, eventType, _, _, _, _ string) { events = append(events, eventType) }
|
||||
dc := NewHostDiskChecker(st, 90, 95, onEvent, quietLog())
|
||||
|
||||
if dc.GetState("h1") != "ok" {
|
||||
t.Fatalf("seed state = %s, want ok", dc.GetState("h1"))
|
||||
}
|
||||
if len(events) != 0 {
|
||||
t.Fatalf("seed must not emit, got %v", events)
|
||||
}
|
||||
|
||||
// 80 → nothing.
|
||||
saveDiskReport(t, st, 80)
|
||||
dc.Check()
|
||||
if len(events) != 0 {
|
||||
t.Fatalf("80%% must not alert, got %v", events)
|
||||
}
|
||||
|
||||
// ok → 91 → warning (one event).
|
||||
saveDiskReport(t, st, 91)
|
||||
dc.Check()
|
||||
if dc.GetState("h1") != bandWarning {
|
||||
t.Fatalf("state = %s, want warning", dc.GetState("h1"))
|
||||
}
|
||||
if len(events) != 1 || events[0] != "host_disk_warning" {
|
||||
t.Fatalf("want one host_disk_warning, got %v", events)
|
||||
}
|
||||
|
||||
// steady warning: no re-emit.
|
||||
saveDiskReport(t, st, 92)
|
||||
dc.Check()
|
||||
if len(events) != 1 {
|
||||
t.Fatalf("steady warning must not re-emit, got %v", events)
|
||||
}
|
||||
|
||||
// warning → 96 → critical (escalation).
|
||||
saveDiskReport(t, st, 96)
|
||||
dc.Check()
|
||||
if dc.GetState("h1") != bandCritical {
|
||||
t.Fatalf("state = %s, want critical", dc.GetState("h1"))
|
||||
}
|
||||
if len(events) != 2 || events[1] != "host_disk_critical" {
|
||||
t.Fatalf("want host_disk_critical escalation, got %v", events)
|
||||
}
|
||||
|
||||
// recovery: 80 clears + re-arms (no recovery event, state back to ok).
|
||||
saveDiskReport(t, st, 80)
|
||||
dc.Check()
|
||||
if dc.GetState("h1") != bandOK {
|
||||
t.Fatalf("state = %s, want ok after recovery", dc.GetState("h1"))
|
||||
}
|
||||
if len(events) != 2 {
|
||||
t.Fatalf("recovery must not emit an event, got %v", events)
|
||||
}
|
||||
|
||||
// re-armed: a fresh 91 breach alerts again.
|
||||
saveDiskReport(t, st, 91)
|
||||
dc.Check()
|
||||
if len(events) != 3 || events[2] != "host_disk_warning" {
|
||||
t.Fatalf("re-arm: a new breach must alert again, got %v", events)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHostDiskChecker_Severity asserts the band→severity mapping the dispatcher requires (it only routes
|
||||
// warning/error): warning band → "warning", critical band → "error" (NOT "critical", which would be dropped).
|
||||
func TestHostDiskChecker_Severity(t *testing.T) {
|
||||
st := newDiskStore(t)
|
||||
saveDiskReport(t, st, 50)
|
||||
var sev []string
|
||||
onEvent := func(_, _, severity, _, _, _ string) { sev = append(sev, severity) }
|
||||
dc := NewHostDiskChecker(st, 90, 95, onEvent, quietLog())
|
||||
|
||||
saveDiskReport(t, st, 92)
|
||||
dc.Check()
|
||||
saveDiskReport(t, st, 97)
|
||||
dc.Check()
|
||||
if len(sev) != 2 || sev[0] != "warning" || sev[1] != "error" {
|
||||
t.Fatalf("severities = %v, want [warning error] (critical band must be 'error' so the dispatcher routes it)", sev)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHostDiskChecker_BornPersistent is the F2 lesson (§7-B): a disk ALREADY over the critical threshold
|
||||
// when the checker (re)starts must alert on the FIRST Check — there is no crossing to observe.
|
||||
func TestHostDiskChecker_BornPersistent(t *testing.T) {
|
||||
st := newDiskStore(t)
|
||||
saveDiskReport(t, st, 97) // already critical at init
|
||||
|
||||
var events []string
|
||||
onEvent := func(_, eventType, _, _, _, _ string) { events = append(events, eventType) }
|
||||
dc := NewHostDiskChecker(st, 90, 95, onEvent, quietLog())
|
||||
|
||||
// Seed must have left the breached host UNSEEDED (so the first Check emits).
|
||||
if dc.GetState("h1") != "unknown" {
|
||||
t.Fatalf("an already-breached host must be left unseeded at init, state = %s", dc.GetState("h1"))
|
||||
}
|
||||
if len(events) != 0 {
|
||||
t.Fatalf("init must not emit directly, got %v", events)
|
||||
}
|
||||
|
||||
dc.Check()
|
||||
if len(events) != 1 || events[0] != "host_disk_critical" {
|
||||
t.Fatalf("born-persistent: first Check MUST emit host_disk_critical, got %v", events)
|
||||
}
|
||||
|
||||
// COMPANION RED-PROOF: a transition-only design seeds ALL hosts at init (including breached ones), so
|
||||
// the breached host's state == current band at first Check → no transition → SILENT forever. We model
|
||||
// that by pre-seeding the state to critical (as a seed-all impl would) on a FRESH checker, then Check.
|
||||
st2 := newDiskStore(t)
|
||||
saveDiskReport(t, st2, 97)
|
||||
var silent []string
|
||||
dc2 := NewHostDiskChecker(st2, 90, 95, func(_, et, _, _, _, _ string) { silent = append(silent, et) }, quietLog())
|
||||
dc2.mu.Lock()
|
||||
dc2.states["h1"] = bandCritical // the WRONG (seed-all) design's seed
|
||||
dc2.mu.Unlock()
|
||||
dc2.Check()
|
||||
if len(silent) != 0 {
|
||||
t.Fatalf("control: a seed-all (transition-only) impl should be silent on the born-breach, got %v", silent)
|
||||
}
|
||||
// → The real checker (which leaves breached hosts unseeded) emitted; the transition-only model stayed
|
||||
// silent. That gap is the bug this design fixes.
|
||||
}
|
||||
|
||||
// TestHostDiskChecker_ThresholdDefaults: an unset/invalid threshold config falls back to 90/95, and a
|
||||
// misordered (crit ≤ warn) config does too — a typo can never silence or invert the alert.
|
||||
func TestHostDiskChecker_ThresholdDefaults(t *testing.T) {
|
||||
cases := []struct{ warn, crit, wantWarn, wantCrit float64 }{
|
||||
{0, 0, 90, 95},
|
||||
{85, 0, 85, 95},
|
||||
{0, 98, 90, 98},
|
||||
{95, 90, 90, 95}, // misordered → defaults
|
||||
{-5, 200, 90, 95}, // out of range → defaults
|
||||
}
|
||||
for _, c := range cases {
|
||||
w, cr := normalizeDiskThresholds(c.warn, c.crit)
|
||||
if w != c.wantWarn || cr != c.wantCrit {
|
||||
t.Errorf("normalizeDiskThresholds(%v,%v) = (%v,%v), want (%v,%v)", c.warn, c.crit, w, cr, c.wantWarn, c.wantCrit)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -61,10 +61,14 @@ var customerMessages = map[string]string{
|
||||
"crossdrive_completed": "A másodlagos mentés sikeresen elkészült.",
|
||||
"crossdrive_failed": "A másodlagos mentés sikertelen!",
|
||||
|
||||
// Disk events
|
||||
// Disk events (GUEST — the controller's own cgroup view)
|
||||
"disk_warning": "A lemezterület 90% felett van — kérjük, szabadíts fel helyet.",
|
||||
"disk_critical": "A lemezterület kritikusan magas (95%+) — azonnali beavatkozás szükséges!",
|
||||
|
||||
// Host disk events (the Proxmox HOST root filesystem — distinct from the guest disk above)
|
||||
"host_disk_warning": "A házszerver alaprendszerének (Proxmox-gazda) gyökérlemeze 90% felett van — kérjük, szabadíts fel helyet (pl. régi biztonsági mentések).",
|
||||
"host_disk_critical": "A házszerver alaprendszerének gyökérlemeze kritikusan tele van (95%+) — azonnali beavatkozás szükséges, különben a rendszer hibázhat!",
|
||||
|
||||
// Storage events
|
||||
"storage_disconnected": "Egy meghajtó leválasztva — a mentések szünetelhetnek.",
|
||||
"storage_reconnected": "A meghajtó újra csatlakoztatva.",
|
||||
|
||||
@@ -1683,6 +1683,54 @@ type HostLeafRow struct {
|
||||
LeafFP string
|
||||
}
|
||||
|
||||
// HostDiskRow is the per-host root-filesystem usage the HostDiskChecker reads: the denormalized
|
||||
// disk_percent column (the threshold signal) plus total/used bytes parsed from report_json (event detail
|
||||
// only). DiskPercent is the HOST root fs (agent HostMetrics) — distinct from the controller's GUEST cgroup
|
||||
// disk. A NULL/absent disk_percent (a pre-disk-reporting agent) yields 0 → the checker treats it as ok.
|
||||
type HostDiskRow struct {
|
||||
HostID string
|
||||
CustomerID string
|
||||
DiskPercent float64
|
||||
DiskTotalBytes int64
|
||||
DiskUsedBytes int64
|
||||
}
|
||||
|
||||
// GetHostDiskUsage returns the latest root-fs usage per host (MAX(id) per host, mirroring
|
||||
// GetHostCapabilities / GetHostLeafFingerprints). disk_percent comes from the denorm column; total/used
|
||||
// bytes are parsed from the report body's host block for the event detail (no schema migration needed).
|
||||
func (s *Store) GetHostDiskUsage() ([]HostDiskRow, error) {
|
||||
rows, err := s.db.Query(`
|
||||
SELECT hr.host_id, hr.customer_id, hr.disk_percent, hr.report_json
|
||||
FROM host_reports hr
|
||||
JOIN (SELECT host_id, MAX(id) AS mx FROM host_reports GROUP BY host_id) latest
|
||||
ON hr.id = latest.mx`)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []HostDiskRow
|
||||
for rows.Next() {
|
||||
var r HostDiskRow
|
||||
var dp sql.NullFloat64
|
||||
var reportJSON string
|
||||
if err := rows.Scan(&r.HostID, &r.CustomerID, &dp, &reportJSON); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
r.DiskPercent = dp.Float64
|
||||
var body struct {
|
||||
Host struct {
|
||||
DiskTotalBytes int64 `json:"disk_total_bytes"`
|
||||
DiskUsedBytes int64 `json:"disk_used_bytes"`
|
||||
} `json:"host"`
|
||||
}
|
||||
_ = json.Unmarshal([]byte(reportJSON), &body) // malformed/old body → zero bytes (detail only)
|
||||
r.DiskTotalBytes = body.Host.DiskTotalBytes
|
||||
r.DiskUsedBytes = body.Host.DiskUsedBytes
|
||||
out = append(out, r)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// GetHostLeafFingerprints returns the latest reported local-API leaf fp per host (mirrors
|
||||
// GetHostCapabilities — MAX(id) per host, parsed from report_json so there is no schema migration).
|
||||
func (s *Store) GetHostLeafFingerprints() ([]HostLeafRow, error) {
|
||||
|
||||
Reference in New Issue
Block a user