admin/felhom-controller
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(disk-health): the alert that never sent — severity, a real Hiba level, and a memory that survives a restart

Browse Source 
Three defects made the disk-health feature silent in exactly the case it
exists for. Evidence: felhom.eu documentation/audits/DIAG-smart-passed-trap-2026-08-14.md

1. SEVERITY (the one that changes whether anything arrives at all).
   NotifyDiskHealthDegraded emitted severity "warn", which is NOT in the
   hub's accepted set {info,warning,error,critical}. The hub coerced it to
   "info" (hub/internal/api/handler.go) and severityNotifies dropped it
   (hub/internal/notify/dispatcher.go), so every Figyelmeztetes-level disk
   alert was filed as an informational notice and emailed to NOBODY, on the
   customer and the operator leg alike. Now "warning". DiskAlertKind.Severity()
   is exported so the contract is checkable from any package.

2. NO LEVEL ABOVE "worth an eye". smart_status.passed CANNOT fail on
   unreadable sectors (attrs 187/197/198 all carry thresh 0 and a normalized
   value floors at 1), so Hiba was unreachable for this whole fault class.
   DiskVerdictFor now takes a DiskPrior and implements a 14-row top-down
   ladder: sustained unreadable sectors, a count too large to be a blip (64),
   unreadable+remapping together, overheating, NVMe critical flag or spent
   endurance all reach Hiba. No fourth label — predicted failure is "Hiba".

3. IT SPOKE ONCE, AND FORGOT ON RESTART. The baseline was in-memory, so a box
   that rebooted while a disk was failing never alerted again; and between 8
   and 352 sectors nothing was emitted at all. State is now persisted
   (disk-health-state.json, atomic tmp+rename), the decision compares against
   the last ALERTED verdict (collapsing flaps to one alert while letting a
   genuine escalation fire immediately), and a disk already at Hiba re-alerts
   once it has BOTH doubled its count and waited out a 24h cooldown.

The card replays the same prior the check used (diskRecord.PriorSawUncorrectable)
so the chip and the email cannot disagree — the property the shared verdict
function exists to guarantee, now pinned rather than asserted.

Tests: 12 scenario groups A-L. Group L builds the Server through web.NewServer,
the same call main.go makes, over a real file.
This commit is contained in:
authentik Default Admin
2026-08-14 08:10:59 +02:00
parent 3e3ee94b7b
commit bb50e1293c
9 changed files with 1534 additions and 228 deletions
Download Patch File Download Diff File Expand all files Collapse all files
controller/internal/web/disk_health.go
+112 -48
View File
@@ -7,13 +7,16 @@ import (
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/agentapi"
"gitea.dooplex.hu/admin/felhom-controller/internal/notify"
)
// Disk-health card + 6-hourly degradation check (v0.169.0). The card and the check share ONE data
// path (the 60s-TTL-cached /disks call) and ONE verdict function (agentapi.DiskVerdictFor), so the
// chip a customer sees and the alert they receive can never disagree. No new smartctl load — the
// agent (v0.94.0) serializes its already-computed SMART; this consumes it and feature-detects
// (nil Smart → "Nincs adat", never alarms). No global banner (CONTEXT ruling) — card + email only.
// Disk-health card + periodic degradation check (v0.169.0; ladder + persistence v0.215.0). The card
// and the check share ONE data path (the 60s-TTL-cached /disks call) and ONE verdict function
// (agentapi.DiskVerdictFor), so the chip a customer sees and the alert they receive can never
// disagree. That shared-function property is load-bearing: BOTH callers must pass the SAME prior, or
// the chip could read Hiba while the email says Figyelmeztetés. No new smartctl load — the agent
// (v0.94.0) serializes its already-computed SMART; this consumes it and feature-detects (nil Smart →
// "Nincs adat", never alarms). No global banner (CONTEXT ruling) — card + email only.
const diskCacheTTL = 60 * time.Second
@@ -23,10 +26,23 @@ type diskHealthState struct {
cacheResp agentapi.DisksResponse
cacheErr error
cacheSet bool
// baseline is the last-seen verdict per disk (in-memory only). UNKNOWN is never recorded. Lost on
// restart → the next check re-baselines silently (accepted; see CONTEXT).
baseline map[string]agentapi.DiskVerdict
baselined bool
// records is the PERSISTED per-disk observation + alert history (v0.215.0), keyed by diskKey.
// It replaced the in-memory-only baseline, whose loss on restart meant a box that rebooted while
// a disk was already failing never alerted again. UNKNOWN is never recorded and never deletes an
// existing record. See disk_health_state.go.
records map[string]*diskRecord
loaded bool
// clock is the injectable time source (nil → time.Now). The 24h re-alert cooldown is computed
// from it, so it must be injectable or the cooldown cannot be tested without sleeping.
clock func() time.Time
}
// now reads the injectable clock.
func (st *diskHealthState) now() time.Time {
if st.clock != nil {
return st.clock()
}
return time.Now()
}
// DiskHealthRow is one rendered card row.
@@ -110,12 +126,23 @@ func (s *Server) diskHealthRows(ctx context.Context) []DiskHealthRow {
if err != nil {
return nil
}
// The card must consult the SAME persisted prior the check uses, or the chip and the email can
// disagree — which is the exact property the shared verdict function exists to guarantee. Read
// only: rendering a page never writes reporting history.
s.diskHealth.mu.Lock()
s.loadDiskStateLocked()
priors := make(map[string]agentapi.DiskPrior, len(resp.Disks))
for _, d := range resp.Disks {
priors[diskKey(d)] = s.cardPriorFor(diskKey(d))
}
s.diskHealth.mu.Unlock()
var rows []DiskHealthRow
for _, d := range resp.Disks {
if !isPhysicalDisk(d) {
continue
}
v := agentapi.DiskVerdictFor(d.Smart)
v := agentapi.DiskVerdictFor(d.Smart, priors[diskKey(d)])
row := DiskHealthRow{Label: diskDisplayLabel(d), ChipLabel: v.Label(), ChipClass: diskChipClass(v)}
if d.Smart != nil && d.Smart.TemperatureC != nil {
row.Temp = strconv.Itoa(*d.Smart.TemperatureC)
@@ -136,31 +163,38 @@ func diskKey(d agentapi.DiskInfo) string {
return "name:" + d.Name
}
// RunDiskHealthCheck is the 6-hourly job. It emits disk_health_degraded ONLY on a degradation
// transition (a disk's verdict WORSENED) against the in-memory baseline. UNKNOWN is excluded both
// directions (never recorded, never a transition endpoint). The FIRST run baselines silently; a
// newly-appeared disk baselines silently; recovery (improvement) notifies nothing. Returns nil even
// when the agent is unreachable (skip quietly — no baseline churn, no alarm).
// RunDiskHealthCheck is the periodic job. It emits disk_health_degraded per the v0.215.0 decision
// rules in diskAlertDecision: escalation against the last ALERTED verdict fires immediately, a disk
// already at Hiba re-alerts when it has both doubled its unreadable-sector count AND waited out the
// 24h cooldown, and everything else is silent.
//
// Behaviours preserved verbatim from v0.169.0:
// - the first verdict ever for a disk baselines silently;
// - UNKNOWN is excluded both directions — never recorded, never a transition endpoint, and it does
// NOT delete an existing record (a transient unreadable SMART must not erase a real disk's
// history and hand it a clean slate);
// - a disk that disappears from the report is dropped, so a reappearance re-baselines silently;
// - recovery (improvement) notifies nothing;
// - an unreachable agent returns nil and changes nothing at all.
func (s *Server) RunDiskHealthCheck(ctx context.Context) error {
// Fetch FRESH (not the 60s card cache): the check runs every 6h, so it must see current SMART, and
// this keeps its transition logic independent of dashboard render timing.
// Fetch FRESH (not the 60s card cache): the check must see current SMART, and this keeps its
// decision logic independent of dashboard render timing.
resp, err := s.fetchDisks(ctx)
if err != nil {
// POSITIVE OBSERVABLE: say the check ran and why it produced nothing. An absent log line is
// equally consistent with "healthy" and "never ran".
if s.logger != nil {
s.logger.Printf("[DEBUG] [web] disk-health check skipped: agent unreachable: %v", err)
}
return nil
}
type degradation struct {
label string
attrs []string
critical bool
}
var fired []degradation
var fired []notify.DiskAlert
evaluated := 0
s.diskHealth.mu.Lock()
if s.diskHealth.baseline == nil {
s.diskHealth.baseline = map[string]agentapi.DiskVerdict{}
}
firstRun := !s.diskHealth.baselined
s.loadDiskStateLocked()
now := s.diskHealth.now()
seen := map[string]bool{}
for _, d := range resp.Disks {
if !isPhysicalDisk(d) {
@@ -168,47 +202,77 @@ func (s *Server) RunDiskHealthCheck(ctx context.Context) error {
}
key := diskKey(d)
seen[key] = true
v := agentapi.DiskVerdictFor(d.Smart)
prev := s.diskHealth.records[key]
prior := s.priorFor(key)
v := agentapi.DiskVerdictFor(d.Smart, prior)
if v == agentapi.DiskVerdictUnknown {
// Excluded both directions: don't record, don't transition, don't drop an existing baseline
// (a transient UNKNOWN blip must not erase history or fire).
// Excluded both directions: don't record, don't transition, don't drop the existing record.
continue
}
prev, had := s.diskHealth.baseline[key]
s.diskHealth.baseline[key] = v
if firstRun || !had {
continue // first verdict ever for this disk → baseline silently
evaluated++
sectors := agentapi.UncorrectableSectors(d.Smart)
emit, worsened := diskAlertDecision(prev, v, sectors, now)
// Build the successor record. The alert history is CARRIED FORWARD across an improvement —
// that is what makes flap damping work: a disk that recovers and degrades again still knows
// what it last told the customer.
rec := &diskRecord{Verdict: int(v), SawUncorrectable: sectors > 0, Sectors: sectors, ChangedAt: now,
PriorSawUncorrectable: prior.SawUncorrectable}
if prev != nil {
rec.Alerted, rec.AlertedVerdict = prev.Alerted, prev.AlertedVerdict
rec.AlertedSectors, rec.AlertedAt = prev.AlertedSectors, prev.AlertedAt
if prev.Verdict == int(v) && !prev.ChangedAt.IsZero() {
rec.ChangedAt = prev.ChangedAt // unchanged verdict keeps its original change time
}
}
if v > prev { // verdict worsened (Unknown=0 < OK=1 < Warn=2 < Fail=3; Unknown excluded above)
fired = append(fired, degradation{
label: diskDisplayLabel(d),
attrs: agentapi.DegradedAttributes(d.Smart),
critical: v == agentapi.DiskVerdictFail,
if emit {
rec.Alerted, rec.AlertedVerdict, rec.AlertedSectors, rec.AlertedAt = true, int(v), sectors, now
fired = append(fired, notify.DiskAlert{
Label: diskDisplayLabel(d),
Attributes: agentapi.DegradedAttributes(d.Smart),
Kind: diskAlertKindFor(d.Smart, v, worsened),
Sectors: sectors,
TemperatureC: temperatureOf(d.Smart),
})
}
s.diskHealth.records[key] = rec
}
// Forget disks no longer reported so a reappearance re-baselines silently.
for k := range s.diskHealth.baseline {
for k := range s.diskHealth.records {
if !seen[k] {
delete(s.diskHealth.baseline, k)
delete(s.diskHealth.records, k)
}
}
s.diskHealth.baselined = true
// ONE write per run, after every disk has been evaluated.
s.saveDiskStateLocked()
s.diskHealth.mu.Unlock()
for _, f := range fired {
s.emitDiskDegraded(f.label, f.attrs, f.critical)
// POSITIVE OBSERVABLE, every cycle: "0 alert(s)" from a check that evaluated N disks is evidence
// of health; silence is evidence of nothing.
if s.logger != nil {
s.logger.Printf("[INFO] [web] disk-health check complete: %d disk(s) evaluated, %d alert(s)", evaluated, len(fired))
}
for _, a := range fired {
s.emitDiskDegraded(a)
}
return nil
}
// emitDiskDegraded routes a degradation to the notifier (or the test seam). nil notifier → no-op.
func (s *Server) emitDiskDegraded(label string, attrs []string, critical bool) {
// temperatureOf is the disk's reported temperature, or 0 when the device reports none.
func temperatureOf(sm *agentapi.SmartSummary) int {
if sm != nil && sm.TemperatureC != nil {
return *sm.TemperatureC
}
return 0
}
// emitDiskDegraded routes an alert to the notifier (or the test seam). nil notifier → no-op.
func (s *Server) emitDiskDegraded(a notify.DiskAlert) {
if s.diskNotifyFn != nil {
s.diskNotifyFn(label, attrs, critical)
s.diskNotifyFn(a)
return
}
if s.notifier != nil {
s.notifier.NotifyDiskHealthDegraded(label, attrs, critical)
s.notifier.NotifyDiskHealthDegraded(a)
}
}