admin/felhom-controller
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(disk-health): the alert that never sent — severity, a real Hiba level, and a memory that survives a restart

Browse Source 
Three defects made the disk-health feature silent in exactly the case it
exists for. Evidence: felhom.eu documentation/audits/DIAG-smart-passed-trap-2026-08-14.md

1. SEVERITY (the one that changes whether anything arrives at all).
   NotifyDiskHealthDegraded emitted severity "warn", which is NOT in the
   hub's accepted set {info,warning,error,critical}. The hub coerced it to
   "info" (hub/internal/api/handler.go) and severityNotifies dropped it
   (hub/internal/notify/dispatcher.go), so every Figyelmeztetes-level disk
   alert was filed as an informational notice and emailed to NOBODY, on the
   customer and the operator leg alike. Now "warning". DiskAlertKind.Severity()
   is exported so the contract is checkable from any package.

2. NO LEVEL ABOVE "worth an eye". smart_status.passed CANNOT fail on
   unreadable sectors (attrs 187/197/198 all carry thresh 0 and a normalized
   value floors at 1), so Hiba was unreachable for this whole fault class.
   DiskVerdictFor now takes a DiskPrior and implements a 14-row top-down
   ladder: sustained unreadable sectors, a count too large to be a blip (64),
   unreadable+remapping together, overheating, NVMe critical flag or spent
   endurance all reach Hiba. No fourth label — predicted failure is "Hiba".

3. IT SPOKE ONCE, AND FORGOT ON RESTART. The baseline was in-memory, so a box
   that rebooted while a disk was failing never alerted again; and between 8
   and 352 sectors nothing was emitted at all. State is now persisted
   (disk-health-state.json, atomic tmp+rename), the decision compares against
   the last ALERTED verdict (collapsing flaps to one alert while letting a
   genuine escalation fire immediately), and a disk already at Hiba re-alerts
   once it has BOTH doubled its count and waited out a 24h cooldown.

The card replays the same prior the check used (diskRecord.PriorSawUncorrectable)
so the chip and the email cannot disagree — the property the shared verdict
function exists to guarantee, now pinned rather than asserted.

Tests: 12 scenario groups A-L. Group L builds the Server through web.NewServer,
the same call main.go makes, over a real file.
This commit is contained in:
authentik Default Admin
2026-08-14 08:10:59 +02:00
parent 3e3ee94b7b
commit bb50e1293c
9 changed files with 1534 additions and 228 deletions
Download Patch File Download Diff File Expand all files Collapse all files
controller/internal/agentapi/diskverdict.go
+136 -22
View File
@@ -1,7 +1,7 @@
package agentapi
// DiskVerdict is the customer-facing disk-health verdict derived from a SmartSummary (v0.169.0).
// It is the SHARED source of truth for both the "Lemezek állapota" dashboard card and the 6-hourly
// It is the SHARED source of truth for both the "Lemezek állapota" dashboard card and the periodic
// degradation check — one pure function so the chip and the alert can never disagree.
type DiskVerdict int
@@ -9,38 +9,144 @@ const (
// DiskVerdictUnknown — no SMART data (nil / UNKNOWN / old agent). Renders "Nincs adat"; NEVER
// alarms and NEVER participates in degradation transitions (excluded both directions).
DiskVerdictUnknown DiskVerdict = iota
DiskVerdictOK // "Rendben" — PASSED, all counters clean
DiskVerdictWarn // "Figyelmeztetés" — PASSED but a wear/relocation counter is non-zero (or NVMe ≥90%)
DiskVerdictFail // "Hiba" — FAILING
DiskVerdictOK // "Rendben" — clean
DiskVerdictWarn // "Figyelmeztetés" — a wear/relocation counter is non-zero, below the Hiba bar
DiskVerdictFail // "Hiba" — FAILING, or failing-but-not-self-reported (v0.215.0)
)
// percentageUsedWarn is the NVMe wear threshold (inclusive) at which a still-PASSED disk warns.
const percentageUsedWarn = 90
// Thresholds. A number without a reason becomes permanent by default, so each carries its provenance.
// The evidence is committed at felhom.eu/documentation/audits/DIAG-smart-passed-trap-2026-08-14.md
// and its two fixtures (ST3000VX010 S/N Z6A07P2G, /dev/sdg on DooPlex, 11-13 Aug 2026).
const (
// percentageUsedWarn / percentageUsedFail — NVMe wear (%). 100 means the vendor's rated endurance
// is spent; that is a declaration, not a trend, so it is Hiba.
percentageUsedWarn = 90
percentageUsedFail = 100
// DiskVerdictFor maps a SmartSummary to a verdict per the v0.169.0 rules:
// uncorrectableFailCount — unreadable sectors too numerous to be a blip.
//
// PROVENANCE: on the one real failing drive observed, the benign excursion peaked at 16 and
// cleared COMPLETELY within an hour (11 Aug 12:28 -> 13:28); the terminal run passed 64 at
// 13 Aug 11:28 and never came back below it. 64 sits above the one observed transient and below
// the observed terminal run. This is a judgement from ONE drive: it is a static BACKSTOP behind
// the sustain rule, not the primary signal, and Phase 3 is expected to replace it with
// growth-rate detection once the box keeps history.
uncorrectableFailCount = 64
// temperatureWarnC / TemperatureFailC — adopted UNCHANGED from the operator's existing Prometheus
// bands on DooPlex, so the two systems cannot disagree about the same drive.
temperatureWarnC = 55
// TemperatureFailC is exported because the alert-copy layer must pick the "overheated" message
// shape from the SAME number the verdict fired on. A second literal elsewhere would be free to
// drift, and the drift would show up as a customer told the wrong reason.
TemperatureFailC = 60
)
// DiskPrior is what the previous check observed for THIS SAME disk. It is the only history the
// verdict consults, and it is passed in rather than read so the function stays pure — the caller
// (internal/web) owns loading it from the persisted per-disk state.
//
// FAILING → Hiba
// PASSED + any(reallocated>0, pending>0, offline_unc>0,
// critical_warning>0, media_errors>0,
// percentage_used >= 90) → Figyelmeztetés
// PASSED otherwise → Rendben
// nil / UNKNOWN / empty → Nincs adat
func DiskVerdictFor(s *SmartSummary) DiskVerdict {
// Plain value type: no methods, no I/O. A zero DiskPrior means "nothing known", which is the correct
// fail-safe — a first-ever observation can only reach Figyelmeztetés from counters, never Hiba.
type DiskPrior struct {
// SawUncorrectable reports whether unreadable sectors (pending OR offline-uncorrectable) were
// present at the previous check. It is what turns a one-off excursion into a sustained fault.
SawUncorrectable bool
}
// DiskVerdictFor maps a SmartSummary plus the previous observation to a verdict. Rules are evaluated
// TOP-DOWN and the FIRST match wins (v0.215.0):
//
// 1. nil / "" / UNKNOWN -> Nincs adat
// 2. Health == FAILING -> Hiba (drive self-reports)
// 3. temperature_c >= 60 -> Hiba
// 4. critical_warning > 0 (NVMe's own flag: a declaration) -> Hiba
// 5. percentage_used >= 100 -> Hiba
// 6. unreadable > 0 AND prior.SawUncorrectable -> Hiba (SUSTAINED)
// 7. unreadable > 0 AND reallocated > 0 -> Hiba (accumulating + remapping)
// 8. unreadable >= 64 -> Hiba (too large to be a blip)
// 9. unreadable > 0 -> Figyelmeztetés (first sighting)
// 10. reallocated > 0 -> Figyelmeztetés
// 11. media_errors > 0 -> Figyelmeztetés
// 12. percentage_used >= 90 -> Figyelmeztetés
// 13. temperature_c >= 55 -> Figyelmeztetés
// 14. otherwise -> Rendben
//
// WHY rows 2-8 exist at all: smart_status.passed CANNOT fail on unreadable sectors. Attributes 187,
// 197 and 198 all carry thresh 0, and a normalized SMART value floors at 1, so it can never drop to
// or below the threshold. The real drive stayed PASSED at 352 pending sectors with 1001 reported
// uncorrectable reads. A verdict built on the drive's own self-assessment is blind to this whole
// class of failure, which is why rows 3-8 read the raw counters instead.
//
// WHY row 6 sits ABOVE row 8: sustain is the PRIMARY rule and the count is the backstop. On the real
// drive sustain fires a full day earlier (12 Aug) than the count threshold (13 Aug). Row 8 exists for
// a box that was powered off or restarted across the sustain window and so has no prior.
//
// Pure: no clock, no I/O, no logging. Everything it needs arrives as an argument.
func DiskVerdictFor(s *SmartSummary, prior DiskPrior) DiskVerdict {
// 1 — no data. Never alarms.
if s == nil || s.Health == "" || s.Health == SmartUnknown {
return DiskVerdictUnknown
}
// 2 — the drive admits failure.
if s.Health == SmartFailing {
return DiskVerdictFail
}
// Health == PASSED (or any non-empty non-FAILING value we treat as passing): inspect the counters.
if positive(s.ReallocatedSectors) || positive(s.PendingSectors) || positive(s.OfflineUncorrectable) ||
positive(s.CriticalWarning) || positive(s.MediaErrors) || atLeast(s.PercentageUsed, percentageUsedWarn) {
// Health == PASSED (or any non-empty non-FAILING value we treat as passing): inspect the counters,
// because the overall verdict is structurally unable to report this class of fault.
switch {
case atLeast(s.TemperatureC, TemperatureFailC): // 3
return DiskVerdictFail
case positive(s.CriticalWarning): // 4
return DiskVerdictFail
case atLeast(s.PercentageUsed, percentageUsedFail): // 5
return DiskVerdictFail
}
unreadable := UncorrectableSectors(s)
switch {
case unreadable > 0 && prior.SawUncorrectable: // 6 — sustained across two consecutive checks
return DiskVerdictFail
case unreadable > 0 && positive(s.ReallocatedSectors): // 7 — accumulating and remapping together
return DiskVerdictFail
case unreadable >= uncorrectableFailCount: // 8 — too large to be a blip
return DiskVerdictFail
case unreadable > 0: // 9 — first sighting, below the bar
return DiskVerdictWarn
case positive(s.ReallocatedSectors): // 10
return DiskVerdictWarn
case positive(s.MediaErrors): // 11
return DiskVerdictWarn
case atLeast(s.PercentageUsed, percentageUsedWarn): // 12
return DiskVerdictWarn
case atLeast(s.TemperatureC, temperatureWarnC): // 13
return DiskVerdictWarn
}
return DiskVerdictOK
return DiskVerdictOK // 14
}
// UncorrectableSectors is the disk's unreadable-sector count: max(pending, offline_uncorrectable).
// The two attributes track the same physical defect and on the real drive moved in lockstep, so the
// larger is the honest figure. 0 when neither is reported (an old agent or a device without them).
// Exported because the alert copy quotes this number and the persisted state remembers it.
func UncorrectableSectors(s *SmartSummary) int {
if s == nil {
return 0
}
n := 0
if s.PendingSectors != nil && *s.PendingSectors > n {
n = *s.PendingSectors
}
if s.OfflineUncorrectable != nil && *s.OfflineUncorrectable > n {
n = *s.OfflineUncorrectable
}
return n
}
// Label is the exact Hungarian customer copy for the verdict (shared by the card chip and the email).
//
// There are FOUR labels and there will not be a fifth: a predicted failure is "Hiba", the same word a
// self-reported failure gets. A fourth word sharing a root with "Figyelmeztetés" would make the MORE
// severe state read as the milder one (settled operator decision, v0.215.0).
func (v DiskVerdict) Label() string {
switch v {
case DiskVerdictOK:
@@ -54,11 +160,15 @@ func (v DiskVerdict) Label() string {
}
}
// DegradedAttributes returns the human-readable Hungarian names of the attribute(s) that pushed a
// PASSED disk to Figyelmeztetés (empty for OK/Fail/Unknown) — for the alert body. FAILING is a
// whole-disk verdict with no single triggering counter, so it returns nil there.
// DegradedAttributes returns the human-readable Hungarian names of the attribute(s) behind a
// degraded verdict, for the alert body.
//
// v0.215.0: this now also names the attributes behind a Hiba REACHED FROM COUNTERS (truth-table rows
// 3 and 6-8), not only a Figyelmeztetés — the alert message needs to say what is wrong, and those
// rows do have a triggering counter. It returns nil ONLY for row 2 (the drive self-reports FAILING,
// a whole-disk verdict with no single triggering counter) and, naturally, for Nincs adat / Rendben.
func DegradedAttributes(s *SmartSummary) []string {
if s == nil {
if s == nil || s.Health == "" || s.Health == SmartUnknown || s.Health == SmartFailing {
return nil
}
var out []string
@@ -80,6 +190,10 @@ func DegradedAttributes(s *SmartSummary) []string {
if atLeast(s.PercentageUsed, percentageUsedWarn) {
out = append(out, "elhasználódás")
}
// Newly able to trigger a verdict on its own (rows 3 and 13), so it must be nameable.
if atLeast(s.TemperatureC, temperatureWarnC) {
out = append(out, "hőmérséklet")
}
return out
}