fix(disk-health): the alert that never sent — severity, a real Hiba level, and a memory that survives a restart

Three defects made the disk-health feature silent in exactly the case it
exists for. Evidence: felhom.eu documentation/audits/DIAG-smart-passed-trap-2026-08-14.md

1. SEVERITY (the one that changes whether anything arrives at all).
   NotifyDiskHealthDegraded emitted severity "warn", which is NOT in the
   hub's accepted set {info,warning,error,critical}. The hub coerced it to
   "info" (hub/internal/api/handler.go) and severityNotifies dropped it
   (hub/internal/notify/dispatcher.go), so every Figyelmeztetes-level disk
   alert was filed as an informational notice and emailed to NOBODY, on the
   customer and the operator leg alike. Now "warning". DiskAlertKind.Severity()
   is exported so the contract is checkable from any package.

2. NO LEVEL ABOVE "worth an eye". smart_status.passed CANNOT fail on
   unreadable sectors (attrs 187/197/198 all carry thresh 0 and a normalized
   value floors at 1), so Hiba was unreachable for this whole fault class.
   DiskVerdictFor now takes a DiskPrior and implements a 14-row top-down
   ladder: sustained unreadable sectors, a count too large to be a blip (64),
   unreadable+remapping together, overheating, NVMe critical flag or spent
   endurance all reach Hiba. No fourth label — predicted failure is "Hiba".

3. IT SPOKE ONCE, AND FORGOT ON RESTART. The baseline was in-memory, so a box
   that rebooted while a disk was failing never alerted again; and between 8
   and 352 sectors nothing was emitted at all. State is now persisted
   (disk-health-state.json, atomic tmp+rename), the decision compares against
   the last ALERTED verdict (collapsing flaps to one alert while letting a
   genuine escalation fire immediately), and a disk already at Hiba re-alerts
   once it has BOTH doubled its count and waited out a 24h cooldown.

The card replays the same prior the check used (diskRecord.PriorSawUncorrectable)
so the chip and the email cannot disagree — the property the shared verdict
function exists to guarantee, now pinned rather than asserted.

Tests: 12 scenario groups A-L. Group L builds the Server through web.NewServer,
the same call main.go makes, over a real file.
This commit is contained in:
2026-08-14 08:10:59 +02:00
parent 3e3ee94b7b
commit bb50e1293c
9 changed files with 1534 additions and 228 deletions
+136 -22
View File
@@ -1,7 +1,7 @@
package agentapi package agentapi
// DiskVerdict is the customer-facing disk-health verdict derived from a SmartSummary (v0.169.0). // DiskVerdict is the customer-facing disk-health verdict derived from a SmartSummary (v0.169.0).
// It is the SHARED source of truth for both the "Lemezek állapota" dashboard card and the 6-hourly // It is the SHARED source of truth for both the "Lemezek állapota" dashboard card and the periodic
// degradation check — one pure function so the chip and the alert can never disagree. // degradation check — one pure function so the chip and the alert can never disagree.
type DiskVerdict int type DiskVerdict int
@@ -9,38 +9,144 @@ const (
// DiskVerdictUnknown — no SMART data (nil / UNKNOWN / old agent). Renders "Nincs adat"; NEVER // DiskVerdictUnknown — no SMART data (nil / UNKNOWN / old agent). Renders "Nincs adat"; NEVER
// alarms and NEVER participates in degradation transitions (excluded both directions). // alarms and NEVER participates in degradation transitions (excluded both directions).
DiskVerdictUnknown DiskVerdict = iota DiskVerdictUnknown DiskVerdict = iota
DiskVerdictOK // "Rendben" — PASSED, all counters clean DiskVerdictOK // "Rendben" — clean
DiskVerdictWarn // "Figyelmeztetés" — PASSED but a wear/relocation counter is non-zero (or NVMe ≥90%) DiskVerdictWarn // "Figyelmeztetés" — a wear/relocation counter is non-zero, below the Hiba bar
DiskVerdictFail // "Hiba" — FAILING DiskVerdictFail // "Hiba" — FAILING, or failing-but-not-self-reported (v0.215.0)
) )
// percentageUsedWarn is the NVMe wear threshold (inclusive) at which a still-PASSED disk warns. // Thresholds. A number without a reason becomes permanent by default, so each carries its provenance.
const percentageUsedWarn = 90 // The evidence is committed at felhom.eu/documentation/audits/DIAG-smart-passed-trap-2026-08-14.md
// and its two fixtures (ST3000VX010 S/N Z6A07P2G, /dev/sdg on DooPlex, 11-13 Aug 2026).
const (
// percentageUsedWarn / percentageUsedFail — NVMe wear (%). 100 means the vendor's rated endurance
// is spent; that is a declaration, not a trend, so it is Hiba.
percentageUsedWarn = 90
percentageUsedFail = 100
// DiskVerdictFor maps a SmartSummary to a verdict per the v0.169.0 rules: // uncorrectableFailCount — unreadable sectors too numerous to be a blip.
//
// PROVENANCE: on the one real failing drive observed, the benign excursion peaked at 16 and
// cleared COMPLETELY within an hour (11 Aug 12:28 -> 13:28); the terminal run passed 64 at
// 13 Aug 11:28 and never came back below it. 64 sits above the one observed transient and below
// the observed terminal run. This is a judgement from ONE drive: it is a static BACKSTOP behind
// the sustain rule, not the primary signal, and Phase 3 is expected to replace it with
// growth-rate detection once the box keeps history.
uncorrectableFailCount = 64
// temperatureWarnC / TemperatureFailC — adopted UNCHANGED from the operator's existing Prometheus
// bands on DooPlex, so the two systems cannot disagree about the same drive.
temperatureWarnC = 55
// TemperatureFailC is exported because the alert-copy layer must pick the "overheated" message
// shape from the SAME number the verdict fired on. A second literal elsewhere would be free to
// drift, and the drift would show up as a customer told the wrong reason.
TemperatureFailC = 60
)
// DiskPrior is what the previous check observed for THIS SAME disk. It is the only history the
// verdict consults, and it is passed in rather than read so the function stays pure — the caller
// (internal/web) owns loading it from the persisted per-disk state.
// //
// FAILING → Hiba // Plain value type: no methods, no I/O. A zero DiskPrior means "nothing known", which is the correct
// PASSED + any(reallocated>0, pending>0, offline_unc>0, // fail-safe — a first-ever observation can only reach Figyelmeztetés from counters, never Hiba.
// critical_warning>0, media_errors>0, type DiskPrior struct {
// percentage_used >= 90) → Figyelmeztetés // SawUncorrectable reports whether unreadable sectors (pending OR offline-uncorrectable) were
// PASSED otherwise → Rendben // present at the previous check. It is what turns a one-off excursion into a sustained fault.
// nil / UNKNOWN / empty → Nincs adat SawUncorrectable bool
func DiskVerdictFor(s *SmartSummary) DiskVerdict { }
// DiskVerdictFor maps a SmartSummary plus the previous observation to a verdict. Rules are evaluated
// TOP-DOWN and the FIRST match wins (v0.215.0):
//
// 1. nil / "" / UNKNOWN -> Nincs adat
// 2. Health == FAILING -> Hiba (drive self-reports)
// 3. temperature_c >= 60 -> Hiba
// 4. critical_warning > 0 (NVMe's own flag: a declaration) -> Hiba
// 5. percentage_used >= 100 -> Hiba
// 6. unreadable > 0 AND prior.SawUncorrectable -> Hiba (SUSTAINED)
// 7. unreadable > 0 AND reallocated > 0 -> Hiba (accumulating + remapping)
// 8. unreadable >= 64 -> Hiba (too large to be a blip)
// 9. unreadable > 0 -> Figyelmeztetés (first sighting)
// 10. reallocated > 0 -> Figyelmeztetés
// 11. media_errors > 0 -> Figyelmeztetés
// 12. percentage_used >= 90 -> Figyelmeztetés
// 13. temperature_c >= 55 -> Figyelmeztetés
// 14. otherwise -> Rendben
//
// WHY rows 2-8 exist at all: smart_status.passed CANNOT fail on unreadable sectors. Attributes 187,
// 197 and 198 all carry thresh 0, and a normalized SMART value floors at 1, so it can never drop to
// or below the threshold. The real drive stayed PASSED at 352 pending sectors with 1001 reported
// uncorrectable reads. A verdict built on the drive's own self-assessment is blind to this whole
// class of failure, which is why rows 3-8 read the raw counters instead.
//
// WHY row 6 sits ABOVE row 8: sustain is the PRIMARY rule and the count is the backstop. On the real
// drive sustain fires a full day earlier (12 Aug) than the count threshold (13 Aug). Row 8 exists for
// a box that was powered off or restarted across the sustain window and so has no prior.
//
// Pure: no clock, no I/O, no logging. Everything it needs arrives as an argument.
func DiskVerdictFor(s *SmartSummary, prior DiskPrior) DiskVerdict {
// 1 — no data. Never alarms.
if s == nil || s.Health == "" || s.Health == SmartUnknown { if s == nil || s.Health == "" || s.Health == SmartUnknown {
return DiskVerdictUnknown return DiskVerdictUnknown
} }
// 2 — the drive admits failure.
if s.Health == SmartFailing { if s.Health == SmartFailing {
return DiskVerdictFail return DiskVerdictFail
} }
// Health == PASSED (or any non-empty non-FAILING value we treat as passing): inspect the counters. // Health == PASSED (or any non-empty non-FAILING value we treat as passing): inspect the counters,
if positive(s.ReallocatedSectors) || positive(s.PendingSectors) || positive(s.OfflineUncorrectable) || // because the overall verdict is structurally unable to report this class of fault.
positive(s.CriticalWarning) || positive(s.MediaErrors) || atLeast(s.PercentageUsed, percentageUsedWarn) { switch {
case atLeast(s.TemperatureC, TemperatureFailC): // 3
return DiskVerdictFail
case positive(s.CriticalWarning): // 4
return DiskVerdictFail
case atLeast(s.PercentageUsed, percentageUsedFail): // 5
return DiskVerdictFail
}
unreadable := UncorrectableSectors(s)
switch {
case unreadable > 0 && prior.SawUncorrectable: // 6 — sustained across two consecutive checks
return DiskVerdictFail
case unreadable > 0 && positive(s.ReallocatedSectors): // 7 — accumulating and remapping together
return DiskVerdictFail
case unreadable >= uncorrectableFailCount: // 8 — too large to be a blip
return DiskVerdictFail
case unreadable > 0: // 9 — first sighting, below the bar
return DiskVerdictWarn
case positive(s.ReallocatedSectors): // 10
return DiskVerdictWarn
case positive(s.MediaErrors): // 11
return DiskVerdictWarn
case atLeast(s.PercentageUsed, percentageUsedWarn): // 12
return DiskVerdictWarn
case atLeast(s.TemperatureC, temperatureWarnC): // 13
return DiskVerdictWarn return DiskVerdictWarn
} }
return DiskVerdictOK return DiskVerdictOK // 14
}
// UncorrectableSectors is the disk's unreadable-sector count: max(pending, offline_uncorrectable).
// The two attributes track the same physical defect and on the real drive moved in lockstep, so the
// larger is the honest figure. 0 when neither is reported (an old agent or a device without them).
// Exported because the alert copy quotes this number and the persisted state remembers it.
func UncorrectableSectors(s *SmartSummary) int {
if s == nil {
return 0
}
n := 0
if s.PendingSectors != nil && *s.PendingSectors > n {
n = *s.PendingSectors
}
if s.OfflineUncorrectable != nil && *s.OfflineUncorrectable > n {
n = *s.OfflineUncorrectable
}
return n
} }
// Label is the exact Hungarian customer copy for the verdict (shared by the card chip and the email). // Label is the exact Hungarian customer copy for the verdict (shared by the card chip and the email).
//
// There are FOUR labels and there will not be a fifth: a predicted failure is "Hiba", the same word a
// self-reported failure gets. A fourth word sharing a root with "Figyelmeztetés" would make the MORE
// severe state read as the milder one (settled operator decision, v0.215.0).
func (v DiskVerdict) Label() string { func (v DiskVerdict) Label() string {
switch v { switch v {
case DiskVerdictOK: case DiskVerdictOK:
@@ -54,11 +160,15 @@ func (v DiskVerdict) Label() string {
} }
} }
// DegradedAttributes returns the human-readable Hungarian names of the attribute(s) that pushed a // DegradedAttributes returns the human-readable Hungarian names of the attribute(s) behind a
// PASSED disk to Figyelmeztetés (empty for OK/Fail/Unknown) — for the alert body. FAILING is a // degraded verdict, for the alert body.
// whole-disk verdict with no single triggering counter, so it returns nil there. //
// v0.215.0: this now also names the attributes behind a Hiba REACHED FROM COUNTERS (truth-table rows
// 3 and 6-8), not only a Figyelmeztetés — the alert message needs to say what is wrong, and those
// rows do have a triggering counter. It returns nil ONLY for row 2 (the drive self-reports FAILING,
// a whole-disk verdict with no single triggering counter) and, naturally, for Nincs adat / Rendben.
func DegradedAttributes(s *SmartSummary) []string { func DegradedAttributes(s *SmartSummary) []string {
if s == nil { if s == nil || s.Health == "" || s.Health == SmartUnknown || s.Health == SmartFailing {
return nil return nil
} }
var out []string var out []string
@@ -80,6 +190,10 @@ func DegradedAttributes(s *SmartSummary) []string {
if atLeast(s.PercentageUsed, percentageUsedWarn) { if atLeast(s.PercentageUsed, percentageUsedWarn) {
out = append(out, "elhasználódás") out = append(out, "elhasználódás")
} }
// Newly able to trigger a verdict on its own (rows 3 and 13), so it must be nameable.
if atLeast(s.TemperatureC, temperatureWarnC) {
out = append(out, "hőmérséklet")
}
return out return out
} }
@@ -0,0 +1,202 @@
package agentapi
import "testing"
// The v0.215.0 severity ladder, verdict half. The event half (emission, damping, cooldown,
// persistence) lives in internal/web — this file pins ONLY what the pure function decides.
//
// Every value used here is taken from the committed evidence:
// felhom.eu/documentation/audits/fixtures/smart-ST3000VX010-failing-2026-08-14.json
// (ST3000VX010-2E3166, S/N Z6A07P2G, /dev/sdg on DooPlex).
// realDrive is the failing drive AS CAPTURED on 2026-08-14: PASSED, 352 pending, 352 offline
// uncorrectable, 0 reallocated, 40 °C. The whole point of the fixture is that Health is PASSED.
func realDrive() *SmartSummary {
return &SmartSummary{
Health: SmartPassed,
PendingSectors: ip(352),
OfflineUncorrectable: ip(352),
ReallocatedSectors: ip(0),
TemperatureC: ip(40),
}
}
// Group A (verdict half) — Scenario A. The real drive on its SECOND observation reaches Hiba, and
// the chip label is exactly "Hiba".
//
// Red-proof: delete truth-table row 6 (the `prior.SawUncorrectable` case) from DiskVerdictFor →
// the drive still reaches Fail via row 8 (352 >= 64), so this test alone does NOT prove row 6.
// TestLadder_SustainIsWhatFires below is the one that isolates it.
func TestLadder_RealDrive_ReachesHiba(t *testing.T) {
got := DiskVerdictFor(realDrive(), DiskPrior{SawUncorrectable: true})
if got != DiskVerdictFail {
t.Fatalf("real failing drive verdict = %d (%s), want Fail/Hiba", got, got.Label())
}
if got.Label() != "Hiba" {
t.Errorf("label = %q, want %q", got.Label(), "Hiba")
}
// The trap this whole change exists for: the drive's own verdict says everything is fine.
if realDrive().Health != SmartPassed {
t.Fatal("fixture drift: the real drive's Health must be PASSED — that IS the defect")
}
}
// Groups B + C (verdict half) — Scenarios B and C. The SAME SmartSummary yields Figyelmeztetés on a
// first sighting and Hiba once it is sustained. This is the pair that isolates row 6: the counters
// are identical and only `prior` differs, so nothing else in the table can be producing the change.
//
// The values are the 11 August excursion (8 sectors), which cleared completely within an hour — a
// count deliberately far below the 64 backstop so row 8 cannot mask row 6.
//
// Red-proof: remove the `prior.SawUncorrectable` clause from row 6 → the sustained case stays Warn.
func TestLadder_SustainIsWhatFires(t *testing.T) {
excursion := func() *SmartSummary {
return &SmartSummary{Health: SmartPassed, PendingSectors: ip(8), OfflineUncorrectable: ip(8), ReallocatedSectors: ip(0)}
}
if got := DiskVerdictFor(excursion(), DiskPrior{}); got != DiskVerdictWarn {
t.Errorf("first sighting of 8 sectors = %d (%s), want Warn/Figyelmeztetés — a single "+
"excursion that clears by itself is normal and must NOT reach Hiba", got, got.Label())
}
if got := DiskVerdictFor(excursion(), DiskPrior{SawUncorrectable: true}); got != DiskVerdictFail {
t.Errorf("SAME 8 sectors, now sustained = %d (%s), want Fail/Hiba", got, got.Label())
}
if got := DiskVerdictFor(excursion(), DiskPrior{}).Label(); got != "Figyelmeztetés" {
t.Errorf("first-sighting label = %q, want Figyelmeztetés", got)
}
}
// Row 8, the backstop — for a box that was powered off or restarted across the sustain window and so
// has NO prior. 63 stays Warn, 64 reaches Hiba. The boundary is inclusive, which is what
// `uncorrectableFailCount` claims and what the real drive did at 13 Aug 11:28 (exactly 64).
//
// Red-proof: change `>=` to `>` in row 8 → the "exactly 64" case reads Warn.
func TestLadder_CountBackstopBoundary(t *testing.T) {
cases := []struct {
pending int
want DiskVerdict
}{
{63, DiskVerdictWarn},
{64, DiskVerdictFail},
{352, DiskVerdictFail},
}
for _, c := range cases {
s := &SmartSummary{Health: SmartPassed, PendingSectors: ip(c.pending)}
if got := DiskVerdictFor(s, DiskPrior{}); got != c.want {
t.Errorf("%d pending sectors, no prior = %d (%s), want %d", c.pending, got, got.Label(), c.want)
}
}
// Row 7 — unreadable AND remapping together is Hiba even at a low count with no prior.
s := &SmartSummary{Health: SmartPassed, PendingSectors: ip(8), ReallocatedSectors: ip(1)}
if got := DiskVerdictFor(s, DiskPrior{}); got != DiskVerdictFail {
t.Errorf("row 7 (unreadable + reallocated) = %d, want Fail", got)
}
}
// Group I — Scenario I, heat. 61 → Hiba, 56 → Figyelmeztetés, 54 → Rendben, with all counters clean.
//
// Red-proof: remove rows 3 and 13 → all three read Rendben.
func TestLadder_Temperature(t *testing.T) {
cases := []struct {
temp int
want DiskVerdict
}{
{54, DiskVerdictOK},
{55, DiskVerdictWarn}, // inclusive boundary
{56, DiskVerdictWarn},
{59, DiskVerdictWarn},
{60, DiskVerdictFail}, // inclusive boundary
{61, DiskVerdictFail},
}
for _, c := range cases {
s := &SmartSummary{Health: SmartPassed, TemperatureC: ip(c.temp), PendingSectors: ip(0), ReallocatedSectors: ip(0)}
if got := DiskVerdictFor(s, DiskPrior{}); got != c.want {
t.Errorf("%d °C = %d (%s), want %d", c.temp, got, got.Label(), c.want)
}
}
}
// Group J (verdict half) — Scenario J. No data never alarms, and a prior must not manufacture one:
// a nil/UNKNOWN SmartSummary reads Nincs adat EVEN WITH SawUncorrectable set. Row 1 is first in the
// table for exactly this reason.
//
// Red-proof: move row 1 below row 6 → the UNKNOWN-with-prior case reads Hiba, i.e. a disk whose
// SMART briefly became unreadable would be reported as failing.
func TestLadder_UnknownNeverAlarms(t *testing.T) {
for _, s := range []*SmartSummary{nil, {Health: ""}, {Health: SmartUnknown}} {
if got := DiskVerdictFor(s, DiskPrior{SawUncorrectable: true}); got != DiskVerdictUnknown {
t.Errorf("no-data disk with a prior = %d (%s), want Unknown/Nincs adat", got, got.Label())
}
}
if got := DiskVerdictFor(&SmartSummary{Health: SmartUnknown}, DiskPrior{}).Label(); got != "Nincs adat" {
t.Errorf("label = %q, want Nincs adat", got)
}
}
// The zero DiskPrior must be the SAFE default: a caller that forgets to load history can only
// under-report (Figyelmeztetés), never over-report (Hiba) on a first sighting. This pins the
// fail-safe direction the persisted-state loader relies on when its file is missing or corrupt.
func TestLadder_ZeroPriorIsFailSafe(t *testing.T) {
s := &SmartSummary{Health: SmartPassed, PendingSectors: ip(8)}
if got := DiskVerdictFor(s, DiskPrior{}); got != DiskVerdictWarn {
t.Fatalf("zero prior must degrade to Warn, not Fail; got %d (%s)", got, got.Label())
}
}
// UncorrectableSectors is max(pending, offline) — the number the alert copy quotes and the persisted
// state remembers. A wrong answer here puts a wrong count in a customer's email.
func TestUncorrectableSectors(t *testing.T) {
cases := []struct {
name string
in *SmartSummary
want int
}{
{"nil summary", nil, 0},
{"neither reported (old agent)", &SmartSummary{Health: SmartPassed}, 0},
{"both zero", &SmartSummary{PendingSectors: ip(0), OfflineUncorrectable: ip(0)}, 0},
{"pending only", &SmartSummary{PendingSectors: ip(8)}, 8},
{"offline only", &SmartSummary{OfflineUncorrectable: ip(24)}, 24},
{"pending larger", &SmartSummary{PendingSectors: ip(40), OfflineUncorrectable: ip(24)}, 40},
{"offline larger", &SmartSummary{PendingSectors: ip(24), OfflineUncorrectable: ip(40)}, 40},
{"the real drive", realDrive(), 352},
}
for _, c := range cases {
if got := UncorrectableSectors(c.in); got != c.want {
t.Errorf("%s: UncorrectableSectors = %d, want %d", c.name, got, c.want)
}
}
}
// DegradedAttributes must NAME the counters behind a Hiba reached from counters (v0.215.0) — the
// alert body is built from this and an empty list produces a message that says nothing is wrong.
// It still returns nil for row 2 (drive-reported FAILING), which has no single triggering counter.
//
// Red-proof: restore the pre-v0.215.0 body (nil for anything at Fail) → the real-drive case returns
// an empty list.
func TestDegradedAttributes_NamesFailCounters(t *testing.T) {
got := DegradedAttributes(realDrive())
if len(got) == 0 {
t.Fatal("a Hiba reached from counters must name its attributes, got none")
}
found := map[string]bool{}
for _, a := range got {
found[a] = true
}
for _, want := range []string{"függőben lévő szektorok", "javíthatatlan szektorok"} {
if !found[want] {
t.Errorf("missing attribute %q in %v", want, got)
}
}
// Row 2 — the drive self-reports FAILING: no single triggering counter, so nil.
if a := DegradedAttributes(&SmartSummary{Health: SmartFailing, PendingSectors: ip(5)}); a != nil {
t.Errorf("FAILING (row 2) must return nil attributes, got %v", a)
}
// Nincs adat must never produce attribute names either.
if a := DegradedAttributes(&SmartSummary{Health: SmartUnknown}); a != nil {
t.Errorf("UNKNOWN must return nil attributes, got %v", a)
}
// Temperature is newly able to trigger on its own, so it must be nameable.
hot := DegradedAttributes(&SmartSummary{Health: SmartPassed, TemperatureC: ip(61)})
if len(hot) != 1 || hot[0] != "hőmérséklet" {
t.Errorf("hot disk attributes = %v, want [hőmérséklet]", hot)
}
}
@@ -4,32 +4,40 @@ import "testing"
func ip(v int) *int { return &v } func ip(v int) *int { return &v }
// Verdict table (Part 2). Red-proof: change the PercentageUsed boundary from `>= 90` to `> 90` in // Verdict table (Part 2, extended v0.215.0). Red-proof: change the PercentageUsed boundary from
// DiskVerdictFor → the "NVMe percentage_used exactly 90 → Figyelmeztetés" case fails. // `>= 90` to `> 90` in DiskVerdictFor → the "NVMe percentage_used exactly 90 → Figyelmeztetés" case
// fails.
//
// v0.215.0 moved ONE pre-existing case deliberately: critical_warning>0 was Figyelmeztetés and is
// now Hiba (truth-table row 4). It is NVMe's own critical flag — a declaration by the device, not a
// counter that might drift back — so it belongs with the self-reported failures, not below them.
func TestDiskVerdictFor(t *testing.T) { func TestDiskVerdictFor(t *testing.T) {
noPrior := DiskPrior{}
cases := []struct { cases := []struct {
name string name string
in *SmartSummary in *SmartSummary
want DiskVerdict prior DiskPrior
want DiskVerdict
}{ }{
{"nil → unknown", nil, DiskVerdictUnknown}, {"nil → unknown", nil, noPrior, DiskVerdictUnknown},
{"empty health → unknown", &SmartSummary{Health: ""}, DiskVerdictUnknown}, {"empty health → unknown", &SmartSummary{Health: ""}, noPrior, DiskVerdictUnknown},
{"UNKNOWN → unknown", &SmartSummary{Health: SmartUnknown}, DiskVerdictUnknown}, {"UNKNOWN → unknown", &SmartSummary{Health: SmartUnknown}, noPrior, DiskVerdictUnknown},
{"FAILING → fail", &SmartSummary{Health: SmartFailing}, DiskVerdictFail}, {"FAILING → fail", &SmartSummary{Health: SmartFailing}, noPrior, DiskVerdictFail},
{"FAILING beats counters", &SmartSummary{Health: SmartFailing, ReallocatedSectors: ip(0)}, DiskVerdictFail}, {"FAILING beats counters", &SmartSummary{Health: SmartFailing, ReallocatedSectors: ip(0)}, noPrior, DiskVerdictFail},
{"PASSED clean → ok", &SmartSummary{Health: SmartPassed, ReallocatedSectors: ip(0), PendingSectors: ip(0), TemperatureC: ip(30)}, DiskVerdictOK}, {"PASSED clean → ok", &SmartSummary{Health: SmartPassed, ReallocatedSectors: ip(0), PendingSectors: ip(0), TemperatureC: ip(30)}, noPrior, DiskVerdictOK},
{"PASSED nil counters → ok", &SmartSummary{Health: SmartPassed}, DiskVerdictOK}, {"PASSED nil counters → ok", &SmartSummary{Health: SmartPassed}, noPrior, DiskVerdictOK},
{"reallocated>0 → warn", &SmartSummary{Health: SmartPassed, ReallocatedSectors: ip(1)}, DiskVerdictWarn}, {"reallocated>0 alone → warn", &SmartSummary{Health: SmartPassed, ReallocatedSectors: ip(1)}, noPrior, DiskVerdictWarn},
{"pending>0 → warn", &SmartSummary{Health: SmartPassed, PendingSectors: ip(5)}, DiskVerdictWarn}, {"pending>0 first sighting → warn", &SmartSummary{Health: SmartPassed, PendingSectors: ip(5)}, noPrior, DiskVerdictWarn},
{"offline_unc>0 → warn", &SmartSummary{Health: SmartPassed, OfflineUncorrectable: ip(2)}, DiskVerdictWarn}, {"offline_unc>0 first sighting → warn", &SmartSummary{Health: SmartPassed, OfflineUncorrectable: ip(2)}, noPrior, DiskVerdictWarn},
{"critical_warning>0 → warn", &SmartSummary{Health: SmartPassed, CriticalWarning: ip(1)}, DiskVerdictWarn}, {"critical_warning>0 → fail (row 4)", &SmartSummary{Health: SmartPassed, CriticalWarning: ip(1)}, noPrior, DiskVerdictFail},
{"media_errors>0 → warn", &SmartSummary{Health: SmartPassed, MediaErrors: ip(3)}, DiskVerdictWarn}, {"media_errors>0 → warn", &SmartSummary{Health: SmartPassed, MediaErrors: ip(3)}, noPrior, DiskVerdictWarn},
{"percentage_used 89 → ok", &SmartSummary{Health: SmartPassed, PercentageUsed: ip(89)}, DiskVerdictOK}, {"percentage_used 89 → ok", &SmartSummary{Health: SmartPassed, PercentageUsed: ip(89)}, noPrior, DiskVerdictOK},
{"percentage_used exactly 90 → warn", &SmartSummary{Health: SmartPassed, PercentageUsed: ip(90)}, DiskVerdictWarn}, {"percentage_used exactly 90 → warn", &SmartSummary{Health: SmartPassed, PercentageUsed: ip(90)}, noPrior, DiskVerdictWarn},
{"percentage_used 95 → warn", &SmartSummary{Health: SmartPassed, PercentageUsed: ip(95)}, DiskVerdictWarn}, {"percentage_used 95 → warn", &SmartSummary{Health: SmartPassed, PercentageUsed: ip(95)}, noPrior, DiskVerdictWarn},
{"percentage_used exactly 100 → fail (row 5)", &SmartSummary{Health: SmartPassed, PercentageUsed: ip(100)}, noPrior, DiskVerdictFail},
} }
for _, c := range cases { for _, c := range cases {
if got := DiskVerdictFor(c.in); got != c.want { if got := DiskVerdictFor(c.in, c.prior); got != c.want {
t.Errorf("%s: DiskVerdictFor = %d, want %d", c.name, got, c.want) t.Errorf("%s: DiskVerdictFor = %d, want %d", c.name, got, c.want)
} }
} }
+143 -18
View File
@@ -5,24 +5,81 @@ import (
"testing" "testing"
) )
// NotifyDiskHealthDegraded emits the right event type + severity + a message carrying the required // hubAcceptedSeverities is the hub's EXACT severity vocabulary, transcribed from
// "Lemez állapot romlás: <label>" phrase and the triggering attribute(s). Uses the pushFn seam. // felhom.eu/hub/internal/api/handler.go — the event-ingest handler's `switch payload.Severity` —
func TestNotifyDiskHealthDegraded(t *testing.T) { // where anything outside this set is silently coerced to "info".
//
// Of these, only warning/error/critical actually reach an email:
// felhom.eu/hub/internal/notify/dispatcher.go, func severityNotifies.
//
// This set is duplicated here ON PURPOSE. A test asserting only `== "warning"` would keep passing if
// the hub's vocabulary changed underneath it; pinning the set means the assertion is about the WIRE
// CONTRACT, and a reader can check both named locations rather than take this on trust.
var hubAcceptedSeverities = map[string]bool{"info": true, "warning": true, "error": true, "critical": true}
// hubSeverityNotifies mirrors felhom.eu/hub/internal/notify/dispatcher.go:severityNotifies.
func hubSeverityNotifies(sev string) bool {
switch sev {
case "warning", "error", "critical":
return true
}
return false
}
// Group K — Scenario K: the severity actually routes.
//
// This is the highest-value assertion in the change. Before v0.215.0 the Figyelmeztetés-level disk
// alert carried severity "warn", which is NOT in the hub's accepted set, so the hub coerced it to
// "info" and severityNotifies dropped it — the alert reached no email on either the customer or the
// operator leg.
//
// Red-proof: restore `return "warn"` in DiskAlertKind.severity() → the exact-match assertion, the
// accepted-set assertion AND the routes-to-email assertion all fail.
func TestNotifyDiskHealthDegraded_SeverityRoutes(t *testing.T) {
n := &Notifier{} n := &Notifier{}
var gotType, gotSev, gotMsg string var gotSev string
var gotDetails interface{} n.pushFn = func(eventType, severity, message string, details interface{}) { gotSev = severity }
n.pushFn = func(eventType, severity, message string, details interface{}) {
gotType, gotSev, gotMsg, gotDetails = eventType, severity, message, details n.NotifyDiskHealthDegraded(DiskAlert{Label: "sdb", Kind: DiskAlertWarn,
Attributes: []string{"függőben lévő szektorok"}})
if gotSev != "warning" {
t.Errorf("Figyelmeztetés severity = %q, want %q", gotSev, "warning")
}
if !hubAcceptedSeverities[gotSev] {
t.Errorf("severity %q is NOT in the hub's accepted set — the hub coerces it to \"info\"", gotSev)
}
if !hubSeverityNotifies(gotSev) {
t.Errorf("severity %q does not route to email — the alert would reach nobody", gotSev)
} }
// Warn (Figyelmeztetés): severity warn, attributes named. // Every Hiba kind must also carry a routing severity.
n.NotifyDiskHealthDegraded("sdb (lassú)", []string{"függőben lévő szektorok"}, false) for _, k := range []DiskAlertKind{DiskAlertFailSelfReported, DiskAlertFailSectors, DiskAlertFailTemperature, DiskAlertFailWorsened} {
n.NotifyDiskHealthDegraded(DiskAlert{Label: "sdc", Kind: k, Sectors: 352, TemperatureC: 61})
if gotSev != "critical" {
t.Errorf("kind %d severity = %q, want critical", k, gotSev)
}
if !hubAcceptedSeverities[gotSev] || !hubSeverityNotifies(gotSev) {
t.Errorf("kind %d severity %q does not route", k, gotSev)
}
}
}
// The event type + details payload + the Figyelmeztetés copy (unchanged wording, now delivered).
func TestNotifyDiskHealthDegraded_WarnShape(t *testing.T) {
n := &Notifier{}
var gotType, gotMsg string
var gotDetails interface{}
n.pushFn = func(eventType, severity, message string, details interface{}) {
gotType, gotMsg, gotDetails = eventType, message, details
}
n.NotifyDiskHealthDegraded(DiskAlert{Label: "sdb (lassú)", Kind: DiskAlertWarn,
Attributes: []string{"függőben lévő szektorok"}})
if gotType != "disk_health_degraded" { if gotType != "disk_health_degraded" {
t.Errorf("event type = %q, want disk_health_degraded", gotType) t.Errorf("event type = %q, want disk_health_degraded", gotType)
} }
if gotSev != "warn" {
t.Errorf("severity = %q, want warn", gotSev)
}
if !strings.Contains(gotMsg, "Lemez állapot romlás: sdb (lassú)") { if !strings.Contains(gotMsg, "Lemez állapot romlás: sdb (lassú)") {
t.Errorf("message missing the required subject phrase: %q", gotMsg) t.Errorf("message missing the required subject phrase: %q", gotMsg)
} }
@@ -32,13 +89,81 @@ func TestNotifyDiskHealthDegraded(t *testing.T) {
if d, ok := gotDetails.(DiskHealthDetails); !ok || d.Disk != "sdb (lassú)" || d.Critical { if d, ok := gotDetails.(DiskHealthDetails); !ok || d.Disk != "sdb (lassú)" || d.Critical {
t.Errorf("details = %+v, want DiskHealthDetails{Disk:sdb (lassú), Critical:false}", gotDetails) t.Errorf("details = %+v, want DiskHealthDetails{Disk:sdb (lassú), Critical:false}", gotDetails)
} }
}
// Critical (Hiba/FAILING): severity critical. // Each Hiba kind produces its OWN message, and each names the fact the customer needs to act on.
n.NotifyDiskHealthDegraded("sdc", nil, true) // A customer told "the drive overheated" must not be told to arrange a replacement, and vice versa.
if gotSev != "critical" { //
t.Errorf("critical severity = %q, want critical", gotSev) // Red-proof: collapse the switch so every Fail kind emits the self-reported sentence → the sector
// count and the temperature disappear from their messages and the substring assertions fail.
func TestNotifyDiskHealthDegraded_FailShapes(t *testing.T) {
n := &Notifier{}
var gotMsg string
var gotDetails interface{}
n.pushFn = func(eventType, severity, message string, details interface{}) { gotMsg, gotDetails = message, details }
cases := []struct {
name string
alert DiskAlert
wantSubstrs []string
notSubstrs []string
}{
{
name: "drive self-reports FAILING",
alert: DiskAlert{Label: "sdc", Kind: DiskAlertFailSelfReported},
wantSubstrs: []string{"Lemez állapot romlás: sdc", "SMART önellenőrzése hibát jelez"},
},
{
name: "reached from sector counters",
alert: DiskAlert{Label: "ST3000VX010-2E3166", Kind: DiskAlertFailSectors, Sectors: 352},
wantSubstrs: []string{"Lemez hiba: ST3000VX010-2E3166", "352 olvashatatlan szektor", "meghajtó cseréjéhez"},
notSubstrs: []string{"SMART önellenőrzése"}, // the drive said PASSED — must not claim otherwise
},
{
name: "reached from heat",
alert: DiskAlert{Label: "sdd", Kind: DiskAlertFailTemperature, TemperatureC: 61},
wantSubstrs: []string{"Lemez hiba: sdd", "túlmelegedett (61 °C)", "szellőzését"},
notSubstrs: []string{"olvashatatlan szektor"},
},
{
name: "already reported, still worsening",
alert: DiskAlert{Label: "sdg", Kind: DiskAlertFailWorsened, Sectors: 352},
wantSubstrs: []string{"Lemez hiba: sdg", "tovább romlott", "352 olvashatatlan szektor"},
},
} }
if !strings.Contains(gotMsg, "SMART önellenőrzése hibát jelez") { for _, c := range cases {
t.Errorf("critical message wrong: %q", gotMsg) n.NotifyDiskHealthDegraded(c.alert)
for _, w := range c.wantSubstrs {
if !strings.Contains(gotMsg, w) {
t.Errorf("%s: message %q missing %q", c.name, gotMsg, w)
}
}
for _, bad := range c.notSubstrs {
if strings.Contains(gotMsg, bad) {
t.Errorf("%s: message %q must not contain %q", c.name, gotMsg, bad)
}
}
if d, ok := gotDetails.(DiskHealthDetails); !ok || !d.Critical {
t.Errorf("%s: details %+v, want Critical:true", c.name, gotDetails)
}
}
}
// No customer-facing disk copy may carry an emoji (design-system hard rule) and every shape must
// name the disk. Cheap, and it covers shapes added later by someone who did not read the rule.
func TestNotifyDiskHealthDegraded_CopyDiscipline(t *testing.T) {
n := &Notifier{}
var gotMsg string
n.pushFn = func(eventType, severity, message string, details interface{}) { gotMsg = message }
for _, k := range []DiskAlertKind{DiskAlertWarn, DiskAlertFailSelfReported, DiskAlertFailSectors, DiskAlertFailTemperature, DiskAlertFailWorsened} {
n.NotifyDiskHealthDegraded(DiskAlert{Label: "TESTDISK", Kind: k, Sectors: 8, TemperatureC: 61})
if !strings.Contains(gotMsg, "TESTDISK") {
t.Errorf("kind %d: message does not name the disk: %q", k, gotMsg)
}
for _, r := range gotMsg {
if r > 0x2100 { // emoji / symbol block — Hungarian accents are all well below this
t.Errorf("kind %d: message contains a non-text rune %q: %s", k, r, gotMsg)
}
}
} }
} }
+71 -17
View File
@@ -556,24 +556,78 @@ type DiskHealthDetails struct {
Critical bool `json:"critical"` Critical bool `json:"critical"`
} }
// NotifyDiskHealthDegraded fires a disk_health_degraded hub event on a disk-health DEGRADATION only // DiskAlertKind selects the customer-facing message shape. A customer's ACTION differs by kind —
// (the controller's 6h check owns the transition logic — never first-run/recovery/UNKNOWN). severity // "back up and call us for a replacement" is not "check the ventilation" — so the kind travels with
// is warn for a Figyelmeztetés, critical for a Hiba (FAILING). The hub applies its own per-event-type // the alert rather than being flattened into a single sentence.
// cooldown. NOTE: the event type "disk_health_degraded" MUST be in the hub's allowedEventTypes (else type DiskAlertKind int
// the hub 400s the POST) — added in the hub's v0.x bump alongside this.
func (n *Notifier) NotifyDiskHealthDegraded(label string, attrs []string, critical bool) { const (
severity := "warn" DiskAlertWarn DiskAlertKind = iota // Figyelmeztetés — worth keeping an eye on
var msg string DiskAlertFailSelfReported // Hiba — the drive's own SMART verdict says FAILING
switch { DiskAlertFailSectors // Hiba — reached from unreadable-sector counters
case critical: DiskAlertFailTemperature // Hiba — reached from heat
severity = "critical" DiskAlertFailWorsened // Hiba — already reported, and still getting worse
msg = fmt.Sprintf("Lemez állapot romlás: %s — a lemez SMART önellenőrzése hibát jelez. Kérjük, mentse az adatait, és vegye fel velünk a kapcsolatot.", label) )
case len(attrs) > 0:
msg = fmt.Sprintf("Lemez állapot romlás: %s — romló érték: %s. Javasolt figyelemmel kísérni.", label, strings.Join(attrs, ", ")) // DiskAlert is the payload for one disk-health alert. It carries enough for the notifier to pick a
default: // message shape and fill in the counts; message CONSTRUCTION stays here because the notifier owns
msg = fmt.Sprintf("Lemez állapot romlás: %s — a lemez állapota romlott. Javasolt figyelemmel kísérni.", label) // customer copy, and moving it to the caller would scatter Hungarian across packages.
type DiskAlert struct {
Label string // customer-facing disk label (device model where known)
Attributes []string // Hungarian attribute names behind the verdict (nil for a self-reported FAILING)
Kind DiskAlertKind
Sectors int // max(pending, offline_uncorrectable) — quoted in the sector/worsened shapes
TemperatureC int // °C — quoted in the temperature shape
}
// Severity is the hub-accepted severity string for this alert.
//
// THE VOCABULARY IS EXACT AND IT IS THE HUB'S, NOT OURS. The hub accepts only
// {"info","warning","error","critical"} and silently COERCES anything else to "info"
// (felhom.eu/hub/internal/api/handler.go, the severity switch in the event-ingest handler); "info" is
// then dropped by severityNotifies (felhom.eu/hub/internal/notify/dispatcher.go), which routes only
// warning/error/critical. So a severity outside that set is stored and emailed to NOBODY — neither
// the customer nor the operator leg.
//
// Exported so any caller — and any test in any package — can check the contract against the two
// named hub locations instead of duplicating the literal.
//
// Until v0.215.0 this function emitted "warn", which is not in the set. Every Figyelmeztetés-level
// disk alert the product ever produced was filed as an informational notice and delivered to no one.
func (k DiskAlertKind) Severity() string {
if k == DiskAlertWarn {
return "warning"
} }
n.emit("disk_health_degraded", severity, msg, DiskHealthDetails{Disk: label, Attributes: attrs, Critical: critical}) return "critical"
}
// NotifyDiskHealthDegraded fires a disk_health_degraded hub event. The controller's periodic check
// owns the decision to call this at all (transitions, flap damping, the re-alert cooldown) — never
// first-run, never recovery, never UNKNOWN.
//
// The hub applies its own per-event-type cooldown ON TOP of ours. NOTE: the event type
// "disk_health_degraded" MUST be in the hub's allowedEventTypes (else the hub 400s the POST) — it is.
func (n *Notifier) NotifyDiskHealthDegraded(a DiskAlert) {
critical := a.Kind != DiskAlertWarn
var msg string
switch a.Kind {
case DiskAlertFailSelfReported:
msg = fmt.Sprintf("Lemez állapot romlás: %s — a lemez SMART önellenőrzése hibát jelez. Kérjük, mentse az adatait, és vegye fel velünk a kapcsolatot.", a.Label)
case DiskAlertFailSectors:
msg = fmt.Sprintf("Lemez hiba: %s — a meghajtón %d olvashatatlan szektor van. Mentse az adatait, és keressen meg minket a meghajtó cseréjéhez.", a.Label, a.Sectors)
case DiskAlertFailTemperature:
msg = fmt.Sprintf("Lemez hiba: %s — a meghajtó túlmelegedett (%d °C). Ellenőrizze a gép szellőzését, és keressen meg minket.", a.Label, a.TemperatureC)
case DiskAlertFailWorsened:
msg = fmt.Sprintf("Lemez hiba: %s — a meghajtó állapota tovább romlott, már %d olvashatatlan szektor van. Ha még nem tette meg, mentse az adatait.", a.Label, a.Sectors)
default: // DiskAlertWarn
if len(a.Attributes) > 0 {
msg = fmt.Sprintf("Lemez állapot romlás: %s — romló érték: %s. Javasolt figyelemmel kísérni.", a.Label, strings.Join(a.Attributes, ", "))
} else {
msg = fmt.Sprintf("Lemez állapot romlás: %s — a lemez állapota romlott. Javasolt figyelemmel kísérni.", a.Label)
}
}
n.emit("disk_health_degraded", a.Kind.Severity(), msg,
DiskHealthDetails{Disk: a.Label, Attributes: a.Attributes, Critical: critical})
} }
// emit sends an event through the test seam if set, else the real async PushEvent. // emit sends an event through the test seam if set, else the real async PushEvent.
+112 -48
View File
@@ -7,13 +7,16 @@ import (
"time" "time"
"gitea.dooplex.hu/admin/felhom-controller/internal/agentapi" "gitea.dooplex.hu/admin/felhom-controller/internal/agentapi"
"gitea.dooplex.hu/admin/felhom-controller/internal/notify"
) )
// Disk-health card + 6-hourly degradation check (v0.169.0). The card and the check share ONE data // Disk-health card + periodic degradation check (v0.169.0; ladder + persistence v0.215.0). The card
// path (the 60s-TTL-cached /disks call) and ONE verdict function (agentapi.DiskVerdictFor), so the // and the check share ONE data path (the 60s-TTL-cached /disks call) and ONE verdict function
// chip a customer sees and the alert they receive can never disagree. No new smartctl load — the // (agentapi.DiskVerdictFor), so the chip a customer sees and the alert they receive can never
// agent (v0.94.0) serializes its already-computed SMART; this consumes it and feature-detects // disagree. That shared-function property is load-bearing: BOTH callers must pass the SAME prior, or
// (nil Smart → "Nincs adat", never alarms). No global banner (CONTEXT ruling) — card + email only. // the chip could read Hiba while the email says Figyelmeztetés. No new smartctl load — the agent
// (v0.94.0) serializes its already-computed SMART; this consumes it and feature-detects (nil Smart →
// "Nincs adat", never alarms). No global banner (CONTEXT ruling) — card + email only.
const diskCacheTTL = 60 * time.Second const diskCacheTTL = 60 * time.Second
@@ -23,10 +26,23 @@ type diskHealthState struct {
cacheResp agentapi.DisksResponse cacheResp agentapi.DisksResponse
cacheErr error cacheErr error
cacheSet bool cacheSet bool
// baseline is the last-seen verdict per disk (in-memory only). UNKNOWN is never recorded. Lost on // records is the PERSISTED per-disk observation + alert history (v0.215.0), keyed by diskKey.
// restart → the next check re-baselines silently (accepted; see CONTEXT). // It replaced the in-memory-only baseline, whose loss on restart meant a box that rebooted while
baseline map[string]agentapi.DiskVerdict // a disk was already failing never alerted again. UNKNOWN is never recorded and never deletes an
baselined bool // existing record. See disk_health_state.go.
records map[string]*diskRecord
loaded bool
// clock is the injectable time source (nil → time.Now). The 24h re-alert cooldown is computed
// from it, so it must be injectable or the cooldown cannot be tested without sleeping.
clock func() time.Time
}
// now reads the injectable clock.
func (st *diskHealthState) now() time.Time {
if st.clock != nil {
return st.clock()
}
return time.Now()
} }
// DiskHealthRow is one rendered card row. // DiskHealthRow is one rendered card row.
@@ -110,12 +126,23 @@ func (s *Server) diskHealthRows(ctx context.Context) []DiskHealthRow {
if err != nil { if err != nil {
return nil return nil
} }
// The card must consult the SAME persisted prior the check uses, or the chip and the email can
// disagree — which is the exact property the shared verdict function exists to guarantee. Read
// only: rendering a page never writes reporting history.
s.diskHealth.mu.Lock()
s.loadDiskStateLocked()
priors := make(map[string]agentapi.DiskPrior, len(resp.Disks))
for _, d := range resp.Disks {
priors[diskKey(d)] = s.cardPriorFor(diskKey(d))
}
s.diskHealth.mu.Unlock()
var rows []DiskHealthRow var rows []DiskHealthRow
for _, d := range resp.Disks { for _, d := range resp.Disks {
if !isPhysicalDisk(d) { if !isPhysicalDisk(d) {
continue continue
} }
v := agentapi.DiskVerdictFor(d.Smart) v := agentapi.DiskVerdictFor(d.Smart, priors[diskKey(d)])
row := DiskHealthRow{Label: diskDisplayLabel(d), ChipLabel: v.Label(), ChipClass: diskChipClass(v)} row := DiskHealthRow{Label: diskDisplayLabel(d), ChipLabel: v.Label(), ChipClass: diskChipClass(v)}
if d.Smart != nil && d.Smart.TemperatureC != nil { if d.Smart != nil && d.Smart.TemperatureC != nil {
row.Temp = strconv.Itoa(*d.Smart.TemperatureC) row.Temp = strconv.Itoa(*d.Smart.TemperatureC)
@@ -136,31 +163,38 @@ func diskKey(d agentapi.DiskInfo) string {
return "name:" + d.Name return "name:" + d.Name
} }
// RunDiskHealthCheck is the 6-hourly job. It emits disk_health_degraded ONLY on a degradation // RunDiskHealthCheck is the periodic job. It emits disk_health_degraded per the v0.215.0 decision
// transition (a disk's verdict WORSENED) against the in-memory baseline. UNKNOWN is excluded both // rules in diskAlertDecision: escalation against the last ALERTED verdict fires immediately, a disk
// directions (never recorded, never a transition endpoint). The FIRST run baselines silently; a // already at Hiba re-alerts when it has both doubled its unreadable-sector count AND waited out the
// newly-appeared disk baselines silently; recovery (improvement) notifies nothing. Returns nil even // 24h cooldown, and everything else is silent.
// when the agent is unreachable (skip quietly — no baseline churn, no alarm). //
// Behaviours preserved verbatim from v0.169.0:
// - the first verdict ever for a disk baselines silently;
// - UNKNOWN is excluded both directions — never recorded, never a transition endpoint, and it does
// NOT delete an existing record (a transient unreadable SMART must not erase a real disk's
// history and hand it a clean slate);
// - a disk that disappears from the report is dropped, so a reappearance re-baselines silently;
// - recovery (improvement) notifies nothing;
// - an unreachable agent returns nil and changes nothing at all.
func (s *Server) RunDiskHealthCheck(ctx context.Context) error { func (s *Server) RunDiskHealthCheck(ctx context.Context) error {
// Fetch FRESH (not the 60s card cache): the check runs every 6h, so it must see current SMART, and // Fetch FRESH (not the 60s card cache): the check must see current SMART, and this keeps its
// this keeps its transition logic independent of dashboard render timing. // decision logic independent of dashboard render timing.
resp, err := s.fetchDisks(ctx) resp, err := s.fetchDisks(ctx)
if err != nil { if err != nil {
// POSITIVE OBSERVABLE: say the check ran and why it produced nothing. An absent log line is
// equally consistent with "healthy" and "never ran".
if s.logger != nil {
s.logger.Printf("[DEBUG] [web] disk-health check skipped: agent unreachable: %v", err)
}
return nil return nil
} }
type degradation struct { var fired []notify.DiskAlert
label string evaluated := 0
attrs []string
critical bool
}
var fired []degradation
s.diskHealth.mu.Lock() s.diskHealth.mu.Lock()
if s.diskHealth.baseline == nil { s.loadDiskStateLocked()
s.diskHealth.baseline = map[string]agentapi.DiskVerdict{} now := s.diskHealth.now()
}
firstRun := !s.diskHealth.baselined
seen := map[string]bool{} seen := map[string]bool{}
for _, d := range resp.Disks { for _, d := range resp.Disks {
if !isPhysicalDisk(d) { if !isPhysicalDisk(d) {
@@ -168,47 +202,77 @@ func (s *Server) RunDiskHealthCheck(ctx context.Context) error {
} }
key := diskKey(d) key := diskKey(d)
seen[key] = true seen[key] = true
v := agentapi.DiskVerdictFor(d.Smart) prev := s.diskHealth.records[key]
prior := s.priorFor(key)
v := agentapi.DiskVerdictFor(d.Smart, prior)
if v == agentapi.DiskVerdictUnknown { if v == agentapi.DiskVerdictUnknown {
// Excluded both directions: don't record, don't transition, don't drop an existing baseline // Excluded both directions: don't record, don't transition, don't drop the existing record.
// (a transient UNKNOWN blip must not erase history or fire).
continue continue
} }
prev, had := s.diskHealth.baseline[key] evaluated++
s.diskHealth.baseline[key] = v sectors := agentapi.UncorrectableSectors(d.Smart)
if firstRun || !had { emit, worsened := diskAlertDecision(prev, v, sectors, now)
continue // first verdict ever for this disk → baseline silently
// Build the successor record. The alert history is CARRIED FORWARD across an improvement —
// that is what makes flap damping work: a disk that recovers and degrades again still knows
// what it last told the customer.
rec := &diskRecord{Verdict: int(v), SawUncorrectable: sectors > 0, Sectors: sectors, ChangedAt: now,
PriorSawUncorrectable: prior.SawUncorrectable}
if prev != nil {
rec.Alerted, rec.AlertedVerdict = prev.Alerted, prev.AlertedVerdict
rec.AlertedSectors, rec.AlertedAt = prev.AlertedSectors, prev.AlertedAt
if prev.Verdict == int(v) && !prev.ChangedAt.IsZero() {
rec.ChangedAt = prev.ChangedAt // unchanged verdict keeps its original change time
}
} }
if v > prev { // verdict worsened (Unknown=0 < OK=1 < Warn=2 < Fail=3; Unknown excluded above) if emit {
fired = append(fired, degradation{ rec.Alerted, rec.AlertedVerdict, rec.AlertedSectors, rec.AlertedAt = true, int(v), sectors, now
label: diskDisplayLabel(d), fired = append(fired, notify.DiskAlert{
attrs: agentapi.DegradedAttributes(d.Smart), Label: diskDisplayLabel(d),
critical: v == agentapi.DiskVerdictFail, Attributes: agentapi.DegradedAttributes(d.Smart),
Kind: diskAlertKindFor(d.Smart, v, worsened),
Sectors: sectors,
TemperatureC: temperatureOf(d.Smart),
}) })
} }
s.diskHealth.records[key] = rec
} }
// Forget disks no longer reported so a reappearance re-baselines silently. // Forget disks no longer reported so a reappearance re-baselines silently.
for k := range s.diskHealth.baseline { for k := range s.diskHealth.records {
if !seen[k] { if !seen[k] {
delete(s.diskHealth.baseline, k) delete(s.diskHealth.records, k)
} }
} }
s.diskHealth.baselined = true // ONE write per run, after every disk has been evaluated.
s.saveDiskStateLocked()
s.diskHealth.mu.Unlock() s.diskHealth.mu.Unlock()
for _, f := range fired { // POSITIVE OBSERVABLE, every cycle: "0 alert(s)" from a check that evaluated N disks is evidence
s.emitDiskDegraded(f.label, f.attrs, f.critical) // of health; silence is evidence of nothing.
if s.logger != nil {
s.logger.Printf("[INFO] [web] disk-health check complete: %d disk(s) evaluated, %d alert(s)", evaluated, len(fired))
}
for _, a := range fired {
s.emitDiskDegraded(a)
} }
return nil return nil
} }
// emitDiskDegraded routes a degradation to the notifier (or the test seam). nil notifier → no-op. // temperatureOf is the disk's reported temperature, or 0 when the device reports none.
func (s *Server) emitDiskDegraded(label string, attrs []string, critical bool) { func temperatureOf(sm *agentapi.SmartSummary) int {
if sm != nil && sm.TemperatureC != nil {
return *sm.TemperatureC
}
return 0
}
// emitDiskDegraded routes an alert to the notifier (or the test seam). nil notifier → no-op.
func (s *Server) emitDiskDegraded(a notify.DiskAlert) {
if s.diskNotifyFn != nil { if s.diskNotifyFn != nil {
s.diskNotifyFn(label, attrs, critical) s.diskNotifyFn(a)
return return
} }
if s.notifier != nil { if s.notifier != nil {
s.notifier.NotifyDiskHealthDegraded(label, attrs, critical) s.notifier.NotifyDiskHealthDegraded(a)
} }
} }
@@ -0,0 +1,251 @@
package web
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/agentapi"
"gitea.dooplex.hu/admin/felhom-controller/internal/notify"
)
// Persisted disk-health reporting state (v0.215.0).
//
// WHY IT IS PERSISTED: until v0.215.0 the baseline was in-memory only, so a controller restart while
// a disk was already failing re-baselined that disk silently — and it never alerted again. A box that
// reboots during exactly the fault this feature exists for was the one case it could not report.
//
// WHAT IT IS NOT: one small record per disk about what has been OBSERVED and REPORTED. It is NOT a
// sample history and must not grow into one — SMART sample series belong in metrics.MetricsStore
// under Phase 2/3, which is a schema migration and a wire change. Keeping this a flat file keeps a
// one-word severity fix off that critical path.
//
// Crash-safety: atomic tmp+rename is sufficient. The state describes reporting history, not a
// mutation in flight, so a lost write costs at most one re-baseline — and the failure direction is a
// DUPLICATE alert rather than a missing one.
const diskStateFileName = "disk-health-state.json"
// diskStateVersion guards the on-disk shape. An unrecognised version is treated as "no prior"
// (fail-safe) rather than mis-parsed into a verdict.
const diskStateVersion = 1
// diskRealertCooldown is the minimum gap between two alerts for a disk already at Hiba. It is ANDed
// with the doubling bar — clearing only one of them emits nothing.
const diskRealertCooldown = 24 * time.Hour
// diskRecord is everything the checker remembers about ONE disk between runs.
type diskRecord struct {
// Verdict is the last non-UNKNOWN verdict observed (agentapi.DiskVerdict as an int).
Verdict int `json:"verdict"`
// SawUncorrectable is THIS observation: it becomes the NEXT check's agentapi.DiskPrior, and is
// what turns a one-off excursion into a sustained fault.
SawUncorrectable bool `json:"saw_uncorrectable"`
// PriorSawUncorrectable is the prior that PRODUCED Verdict — i.e. the previous observation.
//
// It exists so the CARD can reach the same verdict as the check. The two run at different times
// over the same SMART: the check consumes the prior and then overwrites it with the current
// observation, so a card rendering afterwards would consume its own check's write and read one
// level too high — a disk at its first sighting of 8 sectors would show "Hiba" on the dashboard
// while the alert (correctly) said "Figyelmeztetés". Replaying the SAME prior keeps the shared
// verdict function's guarantee intact instead of merely asserting it.
PriorSawUncorrectable bool `json:"prior_saw_uncorrectable,omitempty"`
// Sectors is max(pending, offline_uncorrectable) at the last observation.
Sectors int `json:"sectors,omitempty"`
// ChangedAt is when Verdict last changed value.
ChangedAt time.Time `json:"changed_at,omitempty"`
// --- alert history: survives an intervening improvement, which is what makes flap damping work ---
// Alerted is false until this disk has ever produced an event. Distinguishes "never alerted"
// from "alerted at Nincs adat", which cannot happen but would otherwise share the zero value.
Alerted bool `json:"alerted,omitempty"`
AlertedVerdict int `json:"alerted_verdict,omitempty"`
AlertedSectors int `json:"alerted_sectors,omitempty"`
AlertedAt time.Time `json:"alerted_at,omitempty"`
}
// diskStateFile is the on-disk envelope.
type diskStateFile struct {
Version int `json:"version"`
Disks map[string]*diskRecord `json:"disks"`
}
// diskStatePath is the state file's location, or "" when no data dir is configured (unit tests and
// an unprovisioned guest). An empty path degrades to in-memory-only: correct behaviour within the
// process, simply not durable.
func (s *Server) diskStatePath() string {
if s.cfg == nil || s.cfg.Paths.DataDir == "" {
return ""
}
return filepath.Join(s.cfg.Paths.DataDir, diskStateFileName)
}
// loadDiskStateLocked populates the in-memory records from disk exactly once per process. The caller
// holds s.diskHealth.mu.
//
// NEVER FATAL. A missing file is the normal first-boot case. A corrupt or unreadable one is LOGGED
// and treated as "no prior" — the worst that costs is one silent re-baseline, whereas crashing the
// check would take away disk monitoring entirely.
func (s *Server) loadDiskStateLocked() {
if s.diskHealth.loaded {
return
}
s.diskHealth.loaded = true
if s.diskHealth.records == nil {
s.diskHealth.records = map[string]*diskRecord{}
}
path := s.diskStatePath()
if path == "" {
return
}
data, err := os.ReadFile(path)
if err != nil {
if !os.IsNotExist(err) && s.logger != nil {
s.logger.Printf("[WARN] [web] disk-health state unreadable, continuing with no prior: %v", err)
}
return
}
var f diskStateFile
if err := json.Unmarshal(data, &f); err != nil {
if s.logger != nil {
s.logger.Printf("[WARN] [web] disk-health state corrupt, continuing with no prior: %v", err)
}
return
}
if f.Version != diskStateVersion {
if s.logger != nil {
s.logger.Printf("[WARN] [web] disk-health state version %d unrecognised (want %d), continuing with no prior",
f.Version, diskStateVersion)
}
return
}
for k, r := range f.Disks {
if r != nil {
s.diskHealth.records[k] = r
}
}
if s.logger != nil {
s.logger.Printf("[DEBUG] [web] disk-health state loaded: %d disk(s)", len(s.diskHealth.records))
}
}
// saveDiskStateLocked writes the records atomically (.tmp then rename), following the shape of
// selfupdate.SaveState. Called ONCE per check run, after every disk has been evaluated — not per
// disk. Caller holds s.diskHealth.mu. Errors are logged, never returned: failing to remember is not
// a reason to stop monitoring.
func (s *Server) saveDiskStateLocked() {
path := s.diskStatePath()
if path == "" {
return
}
if err := writeDiskState(path, s.diskHealth.records); err != nil && s.logger != nil {
s.logger.Printf("[WARN] [web] disk-health state not saved (a restart will re-baseline): %v", err)
}
}
func writeDiskState(path string, records map[string]*diskRecord) error {
data, err := json.MarshalIndent(diskStateFile{Version: diskStateVersion, Disks: records}, "", " ")
if err != nil {
return fmt.Errorf("marshaling disk-health state: %w", err)
}
tmp := path + ".tmp"
if err := os.WriteFile(tmp, data, 0644); err != nil {
return fmt.Errorf("writing temp disk-health state: %w", err)
}
if err := os.Rename(tmp, path); err != nil {
os.Remove(tmp)
return fmt.Errorf("renaming disk-health state: %w", err)
}
return nil
}
// priorFor is the agentapi.DiskPrior the CHECK consumes: what the previous check observed. A zero
// DiskPrior ("nothing known") is the fail-safe — a first sighting can then only reach
// Figyelmeztetés, never Hiba. Caller holds the mutex.
func (s *Server) priorFor(key string) agentapi.DiskPrior {
if r := s.diskHealth.records[key]; r != nil {
return agentapi.DiskPrior{SawUncorrectable: r.SawUncorrectable}
}
return agentapi.DiskPrior{}
}
// cardPriorFor is the agentapi.DiskPrior the CARD replays: the prior that produced the stored
// verdict, NOT the observation that check went on to record. See diskRecord.PriorSawUncorrectable —
// using priorFor here would make the dashboard chip read one level more severe than the alert.
// Caller holds the mutex.
func (s *Server) cardPriorFor(key string) agentapi.DiskPrior {
if r := s.diskHealth.records[key]; r != nil {
return agentapi.DiskPrior{SawUncorrectable: r.PriorSawUncorrectable}
}
return agentapi.DiskPrior{}
}
// diskAlertDecision decides whether THIS observation produces an event, per the v0.215.0 rules.
// Pure: everything, including the clock reading, arrives as an argument.
//
// prev == nil -> no event (first verdict ever for this disk: baseline silently)
// v is Rendben (or better) -> no event (recovery is silent — unchanged behaviour)
// v worse than the LAST ALERTED -> EVENT. Escalation is never damped.
// v == last alerted, and v is Hiba,
// and sectors >= 2x the count at
// the last alert, and >= 24h since -> EVENT ("still getting worse")
// anything else -> no event
//
// The second return value reports whether this is the "still getting worse" re-alert, which selects
// a different message shape.
//
// WHY "worse than the last ALERTED verdict" rather than the last OBSERVED one: a disk that goes
// Figyelmeztetés -> Rendben -> Figyelmeztetés has worsened against its last observation twice, and
// today's transition-only logic emits two identical mild alerts on that zero-crossing. That is how a
// customer learns to ignore the message. Comparing against what was last REPORTED collapses the flap
// to one alert while leaving a genuine escalation (Figyelmeztetés -> Hiba) free to fire immediately.
//
// WHY nothing else emits at v == last alerted: the steady-state case and the post-improvement flap
// case are indistinguishable without tracking intervening improvements, and both must stay silent.
// Only the explicit Hiba re-alert above breaks that silence, and only when BOTH its bars are cleared.
func diskAlertDecision(prev *diskRecord, v agentapi.DiskVerdict, sectors int, now time.Time) (emit bool, worsened bool) {
if v <= agentapi.DiskVerdictOK {
return false, false // Nincs adat never reaches here; Rendben is silent
}
if prev == nil {
return false, false // first verdict ever for this disk
}
alerted := agentapi.DiskVerdictUnknown
if prev.Alerted {
alerted = agentapi.DiskVerdict(prev.AlertedVerdict)
}
if v > alerted {
return true, false // escalation — damping never applies
}
if v == alerted && v == agentapi.DiskVerdictFail &&
prev.AlertedSectors > 0 && sectors >= 2*prev.AlertedSectors &&
now.Sub(prev.AlertedAt) >= diskRealertCooldown {
return true, true // still getting worse
}
return false, false
}
// diskAlertKindFor picks the customer-facing message shape for an alert that has already been
// decided. It mirrors the truth-table order so the reason quoted to the customer is the reason the
// verdict actually fired on.
func diskAlertKindFor(sm *agentapi.SmartSummary, v agentapi.DiskVerdict, worsened bool) notify.DiskAlertKind {
if v != agentapi.DiskVerdictFail {
return notify.DiskAlertWarn
}
switch {
case sm != nil && sm.Health == agentapi.SmartFailing:
return notify.DiskAlertFailSelfReported
case sm != nil && sm.TemperatureC != nil && *sm.TemperatureC >= agentapi.TemperatureFailC:
return notify.DiskAlertFailTemperature
case agentapi.UncorrectableSectors(sm) > 0 && worsened:
return notify.DiskAlertFailWorsened
case agentapi.UncorrectableSectors(sm) > 0:
return notify.DiskAlertFailSectors
}
// Hiba with no sector count and no heat: NVMe's own critical flag, or spent rated endurance.
// Both are declarations BY THE DEVICE, so the self-reported wording is the honest one.
return notify.DiskAlertFailSelfReported
}
+585 -98
View File
@@ -2,122 +2,628 @@ package web
import ( import (
"context" "context"
"encoding/json"
"io"
"log"
"os"
"path/filepath"
"testing" "testing"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/agentapi" "gitea.dooplex.hu/admin/felhom-controller/internal/agentapi"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
"gitea.dooplex.hu/admin/felhom-controller/internal/notify"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
) )
// Disk-health severity ladder, event half (v0.215.0). Scenario letters refer to the task spec.
// The verdict half is pinned in internal/agentapi/diskverdict_ladder_test.go.
func smartPtr(v int) *int { return &v } func smartPtr(v int) *int { return &v }
func physDisk(name string, sm *agentapi.SmartSummary) agentapi.DiskInfo { func physDisk(name string, sm *agentapi.SmartSummary) agentapi.DiskInfo {
return agentapi.DiskInfo{Name: name, BackingDevice: "/dev/" + name, DurableID: "uuid:" + name, Smart: sm} return agentapi.DiskInfo{Name: name, BackingDevice: "/dev/" + name, DurableID: "uuid:" + name, Smart: sm}
} }
// diskCheckHarness wires a Server with the disks source + notify sink seams and returns a captured // realDriveSmart is the failing drive as captured on 2026-08-14 — PASSED, 352 unreadable sectors.
// list of emitted disk labels. // felhom.eu/documentation/audits/fixtures/smart-ST3000VX010-failing-2026-08-14.json
func diskCheckHarness(t *testing.T) (*Server, *[]string, *[]agentapi.DiskInfo) { func realDriveSmart() *agentapi.SmartSummary {
return &agentapi.SmartSummary{
Health: agentapi.SmartPassed,
PendingSectors: smartPtr(352),
OfflineUncorrectable: smartPtr(352),
ReallocatedSectors: smartPtr(0),
TemperatureC: smartPtr(40),
}
}
// diskHarness wires a Server with the disks source + notify sink seams, a REAL data dir (so the
// persisted state is exercised, not bypassed) and a controllable clock.
type diskHarness struct {
s *Server
fired *[]notify.DiskAlert
payload *[]agentapi.DiskInfo
dir string
clock *time.Time
}
func newDiskHarness(t *testing.T) *diskHarness {
t.Helper() t.Helper()
s := testServer(t) dir := t.TempDir()
var fired []string lg := log.New(io.Discard, "", 0)
payload := &[]agentapi.DiskInfo{} sett, err := settings.Load(filepath.Join(dir, "settings.json"), lg)
s.diskNotifyFn = func(label string, attrs []string, critical bool) { fired = append(fired, label) } if err != nil {
t.Fatalf("settings: %v", err)
}
cfg := &config.Config{}
cfg.Paths.DataDir = dir
s := &Server{settings: sett, logger: lg, cfg: cfg}
now := time.Date(2026, 8, 14, 12, 0, 0, 0, time.UTC)
h := &diskHarness{s: s, dir: dir, clock: &now, fired: &[]notify.DiskAlert{}, payload: &[]agentapi.DiskInfo{}}
s.diskHealth.clock = func() time.Time { return *h.clock }
s.diskNotifyFn = func(a notify.DiskAlert) { *h.fired = append(*h.fired, a) }
s.disksFn = func(ctx context.Context) (agentapi.DisksResponse, error) { s.disksFn = func(ctx context.Context) (agentapi.DisksResponse, error) {
return agentapi.DisksResponse{Disks: *payload}, nil return agentapi.DisksResponse{Disks: *h.payload}, nil
} }
return s, &fired, payload return h
} }
// Scenario B — first run baselines silently; a real degradation (OK→Warn) emits exactly once; a func (h *diskHarness) set(d ...agentapi.DiskInfo) { *h.payload = d }
// steady-state re-check does not re-emit. Red-proof: remove the `firstRun || !had` guard → the first func (h *diskHarness) run(t *testing.T) {
// run emits and the "no notify on first run" assertion fails. t.Helper()
func TestDiskHealthCheck_DegradationOnce(t *testing.T) { if err := h.s.RunDiskHealthCheck(context.Background()); err != nil {
s, fired, payload := diskCheckHarness(t) t.Fatalf("RunDiskHealthCheck: %v", err)
ctx := context.Background() }
}
func (h *diskHarness) advance(d time.Duration) { *h.clock = h.clock.Add(d) }
func (h *diskHarness) events() []notify.DiskAlert {
return *h.fired
}
func (h *diskHarness) reset() { *h.fired = nil }
// First check: disk PASSED clean (verdict OK). Baseline only. // record reads the persisted state file straight off disk — asserting the FILE, not the in-memory
*payload = []agentapi.DiskInfo{physDisk("sdb", &agentapi.SmartSummary{Health: agentapi.SmartPassed})} // map, because Scenario L depends on what actually landed there.
_ = s.RunDiskHealthCheck(ctx) func (h *diskHarness) record(t *testing.T, key string) *diskRecord {
if len(*fired) != 0 { t.Helper()
t.Fatalf("first run must not notify, got %v", *fired) data, err := os.ReadFile(filepath.Join(h.dir, diskStateFileName))
if err != nil {
t.Fatalf("reading state file: %v", err)
}
var f diskStateFile
if err := json.Unmarshal(data, &f); err != nil {
t.Fatalf("parsing state file: %v", err)
}
return f.Disks[key]
}
// ── Group A — Scenario A: the real drive, second observation ────────────────────────────────────
// The captured failing drive reaches Hiba, renders the crit chip, and emits exactly ONE event at
// severity "critical".
//
// Red-proof: remove truth-table row 6 (prior.SawUncorrectable) → 352 still trips row 8, so use
// TestDiskLadder_SustainDrivesTheEscalation (Group C) to isolate row 6.
func TestDiskCheck_RealDrive_HibaAndOneCriticalEvent(t *testing.T) {
h := newDiskHarness(t)
h.set(physDisk("sdg", realDriveSmart()))
h.run(t) // first ever observation: baselines silently
if n := len(h.events()); n != 0 {
t.Fatalf("first observation must be silent, got %d event(s)", n)
} }
// Degrade: pending sectors 0→5 (OK→Figyelmeztetés). h.run(t) // second observation, now with prior
*payload = []agentapi.DiskInfo{physDisk("sdb", &agentapi.SmartSummary{Health: agentapi.SmartPassed, PendingSectors: smartPtr(5)})} ev := h.events()
_ = s.RunDiskHealthCheck(ctx) if len(ev) != 1 {
if len(*fired) != 1 { t.Fatalf("want exactly ONE event, got %d: %+v", len(ev), ev)
t.Fatalf("degradation must emit exactly once, got %v", *fired) }
if ev[0].Kind.Severity() != "critical" {
t.Errorf("severity = %q, want %q", ev[0].Kind.Severity(), "critical")
}
if ev[0].Kind != notify.DiskAlertFailSectors {
t.Errorf("kind = %d, want DiskAlertFailSectors (the drive says PASSED — we must not claim it self-reported)", ev[0].Kind)
}
if ev[0].Sectors != 352 {
t.Errorf("sectors = %d, want 352", ev[0].Sectors)
} }
// Steady state: still Figyelmeztetés — no repeat. // The card must agree with the alert — same verdict function, same prior.
_ = s.RunDiskHealthCheck(ctx) rows := h.s.diskHealthRows(context.Background())
if len(*fired) != 1 { if len(rows) != 1 {
t.Fatalf("steady-state degraded must not re-emit, got %v", *fired) t.Fatalf("want 1 card row, got %d", len(rows))
}
if rows[0].ChipLabel != "Hiba" {
t.Errorf("chip label = %q, want %q", rows[0].ChipLabel, "Hiba")
}
if rows[0].ChipClass != "state-text-crit" {
t.Errorf("chip class = %q, want %q", rows[0].ChipClass, "state-text-crit")
} }
} }
// Scenario B (cont.) — recovery (Figyelmeztetés→Rendben) notifies nothing. // ── Group B — Scenario B: the transient that cleared (11 August) ────────────────────────────────
func TestDiskHealthCheck_RecoverySilent(t *testing.T) {
s, fired, payload := diskCheckHarness(t)
ctx := context.Background()
*payload = []agentapi.DiskInfo{physDisk("sdb", &agentapi.SmartSummary{Health: agentapi.SmartPassed})} // A first-ever sighting of 8 sectors is Figyelmeztetés, not Hiba. The real drive did exactly this on
_ = s.RunDiskHealthCheck(ctx) // baseline OK // 11 Aug 12:28 and had cleared completely by 13:28.
*payload = []agentapi.DiskInfo{physDisk("sdb", &agentapi.SmartSummary{Health: agentapi.SmartPassed, PendingSectors: smartPtr(5)})} //
_ = s.RunDiskHealthCheck(ctx) // OK→Warn: emits // Red-proof: make truth-table row 9 return Fail → the chip reads Hiba and the kind assertion fails.
if len(*fired) != 1 { func TestDiskCheck_FirstSightingIsWarnOnly(t *testing.T) {
t.Fatalf("expected 1 emit on degradation, got %v", *fired) h := newDiskHarness(t)
excursion := &agentapi.SmartSummary{Health: agentapi.SmartPassed,
PendingSectors: smartPtr(8), OfflineUncorrectable: smartPtr(8), ReallocatedSectors: smartPtr(0)}
h.set(physDisk("sdg", excursion))
h.run(t)
rows := h.s.diskHealthRows(context.Background())
if len(rows) != 1 || rows[0].ChipLabel != "Figyelmeztetés" {
t.Fatalf("first sighting chip = %+v, want Figyelmeztetés", rows)
} }
// Recover back to clean. rec := h.record(t, "uuid:sdg")
*payload = []agentapi.DiskInfo{physDisk("sdb", &agentapi.SmartSummary{Health: agentapi.SmartPassed})} if rec == nil {
_ = s.RunDiskHealthCheck(ctx) t.Fatal("no persisted record for the disk")
if len(*fired) != 1 { }
t.Errorf("recovery must not notify, got %v", *fired) if !rec.SawUncorrectable {
t.Error("persisted state must record SawUncorrectable=true so the NEXT check can sustain")
}
if rec.Sectors != 8 {
t.Errorf("persisted sectors = %d, want 8", rec.Sectors)
} }
} }
// Scenario C — UNKNOWN is excluded both directions: a UNKNOWN disk never baselines/emits, and an // ── Group C — Scenario C: the same disk one check later, still bad ──────────────────────────────
// OK→UNKNOWN→Warn sequence fires on the real OK→Warn (the UNKNOWN blip is ignored, not treated as a
// transition). Red-proof: the truth of "excluded both directions" — if UNKNOWN were recorded as a
// verdict, UNKNOWN→Warn would look like a degradation from a low baseline.
func TestDiskHealthCheck_UnknownExcluded(t *testing.T) {
s, fired, payload := diskCheckHarness(t)
ctx := context.Background()
// A purely-UNKNOWN disk: first run + repeat, never notifies, never records. // The count does NOT grow — only the fact that it persisted. This isolates truth-table row 6: at 8
*payload = []agentapi.DiskInfo{physDisk("sdb", &agentapi.SmartSummary{Health: agentapi.SmartUnknown})} // sectors the count backstop (64) cannot be what fires.
_ = s.RunDiskHealthCheck(ctx) //
_ = s.RunDiskHealthCheck(ctx) // Red-proof: drop the prior argument's effect (always pass a zero DiskPrior in RunDiskHealthCheck)
if len(*fired) != 0 { // → the disk stays at Figyelmeztetés forever and no event is emitted.
t.Fatalf("UNKNOWN disk must never notify, got %v", *fired) func TestDiskLadder_SustainDrivesTheEscalation(t *testing.T) {
h := newDiskHarness(t)
excursion := func() agentapi.DiskInfo {
return physDisk("sdg", &agentapi.SmartSummary{Health: agentapi.SmartPassed,
PendingSectors: smartPtr(8), OfflineUncorrectable: smartPtr(8), ReallocatedSectors: smartPtr(0)})
}
h.set(excursion())
h.run(t) // first sighting → silent baseline at Figyelmeztetés
if n := len(h.events()); n != 0 {
t.Fatalf("first sighting must be silent, got %d", n)
} }
// OK baseline, then a UNKNOWN blip, then Warn — must fire once (OK→Warn), the blip ignored. h.set(excursion())
*payload = []agentapi.DiskInfo{physDisk("sdc", &agentapi.SmartSummary{Health: agentapi.SmartPassed})} h.run(t) // SAME counters, now sustained → Hiba
_ = s.RunDiskHealthCheck(ctx) // baseline OK ev := h.events()
*payload = []agentapi.DiskInfo{physDisk("sdc", &agentapi.SmartSummary{Health: agentapi.SmartUnknown})} if len(ev) != 1 {
_ = s.RunDiskHealthCheck(ctx) // UNKNOWN blip: no change, no emit t.Fatalf("sustained excursion must emit exactly ONE event, got %d: %+v", len(ev), ev)
if len(*fired) != 0 {
t.Fatalf("UNKNOWN blip must not emit, got %v", *fired)
} }
*payload = []agentapi.DiskInfo{physDisk("sdc", &agentapi.SmartSummary{Health: agentapi.SmartPassed, PendingSectors: smartPtr(3)})} if ev[0].Kind.Severity() != "critical" {
_ = s.RunDiskHealthCheck(ctx) // OK→Warn (the blip was ignored): fire once t.Errorf("severity = %q, want critical", ev[0].Kind.Severity())
if len(*fired) != 1 { }
t.Fatalf("real OK→Warn after a UNKNOWN blip must fire once, got %v", *fired) if ev[0].Sectors != 8 {
t.Errorf("sectors = %d, want 8 — the count did not grow; SUSTAIN is what fired", ev[0].Sectors)
}
if rows := h.s.diskHealthRows(context.Background()); rows[0].ChipLabel != "Hiba" {
t.Errorf("chip = %q, want Hiba", rows[0].ChipLabel)
} }
} }
// Scenario C — the card renders gracefully with nil SMART (old agent / no data): the row shows // ── Group D — Scenario D: recovered ─────────────────────────────────────────────────────────────
// "Nincs adat" and never errors; a physical disk with no smart field is still listed.
// Red-proof: make recovery emit → an event appears where zero are expected.
func TestDiskCheck_RecoveryIsSilentAndClearsState(t *testing.T) {
h := newDiskHarness(t)
h.set(physDisk("sdg", &agentapi.SmartSummary{Health: agentapi.SmartPassed, PendingSectors: smartPtr(8)}))
h.run(t)
h.reset()
h.set(physDisk("sdg", &agentapi.SmartSummary{Health: agentapi.SmartPassed,
PendingSectors: smartPtr(0), OfflineUncorrectable: smartPtr(0), ReallocatedSectors: smartPtr(0)}))
h.run(t)
if n := len(h.events()); n != 0 {
t.Errorf("recovery must be silent, got %d event(s): %+v", n, h.events())
}
if rows := h.s.diskHealthRows(context.Background()); rows[0].ChipLabel != "Rendben" {
t.Errorf("chip = %q, want Rendben", rows[0].ChipLabel)
}
rec := h.record(t, "uuid:sdg")
if rec.SawUncorrectable {
t.Error("recovery must clear SawUncorrectable, else the disk re-escalates on the next blip")
}
}
// ── Group E — Scenario E: the flap must not spam ────────────────────────────────────────────────
// Warn (alerted) → OK → Warn again, all inside 24h: exactly ONE event across all three checks.
// Today's transition-only logic emits twice on that zero-crossing, which is how a customer learns to
// ignore the message.
//
// Red-proof: compare against the last OBSERVED verdict instead of the last ALERTED one → two events.
func TestDiskCheck_FlapDamping(t *testing.T) {
h := newDiskHarness(t)
warn := func() agentapi.DiskInfo {
return physDisk("sdb", &agentapi.SmartSummary{Health: agentapi.SmartPassed, ReallocatedSectors: smartPtr(1)})
}
clean := func() agentapi.DiskInfo {
return physDisk("sdb", &agentapi.SmartSummary{Health: agentapi.SmartPassed, ReallocatedSectors: smartPtr(0)})
}
// Baseline clean first so the Warn below is a real transition rather than a first sighting.
h.set(clean())
h.run(t)
h.set(warn())
h.run(t) // OK→Warn: the one legitimate alert
if n := len(h.events()); n != 1 {
t.Fatalf("first degradation must emit once, got %d", n)
}
h.advance(2 * time.Hour)
h.set(clean())
h.run(t) // recovery: silent
h.advance(2 * time.Hour)
h.set(warn())
h.run(t) // re-degradation to the SAME level within 24h: damped
if n := len(h.events()); n != 1 {
t.Errorf("a flap inside 24h must produce exactly ONE event, got %d: %+v", n, h.events())
}
}
// ── Group F — Scenario F: escalation always beats damping ───────────────────────────────────────
// Red-proof: let damping cover escalations (return false whenever a verdict was ever alerted) →
// zero events, and a drive that has started actually failing says nothing.
func TestDiskCheck_EscalationBeatsDamping(t *testing.T) {
h := newDiskHarness(t)
h.set(physDisk("sdb", &agentapi.SmartSummary{Health: agentapi.SmartPassed, ReallocatedSectors: smartPtr(0)}))
h.run(t)
h.set(physDisk("sdb", &agentapi.SmartSummary{Health: agentapi.SmartPassed, ReallocatedSectors: smartPtr(1)}))
h.run(t) // OK→Warn, alerted
if n := len(h.events()); n != 1 {
t.Fatalf("setup: want 1 event, got %d", n)
}
h.reset()
// Two hours later — well inside the damping window — it reaches Hiba.
h.advance(2 * time.Hour)
h.set(physDisk("sdb", &agentapi.SmartSummary{Health: agentapi.SmartFailing, ReallocatedSectors: smartPtr(1)}))
h.run(t)
ev := h.events()
if len(ev) != 1 {
t.Fatalf("an escalation inside the damping window MUST emit, got %d", len(ev))
}
if ev[0].Kind.Severity() != "critical" {
t.Errorf("severity = %q, want critical", ev[0].Kind.Severity())
}
if ev[0].Kind != notify.DiskAlertFailSelfReported {
t.Errorf("kind = %d, want DiskAlertFailSelfReported", ev[0].Kind)
}
}
// ── Group G — Scenario G: still getting worse ───────────────────────────────────────────────────
// A disk already at Hiba, last alerted at 64 sectors, now at 352, 24h later: a SECOND event naming
// the current count. Today's code emits nothing at all between 8 and 352 sectors.
//
// Red-proof: remove the re-alert branch from diskAlertDecision → one event only.
func TestDiskCheck_RealertWhenStillWorsening(t *testing.T) {
h := newDiskHarness(t)
at := func(n int) agentapi.DiskInfo {
return physDisk("sdg", &agentapi.SmartSummary{Health: agentapi.SmartPassed,
PendingSectors: smartPtr(n), OfflineUncorrectable: smartPtr(n), ReallocatedSectors: smartPtr(0)})
}
h.set(at(64))
h.run(t) // first sighting → silent (already Hiba via row 8, but baselined)
h.set(at(64))
h.run(t) // sustained → Hiba, alerted at 64
if n := len(h.events()); n != 1 {
t.Fatalf("setup: want 1 event, got %d", n)
}
h.reset()
h.advance(25 * time.Hour)
h.set(at(352))
h.run(t)
ev := h.events()
if len(ev) != 1 {
t.Fatalf("a doubling after the cooldown must re-alert, got %d event(s)", len(ev))
}
if ev[0].Kind != notify.DiskAlertFailWorsened {
t.Errorf("kind = %d, want DiskAlertFailWorsened", ev[0].Kind)
}
if ev[0].Sectors != 352 {
t.Errorf("re-alert must name the CURRENT count, got %d", ev[0].Sectors)
}
if ev[0].Kind.Severity() != "critical" {
t.Errorf("severity = %q, want critical", ev[0].Kind.Severity())
}
}
// ── Group H — Scenario H: growth below the bar, and inside the cooldown ─────────────────────────
// Red-proof: make the cooldown and the doubling bar an OR instead of an AND → an event appears.
func TestDiskCheck_NoRealertBelowBothBars(t *testing.T) {
at := func(n int) agentapi.DiskInfo {
return physDisk("sdg", &agentapi.SmartSummary{Health: agentapi.SmartPassed,
PendingSectors: smartPtr(n), OfflineUncorrectable: smartPtr(n), ReallocatedSectors: smartPtr(0)})
}
// Neither bar cleared: 352 → 360 after 2h.
h := newDiskHarness(t)
h.set(at(352))
h.run(t)
h.set(at(352))
h.run(t) // alerted at 352
h.reset()
h.advance(2 * time.Hour)
h.set(at(360))
h.run(t)
if n := len(h.events()); n != 0 {
t.Errorf("neither bar cleared → no event, got %d: %+v", n, h.events())
}
// ONLY the cooldown cleared (25h) but growth is tiny → still silent. Pins the AND.
h.advance(25 * time.Hour)
h.set(at(400))
h.run(t)
if n := len(h.events()); n != 0 {
t.Errorf("cooldown alone must not re-alert (AND, not OR), got %d: %+v", n, h.events())
}
// ONLY the doubling cleared (2h later) → still silent. Pins the other half of the AND.
h2 := newDiskHarness(t)
h2.set(at(100))
h2.run(t)
h2.set(at(100))
h2.run(t) // alerted at 100
h2.reset()
h2.advance(2 * time.Hour)
h2.set(at(400))
h2.run(t)
if n := len(h2.events()); n != 0 {
t.Errorf("doubling alone inside the cooldown must not re-alert, got %d: %+v", n, h2.events())
}
}
// ── Group I — Scenario I: heat ──────────────────────────────────────────────────────────────────
// The event half of the temperature ladder: a hot disk alerts with the temperature SHAPE, so the
// customer is told to check ventilation rather than to arrange a replacement.
//
// Red-proof: remove truth-table rows 3 and 13 → the disk reads Rendben and nothing is emitted.
func TestDiskCheck_TemperatureShape(t *testing.T) {
h := newDiskHarness(t)
cool := physDisk("sdd", &agentapi.SmartSummary{Health: agentapi.SmartPassed, TemperatureC: smartPtr(40)})
hot := physDisk("sdd", &agentapi.SmartSummary{Health: agentapi.SmartPassed, TemperatureC: smartPtr(61)})
h.set(cool)
h.run(t)
h.set(hot)
h.run(t)
ev := h.events()
if len(ev) != 1 {
t.Fatalf("overheating must emit once, got %d", len(ev))
}
if ev[0].Kind != notify.DiskAlertFailTemperature {
t.Errorf("kind = %d, want DiskAlertFailTemperature", ev[0].Kind)
}
if ev[0].TemperatureC != 61 {
t.Errorf("temperature = %d, want 61", ev[0].TemperatureC)
}
if rows := h.s.diskHealthRows(context.Background()); rows[0].ChipLabel != "Hiba" {
t.Errorf("chip = %q, want Hiba", rows[0].ChipLabel)
}
}
// ── Group J — Scenario J: no data never alarms, and never erases history ────────────────────────
// Red-proof: let UNKNOWN write or delete the record → the disk re-baselines and its history is lost,
// so the sustained escalation afterwards never fires.
func TestDiskCheck_UnknownNeverAlarmsNorErasesPrior(t *testing.T) {
h := newDiskHarness(t)
// Establish a real record with SawUncorrectable set.
h.set(physDisk("sdg", &agentapi.SmartSummary{Health: agentapi.SmartPassed, PendingSectors: smartPtr(8)}))
h.run(t)
before := h.record(t, "uuid:sdg")
if !before.SawUncorrectable {
t.Fatal("setup: expected SawUncorrectable")
}
h.reset()
// A transient unreadable SMART, and separately a nil one.
for _, sm := range []*agentapi.SmartSummary{{Health: agentapi.SmartUnknown}, nil} {
h.set(physDisk("sdg", sm))
h.run(t)
if n := len(h.events()); n != 0 {
t.Fatalf("no-data disk must never alarm, got %d", n)
}
after := h.record(t, "uuid:sdg")
if after == nil {
t.Fatal("UNKNOWN deleted the prior record — a transient blip must not erase history")
}
if after.SawUncorrectable != before.SawUncorrectable || after.Verdict != before.Verdict {
t.Errorf("UNKNOWN overwrote the prior: before=%+v after=%+v", before, after)
}
if rows := h.s.diskHealthRows(context.Background()); len(rows) != 1 || rows[0].ChipLabel != "Nincs adat" {
t.Errorf("chip = %+v, want Nincs adat", rows)
}
}
// History survived, so the real sustained escalation still fires.
h.set(physDisk("sdg", &agentapi.SmartSummary{Health: agentapi.SmartPassed, PendingSectors: smartPtr(8)}))
h.run(t)
if n := len(h.events()); n != 1 {
t.Errorf("the sustained escalation must still fire after a no-data blip, got %d", n)
}
}
// ── Group L — Scenario L: state survives a restart (SEAM TEST) ──────────────────────────────────
// Constructed through the PRODUCTION path — web.NewServer, exactly as cmd/controller/main.go builds
// it — over the same data dir, reading and writing a real file. An injected-seam test proves the
// component; it never proves the caller. This project has shipped seven built-but-never-wired
// defects, and the severity bug being fixed here is arguably the seventh.
//
// Red-proof: skip loadDiskStateLocked in RunDiskHealthCheck → the restarted controller silently
// re-baselines a disk it has already reported, and never alerts on it again. That is today's bug.
func TestDiskCheck_StateSurvivesRestart_ProductionPath(t *testing.T) {
dir := t.TempDir()
lg := log.New(io.Discard, "", 0)
sett, err := settings.Load(filepath.Join(dir, "settings.json"), lg)
if err != nil {
t.Fatalf("settings: %v", err)
}
cfg := &config.Config{}
cfg.Paths.DataDir = dir
failing := []agentapi.DiskInfo{physDisk("sdg", realDriveSmart())}
newProdServer := func() (*Server, *[]notify.DiskAlert) {
// The same call cmd/controller/main.go makes.
s := NewServer(cfg, nil, nil, nil, nil, sett, nil, nil, nil, lg, "test")
fired := &[]notify.DiskAlert{}
s.diskNotifyFn = func(a notify.DiskAlert) { *fired = append(*fired, a) }
s.disksFn = func(ctx context.Context) (agentapi.DisksResponse, error) {
return agentapi.DisksResponse{Disks: failing}, nil
}
return s, fired
}
s1, fired1 := newProdServer()
_ = s1.RunDiskHealthCheck(context.Background()) // baseline
_ = s1.RunDiskHealthCheck(context.Background()) // sustained → Hiba, one alert
if len(*fired1) != 1 {
t.Fatalf("setup: want 1 alert before the restart, got %d", len(*fired1))
}
if _, err := os.Stat(filepath.Join(dir, diskStateFileName)); err != nil {
t.Fatalf("the production path must have WRITTEN the state file: %v", err)
}
// Restart: a brand-new Server over the same data dir, disk still failing.
s2, fired2 := newProdServer()
_ = s2.RunDiskHealthCheck(context.Background())
if n := len(*fired2); n != 0 {
t.Errorf("a restarted controller must NOT re-alert an already-reported disk, got %d: %+v", n, *fired2)
}
// And it must still know the disk is bad — the chip cannot silently drop to Figyelmeztetés.
rows := s2.diskHealthRows(context.Background())
if len(rows) != 1 || rows[0].ChipLabel != "Hiba" {
t.Errorf("after restart the card = %+v, want Hiba", rows)
}
}
// A corrupt state file must LOG and fall back to "no prior" — never crash the check and never take
// the box down. The fail-safe direction is a duplicate alert, not a missing one.
func TestDiskState_CorruptFileFallsBackToNoPrior(t *testing.T) {
h := newDiskHarness(t)
if err := os.WriteFile(filepath.Join(h.dir, diskStateFileName), []byte("{not json"), 0644); err != nil {
t.Fatal(err)
}
h.set(physDisk("sdg", realDriveSmart()))
h.run(t) // must not panic; treated as first-ever → silent
if n := len(h.events()); n != 0 {
t.Errorf("corrupt state → treated as no prior → silent baseline, got %d", n)
}
// And the next check works normally, proving the check was not left wedged.
h.run(t)
if n := len(h.events()); n != 1 {
t.Errorf("check must keep working after a corrupt state file, got %d events", n)
}
}
// A disk that disappears from the report is forgotten, so a reappearance re-baselines silently
// (behaviour preserved from v0.169.0 — persisting the state must not change it).
func TestDiskCheck_DisappearedDiskIsForgotten(t *testing.T) {
h := newDiskHarness(t)
h.set(physDisk("sdg", &agentapi.SmartSummary{Health: agentapi.SmartPassed, PendingSectors: smartPtr(8)}))
h.run(t)
h.set() // drive removed
h.run(t)
if r := h.record(t, "uuid:sdg"); r != nil {
t.Errorf("a disappeared disk must be dropped, got %+v", r)
}
// Reappears, still bad: first sighting again → silent, NOT an immediate Hiba.
h.set(physDisk("sdg", &agentapi.SmartSummary{Health: agentapi.SmartPassed, PendingSectors: smartPtr(8)}))
h.run(t)
if n := len(h.events()); n != 0 {
t.Errorf("a reappearing disk must re-baseline silently, got %d", n)
}
}
// An unreachable agent changes nothing at all — no state churn, no alarm, no error.
func TestDiskCheck_UnreachableAgentIsInert(t *testing.T) {
h := newDiskHarness(t)
h.set(physDisk("sdg", &agentapi.SmartSummary{Health: agentapi.SmartPassed, PendingSectors: smartPtr(8)}))
h.run(t)
before := h.record(t, "uuid:sdg")
h.s.disksFn = func(ctx context.Context) (agentapi.DisksResponse, error) {
return agentapi.DisksResponse{}, context.DeadlineExceeded
}
if err := h.s.RunDiskHealthCheck(context.Background()); err != nil {
t.Errorf("an unreachable agent must return nil, got %v", err)
}
after := h.record(t, "uuid:sdg")
if after == nil || after.SawUncorrectable != before.SawUncorrectable {
t.Errorf("an unreachable agent must not churn state: before=%+v after=%+v", before, after)
}
if n := len(h.events()); n != 0 {
t.Errorf("an unreachable agent must not alarm, got %d", n)
}
}
// diskAlertDecision is pure — pin its table directly, including the boundaries the scenarios above
// only reach indirectly.
func TestDiskAlertDecision_Table(t *testing.T) {
base := time.Date(2026, 8, 14, 12, 0, 0, 0, time.UTC)
alertedFail := func(sectors int, ago time.Duration) *diskRecord {
return &diskRecord{Verdict: int(agentapi.DiskVerdictFail), Alerted: true,
AlertedVerdict: int(agentapi.DiskVerdictFail), AlertedSectors: sectors, AlertedAt: base.Add(-ago)}
}
cases := []struct {
name string
prev *diskRecord
v agentapi.DiskVerdict
sectors int
wantEmit bool
wantWorsened bool
}{
{"first ever, Warn", nil, agentapi.DiskVerdictWarn, 8, false, false},
{"first ever, Fail", nil, agentapi.DiskVerdictFail, 352, false, false},
{"OK is always silent", &diskRecord{Verdict: int(agentapi.DiskVerdictWarn)}, agentapi.DiskVerdictOK, 0, false, false},
{"never alerted, now Warn", &diskRecord{Verdict: int(agentapi.DiskVerdictOK)}, agentapi.DiskVerdictWarn, 8, true, false},
{"escalate Warn→Fail", &diskRecord{Verdict: int(agentapi.DiskVerdictWarn), Alerted: true,
AlertedVerdict: int(agentapi.DiskVerdictWarn), AlertedAt: base}, agentapi.DiskVerdictFail, 8, true, false},
{"steady Fail, no growth, no time", alertedFail(352, time.Hour), agentapi.DiskVerdictFail, 352, false, false},
{"exactly 2x but only 23h", alertedFail(64, 23 * time.Hour), agentapi.DiskVerdictFail, 128, false, false},
{"exactly 2x at exactly 24h", alertedFail(64, 24 * time.Hour), agentapi.DiskVerdictFail, 128, true, true},
{"25h but only 1.9x", alertedFail(64, 25 * time.Hour), agentapi.DiskVerdictFail, 121, false, false},
{"improved Fail→Warn is silent", alertedFail(352, 48 * time.Hour), agentapi.DiskVerdictWarn, 8, false, false},
{"Fail from heat (0 sectors) never re-alerts on the doubling rule",
alertedFail(0, 48 * time.Hour), agentapi.DiskVerdictFail, 0, false, false},
}
for _, c := range cases {
emit, worsened := diskAlertDecision(c.prev, c.v, c.sectors, base)
if emit != c.wantEmit || worsened != c.wantWorsened {
t.Errorf("%s: got (emit=%v worsened=%v), want (emit=%v worsened=%v)",
c.name, emit, worsened, c.wantEmit, c.wantWorsened)
}
}
}
// ── Preserved v0.169.0/v0.171.0 coverage ────────────────────────────────────────────────────────
// The card renders gracefully with nil SMART (old agent / no data): the row shows "Nincs adat" and
// never errors; a physical disk with no smart field is still listed.
func TestDiskHealthRows_NilSmart(t *testing.T) { func TestDiskHealthRows_NilSmart(t *testing.T) {
s, _, payload := diskCheckHarness(t) h := newDiskHarness(t)
*payload = []agentapi.DiskInfo{ h.set(
{Name: "sdb", BackingDevice: "/dev/sdb", Smart: nil}, // physical, no smart → Nincs adat agentapi.DiskInfo{Name: "sdb", BackingDevice: "/dev/sdb", Smart: nil}, // physical, no smart → Nincs adat
// PBS/LVM carry a default UNKNOWN SMART from the agent — must STILL be excluded (not disks). // PBS/LVM carry a default UNKNOWN SMART from the agent — must STILL be excluded (not disks).
{Name: "felhom-pbs", Type: "pbs", Smart: &agentapi.SmartSummary{Health: agentapi.SmartUnknown}}, agentapi.DiskInfo{Name: "felhom-pbs", Type: "pbs", Smart: &agentapi.SmartSummary{Health: agentapi.SmartUnknown}},
{Name: "local-lvm", Type: "lvmthin", Smart: &agentapi.SmartSummary{Health: agentapi.SmartUnknown}}, agentapi.DiskInfo{Name: "local-lvm", Type: "lvmthin", Smart: &agentapi.SmartSummary{Health: agentapi.SmartUnknown}},
{Name: "sdc", BackingDevice: "/dev/sdc", Smart: &agentapi.SmartSummary{Health: agentapi.SmartPassed, TemperatureC: smartPtr(31)}}, // Rendben, 31°C agentapi.DiskInfo{Name: "sdc", BackingDevice: "/dev/sdc", Smart: &agentapi.SmartSummary{Health: agentapi.SmartPassed, TemperatureC: smartPtr(31)}},
} )
rows := s.diskHealthRows(context.Background()) rows := h.s.diskHealthRows(context.Background())
if len(rows) != 2 { if len(rows) != 2 {
t.Fatalf("want 2 physical-disk rows (pbs+lvm excluded), got %d: %+v", len(rows), rows) t.Fatalf("want 2 physical-disk rows (pbs+lvm excluded), got %d: %+v", len(rows), rows)
} }
@@ -133,26 +639,7 @@ func TestDiskHealthRows_NilSmart(t *testing.T) {
} }
} }
// A degraded verdict is FAILING → critical (Scenario B, Hiba→critical path). // TTL cache: two card fetches inside 60s hit the agent once.
func TestDiskHealthCheck_FailingCritical(t *testing.T) {
s := testServer(t)
var crit []bool
s.diskNotifyFn = func(label string, attrs []string, critical bool) { crit = append(crit, critical) }
payload := &[]agentapi.DiskInfo{}
s.disksFn = func(ctx context.Context) (agentapi.DisksResponse, error) {
return agentapi.DisksResponse{Disks: *payload}, nil
}
ctx := context.Background()
*payload = []agentapi.DiskInfo{physDisk("sdb", &agentapi.SmartSummary{Health: agentapi.SmartPassed})}
_ = s.RunDiskHealthCheck(ctx) // baseline OK
*payload = []agentapi.DiskInfo{physDisk("sdb", &agentapi.SmartSummary{Health: agentapi.SmartFailing})}
_ = s.RunDiskHealthCheck(ctx) // OK→Fail
if len(crit) != 1 || !crit[0] {
t.Fatalf("OK→FAILING must emit one critical event, got %v", crit)
}
}
// TTL cache (Scenario A): two card fetches inside 60s hit the agent once.
func TestCachedDisks_TTL(t *testing.T) { func TestCachedDisks_TTL(t *testing.T) {
s := testServer(t) s := testServer(t)
calls := 0 calls := 0
+5 -4
View File
@@ -180,12 +180,13 @@ type Server struct {
// manual trigger goes through the loop (stop stacks → backup → resume), never a bare agent call. // manual trigger goes through the loop (stop stacks → backup → resume), never a bare agent call.
backupTrigger BackupTrigger backupTrigger BackupTrigger
// Disk-health card + 6h degradation check (v0.169.0). diskHealth holds the 60s /disks TTL cache + // Disk-health card + periodic degradation check (v0.169.0; ladder + persistence v0.215.0).
// the in-memory verdict baseline. disksFn / diskNotifyFn are test seams (nil → the real agent // diskHealth holds the 60s /disks TTL cache + the PERSISTED per-disk observation/alert history
// client Disks() / the real notifier). // (disk_health_state.go). disksFn / diskNotifyFn are test seams (nil → the real agent client
// Disks() / the real notifier).
diskHealth diskHealthState diskHealth diskHealthState
disksFn func(context.Context) (agentapi.DisksResponse, error) disksFn func(context.Context) (agentapi.DisksResponse, error)
diskNotifyFn func(label string, attrs []string, critical bool) diskNotifyFn func(notify.DiskAlert)
// tiersFn is the sibling test seam for the agent's backup-tier view (nil → the real client's // tiersFn is the sibling test seam for the agent's backup-tier view (nil → the real client's
// BackupTiers()). Added with R-114 so the backup-target state — which is the source of a // BackupTiers()). Added with R-114 so the backup-target state — which is the source of a