fix(disk-health): the alert that never sent — severity, a real Hiba level, and a memory that survives a restart

Three defects made the disk-health feature silent in exactly the case it
exists for. Evidence: felhom.eu documentation/audits/DIAG-smart-passed-trap-2026-08-14.md

1. SEVERITY (the one that changes whether anything arrives at all).
   NotifyDiskHealthDegraded emitted severity "warn", which is NOT in the
   hub's accepted set {info,warning,error,critical}. The hub coerced it to
   "info" (hub/internal/api/handler.go) and severityNotifies dropped it
   (hub/internal/notify/dispatcher.go), so every Figyelmeztetes-level disk
   alert was filed as an informational notice and emailed to NOBODY, on the
   customer and the operator leg alike. Now "warning". DiskAlertKind.Severity()
   is exported so the contract is checkable from any package.

2. NO LEVEL ABOVE "worth an eye". smart_status.passed CANNOT fail on
   unreadable sectors (attrs 187/197/198 all carry thresh 0 and a normalized
   value floors at 1), so Hiba was unreachable for this whole fault class.
   DiskVerdictFor now takes a DiskPrior and implements a 14-row top-down
   ladder: sustained unreadable sectors, a count too large to be a blip (64),
   unreadable+remapping together, overheating, NVMe critical flag or spent
   endurance all reach Hiba. No fourth label — predicted failure is "Hiba".

3. IT SPOKE ONCE, AND FORGOT ON RESTART. The baseline was in-memory, so a box
   that rebooted while a disk was failing never alerted again; and between 8
   and 352 sectors nothing was emitted at all. State is now persisted
   (disk-health-state.json, atomic tmp+rename), the decision compares against
   the last ALERTED verdict (collapsing flaps to one alert while letting a
   genuine escalation fire immediately), and a disk already at Hiba re-alerts
   once it has BOTH doubled its count and waited out a 24h cooldown.

The card replays the same prior the check used (diskRecord.PriorSawUncorrectable)
so the chip and the email cannot disagree — the property the shared verdict
function exists to guarantee, now pinned rather than asserted.

Tests: 12 scenario groups A-L. Group L builds the Server through web.NewServer,
the same call main.go makes, over a real file.
This commit is contained in:
2026-08-14 08:10:59 +02:00
parent 3e3ee94b7b
commit bb50e1293c
9 changed files with 1534 additions and 228 deletions
+112 -48
View File
@@ -7,13 +7,16 @@ import (
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/agentapi"
"gitea.dooplex.hu/admin/felhom-controller/internal/notify"
)
// Disk-health card + 6-hourly degradation check (v0.169.0). The card and the check share ONE data
// path (the 60s-TTL-cached /disks call) and ONE verdict function (agentapi.DiskVerdictFor), so the
// chip a customer sees and the alert they receive can never disagree. No new smartctl load — the
// agent (v0.94.0) serializes its already-computed SMART; this consumes it and feature-detects
// (nil Smart → "Nincs adat", never alarms). No global banner (CONTEXT ruling) — card + email only.
// Disk-health card + periodic degradation check (v0.169.0; ladder + persistence v0.215.0). The card
// and the check share ONE data path (the 60s-TTL-cached /disks call) and ONE verdict function
// (agentapi.DiskVerdictFor), so the chip a customer sees and the alert they receive can never
// disagree. That shared-function property is load-bearing: BOTH callers must pass the SAME prior, or
// the chip could read Hiba while the email says Figyelmeztetés. No new smartctl load — the agent
// (v0.94.0) serializes its already-computed SMART; this consumes it and feature-detects (nil Smart →
// "Nincs adat", never alarms). No global banner (CONTEXT ruling) — card + email only.
const diskCacheTTL = 60 * time.Second
@@ -23,10 +26,23 @@ type diskHealthState struct {
cacheResp agentapi.DisksResponse
cacheErr error
cacheSet bool
// baseline is the last-seen verdict per disk (in-memory only). UNKNOWN is never recorded. Lost on
// restart → the next check re-baselines silently (accepted; see CONTEXT).
baseline map[string]agentapi.DiskVerdict
baselined bool
// records is the PERSISTED per-disk observation + alert history (v0.215.0), keyed by diskKey.
// It replaced the in-memory-only baseline, whose loss on restart meant a box that rebooted while
// a disk was already failing never alerted again. UNKNOWN is never recorded and never deletes an
// existing record. See disk_health_state.go.
records map[string]*diskRecord
loaded bool
// clock is the injectable time source (nil → time.Now). The 24h re-alert cooldown is computed
// from it, so it must be injectable or the cooldown cannot be tested without sleeping.
clock func() time.Time
}
// now reads the injectable clock.
func (st *diskHealthState) now() time.Time {
if st.clock != nil {
return st.clock()
}
return time.Now()
}
// DiskHealthRow is one rendered card row.
@@ -110,12 +126,23 @@ func (s *Server) diskHealthRows(ctx context.Context) []DiskHealthRow {
if err != nil {
return nil
}
// The card must consult the SAME persisted prior the check uses, or the chip and the email can
// disagree — which is the exact property the shared verdict function exists to guarantee. Read
// only: rendering a page never writes reporting history.
s.diskHealth.mu.Lock()
s.loadDiskStateLocked()
priors := make(map[string]agentapi.DiskPrior, len(resp.Disks))
for _, d := range resp.Disks {
priors[diskKey(d)] = s.cardPriorFor(diskKey(d))
}
s.diskHealth.mu.Unlock()
var rows []DiskHealthRow
for _, d := range resp.Disks {
if !isPhysicalDisk(d) {
continue
}
v := agentapi.DiskVerdictFor(d.Smart)
v := agentapi.DiskVerdictFor(d.Smart, priors[diskKey(d)])
row := DiskHealthRow{Label: diskDisplayLabel(d), ChipLabel: v.Label(), ChipClass: diskChipClass(v)}
if d.Smart != nil && d.Smart.TemperatureC != nil {
row.Temp = strconv.Itoa(*d.Smart.TemperatureC)
@@ -136,31 +163,38 @@ func diskKey(d agentapi.DiskInfo) string {
return "name:" + d.Name
}
// RunDiskHealthCheck is the 6-hourly job. It emits disk_health_degraded ONLY on a degradation
// transition (a disk's verdict WORSENED) against the in-memory baseline. UNKNOWN is excluded both
// directions (never recorded, never a transition endpoint). The FIRST run baselines silently; a
// newly-appeared disk baselines silently; recovery (improvement) notifies nothing. Returns nil even
// when the agent is unreachable (skip quietly — no baseline churn, no alarm).
// RunDiskHealthCheck is the periodic job. It emits disk_health_degraded per the v0.215.0 decision
// rules in diskAlertDecision: escalation against the last ALERTED verdict fires immediately, a disk
// already at Hiba re-alerts when it has both doubled its unreadable-sector count AND waited out the
// 24h cooldown, and everything else is silent.
//
// Behaviours preserved verbatim from v0.169.0:
// - the first verdict ever for a disk baselines silently;
// - UNKNOWN is excluded both directions — never recorded, never a transition endpoint, and it does
// NOT delete an existing record (a transient unreadable SMART must not erase a real disk's
// history and hand it a clean slate);
// - a disk that disappears from the report is dropped, so a reappearance re-baselines silently;
// - recovery (improvement) notifies nothing;
// - an unreachable agent returns nil and changes nothing at all.
func (s *Server) RunDiskHealthCheck(ctx context.Context) error {
// Fetch FRESH (not the 60s card cache): the check runs every 6h, so it must see current SMART, and
// this keeps its transition logic independent of dashboard render timing.
// Fetch FRESH (not the 60s card cache): the check must see current SMART, and this keeps its
// decision logic independent of dashboard render timing.
resp, err := s.fetchDisks(ctx)
if err != nil {
// POSITIVE OBSERVABLE: say the check ran and why it produced nothing. An absent log line is
// equally consistent with "healthy" and "never ran".
if s.logger != nil {
s.logger.Printf("[DEBUG] [web] disk-health check skipped: agent unreachable: %v", err)
}
return nil
}
type degradation struct {
label string
attrs []string
critical bool
}
var fired []degradation
var fired []notify.DiskAlert
evaluated := 0
s.diskHealth.mu.Lock()
if s.diskHealth.baseline == nil {
s.diskHealth.baseline = map[string]agentapi.DiskVerdict{}
}
firstRun := !s.diskHealth.baselined
s.loadDiskStateLocked()
now := s.diskHealth.now()
seen := map[string]bool{}
for _, d := range resp.Disks {
if !isPhysicalDisk(d) {
@@ -168,47 +202,77 @@ func (s *Server) RunDiskHealthCheck(ctx context.Context) error {
}
key := diskKey(d)
seen[key] = true
v := agentapi.DiskVerdictFor(d.Smart)
prev := s.diskHealth.records[key]
prior := s.priorFor(key)
v := agentapi.DiskVerdictFor(d.Smart, prior)
if v == agentapi.DiskVerdictUnknown {
// Excluded both directions: don't record, don't transition, don't drop an existing baseline
// (a transient UNKNOWN blip must not erase history or fire).
// Excluded both directions: don't record, don't transition, don't drop the existing record.
continue
}
prev, had := s.diskHealth.baseline[key]
s.diskHealth.baseline[key] = v
if firstRun || !had {
continue // first verdict ever for this disk → baseline silently
evaluated++
sectors := agentapi.UncorrectableSectors(d.Smart)
emit, worsened := diskAlertDecision(prev, v, sectors, now)
// Build the successor record. The alert history is CARRIED FORWARD across an improvement —
// that is what makes flap damping work: a disk that recovers and degrades again still knows
// what it last told the customer.
rec := &diskRecord{Verdict: int(v), SawUncorrectable: sectors > 0, Sectors: sectors, ChangedAt: now,
PriorSawUncorrectable: prior.SawUncorrectable}
if prev != nil {
rec.Alerted, rec.AlertedVerdict = prev.Alerted, prev.AlertedVerdict
rec.AlertedSectors, rec.AlertedAt = prev.AlertedSectors, prev.AlertedAt
if prev.Verdict == int(v) && !prev.ChangedAt.IsZero() {
rec.ChangedAt = prev.ChangedAt // unchanged verdict keeps its original change time
}
}
if v > prev { // verdict worsened (Unknown=0 < OK=1 < Warn=2 < Fail=3; Unknown excluded above)
fired = append(fired, degradation{
label: diskDisplayLabel(d),
attrs: agentapi.DegradedAttributes(d.Smart),
critical: v == agentapi.DiskVerdictFail,
if emit {
rec.Alerted, rec.AlertedVerdict, rec.AlertedSectors, rec.AlertedAt = true, int(v), sectors, now
fired = append(fired, notify.DiskAlert{
Label: diskDisplayLabel(d),
Attributes: agentapi.DegradedAttributes(d.Smart),
Kind: diskAlertKindFor(d.Smart, v, worsened),
Sectors: sectors,
TemperatureC: temperatureOf(d.Smart),
})
}
s.diskHealth.records[key] = rec
}
// Forget disks no longer reported so a reappearance re-baselines silently.
for k := range s.diskHealth.baseline {
for k := range s.diskHealth.records {
if !seen[k] {
delete(s.diskHealth.baseline, k)
delete(s.diskHealth.records, k)
}
}
s.diskHealth.baselined = true
// ONE write per run, after every disk has been evaluated.
s.saveDiskStateLocked()
s.diskHealth.mu.Unlock()
for _, f := range fired {
s.emitDiskDegraded(f.label, f.attrs, f.critical)
// POSITIVE OBSERVABLE, every cycle: "0 alert(s)" from a check that evaluated N disks is evidence
// of health; silence is evidence of nothing.
if s.logger != nil {
s.logger.Printf("[INFO] [web] disk-health check complete: %d disk(s) evaluated, %d alert(s)", evaluated, len(fired))
}
for _, a := range fired {
s.emitDiskDegraded(a)
}
return nil
}
// emitDiskDegraded routes a degradation to the notifier (or the test seam). nil notifier → no-op.
func (s *Server) emitDiskDegraded(label string, attrs []string, critical bool) {
// temperatureOf is the disk's reported temperature, or 0 when the device reports none.
func temperatureOf(sm *agentapi.SmartSummary) int {
if sm != nil && sm.TemperatureC != nil {
return *sm.TemperatureC
}
return 0
}
// emitDiskDegraded routes an alert to the notifier (or the test seam). nil notifier → no-op.
func (s *Server) emitDiskDegraded(a notify.DiskAlert) {
if s.diskNotifyFn != nil {
s.diskNotifyFn(label, attrs, critical)
s.diskNotifyFn(a)
return
}
if s.notifier != nil {
s.notifier.NotifyDiskHealthDegraded(label, attrs, critical)
s.notifier.NotifyDiskHealthDegraded(a)
}
}
@@ -0,0 +1,251 @@
package web
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/agentapi"
"gitea.dooplex.hu/admin/felhom-controller/internal/notify"
)
// Persisted disk-health reporting state (v0.215.0).
//
// WHY IT IS PERSISTED: until v0.215.0 the baseline was in-memory only, so a controller restart while
// a disk was already failing re-baselined that disk silently — and it never alerted again. A box that
// reboots during exactly the fault this feature exists for was the one case it could not report.
//
// WHAT IT IS NOT: one small record per disk about what has been OBSERVED and REPORTED. It is NOT a
// sample history and must not grow into one — SMART sample series belong in metrics.MetricsStore
// under Phase 2/3, which is a schema migration and a wire change. Keeping this a flat file keeps a
// one-word severity fix off that critical path.
//
// Crash-safety: atomic tmp+rename is sufficient. The state describes reporting history, not a
// mutation in flight, so a lost write costs at most one re-baseline — and the failure direction is a
// DUPLICATE alert rather than a missing one.
const diskStateFileName = "disk-health-state.json"
// diskStateVersion guards the on-disk shape. An unrecognised version is treated as "no prior"
// (fail-safe) rather than mis-parsed into a verdict.
const diskStateVersion = 1
// diskRealertCooldown is the minimum gap between two alerts for a disk already at Hiba. It is ANDed
// with the doubling bar — clearing only one of them emits nothing.
const diskRealertCooldown = 24 * time.Hour
// diskRecord is everything the checker remembers about ONE disk between runs.
type diskRecord struct {
// Verdict is the last non-UNKNOWN verdict observed (agentapi.DiskVerdict as an int).
Verdict int `json:"verdict"`
// SawUncorrectable is THIS observation: it becomes the NEXT check's agentapi.DiskPrior, and is
// what turns a one-off excursion into a sustained fault.
SawUncorrectable bool `json:"saw_uncorrectable"`
// PriorSawUncorrectable is the prior that PRODUCED Verdict — i.e. the previous observation.
//
// It exists so the CARD can reach the same verdict as the check. The two run at different times
// over the same SMART: the check consumes the prior and then overwrites it with the current
// observation, so a card rendering afterwards would consume its own check's write and read one
// level too high — a disk at its first sighting of 8 sectors would show "Hiba" on the dashboard
// while the alert (correctly) said "Figyelmeztetés". Replaying the SAME prior keeps the shared
// verdict function's guarantee intact instead of merely asserting it.
PriorSawUncorrectable bool `json:"prior_saw_uncorrectable,omitempty"`
// Sectors is max(pending, offline_uncorrectable) at the last observation.
Sectors int `json:"sectors,omitempty"`
// ChangedAt is when Verdict last changed value.
ChangedAt time.Time `json:"changed_at,omitempty"`
// --- alert history: survives an intervening improvement, which is what makes flap damping work ---
// Alerted is false until this disk has ever produced an event. Distinguishes "never alerted"
// from "alerted at Nincs adat", which cannot happen but would otherwise share the zero value.
Alerted bool `json:"alerted,omitempty"`
AlertedVerdict int `json:"alerted_verdict,omitempty"`
AlertedSectors int `json:"alerted_sectors,omitempty"`
AlertedAt time.Time `json:"alerted_at,omitempty"`
}
// diskStateFile is the on-disk envelope.
type diskStateFile struct {
Version int `json:"version"`
Disks map[string]*diskRecord `json:"disks"`
}
// diskStatePath is the state file's location, or "" when no data dir is configured (unit tests and
// an unprovisioned guest). An empty path degrades to in-memory-only: correct behaviour within the
// process, simply not durable.
func (s *Server) diskStatePath() string {
if s.cfg == nil || s.cfg.Paths.DataDir == "" {
return ""
}
return filepath.Join(s.cfg.Paths.DataDir, diskStateFileName)
}
// loadDiskStateLocked populates the in-memory records from disk exactly once per process. The caller
// holds s.diskHealth.mu.
//
// NEVER FATAL. A missing file is the normal first-boot case. A corrupt or unreadable one is LOGGED
// and treated as "no prior" — the worst that costs is one silent re-baseline, whereas crashing the
// check would take away disk monitoring entirely.
func (s *Server) loadDiskStateLocked() {
if s.diskHealth.loaded {
return
}
s.diskHealth.loaded = true
if s.diskHealth.records == nil {
s.diskHealth.records = map[string]*diskRecord{}
}
path := s.diskStatePath()
if path == "" {
return
}
data, err := os.ReadFile(path)
if err != nil {
if !os.IsNotExist(err) && s.logger != nil {
s.logger.Printf("[WARN] [web] disk-health state unreadable, continuing with no prior: %v", err)
}
return
}
var f diskStateFile
if err := json.Unmarshal(data, &f); err != nil {
if s.logger != nil {
s.logger.Printf("[WARN] [web] disk-health state corrupt, continuing with no prior: %v", err)
}
return
}
if f.Version != diskStateVersion {
if s.logger != nil {
s.logger.Printf("[WARN] [web] disk-health state version %d unrecognised (want %d), continuing with no prior",
f.Version, diskStateVersion)
}
return
}
for k, r := range f.Disks {
if r != nil {
s.diskHealth.records[k] = r
}
}
if s.logger != nil {
s.logger.Printf("[DEBUG] [web] disk-health state loaded: %d disk(s)", len(s.diskHealth.records))
}
}
// saveDiskStateLocked writes the records atomically (.tmp then rename), following the shape of
// selfupdate.SaveState. Called ONCE per check run, after every disk has been evaluated — not per
// disk. Caller holds s.diskHealth.mu. Errors are logged, never returned: failing to remember is not
// a reason to stop monitoring.
func (s *Server) saveDiskStateLocked() {
path := s.diskStatePath()
if path == "" {
return
}
if err := writeDiskState(path, s.diskHealth.records); err != nil && s.logger != nil {
s.logger.Printf("[WARN] [web] disk-health state not saved (a restart will re-baseline): %v", err)
}
}
func writeDiskState(path string, records map[string]*diskRecord) error {
data, err := json.MarshalIndent(diskStateFile{Version: diskStateVersion, Disks: records}, "", " ")
if err != nil {
return fmt.Errorf("marshaling disk-health state: %w", err)
}
tmp := path + ".tmp"
if err := os.WriteFile(tmp, data, 0644); err != nil {
return fmt.Errorf("writing temp disk-health state: %w", err)
}
if err := os.Rename(tmp, path); err != nil {
os.Remove(tmp)
return fmt.Errorf("renaming disk-health state: %w", err)
}
return nil
}
// priorFor is the agentapi.DiskPrior the CHECK consumes: what the previous check observed. A zero
// DiskPrior ("nothing known") is the fail-safe — a first sighting can then only reach
// Figyelmeztetés, never Hiba. Caller holds the mutex.
func (s *Server) priorFor(key string) agentapi.DiskPrior {
if r := s.diskHealth.records[key]; r != nil {
return agentapi.DiskPrior{SawUncorrectable: r.SawUncorrectable}
}
return agentapi.DiskPrior{}
}
// cardPriorFor is the agentapi.DiskPrior the CARD replays: the prior that produced the stored
// verdict, NOT the observation that check went on to record. See diskRecord.PriorSawUncorrectable —
// using priorFor here would make the dashboard chip read one level more severe than the alert.
// Caller holds the mutex.
func (s *Server) cardPriorFor(key string) agentapi.DiskPrior {
if r := s.diskHealth.records[key]; r != nil {
return agentapi.DiskPrior{SawUncorrectable: r.PriorSawUncorrectable}
}
return agentapi.DiskPrior{}
}
// diskAlertDecision decides whether THIS observation produces an event, per the v0.215.0 rules.
// Pure: everything, including the clock reading, arrives as an argument.
//
// prev == nil -> no event (first verdict ever for this disk: baseline silently)
// v is Rendben (or better) -> no event (recovery is silent — unchanged behaviour)
// v worse than the LAST ALERTED -> EVENT. Escalation is never damped.
// v == last alerted, and v is Hiba,
// and sectors >= 2x the count at
// the last alert, and >= 24h since -> EVENT ("still getting worse")
// anything else -> no event
//
// The second return value reports whether this is the "still getting worse" re-alert, which selects
// a different message shape.
//
// WHY "worse than the last ALERTED verdict" rather than the last OBSERVED one: a disk that goes
// Figyelmeztetés -> Rendben -> Figyelmeztetés has worsened against its last observation twice, and
// today's transition-only logic emits two identical mild alerts on that zero-crossing. That is how a
// customer learns to ignore the message. Comparing against what was last REPORTED collapses the flap
// to one alert while leaving a genuine escalation (Figyelmeztetés -> Hiba) free to fire immediately.
//
// WHY nothing else emits at v == last alerted: the steady-state case and the post-improvement flap
// case are indistinguishable without tracking intervening improvements, and both must stay silent.
// Only the explicit Hiba re-alert above breaks that silence, and only when BOTH its bars are cleared.
func diskAlertDecision(prev *diskRecord, v agentapi.DiskVerdict, sectors int, now time.Time) (emit bool, worsened bool) {
if v <= agentapi.DiskVerdictOK {
return false, false // Nincs adat never reaches here; Rendben is silent
}
if prev == nil {
return false, false // first verdict ever for this disk
}
alerted := agentapi.DiskVerdictUnknown
if prev.Alerted {
alerted = agentapi.DiskVerdict(prev.AlertedVerdict)
}
if v > alerted {
return true, false // escalation — damping never applies
}
if v == alerted && v == agentapi.DiskVerdictFail &&
prev.AlertedSectors > 0 && sectors >= 2*prev.AlertedSectors &&
now.Sub(prev.AlertedAt) >= diskRealertCooldown {
return true, true // still getting worse
}
return false, false
}
// diskAlertKindFor picks the customer-facing message shape for an alert that has already been
// decided. It mirrors the truth-table order so the reason quoted to the customer is the reason the
// verdict actually fired on.
func diskAlertKindFor(sm *agentapi.SmartSummary, v agentapi.DiskVerdict, worsened bool) notify.DiskAlertKind {
if v != agentapi.DiskVerdictFail {
return notify.DiskAlertWarn
}
switch {
case sm != nil && sm.Health == agentapi.SmartFailing:
return notify.DiskAlertFailSelfReported
case sm != nil && sm.TemperatureC != nil && *sm.TemperatureC >= agentapi.TemperatureFailC:
return notify.DiskAlertFailTemperature
case agentapi.UncorrectableSectors(sm) > 0 && worsened:
return notify.DiskAlertFailWorsened
case agentapi.UncorrectableSectors(sm) > 0:
return notify.DiskAlertFailSectors
}
// Hiba with no sector count and no heat: NVMe's own critical flag, or spent rated endurance.
// Both are declarations BY THE DEVICE, so the self-reported wording is the honest one.
return notify.DiskAlertFailSelfReported
}
+585 -98
View File
@@ -2,122 +2,628 @@ package web
import (
"context"
"encoding/json"
"io"
"log"
"os"
"path/filepath"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/agentapi"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
"gitea.dooplex.hu/admin/felhom-controller/internal/notify"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// Disk-health severity ladder, event half (v0.215.0). Scenario letters refer to the task spec.
// The verdict half is pinned in internal/agentapi/diskverdict_ladder_test.go.
func smartPtr(v int) *int { return &v }
func physDisk(name string, sm *agentapi.SmartSummary) agentapi.DiskInfo {
return agentapi.DiskInfo{Name: name, BackingDevice: "/dev/" + name, DurableID: "uuid:" + name, Smart: sm}
}
// diskCheckHarness wires a Server with the disks source + notify sink seams and returns a captured
// list of emitted disk labels.
func diskCheckHarness(t *testing.T) (*Server, *[]string, *[]agentapi.DiskInfo) {
// realDriveSmart is the failing drive as captured on 2026-08-14 — PASSED, 352 unreadable sectors.
// felhom.eu/documentation/audits/fixtures/smart-ST3000VX010-failing-2026-08-14.json
func realDriveSmart() *agentapi.SmartSummary {
return &agentapi.SmartSummary{
Health: agentapi.SmartPassed,
PendingSectors: smartPtr(352),
OfflineUncorrectable: smartPtr(352),
ReallocatedSectors: smartPtr(0),
TemperatureC: smartPtr(40),
}
}
// diskHarness wires a Server with the disks source + notify sink seams, a REAL data dir (so the
// persisted state is exercised, not bypassed) and a controllable clock.
type diskHarness struct {
s *Server
fired *[]notify.DiskAlert
payload *[]agentapi.DiskInfo
dir string
clock *time.Time
}
func newDiskHarness(t *testing.T) *diskHarness {
t.Helper()
s := testServer(t)
var fired []string
payload := &[]agentapi.DiskInfo{}
s.diskNotifyFn = func(label string, attrs []string, critical bool) { fired = append(fired, label) }
dir := t.TempDir()
lg := log.New(io.Discard, "", 0)
sett, err := settings.Load(filepath.Join(dir, "settings.json"), lg)
if err != nil {
t.Fatalf("settings: %v", err)
}
cfg := &config.Config{}
cfg.Paths.DataDir = dir
s := &Server{settings: sett, logger: lg, cfg: cfg}
now := time.Date(2026, 8, 14, 12, 0, 0, 0, time.UTC)
h := &diskHarness{s: s, dir: dir, clock: &now, fired: &[]notify.DiskAlert{}, payload: &[]agentapi.DiskInfo{}}
s.diskHealth.clock = func() time.Time { return *h.clock }
s.diskNotifyFn = func(a notify.DiskAlert) { *h.fired = append(*h.fired, a) }
s.disksFn = func(ctx context.Context) (agentapi.DisksResponse, error) {
return agentapi.DisksResponse{Disks: *payload}, nil
return agentapi.DisksResponse{Disks: *h.payload}, nil
}
return s, &fired, payload
return h
}
// Scenario B — first run baselines silently; a real degradation (OK→Warn) emits exactly once; a
// steady-state re-check does not re-emit. Red-proof: remove the `firstRun || !had` guard → the first
// run emits and the "no notify on first run" assertion fails.
func TestDiskHealthCheck_DegradationOnce(t *testing.T) {
s, fired, payload := diskCheckHarness(t)
ctx := context.Background()
func (h *diskHarness) set(d ...agentapi.DiskInfo) { *h.payload = d }
func (h *diskHarness) run(t *testing.T) {
t.Helper()
if err := h.s.RunDiskHealthCheck(context.Background()); err != nil {
t.Fatalf("RunDiskHealthCheck: %v", err)
}
}
func (h *diskHarness) advance(d time.Duration) { *h.clock = h.clock.Add(d) }
func (h *diskHarness) events() []notify.DiskAlert {
return *h.fired
}
func (h *diskHarness) reset() { *h.fired = nil }
// First check: disk PASSED clean (verdict OK). Baseline only.
*payload = []agentapi.DiskInfo{physDisk("sdb", &agentapi.SmartSummary{Health: agentapi.SmartPassed})}
_ = s.RunDiskHealthCheck(ctx)
if len(*fired) != 0 {
t.Fatalf("first run must not notify, got %v", *fired)
// record reads the persisted state file straight off disk — asserting the FILE, not the in-memory
// map, because Scenario L depends on what actually landed there.
func (h *diskHarness) record(t *testing.T, key string) *diskRecord {
t.Helper()
data, err := os.ReadFile(filepath.Join(h.dir, diskStateFileName))
if err != nil {
t.Fatalf("reading state file: %v", err)
}
var f diskStateFile
if err := json.Unmarshal(data, &f); err != nil {
t.Fatalf("parsing state file: %v", err)
}
return f.Disks[key]
}
// ── Group A — Scenario A: the real drive, second observation ────────────────────────────────────
// The captured failing drive reaches Hiba, renders the crit chip, and emits exactly ONE event at
// severity "critical".
//
// Red-proof: remove truth-table row 6 (prior.SawUncorrectable) → 352 still trips row 8, so use
// TestDiskLadder_SustainDrivesTheEscalation (Group C) to isolate row 6.
func TestDiskCheck_RealDrive_HibaAndOneCriticalEvent(t *testing.T) {
h := newDiskHarness(t)
h.set(physDisk("sdg", realDriveSmart()))
h.run(t) // first ever observation: baselines silently
if n := len(h.events()); n != 0 {
t.Fatalf("first observation must be silent, got %d event(s)", n)
}
// Degrade: pending sectors 0→5 (OK→Figyelmeztetés).
*payload = []agentapi.DiskInfo{physDisk("sdb", &agentapi.SmartSummary{Health: agentapi.SmartPassed, PendingSectors: smartPtr(5)})}
_ = s.RunDiskHealthCheck(ctx)
if len(*fired) != 1 {
t.Fatalf("degradation must emit exactly once, got %v", *fired)
h.run(t) // second observation, now with prior
ev := h.events()
if len(ev) != 1 {
t.Fatalf("want exactly ONE event, got %d: %+v", len(ev), ev)
}
if ev[0].Kind.Severity() != "critical" {
t.Errorf("severity = %q, want %q", ev[0].Kind.Severity(), "critical")
}
if ev[0].Kind != notify.DiskAlertFailSectors {
t.Errorf("kind = %d, want DiskAlertFailSectors (the drive says PASSED — we must not claim it self-reported)", ev[0].Kind)
}
if ev[0].Sectors != 352 {
t.Errorf("sectors = %d, want 352", ev[0].Sectors)
}
// Steady state: still Figyelmeztetés — no repeat.
_ = s.RunDiskHealthCheck(ctx)
if len(*fired) != 1 {
t.Fatalf("steady-state degraded must not re-emit, got %v", *fired)
// The card must agree with the alert — same verdict function, same prior.
rows := h.s.diskHealthRows(context.Background())
if len(rows) != 1 {
t.Fatalf("want 1 card row, got %d", len(rows))
}
if rows[0].ChipLabel != "Hiba" {
t.Errorf("chip label = %q, want %q", rows[0].ChipLabel, "Hiba")
}
if rows[0].ChipClass != "state-text-crit" {
t.Errorf("chip class = %q, want %q", rows[0].ChipClass, "state-text-crit")
}
}
// Scenario B (cont.) — recovery (Figyelmeztetés→Rendben) notifies nothing.
func TestDiskHealthCheck_RecoverySilent(t *testing.T) {
s, fired, payload := diskCheckHarness(t)
ctx := context.Background()
// ── Group B — Scenario B: the transient that cleared (11 August) ────────────────────────────────
*payload = []agentapi.DiskInfo{physDisk("sdb", &agentapi.SmartSummary{Health: agentapi.SmartPassed})}
_ = s.RunDiskHealthCheck(ctx) // baseline OK
*payload = []agentapi.DiskInfo{physDisk("sdb", &agentapi.SmartSummary{Health: agentapi.SmartPassed, PendingSectors: smartPtr(5)})}
_ = s.RunDiskHealthCheck(ctx) // OK→Warn: emits
if len(*fired) != 1 {
t.Fatalf("expected 1 emit on degradation, got %v", *fired)
// A first-ever sighting of 8 sectors is Figyelmeztetés, not Hiba. The real drive did exactly this on
// 11 Aug 12:28 and had cleared completely by 13:28.
//
// Red-proof: make truth-table row 9 return Fail → the chip reads Hiba and the kind assertion fails.
func TestDiskCheck_FirstSightingIsWarnOnly(t *testing.T) {
h := newDiskHarness(t)
excursion := &agentapi.SmartSummary{Health: agentapi.SmartPassed,
PendingSectors: smartPtr(8), OfflineUncorrectable: smartPtr(8), ReallocatedSectors: smartPtr(0)}
h.set(physDisk("sdg", excursion))
h.run(t)
rows := h.s.diskHealthRows(context.Background())
if len(rows) != 1 || rows[0].ChipLabel != "Figyelmeztetés" {
t.Fatalf("first sighting chip = %+v, want Figyelmeztetés", rows)
}
// Recover back to clean.
*payload = []agentapi.DiskInfo{physDisk("sdb", &agentapi.SmartSummary{Health: agentapi.SmartPassed})}
_ = s.RunDiskHealthCheck(ctx)
if len(*fired) != 1 {
t.Errorf("recovery must not notify, got %v", *fired)
rec := h.record(t, "uuid:sdg")
if rec == nil {
t.Fatal("no persisted record for the disk")
}
if !rec.SawUncorrectable {
t.Error("persisted state must record SawUncorrectable=true so the NEXT check can sustain")
}
if rec.Sectors != 8 {
t.Errorf("persisted sectors = %d, want 8", rec.Sectors)
}
}
// Scenario C — UNKNOWN is excluded both directions: a UNKNOWN disk never baselines/emits, and an
// OK→UNKNOWN→Warn sequence fires on the real OK→Warn (the UNKNOWN blip is ignored, not treated as a
// transition). Red-proof: the truth of "excluded both directions" — if UNKNOWN were recorded as a
// verdict, UNKNOWN→Warn would look like a degradation from a low baseline.
func TestDiskHealthCheck_UnknownExcluded(t *testing.T) {
s, fired, payload := diskCheckHarness(t)
ctx := context.Background()
// ── Group C — Scenario C: the same disk one check later, still bad ──────────────────────────────
// A purely-UNKNOWN disk: first run + repeat, never notifies, never records.
*payload = []agentapi.DiskInfo{physDisk("sdb", &agentapi.SmartSummary{Health: agentapi.SmartUnknown})}
_ = s.RunDiskHealthCheck(ctx)
_ = s.RunDiskHealthCheck(ctx)
if len(*fired) != 0 {
t.Fatalf("UNKNOWN disk must never notify, got %v", *fired)
// The count does NOT grow — only the fact that it persisted. This isolates truth-table row 6: at 8
// sectors the count backstop (64) cannot be what fires.
//
// Red-proof: drop the prior argument's effect (always pass a zero DiskPrior in RunDiskHealthCheck)
// → the disk stays at Figyelmeztetés forever and no event is emitted.
func TestDiskLadder_SustainDrivesTheEscalation(t *testing.T) {
h := newDiskHarness(t)
excursion := func() agentapi.DiskInfo {
return physDisk("sdg", &agentapi.SmartSummary{Health: agentapi.SmartPassed,
PendingSectors: smartPtr(8), OfflineUncorrectable: smartPtr(8), ReallocatedSectors: smartPtr(0)})
}
h.set(excursion())
h.run(t) // first sighting → silent baseline at Figyelmeztetés
if n := len(h.events()); n != 0 {
t.Fatalf("first sighting must be silent, got %d", n)
}
// OK baseline, then a UNKNOWN blip, then Warn — must fire once (OK→Warn), the blip ignored.
*payload = []agentapi.DiskInfo{physDisk("sdc", &agentapi.SmartSummary{Health: agentapi.SmartPassed})}
_ = s.RunDiskHealthCheck(ctx) // baseline OK
*payload = []agentapi.DiskInfo{physDisk("sdc", &agentapi.SmartSummary{Health: agentapi.SmartUnknown})}
_ = s.RunDiskHealthCheck(ctx) // UNKNOWN blip: no change, no emit
if len(*fired) != 0 {
t.Fatalf("UNKNOWN blip must not emit, got %v", *fired)
h.set(excursion())
h.run(t) // SAME counters, now sustained → Hiba
ev := h.events()
if len(ev) != 1 {
t.Fatalf("sustained excursion must emit exactly ONE event, got %d: %+v", len(ev), ev)
}
*payload = []agentapi.DiskInfo{physDisk("sdc", &agentapi.SmartSummary{Health: agentapi.SmartPassed, PendingSectors: smartPtr(3)})}
_ = s.RunDiskHealthCheck(ctx) // OK→Warn (the blip was ignored): fire once
if len(*fired) != 1 {
t.Fatalf("real OK→Warn after a UNKNOWN blip must fire once, got %v", *fired)
if ev[0].Kind.Severity() != "critical" {
t.Errorf("severity = %q, want critical", ev[0].Kind.Severity())
}
if ev[0].Sectors != 8 {
t.Errorf("sectors = %d, want 8 — the count did not grow; SUSTAIN is what fired", ev[0].Sectors)
}
if rows := h.s.diskHealthRows(context.Background()); rows[0].ChipLabel != "Hiba" {
t.Errorf("chip = %q, want Hiba", rows[0].ChipLabel)
}
}
// Scenario C — the card renders gracefully with nil SMART (old agent / no data): the row shows
// "Nincs adat" and never errors; a physical disk with no smart field is still listed.
// ── Group D — Scenario D: recovered ─────────────────────────────────────────────────────────────
// Red-proof: make recovery emit → an event appears where zero are expected.
func TestDiskCheck_RecoveryIsSilentAndClearsState(t *testing.T) {
h := newDiskHarness(t)
h.set(physDisk("sdg", &agentapi.SmartSummary{Health: agentapi.SmartPassed, PendingSectors: smartPtr(8)}))
h.run(t)
h.reset()
h.set(physDisk("sdg", &agentapi.SmartSummary{Health: agentapi.SmartPassed,
PendingSectors: smartPtr(0), OfflineUncorrectable: smartPtr(0), ReallocatedSectors: smartPtr(0)}))
h.run(t)
if n := len(h.events()); n != 0 {
t.Errorf("recovery must be silent, got %d event(s): %+v", n, h.events())
}
if rows := h.s.diskHealthRows(context.Background()); rows[0].ChipLabel != "Rendben" {
t.Errorf("chip = %q, want Rendben", rows[0].ChipLabel)
}
rec := h.record(t, "uuid:sdg")
if rec.SawUncorrectable {
t.Error("recovery must clear SawUncorrectable, else the disk re-escalates on the next blip")
}
}
// ── Group E — Scenario E: the flap must not spam ────────────────────────────────────────────────
// Warn (alerted) → OK → Warn again, all inside 24h: exactly ONE event across all three checks.
// Today's transition-only logic emits twice on that zero-crossing, which is how a customer learns to
// ignore the message.
//
// Red-proof: compare against the last OBSERVED verdict instead of the last ALERTED one → two events.
func TestDiskCheck_FlapDamping(t *testing.T) {
h := newDiskHarness(t)
warn := func() agentapi.DiskInfo {
return physDisk("sdb", &agentapi.SmartSummary{Health: agentapi.SmartPassed, ReallocatedSectors: smartPtr(1)})
}
clean := func() agentapi.DiskInfo {
return physDisk("sdb", &agentapi.SmartSummary{Health: agentapi.SmartPassed, ReallocatedSectors: smartPtr(0)})
}
// Baseline clean first so the Warn below is a real transition rather than a first sighting.
h.set(clean())
h.run(t)
h.set(warn())
h.run(t) // OK→Warn: the one legitimate alert
if n := len(h.events()); n != 1 {
t.Fatalf("first degradation must emit once, got %d", n)
}
h.advance(2 * time.Hour)
h.set(clean())
h.run(t) // recovery: silent
h.advance(2 * time.Hour)
h.set(warn())
h.run(t) // re-degradation to the SAME level within 24h: damped
if n := len(h.events()); n != 1 {
t.Errorf("a flap inside 24h must produce exactly ONE event, got %d: %+v", n, h.events())
}
}
// ── Group F — Scenario F: escalation always beats damping ───────────────────────────────────────
// Red-proof: let damping cover escalations (return false whenever a verdict was ever alerted) →
// zero events, and a drive that has started actually failing says nothing.
func TestDiskCheck_EscalationBeatsDamping(t *testing.T) {
h := newDiskHarness(t)
h.set(physDisk("sdb", &agentapi.SmartSummary{Health: agentapi.SmartPassed, ReallocatedSectors: smartPtr(0)}))
h.run(t)
h.set(physDisk("sdb", &agentapi.SmartSummary{Health: agentapi.SmartPassed, ReallocatedSectors: smartPtr(1)}))
h.run(t) // OK→Warn, alerted
if n := len(h.events()); n != 1 {
t.Fatalf("setup: want 1 event, got %d", n)
}
h.reset()
// Two hours later — well inside the damping window — it reaches Hiba.
h.advance(2 * time.Hour)
h.set(physDisk("sdb", &agentapi.SmartSummary{Health: agentapi.SmartFailing, ReallocatedSectors: smartPtr(1)}))
h.run(t)
ev := h.events()
if len(ev) != 1 {
t.Fatalf("an escalation inside the damping window MUST emit, got %d", len(ev))
}
if ev[0].Kind.Severity() != "critical" {
t.Errorf("severity = %q, want critical", ev[0].Kind.Severity())
}
if ev[0].Kind != notify.DiskAlertFailSelfReported {
t.Errorf("kind = %d, want DiskAlertFailSelfReported", ev[0].Kind)
}
}
// ── Group G — Scenario G: still getting worse ───────────────────────────────────────────────────
// A disk already at Hiba, last alerted at 64 sectors, now at 352, 24h later: a SECOND event naming
// the current count. Today's code emits nothing at all between 8 and 352 sectors.
//
// Red-proof: remove the re-alert branch from diskAlertDecision → one event only.
func TestDiskCheck_RealertWhenStillWorsening(t *testing.T) {
h := newDiskHarness(t)
at := func(n int) agentapi.DiskInfo {
return physDisk("sdg", &agentapi.SmartSummary{Health: agentapi.SmartPassed,
PendingSectors: smartPtr(n), OfflineUncorrectable: smartPtr(n), ReallocatedSectors: smartPtr(0)})
}
h.set(at(64))
h.run(t) // first sighting → silent (already Hiba via row 8, but baselined)
h.set(at(64))
h.run(t) // sustained → Hiba, alerted at 64
if n := len(h.events()); n != 1 {
t.Fatalf("setup: want 1 event, got %d", n)
}
h.reset()
h.advance(25 * time.Hour)
h.set(at(352))
h.run(t)
ev := h.events()
if len(ev) != 1 {
t.Fatalf("a doubling after the cooldown must re-alert, got %d event(s)", len(ev))
}
if ev[0].Kind != notify.DiskAlertFailWorsened {
t.Errorf("kind = %d, want DiskAlertFailWorsened", ev[0].Kind)
}
if ev[0].Sectors != 352 {
t.Errorf("re-alert must name the CURRENT count, got %d", ev[0].Sectors)
}
if ev[0].Kind.Severity() != "critical" {
t.Errorf("severity = %q, want critical", ev[0].Kind.Severity())
}
}
// ── Group H — Scenario H: growth below the bar, and inside the cooldown ─────────────────────────
// Red-proof: make the cooldown and the doubling bar an OR instead of an AND → an event appears.
func TestDiskCheck_NoRealertBelowBothBars(t *testing.T) {
at := func(n int) agentapi.DiskInfo {
return physDisk("sdg", &agentapi.SmartSummary{Health: agentapi.SmartPassed,
PendingSectors: smartPtr(n), OfflineUncorrectable: smartPtr(n), ReallocatedSectors: smartPtr(0)})
}
// Neither bar cleared: 352 → 360 after 2h.
h := newDiskHarness(t)
h.set(at(352))
h.run(t)
h.set(at(352))
h.run(t) // alerted at 352
h.reset()
h.advance(2 * time.Hour)
h.set(at(360))
h.run(t)
if n := len(h.events()); n != 0 {
t.Errorf("neither bar cleared → no event, got %d: %+v", n, h.events())
}
// ONLY the cooldown cleared (25h) but growth is tiny → still silent. Pins the AND.
h.advance(25 * time.Hour)
h.set(at(400))
h.run(t)
if n := len(h.events()); n != 0 {
t.Errorf("cooldown alone must not re-alert (AND, not OR), got %d: %+v", n, h.events())
}
// ONLY the doubling cleared (2h later) → still silent. Pins the other half of the AND.
h2 := newDiskHarness(t)
h2.set(at(100))
h2.run(t)
h2.set(at(100))
h2.run(t) // alerted at 100
h2.reset()
h2.advance(2 * time.Hour)
h2.set(at(400))
h2.run(t)
if n := len(h2.events()); n != 0 {
t.Errorf("doubling alone inside the cooldown must not re-alert, got %d: %+v", n, h2.events())
}
}
// ── Group I — Scenario I: heat ──────────────────────────────────────────────────────────────────
// The event half of the temperature ladder: a hot disk alerts with the temperature SHAPE, so the
// customer is told to check ventilation rather than to arrange a replacement.
//
// Red-proof: remove truth-table rows 3 and 13 → the disk reads Rendben and nothing is emitted.
func TestDiskCheck_TemperatureShape(t *testing.T) {
h := newDiskHarness(t)
cool := physDisk("sdd", &agentapi.SmartSummary{Health: agentapi.SmartPassed, TemperatureC: smartPtr(40)})
hot := physDisk("sdd", &agentapi.SmartSummary{Health: agentapi.SmartPassed, TemperatureC: smartPtr(61)})
h.set(cool)
h.run(t)
h.set(hot)
h.run(t)
ev := h.events()
if len(ev) != 1 {
t.Fatalf("overheating must emit once, got %d", len(ev))
}
if ev[0].Kind != notify.DiskAlertFailTemperature {
t.Errorf("kind = %d, want DiskAlertFailTemperature", ev[0].Kind)
}
if ev[0].TemperatureC != 61 {
t.Errorf("temperature = %d, want 61", ev[0].TemperatureC)
}
if rows := h.s.diskHealthRows(context.Background()); rows[0].ChipLabel != "Hiba" {
t.Errorf("chip = %q, want Hiba", rows[0].ChipLabel)
}
}
// ── Group J — Scenario J: no data never alarms, and never erases history ────────────────────────
// Red-proof: let UNKNOWN write or delete the record → the disk re-baselines and its history is lost,
// so the sustained escalation afterwards never fires.
func TestDiskCheck_UnknownNeverAlarmsNorErasesPrior(t *testing.T) {
h := newDiskHarness(t)
// Establish a real record with SawUncorrectable set.
h.set(physDisk("sdg", &agentapi.SmartSummary{Health: agentapi.SmartPassed, PendingSectors: smartPtr(8)}))
h.run(t)
before := h.record(t, "uuid:sdg")
if !before.SawUncorrectable {
t.Fatal("setup: expected SawUncorrectable")
}
h.reset()
// A transient unreadable SMART, and separately a nil one.
for _, sm := range []*agentapi.SmartSummary{{Health: agentapi.SmartUnknown}, nil} {
h.set(physDisk("sdg", sm))
h.run(t)
if n := len(h.events()); n != 0 {
t.Fatalf("no-data disk must never alarm, got %d", n)
}
after := h.record(t, "uuid:sdg")
if after == nil {
t.Fatal("UNKNOWN deleted the prior record — a transient blip must not erase history")
}
if after.SawUncorrectable != before.SawUncorrectable || after.Verdict != before.Verdict {
t.Errorf("UNKNOWN overwrote the prior: before=%+v after=%+v", before, after)
}
if rows := h.s.diskHealthRows(context.Background()); len(rows) != 1 || rows[0].ChipLabel != "Nincs adat" {
t.Errorf("chip = %+v, want Nincs adat", rows)
}
}
// History survived, so the real sustained escalation still fires.
h.set(physDisk("sdg", &agentapi.SmartSummary{Health: agentapi.SmartPassed, PendingSectors: smartPtr(8)}))
h.run(t)
if n := len(h.events()); n != 1 {
t.Errorf("the sustained escalation must still fire after a no-data blip, got %d", n)
}
}
// ── Group L — Scenario L: state survives a restart (SEAM TEST) ──────────────────────────────────
// Constructed through the PRODUCTION path — web.NewServer, exactly as cmd/controller/main.go builds
// it — over the same data dir, reading and writing a real file. An injected-seam test proves the
// component; it never proves the caller. This project has shipped seven built-but-never-wired
// defects, and the severity bug being fixed here is arguably the seventh.
//
// Red-proof: skip loadDiskStateLocked in RunDiskHealthCheck → the restarted controller silently
// re-baselines a disk it has already reported, and never alerts on it again. That is today's bug.
func TestDiskCheck_StateSurvivesRestart_ProductionPath(t *testing.T) {
dir := t.TempDir()
lg := log.New(io.Discard, "", 0)
sett, err := settings.Load(filepath.Join(dir, "settings.json"), lg)
if err != nil {
t.Fatalf("settings: %v", err)
}
cfg := &config.Config{}
cfg.Paths.DataDir = dir
failing := []agentapi.DiskInfo{physDisk("sdg", realDriveSmart())}
newProdServer := func() (*Server, *[]notify.DiskAlert) {
// The same call cmd/controller/main.go makes.
s := NewServer(cfg, nil, nil, nil, nil, sett, nil, nil, nil, lg, "test")
fired := &[]notify.DiskAlert{}
s.diskNotifyFn = func(a notify.DiskAlert) { *fired = append(*fired, a) }
s.disksFn = func(ctx context.Context) (agentapi.DisksResponse, error) {
return agentapi.DisksResponse{Disks: failing}, nil
}
return s, fired
}
s1, fired1 := newProdServer()
_ = s1.RunDiskHealthCheck(context.Background()) // baseline
_ = s1.RunDiskHealthCheck(context.Background()) // sustained → Hiba, one alert
if len(*fired1) != 1 {
t.Fatalf("setup: want 1 alert before the restart, got %d", len(*fired1))
}
if _, err := os.Stat(filepath.Join(dir, diskStateFileName)); err != nil {
t.Fatalf("the production path must have WRITTEN the state file: %v", err)
}
// Restart: a brand-new Server over the same data dir, disk still failing.
s2, fired2 := newProdServer()
_ = s2.RunDiskHealthCheck(context.Background())
if n := len(*fired2); n != 0 {
t.Errorf("a restarted controller must NOT re-alert an already-reported disk, got %d: %+v", n, *fired2)
}
// And it must still know the disk is bad — the chip cannot silently drop to Figyelmeztetés.
rows := s2.diskHealthRows(context.Background())
if len(rows) != 1 || rows[0].ChipLabel != "Hiba" {
t.Errorf("after restart the card = %+v, want Hiba", rows)
}
}
// A corrupt state file must LOG and fall back to "no prior" — never crash the check and never take
// the box down. The fail-safe direction is a duplicate alert, not a missing one.
func TestDiskState_CorruptFileFallsBackToNoPrior(t *testing.T) {
h := newDiskHarness(t)
if err := os.WriteFile(filepath.Join(h.dir, diskStateFileName), []byte("{not json"), 0644); err != nil {
t.Fatal(err)
}
h.set(physDisk("sdg", realDriveSmart()))
h.run(t) // must not panic; treated as first-ever → silent
if n := len(h.events()); n != 0 {
t.Errorf("corrupt state → treated as no prior → silent baseline, got %d", n)
}
// And the next check works normally, proving the check was not left wedged.
h.run(t)
if n := len(h.events()); n != 1 {
t.Errorf("check must keep working after a corrupt state file, got %d events", n)
}
}
// A disk that disappears from the report is forgotten, so a reappearance re-baselines silently
// (behaviour preserved from v0.169.0 — persisting the state must not change it).
func TestDiskCheck_DisappearedDiskIsForgotten(t *testing.T) {
h := newDiskHarness(t)
h.set(physDisk("sdg", &agentapi.SmartSummary{Health: agentapi.SmartPassed, PendingSectors: smartPtr(8)}))
h.run(t)
h.set() // drive removed
h.run(t)
if r := h.record(t, "uuid:sdg"); r != nil {
t.Errorf("a disappeared disk must be dropped, got %+v", r)
}
// Reappears, still bad: first sighting again → silent, NOT an immediate Hiba.
h.set(physDisk("sdg", &agentapi.SmartSummary{Health: agentapi.SmartPassed, PendingSectors: smartPtr(8)}))
h.run(t)
if n := len(h.events()); n != 0 {
t.Errorf("a reappearing disk must re-baseline silently, got %d", n)
}
}
// An unreachable agent changes nothing at all — no state churn, no alarm, no error.
func TestDiskCheck_UnreachableAgentIsInert(t *testing.T) {
h := newDiskHarness(t)
h.set(physDisk("sdg", &agentapi.SmartSummary{Health: agentapi.SmartPassed, PendingSectors: smartPtr(8)}))
h.run(t)
before := h.record(t, "uuid:sdg")
h.s.disksFn = func(ctx context.Context) (agentapi.DisksResponse, error) {
return agentapi.DisksResponse{}, context.DeadlineExceeded
}
if err := h.s.RunDiskHealthCheck(context.Background()); err != nil {
t.Errorf("an unreachable agent must return nil, got %v", err)
}
after := h.record(t, "uuid:sdg")
if after == nil || after.SawUncorrectable != before.SawUncorrectable {
t.Errorf("an unreachable agent must not churn state: before=%+v after=%+v", before, after)
}
if n := len(h.events()); n != 0 {
t.Errorf("an unreachable agent must not alarm, got %d", n)
}
}
// diskAlertDecision is pure — pin its table directly, including the boundaries the scenarios above
// only reach indirectly.
func TestDiskAlertDecision_Table(t *testing.T) {
base := time.Date(2026, 8, 14, 12, 0, 0, 0, time.UTC)
alertedFail := func(sectors int, ago time.Duration) *diskRecord {
return &diskRecord{Verdict: int(agentapi.DiskVerdictFail), Alerted: true,
AlertedVerdict: int(agentapi.DiskVerdictFail), AlertedSectors: sectors, AlertedAt: base.Add(-ago)}
}
cases := []struct {
name string
prev *diskRecord
v agentapi.DiskVerdict
sectors int
wantEmit bool
wantWorsened bool
}{
{"first ever, Warn", nil, agentapi.DiskVerdictWarn, 8, false, false},
{"first ever, Fail", nil, agentapi.DiskVerdictFail, 352, false, false},
{"OK is always silent", &diskRecord{Verdict: int(agentapi.DiskVerdictWarn)}, agentapi.DiskVerdictOK, 0, false, false},
{"never alerted, now Warn", &diskRecord{Verdict: int(agentapi.DiskVerdictOK)}, agentapi.DiskVerdictWarn, 8, true, false},
{"escalate Warn→Fail", &diskRecord{Verdict: int(agentapi.DiskVerdictWarn), Alerted: true,
AlertedVerdict: int(agentapi.DiskVerdictWarn), AlertedAt: base}, agentapi.DiskVerdictFail, 8, true, false},
{"steady Fail, no growth, no time", alertedFail(352, time.Hour), agentapi.DiskVerdictFail, 352, false, false},
{"exactly 2x but only 23h", alertedFail(64, 23 * time.Hour), agentapi.DiskVerdictFail, 128, false, false},
{"exactly 2x at exactly 24h", alertedFail(64, 24 * time.Hour), agentapi.DiskVerdictFail, 128, true, true},
{"25h but only 1.9x", alertedFail(64, 25 * time.Hour), agentapi.DiskVerdictFail, 121, false, false},
{"improved Fail→Warn is silent", alertedFail(352, 48 * time.Hour), agentapi.DiskVerdictWarn, 8, false, false},
{"Fail from heat (0 sectors) never re-alerts on the doubling rule",
alertedFail(0, 48 * time.Hour), agentapi.DiskVerdictFail, 0, false, false},
}
for _, c := range cases {
emit, worsened := diskAlertDecision(c.prev, c.v, c.sectors, base)
if emit != c.wantEmit || worsened != c.wantWorsened {
t.Errorf("%s: got (emit=%v worsened=%v), want (emit=%v worsened=%v)",
c.name, emit, worsened, c.wantEmit, c.wantWorsened)
}
}
}
// ── Preserved v0.169.0/v0.171.0 coverage ────────────────────────────────────────────────────────
// The card renders gracefully with nil SMART (old agent / no data): the row shows "Nincs adat" and
// never errors; a physical disk with no smart field is still listed.
func TestDiskHealthRows_NilSmart(t *testing.T) {
s, _, payload := diskCheckHarness(t)
*payload = []agentapi.DiskInfo{
{Name: "sdb", BackingDevice: "/dev/sdb", Smart: nil}, // physical, no smart → Nincs adat
h := newDiskHarness(t)
h.set(
agentapi.DiskInfo{Name: "sdb", BackingDevice: "/dev/sdb", Smart: nil}, // physical, no smart → Nincs adat
// PBS/LVM carry a default UNKNOWN SMART from the agent — must STILL be excluded (not disks).
{Name: "felhom-pbs", Type: "pbs", Smart: &agentapi.SmartSummary{Health: agentapi.SmartUnknown}},
{Name: "local-lvm", Type: "lvmthin", Smart: &agentapi.SmartSummary{Health: agentapi.SmartUnknown}},
{Name: "sdc", BackingDevice: "/dev/sdc", Smart: &agentapi.SmartSummary{Health: agentapi.SmartPassed, TemperatureC: smartPtr(31)}}, // Rendben, 31°C
}
rows := s.diskHealthRows(context.Background())
agentapi.DiskInfo{Name: "felhom-pbs", Type: "pbs", Smart: &agentapi.SmartSummary{Health: agentapi.SmartUnknown}},
agentapi.DiskInfo{Name: "local-lvm", Type: "lvmthin", Smart: &agentapi.SmartSummary{Health: agentapi.SmartUnknown}},
agentapi.DiskInfo{Name: "sdc", BackingDevice: "/dev/sdc", Smart: &agentapi.SmartSummary{Health: agentapi.SmartPassed, TemperatureC: smartPtr(31)}},
)
rows := h.s.diskHealthRows(context.Background())
if len(rows) != 2 {
t.Fatalf("want 2 physical-disk rows (pbs+lvm excluded), got %d: %+v", len(rows), rows)
}
@@ -133,26 +639,7 @@ func TestDiskHealthRows_NilSmart(t *testing.T) {
}
}
// A degraded verdict is FAILING → critical (Scenario B, Hiba→critical path).
func TestDiskHealthCheck_FailingCritical(t *testing.T) {
s := testServer(t)
var crit []bool
s.diskNotifyFn = func(label string, attrs []string, critical bool) { crit = append(crit, critical) }
payload := &[]agentapi.DiskInfo{}
s.disksFn = func(ctx context.Context) (agentapi.DisksResponse, error) {
return agentapi.DisksResponse{Disks: *payload}, nil
}
ctx := context.Background()
*payload = []agentapi.DiskInfo{physDisk("sdb", &agentapi.SmartSummary{Health: agentapi.SmartPassed})}
_ = s.RunDiskHealthCheck(ctx) // baseline OK
*payload = []agentapi.DiskInfo{physDisk("sdb", &agentapi.SmartSummary{Health: agentapi.SmartFailing})}
_ = s.RunDiskHealthCheck(ctx) // OK→Fail
if len(crit) != 1 || !crit[0] {
t.Fatalf("OK→FAILING must emit one critical event, got %v", crit)
}
}
// TTL cache (Scenario A): two card fetches inside 60s hit the agent once.
// TTL cache: two card fetches inside 60s hit the agent once.
func TestCachedDisks_TTL(t *testing.T) {
s := testServer(t)
calls := 0
+5 -4
View File
@@ -180,12 +180,13 @@ type Server struct {
// manual trigger goes through the loop (stop stacks → backup → resume), never a bare agent call.
backupTrigger BackupTrigger
// Disk-health card + 6h degradation check (v0.169.0). diskHealth holds the 60s /disks TTL cache +
// the in-memory verdict baseline. disksFn / diskNotifyFn are test seams (nil → the real agent
// client Disks() / the real notifier).
// Disk-health card + periodic degradation check (v0.169.0; ladder + persistence v0.215.0).
// diskHealth holds the 60s /disks TTL cache + the PERSISTED per-disk observation/alert history
// (disk_health_state.go). disksFn / diskNotifyFn are test seams (nil → the real agent client
// Disks() / the real notifier).
diskHealth diskHealthState
disksFn func(context.Context) (agentapi.DisksResponse, error)
diskNotifyFn func(label string, attrs []string, critical bool)
diskNotifyFn func(notify.DiskAlert)
// tiersFn is the sibling test seam for the agent's backup-tier view (nil → the real client's
// BackupTiers()). Added with R-114 so the backup-target state — which is the source of a