From 34d83f5a020b241b4abd0cf919869924a6982574 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Fri, 14 Aug 2026 08:21:55 +0200 Subject: [PATCH] =?UTF-8?q?feat(disk-health):=20poll=20hourly,=20not=206-h?= =?UTF-8?q?ourly=20=E2=80=94=20measured,=20not=20assumed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Part 4 was gated on a measurement. On demo-hp (Tier 0) the controller's real /disks fetch — fetchDisks, the same path the check uses, not the 60s card cache — costs min 0.805s / median 0.821s / max 0.841s over 10 calls, all HTTP 200, across 3 physical disk rows (2 distinct devices). Median is 6x under the 5s bar, so the <5s branch applies and the interval drops 6h -> 1h. Why it matters: the one real failing drive's benign excursion lasted about ONE HOUR and cleared completely. A 6-hourly sampler can land either side of an excursion like that, see nothing, and then catch the terminal run half a day late. The smartd history that produced the whole analysis sampled every 30 minutes and only just resolved the shape. --- controller/cmd/controller/main.go | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/controller/cmd/controller/main.go b/controller/cmd/controller/main.go index 04a9df6..3d915e0 100644 --- a/controller/cmd/controller/main.go +++ b/controller/cmd/controller/main.go @@ -1401,12 +1401,21 @@ func main() { stackMgr.RecoverMigration(ctx) webServer.SetEncryptionKey(encKey) webServer.SetAppExporter(appExporter) - // Disk-health degradation check (v0.169.0): every 6h, compare each physical disk's SMART verdict - // against the in-memory baseline and emit disk_health_degraded on a degradation only (first run - // baselines silently; recovery/UNKNOWN never notify). Only on a provisioned guest (an agent to + // Disk-health degradation check (v0.169.0; hourly + persisted state v0.215.0): compare each + // physical disk's SMART verdict against the PERSISTED per-disk record (disk_health_state.go) and + // emit disk_health_degraded per the v0.215.0 decision rules — escalation against the last ALERTED + // verdict, plus a re-alert for a disk already at Hiba that keeps worsening. A disk's first verdict + // baselines silently; recovery and UNKNOWN never notify. Only on a provisioned guest (an agent to // read /disks from); the check no-ops gracefully if the agent is unreachable. + // + // WHY HOURLY, not the original 6h: the one real failing drive's benign excursion lasted about ONE + // HOUR (11 Aug 12:28 -> 13:28, cleared completely). A 6-hourly sampler can land either side of an + // excursion like that and see nothing, then catch the terminal run half a day late. Measured on + // demo-hp 2026-08-14, the real /disks fetch costs min 0.805s / median 0.821s / max 0.841s over 10 + // calls (3 physical disk rows), so an hourly poll is ~0.02% duty — far under the 5s bar that would + // have kept this at 6h. if cfg.LocalAPI.Endpoint != "" { - sched.Every("disk-health-check", 6*time.Hour, webServer.RunDiskHealthCheck) + sched.Every("disk-health-check", 1*time.Hour, webServer.RunDiskHealthCheck) } // Browser .fab upload (v0.128.0): upload state is in-memory, so a restart strands the .part — // GC stray part files in every registered drive's exports dir at startup.