diff --git a/documentation/audits/DIAG-smart-passed-trap-2026-08-14.md b/documentation/audits/DIAG-smart-passed-trap-2026-08-14.md new file mode 100644 index 0000000..cd7c39b --- /dev/null +++ b/documentation/audits/DIAG-smart-passed-trap-2026-08-14.md @@ -0,0 +1,196 @@ +# DIAG — the SMART `PASSED` trap, and the disk alert that reached nobody + +**Date:** 2026-08-14 +**Drive:** Seagate `ST3000VX010-2E3166`, S/N `Z6A07P2G`, 3.0 TB, `/dev/sdg` on **DooPlex** +**Fixtures:** `fixtures/smart-ST3000VX010-failing-2026-08-14.json` (raw `smartctl -a -j /dev/sdg`), +`fixtures/smartd-history-sdg-2026-08-14.txt` (406 `smartd` journal lines for this device, 11–14 Aug) +**Status:** the three controller defects named here are FIXED in controller **v0.215.0**; the +counterfactual in §5 is derived from source, **not** reproduced live. + +This is the project's first genuinely failing disk. Before it, the capability map recorded the +disk-failure scenario as *"Healthy path only — a genuinely failing disk has never been seen."* + +--- + +## 1. What happened + +On **11 August 12:28** a 3 TB drive in DooPlex reported its first unreadable sectors. By **13 August +21:58** it was at 360, and it was taking down a running service. Throughout the entire episode the +drive's own overall self-assessment read **`PASSED`**, and it still does. + +`smartd` was running the whole time and mailed **local root** — a mailbox nobody reads. The failure +was actually found by a crashlooping pod, not by any alert. + +--- + +## 2. The mechanism — `smart_status.passed` cannot fail on unreadable sectors + +Measured, from the committed fixture: + +| ID | Attribute | value | worst | thresh | raw | +|-----|--------------------------|-------|-------|--------|--------| +| 5 | `Reallocated_Sector_Ct` | 100 | 100 | **10** | 0 | +| 187 | `Reported_Uncorrect` | **1** | **1** | **0** | **1001** | +| 188 | `Command_Timeout` | 100 | 100 | **0** | 0 | +| 197 | `Current_Pending_Sector` | 98 | 98 | **0** | **352** | +| 198 | `Offline_Uncorrectable` | 98 | 98 | **0** | **352** | +| 199 | `UDMA_CRC_Error_Count` | 200 | 200 | **0** | 0 | + +`smart_status.passed` is false only when some attribute's **normalized value** falls **at or below** +its **threshold**. Attributes 187, 197 and 198 — the three that record unreadable sectors — all carry +`thresh: 0`. A normalized SMART value floors at 1 and cannot reach 0. + +> **Therefore `smart_status.passed` is structurally incapable of failing on unreadable sectors.** +> This is not a quirk of this drive or this vendor. It is how a zero threshold behaves. + +Attribute **187 `Reported_Uncorrect` sits at normalized `1` against threshold `0`** with a raw count +of **1001**. It is one point from failing and has no remaining point to give. + +Corroborating, from the same fixture: `ata_smart_error_log.summary.count = 1001`, power-on hours +**60505** (~6.9 years), `smartctl` exit status **64** (bit 6 — *the device error log contains +records of errors*) while `smart_status.passed` is still `true`. + +**Any monitor built on the drive's overall verdict is blind to this entire class of failure.** +Felhom already knows better — `agentapi.DiskVerdictFor` reads the raw counters — which is why it +would have noticed on 11 August, two days early. + +--- + +## 3. Unreadable sectors are not monotonic + +From `smartd-history-sdg-2026-08-14.txt`, `Current_Pending_Sector` over the episode (30-minute +sampling, host clock = CEST): + +``` +Aug 11 12:28 8 first sighting +Aug 11 12:58 16 (+8) +Aug 11 13:28 0 FULL CLEAR — "No more Currently unreadable (pending) sectors, + warning condition reset after 1 email" +Aug 11 20:28 8 returns +Aug 12 01:28 16 → 01:58 8 (-8) +Aug 12 02:28 32 → 02:58 24 (-8) +Aug 12 03:28 24 187 Reported_Uncorrect 100→97; ATA error count 0→3 +Aug 12 03:58 16 (-8) → 04:28 24 (+8) +Aug 12 10:58 32 … steady 32 for ~11h … +Aug 12 21:58 24 (-8) +Aug 13 03:28 40 (+16) → 04:28 24 (-16) +Aug 13 11:28 64 terminal run begins — never returns below 64 +Aug 13 11:58 72 12:58 80 13:28 112 15:58 120 +Aug 13 21:58 360 (+240) +Aug 13 22:28 352 (-8) … steady 352 through 14 Aug … +``` + +Two measured facts carry design weight: + +1. **The 11 August excursion cleared completely within one hour** (12:28 → 13:28). A bare `> 0` + alarm would have fired on a drive that then looked fine for seven hours. This is why the ladder + uses a *sustain* rule rather than a bare non-zero test. +2. **The benign excursion peaked at 16; the terminal run crossed 64 at 13 Aug 11:28 and never came + back.** That is the entire empirical basis for the static count threshold of 64 — see §6. + +--- + +## 4. The three controller defects + +All three are in `felhom-controller` at `3e3ee94` (v0.214.0), the tree audited here. + +### D1 — the alert carries a severity the hub does not recognise *(highest value)* + +`internal/notify/notifier.go:565` — `NotifyDiskHealthDegraded` sets: + +```go +severity := "warn" +``` + +The hub accepts an exact-match lowercase vocabulary and **coerces anything else to `info`**: + +- `felhom.eu/hub/internal/api/handler.go:2121-2126` — `case "info", "warning", "error", "critical":` + … `default: payload.Severity = "info"` +- `felhom.eu/hub/internal/notify/dispatcher.go:89-96` — `severityNotifies` returns true only for + `warning` / `error` / `critical`. + +`"warn"` is not in the accepted set. So the Figyelmeztetés-level disk alert is **stored as an +informational notice and emailed to nobody**, on the customer leg and the operator leg alike. + +The function's own doc comment reads *"The hub applies its own per-event-type cooldown"* — which +presumes it routes. An invariant asserted in a comment with no test pinning it; this project's +recurring shape. + +### D2 — no level above "worth keeping an eye on it" + +`internal/agentapi/diskverdict.go:28-41` returns `Warn` for *any* non-zero counter, and can only +reach `Fail` when `Health == "FAILING"` — which, by §2, this fault class cannot produce. A drive with +one aging sector and a drive at 352 unreadable sectors rendered the identical chip and the identical +mild sentence. + +### D3 — it speaks once, and forgets on restart + +`internal/web/disk_health.go:182` emits only on `v > prev`, against a **in-memory** baseline +(`disk_health.go:26-29`, *"Lost on restart → the next check re-baselines silently"*). Consequences: + +- Between 8 and 352 pending sectors the verdict never changes level, so **nothing further is emitted**. +- A controller restart while a disk is already bad re-baselines it silently — that disk never alerts + again. + +--- + +## 5. Counterfactual — what a customer would have received + +**Derived from the source above plus the §3 timeline. NOT reproduced live.** + +| Date/time | Drive state | Felhom verdict at v0.214.0 | Emitted | Delivered | +|-----------|-------------|-----------------------------|---------|-----------| +| 11 Aug 12:28 | pending 8 | OK → Figyelmeztetés | `disk_health_degraded`, severity `warn` | **nothing** — coerced to `info`, dropped by `severityNotifies` | +| 11 Aug 13:28 | pending 0 | Figyelmeztetés → Rendben | none (recovery is silent) | nothing | +| 11 Aug 20:28 → 13 Aug | 8 → 352 | Figyelmeztetés throughout | none (no level change) | nothing | +| 13 Aug 21:58 | pending 360 | Figyelmeztetés | none | nothing | + +> **Felhom would have emitted zero emails about this drive.** The one event it did produce was filed +> at `info` and delivered to no one. + +Note the two defects compound: even had D1 been fixed alone, the customer would have received a +single mild "Javasolt figyelemmel kísérni" at 8 sectors on 11 August and then silence through 352. + +--- + +## 6. Provenance of the thresholds chosen in v0.215.0 + +- **64 unreadable sectors → Hiba.** The observed benign excursion peaked at **16** and cleared inside + an hour; the terminal run passed **64** at 13 Aug 11:28 and never returned below it. 64 sits above + the one observed transient and below the observed terminal run. **This is a judgement from ONE + drive.** It is a static backstop and is expected to be replaced in Phase 3 by growth-rate detection + over real history. +- **Sustain before count.** The primary rule is "unreadable sectors still present at the next check"; + the count is the backstop. On this drive sustain fires **12 Aug**, the count not until **13 Aug** — + a full day earlier. The backstop exists for a box that was powered off or restarted across the + sustain window. +- **55 / 60 °C.** Adopted unchanged from the operator's existing Prometheus bands on DooPlex, so the + two systems cannot disagree about the same drive. + +--- + +## 7. Measured vs inferred + +**Measured** (reproducible from the committed fixtures): +- The attribute table, thresholds and raw values in §2; `passed: true` at 352 pending sectors. +- The full non-monotonic timeline in §3, including the one-hour full clear. +- The three defect locators in §4 — read from live source on both the controller and the hub side. + +**Inferred** (source-derived, not executed): +- The §5 counterfactual. It follows from the §4 locators and the §3 timeline; **no email path was + exercised against this drive**, and no `disk_health_degraded` event for it exists in the hub. + +**Not covered here:** +- Attributes **187**, **199** and **188** are not on the agent→controller wire today. Adding them is + Phase 2 (a declared wire change, so the hub models them in the same session under G-1). Everything + the v0.215.0 fix needs was already on the wire. +- The new Fail-from-counters path has **not** fired on real hardware — only against this fixture's + values in unit tests. + +--- + +## 8. One layer out + +`smartd` on DooPlex did its job and mailed local root, where nothing reads. That is the same shape +as D1 — a correct detection with a delivery path to nowhere — one layer outside the product. Tracked +separately as DooPlex hygiene. diff --git a/documentation/audits/fixtures/smart-ST3000VX010-failing-2026-08-14.json b/documentation/audits/fixtures/smart-ST3000VX010-failing-2026-08-14.json new file mode 100644 index 0000000..2acff34 --- /dev/null +++ b/documentation/audits/fixtures/smart-ST3000VX010-failing-2026-08-14.json @@ -0,0 +1,1025 @@ +{ + "json_format_version": [ + 1, + 0 + ], + "smartctl": { + "version": [ + 7, + 5 + ], + "pre_release": false, + "svn_revision": "5714", + "platform_info": "x86_64-linux-6.12.57+deb13-amd64", + "build_info": "(local build)", + "argv": [ + "smartctl", + "-a", + "-j", + "/dev/sdg" + ], + "drive_database_version": { + "string": "7.5/5706" + }, + "exit_status": 64 + }, + "local_time": { + "time_t": 1786686799, + "asctime": "Fri Aug 14 07:53:19 2026 CEST" + }, + "device": { + "name": "/dev/sdg", + "info_name": "/dev/sdg [SAT]", + "type": "sat", + "protocol": "ATA" + }, + "model_family": "Seagate Skyhawk", + "model_name": "ST3000VX010-2E3166", + "serial_number": "Z6A07P2G", + "wwn": { + "naa": 5, + "oui": 3152, + "id": 2475315021 + }, + "firmware_version": "CV12", + "user_capacity": { + "blocks": 5860533168, + "bytes": 3000592982016 + }, + "logical_block_size": 512, + "physical_block_size": 4096, + "rotation_rate": 5900, + "form_factor": { + "ata_value": 2, + "name": "3.5 inches" + }, + "trim": { + "supported": false + }, + "in_smartctl_database": true, + "ata_version": { + "string": "ACS-2, ACS-3 T13/2161-D revision 3b", + "major_value": 1008, + "minor_value": 31 + }, + "sata_version": { + "string": "SATA 3.1", + "value": 126 + }, + "interface_speed": { + "max": { + "sata_value": 14, + "string": "6.0 Gb/s", + "units_per_second": 60, + "bits_per_unit": 100000000 + }, + "current": { + "sata_value": 3, + "string": "6.0 Gb/s", + "units_per_second": 60, + "bits_per_unit": 100000000 + } + }, + "smart_support": { + "available": true, + "enabled": true + }, + "smart_status": { + "passed": true + }, + "ata_smart_data": { + "offline_data_collection": { + "status": { + "value": 130, + "string": "was completed without error", + "passed": true + }, + "completion_seconds": 97 + }, + "self_test": { + "status": { + "value": 0, + "string": "completed without error", + "passed": true + }, + "polling_minutes": { + "short": 1, + "extended": 373, + "conveyance": 2 + } + }, + "capabilities": { + "values": [ + 123, + 3 + ], + "exec_offline_immediate_supported": true, + "offline_is_aborted_upon_new_cmd": false, + "offline_surface_scan_supported": true, + "self_tests_supported": true, + "conveyance_self_test_supported": true, + "selective_self_test_supported": true, + "attribute_autosave_enabled": true, + "error_logging_supported": true, + "gp_logging_supported": true + } + }, + "ata_sct_capabilities": { + "value": 4281, + "error_recovery_control_supported": true, + "feature_control_supported": true, + "data_table_supported": true + }, + "ata_smart_attributes": { + "revision": 10, + "table": [ + { + "id": 1, + "name": "Raw_Read_Error_Rate", + "value": 117, + "worst": 82, + "thresh": 6, + "when_failed": "", + "flags": { + "value": 15, + "string": "POSR-- ", + "prefailure": true, + "updated_online": true, + "performance": true, + "error_rate": true, + "event_count": false, + "auto_keep": false + }, + "raw": { + "value": 131764672, + "string": "131764672" + } + }, + { + "id": 3, + "name": "Spin_Up_Time", + "value": 93, + "worst": 93, + "thresh": 0, + "when_failed": "", + "flags": { + "value": 3, + "string": "PO---- ", + "prefailure": true, + "updated_online": true, + "performance": false, + "error_rate": false, + "event_count": false, + "auto_keep": false + }, + "raw": { + "value": 0, + "string": "0" + } + }, + { + "id": 4, + "name": "Start_Stop_Count", + "value": 97, + "worst": 97, + "thresh": 20, + "when_failed": "", + "flags": { + "value": 50, + "string": "-O--CK ", + "prefailure": false, + "updated_online": true, + "performance": false, + "error_rate": false, + "event_count": true, + "auto_keep": true + }, + "raw": { + "value": 3245, + "string": "3245" + } + }, + { + "id": 5, + "name": "Reallocated_Sector_Ct", + "value": 100, + "worst": 100, + "thresh": 10, + "when_failed": "", + "flags": { + "value": 51, + "string": "PO--CK ", + "prefailure": true, + "updated_online": true, + "performance": false, + "error_rate": false, + "event_count": true, + "auto_keep": true + }, + "raw": { + "value": 0, + "string": "0" + } + }, + { + "id": 7, + "name": "Seek_Error_Rate", + "value": 79, + "worst": 60, + "thresh": 30, + "when_failed": "", + "flags": { + "value": 15, + "string": "POSR-- ", + "prefailure": true, + "updated_online": true, + "performance": true, + "error_rate": true, + "event_count": false, + "auto_keep": false + }, + "raw": { + "value": 93236838, + "string": "93236838" + } + }, + { + "id": 9, + "name": "Power_On_Hours", + "value": 31, + "worst": 31, + "thresh": 0, + "when_failed": "", + "flags": { + "value": 50, + "string": "-O--CK ", + "prefailure": false, + "updated_online": true, + "performance": false, + "error_rate": false, + "event_count": true, + "auto_keep": true + }, + "raw": { + "value": 60505, + "string": "60505h+00m+00.000s" + } + }, + { + "id": 10, + "name": "Spin_Retry_Count", + "value": 100, + "worst": 100, + "thresh": 97, + "when_failed": "", + "flags": { + "value": 19, + "string": "PO--C- ", + "prefailure": true, + "updated_online": true, + "performance": false, + "error_rate": false, + "event_count": true, + "auto_keep": false + }, + "raw": { + "value": 0, + "string": "0" + } + }, + { + "id": 12, + "name": "Power_Cycle_Count", + "value": 100, + "worst": 100, + "thresh": 20, + "when_failed": "", + "flags": { + "value": 50, + "string": "-O--CK ", + "prefailure": false, + "updated_online": true, + "performance": false, + "error_rate": false, + "event_count": true, + "auto_keep": true + }, + "raw": { + "value": 75, + "string": "75" + } + }, + { + "id": 184, + "name": "End-to-End_Error", + "value": 100, + "worst": 100, + "thresh": 99, + "when_failed": "", + "flags": { + "value": 50, + "string": "-O--CK ", + "prefailure": false, + "updated_online": true, + "performance": false, + "error_rate": false, + "event_count": true, + "auto_keep": true + }, + "raw": { + "value": 0, + "string": "0" + } + }, + { + "id": 187, + "name": "Reported_Uncorrect", + "value": 1, + "worst": 1, + "thresh": 0, + "when_failed": "", + "flags": { + "value": 50, + "string": "-O--CK ", + "prefailure": false, + "updated_online": true, + "performance": false, + "error_rate": false, + "event_count": true, + "auto_keep": true + }, + "raw": { + "value": 1001, + "string": "1001" + } + }, + { + "id": 188, + "name": "Command_Timeout", + "value": 100, + "worst": 100, + "thresh": 0, + "when_failed": "", + "flags": { + "value": 50, + "string": "-O--CK ", + "prefailure": false, + "updated_online": true, + "performance": false, + "error_rate": false, + "event_count": true, + "auto_keep": true + }, + "raw": { + "value": 0, + "string": "0" + } + }, + { + "id": 189, + "name": "High_Fly_Writes", + "value": 43, + "worst": 43, + "thresh": 0, + "when_failed": "", + "flags": { + "value": 58, + "string": "-O-RCK ", + "prefailure": false, + "updated_online": true, + "performance": false, + "error_rate": true, + "event_count": true, + "auto_keep": true + }, + "raw": { + "value": 57, + "string": "57" + } + }, + { + "id": 190, + "name": "Airflow_Temperature_Cel", + "value": 60, + "worst": 52, + "thresh": 45, + "when_failed": "", + "flags": { + "value": 34, + "string": "-O---K ", + "prefailure": false, + "updated_online": true, + "performance": false, + "error_rate": false, + "event_count": false, + "auto_keep": true + }, + "raw": { + "value": 723583016, + "string": "40 (Min/Max 33/43)" + } + }, + { + "id": 191, + "name": "G-Sense_Error_Rate", + "value": 100, + "worst": 100, + "thresh": 0, + "when_failed": "", + "flags": { + "value": 50, + "string": "-O--CK ", + "prefailure": false, + "updated_online": true, + "performance": false, + "error_rate": false, + "event_count": true, + "auto_keep": true + }, + "raw": { + "value": 0, + "string": "0" + } + }, + { + "id": 192, + "name": "Power-Off_Retract_Count", + "value": 100, + "worst": 100, + "thresh": 0, + "when_failed": "", + "flags": { + "value": 50, + "string": "-O--CK ", + "prefailure": false, + "updated_online": true, + "performance": false, + "error_rate": false, + "event_count": true, + "auto_keep": true + }, + "raw": { + "value": 33, + "string": "33" + } + }, + { + "id": 193, + "name": "Load_Cycle_Count", + "value": 99, + "worst": 99, + "thresh": 0, + "when_failed": "", + "flags": { + "value": 50, + "string": "-O--CK ", + "prefailure": false, + "updated_online": true, + "performance": false, + "error_rate": false, + "event_count": true, + "auto_keep": true + }, + "raw": { + "value": 3375, + "string": "3375" + } + }, + { + "id": 194, + "name": "Temperature_Celsius", + "value": 40, + "worst": 48, + "thresh": 0, + "when_failed": "", + "flags": { + "value": 34, + "string": "-O---K ", + "prefailure": false, + "updated_online": true, + "performance": false, + "error_rate": false, + "event_count": false, + "auto_keep": true + }, + "raw": { + "value": 94489280552, + "string": "40 (0 22 0 0 0)" + } + }, + { + "id": 197, + "name": "Current_Pending_Sector", + "value": 98, + "worst": 98, + "thresh": 0, + "when_failed": "", + "flags": { + "value": 18, + "string": "-O--C- ", + "prefailure": false, + "updated_online": true, + "performance": false, + "error_rate": false, + "event_count": true, + "auto_keep": false + }, + "raw": { + "value": 352, + "string": "352" + } + }, + { + "id": 198, + "name": "Offline_Uncorrectable", + "value": 98, + "worst": 98, + "thresh": 0, + "when_failed": "", + "flags": { + "value": 16, + "string": "----C- ", + "prefailure": false, + "updated_online": false, + "performance": false, + "error_rate": false, + "event_count": true, + "auto_keep": false + }, + "raw": { + "value": 352, + "string": "352" + } + }, + { + "id": 199, + "name": "UDMA_CRC_Error_Count", + "value": 200, + "worst": 200, + "thresh": 0, + "when_failed": "", + "flags": { + "value": 62, + "string": "-OSRCK ", + "prefailure": false, + "updated_online": true, + "performance": true, + "error_rate": true, + "event_count": true, + "auto_keep": true + }, + "raw": { + "value": 0, + "string": "0" + } + } + ] + }, + "spare_available": { + "current_percent": 100, + "threshold_percent": 10 + }, + "power_on_time": { + "hours": 60505, + "minutes": 0 + }, + "power_cycle_count": 75, + "temperature": { + "current": 40 + }, + "ata_smart_error_log": { + "summary": { + "revision": 1, + "count": 1001, + "logged_count": 5, + "table": [ + { + "error_number": 1001, + "lifetime_hours": 60496, + "completion_registers": { + "error": 64, + "status": 83, + "count": 0, + "lba": 16777215, + "device": 15 + }, + "error_description": "Error: UNC at LBA = 0x0fffffff = 268435455", + "previous_commands": [ + { + "registers": { + "command": 96, + "features": 0, + "count": 8, + "lba": 16777215, + "device": 79, + "device_control": 0 + }, + "powerup_milliseconds": 1253657807, + "command_name": "READ FPDMA QUEUED" + }, + { + "registers": { + "command": 239, + "features": 16, + "count": 2, + "lba": 0, + "device": 160, + "device_control": 0 + }, + "powerup_milliseconds": 1253657794, + "command_name": "SET FEATURES [Enable SATA feature]" + }, + { + "registers": { + "command": 39, + "features": 0, + "count": 0, + "lba": 0, + "device": 224, + "device_control": 0 + }, + "powerup_milliseconds": 1253657794, + "command_name": "READ NATIVE MAX ADDRESS EXT [OBS-ACS-3]" + }, + { + "registers": { + "command": 236, + "features": 0, + "count": 0, + "lba": 0, + "device": 160, + "device_control": 0 + }, + "powerup_milliseconds": 1253657793, + "command_name": "IDENTIFY DEVICE" + }, + { + "registers": { + "command": 239, + "features": 3, + "count": 70, + "lba": 0, + "device": 160, + "device_control": 0 + }, + "powerup_milliseconds": 1253657793, + "command_name": "SET FEATURES [Set transfer mode]" + } + ] + }, + { + "error_number": 1000, + "lifetime_hours": 60496, + "completion_registers": { + "error": 64, + "status": 83, + "count": 0, + "lba": 16777215, + "device": 15 + }, + "error_description": "Error: UNC at LBA = 0x0fffffff = 268435455", + "previous_commands": [ + { + "registers": { + "command": 96, + "features": 0, + "count": 8, + "lba": 16777215, + "device": 79, + "device_control": 0 + }, + "powerup_milliseconds": 1253653780, + "command_name": "READ FPDMA QUEUED" + }, + { + "registers": { + "command": 239, + "features": 16, + "count": 2, + "lba": 0, + "device": 160, + "device_control": 0 + }, + "powerup_milliseconds": 1253653746, + "command_name": "SET FEATURES [Enable SATA feature]" + }, + { + "registers": { + "command": 39, + "features": 0, + "count": 0, + "lba": 0, + "device": 224, + "device_control": 0 + }, + "powerup_milliseconds": 1253653745, + "command_name": "READ NATIVE MAX ADDRESS EXT [OBS-ACS-3]" + }, + { + "registers": { + "command": 236, + "features": 0, + "count": 0, + "lba": 0, + "device": 160, + "device_control": 0 + }, + "powerup_milliseconds": 1253653745, + "command_name": "IDENTIFY DEVICE" + }, + { + "registers": { + "command": 239, + "features": 3, + "count": 70, + "lba": 0, + "device": 160, + "device_control": 0 + }, + "powerup_milliseconds": 1253653745, + "command_name": "SET FEATURES [Set transfer mode]" + } + ] + }, + { + "error_number": 999, + "lifetime_hours": 60496, + "completion_registers": { + "error": 64, + "status": 83, + "count": 0, + "lba": 16777215, + "device": 15 + }, + "error_description": "Error: UNC at LBA = 0x0fffffff = 268435455", + "previous_commands": [ + { + "registers": { + "command": 96, + "features": 0, + "count": 8, + "lba": 16777215, + "device": 79, + "device_control": 0 + }, + "powerup_milliseconds": 1253649720, + "command_name": "READ FPDMA QUEUED" + }, + { + "registers": { + "command": 239, + "features": 16, + "count": 2, + "lba": 0, + "device": 160, + "device_control": 0 + }, + "powerup_milliseconds": 1253649705, + "command_name": "SET FEATURES [Enable SATA feature]" + }, + { + "registers": { + "command": 39, + "features": 0, + "count": 0, + "lba": 0, + "device": 224, + "device_control": 0 + }, + "powerup_milliseconds": 1253649705, + "command_name": "READ NATIVE MAX ADDRESS EXT [OBS-ACS-3]" + }, + { + "registers": { + "command": 236, + "features": 0, + "count": 0, + "lba": 0, + "device": 160, + "device_control": 0 + }, + "powerup_milliseconds": 1253649704, + "command_name": "IDENTIFY DEVICE" + }, + { + "registers": { + "command": 239, + "features": 3, + "count": 70, + "lba": 0, + "device": 160, + "device_control": 0 + }, + "powerup_milliseconds": 1253649704, + "command_name": "SET FEATURES [Set transfer mode]" + } + ] + }, + { + "error_number": 998, + "lifetime_hours": 60496, + "completion_registers": { + "error": 64, + "status": 83, + "count": 0, + "lba": 16777215, + "device": 15 + }, + "error_description": "Error: UNC at LBA = 0x0fffffff = 268435455", + "previous_commands": [ + { + "registers": { + "command": 96, + "features": 0, + "count": 8, + "lba": 16777215, + "device": 79, + "device_control": 0 + }, + "powerup_milliseconds": 1253645687, + "command_name": "READ FPDMA QUEUED" + }, + { + "registers": { + "command": 239, + "features": 16, + "count": 2, + "lba": 0, + "device": 160, + "device_control": 0 + }, + "powerup_milliseconds": 1253645647, + "command_name": "SET FEATURES [Enable SATA feature]" + }, + { + "registers": { + "command": 39, + "features": 0, + "count": 0, + "lba": 0, + "device": 224, + "device_control": 0 + }, + "powerup_milliseconds": 1253645647, + "command_name": "READ NATIVE MAX ADDRESS EXT [OBS-ACS-3]" + }, + { + "registers": { + "command": 236, + "features": 0, + "count": 0, + "lba": 0, + "device": 160, + "device_control": 0 + }, + "powerup_milliseconds": 1253645647, + "command_name": "IDENTIFY DEVICE" + }, + { + "registers": { + "command": 239, + "features": 3, + "count": 70, + "lba": 0, + "device": 160, + "device_control": 0 + }, + "powerup_milliseconds": 1253645647, + "command_name": "SET FEATURES [Set transfer mode]" + } + ] + }, + { + "error_number": 997, + "lifetime_hours": 60496, + "completion_registers": { + "error": 64, + "status": 83, + "count": 0, + "lba": 16777215, + "device": 15 + }, + "error_description": "Error: UNC at LBA = 0x0fffffff = 268435455", + "previous_commands": [ + { + "registers": { + "command": 96, + "features": 0, + "count": 8, + "lba": 16777215, + "device": 79, + "device_control": 0 + }, + "powerup_milliseconds": 1253641683, + "command_name": "READ FPDMA QUEUED" + }, + { + "registers": { + "command": 239, + "features": 16, + "count": 2, + "lba": 0, + "device": 160, + "device_control": 0 + }, + "powerup_milliseconds": 1253641644, + "command_name": "SET FEATURES [Enable SATA feature]" + }, + { + "registers": { + "command": 39, + "features": 0, + "count": 0, + "lba": 0, + "device": 224, + "device_control": 0 + }, + "powerup_milliseconds": 1253641643, + "command_name": "READ NATIVE MAX ADDRESS EXT [OBS-ACS-3]" + }, + { + "registers": { + "command": 236, + "features": 0, + "count": 0, + "lba": 0, + "device": 160, + "device_control": 0 + }, + "powerup_milliseconds": 1253641643, + "command_name": "IDENTIFY DEVICE" + }, + { + "registers": { + "command": 239, + "features": 3, + "count": 70, + "lba": 0, + "device": 160, + "device_control": 0 + }, + "powerup_milliseconds": 1253641643, + "command_name": "SET FEATURES [Set transfer mode]" + } + ] + } + ] + } + }, + "ata_smart_self_test_log": { + "standard": { + "revision": 1, + "count": 0 + } + }, + "ata_smart_selective_self_test_log": { + "revision": 1, + "table": [ + { + "lba_min": 0, + "lba_max": 0, + "status": { + "value": 0, + "string": "Not_testing" + } + }, + { + "lba_min": 0, + "lba_max": 0, + "status": { + "value": 0, + "string": "Not_testing" + } + }, + { + "lba_min": 0, + "lba_max": 0, + "status": { + "value": 0, + "string": "Not_testing" + } + }, + { + "lba_min": 0, + "lba_max": 0, + "status": { + "value": 0, + "string": "Not_testing" + } + }, + { + "lba_min": 0, + "lba_max": 0, + "status": { + "value": 0, + "string": "Not_testing" + } + } + ], + "flags": { + "value": 0, + "remainder_scan_enabled": false + }, + "power_up_scan_resume_minutes": 0 + } +} diff --git a/documentation/audits/fixtures/smartd-history-sdg-2026-08-14.txt b/documentation/audits/fixtures/smartd-history-sdg-2026-08-14.txt new file mode 100644 index 0000000..29e8cc4 --- /dev/null +++ b/documentation/audits/fixtures/smartd-history-sdg-2026-08-14.txt @@ -0,0 +1,406 @@ +Aug 11 01:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 61 to 60 +Aug 11 01:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 39 to 40 +Aug 11 01:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 61 +Aug 11 01:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 39 +Aug 11 02:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 61 to 60 +Aug 11 02:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 39 to 40 +Aug 11 03:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 117 to 118 +Aug 11 03:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 61 +Aug 11 03:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 39 +Aug 11 10:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 61 to 60 +Aug 11 10:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 39 to 40 +Aug 11 10:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 61 +Aug 11 10:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 39 +Aug 11 11:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 61 to 60 +Aug 11 11:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 39 to 40 +Aug 11 12:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Currently unreadable (pending) sectors +Aug 11 12:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Offline uncorrectable sectors +Aug 11 12:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 16 Currently unreadable (pending) sectors (changed +8) +Aug 11 12:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 16 Offline uncorrectable sectors (changed +8) +Aug 11 13:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], No more Currently unreadable (pending) sectors, warning condition reset after 1 email +Aug 11 13:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], No more Offline uncorrectable sectors, warning condition reset after 1 email +Aug 11 14:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 61 +Aug 11 14:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 39 +Aug 11 15:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 118 to 119 +Aug 11 15:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 61 to 60 +Aug 11 15:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 39 to 40 +Aug 11 16:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 59 +Aug 11 16:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 41 +Aug 11 17:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 60 +Aug 11 17:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 41 to 40 +Aug 11 18:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 9 Power_On_Hours changed from 32 to 31 +Aug 11 20:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Currently unreadable (pending) sectors +Aug 11 20:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Offline uncorrectable sectors +Aug 11 20:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Currently unreadable (pending) sectors +Aug 11 20:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Offline uncorrectable sectors +Aug 11 21:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Currently unreadable (pending) sectors +Aug 11 21:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Offline uncorrectable sectors +Aug 11 21:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Currently unreadable (pending) sectors +Aug 11 21:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Offline uncorrectable sectors +Aug 11 22:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Currently unreadable (pending) sectors +Aug 11 22:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Offline uncorrectable sectors +Aug 11 22:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Currently unreadable (pending) sectors +Aug 11 22:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Offline uncorrectable sectors +Aug 11 23:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Currently unreadable (pending) sectors +Aug 11 23:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Offline uncorrectable sectors +Aug 11 23:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 61 +Aug 11 23:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 39 +Aug 11 23:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Currently unreadable (pending) sectors +Aug 11 23:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Offline uncorrectable sectors +Aug 12 00:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Currently unreadable (pending) sectors +Aug 12 00:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Offline uncorrectable sectors +Aug 12 00:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Currently unreadable (pending) sectors +Aug 12 00:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Offline uncorrectable sectors +Aug 12 01:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 16 Currently unreadable (pending) sectors (changed +8) +Aug 12 01:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 16 Offline uncorrectable sectors (changed +8) +Aug 12 01:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Currently unreadable (pending) sectors (changed -8) +Aug 12 01:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Offline uncorrectable sectors (changed -8) +Aug 12 02:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors (changed +24) +Aug 12 02:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors (changed +24) +Aug 12 02:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 189 High_Fly_Writes changed from 51 to 50 +Aug 12 02:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors (changed -8) +Aug 12 02:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors (changed -8) +Aug 12 03:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 12 03:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 12 03:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 119 to 115 +Aug 12 03:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 187 Reported_Uncorrect changed from 100 to 97 +Aug 12 03:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 0 to 3 +Aug 12 03:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 16 Currently unreadable (pending) sectors (changed -8) +Aug 12 03:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 16 Offline uncorrectable sectors (changed -8) +Aug 12 03:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 189 High_Fly_Writes changed from 50 to 49 +Aug 12 04:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors (changed +8) +Aug 12 04:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors (changed +8) +Aug 12 04:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 115 to 113 +Aug 12 04:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 187 Reported_Uncorrect changed from 97 to 95 +Aug 12 04:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 189 High_Fly_Writes changed from 49 to 47 +Aug 12 04:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 3 to 5 +Aug 12 04:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 12 04:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 12 04:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 61 to 60 +Aug 12 04:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 39 to 40 +Aug 12 05:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 12 05:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 12 05:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 12 05:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 12 05:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 59 +Aug 12 05:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 41 +Aug 12 06:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 12 06:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 12 06:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 12 06:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 12 07:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 12 07:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 12 07:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 12 07:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 12 08:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 12 08:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 12 08:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 58 +Aug 12 08:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 41 to 42 +Aug 12 08:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 12 08:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 12 08:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 58 to 59 +Aug 12 08:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 42 to 41 +Aug 12 09:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 12 09:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 12 09:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 60 +Aug 12 09:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 41 to 40 +Aug 12 09:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 12 09:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 12 10:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 12 10:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 12 10:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors (changed +8) +Aug 12 10:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors (changed +8) +Aug 12 10:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 189 High_Fly_Writes changed from 47 to 46 +Aug 12 10:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 59 +Aug 12 10:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 41 +Aug 12 11:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors +Aug 12 11:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors +Aug 12 11:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors +Aug 12 11:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors +Aug 12 12:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors +Aug 12 12:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors +Aug 12 12:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 113 to 100 +Aug 12 12:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors +Aug 12 12:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors +Aug 12 12:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 100 to 101 +Aug 12 12:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 58 +Aug 12 12:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 41 to 42 +Aug 12 13:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors +Aug 12 13:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors +Aug 12 13:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 101 to 102 +Aug 12 13:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors +Aug 12 13:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors +Aug 12 14:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors +Aug 12 14:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors +Aug 12 14:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 102 to 106 +Aug 12 14:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 58 to 59 +Aug 12 14:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 42 to 41 +Aug 12 14:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors +Aug 12 14:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors +Aug 12 14:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 106 to 107 +Aug 12 14:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 60 +Aug 12 14:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 41 to 40 +Aug 12 15:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors +Aug 12 15:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors +Aug 12 15:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors +Aug 12 15:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors +Aug 12 15:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 107 to 108 +Aug 12 16:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors +Aug 12 16:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors +Aug 12 16:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 189 High_Fly_Writes changed from 46 to 45 +Aug 12 16:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors +Aug 12 16:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors +Aug 12 17:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors +Aug 12 17:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors +Aug 12 17:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors +Aug 12 17:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors +Aug 12 17:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 108 to 110 +Aug 12 17:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 59 +Aug 12 17:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 41 +Aug 12 18:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors +Aug 12 18:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors +Aug 12 18:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 110 to 111 +Aug 12 18:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 60 +Aug 12 18:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 41 to 40 +Aug 12 18:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors +Aug 12 18:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors +Aug 12 19:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors +Aug 12 19:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors +Aug 12 19:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors +Aug 12 19:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors +Aug 12 20:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors +Aug 12 20:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors +Aug 12 20:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors +Aug 12 20:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors +Aug 12 21:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors +Aug 12 21:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors +Aug 12 21:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 59 +Aug 12 21:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 41 +Aug 12 21:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors (changed -8) +Aug 12 21:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors (changed -8) +Aug 12 22:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 12 22:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 12 22:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 12 22:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 12 22:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 60 +Aug 12 22:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 41 to 40 +Aug 12 23:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 12 23:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 12 23:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 12 23:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 12 23:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 111 to 112 +Aug 13 00:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 13 00:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 13 00:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 13 00:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 13 01:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 13 01:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 13 01:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 61 +Aug 13 01:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 39 +Aug 13 01:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 13 01:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 13 02:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 13 02:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 13 02:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 112 to 113 +Aug 13 02:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 13 02:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 13 03:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 40 Currently unreadable (pending) sectors (changed +16) +Aug 13 03:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 40 Offline uncorrectable sectors (changed +16) +Aug 13 03:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 113 to 103 +Aug 13 03:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 187 Reported_Uncorrect changed from 95 to 83 +Aug 13 03:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 5 to 17 +Aug 13 03:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 40 Currently unreadable (pending) sectors +Aug 13 03:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 40 Offline uncorrectable sectors +Aug 13 04:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors (changed -16) +Aug 13 04:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors (changed -16) +Aug 13 04:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 187 Reported_Uncorrect changed from 83 to 82 +Aug 13 04:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 17 to 18 +Aug 13 04:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 13 04:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 13 05:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 13 05:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 13 05:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 103 to 104 +Aug 13 05:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 13 05:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 13 06:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 13 06:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 13 06:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 13 06:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 13 07:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 13 07:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 13 07:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 13 07:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 13 08:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 13 08:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 13 08:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 13 08:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 13 08:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 104 to 105 +Aug 13 09:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 13 09:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 13 09:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 13 09:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 13 10:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 13 10:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 13 10:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 61 to 60 +Aug 13 10:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 39 to 40 +Aug 13 10:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors +Aug 13 10:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors +Aug 13 11:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 64 Currently unreadable (pending) sectors (changed +40) +Aug 13 11:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 64 Offline uncorrectable sectors (changed +40) +Aug 13 11:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 189 High_Fly_Writes changed from 45 to 43 +Aug 13 11:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 59 +Aug 13 11:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 41 +Aug 13 11:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 72 Currently unreadable (pending) sectors (changed +8) +Aug 13 11:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 72 Offline uncorrectable sectors (changed +8) +Aug 13 12:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 72 Currently unreadable (pending) sectors +Aug 13 12:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 72 Offline uncorrectable sectors +Aug 13 12:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 80 Currently unreadable (pending) sectors (changed +8) +Aug 13 12:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 80 Offline uncorrectable sectors (changed +8) +Aug 13 13:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 112 Currently unreadable (pending) sectors (changed +32) +Aug 13 13:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 112 Offline uncorrectable sectors (changed +32) +Aug 13 13:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 105 to 104 +Aug 13 13:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 187 Reported_Uncorrect changed from 82 to 80 +Aug 13 13:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 18 to 20 +Aug 13 13:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 112 Currently unreadable (pending) sectors +Aug 13 13:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 112 Offline uncorrectable sectors +Aug 13 13:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 104 to 99 +Aug 13 13:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 187 Reported_Uncorrect changed from 80 to 25 +Aug 13 13:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 58 +Aug 13 13:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 41 to 42 +Aug 13 13:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 20 to 75 +Aug 13 14:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 112 Currently unreadable (pending) sectors +Aug 13 14:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 112 Offline uncorrectable sectors +Aug 13 14:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 99 to 96 +Aug 13 14:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 187 Reported_Uncorrect changed from 25 to 1 +Aug 13 14:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 75 to 129 +Aug 13 14:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 112 Currently unreadable (pending) sectors +Aug 13 14:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 112 Offline uncorrectable sectors +Aug 13 14:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 96 to 95 +Aug 13 14:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 129 to 184 +Aug 13 15:28:40 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 112 Currently unreadable (pending) sectors +Aug 13 15:28:40 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 112 Offline uncorrectable sectors +Aug 13 15:28:40 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 95 to 94 +Aug 13 15:28:40 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 58 to 57 +Aug 13 15:28:40 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 42 to 43 +Aug 13 15:28:40 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 184 to 239 +Aug 13 15:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Currently unreadable (pending) sectors (changed +8) +Aug 13 15:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Offline uncorrectable sectors (changed +8) +Aug 13 15:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 239 to 292 +Aug 13 16:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Currently unreadable (pending) sectors +Aug 13 16:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Offline uncorrectable sectors +Aug 13 16:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 292 to 344 +Aug 13 16:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Currently unreadable (pending) sectors +Aug 13 16:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Offline uncorrectable sectors +Aug 13 16:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 94 to 93 +Aug 13 16:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 57 to 58 +Aug 13 16:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 43 to 42 +Aug 13 16:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 344 to 399 +Aug 13 17:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Currently unreadable (pending) sectors +Aug 13 17:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Offline uncorrectable sectors +Aug 13 17:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 93 to 94 +Aug 13 17:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 58 to 59 +Aug 13 17:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 42 to 41 +Aug 13 17:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 399 to 450 +Aug 13 17:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Currently unreadable (pending) sectors +Aug 13 17:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Offline uncorrectable sectors +Aug 13 17:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 94 to 91 +Aug 13 17:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 450 to 501 +Aug 13 18:28:40 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Currently unreadable (pending) sectors +Aug 13 18:28:40 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Offline uncorrectable sectors +Aug 13 18:28:40 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 91 to 90 +Aug 13 18:28:40 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 501 to 554 +Aug 13 18:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Currently unreadable (pending) sectors +Aug 13 18:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Offline uncorrectable sectors +Aug 13 18:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 90 to 89 +Aug 13 18:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 554 to 606 +Aug 13 19:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Currently unreadable (pending) sectors +Aug 13 19:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Offline uncorrectable sectors +Aug 13 19:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 89 to 93 +Aug 13 19:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 606 to 655 +Aug 13 19:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Currently unreadable (pending) sectors +Aug 13 19:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Offline uncorrectable sectors +Aug 13 19:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 60 +Aug 13 19:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 41 to 40 +Aug 13 19:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 655 to 665 +Aug 13 20:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Currently unreadable (pending) sectors +Aug 13 20:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Offline uncorrectable sectors +Aug 13 20:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 93 to 96 +Aug 13 20:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 59 +Aug 13 20:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 41 +Aug 13 20:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Currently unreadable (pending) sectors +Aug 13 20:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Offline uncorrectable sectors +Aug 13 20:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 96 to 105 +Aug 13 20:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 58 +Aug 13 20:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 41 to 42 +Aug 13 21:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Currently unreadable (pending) sectors +Aug 13 21:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Offline uncorrectable sectors +Aug 13 21:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 105 to 114 +Aug 13 21:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 360 Currently unreadable (pending) sectors (changed +240) +Aug 13 21:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 360 Offline uncorrectable sectors (changed +240) +Aug 13 21:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 114 to 84 +Aug 13 21:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 58 to 59 +Aug 13 21:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 42 to 41 +Aug 13 21:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 197 Current_Pending_Sector changed from 100 to 98 +Aug 13 21:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 198 Offline_Uncorrectable changed from 100 to 98 +Aug 13 21:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 665 to 768 +Aug 13 22:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors (changed -8) +Aug 13 22:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors (changed -8) +Aug 13 22:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 84 to 91 +Aug 13 22:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 58 +Aug 13 22:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 41 to 42 +Aug 13 22:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 768 to 1001 +Aug 13 22:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors +Aug 13 22:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors +Aug 13 22:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 91 to 118 +Aug 13 23:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors +Aug 13 23:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors +Aug 13 23:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 118 to 109 +Aug 13 23:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors +Aug 13 23:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors +Aug 13 23:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 109 to 117 +Aug 13 23:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 58 to 57 +Aug 13 23:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 42 to 43 +Aug 14 00:28:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors +Aug 14 00:28:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors +Aug 14 00:28:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 57 to 59 +Aug 14 00:28:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 43 to 41 +Aug 14 00:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors +Aug 14 00:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors +Aug 14 01:28:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors +Aug 14 01:28:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors +Aug 14 01:58:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors +Aug 14 01:58:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors +Aug 14 02:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors +Aug 14 02:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors +Aug 14 02:58:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors +Aug 14 02:58:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors +Aug 14 03:28:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors +Aug 14 03:28:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors +Aug 14 03:58:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors +Aug 14 03:58:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors +Aug 14 04:28:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors +Aug 14 04:28:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors +Aug 14 04:28:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 60 +Aug 14 04:28:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 41 to 40 +Aug 14 04:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors +Aug 14 04:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors +Aug 14 04:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 61 +Aug 14 04:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 39 +Aug 14 05:28:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors +Aug 14 05:28:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors +Aug 14 05:28:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 61 to 60 +Aug 14 05:28:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 39 to 40 +Aug 14 05:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors +Aug 14 05:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors +Aug 14 05:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 61 +Aug 14 05:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 39 +Aug 14 06:28:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors +Aug 14 06:28:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors +Aug 14 06:28:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 61 to 60 +Aug 14 06:28:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 39 to 40 +Aug 14 06:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors +Aug 14 06:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors +Aug 14 07:28:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors +Aug 14 07:28:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors +Aug 14 07:28:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 61 +Aug 14 07:28:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 39