admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(audits): first genuinely failing disk — the SMART PASSED trap
gates / gates (push) Successful in 12s
Details

Browse Source 
Commit the raw evidence from ST3000VX010 S/N Z6A07P2G (/dev/sdg on DooPlex),
which went 8 -> 352 unreadable sectors 11-13 Aug while smart_status.passed
stayed true throughout.

- fixtures/smart-ST3000VX010-failing-2026-08-14.json: raw smartctl -a -j, verbatim
- fixtures/smartd-history-sdg-2026-08-14.txt: 406 smartd journal lines, 11-14 Aug
- DIAG-smart-passed-trap-2026-08-14.md: the mechanism (attrs 187/197/198 all carry
  thresh 0, so a normalized value that floors at 1 can never fail the overall
  verdict on unreadable sectors), the non-monotonic timeline, the three controller
  defects with locators, and the counterfactual: zero emails would have been sent.
This commit is contained in:
authentik Default Admin
2026-08-14 07:57:07 +02:00
parent e0b56c976f
commit 848de8153d
3 changed files with 1627 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/audits/DIAG-smart-passed-trap-2026-08-14.md
+196
View File
@@ -0,0 +1,196 @@
# DIAG — the SMART `PASSED` trap, and the disk alert that reached nobody
**Date:** 2026-08-14
**Drive:** Seagate `ST3000VX010-2E3166`, S/N `Z6A07P2G`, 3.0 TB, `/dev/sdg` on **DooPlex**
**Fixtures:** `fixtures/smart-ST3000VX010-failing-2026-08-14.json` (raw `smartctl -a -j /dev/sdg`),
`fixtures/smartd-history-sdg-2026-08-14.txt` (406 `smartd` journal lines for this device, 11–14 Aug)
**Status:** the three controller defects named here are FIXED in controller **v0.215.0**; the
counterfactual in §5 is derived from source, **not** reproduced live.
This is the project's first genuinely failing disk. Before it, the capability map recorded the
disk-failure scenario as *"Healthy path only — a genuinely failing disk has never been seen."*
---
## 1. What happened
On **11 August 12:28** a 3 TB drive in DooPlex reported its first unreadable sectors. By **13 August
21:58** it was at 360, and it was taking down a running service. Throughout the entire episode the
drive's own overall self-assessment read **`PASSED`**, and it still does.
`smartd` was running the whole time and mailed **local root** — a mailbox nobody reads. The failure
was actually found by a crashlooping pod, not by any alert.
---
## 2. The mechanism — `smart_status.passed` cannot fail on unreadable sectors
Measured, from the committed fixture:
| ID | Attribute | value | worst | thresh | raw |
|-----|--------------------------|-------|-------|--------|--------|
| 5 | `Reallocated_Sector_Ct` | 100 | 100 | **10** | 0 |
| 187 | `Reported_Uncorrect` | **1** | **1** | **0** | **1001** |
| 188 | `Command_Timeout` | 100 | 100 | **0** | 0 |
| 197 | `Current_Pending_Sector` | 98 | 98 | **0** | **352** |
| 198 | `Offline_Uncorrectable` | 98 | 98 | **0** | **352** |
| 199 | `UDMA_CRC_Error_Count` | 200 | 200 | **0** | 0 |
`smart_status.passed` is false only when some attribute's **normalized value** falls **at or below**
its **threshold**. Attributes 187, 197 and 198 — the three that record unreadable sectors — all carry
`thresh: 0`. A normalized SMART value floors at 1 and cannot reach 0.
> **Therefore `smart_status.passed` is structurally incapable of failing on unreadable sectors.**
> This is not a quirk of this drive or this vendor. It is how a zero threshold behaves.
Attribute **187 `Reported_Uncorrect` sits at normalized `1` against threshold `0`** with a raw count
of **1001**. It is one point from failing and has no remaining point to give.
Corroborating, from the same fixture: `ata_smart_error_log.summary.count = 1001`, power-on hours
**60505** (~6.9 years), `smartctl` exit status **64** (bit 6 — *the device error log contains
records of errors*) while `smart_status.passed` is still `true`.
**Any monitor built on the drive's overall verdict is blind to this entire class of failure.**
Felhom already knows better — `agentapi.DiskVerdictFor` reads the raw counters — which is why it
would have noticed on 11 August, two days early.
---
## 3. Unreadable sectors are not monotonic
From `smartd-history-sdg-2026-08-14.txt`, `Current_Pending_Sector` over the episode (30-minute
sampling, host clock = CEST):
```
Aug 11 12:28 8 first sighting
Aug 11 12:58 16 (+8)
Aug 11 13:28 0 FULL CLEAR — "No more Currently unreadable (pending) sectors,
warning condition reset after 1 email"
Aug 11 20:28 8 returns
Aug 12 01:28 16 → 01:58 8 (-8)
Aug 12 02:28 32 → 02:58 24 (-8)
Aug 12 03:28 24 187 Reported_Uncorrect 100→97; ATA error count 0→3
Aug 12 03:58 16 (-8) → 04:28 24 (+8)
Aug 12 10:58 32 … steady 32 for ~11h …
Aug 12 21:58 24 (-8)
Aug 13 03:28 40 (+16) → 04:28 24 (-16)
Aug 13 11:28 64 terminal run begins — never returns below 64
Aug 13 11:58 72 12:58 80 13:28 112 15:58 120
Aug 13 21:58 360 (+240)
Aug 13 22:28 352 (-8) … steady 352 through 14 Aug …
```
Two measured facts carry design weight:
1. **The 11 August excursion cleared completely within one hour** (12:28 → 13:28). A bare `> 0`
alarm would have fired on a drive that then looked fine for seven hours. This is why the ladder
uses a *sustain* rule rather than a bare non-zero test.
2. **The benign excursion peaked at 16; the terminal run crossed 64 at 13 Aug 11:28 and never came
back.** That is the entire empirical basis for the static count threshold of 64 — see §6.
---
## 4. The three controller defects
All three are in `felhom-controller` at `3e3ee94` (v0.214.0), the tree audited here.
### D1 — the alert carries a severity the hub does not recognise *(highest value)*
`internal/notify/notifier.go:565` — `NotifyDiskHealthDegraded` sets:
```go
severity := "warn"
```
The hub accepts an exact-match lowercase vocabulary and **coerces anything else to `info`**:
- `felhom.eu/hub/internal/api/handler.go:2121-2126` — `case "info", "warning", "error", "critical":`
… `default: payload.Severity = "info"`
- `felhom.eu/hub/internal/notify/dispatcher.go:89-96` — `severityNotifies` returns true only for
`warning` / `error` / `critical`.
`"warn"` is not in the accepted set. So the Figyelmeztetés-level disk alert is **stored as an
informational notice and emailed to nobody**, on the customer leg and the operator leg alike.
The function's own doc comment reads *"The hub applies its own per-event-type cooldown"* — which
presumes it routes. An invariant asserted in a comment with no test pinning it; this project's
recurring shape.
### D2 — no level above "worth keeping an eye on it"
`internal/agentapi/diskverdict.go:28-41` returns `Warn` for *any* non-zero counter, and can only
reach `Fail` when `Health == "FAILING"` — which, by §2, this fault class cannot produce. A drive with
one aging sector and a drive at 352 unreadable sectors rendered the identical chip and the identical
mild sentence.
### D3 — it speaks once, and forgets on restart
`internal/web/disk_health.go:182` emits only on `v > prev`, against a **in-memory** baseline
(`disk_health.go:26-29`, *"Lost on restart → the next check re-baselines silently"*). Consequences:
- Between 8 and 352 pending sectors the verdict never changes level, so **nothing further is emitted**.
- A controller restart while a disk is already bad re-baselines it silently — that disk never alerts
again.
---
## 5. Counterfactual — what a customer would have received
**Derived from the source above plus the §3 timeline. NOT reproduced live.**
| Date/time | Drive state | Felhom verdict at v0.214.0 | Emitted | Delivered |
|-----------|-------------|-----------------------------|---------|-----------|
| 11 Aug 12:28 | pending 8 | OK → Figyelmeztetés | `disk_health_degraded`, severity `warn` | **nothing** — coerced to `info`, dropped by `severityNotifies` |
| 11 Aug 13:28 | pending 0 | Figyelmeztetés → Rendben | none (recovery is silent) | nothing |
| 11 Aug 20:28 → 13 Aug | 8 → 352 | Figyelmeztetés throughout | none (no level change) | nothing |
| 13 Aug 21:58 | pending 360 | Figyelmeztetés | none | nothing |
> **Felhom would have emitted zero emails about this drive.** The one event it did produce was filed
> at `info` and delivered to no one.
Note the two defects compound: even had D1 been fixed alone, the customer would have received a
single mild "Javasolt figyelemmel kísérni" at 8 sectors on 11 August and then silence through 352.
---
## 6. Provenance of the thresholds chosen in v0.215.0
- **64 unreadable sectors → Hiba.** The observed benign excursion peaked at **16** and cleared inside
an hour; the terminal run passed **64** at 13 Aug 11:28 and never returned below it. 64 sits above
the one observed transient and below the observed terminal run. **This is a judgement from ONE
drive.** It is a static backstop and is expected to be replaced in Phase 3 by growth-rate detection
over real history.
- **Sustain before count.** The primary rule is "unreadable sectors still present at the next check";
the count is the backstop. On this drive sustain fires **12 Aug**, the count not until **13 Aug** —
a full day earlier. The backstop exists for a box that was powered off or restarted across the
sustain window.
- **55 / 60 °C.** Adopted unchanged from the operator's existing Prometheus bands on DooPlex, so the
two systems cannot disagree about the same drive.
---
## 7. Measured vs inferred
**Measured** (reproducible from the committed fixtures):
- The attribute table, thresholds and raw values in §2; `passed: true` at 352 pending sectors.
- The full non-monotonic timeline in §3, including the one-hour full clear.
- The three defect locators in §4 — read from live source on both the controller and the hub side.
**Inferred** (source-derived, not executed):
- The §5 counterfactual. It follows from the §4 locators and the §3 timeline; **no email path was
exercised against this drive**, and no `disk_health_degraded` event for it exists in the hub.
**Not covered here:**
- Attributes **187**, **199** and **188** are not on the agent→controller wire today. Adding them is
Phase 2 (a declared wire change, so the hub models them in the same session under G-1). Everything
the v0.215.0 fix needs was already on the wire.
- The new Fail-from-counters path has **not** fired on real hardware — only against this fixture's
values in unit tests.
---
## 8. One layer out
`smartd` on DooPlex did its job and mailed local root, where nothing reads. That is the same shape
as D1 — a correct detection with a delivery path to nowhere — one layer outside the product. Tracked
separately as DooPlex hygiene.
documentation/audits/fixtures/smart-ST3000VX010-failing-2026-08-14.json
+1025
View File
File diff suppressed because it is too large Load Diff
documentation/audits/fixtures/smartd-history-sdg-2026-08-14.txt
+406
View File
@@ -0,0 +1,406 @@
Aug 11 01:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 61 to 60
Aug 11 01:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 39 to 40
Aug 11 01:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 61
Aug 11 01:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 39
Aug 11 02:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 61 to 60
Aug 11 02:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 39 to 40
Aug 11 03:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 117 to 118
Aug 11 03:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 61
Aug 11 03:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 39
Aug 11 10:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 61 to 60
Aug 11 10:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 39 to 40
Aug 11 10:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 61
Aug 11 10:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 39
Aug 11 11:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 61 to 60
Aug 11 11:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 39 to 40
Aug 11 12:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Currently unreadable (pending) sectors
Aug 11 12:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Offline uncorrectable sectors
Aug 11 12:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 16 Currently unreadable (pending) sectors (changed +8)
Aug 11 12:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 16 Offline uncorrectable sectors (changed +8)
Aug 11 13:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], No more Currently unreadable (pending) sectors, warning condition reset after 1 email
Aug 11 13:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], No more Offline uncorrectable sectors, warning condition reset after 1 email
Aug 11 14:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 61
Aug 11 14:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 39
Aug 11 15:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 118 to 119
Aug 11 15:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 61 to 60
Aug 11 15:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 39 to 40
Aug 11 16:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 59
Aug 11 16:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 41
Aug 11 17:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 60
Aug 11 17:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 41 to 40
Aug 11 18:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 9 Power_On_Hours changed from 32 to 31
Aug 11 20:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Currently unreadable (pending) sectors
Aug 11 20:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Offline uncorrectable sectors
Aug 11 20:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Currently unreadable (pending) sectors
Aug 11 20:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Offline uncorrectable sectors
Aug 11 21:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Currently unreadable (pending) sectors
Aug 11 21:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Offline uncorrectable sectors
Aug 11 21:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Currently unreadable (pending) sectors
Aug 11 21:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Offline uncorrectable sectors
Aug 11 22:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Currently unreadable (pending) sectors
Aug 11 22:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Offline uncorrectable sectors
Aug 11 22:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Currently unreadable (pending) sectors
Aug 11 22:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Offline uncorrectable sectors
Aug 11 23:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Currently unreadable (pending) sectors
Aug 11 23:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Offline uncorrectable sectors
Aug 11 23:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 61
Aug 11 23:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 39
Aug 11 23:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Currently unreadable (pending) sectors
Aug 11 23:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Offline uncorrectable sectors
Aug 12 00:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Currently unreadable (pending) sectors
Aug 12 00:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Offline uncorrectable sectors
Aug 12 00:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Currently unreadable (pending) sectors
Aug 12 00:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Offline uncorrectable sectors
Aug 12 01:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 16 Currently unreadable (pending) sectors (changed +8)
Aug 12 01:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 16 Offline uncorrectable sectors (changed +8)
Aug 12 01:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Currently unreadable (pending) sectors (changed -8)
Aug 12 01:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 8 Offline uncorrectable sectors (changed -8)
Aug 12 02:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors (changed +24)
Aug 12 02:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors (changed +24)
Aug 12 02:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 189 High_Fly_Writes changed from 51 to 50
Aug 12 02:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors (changed -8)
Aug 12 02:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors (changed -8)
Aug 12 03:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 12 03:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 12 03:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 119 to 115
Aug 12 03:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 187 Reported_Uncorrect changed from 100 to 97
Aug 12 03:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 0 to 3
Aug 12 03:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 16 Currently unreadable (pending) sectors (changed -8)
Aug 12 03:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 16 Offline uncorrectable sectors (changed -8)
Aug 12 03:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 189 High_Fly_Writes changed from 50 to 49
Aug 12 04:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors (changed +8)
Aug 12 04:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors (changed +8)
Aug 12 04:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 115 to 113
Aug 12 04:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 187 Reported_Uncorrect changed from 97 to 95
Aug 12 04:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 189 High_Fly_Writes changed from 49 to 47
Aug 12 04:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 3 to 5
Aug 12 04:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 12 04:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 12 04:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 61 to 60
Aug 12 04:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 39 to 40
Aug 12 05:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 12 05:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 12 05:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 12 05:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 12 05:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 59
Aug 12 05:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 41
Aug 12 06:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 12 06:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 12 06:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 12 06:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 12 07:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 12 07:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 12 07:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 12 07:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 12 08:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 12 08:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 12 08:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 58
Aug 12 08:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 41 to 42
Aug 12 08:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 12 08:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 12 08:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 58 to 59
Aug 12 08:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 42 to 41
Aug 12 09:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 12 09:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 12 09:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 60
Aug 12 09:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 41 to 40
Aug 12 09:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 12 09:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 12 10:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 12 10:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 12 10:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors (changed +8)
Aug 12 10:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors (changed +8)
Aug 12 10:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 189 High_Fly_Writes changed from 47 to 46
Aug 12 10:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 59
Aug 12 10:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 41
Aug 12 11:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors
Aug 12 11:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors
Aug 12 11:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors
Aug 12 11:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors
Aug 12 12:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors
Aug 12 12:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors
Aug 12 12:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 113 to 100
Aug 12 12:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors
Aug 12 12:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors
Aug 12 12:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 100 to 101
Aug 12 12:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 58
Aug 12 12:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 41 to 42
Aug 12 13:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors
Aug 12 13:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors
Aug 12 13:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 101 to 102
Aug 12 13:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors
Aug 12 13:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors
Aug 12 14:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors
Aug 12 14:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors
Aug 12 14:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 102 to 106
Aug 12 14:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 58 to 59
Aug 12 14:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 42 to 41
Aug 12 14:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors
Aug 12 14:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors
Aug 12 14:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 106 to 107
Aug 12 14:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 60
Aug 12 14:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 41 to 40
Aug 12 15:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors
Aug 12 15:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors
Aug 12 15:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors
Aug 12 15:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors
Aug 12 15:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 107 to 108
Aug 12 16:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors
Aug 12 16:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors
Aug 12 16:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 189 High_Fly_Writes changed from 46 to 45
Aug 12 16:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors
Aug 12 16:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors
Aug 12 17:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors
Aug 12 17:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors
Aug 12 17:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors
Aug 12 17:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors
Aug 12 17:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 108 to 110
Aug 12 17:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 59
Aug 12 17:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 41
Aug 12 18:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors
Aug 12 18:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors
Aug 12 18:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 110 to 111
Aug 12 18:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 60
Aug 12 18:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 41 to 40
Aug 12 18:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors
Aug 12 18:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors
Aug 12 19:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors
Aug 12 19:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors
Aug 12 19:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors
Aug 12 19:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors
Aug 12 20:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors
Aug 12 20:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors
Aug 12 20:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors
Aug 12 20:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors
Aug 12 21:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Currently unreadable (pending) sectors
Aug 12 21:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 32 Offline uncorrectable sectors
Aug 12 21:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 59
Aug 12 21:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 41
Aug 12 21:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors (changed -8)
Aug 12 21:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors (changed -8)
Aug 12 22:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 12 22:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 12 22:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 12 22:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 12 22:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 60
Aug 12 22:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 41 to 40
Aug 12 23:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 12 23:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 12 23:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 12 23:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 12 23:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 111 to 112
Aug 13 00:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 13 00:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 13 00:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 13 00:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 13 01:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 13 01:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 13 01:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 61
Aug 13 01:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 39
Aug 13 01:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 13 01:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 13 02:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 13 02:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 13 02:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 112 to 113
Aug 13 02:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 13 02:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 13 03:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 40 Currently unreadable (pending) sectors (changed +16)
Aug 13 03:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 40 Offline uncorrectable sectors (changed +16)
Aug 13 03:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 113 to 103
Aug 13 03:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 187 Reported_Uncorrect changed from 95 to 83
Aug 13 03:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 5 to 17
Aug 13 03:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 40 Currently unreadable (pending) sectors
Aug 13 03:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 40 Offline uncorrectable sectors
Aug 13 04:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors (changed -16)
Aug 13 04:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors (changed -16)
Aug 13 04:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 187 Reported_Uncorrect changed from 83 to 82
Aug 13 04:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 17 to 18
Aug 13 04:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 13 04:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 13 05:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 13 05:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 13 05:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 103 to 104
Aug 13 05:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 13 05:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 13 06:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 13 06:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 13 06:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 13 06:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 13 07:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 13 07:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 13 07:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 13 07:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 13 08:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 13 08:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 13 08:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 13 08:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 13 08:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 104 to 105
Aug 13 09:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 13 09:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 13 09:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 13 09:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 13 10:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 13 10:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 13 10:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 61 to 60
Aug 13 10:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 39 to 40
Aug 13 10:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Currently unreadable (pending) sectors
Aug 13 10:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 24 Offline uncorrectable sectors
Aug 13 11:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 64 Currently unreadable (pending) sectors (changed +40)
Aug 13 11:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 64 Offline uncorrectable sectors (changed +40)
Aug 13 11:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 189 High_Fly_Writes changed from 45 to 43
Aug 13 11:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 59
Aug 13 11:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 41
Aug 13 11:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 72 Currently unreadable (pending) sectors (changed +8)
Aug 13 11:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 72 Offline uncorrectable sectors (changed +8)
Aug 13 12:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 72 Currently unreadable (pending) sectors
Aug 13 12:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 72 Offline uncorrectable sectors
Aug 13 12:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 80 Currently unreadable (pending) sectors (changed +8)
Aug 13 12:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 80 Offline uncorrectable sectors (changed +8)
Aug 13 13:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 112 Currently unreadable (pending) sectors (changed +32)
Aug 13 13:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 112 Offline uncorrectable sectors (changed +32)
Aug 13 13:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 105 to 104
Aug 13 13:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 187 Reported_Uncorrect changed from 82 to 80
Aug 13 13:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 18 to 20
Aug 13 13:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 112 Currently unreadable (pending) sectors
Aug 13 13:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 112 Offline uncorrectable sectors
Aug 13 13:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 104 to 99
Aug 13 13:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 187 Reported_Uncorrect changed from 80 to 25
Aug 13 13:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 58
Aug 13 13:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 41 to 42
Aug 13 13:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 20 to 75
Aug 13 14:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 112 Currently unreadable (pending) sectors
Aug 13 14:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 112 Offline uncorrectable sectors
Aug 13 14:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 99 to 96
Aug 13 14:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 187 Reported_Uncorrect changed from 25 to 1
Aug 13 14:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 75 to 129
Aug 13 14:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 112 Currently unreadable (pending) sectors
Aug 13 14:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 112 Offline uncorrectable sectors
Aug 13 14:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 96 to 95
Aug 13 14:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 129 to 184
Aug 13 15:28:40 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 112 Currently unreadable (pending) sectors
Aug 13 15:28:40 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 112 Offline uncorrectable sectors
Aug 13 15:28:40 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 95 to 94
Aug 13 15:28:40 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 58 to 57
Aug 13 15:28:40 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 42 to 43
Aug 13 15:28:40 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 184 to 239
Aug 13 15:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Currently unreadable (pending) sectors (changed +8)
Aug 13 15:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Offline uncorrectable sectors (changed +8)
Aug 13 15:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 239 to 292
Aug 13 16:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Currently unreadable (pending) sectors
Aug 13 16:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Offline uncorrectable sectors
Aug 13 16:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 292 to 344
Aug 13 16:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Currently unreadable (pending) sectors
Aug 13 16:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Offline uncorrectable sectors
Aug 13 16:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 94 to 93
Aug 13 16:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 57 to 58
Aug 13 16:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 43 to 42
Aug 13 16:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 344 to 399
Aug 13 17:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Currently unreadable (pending) sectors
Aug 13 17:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Offline uncorrectable sectors
Aug 13 17:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 93 to 94
Aug 13 17:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 58 to 59
Aug 13 17:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 42 to 41
Aug 13 17:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 399 to 450
Aug 13 17:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Currently unreadable (pending) sectors
Aug 13 17:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Offline uncorrectable sectors
Aug 13 17:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 94 to 91
Aug 13 17:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 450 to 501
Aug 13 18:28:40 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Currently unreadable (pending) sectors
Aug 13 18:28:40 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Offline uncorrectable sectors
Aug 13 18:28:40 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 91 to 90
Aug 13 18:28:40 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 501 to 554
Aug 13 18:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Currently unreadable (pending) sectors
Aug 13 18:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Offline uncorrectable sectors
Aug 13 18:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 90 to 89
Aug 13 18:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 554 to 606
Aug 13 19:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Currently unreadable (pending) sectors
Aug 13 19:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Offline uncorrectable sectors
Aug 13 19:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 89 to 93
Aug 13 19:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 606 to 655
Aug 13 19:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Currently unreadable (pending) sectors
Aug 13 19:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Offline uncorrectable sectors
Aug 13 19:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 60
Aug 13 19:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 41 to 40
Aug 13 19:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 655 to 665
Aug 13 20:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Currently unreadable (pending) sectors
Aug 13 20:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Offline uncorrectable sectors
Aug 13 20:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 93 to 96
Aug 13 20:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 59
Aug 13 20:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 41
Aug 13 20:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Currently unreadable (pending) sectors
Aug 13 20:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Offline uncorrectable sectors
Aug 13 20:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 96 to 105
Aug 13 20:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 58
Aug 13 20:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 41 to 42
Aug 13 21:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Currently unreadable (pending) sectors
Aug 13 21:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 120 Offline uncorrectable sectors
Aug 13 21:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 105 to 114
Aug 13 21:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 360 Currently unreadable (pending) sectors (changed +240)
Aug 13 21:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 360 Offline uncorrectable sectors (changed +240)
Aug 13 21:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 114 to 84
Aug 13 21:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 58 to 59
Aug 13 21:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 42 to 41
Aug 13 21:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 197 Current_Pending_Sector changed from 100 to 98
Aug 13 21:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 198 Offline_Uncorrectable changed from 100 to 98
Aug 13 21:58:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 665 to 768
Aug 13 22:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors (changed -8)
Aug 13 22:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors (changed -8)
Aug 13 22:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 84 to 91
Aug 13 22:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 58
Aug 13 22:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 41 to 42
Aug 13 22:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], ATA error count increased from 768 to 1001
Aug 13 22:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors
Aug 13 22:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors
Aug 13 22:58:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 91 to 118
Aug 13 23:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors
Aug 13 23:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors
Aug 13 23:28:37 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 118 to 109
Aug 13 23:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors
Aug 13 23:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors
Aug 13 23:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 109 to 117
Aug 13 23:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 58 to 57
Aug 13 23:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 42 to 43
Aug 14 00:28:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors
Aug 14 00:28:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors
Aug 14 00:28:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 57 to 59
Aug 14 00:28:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 43 to 41
Aug 14 00:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors
Aug 14 00:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors
Aug 14 01:28:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors
Aug 14 01:28:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors
Aug 14 01:58:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors
Aug 14 01:58:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors
Aug 14 02:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors
Aug 14 02:28:36 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors
Aug 14 02:58:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors
Aug 14 02:58:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors
Aug 14 03:28:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors
Aug 14 03:28:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors
Aug 14 03:58:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors
Aug 14 03:58:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors
Aug 14 04:28:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors
Aug 14 04:28:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors
Aug 14 04:28:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 60
Aug 14 04:28:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 41 to 40
Aug 14 04:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors
Aug 14 04:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors
Aug 14 04:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 61
Aug 14 04:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 39
Aug 14 05:28:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors
Aug 14 05:28:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors
Aug 14 05:28:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 61 to 60
Aug 14 05:28:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 39 to 40
Aug 14 05:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors
Aug 14 05:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors
Aug 14 05:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 61
Aug 14 05:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 39
Aug 14 06:28:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors
Aug 14 06:28:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors
Aug 14 06:28:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 61 to 60
Aug 14 06:28:42 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 39 to 40
Aug 14 06:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors
Aug 14 06:58:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors
Aug 14 07:28:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Currently unreadable (pending) sectors
Aug 14 07:28:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], 352 Offline uncorrectable sectors
Aug 14 07:28:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 61
Aug 14 07:28:41 dooplex smartd[2694634]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 39