evening Part A: disk Z6A07P2G silenced 30 days by serial, Longhorn HDD4 removed, removal checklist + R-934; R-231 token out of the URL

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-12 16:36:00 +02:00
parent 0d463a859e
commit 6c2c3dd574
3 changed files with 74 additions and 2 deletions
@@ -0,0 +1,21 @@
+# Disk ST3000VX010 S/N Z6A07P2G on DooPlex — read 2026-10-12 ~14:30Z, then three changes after the operator's yes in chat
+alerts (Alertmanager active): DiskUncorrectableSectors and DiskOfflineUncorrectable, labels component=disk,
+  device=ata-ST3000VX010-2E3166_Z6A07P2G, severity critical, receiver email-notifications; expressions
+  max by (device)(smartctl_device_attribute{attribute_name="Current_Pending_Sector"|"Offline_Uncorrectable",raw}) > 0;
+  startsAt 2026-10-12T11:49:06Z (the Prometheus restart of the reloader change; first fired 2026-10-10 08:21Z).
+SMART history (Prometheus, 2026-10-09 → 2026-10-12, 6 h step): Current_Pending_Sector 352 flat, Offline_Uncorrectable 352 flat, Reallocated 0.
+device: /dev/disk/by-id/ata-ST3000VX010-2E3166_Z6A07P2G -> sdg; by-path pci-0000:00:1f.2-ata-6; kernel ata10; sdg2 ext4 UUID 9cf20140-…; NOT mounted.
+twin: ST3000VX010 S/N Z6A070XZ = sdd, mounted /mnt/3_hdd — in use.
+fstab line 16: "# DISABLED 2026-08-14, drive failed (… Z6A07P2G): UUID=9cf20140-… /mnt/4_hdd …" (commented out).
+k3s.service.d/wait-for-mounts.conf: RequiresMountsFor=/mnt/2_hdd /mnt/ssd_2 /mnt/1_hdd /mnt/3_hdd (no 4_hdd).
+Longhorn node dooplex, before: disk HDD4 path /mnt/4_hdd/data/longhorn allowScheduling false, 0 replicas, Ready False
+  (spec kept here for undo: {"allowScheduling":false,"diskType":"filesystem","path":"/mnt/4_hdd/data/longhorn","storageReserved":536870912000,"tags":["hdd","hdd4"]}).
+backup scripts (live and role), manifests, PVs, pod hostPaths: no path on /mnt/4_hdd (only comments).
+SMART exporter: admin-system/smartctl-exporter.yaml line 63 names the disk by id (to remove after the pull-out).
+CHANGE 1 (yes): silence 1e18c10a-9398-4a67-9769-03e5405fce1b, matcher device=~".*_Z6A07P2G", until 2026-11-11T14:32:06Z,
+  comment "disk Z6A07P2G evacuated 2026-10-10, waiting for physical removal". Read back: both alerts "suppressed".
+  Control: of the 7 disks Prometheus sees (smartctl_device_temperature), the pattern matches exactly 1.
+CHANGE 2 (yes): `kubectl patch nodes.longhorn.io dooplex` disks.HDD4=null. Read back: spec disks + status disks = Crucial-SSD-240G, HDD2,
+  default-disk; node conditions identical to before; 78 Longhorn volumes, all healthy.
+CHANGE 3 (yes, R-231): backup-manifests.sh sends the token as a header (homelab-manifests 1943a70, installed by the role, live file = git).
+  Read-only proof: git ls-remote of the private repo with the header -> HEAD d0767fe; without -> "could not read Username" (refused).