evening Part A: disk Z6A07P2G silenced 30 days by serial, Longhorn HDD4 removed, removal checklist + R-934; R-231 token out of the URL

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-12 16:36:00 +02:00
parent 0d463a859e
commit 6c2c3dd574
3 changed files with 74 additions and 2 deletions
@@ -0,0 +1,21 @@
+# Disk ST3000VX010 S/N Z6A07P2G on DooPlex — read 2026-10-12 ~14:30Z, then three changes after the operator's yes in chat
+alerts (Alertmanager active): DiskUncorrectableSectors and DiskOfflineUncorrectable, labels component=disk,
+  device=ata-ST3000VX010-2E3166_Z6A07P2G, severity critical, receiver email-notifications; expressions
+  max by (device)(smartctl_device_attribute{attribute_name="Current_Pending_Sector"|"Offline_Uncorrectable",raw}) > 0;
+  startsAt 2026-10-12T11:49:06Z (the Prometheus restart of the reloader change; first fired 2026-10-10 08:21Z).
+SMART history (Prometheus, 2026-10-09 → 2026-10-12, 6 h step): Current_Pending_Sector 352 flat, Offline_Uncorrectable 352 flat, Reallocated 0.
+device: /dev/disk/by-id/ata-ST3000VX010-2E3166_Z6A07P2G -> sdg; by-path pci-0000:00:1f.2-ata-6; kernel ata10; sdg2 ext4 UUID 9cf20140-…; NOT mounted.
+twin: ST3000VX010 S/N Z6A070XZ = sdd, mounted /mnt/3_hdd — in use.
+fstab line 16: "# DISABLED 2026-08-14, drive failed (… Z6A07P2G): UUID=9cf20140-… /mnt/4_hdd …" (commented out).
+k3s.service.d/wait-for-mounts.conf: RequiresMountsFor=/mnt/2_hdd /mnt/ssd_2 /mnt/1_hdd /mnt/3_hdd (no 4_hdd).
+Longhorn node dooplex, before: disk HDD4 path /mnt/4_hdd/data/longhorn allowScheduling false, 0 replicas, Ready False
+  (spec kept here for undo: {"allowScheduling":false,"diskType":"filesystem","path":"/mnt/4_hdd/data/longhorn","storageReserved":536870912000,"tags":["hdd","hdd4"]}).
+backup scripts (live and role), manifests, PVs, pod hostPaths: no path on /mnt/4_hdd (only comments).
+SMART exporter: admin-system/smartctl-exporter.yaml line 63 names the disk by id (to remove after the pull-out).
+CHANGE 1 (yes): silence 1e18c10a-9398-4a67-9769-03e5405fce1b, matcher device=~".*_Z6A07P2G", until 2026-11-11T14:32:06Z,
+  comment "disk Z6A07P2G evacuated 2026-10-10, waiting for physical removal". Read back: both alerts "suppressed".
+  Control: of the 7 disks Prometheus sees (smartctl_device_temperature), the pattern matches exactly 1.
+CHANGE 2 (yes): `kubectl patch nodes.longhorn.io dooplex` disks.HDD4=null. Read back: spec disks + status disks = Crucial-SSD-240G, HDD2,
+  default-disk; node conditions identical to before; 78 Longhorn volumes, all healthy.
+CHANGE 3 (yes, R-231): backup-manifests.sh sends the token as a header (homelab-manifests 1943a70, installed by the role, live file = git).
+  Read-only proof: git ls-remote of the private repo with the header -> HEAD d0767fe; without -> "could not read Username" (refused).
@@ -0,0 +1,50 @@
+# RUNBOOK — pull the failed disk Z6A07P2G out of DooPlex
+
+> Written 2026-10-12 by CC from read-only checks (`audits/answers-2026-10-12/disk-Z6A07P2G.txt`). Operator's job: the
+> power-off and the hands. Register row R-934.
+
+## The disk — by SERIAL, never by `sdX`
+
+| | |
+|---|---|
+| Model | Seagate Skyhawk **ST3000VX010-2E3166**, 3 TB |
+| Serial | **Z6A07P2G** |
+| Port (2026-10-12) | the onboard SATA controller `0000:00:1f.2`, port **ata-6** (`/dev/disk/by-path/pci-0000:00:1f.2-ata-6`); kernel `ata10` |
+| Fault | 352 unreadable (pending) + 352 offline-uncorrectable sectors, flat since 2026-10-09 |
+
+**⚠ THE TWIN.** DooPlex has a SECOND disk of the same model: **ST3000VX010, serial Z6A070XZ — IN USE as `/mnt/3_hdd`, keep it.**
+`Z6A07P2G` (pull) and `Z6A070XZ` (keep) differ only in the last four characters. **Read the serial on the disk's
+label before you unplug anything.** If you cannot read it, follow the cable to SATA port 6 on the board. Do not
+make the disk busy to find it by its LED: no reads on a failing disk.
+
+## Before power-off (already true on 2026-10-12 — re-check in one minute)
+
+```bash
+lsblk -o NAME,SERIAL,MOUNTPOINTS | grep Z6A07P2G      # no mountpoint
+grep -n 9cf20140 /etc/fstab                          # only a '# DISABLED 2026-08-14' line
+grep RequiresMountsFor /etc/systemd/system/k3s.service.d/wait-for-mounts.conf   # no /mnt/4_hdd
+sudo kubectl -n longhorn-system get nodes.longhorn.io dooplex -o jsonpath='{.spec.disks}' | grep -c 4_hdd   # 0 (HDD4 entry removed 2026-10-12)
+```
+
+Nothing else reads the disk: no backup script, PV, hostPath or running app names `/mnt/4_hdd` (2026-10-12).
+
+## Pull it
+
+1. Plan a window: DooPlex runs Gitea, the registry, the hub and Longhorn. A box keeps running while the hub is off.
+2. `sudo poweroff`. Unplug power.
+3. Find the disk with serial **Z6A07P2G** (label), on SATA port 6. Unplug its data and power cable. Remove it.
+4. Power on.
+
+## After the first boot
+
+1. All mounts back: `findmnt /mnt/1_hdd /mnt/2_hdd /mnt/3_hdd /mnt/5_hdd /mnt/ssd_2` — five lines.
+2. k3s up and Longhorn healthy: `sudo kubectl -n longhorn-system get nodes.longhorn.io` READY True; all volumes
+   `healthy` (`sudo kubectl -n longhorn-system get volumes.longhorn.io`).
+3. The hub answers (`https://hub.felhom.eu`), Gitea answers.
+4. The SMART exporter: delete the line `--smartctl.device=/dev/disk/by-id/ata-ST3000VX010-2E3166_Z6A07P2G` in
+   `homelab-manifests/admin-system/smartctl-exporter.yaml`, commit, sync — otherwise it keeps asking for a disk that
+   is gone.
+5. The next DooPlex backup (03:10) is green: `systemctl show dooplex-backup.service -p Result -p ExecMainStatus`.
+6. Delete the Alertmanager silence for `device=~".*_Z6A07P2G"` (it ends by itself on **2026-11-11 14:32 UTC**).
+   If the disk is still in the machine on that date, the two alarms come back **on purpose**.
+7. Tell CC: the row closes.