felhom.eu/documentation/runbooks/dooplex-remove-disk-Z6A07P2G.md

RUNBOOK — pull the failed disk Z6A07P2G out of DooPlex

Written 2026-10-12 by CC from read-only checks (audits/answers-2026-10-12/disk-Z6A07P2G.txt). Operator's job: the power-off and the hands. Register row R-934.

The disk — by SERIAL, never by sdX

Model	Seagate Skyhawk ST3000VX010-2E3166, 3 TB
Serial	Z6A07P2G
Port (2026-10-12)	the onboard SATA controller 0000:00:1f.2, port ata-6 (/dev/disk/by-path/pci-0000:00:1f.2-ata-6); kernel ata10
Fault	352 unreadable (pending) + 352 offline-uncorrectable sectors, flat since 2026-10-09

⚠ THE TWIN. DooPlex has a SECOND disk of the same model: ST3000VX010, serial Z6A070XZ — IN USE as /mnt/3_hdd, keep it. Z6A07P2G (pull) and Z6A070XZ (keep) differ only in the last four characters. Read the serial on the disk's label before you unplug anything. If you cannot read it, follow the cable to SATA port 6 on the board. Do not make the disk busy to find it by its LED: no reads on a failing disk.

Before power-off (already true on 2026-10-12 — re-check in one minute)

lsblk -o NAME,SERIAL,MOUNTPOINTS | grep Z6A07P2G      # no mountpoint
grep -n 9cf20140 /etc/fstab                          # only a '# DISABLED 2026-08-14' line
grep RequiresMountsFor /etc/systemd/system/k3s.service.d/wait-for-mounts.conf   # no /mnt/4_hdd
sudo kubectl -n longhorn-system get nodes.longhorn.io dooplex -o jsonpath='{.spec.disks}' | grep -c 4_hdd   # 0 (HDD4 entry removed 2026-10-12)

Nothing else reads the disk: no backup script, PV, hostPath or running app names /mnt/4_hdd (2026-10-12).

Pull it

1. Plan a window: DooPlex runs Gitea, the registry, the hub and Longhorn. A box keeps running while the hub is off.
2. sudo poweroff. Unplug power.
3. Find the disk with serial Z6A07P2G (label), on SATA port 6. Unplug its data and power cable. Remove it.
4. Power on.

After the first boot

1. All mounts back: findmnt /mnt/1_hdd /mnt/2_hdd /mnt/3_hdd /mnt/5_hdd /mnt/ssd_2 — five lines.
2. k3s up and Longhorn healthy: sudo kubectl -n longhorn-system get nodes.longhorn.io READY True; all volumes healthy (sudo kubectl -n longhorn-system get volumes.longhorn.io).
3. The hub answers (https://hub.felhom.eu), Gitea answers.
4. The SMART exporter: delete the line --smartctl.device=/dev/disk/by-id/ata-ST3000VX010-2E3166_Z6A07P2G in homelab-manifests/admin-system/smartctl-exporter.yaml, commit, sync — otherwise it keeps asking for a disk that is gone.
5. The next DooPlex backup (03:10) is green: systemctl show dooplex-backup.service -p Result -p ExecMainStatus.
6. Delete the Alertmanager silence for device=~".*_Z6A07P2G" (it ends by itself on 2026-11-11 14:32 UTC). If the disk is still in the machine on that date, the two alarms come back on purpose.
7. Tell CC: the row closes.