d
6c2c3dd574
gates / gates (push) Successful in 6m24s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
51 lines
2.8 KiB
Markdown
51 lines
2.8 KiB
Markdown
# RUNBOOK — pull the failed disk Z6A07P2G out of DooPlex
|
|
|
|
> Written 2026-10-12 by CC from read-only checks (`audits/answers-2026-10-12/disk-Z6A07P2G.txt`). Operator's job: the
|
|
> power-off and the hands. Register row R-934.
|
|
|
|
## The disk — by SERIAL, never by `sdX`
|
|
|
|
| | |
|
|
|---|---|
|
|
| Model | Seagate Skyhawk **ST3000VX010-2E3166**, 3 TB |
|
|
| Serial | **Z6A07P2G** |
|
|
| Port (2026-10-12) | the onboard SATA controller `0000:00:1f.2`, port **ata-6** (`/dev/disk/by-path/pci-0000:00:1f.2-ata-6`); kernel `ata10` |
|
|
| Fault | 352 unreadable (pending) + 352 offline-uncorrectable sectors, flat since 2026-10-09 |
|
|
|
|
**⚠ THE TWIN.** DooPlex has a SECOND disk of the same model: **ST3000VX010, serial Z6A070XZ — IN USE as `/mnt/3_hdd`, keep it.**
|
|
`Z6A07P2G` (pull) and `Z6A070XZ` (keep) differ only in the last four characters. **Read the serial on the disk's
|
|
label before you unplug anything.** If you cannot read it, follow the cable to SATA port 6 on the board. Do not
|
|
make the disk busy to find it by its LED: no reads on a failing disk.
|
|
|
|
## Before power-off (already true on 2026-10-12 — re-check in one minute)
|
|
|
|
```bash
|
|
lsblk -o NAME,SERIAL,MOUNTPOINTS | grep Z6A07P2G # no mountpoint
|
|
grep -n 9cf20140 /etc/fstab # only a '# DISABLED 2026-08-14' line
|
|
grep RequiresMountsFor /etc/systemd/system/k3s.service.d/wait-for-mounts.conf # no /mnt/4_hdd
|
|
sudo kubectl -n longhorn-system get nodes.longhorn.io dooplex -o jsonpath='{.spec.disks}' | grep -c 4_hdd # 0 (HDD4 entry removed 2026-10-12)
|
|
```
|
|
|
|
Nothing else reads the disk: no backup script, PV, hostPath or running app names `/mnt/4_hdd` (2026-10-12).
|
|
|
|
## Pull it
|
|
|
|
1. Plan a window: DooPlex runs Gitea, the registry, the hub and Longhorn. A box keeps running while the hub is off.
|
|
2. `sudo poweroff`. Unplug power.
|
|
3. Find the disk with serial **Z6A07P2G** (label), on SATA port 6. Unplug its data and power cable. Remove it.
|
|
4. Power on.
|
|
|
|
## After the first boot
|
|
|
|
1. All mounts back: `findmnt /mnt/1_hdd /mnt/2_hdd /mnt/3_hdd /mnt/5_hdd /mnt/ssd_2` — five lines.
|
|
2. k3s up and Longhorn healthy: `sudo kubectl -n longhorn-system get nodes.longhorn.io` READY True; all volumes
|
|
`healthy` (`sudo kubectl -n longhorn-system get volumes.longhorn.io`).
|
|
3. The hub answers (`https://hub.felhom.eu`), Gitea answers.
|
|
4. The SMART exporter: delete the line `--smartctl.device=/dev/disk/by-id/ata-ST3000VX010-2E3166_Z6A07P2G` in
|
|
`homelab-manifests/admin-system/smartctl-exporter.yaml`, commit, sync — otherwise it keeps asking for a disk that
|
|
is gone.
|
|
5. The next DooPlex backup (03:10) is green: `systemctl show dooplex-backup.service -p Result -p ExecMainStatus`.
|
|
6. Delete the Alertmanager silence for `device=~".*_Z6A07P2G"` (it ends by itself on **2026-11-11 14:32 UTC**).
|
|
If the disk is still in the machine on that date, the two alarms come back **on purpose**.
|
|
7. Tell CC: the row closes.
|