admin
6b5804c63c
gates / gates (push) Successful in 2m18s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
63 lines
3.7 KiB
Markdown
63 lines
3.7 KiB
Markdown
# Runbook — delete a phantom backup snapshot (an aborted upload's leftover) on the off-site backup server
|
|
|
|
**Why it exists:** operator ruling 2026-10-06 (`architecture/09-update-architecture.md` §3 decision 140, R-99): *"Leftovers
|
|
should be deleted."* A PBS daemon killed in the middle of an upload leaves a snapshot that holds nothing (measured 1 B,
|
|
2026-07-28, F-CRIT-2). The box already refuses to count it as a backup (felhom-agent `internal/backup/runner.go`,
|
|
`archivePlausiblyComplete`, the 1 MiB floor) and logs it once at WARN. Server-side prune never removes it (dry-run
|
|
2026-07-28: keep-last 2 kept two real snapshots PLUS the phantom). So one accumulates per aborted upload, until a person
|
|
deletes it — by this runbook, when one is seen. **ep0 is Tier 2, protected** (`runbooks/target-selection.md`): this runbook
|
|
is the ONLY deletion the ruling allows there, and only of a snapshot this runbook proves to be a phantom.
|
|
|
|
## When you see one
|
|
|
|
The box's agent log says `… size N B is below the 1048576 B plausibility floor — an aborted/incomplete archive, not a
|
|
successful backup` (once per volid). Or a session reads one in step 1.
|
|
|
|
## 1. List — read only
|
|
|
|
From DooPlex (`ssh root@167.233.158.164`; never via `felhom-pve → 10.77.0.1`):
|
|
|
|
```bash
|
|
# the server's own view, every namespace (size = what the manifest references; verification state)
|
|
for ns in $(ls /mnt/pbs-datastore/ns); do
|
|
proxmox-backup-debug api get /admin/datastore/felhom-offsite/snapshots --ns "$ns" --output-format json-pretty
|
|
done
|
|
# the directories themselves (a snapshot with no manifest may not be listed by the API at all)
|
|
python3 - < documentation/runbooks/pbs-phantom-list.py # run on ep0: ssh root@… python3 - < that file
|
|
```
|
|
|
|
**A snapshot is a PHANTOM only when BOTH hold:**
|
|
1. the server reports `size` below **1 MiB** (1,048,576 B) — or the directory has **no `index.json.blob`** (no manifest);
|
|
2. it is **not** `protected`, and it is **not the newest** snapshot of its group while an upload for that group may still be
|
|
running (check `proxmox-backup-manager task list --all --limit 20` — no running `backup` task for that group).
|
|
|
|
The smallest real backup ever measured is ~584 MiB; real ones on 2026-10-06 were ≥ 352 MiB. Anything between 1 MiB and the
|
|
smallest real size is **NOT** a phantom by this runbook — leave it and ask the operator.
|
|
|
|
**Write down, per namespace, the count of real snapshots** (everything that is not a phantom). This count is the control.
|
|
|
|
## 2. Delete — one snapshot at a time, the command shown first
|
|
|
|
```bash
|
|
# print it, read it, then run it
|
|
echo proxmox-backup-debug api delete /admin/datastore/felhom-offsite/snapshots \
|
|
--ns <namespace> --backup-type <ct|vm|host> --backup-id <id> --backup-time <unix time>
|
|
proxmox-backup-debug api delete /admin/datastore/felhom-offsite/snapshots \
|
|
--ns <namespace> --backup-type <ct|vm|host> --backup-id <id> --backup-time <unix time>
|
|
```
|
|
|
|
Never `rm -rf` a snapshot directory, never `prune`, never touch a datastore, a namespace or a group. One `delete` per
|
|
proven phantom.
|
|
|
|
## 3. Prove nothing real moved
|
|
|
|
Re-run step 1. **The count of real snapshots in every namespace must be exactly what it was before.** The deleted
|
|
phantom is gone; nothing else changed. If a count moved, stop and tell the operator at once. Save both listings (before,
|
|
after) under `documentation/audits/<session>/`.
|
|
|
|
## The record of each use
|
|
|
|
| Date | Namespace / group / time | Size | Why a phantom | Real counts before = after |
|
|
|---|---|---|---|---|
|
|
| 2026-10-06 | — | — | **No phantom existed** (9 snapshots, all ≥ 369,808,250 B, all verification `ok`; no directory without a manifest) — nothing deleted | Tester-2 1, demo-felhom 3, demo-hp 2, operator 2, tester-1 1 (`audits/ten-answers-2026-10-06/r99-ep0-listing.txt`) |
|