felhom.eu/documentation/runbooks/pbs-phantom-cleanup.md

Runbook — delete a phantom backup snapshot (an aborted upload's leftover) on the off-site backup server

Why it exists: operator ruling 2026-10-06 (architecture/09-update-architecture.md §3 decision 140, R-99): "Leftovers should be deleted." A PBS daemon killed in the middle of an upload leaves a snapshot that holds nothing (measured 1 B, 2026-07-28, F-CRIT-2). The box already refuses to count it as a backup (felhom-agent internal/backup/runner.go, archivePlausiblyComplete, the 1 MiB floor) and logs it once at WARN. Server-side prune never removes it (dry-run 2026-07-28: keep-last 2 kept two real snapshots PLUS the phantom). So one accumulates per aborted upload, until a person deletes it — by this runbook, when one is seen. ep0 is Tier 2, protected (runbooks/target-selection.md): this runbook is the ONLY deletion the ruling allows there, and only of a snapshot this runbook proves to be a phantom.

When you see one

The box's agent log says … size N B is below the 1048576 B plausibility floor — an aborted/incomplete archive, not a successful backup (once per volid). Or a session reads one in step 1.

1. List — read only

From DooPlex (ssh root@167.233.158.164; never via felhom-pve → 10.77.0.1):

# the server's own view, every namespace (size = what the manifest references; verification state)
for ns in $(ls /mnt/pbs-datastore/ns); do
  proxmox-backup-debug api get /admin/datastore/felhom-offsite/snapshots --ns "$ns" --output-format json-pretty
done
# the directories themselves (a snapshot with no manifest may not be listed by the API at all)
python3 - < documentation/runbooks/pbs-phantom-list.py      # run on ep0: ssh root@… python3 - < that file

A snapshot is a PHANTOM only when BOTH hold:

1. the server reports size below 1 MiB (1,048,576 B) — or the directory has no index.json.blob (no manifest);
2. it is not protected, and it is not the newest snapshot of its group while an upload for that group may still be running (check proxmox-backup-manager task list --all --limit 20 — no running backup task for that group).

The smallest real backup ever measured is ~584 MiB; real ones on 2026-10-06 were ≥ 352 MiB. Anything between 1 MiB and the smallest real size is NOT a phantom by this runbook — leave it and ask the operator.

Write down, per namespace, the count of real snapshots (everything that is not a phantom). This count is the control.

2. Delete — one snapshot at a time, the command shown first

# print it, read it, then run it
echo proxmox-backup-debug api delete /admin/datastore/felhom-offsite/snapshots \
  --ns <namespace> --backup-type <ct|vm|host> --backup-id <id> --backup-time <unix time>
proxmox-backup-debug api delete /admin/datastore/felhom-offsite/snapshots \
  --ns <namespace> --backup-type <ct|vm|host> --backup-id <id> --backup-time <unix time>

Never rm -rf a snapshot directory, never prune, never touch a datastore, a namespace or a group. One delete per proven phantom.

3. Prove nothing real moved

Re-run step 1. The count of real snapshots in every namespace must be exactly what it was before. The deleted phantom is gone; nothing else changed. If a count moved, stop and tell the operator at once. Save both listings (before, after) under documentation/audits/<session>/.

The record of each use

Date	Namespace / group / time	Size	Why a phantom	Real counts before = after
2026-10-06	—	—	No phantom existed (9 snapshots, all ≥ 369,808,250 B, all verification ok; no directory without a manifest) — nothing deleted	Tester-2 1, demo-felhom 3, demo-hp 2, operator 2, tester-1 1 (audits/ten-answers-2026-10-06/r99-ep0-listing.txt)