# Runbook — delete a phantom backup snapshot (an aborted upload's leftover) on the off-site backup server

**Why it exists:** operator ruling 2026-10-06 (`architecture/09-update-architecture.md` §3 decision 140, R-99): *"Leftovers
should be deleted."* A PBS daemon killed in the middle of an upload leaves a snapshot that holds nothing (measured 1 B,
2026-07-28, F-CRIT-2). The box already refuses to count it as a backup (felhom-agent `internal/backup/runner.go`,
`archivePlausiblyComplete`, the 1 MiB floor) and logs it once at WARN. Server-side prune never removes it (dry-run
2026-07-28: keep-last 2 kept two real snapshots PLUS the phantom). So one accumulates per aborted upload, until a person
deletes it — by this runbook, when one is seen. **ep0 is Tier 2, protected** (`runbooks/target-selection.md`): this runbook
is the ONLY deletion the ruling allows there, and only of a snapshot this runbook proves to be a phantom.

## When you see one

The box's agent log says `… size N B is below the 1048576 B plausibility floor — an aborted/incomplete archive, not a
successful backup` (once per volid). Or a session reads one in step 1.

## 1. List — read only

From DooPlex (`ssh root@167.233.158.164`; never via `felhom-pve → 10.77.0.1`):

```bash
# the server's own view, every namespace (size = what the manifest references; verification state)
for ns in $(ls /mnt/pbs-datastore/ns); do
  proxmox-backup-debug api get /admin/datastore/felhom-offsite/snapshots --ns "$ns" --output-format json-pretty
done
# the directories themselves (a snapshot with no manifest may not be listed by the API at all)
python3 - < documentation/runbooks/pbs-phantom-list.py      # run on ep0: ssh root@… python3 - < that file
```

**A snapshot is a PHANTOM only when BOTH hold:**
1. the server reports `size` below **1 MiB** (1,048,576 B) — or the directory has **no `index.json.blob`** (no manifest);
2. it is **not** `protected`, and it is **not the newest** snapshot of its group while an upload for that group may still be
   running (check `proxmox-backup-manager task list --all --limit 20` — no running `backup` task for that group).

The smallest real backup ever measured is ~584 MiB; real ones on 2026-10-06 were ≥ 352 MiB. Anything between 1 MiB and the
smallest real size is **NOT** a phantom by this runbook — leave it and ask the operator.

**Write down, per namespace, the count of real snapshots** (everything that is not a phantom). This count is the control.

## 2. Delete — one snapshot at a time, the command shown first

```bash
# print it, read it, then run it
echo proxmox-backup-debug api delete /admin/datastore/felhom-offsite/snapshots \
  --ns <namespace> --backup-type <ct|vm|host> --backup-id <id> --backup-time <unix time>
proxmox-backup-debug api delete /admin/datastore/felhom-offsite/snapshots \
  --ns <namespace> --backup-type <ct|vm|host> --backup-id <id> --backup-time <unix time>
```

Never `rm -rf` a snapshot directory, never `prune`, never touch a datastore, a namespace or a group. One `delete` per
proven phantom.

## 3. Prove nothing real moved

Re-run step 1. **The count of real snapshots in every namespace must be exactly what it was before.** The deleted
phantom is gone; nothing else changed. If a count moved, stop and tell the operator at once. Save both listings (before,
after) under `documentation/audits/<session>/`.

## The record of each use

| Date | Namespace / group / time | Size | Why a phantom | Real counts before = after |
|---|---|---|---|---|
| 2026-10-06 | — | — | **No phantom existed** (9 snapshots, all ≥ 369,808,250 B, all verification `ok`; no directory without a manifest) — nothing deleted | Tester-2 1, demo-felhom 3, demo-hp 2, operator 2, tester-1 1 (`audits/ten-answers-2026-10-06/r99-ep0-listing.txt`) |