R-99 runbook (pbs-phantom-cleanup) + read-only ep0 listing: no phantom today, nothing deleted; R-444 trim measured on demo-hp; R-618 closed by ruling (150 -> 149)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS

documentation/runbooks/pbs-phantom-cleanup.md

@@ -0,0 +1,62 @@
+# Runbook — delete a phantom backup snapshot (an aborted upload's leftover) on the off-site backup server
+
+**Why it exists:** operator ruling 2026-10-06 (`architecture/09-update-architecture.md` §3 decision 140, R-99): *"Leftovers
+should be deleted."* A PBS daemon killed in the middle of an upload leaves a snapshot that holds nothing (measured 1 B,
+2026-07-28, F-CRIT-2). The box already refuses to count it as a backup (felhom-agent `internal/backup/runner.go`,
+`archivePlausiblyComplete`, the 1 MiB floor) and logs it once at WARN. Server-side prune never removes it (dry-run
+2026-07-28: keep-last 2 kept two real snapshots PLUS the phantom). So one accumulates per aborted upload, until a person
+deletes it — by this runbook, when one is seen. **ep0 is Tier 2, protected** (`runbooks/target-selection.md`): this runbook
+is the ONLY deletion the ruling allows there, and only of a snapshot this runbook proves to be a phantom.
+
+## When you see one
+
+The box's agent log says `… size N B is below the 1048576 B plausibility floor — an aborted/incomplete archive, not a
+successful backup` (once per volid). Or a session reads one in step 1.
+
+## 1. List — read only
+
+From DooPlex (`ssh root@167.233.158.164`; never via `felhom-pve → 10.77.0.1`):
+
+```bash
+# the server's own view, every namespace (size = what the manifest references; verification state)
+for ns in $(ls /mnt/pbs-datastore/ns); do
+  proxmox-backup-debug api get /admin/datastore/felhom-offsite/snapshots --ns "$ns" --output-format json-pretty
+done
+# the directories themselves (a snapshot with no manifest may not be listed by the API at all)
+python3 - < documentation/runbooks/pbs-phantom-list.py      # run on ep0: ssh root@… python3 - < that file
+```
+
+**A snapshot is a PHANTOM only when BOTH hold:**
+1. the server reports `size` below **1 MiB** (1,048,576 B) — or the directory has **no `index.json.blob`** (no manifest);
+2. it is **not** `protected`, and it is **not the newest** snapshot of its group while an upload for that group may still be
+   running (check `proxmox-backup-manager task list --all --limit 20` — no running `backup` task for that group).
+
+The smallest real backup ever measured is ~584 MiB; real ones on 2026-10-06 were ≥ 352 MiB. Anything between 1 MiB and the
+smallest real size is **NOT** a phantom by this runbook — leave it and ask the operator.
+
+**Write down, per namespace, the count of real snapshots** (everything that is not a phantom). This count is the control.
+
+## 2. Delete — one snapshot at a time, the command shown first
+
+```bash
+# print it, read it, then run it
+echo proxmox-backup-debug api delete /admin/datastore/felhom-offsite/snapshots \
+  --ns <namespace> --backup-type <ct|vm|host> --backup-id <id> --backup-time <unix time>
+proxmox-backup-debug api delete /admin/datastore/felhom-offsite/snapshots \
+  --ns <namespace> --backup-type <ct|vm|host> --backup-id <id> --backup-time <unix time>
+```
+
+Never `rm -rf` a snapshot directory, never `prune`, never touch a datastore, a namespace or a group. One `delete` per
+proven phantom.
+
+## 3. Prove nothing real moved
+
+Re-run step 1. **The count of real snapshots in every namespace must be exactly what it was before.** The deleted
+phantom is gone; nothing else changed. If a count moved, stop and tell the operator at once. Save both listings (before,
+after) under `documentation/audits/<session>/`.
+
+## The record of each use
+
+| Date | Namespace / group / time | Size | Why a phantom | Real counts before = after |
+|---|---|---|---|---|
+| 2026-10-06 | — | — | **No phantom existed** (9 snapshots, all ≥ 369,808,250 B, all verification `ok`; no directory without a manifest) — nothing deleted | Tester-2 1, demo-felhom 3, demo-hp 2, operator 2, tester-1 1 (`audits/ten-answers-2026-10-06/r99-ep0-listing.txt`) |

documentation/runbooks/pbs-phantom-list.py

@@ -0,0 +1,23 @@
+import os, re, json
+root = "/mnt/pbs-datastore"
+rows = []
+def walk_ns(base, ns):
+    for typ in ("ct", "vm", "host"):
+        tdir = os.path.join(base, typ)
+        if not os.path.isdir(tdir): continue
+        for gid in sorted(os.listdir(tdir)):
+            gdir = os.path.join(tdir, gid)
+            for snap in sorted(os.listdir(gdir)):
+                sdir = os.path.join(gdir, snap)
+                if not os.path.isdir(sdir) or not re.match(r"\d{4}-\d{2}-\d{2}T", snap): continue
+                files = os.listdir(sdir)
+                size = sum(os.path.getsize(os.path.join(sdir, f)) for f in files)
+                man = "index.json.blob" in files
+                manifest_files = []
+                rows.append({"ns": ns or "(root)", "group": f"{typ}/{gid}", "snap": snap, "files": sorted(files), "dir_bytes": size, "manifest": man})
+    nsdir = os.path.join(base, "ns")
+    if os.path.isdir(nsdir):
+        for n in sorted(os.listdir(nsdir)):
+            walk_ns(os.path.join(nsdir, n), (ns + "/" if ns else "") + n)
+walk_ns(root, "")
+print(json.dumps(rows))