felhom.eu/documentation/runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md

RUNBOOK RECORD — PBS prune moved server-side, write proof closed, GC scheduled (2026-07-27)

Class: supervised operational run. No code, no version bump. Follows RUNBOOK-ep0-cleanup-2026-07-27.md (Parts A–C complete; Part D stopped correctly on the prune defect).

Targets: ep0 / felhom-hetzner, demo-felhom (N100), demo-hp (HP t740).

Timezone note, corrected from the runbook's standing rule. ep0 runs Etc/UTC, not CEST — timedatectl reports Time zone: Etc/UTC (UTC, +0000), NTP-synced. The CEST↔UTC split applies to the boxes (demo-felhom, demo-hp are CEST), not to ep0. PBS calendar events therefore resolve in UTC on this host. Both zones are still stated for every schedule below.

---

Outcome

Item	Result
Prune attempt by boxes	disabled — config only, no grant, no code
Server-side prune	2 jobs live, daily 03:30 UTC, keep-last 2
Write proof	CLOSED — TASK OK, no job errors
GC	scheduled sun 04:30 UTC — NOT run
verify-new	enabled (operator ruling)
Legacy demo-felhom-01	deleted — namespace + ACLs + token (operator ruling, confirmed twice)
Old 13 GB copy	still in place

---

1. Part 1 — the prune is CONFIG-GATED (branch taken: turn it off)

Source landmark — internal/backup/runner.go:104, inside localPruneSpec:

if s.Type == "pbs" && !r.allowPBSPrune {
    // Not opted in → never prune the offsite DR
    return ""
}

allowPBSPrune is derived, not a config key — cmd/felhom-agent/main.go:1348:

allowPBSPrune := !t.Primary && t.KeepLast > 0

So the off-switch is the tier's keep_last. Safety checked before changing it: BackupTiers() (internal/config/config.go:453-495) rejects a tier only for an empty/duplicate target_id or cadence_seconds <= 0. keep_last is not a rejection condition, and for extra tiers keep < 0 → 0 is legal and preserved. Only the primary tier clamps to ≥1 (KeepLast()), and that is the local tier, untouched. So keep_last: 0 disarms the prune without disarming the tier.

agent.json diff — identical on both boxes (backed up to agent.json.pre-prunegate.bak):

29c29
<         "keep_last": 2
---
>         "keep_last": 0

No in-flight restore-test or backup on either box at the time; both agents restarted cleanly.

Verified after restart — both boxes:

backup tier armed target=local       cadence=24h0m0s  keep_last=3 prune_pbs_allowed=false primary=true
backup tier armed target=felhom-pbs  cadence=168h0m0s keep_last=0 prune_pbs_allowed=false primary=false
backup: restore-test scheduler starting cadence=84h0m0s

The PBS tier is still ARMED (cadence 168h, no tier REJECTED line), the local tier is unchanged, and the restore-test cadence is still the reverted 84h. No prune right was granted to any box.

---

2. Part 2 — server-side prune jobs on ep0

Namespace-path form: prune-job create takes --ns <plain-namespace-name> — not /datastore/<ds>/<ns> and not /ns/<ns>. That is the form used. --max-depth 0 = non-recursive.

proxmox-backup-manager prune-job create prune-demo-felhom \
  --store felhom-offsite --ns demo-felhom --max-depth 0 \
  --keep-last 2 --schedule "03:30" \
  --comment "R-82 retention keep-last=2, server-side (box tokens are write-only)"
# same for prune-demo-hp

id	store	ns	schedule	max-depth	keep-last
prune-demo-felhom	felhom-offsite	demo-felhom	03:30	0	2
prune-demo-hp	felhom-offsite	demo-hp	03:30	0	2

Schedules, both zones:

Job	ep0 local (UTC)	CEST
prune (daily)	03:30	05:30
GC (weekly, Sun)	04:30	06:30

Ordering holds: prune runs daily including Sunday, one hour before the weekly GC on the same morning — so GC always sees the freshest prune result. Daily was chosen deliberately: with weekly backups the prune is usually a no-op, and a no-op that runs beats a weekly one that misses.

The weekly backup is due-based and therefore drifts, so no fixed schedule can guarantee non-overlap. Occasional overlap is possible and is accepted, not designed away.

---

3. Part 3 — dry run, then the real prune

Dry run (verbatim) — DELETED NOTHING

Run with a temporary root@pam token granted DatastoreAdmin, removed on exit (the pattern felhom-tenantsync.sh already uses). prune-job run has no --dry-run, so the client-side proxmox-backup-client prune … --dry-run was used.

############ DRY RUN  ns=demo-felhom  keep-last=2 ############
| snapshot                     |                     date | action |
| ct/9201/2026-07-18T18:31:06Z | Sat Jul 18 18:31:06 2026 |   keep |
| ct/9201/2026-07-26T12:21:48Z | Sun Jul 26 12:21:48 2026 |   keep |

############ DRY RUN  ns=demo-hp  keep-last=2 ############
| snapshot                     |                     date | action |
| ct/9201/2026-07-26T15:42:42Z | Sun Jul 26 15:42:42 2026 | remove |
| ct/9201/2026-07-26T17:02:57Z | Sun Jul 26 17:02:57 2026 |   keep |
| ct/9201/2026-07-27T08:25:47Z | Mon Jul 27 08:25:47 2026 |   keep |

Gate passed exactly as specified: demo-hp loses only its excess above 2; demo-felhom loses nothing. The STOP condition was not triggered.

Real run — both TASK OK

prune job 'prune-demo-felhom'   → keep, keep                       → TASK OK
prune job 'prune-demo-hp'       → remove ct/9201/2026-07-26T15:42:42Z, keep, keep → TASK OK

Namespace	Before	After	Newest snapshot
demo-felhom	2	2	2026-07-26T12:21:48Z — unchanged
demo-hp	3	2	2026-07-27T08:25:47Z — unchanged
demo-felhom-01	3	3	untouched (no job)

Chunks: 9,787 → 9,787 — unchanged. That is the load-bearing confirmation: prune removed snapshot indexes and reclaimed no chunks.

df moved by −77,824 B (−76 KB). The runbook predicted no movement; the substance is right (no chunks freed) but not literally zero, because the removed snapshot's own index files (root.pxar.didx ~57 KB, catalog.pcat1.didx, index.json.blob, pct.conf.blob) occupy disk and go with it. Nothing other than prune ran. Chunk count, not df, is the correct probe for "did prune reclaim data".

---

4. Part 4 — the write proof, CLOSED

Driven through the real flow: POST /api/guest-backup/trigger → TriggerNow() (the UI's „Mentés most"), which deliberately covers every tier in one quiesce window. Not --selftest, not raw vzdump.

Before vs after, same box, same path:

07-27 08:25:47 UTC  vzdump (felhom-pbs) -> job errors     ← prune denied
07-27 09:37:29 UTC  vzdump (felhom-pbs) -> OK             ← after the Part 1 change

INFO: creating Proxmox Backup Server archive 'ct/9201/2026-07-27T09:37:29Z'
INFO: root.pxar: had to backup 123.95 MiB of 3.975 GiB (compressed 30.431 MiB) in 44.64 s
INFO: root.pxar: backup was done incrementally, reused 3.854 GiB (97.0%)
INFO: Duration: 45.80s
TASK OK

PBS side: UPID:…backup:felhom-offsite:ct-9201:felhom@pbs!demo-hp → OK (09:37:36 → 09:38:22).

Evidence	Result
Snapshot identity	ns/demo-hp/ct/9201/**2026-07-27T09:37:29Z**
Contents	root.pxar.didx 57,416 B, catalog.pcat1.didx, index.json.blob, pct.conf.blob, client.log.blob — real, not an empty dir
Chunk delta	9,787 → 9,813 (+26)
df delta	13,579,243,520 → 13,612,040,192 (+32.8 MB)
Prune step	absent entirely — keep_last: 0 means no --prune-backups flag, so the permission check is never reached
Job errors	none

The hub PBS-DR gauge evidence is NOT satisfied. A +32.8 MB delta is below the gauge's 0.1 GB display granularity, so it still reads 12.6 GB / 13 %. Stating that plainly rather than dressing it up — the same call as last time, and it still stands.

---

5. Part 5 — GC scheduled, NOT run

proxmox-backup-manager datastore update felhom-offsite --gc-schedule "sun 04:30"

| name        | felhom-offsite     |
| path        | /mnt/pbs-datastore |
| gc-schedule | sun 04:30          |
| verify-new  | 1                  |

GC was NOT run, confirmed: garbage-collection status reports upid empty and disk-chunks 0 — it has never executed on this datastore.

Deliberate: every chunk still carries a fresh atime from this morning's migration copy, and GC only reclaims chunks whose atime predates the cutoff. A run today would reclaim nothing regardless of what Part 3 pruned. The first meaningful GC is the scheduled one.

---

6. Part 6 — verify-new ruling: ENABLED

Operator ruling 2026-07-27: enable it.

proxmox-backup-manager datastore update felhom-offsite --verify-new true

Rationale recorded: nothing on this PBS ever verified, so verify_state stayed none forever and the hub's "newest PBS snapshot failed verification" alarm could not fire — inert, in the same class as the hub 400ing an unknown event type. verify-new verifies each snapshot as it is created; with keep-last 2 that covers essentially the whole datastore and turns a dead check live, for a few minutes of ep0 CPU per weekly backup.

Watch item (superseded 2026-08-03: ep0 is now a CX33, 8 GB RAM, and it HAS a 4 GiB swapfile which survived the resize — so the pressure below is much reduced, though the shape of the concern stands). As written: ep0 is a 3.7 GB CX23 with no swap, and inline verification runs within the backup window. Today's full forced verify completed fine (~250 MiB/s, 0 errors), but see RUNBOOK-ep0-datastore-volume-2026-07-27.md for the rsync OOM on this same box.

---

7. Legacy demo-felhom-01 namespace — DELETED (operator ruling, confirmed twice)

Not in the original runbook; surfaced because "a prune job per customer namespace" would have applied keep-last 2 to a retired install's archive. Flagged rather than actioned silently, then ruled on.

Destroyed: 3 snapshots, all 2026-07-04 (14:37:16Z, 14:55:44Z, 17:51:34Z), from the retired demo-felhom-01 install. No live box had written there since 07-04.

Removed with it, so nothing dangles (the same shape as this morning's scratch datastore):

Object	Status
namespace demo-felhom-01	deleted (namespace delete --delete-groups true)
ACL felhom@pbs → /datastore/felhom-offsite/demo-felhom-01	deleted
ACL felhom@pbs!demo-felhom-01 → same path	deleted
token felhom@pbs!demo-felhom-01	deleted

Final ACL state — four entries, write-only, live namespaces only:

felhom@pbs              /datastore/felhom-offsite/demo-felhom  DatastoreBackup
felhom@pbs              /datastore/felhom-offsite/demo-hp      DatastoreBackup
felhom@pbs!demo-felhom  /datastore/felhom-offsite/demo-felhom  DatastoreBackup
felhom@pbs!demo-hp      /datastore/felhom-offsite/demo-hp      DatastoreBackup

Live namespaces verified intact throughout (demo-felhom 2, demo-hp 3). Datastore name and path unchanged. The temporary root@pam!nsdel token and its ACL were both removed.

It freed almost nothing today, and that was stated before the confirmation: the namespace directory was 620 KB (index files only). Its chunks live in the shared .chunks store and come back only when GC's atime cutoff passes — weeks away, not today.

---

8. The demo-felhom prediction — CLOSED

The falsifiable claim from the previous runbook was: demo-felhom sits at exactly 2 snapshots, so its prune has silently had nothing to do; its next weekly backup makes 3 and will fail identically.

It is now neutralised on both halves, independently:

1. The box no longer attempts prune — prune_pbs_allowed=false on demo-felhom, so no --prune-backups flag is passed and the permission check is unreachable.
2. Server-side prune covers the namespace — prune-demo-felhom runs daily at 03:30 UTC and was verified live (TASK OK, both snapshots kept).

So demo-felhom's third snapshot will be pruned by ep0 under root@pam, not by the box, and the failure mode cannot recur. Prediction closed — the failure will not reproduce.

---

9. Old datastore copy — STILL IN PLACE

drwxr-xr-x 4 backup backup 4096 Jul  3 21:48 /srv/pbs-felhom
13G     /srv/pbs-felhom

Untouched. Rollback remains a two-line datastore.cfg revert. Volume: 98 G, 13 G used, 13 %.

---

10. Deferred / open

1. R-89 (below) — hub-owned retention policy. Today's prune jobs are the first increment of it, not a stopgap.
2. The restic delete-rights question carried in R-89 — unanswered, and the more urgent half.
3. GC has still never run. First scheduled execution Sunday 04:30 UTC. Its first run is worth watching: it is the only thing that reclaims chunks, and nothing has ever exercised it here.
4. Hub PBS-DR gauge granularity — 0.1 GB steps mean routine incremental backups are invisible to it. Not a fault, but it cannot be used as write-proof evidence for small deltas.
5. ep0 has no swap (3.7 GB CX23) — corrected 2026-08-03: ep0 is a CX33 with 8 GB RAM and an active 4 GiB swapfile. See the volume runbook's OOM for the original incident.

11. Observations

• keep_last: 0 is the correct off-switch precisely because it is derived, not a flag. Setting it disables both the --prune-backups value and the allowPBSPrune gate in one edit, and the tier stays armed. A dedicated "don't prune" boolean would have been a second source of truth.
• The false-negative was the real damage, not the unpruned snapshots. demo-hp's PBS tier had reported failure on every backup since the tier was created on 07-26, while the data landed correctly every time. A tier that cries wolf on every success makes a genuine failure invisible — which is exactly what happened at 07:13 UTC, when a real outage produced the same-looking result.
• Chunk count is the honest probe for prune, not df. They answer different questions, and df moved here for a reason that has nothing to do with reclamation.
• prune-job run has no --dry-run; the gate must come from proxmox-backup-client prune --dry-run, which needs a token. The temporary-admin-token-then-delete pattern is the right shape and is already established in felhom-tenantsync.sh.