# RUNBOOK RECORD — PBS prune moved server-side, write proof closed, GC scheduled (2026-07-27)

**Class:** supervised operational run. **No code, no version bump.** Follows
`RUNBOOK-ep0-cleanup-2026-07-27.md` (Parts A–C complete; Part D stopped correctly on the prune defect).

**Targets:** `ep0` / `felhom-hetzner`, `demo-felhom` (N100), `demo-hp` (HP t740).

> **Timezone note, corrected from the runbook's standing rule.** `ep0` runs **`Etc/UTC`**, not CEST —
> `timedatectl` reports `Time zone: Etc/UTC (UTC, +0000)`, NTP-synced. The CEST↔UTC split applies to
> the **boxes** (`demo-felhom`, `demo-hp` are CEST), not to ep0. PBS calendar events therefore resolve
> in **UTC** on this host. Both zones are still stated for every schedule below.

---

## Outcome

| Item | Result |
|---|---|
| Prune attempt by boxes | **disabled** — config only, no grant, no code |
| Server-side prune | **2 jobs live**, daily `03:30` UTC, `keep-last 2` |
| Write proof | **CLOSED — `TASK OK`, no job errors** |
| GC | **scheduled** `sun 04:30` UTC — **NOT run** |
| `verify-new` | **enabled** (operator ruling) |
| Legacy `demo-felhom-01` | **deleted** — namespace + ACLs + token (operator ruling, confirmed twice) |
| Old 13 GB copy | **still in place** |

---

## 1. Part 1 — the prune is CONFIG-GATED (branch taken: turn it off)

**Source landmark** — `internal/backup/runner.go:104`, inside `localPruneSpec`:

```go
if s.Type == "pbs" && !r.allowPBSPrune {
    // Not opted in → never prune the offsite DR
    return ""
}
```

`allowPBSPrune` is **derived, not a config key** — `cmd/felhom-agent/main.go:1348`:

```go
allowPBSPrune := !t.Primary && t.KeepLast > 0
```

So the off-switch is the tier's `keep_last`. **Safety checked before changing it:** `BackupTiers()`
(`internal/config/config.go:453-495`) rejects a tier only for an empty/duplicate `target_id` or
`cadence_seconds <= 0`. `keep_last` is **not** a rejection condition, and for extra tiers
`keep < 0 → 0` is legal and preserved. Only the **primary** tier clamps to ≥1 (`KeepLast()`), and
that is the local tier, untouched. So `keep_last: 0` disarms the prune **without disarming the tier**.

**`agent.json` diff — identical on both boxes** (backed up to `agent.json.pre-prunegate.bak`):

```diff
29c29
<         "keep_last": 2
---
>         "keep_last": 0
```

No in-flight restore-test or backup on either box at the time; both agents restarted cleanly.

**Verified after restart — both boxes:**

```
backup tier armed target=local       cadence=24h0m0s  keep_last=3 prune_pbs_allowed=false primary=true
backup tier armed target=felhom-pbs  cadence=168h0m0s keep_last=0 prune_pbs_allowed=false primary=false
backup: restore-test scheduler starting cadence=84h0m0s
```

**The PBS tier is still ARMED** (cadence 168h, no `tier REJECTED` line), the local tier is unchanged,
and the restore-test cadence is still the reverted 84h. **No prune right was granted to any box.**

---

## 2. Part 2 — server-side prune jobs on ep0

**Namespace-path form:** `prune-job create` takes `--ns <plain-namespace-name>` — **not**
`/datastore/<ds>/<ns>` and **not** `/ns/<ns>`. That is the form used. `--max-depth 0` = non-recursive.

```sh
proxmox-backup-manager prune-job create prune-demo-felhom \
  --store felhom-offsite --ns demo-felhom --max-depth 0 \
  --keep-last 2 --schedule "03:30" \
  --comment "R-82 retention keep-last=2, server-side (box tokens are write-only)"
# same for prune-demo-hp
```

| id | store | ns | schedule | max-depth | keep-last |
|---|---|---|---|---|---|
| `prune-demo-felhom` | felhom-offsite | demo-felhom | `03:30` | 0 | 2 |
| `prune-demo-hp` | felhom-offsite | demo-hp | `03:30` | 0 | 2 |

**Schedules, both zones:**

| Job | ep0 local (UTC) | CEST |
|---|---|---|
| prune (daily) | **03:30** | 05:30 |
| GC (weekly, Sun) | **04:30** | 06:30 |

**Ordering holds:** prune runs daily including Sunday, one hour before the weekly GC on the same
morning — so GC always sees the freshest prune result. **Daily was chosen deliberately**: with weekly
backups the prune is usually a no-op, and a no-op that runs beats a weekly one that misses.

> **The weekly backup is due-based and therefore drifts**, so no fixed schedule can guarantee
> non-overlap. Occasional overlap is possible and is accepted, not designed away.

---

## 3. Part 3 — dry run, then the real prune

### Dry run (verbatim) — DELETED NOTHING

Run with a **temporary `root@pam` token** granted `DatastoreAdmin`, removed on exit (the pattern
`felhom-tenantsync.sh` already uses). `prune-job run` has **no `--dry-run`**, so the client-side
`proxmox-backup-client prune … --dry-run` was used.

```
############ DRY RUN  ns=demo-felhom  keep-last=2 ############
| snapshot                     |                     date | action |
| ct/9201/2026-07-18T18:31:06Z | Sat Jul 18 18:31:06 2026 |   keep |
| ct/9201/2026-07-26T12:21:48Z | Sun Jul 26 12:21:48 2026 |   keep |

############ DRY RUN  ns=demo-hp  keep-last=2 ############
| snapshot                     |                     date | action |
| ct/9201/2026-07-26T15:42:42Z | Sun Jul 26 15:42:42 2026 | remove |
| ct/9201/2026-07-26T17:02:57Z | Sun Jul 26 17:02:57 2026 |   keep |
| ct/9201/2026-07-27T08:25:47Z | Mon Jul 27 08:25:47 2026 |   keep |
```

**Gate passed exactly as specified:** demo-hp loses only its excess above 2; **demo-felhom loses
nothing**. The STOP condition was not triggered.

### Real run — both `TASK OK`

```
prune job 'prune-demo-felhom'   → keep, keep                       → TASK OK
prune job 'prune-demo-hp'       → remove ct/9201/2026-07-26T15:42:42Z, keep, keep → TASK OK
```

| Namespace | Before | After | Newest snapshot |
|---|---|---|---|
| demo-felhom | 2 | **2** | `2026-07-26T12:21:48Z` — unchanged |
| demo-hp | 3 | **2** | `2026-07-27T08:25:47Z` — unchanged |
| demo-felhom-01 | 3 | 3 | untouched (no job) |

**Chunks: 9,787 → 9,787 — unchanged.** That is the load-bearing confirmation: prune removed snapshot
indexes and reclaimed no chunks.

**`df` moved by −77,824 B (−76 KB).** The runbook predicted no movement; the substance is right (no
chunks freed) but not literally zero, because the removed snapshot's **own index files**
(`root.pxar.didx` ~57 KB, `catalog.pcat1.didx`, `index.json.blob`, `pct.conf.blob`) occupy disk and go
with it. Nothing other than prune ran. **Chunk count, not `df`, is the correct probe for "did prune
reclaim data".**

---

## 4. Part 4 — the write proof, CLOSED

Driven through the real flow: `POST /api/guest-backup/trigger` → `TriggerNow()` (the UI's
„Mentés most"), which deliberately covers **every** tier in one quiesce window. **Not `--selftest`,
not raw `vzdump`.**

**Before vs after, same box, same path:**

```
07-27 08:25:47 UTC  vzdump (felhom-pbs) -> job errors     ← prune denied
07-27 09:37:29 UTC  vzdump (felhom-pbs) -> OK             ← after the Part 1 change
```

```
INFO: creating Proxmox Backup Server archive 'ct/9201/2026-07-27T09:37:29Z'
INFO: root.pxar: had to backup 123.95 MiB of 3.975 GiB (compressed 30.431 MiB) in 44.64 s
INFO: root.pxar: backup was done incrementally, reused 3.854 GiB (97.0%)
INFO: Duration: 45.80s
TASK OK
```

**PBS side:** `UPID:…backup:felhom-offsite:ct-9201:felhom@pbs!demo-hp` → **OK** (09:37:36 → 09:38:22).

| Evidence | Result |
|---|---|
| Snapshot identity | `ns/demo-hp/ct/9201/**2026-07-27T09:37:29Z**` |
| Contents | `root.pxar.didx` 57,416 B, `catalog.pcat1.didx`, `index.json.blob`, `pct.conf.blob`, `client.log.blob` — real, not an empty dir |
| Chunk delta | 9,787 → **9,813** (+26) |
| `df` delta | 13,579,243,520 → 13,612,040,192 (**+32.8 MB**) |
| Prune step | **absent entirely** — `keep_last: 0` means no `--prune-backups` flag, so the permission check is never reached |
| Job errors | **none** |

**The hub PBS-DR gauge evidence is NOT satisfied.** A +32.8 MB delta is below the gauge's 0.1 GB
display granularity, so it still reads 12.6 GB / 13 %. Stating that plainly rather than dressing it
up — the same call as last time, and it still stands.

---

## 5. Part 5 — GC scheduled, NOT run

```sh
proxmox-backup-manager datastore update felhom-offsite --gc-schedule "sun 04:30"
```

```
| name        | felhom-offsite     |
| path        | /mnt/pbs-datastore |
| gc-schedule | sun 04:30          |
| verify-new  | 1                  |
```

**GC was NOT run**, confirmed: `garbage-collection status` reports `upid` empty and `disk-chunks 0` —
it has never executed on this datastore.

Deliberate: every chunk still carries a **fresh atime from this morning's migration copy**, and GC
only reclaims chunks whose atime predates the cutoff. A run today would reclaim nothing regardless of
what Part 3 pruned. **The first meaningful GC is the scheduled one.**

---

## 6. Part 6 — `verify-new` ruling: ENABLED

**Operator ruling 2026-07-27: enable it.**

```sh
proxmox-backup-manager datastore update felhom-offsite --verify-new true
```

Rationale recorded: nothing on this PBS ever verified, so `verify_state` stayed `none` forever and the
hub's *"newest PBS snapshot failed verification"* alarm **could not fire** — inert, in the same class
as the hub 400ing an unknown event type. `verify-new` verifies each snapshot as it is created; with
`keep-last 2` that covers essentially the whole datastore and turns a dead check live, for a few
minutes of ep0 CPU per weekly backup.

> Watch item: ep0 is a 3.7 GB CX23 with **no swap**, and inline verification runs within the backup
> window. Today's full forced verify completed fine (~250 MiB/s, 0 errors), but see
> `RUNBOOK-ep0-datastore-volume-2026-07-27.md` for the rsync OOM on this same box.

---

## 7. Legacy `demo-felhom-01` namespace — DELETED (operator ruling, confirmed twice)

Not in the original runbook; surfaced because "a prune job per customer namespace" would have applied
`keep-last 2` to a retired install's archive. Flagged rather than actioned silently, then ruled on.

**Destroyed:** 3 snapshots, all 2026-07-04 (`14:37:16Z`, `14:55:44Z`, `17:51:34Z`), from the retired
`demo-felhom-01` install. No live box had written there since 07-04.

**Removed with it, so nothing dangles** (the same shape as this morning's `scratch` datastore):

| Object | Status |
|---|---|
| namespace `demo-felhom-01` | deleted (`namespace delete --delete-groups true`) |
| ACL `felhom@pbs` → `/datastore/felhom-offsite/demo-felhom-01` | deleted |
| ACL `felhom@pbs!demo-felhom-01` → same path | deleted |
| token `felhom@pbs!demo-felhom-01` | deleted |

**Final ACL state — four entries, write-only, live namespaces only:**

```
felhom@pbs              /datastore/felhom-offsite/demo-felhom  DatastoreBackup
felhom@pbs              /datastore/felhom-offsite/demo-hp      DatastoreBackup
felhom@pbs!demo-felhom  /datastore/felhom-offsite/demo-felhom  DatastoreBackup
felhom@pbs!demo-hp      /datastore/felhom-offsite/demo-hp      DatastoreBackup
```

Live namespaces verified intact throughout (demo-felhom 2, demo-hp 3). Datastore name and path
unchanged. The temporary `root@pam!nsdel` token **and its ACL** were both removed.

> **It freed almost nothing today, and that was stated before the confirmation:** the namespace
> directory was **620 KB** (index files only). Its chunks live in the shared `.chunks` store and come
> back only when GC's atime cutoff passes — weeks away, not today.

---

## 8. The demo-felhom prediction — CLOSED

The falsifiable claim from the previous runbook was: *demo-felhom sits at exactly 2 snapshots, so its
prune has silently had nothing to do; its next weekly backup makes 3 and will fail identically.*

**It is now neutralised on both halves, independently:**

1. **The box no longer attempts prune** — `prune_pbs_allowed=false` on demo-felhom, so no
   `--prune-backups` flag is passed and the permission check is unreachable.
2. **Server-side prune covers the namespace** — `prune-demo-felhom` runs daily at 03:30 UTC and was
   **verified live** (`TASK OK`, both snapshots kept).

So demo-felhom's third snapshot will be pruned by ep0 under `root@pam`, not by the box, and the
failure mode cannot recur. **Prediction closed — the failure will not reproduce.**

---

## 9. Old datastore copy — STILL IN PLACE

```
drwxr-xr-x 4 backup backup 4096 Jul  3 21:48 /srv/pbs-felhom
13G     /srv/pbs-felhom
```

Untouched. Rollback remains a two-line `datastore.cfg` revert. Volume: 98 G, 13 G used, 13 %.

---

## 10. Deferred / open

1. **R-89** (below) — hub-owned retention policy. Today's prune jobs are the first increment of it,
   not a stopgap.
2. **The restic delete-rights question** carried in R-89 — unanswered, and the more urgent half.
3. **GC has still never run.** First scheduled execution Sunday 04:30 UTC. Its first run is worth
   watching: it is the only thing that reclaims chunks, and nothing has ever exercised it here.
4. **Hub PBS-DR gauge granularity** — 0.1 GB steps mean routine incremental backups are invisible to
   it. Not a fault, but it cannot be used as write-proof evidence for small deltas.
5. **ep0 has no swap** (3.7 GB CX23) — see the volume runbook's OOM.

## 11. Observations

- **`keep_last: 0` is the correct off-switch precisely because it is derived, not a flag.** Setting it
  disables both the `--prune-backups` value and the `allowPBSPrune` gate in one edit, and the tier
  stays armed. A dedicated "don't prune" boolean would have been a second source of truth.
- **The false-negative was the real damage, not the unpruned snapshots.** demo-hp's PBS tier had
  reported failure on *every* backup since the tier was created on 07-26, while the data landed
  correctly every time. A tier that cries wolf on every success makes a genuine failure invisible —
  which is exactly what happened at 07:13 UTC, when a real outage produced the same-looking result.
- **Chunk count is the honest probe for prune, not `df`.** They answer different questions, and `df`
  moved here for a reason that has nothing to do with reclamation.
- `prune-job run` has no `--dry-run`; the gate must come from `proxmox-backup-client prune --dry-run`,
  which needs a token. The temporary-admin-token-then-delete pattern is the right shape and is already
  established in `felhom-tenantsync.sh`.