admin
a31872ea24
Supervised runbook execution. No code, no version bump. The felhom-pbs tier had reported `job errors` on EVERY demo-hp backup since the tier was created on 07-26, while the data landed correctly every time: `DatastoreBackup` grants Datastore.Backup but not Datastore.Prune, so the box's keep_last=2 prune was denied. Operator ruling: retention is a COMMERCIAL attribute owned by the hub; ep0 executes. Box tokens therefore stay write-only - a compromised box must not be able to delete its own offsite backups. No grant was widened and felhom-tenantsync.sh is unchanged (the ruling makes it correct). Increment 1: - boxes stop attempting prune. allowPBSPrune is DERIVED (`!t.Primary && t.KeepLast > 0`), so keep_last: 0 on the PBS tier disables both the --prune-backups value and the gate in one config edit, and the tier stays armed. Verified prune_pbs_allowed=false on both boxes with no tier REJECTED line. - per-namespace prune jobs on ep0, keep-last 2, daily 03:30 UTC (05:30 CEST), dry-run gated. demo-hp 3->2, demo-felhom untouched, chunk count unchanged (prune removes indexes, not chunks). Write proof CLOSED: 08:25:47 job errors -> 09:37:29 TASK OK, snapshot 2026-07-27T09:37:29Z, chunks 9787->9813, prune step absent entirely. Driven through POST /api/guest-backup/trigger (the UI path), not --selftest and not raw vzdump. Hub gauge evidence explicitly NOT satisfied - the delta is below its 0.1 GB display granularity. GC scheduled sun 04:30 UTC and deliberately NOT run: every chunk still carries a fresh atime from the migration copy, so a run today would reclaim nothing. verify-new enabled per operator ruling, turning an inert hub alarm live. Legacy demo-felhom-01 namespace deleted with its two ACL entries and its token (operator ruling, confirmed twice) so nothing dangles. R-89 records the target architecture and carries the unanswered parallel question: does the restic key on storage-box-pool-1 have DELETE rights? If so the daily app-data tier has the identical exposure and append-only is the equivalent answer. ep0 is Etc/UTC, not CEST - corrected in the record.
326 lines
14 KiB
Markdown
326 lines
14 KiB
Markdown
# RUNBOOK RECORD — PBS prune moved server-side, write proof closed, GC scheduled (2026-07-27)
|
||
|
||
**Class:** supervised operational run. **No code, no version bump.** Follows
|
||
`RUNBOOK-ep0-cleanup-2026-07-27.md` (Parts A–C complete; Part D stopped correctly on the prune defect).
|
||
|
||
**Targets:** `ep0` / `felhom-hetzner`, `demo-felhom` (N100), `demo-hp` (HP t740).
|
||
|
||
> **Timezone note, corrected from the runbook's standing rule.** `ep0` runs **`Etc/UTC`**, not CEST —
|
||
> `timedatectl` reports `Time zone: Etc/UTC (UTC, +0000)`, NTP-synced. The CEST↔UTC split applies to
|
||
> the **boxes** (`demo-felhom`, `demo-hp` are CEST), not to ep0. PBS calendar events therefore resolve
|
||
> in **UTC** on this host. Both zones are still stated for every schedule below.
|
||
|
||
---
|
||
|
||
## Outcome
|
||
|
||
| Item | Result |
|
||
|---|---|
|
||
| Prune attempt by boxes | **disabled** — config only, no grant, no code |
|
||
| Server-side prune | **2 jobs live**, daily `03:30` UTC, `keep-last 2` |
|
||
| Write proof | **CLOSED — `TASK OK`, no job errors** |
|
||
| GC | **scheduled** `sun 04:30` UTC — **NOT run** |
|
||
| `verify-new` | **enabled** (operator ruling) |
|
||
| Legacy `demo-felhom-01` | **deleted** — namespace + ACLs + token (operator ruling, confirmed twice) |
|
||
| Old 13 GB copy | **still in place** |
|
||
|
||
---
|
||
|
||
## 1. Part 1 — the prune is CONFIG-GATED (branch taken: turn it off)
|
||
|
||
**Source landmark** — `internal/backup/runner.go:104`, inside `localPruneSpec`:
|
||
|
||
```go
|
||
if s.Type == "pbs" && !r.allowPBSPrune {
|
||
// Not opted in → never prune the offsite DR
|
||
return ""
|
||
}
|
||
```
|
||
|
||
`allowPBSPrune` is **derived, not a config key** — `cmd/felhom-agent/main.go:1348`:
|
||
|
||
```go
|
||
allowPBSPrune := !t.Primary && t.KeepLast > 0
|
||
```
|
||
|
||
So the off-switch is the tier's `keep_last`. **Safety checked before changing it:** `BackupTiers()`
|
||
(`internal/config/config.go:453-495`) rejects a tier only for an empty/duplicate `target_id` or
|
||
`cadence_seconds <= 0`. `keep_last` is **not** a rejection condition, and for extra tiers
|
||
`keep < 0 → 0` is legal and preserved. Only the **primary** tier clamps to ≥1 (`KeepLast()`), and
|
||
that is the local tier, untouched. So `keep_last: 0` disarms the prune **without disarming the tier**.
|
||
|
||
**`agent.json` diff — identical on both boxes** (backed up to `agent.json.pre-prunegate.bak`):
|
||
|
||
```diff
|
||
29c29
|
||
< "keep_last": 2
|
||
---
|
||
> "keep_last": 0
|
||
```
|
||
|
||
No in-flight restore-test or backup on either box at the time; both agents restarted cleanly.
|
||
|
||
**Verified after restart — both boxes:**
|
||
|
||
```
|
||
backup tier armed target=local cadence=24h0m0s keep_last=3 prune_pbs_allowed=false primary=true
|
||
backup tier armed target=felhom-pbs cadence=168h0m0s keep_last=0 prune_pbs_allowed=false primary=false
|
||
backup: restore-test scheduler starting cadence=84h0m0s
|
||
```
|
||
|
||
**The PBS tier is still ARMED** (cadence 168h, no `tier REJECTED` line), the local tier is unchanged,
|
||
and the restore-test cadence is still the reverted 84h. **No prune right was granted to any box.**
|
||
|
||
---
|
||
|
||
## 2. Part 2 — server-side prune jobs on ep0
|
||
|
||
**Namespace-path form:** `prune-job create` takes `--ns <plain-namespace-name>` — **not**
|
||
`/datastore/<ds>/<ns>` and **not** `/ns/<ns>`. That is the form used. `--max-depth 0` = non-recursive.
|
||
|
||
```sh
|
||
proxmox-backup-manager prune-job create prune-demo-felhom \
|
||
--store felhom-offsite --ns demo-felhom --max-depth 0 \
|
||
--keep-last 2 --schedule "03:30" \
|
||
--comment "R-82 retention keep-last=2, server-side (box tokens are write-only)"
|
||
# same for prune-demo-hp
|
||
```
|
||
|
||
| id | store | ns | schedule | max-depth | keep-last |
|
||
|---|---|---|---|---|---|
|
||
| `prune-demo-felhom` | felhom-offsite | demo-felhom | `03:30` | 0 | 2 |
|
||
| `prune-demo-hp` | felhom-offsite | demo-hp | `03:30` | 0 | 2 |
|
||
|
||
**Schedules, both zones:**
|
||
|
||
| Job | ep0 local (UTC) | CEST |
|
||
|---|---|---|
|
||
| prune (daily) | **03:30** | 05:30 |
|
||
| GC (weekly, Sun) | **04:30** | 06:30 |
|
||
|
||
**Ordering holds:** prune runs daily including Sunday, one hour before the weekly GC on the same
|
||
morning — so GC always sees the freshest prune result. **Daily was chosen deliberately**: with weekly
|
||
backups the prune is usually a no-op, and a no-op that runs beats a weekly one that misses.
|
||
|
||
> **The weekly backup is due-based and therefore drifts**, so no fixed schedule can guarantee
|
||
> non-overlap. Occasional overlap is possible and is accepted, not designed away.
|
||
|
||
---
|
||
|
||
## 3. Part 3 — dry run, then the real prune
|
||
|
||
### Dry run (verbatim) — DELETED NOTHING
|
||
|
||
Run with a **temporary `root@pam` token** granted `DatastoreAdmin`, removed on exit (the pattern
|
||
`felhom-tenantsync.sh` already uses). `prune-job run` has **no `--dry-run`**, so the client-side
|
||
`proxmox-backup-client prune … --dry-run` was used.
|
||
|
||
```
|
||
############ DRY RUN ns=demo-felhom keep-last=2 ############
|
||
| snapshot | date | action |
|
||
| ct/9201/2026-07-18T18:31:06Z | Sat Jul 18 18:31:06 2026 | keep |
|
||
| ct/9201/2026-07-26T12:21:48Z | Sun Jul 26 12:21:48 2026 | keep |
|
||
|
||
############ DRY RUN ns=demo-hp keep-last=2 ############
|
||
| snapshot | date | action |
|
||
| ct/9201/2026-07-26T15:42:42Z | Sun Jul 26 15:42:42 2026 | remove |
|
||
| ct/9201/2026-07-26T17:02:57Z | Sun Jul 26 17:02:57 2026 | keep |
|
||
| ct/9201/2026-07-27T08:25:47Z | Mon Jul 27 08:25:47 2026 | keep |
|
||
```
|
||
|
||
**Gate passed exactly as specified:** demo-hp loses only its excess above 2; **demo-felhom loses
|
||
nothing**. The STOP condition was not triggered.
|
||
|
||
### Real run — both `TASK OK`
|
||
|
||
```
|
||
prune job 'prune-demo-felhom' → keep, keep → TASK OK
|
||
prune job 'prune-demo-hp' → remove ct/9201/2026-07-26T15:42:42Z, keep, keep → TASK OK
|
||
```
|
||
|
||
| Namespace | Before | After | Newest snapshot |
|
||
|---|---|---|---|
|
||
| demo-felhom | 2 | **2** | `2026-07-26T12:21:48Z` — unchanged |
|
||
| demo-hp | 3 | **2** | `2026-07-27T08:25:47Z` — unchanged |
|
||
| demo-felhom-01 | 3 | 3 | untouched (no job) |
|
||
|
||
**Chunks: 9,787 → 9,787 — unchanged.** That is the load-bearing confirmation: prune removed snapshot
|
||
indexes and reclaimed no chunks.
|
||
|
||
**`df` moved by −77,824 B (−76 KB).** The runbook predicted no movement; the substance is right (no
|
||
chunks freed) but not literally zero, because the removed snapshot's **own index files**
|
||
(`root.pxar.didx` ~57 KB, `catalog.pcat1.didx`, `index.json.blob`, `pct.conf.blob`) occupy disk and go
|
||
with it. Nothing other than prune ran. **Chunk count, not `df`, is the correct probe for "did prune
|
||
reclaim data".**
|
||
|
||
---
|
||
|
||
## 4. Part 4 — the write proof, CLOSED
|
||
|
||
Driven through the real flow: `POST /api/guest-backup/trigger` → `TriggerNow()` (the UI's
|
||
„Mentés most"), which deliberately covers **every** tier in one quiesce window. **Not `--selftest`,
|
||
not raw `vzdump`.**
|
||
|
||
**Before vs after, same box, same path:**
|
||
|
||
```
|
||
07-27 08:25:47 UTC vzdump (felhom-pbs) -> job errors ← prune denied
|
||
07-27 09:37:29 UTC vzdump (felhom-pbs) -> OK ← after the Part 1 change
|
||
```
|
||
|
||
```
|
||
INFO: creating Proxmox Backup Server archive 'ct/9201/2026-07-27T09:37:29Z'
|
||
INFO: root.pxar: had to backup 123.95 MiB of 3.975 GiB (compressed 30.431 MiB) in 44.64 s
|
||
INFO: root.pxar: backup was done incrementally, reused 3.854 GiB (97.0%)
|
||
INFO: Duration: 45.80s
|
||
TASK OK
|
||
```
|
||
|
||
**PBS side:** `UPID:…backup:felhom-offsite:ct-9201:felhom@pbs!demo-hp` → **OK** (09:37:36 → 09:38:22).
|
||
|
||
| Evidence | Result |
|
||
|---|---|
|
||
| Snapshot identity | `ns/demo-hp/ct/9201/**2026-07-27T09:37:29Z**` |
|
||
| Contents | `root.pxar.didx` 57,416 B, `catalog.pcat1.didx`, `index.json.blob`, `pct.conf.blob`, `client.log.blob` — real, not an empty dir |
|
||
| Chunk delta | 9,787 → **9,813** (+26) |
|
||
| `df` delta | 13,579,243,520 → 13,612,040,192 (**+32.8 MB**) |
|
||
| Prune step | **absent entirely** — `keep_last: 0` means no `--prune-backups` flag, so the permission check is never reached |
|
||
| Job errors | **none** |
|
||
|
||
**The hub PBS-DR gauge evidence is NOT satisfied.** A +32.8 MB delta is below the gauge's 0.1 GB
|
||
display granularity, so it still reads 12.6 GB / 13 %. Stating that plainly rather than dressing it
|
||
up — the same call as last time, and it still stands.
|
||
|
||
---
|
||
|
||
## 5. Part 5 — GC scheduled, NOT run
|
||
|
||
```sh
|
||
proxmox-backup-manager datastore update felhom-offsite --gc-schedule "sun 04:30"
|
||
```
|
||
|
||
```
|
||
| name | felhom-offsite |
|
||
| path | /mnt/pbs-datastore |
|
||
| gc-schedule | sun 04:30 |
|
||
| verify-new | 1 |
|
||
```
|
||
|
||
**GC was NOT run**, confirmed: `garbage-collection status` reports `upid` empty and `disk-chunks 0` —
|
||
it has never executed on this datastore.
|
||
|
||
Deliberate: every chunk still carries a **fresh atime from this morning's migration copy**, and GC
|
||
only reclaims chunks whose atime predates the cutoff. A run today would reclaim nothing regardless of
|
||
what Part 3 pruned. **The first meaningful GC is the scheduled one.**
|
||
|
||
---
|
||
|
||
## 6. Part 6 — `verify-new` ruling: ENABLED
|
||
|
||
**Operator ruling 2026-07-27: enable it.**
|
||
|
||
```sh
|
||
proxmox-backup-manager datastore update felhom-offsite --verify-new true
|
||
```
|
||
|
||
Rationale recorded: nothing on this PBS ever verified, so `verify_state` stayed `none` forever and the
|
||
hub's *"newest PBS snapshot failed verification"* alarm **could not fire** — inert, in the same class
|
||
as the hub 400ing an unknown event type. `verify-new` verifies each snapshot as it is created; with
|
||
`keep-last 2` that covers essentially the whole datastore and turns a dead check live, for a few
|
||
minutes of ep0 CPU per weekly backup.
|
||
|
||
> Watch item: ep0 is a 3.7 GB CX23 with **no swap**, and inline verification runs within the backup
|
||
> window. Today's full forced verify completed fine (~250 MiB/s, 0 errors), but see
|
||
> `RUNBOOK-ep0-datastore-volume-2026-07-27.md` for the rsync OOM on this same box.
|
||
|
||
---
|
||
|
||
## 7. Legacy `demo-felhom-01` namespace — DELETED (operator ruling, confirmed twice)
|
||
|
||
Not in the original runbook; surfaced because "a prune job per customer namespace" would have applied
|
||
`keep-last 2` to a retired install's archive. Flagged rather than actioned silently, then ruled on.
|
||
|
||
**Destroyed:** 3 snapshots, all 2026-07-04 (`14:37:16Z`, `14:55:44Z`, `17:51:34Z`), from the retired
|
||
`demo-felhom-01` install. No live box had written there since 07-04.
|
||
|
||
**Removed with it, so nothing dangles** (the same shape as this morning's `scratch` datastore):
|
||
|
||
| Object | Status |
|
||
|---|---|
|
||
| namespace `demo-felhom-01` | deleted (`namespace delete --delete-groups true`) |
|
||
| ACL `felhom@pbs` → `/datastore/felhom-offsite/demo-felhom-01` | deleted |
|
||
| ACL `felhom@pbs!demo-felhom-01` → same path | deleted |
|
||
| token `felhom@pbs!demo-felhom-01` | deleted |
|
||
|
||
**Final ACL state — four entries, write-only, live namespaces only:**
|
||
|
||
```
|
||
felhom@pbs /datastore/felhom-offsite/demo-felhom DatastoreBackup
|
||
felhom@pbs /datastore/felhom-offsite/demo-hp DatastoreBackup
|
||
felhom@pbs!demo-felhom /datastore/felhom-offsite/demo-felhom DatastoreBackup
|
||
felhom@pbs!demo-hp /datastore/felhom-offsite/demo-hp DatastoreBackup
|
||
```
|
||
|
||
Live namespaces verified intact throughout (demo-felhom 2, demo-hp 3). Datastore name and path
|
||
unchanged. The temporary `root@pam!nsdel` token **and its ACL** were both removed.
|
||
|
||
> **It freed almost nothing today, and that was stated before the confirmation:** the namespace
|
||
> directory was **620 KB** (index files only). Its chunks live in the shared `.chunks` store and come
|
||
> back only when GC's atime cutoff passes — weeks away, not today.
|
||
|
||
---
|
||
|
||
## 8. The demo-felhom prediction — CLOSED
|
||
|
||
The falsifiable claim from the previous runbook was: *demo-felhom sits at exactly 2 snapshots, so its
|
||
prune has silently had nothing to do; its next weekly backup makes 3 and will fail identically.*
|
||
|
||
**It is now neutralised on both halves, independently:**
|
||
|
||
1. **The box no longer attempts prune** — `prune_pbs_allowed=false` on demo-felhom, so no
|
||
`--prune-backups` flag is passed and the permission check is unreachable.
|
||
2. **Server-side prune covers the namespace** — `prune-demo-felhom` runs daily at 03:30 UTC and was
|
||
**verified live** (`TASK OK`, both snapshots kept).
|
||
|
||
So demo-felhom's third snapshot will be pruned by ep0 under `root@pam`, not by the box, and the
|
||
failure mode cannot recur. **Prediction closed — the failure will not reproduce.**
|
||
|
||
---
|
||
|
||
## 9. Old datastore copy — STILL IN PLACE
|
||
|
||
```
|
||
drwxr-xr-x 4 backup backup 4096 Jul 3 21:48 /srv/pbs-felhom
|
||
13G /srv/pbs-felhom
|
||
```
|
||
|
||
Untouched. Rollback remains a two-line `datastore.cfg` revert. Volume: 98 G, 13 G used, 13 %.
|
||
|
||
---
|
||
|
||
## 10. Deferred / open
|
||
|
||
1. **R-89** (below) — hub-owned retention policy. Today's prune jobs are the first increment of it,
|
||
not a stopgap.
|
||
2. **The restic delete-rights question** carried in R-89 — unanswered, and the more urgent half.
|
||
3. **GC has still never run.** First scheduled execution Sunday 04:30 UTC. Its first run is worth
|
||
watching: it is the only thing that reclaims chunks, and nothing has ever exercised it here.
|
||
4. **Hub PBS-DR gauge granularity** — 0.1 GB steps mean routine incremental backups are invisible to
|
||
it. Not a fault, but it cannot be used as write-proof evidence for small deltas.
|
||
5. **ep0 has no swap** (3.7 GB CX23) — see the volume runbook's OOM.
|
||
|
||
## 11. Observations
|
||
|
||
- **`keep_last: 0` is the correct off-switch precisely because it is derived, not a flag.** Setting it
|
||
disables both the `--prune-backups` value and the `allowPBSPrune` gate in one edit, and the tier
|
||
stays armed. A dedicated "don't prune" boolean would have been a second source of truth.
|
||
- **The false-negative was the real damage, not the unpruned snapshots.** demo-hp's PBS tier had
|
||
reported failure on *every* backup since the tier was created on 07-26, while the data landed
|
||
correctly every time. A tier that cries wolf on every success makes a genuine failure invisible —
|
||
which is exactly what happened at 07:13 UTC, when a real outage produced the same-looking result.
|
||
- **Chunk count is the honest probe for prune, not `df`.** They answer different questions, and `df`
|
||
moved here for a reason that has nothing to do with reclamation.
|
||
- `prune-job run` has no `--dry-run`; the gate must come from `proxmox-backup-client prune --dry-run`,
|
||
which needs a token. The temporary-admin-token-then-delete pattern is the right shape and is already
|
||
established in `felhom-tenantsync.sh`.
|