docs(pbs): move PBS prune server-side, close the write proof, schedule GC
Supervised runbook execution. No code, no version bump. The felhom-pbs tier had reported `job errors` on EVERY demo-hp backup since the tier was created on 07-26, while the data landed correctly every time: `DatastoreBackup` grants Datastore.Backup but not Datastore.Prune, so the box's keep_last=2 prune was denied. Operator ruling: retention is a COMMERCIAL attribute owned by the hub; ep0 executes. Box tokens therefore stay write-only - a compromised box must not be able to delete its own offsite backups. No grant was widened and felhom-tenantsync.sh is unchanged (the ruling makes it correct). Increment 1: - boxes stop attempting prune. allowPBSPrune is DERIVED (`!t.Primary && t.KeepLast > 0`), so keep_last: 0 on the PBS tier disables both the --prune-backups value and the gate in one config edit, and the tier stays armed. Verified prune_pbs_allowed=false on both boxes with no tier REJECTED line. - per-namespace prune jobs on ep0, keep-last 2, daily 03:30 UTC (05:30 CEST), dry-run gated. demo-hp 3->2, demo-felhom untouched, chunk count unchanged (prune removes indexes, not chunks). Write proof CLOSED: 08:25:47 job errors -> 09:37:29 TASK OK, snapshot 2026-07-27T09:37:29Z, chunks 9787->9813, prune step absent entirely. Driven through POST /api/guest-backup/trigger (the UI path), not --selftest and not raw vzdump. Hub gauge evidence explicitly NOT satisfied - the delta is below its 0.1 GB display granularity. GC scheduled sun 04:30 UTC and deliberately NOT run: every chunk still carries a fresh atime from the migration copy, so a run today would reclaim nothing. verify-new enabled per operator ruling, turning an inert hub alarm live. Legacy demo-felhom-01 namespace deleted with its two ACL entries and its token (operator ruling, confirmed twice) so nothing dangles. R-89 records the target architecture and carries the unanswered parallel question: does the restic key on storage-box-pool-1 have DELETE rights? If so the daily app-data tier has the identical exposure and append-only is the equivalent answer. ep0 is Etc/UTC, not CEST - corrected in the record.
This commit is contained in:
+11
@@ -57,6 +57,17 @@ or retiring it is → **R-83**.
|
||||
thresholds (host 26h / offsite 8d); fresh-install default; an unprovisioned tier DEFERS.
|
||||
**Operator rulings:** 2-week offsite retention, first backup runs as long as it needs, one backup
|
||||
at a time per guest, drill box dropped from the rollout.
|
||||
**RETENTION IS A COMMERCIAL ATTRIBUTE — the hub decides, ep0 executes (operator ruling 2026-07-27,
|
||||
R-89).** A paid tier may buy longer retention, so the policy belongs with customer config on the
|
||||
hub, never in ep0's PBS config and never in a box's config. Execution stays server-side: a
|
||||
reconciler writes a **PBS prune job** and PBS's own scheduler runs it, so hub downtime leaves the
|
||||
last-known policy running rather than silently stopping retention. **Box tokens stay write-only
|
||||
(`DatastoreBackup`) — never widen a grant to fix a prune error:** a compromised box must not be
|
||||
able to delete its own offsite backups, which is the scenario offsite DR exists to survive.
|
||||
Increment 1 shipped 2026-07-27 (boxes stop attempting prune via `keep_last: 0`; per-namespace prune
|
||||
jobs on ep0, daily 03:30 UTC) — `runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md`. This closed a
|
||||
live false-negative: **every** demo-hp PBS backup since 07-26 reported `job errors` while the data
|
||||
landed correctly, because `DatastoreBackup` carries no `Datastore.Prune`.
|
||||
**Four defects found by RUNNING it, not reviewing it** — a 30-min wait bound against a 41-min
|
||||
backup (the agent recorded `success:false` while the backup was still going); the restore tier read
|
||||
from the configured target instead of the archive (**a silent regression of the S4.1 fix** — the
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
# REPORT — PBS prune moved server-side, write proof closed (2026-07-27)
|
||||
|
||||
**Class:** supervised operational run. **No code, no version bump.** Topic-scoped per the
|
||||
parallel-session rule; shared `REPORT.md` untouched.
|
||||
|
||||
**Full record:** `documentation/runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md`
|
||||
|
||||
---
|
||||
|
||||
## Outcome — all parts complete
|
||||
|
||||
| Part | Result |
|
||||
|---|---|
|
||||
| 1 — prune gate | **Config-gated.** `keep_last: 0` on the PBS tier, both boxes → `prune_pbs_allowed=false`. No code, **no grant** |
|
||||
| 2 — prune jobs | 2 jobs, per live namespace, `keep-last 2`, daily **03:30 UTC / 05:30 CEST** |
|
||||
| 3 — dry run → real | Gate passed; both `TASK OK`; demo-hp 3→2, demo-felhom untouched |
|
||||
| 4 — write proof | **CLOSED — `TASK OK`, no job errors** |
|
||||
| 5 — GC | Scheduled `sun 04:30 UTC / 06:30 CEST`. **NOT run** |
|
||||
| 6 — `verify-new` | **Enabled** (operator ruling) |
|
||||
| — legacy ns | `demo-felhom-01` deleted with its ACLs + token (operator ruling, confirmed twice) |
|
||||
| 7 — roadmap | **R-89** + CONTEXT.md note |
|
||||
|
||||
## The fix, in one line
|
||||
|
||||
`allowPBSPrune := !t.Primary && t.KeepLast > 0` — so setting the PBS tier's `keep_last` to `0`
|
||||
disables both the `--prune-backups` value and the gate, in one config edit, **while the tier stays
|
||||
armed**. Verified: `backup tier armed target=felhom-pbs cadence=168h0m0s keep_last=0
|
||||
prune_pbs_allowed=false`, no `tier REJECTED` line.
|
||||
|
||||
## The proof
|
||||
|
||||
```
|
||||
07-27 08:25:47 UTC vzdump (felhom-pbs) -> job errors ← prune denied
|
||||
07-27 09:37:29 UTC vzdump (felhom-pbs) -> OK ← after the change
|
||||
```
|
||||
|
||||
New snapshot `ns/demo-hp/ct/9201/2026-07-27T09:37:29Z`, chunks 9,787 → **9,813**, 97.0 % reused,
|
||||
45.80 s, **prune step absent entirely**. Driven via `POST /api/guest-backup/trigger` → `TriggerNow()`
|
||||
— the UI's „Mentés most" path, not `--selftest`, not raw `vzdump`.
|
||||
|
||||
**Hub gauge evidence NOT satisfied** — a +32.8 MB delta is below its 0.1 GB display granularity, so it
|
||||
still reads 12.6 GB / 13 %. Stated plainly rather than dressed up.
|
||||
|
||||
## The demo-felhom prediction — CLOSED
|
||||
|
||||
The claim was that demo-felhom's next weekly backup would make 3 snapshots and reproduce the prune
|
||||
failure. Neutralised on both halves: the box no longer attempts prune, and `prune-demo-felhom` covers
|
||||
the namespace server-side (verified live, `TASK OK`). **It will not reproduce.**
|
||||
|
||||
## Why it mattered more than the unpruned snapshots
|
||||
|
||||
demo-hp's PBS tier had reported failure on **every** backup since the tier was created on 07-26, while
|
||||
the data landed correctly every time. A tier that cries wolf on every success makes a genuine failure
|
||||
invisible — which is precisely what happened at 07:13 UTC, when a real outage produced an
|
||||
indistinguishable result.
|
||||
|
||||
## Security property preserved
|
||||
|
||||
**No prune right was granted to any box.** Final ACLs are four entries, write-only
|
||||
(`DatastoreBackup`), live namespaces only. A compromised box still cannot delete its own offsite
|
||||
backups. `felhom-tenantsync.sh` was **not** edited — the ruling makes its current grant correct.
|
||||
|
||||
## Open
|
||||
|
||||
1. **R-89** — hub-owned retention policy (today's jobs are increment 1, not a stopgap).
|
||||
2. **Does the restic key on `storage-box-pool-1` have DELETE rights?** Unanswered, carried in R-89,
|
||||
and the more urgent half — if so, the daily app-data tier has the identical exposure and
|
||||
append-only mode is the equivalent answer. Rule once for both tiers.
|
||||
3. **GC has still never run.** First execution Sunday 04:30 UTC; worth watching, as nothing has ever
|
||||
exercised it here.
|
||||
4. Old 13 GB datastore copy still at `/srv/pbs-felhom` — rollback intact.
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,325 @@
|
||||
# RUNBOOK RECORD — PBS prune moved server-side, write proof closed, GC scheduled (2026-07-27)
|
||||
|
||||
**Class:** supervised operational run. **No code, no version bump.** Follows
|
||||
`RUNBOOK-ep0-cleanup-2026-07-27.md` (Parts A–C complete; Part D stopped correctly on the prune defect).
|
||||
|
||||
**Targets:** `ep0` / `felhom-hetzner`, `demo-felhom` (N100), `demo-hp` (HP t740).
|
||||
|
||||
> **Timezone note, corrected from the runbook's standing rule.** `ep0` runs **`Etc/UTC`**, not CEST —
|
||||
> `timedatectl` reports `Time zone: Etc/UTC (UTC, +0000)`, NTP-synced. The CEST↔UTC split applies to
|
||||
> the **boxes** (`demo-felhom`, `demo-hp` are CEST), not to ep0. PBS calendar events therefore resolve
|
||||
> in **UTC** on this host. Both zones are still stated for every schedule below.
|
||||
|
||||
---
|
||||
|
||||
## Outcome
|
||||
|
||||
| Item | Result |
|
||||
|---|---|
|
||||
| Prune attempt by boxes | **disabled** — config only, no grant, no code |
|
||||
| Server-side prune | **2 jobs live**, daily `03:30` UTC, `keep-last 2` |
|
||||
| Write proof | **CLOSED — `TASK OK`, no job errors** |
|
||||
| GC | **scheduled** `sun 04:30` UTC — **NOT run** |
|
||||
| `verify-new` | **enabled** (operator ruling) |
|
||||
| Legacy `demo-felhom-01` | **deleted** — namespace + ACLs + token (operator ruling, confirmed twice) |
|
||||
| Old 13 GB copy | **still in place** |
|
||||
|
||||
---
|
||||
|
||||
## 1. Part 1 — the prune is CONFIG-GATED (branch taken: turn it off)
|
||||
|
||||
**Source landmark** — `internal/backup/runner.go:104`, inside `localPruneSpec`:
|
||||
|
||||
```go
|
||||
if s.Type == "pbs" && !r.allowPBSPrune {
|
||||
// Not opted in → never prune the offsite DR
|
||||
return ""
|
||||
}
|
||||
```
|
||||
|
||||
`allowPBSPrune` is **derived, not a config key** — `cmd/felhom-agent/main.go:1348`:
|
||||
|
||||
```go
|
||||
allowPBSPrune := !t.Primary && t.KeepLast > 0
|
||||
```
|
||||
|
||||
So the off-switch is the tier's `keep_last`. **Safety checked before changing it:** `BackupTiers()`
|
||||
(`internal/config/config.go:453-495`) rejects a tier only for an empty/duplicate `target_id` or
|
||||
`cadence_seconds <= 0`. `keep_last` is **not** a rejection condition, and for extra tiers
|
||||
`keep < 0 → 0` is legal and preserved. Only the **primary** tier clamps to ≥1 (`KeepLast()`), and
|
||||
that is the local tier, untouched. So `keep_last: 0` disarms the prune **without disarming the tier**.
|
||||
|
||||
**`agent.json` diff — identical on both boxes** (backed up to `agent.json.pre-prunegate.bak`):
|
||||
|
||||
```diff
|
||||
29c29
|
||||
< "keep_last": 2
|
||||
---
|
||||
> "keep_last": 0
|
||||
```
|
||||
|
||||
No in-flight restore-test or backup on either box at the time; both agents restarted cleanly.
|
||||
|
||||
**Verified after restart — both boxes:**
|
||||
|
||||
```
|
||||
backup tier armed target=local cadence=24h0m0s keep_last=3 prune_pbs_allowed=false primary=true
|
||||
backup tier armed target=felhom-pbs cadence=168h0m0s keep_last=0 prune_pbs_allowed=false primary=false
|
||||
backup: restore-test scheduler starting cadence=84h0m0s
|
||||
```
|
||||
|
||||
**The PBS tier is still ARMED** (cadence 168h, no `tier REJECTED` line), the local tier is unchanged,
|
||||
and the restore-test cadence is still the reverted 84h. **No prune right was granted to any box.**
|
||||
|
||||
---
|
||||
|
||||
## 2. Part 2 — server-side prune jobs on ep0
|
||||
|
||||
**Namespace-path form:** `prune-job create` takes `--ns <plain-namespace-name>` — **not**
|
||||
`/datastore/<ds>/<ns>` and **not** `/ns/<ns>`. That is the form used. `--max-depth 0` = non-recursive.
|
||||
|
||||
```sh
|
||||
proxmox-backup-manager prune-job create prune-demo-felhom \
|
||||
--store felhom-offsite --ns demo-felhom --max-depth 0 \
|
||||
--keep-last 2 --schedule "03:30" \
|
||||
--comment "R-82 retention keep-last=2, server-side (box tokens are write-only)"
|
||||
# same for prune-demo-hp
|
||||
```
|
||||
|
||||
| id | store | ns | schedule | max-depth | keep-last |
|
||||
|---|---|---|---|---|---|
|
||||
| `prune-demo-felhom` | felhom-offsite | demo-felhom | `03:30` | 0 | 2 |
|
||||
| `prune-demo-hp` | felhom-offsite | demo-hp | `03:30` | 0 | 2 |
|
||||
|
||||
**Schedules, both zones:**
|
||||
|
||||
| Job | ep0 local (UTC) | CEST |
|
||||
|---|---|---|
|
||||
| prune (daily) | **03:30** | 05:30 |
|
||||
| GC (weekly, Sun) | **04:30** | 06:30 |
|
||||
|
||||
**Ordering holds:** prune runs daily including Sunday, one hour before the weekly GC on the same
|
||||
morning — so GC always sees the freshest prune result. **Daily was chosen deliberately**: with weekly
|
||||
backups the prune is usually a no-op, and a no-op that runs beats a weekly one that misses.
|
||||
|
||||
> **The weekly backup is due-based and therefore drifts**, so no fixed schedule can guarantee
|
||||
> non-overlap. Occasional overlap is possible and is accepted, not designed away.
|
||||
|
||||
---
|
||||
|
||||
## 3. Part 3 — dry run, then the real prune
|
||||
|
||||
### Dry run (verbatim) — DELETED NOTHING
|
||||
|
||||
Run with a **temporary `root@pam` token** granted `DatastoreAdmin`, removed on exit (the pattern
|
||||
`felhom-tenantsync.sh` already uses). `prune-job run` has **no `--dry-run`**, so the client-side
|
||||
`proxmox-backup-client prune … --dry-run` was used.
|
||||
|
||||
```
|
||||
############ DRY RUN ns=demo-felhom keep-last=2 ############
|
||||
| snapshot | date | action |
|
||||
| ct/9201/2026-07-18T18:31:06Z | Sat Jul 18 18:31:06 2026 | keep |
|
||||
| ct/9201/2026-07-26T12:21:48Z | Sun Jul 26 12:21:48 2026 | keep |
|
||||
|
||||
############ DRY RUN ns=demo-hp keep-last=2 ############
|
||||
| snapshot | date | action |
|
||||
| ct/9201/2026-07-26T15:42:42Z | Sun Jul 26 15:42:42 2026 | remove |
|
||||
| ct/9201/2026-07-26T17:02:57Z | Sun Jul 26 17:02:57 2026 | keep |
|
||||
| ct/9201/2026-07-27T08:25:47Z | Mon Jul 27 08:25:47 2026 | keep |
|
||||
```
|
||||
|
||||
**Gate passed exactly as specified:** demo-hp loses only its excess above 2; **demo-felhom loses
|
||||
nothing**. The STOP condition was not triggered.
|
||||
|
||||
### Real run — both `TASK OK`
|
||||
|
||||
```
|
||||
prune job 'prune-demo-felhom' → keep, keep → TASK OK
|
||||
prune job 'prune-demo-hp' → remove ct/9201/2026-07-26T15:42:42Z, keep, keep → TASK OK
|
||||
```
|
||||
|
||||
| Namespace | Before | After | Newest snapshot |
|
||||
|---|---|---|---|
|
||||
| demo-felhom | 2 | **2** | `2026-07-26T12:21:48Z` — unchanged |
|
||||
| demo-hp | 3 | **2** | `2026-07-27T08:25:47Z` — unchanged |
|
||||
| demo-felhom-01 | 3 | 3 | untouched (no job) |
|
||||
|
||||
**Chunks: 9,787 → 9,787 — unchanged.** That is the load-bearing confirmation: prune removed snapshot
|
||||
indexes and reclaimed no chunks.
|
||||
|
||||
**`df` moved by −77,824 B (−76 KB).** The runbook predicted no movement; the substance is right (no
|
||||
chunks freed) but not literally zero, because the removed snapshot's **own index files**
|
||||
(`root.pxar.didx` ~57 KB, `catalog.pcat1.didx`, `index.json.blob`, `pct.conf.blob`) occupy disk and go
|
||||
with it. Nothing other than prune ran. **Chunk count, not `df`, is the correct probe for "did prune
|
||||
reclaim data".**
|
||||
|
||||
---
|
||||
|
||||
## 4. Part 4 — the write proof, CLOSED
|
||||
|
||||
Driven through the real flow: `POST /api/guest-backup/trigger` → `TriggerNow()` (the UI's
|
||||
„Mentés most"), which deliberately covers **every** tier in one quiesce window. **Not `--selftest`,
|
||||
not raw `vzdump`.**
|
||||
|
||||
**Before vs after, same box, same path:**
|
||||
|
||||
```
|
||||
07-27 08:25:47 UTC vzdump (felhom-pbs) -> job errors ← prune denied
|
||||
07-27 09:37:29 UTC vzdump (felhom-pbs) -> OK ← after the Part 1 change
|
||||
```
|
||||
|
||||
```
|
||||
INFO: creating Proxmox Backup Server archive 'ct/9201/2026-07-27T09:37:29Z'
|
||||
INFO: root.pxar: had to backup 123.95 MiB of 3.975 GiB (compressed 30.431 MiB) in 44.64 s
|
||||
INFO: root.pxar: backup was done incrementally, reused 3.854 GiB (97.0%)
|
||||
INFO: Duration: 45.80s
|
||||
TASK OK
|
||||
```
|
||||
|
||||
**PBS side:** `UPID:…backup:felhom-offsite:ct-9201:felhom@pbs!demo-hp` → **OK** (09:37:36 → 09:38:22).
|
||||
|
||||
| Evidence | Result |
|
||||
|---|---|
|
||||
| Snapshot identity | `ns/demo-hp/ct/9201/**2026-07-27T09:37:29Z**` |
|
||||
| Contents | `root.pxar.didx` 57,416 B, `catalog.pcat1.didx`, `index.json.blob`, `pct.conf.blob`, `client.log.blob` — real, not an empty dir |
|
||||
| Chunk delta | 9,787 → **9,813** (+26) |
|
||||
| `df` delta | 13,579,243,520 → 13,612,040,192 (**+32.8 MB**) |
|
||||
| Prune step | **absent entirely** — `keep_last: 0` means no `--prune-backups` flag, so the permission check is never reached |
|
||||
| Job errors | **none** |
|
||||
|
||||
**The hub PBS-DR gauge evidence is NOT satisfied.** A +32.8 MB delta is below the gauge's 0.1 GB
|
||||
display granularity, so it still reads 12.6 GB / 13 %. Stating that plainly rather than dressing it
|
||||
up — the same call as last time, and it still stands.
|
||||
|
||||
---
|
||||
|
||||
## 5. Part 5 — GC scheduled, NOT run
|
||||
|
||||
```sh
|
||||
proxmox-backup-manager datastore update felhom-offsite --gc-schedule "sun 04:30"
|
||||
```
|
||||
|
||||
```
|
||||
| name | felhom-offsite |
|
||||
| path | /mnt/pbs-datastore |
|
||||
| gc-schedule | sun 04:30 |
|
||||
| verify-new | 1 |
|
||||
```
|
||||
|
||||
**GC was NOT run**, confirmed: `garbage-collection status` reports `upid` empty and `disk-chunks 0` —
|
||||
it has never executed on this datastore.
|
||||
|
||||
Deliberate: every chunk still carries a **fresh atime from this morning's migration copy**, and GC
|
||||
only reclaims chunks whose atime predates the cutoff. A run today would reclaim nothing regardless of
|
||||
what Part 3 pruned. **The first meaningful GC is the scheduled one.**
|
||||
|
||||
---
|
||||
|
||||
## 6. Part 6 — `verify-new` ruling: ENABLED
|
||||
|
||||
**Operator ruling 2026-07-27: enable it.**
|
||||
|
||||
```sh
|
||||
proxmox-backup-manager datastore update felhom-offsite --verify-new true
|
||||
```
|
||||
|
||||
Rationale recorded: nothing on this PBS ever verified, so `verify_state` stayed `none` forever and the
|
||||
hub's *"newest PBS snapshot failed verification"* alarm **could not fire** — inert, in the same class
|
||||
as the hub 400ing an unknown event type. `verify-new` verifies each snapshot as it is created; with
|
||||
`keep-last 2` that covers essentially the whole datastore and turns a dead check live, for a few
|
||||
minutes of ep0 CPU per weekly backup.
|
||||
|
||||
> Watch item: ep0 is a 3.7 GB CX23 with **no swap**, and inline verification runs within the backup
|
||||
> window. Today's full forced verify completed fine (~250 MiB/s, 0 errors), but see
|
||||
> `RUNBOOK-ep0-datastore-volume-2026-07-27.md` for the rsync OOM on this same box.
|
||||
|
||||
---
|
||||
|
||||
## 7. Legacy `demo-felhom-01` namespace — DELETED (operator ruling, confirmed twice)
|
||||
|
||||
Not in the original runbook; surfaced because "a prune job per customer namespace" would have applied
|
||||
`keep-last 2` to a retired install's archive. Flagged rather than actioned silently, then ruled on.
|
||||
|
||||
**Destroyed:** 3 snapshots, all 2026-07-04 (`14:37:16Z`, `14:55:44Z`, `17:51:34Z`), from the retired
|
||||
`demo-felhom-01` install. No live box had written there since 07-04.
|
||||
|
||||
**Removed with it, so nothing dangles** (the same shape as this morning's `scratch` datastore):
|
||||
|
||||
| Object | Status |
|
||||
|---|---|
|
||||
| namespace `demo-felhom-01` | deleted (`namespace delete --delete-groups true`) |
|
||||
| ACL `felhom@pbs` → `/datastore/felhom-offsite/demo-felhom-01` | deleted |
|
||||
| ACL `felhom@pbs!demo-felhom-01` → same path | deleted |
|
||||
| token `felhom@pbs!demo-felhom-01` | deleted |
|
||||
|
||||
**Final ACL state — four entries, write-only, live namespaces only:**
|
||||
|
||||
```
|
||||
felhom@pbs /datastore/felhom-offsite/demo-felhom DatastoreBackup
|
||||
felhom@pbs /datastore/felhom-offsite/demo-hp DatastoreBackup
|
||||
felhom@pbs!demo-felhom /datastore/felhom-offsite/demo-felhom DatastoreBackup
|
||||
felhom@pbs!demo-hp /datastore/felhom-offsite/demo-hp DatastoreBackup
|
||||
```
|
||||
|
||||
Live namespaces verified intact throughout (demo-felhom 2, demo-hp 3). Datastore name and path
|
||||
unchanged. The temporary `root@pam!nsdel` token **and its ACL** were both removed.
|
||||
|
||||
> **It freed almost nothing today, and that was stated before the confirmation:** the namespace
|
||||
> directory was **620 KB** (index files only). Its chunks live in the shared `.chunks` store and come
|
||||
> back only when GC's atime cutoff passes — weeks away, not today.
|
||||
|
||||
---
|
||||
|
||||
## 8. The demo-felhom prediction — CLOSED
|
||||
|
||||
The falsifiable claim from the previous runbook was: *demo-felhom sits at exactly 2 snapshots, so its
|
||||
prune has silently had nothing to do; its next weekly backup makes 3 and will fail identically.*
|
||||
|
||||
**It is now neutralised on both halves, independently:**
|
||||
|
||||
1. **The box no longer attempts prune** — `prune_pbs_allowed=false` on demo-felhom, so no
|
||||
`--prune-backups` flag is passed and the permission check is unreachable.
|
||||
2. **Server-side prune covers the namespace** — `prune-demo-felhom` runs daily at 03:30 UTC and was
|
||||
**verified live** (`TASK OK`, both snapshots kept).
|
||||
|
||||
So demo-felhom's third snapshot will be pruned by ep0 under `root@pam`, not by the box, and the
|
||||
failure mode cannot recur. **Prediction closed — the failure will not reproduce.**
|
||||
|
||||
---
|
||||
|
||||
## 9. Old datastore copy — STILL IN PLACE
|
||||
|
||||
```
|
||||
drwxr-xr-x 4 backup backup 4096 Jul 3 21:48 /srv/pbs-felhom
|
||||
13G /srv/pbs-felhom
|
||||
```
|
||||
|
||||
Untouched. Rollback remains a two-line `datastore.cfg` revert. Volume: 98 G, 13 G used, 13 %.
|
||||
|
||||
---
|
||||
|
||||
## 10. Deferred / open
|
||||
|
||||
1. **R-89** (below) — hub-owned retention policy. Today's prune jobs are the first increment of it,
|
||||
not a stopgap.
|
||||
2. **The restic delete-rights question** carried in R-89 — unanswered, and the more urgent half.
|
||||
3. **GC has still never run.** First scheduled execution Sunday 04:30 UTC. Its first run is worth
|
||||
watching: it is the only thing that reclaims chunks, and nothing has ever exercised it here.
|
||||
4. **Hub PBS-DR gauge granularity** — 0.1 GB steps mean routine incremental backups are invisible to
|
||||
it. Not a fault, but it cannot be used as write-proof evidence for small deltas.
|
||||
5. **ep0 has no swap** (3.7 GB CX23) — see the volume runbook's OOM.
|
||||
|
||||
## 11. Observations
|
||||
|
||||
- **`keep_last: 0` is the correct off-switch precisely because it is derived, not a flag.** Setting it
|
||||
disables both the `--prune-backups` value and the `allowPBSPrune` gate in one edit, and the tier
|
||||
stays armed. A dedicated "don't prune" boolean would have been a second source of truth.
|
||||
- **The false-negative was the real damage, not the unpruned snapshots.** demo-hp's PBS tier had
|
||||
reported failure on *every* backup since the tier was created on 07-26, while the data landed
|
||||
correctly every time. A tier that cries wolf on every success makes a genuine failure invisible —
|
||||
which is exactly what happened at 07:13 UTC, when a real outage produced the same-looking result.
|
||||
- **Chunk count is the honest probe for prune, not `df`.** They answer different questions, and `df`
|
||||
moved here for a reason that has nothing to do with reclamation.
|
||||
- `prune-job run` has no `--dry-run`; the gate must come from `proxmox-backup-client prune --dry-run`,
|
||||
which needs a token. The temporary-admin-token-then-delete pattern is the right shape and is already
|
||||
established in `felhom-tenantsync.sh`.
|
||||
Reference in New Issue
Block a user