Files

T

admin a31872ea24 docs(pbs): move PBS prune server-side, close the write proof, schedule GC

Supervised runbook execution. No code, no version bump.

The felhom-pbs tier had reported `job errors` on EVERY demo-hp backup
since the tier was created on 07-26, while the data landed correctly
every time: `DatastoreBackup` grants Datastore.Backup but not
Datastore.Prune, so the box's keep_last=2 prune was denied.

Operator ruling: retention is a COMMERCIAL attribute owned by the hub;
ep0 executes. Box tokens therefore stay write-only - a compromised box
must not be able to delete its own offsite backups. No grant was widened
and felhom-tenantsync.sh is unchanged (the ruling makes it correct).

Increment 1:
- boxes stop attempting prune. allowPBSPrune is DERIVED
  (`!t.Primary && t.KeepLast > 0`), so keep_last: 0 on the PBS tier
  disables both the --prune-backups value and the gate in one config
  edit, and the tier stays armed. Verified prune_pbs_allowed=false on
  both boxes with no tier REJECTED line.
- per-namespace prune jobs on ep0, keep-last 2, daily 03:30 UTC
  (05:30 CEST), dry-run gated. demo-hp 3->2, demo-felhom untouched,
  chunk count unchanged (prune removes indexes, not chunks).

Write proof CLOSED: 08:25:47 job errors -> 09:37:29 TASK OK, snapshot
2026-07-27T09:37:29Z, chunks 9787->9813, prune step absent entirely.
Driven through POST /api/guest-backup/trigger (the UI path), not
--selftest and not raw vzdump. Hub gauge evidence explicitly NOT
satisfied - the delta is below its 0.1 GB display granularity.

GC scheduled sun 04:30 UTC and deliberately NOT run: every chunk still
carries a fresh atime from the migration copy, so a run today would
reclaim nothing. verify-new enabled per operator ruling, turning an
inert hub alarm live.

Legacy demo-felhom-01 namespace deleted with its two ACL entries and its
token (operator ruling, confirmed twice) so nothing dangles.

R-89 records the target architecture and carries the unanswered parallel
question: does the restic key on storage-box-pool-1 have DELETE rights?
If so the daily app-data tier has the identical exposure and append-only
is the equivalent answer.

ep0 is Etc/UTC, not CEST - corrected in the record.

2026-07-27 15:16:11 +02:00

3.5 KiB

Raw Blame History

REPORT — PBS prune moved server-side, write proof closed (2026-07-27)

Class: supervised operational run. No code, no version bump. Topic-scoped per the parallel-session rule; shared REPORT.md untouched.

Full record: documentation/runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md

Outcome — all parts complete

Part	Result
1 — prune gate	Config-gated. `keep_last: 0` on the PBS tier, both boxes → `prune_pbs_allowed=false`. No code, no grant
2 — prune jobs	2 jobs, per live namespace, `keep-last 2`, daily 03:30 UTC / 05:30 CEST
3 — dry run → real	Gate passed; both `TASK OK`; demo-hp 3→2, demo-felhom untouched
4 — write proof	CLOSED — `TASK OK`, no job errors
5 — GC	Scheduled `sun 04:30 UTC / 06:30 CEST`. NOT run
6 — `verify-new`	Enabled (operator ruling)
— legacy ns	`demo-felhom-01` deleted with its ACLs + token (operator ruling, confirmed twice)
7 — roadmap	R-89 + CONTEXT.md note

The fix, in one line

allowPBSPrune := !t.Primary && t.KeepLast > 0 — so setting the PBS tier's keep_last to 0 disables both the --prune-backups value and the gate, in one config edit, while the tier stays armed. Verified: backup tier armed target=felhom-pbs cadence=168h0m0s keep_last=0 prune_pbs_allowed=false, no tier REJECTED line.

The proof

07-27 08:25:47 UTC  vzdump (felhom-pbs) -> job errors     ← prune denied
07-27 09:37:29 UTC  vzdump (felhom-pbs) -> OK             ← after the change

New snapshot ns/demo-hp/ct/9201/2026-07-27T09:37:29Z, chunks 9,787 → 9,813, 97.0 % reused, 45.80 s, prune step absent entirely. Driven via POST /api/guest-backup/trigger → TriggerNow() — the UI's „Mentés most" path, not --selftest, not raw vzdump.

Hub gauge evidence NOT satisfied — a +32.8 MB delta is below its 0.1 GB display granularity, so it still reads 12.6 GB / 13 %. Stated plainly rather than dressed up.

The demo-felhom prediction — CLOSED

The claim was that demo-felhom's next weekly backup would make 3 snapshots and reproduce the prune failure. Neutralised on both halves: the box no longer attempts prune, and prune-demo-felhom covers the namespace server-side (verified live, TASK OK). It will not reproduce.

Why it mattered more than the unpruned snapshots

demo-hp's PBS tier had reported failure on every backup since the tier was created on 07-26, while the data landed correctly every time. A tier that cries wolf on every success makes a genuine failure invisible — which is precisely what happened at 07:13 UTC, when a real outage produced an indistinguishable result.

Security property preserved

No prune right was granted to any box. Final ACLs are four entries, write-only (DatastoreBackup), live namespaces only. A compromised box still cannot delete its own offsite backups. felhom-tenantsync.sh was not edited — the ruling makes its current grant correct.

Open

R-89 — hub-owned retention policy (today's jobs are increment 1, not a stopgap).
Does the restic key on storage-box-pool-1 have DELETE rights? Unanswered, carried in R-89, and the more urgent half — if so, the daily app-data tier has the identical exposure and append-only mode is the equivalent answer. Rule once for both tiers.
GC has still never run. First execution Sunday 04:30 UTC; worth watching, as nothing has ever exercised it here.
Old 13 GB datastore copy still at /srv/pbs-felhom — rollback intact.

3.5 KiB Raw Blame History