admin
a31872ea24
Supervised runbook execution. No code, no version bump. The felhom-pbs tier had reported `job errors` on EVERY demo-hp backup since the tier was created on 07-26, while the data landed correctly every time: `DatastoreBackup` grants Datastore.Backup but not Datastore.Prune, so the box's keep_last=2 prune was denied. Operator ruling: retention is a COMMERCIAL attribute owned by the hub; ep0 executes. Box tokens therefore stay write-only - a compromised box must not be able to delete its own offsite backups. No grant was widened and felhom-tenantsync.sh is unchanged (the ruling makes it correct). Increment 1: - boxes stop attempting prune. allowPBSPrune is DERIVED (`!t.Primary && t.KeepLast > 0`), so keep_last: 0 on the PBS tier disables both the --prune-backups value and the gate in one config edit, and the tier stays armed. Verified prune_pbs_allowed=false on both boxes with no tier REJECTED line. - per-namespace prune jobs on ep0, keep-last 2, daily 03:30 UTC (05:30 CEST), dry-run gated. demo-hp 3->2, demo-felhom untouched, chunk count unchanged (prune removes indexes, not chunks). Write proof CLOSED: 08:25:47 job errors -> 09:37:29 TASK OK, snapshot 2026-07-27T09:37:29Z, chunks 9787->9813, prune step absent entirely. Driven through POST /api/guest-backup/trigger (the UI path), not --selftest and not raw vzdump. Hub gauge evidence explicitly NOT satisfied - the delta is below its 0.1 GB display granularity. GC scheduled sun 04:30 UTC and deliberately NOT run: every chunk still carries a fresh atime from the migration copy, so a run today would reclaim nothing. verify-new enabled per operator ruling, turning an inert hub alarm live. Legacy demo-felhom-01 namespace deleted with its two ACL entries and its token (operator ruling, confirmed twice) so nothing dangles. R-89 records the target architecture and carries the unanswered parallel question: does the restic key on storage-box-pool-1 have DELETE rights? If so the daily app-data tier has the identical exposure and append-only is the equivalent answer. ep0 is Etc/UTC, not CEST - corrected in the record.
72 lines
3.5 KiB
Markdown
72 lines
3.5 KiB
Markdown
# REPORT — PBS prune moved server-side, write proof closed (2026-07-27)
|
|
|
|
**Class:** supervised operational run. **No code, no version bump.** Topic-scoped per the
|
|
parallel-session rule; shared `REPORT.md` untouched.
|
|
|
|
**Full record:** `documentation/runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md`
|
|
|
|
---
|
|
|
|
## Outcome — all parts complete
|
|
|
|
| Part | Result |
|
|
|---|---|
|
|
| 1 — prune gate | **Config-gated.** `keep_last: 0` on the PBS tier, both boxes → `prune_pbs_allowed=false`. No code, **no grant** |
|
|
| 2 — prune jobs | 2 jobs, per live namespace, `keep-last 2`, daily **03:30 UTC / 05:30 CEST** |
|
|
| 3 — dry run → real | Gate passed; both `TASK OK`; demo-hp 3→2, demo-felhom untouched |
|
|
| 4 — write proof | **CLOSED — `TASK OK`, no job errors** |
|
|
| 5 — GC | Scheduled `sun 04:30 UTC / 06:30 CEST`. **NOT run** |
|
|
| 6 — `verify-new` | **Enabled** (operator ruling) |
|
|
| — legacy ns | `demo-felhom-01` deleted with its ACLs + token (operator ruling, confirmed twice) |
|
|
| 7 — roadmap | **R-89** + CONTEXT.md note |
|
|
|
|
## The fix, in one line
|
|
|
|
`allowPBSPrune := !t.Primary && t.KeepLast > 0` — so setting the PBS tier's `keep_last` to `0`
|
|
disables both the `--prune-backups` value and the gate, in one config edit, **while the tier stays
|
|
armed**. Verified: `backup tier armed target=felhom-pbs cadence=168h0m0s keep_last=0
|
|
prune_pbs_allowed=false`, no `tier REJECTED` line.
|
|
|
|
## The proof
|
|
|
|
```
|
|
07-27 08:25:47 UTC vzdump (felhom-pbs) -> job errors ← prune denied
|
|
07-27 09:37:29 UTC vzdump (felhom-pbs) -> OK ← after the change
|
|
```
|
|
|
|
New snapshot `ns/demo-hp/ct/9201/2026-07-27T09:37:29Z`, chunks 9,787 → **9,813**, 97.0 % reused,
|
|
45.80 s, **prune step absent entirely**. Driven via `POST /api/guest-backup/trigger` → `TriggerNow()`
|
|
— the UI's „Mentés most" path, not `--selftest`, not raw `vzdump`.
|
|
|
|
**Hub gauge evidence NOT satisfied** — a +32.8 MB delta is below its 0.1 GB display granularity, so it
|
|
still reads 12.6 GB / 13 %. Stated plainly rather than dressed up.
|
|
|
|
## The demo-felhom prediction — CLOSED
|
|
|
|
The claim was that demo-felhom's next weekly backup would make 3 snapshots and reproduce the prune
|
|
failure. Neutralised on both halves: the box no longer attempts prune, and `prune-demo-felhom` covers
|
|
the namespace server-side (verified live, `TASK OK`). **It will not reproduce.**
|
|
|
|
## Why it mattered more than the unpruned snapshots
|
|
|
|
demo-hp's PBS tier had reported failure on **every** backup since the tier was created on 07-26, while
|
|
the data landed correctly every time. A tier that cries wolf on every success makes a genuine failure
|
|
invisible — which is precisely what happened at 07:13 UTC, when a real outage produced an
|
|
indistinguishable result.
|
|
|
|
## Security property preserved
|
|
|
|
**No prune right was granted to any box.** Final ACLs are four entries, write-only
|
|
(`DatastoreBackup`), live namespaces only. A compromised box still cannot delete its own offsite
|
|
backups. `felhom-tenantsync.sh` was **not** edited — the ruling makes its current grant correct.
|
|
|
|
## Open
|
|
|
|
1. **R-89** — hub-owned retention policy (today's jobs are increment 1, not a stopgap).
|
|
2. **Does the restic key on `storage-box-pool-1` have DELETE rights?** Unanswered, carried in R-89,
|
|
and the more urgent half — if so, the daily app-data tier has the identical exposure and
|
|
append-only mode is the equivalent answer. Rule once for both tiers.
|
|
3. **GC has still never run.** First execution Sunday 04:30 UTC; worth watching, as nothing has ever
|
|
exercised it here.
|
|
4. Old 13 GB datastore copy still at `/srv/pbs-felhom` — rollback intact.
|