docs(pbs): move PBS prune server-side, close the write proof, schedule GC
Supervised runbook execution. No code, no version bump. The felhom-pbs tier had reported `job errors` on EVERY demo-hp backup since the tier was created on 07-26, while the data landed correctly every time: `DatastoreBackup` grants Datastore.Backup but not Datastore.Prune, so the box's keep_last=2 prune was denied. Operator ruling: retention is a COMMERCIAL attribute owned by the hub; ep0 executes. Box tokens therefore stay write-only - a compromised box must not be able to delete its own offsite backups. No grant was widened and felhom-tenantsync.sh is unchanged (the ruling makes it correct). Increment 1: - boxes stop attempting prune. allowPBSPrune is DERIVED (`!t.Primary && t.KeepLast > 0`), so keep_last: 0 on the PBS tier disables both the --prune-backups value and the gate in one config edit, and the tier stays armed. Verified prune_pbs_allowed=false on both boxes with no tier REJECTED line. - per-namespace prune jobs on ep0, keep-last 2, daily 03:30 UTC (05:30 CEST), dry-run gated. demo-hp 3->2, demo-felhom untouched, chunk count unchanged (prune removes indexes, not chunks). Write proof CLOSED: 08:25:47 job errors -> 09:37:29 TASK OK, snapshot 2026-07-27T09:37:29Z, chunks 9787->9813, prune step absent entirely. Driven through POST /api/guest-backup/trigger (the UI path), not --selftest and not raw vzdump. Hub gauge evidence explicitly NOT satisfied - the delta is below its 0.1 GB display granularity. GC scheduled sun 04:30 UTC and deliberately NOT run: every chunk still carries a fresh atime from the migration copy, so a run today would reclaim nothing. verify-new enabled per operator ruling, turning an inert hub alarm live. Legacy demo-felhom-01 namespace deleted with its two ACL entries and its token (operator ruling, confirmed twice) so nothing dangles. R-89 records the target architecture and carries the unanswered parallel question: does the restic key on storage-box-pool-1 have DELETE rights? If so the daily app-data tier has the identical exposure and append-only is the equivalent answer. ep0 is Etc/UTC, not CEST - corrected in the record.
This commit is contained in:
1 parent
a16896af86
commit
a31872ea24
4 files changed
+408
No files matched your search
+11
@@ -57,6 +57,17 @@ or retiring it is → **R-83**.
|
||||
thresholds (host 26h / offsite 8d); fresh-install default; an unprovisioned tier DEFERS.
|
||||
**Operator rulings:** 2-week offsite retention, first backup runs as long as it needs, one backup
|
||||
at a time per guest, drill box dropped from the rollout.
|
||||
**RETENTION IS A COMMERCIAL ATTRIBUTE — the hub decides, ep0 executes (operator ruling 2026-07-27,
|
||||
R-89).** A paid tier may buy longer retention, so the policy belongs with customer config on the
|
||||
hub, never in ep0's PBS config and never in a box's config. Execution stays server-side: a
|
||||
reconciler writes a **PBS prune job** and PBS's own scheduler runs it, so hub downtime leaves the
|
||||
last-known policy running rather than silently stopping retention. **Box tokens stay write-only
|
||||
(`DatastoreBackup`) — never widen a grant to fix a prune error:** a compromised box must not be
|
||||
able to delete its own offsite backups, which is the scenario offsite DR exists to survive.
|
||||
Increment 1 shipped 2026-07-27 (boxes stop attempting prune via `keep_last: 0`; per-namespace prune
|
||||
jobs on ep0, daily 03:30 UTC) — `runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md`. This closed a
|
||||
live false-negative: **every** demo-hp PBS backup since 07-26 reported `job errors` while the data
|
||||
landed correctly, because `DatastoreBackup` carries no `Datastore.Prune`.
|
||||
**Four defects found by RUNNING it, not reviewing it** — a 30-min wait bound against a 41-min
|
||||
backup (the agent recorded `success:false` while the backup was still going); the restore tier read
|
||||
from the configured target instead of the archive (**a silent regression of the S4.1 fix** — the
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
# REPORT — PBS prune moved server-side, write proof closed (2026-07-27)
|
||||
|
||||
**Class:** supervised operational run. **No code, no version bump.** Topic-scoped per the
|
||||
parallel-session rule; shared `REPORT.md` untouched.
|
||||
|
||||
**Full record:** `documentation/runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md`
|
||||
|
||||
---
|
||||
|
||||
## Outcome — all parts complete
|
||||
|
||||
| Part | Result |
|
||||
|---|---|
|
||||
| 1 — prune gate | **Config-gated.** `keep_last: 0` on the PBS tier, both boxes → `prune_pbs_allowed=false`. No code, **no grant** |
|
||||
| 2 — prune jobs | 2 jobs, per live namespace, `keep-last 2`, daily **03:30 UTC / 05:30 CEST** |
|
||||
| 3 — dry run → real | Gate passed; both `TASK OK`; demo-hp 3→2, demo-felhom untouched |
|
||||
| 4 — write proof | **CLOSED — `TASK OK`, no job errors** |
|
||||
| 5 — GC | Scheduled `sun 04:30 UTC / 06:30 CEST`. **NOT run** |
|
||||
| 6 — `verify-new` | **Enabled** (operator ruling) |
|
||||
| — legacy ns | `demo-felhom-01` deleted with its ACLs + token (operator ruling, confirmed twice) |
|
||||
| 7 — roadmap | **R-89** + CONTEXT.md note |
|
||||
|
||||
## The fix, in one line
|
||||
|
||||
`allowPBSPrune := !t.Primary && t.KeepLast > 0` — so setting the PBS tier's `keep_last` to `0`
|
||||
disables both the `--prune-backups` value and the gate, in one config edit, **while the tier stays
|
||||
armed**. Verified: `backup tier armed target=felhom-pbs cadence=168h0m0s keep_last=0
|
||||
prune_pbs_allowed=false`, no `tier REJECTED` line.
|
||||
|
||||
## The proof
|
||||
|
||||
```
|
||||
07-27 08:25:47 UTC vzdump (felhom-pbs) -> job errors ← prune denied
|
||||
07-27 09:37:29 UTC vzdump (felhom-pbs) -> OK ← after the change
|
||||
```
|
||||
|
||||
New snapshot `ns/demo-hp/ct/9201/2026-07-27T09:37:29Z`, chunks 9,787 → **9,813**, 97.0 % reused,
|
||||
45.80 s, **prune step absent entirely**. Driven via `POST /api/guest-backup/trigger` → `TriggerNow()`
|
||||
— the UI's „Mentés most" path, not `--selftest`, not raw `vzdump`.
|
||||
|
||||
**Hub gauge evidence NOT satisfied** — a +32.8 MB delta is below its 0.1 GB display granularity, so it
|
||||
still reads 12.6 GB / 13 %. Stated plainly rather than dressed up.
|
||||
|
||||
## The demo-felhom prediction — CLOSED
|
||||
|
||||
The claim was that demo-felhom's next weekly backup would make 3 snapshots and reproduce the prune
|
||||
failure. Neutralised on both halves: the box no longer attempts prune, and `prune-demo-felhom` covers
|
||||
the namespace server-side (verified live, `TASK OK`). **It will not reproduce.**
|
||||
|
||||
## Why it mattered more than the unpruned snapshots
|
||||
|
||||
demo-hp's PBS tier had reported failure on **every** backup since the tier was created on 07-26, while
|
||||
the data landed correctly every time. A tier that cries wolf on every success makes a genuine failure
|
||||
invisible — which is precisely what happened at 07:13 UTC, when a real outage produced an
|
||||
indistinguishable result.
|
||||
|
||||
## Security property preserved
|
||||
|
||||
**No prune right was granted to any box.** Final ACLs are four entries, write-only
|
||||
(`DatastoreBackup`), live namespaces only. A compromised box still cannot delete its own offsite
|
||||
backups. `felhom-tenantsync.sh` was **not** edited — the ruling makes its current grant correct.
|
||||
|
||||
## Open
|
||||
|
||||
1. **R-89** — hub-owned retention policy (today's jobs are increment 1, not a stopgap).
|
||||
2. **Does the restic key on `storage-box-pool-1` have DELETE rights?** Unanswered, carried in R-89,
|
||||
and the more urgent half — if so, the daily app-data tier has the identical exposure and
|
||||
append-only mode is the equivalent answer. Rule once for both tiers.
|
||||
3. **GC has still never run.** First execution Sunday 04:30 UTC; worth watching, as nothing has ever
|
||||
exercised it here.
|
||||
4. Old 13 GB datastore copy still at `/srv/pbs-felhom` — rollback intact.
|
||||
@@ -123,6 +123,7 @@ Each attempt runs the **full quiesce cycle**, so every customer app stack is STO
|
||||
**The amplifier, and the part worth designing against:** the agent answers `Due: true, Reason: "no successful backup recorded yet", AgeSecs: nil`, and that **nil age does double duty**. `scheduledRunAllowed` (`quiesce.go:466-480`) returns `true` whenever `lastAgeSecs == nil` — *"no recorded backup yet — never withhold the first one"* — so the same nil that makes every poll due **also bypasses the time-of-day gate** `[W+2h, W+6h)`. On the live box the gate was `[04:30, 08:30)` and the cycles ran at 09:02–09:12 Budapest, i.e. **outside the backup window entirely**. So the fault stops customer apps every 5 minutes *at any hour, including business hours* — the one protection specifically built to prevent that is switched off by the same missing value. A safety valve written for a genuine first-ever backup is being triggered by an unreachable storage read, which is not the same thing at all.
|
||||
|
||||
Self-resolves the moment the target answers (the storage read succeeds, sees the archive, tier stops being due) — which is why it can hide indefinitely: it needs an offsite outage to appear at all. **Shape:** distinguish *storage unreachable* from *storage readable and empty*. Unreachable is UNKNOWN — defer the due-verdict rather than resolving it either way, exactly as R-81 made the hub do with a missing report. Only a target that is reachable AND has no archive is genuinely due. **Fix the window bypass in the same slice:** `AgeSecs == nil` must stop meaning "run now regardless of the hour". Either the agent distinguishes *never backed up* from *cannot tell* in what it reports, or `scheduledRunAllowed` gates on the former only — otherwise any future nil-age path re-opens the same hole. Note this does NOT weaken R-84's fail-safe intent: a tier whose storage is merely slow or briefly unreadable should still err toward backing up — it is specifically the **cold-store + unreachable** pair that must defer, because there the fallback has no information at all, only an empty default that looks like a fact. |
|
||||
| R-89 | **Retention is a COMMERCIAL attribute — it belongs to the hub, not to ep0 or a box** | M | idea — operator ruling 2026-07-27, first increment SHIPPED same day | **Ruling (2026-07-27):** retention is a per-customer *commercial* attribute (a paid tier may buy longer retention), so the **hub owns the policy** and ep0 merely executes it. Execution stays **server-side**: a reconciler writes a **PBS prune job** and PBS's own scheduler runs it, so hub downtime leaves the last-known policy running rather than silently stopping retention. **Increment 1 SHIPPED 2026-07-27** (`runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md`): boxes no longer attempt prune (`keep_last: 0` → `allowPBSPrune=false`, config only — no code, no grant), and per-namespace prune jobs run on ep0 daily 03:30 UTC (`keep-last 2`), dry-run gated and verified `TASK OK`. This also closed a live defect — **every** demo-hp PBS backup since the tier was created on 07-26 had reported `job errors` while the data landed correctly, because `DatastoreBackup` grants `Datastore.Backup` but not `Datastore.Prune`; a tier that cries wolf on every success makes a genuine failure invisible, which is exactly what happened during the 07-27 migration window. **Remaining work:** retention becomes a per-customer attribute on the hub — a **policy** (keep-last/daily/weekly/monthly), NOT a bare number, since a paid tier will want "12 weekly + 6 monthly"; a reconciler converges it into the ep0 prune job in the same descriptor-and-converge shape as `pbs_dr`. **SECURITY PROPERTY TO PRESERVE — do not "fix" a future prune error by widening the grant:** box tokens stay **write-only** (`DatastoreBackup`), never `DatastorePowerUser`/`DatastoreAdmin`. A compromised box must not be able to delete its own offsite backups — that is the scenario offsite DR exists to survive. **PARALLEL QUESTION, unanswered and arguably the more urgent half: does the restic key on `storage-box-pool-1` (`u629488`) have DELETE rights?** If so the daily app-data offsite tier carries the identical exposure, and restic's **append-only** mode is the equivalent answer. **Rule once for both tiers.** Flips the capability-map row for offsite retention (currently: retention configured but unenforceable) |
|
||||
| R-76 | **FileBrowser-created folders break the setgid chain, and a drop-zone's mode is not stable** | S | idea (surfaced by the R-75 spike, 2026-07-26) | Two related findings from `audits/SPIKE-catalog-data-paths-2026-07-26.md` P3/P5, both **pre-existing** and deliberately left alone by that spike. **(a)** FileBrowser Quantum 1.3.3 creates files `0644` and folders `0755` and does **not** propagate the setgid bit — even though the entrypoint wrapper's `umask 002` really is in effect (`/proc/1/status` `Umask: 0002`). Group inheritance itself works (a file uploaded into a 2775 group-100 dir landed group 100, not the process gid 1000), so the convention's *group* half holds and only its *mode* half is lost. The consequence is proven with a control: inside a UI-created `0755` folder a gid-1000 process's file landed group **1000**, while the identical write into the 2775 parent landed group **100**. So **any folder a customer creates through FileBrowser breaks the shared-group chain one level down.** Latent today — every userdata-touching catalog app that declares an identity declares uid/gid **1000**, the same uid FileBrowser runs as, so owner permissions mask it; it bites the day a content app runs as a different non-root uid with gid 1000. The comment at `infra/infra.go:156` is right that the image ignores `-e UMASK` but does not say the wrapper fails to achieve the intended mode either. **(b)** `import/calibre` is live on demo-felhom at `755 1000:1000` where every sibling is `2775 root:1000` — with `media/books` on the same box, same app, same deploy, at `2775` as the control, and no parser asymmetry (checked: `ParseComposeUserdataMounts` picks up both calibre-web binds). Consistent with the consuming app rewriting the mode of its own ingest dir after the deploy belt sets it; **not confirmed causally**. Together they mean **a drop-zone directory's mode is not stable against either the customer or the consuming app** — verify before building anything that assumes `import/*` stays 2775. Flips no capability-map row today (latent); would become customer-visible the moment a non-1000 content app enters the catalog |
|
||||
| R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → universal secret-free / operator-bind (option C) | XL | **SHIPPED + PHYSICALLY CLOSED (slices A+B+C; rehearsal executed 2026-07-18)** | **PHYSICAL CLOSURE 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the generic pairing ISO v1.20.0 (`--loader mkimage`, SB off) **booted the very AMI board that F1 blocked**, installed unattended, and the box self-registered as an unclaimed appliance the same second it first booted (16:17:14) → self-bind → credential → day-0 SUCCESS 16:32:32 → floor-lifted to current. **F1 is closed on physical hardware and the rehearsal dependency on this item is discharged.** Two residual notes stay open, neither blocking: PXE/network-boot is still unbuilt (the third F1 option, for boards where even USB-mkimage fails), and mkimage remains unsigned → **SB must be OFF**. The installer's GRUB menu still offers interactive installers → new item **R-38**. — Prior: **PHYSICAL RUN 2026-07-16 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`):** demo N100 reinstalled clean-slate from a pipeline ISO → chain reached **rc-0 first try on real hardware** (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. **F1 (HIGH, slice-B input):** this cheap AMI `AN3PLUS 0.01` firmware won't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`) — SB-off/shim-bypass don't help; worked around live with a `grub-mkimage` loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B — SHIPPED (scripts v1.18.0, 2026-07-17):** the F1 firmware fix is now a first-class pipeline mode `build-felhom-iso.sh --loader shim|mkimage` (default shim; `mkimage` = monolithic grub-mkimage loader from the ISO's own GRUB, recipe from the run evidence). RUNBOOK-B legs on nested VM 311 proved it: shim boots+installs under OVMF SB-enforcing + SeaBIOS; mkimage boots+installs under OVMF SB-off; mkimage under SB-enforcing FAILS `Access Denied` (unsigned → **SB must be OFF**, documented); surgery byte-identical payload. **Physical N100 boot on the real board still pending** → folds into the supervised rehearsal (R-1; an `n100-safety` match-nothing ISO is built + sha-recorded for a zero-risk pre-flight). **PXE/network-boot** (the third F1 option, for boards where even USB-mkimage fails) stays a deferred note under this item — not built. **Slice C — SHIPPED (hub v0.62.0 + scripts v1.19.0, 2026-07-17):** the GENERIC secret-free universal ISO (`build-felhom-iso.sh --pairing`). The box self-registers as an unclaimed appliance (keyed by SMBIOS-uuid + MAC set — the DMI-verdict tiebreaker), the operator BINDS it to a customer on the Hosts page, and the hub delivers customer-id + retrieval passphrase ONCE (`/api/v1/appliance/register` + one-shot poll, 404-no-oracle — all live-verified through the public ingress); the bootstrap then falls through to the slice-A direct path. **One unit, two modes** (direct = byte-identical, regression-proven zero-appliance-calls). Artifact proven secret-free (baked env = hub URL only; manifest `secret-bearing: no`). The **bind is operator-password-gated** → the live boot→register→bind→day-0 composition (with a Viktor-created drill customer) + the physical N100 boot fold into the supervised rehearsal — **which now runs the COMPLETE final product flow in one pass: RESET the demo → boot the generic ISO → bind → day-0**. SSH-host-key pinning: stored + fingerprints displayed (attaching to the host on bind = future, no clean hand-off surface today). **Customer-facing self-bind page = R-27 (future).** Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needLine truncated
|
||||
| R-29 | **The design-v2 green gates are not enforced anywhere — one has been RED for 16 releases.** `controller/scripts/docker_run_volume_path_gate.py` has failed continuously since **2026-07-14 (v0.129.0)** and nobody noticed until R-7b's close-out ran it by hand at v0.145.0. Two separable parts. **(a) The finding itself is benign and the fix is 3 lines.** The flagged call is `internal/appexport/estimate.go:179` `docker run --rm -v <volumeName>:/vol:ro alpine du` — a **NAMED-VOLUME** mount, i.e. daemon-side with no host path, which is the *safe* shape and byte-for-byte the same pattern as three entries already on the gate's ALLOWLIST (`export.go` `volName+":/vol"`, `backup.go` `volName+":/vol:ro"`, `restore.go` `volName+":/vol"`). It is NOT the v0.124.0 path-strand class the gate exists to catch — the author of the v0.129.0 F-A fix explicitly avoided that class (see the function's own comment) and simply never added the allowlist entry. So the fix is an ALLOWLIST addition WITH ITS WHY, **not** a docker-cp rewrite; anyone who 'fixes' this by rewriting the call has misread the gate. **(b) The systemic half is the real item:** the gates run only when a human remembers to run them, so a gate can sit red across 16 releases while every REPORT says 'green'. This is the SECOND instance of the class — cf. the v0.123.0 note *'Windows green gate silently red (read-only fsync)'*. Decide where they run (pre-push hook, `build.sh` step, or a CI job) and make a red gate block the train the way the Go green gate does. | S (a) / M (b) | idea | Origin: R-7b close-out, `felhom-controller` REPORT §4(f) — CC correctly left it alone as out-of-scope and pre-existing, and verified by stashing that it fails identically on the unmodified tree. Flips no capability-map row (engineering hygiene, no customer-visible behaviour). Affected gates to audit for the same rot: controller `template_id_gate` / `emoji_gate` / `native_confirm_gate` / `offbox_rename_gate` / `mojibake_gate` / `app_row_dedup_gate` / `docker_run_volume_path_gate`, hub `hub_confirm_gate`, manifests `manifest_bearer_gate`, website `site_gates`. **Do not bundle (a) into an unrelated feature commit** — it is a one-line behavioural claim about a mount's safety and deserves its own reviewed diff. **2026-07-18 rehearsal note:** the run's finding list independently re-raised "assign the pre-existing `docker_run_volume_path_gate` failure its ID so red stops normalizing" — **that is this item; no second ID was minted.** |
|
||||
|
||||
@@ -0,0 +1,325 @@
|
||||
# RUNBOOK RECORD — PBS prune moved server-side, write proof closed, GC scheduled (2026-07-27)
|
||||
|
||||
**Class:** supervised operational run. **No code, no version bump.** Follows
|
||||
`RUNBOOK-ep0-cleanup-2026-07-27.md` (Parts A–C complete; Part D stopped correctly on the prune defect).
|
||||
|
||||
**Targets:** `ep0` / `felhom-hetzner`, `demo-felhom` (N100), `demo-hp` (HP t740).
|
||||
|
||||
> **Timezone note, corrected from the runbook's standing rule.** `ep0` runs **`Etc/UTC`**, not CEST —
|
||||
> `timedatectl` reports `Time zone: Etc/UTC (UTC, +0000)`, NTP-synced. The CEST↔UTC split applies to
|
||||
> the **boxes** (`demo-felhom`, `demo-hp` are CEST), not to ep0. PBS calendar events therefore resolve
|
||||
> in **UTC** on this host. Both zones are still stated for every schedule below.
|
||||
|
||||
---
|
||||
|
||||
## Outcome
|
||||
|
||||
| Item | Result |
|
||||
|---|---|
|
||||
| Prune attempt by boxes | **disabled** — config only, no grant, no code |
|
||||
| Server-side prune | **2 jobs live**, daily `03:30` UTC, `keep-last 2` |
|
||||
| Write proof | **CLOSED — `TASK OK`, no job errors** |
|
||||
| GC | **scheduled** `sun 04:30` UTC — **NOT run** |
|
||||
| `verify-new` | **enabled** (operator ruling) |
|
||||
| Legacy `demo-felhom-01` | **deleted** — namespace + ACLs + token (operator ruling, confirmed twice) |
|
||||
| Old 13 GB copy | **still in place** |
|
||||
|
||||
---
|
||||
|
||||
## 1. Part 1 — the prune is CONFIG-GATED (branch taken: turn it off)
|
||||
|
||||
**Source landmark** — `internal/backup/runner.go:104`, inside `localPruneSpec`:
|
||||
|
||||
```go
|
||||
if s.Type == "pbs" && !r.allowPBSPrune {
|
||||
// Not opted in → never prune the offsite DR
|
||||
return ""
|
||||
}
|
||||
```
|
||||
|
||||
`allowPBSPrune` is **derived, not a config key** — `cmd/felhom-agent/main.go:1348`:
|
||||
|
||||
```go
|
||||
allowPBSPrune := !t.Primary && t.KeepLast > 0
|
||||
```
|
||||
|
||||
So the off-switch is the tier's `keep_last`. **Safety checked before changing it:** `BackupTiers()`
|
||||
(`internal/config/config.go:453-495`) rejects a tier only for an empty/duplicate `target_id` or
|
||||
`cadence_seconds <= 0`. `keep_last` is **not** a rejection condition, and for extra tiers
|
||||
`keep < 0 → 0` is legal and preserved. Only the **primary** tier clamps to ≥1 (`KeepLast()`), and
|
||||
that is the local tier, untouched. So `keep_last: 0` disarms the prune **without disarming the tier**.
|
||||
|
||||
**`agent.json` diff — identical on both boxes** (backed up to `agent.json.pre-prunegate.bak`):
|
||||
|
||||
```diff
|
||||
29c29
|
||||
< "keep_last": 2
|
||||
---
|
||||
> "keep_last": 0
|
||||
```
|
||||
|
||||
No in-flight restore-test or backup on either box at the time; both agents restarted cleanly.
|
||||
|
||||
**Verified after restart — both boxes:**
|
||||
|
||||
```
|
||||
backup tier armed target=local cadence=24h0m0s keep_last=3 prune_pbs_allowed=false primary=true
|
||||
backup tier armed target=felhom-pbs cadence=168h0m0s keep_last=0 prune_pbs_allowed=false primary=false
|
||||
backup: restore-test scheduler starting cadence=84h0m0s
|
||||
```
|
||||
|
||||
**The PBS tier is still ARMED** (cadence 168h, no `tier REJECTED` line), the local tier is unchanged,
|
||||
and the restore-test cadence is still the reverted 84h. **No prune right was granted to any box.**
|
||||
|
||||
---
|
||||
|
||||
## 2. Part 2 — server-side prune jobs on ep0
|
||||
|
||||
**Namespace-path form:** `prune-job create` takes `--ns <plain-namespace-name>` — **not**
|
||||
`/datastore/<ds>/<ns>` and **not** `/ns/<ns>`. That is the form used. `--max-depth 0` = non-recursive.
|
||||
|
||||
```sh
|
||||
proxmox-backup-manager prune-job create prune-demo-felhom \
|
||||
--store felhom-offsite --ns demo-felhom --max-depth 0 \
|
||||
--keep-last 2 --schedule "03:30" \
|
||||
--comment "R-82 retention keep-last=2, server-side (box tokens are write-only)"
|
||||
# same for prune-demo-hp
|
||||
```
|
||||
|
||||
| id | store | ns | schedule | max-depth | keep-last |
|
||||
|---|---|---|---|---|---|
|
||||
| `prune-demo-felhom` | felhom-offsite | demo-felhom | `03:30` | 0 | 2 |
|
||||
| `prune-demo-hp` | felhom-offsite | demo-hp | `03:30` | 0 | 2 |
|
||||
|
||||
**Schedules, both zones:**
|
||||
|
||||
| Job | ep0 local (UTC) | CEST |
|
||||
|---|---|---|
|
||||
| prune (daily) | **03:30** | 05:30 |
|
||||
| GC (weekly, Sun) | **04:30** | 06:30 |
|
||||
|
||||
**Ordering holds:** prune runs daily including Sunday, one hour before the weekly GC on the same
|
||||
morning — so GC always sees the freshest prune result. **Daily was chosen deliberately**: with weekly
|
||||
backups the prune is usually a no-op, and a no-op that runs beats a weekly one that misses.
|
||||
|
||||
> **The weekly backup is due-based and therefore drifts**, so no fixed schedule can guarantee
|
||||
> non-overlap. Occasional overlap is possible and is accepted, not designed away.
|
||||
|
||||
---
|
||||
|
||||
## 3. Part 3 — dry run, then the real prune
|
||||
|
||||
### Dry run (verbatim) — DELETED NOTHING
|
||||
|
||||
Run with a **temporary `root@pam` token** granted `DatastoreAdmin`, removed on exit (the pattern
|
||||
`felhom-tenantsync.sh` already uses). `prune-job run` has **no `--dry-run`**, so the client-side
|
||||
`proxmox-backup-client prune … --dry-run` was used.
|
||||
|
||||
```
|
||||
############ DRY RUN ns=demo-felhom keep-last=2 ############
|
||||
| snapshot | date | action |
|
||||
| ct/9201/2026-07-18T18:31:06Z | Sat Jul 18 18:31:06 2026 | keep |
|
||||
| ct/9201/2026-07-26T12:21:48Z | Sun Jul 26 12:21:48 2026 | keep |
|
||||
|
||||
############ DRY RUN ns=demo-hp keep-last=2 ############
|
||||
| snapshot | date | action |
|
||||
| ct/9201/2026-07-26T15:42:42Z | Sun Jul 26 15:42:42 2026 | remove |
|
||||
| ct/9201/2026-07-26T17:02:57Z | Sun Jul 26 17:02:57 2026 | keep |
|
||||
| ct/9201/2026-07-27T08:25:47Z | Mon Jul 27 08:25:47 2026 | keep |
|
||||
```
|
||||
|
||||
**Gate passed exactly as specified:** demo-hp loses only its excess above 2; **demo-felhom loses
|
||||
nothing**. The STOP condition was not triggered.
|
||||
|
||||
### Real run — both `TASK OK`
|
||||
|
||||
```
|
||||
prune job 'prune-demo-felhom' → keep, keep → TASK OK
|
||||
prune job 'prune-demo-hp' → remove ct/9201/2026-07-26T15:42:42Z, keep, keep → TASK OK
|
||||
```
|
||||
|
||||
| Namespace | Before | After | Newest snapshot |
|
||||
|---|---|---|---|
|
||||
| demo-felhom | 2 | **2** | `2026-07-26T12:21:48Z` — unchanged |
|
||||
| demo-hp | 3 | **2** | `2026-07-27T08:25:47Z` — unchanged |
|
||||
| demo-felhom-01 | 3 | 3 | untouched (no job) |
|
||||
|
||||
**Chunks: 9,787 → 9,787 — unchanged.** That is the load-bearing confirmation: prune removed snapshot
|
||||
indexes and reclaimed no chunks.
|
||||
|
||||
**`df` moved by −77,824 B (−76 KB).** The runbook predicted no movement; the substance is right (no
|
||||
chunks freed) but not literally zero, because the removed snapshot's **own index files**
|
||||
(`root.pxar.didx` ~57 KB, `catalog.pcat1.didx`, `index.json.blob`, `pct.conf.blob`) occupy disk and go
|
||||
with it. Nothing other than prune ran. **Chunk count, not `df`, is the correct probe for "did prune
|
||||
reclaim data".**
|
||||
|
||||
---
|
||||
|
||||
## 4. Part 4 — the write proof, CLOSED
|
||||
|
||||
Driven through the real flow: `POST /api/guest-backup/trigger` → `TriggerNow()` (the UI's
|
||||
„Mentés most"), which deliberately covers **every** tier in one quiesce window. **Not `--selftest`,
|
||||
not raw `vzdump`.**
|
||||
|
||||
**Before vs after, same box, same path:**
|
||||
|
||||
```
|
||||
07-27 08:25:47 UTC vzdump (felhom-pbs) -> job errors ← prune denied
|
||||
07-27 09:37:29 UTC vzdump (felhom-pbs) -> OK ← after the Part 1 change
|
||||
```
|
||||
|
||||
```
|
||||
INFO: creating Proxmox Backup Server archive 'ct/9201/2026-07-27T09:37:29Z'
|
||||
INFO: root.pxar: had to backup 123.95 MiB of 3.975 GiB (compressed 30.431 MiB) in 44.64 s
|
||||
INFO: root.pxar: backup was done incrementally, reused 3.854 GiB (97.0%)
|
||||
INFO: Duration: 45.80s
|
||||
TASK OK
|
||||
```
|
||||
|
||||
**PBS side:** `UPID:…backup:felhom-offsite:ct-9201:felhom@pbs!demo-hp` → **OK** (09:37:36 → 09:38:22).
|
||||
|
||||
| Evidence | Result |
|
||||
|---|---|
|
||||
| Snapshot identity | `ns/demo-hp/ct/9201/**2026-07-27T09:37:29Z**` |
|
||||
| Contents | `root.pxar.didx` 57,416 B, `catalog.pcat1.didx`, `index.json.blob`, `pct.conf.blob`, `client.log.blob` — real, not an empty dir |
|
||||
| Chunk delta | 9,787 → **9,813** (+26) |
|
||||
| `df` delta | 13,579,243,520 → 13,612,040,192 (**+32.8 MB**) |
|
||||
| Prune step | **absent entirely** — `keep_last: 0` means no `--prune-backups` flag, so the permission check is never reached |
|
||||
| Job errors | **none** |
|
||||
|
||||
**The hub PBS-DR gauge evidence is NOT satisfied.** A +32.8 MB delta is below the gauge's 0.1 GB
|
||||
display granularity, so it still reads 12.6 GB / 13 %. Stating that plainly rather than dressing it
|
||||
up — the same call as last time, and it still stands.
|
||||
|
||||
---
|
||||
|
||||
## 5. Part 5 — GC scheduled, NOT run
|
||||
|
||||
```sh
|
||||
proxmox-backup-manager datastore update felhom-offsite --gc-schedule "sun 04:30"
|
||||
```
|
||||
|
||||
```
|
||||
| name | felhom-offsite |
|
||||
| path | /mnt/pbs-datastore |
|
||||
| gc-schedule | sun 04:30 |
|
||||
| verify-new | 1 |
|
||||
```
|
||||
|
||||
**GC was NOT run**, confirmed: `garbage-collection status` reports `upid` empty and `disk-chunks 0` —
|
||||
it has never executed on this datastore.
|
||||
|
||||
Deliberate: every chunk still carries a **fresh atime from this morning's migration copy**, and GC
|
||||
only reclaims chunks whose atime predates the cutoff. A run today would reclaim nothing regardless of
|
||||
what Part 3 pruned. **The first meaningful GC is the scheduled one.**
|
||||
|
||||
---
|
||||
|
||||
## 6. Part 6 — `verify-new` ruling: ENABLED
|
||||
|
||||
**Operator ruling 2026-07-27: enable it.**
|
||||
|
||||
```sh
|
||||
proxmox-backup-manager datastore update felhom-offsite --verify-new true
|
||||
```
|
||||
|
||||
Rationale recorded: nothing on this PBS ever verified, so `verify_state` stayed `none` forever and the
|
||||
hub's *"newest PBS snapshot failed verification"* alarm **could not fire** — inert, in the same class
|
||||
as the hub 400ing an unknown event type. `verify-new` verifies each snapshot as it is created; with
|
||||
`keep-last 2` that covers essentially the whole datastore and turns a dead check live, for a few
|
||||
minutes of ep0 CPU per weekly backup.
|
||||
|
||||
> Watch item: ep0 is a 3.7 GB CX23 with **no swap**, and inline verification runs within the backup
|
||||
> window. Today's full forced verify completed fine (~250 MiB/s, 0 errors), but see
|
||||
> `RUNBOOK-ep0-datastore-volume-2026-07-27.md` for the rsync OOM on this same box.
|
||||
|
||||
---
|
||||
|
||||
## 7. Legacy `demo-felhom-01` namespace — DELETED (operator ruling, confirmed twice)
|
||||
|
||||
Not in the original runbook; surfaced because "a prune job per customer namespace" would have applied
|
||||
`keep-last 2` to a retired install's archive. Flagged rather than actioned silently, then ruled on.
|
||||
|
||||
**Destroyed:** 3 snapshots, all 2026-07-04 (`14:37:16Z`, `14:55:44Z`, `17:51:34Z`), from the retired
|
||||
`demo-felhom-01` install. No live box had written there since 07-04.
|
||||
|
||||
**Removed with it, so nothing dangles** (the same shape as this morning's `scratch` datastore):
|
||||
|
||||
| Object | Status |
|
||||
|---|---|
|
||||
| namespace `demo-felhom-01` | deleted (`namespace delete --delete-groups true`) |
|
||||
| ACL `felhom@pbs` → `/datastore/felhom-offsite/demo-felhom-01` | deleted |
|
||||
| ACL `felhom@pbs!demo-felhom-01` → same path | deleted |
|
||||
| token `felhom@pbs!demo-felhom-01` | deleted |
|
||||
|
||||
**Final ACL state — four entries, write-only, live namespaces only:**
|
||||
|
||||
```
|
||||
felhom@pbs /datastore/felhom-offsite/demo-felhom DatastoreBackup
|
||||
felhom@pbs /datastore/felhom-offsite/demo-hp DatastoreBackup
|
||||
felhom@pbs!demo-felhom /datastore/felhom-offsite/demo-felhom DatastoreBackup
|
||||
felhom@pbs!demo-hp /datastore/felhom-offsite/demo-hp DatastoreBackup
|
||||
```
|
||||
|
||||
Live namespaces verified intact throughout (demo-felhom 2, demo-hp 3). Datastore name and path
|
||||
unchanged. The temporary `root@pam!nsdel` token **and its ACL** were both removed.
|
||||
|
||||
> **It freed almost nothing today, and that was stated before the confirmation:** the namespace
|
||||
> directory was **620 KB** (index files only). Its chunks live in the shared `.chunks` store and come
|
||||
> back only when GC's atime cutoff passes — weeks away, not today.
|
||||
|
||||
---
|
||||
|
||||
## 8. The demo-felhom prediction — CLOSED
|
||||
|
||||
The falsifiable claim from the previous runbook was: *demo-felhom sits at exactly 2 snapshots, so its
|
||||
prune has silently had nothing to do; its next weekly backup makes 3 and will fail identically.*
|
||||
|
||||
**It is now neutralised on both halves, independently:**
|
||||
|
||||
1. **The box no longer attempts prune** — `prune_pbs_allowed=false` on demo-felhom, so no
|
||||
`--prune-backups` flag is passed and the permission check is unreachable.
|
||||
2. **Server-side prune covers the namespace** — `prune-demo-felhom` runs daily at 03:30 UTC and was
|
||||
**verified live** (`TASK OK`, both snapshots kept).
|
||||
|
||||
So demo-felhom's third snapshot will be pruned by ep0 under `root@pam`, not by the box, and the
|
||||
failure mode cannot recur. **Prediction closed — the failure will not reproduce.**
|
||||
|
||||
---
|
||||
|
||||
## 9. Old datastore copy — STILL IN PLACE
|
||||
|
||||
```
|
||||
drwxr-xr-x 4 backup backup 4096 Jul 3 21:48 /srv/pbs-felhom
|
||||
13G /srv/pbs-felhom
|
||||
```
|
||||
|
||||
Untouched. Rollback remains a two-line `datastore.cfg` revert. Volume: 98 G, 13 G used, 13 %.
|
||||
|
||||
---
|
||||
|
||||
## 10. Deferred / open
|
||||
|
||||
1. **R-89** (below) — hub-owned retention policy. Today's prune jobs are the first increment of it,
|
||||
not a stopgap.
|
||||
2. **The restic delete-rights question** carried in R-89 — unanswered, and the more urgent half.
|
||||
3. **GC has still never run.** First scheduled execution Sunday 04:30 UTC. Its first run is worth
|
||||
watching: it is the only thing that reclaims chunks, and nothing has ever exercised it here.
|
||||
4. **Hub PBS-DR gauge granularity** — 0.1 GB steps mean routine incremental backups are invisible to
|
||||
it. Not a fault, but it cannot be used as write-proof evidence for small deltas.
|
||||
5. **ep0 has no swap** (3.7 GB CX23) — see the volume runbook's OOM.
|
||||
|
||||
## 11. Observations
|
||||
|
||||
- **`keep_last: 0` is the correct off-switch precisely because it is derived, not a flag.** Setting it
|
||||
disables both the `--prune-backups` value and the `allowPBSPrune` gate in one edit, and the tier
|
||||
stays armed. A dedicated "don't prune" boolean would have been a second source of truth.
|
||||
- **The false-negative was the real damage, not the unpruned snapshots.** demo-hp's PBS tier had
|
||||
reported failure on *every* backup since the tier was created on 07-26, while the data landed
|
||||
correctly every time. A tier that cries wolf on every success makes a genuine failure invisible —
|
||||
which is exactly what happened at 07:13 UTC, when a real outage produced the same-looking result.
|
||||
- **Chunk count is the honest probe for prune, not `df`.** They answer different questions, and `df`
|
||||
moved here for a reason that has nothing to do with reclamation.
|
||||
- `prune-job run` has no `--dry-run`; the gate must come from `proxmox-backup-client prune --dry-run`,
|
||||
which needs a token. The temporary-admin-token-then-delete pattern is the right shape and is already
|
||||
established in `felhom-tenantsync.sh`.
|
||||
Reference in new issue
Block a user