From a31872ea2417e5ea6c2b0008c980cfb8f60ca8c0 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 27 Jul 2026 15:16:11 +0200 Subject: [PATCH] docs(pbs): move PBS prune server-side, close the write proof, schedule GC Supervised runbook execution. No code, no version bump. The felhom-pbs tier had reported `job errors` on EVERY demo-hp backup since the tier was created on 07-26, while the data landed correctly every time: `DatastoreBackup` grants Datastore.Backup but not Datastore.Prune, so the box's keep_last=2 prune was denied. Operator ruling: retention is a COMMERCIAL attribute owned by the hub; ep0 executes. Box tokens therefore stay write-only - a compromised box must not be able to delete its own offsite backups. No grant was widened and felhom-tenantsync.sh is unchanged (the ruling makes it correct). Increment 1: - boxes stop attempting prune. allowPBSPrune is DERIVED (`!t.Primary && t.KeepLast > 0`), so keep_last: 0 on the PBS tier disables both the --prune-backups value and the gate in one config edit, and the tier stays armed. Verified prune_pbs_allowed=false on both boxes with no tier REJECTED line. - per-namespace prune jobs on ep0, keep-last 2, daily 03:30 UTC (05:30 CEST), dry-run gated. demo-hp 3->2, demo-felhom untouched, chunk count unchanged (prune removes indexes, not chunks). Write proof CLOSED: 08:25:47 job errors -> 09:37:29 TASK OK, snapshot 2026-07-27T09:37:29Z, chunks 9787->9813, prune step absent entirely. Driven through POST /api/guest-backup/trigger (the UI path), not --selftest and not raw vzdump. Hub gauge evidence explicitly NOT satisfied - the delta is below its 0.1 GB display granularity. GC scheduled sun 04:30 UTC and deliberately NOT run: every chunk still carries a fresh atime from the migration copy, so a run today would reclaim nothing. verify-new enabled per operator ruling, turning an inert hub alarm live. Legacy demo-felhom-01 namespace deleted with its two ACL entries and its token (operator ruling, confirmed twice) so nothing dangles. R-89 records the target architecture and carries the unanswered parallel question: does the restic key on storage-box-pool-1 have DELETE rights? If so the daily app-data tier has the identical exposure and append-only is the equivalent answer. ep0 is Etc/UTC, not CEST - corrected in the record. --- CONTEXT.md | 11 + REPORT-pbs-prune-serverside-2026-07-27.md | 71 ++++ documentation/backlog/ROADMAP.md | 1 + ...RUNBOOK-pbs-prune-serverside-2026-07-27.md | 325 ++++++++++++++++++ 4 files changed, 408 insertions(+) create mode 100644 REPORT-pbs-prune-serverside-2026-07-27.md create mode 100644 documentation/runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md diff --git a/CONTEXT.md b/CONTEXT.md index 3c6b561..59f4edb 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -57,6 +57,17 @@ or retiring it is → **R-83**. thresholds (host 26h / offsite 8d); fresh-install default; an unprovisioned tier DEFERS. **Operator rulings:** 2-week offsite retention, first backup runs as long as it needs, one backup at a time per guest, drill box dropped from the rollout. + **RETENTION IS A COMMERCIAL ATTRIBUTE — the hub decides, ep0 executes (operator ruling 2026-07-27, + R-89).** A paid tier may buy longer retention, so the policy belongs with customer config on the + hub, never in ep0's PBS config and never in a box's config. Execution stays server-side: a + reconciler writes a **PBS prune job** and PBS's own scheduler runs it, so hub downtime leaves the + last-known policy running rather than silently stopping retention. **Box tokens stay write-only + (`DatastoreBackup`) — never widen a grant to fix a prune error:** a compromised box must not be + able to delete its own offsite backups, which is the scenario offsite DR exists to survive. + Increment 1 shipped 2026-07-27 (boxes stop attempting prune via `keep_last: 0`; per-namespace prune + jobs on ep0, daily 03:30 UTC) — `runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md`. This closed a + live false-negative: **every** demo-hp PBS backup since 07-26 reported `job errors` while the data + landed correctly, because `DatastoreBackup` carries no `Datastore.Prune`. **Four defects found by RUNNING it, not reviewing it** — a 30-min wait bound against a 41-min backup (the agent recorded `success:false` while the backup was still going); the restore tier read from the configured target instead of the archive (**a silent regression of the S4.1 fix** — the diff --git a/REPORT-pbs-prune-serverside-2026-07-27.md b/REPORT-pbs-prune-serverside-2026-07-27.md new file mode 100644 index 0000000..476b171 --- /dev/null +++ b/REPORT-pbs-prune-serverside-2026-07-27.md @@ -0,0 +1,71 @@ +# REPORT — PBS prune moved server-side, write proof closed (2026-07-27) + +**Class:** supervised operational run. **No code, no version bump.** Topic-scoped per the +parallel-session rule; shared `REPORT.md` untouched. + +**Full record:** `documentation/runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md` + +--- + +## Outcome — all parts complete + +| Part | Result | +|---|---| +| 1 — prune gate | **Config-gated.** `keep_last: 0` on the PBS tier, both boxes → `prune_pbs_allowed=false`. No code, **no grant** | +| 2 — prune jobs | 2 jobs, per live namespace, `keep-last 2`, daily **03:30 UTC / 05:30 CEST** | +| 3 — dry run → real | Gate passed; both `TASK OK`; demo-hp 3→2, demo-felhom untouched | +| 4 — write proof | **CLOSED — `TASK OK`, no job errors** | +| 5 — GC | Scheduled `sun 04:30 UTC / 06:30 CEST`. **NOT run** | +| 6 — `verify-new` | **Enabled** (operator ruling) | +| — legacy ns | `demo-felhom-01` deleted with its ACLs + token (operator ruling, confirmed twice) | +| 7 — roadmap | **R-89** + CONTEXT.md note | + +## The fix, in one line + +`allowPBSPrune := !t.Primary && t.KeepLast > 0` — so setting the PBS tier's `keep_last` to `0` +disables both the `--prune-backups` value and the gate, in one config edit, **while the tier stays +armed**. Verified: `backup tier armed target=felhom-pbs cadence=168h0m0s keep_last=0 +prune_pbs_allowed=false`, no `tier REJECTED` line. + +## The proof + +``` +07-27 08:25:47 UTC vzdump (felhom-pbs) -> job errors ← prune denied +07-27 09:37:29 UTC vzdump (felhom-pbs) -> OK ← after the change +``` + +New snapshot `ns/demo-hp/ct/9201/2026-07-27T09:37:29Z`, chunks 9,787 → **9,813**, 97.0 % reused, +45.80 s, **prune step absent entirely**. Driven via `POST /api/guest-backup/trigger` → `TriggerNow()` +— the UI's „Mentés most" path, not `--selftest`, not raw `vzdump`. + +**Hub gauge evidence NOT satisfied** — a +32.8 MB delta is below its 0.1 GB display granularity, so it +still reads 12.6 GB / 13 %. Stated plainly rather than dressed up. + +## The demo-felhom prediction — CLOSED + +The claim was that demo-felhom's next weekly backup would make 3 snapshots and reproduce the prune +failure. Neutralised on both halves: the box no longer attempts prune, and `prune-demo-felhom` covers +the namespace server-side (verified live, `TASK OK`). **It will not reproduce.** + +## Why it mattered more than the unpruned snapshots + +demo-hp's PBS tier had reported failure on **every** backup since the tier was created on 07-26, while +the data landed correctly every time. A tier that cries wolf on every success makes a genuine failure +invisible — which is precisely what happened at 07:13 UTC, when a real outage produced an +indistinguishable result. + +## Security property preserved + +**No prune right was granted to any box.** Final ACLs are four entries, write-only +(`DatastoreBackup`), live namespaces only. A compromised box still cannot delete its own offsite +backups. `felhom-tenantsync.sh` was **not** edited — the ruling makes its current grant correct. + +## Open + +1. **R-89** — hub-owned retention policy (today's jobs are increment 1, not a stopgap). +2. **Does the restic key on `storage-box-pool-1` have DELETE rights?** Unanswered, carried in R-89, + and the more urgent half — if so, the daily app-data tier has the identical exposure and + append-only mode is the equivalent answer. Rule once for both tiers. +3. **GC has still never run.** First execution Sunday 04:30 UTC; worth watching, as nothing has ever + exercised it here. +4. Old 13 GB datastore copy still at `/srv/pbs-felhom` — rollback intact. diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 6377d61..299565c 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -123,6 +123,7 @@ Each attempt runs the **full quiesce cycle**, so every customer app stack is STO **The amplifier, and the part worth designing against:** the agent answers `Due: true, Reason: "no successful backup recorded yet", AgeSecs: nil`, and that **nil age does double duty**. `scheduledRunAllowed` (`quiesce.go:466-480`) returns `true` whenever `lastAgeSecs == nil` — *"no recorded backup yet — never withhold the first one"* — so the same nil that makes every poll due **also bypasses the time-of-day gate** `[W+2h, W+6h)`. On the live box the gate was `[04:30, 08:30)` and the cycles ran at 09:02–09:12 Budapest, i.e. **outside the backup window entirely**. So the fault stops customer apps every 5 minutes *at any hour, including business hours* — the one protection specifically built to prevent that is switched off by the same missing value. A safety valve written for a genuine first-ever backup is being triggered by an unreachable storage read, which is not the same thing at all. Self-resolves the moment the target answers (the storage read succeeds, sees the archive, tier stops being due) — which is why it can hide indefinitely: it needs an offsite outage to appear at all. **Shape:** distinguish *storage unreachable* from *storage readable and empty*. Unreachable is UNKNOWN — defer the due-verdict rather than resolving it either way, exactly as R-81 made the hub do with a missing report. Only a target that is reachable AND has no archive is genuinely due. **Fix the window bypass in the same slice:** `AgeSecs == nil` must stop meaning "run now regardless of the hour". Either the agent distinguishes *never backed up* from *cannot tell* in what it reports, or `scheduledRunAllowed` gates on the former only — otherwise any future nil-age path re-opens the same hole. Note this does NOT weaken R-84's fail-safe intent: a tier whose storage is merely slow or briefly unreadable should still err toward backing up — it is specifically the **cold-store + unreachable** pair that must defer, because there the fallback has no information at all, only an empty default that looks like a fact. | +| R-89 | **Retention is a COMMERCIAL attribute — it belongs to the hub, not to ep0 or a box** | M | idea — operator ruling 2026-07-27, first increment SHIPPED same day | **Ruling (2026-07-27):** retention is a per-customer *commercial* attribute (a paid tier may buy longer retention), so the **hub owns the policy** and ep0 merely executes it. Execution stays **server-side**: a reconciler writes a **PBS prune job** and PBS's own scheduler runs it, so hub downtime leaves the last-known policy running rather than silently stopping retention. **Increment 1 SHIPPED 2026-07-27** (`runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md`): boxes no longer attempt prune (`keep_last: 0` → `allowPBSPrune=false`, config only — no code, no grant), and per-namespace prune jobs run on ep0 daily 03:30 UTC (`keep-last 2`), dry-run gated and verified `TASK OK`. This also closed a live defect — **every** demo-hp PBS backup since the tier was created on 07-26 had reported `job errors` while the data landed correctly, because `DatastoreBackup` grants `Datastore.Backup` but not `Datastore.Prune`; a tier that cries wolf on every success makes a genuine failure invisible, which is exactly what happened during the 07-27 migration window. **Remaining work:** retention becomes a per-customer attribute on the hub — a **policy** (keep-last/daily/weekly/monthly), NOT a bare number, since a paid tier will want "12 weekly + 6 monthly"; a reconciler converges it into the ep0 prune job in the same descriptor-and-converge shape as `pbs_dr`. **SECURITY PROPERTY TO PRESERVE — do not "fix" a future prune error by widening the grant:** box tokens stay **write-only** (`DatastoreBackup`), never `DatastorePowerUser`/`DatastoreAdmin`. A compromised box must not be able to delete its own offsite backups — that is the scenario offsite DR exists to survive. **PARALLEL QUESTION, unanswered and arguably the more urgent half: does the restic key on `storage-box-pool-1` (`u629488`) have DELETE rights?** If so the daily app-data offsite tier carries the identical exposure, and restic's **append-only** mode is the equivalent answer. **Rule once for both tiers.** Flips the capability-map row for offsite retention (currently: retention configured but unenforceable) | | R-76 | **FileBrowser-created folders break the setgid chain, and a drop-zone's mode is not stable** | S | idea (surfaced by the R-75 spike, 2026-07-26) | Two related findings from `audits/SPIKE-catalog-data-paths-2026-07-26.md` P3/P5, both **pre-existing** and deliberately left alone by that spike. **(a)** FileBrowser Quantum 1.3.3 creates files `0644` and folders `0755` and does **not** propagate the setgid bit — even though the entrypoint wrapper's `umask 002` really is in effect (`/proc/1/status` `Umask: 0002`). Group inheritance itself works (a file uploaded into a 2775 group-100 dir landed group 100, not the process gid 1000), so the convention's *group* half holds and only its *mode* half is lost. The consequence is proven with a control: inside a UI-created `0755` folder a gid-1000 process's file landed group **1000**, while the identical write into the 2775 parent landed group **100**. So **any folder a customer creates through FileBrowser breaks the shared-group chain one level down.** Latent today — every userdata-touching catalog app that declares an identity declares uid/gid **1000**, the same uid FileBrowser runs as, so owner permissions mask it; it bites the day a content app runs as a different non-root uid with gid 1000. The comment at `infra/infra.go:156` is right that the image ignores `-e UMASK` but does not say the wrapper fails to achieve the intended mode either. **(b)** `import/calibre` is live on demo-felhom at `755 1000:1000` where every sibling is `2775 root:1000` — with `media/books` on the same box, same app, same deploy, at `2775` as the control, and no parser asymmetry (checked: `ParseComposeUserdataMounts` picks up both calibre-web binds). Consistent with the consuming app rewriting the mode of its own ingest dir after the deploy belt sets it; **not confirmed causally**. Together they mean **a drop-zone directory's mode is not stable against either the customer or the consuming app** — verify before building anything that assumes `import/*` stays 2775. Flips no capability-map row today (latent); would become customer-visible the moment a non-1000 content app enters the catalog | | R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → universal secret-free / operator-bind (option C) | XL | **SHIPPED + PHYSICALLY CLOSED (slices A+B+C; rehearsal executed 2026-07-18)** | **PHYSICAL CLOSURE 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the generic pairing ISO v1.20.0 (`--loader mkimage`, SB off) **booted the very AMI board that F1 blocked**, installed unattended, and the box self-registered as an unclaimed appliance the same second it first booted (16:17:14) → self-bind → credential → day-0 SUCCESS 16:32:32 → floor-lifted to current. **F1 is closed on physical hardware and the rehearsal dependency on this item is discharged.** Two residual notes stay open, neither blocking: PXE/network-boot is still unbuilt (the third F1 option, for boards where even USB-mkimage fails), and mkimage remains unsigned → **SB must be OFF**. The installer's GRUB menu still offers interactive installers → new item **R-38**. — Prior: **PHYSICAL RUN 2026-07-16 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`):** demo N100 reinstalled clean-slate from a pipeline ISO → chain reached **rc-0 first try on real hardware** (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. **F1 (HIGH, slice-B input):** this cheap AMI `AN3PLUS 0.01` firmware won't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`) — SB-off/shim-bypass don't help; worked around live with a `grub-mkimage` loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B — SHIPPED (scripts v1.18.0, 2026-07-17):** the F1 firmware fix is now a first-class pipeline mode `build-felhom-iso.sh --loader shim|mkimage` (default shim; `mkimage` = monolithic grub-mkimage loader from the ISO's own GRUB, recipe from the run evidence). RUNBOOK-B legs on nested VM 311 proved it: shim boots+installs under OVMF SB-enforcing + SeaBIOS; mkimage boots+installs under OVMF SB-off; mkimage under SB-enforcing FAILS `Access Denied` (unsigned → **SB must be OFF**, documented); surgery byte-identical payload. **Physical N100 boot on the real board still pending** → folds into the supervised rehearsal (R-1; an `n100-safety` match-nothing ISO is built + sha-recorded for a zero-risk pre-flight). **PXE/network-boot** (the third F1 option, for boards where even USB-mkimage fails) stays a deferred note under this item — not built. **Slice C — SHIPPED (hub v0.62.0 + scripts v1.19.0, 2026-07-17):** the GENERIC secret-free universal ISO (`build-felhom-iso.sh --pairing`). The box self-registers as an unclaimed appliance (keyed by SMBIOS-uuid + MAC set — the DMI-verdict tiebreaker), the operator BINDS it to a customer on the Hosts page, and the hub delivers customer-id + retrieval passphrase ONCE (`/api/v1/appliance/register` + one-shot poll, 404-no-oracle — all live-verified through the public ingress); the bootstrap then falls through to the slice-A direct path. **One unit, two modes** (direct = byte-identical, regression-proven zero-appliance-calls). Artifact proven secret-free (baked env = hub URL only; manifest `secret-bearing: no`). The **bind is operator-password-gated** → the live boot→register→bind→day-0 composition (with a Viktor-created drill customer) + the physical N100 boot fold into the supervised rehearsal — **which now runs the COMPLETE final product flow in one pass: RESET the demo → boot the generic ISO → bind → day-0**. SSH-host-key pinning: stored + fingerprints displayed (attaching to the host on bind = future, no clean hand-off surface today). **Customer-facing self-bind page = R-27 (future).** Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* | **SECOND-HARDWARE PROOF 2026-07-21 (slice C, demo-hp): the pairing flow is no longer a one-board result.** The universal secret-free ISO was booted on a completely different machine — an **HP t740 (Ryzen V1756B, AMI M42 firmware)** versus the N100 it was proven on — and the whole chain ran on **virgin hardware in one pass**: armed install → self-registration as an unclaimed appliance → operator bind → day-0, ending with a running customer guest 9201 and agent 0.92.1 checking in as host `demo-hp-bb76ea`. **Two things generalise from the second board specifically:** the **shim** loader booted with **Secure Boot ENABLED** (`mokutil --sb-state` → `SecureBoot enabled`), confirming the mkimage/SB-off dance is an N100-firmware workaround and NOT a Felhom requirement; and the exact-serial disk filter selected the SanDisk system SSD while leaving the box's **1TB NVMe untouched and unenrolled** (its prior NTFS partition is still intact, unmounted, in no LVM/ZFS) — the destructive path stayed inside its filter on hardware it had never seen. **Not clean, and the failures are filed:** the install got no DHCP on the 4-port NIC and baked a static fallback rather than aborting (**R-59**), which cost a cable move and a hand-repair; and the console was unreachable because the baked root password is unknowable (**R-61**) | R-29 | **The design-v2 green gates are not enforced anywhere — one has been RED for 16 releases.** `controller/scripts/docker_run_volume_path_gate.py` has failed continuously since **2026-07-14 (v0.129.0)** and nobody noticed until R-7b's close-out ran it by hand at v0.145.0. Two separable parts. **(a) The finding itself is benign and the fix is 3 lines.** The flagged call is `internal/appexport/estimate.go:179` `docker run --rm -v :/vol:ro alpine du` — a **NAMED-VOLUME** mount, i.e. daemon-side with no host path, which is the *safe* shape and byte-for-byte the same pattern as three entries already on the gate's ALLOWLIST (`export.go` `volName+":/vol"`, `backup.go` `volName+":/vol:ro"`, `restore.go` `volName+":/vol"`). It is NOT the v0.124.0 path-strand class the gate exists to catch — the author of the v0.129.0 F-A fix explicitly avoided that class (see the function's own comment) and simply never added the allowlist entry. So the fix is an ALLOWLIST addition WITH ITS WHY, **not** a docker-cp rewrite; anyone who 'fixes' this by rewriting the call has misread the gate. **(b) The systemic half is the real item:** the gates run only when a human remembers to run them, so a gate can sit red across 16 releases while every REPORT says 'green'. This is the SECOND instance of the class — cf. the v0.123.0 note *'Windows green gate silently red (read-only fsync)'*. Decide where they run (pre-push hook, `build.sh` step, or a CI job) and make a red gate block the train the way the Go green gate does. | S (a) / M (b) | idea | Origin: R-7b close-out, `felhom-controller` REPORT §4(f) — CC correctly left it alone as out-of-scope and pre-existing, and verified by stashing that it fails identically on the unmodified tree. Flips no capability-map row (engineering hygiene, no customer-visible behaviour). Affected gates to audit for the same rot: controller `template_id_gate` / `emoji_gate` / `native_confirm_gate` / `offbox_rename_gate` / `mojibake_gate` / `app_row_dedup_gate` / `docker_run_volume_path_gate`, hub `hub_confirm_gate`, manifests `manifest_bearer_gate`, website `site_gates`. **Do not bundle (a) into an unrelated feature commit** — it is a one-line behavioural claim about a mount's safety and deserves its own reviewed diff. **2026-07-18 rehearsal note:** the run's finding list independently re-raised "assign the pre-existing `docker_run_volume_path_gate` failure its ID so red stops normalizing" — **that is this item; no second ID was minted.** | diff --git a/documentation/runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md b/documentation/runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md new file mode 100644 index 0000000..e3f2056 --- /dev/null +++ b/documentation/runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md @@ -0,0 +1,325 @@ +# RUNBOOK RECORD — PBS prune moved server-side, write proof closed, GC scheduled (2026-07-27) + +**Class:** supervised operational run. **No code, no version bump.** Follows +`RUNBOOK-ep0-cleanup-2026-07-27.md` (Parts A–C complete; Part D stopped correctly on the prune defect). + +**Targets:** `ep0` / `felhom-hetzner`, `demo-felhom` (N100), `demo-hp` (HP t740). + +> **Timezone note, corrected from the runbook's standing rule.** `ep0` runs **`Etc/UTC`**, not CEST — +> `timedatectl` reports `Time zone: Etc/UTC (UTC, +0000)`, NTP-synced. The CEST↔UTC split applies to +> the **boxes** (`demo-felhom`, `demo-hp` are CEST), not to ep0. PBS calendar events therefore resolve +> in **UTC** on this host. Both zones are still stated for every schedule below. + +--- + +## Outcome + +| Item | Result | +|---|---| +| Prune attempt by boxes | **disabled** — config only, no grant, no code | +| Server-side prune | **2 jobs live**, daily `03:30` UTC, `keep-last 2` | +| Write proof | **CLOSED — `TASK OK`, no job errors** | +| GC | **scheduled** `sun 04:30` UTC — **NOT run** | +| `verify-new` | **enabled** (operator ruling) | +| Legacy `demo-felhom-01` | **deleted** — namespace + ACLs + token (operator ruling, confirmed twice) | +| Old 13 GB copy | **still in place** | + +--- + +## 1. Part 1 — the prune is CONFIG-GATED (branch taken: turn it off) + +**Source landmark** — `internal/backup/runner.go:104`, inside `localPruneSpec`: + +```go +if s.Type == "pbs" && !r.allowPBSPrune { + // Not opted in → never prune the offsite DR + return "" +} +``` + +`allowPBSPrune` is **derived, not a config key** — `cmd/felhom-agent/main.go:1348`: + +```go +allowPBSPrune := !t.Primary && t.KeepLast > 0 +``` + +So the off-switch is the tier's `keep_last`. **Safety checked before changing it:** `BackupTiers()` +(`internal/config/config.go:453-495`) rejects a tier only for an empty/duplicate `target_id` or +`cadence_seconds <= 0`. `keep_last` is **not** a rejection condition, and for extra tiers +`keep < 0 → 0` is legal and preserved. Only the **primary** tier clamps to ≥1 (`KeepLast()`), and +that is the local tier, untouched. So `keep_last: 0` disarms the prune **without disarming the tier**. + +**`agent.json` diff — identical on both boxes** (backed up to `agent.json.pre-prunegate.bak`): + +```diff +29c29 +< "keep_last": 2 +--- +> "keep_last": 0 +``` + +No in-flight restore-test or backup on either box at the time; both agents restarted cleanly. + +**Verified after restart — both boxes:** + +``` +backup tier armed target=local cadence=24h0m0s keep_last=3 prune_pbs_allowed=false primary=true +backup tier armed target=felhom-pbs cadence=168h0m0s keep_last=0 prune_pbs_allowed=false primary=false +backup: restore-test scheduler starting cadence=84h0m0s +``` + +**The PBS tier is still ARMED** (cadence 168h, no `tier REJECTED` line), the local tier is unchanged, +and the restore-test cadence is still the reverted 84h. **No prune right was granted to any box.** + +--- + +## 2. Part 2 — server-side prune jobs on ep0 + +**Namespace-path form:** `prune-job create` takes `--ns ` — **not** +`/datastore//` and **not** `/ns/`. That is the form used. `--max-depth 0` = non-recursive. + +```sh +proxmox-backup-manager prune-job create prune-demo-felhom \ + --store felhom-offsite --ns demo-felhom --max-depth 0 \ + --keep-last 2 --schedule "03:30" \ + --comment "R-82 retention keep-last=2, server-side (box tokens are write-only)" +# same for prune-demo-hp +``` + +| id | store | ns | schedule | max-depth | keep-last | +|---|---|---|---|---|---| +| `prune-demo-felhom` | felhom-offsite | demo-felhom | `03:30` | 0 | 2 | +| `prune-demo-hp` | felhom-offsite | demo-hp | `03:30` | 0 | 2 | + +**Schedules, both zones:** + +| Job | ep0 local (UTC) | CEST | +|---|---|---| +| prune (daily) | **03:30** | 05:30 | +| GC (weekly, Sun) | **04:30** | 06:30 | + +**Ordering holds:** prune runs daily including Sunday, one hour before the weekly GC on the same +morning — so GC always sees the freshest prune result. **Daily was chosen deliberately**: with weekly +backups the prune is usually a no-op, and a no-op that runs beats a weekly one that misses. + +> **The weekly backup is due-based and therefore drifts**, so no fixed schedule can guarantee +> non-overlap. Occasional overlap is possible and is accepted, not designed away. + +--- + +## 3. Part 3 — dry run, then the real prune + +### Dry run (verbatim) — DELETED NOTHING + +Run with a **temporary `root@pam` token** granted `DatastoreAdmin`, removed on exit (the pattern +`felhom-tenantsync.sh` already uses). `prune-job run` has **no `--dry-run`**, so the client-side +`proxmox-backup-client prune … --dry-run` was used. + +``` +############ DRY RUN ns=demo-felhom keep-last=2 ############ +| snapshot | date | action | +| ct/9201/2026-07-18T18:31:06Z | Sat Jul 18 18:31:06 2026 | keep | +| ct/9201/2026-07-26T12:21:48Z | Sun Jul 26 12:21:48 2026 | keep | + +############ DRY RUN ns=demo-hp keep-last=2 ############ +| snapshot | date | action | +| ct/9201/2026-07-26T15:42:42Z | Sun Jul 26 15:42:42 2026 | remove | +| ct/9201/2026-07-26T17:02:57Z | Sun Jul 26 17:02:57 2026 | keep | +| ct/9201/2026-07-27T08:25:47Z | Mon Jul 27 08:25:47 2026 | keep | +``` + +**Gate passed exactly as specified:** demo-hp loses only its excess above 2; **demo-felhom loses +nothing**. The STOP condition was not triggered. + +### Real run — both `TASK OK` + +``` +prune job 'prune-demo-felhom' → keep, keep → TASK OK +prune job 'prune-demo-hp' → remove ct/9201/2026-07-26T15:42:42Z, keep, keep → TASK OK +``` + +| Namespace | Before | After | Newest snapshot | +|---|---|---|---| +| demo-felhom | 2 | **2** | `2026-07-26T12:21:48Z` — unchanged | +| demo-hp | 3 | **2** | `2026-07-27T08:25:47Z` — unchanged | +| demo-felhom-01 | 3 | 3 | untouched (no job) | + +**Chunks: 9,787 → 9,787 — unchanged.** That is the load-bearing confirmation: prune removed snapshot +indexes and reclaimed no chunks. + +**`df` moved by −77,824 B (−76 KB).** The runbook predicted no movement; the substance is right (no +chunks freed) but not literally zero, because the removed snapshot's **own index files** +(`root.pxar.didx` ~57 KB, `catalog.pcat1.didx`, `index.json.blob`, `pct.conf.blob`) occupy disk and go +with it. Nothing other than prune ran. **Chunk count, not `df`, is the correct probe for "did prune +reclaim data".** + +--- + +## 4. Part 4 — the write proof, CLOSED + +Driven through the real flow: `POST /api/guest-backup/trigger` → `TriggerNow()` (the UI's +„Mentés most"), which deliberately covers **every** tier in one quiesce window. **Not `--selftest`, +not raw `vzdump`.** + +**Before vs after, same box, same path:** + +``` +07-27 08:25:47 UTC vzdump (felhom-pbs) -> job errors ← prune denied +07-27 09:37:29 UTC vzdump (felhom-pbs) -> OK ← after the Part 1 change +``` + +``` +INFO: creating Proxmox Backup Server archive 'ct/9201/2026-07-27T09:37:29Z' +INFO: root.pxar: had to backup 123.95 MiB of 3.975 GiB (compressed 30.431 MiB) in 44.64 s +INFO: root.pxar: backup was done incrementally, reused 3.854 GiB (97.0%) +INFO: Duration: 45.80s +TASK OK +``` + +**PBS side:** `UPID:…backup:felhom-offsite:ct-9201:felhom@pbs!demo-hp` → **OK** (09:37:36 → 09:38:22). + +| Evidence | Result | +|---|---| +| Snapshot identity | `ns/demo-hp/ct/9201/**2026-07-27T09:37:29Z**` | +| Contents | `root.pxar.didx` 57,416 B, `catalog.pcat1.didx`, `index.json.blob`, `pct.conf.blob`, `client.log.blob` — real, not an empty dir | +| Chunk delta | 9,787 → **9,813** (+26) | +| `df` delta | 13,579,243,520 → 13,612,040,192 (**+32.8 MB**) | +| Prune step | **absent entirely** — `keep_last: 0` means no `--prune-backups` flag, so the permission check is never reached | +| Job errors | **none** | + +**The hub PBS-DR gauge evidence is NOT satisfied.** A +32.8 MB delta is below the gauge's 0.1 GB +display granularity, so it still reads 12.6 GB / 13 %. Stating that plainly rather than dressing it +up — the same call as last time, and it still stands. + +--- + +## 5. Part 5 — GC scheduled, NOT run + +```sh +proxmox-backup-manager datastore update felhom-offsite --gc-schedule "sun 04:30" +``` + +``` +| name | felhom-offsite | +| path | /mnt/pbs-datastore | +| gc-schedule | sun 04:30 | +| verify-new | 1 | +``` + +**GC was NOT run**, confirmed: `garbage-collection status` reports `upid` empty and `disk-chunks 0` — +it has never executed on this datastore. + +Deliberate: every chunk still carries a **fresh atime from this morning's migration copy**, and GC +only reclaims chunks whose atime predates the cutoff. A run today would reclaim nothing regardless of +what Part 3 pruned. **The first meaningful GC is the scheduled one.** + +--- + +## 6. Part 6 — `verify-new` ruling: ENABLED + +**Operator ruling 2026-07-27: enable it.** + +```sh +proxmox-backup-manager datastore update felhom-offsite --verify-new true +``` + +Rationale recorded: nothing on this PBS ever verified, so `verify_state` stayed `none` forever and the +hub's *"newest PBS snapshot failed verification"* alarm **could not fire** — inert, in the same class +as the hub 400ing an unknown event type. `verify-new` verifies each snapshot as it is created; with +`keep-last 2` that covers essentially the whole datastore and turns a dead check live, for a few +minutes of ep0 CPU per weekly backup. + +> Watch item: ep0 is a 3.7 GB CX23 with **no swap**, and inline verification runs within the backup +> window. Today's full forced verify completed fine (~250 MiB/s, 0 errors), but see +> `RUNBOOK-ep0-datastore-volume-2026-07-27.md` for the rsync OOM on this same box. + +--- + +## 7. Legacy `demo-felhom-01` namespace — DELETED (operator ruling, confirmed twice) + +Not in the original runbook; surfaced because "a prune job per customer namespace" would have applied +`keep-last 2` to a retired install's archive. Flagged rather than actioned silently, then ruled on. + +**Destroyed:** 3 snapshots, all 2026-07-04 (`14:37:16Z`, `14:55:44Z`, `17:51:34Z`), from the retired +`demo-felhom-01` install. No live box had written there since 07-04. + +**Removed with it, so nothing dangles** (the same shape as this morning's `scratch` datastore): + +| Object | Status | +|---|---| +| namespace `demo-felhom-01` | deleted (`namespace delete --delete-groups true`) | +| ACL `felhom@pbs` → `/datastore/felhom-offsite/demo-felhom-01` | deleted | +| ACL `felhom@pbs!demo-felhom-01` → same path | deleted | +| token `felhom@pbs!demo-felhom-01` | deleted | + +**Final ACL state — four entries, write-only, live namespaces only:** + +``` +felhom@pbs /datastore/felhom-offsite/demo-felhom DatastoreBackup +felhom@pbs /datastore/felhom-offsite/demo-hp DatastoreBackup +felhom@pbs!demo-felhom /datastore/felhom-offsite/demo-felhom DatastoreBackup +felhom@pbs!demo-hp /datastore/felhom-offsite/demo-hp DatastoreBackup +``` + +Live namespaces verified intact throughout (demo-felhom 2, demo-hp 3). Datastore name and path +unchanged. The temporary `root@pam!nsdel` token **and its ACL** were both removed. + +> **It freed almost nothing today, and that was stated before the confirmation:** the namespace +> directory was **620 KB** (index files only). Its chunks live in the shared `.chunks` store and come +> back only when GC's atime cutoff passes — weeks away, not today. + +--- + +## 8. The demo-felhom prediction — CLOSED + +The falsifiable claim from the previous runbook was: *demo-felhom sits at exactly 2 snapshots, so its +prune has silently had nothing to do; its next weekly backup makes 3 and will fail identically.* + +**It is now neutralised on both halves, independently:** + +1. **The box no longer attempts prune** — `prune_pbs_allowed=false` on demo-felhom, so no + `--prune-backups` flag is passed and the permission check is unreachable. +2. **Server-side prune covers the namespace** — `prune-demo-felhom` runs daily at 03:30 UTC and was + **verified live** (`TASK OK`, both snapshots kept). + +So demo-felhom's third snapshot will be pruned by ep0 under `root@pam`, not by the box, and the +failure mode cannot recur. **Prediction closed — the failure will not reproduce.** + +--- + +## 9. Old datastore copy — STILL IN PLACE + +``` +drwxr-xr-x 4 backup backup 4096 Jul 3 21:48 /srv/pbs-felhom +13G /srv/pbs-felhom +``` + +Untouched. Rollback remains a two-line `datastore.cfg` revert. Volume: 98 G, 13 G used, 13 %. + +--- + +## 10. Deferred / open + +1. **R-89** (below) — hub-owned retention policy. Today's prune jobs are the first increment of it, + not a stopgap. +2. **The restic delete-rights question** carried in R-89 — unanswered, and the more urgent half. +3. **GC has still never run.** First scheduled execution Sunday 04:30 UTC. Its first run is worth + watching: it is the only thing that reclaims chunks, and nothing has ever exercised it here. +4. **Hub PBS-DR gauge granularity** — 0.1 GB steps mean routine incremental backups are invisible to + it. Not a fault, but it cannot be used as write-proof evidence for small deltas. +5. **ep0 has no swap** (3.7 GB CX23) — see the volume runbook's OOM. + +## 11. Observations + +- **`keep_last: 0` is the correct off-switch precisely because it is derived, not a flag.** Setting it + disables both the `--prune-backups` value and the `allowPBSPrune` gate in one edit, and the tier + stays armed. A dedicated "don't prune" boolean would have been a second source of truth. +- **The false-negative was the real damage, not the unpruned snapshots.** demo-hp's PBS tier had + reported failure on *every* backup since the tier was created on 07-26, while the data landed + correctly every time. A tier that cries wolf on every success makes a genuine failure invisible — + which is exactly what happened at 07:13 UTC, when a real outage produced the same-looking result. +- **Chunk count is the honest probe for prune, not `df`.** They answer different questions, and `df` + moved here for a reason that has nothing to do with reclamation. +- `prune-job run` has no `--dry-run`; the gate must come from `proxmox-backup-client prune --dry-run`, + which needs a token. The temporary-admin-token-then-delete pattern is the right shape and is already + established in `felhom-tenantsync.sh`.