From 51871a7ea6a2209e050cc2fe81399ef9fd28754f Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 4 Aug 2026 09:46:09 +0200 Subject: [PATCH] installer 1.25.0: the off-site tier stops asking to prune (R-191) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every weekly off-site run uploaded successfully and then failed the job on a prune the box's token is deliberately refused — R-89 moved off-site pruning server-side to ep0 and box tokens stay write-only. The 2026-07-26 'two weeks' ruling was not reversed; where it is enforced moved, and keep_last: 2 did not follow. Now 0, which the agent's existing guard already reads as 'never prune from the box'. Verified read-only on ep0 before changing it: both namespaces have a prune job at 03:30 keep-last 2 that has run every day since 2026-07-27 — 18 tasks, all OK, the newest keeping exactly two. Without that check this would have traded a weekly false alarm for unbounded growth. A gate asserts the offsite tier carries no client-side prune. The local tier is untouched. --- scripts/CHANGELOG.md | 23 +++++++++++++++++++++++ scripts/felhom-host-install.sh | 30 ++++++++++++++++++++++++++---- scripts/hostinstall_gates.py | 26 ++++++++++++++++++++++++++ 3 files changed, 75 insertions(+), 4 deletions(-) diff --git a/scripts/CHANGELOG.md b/scripts/CHANGELOG.md index 5c112b5..dae8554 100644 --- a/scripts/CHANGELOG.md +++ b/scripts/CHANGELOG.md @@ -1,3 +1,26 @@ +## 1.25.0 — the off-site tier stops asking to prune (2026-08-04, R-191) + +**A backup that worked must not report failure.** The off-site tier was written with `keep_last: 2`, +so every weekly run uploaded its snapshot successfully and then failed the whole job on a prune the +box's token is deliberately refused: `prune 'ct/9201': permission check failed - missing +Datastore.Modify|Datastore.Prune` → `TASK ERROR: job errors` → `whole_guest_backup_failed` in the +operator's inbox. Every week, on both boxes, about a backup that had already succeeded. + +**R-89 moved off-site pruning SERVER-SIDE** — ep0 runs a per-namespace prune job and box tokens stay +write-only, so a box can never delete its own off-site history. The 2026-07-26 "two weeks" ruling was +not reversed; where it is ENFORCED moved, and this value did not follow. The tier now writes +`keep_last: 0`, which the agent's existing guard (`allowPBSPrune = !primary && keep_last > 0`) already +reads as "never prune from the box" — no agent change needed. + +**VERIFIED BEFORE CHANGING IT** (read-only on ep0, 2026-08-04): prune jobs `prune-demo-felhom` and +`prune-demo-hp` exist on datastore `felhom-offsite`, one per namespace, schedule 03:30, keep-last 2, +and have run **every day since 2026-07-27 — 18 tasks, all `status=OK`**, the newest showing +`retention options: --ns demo-felhom --max-depth 0 --keep-last 2` and keeping exactly two. Disabling +the client-side prune without that check would have traded a weekly false alarm for unbounded growth. + +A gate now asserts the off-site tier carries no client-side prune, so the value cannot drift back +quietly. The local tier's retention is untouched. + ## 1.24.0 — a pre-existing backup target is granted too (2026-08-03, R-185) **`configure_backup_target` has two arms and only one of them granted.** The Case A arm creates the diff --git a/scripts/felhom-host-install.sh b/scripts/felhom-host-install.sh index e6fabd7..0770978 100644 --- a/scripts/felhom-host-install.sh +++ b/scripts/felhom-host-install.sh @@ -184,7 +184,7 @@ set -euo pipefail -SCRIPT_VERSION="1.24.0" # the SINGLE version source (F-1): -h and the run banners follow it. +SCRIPT_VERSION="1.25.0" # the SINGLE version source (F-1): -h and the run banners follow it. # The hub used to carry a copy for its Setup tab; R-94 DELETED it # (2026-08-02) because the hub cannot know which version a box runs — # the Setup command fetches this script at run time. scripts/ @@ -2479,8 +2479,30 @@ for _k,_v in {"unit_dir":"/etc/systemd/system","stage_dir":"/var/lib/felhom-agen base.setdefault('storage', {"watchdog_interval_seconds":5,"watchdog_debounce_seconds":15,"known_refresh_seconds":20}) # R-82: local DAILY + offsite WEEKLY. The two tiers carry SEPARATE cadences and retentions — # keep_last=3 is three DAYS on the daily tier and three WEEKS on a weekly one, so one shared knob -# would guarantee that one of them is wrong. keep_last=2 on the offsite tier = two weeks (operator -# ruling 2026-07-26). +# would guarantee that one of them is wrong. +# +# THE OFFSITE TIER CARRIES NO CLIENT-SIDE RETENTION, AND THAT IS THE POINT (R-191, 2026-08-04). +# It used to be written `keep_last: 2` ("two weeks", operator ruling 2026-07-26). **R-89 then moved +# offsite pruning SERVER-SIDE** — ep0 runs a per-namespace prune job and box tokens stay write-only, +# deliberately, so that a box can never delete its own offsite history. The 2026-07-26 ruling was not +# reversed; where it is ENFORCED moved, and this value did not follow. +# +# The consequence was weekly and wrong in the worst direction: vzdump UPLOADED the snapshot fine and +# then failed the whole job on the prune the token is refused — +# `prune 'ct/9201': permission check failed - missing Datastore.Modify|Datastore.Prune` +# → `TASK ERROR: job errors` → `whole_guest_backup_failed` in the operator's inbox. Every week, on +# both boxes, about a backup that had already succeeded. A tier that cries wolf weekly is a tier +# whose real failure nobody will see. +# +# VERIFIED BEFORE CHANGING IT (2026-08-04, read-only on ep0): prune jobs `prune-demo-felhom` and +# `prune-demo-hp` exist on datastore `felhom-offsite`, one per namespace, schedule 03:30, keep-last 2, +# and have run EVERY DAY since 2026-07-27 — 18 tasks, all `status=OK`, the newest showing +# `retention options: --ns demo-felhom --max-depth 0 --keep-last 2` and keeping exactly two. Retention +# happens; it happens THERE. **If that ever stops being true, this zero is unbounded growth** — check +# ep0's prune jobs before assuming the offsite tier is retained. +# +# `keep_last: 0` means "never prune from the box" and is the value the agent's own guard reads +# (allowPBSPrune = !primary && keep_last > 0), so no agent change is needed to honour it. # # The offsite tier is written even though `felhom-pbs` does not exist yet: that storage appears only # when the hub provisions the DR tier. The agent DEFERS a tier whose target storage is absent @@ -2491,7 +2513,7 @@ base.setdefault('storage', {"watchdog_interval_seconds":5,"watchdog_debounce_sec # setdefault: an EXISTING box's backup block is preserved WHOLE. Upgrades never gain the tier here — # they are migrated explicitly (R-82 Slice D.2), so an in-place upgrade can never silently start # writing to an offsite datastore. -base.setdefault('backup', {"local_backup_target":os.environ.get('BACKUP_TARGET_RESOLVED','local'),"local_backup_retention":3,"restore_storage":"local-lvm","restore_test_cadence_seconds":0,"scratch_vmid_min":990000,"scratch_vmid_max":990009,"pbs_secret_dir":"/etc/pve/priv/storage","backup_cadence_seconds":0,"backup_targets":[{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":2}]}) +base.setdefault('backup', {"local_backup_target":os.environ.get('BACKUP_TARGET_RESOLVED','local'),"local_backup_retention":3,"restore_storage":"local-lvm","restore_test_cadence_seconds":0,"scratch_vmid_min":990000,"scratch_vmid_max":990009,"pbs_secret_dir":"/etc/pve/priv/storage","backup_cadence_seconds":0,"backup_targets":[{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":0}]}) base.setdefault('local_api', {}) base['local_api'].setdefault('enable', True) base['local_api']['listen_addr'] = os.environ['BRIDGE_ADDR'] diff --git a/scripts/hostinstall_gates.py b/scripts/hostinstall_gates.py index 96c5ed6..cdbc3b4 100644 --- a/scripts/hostinstall_gates.py +++ b/scripts/hostinstall_gates.py @@ -233,6 +233,32 @@ else: "invisible to the agent, it is never restore-tested, and an empty listing looks exactly " "like a brand-new tier." % (resolutions, grants)) +# ── R-191: the OFFSITE tier must not arm a client-side prune ───────────────────────────────── +# +# R-89 moved offsite pruning SERVER-SIDE — ep0 runs a per-namespace prune job and box tokens stay +# write-only, so the box is REFUSED if it asks. When this default was `keep_last: 2` the effect was a +# weekly lie: vzdump uploaded the snapshot, then failed the whole job on the prune, and the operator +# was told the offsite backup had failed when it had succeeded. +# +# The assertion is on the OFFSITE entry only. The local tier's `local_backup_retention` is untouched +# and must stay untouched — it prunes correctly and is allowed to. +m = re.search(r'"backup_targets":\s*\[(.*?)\]', src, re.S) +if not m: + fail("cannot find backup_targets in the rendered agent.json defaults — the offsite-retention " + "assertion cannot run, and a check that cannot run must never report OK (R-191)") +else: + targets = m.group(1) + kl = re.search(r'"keep_last"\s*:\s*(\d+)', targets) + if not kl: + fail("the offsite backup_target carries no keep_last at all — expected an explicit 0 " + "(R-191: 0 means 'never prune from the box'; absent is not the same statement)") + elif kl.group(1) != "0": + fail("the offsite backup_target arms a CLIENT-SIDE prune (keep_last=%s). R-89 moved offsite " + "pruning server-side to ep0 and box tokens are write-only, so every weekly run will " + "upload successfully and then FAIL the job on a refused prune (R-191)." % kl.group(1)) + else: + ok("the offsite tier arms no client-side prune (keep_last=0; retention is ep0's prune jobs)") + print() if fails: print("hostinstall gates: %d FAILURE(S)" % len(fails))