installer 1.25.0: the off-site tier stops asking to prune (R-191)
gates / gates (push) Successful in 8s
gates / gates (push) Successful in 8s
Every weekly off-site run uploaded successfully and then failed the job on a prune the box's token is deliberately refused — R-89 moved off-site pruning server-side to ep0 and box tokens stay write-only. The 2026-07-26 'two weeks' ruling was not reversed; where it is enforced moved, and keep_last: 2 did not follow. Now 0, which the agent's existing guard already reads as 'never prune from the box'. Verified read-only on ep0 before changing it: both namespaces have a prune job at 03:30 keep-last 2 that has run every day since 2026-07-27 — 18 tasks, all OK, the newest keeping exactly two. Without that check this would have traded a weekly false alarm for unbounded growth. A gate asserts the offsite tier carries no client-side prune. The local tier is untouched.
This commit is contained in:
@@ -1,3 +1,26 @@
|
|||||||
|
## 1.25.0 — the off-site tier stops asking to prune (2026-08-04, R-191)
|
||||||
|
|
||||||
|
**A backup that worked must not report failure.** The off-site tier was written with `keep_last: 2`,
|
||||||
|
so every weekly run uploaded its snapshot successfully and then failed the whole job on a prune the
|
||||||
|
box's token is deliberately refused: `prune 'ct/9201': permission check failed - missing
|
||||||
|
Datastore.Modify|Datastore.Prune` → `TASK ERROR: job errors` → `whole_guest_backup_failed` in the
|
||||||
|
operator's inbox. Every week, on both boxes, about a backup that had already succeeded.
|
||||||
|
|
||||||
|
**R-89 moved off-site pruning SERVER-SIDE** — ep0 runs a per-namespace prune job and box tokens stay
|
||||||
|
write-only, so a box can never delete its own off-site history. The 2026-07-26 "two weeks" ruling was
|
||||||
|
not reversed; where it is ENFORCED moved, and this value did not follow. The tier now writes
|
||||||
|
`keep_last: 0`, which the agent's existing guard (`allowPBSPrune = !primary && keep_last > 0`) already
|
||||||
|
reads as "never prune from the box" — no agent change needed.
|
||||||
|
|
||||||
|
**VERIFIED BEFORE CHANGING IT** (read-only on ep0, 2026-08-04): prune jobs `prune-demo-felhom` and
|
||||||
|
`prune-demo-hp` exist on datastore `felhom-offsite`, one per namespace, schedule 03:30, keep-last 2,
|
||||||
|
and have run **every day since 2026-07-27 — 18 tasks, all `status=OK`**, the newest showing
|
||||||
|
`retention options: --ns demo-felhom --max-depth 0 --keep-last 2` and keeping exactly two. Disabling
|
||||||
|
the client-side prune without that check would have traded a weekly false alarm for unbounded growth.
|
||||||
|
|
||||||
|
A gate now asserts the off-site tier carries no client-side prune, so the value cannot drift back
|
||||||
|
quietly. The local tier's retention is untouched.
|
||||||
|
|
||||||
## 1.24.0 — a pre-existing backup target is granted too (2026-08-03, R-185)
|
## 1.24.0 — a pre-existing backup target is granted too (2026-08-03, R-185)
|
||||||
|
|
||||||
**`configure_backup_target` has two arms and only one of them granted.** The Case A arm creates the
|
**`configure_backup_target` has two arms and only one of them granted.** The Case A arm creates the
|
||||||
|
|||||||
@@ -184,7 +184,7 @@
|
|||||||
|
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
SCRIPT_VERSION="1.24.0" # the SINGLE version source (F-1): -h and the run banners follow it.
|
SCRIPT_VERSION="1.25.0" # the SINGLE version source (F-1): -h and the run banners follow it.
|
||||||
# The hub used to carry a copy for its Setup tab; R-94 DELETED it
|
# The hub used to carry a copy for its Setup tab; R-94 DELETED it
|
||||||
# (2026-08-02) because the hub cannot know which version a box runs —
|
# (2026-08-02) because the hub cannot know which version a box runs —
|
||||||
# the Setup command fetches this script at run time. scripts/
|
# the Setup command fetches this script at run time. scripts/
|
||||||
@@ -2479,8 +2479,30 @@ for _k,_v in {"unit_dir":"/etc/systemd/system","stage_dir":"/var/lib/felhom-agen
|
|||||||
base.setdefault('storage', {"watchdog_interval_seconds":5,"watchdog_debounce_seconds":15,"known_refresh_seconds":20})
|
base.setdefault('storage', {"watchdog_interval_seconds":5,"watchdog_debounce_seconds":15,"known_refresh_seconds":20})
|
||||||
# R-82: local DAILY + offsite WEEKLY. The two tiers carry SEPARATE cadences and retentions —
|
# R-82: local DAILY + offsite WEEKLY. The two tiers carry SEPARATE cadences and retentions —
|
||||||
# keep_last=3 is three DAYS on the daily tier and three WEEKS on a weekly one, so one shared knob
|
# keep_last=3 is three DAYS on the daily tier and three WEEKS on a weekly one, so one shared knob
|
||||||
# would guarantee that one of them is wrong. keep_last=2 on the offsite tier = two weeks (operator
|
# would guarantee that one of them is wrong.
|
||||||
# ruling 2026-07-26).
|
#
|
||||||
|
# THE OFFSITE TIER CARRIES NO CLIENT-SIDE RETENTION, AND THAT IS THE POINT (R-191, 2026-08-04).
|
||||||
|
# It used to be written `keep_last: 2` ("two weeks", operator ruling 2026-07-26). **R-89 then moved
|
||||||
|
# offsite pruning SERVER-SIDE** — ep0 runs a per-namespace prune job and box tokens stay write-only,
|
||||||
|
# deliberately, so that a box can never delete its own offsite history. The 2026-07-26 ruling was not
|
||||||
|
# reversed; where it is ENFORCED moved, and this value did not follow.
|
||||||
|
#
|
||||||
|
# The consequence was weekly and wrong in the worst direction: vzdump UPLOADED the snapshot fine and
|
||||||
|
# then failed the whole job on the prune the token is refused —
|
||||||
|
# `prune 'ct/9201': permission check failed - missing Datastore.Modify|Datastore.Prune`
|
||||||
|
# → `TASK ERROR: job errors` → `whole_guest_backup_failed` in the operator's inbox. Every week, on
|
||||||
|
# both boxes, about a backup that had already succeeded. A tier that cries wolf weekly is a tier
|
||||||
|
# whose real failure nobody will see.
|
||||||
|
#
|
||||||
|
# VERIFIED BEFORE CHANGING IT (2026-08-04, read-only on ep0): prune jobs `prune-demo-felhom` and
|
||||||
|
# `prune-demo-hp` exist on datastore `felhom-offsite`, one per namespace, schedule 03:30, keep-last 2,
|
||||||
|
# and have run EVERY DAY since 2026-07-27 — 18 tasks, all `status=OK`, the newest showing
|
||||||
|
# `retention options: --ns demo-felhom --max-depth 0 --keep-last 2` and keeping exactly two. Retention
|
||||||
|
# happens; it happens THERE. **If that ever stops being true, this zero is unbounded growth** — check
|
||||||
|
# ep0's prune jobs before assuming the offsite tier is retained.
|
||||||
|
#
|
||||||
|
# `keep_last: 0` means "never prune from the box" and is the value the agent's own guard reads
|
||||||
|
# (allowPBSPrune = !primary && keep_last > 0), so no agent change is needed to honour it.
|
||||||
#
|
#
|
||||||
# The offsite tier is written even though `felhom-pbs` does not exist yet: that storage appears only
|
# The offsite tier is written even though `felhom-pbs` does not exist yet: that storage appears only
|
||||||
# when the hub provisions the DR tier. The agent DEFERS a tier whose target storage is absent
|
# when the hub provisions the DR tier. The agent DEFERS a tier whose target storage is absent
|
||||||
@@ -2491,7 +2513,7 @@ base.setdefault('storage', {"watchdog_interval_seconds":5,"watchdog_debounce_sec
|
|||||||
# setdefault: an EXISTING box's backup block is preserved WHOLE. Upgrades never gain the tier here —
|
# setdefault: an EXISTING box's backup block is preserved WHOLE. Upgrades never gain the tier here —
|
||||||
# they are migrated explicitly (R-82 Slice D.2), so an in-place upgrade can never silently start
|
# they are migrated explicitly (R-82 Slice D.2), so an in-place upgrade can never silently start
|
||||||
# writing to an offsite datastore.
|
# writing to an offsite datastore.
|
||||||
base.setdefault('backup', {"local_backup_target":os.environ.get('BACKUP_TARGET_RESOLVED','local'),"local_backup_retention":3,"restore_storage":"local-lvm","restore_test_cadence_seconds":0,"scratch_vmid_min":990000,"scratch_vmid_max":990009,"pbs_secret_dir":"/etc/pve/priv/storage","backup_cadence_seconds":0,"backup_targets":[{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":2}]})
|
base.setdefault('backup', {"local_backup_target":os.environ.get('BACKUP_TARGET_RESOLVED','local'),"local_backup_retention":3,"restore_storage":"local-lvm","restore_test_cadence_seconds":0,"scratch_vmid_min":990000,"scratch_vmid_max":990009,"pbs_secret_dir":"/etc/pve/priv/storage","backup_cadence_seconds":0,"backup_targets":[{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":0}]})
|
||||||
base.setdefault('local_api', {})
|
base.setdefault('local_api', {})
|
||||||
base['local_api'].setdefault('enable', True)
|
base['local_api'].setdefault('enable', True)
|
||||||
base['local_api']['listen_addr'] = os.environ['BRIDGE_ADDR']
|
base['local_api']['listen_addr'] = os.environ['BRIDGE_ADDR']
|
||||||
|
|||||||
@@ -233,6 +233,32 @@ else:
|
|||||||
"invisible to the agent, it is never restore-tested, and an empty listing looks exactly "
|
"invisible to the agent, it is never restore-tested, and an empty listing looks exactly "
|
||||||
"like a brand-new tier." % (resolutions, grants))
|
"like a brand-new tier." % (resolutions, grants))
|
||||||
|
|
||||||
|
# ── R-191: the OFFSITE tier must not arm a client-side prune ─────────────────────────────────
|
||||||
|
#
|
||||||
|
# R-89 moved offsite pruning SERVER-SIDE — ep0 runs a per-namespace prune job and box tokens stay
|
||||||
|
# write-only, so the box is REFUSED if it asks. When this default was `keep_last: 2` the effect was a
|
||||||
|
# weekly lie: vzdump uploaded the snapshot, then failed the whole job on the prune, and the operator
|
||||||
|
# was told the offsite backup had failed when it had succeeded.
|
||||||
|
#
|
||||||
|
# The assertion is on the OFFSITE entry only. The local tier's `local_backup_retention` is untouched
|
||||||
|
# and must stay untouched — it prunes correctly and is allowed to.
|
||||||
|
m = re.search(r'"backup_targets":\s*\[(.*?)\]', src, re.S)
|
||||||
|
if not m:
|
||||||
|
fail("cannot find backup_targets in the rendered agent.json defaults — the offsite-retention "
|
||||||
|
"assertion cannot run, and a check that cannot run must never report OK (R-191)")
|
||||||
|
else:
|
||||||
|
targets = m.group(1)
|
||||||
|
kl = re.search(r'"keep_last"\s*:\s*(\d+)', targets)
|
||||||
|
if not kl:
|
||||||
|
fail("the offsite backup_target carries no keep_last at all — expected an explicit 0 "
|
||||||
|
"(R-191: 0 means 'never prune from the box'; absent is not the same statement)")
|
||||||
|
elif kl.group(1) != "0":
|
||||||
|
fail("the offsite backup_target arms a CLIENT-SIDE prune (keep_last=%s). R-89 moved offsite "
|
||||||
|
"pruning server-side to ep0 and box tokens are write-only, so every weekly run will "
|
||||||
|
"upload successfully and then FAIL the job on a refused prune (R-191)." % kl.group(1))
|
||||||
|
else:
|
||||||
|
ok("the offsite tier arms no client-side prune (keep_last=0; retention is ep0's prune jobs)")
|
||||||
|
|
||||||
print()
|
print()
|
||||||
if fails:
|
if fails:
|
||||||
print("hostinstall gates: %d FAILURE(S)" % len(fails))
|
print("hostinstall gates: %d FAILURE(S)" % len(fails))
|
||||||
|
|||||||
Reference in New Issue
Block a user