From e06920d930cc57f63bd83f7d3fd272be81bf4e04 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 4 Aug 2026 10:08:10 +0200 Subject: [PATCH] R-188: the reorder moved the race rather than removing it (observed on the v0.124.0 release) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI run 28 failed and run 29 passed on the same sha. The tag was created locally at 07:38:36Z; run 28 started 07:38:31Z (commit push) and run 29 at 07:38:41Z (tag push). The old race was 'tag before package'; the new one is 'package before its tag is pushed' — the window the converse probe exists to detect, and it did. Not confirmed from the run log (the endpoint 404s), so recorded as the timing-consistent explanation rather than a proven one. The gate passes locally with all 7 versions installable and none untagged. Both obvious closures are the 'make the gate blind for a window' shape this row refused, so the options are to accept a seconds-wide window or to make releases quiet to CI another way. --- documentation/backlog/OPEN-ITEMS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 6b08f38..eaf3248 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -31,7 +31,7 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-185** | **The agent cannot see the host backup tier's archives on demo-felhom — the PVE token has no ACL on `/storage/felhom-backup`, so the content listing returns EMPTY where root sees three archives.** Found 2026-08-03 while live-validating R-86. `pveum acl list` grants `FelhomAgentStore` on `/storage/{local,local-lvm,felhom-pbs}` and **not** on `felhom-backup`, which is the box's actual `local_backup_target`. Verified three ways: `pvesh` as root lists 3 archives (6.1–6.3 GB, 08-01/02/03); the same endpoint with the agent's token returns `{"data":[]}`; and `local` — which HAS a grant — returns its archives through the same token | **CLOSED — SHIPPED + PROVEN-LIVE 2026-08-03** (agent **v0.123.0**, installer **1.24.0**) | — | **Pre-existing and independent of R-86** (it is a property of the ACL, and the R-85 rotation had the same blindness). **Consequences:** the host tier has never been restore-testable on that box, and R-85's *"an empty tier is skipped, not failed"* rule made that silent. **The part worth fixing is the silence, not only the grant:** a permission-blinded tier is today INDISTINGUISHABLE from a newborn one — both report *"no settled archive yet"* — which is this project's own absence-is-not-evidence rule failing in a new place. The agent already knows better: it RECORDS successful backups to that target, so *"I wrote archives here and the tier lists none"* is a contradiction it can detect and should say loudly. **Do not fix by widening the token blind:** decide whether the host-install ACL set should follow `local_backup_target` (it currently hardcodes `local`), which is where the drift began **BOTH HALVES FIXED, AND THE ROOT CAUSE WAS NOT WHERE IT LOOKED.** **(1) The silence.** `Client.Permissions` asks `GET /access/permissions?path=/storage/` **as the agent's own token**, and one `capability.Status` per configured tier reports it — composed around the sudo prober the way the pool-read check already is, with the `Status` wire shape untouched so the hub's existing critical-degraded alert applies with **no hub change**. **MEASURED BEFORE IT WAS DESIGNED AGAINST, and the obvious reading is wrong:** an ungranted path answers neither empty nor 403 but with the privileges INHERITED from the box-wide `/` grant — `felhom-pbs` → `{Datastore.Allocate:1, Datastore.AllocateSpace:1}` vs `felhom-backup` → `{Sys.Audit:1, SDN.Use:1, Datastore.Audit:1}`. A probe asking *"is the path present?"* or *"does it have Datastore.Audit?"* would report the blinded storage HEALTHY; the probe tests `Datastore.AllocateSpace` and a red-proof pins that. The probed set comes from the box's own `BackupTiers()`, never a fixed list — a hardcoded probe list is the defect reproduced inside the fix. Critical (the hub alerts only on critical), **except** the `local` fallback target, which is reported but does not page: turning host-install's own documented DEGRADED configuration into an alert is how a signal becomes something an operator archives unread. It never looks at content, so it cannot alarm on a newborn tier by construction; it never reports ok when it could not ask. **(2) The installer.** The row and the task both assumed `PVE_STORAGES` was the culprit. It is not: `configure_backup_target`'s **Case A** arm creates the storage and grants in the same breath, so a box that builds its own target was always fine. The **Scenario-F** arm — *"the target already exists, leave it exactly as it is"* — returned **without granting**, which is why a box whose `felhom-backup` pre-dated the install (the vzdump-target-move runbook, or a reinstall) pointed `local_backup_target` at a storage its token could not read. That arm now ensures the ACL through the same guarded wrapper; the storage DEFINITION is still untouched, so Scenario F is unviolated, and `pveum acl modify` is idempotent. `$BACKUP_TARGET_ID` is deliberately still NOT in `PVE_STORAGES` — that list is granted in step 4/5, before the target is resolved in step 6, and `--acl-storages` entries are preflight-checked for existence. A **gate** now asserts every arm that resolves the target also grants on it (red-proved: *resolves in 2 place(s) but grants in only 1*). **LIVE, in order:** the three-way observation reproduced unchanged (root 3 archives / token `{"data":[]}` / `local` lists through the same token) → agent v0.123.0 deployed → **`capability DEGRADED … pve:store-grant:felhom-backup … lacks Datastore.AllocateSpace … INVISIBLE to the agent and it is never restore-tested`, critical=true**, and the hub raised `agent_capability_degraded` **and emailed the operator** → grant applied (user AND token) → the token lists **3 archives** → `ok=70 total=70 degraded=0` and the hub logged `degraded → ok (agent_capability_recovered)` → **the host tier is a due-check candidate for the first time on that box**, correctly picking the 08-02 archive because the 08-03 one has not settled 24 h. **demo-hp CARRIES THE SAME DRIFT AND WAS FIXED TOO** — `local_backup_target=felhom-backup`, no ACL row, now granted and listing 4 archives. §8.6 assumed one box was affected; the same one-line additive idempotent fix applies to the other, and leaving a known-blind backup tier on a Tier-0 box after finding it would be this row happening twice. The tester's box was NOT touched (Tier 2) — see the note in this row's outcome. **CORRECTION + WIDER IMPACT, measured 2026-08-03 during the R-185 close-out run.** This row (and the runbook annotation written with it) said the drift was invisible because writes went through a root path. **That is wrong.** demo-felhom's local-api backup jobs 403'd **six times** between 09:24 and 17:34 CEST — `POST …/vzdump -> HTTP 403: permission denied at /storage/felhom-backup (missing privilege Datastore.Allocate)` — and the hub raised `whole_guest_backup_failed` at the first (*"retrying with backoff"*), edge-triggering suppressing the rest. So the impact was not only an unreadable tier: **the agent's own whole-guest backups to it were failing**, and the operator was told once. The failures stop at the grant. **AND THE GRANT WAS LOST, NOT MERELY NEVER ISSUED — on this box.** A vzdump by `felhom-agent@pve!agent` with `--storage felhom-backup` **succeeded at 04:44:50 CEST the same day**, and the first 403 is 09:24:56. Ruled out: a host reinstall (uptime 12 days), any logged `pveum`/ACL activity (none in syslog 04:00–10:00, none in the cluster log). Correlated but NOT established as causal: `host_leaf_changed` 09:15 and `controller_started` 09:19, i.e. guest 9201 was reprovisioned nine minutes before the first failure. **How the grant disappeared is unexplained → R-190.** | CC | | **R-186** | **A released agent binary's sha256 cannot be reproduced from its tag.** `release-agent.sh` builds at step 3 and tags at step 4, so Go's VCS stamp records a PSEUDO-version (`v0.120.1-0.20260803130452-4d825910…`) in the published bytes, while any rebuild after the tag exists stamps `v0.121.0` — a different binary. Measured 2026-08-03 on v0.121.0: published `b2128f3c…` (14 081 336 B) vs rebuild-at-tag `8302e396…` (14 077 240 B), identical source, identical toolchain, 4 096 bytes apart | **CLOSED — SHIPPED + MEASURED 2026-08-03** (agent **v0.122.0**) | — | **Why it matters:** the sha the operator vouches is the one thing tying a machine to a binary, and today nobody can independently rebuild it to check. **The build order is deliberate** (the script's own comment: a tag with no package is caught by `check-published-versions.py`, a package with no tag is invisible to it), so the fix is not to swap the steps blind. Candidates: `-buildvcs=false` or `-trimpath` for a version-stable stamp, or tag-then-build with the tag deleted on a failed publish. **Mitigation used this session:** the DEPLOYED binary is the PUBLISHED artifact, downloaded from Gitea — not a local rebuild — so the running bytes are the vouchable ones **FIXED by removing the cause, not by sequencing around it.** `-buildvcs=false` drops the VCS stamp (nothing in the repo reads it — no `ReadBuildInfo` caller) and `-trimpath` removes absolute build paths, so the same source produces the same bytes whether or not the tag exists yet and regardless of checkout directory. **Measured before the change**, same source, same toolchain, one commit: default flags with no tag `18f4a495…` 14 085 464 B (mod `v0.121.2-0.2026…`) vs default flags tagged `4a38f394…` 14 085 440 B (mod `v0.121.99`) — different bytes; with the new flags, `7ffcdf1d…` 14 064 574 B **identical both ways**. **Measured after the change on the real release:** published `d5f294e5…` 14 076 649 B = rebuild-at-tag #1 = rebuild-at-tag #2, **all three identical**. **A second discrepancy fell out of it:** `publish-agent.sh`'s fallback build forced `CGO_ENABLED=0` and therefore produced a binary **74 KB smaller** (13 990 236 B vs 14 064 574 B) than the release path built for the same version — one version name, two binaries, decided by which entry point ran. Both paths now build identically, with a comment in each naming the other. **The operator can now check a published sha without being there:** the exact command is in `felhom-agent/CLAUDE.md`. | CC | | **R-187** | **R-115's one-command release had never actually run its publish leg — the first real use died there.** `scripts/publish-agent.sh` has been mode `0644` since it was created (2026-06-28), because every earlier caller invoked it as `bash scripts/publish-agent.sh`; `release-agent.sh` (written 2026-08-03) called it directly and got `Permission denied` on v0.121.0's release | **CLOSED — SHIPPED 2026-08-03** (`felhom-agent`) | — | **Fixed both ways in one commit:** the executable bit restored, and the caller changed to `bash "$REPO_ROOT/scripts/publish-agent.sh"` so the release no longer depends on a file mode — the kind of thing a checkout, an archive or a copy silently loses again. **The lesson is R-115's own, one level up:** the mechanism written to make a step unforgettable was itself never exercised end-to-end, so it failed the first time it mattered. A mechanism that has not been RUN is a note with better formatting | CC | -| **R-188** | **Every agent release has a ~50 % chance of emailing the operator a CI failure for a release that is correct.** `on: [push]` fires the gates workflow on the **tag** push too, and `release-agent.sh` pushes the tag BEFORE publishing the package (deliberately — a tag with no package is caught by `check-published-versions.py`, a package with no tag is invisible to it). So CI can run the published-versions gate inside the seconds-long window where `v` exists and the package does not, and it correctly reports *"every released agent version must be INSTALLABLE"* — for a release that completes normally a moment later | **CLOSED — SHIPPED 2026-08-03** (agent **v0.122.0**) | — | **MEASURED, 2026-08-03, across two releases in one session:** v0.121.0 → runs #12 (id 79) **success** and #13 (id 81) **failure** on the same sha `4618169`; v0.121.1 → #17 (id 88) **failure** and #18 (id 89) **success** on the same sha `cd1b087`; and v0.121.0's second pair (#14/#15, `4d82591`) both **success** — i.e. it is a race, not a rule. Local gate afterwards: all three versions verify. **Why it matters more than it looks:** R-168 made CI email the operator on failure precisely so a red gate cannot be missed. A signal that cries wolf on every second correct release is how that mail becomes something you archive unread — the R-29 failure mode one layer up, and R-100's corollary in a new place (an alarm whose text is true but whose trigger is not the thing you would act on). **Fix shapes, cheapest first:** have the workflow skip tag refs (`if: !startsWith(github.ref, 'refs/tags/')` — the commit push already covers the content, and the tag adds no new code); or make `check-published-versions.py` tolerate a tag younger than ~2 minutes; or publish before tagging, which the release script's own comment argues against. **Do not "fix" it by quieting the alarm** **FIXED by moving ONLY the tag push** (build → tag locally → publish → push tag). The tag is still created before anything is published, so the build and the tag still describe the same commit; it becomes VISIBLE — to CI, and to any `raw/tag/…` fetch — only once the package is downloadable. **The invariant the old order protected was not traded away:** `check-published-versions.py` now asserts the converse directly — *no published version may be missing its tag* — as a BOUNDED probe of the frontier (where a failed tag push leaves an orphan) and of patch gaps, printing its probe set on every run because a check whose coverage is invisible reads as a guarantee it is not making. The package listing api still answers **401** without a token (re-measured 2026-08-03), so absence still cannot be enumerated and the script says so. **Both directions demonstrated:** a real published-but-untagged fixture (0.121.2, published then deleted — HTTP 201 then 204, absence re-verified by a 404) made the gate FAIL and print the one-line recovery; removing the converse block made it pass over that same orphan. A tag with no package still fails, shown against a local stand-in rather than by pushing a probe tag — pushing one would have emailed the operator a TRUE alarm about a fixture, which is the same attention cost this row removes. **A half-done release is now loud:** publish-succeeded-then-push-failed dies naming the exact recovery command, and a FAILED publish removes the local-only tag so the release can simply be retried instead of colliding with step 2's re-release guard. **Scenario E, live:** v0.122.0's own release produced CI runs **21 (task id 96)** and **22 (task id 97)** on the same sha — **both green**, where the same shape produced one red run on each of the two previous releases. | CC | +| **R-188** | **Every agent release has a ~50 % chance of emailing the operator a CI failure for a release that is correct.** `on: [push]` fires the gates workflow on the **tag** push too, and `release-agent.sh` pushes the tag BEFORE publishing the package (deliberately — a tag with no package is caught by `check-published-versions.py`, a package with no tag is invisible to it). So CI can run the published-versions gate inside the seconds-long window where `v` exists and the package does not, and it correctly reports *"every released agent version must be INSTALLABLE"* — for a release that completes normally a moment later | **CLOSED — SHIPPED 2026-08-03** (agent **v0.122.0**) | — | **MEASURED, 2026-08-03, across two releases in one session:** v0.121.0 → runs #12 (id 79) **success** and #13 (id 81) **failure** on the same sha `4618169`; v0.121.1 → #17 (id 88) **failure** and #18 (id 89) **success** on the same sha `cd1b087`; and v0.121.0's second pair (#14/#15, `4d82591`) both **success** — i.e. it is a race, not a rule. Local gate afterwards: all three versions verify. **Why it matters more than it looks:** R-168 made CI email the operator on failure precisely so a red gate cannot be missed. A signal that cries wolf on every second correct release is how that mail becomes something you archive unread — the R-29 failure mode one layer up, and R-100's corollary in a new place (an alarm whose text is true but whose trigger is not the thing you would act on). **Fix shapes, cheapest first:** have the workflow skip tag refs (`if: !startsWith(github.ref, 'refs/tags/')` — the commit push already covers the content, and the tag adds no new code); or make `check-published-versions.py` tolerate a tag younger than ~2 minutes; or publish before tagging, which the release script's own comment argues against. **Do not "fix" it by quieting the alarm** **FIXED by moving ONLY the tag push** (build → tag locally → publish → push tag). The tag is still created before anything is published, so the build and the tag still describe the same commit; it becomes VISIBLE — to CI, and to any `raw/tag/…` fetch — only once the package is downloadable. **The invariant the old order protected was not traded away:** `check-published-versions.py` now asserts the converse directly — *no published version may be missing its tag* — as a BOUNDED probe of the frontier (where a failed tag push leaves an orphan) and of patch gaps, printing its probe set on every run because a check whose coverage is invisible reads as a guarantee it is not making. The package listing api still answers **401** without a token (re-measured 2026-08-03), so absence still cannot be enumerated and the script says so. **Both directions demonstrated:** a real published-but-untagged fixture (0.121.2, published then deleted — HTTP 201 then 204, absence re-verified by a 404) made the gate FAIL and print the one-line recovery; removing the converse block made it pass over that same orphan. A tag with no package still fails, shown against a local stand-in rather than by pushing a probe tag — pushing one would have emailed the operator a TRUE alarm about a fixture, which is the same attention cost this row removes. **A half-done release is now loud:** publish-succeeded-then-push-failed dies naming the exact recovery command, and a FAILED publish removes the local-only tag so the release can simply be retried instead of colliding with step 2's re-release guard. **Scenario E, live:** v0.122.0's own release produced CI runs **21 (task id 96)** and **22 (task id 97)** on the same sha — **both green**, where the same shape produced one red run on each of the two previous releases. **THE REORDER MOVED THE RACE RATHER THAN REMOVING IT — observed 2026-08-04, and it is narrower but real.** Releasing v0.124.0 produced CI run **28 (id 115) FAILED** and run **29 (id 116) SUCCESS** on the same sha `257c4d85`. Timing: the tag was created locally at **07:38:36Z**; run 28 started **07:38:31Z** (the COMMIT push, five seconds earlier) and run 29 at **07:38:41Z** (the tag push). The old race was *tag visible before its package*; the new one is *package published before its tag is pushed* — the exact window the converse probe was added to detect, and it detected it. **NOT CONFIRMED from the run log** (Gitea's run-log endpoint 404s for this runner), so this is the timing-consistent explanation rather than a proven one; the gate passes locally with all 7 released versions installable and none untagged. **What it means in practice:** a release can still produce one red run and one operator e-mail, now from the opposite side. The window is seconds rather than the previous coin-flip, and it is self-clearing. **If it is worth closing:** have the workflow skip runs whose ref is a branch push occurring inside a release, or have the converse probe tolerate a package younger than ~2 minutes — but note that BOTH are the 'make the gate blind for a window' shape this row explicitly refused, so the honest options are to accept the narrow window or to make releases quiet to CI another way. | CC | | **R-189** | **A passing restore-test can be invisible to the hub forever — and R-86 made that window a week instead of a day.** The agent reports `restore_tests[]` from `internal/backup.Store`, which is **in-memory** (its own comment: *"lost on restart; the cadence re-populates"*), and the hub's report interval is 900 s. An agent restart inside that window drops the result. That used to self-heal within 24 h because the next cadence re-tested the same tier; **under per-archive due-ness the agent will NOT re-test a proven archive**, so the hub can stay ignorant of a proof until the NEXT archive generation — a week on the offsite tier. **Observed live, 2026-08-03:** the 15:25:14 PASS on `felhom-pbs` (a real 14.5 GB offsite restore) reached **no** host-report — the hub logged `0 restore-tests` at 15:28:00 and 15:32:14 — because the agent was restarted at 15:27:57 for the v0.121.1 deploy, 2 m 43 s after the run and before the next report | **CLOSED — SHIPPED + PROVEN-LIVE 2026-08-03** (agent **v0.122.0**) | — | **The persisted proof already exists and is not reported.** `RestoreTestState` (on disk, per tier, R-86: with the archive id) survives exactly this, and `RestoreTestState.Snapshot()` carries the doc comment *"for the host-report gauge"* — **and has no production caller at all** (grep, 2026-08-03). That is this project's own *seam built but never wired* class, fifth instance, and a comment asserting an invariant with no test pinning it, both in one method. **Severity is bounded, and worth stating so it is not over-ranked:** the hub scans its retained window, archives keep arriving, and R-86's per-tier window for the offsite tier is 12 days against a 7-day archive rhythm — so ONE lost report is tolerated. What is not tolerable is that the loss is **silent** and that the box will not repeat the work. **Fix shape:** report the persisted per-tier proof (target, archive, proven_at) alongside the in-memory latest run, and let the hub's `lastProvenPerTier` read it — a report-contract change, so it touches `internal/hub/report.go` + `contract_test.go` + the hub's checker, and it is a spec, not a one-liner **FIXED.** `RestoreTestState` now stores the **tier** and what was **verified** beside the archive (recorded at proof time, from the run's own result — deriving them later would need a storage lookup that can fail on the one path where failing means mislabelling a proof), exposes `ProvenRestoreTests`, and `Collector.SetProvenRestoreTests` merges it with the in-memory result. **Merge rule: one entry per tier, newest by `TestedAt` wins** — a fresh failure beats a stored success (the failure is the news and lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier never appears twice (two entries would read at the hub as two tests). An unparseable timestamp counts as OLDER, so a malformed entry cannot displace a good one. **It refuses to lie:** a record missing the archive OR the tier produces NO entry, and run mechanics (scratch VMID, duration) are not re-invented — an absent duration is not a claim, a fabricated one would be. **The asymmetry is now written where it will be read:** only successes are persisted, because a success SUPPRESSES future work (a proven archive is never re-tested, so a lost proof leaves the box quietly less tested than it believes) while a failure CAUSES future work and heals itself at the next evaluation. **`Store`'s comment was corrected in place** — *"lost on restart; the cadence re-populates"* was true under a timer and became false the moment R-86 shipped; leaving it is how the next reader concludes this is handled. **The seam that had no caller is now pinned by an AST test** (`TestMainWiresTheDurableRestoreTestProof`), because `Snapshot()` claimed a *"host-report gauge"* in its doc comment for weeks with nothing calling it. **MIGRATION, stated because it is visible on the live box:** a pre-R-189 record carries an archive but no tier and is therefore NOT reportable — upgrading does not retroactively make an old proof visible to the hub; the tier's next real proof fills it in. Observed on demo-felhom immediately after the upgrade: still `0 restore-tests` until a fresh run. | CC | | **R-190** | **A storage ACL that demonstrably WORKED in the morning was gone by mid-morning, and nothing recorded its removal.** On demo-felhom, a `vzdump` by `felhom-agent@pve!agent` with `--storage felhom-backup` completed **OK at 04:44:50 CEST 2026-08-03** (task log read in full). From **09:24:56** the same path returned `HTTP 403 … missing privilege Datastore.Allocate at /storage/felhom-backup`, six times through the day, until the grant was re-applied by hand at 18:54. By ~14:50 `pveum acl list` showed **no row at all** for that path | **MITIGATION SHIPPED 2026-08-04** (agent **v0.124.0 → v0.124.1**) — **MECHANISM STILL OPEN** | — | **Why this is not just R-185 restated:** R-185's mechanism (the installer's Scenario-F arm resolves a pre-existing target without granting) explains a box that NEVER had the grant. This box HAD it and lost it, inside five hours, with the machine up throughout. **Ruled out, each by measurement:** a host reinstall (`uptime` = 12 days); any `pveum`/ACL/`user.cfg` activity in syslog between 04:00 and 10:00 (none); any ACL entry in `/cluster/log` (none). **Correlated, not established:** `host_leaf_changed` at 09:15 and `controller_started` at 09:19 — guest 9201 was reprovisioned nine minutes before the first 403. PVE removes ACLs at `/vms/` when a guest is destroyed (`AccessControl::remove_vm_access`, the F-LEAK mechanism); whether any path can take a `/storage/` row with it has NOT been established and is the first thing to check. **Why it matters more than the grant did:** a permission that can vanish silently makes every ACL-based guarantee on these hosts provisional, and the agent's new store-grant probe (v0.123.0) now detects the STATE but says nothing about the TRANSITION. **Worth pairing with:** whether the probe should report a grant it once had and no longer has as a distinct, louder signal than one it never had **THE ROW NOW REFLECTS THE MITIGATION, NOT THE CAUSE — stated plainly because the two are different things.** The box is resilient; the loss is still unexplained. **Mitigation:** when the store-grant probe finds the grant absent, the agent runs the EXISTING root wrapper (`felhom-backup-target-apply grant `) and **re-reads once** to confirm — the pbsdr R-22 self-grant shape, including its restraint. **No new privileged surface:** the sudoers vector `grant *` already covers any storage id (confirmed in `configs/felhom-agent.sudoers`, not assumed), and the verb already grants BOTH user and token. The verb existed, was permitted, and had only ever been called at storage CREATION — the *built but never wired* shape in a verb rather than a seam, this project's seventh instance. Bounded at one attempt per tier per hour (a storage can be unreadable for reasons an ACL cannot fix; re-granting every cycle is a repair loop wearing a fix's clothes). **THE RECORD IS THE HALF THIS ROW IS ABOUT, and v0.124.0 got it wrong in production while every unit test passed.** It reported degraded for *one cycle* — meaning the probe call that repaired. But `probeAll` is invoked INDEPENDENTLY by the self-check log and by the collector building a host-report: on the box the repairing call was the log's (`09:39:34`, journal shows the repair and `degraded=1`) and the report three seconds later found the grant present and sent **`ok`**. The agent's journal had the record, the hub had nothing, and the operator would have learned nothing — the exact silence this row exists for, re-created inside its own mitigation. **v0.124.1** replaces it with a latch on TIME (20 min > the 900 s report interval), so at least one report must carry it. **PROVEN LIVE, twice, on demo-felhom** (grant deleted by hand, both rows): agent logs `store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190) target=felhom-backup privilege=Datastore.AllocateSpace action="felhom-backup-target-apply grant felhom-backup" confirmed_by=re-read`; the ACL rows return; and on v0.124.1 the **host-report at 08:00:30Z carried `status=degraded`** with the explanation and the hub raised `agent_capability_degraded` and **e-mailed the operator** at 08:00:40. Nothing new was built to carry it — the hub's existing ok→degraded→ok edge is the channel, and the text rides `Feature` because that is the field the hub interpolates into the e-mail (`Reason` does not travel). **PART 3 — the single bounded pass at the mechanism, with the negatives named.** The lead §8.6 nominated is real as a CLASS and is documented in our own installer: *"`pveum user token remove` purges the token's ACL, so re-applying post-rotate is mandatory"*. **It does NOT fit this box.** A rotation purges ALL of the token's ACLs and mints a NEW secret; demo-felhom's token still authenticates with the same secret (`--selftest` OK), it retained its other three storage grants throughout, and only `felhom-backup` was refused. No installer run is evidenced (no 2026-08-03 install log; host uptime 12 days at the time). Previously ruled out and unchanged: a host reinstall, any `pveum`/ACL/`user.cfg` activity in syslog 04:00–10:00, any cluster-log ACL entry. **Ruled out on THIS box; NOT ruled out fleet-wide** — any installer run still purges and re-grants only the hardcoded `PVE_STORAGES` set, though installer 1.24.0's reuse-arm fix now re-grants the backup target on that path. **A NEW OBSERVATION FROM THE LIVE RUNS, relevant to the timeline:** PVE **caches permissions** — after deleting both ACL rows the probe still read the privilege as present for ~40 s in one run and ~16 min in another. Detection is only as prompt as that cache, and a cache expiry could equally explain why a box kept working for hours after a grant was removed. → **R-194**. | CC | | **R-191** | **Every weekly offsite backup UPLOADS successfully and then FAILS the job on a prune the box is deliberately not allowed to do — on both demo boxes.** Measured on demo-felhom 2026-08-04 06:49–06:53: the upload completed (223 s, 629 MiB of 1.874 GiB, **67.2 % reused incrementally**), then `ERROR: prune 'ct/9201': proxmox-backup-client failed: Error: permission check failed - missing Datastore.Modify\|Datastore.Prune on /datastore/felhom-offsite/demo-felhom` → `ERROR: Backup of VM 9201 failed - error pruning backups` → `TASK ERROR: job errors`. The hub raised `whole_guest_backup_failed` | **CLOSED — SHIPPED 2026-08-04** (installer **1.25.0**; both live boxes corrected) | — | **This is R-89's rule not reaching the config.** R-89 moved PBS pruning SERVER-SIDE — *"boxes set `keep_last: 0`, ep0 runs prune jobs; box tokens stay write-only, never widen the grant"*. The token behaves exactly as designed: it refuses. But **both** demo boxes still arm the offsite tier with `keep_last=2 prune_pbs_allowed=true` (`backup_targets: [{target_id: felhom-pbs, cadence_seconds: 604800, keep_last: 2}]`), so every run asks for a prune that must fail. **The data is SAFE and that is why this is not a P1:** the snapshot lands before the prune is attempted; what is wrong is the job's VERDICT and the weekly operator e-mail it produces. **But it is corrosive in the specific way this project keeps finding:** a backup that reports FAILED while succeeding trains the operator to discount `whole_guest_backup_failed`, which is the same alert that would carry a real one — and it is exactly the failure the R-100 corollary warns about, an alarm whose text is true and whose trigger is not the thing you would act on. **Fix is one config line per box** (`keep_last: 0` on the PBS tier) plus whatever writes it on a fresh install; **deliberately NOT applied in this session** — the session was a runbook with an explicit "change nothing, and if a change appears necessary, stop and report" rule, and a retention field on a live backup tier is not a change to slip into an observation run. **Check before fixing:** whether ep0's prune jobs actually cover these two namespaces, or the snapshots simply accumulate once the box stops asking **THE GATE WAS RUN FIRST, AND IT MATTERED.** Before disabling anything, ep0 was read (read-only, Tier 2): prune jobs `prune-demo-felhom` and `prune-demo-hp` exist on datastore `felhom-offsite`, one per namespace, `schedule 03:30`, `keep-last 2`, comment *"R-82 retention keep-last=2, server-side (box tokens are write-only)"* — and they have run **every day since 2026-07-27: 18 tasks, all `status=OK`**. The newest task log reads `retention options: --ns demo-felhom --max-depth 0 --keep-last 2` / `keep ct/9201/2026-07-27…` / `keep ct/9201/2026-07-28…` / `TASK OK`. Retention happens, and it happens there. **A METHODOLOGICAL WARNING WORTH MORE THAN THE FIX.** Three separate queries said the OPPOSITE — *no prune jobs have ever run* — and **all three were broken instruments**: `worker-type` where the field is `worker_type`; the value `prune` where the worker type is `prunejob`; and `journalctl -u proxmox-backup` where the unit is `proxmox-backup-proxy`. A fourth reading (3 snapshots under keep-last 2) was mis-framed by CC and self-corrected — the third snapshot had landed AFTER that day's 03:30 window. Acting on any of them would have disabled the only pruning ATTEMPT while reporting that nothing prunes: a weekly false alarm traded for unbounded growth on the protected endpoint, invisible for months. **The gate is what caught it, and only because it demanded evidence rather than a verdict.** **Shipped:** installer **1.25.0** writes `keep_last: 0` on the offsite tier (the agent's existing guard `allowPBSPrune = !primary && keep_last > 0` already reads that as *never prune from the box* — no agent change), the justifying paragraph is rewritten to say where retention lives and cite R-89, and `hostinstall_gates.py` asserts it (red-proved: pinning `keep_last: 2` back fails the gate). **Both live boxes corrected in their own config** — `backup tier armed target=felhom-pbs … keep_last=0 … prune_pbs_allowed=false` on demo-felhom and demo-hp, with the local tier untouched at `keep_last=3`. Served over HTTPS at `1.25.0` with `"keep_last":0` in the served bytes. **STILL TO OBSERVE:** the next weekly offsite run completing OK end-to-end. The change removes the failing step; the *schedule* proving it is next week's event, and this row should carry that line when it happens. | CC |