R-1295: the PBS DR set-up waits out PBS's 5-s permission cache. Cause measured on a throwaway PBS 4.2.5 under load (v1.3.0 failed 11/30, v1.4.0 0/30): tenantsync v1.4.0 re-checks the same token (and the admin token) up to 15 s; hub v0.172.0 retries a self_check_failed rollback 3x, 7 s apart, and a final failure mails the operator (pbsdr_provision_failed, operator-only). Red-proved. Commit only; ep0 unchanged.

documentation/audits/r1295-pbsdr-selfcheck-2026-10-26/README.md

@@ -0,0 +1,44 @@
+# R-1295 — a fresh install's whole-guest off-site tier failed to set up, silently (diagnosis + fix, 2026-10-26)
+
+## What happened (2026-10-26, Tester4's fresh install, host `Tester4-eb7371`)
+
+Hub 10:31:15 CET: `[ERROR] pbsdr auto-provision for Tester4 (WG-registration hook): pbsdr: F-14 auto re-issue for
+Tester4: tenantsync: endpoint refused: own-namespace list as felhom@pbs!Tester4 failed twice — rolled back (code
+self_check_failed)`. No operator mail.
+
+ep0, read only (`journalctl`, root SSH from DooPlex): 09:31:14Z `felhom-tenantsync` (provision → token_exists), 09:31:15Z
+`felhom-tenantsync` (reissue), and in the same second two proxy lines `GET …/snapshots?ns=Tester4: 403 Forbidden …
+permission check failed - missing Datastore.Audit|Datastore.Backup on /datastore/felhom-offsite/Tester4`. Since
+2026-09-01 the only 403s of this kind on ep0: Tester4 2026-10-25 16:52:15Z (one — the script's retry then passed) and
+these two. After the rollback Tester4 has no token and no ACL on ep0.
+
+## Candidates, each ruled in or out with evidence
+
+| Candidate | Verdict | Evidence |
+|---|---|---|
+| a version mismatch (new CLI, old running proxy) | OUT | ep0 runs AND has installed 4.2.5 (dpkg/apt history 2026-08-18; the „4.2.8" in `versions` is the repository's newest) |
+| the proxy mapped a stale shared-memory version file | OUT | `/run/proxmox-backup/shmem/config-versions` inode 11 = the inode both proxy and api map |
+| the capital letter in `Tester4` | OUT | 40 + 20 runs on a throwaway PBS with `Tester4`: 0 failures without load |
+| a namespace/token left by the deleted host, or the ACL layout | OUT | ep0's ACL layout copied into the throwaway PBS (felhom@pbs DatastoreAdmin on the datastore, a reader token): 0 failures without load |
+| **the proxy's permission cache under concurrent requests** | **IN** | throwaway PBS 4.2.5 + a second client listing every 50 ms: v1.3.0 failed 9 of 20 first checks, 1 rolled back; the stale window measured 4.2–5.0 s (`MEASURE`, 25 runs); the old retry comes within 1 s |
+
+**The cause:** PBS's proxy caches permissions for up to 5 s, and an ACL-only change does not clear that cache. A request
+from another client between the token create and its grant leaves „new token, old ACLs" for up to ~5 s. The script's
+one quick retry falls inside the same window, so it rolls the token back. The same cache also hits the script's own
+transient ADMIN token: under load its first namespace list was refused and a wrong `namespace create` then died on
+„missing Datastore.Modify".
+
+## The fix
+
+- `scripts/felhom-tenantsync.sh` v1.4.0: the SAME token is re-checked every 0.5 s for up to 15 s (a log line per try),
+  then the old regenerate-once-and-rollback path; the admin token waits the same way before its first namespace list.
+  Test `scripts/test_tenantsync_selfcheck.py` (19 checks; v1.3.0 fails 11). **Live on the throwaway PBS 4.2.5 under the
+  same load, the hub's exact order (provision → reissue), 30 pairs: v1.3.0 failed 11, v1.4.0 failed 0.**
+  Installing it on ep0 is an ep0 change — the operator's yes.
+- hub v0.172.0: a `self_check_failed` rollback is retried (3 attempts in all, 7 s apart, a log line per try) in both the
+  provision and the F-14 re-issue paths; a final failure of the WG-registration hook raises the operator-only event
+  `pbsdr_provision_failed` (stored + dispatched; subject „… the whole-box off-site copy (PBS DR) could not be set up —
+  save the customer config to retry"). Red-proof `redproof-hub.txt`. With the hub alone (ep0 unchanged) each attempt has
+  the measured ~1-in-20 chance under load, so 3 attempts ≈ 1 in 8000; and a failure is never silent again.
+
+Throwaway: LXC 9374 on demo-hp (nvme-scratch), PBS 4.2.5 from the Proxmox repository; destroyed at the end of this run.

documentation/audits/r1295-pbsdr-selfcheck-2026-10-26/redproof-hub.txt

@@ -0,0 +1,10 @@
+# R-1295 hub red-proof (2026-10-26). The code reverted (no tenancyWithRetry, no pbsdrProvisionFailed call, no
+# operator-only entry, no subject case, no typed self_check_failed), the tests kept:
+--- FAIL: TestR1295_SelfCheckFailedIsTyped — got "tenantsync: endpoint refused: … (code self_check_failed)" (untyped)
+--- FAIL: TestR1295_ARolledBackSelfCheckIsRetriedAndTheTierIsSetUp — descriptor <nil>
+--- FAIL: TestR1295_TheF14ReissuePathIsRetriedToo — descriptor <nil>
+--- FAIL: TestR1295_AFinalFailureReachesTheOperator — provision calls = 1, want 3; no pbsdr_provision_failed event
+--- FAIL: TestR1295_TheMail — pbsdr_provision_failed must be operator-only
+--- FAIL: TestR1295_TheFailedSetUpMailsTheOperatorOnly (notify)
+(TestR1295_OtherFailuresAreNotRetried passes both ways — a guard.)
+# Restored: ok tenantsync, ok web, ok notify.