R-1295 — a fresh install's whole-guest off-site tier failed to set up, silently (diagnosis + fix, 2026-10-26)
What happened (2026-10-26, Tester4's fresh install, host Tester4-eb7371)
Hub 10:31:15 CET: [ERROR] pbsdr auto-provision for Tester4 (WG-registration hook): pbsdr: F-14 auto re-issue for Tester4: tenantsync: endpoint refused: own-namespace list as felhom@pbs!Tester4 failed twice — rolled back (code self_check_failed). No operator mail.
ep0, read only (journalctl, root SSH from DooPlex): 09:31:14Z felhom-tenantsync (provision → token_exists), 09:31:15Z
felhom-tenantsync (reissue), and in the same second two proxy lines GET …/snapshots?ns=Tester4: 403 Forbidden … permission check failed - missing Datastore.Audit|Datastore.Backup on /datastore/felhom-offsite/Tester4. Since
2026-09-01 the only 403s of this kind on ep0: Tester4 2026-10-25 16:52:15Z (one — the script's retry then passed) and
these two. After the rollback Tester4 has no token and no ACL on ep0.
Candidates, each ruled in or out with evidence
| Candidate | Verdict | Evidence |
|---|---|---|
| a version mismatch (new CLI, old running proxy) | OUT | ep0 runs AND has installed 4.2.5 (dpkg/apt history 2026-08-18; the „4.2.8" in versions is the repository's newest) |
| the proxy mapped a stale shared-memory version file | OUT | /run/proxmox-backup/shmem/config-versions inode 11 = the inode both proxy and api map |
the capital letter in Tester4 |
OUT | 40 + 20 runs on a throwaway PBS with Tester4: 0 failures without load |
| a namespace/token left by the deleted host, or the ACL layout | OUT | ep0's ACL layout copied into the throwaway PBS (felhom@pbs DatastoreAdmin on the datastore, a reader token): 0 failures without load |
| the proxy's permission cache under concurrent requests | IN | throwaway PBS 4.2.5 + a second client listing every 50 ms: v1.3.0 failed 9 of 20 first checks, 1 rolled back; the stale window measured 4.2–5.0 s (MEASURE, 25 runs); the old retry comes within 1 s |
The cause: PBS's proxy caches permissions for up to 5 s, and an ACL-only change does not clear that cache. A request
from another client between the token create and its grant leaves „new token, old ACLs" for up to ~5 s. The script's
one quick retry falls inside the same window, so it rolls the token back. The same cache also hits the script's own
transient ADMIN token: under load its first namespace list was refused and a wrong namespace create then died on
„missing Datastore.Modify".
The fix
scripts/felhom-tenantsync.shv1.4.0: the SAME token is re-checked every 0.5 s for up to 15 s (a log line per try), then the old regenerate-once-and-rollback path; the admin token waits the same way before its first namespace list. Testscripts/test_tenantsync_selfcheck.py(19 checks; v1.3.0 fails 11). Live on the throwaway PBS 4.2.5 under the same load, the hub's exact order (provision → reissue), 30 pairs: v1.3.0 failed 11, v1.4.0 failed 0. Installing it on ep0 is an ep0 change — the operator's yes.- hub v0.172.0: a
self_check_failedrollback is retried (3 attempts in all, 7 s apart, a log line per try) in both the provision and the F-14 re-issue paths; a final failure of the WG-registration hook raises the operator-only eventpbsdr_provision_failed(stored + dispatched; subject „… the whole-box off-site copy (PBS DR) could not be set up — save the customer config to retry"). Red-proofredproof-hub.txt. With the hub alone (ep0 unchanged) each attempt has the measured ~1-in-20 chance under load, so 3 attempts ≈ 1 in 8000; and a failure is never silent again.
Throwaway: LXC 9374 on demo-hp (nvme-scratch), PBS 4.2.5 from the Proxmox repository; destroyed at the end of this run.