45 lines
3.7 KiB
Markdown
45 lines
3.7 KiB
Markdown
# R-1295 — a fresh install's whole-guest off-site tier failed to set up, silently (diagnosis + fix, 2026-10-26)
|
||
|
||
## What happened (2026-10-26, Tester4's fresh install, host `Tester4-eb7371`)
|
||
|
||
Hub 10:31:15 CET: `[ERROR] pbsdr auto-provision for Tester4 (WG-registration hook): pbsdr: F-14 auto re-issue for
|
||
Tester4: tenantsync: endpoint refused: own-namespace list as felhom@pbs!Tester4 failed twice — rolled back (code
|
||
self_check_failed)`. No operator mail.
|
||
|
||
ep0, read only (`journalctl`, root SSH from DooPlex): 09:31:14Z `felhom-tenantsync` (provision → token_exists), 09:31:15Z
|
||
`felhom-tenantsync` (reissue), and in the same second two proxy lines `GET …/snapshots?ns=Tester4: 403 Forbidden …
|
||
permission check failed - missing Datastore.Audit|Datastore.Backup on /datastore/felhom-offsite/Tester4`. Since
|
||
2026-09-01 the only 403s of this kind on ep0: Tester4 2026-10-25 16:52:15Z (one — the script's retry then passed) and
|
||
these two. After the rollback Tester4 has no token and no ACL on ep0.
|
||
|
||
## Candidates, each ruled in or out with evidence
|
||
|
||
| Candidate | Verdict | Evidence |
|
||
|---|---|---|
|
||
| a version mismatch (new CLI, old running proxy) | OUT | ep0 runs AND has installed 4.2.5 (dpkg/apt history 2026-08-18; the „4.2.8" in `versions` is the repository's newest) |
|
||
| the proxy mapped a stale shared-memory version file | OUT | `/run/proxmox-backup/shmem/config-versions` inode 11 = the inode both proxy and api map |
|
||
| the capital letter in `Tester4` | OUT | 40 + 20 runs on a throwaway PBS with `Tester4`: 0 failures without load |
|
||
| a namespace/token left by the deleted host, or the ACL layout | OUT | ep0's ACL layout copied into the throwaway PBS (felhom@pbs DatastoreAdmin on the datastore, a reader token): 0 failures without load |
|
||
| **the proxy's permission cache under concurrent requests** | **IN** | throwaway PBS 4.2.5 + a second client listing every 50 ms: v1.3.0 failed 9 of 20 first checks, 1 rolled back; the stale window measured 4.2–5.0 s (`MEASURE`, 25 runs); the old retry comes within 1 s |
|
||
|
||
**The cause:** PBS's proxy caches permissions for up to 5 s, and an ACL-only change does not clear that cache. A request
|
||
from another client between the token create and its grant leaves „new token, old ACLs" for up to ~5 s. The script's
|
||
one quick retry falls inside the same window, so it rolls the token back. The same cache also hits the script's own
|
||
transient ADMIN token: under load its first namespace list was refused and a wrong `namespace create` then died on
|
||
„missing Datastore.Modify".
|
||
|
||
## The fix
|
||
|
||
- `scripts/felhom-tenantsync.sh` v1.4.0: the SAME token is re-checked every 0.5 s for up to 15 s (a log line per try),
|
||
then the old regenerate-once-and-rollback path; the admin token waits the same way before its first namespace list.
|
||
Test `scripts/test_tenantsync_selfcheck.py` (19 checks; v1.3.0 fails 11). **Live on the throwaway PBS 4.2.5 under the
|
||
same load, the hub's exact order (provision → reissue), 30 pairs: v1.3.0 failed 11, v1.4.0 failed 0.**
|
||
Installing it on ep0 is an ep0 change — the operator's yes.
|
||
- hub v0.172.0: a `self_check_failed` rollback is retried (3 attempts in all, 7 s apart, a log line per try) in both the
|
||
provision and the F-14 re-issue paths; a final failure of the WG-registration hook raises the operator-only event
|
||
`pbsdr_provision_failed` (stored + dispatched; subject „… the whole-box off-site copy (PBS DR) could not be set up —
|
||
save the customer config to retry"). Red-proof `redproof-hub.txt`. With the hub alone (ep0 unchanged) each attempt has
|
||
the measured ~1-in-20 chance under load, so 3 attempts ≈ 1 in 8000; and a failure is never silent again.
|
||
|
||
Throwaway: LXC 9374 on demo-hp (nvme-scratch), PBS 4.2.5 from the Proxmox repository; destroyed at the end of this run.
|