docs(offsite): S4 SHIPPED — per-customer PBS tenancy + v4-pin/watchdog (live 2026-07-04)
- runbook offsite-endpoint.md: new §4a (endpoint per-customer tenancy) + §4b (box-side storage + agent wiring). Confirmed minimal ACL: DatastoreBackup on /datastore/<ds>/<ns> (NOT /ns/<ns>) to BOTH user felhom@pbs AND token felhom@pbs!<ns> (PBS privsep = intersection); cross-tenant 403; ns-scoped verify works with DatastoreBackup; DatastoreBackup can't prune (safety). Box: box-born encryption key, argv-safe .pw/.enc + storage.cfg, agent PVE FelhomAgentStore grant, pbs-secrets WARN fix, local_backup_target retarget. - doc-06 §3.4 SHIPPED (ns-aware PBS client v0.67.0 rationale); §4.2 v4-pin + watchdog SHIPPED v0.66.0 (closes the dual-stack trap); S4 roadmap row done + follow-ups (restore-test WAN deadline + scratch VM.Allocate; Tier-1/Tier-2 target split). Live: real vzdump of 9201 → ciphertext in ns over the tunnel; verify=ok under the box's own scoped token; WARN gone; restore decrypts with the box-born key. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
@@ -132,6 +132,102 @@ datastore parent is `/srv` (world-traversable — a datastore under `/root` fail
|
||||
`Permission denied` for the `backup` user). GUI: reachable ONLY via SSH port-forward
|
||||
(`ssh -L 8007:127.0.0.1:8007 root@<box>`) or through the tunnel — by design, no public 8007.
|
||||
|
||||
## 4a. Per-customer PBS tenancy (S4 — live-confirmed 2026-07-04 on felhom-hetzner)
|
||||
|
||||
Each customer box gets ITS OWN namespace + a privilege-separated token scoped to that namespace,
|
||||
on the single shared `felhom-offsite` datastore (doc 06 D5). One shared PBS user `felhom@pbs`
|
||||
holds a per-customer token each; cross-tenant isolation is enforced by the per-token ACL.
|
||||
|
||||
**On the endpoint, per customer `<ns>` (e.g. `demo-felhom-01` = the box's `host_id`):**
|
||||
```sh
|
||||
DS=felhom-offsite
|
||||
# 1. namespace — created via a THROWAWAY root admin token (root@pam has no password on the cloud
|
||||
# image; proxmox-backup-manager has NO `namespace` subcommand — namespaces are client-side):
|
||||
proxmox-backup-manager user generate-token root@pam nsadmin > /root/.nsadmin.raw # umask 077
|
||||
NSADMIN=$(sed -n 's/.*"value": "\([^"]*\)".*/\1/p' /root/.nsadmin.raw); rm -f /root/.nsadmin.raw
|
||||
proxmox-backup-manager acl update /datastore/$DS DatastoreAdmin --auth-id 'root@pam!nsadmin'
|
||||
export PBS_FINGERPRINT="$(proxmox-backup-manager cert info | awk '/Fingerprint/{print $3}')"
|
||||
PBS_PASSWORD="$NSADMIN" proxmox-backup-client namespace create <ns> --repository "root@pam!nsadmin@localhost:$DS"
|
||||
# 2. the shared user (once) + this customer's privilege-separated token (secret → root-only file):
|
||||
proxmox-backup-manager user create felhom@pbs --comment 'offsite tenancy' 2>/dev/null || true
|
||||
proxmox-backup-manager user generate-token felhom@pbs <ns> > /root/.tok-<ns>.raw
|
||||
sed -n 's/.*"value": "\([^"]*\)".*/\1/p' /root/.tok-<ns>.raw > /root/.tok-<ns>.secret # 0600; rm the .raw
|
||||
# 3. THE ACL — the load-bearing gotcha (see below): DatastoreBackup on the NAMESPACE PATH, to BOTH
|
||||
# the user AND the token:
|
||||
proxmox-backup-manager acl update /datastore/$DS/<ns> DatastoreBackup --auth-id 'felhom@pbs'
|
||||
proxmox-backup-manager acl update /datastore/$DS/<ns> DatastoreBackup --auth-id 'felhom@pbs!<ns>'
|
||||
# 4. clean up the throwaway admin token:
|
||||
proxmox-backup-manager user delete-token root@pam nsadmin
|
||||
proxmox-backup-manager acl update /datastore/$DS DatastoreAdmin --auth-id 'root@pam!nsadmin' --delete
|
||||
```
|
||||
Hand `/root/.tok-<ns>.secret` + the cert fingerprint to the box out-of-band (they become the PVE
|
||||
storage `.pw` + `fingerprint` — §4b).
|
||||
|
||||
**The confirmed minimal ACL + the gotchas (all live-proven, don't re-derive):**
|
||||
- **ACL path is `/datastore/<ds>/<ns>` — NOT `/datastore/<ds>/ns/<ns>`.** The `/ns/` form is the
|
||||
*client repository* path; the *ACL object* path drops it. Granting `/ns/<ns>` silently no-ops
|
||||
(the permission check runs against `/datastore/<ds>/<ns>`).
|
||||
- **PBS privsep tokens = intersection(user perms, token perms).** The token's own ACL alone is
|
||||
NOT enough — the **user** `felhom@pbs` must ALSO be granted on the namespace, or the token has
|
||||
the empty intersection and every op 403s (`missing Datastore.Backup`). Isolation is preserved
|
||||
anyway: token B's effective perms = user(has A's + B's ns) ∩ token-B(only B's ns) = B's ns only.
|
||||
Adversarially confirmed: token A → list/backup ns B ⇒ **403**.
|
||||
- **`DatastoreBackup` is the whole grant** — it covers backup, list-own, restore-own, AND
|
||||
**namespace-scoped verify** (`POST /verify` with `ns=<ns>` — no `Datastore.Verify`/admin needed).
|
||||
It deliberately does **NOT** cover prune/forget (`Datastore.Modify|Prune`) — the box **cannot
|
||||
delete its own backups**; pruning is an operator/endpoint job (matches "the runner never prunes
|
||||
PBS"). No token ever exceeds `DatastoreBackup`; no admin token lives on the endpoint for a box.
|
||||
|
||||
## 4b. Box-side storage + agent wiring (S4 — on the customer box, as root)
|
||||
|
||||
```sh
|
||||
FP=<endpoint cert fingerprint>; UN='felhom@pbs!<ns>'; PWSTORE=/etc/pve/priv/storage
|
||||
# 1. encryption key — BORN ON THE BOX (never leaves as plaintext; the endpoint only ever sees
|
||||
# ciphertext). Its fingerprint goes in storage.cfg:
|
||||
proxmox-backup-client key create $PWSTORE/felhom-offsite.enc --kdf none
|
||||
ENCFP=$(sed -n 's/.*"fingerprint":"\([^"]*\)".*/\1/p' $PWSTORE/felhom-offsite.enc)
|
||||
chown root:www-data $PWSTORE/felhom-offsite.enc && chmod 600 $PWSTORE/felhom-offsite.enc
|
||||
# 2. token secret → PVE .pw store — ARGV-SAFE (pvesm add only takes --password on argv AND it
|
||||
# validates the connection, so a dummy won't do; write the .pw file + storage.cfg directly):
|
||||
install -o root -g www-data -m 600 /root/.tok-<ns>.secret $PWSTORE/felhom-offsite.pw
|
||||
cat >> /etc/pve/storage.cfg <<CFG
|
||||
|
||||
pbs: felhom-offsite
|
||||
datastore felhom-offsite
|
||||
server 10.77.0.1
|
||||
content backup
|
||||
fingerprint $FP
|
||||
namespace <ns>
|
||||
encryption-key $ENCFP
|
||||
username $UN
|
||||
CFG
|
||||
pvesm status --storage felhom-offsite # expect: active (PVE connects over the tunnel)
|
||||
# 3. agent PVE token needs the new storage (dual-grant, PVE privsep like PBS):
|
||||
pveum acl modify /storage/felhom-offsite --roles FelhomAgentStore --users 'felhom-agent@pve'
|
||||
pveum acl modify /storage/felhom-offsite --roles FelhomAgentStore --tokens 'felhom-agent@pve!agent'
|
||||
# 4. THE WARN FIX — the non-root agent cannot read /etc/pve/priv (root 0700); give it an owned copy:
|
||||
install -d -o felhom-agent -g felhom-agent -m 0700 /var/lib/felhom-agent/pbs-secrets
|
||||
for ID in felhom-pbs felhom-offsite; do
|
||||
install -o felhom-agent -g felhom-agent -m 0600 /etc/pve/priv/storage/$ID.pw /var/lib/felhom-agent/pbs-secrets/$ID.pw
|
||||
done
|
||||
# then in agent.json backup{}: "pbs_secret_dir":"/var/lib/felhom-agent/pbs-secrets"
|
||||
# 5. RETARGET the box's backups offsite — the field is backup.local_backup_target (BackupTarget()):
|
||||
# agent.json backup{}: "local_backup_target":"felhom-offsite" (defaultBackupTarget stays felhom-pbs in code)
|
||||
systemctl restart felhom-agent # verify: journal has ZERO "cannot read token secret"; verify loop
|
||||
# logs `verify cycle complete datastore=felhom-offsite`
|
||||
```
|
||||
Verified live 2026-07-04: a real vzdump of guest 9201 (crash-consistent, ~4.6 GB compressed over
|
||||
the tunnel, ~20 min) lands as **ciphertext** in ns `demo-felhom-01` (`root.pxar` crypt-mode
|
||||
`encrypt`); the ns-scoped verify runs under the box's own `felhom@pbs!demo-felhom-01` token and
|
||||
returns `verify=ok`; the WARN is gone; a restore reads it back decrypting with the on-box key.
|
||||
|
||||
**Provisioning notes / follow-ups (S4):** (a) the `.pw` copy in `pbs-secrets` is a STATIC snapshot
|
||||
of the PVE token secret — a token rotation must re-copy it (a sync step for the hub-driven-tenancy
|
||||
slice). (b) The restore-test scheduler's internal restore-task deadline (~10 min) is too short for
|
||||
a WAN restore of a large volume, and the agent PVE token lacks `VM.Allocate` on the scratch VMID
|
||||
band for teardown — both are box-side config fixes needed before the offsite restore-test can run
|
||||
unattended (raise the restore deadline for the `pbs` tier; grant the scratch-band VM.Allocate).
|
||||
|
||||
## 5. Peersync surface
|
||||
|
||||
```sh
|
||||
|
||||
Reference in New Issue
Block a user