# SPIKE β€” PBS DR tier auto-provisioning, SLICE 0: the three load-bearing mechanisms **Date:** 2026-07-10 Β· **Class:** spike (validate before the epic's implementation slices) Β· **Executor:** Claude Code live on the demo host (felhom-pve) + the build-server PBS (180) + **ep0 (felhom-hetzner, Viktor-approved πŸ›‘ touch)**. **No production code shipped; every probe artifact was `spike-*`-named and torn down** (post-teardown asserts in Β§6). The epic: "PBS DR" on the hub customer page β†’ WG peer + per-customer PBS tenancy on ep0 β†’ the agent applies the storage entry host-side β†’ K exists β†’ the standard ceremony covers it. --- ## 1. VERDICT | Probe | Question | Verdict | Settled mechanism | |---|---|---|---| | **1** | Can a path-scoped PVE token create the storage entry? | **NO β€” impossible.** Create/modify/delete ALL check `Datastore.Allocate` on the **`/storage` root**; path-scoped ACLs govern only *using* a storage. | **Sudoers vector** (1b), storage-id **pinned** in the alias β€” full add/set/acl/remove cycle proven as `felhom-agent` via `sudo -n`; unpinned id denied. | | **2** | Can tenancy ops ride the peersync channel? | **Not today** (forced command + single-alias sudoers block everything else β€” live-proven), **but the op-set + one-time-secret-over-SSH-stdout transport is PROVEN on ep0** with a throwaway tenant, incl. read/write cross-tenant 403s. | **Slice 1 ships a second surface** (`felhom-tenantsync`: own keypair + own forced-command line + own sudoers entry β€” peersync untouched, one-script-one-job preserved). | | **3** | Does `--encryption-key autogen` give us K at apply time? | **YES.** Entry active, **K born at `/etc/pve/priv/storage/.enc`** (0600 root:www-data), vzdump lands as **ciphertext** on the PBS side (`crypt-mode: encrypt`), `escrow-create` would find it. | `pvesm add pbs … --encryption-key autogen` (via the Probe-1b sudo wrapper), fingerprint from Probe 2's retrieval. | The epic is now mechanical. Slice map in Β§5. ## 2. Probe 1 β€” PVE permission model (demo host) - Fixture: `FelhomAgentStore = Datastore.Allocate,Datastore.AllocateSpace`; agent token `felhom-agent@pve!agent` is `privsep=1` β†’ dual-grant (user AND token) required, as everywhere. - ACL on the **not-yet-existing** path `/storage/spike-pbs` is allowed (paths are logical) β€” granted to both user and token. - **CREATE as the token** (`POST /api2/json/storage`, type dir, minimal) β†’ **`403 Permission check failed (/storage, Datastore.Allocate)`** β€” the check runs against the `/storage` ROOT, verbatim. Granting that root-wide = the agent could create/delete ANY storage β†’ unacceptable. - **MODIFY (`PUT /storage/spike-pbs`) and DELETE as the token** against a root-created entry β†’ **the SAME 403 on `/storage`**. The whole entry-lifecycle CRUD ignores path-scoped grants; `/storage/` ACLs matter only for content/usage ops (AllocateSpace, backup, audit). ### 2b. The sudoers vector (the fallback pattern β€” PROVEN) Temp drop-in (`visudo -cf` OK), storage-id **PINNED**, run as the `felhom-agent` user via `sudo -n`: ``` Cmnd_Alias FELHOM_SPIKE_STORE = \ /usr/sbin/pvesm add pbs spike-pbs *, \ /usr/sbin/pvesm add dir spike-pbs *, \ /usr/sbin/pvesm set spike-pbs *, \ /usr/sbin/pvesm remove spike-pbs, \ /usr/sbin/pveum acl modify /storage/spike-pbs * felhom-agent ALL=(root) NOPASSWD: FELHOM_SPIKE_STORE ``` - add β†’ set β†’ `pveum acl modify /storage/spike-pbs` (the self-grant) β†’ remove: **all OK** as the agent user. - **Containment held:** `sudo -n pvesm remove felhom-usb` (an unpinned real id) β†’ denied (`a password is required`). In-agent fine validation per the existing sudoers philosophy (the drop-in is the coarse gate). - **⚠ sudo logs full argv to auth.log** β†’ the PBS token secret must **NEVER** ride `sudo pvesm add … --password `. Epic shape: a root **wrapper script** (installed like the existing units/install pattern, sudoers-pinned by name) that reads the secret on **STDIN** and calls `pvesm` internally β€” the sudo log then shows only the wrapper invocation. (Matches the Β§4b runbook precedent that already avoided password-on-argv, and the agent's `runCommandStdin` seam.) ## 3. Probe 2 β€” ep0 tenancy ops + the one-time-secret channel (πŸ›‘ approved) **Recon (read-only, recorded):** ep0 = `felhom-hetzner` (167.233.158.164), peersync script v1.0.1. `felhom-peersync`'s authorized_keys = ONE line, `restrict,command="sudo /usr/local/bin/felhom-peersync"`; sudoers = exactly that script; **live-proven the user can sudo nothing else** (`sudo -n proxmox-backup-manager user list` β†’ password required). So the current channel cannot carry tenancy ops β€” by design. **The op-set, proven with throwaway `spike-tenant` on the real `felhom-offsite` datastore** (Β§4a dance, spike-named; run as root = what the future sudo-pinned tenantsync script is): 1. throwaway `root@pam!spikeadm` admin token + `DatastoreAdmin` on the datastore β†’ `proxmox-backup-client namespace create spike-tenant` (namespace ops are client-side; `PBS_FINGERPRINT` required even on localhost) 2. `proxmox-backup-manager user generate-token felhom@pbs spike-tenant` β€” **the secret returned over the SSH session stdout (36 bytes observed by the caller, never persisted on ep0, never displayed)** β€” the one-time-secret transport the hub will consume-once 3. dual-grant `DatastoreBackup` on `/datastore/felhom-offsite/spike-tenant` to **both** `felhom@pbs` and `felhom@pbs!spike-tenant` 4. **S4 isolation asserts (all green):** own-ns list OK Β· `demo-felhom-01` list β†’ **403** Β· own-ns **write** (real `proxmox-backup-client backup`) OK Β· cross-tenant **write** β†’ **403** 5. fingerprint retrieval (the descriptor field): `proxmox-backup-manager cert info | awk '/Fingerprint/{print $3}'` 6. teardown: `namespace delete spike-tenant --delete-groups true`, delete both tokens, delete all spike ACLs **Ordering/behavior gotchas (live-hit):** - **Token must exist BEFORE its ACL grant** β€” `acl update --auth-id '…!tok'` on a not-yet-existing token fails `no such API token` (PBS validates the auth-id; PVE does not β€” Probe 1 granted an unborn path fine). - **`delete-token` purges the token's ACLs** (PVE-like) β€” a regen is delete + generate + **re-grant**. - `generate-token` has **no `--output-format`** β€” parse its default JSON block (`sed -n 's/.*"value": "…'`). There is no regenerate subcommand on PBS 4.2.2. - One transient: the first grantβ†’assert pass 403'd on the OWN namespace despite both ACLs listing correctly; a delete+regen+re-grant converged and the identical sequence then passed grantβ†’assert ~1s apart. Cause not identified. **Operational note for slice 1: post-provision, probe own-ns list as the new token; on 403, regen once.** - `proxmox-backup-client` is a **separate package** (the runbook says so; 180 didn't have it β€” installed). **The slice-1 ep0 surface (recommendation):** a second keypair (`hub-tenancy`) + second authorized_keys line with forced command `sudo /usr/local/bin/felhom-tenantsync` + its own single-line sudoers entry. JSON-on-stdin contract like peersync (`{"op":"provision|deprovision","ns":""}`, ns validated `^[a-z0-9-]{1,32}$` and never a reserved name), response JSON on stdout carrying `{token_secret, fingerprint, datastore, ns}`. The script holds the throwaway admin secret in memory only (no `/root/.spikeadm.raw` even transiently). Peersync itself stays untouched (one script, one job). ## 4. Probe 3 β€” K autogen + fingerprint at apply time (demo host) Target: the LAN PBS (192.168.0.180, datastore `felhom-spike`, spike-tenant ns created + torn down there too β€” ep0 stayed out of this probe). ``` pvesm add pbs spike-pbs --server 192.168.0.180 --datastore felhom-spike --namespace spike-tenant \ --username 'felhom@pbs!spike-tenant' --password --fingerprint --encryption-key autogen ``` - Entry **active** immediately (`pvesm status`). - **K born at `/etc/pve/priv/storage/spike-pbs.enc`** β€” 0600 root:www-data, 255 bytes, same shape as the real `felhom-pbs.enc`/`felhom-offsite.enc`; storage.cfg gets `encryption-key `; the token secret lands as `spike-pbs.pw` alongside. - **vzdump-class probe:** guest 9001 (`spike-lxc`, stop-mode, 26s) β†’ PBS side sees `pct.conf.blob encrypt Β· root.pxar.didx encrypt Β· catalog.pcat1.didx encrypt Β· index.json.blob sign-only Β· client.log.blob none` β€” **the doc-06 property: ciphertext client-side, K never left the box.** (Manifest sign-only + plaintext client log are standard PBS semantics.) - **`escrow-create --storage spike-pbs` would find K:** `runSelftestEscrowCreate` stats `cfg.Backup.PBSEncKeyPath(storage)`; demo's `pbs_secret_dir` = `/etc/pve/priv/storage` (the default) β†’ the exact autogen path. **⚠ On boxes that override `pbs_secret_dir` (the Β§4b WARN-fix copy dir), only `.pw` is copied today β€” slice 2 must also place/copy `.enc` there or escrow-create misses it.** - **⚠ `pvesm remove ` DELETES `.enc` AND `.pw`** β€” removing the entry **destroys K**. The epic's re-apply path must be `pvesm set`-only, NEVER remove+re-add after a ceremony (the ApplyOffsiteTarget EscrowState lesson, storage-entry edition). The remove vector stays in the sudoers alias only for the explicit decommission flow, gated on escrow/DR policy. ## 5. The epic's slice map (recommend, don't build) - **Slice 1 (hub):** customer-config "PBS DR tier" enable β†’ WG peer assign (existing allocator) + tenancy provision over the NEW `felhom-tenantsync` surface (Β§3) β†’ token secret stored **consume-once, HOST-scoped** (the agent fetches it with its host api_key β€” the controller consume-once precedent, host-side twin) β†’ non-secret coords into desired-state: `pbs_tunnel_ip` (10.77.0.1), `datastore` (felhom-offsite), `namespace` (= host_id), `fingerprint` (from the tenantsync response). - **Slice 2 (agent):** the apply-bridge β€” on the desired-state pbs block: ensure wg-felhom up β†’ verify the PBS fingerprint β†’ consume the token β†’ **root wrapper (sudoers-pinned, secret on stdin, Β§2b)** runs `pvesm add pbs felhom-pbs … --encryption-key autogen` β†’ self-grant `/storage/felhom-pbs` via the pinned `pveum acl modify` (dual-grant: user + token) β†’ `pvesm status` reachability probe β†’ capability report clears `wg-handshake-read` degradation. Idempotent re-apply = `pvesm set` only (never remove β€” K, Β§4). Marker + fail-safe per the offsite bridge precedent; consume-once + argv/log-hygiene red-proofs mandatory. **Naming note:** the epic's id `felhom-pbs` already exists on the DEMO host (the LAN dev PBS) β€” fresh customer boxes (Peti) are clean; a demo migration needs a rename/coexistence decision first. - **Slice 3:** DR schedule + recipe coords + monitoring wiring. Much exists: the agent's PBS verify loop + tier-aware unattended restore-test (S4.1) already run against pbs-type storages on demo; hub-side needs the event wiring only (the restic `OffsiteChecker` is offbox-specific). - **Sequencing:** after slice 2 lands on Peti (floor + agent update): his box gains `felhom-pbs` + K β†’ **the standard ceremony one-liner** seals K + identity under one fresh R β†’ auto-confirm β†’ arc closed. ## 6. Teardown + environment observations **Teardown verified:** demo β€” spike storage entry, `.enc`/`.pw`, both spike ACLs, sudoers drop-in, `/tmp` artifacts, staged secret all gone (`pveum acl list` + storage.cfg grep clean; the surviving `spike` grep hit is `datastore felhom-spike`, the real entry's datastore name). 180 β€” spike namespace+snapshots, token, ACLs, staged secrets gone; only `felhom@pbs!n100` remains. ep0 β€” spike namespace+snapshots, both tokens, all spike ACLs, `/tmp/spikewrite` gone; only the real `demo-felhom-01` tenancy remains. Demo's real `felhom-offsite` + `felhom-pbs` storages re-verified **active** post-spike. **Flags for Viktor (pre-existing, not touched):** 1. **ep0 has an orphan-looking `root@pam!spike` token + `DatastoreAdmin` ACL on `/datastore/scratch`** β€” from the S4-era spikes; cleanup candidate. 2. **The demo host has NO `/storage/felhom-pbs` or `/storage/felhom-offsite` FelhomAgentStore grants today** (only `local` + `local-lvm`) β€” the Β§4b step-3 grants are absent, most likely the token-remove-purges-ACL gotcha. Backups demonstrably still work (the verify loop is green), so either the grants are not actually load-bearing for the current vzdump path or something re-asserts lazily β€” worth a deliberate re-assert + a check of which ops actually need them before slice 2 codifies the self-grant. 3. The LAN PBS (180) real token `felhom@pbs!n100` holds **datastore-wide `DatastoreAdmin`** β€” dev-grade, much broader than ep0's ns-scoped `DatastoreBackup` model; fine for dev, don't copy the pattern.