felhom.eu/documentation/audits/SPIKE-pbs-tier-provisioning-2026-07-10.md

SPIKE — PBS DR tier auto-provisioning, SLICE 0: the three load-bearing mechanisms

Date: 2026-07-10 · Class: spike (validate before the epic's implementation slices) · Executor: Claude Code live on the demo host (felhom-pve) + the build-server PBS (180) + ep0 (felhom-hetzner, Viktor-approved 🛑 touch). No production code shipped; every probe artifact was spike-*-named and torn down (post-teardown asserts in §6). The epic: "PBS DR" on the hub customer page → WG peer + per-customer PBS tenancy on ep0 → the agent applies the storage entry host-side → K exists → the standard ceremony covers it.

---

1. VERDICT

Probe	Question	Verdict	Settled mechanism
1	Can a path-scoped PVE token create the storage entry?	NO — impossible. Create/modify/delete ALL check Datastore.Allocate on the /storage root; path-scoped ACLs govern only using a storage.	Sudoers vector (1b), storage-id pinned in the alias — full add/set/acl/remove cycle proven as felhom-agent via sudo -n; unpinned id denied.
2	Can tenancy ops ride the peersync channel?	Not today (forced command + single-alias sudoers block everything else — live-proven), but the op-set + one-time-secret-over-SSH-stdout transport is PROVEN on ep0 with a throwaway tenant, incl. read/write cross-tenant 403s.	Slice 1 ships a second surface (felhom-tenantsync: own keypair + own forced-command line + own sudoers entry — peersync untouched, one-script-one-job preserved).
3	Does --encryption-key autogen give us K at apply time?	YES. Entry active, K born at /etc/pve/priv/storage/<id>.enc (0600 root:www-data), vzdump lands as ciphertext on the PBS side (crypt-mode: encrypt), escrow-create would find it.	pvesm add pbs <id> … --encryption-key autogen (via the Probe-1b sudo wrapper), fingerprint from Probe 2's retrieval.

The epic is now mechanical. Slice map in §5.

2. Probe 1 — PVE permission model (demo host)

• Fixture: FelhomAgentStore = Datastore.Allocate,Datastore.AllocateSpace; agent token felhom-agent@pve!agent is privsep=1 → dual-grant (user AND token) required, as everywhere.
• ACL on the not-yet-existing path /storage/spike-pbs is allowed (paths are logical) — granted to both user and token.
• CREATE as the token (POST /api2/json/storage, type dir, minimal) → 403 Permission check failed (/storage, Datastore.Allocate) — the check runs against the /storage ROOT, verbatim. Granting that root-wide = the agent could create/delete ANY storage → unacceptable.
• MODIFY (PUT /storage/spike-pbs) and DELETE as the token against a root-created entry → the SAME 403 on /storage. The whole entry-lifecycle CRUD ignores path-scoped grants; /storage/<id> ACLs matter only for content/usage ops (AllocateSpace, backup, audit).

2b. The sudoers vector (the fallback pattern — PROVEN)

Temp drop-in (visudo -cf OK), storage-id PINNED, run as the felhom-agent user via sudo -n:

Cmnd_Alias FELHOM_SPIKE_STORE = \
  /usr/sbin/pvesm add pbs spike-pbs *, \
  /usr/sbin/pvesm add dir spike-pbs *, \
  /usr/sbin/pvesm set spike-pbs *, \
  /usr/sbin/pvesm remove spike-pbs, \
  /usr/sbin/pveum acl modify /storage/spike-pbs *
felhom-agent ALL=(root) NOPASSWD: FELHOM_SPIKE_STORE

• add → set → pveum acl modify /storage/spike-pbs (the self-grant) → remove: all OK as the agent user.
• Containment held: sudo -n pvesm remove felhom-usb (an unpinned real id) → denied (a password is required). In-agent fine validation per the existing sudoers philosophy (the drop-in is the coarse gate).
• ⚠ sudo logs full argv to auth.log → the PBS token secret must NEVER ride sudo pvesm add … --password <secret>. Epic shape: a root wrapper script (installed like the existing units/install pattern, sudoers-pinned by name) that reads the secret on STDIN and calls pvesm internally — the sudo log then shows only the wrapper invocation. (Matches the §4b runbook precedent that already avoided password-on-argv, and the agent's runCommandStdin seam.)

3. Probe 2 — ep0 tenancy ops + the one-time-secret channel (🛑 approved)

Recon (read-only, recorded): ep0 = felhom-hetzner (167.233.158.164), peersync script v1.0.1. felhom-peersync's authorized_keys = ONE line, restrict,command="sudo /usr/local/bin/felhom-peersync"; sudoers = exactly that script; live-proven the user can sudo nothing else (sudo -n proxmox-backup-manager user list → password required). So the current channel cannot carry tenancy ops — by design.

The op-set, proven with throwaway spike-tenant on the real felhom-offsite datastore (§4a dance, spike-named; run as root = what the future sudo-pinned tenantsync script is):

1. throwaway root@pam!spikeadm admin token + DatastoreAdmin on the datastore → proxmox-backup-client namespace create spike-tenant (namespace ops are client-side; PBS_FINGERPRINT required even on localhost)
2. proxmox-backup-manager user generate-token felhom@pbs spike-tenant — the secret returned over the SSH session stdout (36 bytes observed by the caller, never persisted on ep0, never displayed) — the one-time-secret transport the hub will consume-once
3. dual-grant DatastoreBackup on /datastore/felhom-offsite/spike-tenant to both felhom@pbs and felhom@pbs!spike-tenant
4. S4 isolation asserts (all green): own-ns list OK · demo-felhom-01 list → 403 · own-ns write (real proxmox-backup-client backup) OK · cross-tenant write → 403
5. fingerprint retrieval (the descriptor field): proxmox-backup-manager cert info | awk '/Fingerprint/{print $3}'
6. teardown: namespace delete spike-tenant --delete-groups true, delete both tokens, delete all spike ACLs

Ordering/behavior gotchas (live-hit):

• Token must exist BEFORE its ACL grant — acl update --auth-id '…!tok' on a not-yet-existing token fails no such API token (PBS validates the auth-id; PVE does not — Probe 1 granted an unborn path fine).
• delete-token purges the token's ACLs (PVE-like) — a regen is delete + generate + re-grant.
• generate-token has no --output-format — parse its default JSON block (sed -n 's/.*"value": "…'). There is no regenerate subcommand on PBS 4.2.2.
• One transient: the first grant→assert pass 403'd on the OWN namespace despite both ACLs listing correctly; a delete+regen+re-grant converged and the identical sequence then passed grant→assert ~1s apart. Cause not identified. Operational note for slice 1: post-provision, probe own-ns list as the new token; on 403, regen once.
• proxmox-backup-client is a separate package (the runbook says so; 180 didn't have it — installed).

The slice-1 ep0 surface (recommendation): a second keypair (hub-tenancy) + second authorized_keys line with forced command sudo /usr/local/bin/felhom-tenantsync + its own single-line sudoers entry. JSON-on-stdin contract like peersync ({"op":"provision|deprovision","ns":"<host_id>"}, ns validated ^[a-z0-9-]{1,32}$ and never a reserved name), response JSON on stdout carrying {token_secret, fingerprint, datastore, ns}. The script holds the throwaway admin secret in memory only (no /root/.spikeadm.raw even transiently). Peersync itself stays untouched (one script, one job).

4. Probe 3 — K autogen + fingerprint at apply time (demo host)

Target: the LAN PBS (192.168.0.180, datastore felhom-spike, spike-tenant ns created + torn down there too — ep0 stayed out of this probe).

pvesm add pbs spike-pbs --server 192.168.0.180 --datastore felhom-spike --namespace spike-tenant \
  --username 'felhom@pbs!spike-tenant' --password <secret> --fingerprint <fp> --encryption-key autogen

• Entry active immediately (pvesm status).
• K born at /etc/pve/priv/storage/spike-pbs.enc — 0600 root:www-data, 255 bytes, same shape as the real felhom-pbs.enc/felhom-offsite.enc; storage.cfg gets encryption-key <K's own fingerprint>; the token secret lands as spike-pbs.pw alongside.
• vzdump-class probe: guest 9001 (spike-lxc, stop-mode, 26s) → PBS side sees pct.conf.blob encrypt · root.pxar.didx encrypt · catalog.pcat1.didx encrypt · index.json.blob sign-only · client.log.blob none — the doc-06 property: ciphertext client-side, K never left the box. (Manifest sign-only + plaintext client log are standard PBS semantics.)
• escrow-create --storage spike-pbs would find K: runSelftestEscrowCreate stats cfg.Backup.PBSEncKeyPath(storage); demo's pbs_secret_dir = /etc/pve/priv/storage (the default) → the exact autogen path. ⚠ On boxes that override pbs_secret_dir (the §4b WARN-fix copy dir), only .pw is copied today — slice 2 must also place/copy .enc there or escrow-create misses it.
• ⚠ pvesm remove <id> DELETES <id>.enc AND <id>.pw — removing the entry destroys K. The epic's re-apply path must be pvesm set-only, NEVER remove+re-add after a ceremony (the ApplyOffsiteTarget EscrowState lesson, storage-entry edition). The remove vector stays in the sudoers alias only for the explicit decommission flow, gated on escrow/DR policy.

5. The epic's slice map (recommend, don't build)

• Slice 1 (hub): customer-config "PBS DR tier" enable → WG peer assign (existing allocator) + tenancy provision over the NEW felhom-tenantsync surface (§3) → token secret stored consume-once, HOST-scoped (the agent fetches it with its host api_key — the controller consume-once precedent, host-side twin) → non-secret coords into desired-state: pbs_tunnel_ip (10.77.0.1), datastore (felhom-offsite), namespace (= host_id), fingerprint (from the tenantsync response).
• Slice 2 (agent): the apply-bridge — on the desired-state pbs block: ensure wg-felhom up → verify the PBS fingerprint → consume the token → root wrapper (sudoers-pinned, secret on stdin, §2b) runs pvesm add pbs felhom-pbs … --encryption-key autogen → self-grant /storage/felhom-pbs via the pinned pveum acl modify (dual-grant: user + token) → pvesm status reachability probe → capability report clears wg-handshake-read degradation. Idempotent re-apply = pvesm set only (never remove — K, §4). Marker + fail-safe per the offsite bridge precedent; consume-once + argv/log-hygiene red-proofs mandatory. Naming note: the epic's id felhom-pbs already exists on the DEMO host (the LAN dev PBS) — fresh customer boxes (Peti) are clean; a demo migration needs a rename/coexistence decision first.
• Slice 3: DR schedule + recipe coords + monitoring wiring. Much exists: the agent's PBS verify loop + tier-aware unattended restore-test (S4.1) already run against pbs-type storages on demo; hub-side needs the event wiring only (the restic OffsiteChecker is offbox-specific).
• Sequencing: after slice 2 lands on Peti (floor + agent update): his box gains felhom-pbs + K → the standard ceremony one-liner seals K + identity under one fresh R → auto-confirm → arc closed.

6. Teardown + environment observations

Teardown verified: demo — spike storage entry, .enc/.pw, both spike ACLs, sudoers drop-in, /tmp artifacts, staged secret all gone (pveum acl list + storage.cfg grep clean; the surviving spike grep hit is datastore felhom-spike, the real entry's datastore name). 180 — spike namespace+snapshots, token, ACLs, staged secrets gone; only felhom@pbs!n100 remains. ep0 — spike namespace+snapshots, both tokens, all spike ACLs, /tmp/spikewrite gone; only the real demo-felhom-01 tenancy remains. Demo's real felhom-offsite + felhom-pbs storages re-verified active post-spike.

Flags for Viktor (pre-existing, not touched):

1. ep0 has an orphan-looking root@pam!spike token + DatastoreAdmin ACL on /datastore/scratch — from the S4-era spikes; cleanup candidate.
2. The demo host has NO /storage/felhom-pbs or /storage/felhom-offsite FelhomAgentStore grants today (only local + local-lvm) — the §4b step-3 grants are absent, most likely the token-remove-purges-ACL gotcha. Backups demonstrably still work (the verify loop is green), so either the grants are not actually load-bearing for the current vzdump path or something re-asserts lazily — worth a deliberate re-assert + a check of which ops actually need them before slice 2 codifies the self-grant.
3. The LAN PBS (180) real token felhom@pbs!n100 holds datastore-wide DatastoreAdmin — dev-grade, much broader than ep0's ns-scoped DatastoreBackup model; fine for dev, don't copy the pattern.