# SPIKE — PBS DR tier auto-provisioning, SLICE 0: the three load-bearing mechanisms

**Date:** 2026-07-10 · **Class:** spike (validate before the epic's implementation slices) · **Executor:**
Claude Code live on the demo host (felhom-pve) + the build-server PBS (180) + **ep0 (felhom-hetzner,
Viktor-approved 🛑 touch)**. **No production code shipped; every probe artifact was `spike-*`-named and torn
down** (post-teardown asserts in §6). The epic: "PBS DR" on the hub customer page → WG peer + per-customer
PBS tenancy on ep0 → the agent applies the storage entry host-side → K exists → the standard ceremony covers it.

---

## 1. VERDICT

| Probe | Question | Verdict | Settled mechanism |
|---|---|---|---|
| **1** | Can a path-scoped PVE token create the storage entry? | **NO — impossible.** Create/modify/delete ALL check `Datastore.Allocate` on the **`/storage` root**; path-scoped ACLs govern only *using* a storage. | **Sudoers vector** (1b), storage-id **pinned** in the alias — full add/set/acl/remove cycle proven as `felhom-agent` via `sudo -n`; unpinned id denied. |
| **2** | Can tenancy ops ride the peersync channel? | **Not today** (forced command + single-alias sudoers block everything else — live-proven), **but the op-set + one-time-secret-over-SSH-stdout transport is PROVEN on ep0** with a throwaway tenant, incl. read/write cross-tenant 403s. | **Slice 1 ships a second surface** (`felhom-tenantsync`: own keypair + own forced-command line + own sudoers entry — peersync untouched, one-script-one-job preserved). |
| **3** | Does `--encryption-key autogen` give us K at apply time? | **YES.** Entry active, **K born at `/etc/pve/priv/storage/<id>.enc`** (0600 root:www-data), vzdump lands as **ciphertext** on the PBS side (`crypt-mode: encrypt`), `escrow-create` would find it. | `pvesm add pbs <id> … --encryption-key autogen` (via the Probe-1b sudo wrapper), fingerprint from Probe 2's retrieval. |

The epic is now mechanical. Slice map in §5.

## 2. Probe 1 — PVE permission model (demo host)

- Fixture: `FelhomAgentStore = Datastore.Allocate,Datastore.AllocateSpace`; agent token `felhom-agent@pve!agent`
  is `privsep=1` → dual-grant (user AND token) required, as everywhere.
- ACL on the **not-yet-existing** path `/storage/spike-pbs` is allowed (paths are logical) — granted to both
  user and token.
- **CREATE as the token** (`POST /api2/json/storage`, type dir, minimal) →
  **`403 Permission check failed (/storage, Datastore.Allocate)`** — the check runs against the `/storage`
  ROOT, verbatim. Granting that root-wide = the agent could create/delete ANY storage → unacceptable.
- **MODIFY (`PUT /storage/spike-pbs`) and DELETE as the token** against a root-created entry → **the SAME
  403 on `/storage`**. The whole entry-lifecycle CRUD ignores path-scoped grants; `/storage/<id>` ACLs matter
  only for content/usage ops (AllocateSpace, backup, audit).

### 2b. The sudoers vector (the fallback pattern — PROVEN)

Temp drop-in (`visudo -cf` OK), storage-id **PINNED**, run as the `felhom-agent` user via `sudo -n`:

```
Cmnd_Alias FELHOM_SPIKE_STORE = \
  /usr/sbin/pvesm add pbs spike-pbs *, \
  /usr/sbin/pvesm add dir spike-pbs *, \
  /usr/sbin/pvesm set spike-pbs *, \
  /usr/sbin/pvesm remove spike-pbs, \
  /usr/sbin/pveum acl modify /storage/spike-pbs *
felhom-agent ALL=(root) NOPASSWD: FELHOM_SPIKE_STORE
```

- add → set → `pveum acl modify /storage/spike-pbs` (the self-grant) → remove: **all OK** as the agent user.
- **Containment held:** `sudo -n pvesm remove felhom-usb` (an unpinned real id) → denied (`a password is
  required`). In-agent fine validation per the existing sudoers philosophy (the drop-in is the coarse gate).
- **⚠ sudo logs full argv to auth.log** → the PBS token secret must **NEVER** ride
  `sudo pvesm add … --password <secret>`. Epic shape: a root **wrapper script** (installed like the existing
  units/install pattern, sudoers-pinned by name) that reads the secret on **STDIN** and calls `pvesm`
  internally — the sudo log then shows only the wrapper invocation. (Matches the §4b runbook precedent that
  already avoided password-on-argv, and the agent's `runCommandStdin` seam.)

## 3. Probe 2 — ep0 tenancy ops + the one-time-secret channel (🛑 approved)

**Recon (read-only, recorded):** ep0 = `felhom-hetzner` (167.233.158.164), peersync script v1.0.1.
`felhom-peersync`'s authorized_keys = ONE line, `restrict,command="sudo /usr/local/bin/felhom-peersync"`;
sudoers = exactly that script; **live-proven the user can sudo nothing else** (`sudo -n proxmox-backup-manager
user list` → password required). So the current channel cannot carry tenancy ops — by design.

**The op-set, proven with throwaway `spike-tenant` on the real `felhom-offsite` datastore** (§4a dance,
spike-named; run as root = what the future sudo-pinned tenantsync script is):

1. throwaway `root@pam!spikeadm` admin token + `DatastoreAdmin` on the datastore → `proxmox-backup-client
   namespace create spike-tenant` (namespace ops are client-side; `PBS_FINGERPRINT` required even on localhost)
2. `proxmox-backup-manager user generate-token felhom@pbs spike-tenant` — **the secret returned over the SSH
   session stdout (36 bytes observed by the caller, never persisted on ep0, never displayed)** — the
   one-time-secret transport the hub will consume-once
3. dual-grant `DatastoreBackup` on `/datastore/felhom-offsite/spike-tenant` to **both** `felhom@pbs` and
   `felhom@pbs!spike-tenant`
4. **S4 isolation asserts (all green):** own-ns list OK · `demo-felhom-01` list → **403** · own-ns **write**
   (real `proxmox-backup-client backup`) OK · cross-tenant **write** → **403**
5. fingerprint retrieval (the descriptor field): `proxmox-backup-manager cert info | awk '/Fingerprint/{print $3}'`
6. teardown: `namespace delete spike-tenant --delete-groups true`, delete both tokens, delete all spike ACLs

**Ordering/behavior gotchas (live-hit):**
- **Token must exist BEFORE its ACL grant** — `acl update --auth-id '…!tok'` on a not-yet-existing token
  fails `no such API token` (PBS validates the auth-id; PVE does not — Probe 1 granted an unborn path fine).
- **`delete-token` purges the token's ACLs** (PVE-like) — a regen is delete + generate + **re-grant**.
- `generate-token` has **no `--output-format`** — parse its default JSON block (`sed -n 's/.*"value": "…'`).
  There is no regenerate subcommand on PBS 4.2.2.
- One transient: the first grant→assert pass 403'd on the OWN namespace despite both ACLs listing correctly;
  a delete+regen+re-grant converged and the identical sequence then passed grant→assert ~1s apart. Cause not
  identified. **Operational note for slice 1: post-provision, probe own-ns list as the new token; on 403,
  regen once.**
- `proxmox-backup-client` is a **separate package** (the runbook says so; 180 didn't have it — installed).

**The slice-1 ep0 surface (recommendation):** a second keypair (`hub-tenancy`) + second authorized_keys line
with forced command `sudo /usr/local/bin/felhom-tenantsync` + its own single-line sudoers entry. JSON-on-stdin
contract like peersync (`{"op":"provision|deprovision","ns":"<host_id>"}`, ns validated `^[a-z0-9-]{1,32}$`
and never a reserved name), response JSON on stdout carrying `{token_secret, fingerprint, datastore, ns}`.
The script holds the throwaway admin secret in memory only (no `/root/.spikeadm.raw` even transiently).
Peersync itself stays untouched (one script, one job).

## 4. Probe 3 — K autogen + fingerprint at apply time (demo host)

Target: the LAN PBS (192.168.0.180, datastore `felhom-spike`, spike-tenant ns created + torn down there too —
ep0 stayed out of this probe).

```
pvesm add pbs spike-pbs --server 192.168.0.180 --datastore felhom-spike --namespace spike-tenant \
  --username 'felhom@pbs!spike-tenant' --password <secret> --fingerprint <fp> --encryption-key autogen
```

- Entry **active** immediately (`pvesm status`).
- **K born at `/etc/pve/priv/storage/spike-pbs.enc`** — 0600 root:www-data, 255 bytes, same shape as the real
  `felhom-pbs.enc`/`felhom-offsite.enc`; storage.cfg gets `encryption-key <K's own fingerprint>`; the token
  secret lands as `spike-pbs.pw` alongside.
- **vzdump-class probe:** guest 9001 (`spike-lxc`, stop-mode, 26s) → PBS side sees
  `pct.conf.blob encrypt · root.pxar.didx encrypt · catalog.pcat1.didx encrypt · index.json.blob sign-only ·
  client.log.blob none` — **the doc-06 property: ciphertext client-side, K never left the box.** (Manifest
  sign-only + plaintext client log are standard PBS semantics.)
- **`escrow-create --storage spike-pbs` would find K:** `runSelftestEscrowCreate` stats
  `cfg.Backup.PBSEncKeyPath(storage)`; demo's `pbs_secret_dir` = `/etc/pve/priv/storage` (the default) → the
  exact autogen path. **⚠ On boxes that override `pbs_secret_dir` (the §4b WARN-fix copy dir), only `.pw` is
  copied today — slice 2 must also place/copy `.enc` there or escrow-create misses it.**
- **⚠ `pvesm remove <id>` DELETES `<id>.enc` AND `<id>.pw`** — removing the entry **destroys K**. The epic's
  re-apply path must be `pvesm set`-only, NEVER remove+re-add after a ceremony (the ApplyOffsiteTarget
  EscrowState lesson, storage-entry edition). The remove vector stays in the sudoers alias only for the
  explicit decommission flow, gated on escrow/DR policy.

## 5. The epic's slice map (recommend, don't build)

- **Slice 1 (hub):** customer-config "PBS DR tier" enable → WG peer assign (existing allocator) + tenancy
  provision over the NEW `felhom-tenantsync` surface (§3) → token secret stored **consume-once, HOST-scoped**
  (the agent fetches it with its host api_key — the controller consume-once precedent, host-side twin) →
  non-secret coords into desired-state: `pbs_tunnel_ip` (10.77.0.1), `datastore` (felhom-offsite),
  `namespace` (= host_id), `fingerprint` (from the tenantsync response).
- **Slice 2 (agent):** the apply-bridge — on the desired-state pbs block: ensure wg-felhom up → verify the
  PBS fingerprint → consume the token → **root wrapper (sudoers-pinned, secret on stdin, §2b)** runs
  `pvesm add pbs felhom-pbs … --encryption-key autogen` → self-grant `/storage/felhom-pbs` via the pinned
  `pveum acl modify` (dual-grant: user + token) → `pvesm status` reachability probe → capability report clears
  `wg-handshake-read` degradation. Idempotent re-apply = `pvesm set` only (never remove — K, §4). Marker +
  fail-safe per the offsite bridge precedent; consume-once + argv/log-hygiene red-proofs mandatory.
  **Naming note:** the epic's id `felhom-pbs` already exists on the DEMO host (the LAN dev PBS) — fresh
  customer boxes (Peti) are clean; a demo migration needs a rename/coexistence decision first.
- **Slice 3:** DR schedule + recipe coords + monitoring wiring. Much exists: the agent's PBS verify loop +
  tier-aware unattended restore-test (S4.1) already run against pbs-type storages on demo; hub-side needs the
  event wiring only (the restic `OffsiteChecker` is offbox-specific).
- **Sequencing:** after slice 2 lands on Peti (floor + agent update): his box gains `felhom-pbs` + K → **the
  standard ceremony one-liner** seals K + identity under one fresh R → auto-confirm → arc closed.

## 6. Teardown + environment observations

**Teardown verified:** demo — spike storage entry, `.enc`/`.pw`, both spike ACLs, sudoers drop-in, `/tmp`
artifacts, staged secret all gone (`pveum acl list` + storage.cfg grep clean; the surviving `spike` grep hit
is `datastore felhom-spike`, the real entry's datastore name). 180 — spike namespace+snapshots, token, ACLs,
staged secrets gone; only `felhom@pbs!n100` remains. ep0 — spike namespace+snapshots, both tokens, all spike
ACLs, `/tmp/spikewrite` gone; only the real `demo-felhom-01` tenancy remains. Demo's real `felhom-offsite` +
`felhom-pbs` storages re-verified **active** post-spike.

**Flags for Viktor (pre-existing, not touched):**
1. **ep0 has an orphan-looking `root@pam!spike` token + `DatastoreAdmin` ACL on `/datastore/scratch`** — from
   the S4-era spikes; cleanup candidate.
2. **The demo host has NO `/storage/felhom-pbs` or `/storage/felhom-offsite` FelhomAgentStore grants today**
   (only `local` + `local-lvm`) — the §4b step-3 grants are absent, most likely the token-remove-purges-ACL
   gotcha. Backups demonstrably still work (the verify loop is green), so either the grants are not actually
   load-bearing for the current vzdump path or something re-asserts lazily — worth a deliberate re-assert +
   a check of which ops actually need them before slice 2 codifies the self-grant.
3. The LAN PBS (180) real token `felhom@pbs!n100` holds **datastore-wide `DatastoreAdmin`** — dev-grade,
   much broader than ep0's ns-scoped `DatastoreBackup` model; fine for dev, don't copy the pattern.
