spike: PBS DR tier provisioning SLICE 0 — all 3 mechanisms settled (token-create impossible -> pinned sudoers; ep0 tenancy op-set + stdout secret transport proven; encryption-key autogen births K, pvesm remove deletes it)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6

CONTEXT.md

@@ -3,6 +3,17 @@
 > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
 > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
 
+- **2026-07-10 — SPIKE: PBS DR tier auto-provisioning SLICE 0 — all three mechanisms SETTLED.** Record:
+  `documentation/audits/SPIKE-pbs-tier-provisioning-2026-07-10.md`. (1) PVE token storage-create **impossible**
+  (create/modify/delete all check `/storage` root) → **sudoers vector proven** (pinned-id alias, cycle green as
+  the agent user, unpinned denied; secret must ride a stdin-wrapper, never sudo argv). (2) ep0 tenancy op-set +
+  one-time-secret-over-SSH-stdout **proven live on ep0** (🛑 Viktor-approved; read/write cross-tenant 403s);
+  current peersync channel can't carry it → slice 1 ships a second forced-command surface (`felhom-tenantsync`).
+  (3) `pvesm add pbs --encryption-key autogen` births K at `PBSEncKeyPath` + vzdump lands ciphertext (doc-06
+  property) + escrow-create finds K. **⚠ `pvesm remove` DELETES K** → re-apply must be set-only. All spike
+  artifacts torn down; flags: orphan `root@pam!spike` on ep0 scratch; demo's §4b step-3 storage grants absent.
+  Epic slice map in the record §5.
+
 - **2026-07-10 — new-box restore round-trip VALIDATED + old-box archive RETIRED.** Record:
   `documentation/audits/VALIDATION-newbox-restore-2026-07-10.md`. Both toggled apps restored from the
   production pool box via the real endpoint — **9/9 sha256-identical** (immich 44MB db-dump exact,

REPORT.md

@@ -2,22 +2,34 @@
 
 > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
 
-## TASK — hub v0.43.1: Git Sync form hint (credentials optional) — 2026-07-10
+## SPIKE — PBS DR tier auto-provisioning, SLICE 0: the three load-bearing mechanisms — 2026-07-10
 
-**Pairs with controller v0.112.0 (anonymous registry self-update — full report in
+**No production code — findings record at
-felhom-controller/REPORT.md).** One template string, no behavior change: the config editor's Git Sync
+`documentation/audits/SPIKE-pbs-tier-provisioning-2026-07-10.md`.** All three unproven mechanisms of the
-section looked load-bearing, but the credentials matter only for a PRIVATE app catalog — since
+"PBS DR tier" epic got empirical answers; every probe artifact was `spike-*`-named and torn down (asserted);
-controller v0.112.0 version discovery + self-update work without them (anonymous Docker v2 token flow).
+the demo's real `felhom-offsite`/`felhom-pbs` storages re-verified active afterwards. Viktor approved the
+one 🛑 (the ep0 touch) mid-run.
 
-- `config_form.html` Git Sync section gains the hint: **"Opcionális — csak privát
+- **Probe 1 (demo host): path-scoped token create is IMPOSSIBLE.** `POST/PUT/DELETE /storage*` all 403 with
-  alkalmazás-katalógushoz. A verziófrissítés enélkül is működik."**
+  `Permission check failed (/storage, Datastore.Allocate)` — the whole entry-lifecycle CRUD checks the
-- Deployed via GitOps: image `0.43.1`, manifest bump, ArgoCD sync → Synced/Healthy, rollout complete.
+  `/storage` ROOT; `/storage/<id>` ACLs govern usage only. **Probe 1b settled the fallback:** a pinned-id
-- Green gate green; render tests cover template parse.
+  sudoers alias (`pvesm add/set/remove <id>`, `pveum acl modify /storage/<id>`) — full cycle green as the
-
+  `felhom-agent` user via `sudo -n`, unpinned id denied. Gotcha: sudo logs argv → the token secret must go
-**Pending operator step (per the controller task spec):** after the floor bump rolls Peti to 0.112.0,
+  through a stdin-reading root wrapper, never `sudo pvesm add --password …`.
-delete the temporary Git Sync creds from Peti's hub config → next config pull → Peti's settings page
+- **Probe 2 (ep0, 🛑): the tenancy op-set + secret transport PROVEN.** Recon recorded the surface (one
-shows "Registry: nyilvános (hitelesítés nélkül)" with zero credentials on the box. (Verified live on
+  `restrict`+forced-command key, one sudoers line; the peersync user can sudo nothing else — live-proven).
-the credential-less demo box already: anonymous check OK, latest resolved, mode line renders.)
+  Throwaway `spike-tenant` on the real `felhom-offsite`: namespace + privsep token + dual-grant, the token
-
+  secret returned over SSH stdout (36 bytes, never persisted on ep0, never displayed), S4 isolation asserts
-*(Previous report — hub v0.43.0 remote app-log diagnostics incl. the completed live tail round-trip —
+  green incl. a real cross-tenant WRITE 403. Ordering gotchas recorded (token-before-ACL; delete-token purges
-is recorded in hub/CHANGELOG.md v0.43.0 and the git history of this file.)*
+  its ACLs; no `--output-format` on generate-token). Slice-1 recommendation: a second forced-command surface
+  (`felhom-tenantsync`, own keypair + sudoers line), peersync untouched.
+- **Probe 3 (demo host): `pvesm add pbs … --encryption-key autogen` works end-to-end.** Entry active, K born
+  at `/etc/pve/priv/storage/<id>.enc` (0600, = `PBSEncKeyPath` on demo), vzdump of a small guest lands as
+  **ciphertext** PBS-side (`crypt-mode: encrypt` — the doc-06 property), `escrow-create` would find K.
+  **⚠ `pvesm remove` DELETES the `.enc` (K!) + `.pw`** → the epic's re-apply must be `pvesm set`-only.
+- **Flags surfaced (pre-existing):** orphan `root@pam!spike` DatastoreAdmin on ep0 `/datastore/scratch`;
+  the demo host is missing the §4b step-3 `/storage/felhom-pbs|felhom-offsite` FelhomAgentStore grants
+  (backups still green — needs a deliberate re-assert before slice 2 codifies the self-grant); 180's
+  `felhom@pbs!n100` is datastore-wide DatastoreAdmin (dev-grade, don't copy).
+- **Epic slice map** (hub tenantsync + consume-once → agent apply-bridge with stdin-wrapper + set-only
+  re-apply → DR schedule/monitoring → Peti ceremony one-liner) in the record §5.

documentation/audits/SPIKE-pbs-tier-provisioning-2026-07-10.md

@@ -0,0 +1,164 @@
+# SPIKE — PBS DR tier auto-provisioning, SLICE 0: the three load-bearing mechanisms
+
+**Date:** 2026-07-10 · **Class:** spike (validate before the epic's implementation slices) · **Executor:**
+Claude Code live on the demo host (felhom-pve) + the build-server PBS (180) + **ep0 (felhom-hetzner,
+Viktor-approved 🛑 touch)**. **No production code shipped; every probe artifact was `spike-*`-named and torn
+down** (post-teardown asserts in §6). The epic: "PBS DR" on the hub customer page → WG peer + per-customer
+PBS tenancy on ep0 → the agent applies the storage entry host-side → K exists → the standard ceremony covers it.
+
+---
+
+## 1. VERDICT
+
+| Probe | Question | Verdict | Settled mechanism |
+|---|---|---|---|
+| **1** | Can a path-scoped PVE token create the storage entry? | **NO — impossible.** Create/modify/delete ALL check `Datastore.Allocate` on the **`/storage` root**; path-scoped ACLs govern only *using* a storage. | **Sudoers vector** (1b), storage-id **pinned** in the alias — full add/set/acl/remove cycle proven as `felhom-agent` via `sudo -n`; unpinned id denied. |
+| **2** | Can tenancy ops ride the peersync channel? | **Not today** (forced command + single-alias sudoers block everything else — live-proven), **but the op-set + one-time-secret-over-SSH-stdout transport is PROVEN on ep0** with a throwaway tenant, incl. read/write cross-tenant 403s. | **Slice 1 ships a second surface** (`felhom-tenantsync`: own keypair + own forced-command line + own sudoers entry — peersync untouched, one-script-one-job preserved). |
+| **3** | Does `--encryption-key autogen` give us K at apply time? | **YES.** Entry active, **K born at `/etc/pve/priv/storage/<id>.enc`** (0600 root:www-data), vzdump lands as **ciphertext** on the PBS side (`crypt-mode: encrypt`), `escrow-create` would find it. | `pvesm add pbs <id> … --encryption-key autogen` (via the Probe-1b sudo wrapper), fingerprint from Probe 2's retrieval. |
+
+The epic is now mechanical. Slice map in §5.
+
+## 2. Probe 1 — PVE permission model (demo host)
+
+- Fixture: `FelhomAgentStore = Datastore.Allocate,Datastore.AllocateSpace`; agent token `felhom-agent@pve!agent`
+  is `privsep=1` → dual-grant (user AND token) required, as everywhere.
+- ACL on the **not-yet-existing** path `/storage/spike-pbs` is allowed (paths are logical) — granted to both
+  user and token.
+- **CREATE as the token** (`POST /api2/json/storage`, type dir, minimal) →
+  **`403 Permission check failed (/storage, Datastore.Allocate)`** — the check runs against the `/storage`
+  ROOT, verbatim. Granting that root-wide = the agent could create/delete ANY storage → unacceptable.
+- **MODIFY (`PUT /storage/spike-pbs`) and DELETE as the token** against a root-created entry → **the SAME
+  403 on `/storage`**. The whole entry-lifecycle CRUD ignores path-scoped grants; `/storage/<id>` ACLs matter
+  only for content/usage ops (AllocateSpace, backup, audit).
+
+### 2b. The sudoers vector (the fallback pattern — PROVEN)
+
+Temp drop-in (`visudo -cf` OK), storage-id **PINNED**, run as the `felhom-agent` user via `sudo -n`:
+
+```
+Cmnd_Alias FELHOM_SPIKE_STORE = \
+  /usr/sbin/pvesm add pbs spike-pbs *, \
+  /usr/sbin/pvesm add dir spike-pbs *, \
+  /usr/sbin/pvesm set spike-pbs *, \
+  /usr/sbin/pvesm remove spike-pbs, \
+  /usr/sbin/pveum acl modify /storage/spike-pbs *
+felhom-agent ALL=(root) NOPASSWD: FELHOM_SPIKE_STORE
+```
+
+- add → set → `pveum acl modify /storage/spike-pbs` (the self-grant) → remove: **all OK** as the agent user.
+- **Containment held:** `sudo -n pvesm remove felhom-usb` (an unpinned real id) → denied (`a password is
+  required`). In-agent fine validation per the existing sudoers philosophy (the drop-in is the coarse gate).
+- **⚠ sudo logs full argv to auth.log** → the PBS token secret must **NEVER** ride
+  `sudo pvesm add … --password <secret>`. Epic shape: a root **wrapper script** (installed like the existing
+  units/install pattern, sudoers-pinned by name) that reads the secret on **STDIN** and calls `pvesm`
+  internally — the sudo log then shows only the wrapper invocation. (Matches the §4b runbook precedent that
+  already avoided password-on-argv, and the agent's `runCommandStdin` seam.)
+
+## 3. Probe 2 — ep0 tenancy ops + the one-time-secret channel (🛑 approved)
+
+**Recon (read-only, recorded):** ep0 = `felhom-hetzner` (167.233.158.164), peersync script v1.0.1.
+`felhom-peersync`'s authorized_keys = ONE line, `restrict,command="sudo /usr/local/bin/felhom-peersync"`;
+sudoers = exactly that script; **live-proven the user can sudo nothing else** (`sudo -n proxmox-backup-manager
+user list` → password required). So the current channel cannot carry tenancy ops — by design.
+
+**The op-set, proven with throwaway `spike-tenant` on the real `felhom-offsite` datastore** (§4a dance,
+spike-named; run as root = what the future sudo-pinned tenantsync script is):
+
+1. throwaway `root@pam!spikeadm` admin token + `DatastoreAdmin` on the datastore → `proxmox-backup-client
+   namespace create spike-tenant` (namespace ops are client-side; `PBS_FINGERPRINT` required even on localhost)
+2. `proxmox-backup-manager user generate-token felhom@pbs spike-tenant` — **the secret returned over the SSH
+   session stdout (36 bytes observed by the caller, never persisted on ep0, never displayed)** — the
+   one-time-secret transport the hub will consume-once
+3. dual-grant `DatastoreBackup` on `/datastore/felhom-offsite/spike-tenant` to **both** `felhom@pbs` and
+   `felhom@pbs!spike-tenant`
+4. **S4 isolation asserts (all green):** own-ns list OK · `demo-felhom-01` list → **403** · own-ns **write**
+   (real `proxmox-backup-client backup`) OK · cross-tenant **write** → **403**
+5. fingerprint retrieval (the descriptor field): `proxmox-backup-manager cert info | awk '/Fingerprint/{print $3}'`
+6. teardown: `namespace delete spike-tenant --delete-groups true`, delete both tokens, delete all spike ACLs
+
+**Ordering/behavior gotchas (live-hit):**
+- **Token must exist BEFORE its ACL grant** — `acl update --auth-id '…!tok'` on a not-yet-existing token
+  fails `no such API token` (PBS validates the auth-id; PVE does not — Probe 1 granted an unborn path fine).
+- **`delete-token` purges the token's ACLs** (PVE-like) — a regen is delete + generate + **re-grant**.
+- `generate-token` has **no `--output-format`** — parse its default JSON block (`sed -n 's/.*"value": "…'`).
+  There is no regenerate subcommand on PBS 4.2.2.
+- One transient: the first grant→assert pass 403'd on the OWN namespace despite both ACLs listing correctly;
+  a delete+regen+re-grant converged and the identical sequence then passed grant→assert ~1s apart. Cause not
+  identified. **Operational note for slice 1: post-provision, probe own-ns list as the new token; on 403,
+  regen once.**
+- `proxmox-backup-client` is a **separate package** (the runbook says so; 180 didn't have it — installed).
+
+**The slice-1 ep0 surface (recommendation):** a second keypair (`hub-tenancy`) + second authorized_keys line
+with forced command `sudo /usr/local/bin/felhom-tenantsync` + its own single-line sudoers entry. JSON-on-stdin
+contract like peersync (`{"op":"provision|deprovision","ns":"<host_id>"}`, ns validated `^[a-z0-9-]{1,32}$`
+and never a reserved name), response JSON on stdout carrying `{token_secret, fingerprint, datastore, ns}`.
+The script holds the throwaway admin secret in memory only (no `/root/.spikeadm.raw` even transiently).
+Peersync itself stays untouched (one script, one job).
+
+## 4. Probe 3 — K autogen + fingerprint at apply time (demo host)
+
+Target: the LAN PBS (192.168.0.180, datastore `felhom-spike`, spike-tenant ns created + torn down there too —
+ep0 stayed out of this probe).
+
+```
+pvesm add pbs spike-pbs --server 192.168.0.180 --datastore felhom-spike --namespace spike-tenant \
+  --username 'felhom@pbs!spike-tenant' --password <secret> --fingerprint <fp> --encryption-key autogen
+```
+
+- Entry **active** immediately (`pvesm status`).
+- **K born at `/etc/pve/priv/storage/spike-pbs.enc`** — 0600 root:www-data, 255 bytes, same shape as the real
+  `felhom-pbs.enc`/`felhom-offsite.enc`; storage.cfg gets `encryption-key <K's own fingerprint>`; the token
+  secret lands as `spike-pbs.pw` alongside.
+- **vzdump-class probe:** guest 9001 (`spike-lxc`, stop-mode, 26s) → PBS side sees
+  `pct.conf.blob encrypt · root.pxar.didx encrypt · catalog.pcat1.didx encrypt · index.json.blob sign-only ·
+  client.log.blob none` — **the doc-06 property: ciphertext client-side, K never left the box.** (Manifest
+  sign-only + plaintext client log are standard PBS semantics.)
+- **`escrow-create --storage spike-pbs` would find K:** `runSelftestEscrowCreate` stats
+  `cfg.Backup.PBSEncKeyPath(storage)`; demo's `pbs_secret_dir` = `/etc/pve/priv/storage` (the default) → the
+  exact autogen path. **⚠ On boxes that override `pbs_secret_dir` (the §4b WARN-fix copy dir), only `.pw` is
+  copied today — slice 2 must also place/copy `.enc` there or escrow-create misses it.**
+- **⚠ `pvesm remove <id>` DELETES `<id>.enc` AND `<id>.pw`** — removing the entry **destroys K**. The epic's
+  re-apply path must be `pvesm set`-only, NEVER remove+re-add after a ceremony (the ApplyOffsiteTarget
+  EscrowState lesson, storage-entry edition). The remove vector stays in the sudoers alias only for the
+  explicit decommission flow, gated on escrow/DR policy.
+
+## 5. The epic's slice map (recommend, don't build)
+
+- **Slice 1 (hub):** customer-config "PBS DR tier" enable → WG peer assign (existing allocator) + tenancy
+  provision over the NEW `felhom-tenantsync` surface (§3) → token secret stored **consume-once, HOST-scoped**
+  (the agent fetches it with its host api_key — the controller consume-once precedent, host-side twin) →
+  non-secret coords into desired-state: `pbs_tunnel_ip` (10.77.0.1), `datastore` (felhom-offsite),
+  `namespace` (= host_id), `fingerprint` (from the tenantsync response).
+- **Slice 2 (agent):** the apply-bridge — on the desired-state pbs block: ensure wg-felhom up → verify the
+  PBS fingerprint → consume the token → **root wrapper (sudoers-pinned, secret on stdin, §2b)** runs
+  `pvesm add pbs felhom-pbs … --encryption-key autogen` → self-grant `/storage/felhom-pbs` via the pinned
+  `pveum acl modify` (dual-grant: user + token) → `pvesm status` reachability probe → capability report clears
+  `wg-handshake-read` degradation. Idempotent re-apply = `pvesm set` only (never remove — K, §4). Marker +
+  fail-safe per the offsite bridge precedent; consume-once + argv/log-hygiene red-proofs mandatory.
+  **Naming note:** the epic's id `felhom-pbs` already exists on the DEMO host (the LAN dev PBS) — fresh
+  customer boxes (Peti) are clean; a demo migration needs a rename/coexistence decision first.
+- **Slice 3:** DR schedule + recipe coords + monitoring wiring. Much exists: the agent's PBS verify loop +
+  tier-aware unattended restore-test (S4.1) already run against pbs-type storages on demo; hub-side needs the
+  event wiring only (the restic `OffsiteChecker` is offbox-specific).
+- **Sequencing:** after slice 2 lands on Peti (floor + agent update): his box gains `felhom-pbs` + K → **the
+  standard ceremony one-liner** seals K + identity under one fresh R → auto-confirm → arc closed.
+
+## 6. Teardown + environment observations
+
+**Teardown verified:** demo — spike storage entry, `.enc`/`.pw`, both spike ACLs, sudoers drop-in, `/tmp`
+artifacts, staged secret all gone (`pveum acl list` + storage.cfg grep clean; the surviving `spike` grep hit
+is `datastore felhom-spike`, the real entry's datastore name). 180 — spike namespace+snapshots, token, ACLs,
+staged secrets gone; only `felhom@pbs!n100` remains. ep0 — spike namespace+snapshots, both tokens, all spike
+ACLs, `/tmp/spikewrite` gone; only the real `demo-felhom-01` tenancy remains. Demo's real `felhom-offsite` +
+`felhom-pbs` storages re-verified **active** post-spike.
+
+**Flags for Viktor (pre-existing, not touched):**
+1. **ep0 has an orphan-looking `root@pam!spike` token + `DatastoreAdmin` ACL on `/datastore/scratch`** — from
+   the S4-era spikes; cleanup candidate.
+2. **The demo host has NO `/storage/felhom-pbs` or `/storage/felhom-offsite` FelhomAgentStore grants today**
+   (only `local` + `local-lvm`) — the §4b step-3 grants are absent, most likely the token-remove-purges-ACL
+   gotcha. Backups demonstrably still work (the verify loop is green), so either the grants are not actually
+   load-bearing for the current vzdump path or something re-asserts lazily — worth a deliberate re-assert +
+   a check of which ops actually need them before slice 2 codifies the self-grant.
+3. The LAN PBS (180) real token `felhom@pbs!n100` holds **datastore-wide `DatastoreAdmin`** — dev-grade,
+   much broader than ep0's ns-scoped `DatastoreBackup` model; fine for dev, don't copy the pattern.