From 00afadc1feac9d5941e075fc74d46da562929e2c Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Fri, 10 Jul 2026 20:12:26 +0200 Subject: [PATCH] =?UTF-8?q?spike:=20PBS=20DR=20tier=20provisioning=20SLICE?= =?UTF-8?q?=200=20=E2=80=94=20all=203=20mechanisms=20settled=20(token-crea?= =?UTF-8?q?te=20impossible=20->=20pinned=20sudoers;=20ep0=20tenancy=20op-s?= =?UTF-8?q?et=20+=20stdout=20secret=20transport=20proven;=20encryption-key?= =?UTF-8?q?=20autogen=20births=20K,=20pvesm=20remove=20deletes=20it)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 --- CONTEXT.md | 11 ++ REPORT.md | 46 +++-- .../SPIKE-pbs-tier-provisioning-2026-07-10.md | 164 ++++++++++++++++++ 3 files changed, 204 insertions(+), 17 deletions(-) create mode 100644 documentation/audits/SPIKE-pbs-tier-provisioning-2026-07-10.md diff --git a/CONTEXT.md b/CONTEXT.md index 0837f51..cba24db 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,6 +3,17 @@ > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. +- **2026-07-10 — SPIKE: PBS DR tier auto-provisioning SLICE 0 — all three mechanisms SETTLED.** Record: + `documentation/audits/SPIKE-pbs-tier-provisioning-2026-07-10.md`. (1) PVE token storage-create **impossible** + (create/modify/delete all check `/storage` root) → **sudoers vector proven** (pinned-id alias, cycle green as + the agent user, unpinned denied; secret must ride a stdin-wrapper, never sudo argv). (2) ep0 tenancy op-set + + one-time-secret-over-SSH-stdout **proven live on ep0** (🛑 Viktor-approved; read/write cross-tenant 403s); + current peersync channel can't carry it → slice 1 ships a second forced-command surface (`felhom-tenantsync`). + (3) `pvesm add pbs --encryption-key autogen` births K at `PBSEncKeyPath` + vzdump lands ciphertext (doc-06 + property) + escrow-create finds K. **⚠ `pvesm remove` DELETES K** → re-apply must be set-only. All spike + artifacts torn down; flags: orphan `root@pam!spike` on ep0 scratch; demo's §4b step-3 storage grants absent. + Epic slice map in the record §5. + - **2026-07-10 — new-box restore round-trip VALIDATED + old-box archive RETIRED.** Record: `documentation/audits/VALIDATION-newbox-restore-2026-07-10.md`. Both toggled apps restored from the production pool box via the real endpoint — **9/9 sha256-identical** (immich 44MB db-dump exact, diff --git a/REPORT.md b/REPORT.md index 6756e81..fee9747 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,22 +2,34 @@ > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md). -## TASK — hub v0.43.1: Git Sync form hint (credentials optional) — 2026-07-10 +## SPIKE — PBS DR tier auto-provisioning, SLICE 0: the three load-bearing mechanisms — 2026-07-10 -**Pairs with controller v0.112.0 (anonymous registry self-update — full report in -felhom-controller/REPORT.md).** One template string, no behavior change: the config editor's Git Sync -section looked load-bearing, but the credentials matter only for a PRIVATE app catalog — since -controller v0.112.0 version discovery + self-update work without them (anonymous Docker v2 token flow). +**No production code — findings record at +`documentation/audits/SPIKE-pbs-tier-provisioning-2026-07-10.md`.** All three unproven mechanisms of the +"PBS DR tier" epic got empirical answers; every probe artifact was `spike-*`-named and torn down (asserted); +the demo's real `felhom-offsite`/`felhom-pbs` storages re-verified active afterwards. Viktor approved the +one 🛑 (the ep0 touch) mid-run. -- `config_form.html` Git Sync section gains the hint: **"Opcionális — csak privát - alkalmazás-katalógushoz. A verziófrissítés enélkül is működik."** -- Deployed via GitOps: image `0.43.1`, manifest bump, ArgoCD sync → Synced/Healthy, rollout complete. -- Green gate green; render tests cover template parse. - -**Pending operator step (per the controller task spec):** after the floor bump rolls Peti to 0.112.0, -delete the temporary Git Sync creds from Peti's hub config → next config pull → Peti's settings page -shows "Registry: nyilvános (hitelesítés nélkül)" with zero credentials on the box. (Verified live on -the credential-less demo box already: anonymous check OK, latest resolved, mode line renders.) - -*(Previous report — hub v0.43.0 remote app-log diagnostics incl. the completed live tail round-trip — -is recorded in hub/CHANGELOG.md v0.43.0 and the git history of this file.)* +- **Probe 1 (demo host): path-scoped token create is IMPOSSIBLE.** `POST/PUT/DELETE /storage*` all 403 with + `Permission check failed (/storage, Datastore.Allocate)` — the whole entry-lifecycle CRUD checks the + `/storage` ROOT; `/storage/` ACLs govern usage only. **Probe 1b settled the fallback:** a pinned-id + sudoers alias (`pvesm add/set/remove `, `pveum acl modify /storage/`) — full cycle green as the + `felhom-agent` user via `sudo -n`, unpinned id denied. Gotcha: sudo logs argv → the token secret must go + through a stdin-reading root wrapper, never `sudo pvesm add --password …`. +- **Probe 2 (ep0, 🛑): the tenancy op-set + secret transport PROVEN.** Recon recorded the surface (one + `restrict`+forced-command key, one sudoers line; the peersync user can sudo nothing else — live-proven). + Throwaway `spike-tenant` on the real `felhom-offsite`: namespace + privsep token + dual-grant, the token + secret returned over SSH stdout (36 bytes, never persisted on ep0, never displayed), S4 isolation asserts + green incl. a real cross-tenant WRITE 403. Ordering gotchas recorded (token-before-ACL; delete-token purges + its ACLs; no `--output-format` on generate-token). Slice-1 recommendation: a second forced-command surface + (`felhom-tenantsync`, own keypair + sudoers line), peersync untouched. +- **Probe 3 (demo host): `pvesm add pbs … --encryption-key autogen` works end-to-end.** Entry active, K born + at `/etc/pve/priv/storage/.enc` (0600, = `PBSEncKeyPath` on demo), vzdump of a small guest lands as + **ciphertext** PBS-side (`crypt-mode: encrypt` — the doc-06 property), `escrow-create` would find K. + **⚠ `pvesm remove` DELETES the `.enc` (K!) + `.pw`** → the epic's re-apply must be `pvesm set`-only. +- **Flags surfaced (pre-existing):** orphan `root@pam!spike` DatastoreAdmin on ep0 `/datastore/scratch`; + the demo host is missing the §4b step-3 `/storage/felhom-pbs|felhom-offsite` FelhomAgentStore grants + (backups still green — needs a deliberate re-assert before slice 2 codifies the self-grant); 180's + `felhom@pbs!n100` is datastore-wide DatastoreAdmin (dev-grade, don't copy). +- **Epic slice map** (hub tenantsync + consume-once → agent apply-bridge with stdin-wrapper + set-only + re-apply → DR schedule/monitoring → Peti ceremony one-liner) in the record §5. diff --git a/documentation/audits/SPIKE-pbs-tier-provisioning-2026-07-10.md b/documentation/audits/SPIKE-pbs-tier-provisioning-2026-07-10.md new file mode 100644 index 0000000..a290f50 --- /dev/null +++ b/documentation/audits/SPIKE-pbs-tier-provisioning-2026-07-10.md @@ -0,0 +1,164 @@ +# SPIKE — PBS DR tier auto-provisioning, SLICE 0: the three load-bearing mechanisms + +**Date:** 2026-07-10 · **Class:** spike (validate before the epic's implementation slices) · **Executor:** +Claude Code live on the demo host (felhom-pve) + the build-server PBS (180) + **ep0 (felhom-hetzner, +Viktor-approved 🛑 touch)**. **No production code shipped; every probe artifact was `spike-*`-named and torn +down** (post-teardown asserts in §6). The epic: "PBS DR" on the hub customer page → WG peer + per-customer +PBS tenancy on ep0 → the agent applies the storage entry host-side → K exists → the standard ceremony covers it. + +--- + +## 1. VERDICT + +| Probe | Question | Verdict | Settled mechanism | +|---|---|---|---| +| **1** | Can a path-scoped PVE token create the storage entry? | **NO — impossible.** Create/modify/delete ALL check `Datastore.Allocate` on the **`/storage` root**; path-scoped ACLs govern only *using* a storage. | **Sudoers vector** (1b), storage-id **pinned** in the alias — full add/set/acl/remove cycle proven as `felhom-agent` via `sudo -n`; unpinned id denied. | +| **2** | Can tenancy ops ride the peersync channel? | **Not today** (forced command + single-alias sudoers block everything else — live-proven), **but the op-set + one-time-secret-over-SSH-stdout transport is PROVEN on ep0** with a throwaway tenant, incl. read/write cross-tenant 403s. | **Slice 1 ships a second surface** (`felhom-tenantsync`: own keypair + own forced-command line + own sudoers entry — peersync untouched, one-script-one-job preserved). | +| **3** | Does `--encryption-key autogen` give us K at apply time? | **YES.** Entry active, **K born at `/etc/pve/priv/storage/.enc`** (0600 root:www-data), vzdump lands as **ciphertext** on the PBS side (`crypt-mode: encrypt`), `escrow-create` would find it. | `pvesm add pbs … --encryption-key autogen` (via the Probe-1b sudo wrapper), fingerprint from Probe 2's retrieval. | + +The epic is now mechanical. Slice map in §5. + +## 2. Probe 1 — PVE permission model (demo host) + +- Fixture: `FelhomAgentStore = Datastore.Allocate,Datastore.AllocateSpace`; agent token `felhom-agent@pve!agent` + is `privsep=1` → dual-grant (user AND token) required, as everywhere. +- ACL on the **not-yet-existing** path `/storage/spike-pbs` is allowed (paths are logical) — granted to both + user and token. +- **CREATE as the token** (`POST /api2/json/storage`, type dir, minimal) → + **`403 Permission check failed (/storage, Datastore.Allocate)`** — the check runs against the `/storage` + ROOT, verbatim. Granting that root-wide = the agent could create/delete ANY storage → unacceptable. +- **MODIFY (`PUT /storage/spike-pbs`) and DELETE as the token** against a root-created entry → **the SAME + 403 on `/storage`**. The whole entry-lifecycle CRUD ignores path-scoped grants; `/storage/` ACLs matter + only for content/usage ops (AllocateSpace, backup, audit). + +### 2b. The sudoers vector (the fallback pattern — PROVEN) + +Temp drop-in (`visudo -cf` OK), storage-id **PINNED**, run as the `felhom-agent` user via `sudo -n`: + +``` +Cmnd_Alias FELHOM_SPIKE_STORE = \ + /usr/sbin/pvesm add pbs spike-pbs *, \ + /usr/sbin/pvesm add dir spike-pbs *, \ + /usr/sbin/pvesm set spike-pbs *, \ + /usr/sbin/pvesm remove spike-pbs, \ + /usr/sbin/pveum acl modify /storage/spike-pbs * +felhom-agent ALL=(root) NOPASSWD: FELHOM_SPIKE_STORE +``` + +- add → set → `pveum acl modify /storage/spike-pbs` (the self-grant) → remove: **all OK** as the agent user. +- **Containment held:** `sudo -n pvesm remove felhom-usb` (an unpinned real id) → denied (`a password is + required`). In-agent fine validation per the existing sudoers philosophy (the drop-in is the coarse gate). +- **⚠ sudo logs full argv to auth.log** → the PBS token secret must **NEVER** ride + `sudo pvesm add … --password `. Epic shape: a root **wrapper script** (installed like the existing + units/install pattern, sudoers-pinned by name) that reads the secret on **STDIN** and calls `pvesm` + internally — the sudo log then shows only the wrapper invocation. (Matches the §4b runbook precedent that + already avoided password-on-argv, and the agent's `runCommandStdin` seam.) + +## 3. Probe 2 — ep0 tenancy ops + the one-time-secret channel (🛑 approved) + +**Recon (read-only, recorded):** ep0 = `felhom-hetzner` (167.233.158.164), peersync script v1.0.1. +`felhom-peersync`'s authorized_keys = ONE line, `restrict,command="sudo /usr/local/bin/felhom-peersync"`; +sudoers = exactly that script; **live-proven the user can sudo nothing else** (`sudo -n proxmox-backup-manager +user list` → password required). So the current channel cannot carry tenancy ops — by design. + +**The op-set, proven with throwaway `spike-tenant` on the real `felhom-offsite` datastore** (§4a dance, +spike-named; run as root = what the future sudo-pinned tenantsync script is): + +1. throwaway `root@pam!spikeadm` admin token + `DatastoreAdmin` on the datastore → `proxmox-backup-client + namespace create spike-tenant` (namespace ops are client-side; `PBS_FINGERPRINT` required even on localhost) +2. `proxmox-backup-manager user generate-token felhom@pbs spike-tenant` — **the secret returned over the SSH + session stdout (36 bytes observed by the caller, never persisted on ep0, never displayed)** — the + one-time-secret transport the hub will consume-once +3. dual-grant `DatastoreBackup` on `/datastore/felhom-offsite/spike-tenant` to **both** `felhom@pbs` and + `felhom@pbs!spike-tenant` +4. **S4 isolation asserts (all green):** own-ns list OK · `demo-felhom-01` list → **403** · own-ns **write** + (real `proxmox-backup-client backup`) OK · cross-tenant **write** → **403** +5. fingerprint retrieval (the descriptor field): `proxmox-backup-manager cert info | awk '/Fingerprint/{print $3}'` +6. teardown: `namespace delete spike-tenant --delete-groups true`, delete both tokens, delete all spike ACLs + +**Ordering/behavior gotchas (live-hit):** +- **Token must exist BEFORE its ACL grant** — `acl update --auth-id '…!tok'` on a not-yet-existing token + fails `no such API token` (PBS validates the auth-id; PVE does not — Probe 1 granted an unborn path fine). +- **`delete-token` purges the token's ACLs** (PVE-like) — a regen is delete + generate + **re-grant**. +- `generate-token` has **no `--output-format`** — parse its default JSON block (`sed -n 's/.*"value": "…'`). + There is no regenerate subcommand on PBS 4.2.2. +- One transient: the first grant→assert pass 403'd on the OWN namespace despite both ACLs listing correctly; + a delete+regen+re-grant converged and the identical sequence then passed grant→assert ~1s apart. Cause not + identified. **Operational note for slice 1: post-provision, probe own-ns list as the new token; on 403, + regen once.** +- `proxmox-backup-client` is a **separate package** (the runbook says so; 180 didn't have it — installed). + +**The slice-1 ep0 surface (recommendation):** a second keypair (`hub-tenancy`) + second authorized_keys line +with forced command `sudo /usr/local/bin/felhom-tenantsync` + its own single-line sudoers entry. JSON-on-stdin +contract like peersync (`{"op":"provision|deprovision","ns":""}`, ns validated `^[a-z0-9-]{1,32}$` +and never a reserved name), response JSON on stdout carrying `{token_secret, fingerprint, datastore, ns}`. +The script holds the throwaway admin secret in memory only (no `/root/.spikeadm.raw` even transiently). +Peersync itself stays untouched (one script, one job). + +## 4. Probe 3 — K autogen + fingerprint at apply time (demo host) + +Target: the LAN PBS (192.168.0.180, datastore `felhom-spike`, spike-tenant ns created + torn down there too — +ep0 stayed out of this probe). + +``` +pvesm add pbs spike-pbs --server 192.168.0.180 --datastore felhom-spike --namespace spike-tenant \ + --username 'felhom@pbs!spike-tenant' --password --fingerprint --encryption-key autogen +``` + +- Entry **active** immediately (`pvesm status`). +- **K born at `/etc/pve/priv/storage/spike-pbs.enc`** — 0600 root:www-data, 255 bytes, same shape as the real + `felhom-pbs.enc`/`felhom-offsite.enc`; storage.cfg gets `encryption-key `; the token + secret lands as `spike-pbs.pw` alongside. +- **vzdump-class probe:** guest 9001 (`spike-lxc`, stop-mode, 26s) → PBS side sees + `pct.conf.blob encrypt · root.pxar.didx encrypt · catalog.pcat1.didx encrypt · index.json.blob sign-only · + client.log.blob none` — **the doc-06 property: ciphertext client-side, K never left the box.** (Manifest + sign-only + plaintext client log are standard PBS semantics.) +- **`escrow-create --storage spike-pbs` would find K:** `runSelftestEscrowCreate` stats + `cfg.Backup.PBSEncKeyPath(storage)`; demo's `pbs_secret_dir` = `/etc/pve/priv/storage` (the default) → the + exact autogen path. **⚠ On boxes that override `pbs_secret_dir` (the §4b WARN-fix copy dir), only `.pw` is + copied today — slice 2 must also place/copy `.enc` there or escrow-create misses it.** +- **⚠ `pvesm remove ` DELETES `.enc` AND `.pw`** — removing the entry **destroys K**. The epic's + re-apply path must be `pvesm set`-only, NEVER remove+re-add after a ceremony (the ApplyOffsiteTarget + EscrowState lesson, storage-entry edition). The remove vector stays in the sudoers alias only for the + explicit decommission flow, gated on escrow/DR policy. + +## 5. The epic's slice map (recommend, don't build) + +- **Slice 1 (hub):** customer-config "PBS DR tier" enable → WG peer assign (existing allocator) + tenancy + provision over the NEW `felhom-tenantsync` surface (§3) → token secret stored **consume-once, HOST-scoped** + (the agent fetches it with its host api_key — the controller consume-once precedent, host-side twin) → + non-secret coords into desired-state: `pbs_tunnel_ip` (10.77.0.1), `datastore` (felhom-offsite), + `namespace` (= host_id), `fingerprint` (from the tenantsync response). +- **Slice 2 (agent):** the apply-bridge — on the desired-state pbs block: ensure wg-felhom up → verify the + PBS fingerprint → consume the token → **root wrapper (sudoers-pinned, secret on stdin, §2b)** runs + `pvesm add pbs felhom-pbs … --encryption-key autogen` → self-grant `/storage/felhom-pbs` via the pinned + `pveum acl modify` (dual-grant: user + token) → `pvesm status` reachability probe → capability report clears + `wg-handshake-read` degradation. Idempotent re-apply = `pvesm set` only (never remove — K, §4). Marker + + fail-safe per the offsite bridge precedent; consume-once + argv/log-hygiene red-proofs mandatory. + **Naming note:** the epic's id `felhom-pbs` already exists on the DEMO host (the LAN dev PBS) — fresh + customer boxes (Peti) are clean; a demo migration needs a rename/coexistence decision first. +- **Slice 3:** DR schedule + recipe coords + monitoring wiring. Much exists: the agent's PBS verify loop + + tier-aware unattended restore-test (S4.1) already run against pbs-type storages on demo; hub-side needs the + event wiring only (the restic `OffsiteChecker` is offbox-specific). +- **Sequencing:** after slice 2 lands on Peti (floor + agent update): his box gains `felhom-pbs` + K → **the + standard ceremony one-liner** seals K + identity under one fresh R → auto-confirm → arc closed. + +## 6. Teardown + environment observations + +**Teardown verified:** demo — spike storage entry, `.enc`/`.pw`, both spike ACLs, sudoers drop-in, `/tmp` +artifacts, staged secret all gone (`pveum acl list` + storage.cfg grep clean; the surviving `spike` grep hit +is `datastore felhom-spike`, the real entry's datastore name). 180 — spike namespace+snapshots, token, ACLs, +staged secrets gone; only `felhom@pbs!n100` remains. ep0 — spike namespace+snapshots, both tokens, all spike +ACLs, `/tmp/spikewrite` gone; only the real `demo-felhom-01` tenancy remains. Demo's real `felhom-offsite` + +`felhom-pbs` storages re-verified **active** post-spike. + +**Flags for Viktor (pre-existing, not touched):** +1. **ep0 has an orphan-looking `root@pam!spike` token + `DatastoreAdmin` ACL on `/datastore/scratch`** — from + the S4-era spikes; cleanup candidate. +2. **The demo host has NO `/storage/felhom-pbs` or `/storage/felhom-offsite` FelhomAgentStore grants today** + (only `local` + `local-lvm`) — the §4b step-3 grants are absent, most likely the token-remove-purges-ACL + gotcha. Backups demonstrably still work (the verify loop is green), so either the grants are not actually + load-bearing for the current vzdump path or something re-asserts lazily — worth a deliberate re-assert + + a check of which ops actually need them before slice 2 codifies the self-grant. +3. The LAN PBS (180) real token `felhom@pbs!n100` holds **datastore-wide `DatastoreAdmin`** — dev-grade, + much broader than ep0's ns-scoped `DatastoreBackup` model; fine for dev, don't copy the pattern.