diff --git a/CONTEXT.md b/CONTEXT.md index cba24db..35afbf2 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,6 +3,17 @@ > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. +- **2026-07-10 — PBS DR tier SLICE 1 SHIPPED+LIVE (hub v0.44.0): ep0 tenantsync surface + hub + provisioning.** `felhom-tenantsync` v1.0.0 installed on ep0 (second forced-command key, peersync + untouched — hashes asserted; smoke caught + fixed the client `{"data":[…]}` json wrap; orphan + `root@pam!spike` cleaned). Hub: `internal/tenantsync` (pinned-SSH twin of wgsync; errors never + embed stdout), config-form "PBS DR tier (ep0)" section → WG-peer check (fail-closed) → provision → + `host_pbs_secrets` consume-once (agent fetches `POST /api/v1/hosts/{id}/pbs/consume-token` — + note: /hosts/, not the task's /host/) → `pbs_dr` descriptor into HOST desired_json + gen bump; + idempotent re-save; Re-issue button (F4 twin). Red-proofs: consume-once / fail-closed / + idempotency all mutation-proven. **NOT live-provisioned — slice 2 (agent bridge) + supervised + Peti enable pending.** Runbook §10 addendum in offsite-endpoint.md. Commits ce6a566 + 32ca1f6. + - **2026-07-10 — SPIKE: PBS DR tier auto-provisioning SLICE 0 — all three mechanisms SETTLED.** Record: `documentation/audits/SPIKE-pbs-tier-provisioning-2026-07-10.md`. (1) PVE token storage-create **impossible** (create/modify/delete all check `/storage` root) → **sudoers vector proven** (pinned-id alias, cycle green as diff --git a/REPORT.md b/REPORT.md index fee9747..dd13aa6 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,34 +2,59 @@ > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md). -## SPIKE — PBS DR tier auto-provisioning, SLICE 0: the three load-bearing mechanisms — 2026-07-10 +## TASK — PBS DR tier SLICE 1: ep0 tenantsync surface + hub provisioning (hub v0.44.0) — 2026-07-10 -**No production code — findings record at -`documentation/audits/SPIKE-pbs-tier-provisioning-2026-07-10.md`.** All three unproven mechanisms of the -"PBS DR tier" epic got empirical answers; every probe artifact was `spike-*`-named and torn down (asserted); -the demo's real `felhom-offsite`/`felhom-pbs` storages re-verified active afterwards. Viktor approved the -one 🛑 (the ep0 touch) mid-run. +**LIVE on k3s (Synced/Healthy, image 0.44.0, startup logs `WG peer-sync enabled` + `PBS DR tenantsync +enabled`). NOT live-provisioned — no real customer was provisioned from this task; slice 2 (the agent +apply-bridge) + the supervised Peti enable are pending.** Builds on SPIKE-pbs-tier-provisioning +(00afadc). Commits: ce6a566 (code) + 32ca1f6 (live fix + image bump). -- **Probe 1 (demo host): path-scoped token create is IMPOSSIBLE.** `POST/PUT/DELETE /storage*` all 403 with - `Permission check failed (/storage, Datastore.Allocate)` — the whole entry-lifecycle CRUD checks the - `/storage` ROOT; `/storage/` ACLs govern usage only. **Probe 1b settled the fallback:** a pinned-id - sudoers alias (`pvesm add/set/remove `, `pveum acl modify /storage/`) — full cycle green as the - `felhom-agent` user via `sudo -n`, unpinned id denied. Gotcha: sudo logs argv → the token secret must go - through a stdin-reading root wrapper, never `sudo pvesm add --password …`. -- **Probe 2 (ep0, 🛑): the tenancy op-set + secret transport PROVEN.** Recon recorded the surface (one - `restrict`+forced-command key, one sudoers line; the peersync user can sudo nothing else — live-proven). - Throwaway `spike-tenant` on the real `felhom-offsite`: namespace + privsep token + dual-grant, the token - secret returned over SSH stdout (36 bytes, never persisted on ep0, never displayed), S4 isolation asserts - green incl. a real cross-tenant WRITE 403. Ordering gotchas recorded (token-before-ACL; delete-token purges - its ACLs; no `--output-format` on generate-token). Slice-1 recommendation: a second forced-command surface - (`felhom-tenantsync`, own keypair + sudoers line), peersync untouched. -- **Probe 3 (demo host): `pvesm add pbs … --encryption-key autogen` works end-to-end.** Entry active, K born - at `/etc/pve/priv/storage/.enc` (0600, = `PBSEncKeyPath` on demo), vzdump of a small guest lands as - **ciphertext** PBS-side (`crypt-mode: encrypt` — the doc-06 property), `escrow-create` would find K. - **⚠ `pvesm remove` DELETES the `.enc` (K!) + `.pw`** → the epic's re-apply must be `pvesm set`-only. -- **Flags surfaced (pre-existing):** orphan `root@pam!spike` DatastoreAdmin on ep0 `/datastore/scratch`; - the demo host is missing the §4b step-3 `/storage/felhom-pbs|felhom-offsite` FelhomAgentStore grants - (backups still green — needs a deliberate re-assert before slice 2 codifies the self-grant); 180's - `felhom@pbs!n100` is datastore-wide DatastoreAdmin (dev-grade, don't copy). -- **Epic slice map** (hub tenantsync + consume-once → agent apply-bridge with stdin-wrapper + set-only - re-apply → DR schedule/monitoring → Peti ceremony one-liner) in the record §5. +### What shipped +- **ep0 surface (🛑 Viktor-approved, installed + smoked live):** `scripts/felhom-tenantsync.sh` v1.0.0 + as a SECOND forced-command key for `felhom-peersync` (own sudoers drop-in; **peersync script + + sudoers sha256 asserted byte-identical before/after**). Ops: `provision` (existing token → typed + `token_exists`; own-ns self-check with one regen retry then rollback), `reissue` (delete→recreate→ + re-grant — delete purges token ACLs, the spike fact), `fingerprint`. Secret hygiene: the token + secret rides stdout ONLY (all tool stdout → stderr; hub error paths never embed stdout). + Smoke (throwaway `spike-smoke`, torn down): all ops green; **live fix caught by the smoke** — + `proxmox-backup-client --output-format json` wraps as `{"data":[…]}` → ns-exists check fixed to + `(.data // .)` and reinstalled before the hub ever used the surface. + **Orphan cleanup (spike flag #1):** `root@pam!spike` token + `/datastore/scratch` DatastoreAdmin + ACL removed; ep0 now holds exactly the real `demo-felhom-01` tenancy. +- **Hub `internal/tenantsync`:** wgsync-twin SSH client (pinned host key, constrained + HostKeyAlgorithms, per-op JSON exec, typed `ErrTokenExists`). Contract test + `TestErrors_NeverEmbedStdout` pins the no-stdout-in-errors rule. +- **Provisioning flow (`web/pbsdr.go`):** config-form section "PBS DR tier (ep0)" → on enable+save + (detached-ctx atom): WG peer existence check (the agent self-registers its key; absence + fail-closes with "has not reported a WG key yet") → tenantsync provision → secret stored + consume-once HOST-scoped (`host_pbs_secrets`, the `one_time_secrets` twin) → non-secret `pbs_dr` + descriptor (`enabled, storage_id, pbs_tunnel_ip, datastore, namespace, token_id, fingerprint`) + merged into the HOST `desired_json` + `SetHostDesired` generation bump. Already-provisioned + re-save = success-no-op; disable = `enabled:false` descriptor, tenancy kept; **Re-issue PBS + credentials** button = the F4 precedent (fresh consume-once secret + bump). +- **Consume endpoint:** `POST /api/v1/hosts/{id}/pbs/consume-token` — per-host key self-scoped + (global = operator recovery); 200 exactly once → 404; a foreign key's 403 does NOT burn the + secret. *(Deviation: task spec wrote `/host/{id}/…`; implemented under `/hosts/` matching every + other agent-facing route — slice 2 must use the `/hosts/` form.)* +- **Deploy:** `TENANTSYNC_SSH_KEY_FILE` env + optional `Secret/tenantsync` mount (key generated on + 180, moved into the Secret, local copy shredded); same endpoint addr + pinned host key as peersync. + +### Tests + red-proofs (all run, mutations reverted, gate green) +- Happy path: descriptor lands in desired_json + gen bump exactly 1; secret consumable once and + absent from ConfigJSON/desired-state/captured logs (grep-asserts). +- **Red-proof consume-once:** dropping the `consumed_at` UPDATE → store test re-served `"secret-1"`, + API test returned 200 with the secret again → both FAIL as designed. +- **Red-proof fail-closed:** swallowing the applyPBSDR error in handleConfigUpdate → all four + fail-closed subtests FAIL with 303 (half-enabled save). +- **Red-proof idempotency:** dropping the already-provisioned short-circuit → re-save re-keyed the + token (2 provision calls), staged a fresh secret, bumped spuriously → FAIL. +- Also covered: foreign-host 403 non-burning, reissue rotation (old secret dead), disable no-op on + second save, storage-id edit updates descriptor without re-provision, tenantsync client + round-trip against an in-process SSH server incl. wrong-host-key refusal. +- UI leg: render tests (`TestPBSDR_FormRendersState` + suite) — the hub UI is password-gated, so + render tests are the verification method per repo convention. + +### Operator follow-ups +- Slice 2 (agent apply-bridge) then the supervised Peti enable (floor + agent update first). +- The demo host's missing §4b step-3 storage ACL grants (spike flag #2) — re-assert before slice 2 + codifies the self-grant. diff --git a/documentation/runbooks/offsite-endpoint.md b/documentation/runbooks/offsite-endpoint.md index b32f569..5739336 100644 --- a/documentation/runbooks/offsite-endpoint.md +++ b/documentation/runbooks/offsite-endpoint.md @@ -346,3 +346,50 @@ in `agent.json`. The agent renders the config, claims a port, writes `felhom-op` from `oob_operator_ssh_key`, and fills the belt sets. Verify: from the operator peer, `ssh -p felhom-op@`; the host belt drops any non-operator tunnel source. Re-verify PBS (`pvesm status --storage felhom-offsite` on the box) after the forward change. + +## 10. The felhom-tenantsync surface (PBS DR tier SLICE 1 — installed 2026-07-10) + +The hub's SECOND forced-command surface on the endpoint: per-customer PBS tenancy ops (namespace + +privilege-separated token + dual-grant, the §4a dance automated). Same low-priv user as peersync, its +OWN keypair + script + sudoers drop-in — **the peersync files are untouched** (one script, one job; +peersync script + sudoers sha256 asserted identical before/after at install). Script source of truth: +`scripts/felhom-tenantsync.sh` (v1.0.0; JSON stdin/stdout, ops `provision`/`reissue`/`fingerprint`, +NO deprovision — data deletion stays a deliberate decision). Hub client: `hub/internal/tenantsync`. + +On the build server (the hub's credential; mirror of step 6): +```sh +ssh-keygen -t ed25519 -f tenantsync-ssh -N "" -C hub-tenancy@felhom +sudo kubectl -n felhom-system create secret generic tenantsync --from-file=key=tenantsync-ssh +shred -u tenantsync-ssh tenantsync-ssh.pub # from here the key exists ONLY in the Secret +``` +(The pinned host key + endpoint address are REUSED from the peersync env — no new hostkey entry. +`manifests/hub.yaml` mounts the Secret at `/etc/hub-secrets/tenantsync/key` via +`TENANTSYNC_SSH_KEY_FILE`; absent Secret → the hub logs "PBS DR tenantsync disabled".) + +On the box, as root: +```sh +# script: stage → strip CRLF if via a Windows checkout → syntax-check → install → THEN activate +tr -d '\r' < felhom-tenantsync.sh > /tmp/fts && bash -n /tmp/fts +install -o root -g root -m 0755 /tmp/fts /usr/local/bin/felhom-tenantsync && rm /tmp/fts +cat > /etc/sudoers.d/felhom-tenantsync <<'EOF' +felhom-peersync ALL=(root) NOPASSWD: /usr/local/bin/felhom-tenantsync +EOF +chmod 0440 /etc/sudoers.d/felhom-tenantsync && visudo -cf /etc/sudoers.d/felhom-tenantsync +printf 'restrict,command="sudo /usr/local/bin/felhom-tenantsync" %s\n' "" \ + >> /home/felhom-peersync/.ssh/authorized_keys +``` + +Verify (the install's smoke, run from the build server with the key before it is shredded — pin the +host key from step 6's on-box read, never keyscan): +- `{"op":"fingerprint"}` → `{"status":"ok","fingerprint":""}` +- `{"op":"provision","customer_id":"spike-smoke"}` → ok-shape; **assert `token_secret` length only, + never print it**; a second provision → `{"code":"token_exists"}`; `reissue` → fresh ok-shape; + unknown op → `bad_request`. Then tear the throwaway tenant down (delete token, ns ACL, namespace + via a transient root admin token — §4a teardown shape). + +Install-run facts (2026-07-10): smoke exposed that `proxmox-backup-client … --output-format json` +wraps output as `{"data":[…]}` (the MANAGER returns bare arrays) — script fixed to `(.data // .)` +before the hub ever used the surface. The §6-era orphan `root@pam!spike` token + its +`/datastore/scratch` DatastoreAdmin ACL were removed in the same session (spike flag #1); after +teardown the endpoint holds exactly the real `demo-felhom-01` tenancy. Hub v0.44.0 logs +`PBS DR tenantsync enabled (endpoint 167.233.158.164:22, user felhom-peersync)` on start.