docs: PBS DR slice-1 runbook §10 addendum + REPORT/CONTEXT (hub v0.44.0 live)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6

CONTEXT.md

@@ -3,6 +3,17 @@
 > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
 > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
 
+- **2026-07-10 — PBS DR tier SLICE 1 SHIPPED+LIVE (hub v0.44.0): ep0 tenantsync surface + hub
+  provisioning.** `felhom-tenantsync` v1.0.0 installed on ep0 (second forced-command key, peersync
+  untouched — hashes asserted; smoke caught + fixed the client `{"data":[…]}` json wrap; orphan
+  `root@pam!spike` cleaned). Hub: `internal/tenantsync` (pinned-SSH twin of wgsync; errors never
+  embed stdout), config-form "PBS DR tier (ep0)" section → WG-peer check (fail-closed) → provision →
+  `host_pbs_secrets` consume-once (agent fetches `POST /api/v1/hosts/{id}/pbs/consume-token` —
+  note: /hosts/, not the task's /host/) → `pbs_dr` descriptor into HOST desired_json + gen bump;
+  idempotent re-save; Re-issue button (F4 twin). Red-proofs: consume-once / fail-closed /
+  idempotency all mutation-proven. **NOT live-provisioned — slice 2 (agent bridge) + supervised
+  Peti enable pending.** Runbook §10 addendum in offsite-endpoint.md. Commits ce6a566 + 32ca1f6.
+
 - **2026-07-10 — SPIKE: PBS DR tier auto-provisioning SLICE 0 — all three mechanisms SETTLED.** Record:
   `documentation/audits/SPIKE-pbs-tier-provisioning-2026-07-10.md`. (1) PVE token storage-create **impossible**
   (create/modify/delete all check `/storage` root) → **sudoers vector proven** (pinned-id alias, cycle green as

REPORT.md

@@ -2,34 +2,59 @@
 
 > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
 
-## SPIKE — PBS DR tier auto-provisioning, SLICE 0: the three load-bearing mechanisms — 2026-07-10
+## TASK — PBS DR tier SLICE 1: ep0 tenantsync surface + hub provisioning (hub v0.44.0) — 2026-07-10
 
-**No production code — findings record at
-`documentation/audits/SPIKE-pbs-tier-provisioning-2026-07-10.md`.** All three unproven mechanisms of the
-"PBS DR tier" epic got empirical answers; every probe artifact was `spike-*`-named and torn down (asserted);
-the demo's real `felhom-offsite`/`felhom-pbs` storages re-verified active afterwards. Viktor approved the
-one 🛑 (the ep0 touch) mid-run.
+**LIVE on k3s (Synced/Healthy, image 0.44.0, startup logs `WG peer-sync enabled` + `PBS DR tenantsync
+enabled`). NOT live-provisioned — no real customer was provisioned from this task; slice 2 (the agent
+apply-bridge) + the supervised Peti enable are pending.** Builds on SPIKE-pbs-tier-provisioning
+(00afadc). Commits: ce6a566 (code) + 32ca1f6 (live fix + image bump).
 
-- **Probe 1 (demo host): path-scoped token create is IMPOSSIBLE.** `POST/PUT/DELETE /storage*` all 403 with
-  `Permission check failed (/storage, Datastore.Allocate)` — the whole entry-lifecycle CRUD checks the
-  `/storage` ROOT; `/storage/<id>` ACLs govern usage only. **Probe 1b settled the fallback:** a pinned-id
-  sudoers alias (`pvesm add/set/remove <id>`, `pveum acl modify /storage/<id>`) — full cycle green as the
-  `felhom-agent` user via `sudo -n`, unpinned id denied. Gotcha: sudo logs argv → the token secret must go
-  through a stdin-reading root wrapper, never `sudo pvesm add --password …`.
-- **Probe 2 (ep0, 🛑): the tenancy op-set + secret transport PROVEN.** Recon recorded the surface (one
-  `restrict`+forced-command key, one sudoers line; the peersync user can sudo nothing else — live-proven).
-  Throwaway `spike-tenant` on the real `felhom-offsite`: namespace + privsep token + dual-grant, the token
-  secret returned over SSH stdout (36 bytes, never persisted on ep0, never displayed), S4 isolation asserts
-  green incl. a real cross-tenant WRITE 403. Ordering gotchas recorded (token-before-ACL; delete-token purges
-  its ACLs; no `--output-format` on generate-token). Slice-1 recommendation: a second forced-command surface
-  (`felhom-tenantsync`, own keypair + sudoers line), peersync untouched.
-- **Probe 3 (demo host): `pvesm add pbs … --encryption-key autogen` works end-to-end.** Entry active, K born
-  at `/etc/pve/priv/storage/<id>.enc` (0600, = `PBSEncKeyPath` on demo), vzdump of a small guest lands as
-  **ciphertext** PBS-side (`crypt-mode: encrypt` — the doc-06 property), `escrow-create` would find K.
-  **⚠ `pvesm remove` DELETES the `.enc` (K!) + `.pw`** → the epic's re-apply must be `pvesm set`-only.
-- **Flags surfaced (pre-existing):** orphan `root@pam!spike` DatastoreAdmin on ep0 `/datastore/scratch`;
-  the demo host is missing the §4b step-3 `/storage/felhom-pbs|felhom-offsite` FelhomAgentStore grants
-  (backups still green — needs a deliberate re-assert before slice 2 codifies the self-grant); 180's
-  `felhom@pbs!n100` is datastore-wide DatastoreAdmin (dev-grade, don't copy).
-- **Epic slice map** (hub tenantsync + consume-once → agent apply-bridge with stdin-wrapper + set-only
-  re-apply → DR schedule/monitoring → Peti ceremony one-liner) in the record §5.
+### What shipped
+- **ep0 surface (🛑 Viktor-approved, installed + smoked live):** `scripts/felhom-tenantsync.sh` v1.0.0
+  as a SECOND forced-command key for `felhom-peersync` (own sudoers drop-in; **peersync script +
+  sudoers sha256 asserted byte-identical before/after**). Ops: `provision` (existing token → typed
+  `token_exists`; own-ns self-check with one regen retry then rollback), `reissue` (delete→recreate→
+  re-grant — delete purges token ACLs, the spike fact), `fingerprint`. Secret hygiene: the token
+  secret rides stdout ONLY (all tool stdout → stderr; hub error paths never embed stdout).
+  Smoke (throwaway `spike-smoke`, torn down): all ops green; **live fix caught by the smoke** —
+  `proxmox-backup-client --output-format json` wraps as `{"data":[…]}` → ns-exists check fixed to
+  `(.data // .)` and reinstalled before the hub ever used the surface.
+  **Orphan cleanup (spike flag #1):** `root@pam!spike` token + `/datastore/scratch` DatastoreAdmin
+  ACL removed; ep0 now holds exactly the real `demo-felhom-01` tenancy.
+- **Hub `internal/tenantsync`:** wgsync-twin SSH client (pinned host key, constrained
+  HostKeyAlgorithms, per-op JSON exec, typed `ErrTokenExists`). Contract test
+  `TestErrors_NeverEmbedStdout` pins the no-stdout-in-errors rule.
+- **Provisioning flow (`web/pbsdr.go`):** config-form section "PBS DR tier (ep0)" → on enable+save
+  (detached-ctx atom): WG peer existence check (the agent self-registers its key; absence
+  fail-closes with "has not reported a WG key yet") → tenantsync provision → secret stored
+  consume-once HOST-scoped (`host_pbs_secrets`, the `one_time_secrets` twin) → non-secret `pbs_dr`
+  descriptor (`enabled, storage_id, pbs_tunnel_ip, datastore, namespace, token_id, fingerprint`)
+  merged into the HOST `desired_json` + `SetHostDesired` generation bump. Already-provisioned
+  re-save = success-no-op; disable = `enabled:false` descriptor, tenancy kept; **Re-issue PBS
+  credentials** button = the F4 precedent (fresh consume-once secret + bump).
+- **Consume endpoint:** `POST /api/v1/hosts/{id}/pbs/consume-token` — per-host key self-scoped
+  (global = operator recovery); 200 exactly once → 404; a foreign key's 403 does NOT burn the
+  secret. *(Deviation: task spec wrote `/host/{id}/…`; implemented under `/hosts/` matching every
+  other agent-facing route — slice 2 must use the `/hosts/` form.)*
+- **Deploy:** `TENANTSYNC_SSH_KEY_FILE` env + optional `Secret/tenantsync` mount (key generated on
+  180, moved into the Secret, local copy shredded); same endpoint addr + pinned host key as peersync.
+
+### Tests + red-proofs (all run, mutations reverted, gate green)
+- Happy path: descriptor lands in desired_json + gen bump exactly 1; secret consumable once and
+  absent from ConfigJSON/desired-state/captured logs (grep-asserts).
+- **Red-proof consume-once:** dropping the `consumed_at` UPDATE → store test re-served `"secret-1"`,
+  API test returned 200 with the secret again → both FAIL as designed.
+- **Red-proof fail-closed:** swallowing the applyPBSDR error in handleConfigUpdate → all four
+  fail-closed subtests FAIL with 303 (half-enabled save).
+- **Red-proof idempotency:** dropping the already-provisioned short-circuit → re-save re-keyed the
+  token (2 provision calls), staged a fresh secret, bumped spuriously → FAIL.
+- Also covered: foreign-host 403 non-burning, reissue rotation (old secret dead), disable no-op on
+  second save, storage-id edit updates descriptor without re-provision, tenantsync client
+  round-trip against an in-process SSH server incl. wrong-host-key refusal.
+- UI leg: render tests (`TestPBSDR_FormRendersState` + suite) — the hub UI is password-gated, so
+  render tests are the verification method per repo convention.
+
+### Operator follow-ups
+- Slice 2 (agent apply-bridge) then the supervised Peti enable (floor + agent update first).
+- The demo host's missing §4b step-3 storage ACL grants (spike flag #2) — re-assert before slice 2
+  codifies the self-grant.

documentation/runbooks/offsite-endpoint.md

@@ -346,3 +346,50 @@ in `agent.json`. The agent renders the config, claims a port, writes `felhom-op`
 from `oob_operator_ssh_key`, and fills the belt sets. Verify: from the operator peer,
 `ssh -p <claimed-port> felhom-op@<box tunnel IP>`; the host belt drops any non-operator tunnel source.
 Re-verify PBS (`pvesm status --storage felhom-offsite` on the box) after the forward change.
+
+## 10. The felhom-tenantsync surface (PBS DR tier SLICE 1 — installed 2026-07-10)
+
+The hub's SECOND forced-command surface on the endpoint: per-customer PBS tenancy ops (namespace +
+privilege-separated token + dual-grant, the §4a dance automated). Same low-priv user as peersync, its
+OWN keypair + script + sudoers drop-in — **the peersync files are untouched** (one script, one job;
+peersync script + sudoers sha256 asserted identical before/after at install). Script source of truth:
+`scripts/felhom-tenantsync.sh` (v1.0.0; JSON stdin/stdout, ops `provision`/`reissue`/`fingerprint`,
+NO deprovision — data deletion stays a deliberate decision). Hub client: `hub/internal/tenantsync`.
+
+On the build server (the hub's credential; mirror of step 6):
+```sh
+ssh-keygen -t ed25519 -f tenantsync-ssh -N "" -C hub-tenancy@felhom
+sudo kubectl -n felhom-system create secret generic tenantsync --from-file=key=tenantsync-ssh
+shred -u tenantsync-ssh tenantsync-ssh.pub   # from here the key exists ONLY in the Secret
+```
+(The pinned host key + endpoint address are REUSED from the peersync env — no new hostkey entry.
+`manifests/hub.yaml` mounts the Secret at `/etc/hub-secrets/tenantsync/key` via
+`TENANTSYNC_SSH_KEY_FILE`; absent Secret → the hub logs "PBS DR tenantsync disabled".)
+
+On the box, as root:
+```sh
+# script: stage → strip CRLF if via a Windows checkout → syntax-check → install → THEN activate
+tr -d '\r' < felhom-tenantsync.sh > /tmp/fts && bash -n /tmp/fts
+install -o root -g root -m 0755 /tmp/fts /usr/local/bin/felhom-tenantsync && rm /tmp/fts
+cat > /etc/sudoers.d/felhom-tenantsync <<'EOF'
+felhom-peersync ALL=(root) NOPASSWD: /usr/local/bin/felhom-tenantsync
+EOF
+chmod 0440 /etc/sudoers.d/felhom-tenantsync && visudo -cf /etc/sudoers.d/felhom-tenantsync
+printf 'restrict,command="sudo /usr/local/bin/felhom-tenantsync" %s\n' "<tenantsync pubkey line>" \
+  >> /home/felhom-peersync/.ssh/authorized_keys
+```
+
+Verify (the install's smoke, run from the build server with the key before it is shredded — pin the
+host key from step 6's on-box read, never keyscan):
+- `{"op":"fingerprint"}` → `{"status":"ok","fingerprint":"<PBS cert sha256>"}`
+- `{"op":"provision","customer_id":"spike-smoke"}` → ok-shape; **assert `token_secret` length only,
+  never print it**; a second provision → `{"code":"token_exists"}`; `reissue` → fresh ok-shape;
+  unknown op → `bad_request`. Then tear the throwaway tenant down (delete token, ns ACL, namespace
+  via a transient root admin token — §4a teardown shape).
+
+Install-run facts (2026-07-10): smoke exposed that `proxmox-backup-client … --output-format json`
+wraps output as `{"data":[…]}` (the MANAGER returns bare arrays) — script fixed to `(.data // .)`
+before the hub ever used the surface. The §6-era orphan `root@pam!spike` token + its
+`/datastore/scratch` DatastoreAdmin ACL were removed in the same session (spike flag #1); after
+teardown the endpoint holds exactly the real `demo-felhom-01` tenancy. Hub v0.44.0 logs
+`PBS DR tenantsync enabled (endpoint 167.233.158.164:22, user felhom-peersync)` on start.