docs: PBS DR slice-1 runbook §10 addendum + REPORT/CONTEXT (hub v0.44.0 live)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6

REPORT.md

@@ -2,34 +2,59 @@
 
 > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
 
-## SPIKE — PBS DR tier auto-provisioning, SLICE 0: the three load-bearing mechanisms — 2026-07-10
+## TASK — PBS DR tier SLICE 1: ep0 tenantsync surface + hub provisioning (hub v0.44.0) — 2026-07-10
 
-**No production code — findings record at
-`documentation/audits/SPIKE-pbs-tier-provisioning-2026-07-10.md`.** All three unproven mechanisms of the
-"PBS DR tier" epic got empirical answers; every probe artifact was `spike-*`-named and torn down (asserted);
-the demo's real `felhom-offsite`/`felhom-pbs` storages re-verified active afterwards. Viktor approved the
-one 🛑 (the ep0 touch) mid-run.
+**LIVE on k3s (Synced/Healthy, image 0.44.0, startup logs `WG peer-sync enabled` + `PBS DR tenantsync
+enabled`). NOT live-provisioned — no real customer was provisioned from this task; slice 2 (the agent
+apply-bridge) + the supervised Peti enable are pending.** Builds on SPIKE-pbs-tier-provisioning
+(00afadc). Commits: ce6a566 (code) + 32ca1f6 (live fix + image bump).
 
-- **Probe 1 (demo host): path-scoped token create is IMPOSSIBLE.** `POST/PUT/DELETE /storage*` all 403 with
-  `Permission check failed (/storage, Datastore.Allocate)` — the whole entry-lifecycle CRUD checks the
-  `/storage` ROOT; `/storage/<id>` ACLs govern usage only. **Probe 1b settled the fallback:** a pinned-id
-  sudoers alias (`pvesm add/set/remove <id>`, `pveum acl modify /storage/<id>`) — full cycle green as the
-  `felhom-agent` user via `sudo -n`, unpinned id denied. Gotcha: sudo logs argv → the token secret must go
-  through a stdin-reading root wrapper, never `sudo pvesm add --password …`.
-- **Probe 2 (ep0, 🛑): the tenancy op-set + secret transport PROVEN.** Recon recorded the surface (one
-  `restrict`+forced-command key, one sudoers line; the peersync user can sudo nothing else — live-proven).
-  Throwaway `spike-tenant` on the real `felhom-offsite`: namespace + privsep token + dual-grant, the token
-  secret returned over SSH stdout (36 bytes, never persisted on ep0, never displayed), S4 isolation asserts
-  green incl. a real cross-tenant WRITE 403. Ordering gotchas recorded (token-before-ACL; delete-token purges
-  its ACLs; no `--output-format` on generate-token). Slice-1 recommendation: a second forced-command surface
-  (`felhom-tenantsync`, own keypair + sudoers line), peersync untouched.
-- **Probe 3 (demo host): `pvesm add pbs … --encryption-key autogen` works end-to-end.** Entry active, K born
-  at `/etc/pve/priv/storage/<id>.enc` (0600, = `PBSEncKeyPath` on demo), vzdump of a small guest lands as
-  **ciphertext** PBS-side (`crypt-mode: encrypt` — the doc-06 property), `escrow-create` would find K.
-  **⚠ `pvesm remove` DELETES the `.enc` (K!) + `.pw`** → the epic's re-apply must be `pvesm set`-only.
-- **Flags surfaced (pre-existing):** orphan `root@pam!spike` DatastoreAdmin on ep0 `/datastore/scratch`;
-  the demo host is missing the §4b step-3 `/storage/felhom-pbs|felhom-offsite` FelhomAgentStore grants
-  (backups still green — needs a deliberate re-assert before slice 2 codifies the self-grant); 180's
-  `felhom@pbs!n100` is datastore-wide DatastoreAdmin (dev-grade, don't copy).
-- **Epic slice map** (hub tenantsync + consume-once → agent apply-bridge with stdin-wrapper + set-only
-  re-apply → DR schedule/monitoring → Peti ceremony one-liner) in the record §5.
+### What shipped
+- **ep0 surface (🛑 Viktor-approved, installed + smoked live):** `scripts/felhom-tenantsync.sh` v1.0.0
+  as a SECOND forced-command key for `felhom-peersync` (own sudoers drop-in; **peersync script +
+  sudoers sha256 asserted byte-identical before/after**). Ops: `provision` (existing token → typed
+  `token_exists`; own-ns self-check with one regen retry then rollback), `reissue` (delete→recreate→
+  re-grant — delete purges token ACLs, the spike fact), `fingerprint`. Secret hygiene: the token
+  secret rides stdout ONLY (all tool stdout → stderr; hub error paths never embed stdout).
+  Smoke (throwaway `spike-smoke`, torn down): all ops green; **live fix caught by the smoke** —
+  `proxmox-backup-client --output-format json` wraps as `{"data":[…]}` → ns-exists check fixed to
+  `(.data // .)` and reinstalled before the hub ever used the surface.
+  **Orphan cleanup (spike flag #1):** `root@pam!spike` token + `/datastore/scratch` DatastoreAdmin
+  ACL removed; ep0 now holds exactly the real `demo-felhom-01` tenancy.
+- **Hub `internal/tenantsync`:** wgsync-twin SSH client (pinned host key, constrained
+  HostKeyAlgorithms, per-op JSON exec, typed `ErrTokenExists`). Contract test
+  `TestErrors_NeverEmbedStdout` pins the no-stdout-in-errors rule.
+- **Provisioning flow (`web/pbsdr.go`):** config-form section "PBS DR tier (ep0)" → on enable+save
+  (detached-ctx atom): WG peer existence check (the agent self-registers its key; absence
+  fail-closes with "has not reported a WG key yet") → tenantsync provision → secret stored
+  consume-once HOST-scoped (`host_pbs_secrets`, the `one_time_secrets` twin) → non-secret `pbs_dr`
+  descriptor (`enabled, storage_id, pbs_tunnel_ip, datastore, namespace, token_id, fingerprint`)
+  merged into the HOST `desired_json` + `SetHostDesired` generation bump. Already-provisioned
+  re-save = success-no-op; disable = `enabled:false` descriptor, tenancy kept; **Re-issue PBS
+  credentials** button = the F4 precedent (fresh consume-once secret + bump).
+- **Consume endpoint:** `POST /api/v1/hosts/{id}/pbs/consume-token` — per-host key self-scoped
+  (global = operator recovery); 200 exactly once → 404; a foreign key's 403 does NOT burn the
+  secret. *(Deviation: task spec wrote `/host/{id}/…`; implemented under `/hosts/` matching every
+  other agent-facing route — slice 2 must use the `/hosts/` form.)*
+- **Deploy:** `TENANTSYNC_SSH_KEY_FILE` env + optional `Secret/tenantsync` mount (key generated on
+  180, moved into the Secret, local copy shredded); same endpoint addr + pinned host key as peersync.
+
+### Tests + red-proofs (all run, mutations reverted, gate green)
+- Happy path: descriptor lands in desired_json + gen bump exactly 1; secret consumable once and
+  absent from ConfigJSON/desired-state/captured logs (grep-asserts).
+- **Red-proof consume-once:** dropping the `consumed_at` UPDATE → store test re-served `"secret-1"`,
+  API test returned 200 with the secret again → both FAIL as designed.
+- **Red-proof fail-closed:** swallowing the applyPBSDR error in handleConfigUpdate → all four
+  fail-closed subtests FAIL with 303 (half-enabled save).
+- **Red-proof idempotency:** dropping the already-provisioned short-circuit → re-save re-keyed the
+  token (2 provision calls), staged a fresh secret, bumped spuriously → FAIL.
+- Also covered: foreign-host 403 non-burning, reissue rotation (old secret dead), disable no-op on
+  second save, storage-id edit updates descriptor without re-provision, tenantsync client
+  round-trip against an in-process SSH server incl. wrong-host-key refusal.
+- UI leg: render tests (`TestPBSDR_FormRendersState` + suite) — the hub UI is password-gated, so
+  render tests are the verification method per repo convention.
+
+### Operator follow-ups
+- Slice 2 (agent apply-bridge) then the supervised Peti enable (floor + agent update first).
+- The demo host's missing §4b step-3 storage ACL grants (spike flag #2) — re-assert before slice 2
+  codifies the self-grant.