# RUNBOOK — offsite endpoint provisioning (WG server + PBS + peersync surface)

> **What this creates:** the Felhom offsite endpoint (doc 06 §5) on a blank Debian 13 VM — the
> WG server on **443/UDP**, the offsite PBS with a `/srv` datastore, a drop-everything-else
> firewall, and the hub-driven `felhom-peersync` reconcile surface. Re-running it on a fresh VM
> re-creates the endpoint from nothing (that is the DR story, step 8).
>
> **Validated:** 2026-07-03 on the dev/test endpoint `felhom-hetzner` (Hetzner CX23 **at the time — rescaled
> to a CX33, 4 vCPU / 8 GB RAM, on 2026-08-03; the 40 GB local disk is unchanged**, Debian 13,
> `167.233.158.164` / `2a01:4f8:1c16:7aa1::1`) with hub v0.32.0. The production endpoint is a
> later re-run of this runbook on a production VM.
>
> Companion piece: the hub side (env + Secret) is `manifests/hub.yaml` + step 6/7 below.
> The peersync script's source of truth is `scripts/felhom-peersync.sh` (v1.0.0).

Parameters used throughout (adjust for a new endpoint):

| Param | Dev value |
|---|---|
| VM | `felhom-hetzner`, `167.233.158.164` (**NOT the live jarrs.eu box** — different server) |
| DNS name | `ep0.felhom.eu` (A + AAAA, **DNS-only / grey-cloud** — proxying breaks WG/UDP + PBS) |
| WG port | **443/udp** (spike P4: handshakes + carries identically to 51820) |
| Tunnel subnet | `10.77.0.0/24`; endpoint in-tunnel IP `10.77.0.1` |
| PBS datastore | `felhom-offsite` at `/srv/pbs-felhom` (**never under `/root`** — the `backup` user cannot traverse it; spike gotcha) |

## 0. Preconditions (operator)

- [ ] The VM is powered on and root-SSH-reachable.
- [ ] DNS `ep0.felhom.eu` → A `167.233.158.164` + AAAA `2a01:4f8:1c16:7aa1::1`, **grey-cloud**.
      Verify: `nslookup ep0.felhom.eu 1.1.1.1` returns the A record. **The AAAA must be the
      box's full `::1` address** — a bare `...:7aa1::` (the subnet zero address) resolves but
      points at nothing (live-run finding). Home-resolver propagation can lag public DNS by
      minutes — a client-side `wg-quick up` that fails to resolve right after record creation
      just needs a retry.
- [ ] Sanity: `ssh root@167.233.158.164 hostname` → `felhom-hetzner` (the throwaway box, **CX33 since 2026-08-03**), not
      any production box.

## 1. Base (on the box, as root)

```sh
apt update && DEBIAN_FRONTEND=noninteractive apt full-upgrade -y
printf 'PasswordAuthentication no\n' > /etc/ssh/sshd_config.d/90-felhom.conf && systemctl reload ssh
sshd -T | grep -i '^passwordauthentication'   # must print: passwordauthentication no
```
(The explicit drop-in beats auditing defaults — Hetzner images have no explicit line at all.)

## 2. WG server

```sh
apt install -y wireguard-tools          # module ships in the stock Debian 13 kernel
umask 077
wg genkey > /etc/wireguard/wg0.key      # server private key — NEVER displayed/copied anywhere
wg pubkey < /etc/wireguard/wg0.key      # RECORD this pubkey → hub registration (step 7)
cat > /etc/wireguard/wg0.conf.head <<EOF
[Interface]
Address = 10.77.0.1/24
ListenPort = 443
MTU = 1420
PrivateKey = $(cat /etc/wireguard/wg0.key)
EOF
chmod 600 /etc/wireguard/wg0.conf.head
cp -p /etc/wireguard/wg0.conf.head /etc/wireguard/wg0.conf   # zero peers initially
systemctl enable --now wg-quick@wg0
wg show wg0
```
Verify: `wg show wg0` → `listening port: 443`, no peers. **No `SaveConfig`** anywhere — the
peersync script owns persistence (head-file + generated peers; `wg-quick save` would rewrite
the conf nondeterministically).

> **NEVER run `wg show <if> dump` in a logged/shared session** — field 1 of its interface line
> is the **private key**. Plain `wg show` prints only the public key. (Live-run lesson: a
> `dump | cut` leaked the first server key into a session log → the key was rotated on the
> spot. If it happens to you: regenerate the keypair, rebuild `wg0.conf.head` + `wg0.conf`,
> restart `wg-quick@wg0`, re-`PUT` the new pubkey to the hub — 2 minutes, zero peer downtime
> beyond the restart.)

## 3. Firewall (nftables, v4+v6)

```sh
cat > /etc/nftables.conf <<'EOF'
#!/usr/sbin/nft -f
flush ruleset
table inet filter {
    chain input {
        type filter hook input priority 0; policy drop;
        ct state established,related accept
        iif "lo" accept
        tcp dport 22 accept
        udp dport 443 accept
        tcp dport 8007 iifname "wg0" accept
        ip protocol icmp accept
        meta l4proto ipv6-icmp accept
    }
    chain forward {
        type filter hook forward priority 0; policy drop;
    }
}
EOF
systemctl enable --now nftables
nft list ruleset | head -20
sysctl net.ipv4.ip_forward
```
Verify (doc 06 §4.5 — no inter-peer routing by topology): forward chain policy **drop**,
`net.ipv4.ip_forward = 0`. From an outside vantage: `ssh` (22) connects; **8007 publicly
CLOSED** — `curl -sk --max-time 8 https://<public-ip>:8007/` must time out (the spike-P2b
firewall proof, repeated). 443/udp is proven by the first WG handshake in step 7's validation.

## 4. PBS

```sh
curl -fsSL https://enterprise.proxmox.com/debian/proxmox-release-trixie.gpg \
  -o /usr/share/keyrings/proxmox-archive-keyring.gpg
echo 'deb [signed-by=/usr/share/keyrings/proxmox-archive-keyring.gpg] http://download.proxmox.com/debian/pbs trixie pbs-no-subscription' \
  > /etc/apt/sources.list.d/pbs.list
apt update && DEBIAN_FRONTEND=noninteractive apt install -y --no-install-recommends \
  proxmox-backup-server proxmox-backup-client   # client is a SEPARATE package (namespace ops need it)
rm -f /etc/apt/sources.list.d/pbs-enterprise.sources   # the install adds the ENTERPRISE repo → 401s every apt update
apt update
proxmox-backup-manager datastore create felhom-offsite /srv/pbs-felhom
# prove namespace ops (per-customer tenancy, doc 06 D5) with a THROWAWAY token, then remove it
# (root@pam has no password on a cloud image; S4 wires the real per-customer tokens):
umask 077
proxmox-backup-manager user generate-token root@pam nsproof > /root/.nsproof
proxmox-backup-manager acl update /datastore/felhom-offsite DatastoreAdmin --auth-id 'root@pam!nsproof'
export PBS_PASSWORD="$(tr -d '\n' < /root/.nsproof | sed -n 's/.*"value": "\([^"]*\)".*/\1/p')"
export PBS_FINGERPRINT="$(proxmox-backup-manager cert info | grep -i fingerprint | awk '{print $3}')"
proxmox-backup-client namespace create test-ns --repository 'root@pam!nsproof@localhost:felhom-offsite'
proxmox-backup-client namespace delete test-ns --repository 'root@pam!nsproof@localhost:felhom-offsite'
proxmox-backup-manager user delete-token root@pam nsproof && rm -f /root/.nsproof
```
Verify: `systemctl is-active proxmox-backup proxmox-backup-proxy` → both `active`; the
datastore parent is `/srv` (world-traversable — a datastore under `/root` fails with
`Permission denied` for the `backup` user). GUI: reachable ONLY via SSH port-forward
(`ssh -L 8007:127.0.0.1:8007 root@<box>`) or through the tunnel — by design, no public 8007.

## 4a. Per-customer PBS tenancy (S4 — live-confirmed 2026-07-04 on felhom-hetzner)

Each customer box gets ITS OWN namespace + a privilege-separated token scoped to that namespace,
on the single shared `felhom-offsite` datastore (doc 06 D5). One shared PBS user `felhom@pbs`
holds a per-customer token each; cross-tenant isolation is enforced by the per-token ACL.

**On the endpoint, per customer `<ns>` (e.g. `demo-felhom-01` = the box's `host_id`):**
```sh
DS=felhom-offsite
# 1. namespace — created via a THROWAWAY root admin token (root@pam has no password on the cloud
#    image; proxmox-backup-manager has NO `namespace` subcommand — namespaces are client-side):
proxmox-backup-manager user generate-token root@pam nsadmin > /root/.nsadmin.raw   # umask 077
NSADMIN=$(sed -n 's/.*"value": "\([^"]*\)".*/\1/p' /root/.nsadmin.raw); rm -f /root/.nsadmin.raw
proxmox-backup-manager acl update /datastore/$DS DatastoreAdmin --auth-id 'root@pam!nsadmin'
export PBS_FINGERPRINT="$(proxmox-backup-manager cert info | awk '/Fingerprint/{print $3}')"
PBS_PASSWORD="$NSADMIN" proxmox-backup-client namespace create <ns> --repository "root@pam!nsadmin@localhost:$DS"
# 2. the shared user (once) + this customer's privilege-separated token (secret → root-only file):
proxmox-backup-manager user create felhom@pbs --comment 'offsite tenancy' 2>/dev/null || true
proxmox-backup-manager user generate-token felhom@pbs <ns> > /root/.tok-<ns>.raw
sed -n 's/.*"value": "\([^"]*\)".*/\1/p' /root/.tok-<ns>.raw > /root/.tok-<ns>.secret   # 0600; rm the .raw
# 3. THE ACL — the load-bearing gotcha (see below): DatastoreBackup on the NAMESPACE PATH, to BOTH
#    the user AND the token:
proxmox-backup-manager acl update /datastore/$DS/<ns> DatastoreBackup --auth-id 'felhom@pbs'
proxmox-backup-manager acl update /datastore/$DS/<ns> DatastoreBackup --auth-id 'felhom@pbs!<ns>'
# 4. clean up the throwaway admin token:
proxmox-backup-manager user delete-token root@pam nsadmin
proxmox-backup-manager acl update /datastore/$DS DatastoreAdmin --auth-id 'root@pam!nsadmin' --delete
```
Hand `/root/.tok-<ns>.secret` + the cert fingerprint to the box out-of-band (they become the PVE
storage `.pw` + `fingerprint` — §4b).

**The confirmed minimal ACL + the gotchas (all live-proven, don't re-derive):**
- **ACL path is `/datastore/<ds>/<ns>` — NOT `/datastore/<ds>/ns/<ns>`.** The `/ns/` form is the
  *client repository* path; the *ACL object* path drops it. Granting `/ns/<ns>` silently no-ops
  (the permission check runs against `/datastore/<ds>/<ns>`).
- **PBS privsep tokens = intersection(user perms, token perms).** The token's own ACL alone is
  NOT enough — the **user** `felhom@pbs` must ALSO be granted on the namespace, or the token has
  the empty intersection and every op 403s (`missing Datastore.Backup`). Isolation is preserved
  anyway: token B's effective perms = user(has A's + B's ns) ∩ token-B(only B's ns) = B's ns only.
  Adversarially confirmed: token A → list/backup ns B ⇒ **403**.
- **`DatastoreBackup` is the whole grant** — it covers backup, list-own, restore-own, AND
  **namespace-scoped verify** (`POST /verify` with `ns=<ns>` — no `Datastore.Verify`/admin needed).
  It deliberately does **NOT** cover prune/forget (`Datastore.Modify|Prune`) — the box **cannot
  delete its own backups**; pruning is an operator/endpoint job (matches "the runner never prunes
  PBS"). No token ever exceeds `DatastoreBackup`; no admin token lives on the endpoint for a box.

## 4b. Box-side storage + agent wiring (S4 — on the customer box, as root)

```sh
FP=<endpoint cert fingerprint>; UN='felhom@pbs!<ns>'; PWSTORE=/etc/pve/priv/storage
# 1. encryption key — BORN ON THE BOX (never leaves as plaintext; the endpoint only ever sees
#    ciphertext). Its fingerprint goes in storage.cfg:
proxmox-backup-client key create $PWSTORE/felhom-offsite.enc --kdf none
ENCFP=$(sed -n 's/.*"fingerprint":"\([^"]*\)".*/\1/p' $PWSTORE/felhom-offsite.enc)
chown root:www-data $PWSTORE/felhom-offsite.enc && chmod 600 $PWSTORE/felhom-offsite.enc
# 2. token secret → PVE .pw store — ARGV-SAFE (pvesm add only takes --password on argv AND it
#    validates the connection, so a dummy won't do; write the .pw file + storage.cfg directly):
install -o root -g www-data -m 600 /root/.tok-<ns>.secret $PWSTORE/felhom-offsite.pw
cat >> /etc/pve/storage.cfg <<CFG

pbs: felhom-offsite
	datastore felhom-offsite
	server 10.77.0.1
	content backup
	fingerprint $FP
	namespace <ns>
	encryption-key $ENCFP
	username $UN
CFG
pvesm status --storage felhom-offsite   # expect: active (PVE connects over the tunnel)
# 3. agent PVE token needs the new storage (dual-grant, PVE privsep like PBS):
pveum acl modify /storage/felhom-offsite --roles FelhomAgentStore --users 'felhom-agent@pve'
pveum acl modify /storage/felhom-offsite --roles FelhomAgentStore --tokens 'felhom-agent@pve!agent'
# 4. THE WARN FIX — the non-root agent cannot read /etc/pve/priv (root 0700); give it an owned copy:
install -d -o felhom-agent -g felhom-agent -m 0700 /var/lib/felhom-agent/pbs-secrets
for ID in felhom-pbs felhom-offsite; do
  install -o felhom-agent -g felhom-agent -m 0600 /etc/pve/priv/storage/$ID.pw /var/lib/felhom-agent/pbs-secrets/$ID.pw
done
# then in agent.json backup{}: "pbs_secret_dir":"/var/lib/felhom-agent/pbs-secrets"
# 5. RETARGET the box's backups offsite — the field is backup.local_backup_target (BackupTarget()):
#    agent.json backup{}: "local_backup_target":"felhom-offsite"   (defaultBackupTarget stays felhom-pbs in code)
systemctl restart felhom-agent   # verify: journal has ZERO "cannot read token secret"; verify loop
                                 # logs `verify cycle complete datastore=felhom-offsite`
```
Verified live 2026-07-04: a real vzdump of guest 9201 (crash-consistent, ~4.6 GB compressed over
the tunnel, ~20 min) lands as **ciphertext** in ns `demo-felhom-01` (`root.pxar` crypt-mode
`encrypt`); the ns-scoped verify runs under the box's own `felhom@pbs!demo-felhom-01` token and
returns `verify=ok`; the WARN is gone; a restore reads it back decrypting with the on-box key.

**Provisioning notes / follow-ups (S4):** (a) the `.pw` copy in `pbs-secrets` is a STATIC snapshot
of the PVE token secret — a token rotation must re-copy it (a sync step for the hub-driven-tenancy
slice). (b) The offsite restore-test now runs **unattended** (S4.1, agent v0.68.0): the restore-task
wait is tier-aware (`restore_test_pbs_restore_timeout_seconds`, default 120m for the `pbs` tier;
local stays 10m). The earlier "grant scratch-band `VM.Allocate`" idea was a **phantom** — the scratch
is restored INTO `/pool/felhom` (whose ACL already grants `VM.Allocate`), so teardown is authorized
once the restore completes; the earlier 403 was the 10m-timeout firing teardown against a
not-yet-pooled guest. Live-proven on the agent-token path: `pass:true`, teardown clean, band empty.

## 5. Peersync surface

```sh
useradd -m -s /bin/sh felhom-peersync   # real shell required for the forced command;
                                        # access is bounded by the key options + sudoers, not the shell
apt install -y jq
# install the script (from the repo; strip CRLF if staging via a Windows checkout):
tr -d '\r' < scripts/felhom-peersync.sh > /usr/local/bin/felhom-peersync
chown root:root /usr/local/bin/felhom-peersync && chmod 0755 /usr/local/bin/felhom-peersync
cat > /etc/sudoers.d/felhom-peersync <<'EOF'
felhom-peersync ALL=(root) NOPASSWD: /usr/local/bin/felhom-peersync
EOF
visudo -c
```
Verify: `visudo -c` → "parsed OK". The authorized_keys entry is written in step 6 (it needs
the hub's pubkey):

```
restrict,command="sudo /usr/local/bin/felhom-peersync" ssh-ed25519 AAAA... hub@felhom
```

(`restrict` kills pty/forwarding/X11/agent in one word; the forced command overrides whatever
the client asks to run.)

## 6. Hub credential (build server + the box + k3s)

On the build server (192.168.0.180):

```sh
ssh-keygen -t ed25519 -f wg-endpoint-ssh -N "" -C hub@felhom
```

- The **pubkey** (`wg-endpoint-ssh.pub`) goes into step 5's authorized_keys line on the box:
  `printf 'restrict,command="sudo /usr/local/bin/felhom-peersync" %s\n' "$(cat wg-endpoint-ssh.pub)" > /home/felhom-peersync/.ssh/authorized_keys`
  (as root on the box; `.ssh` dir 0700, file 0600, owner `felhom-peersync`).
- Read the endpoint's **host key on the box itself** (never trust `ssh-keyscan` alone):
  `cat /etc/ssh/ssh_host_ed25519_key.pub`
- Create the k8s Secret, then **shred the local private key** — from this point it exists ONLY
  in the Secret:

```sh
sudo kubectl -n felhom-system create secret generic wg-endpoint-ssh \
  --from-file=key=wg-endpoint-ssh \
  --from-literal=hostkey='ssh-ed25519 AAAA...the-host-key-line'
shred -u wg-endpoint-ssh wg-endpoint-ssh.pub
```

Then apply the manifest + roll the hub (`manifests/hub.yaml` already mounts the Secret as
optional): expect `[INFO] WG peer-sync enabled (endpoint 167.233.158.164:22, ...)` in the hub
log (before the Secret existed it says `disabled (endpoint not configured)`).

## 7. Register the endpoint in the hub

```sh
curl -s -X PUT https://hub.felhom.eu/api/v1/admin/wg/endpoint \
  -H "Authorization: Bearer <GLOBAL-KEY out-of-band>" \
  -d '{"dns_name":"ep0.felhom.eu","wg_port":443,"server_pubkey":"<step-2 pubkey>",
       "tunnel_subnet":"10.77.0.0/24","pbs_tunnel_ip":"10.77.0.1"}'
```

Validation loop (the S1 done-criterion): `POST /api/v1/admin/wg/peers {"pubkey":...}` with a
throwaway pubkey → response `sync:"ok"` → on the box `wg show wg0 peers` lists it →
`DELETE /api/v1/admin/wg/peers {"pubkey":...}` → gone from `wg show`. A live handshake +
`curl -sk https://10.77.0.1:8007/` through a throwaway client tunnel proves 443/udp + the
wg0-only 8007 rule end-to-end.

## 8. Re-provision from nothing (the DR story)

The rebuild is **steps 1–7 on a fresh VM**. What is lost vs regenerable:

- **WG server keypair — regenerable**, but every existing peer's config then points at a dead
  server pubkey. Until S2/S3 deliver endpoint coords via desired-state, re-keying is a
  **manual client-side update** (dev-acceptable; note per doc 06 §7). Register the new pubkey
  via step 7; peers re-converge as they're updated.
- **The hub's peer registry survives** (it lives in the hub DB): a rebuilt endpoint converges
  on the first reconciler tick (≤5 min) or the next mutation — no re-registration needed.
- **The PBS datastore is the real loss** — customer ciphertext. PBS-side redundancy is a
  deferred economics call (doc 06 §7); until then, endpoint loss = offsite history loss
  (customers still hold local backups + a re-seedable offsite).
- The hub's SSH credential + pinned host key must be **rotated on rebuild** (new box = new
  host key): repeat step 6 (`kubectl delete secret wg-endpoint-ssh` first), roll the hub.

## 9. OOB operator forwarding (TASK H1 — 2026-07-05)

The endpoint gains a **per-pair operator→box forward** posture so the operator peer can reach each
box's `felhom-sshd` (doc 06 §4.5 amended: forwarding ON but per-pair allow-listed; box↔box drop is
now explicit). Peersync is UNCHANGED — it still manages only the peer *list*; these forward rules are
STATIC endpoint config.

```sh
# 1. permanent forwarding
echo 'net.ipv4.ip_forward = 1' > /etc/sysctl.d/99-felhom-oob.conf
sysctl -w net.ipv4.ip_forward=1

# 2. forward posture in the STATIC nftables filter forward chain (add to /etc/nftables.conf's
#    `chain forward` — which keeps `policy drop`). ONE accept rule per (operator, box) pair; the
#    box↔box drop is explicit. Reload path so replies + the PBS path are unaffected (INPUT hook).
#    <operator/32> = GET /api/v1/admin/wg/operator-peer ; <box/32> = each host's assigned_ip.
    ct state established,related accept
    iifname "wg0" oifname "wg0" ip saddr <operator/32> ip daddr <box/32> counter accept
    iifname "wg0" oifname "wg0" counter drop
```

**Register the operator peer** (hub, global key) — it becomes an UNBOUND `wg_peers` row (peersync
pushes it to wg0) and its `/32` flows to every box as `oob_peer_ip`:
```sh
curl -s -X PUT https://hub.felhom.eu/api/v1/admin/wg/operator-peer \
  -H "Authorization: Bearer <GLOBAL-KEY>" \
  -d '{"pubkey":"<operator WG pubkey>","assigned_ip":"10.77.0.250","ssh_pubkey":"ssh-ed25519 AAAA… operator"}'
```

**Box side:** `felhom-host-install … --enable-oob` (static felhom-sshd + belt) + `oob.enabled=true`
in `agent.json`. The agent renders the config, claims a port, writes `felhom-op`'s authorized_keys
from `oob_operator_ssh_key`, and fills the belt sets. Verify: from the operator peer,
`ssh -p <claimed-port> felhom-op@<box tunnel IP>`; the host belt drops any non-operator tunnel source.
Re-verify PBS (`pvesm status --storage felhom-offsite` on the box) after the forward change.

## 10. The felhom-tenantsync surface (PBS DR tier SLICE 1 — installed 2026-07-10)

The hub's SECOND forced-command surface on the endpoint: per-customer PBS tenancy ops (namespace +
privilege-separated token + dual-grant, the §4a dance automated). Same low-priv user as peersync, its
OWN keypair + script + sudoers drop-in — **the peersync files are untouched** (one script, one job;
peersync script + sudoers sha256 asserted identical before/after at install). Script source of truth:
`scripts/felhom-tenantsync.sh` (**v1.2.0**; JSON stdin/stdout, ops
`provision`/`reissue`/`deprovision`/`fingerprint`/`usage`). Hub client: `hub/internal/tenantsync`.

> **v1.2.0 (2026-07-17, hub R-5 PBS visibility):** adds the read-only **`usage`** op —
> `{"op":"usage"}` → `{"status":"ok","total":<bytes>,"used":<bytes>,"avail":<bytes>}` (a `df` on the
> `felhom-offsite` datastore path). **No customer_id, no admin token, NO mutation** — the structural twin
> of `fingerprint`. The hub's PBS-DR checker polls it on a 15-min throttle; the Offsite "PBS DR" tab +
> dashboard PBS gauge render it. **This update needs NO sudoers / authorized_keys change** — SAME forced
> command, new op. Update = re-run the on-box script install below (`.bak-<ver>` kept). The hub tolerates
> an ep0 STILL on v1.1.0 (unknown-op → the gauge shows "n/a" until updated; no hub redeploy needed on
> update). Verify after install: `echo '{"op":"usage"}' | ssh -i tenantsync-ssh felhom-peersync@ep0`
> (or the box-local `echo '{"op":"usage"}' | sudo /usr/local/bin/felhom-tenantsync`) returns the totals.

> **v1.1.0 (2026-07-17, deployed to ep0):** adds the `deprovision` op — the deliberate data-deletion
> the slice-1 header reserved, now driven by the hub-side **customer RESET** (hub v0.61.0, operator
> ack-gated). It destroys the customer's namespace + all backup groups + token
> (`namespace delete --delete-groups true`); the shared `felhom@pbs` user is never touched; idempotent
> (`deleted:false` when nothing was there). To update the on-box script, re-run the install block below
> with the new source (a `.bak-<ver>` is kept). Live-drilled 2026-07-17: provision → real backup →
> deprovision (`deleted:true`, group destroyed) → idempotent re-run → all real tenants survived.

On the build server (the hub's credential; mirror of step 6):
```sh
ssh-keygen -t ed25519 -f tenantsync-ssh -N "" -C hub-tenancy@felhom
sudo kubectl -n felhom-system create secret generic tenantsync --from-file=key=tenantsync-ssh
shred -u tenantsync-ssh tenantsync-ssh.pub   # from here the key exists ONLY in the Secret
```
(The pinned host key + endpoint address are REUSED from the peersync env — no new hostkey entry.
`manifests/hub.yaml` mounts the Secret at `/etc/hub-secrets/tenantsync/key` via
`TENANTSYNC_SSH_KEY_FILE`; absent Secret → the hub logs "PBS DR tenantsync disabled".)

On the box, as root:
```sh
# script: stage → strip CRLF if via a Windows checkout → syntax-check → install → THEN activate
tr -d '\r' < felhom-tenantsync.sh > /tmp/fts && bash -n /tmp/fts
install -o root -g root -m 0755 /tmp/fts /usr/local/bin/felhom-tenantsync && rm /tmp/fts
cat > /etc/sudoers.d/felhom-tenantsync <<'EOF'
felhom-peersync ALL=(root) NOPASSWD: /usr/local/bin/felhom-tenantsync
EOF
chmod 0440 /etc/sudoers.d/felhom-tenantsync && visudo -cf /etc/sudoers.d/felhom-tenantsync
printf 'restrict,command="sudo /usr/local/bin/felhom-tenantsync" %s\n' "<tenantsync pubkey line>" \
  >> /home/felhom-peersync/.ssh/authorized_keys
```

Verify (the install's smoke, run from the build server with the key before it is shredded — pin the
host key from step 6's on-box read, never keyscan):
- `{"op":"fingerprint"}` → `{"status":"ok","fingerprint":"<PBS cert sha256>"}`
- `{"op":"provision","customer_id":"spike-smoke"}` → ok-shape; **assert `token_secret` length only,
  never print it**; a second provision → `{"code":"token_exists"}`; `reissue` → fresh ok-shape;
  unknown op → `bad_request`. Then tear the throwaway tenant down (delete token, ns ACL, namespace
  via a transient root admin token — §4a teardown shape).

Install-run facts (2026-07-10): smoke exposed that `proxmox-backup-client … --output-format json`
wraps output as `{"data":[…]}` (the MANAGER returns bare arrays) — script fixed to `(.data // .)`
before the hub ever used the surface. The §6-era orphan `root@pam!spike` token + its
`/datastore/scratch` DatastoreAdmin ACL were removed in the same session (spike flag #1); after
teardown the endpoint holds exactly the real `demo-felhom-01` tenancy. Hub v0.44.0 logs
`PBS DR tenantsync enabled (endpoint 167.233.158.164:22, user felhom-peersync)` on start.

## 11. The felhom-poke surface (agent-plane immediate-sync — Direction-2a, v0.59.0)

The hub's THIRD forced-command surface: a **contentless UDP "sync now" nudge** to a registered
box's WireGuard /32, so a user-triggered agent-plane change (a pbsdr descriptor, a MinAgent floor)
lands in **seconds** instead of the ≤15-min report cycle. Proven end-to-end by
`documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md` (~0.42 s/poke; reserved there
for the agent plane). Same low-priv `felhom-peersync` user + REUSED endpoint address & pinned host
key, its OWN keypair + script — **peersync/tenantsync files untouched**. Script source of truth:
`scripts/felhom-poke.sh` (v1.0.0). Hub client: `hub/internal/poke`.

**No sudoers grant** — unlike peersync/tenantsync, sending a datagram needs no privilege, so the
forced command runs as `felhom-peersync` directly (`command="/usr/local/bin/felhom-poke"`, no
`sudo`). The fixed poke port is **51822** — a shared cross-repo constant (`felhom-agent`
`internal/poke.Port`, this script's `POKE_PORT`, and the port the listener binds on the box's WG
/32); change it in all three or nowhere.

On the build server (the hub's credential; mirror of §6/§10 — the PRIVATE key ends only in the
Secret):
```sh
ssh-keygen -t ed25519 -f agent-poke-ssh -N "" -C hub-poke@felhom
sudo kubectl -n felhom-system create secret generic agent-poke --from-file=key=agent-poke-ssh
# keep agent-poke-ssh.pub for the box step below, THEN:
shred -u agent-poke-ssh   # the private key now exists ONLY in the Secret
```
(The pinned host key + endpoint address are REUSED from the peersync env — no new hostkey entry.
`manifests/hub.yaml` mounts the Secret at `/etc/hub-secrets/agent-poke/key` via `POKE_SSH_KEY_FILE`;
absent Secret → the hub logs "agent-plane poke disabled".)

On the box (ep0), as root — additive; the peersync `authorized_keys` lines are untouched:
```sh
tr -d '\r' < felhom-poke.sh > /tmp/fp && sh -n /tmp/fp
install -o root -g root -m 0755 /tmp/fp /usr/local/bin/felhom-poke && rm /tmp/fp
printf 'restrict,command="/usr/local/bin/felhom-poke" %s\n' "$(cat agent-poke-ssh.pub)" \
  >> /home/felhom-peersync/.ssh/authorized_keys
```

Verify (from the build server with the key before it is shredded — pin the host key from §6's on-box
read, never keyscan): `ssh -i agent-poke-ssh felhom-peersync@<ep0> 10.77.0.<box>` → prints
`poke-fired`, and the box's agent journal logs `poke received → triggering an immediate
desired-state cycle`. A non-WG target (`ssh … 8.8.8.8`) → `refused non-WG target`, non-zero exit.
Hub v0.59.0 logs `agent-plane poke enabled (endpoint 167.233.158.164:22, user felhom-peersync)` on
start.