felhom.eu/documentation/runbooks/offsite-endpoint.md

RUNBOOK — offsite endpoint provisioning (WG server + PBS + peersync surface)

What this creates: the Felhom offsite endpoint (doc 06 §5) on a blank Debian 13 VM — the WG server on 443/UDP, the offsite PBS with a /srv datastore, a drop-everything-else firewall, and the hub-driven felhom-peersync reconcile surface. Re-running it on a fresh VM re-creates the endpoint from nothing (that is the DR story, step 8).

Validated: 2026-07-03 on the dev/test endpoint felhom-hetzner (Hetzner CX23, Debian 13, 167.233.158.164 / 2a01:4f8:1c16:7aa1::1) with hub v0.32.0. The production endpoint is a later re-run of this runbook on a production VM.

Companion piece: the hub side (env + Secret) is manifests/hub.yaml + step 6/7 below. The peersync script's source of truth is scripts/felhom-peersync.sh (v1.0.0).

Parameters used throughout (adjust for a new endpoint):

Param	Dev value
VM	felhom-hetzner, 167.233.158.164 (NOT the live jarrs.eu box — different server)
DNS name	ep0.felhom.eu (A + AAAA, DNS-only / grey-cloud — proxying breaks WG/UDP + PBS)
WG port	443/udp (spike P4: handshakes + carries identically to 51820)
Tunnel subnet	10.77.0.0/24; endpoint in-tunnel IP 10.77.0.1
PBS datastore	felhom-offsite at /srv/pbs-felhom (never under /root — the backup user cannot traverse it; spike gotcha)

0. Preconditions (operator)

• The VM is powered on and root-SSH-reachable.
• DNS ep0.felhom.eu → A 167.233.158.164 + AAAA 2a01:4f8:1c16:7aa1::1, grey-cloud. Verify: nslookup ep0.felhom.eu 1.1.1.1 returns the A record. The AAAA must be the box's full ::1 address — a bare ...:7aa1:: (the subnet zero address) resolves but points at nothing (live-run finding). Home-resolver propagation can lag public DNS by minutes — a client-side wg-quick up that fails to resolve right after record creation just needs a retry.
• Sanity: ssh root@167.233.158.164 hostname → felhom-hetzner (the throwaway CX23), not any production box.

1. Base (on the box, as root)

apt update && DEBIAN_FRONTEND=noninteractive apt full-upgrade -y
printf 'PasswordAuthentication no\n' > /etc/ssh/sshd_config.d/90-felhom.conf && systemctl reload ssh
sshd -T | grep -i '^passwordauthentication'   # must print: passwordauthentication no

(The explicit drop-in beats auditing defaults — Hetzner images have no explicit line at all.)

2. WG server

apt install -y wireguard-tools          # module ships in the stock Debian 13 kernel
umask 077
wg genkey > /etc/wireguard/wg0.key      # server private key — NEVER displayed/copied anywhere
wg pubkey < /etc/wireguard/wg0.key      # RECORD this pubkey → hub registration (step 7)
cat > /etc/wireguard/wg0.conf.head <<EOF
[Interface]
Address = 10.77.0.1/24
ListenPort = 443
MTU = 1420
PrivateKey = $(cat /etc/wireguard/wg0.key)
EOF
chmod 600 /etc/wireguard/wg0.conf.head
cp -p /etc/wireguard/wg0.conf.head /etc/wireguard/wg0.conf   # zero peers initially
systemctl enable --now wg-quick@wg0
wg show wg0

Verify: wg show wg0 → listening port: 443, no peers. No SaveConfig anywhere — the peersync script owns persistence (head-file + generated peers; wg-quick save would rewrite the conf nondeterministically).

NEVER run wg show <if> dump in a logged/shared session — field 1 of its interface line is the private key. Plain wg show prints only the public key. (Live-run lesson: a dump | cut leaked the first server key into a session log → the key was rotated on the spot. If it happens to you: regenerate the keypair, rebuild wg0.conf.head + wg0.conf, restart wg-quick@wg0, re-PUT the new pubkey to the hub — 2 minutes, zero peer downtime beyond the restart.)

3. Firewall (nftables, v4+v6)

cat > /etc/nftables.conf <<'EOF'
#!/usr/sbin/nft -f
flush ruleset
table inet filter {
    chain input {
        type filter hook input priority 0; policy drop;
        ct state established,related accept
        iif "lo" accept
        tcp dport 22 accept
        udp dport 443 accept
        tcp dport 8007 iifname "wg0" accept
        ip protocol icmp accept
        meta l4proto ipv6-icmp accept
    }
    chain forward {
        type filter hook forward priority 0; policy drop;
    }
}
EOF
systemctl enable --now nftables
nft list ruleset | head -20
sysctl net.ipv4.ip_forward

Verify (doc 06 §4.5 — no inter-peer routing by topology): forward chain policy drop, net.ipv4.ip_forward = 0. From an outside vantage: ssh (22) connects; 8007 publicly CLOSED — curl -sk --max-time 8 https://<public-ip>:8007/ must time out (the spike-P2b firewall proof, repeated). 443/udp is proven by the first WG handshake in step 7's validation.

4. PBS

curl -fsSL https://enterprise.proxmox.com/debian/proxmox-release-trixie.gpg \
  -o /usr/share/keyrings/proxmox-archive-keyring.gpg
echo 'deb [signed-by=/usr/share/keyrings/proxmox-archive-keyring.gpg] http://download.proxmox.com/debian/pbs trixie pbs-no-subscription' \
  > /etc/apt/sources.list.d/pbs.list
apt update && DEBIAN_FRONTEND=noninteractive apt install -y --no-install-recommends \
  proxmox-backup-server proxmox-backup-client   # client is a SEPARATE package (namespace ops need it)
rm -f /etc/apt/sources.list.d/pbs-enterprise.sources   # the install adds the ENTERPRISE repo → 401s every apt update
apt update
proxmox-backup-manager datastore create felhom-offsite /srv/pbs-felhom
# prove namespace ops (per-customer tenancy, doc 06 D5) with a THROWAWAY token, then remove it
# (root@pam has no password on a cloud image; S4 wires the real per-customer tokens):
umask 077
proxmox-backup-manager user generate-token root@pam nsproof > /root/.nsproof
proxmox-backup-manager acl update /datastore/felhom-offsite DatastoreAdmin --auth-id 'root@pam!nsproof'
export PBS_PASSWORD="$(tr -d '\n' < /root/.nsproof | sed -n 's/.*"value": "\([^"]*\)".*/\1/p')"
export PBS_FINGERPRINT="$(proxmox-backup-manager cert info | grep -i fingerprint | awk '{print $3}')"
proxmox-backup-client namespace create test-ns --repository 'root@pam!nsproof@localhost:felhom-offsite'
proxmox-backup-client namespace delete test-ns --repository 'root@pam!nsproof@localhost:felhom-offsite'
proxmox-backup-manager user delete-token root@pam nsproof && rm -f /root/.nsproof

Verify: systemctl is-active proxmox-backup proxmox-backup-proxy → both active; the datastore parent is /srv (world-traversable — a datastore under /root fails with Permission denied for the backup user). GUI: reachable ONLY via SSH port-forward (ssh -L 8007:127.0.0.1:8007 root@<box>) or through the tunnel — by design, no public 8007.

4a. Per-customer PBS tenancy (S4 — live-confirmed 2026-07-04 on felhom-hetzner)

Each customer box gets ITS OWN namespace + a privilege-separated token scoped to that namespace, on the single shared felhom-offsite datastore (doc 06 D5). One shared PBS user felhom@pbs holds a per-customer token each; cross-tenant isolation is enforced by the per-token ACL.

On the endpoint, per customer <ns> (e.g. demo-felhom-01 = the box's host_id):

DS=felhom-offsite
# 1. namespace — created via a THROWAWAY root admin token (root@pam has no password on the cloud
#    image; proxmox-backup-manager has NO `namespace` subcommand — namespaces are client-side):
proxmox-backup-manager user generate-token root@pam nsadmin > /root/.nsadmin.raw   # umask 077
NSADMIN=$(sed -n 's/.*"value": "\([^"]*\)".*/\1/p' /root/.nsadmin.raw); rm -f /root/.nsadmin.raw
proxmox-backup-manager acl update /datastore/$DS DatastoreAdmin --auth-id 'root@pam!nsadmin'
export PBS_FINGERPRINT="$(proxmox-backup-manager cert info | awk '/Fingerprint/{print $3}')"
PBS_PASSWORD="$NSADMIN" proxmox-backup-client namespace create <ns> --repository "root@pam!nsadmin@localhost:$DS"
# 2. the shared user (once) + this customer's privilege-separated token (secret → root-only file):
proxmox-backup-manager user create felhom@pbs --comment 'offsite tenancy' 2>/dev/null || true
proxmox-backup-manager user generate-token felhom@pbs <ns> > /root/.tok-<ns>.raw
sed -n 's/.*"value": "\([^"]*\)".*/\1/p' /root/.tok-<ns>.raw > /root/.tok-<ns>.secret   # 0600; rm the .raw
# 3. THE ACL — the load-bearing gotcha (see below): DatastoreBackup on the NAMESPACE PATH, to BOTH
#    the user AND the token:
proxmox-backup-manager acl update /datastore/$DS/<ns> DatastoreBackup --auth-id 'felhom@pbs'
proxmox-backup-manager acl update /datastore/$DS/<ns> DatastoreBackup --auth-id 'felhom@pbs!<ns>'
# 4. clean up the throwaway admin token:
proxmox-backup-manager user delete-token root@pam nsadmin
proxmox-backup-manager acl update /datastore/$DS DatastoreAdmin --auth-id 'root@pam!nsadmin' --delete

Hand /root/.tok-<ns>.secret + the cert fingerprint to the box out-of-band (they become the PVE storage .pw + fingerprint — §4b).

The confirmed minimal ACL + the gotchas (all live-proven, don't re-derive):

• ACL path is /datastore/<ds>/<ns> — NOT /datastore/<ds>/ns/<ns>. The /ns/ form is the client repository path; the ACL object path drops it. Granting /ns/<ns> silently no-ops (the permission check runs against /datastore/<ds>/<ns>).
• PBS privsep tokens = intersection(user perms, token perms). The token's own ACL alone is NOT enough — the user felhom@pbs must ALSO be granted on the namespace, or the token has the empty intersection and every op 403s (missing Datastore.Backup). Isolation is preserved anyway: token B's effective perms = user(has A's + B's ns) ∩ token-B(only B's ns) = B's ns only. Adversarially confirmed: token A → list/backup ns B ⇒ 403.
• DatastoreBackup is the whole grant — it covers backup, list-own, restore-own, AND namespace-scoped verify (POST /verify with ns=<ns> — no Datastore.Verify/admin needed). It deliberately does NOT cover prune/forget (Datastore.Modify|Prune) — the box cannot delete its own backups; pruning is an operator/endpoint job (matches "the runner never prunes PBS"). No token ever exceeds DatastoreBackup; no admin token lives on the endpoint for a box.

4b. Box-side storage + agent wiring (S4 — on the customer box, as root)

FP=<endpoint cert fingerprint>; UN='felhom@pbs!<ns>'; PWSTORE=/etc/pve/priv/storage
# 1. encryption key — BORN ON THE BOX (never leaves as plaintext; the endpoint only ever sees
#    ciphertext). Its fingerprint goes in storage.cfg:
proxmox-backup-client key create $PWSTORE/felhom-offsite.enc --kdf none
ENCFP=$(sed -n 's/.*"fingerprint":"\([^"]*\)".*/\1/p' $PWSTORE/felhom-offsite.enc)
chown root:www-data $PWSTORE/felhom-offsite.enc && chmod 600 $PWSTORE/felhom-offsite.enc
# 2. token secret → PVE .pw store — ARGV-SAFE (pvesm add only takes --password on argv AND it
#    validates the connection, so a dummy won't do; write the .pw file + storage.cfg directly):
install -o root -g www-data -m 600 /root/.tok-<ns>.secret $PWSTORE/felhom-offsite.pw
cat >> /etc/pve/storage.cfg <<CFG

pbs: felhom-offsite
	datastore felhom-offsite
	server 10.77.0.1
	content backup
	fingerprint $FP
	namespace <ns>
	encryption-key $ENCFP
	username $UN
CFG
pvesm status --storage felhom-offsite   # expect: active (PVE connects over the tunnel)
# 3. agent PVE token needs the new storage (dual-grant, PVE privsep like PBS):
pveum acl modify /storage/felhom-offsite --roles FelhomAgentStore --users 'felhom-agent@pve'
pveum acl modify /storage/felhom-offsite --roles FelhomAgentStore --tokens 'felhom-agent@pve!agent'
# 4. THE WARN FIX — the non-root agent cannot read /etc/pve/priv (root 0700); give it an owned copy:
install -d -o felhom-agent -g felhom-agent -m 0700 /var/lib/felhom-agent/pbs-secrets
for ID in felhom-pbs felhom-offsite; do
  install -o felhom-agent -g felhom-agent -m 0600 /etc/pve/priv/storage/$ID.pw /var/lib/felhom-agent/pbs-secrets/$ID.pw
done
# then in agent.json backup{}: "pbs_secret_dir":"/var/lib/felhom-agent/pbs-secrets"
# 5. RETARGET the box's backups offsite — the field is backup.local_backup_target (BackupTarget()):
#    agent.json backup{}: "local_backup_target":"felhom-offsite"   (defaultBackupTarget stays felhom-pbs in code)
systemctl restart felhom-agent   # verify: journal has ZERO "cannot read token secret"; verify loop
                                 # logs `verify cycle complete datastore=felhom-offsite`

Verified live 2026-07-04: a real vzdump of guest 9201 (crash-consistent, ~4.6 GB compressed over the tunnel, ~20 min) lands as ciphertext in ns demo-felhom-01 (root.pxar crypt-mode encrypt); the ns-scoped verify runs under the box's own felhom@pbs!demo-felhom-01 token and returns verify=ok; the WARN is gone; a restore reads it back decrypting with the on-box key.

Provisioning notes / follow-ups (S4): (a) the .pw copy in pbs-secrets is a STATIC snapshot of the PVE token secret — a token rotation must re-copy it (a sync step for the hub-driven-tenancy slice). (b) The offsite restore-test now runs unattended (S4.1, agent v0.68.0): the restore-task wait is tier-aware (restore_test_pbs_restore_timeout_seconds, default 120m for the pbs tier; local stays 10m). The earlier "grant scratch-band VM.Allocate" idea was a phantom — the scratch is restored INTO /pool/felhom (whose ACL already grants VM.Allocate), so teardown is authorized once the restore completes; the earlier 403 was the 10m-timeout firing teardown against a not-yet-pooled guest. Live-proven on the agent-token path: pass:true, teardown clean, band empty.

5. Peersync surface

useradd -m -s /bin/sh felhom-peersync   # real shell required for the forced command;
                                        # access is bounded by the key options + sudoers, not the shell
apt install -y jq
# install the script (from the repo; strip CRLF if staging via a Windows checkout):
tr -d '\r' < scripts/felhom-peersync.sh > /usr/local/bin/felhom-peersync
chown root:root /usr/local/bin/felhom-peersync && chmod 0755 /usr/local/bin/felhom-peersync
cat > /etc/sudoers.d/felhom-peersync <<'EOF'
felhom-peersync ALL=(root) NOPASSWD: /usr/local/bin/felhom-peersync
EOF
visudo -c

Verify: visudo -c → "parsed OK". The authorized_keys entry is written in step 6 (it needs the hub's pubkey):

restrict,command="sudo /usr/local/bin/felhom-peersync" ssh-ed25519 AAAA... hub@felhom

(restrict kills pty/forwarding/X11/agent in one word; the forced command overrides whatever the client asks to run.)

6. Hub credential (build server + the box + k3s)

On the build server (192.168.0.180):

ssh-keygen -t ed25519 -f wg-endpoint-ssh -N "" -C hub@felhom

• The pubkey (wg-endpoint-ssh.pub) goes into step 5's authorized_keys line on the box: printf 'restrict,command="sudo /usr/local/bin/felhom-peersync" %s\n' "$(cat wg-endpoint-ssh.pub)" > /home/felhom-peersync/.ssh/authorized_keys (as root on the box; .ssh dir 0700, file 0600, owner felhom-peersync).
• Read the endpoint's host key on the box itself (never trust ssh-keyscan alone): cat /etc/ssh/ssh_host_ed25519_key.pub
• Create the k8s Secret, then shred the local private key — from this point it exists ONLY in the Secret:

sudo kubectl -n felhom-system create secret generic wg-endpoint-ssh \
  --from-file=key=wg-endpoint-ssh \
  --from-literal=hostkey='ssh-ed25519 AAAA...the-host-key-line'
shred -u wg-endpoint-ssh wg-endpoint-ssh.pub

Then apply the manifest + roll the hub (manifests/hub.yaml already mounts the Secret as optional): expect [INFO] WG peer-sync enabled (endpoint 167.233.158.164:22, ...) in the hub log (before the Secret existed it says disabled (endpoint not configured)).

7. Register the endpoint in the hub

curl -s -X PUT https://hub.felhom.eu/api/v1/admin/wg/endpoint \
  -H "Authorization: Bearer <GLOBAL-KEY out-of-band>" \
  -d '{"dns_name":"ep0.felhom.eu","wg_port":443,"server_pubkey":"<step-2 pubkey>",
       "tunnel_subnet":"10.77.0.0/24","pbs_tunnel_ip":"10.77.0.1"}'

Validation loop (the S1 done-criterion): POST /api/v1/admin/wg/peers {"pubkey":...} with a throwaway pubkey → response sync:"ok" → on the box wg show wg0 peers lists it → DELETE /api/v1/admin/wg/peers {"pubkey":...} → gone from wg show. A live handshake + curl -sk https://10.77.0.1:8007/ through a throwaway client tunnel proves 443/udp + the wg0-only 8007 rule end-to-end.

8. Re-provision from nothing (the DR story)

The rebuild is steps 1–7 on a fresh VM. What is lost vs regenerable:

• WG server keypair — regenerable, but every existing peer's config then points at a dead server pubkey. Until S2/S3 deliver endpoint coords via desired-state, re-keying is a manual client-side update (dev-acceptable; note per doc 06 §7). Register the new pubkey via step 7; peers re-converge as they're updated.
• The hub's peer registry survives (it lives in the hub DB): a rebuilt endpoint converges on the first reconciler tick (≤5 min) or the next mutation — no re-registration needed.
• The PBS datastore is the real loss — customer ciphertext. PBS-side redundancy is a deferred economics call (doc 06 §7); until then, endpoint loss = offsite history loss (customers still hold local backups + a re-seedable offsite).
• The hub's SSH credential + pinned host key must be rotated on rebuild (new box = new host key): repeat step 6 (kubectl delete secret wg-endpoint-ssh first), roll the hub.

9. OOB operator forwarding (TASK H1 — 2026-07-05)

The endpoint gains a per-pair operator→box forward posture so the operator peer can reach each box's felhom-sshd (doc 06 §4.5 amended: forwarding ON but per-pair allow-listed; box↔box drop is now explicit). Peersync is UNCHANGED — it still manages only the peer list; these forward rules are STATIC endpoint config.

# 1. permanent forwarding
echo 'net.ipv4.ip_forward = 1' > /etc/sysctl.d/99-felhom-oob.conf
sysctl -w net.ipv4.ip_forward=1

# 2. forward posture in the STATIC nftables filter forward chain (add to /etc/nftables.conf's
#    `chain forward` — which keeps `policy drop`). ONE accept rule per (operator, box) pair; the
#    box↔box drop is explicit. Reload path so replies + the PBS path are unaffected (INPUT hook).
#    <operator/32> = GET /api/v1/admin/wg/operator-peer ; <box/32> = each host's assigned_ip.
    ct state established,related accept
    iifname "wg0" oifname "wg0" ip saddr <operator/32> ip daddr <box/32> counter accept
    iifname "wg0" oifname "wg0" counter drop

Register the operator peer (hub, global key) — it becomes an UNBOUND wg_peers row (peersync pushes it to wg0) and its /32 flows to every box as oob_peer_ip:

curl -s -X PUT https://hub.felhom.eu/api/v1/admin/wg/operator-peer \
  -H "Authorization: Bearer <GLOBAL-KEY>" \
  -d '{"pubkey":"<operator WG pubkey>","assigned_ip":"10.77.0.250","ssh_pubkey":"ssh-ed25519 AAAA… operator"}'

Box side: felhom-host-install … --enable-oob (static felhom-sshd + belt) + oob.enabled=true in agent.json. The agent renders the config, claims a port, writes felhom-op's authorized_keys from oob_operator_ssh_key, and fills the belt sets. Verify: from the operator peer, ssh -p <claimed-port> felhom-op@<box tunnel IP>; the host belt drops any non-operator tunnel source. Re-verify PBS (pvesm status --storage felhom-offsite on the box) after the forward change.

10. The felhom-tenantsync surface (PBS DR tier SLICE 1 — installed 2026-07-10)

The hub's SECOND forced-command surface on the endpoint: per-customer PBS tenancy ops (namespace + privilege-separated token + dual-grant, the §4a dance automated). Same low-priv user as peersync, its OWN keypair + script + sudoers drop-in — the peersync files are untouched (one script, one job; peersync script + sudoers sha256 asserted identical before/after at install). Script source of truth: scripts/felhom-tenantsync.sh (v1.0.0; JSON stdin/stdout, ops provision/reissue/fingerprint, NO deprovision — data deletion stays a deliberate decision). Hub client: hub/internal/tenantsync.

On the build server (the hub's credential; mirror of step 6):

ssh-keygen -t ed25519 -f tenantsync-ssh -N "" -C hub-tenancy@felhom
sudo kubectl -n felhom-system create secret generic tenantsync --from-file=key=tenantsync-ssh
shred -u tenantsync-ssh tenantsync-ssh.pub   # from here the key exists ONLY in the Secret

(The pinned host key + endpoint address are REUSED from the peersync env — no new hostkey entry. manifests/hub.yaml mounts the Secret at /etc/hub-secrets/tenantsync/key via TENANTSYNC_SSH_KEY_FILE; absent Secret → the hub logs "PBS DR tenantsync disabled".)

On the box, as root:

# script: stage → strip CRLF if via a Windows checkout → syntax-check → install → THEN activate
tr -d '\r' < felhom-tenantsync.sh > /tmp/fts && bash -n /tmp/fts
install -o root -g root -m 0755 /tmp/fts /usr/local/bin/felhom-tenantsync && rm /tmp/fts
cat > /etc/sudoers.d/felhom-tenantsync <<'EOF'
felhom-peersync ALL=(root) NOPASSWD: /usr/local/bin/felhom-tenantsync
EOF
chmod 0440 /etc/sudoers.d/felhom-tenantsync && visudo -cf /etc/sudoers.d/felhom-tenantsync
printf 'restrict,command="sudo /usr/local/bin/felhom-tenantsync" %s\n' "<tenantsync pubkey line>" \
  >> /home/felhom-peersync/.ssh/authorized_keys

Verify (the install's smoke, run from the build server with the key before it is shredded — pin the host key from step 6's on-box read, never keyscan):

• {"op":"fingerprint"} → {"status":"ok","fingerprint":"<PBS cert sha256>"}
• {"op":"provision","customer_id":"spike-smoke"} → ok-shape; assert token_secret length only, never print it; a second provision → {"code":"token_exists"}; reissue → fresh ok-shape; unknown op → bad_request. Then tear the throwaway tenant down (delete token, ns ACL, namespace via a transient root admin token — §4a teardown shape).

Install-run facts (2026-07-10): smoke exposed that proxmox-backup-client … --output-format json wraps output as {"data":[…]} (the MANAGER returns bare arrays) — script fixed to (.data // .) before the hub ever used the surface. The §6-era orphan root@pam!spike token + its /datastore/scratch DatastoreAdmin ACL were removed in the same session (spike flag #1); after teardown the endpoint holds exactly the real demo-felhom-01 tenancy. Hub v0.44.0 logs PBS DR tenantsync enabled (endpoint 167.233.158.164:22, user felhom-peersync) on start.

11. The felhom-poke surface (agent-plane immediate-sync — Direction-2a, v0.59.0)

The hub's THIRD forced-command surface: a contentless UDP "sync now" nudge to a registered box's WireGuard /32, so a user-triggered agent-plane change (a pbsdr descriptor, a MinAgent floor) lands in seconds instead of the ≤15-min report cycle. Proven end-to-end by documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md (~0.42 s/poke; reserved there for the agent plane). Same low-priv felhom-peersync user + REUSED endpoint address & pinned host key, its OWN keypair + script — peersync/tenantsync files untouched. Script source of truth: scripts/felhom-poke.sh (v1.0.0). Hub client: hub/internal/poke.

No sudoers grant — unlike peersync/tenantsync, sending a datagram needs no privilege, so the forced command runs as felhom-peersync directly (command="/usr/local/bin/felhom-poke", no sudo). The fixed poke port is 51822 — a shared cross-repo constant (felhom-agent internal/poke.Port, this script's POKE_PORT, and the port the listener binds on the box's WG /32); change it in all three or nowhere.

On the build server (the hub's credential; mirror of §6/§10 — the PRIVATE key ends only in the Secret):

ssh-keygen -t ed25519 -f agent-poke-ssh -N "" -C hub-poke@felhom
sudo kubectl -n felhom-system create secret generic agent-poke --from-file=key=agent-poke-ssh
# keep agent-poke-ssh.pub for the box step below, THEN:
shred -u agent-poke-ssh   # the private key now exists ONLY in the Secret

(The pinned host key + endpoint address are REUSED from the peersync env — no new hostkey entry. manifests/hub.yaml mounts the Secret at /etc/hub-secrets/agent-poke/key via POKE_SSH_KEY_FILE; absent Secret → the hub logs "agent-plane poke disabled".)

On the box (ep0), as root — additive; the peersync authorized_keys lines are untouched:

tr -d '\r' < felhom-poke.sh > /tmp/fp && sh -n /tmp/fp
install -o root -g root -m 0755 /tmp/fp /usr/local/bin/felhom-poke && rm /tmp/fp
printf 'restrict,command="/usr/local/bin/felhom-poke" %s\n' "$(cat agent-poke-ssh.pub)" \
  >> /home/felhom-peersync/.ssh/authorized_keys

Verify (from the build server with the key before it is shredded — pin the host key from §6's on-box read, never keyscan): ssh -i agent-poke-ssh felhom-peersync@<ep0> 10.77.0.<box> → prints poke-fired, and the box's agent journal logs poke received → triggering an immediate desired-state cycle. A non-WG target (ssh … 8.8.8.8) → refused non-WG target, non-zero exit. Hub v0.59.0 logs agent-plane poke enabled (endpoint 167.233.158.164:22, user felhom-peersync) on start.