felhom.eu/documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md

# SPIKE — immediate-sync transport (hub→box "sync now" signal) — 2026-07-16

> **STATUS: COMPLETE — verdict below.** Empirically measures the two candidate transports for the
> hub→box immediacy signal BEFORE any production spec. No production code shipped; no
> agent/controller/hub image; every live mutation inventoried and reverted (§CLEANUP, verified).
> The question under test (Viktor's ruling): a user-triggered config change must round-trip in
> **seconds**, not the 15-min report cycle — so the hub needs a "sync now" wake-up and the box then
> pulls config via the NORMAL channel (report ACK / desired-state fetch; the hub never delivers
> config by connecting inbound).

**Class:** SPIKE (empirical; no product code). **Repos:** `felhom.eu` (this doc only). All others
read-only for grounding: `hub/internal/wgsync/{client,reconciler}.go`, `hub/cmd/hub/main.go`,
`manifests/hub.yaml`, `felhom-agent/internal/desired/syncer.go`.

**Baselines re-confirmed at session start (git HEAD):** felhom.eu `a50822af` v0.57.0 · felhom-agent
`c040c180` v0.88.0 · felhom-controller `8f3564c1` v0.138.0. Live hub image `:0.57.0`.

**Probe ends (demo arc ONLY; Peti's arc untouched):** box = `felhom-pve`/demo-felhom
(`192.168.0.162`, LAN SSH = safety line) · endpoint = `felhom-hetzner`/ep0 (`167.233.158.164`,
root SSH, wg0 `10.77.0.1`) · hub pod on DooPlex k3s (`192.168.0.180`, `kubectl -n felhom-system`).

---

## Verdict (one line)

**Both transports are GO and they are COMPLEMENTARY, not competing.** (b) **long-poll** delivers to
the **controller** (the config puller) with a **hub-only** change and **zero ep0 coupling**, and is
the lighter fit for Viktor's user-config round-trip — but only after two 60 s ceilings are raised
(nginx `proxy_read_timeout` **measured at 60 s**, and the hub's in-code `WriteTimeout` **60 s**);
today a naive long-poll is capped at ~55 s → ~1 reconnect/box/min. (a) **ep0-relayed poke** delivers
to the **agent** (WG terminates host-side) in **~0.42 s per fresh SSH session** (≈27 ms+18 ms if the
session is pooled), needs **zero infra change on ep0 or the box** (origination from wg0's own IP
never touches the forward chain — proven), and is cryptographically confined to registered peers —
but it needs an **agent train** (a new UDP listener→heartbeat trigger) and a hub forced-command key
on ep0. Recommend **(b) as the primary immediacy path for customer/app config** (hub+controller
only), with **(a) reserved as the agent-plane desired-state nudge** already aligned with the OOB /
mutual-repair arc. **Interval shortening stays REJECTED.**

---

## P0 — ground truth (no changes) — GATE PASSED

All three ends reachable → proceed. Redacted reads only; `wg show <if> dump` never run (S1 key-leak
ban); private keys never printed.

### DNS / fronting — **Viktor's Q2 answered: DNS-only, NO Cloudflare**
- **Public:** `hub.felhom.eu` → CNAME `dooplex.hopto.org` → **37.191.56.193** (the operator's
  One-Hungary fixed-cable line — the SAME public IPv4 the demo box sits behind). No `cf-ray`, no
  `server: cloudflare`.
- **On-LAN split-horizon** (resolver `192.168.0.250`): `hub.felhom.eu` → **192.168.0.192** (the
  `nginx-internal` ingress LB, `nginx-system/ingress-nginx-internal-controller`).
- `curl -sI` on-LAN `/healthz` → `HTTP/1.1 200`, HSTS, no CDN headers. From ep0 via the public IP
  (`--resolve hub:443:37.191.56.193`) → `HTTP/2 403` = the **HU geo-gate** (ep0 is in DE; the gate
  is a location-level `configuration-snippet` on the hub `/` path). The path reaches nginx over
  HTTP/2 and returns immediately.
- **Real ingress chain:** `[client] → internet → 37.191.56.193 (operator home-router NAT) → k3s
  nginx-internal → hub pod:8080`. Self-hosted, no CDN in front. (Controller→hub and agent→hub both
  ride this same DNS-only nginx path — the controller's Cloudflare tunnel is its *inbound* edge only.)

### Box (`felhom-pve` / demo-felhom)
`wg-felhom`: `Address 10.77.0.2/32`, MTU 1280; peer `f3d1ZI7…`(ep0) `Endpoint 167.233.158.164:443`,
`AllowedIPs 10.77.0.1/32, 10.77.0.250/32`, `PersistentKeepalive 25`. `registered.json`: assigned
`10.77.0.2/32`, **generation 11**, pubkey `aqMpucg…`. Handshake fresh (~1 min); `wg-quick@wg-felhom`
+ `felhom-agent` both active; **agent 0.88.0**. Box **nft empty** (no host belt; pve-firewall
disabled) — matches the 07-05 baseline.
- *Note vs 07-05:* the box's WG **identity rotated** (new pubkey) on the R-21 ISO reinstall but the
  assigned IP was preserved (`10.77.0.2`). The box `AllowedIPs` now **already carries
  `10.77.0.250/32`** — i.e. the 07-05 spike's finding-1 gap ("render the operator /32 into the agent
  conf") has since been **closed**; the OOB operator peer is durably rendered.

### Endpoint (`felhom-hetzner`, wg0 :443)
Three peers: box `aqMpucg… → 10.77.0.2/32` (fresh); a **second real box** `zxy… → 10.77.0.3/32`
(handshake ~3h49m — **not touched**); operator `uDPlMa0… → 10.77.0.250/32` (no endpoint — dormant
OOB peer). **`ip_forward = 1`** (ipv6 fwd 0). `nft inet filter`: `input` policy DROP with allows
(established, lo, tcp/22, udp/443, tcp/8007 iif wg0, icmp); `forward` policy DROP and **empty** (no
accept rules). `authorized_keys`: root **2** lines, `felhom-peersync` **2** lines.
- *Note vs 07-05:* `ip_forward` was **0** at the 07-05 baseline and is **1** now — the OOB
  operator-peer feature has shipped in the interim. **The live baseline (`ip_forward=1`, forward
  chain empty) is therefore the cleanup target — NOT the stale "must be 0" assumption in the plan.**

### Hub pod
`hub-7b97cfb48d-qrbpb` Running, image `:0.57.0`. `exec` works (Alpine, root). **Reconciler ALIVE:**
`[INFO] wgsync: pushed 3 peers to 167.233.158.164:22` every 5 min — so **hub-pod → ep0:22 already
works in production** (the forced-command peersync channel a poke-relay would clone).

---

## P1 — ep0→box UDP origination (the poke's last hop) — **GO, zero endpoint change**

Listener on the box bound to its WG IP (`10.77.0.2:19999`) + `tcpdump -ni wg-felhom`; ep0 originates
from wg0.

1. **Delivery, packet-proven:** ep0 `python3 sendto("10.77.0.2",19999)` from `10.77.0.1`. Box:
   `RECV b'sync-poke-p1' from ('10.77.0.1', 47280)`. tcpdump captured **2** pkts
   `10.77.0.1.* > 10.77.0.2.19999` (python + a `nc` variant).
2. **The load-bearing claim — ZERO mutation needed:** `ip_forward` **1 → 1 unchanged**; `forward`
   chain **still empty policy-drop**; no nft/peer/route change. Endpoint-**originated** traffic
   leaves via wg0's output path and **never traverses the `forward` chain** — so the poke needs no
   `ip_forward`, no forward rule, no nft edit. (This is exactly why forwarding being on or off is
   irrelevant to the relay: the poke is origination, not transit.)
3. **Latency over the tunnel:** ping **avg 29.0 ms** (min 22.6 / max 36.9 / mdev 4.8); HTTP
   round-trip connect ~23–31 ms, total ~55–66 ms (5/5 `http=200`).
4. **Negative — cryptographically confined to registered peers:** poking an **unregistered**
   in-subnet IP (`10.77.0.199`; the route exists, `dev wg0 src 10.77.0.1`) fails at the kernel:
   `OSError: [Errno 126] Required key not available` (EKEYREJECTED) — WireGuard refuses to encrypt
   to an IP no peer owns. wg0 peer count unchanged (3). The future forced command can therefore
   **only** ever reach a registered peer's `/32`; there is no "spray the subnet" failure mode.

---

## P2 — hub-pod → ep0 → box poke round-trip — **GO, ~0.42 s per poke (target was ≤2–3 s)**

Throwaway keypair generated **on the DooPlex host** (`/tmp/spike-poke-key`, 0600, private key never
printed); **public** key installed on ep0 root `authorized_keys` as
`command="/root/spike-poke.sh",restrict …` (root ak **2 → 3**, 1 spike line). The forced command is
one UDP `sendto` to the box.

1. **Pod capability:** the hub image is **Alpine** — it has `nc` but **no `ssh`, no `bash`, no
   `python3`**. So a production poke MUST reuse the **in-process `golang.org/x/crypto/ssh`** client
   (the `wgsync` pattern), not a shelled-out binary. Pod→ep0:22 reachability from the **pod netns**
   confirmed: `nc -z -w4 167.233.158.164 22` = OK (and the reconciler proves it continuously).
2. **End-to-end, 5 timed pokes** (DooPlex host `--ssh--> ep0 forced-cmd --> UDP --> box`): all 5
   delivered (box `RECV #1–5 b'hub-poke' from 10.77.0.1`), remote returned `poke-fired`.
   `ssh_wall`: **0.636 s (cold), 0.423, 0.439, 0.414, 0.412** → warm median **~0.42 s**.
3. **Cost decomposition:** forced-command body (python startup + `sendto`) = **18–20 ms** (bare
   python3 = 11–12 ms); DooPlex→ep0 net RTT = **27 ms**. So **~0.40 s of the 0.42 s is SSH
   TCP+KEX+auth+shell-spawn** — the session setup dominates, the poke itself is negligible.
   **Task-2 lever:** a **pooled/persistent** SSH session (or the in-process client holding one
   channel) collapses per-poke to ~**1 RTT (27 ms) + body**; even a fresh session per poke is already
   sub-second.
4. **Failure behavior:** with **no box listener**, the poke still fires on ep0 and the datagram is
   silently discarded at the box — **no error path back to the hub** (a lost poke is harmless by
   construction; the next scheduled report reconciles). **Stale handshake:** NOT force-tested —
   forcing it means stopping keepalive/the tunnel, which risks the live offsite/PBS path (STOP
   rule). From WG semantics + observation: `PersistentKeepalive 25` kept the handshake < 125 s fresh
   throughout; a poke arriving on a cold handshake triggers WG's auto-rehandshake on first tx (~1
   extra RTT), datagram buffered then delivered — **not dropped**.

---

## P3 — long-poll hold-time through the REAL ingress chain

**Method (why no DNS record was created):** a throwaway **sleeper** (a `python:3.14-bookworm` pod
that `sleep`s `?hold=N` before a 200) + Service + **Ingress on host `probe.felhom.eu`, class
`nginx-internal`**, in `felhom-system`. Reached with `curl --resolve probe.felhom.eu:443:<ip> -k`
from each vantage — so **no DNS record was needed** (the plan's fallback "ask Viktor for an A
record" was avoided), **the live hub Ingress was never touched**, and the sleeper's own server block
carries **no geo snippet** (so the remote ep0 vantage is not geo-blocked). Default cert → `curl -k`.
The `nginx-internal` ConfigMap sets **no** custom proxy timeouts → nginx built-in defaults.

### Pass 1 — DEFAULT ingress, on-LAN box (direct nginx, no NAT)
| hold N | result |
|---|---|
| 30 s | `200 @ 30.0 s` |
| 55 s | `200 @ 55.0 s` |
| 70 s | **KILLED** |
| 100 s | **KILLED** |

nginx error log (authoritative): `upstream timed out (110: Operation timed out) while reading
response header from upstream … GET /?hold=70 HTTP/2.0`. **Killing layer = nginx
`proxy_read_timeout` = 60 s** (built-in default). Each request appears **twice** ~60 s apart =
ingress-nginx's default single upstream **retry on timeout**, so the client-visible 504 lands at
**~120 s** (2 × 60 s); the client's own cut at 110 s showed `http=000` just before it. The sleeper's
`BrokenPipeError` confirms nginx abandoned the read at 60 s.
- **H(default) = ~55 s usable / hard kill at 60 s** → naive per-box reconnect rate ≈ **1/min**.
- **In-code twin ceiling:** the hub's own `http.Server{WriteTimeout: 60s}` (`hub/cmd/hub/main.go:339–341`)
  caps a held response at ~60 s regardless of ingress — coincident with nginx's 60 s. **Both must be
  raised** for any long-poll (nginx via annotation; the hub per-endpoint via `http.ResponseController`).

### Pass 2 — TUNED sleeper ingress (`proxy-read-timeout`/`proxy-send-timeout`=3600, `proxy-next-upstream`=off)
- **on-LAN** `hold=120` → `200 @ 120.0 s` (was killed under default → **nginx WAS the layer**, and
  raising it lifts the ceiling cleanly; on-LAN there is no further layer below the pure-nginx value).
- **on-LAN** `hold=600` → `200 @ 600.0 s` (pure tuned-nginx holds a full **10 min** cleanly — no
  further on-LAN layer below 600 s).
- **ep0 remote** (through the operator home-router NAT `37.191.56.193`): `hold=2` → `200 @ 2.1 s`
  (routes, not geo-blocked); `hold=200` → `200 @ 200.09 s`; **`hold=600` → `200 @ 600.09 s`** — the
  **operator home NAT holds an idle held connection for the full 10 min** without eviction. So on
  this arc the NAT is **not** the binding constraint (≤ 600 s); the only ceiling today is the twin
  60 s nginx+hub limits, **both config-fixable**. Once raised, a long-poll can hold **many minutes**
  even through the operator NAT → reconnect rate drops from ~1/min toward ~1/10-min.

### Vantage caveat (honest ledger)
The demo box is **on-LAN, co-located behind the SAME operator NAT** as the hub, so it does **not**
traverse a real customer's `home-NAT → internet → operator-NAT` double-NAT path. The ep0 vantage
represents the **ingress-side (operator) NAT** only; a **customer-side home-router idle timeout** is
**unrepresented** by this arc. The load-bearing measurement — that **nginx's 60 s default is the
binding ceiling today, and it is a config lift** — holds regardless of vantage.

---

## P4 — verdict: (a) poke-relay vs (b) long-poll

| Dimension | **(a) ep0-relayed poke** | **(b) long-poll** |
|---|---|---|
| Measured latency | **~0.42 s** per fresh SSH session (≈**27 ms + 18 ms** if pooled); last hop ep0→box **~29 ms** | delivers on the next held-GET completion; today **capped at ~55 s** by the nginx+hub 60 s ceilings (both fixable) |
| Reaches | the **agent** (WG terminates host-side) → desired-state nudge | the **controller** (the config puller) directly — also usable on the agent channel |
| Hub change | new forced-command poke key + in-process SSH poke call | new hanging-GET endpoint + **raise nginx & hub WriteTimeout** |
| ep0 change | **none** (P1: origination needs no forward/nft/sysctl) | **none** |
| Box/agent change | **agent train** — new UDP listener → "heartbeat now" trigger | **none** if the controller holds the GET (it already pulls config); or the agent |
| ep0 coupling | **yes** (rides the peersync SSH pattern) | **no** (direct hub↔box outbound over the existing ingress) |
| WG down | poke fails (WG is the transport) — but agent self-heals WG in ~15 s (07-05) and a lost poke is harmless; next report catches up | **unaffected** (uses the ingress path, not WG) |
| Hub restart | forced-command is stateless; resumes on next trigger | held GETs drop → box must reconnect (reconnect logic needed) |
| NAT rebind | WG keepalive re-handshakes; poke buffered | held GET dies silently on rebind → client hold must be < NAT idle timeout |

**Recommended Task-2 shape — complementary, not either/or:**
- **(b) long-poll is the primary immediacy path for Viktor's actual ask** (user-triggered
  customer/app **config**). It reaches the **controller — the component that already pulls config** —
  with a **hub-only** change (a hanging-GET endpoint) plus raising the two 60 s ceilings, **no agent
  train, no ep0 coupling, no WG dependency**. The box holds a GET with a hold window comfortably
  under the ingress/NAT ceiling (e.g. 45–55 s today, or longer once the ceilings are raised) and
  reconnects; the hub completes the GET the instant operator intent changes.
- **(a) poke-relay is the natural agent-plane nudge** for host-domain desired-state (guests,
  wireguard) — it reaches the **agent**, needs **zero ep0/box infra change** (P1), rides the OOB /
  mutual-repair arc already in place, and is **sub-second**. Its cost is an agent train (UDP
  listener → heartbeat trigger). Reserve it for the agent's desired-state, where the long-poll's
  "reaches the controller" advantage doesn't apply.

**Interval shortening stays REJECTED.** Shrinking the 900 s hub poll / 15-min report cycle
fleet-wide multiplies steady-state load on **every** box and the hub **continuously**, for an event
that is **rare and user-triggered**, and still does not deliver true immediacy (latency is bounded by
the shortened interval, not seconds). A wake signal delivers seconds-latency at ~zero idle cost.
Rejected.

---

## CLEANUP — inventory & assertion

Every live mutation, reverted (revised to the LIVE P0 baseline, not the stale 07-05 assumptions):
- **ep0:** spike forced-command line removed from `/root/.ssh/authorized_keys` (`grep -c
  spike-immediate-sync` = **0**, back to **2** lines); `/root/spike-poke.sh` `shred -u`'d;
  **`ip_forward` still 1** (never touched — the poke needs none); `forward` chain still empty; wg0
  peer list unchanged (3, incl. the untouched second box `10.77.0.3` and dormant operator
  `10.77.0.250`).
- **DooPlex / k8s:** sleeper `Deployment` + `Service` + `Ingress` + `ConfigMap` (all labelled
  `app: spike-immediate-sync`) deleted; `/tmp/spike-sleeper.yaml` and `/tmp/spike-poke-key*` removed.
  **No change to the hub Ingress, the hub Deployment, or any felhom-system object.**
- **Box:** all spike listeners / tcpdumps / http.servers gone (timeout-bounded); `/tmp/spike-*`
  removed; **nft still empty**; `wg-felhom` unchanged.
- **Final re-verify (all green):** box `wg-quick@wg-felhom` + `felhom-agent` **active**, tunnel live
  (`pvesm status felhom-offsite` = **active**; box↔ep0 last-handshake epoch matches on both ends),
  guest 9201 **running** + `felhom-controller Up (healthy)`; ep0 wg0 **3 peers unchanged**
  (second box `10.77.0.3` + dormant operator `10.77.0.250` untouched); hub pod **1/1 Running**,
  reconciler still pushing 3 peers. ep0 `ip_forward` **1** (never changed), `forward` chain empty,
  root `authorized_keys` back to **2** lines (0 spike lines).
- **No repo/hub/agent/manifest production change** — this commit is docs-only.

---

## Method bar honored

`wg show <if> dump` never run (S1 key-leak ban); only `latest-handshakes`/redacted conf reads.
Throwaway keys generated on the box that owns them, 0600, private keys never printed. Every negative
carries kernel/nft/packet-level evidence (EKEYREJECTED errno; nginx upstream-timeout log line;
tcpdump counts), not bare timeouts. The live baseline was re-measured rather than assumed — the two
places the plan's assumptions had gone stale (`ip_forward` now 1; the operator `/32` now rendered)
are recorded as findings. No probe touched Peti's arc, the hub DB, the live peer list, or the
box's rendered WG conf.