From 10e07f57473c5c34d28218b95ee0ea664fad92cc Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Thu, 16 Jul 2026 20:00:01 +0200 Subject: [PATCH] =?UTF-8?q?docs(spike):=20immediate-sync=20transport=20?= =?UTF-8?q?=E2=80=94=20poke-relay=20vs=20long-poll,=20both=20GO/complement?= =?UTF-8?q?ary?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Empirical spike (demo arc, docs-only, no product code). Measures the two hub->box 'sync now' transports. Verdict: (b) long-poll = primary for user config (hub-only, reaches controller, no ep0/WG/agent coupling; binding ceiling = twin 60s nginx proxy_read_timeout + hub WriteTimeout, both config lifts); (a) ep0-relayed poke = agent-plane nudge (~0.42s, zero ep0/box infra change, EKEYREJECTED-confined). Viktor Q2 answered: hub.felhom.eu is DNS-only, no Cloudflare. All live mutations reverted; arc re-verified green. --- REPORT.md | 101 ++++--- ...IKE-immediate-sync-transport-2026-07-16.md | 257 ++++++++++++++++++ 2 files changed, 304 insertions(+), 54 deletions(-) create mode 100644 documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md diff --git a/REPORT.md b/REPORT.md index 9e9f1b1..3c35f26 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,63 +2,56 @@ > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md). -## Reinstall-of-existing-customer arc — F2/F3/F4/2.3 (hub v0.57.0) — 2026-07-16 +## SPIKE — immediate-sync transport (hub→box "sync now") — 2026-07-16 -Closes the N100 physical-run findings (`documentation/tests/VALIDATION-n100-baremetal-2026-07-16.md`). -**Only the hub shipped** — controller + scripts unchanged (both spec premises contradicted by source, -below). Baseline: felhom.eu @ `dd961a6`, hub v0.56.0 → **v0.57.0**. +**Docs-only spike.** No production code, no image, no manifest change. Every live mutation on the +demo arc was inventoried and reverted; the arc was re-verified green. Findings doc: +[`documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md`](documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md). +Baselines re-confirmed at session start: felhom.eu `a50822af` v0.57.0 · felhom-agent `c040c180` +v0.88.0 · felhom-controller `8f3564c1` v0.138.0. -### Fork verdicts (source-verified) -- **2.1 = FORK B.** `hub/internal/claim/engine.go` + `handler.go:413-429` (`MarkClaimed`, set-only): the - hub stores the claim code + a **claimed boolean**, never the password hash (controller-owned by the - arc's design). Fork A (hub delivers the hash) would violate the design → **B** (bump generation + email). -- **3.1 = no controller change (spec premise contradicted).** `controller/internal/agentapi/escrow.go` - `EscrowPreflight` fetches the prereq checklist **live from the agent** (`GET /escrow/preflight`); it - never sources `pbs_storage_id` from hub config. The agent seeds it on PBS-DR convergence. The fresh - box "lacked" it only because F4's 403 blocked convergence — Part 0 + the demo grant restore it. **No - controller v0.138.0.** +### Question +Viktor's ruling: a user-triggered config change must round-trip in **seconds**, not the 15-min +report cycle. Measure the two candidate "sync now" transports empirically before any Task-2 spec. -### Part 0 — live unblock (done, reversible) -Granted `FelhomAgentStore` on `/storage/felhom-offsite` (user + token) on felhom-pve. The agent's next -tick converged: created the storage, ran `felhom-pbs-apply grant`, **seeded `escrow.pbs_storage_id`**, -`pbsdr: converged state=applied`; `pvesm` shows **felhom-offsite pbs active**. The 403s stopped. +### Verdict — both GO, and complementary (not competing) +- **(b) long-poll — recommended primary for user config.** A hanging GET the hub completes on + change. Reaches the **controller** (the config puller) with a **hub-only** change, **no ep0 + coupling, no WG dependency, no agent train**. **Binding ceiling today = 60 s**, from TWO places + measured/sourced: nginx `proxy_read_timeout` **60 s** (proven from the upstream-timeout log line; + ingress-nginx's default single retry pushes the client-visible 504 to ~120 s) **and** the hub's + in-code `http.Server{WriteTimeout: 60s}` (`hub/cmd/hub/main.go:339-341`). Both are config lifts. +- **(a) ep0-relayed poke — reserved as the agent-plane nudge.** Hub → forced-command on ep0 → ep0 + originates a contentless UDP datagram → box. Reaches the **agent** (WG terminates host-side), + **~0.42 s** per fresh SSH session (~0.40 s of that is SSH setup; a pooled session → ~27 ms + 18 ms). + Needs **zero infra change on ep0 or the box** and is cryptographically confined to registered + peers — but needs an agent train (UDP listener → heartbeat trigger). +- **Interval shortening stays REJECTED** (fleet-wide continuous load, no true immediacy). -### Part 2 — hub v0.57.0 (shipped) -- **F2 claim continuity** — `claim.ReissueForReenroll` (claimed → bump generation once + reset email; - no-op unclaimed), hooked at the host-enroll **mint path** (single-bump-per-re-enroll). Event - `claim_reissued_reenroll`. -- **F3 offsite continuity** — the re-enroll path calls `web.ReissueOffsiteForCustomer` (same machinery - as the manual button; wired via `SetOffsiteReissuer`). Event `offsite_reissued`. -- **2.3 escrow honesty** — `offsite.ReissueCredentials` marks the escrow stale (`store.MarkEscrowStale`; - cleared by the next ceremony); the ACK **withholds** the mismatched `restic_pw_sha256`; the DR - checklist shows **stale** not "done." Event `escrow_stale`. +### Evidence highlights +- **Viktor's Q2:** `hub.felhom.eu` is **DNS-only** (CNAME → `dooplex.hopto.org` → the operator home + line `37.191.56.193`), **no Cloudflare**; split-horizon to the `nginx-internal` LB `192.168.0.192` + on-LAN. Real chain: client → operator home-NAT → nginx-internal → hub pod. +- **P1** ep0→box origination works with **`ip_forward` and the `forward` chain untouched** (endpoint- + originated traffic never transits the forward chain); ~29 ms last hop; negative proven at the + kernel crypto layer (`EKEYREJECTED` to an unregistered `/32`). +- **P2** 5/5 pokes delivered end-to-end; the hub image is Alpine (no ssh/bash/python) → a production + poke must reuse the in-process `x/crypto/ssh` client (the `wgsync` pattern); a lost poke is + harmless by construction. +- **P3** tuned ingress holds cleanly to **600 s on-LAN**, and **the operator home NAT holds an idle + held connection the full 600 s** — so the NAT is not the constraint (≤10 min); only the twin 60 s + config limits are. -### Scenarios / red-proofs -- **A (claim)**: `TestReissueForReenroll` — claimed rotates + reset email; unclaimed no-op. PASS. -- **C (escrow, red-proof)**: `TestReissue_InvalidatesEscrow` — after a re-issue the escrow is STALE and - the hash is withheld. **Red-proof run-fail-revert:** with `MarkEscrowStale` gutted → FAIL ("escrow - must be STALE… advertising ceremony-done against a key the repo no longer accepts"); restored → PASS. -- Green gate: `go build ./... && go vet ./... && go test ./...` all clean. -- **Deployed:** `felhom-hub:0.57.0` (manifest `dd563a0`, ArgoCD Synced/Healthy, rolled out, clean - startup incl. the `stale_at` migration). Controller/scripts/agent unchanged. -- **Scenario E (live demo):** the agent's `/escrow/preflight` is **all 6 green** — `pbs_storage_id` - (felhom-offsite), `dr_tier` applied, `age_binary`, `hub_upload`, `staged_secret`, `sudo_grant`; - overall `ok`. The wizard's start is offered. **CC STOPPED — the ceremony is Viktor's step.** +### Ground-truth drift caught since the 07-05 OOB spike (recorded as findings) +- ep0 `ip_forward` is now **1** (was 0) — the OOB operator-peer feature has shipped; a **second real + box** (`10.77.0.3`) is now on the arc (untouched). +- The operator `/32` (`10.77.0.250`) is now **durably rendered** into the box's `wg-felhom` conf — + the 07-05 finding-1 gap is closed. +- Cleanup bar was set to the **live** baseline (`ip_forward=1`, root ak = 2), not the plan's stale + "must be 0" assumption. -### Observation (agent-side, recorded not acted on) -The agent seeds `escrow.pbs_storage_id` into agent.json on PBS-DR convergence, but the **running** -agent's escrow-preflight kept reporting "not configured" until a `systemctl restart felhom-agent` -(config reload). The seed writes the file but doesn't propagate to the live preflight path — a small -agent-reload gap to fold into the R-22 agent train. - -### F4 — NOT an installer fix (reported) -The DR storage id lives in the agent-domain **pbs_dr descriptor** (`web/pbsdr.go`), provisioned post-WG -— not in the installer-fetched config — so `step_agent_config()` can't derive it. The block is a -bootstrap circularity: the agent's token-auth pre-check `GET /storage/` 403s and aborts before its -own root-run `felhom-pbs-apply grant`. **Root fix is agent-side → ROADMAP R-22 (agent train)**; the demo -was unblocked live (Part 0). Default-storage-id customers (all new/Peti installs) already work — F4 only -bites non-default ids. Per Viktor: do NOT flip the demo's descriptor id. - -### NOT done -The escrow **ceremony itself** — Viktor's S10 step (CC's boundary is prereqs-green; CC never runs the -ceremony, touches R, or mints a recovery code). +### Cleanup (verified) +ep0 forced-command line removed (root ak 2→2, 0 spike lines), poke script shredded, `ip_forward` +still 1, forward chain empty; k8s sleeper (deploy/svc/ingress/cm) deleted; throwaway keypair +shredded; box nft empty, no listeners, no temp files. Arc green: tunnel live (`felhom-offsite` +active), guest 9201 controller healthy, hub pod 1/1, reconciler still pushing 3 peers. diff --git a/documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md b/documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md new file mode 100644 index 0000000..b3a439b --- /dev/null +++ b/documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md @@ -0,0 +1,257 @@ +# SPIKE — immediate-sync transport (hub→box "sync now" signal) — 2026-07-16 + +> **STATUS: COMPLETE — verdict below.** Empirically measures the two candidate transports for the +> hub→box immediacy signal BEFORE any production spec. No production code shipped; no +> agent/controller/hub image; every live mutation inventoried and reverted (§CLEANUP, verified). +> The question under test (Viktor's ruling): a user-triggered config change must round-trip in +> **seconds**, not the 15-min report cycle — so the hub needs a "sync now" wake-up and the box then +> pulls config via the NORMAL channel (report ACK / desired-state fetch; the hub never delivers +> config by connecting inbound). + +**Class:** SPIKE (empirical; no product code). **Repos:** `felhom.eu` (this doc only). All others +read-only for grounding: `hub/internal/wgsync/{client,reconciler}.go`, `hub/cmd/hub/main.go`, +`manifests/hub.yaml`, `felhom-agent/internal/desired/syncer.go`. + +**Baselines re-confirmed at session start (git HEAD):** felhom.eu `a50822af` v0.57.0 · felhom-agent +`c040c180` v0.88.0 · felhom-controller `8f3564c1` v0.138.0. Live hub image `:0.57.0`. + +**Probe ends (demo arc ONLY; Peti's arc untouched):** box = `felhom-pve`/demo-felhom +(`192.168.0.162`, LAN SSH = safety line) · endpoint = `felhom-hetzner`/ep0 (`167.233.158.164`, +root SSH, wg0 `10.77.0.1`) · hub pod on DooPlex k3s (`192.168.0.180`, `kubectl -n felhom-system`). + +--- + +## Verdict (one line) + +**Both transports are GO and they are COMPLEMENTARY, not competing.** (b) **long-poll** delivers to +the **controller** (the config puller) with a **hub-only** change and **zero ep0 coupling**, and is +the lighter fit for Viktor's user-config round-trip — but only after two 60 s ceilings are raised +(nginx `proxy_read_timeout` **measured at 60 s**, and the hub's in-code `WriteTimeout` **60 s**); +today a naive long-poll is capped at ~55 s → ~1 reconnect/box/min. (a) **ep0-relayed poke** delivers +to the **agent** (WG terminates host-side) in **~0.42 s per fresh SSH session** (≈27 ms+18 ms if the +session is pooled), needs **zero infra change on ep0 or the box** (origination from wg0's own IP +never touches the forward chain — proven), and is cryptographically confined to registered peers — +but it needs an **agent train** (a new UDP listener→heartbeat trigger) and a hub forced-command key +on ep0. Recommend **(b) as the primary immediacy path for customer/app config** (hub+controller +only), with **(a) reserved as the agent-plane desired-state nudge** already aligned with the OOB / +mutual-repair arc. **Interval shortening stays REJECTED.** + +--- + +## P0 — ground truth (no changes) — GATE PASSED + +All three ends reachable → proceed. Redacted reads only; `wg show dump` never run (S1 key-leak +ban); private keys never printed. + +### DNS / fronting — **Viktor's Q2 answered: DNS-only, NO Cloudflare** +- **Public:** `hub.felhom.eu` → CNAME `dooplex.hopto.org` → **37.191.56.193** (the operator's + One-Hungary fixed-cable line — the SAME public IPv4 the demo box sits behind). No `cf-ray`, no + `server: cloudflare`. +- **On-LAN split-horizon** (resolver `192.168.0.250`): `hub.felhom.eu` → **192.168.0.192** (the + `nginx-internal` ingress LB, `nginx-system/ingress-nginx-internal-controller`). +- `curl -sI` on-LAN `/healthz` → `HTTP/1.1 200`, HSTS, no CDN headers. From ep0 via the public IP + (`--resolve hub:443:37.191.56.193`) → `HTTP/2 403` = the **HU geo-gate** (ep0 is in DE; the gate + is a location-level `configuration-snippet` on the hub `/` path). The path reaches nginx over + HTTP/2 and returns immediately. +- **Real ingress chain:** `[client] → internet → 37.191.56.193 (operator home-router NAT) → k3s + nginx-internal → hub pod:8080`. Self-hosted, no CDN in front. (Controller→hub and agent→hub both + ride this same DNS-only nginx path — the controller's Cloudflare tunnel is its *inbound* edge only.) + +### Box (`felhom-pve` / demo-felhom) +`wg-felhom`: `Address 10.77.0.2/32`, MTU 1280; peer `f3d1ZI7…`(ep0) `Endpoint 167.233.158.164:443`, +`AllowedIPs 10.77.0.1/32, 10.77.0.250/32`, `PersistentKeepalive 25`. `registered.json`: assigned +`10.77.0.2/32`, **generation 11**, pubkey `aqMpucg…`. Handshake fresh (~1 min); `wg-quick@wg-felhom` ++ `felhom-agent` both active; **agent 0.88.0**. Box **nft empty** (no host belt; pve-firewall +disabled) — matches the 07-05 baseline. +- *Note vs 07-05:* the box's WG **identity rotated** (new pubkey) on the R-21 ISO reinstall but the + assigned IP was preserved (`10.77.0.2`). The box `AllowedIPs` now **already carries + `10.77.0.250/32`** — i.e. the 07-05 spike's finding-1 gap ("render the operator /32 into the agent + conf") has since been **closed**; the OOB operator peer is durably rendered. + +### Endpoint (`felhom-hetzner`, wg0 :443) +Three peers: box `aqMpucg… → 10.77.0.2/32` (fresh); a **second real box** `zxy… → 10.77.0.3/32` +(handshake ~3h49m — **not touched**); operator `uDPlMa0… → 10.77.0.250/32` (no endpoint — dormant +OOB peer). **`ip_forward = 1`** (ipv6 fwd 0). `nft inet filter`: `input` policy DROP with allows +(established, lo, tcp/22, udp/443, tcp/8007 iif wg0, icmp); `forward` policy DROP and **empty** (no +accept rules). `authorized_keys`: root **2** lines, `felhom-peersync` **2** lines. +- *Note vs 07-05:* `ip_forward` was **0** at the 07-05 baseline and is **1** now — the OOB + operator-peer feature has shipped in the interim. **The live baseline (`ip_forward=1`, forward + chain empty) is therefore the cleanup target — NOT the stale "must be 0" assumption in the plan.** + +### Hub pod +`hub-7b97cfb48d-qrbpb` Running, image `:0.57.0`. `exec` works (Alpine, root). **Reconciler ALIVE:** +`[INFO] wgsync: pushed 3 peers to 167.233.158.164:22` every 5 min — so **hub-pod → ep0:22 already +works in production** (the forced-command peersync channel a poke-relay would clone). + +--- + +## P1 — ep0→box UDP origination (the poke's last hop) — **GO, zero endpoint change** + +Listener on the box bound to its WG IP (`10.77.0.2:19999`) + `tcpdump -ni wg-felhom`; ep0 originates +from wg0. + +1. **Delivery, packet-proven:** ep0 `python3 sendto("10.77.0.2",19999)` from `10.77.0.1`. Box: + `RECV b'sync-poke-p1' from ('10.77.0.1', 47280)`. tcpdump captured **2** pkts + `10.77.0.1.* > 10.77.0.2.19999` (python + a `nc` variant). +2. **The load-bearing claim — ZERO mutation needed:** `ip_forward` **1 → 1 unchanged**; `forward` + chain **still empty policy-drop**; no nft/peer/route change. Endpoint-**originated** traffic + leaves via wg0's output path and **never traverses the `forward` chain** — so the poke needs no + `ip_forward`, no forward rule, no nft edit. (This is exactly why forwarding being on or off is + irrelevant to the relay: the poke is origination, not transit.) +3. **Latency over the tunnel:** ping **avg 29.0 ms** (min 22.6 / max 36.9 / mdev 4.8); HTTP + round-trip connect ~23–31 ms, total ~55–66 ms (5/5 `http=200`). +4. **Negative — cryptographically confined to registered peers:** poking an **unregistered** + in-subnet IP (`10.77.0.199`; the route exists, `dev wg0 src 10.77.0.1`) fails at the kernel: + `OSError: [Errno 126] Required key not available` (EKEYREJECTED) — WireGuard refuses to encrypt + to an IP no peer owns. wg0 peer count unchanged (3). The future forced command can therefore + **only** ever reach a registered peer's `/32`; there is no "spray the subnet" failure mode. + +--- + +## P2 — hub-pod → ep0 → box poke round-trip — **GO, ~0.42 s per poke (target was ≤2–3 s)** + +Throwaway keypair generated **on the DooPlex host** (`/tmp/spike-poke-key`, 0600, private key never +printed); **public** key installed on ep0 root `authorized_keys` as +`command="/root/spike-poke.sh",restrict …` (root ak **2 → 3**, 1 spike line). The forced command is +one UDP `sendto` to the box. + +1. **Pod capability:** the hub image is **Alpine** — it has `nc` but **no `ssh`, no `bash`, no + `python3`**. So a production poke MUST reuse the **in-process `golang.org/x/crypto/ssh`** client + (the `wgsync` pattern), not a shelled-out binary. Pod→ep0:22 reachability from the **pod netns** + confirmed: `nc -z -w4 167.233.158.164 22` = OK (and the reconciler proves it continuously). +2. **End-to-end, 5 timed pokes** (DooPlex host `--ssh--> ep0 forced-cmd --> UDP --> box`): all 5 + delivered (box `RECV #1–5 b'hub-poke' from 10.77.0.1`), remote returned `poke-fired`. + `ssh_wall`: **0.636 s (cold), 0.423, 0.439, 0.414, 0.412** → warm median **~0.42 s**. +3. **Cost decomposition:** forced-command body (python startup + `sendto`) = **18–20 ms** (bare + python3 = 11–12 ms); DooPlex→ep0 net RTT = **27 ms**. So **~0.40 s of the 0.42 s is SSH + TCP+KEX+auth+shell-spawn** — the session setup dominates, the poke itself is negligible. + **Task-2 lever:** a **pooled/persistent** SSH session (or the in-process client holding one + channel) collapses per-poke to ~**1 RTT (27 ms) + body**; even a fresh session per poke is already + sub-second. +4. **Failure behavior:** with **no box listener**, the poke still fires on ep0 and the datagram is + silently discarded at the box — **no error path back to the hub** (a lost poke is harmless by + construction; the next scheduled report reconciles). **Stale handshake:** NOT force-tested — + forcing it means stopping keepalive/the tunnel, which risks the live offsite/PBS path (STOP + rule). From WG semantics + observation: `PersistentKeepalive 25` kept the handshake < 125 s fresh + throughout; a poke arriving on a cold handshake triggers WG's auto-rehandshake on first tx (~1 + extra RTT), datagram buffered then delivered — **not dropped**. + +--- + +## P3 — long-poll hold-time through the REAL ingress chain + +**Method (why no DNS record was created):** a throwaway **sleeper** (a `python:3.14-bookworm` pod +that `sleep`s `?hold=N` before a 200) + Service + **Ingress on host `probe.felhom.eu`, class +`nginx-internal`**, in `felhom-system`. Reached with `curl --resolve probe.felhom.eu:443: -k` +from each vantage — so **no DNS record was needed** (the plan's fallback "ask Viktor for an A +record" was avoided), **the live hub Ingress was never touched**, and the sleeper's own server block +carries **no geo snippet** (so the remote ep0 vantage is not geo-blocked). Default cert → `curl -k`. +The `nginx-internal` ConfigMap sets **no** custom proxy timeouts → nginx built-in defaults. + +### Pass 1 — DEFAULT ingress, on-LAN box (direct nginx, no NAT) +| hold N | result | +|---|---| +| 30 s | `200 @ 30.0 s` | +| 55 s | `200 @ 55.0 s` | +| 70 s | **KILLED** | +| 100 s | **KILLED** | + +nginx error log (authoritative): `upstream timed out (110: Operation timed out) while reading +response header from upstream … GET /?hold=70 HTTP/2.0`. **Killing layer = nginx +`proxy_read_timeout` = 60 s** (built-in default). Each request appears **twice** ~60 s apart = +ingress-nginx's default single upstream **retry on timeout**, so the client-visible 504 lands at +**~120 s** (2 × 60 s); the client's own cut at 110 s showed `http=000` just before it. The sleeper's +`BrokenPipeError` confirms nginx abandoned the read at 60 s. +- **H(default) = ~55 s usable / hard kill at 60 s** → naive per-box reconnect rate ≈ **1/min**. +- **In-code twin ceiling:** the hub's own `http.Server{WriteTimeout: 60s}` (`hub/cmd/hub/main.go:339–341`) + caps a held response at ~60 s regardless of ingress — coincident with nginx's 60 s. **Both must be + raised** for any long-poll (nginx via annotation; the hub per-endpoint via `http.ResponseController`). + +### Pass 2 — TUNED sleeper ingress (`proxy-read-timeout`/`proxy-send-timeout`=3600, `proxy-next-upstream`=off) +- **on-LAN** `hold=120` → `200 @ 120.0 s` (was killed under default → **nginx WAS the layer**, and + raising it lifts the ceiling cleanly; on-LAN there is no further layer below the pure-nginx value). +- **on-LAN** `hold=600` → `200 @ 600.0 s` (pure tuned-nginx holds a full **10 min** cleanly — no + further on-LAN layer below 600 s). +- **ep0 remote** (through the operator home-router NAT `37.191.56.193`): `hold=2` → `200 @ 2.1 s` + (routes, not geo-blocked); `hold=200` → `200 @ 200.09 s`; **`hold=600` → `200 @ 600.09 s`** — the + **operator home NAT holds an idle held connection for the full 10 min** without eviction. So on + this arc the NAT is **not** the binding constraint (≤ 600 s); the only ceiling today is the twin + 60 s nginx+hub limits, **both config-fixable**. Once raised, a long-poll can hold **many minutes** + even through the operator NAT → reconnect rate drops from ~1/min toward ~1/10-min. + +### Vantage caveat (honest ledger) +The demo box is **on-LAN, co-located behind the SAME operator NAT** as the hub, so it does **not** +traverse a real customer's `home-NAT → internet → operator-NAT` double-NAT path. The ep0 vantage +represents the **ingress-side (operator) NAT** only; a **customer-side home-router idle timeout** is +**unrepresented** by this arc. The load-bearing measurement — that **nginx's 60 s default is the +binding ceiling today, and it is a config lift** — holds regardless of vantage. + +--- + +## P4 — verdict: (a) poke-relay vs (b) long-poll + +| Dimension | **(a) ep0-relayed poke** | **(b) long-poll** | +|---|---|---| +| Measured latency | **~0.42 s** per fresh SSH session (≈**27 ms + 18 ms** if pooled); last hop ep0→box **~29 ms** | delivers on the next held-GET completion; today **capped at ~55 s** by the nginx+hub 60 s ceilings (both fixable) | +| Reaches | the **agent** (WG terminates host-side) → desired-state nudge | the **controller** (the config puller) directly — also usable on the agent channel | +| Hub change | new forced-command poke key + in-process SSH poke call | new hanging-GET endpoint + **raise nginx & hub WriteTimeout** | +| ep0 change | **none** (P1: origination needs no forward/nft/sysctl) | **none** | +| Box/agent change | **agent train** — new UDP listener → "heartbeat now" trigger | **none** if the controller holds the GET (it already pulls config); or the agent | +| ep0 coupling | **yes** (rides the peersync SSH pattern) | **no** (direct hub↔box outbound over the existing ingress) | +| WG down | poke fails (WG is the transport) — but agent self-heals WG in ~15 s (07-05) and a lost poke is harmless; next report catches up | **unaffected** (uses the ingress path, not WG) | +| Hub restart | forced-command is stateless; resumes on next trigger | held GETs drop → box must reconnect (reconnect logic needed) | +| NAT rebind | WG keepalive re-handshakes; poke buffered | held GET dies silently on rebind → client hold must be < NAT idle timeout | + +**Recommended Task-2 shape — complementary, not either/or:** +- **(b) long-poll is the primary immediacy path for Viktor's actual ask** (user-triggered + customer/app **config**). It reaches the **controller — the component that already pulls config** — + with a **hub-only** change (a hanging-GET endpoint) plus raising the two 60 s ceilings, **no agent + train, no ep0 coupling, no WG dependency**. The box holds a GET with a hold window comfortably + under the ingress/NAT ceiling (e.g. 45–55 s today, or longer once the ceilings are raised) and + reconnects; the hub completes the GET the instant operator intent changes. +- **(a) poke-relay is the natural agent-plane nudge** for host-domain desired-state (guests, + wireguard) — it reaches the **agent**, needs **zero ep0/box infra change** (P1), rides the OOB / + mutual-repair arc already in place, and is **sub-second**. Its cost is an agent train (UDP + listener → heartbeat trigger). Reserve it for the agent's desired-state, where the long-poll's + "reaches the controller" advantage doesn't apply. + +**Interval shortening stays REJECTED.** Shrinking the 900 s hub poll / 15-min report cycle +fleet-wide multiplies steady-state load on **every** box and the hub **continuously**, for an event +that is **rare and user-triggered**, and still does not deliver true immediacy (latency is bounded by +the shortened interval, not seconds). A wake signal delivers seconds-latency at ~zero idle cost. +Rejected. + +--- + +## CLEANUP — inventory & assertion + +Every live mutation, reverted (revised to the LIVE P0 baseline, not the stale 07-05 assumptions): +- **ep0:** spike forced-command line removed from `/root/.ssh/authorized_keys` (`grep -c + spike-immediate-sync` = **0**, back to **2** lines); `/root/spike-poke.sh` `shred -u`'d; + **`ip_forward` still 1** (never touched — the poke needs none); `forward` chain still empty; wg0 + peer list unchanged (3, incl. the untouched second box `10.77.0.3` and dormant operator + `10.77.0.250`). +- **DooPlex / k8s:** sleeper `Deployment` + `Service` + `Ingress` + `ConfigMap` (all labelled + `app: spike-immediate-sync`) deleted; `/tmp/spike-sleeper.yaml` and `/tmp/spike-poke-key*` removed. + **No change to the hub Ingress, the hub Deployment, or any felhom-system object.** +- **Box:** all spike listeners / tcpdumps / http.servers gone (timeout-bounded); `/tmp/spike-*` + removed; **nft still empty**; `wg-felhom` unchanged. +- **Final re-verify (all green):** box `wg-quick@wg-felhom` + `felhom-agent` **active**, tunnel live + (`pvesm status felhom-offsite` = **active**; box↔ep0 last-handshake epoch matches on both ends), + guest 9201 **running** + `felhom-controller Up (healthy)`; ep0 wg0 **3 peers unchanged** + (second box `10.77.0.3` + dormant operator `10.77.0.250` untouched); hub pod **1/1 Running**, + reconciler still pushing 3 peers. ep0 `ip_forward` **1** (never changed), `forward` chain empty, + root `authorized_keys` back to **2** lines (0 spike lines). +- **No repo/hub/agent/manifest production change** — this commit is docs-only. + +--- + +## Method bar honored + +`wg show dump` never run (S1 key-leak ban); only `latest-handshakes`/redacted conf reads. +Throwaway keys generated on the box that owns them, 0600, private keys never printed. Every negative +carries kernel/nft/packet-level evidence (EKEYREJECTED errno; nginx upstream-timeout log line; +tcpdump counts), not bare timeouts. The live baseline was re-measured rather than assumed — the two +places the plan's assumptions had gone stale (`ip_forward` now 1; the operator `/32` now rendered) +are recorded as findings. No probe touched Peti's arc, the hub DB, the live peer list, or the +box's rendered WG conf.