felhom.eu/documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md

SPIKE — offsite-backup connectivity: WireGuard vs the home-NAT reality (2026-07-03)

STATUS: COMPLETE — verdict below. This spike validates the offsite-backup transport before any production connectivity spec is written. No production code shipped; no Headscale on the real hub; no agent binary change; no persistent config on felhom-pve. All probes ran on a throwaway Hetzner box (felhom-hetzner, destroyed-or-parked after — see §6) and as additive, fully-removed interfaces/services on the demo-felhom host. The real DooPlex PBS and its customer snapshots were never a probe target.

Class: SPIKE (empirical validation; no product code). Repos: felhom.eu (this doc only); felhom-agent read-only for grounding (internal/pbs/{client,pin}.go, configs/build-golden.sh, internal/hub/cloudflared.go, internal/escrow/identity.go). Probe ends: felhom-hetzner = Hetzner CX23 (note added 2026-08-03: rescaled to a CX33, 8 GB RAM — this spike records the machine as probed and its body is deliberately unchanged), Debian 13.4, public IPv4 167.233.158.164, global IPv6 2a01:4f8:...::/64 (throwaway — NOT the live jarrs.eu box) ⟷ demo-felhom = the real PVE 9.2.2 host on the operator's home line (One Hungary fixed cable, Budapest), driven over the existing LAN SSH path; the tunnel itself always dialed out.

Verdict (one line): GO at the lowest rung — H1: plain WireGuard, box-dials-out to a small public endpoint, holds through the home NAT with zero coordinator/relay, carries the PBS API with the TLS pin intact at ~wire-speed of the home uplink; Headscale is a separable fleet-management option, not a traversal necessity — with the honest caveat that the operator's line turned out to be plain-NAT (public IP), not CGNAT, so the CGNAT-traversal claim rests on mechanism + a deferred second-ISP confirmation (Peti's VM 110), not on this line's evidence.

---

1. Why this spike exists (the gap, grounded)

Offsite backup — the paid DR substrate (architecture/01-topology-and-trust.md §8) — is structurally unreachable for a real customer today:

• Backup is outbound push to PBS (§7/§8), and the only PBS lives at a LAN address, 192.168.0.180:8007 (felhom-agent/internal/pbs/client.go).
• The public edge is Cloudflare-Tunnel → nginx-internal — HTTP(S) only, no public inbound UDP anywhere; DooPlex has no public IP.
• So offsite backup works only because the demo box shares a LAN with PBS. A real customer's box cannot reach 192.168.0.180 at all. The DR crypto/identity side is done and drilled (SPIKE-dr-recipe-2026-06-16.md); transport is the only missing piece.

Architectural corollary (recorded as a finding): production offsite backup requires a publicly-reachable endpoint that today's infrastructure does not provide. The throwaway Hetzner box in this spike is a preview of that production endpoint, not just a test rig (§4.4).

The hypothesis ladder tested, cheapest-first (the roadmap's "Headscale + DERP" sketch assumes the hard problem — peer-to-peer traversal between two NAT'd hosts — but backup is hub-and-spoke to a public endpoint, which any NAT'd box can reach outbound):

Rung	Hypothesis	Result
H0	both ends global IPv6 → direct, no NAT	N/A — the home side has no IPv6 at all (P0/P1)
H1	plain WG over IPv4, box→public endpoint, keepalive	CONFIRMED on this line (P2/P3)
H2	outbound UDP blocked → TCP/443 fallback needed	not needed — UDP 51820 and 443 both pass (P4)
H3	fleet needs → Headscale	separable — fleet layer, not traversal (P5)

---

2. Probe results (verbatim, redacted)

P0 — NAT classification of the demo-felhom line — plain single-NAT with a public IPv4, no IPv6

• Egress IP seen by ifconfig.me from the host: 37.191.56.193.

• Source IP observed at the Hetzner box for an inbound TCP connection from demo-felhom (HTTP listener log): 37.191.56.193 — same address, i.e. one NAT, stable mapping.

• ipinfo.io: 37.191.56.193 = catv-37-191-56-193.catv.fixed.one.hu, AS21334 One Hungary Ltd. — a fixed (subscriber-routable) cable address, not shared-pool space.

• Traceroute from the host (first hops):

   1  192.168.0.1      0.239 ms      ← home router
   2  89.133.164.70   12.485 ms      ← already PUBLIC ISP space (catv...one.hu, same AS)
   3  89.135.220.78   22.316 ms
  

  No 100.64.0.0/10 (or private) hop after the home router — no CGNAT tier on the path.

• IPv6: ip -6 addr shows link-local only; curl -6 fails (NO-IPV6-EGRESS); a 15-minute passive tcpdump capture for router advertisements on vmbr0 caught zero RAs. (Caveat noted: accept_ra=1 + forwarding=1 means the kernel would ignore RAs anyway, which is why the wire-level capture, not the address absence, is the evidence.) No IPv6 exists on this LAN.

Consequences: the operator's line is a softer premise than the task assumed — H1 success here does not prove CGNAT traversal (see §5). H0 (IPv6-direct) is dead on arrival for this line and cannot be the plan's foundation: production must assume v4-only homes.

P1 — H0 IPv6-direct — N/A (recorded gap)

Not testable: the home side has no global IPv6 (P0). The Hetzner side has native v6, so the server-side of a dual-stack endpoint is free — worth keeping the endpoint dual-stack so v6-capable customer lines use it, but it solves nothing this spike needed solving.

P2 — H1: plain WG over IPv4, outbound-initiated — CONFIRMED, first try, no coordinator/relay

Setup: WG keypair per side (private keys generated on-box, 0600, never transited or logged); Hetzner wgspike = 10.99.0.1/24, ListenPort 51820; demo-felhom wgspike = 10.99.0.2/24, Endpoint 167.233.158.164:51820, PersistentKeepalive = 25, config under /tmp (nothing in /etc/wireguard). PVE 9.2 needed nothing beyond modprobe wireguard (module ships in the kernel)

• the wireguard-tools package.

demo-felhom# wg-quick up /tmp/wgspike.conf && ping -c 3 10.99.0.1
64 bytes from 10.99.0.1: icmp_seq=1 ttl=64 time=51.2 ms
3 packets transmitted, 3 received, 0% packet loss
rtt min/avg/max/mdev = 33.411/40.659/51.204/7.628 ms

Server-side view: peer endpoint 37.191.56.193:32873 (the NAT-translated source port), i.e. a classic outbound UDP mapping — exactly the mechanism that also works through CGNAT.

Keepalive hold (idle test): the link carried zero data traffic for 11.4 minutes (keepalives only — total transfer counters after the window: 1 888 B rx / 3 744 B tx). Result: handshake stayed fresh throughout (age 120 s at check — keepalive-driven rekeys continuing on schedule), the NAT mapping never moved (37.191.56.193:32873 unchanged), and the first ping after the idle window went straight through:

3 packets transmitted, 3 received, 0% packet loss
rtt min/avg/max/mdev = 30.119/30.812/31.279/0.499 ms

The tunnel holds through idle with no help — the single mechanism CGNAT UDP timeouts would threaten, working as designed.

P2b — PBS :8007 through the tunnel, TLS pin intact — CONFIRMED (positive + negative)

A real PBS 4.2.2 was installed on the throwaway box (datastore scratch at /srv/pbs-scratch, API token root@pam!spike, secret 36 chars — never logged; box firewalled to SSH + WG/iperf ports only, so :8007 is not publicly reachable — tunnel-only by construction). From demo-felhom, with PBS_FINGERPRINT = the scratch PBS leaf-cert SHA-256 (the exact pin model of felhom-agent/internal/pbs/pin.go):

=== public :8007 reachability (must FAIL, firewall proof):
000 / UNREACHABLE (as intended)
=== tunnel :8007 API version (pin intact):
client version: 4.2.0
server version: 4.2.2
=== NEGATIVE pin test (wrong fingerprint, must FAIL):
WARNING: certificate fingerprint does not match expected fingerprint!

The exact-cert pin validates through the tunnel (WG is below TLS; the pinned leaf is unchanged), and a wrong pin is detected — the red-proof for the pin surviving the transport change. (Deploy gotcha found + fixed on the scratch box: a datastore under /root fails with Permission denied — the PBS backup user can't traverse /root; use a world-traversable parent like /srv.)

P3 — throughput — home uplink is the bottleneck; tunnel overhead ≈ 5% on the backup direction

Direct (no tunnel) iperf3 demo-felhom ⟷ Hetzner (two runs each):

UPLOAD  (home→Hetzner):  42.1 / 42.0 Mbit/s sender    (receiver 40.7 both runs)
DOWNLOAD (Hetzner→home):  203 / 230 Mbit/s            (retransmits: 3 / 285)

Through the WG tunnel:

UPLOAD  (tunnel):   39.1 Mbit/s sender / 38.4 receiver   ← ~5% under direct
DOWNLOAD (tunnel):  565 Mbit/s, 0 retransmits            ← FASTER than direct TCP

The download anomaly is real and repeatable-in-session: plain TCP from Hetzner→home tops out at ~200–230 Mbit/s with hundreds of retransmits, while the same TCP stream inside the WG/UDP encapsulation ran at 565 Mbit/s with zero retransmits — whatever middlebox/shaping hurts inbound TCP on this cable line does not see the UDP tunnel. Download is the restore direction; a pleasant surprise, not a design input. (Single-session observation; not load-tested further.)

PBS-native measurements over the tunnel (the actual workload):

proxmox-backup-client benchmark:  TLS (maximal backup upload speed): 4.94 MB/s
                                  AES256-GCM 3 904 MB/s, SHA256 2 152 MB/s (CPU is never the limit)

REAL backup, 2 GiB of /dev/urandom (worst case — incompressible, non-dedupable):
  spike.pxar: had to backup 2 GiB of 2 GiB in 480.75 s (average 4.26 MiB/s)
  Duration: 481.26s — sustained, zero stalls, saturating the uplink for 8 minutes straight

4.94 MB/s TLS-in-tunnel ≈ 41 Mbit/s = the full measured uplink: the tunnel + TLS stack costs effectively nothing against the line's own ceiling.

First-backup window math (at the measured ~40 Mbit/s ≈ 5 MB/s effective uplink): 10 GB ≈ 35 min, 50 GB ≈ 2.8 h, 100 GB ≈ 5.6 h — a realistic first backup fits an overnight window; incrementals (PBS dedup) are far smaller. The download direction (restore) is ~5× faster on this line.

P4 — H2: outbound UDP reality — 51820 AND 443/UDP both pass

UDP 51820 egress is proven by P2 itself. The same tunnel re-pointed at :443/UDP (wg set … listen-port 443 server-side, endpoint updated client-side) handshook immediately and carried pings at the same RTT:

3 packets transmitted, 3 received, 0% packet loss
rtt min/avg/max/mdev = 30.300/32.612/35.044/1.938 ms
endpoint: 167.233.158.164:443

No TCP/443 fallback (DERP-over-HTTPS etc.) is needed on this ISP; and 443/UDP working means even a hypothetical 51820-blocking ISP has a first, zero-cost escalation (same protocol, friendlier port) before any TCP fallback machinery is justified.

P5 — H3: what Headscale actually buys — a fleet layer, not traversal

Headscale 0.29.2 stood up on the throwaway box (systemd service + /etc/headscale/config.yaml + sqlite at /var/lib/headscale/db.sqlite): created users customer-a/customer-b, minted a preauth key, inspected the config surface. Observations:

• Operational surface: a public control-plane service (TLS required — the embedded DERP server refuses a non-HTTPS server_url), a database to back up, user + preauthkey + node lifecycle, an ACL policy file, plus a client-side daemon (tailscaled) on every box. The default DERP config points at Tailscale's public derpmap (controlplane.tailscale.com/derpmap/default) — a third-party dependency unless you run your own DERP.
• For hub-and-spoke backup it is redundant: traversal is already solved by rung H1 (the spoke dials a public endpoint). Headscale's real value is fleet mechanics — key rotation, node expiry, per-customer users/ACLs, automatic endpoint re-resolution, and DERP fallback for UDP-hostile lines (H2 — which this ISP does not need).
• Recorded gap: no client was joined to the throwaway headscale (joining = installing tailscaled on the production PVE host; declined). Surface inspection + docs only.

P6 — per-customer isolation

• Plain WG (recommended rung): isolation is inherent in hub-and-spoke: the server holds one peer entry per box with AllowedIPs = <that box's /32> — cross-peer traffic requires the server to forward between WG peers, which the endpoint's firewall simply does not allow (forwarding stays off; only INPUT to the PBS port is open on the WG interface). A box can reach the PBS API and nothing else; boxes cannot see each other. PBS-side tenancy = the existing per-customer namespace + token model (unchanged).
• Headscale (if ever adopted): the same property must be written as ACL policy (per-customer users + a policy file allowing only spoke→PBS) — more expressive, but a policy to maintain and test rather than a topology that cannot express the attack.

P7 — guest-side WG refutation — hypothesis REFUTED: kernel WG works in the unprivileged guest; host placement stands on architecture, not infeasibility

Against the real guest 9201 (features: nesting=1,keyctl=1, unprivileged: 1):

# pct exec 9201 -- ls -la /dev/net/
ls: cannot access '/dev/net/': No such file or directory     ← no TUN, as expected
# pct exec 9201 -- ip link add wgtest type wireguard
rc=0                                                          ← SUCCEEDS
wgtest           DOWN           <POINTOPOINT,NOARP>
# pct exec 9201 -- ip link del wgtest                         ← removed immediately

Kernel WireGuard is not a TUN consumer — an unprivileged LXC with the stock golden features can create a WG interface as long as the host kernel has the module. So the "no TUN → guest can't" argument is dead. The host-placement recommendation stands on architecture instead (§4.2): backup traffic originates on the host (the agent drives vzdump/PBS push with host-side credentials), the tunnel must survive guest death/rebuild by construction (the cloudflared precedent, 01 §7), and the guest must keep holding zero infra credentials (01 §3). A guest-side WG key would violate the trust model, not the kernel. (/dev/net/tun absence does still block a guest-side tailscaled, relevant only if the Headscale rung were ever chosen.)

---

3. The twelve §7 answers

1. CGNAT or plain-NAT? Plain single-NAT with a fixed public IPv4 (37.191.56.193, One Hungary catv-fixed); no IPv6 at all (no address, no egress, no RAs on the wire). The CGNAT premise did not hold on this line.
2. H0 (v6-direct)? N/A — no home IPv6. Keep the production endpoint dual-stack, but v4 is the planning baseline.
3. H1 (plain WG through the NAT)? Yes — outbound-initiated, PersistentKeepalive=25, zero coordinator/relay, held through an 11.4-minute fully-idle window with a stable NAT mapping, and carried the PBS API with the exact-cert pin validating through the tunnel (positive + negative pin proof, P2b).
4. Throughput? Tunnel costs ~5% on the ~40 Mbit/s uplink bottleneck; PBS-native TLS speed through the tunnel = the full uplink (4.94 MB/s); a real worst-case 2 GiB backup sustained 4.26 MiB/s for 8 min with zero stalls. First backup of 50–100 GB ≈ 3–6 h (overnight window); incrementals trivial.
5. H2 (UDP blocked)? No — UDP 51820 and 443 both pass on this ISP. TCP/443 fallback is a contingency for other ISPs, not a launch requirement; note it as the escalation if a customer line ever proves UDP-hostile.
6. H3 (Headscale needed)? Not for backup. It is a separable fleet-management layer (key lifecycle, ACLs, endpoint re-resolution, DERP fallback) purchasable later without re-architecting — the spoke config is the same WG primitive either way. Its cost: a public TLS control plane, a DB, tailscaled on every box, and (by default) a Tailscale-operated DERP dependency.
7. Isolation? Plain WG hub-and-spoke: per-peer /32 AllowedIPs + no forwarding on the endpoint = no lateral path by topology (P6).
8. Placement? Host — same placement as cloudflared (agent-managed systemd service). The P7 probe refuted the infeasibility argument (kernel WG works in the guest), so the decision rests on: backups originate host-side, tunnel survives guest death, guest keeps zero infra creds.
9. Key custody + DR: mint the box's WG keypair at install into the agent state dir (0600, felhom-agent user), register the pubkey with the hub, and add the private key to the escrowed IdentityBundle (felhom-agent/internal/escrow/identity.go) so a re-provisioned box re-establishes its tunnel from the same recovery flow that already restores PBS identity. The endpoint's own pubkey+address are non-secret config in the hub record.
10. Where does the production endpoint live? A small public relay VM (this spike's CX23-class box ≈ €4–8/mo) — see §4.4 for the two variants (PBS-on-VM vs VM-as-rendezvous-relay with DooPlex PBS dialing out too). DooPlex itself cannot be the endpoint (no public IP/UDP).
11. Trust model preserved? Yes — the box only ever dials out (UDP to the endpoint); no new inbound listener, no new inbound attack surface on the box; the endpoint exposes exactly one UDP port publicly. Operator-inbound access (B) and game-server ingress (C — public players are not tunnel members) are explicitly separate problems, out of scope here, and nothing in this design presumes them.
12. Go/no-go: GO on rung H1 — plain WireGuard, host-side, box-dials-out, to a small public endpoint carrying PBS. Headscale/DERP only if (a) a real customer ISP proves UDP-hostile (H2) or (b) fleet size makes static peer config painful (H3). No pivot to an outbound-mTLS PBS proxy needed — the simplest thing worked.

---

4. Architecture recommendation (input to the production spec — NOT the spec)

4.1 Transport rung

Plain kernel WireGuard, one tunnel per box, box-initiated, PersistentKeepalive=25, endpoint = a public relay VM. No coordinator, no relay, no mesh.

4.2 Placement

On the Proxmox host, as an agent-managed systemd service — the exact cloudflared precedent (01 §7, internal/hub/cloudflared.go prober pattern). Interface + config under the agent's ownership; guest untouched.

4.3 Identity & custody

Per-box keypair minted at install; private key 0600 in the agent state dir; pubkey → hub record; private key → escrowed IdentityBundle. Server pubkey/endpoint = plain config from the hub. Revocation = delete the peer entry at the endpoint (hub-driven, auditable).

4.4 The production public endpoint (the §1 corollary)

Two viable shapes, decision deferred to the production spec:

• (a) PBS on the public VM — simplest ops; storage cost scales on the VM; client-side encryption means at-rest exposure is ciphertext-only anyway.
• (b) VM as dumb rendezvous relay — DooPlex PBS also dials out a WG spoke to the VM; the VM forwards spoke↔PBS traffic and stores nothing. Keeps bulk storage on DooPlex's existing disks; the VM needs only bandwidth; both sides remain outbound-only (this also solves DooPlex's own no-public-UDP problem). Ciphertext-only transits the VM. Either way the endpoint is small, dual-stack, firewalled to WG-UDP-only, and is the thing the subscription's offsite promise runs through — it becomes managed infrastructure, not a lab box.

4.5 Isolation

Per-peer /32 AllowedIPs; endpoint forwarding off; WG-interface firewall admits only the PBS port. Per-customer PBS namespaces/tokens unchanged on top.

---

5. Scope caveats + NOT-validated list

• The CGNAT claim is NOT proven by this spike — the operator's line turned out to be plain-NAT. What is proven: outbound-initiated WG through a NAT with keepalive, which is the same mechanism CGNAT permits (CGNAT breaks inbound and p2p, not outbound-to-public). The honest residual risk is an ISP that breaks long-lived UDP mappings aggressively — exactly what the deferred second-ISP vantage (Peti's Debian VM 110, currently unreachable — "No route to host") should confirm once fixed. Follow-up recorded.
• Single ISP (One Hungary catv), single evening, single geography. No long-horizon (days) hold data; no home-IP-rotation event observed (WG roaming handles it by design — untested here).
• No Headscale client join (tailscaled kept off the production host); Headscale assessed on server-side surface + docs only.
• No real vzdump→PBS run over the tunnel (synthetic proxmox-backup-client traffic only — real guest snapshots stay on the real PBS).
• IPv6 absence is a One-Hungary-line fact, not a market fact; the endpoint should stay dual-stack.
• The tunnel-download anomaly (UDP encapsulation outrunning direct TCP) is a single-session observation — noted, not relied on.
• Restore-direction (PBS → box) bulk transfer was measured only as iperf3, not as a proxmox-backup-client restore run.
• Relayed-mode (DERP) throughput was NOT measured — moot, since no relay is recommended.

6. Cleanup assertion

demo-felhom host (all verified after removal): wgspike interface deleted (wg-quick down, "Device does not exist" confirmed); WG private key + PBS token secret shredded; /tmp spike files (config, RA capture log, 2 GiB dataset) removed; the iperf3 package I installed purged; wireguard kernel module unloaded (lsmod count 0); /etc/wireguard/ empty and untouched (mtime Apr 2025); guest 9201's probe link wgtest deleted seconds after creation ("Device does not exist" confirmed); no persistent config, unit, or package change remains from this spike (wireguard-tools was already installed before the spike and was left as found).

Throwaway Hetzner box (felhom-hetzner, 167.233.158.164): WG interface down + config/private key/PBS token shredded; headscale + PBS services stopped and disabled; headscale DB (/var/lib/headscale) and the scratch datastore (/srv/pbs-scratch, random-data chunks only) deleted; then the box was powered off (shutdown -h). Disposition: kept powered-off for the deferred second-ISP follow-up (Peti's VM 110) — the operator can start it from the Hetzner console for that probe or destroy it outright; nothing secret survives on it either way. The live jarrs.eu Hetzner server was never touched.

No repo/hub/agent/manifest production change — this commit is docs-only.

---

Backlog resolution

runbooks/day0-install.md "PBS host is LAN-only until the Headscale/WireGuard work lands" → now points at this spike: the transport decision is made (plain WG to a public endpoint); what remains is the production connectivity spec + implementation (endpoint VM, agent-managed WG service, escrow join, hub peer registry).