P0-P7 probe ladder on real ends (demo-felhom PVE host <-> throwaway Hetzner).
Verdict: GO at H1 — plain host-side WG, box-dials-out to a small public endpoint;
Headscale is a separable fleet layer, not a traversal necessity. Line is plain-NAT
(not CGNAT) with zero IPv6 — recorded honestly; CGNAT vantage deferred to Peti VM 110.
Real 2 GiB worst-case PBS backup over the tunnel at the full home uplink, pin intact.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
AES256-GCM 3 904 MB/s, SHA256 2 152 MB/s (CPU is never the limit)
REAL backup, 2 GiB of /dev/urandom (worst case — incompressible, non-dedupable):
spike.pxar: had to backup 2 GiB of 2 GiB in 480.75 s (average 4.26 MiB/s)
Duration: 481.26s — sustained, zero stalls, saturating the uplink for 8 minutes straight
```
4.94 MB/s TLS-in-tunnel ≈ 41 Mbit/s = the full measured uplink: **the tunnel + TLS stack costs
effectively nothing against the line's own ceiling.**
**First-backup window math (at the measured ~40 Mbit/s ≈ 5 MB/s effective uplink):** 10 GB ≈ 35 min,
50 GB ≈ 2.8 h, 100 GB ≈ 5.6 h — a realistic first backup fits an overnight window; incrementals
(PBS dedup) are far smaller. The *download* direction (restore) is ~5× faster on this line.
### P4 — H2: outbound UDP reality — **51820 AND 443/UDP both pass**
UDP 51820 egress is proven by P2 itself. The same tunnel re-pointed at **`:443/UDP`**
(`wg set … listen-port 443` server-side, endpoint updated client-side) handshook immediately and
carried pings at the same RTT:
```
3 packets transmitted, 3 received, 0% packet loss
rtt min/avg/max/mdev = 30.300/32.612/35.044/1.938 ms
endpoint: 167.233.158.164:443
```
No TCP/443 fallback (DERP-over-HTTPS etc.) is needed on this ISP; and 443/UDP working means even a
hypothetical 51820-blocking ISP has a first, zero-cost escalation (same protocol, friendlier port)
before any TCP fallback machinery is justified.
### P5 — H3: what Headscale actually buys — **a fleet layer, not traversal**
Headscale 0.29.2 stood up on the throwaway box (systemd service + `/etc/headscale/config.yaml` +
sqlite at `/var/lib/headscale/db.sqlite`): created users `customer-a`/`customer-b`, minted a
preauth key, inspected the config surface. Observations:
- **Operational surface:** a public control-plane service (TLS **required** — the embedded DERP
server refuses a non-HTTPS `server_url`), a database to back up, user + preauthkey + node
lifecycle, an ACL policy file, plus a **client-side daemon (tailscaled) on every box**. The
default DERP config points at **Tailscale's public derpmap**
(`controlplane.tailscale.com/derpmap/default`) — a third-party dependency unless you run your
own DERP.
- **For hub-and-spoke backup it is redundant:** traversal is already solved by rung H1 (the spoke
dials a public endpoint). Headscale's real value is fleet mechanics — key rotation, node expiry,
per-customer users/ACLs, automatic endpoint re-resolution, and DERP *fallback* for UDP-hostile
lines (H2 — which this ISP does not need).
- **Recorded gap:** no client was joined to the throwaway headscale (joining = installing
tailscaled on the production PVE host; declined). Surface inspection + docs only.
### P6 — per-customer isolation
- **Plain WG (recommended rung):** isolation is inherent in hub-and-spoke: the server holds one
peer entry per box with `AllowedIPs = <that box's /32>` — cross-peer traffic requires the
*server* to forward between WG peers, which the endpoint's firewall simply does not allow
(forwarding stays off; only `INPUT` to the PBS port is open on the WG interface). A box can
reach the PBS API and nothing else; boxes cannot see each other. PBS-side tenancy = the existing
per-customer namespace + token model (unchanged).
- **Headscale (if ever adopted):** the same property must be *written* as ACL policy
(per-customer users + a policy file allowing only spoke→PBS) — more expressive, but a policy to
maintain and test rather than a topology that cannot express the attack.
### P7 — guest-side WG refutation — **hypothesis REFUTED: kernel WG works in the unprivileged guest; host placement stands on architecture, not infeasibility**
Against the real guest 9201 (`features: nesting=1,keyctl=1`, `unprivileged: 1`):
```
# pct exec 9201 -- ls -la /dev/net/
ls: cannot access '/dev/net/': No such file or directory ← no TUN, as expected
# pct exec 9201 -- ip link add wgtest type wireguard
rc=0 ← SUCCEEDS
wgtest DOWN <POINTOPOINT,NOARP>
# pct exec 9201 -- ip link del wgtest ← removed immediately
```
Kernel WireGuard is **not** a TUN consumer — an unprivileged LXC with the stock golden features can
create a WG interface as long as the host kernel has the module. So the "no TUN → guest can't"
argument is dead. The **host-placement recommendation stands on architecture instead** (§4.2):
backup traffic *originates on the host* (the agent drives vzdump/PBS push with host-side
credentials), the tunnel must survive guest death/rebuild by construction (the `cloudflared`
precedent, 01 §7), and the guest must keep holding **zero** infra credentials (01 §3). A guest-side
WG key would violate the trust model, not the kernel. (`/dev/net/tun` absence *does* still block a
guest-side **tailscaled**, relevant only if the Headscale rung were ever chosen.)
---
## 3. The twelve §7 answers
1. **CGNAT or plain-NAT?** Plain single-NAT with a fixed public IPv4 (`37.191.56.193`, One Hungary
catv-fixed); **no IPv6 at all** (no address, no egress, no RAs on the wire). The CGNAT premise
did not hold *on this line*.
2. **H0 (v6-direct)?** N/A — no home IPv6. Keep the production endpoint dual-stack, but v4 is the
planning baseline.
3. **H1 (plain WG through the NAT)?** Yes — outbound-initiated, `PersistentKeepalive=25`, zero
coordinator/relay, held through an 11.4-minute fully-idle window with a stable NAT mapping,
and carried the PBS API with the exact-cert pin validating through the tunnel (positive +
negative pin proof, P2b).
4. **Throughput?** Tunnel costs ~5% on the ~40 Mbit/s uplink bottleneck; PBS-native TLS speed
through the tunnel = the full uplink (4.94 MB/s); a real worst-case 2 GiB backup sustained
4.26 MiB/s for 8 min with zero stalls. First backup of 50–100 GB ≈ 3–6 h (overnight window);
incrementals trivial.
5. **H2 (UDP blocked)?** No — UDP 51820 **and** 443 both pass on this ISP. TCP/443 fallback is a
*contingency for other ISPs*, not a launch requirement; note it as the escalation if a customer
line ever proves UDP-hostile.
6. **H3 (Headscale needed)?** Not for backup. It is a separable fleet-management layer (key
lifecycle, ACLs, endpoint re-resolution, DERP fallback) purchasable later without re-architecting
— the spoke config is the same WG primitive either way. Its cost: a public TLS control plane,
a DB, tailscaled on every box, and (by default) a Tailscale-operated DERP dependency.
7. **Isolation?** Plain WG hub-and-spoke: per-peer `/32` `AllowedIPs` + no forwarding on the
endpoint = no lateral path by topology (P6).
8. **Placement?** Host — same placement as `cloudflared` (agent-managed systemd service). The P7
probe *refuted* the infeasibility argument (kernel WG works in the guest), so the decision rests
on: backups originate host-side, tunnel survives guest death, guest keeps zero infra creds.
9. **Key custody + DR:** mint the box's WG keypair at install into the agent state dir (0600,
`felhom-agent` user), register the pubkey with the hub, and add the private key to the
escrowed `IdentityBundle` (`felhom-agent/internal/escrow/identity.go`) so a re-provisioned box
re-establishes its tunnel from the same recovery flow that already restores PBS identity. The
endpoint's own pubkey+address are non-secret config in the hub record.
10. **Where does the production endpoint live?** A small public relay VM (this spike's CX23-class
box ≈ €4–8/mo) — see §4.4 for the two variants (PBS-on-VM vs VM-as-rendezvous-relay with
DooPlex PBS dialing out too). DooPlex itself cannot be the endpoint (no public IP/UDP).
11. **Trust model preserved?** Yes — the box only ever dials **out** (UDP to the endpoint); no new
inbound listener, no new inbound attack surface on the box; the endpoint exposes exactly one
UDP port publicly. Operator-inbound access (B) and game-server ingress (C — public players are
not tunnel members) are **explicitly separate problems, out of scope here**, and nothing in
this design presumes them.
12. **Go/no-go:** **GO** on rung **H1** — plain WireGuard, host-side, box-dials-out, to a small
public endpoint carrying PBS. Headscale/DERP only if (a) a real customer ISP proves
UDP-hostile (H2) or (b) fleet size makes static peer config painful (H3). No pivot to an
outbound-mTLS PBS proxy needed — the simplest thing worked.
---
## 4. Architecture recommendation (input to the production spec — NOT the spec)
### 4.1 Transport rung
Plain kernel WireGuard, one tunnel per box, box-initiated, `PersistentKeepalive=25`, endpoint =
a public relay VM. No coordinator, no relay, no mesh.
### 4.2 Placement
On the Proxmox **host**, as an **agent-managed systemd service** — the exact `cloudflared`
precedent (01 §7, `internal/hub/cloudflared.go` prober pattern). Interface + config under the
agent's ownership; guest untouched.
### 4.3 Identity & custody
Per-box keypair minted at install; private key 0600 in the agent state dir; pubkey → hub record;
private key → escrowed `IdentityBundle`. Server pubkey/endpoint = plain config from the hub.
Revocation = delete the peer entry at the endpoint (hub-driven, auditable).
### 4.4 The production public endpoint (the §1 corollary)
Two viable shapes, decision deferred to the production spec:
- **(a) PBS on the public VM** — simplest ops; storage cost scales on the VM; client-side
encryption means at-rest exposure is ciphertext-only anyway.
- **(b) VM as dumb rendezvous relay** — DooPlex PBS *also* dials out a WG spoke to the VM; the VM
forwards spoke↔PBS traffic and stores nothing. Keeps bulk storage on DooPlex's existing disks;
the VM needs only bandwidth; **both** sides remain outbound-only (this also solves DooPlex's own
no-public-UDP problem). Ciphertext-only transits the VM.
Either way the endpoint is small, dual-stack, firewalled to WG-UDP-only, and *is* the thing the
subscription's offsite promise runs through — it becomes managed infrastructure, not a lab box.
### 4.5 Isolation
Per-peer `/32` AllowedIPs; endpoint forwarding off; WG-interface firewall admits only the PBS port.
Per-customer PBS namespaces/tokens unchanged on top.
---
## 5. Scope caveats + NOT-validated list
- **The CGNAT claim is NOT proven by this spike** — the operator's line turned out to be
plain-NAT. What *is* proven: outbound-initiated WG through a NAT with keepalive, which is the
same mechanism CGNAT permits (CGNAT breaks *inbound* and *p2p*, not outbound-to-public). The
honest residual risk is an ISP that breaks long-lived UDP mappings aggressively — exactly what
the deferred second-ISP vantage (**Peti's Debian VM 110**, currently unreachable — "No route to
host") should confirm once fixed. **Follow-up recorded.**
- Single ISP (One Hungary catv), single evening, single geography. No long-horizon (days) hold
data; no home-IP-rotation event observed (WG roaming handles it by design — untested here).
- No Headscale client join (tailscaled kept off the production host); Headscale assessed on
server-side surface + docs only.
- No real vzdump→PBS run over the tunnel (synthetic `proxmox-backup-client` traffic only — real
guest snapshots stay on the real PBS).
- IPv6 absence is a One-Hungary-line fact, not a market fact; the endpoint should stay dual-stack.
- The tunnel-download anomaly (UDP encapsulation outrunning direct TCP) is a single-session
observation — noted, not relied on.
- Restore-direction (PBS → box) bulk transfer was measured only as iperf3, not as a
`proxmox-backup-client restore` run.
- Relayed-mode (DERP) throughput was NOT measured — moot, since no relay is recommended.
`documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md` — the offsite-backup transport
decision, empirically grounded on both real ends (demo-felhom PVE host ⟷ throwaway Hetzner box,
end state: powered off, secrets shredded). Headline results: the operator's line is **plain-NAT
with a fixed public IP, not CGNAT, and has zero IPv6** (P0 honesty — CGNAT confirmation deferred
to Peti's VM 110); plain outbound WG held an 11.4-min idle window and carried a **real 2 GiB
worst-case PBS backup at 4.26 MiB/s = the full home uplink** (~5% tunnel overhead), TLS pin
intact through the tunnel (positive + negative proof); UDP 51820 **and** 443 both pass; kernel WG
surprisingly *works* inside the unprivileged guest (P7 — host placement stands on architecture,
not infeasibility). Recommendation: host-side agent-managed WG (cloudflared pattern), key in the
escrowed IdentityBundle, small public endpoint VM (PBS-on-VM vs rendezvous-relay deferred to the
spec). `runbooks/day0-install.md` backlog line resolved to point here; `CONTEXT.md` notes the
DR-completeness task is unblocked (next: the production connectivity spec).
## skills — NEW: felhom-app-catalog (4th skill) + SparkyFitness as its worked example (2026-07-03)
`skills/felhom-app-catalog/SKILL.md` — the catalog **authoring workflow** (research → inspect the
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
