docs: architecture Part 06 — offsite-connectivity design-of-record (slice roadmap S1-S6)
Records the settled transport decisions (plain WG, host-side agent-managed pilot, one hub-driven endpoint VM running WG+PBS, hub source-of-truth over WireDesiredState, one datastore + per-customer namespaces, relay-through-DooPlex rejected). Grounded at file:line vs felhom.eu@bf099f6 + felhom-agent@4ba1b14. Backlog line resolved to cite spike + design doc. Docs-only; slices are separate future tasks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
@@ -4,69 +4,59 @@
|
||||
|
||||
---
|
||||
|
||||
# REPORT — SPIKE: offsite-backup connectivity (WireGuard, home-NAT traversal) (2026-07-03)
|
||||
# REPORT — architecture Part 06: offsite-connectivity design-of-record (2026-07-03)
|
||||
|
||||
Docs-only run; evidence doc: `documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md`.
|
||||
No production code, no Headscale on the real hub, no agent change, no persistent felhom-pve config.
|
||||
Docs-only. Created `documentation/architecture/06-offsite-connectivity.md` (design-of-record for
|
||||
the offsite-backup transport); resolved the `day0-install.md` backlog line to cite spike + design
|
||||
doc; CONTEXT + scripts/CHANGELOG updated. No code, no version bump, no deploy, no endpoint
|
||||
provisioning.
|
||||
|
||||
## Verdict
|
||||
## Baselines (verified fresh-pulled before writing)
|
||||
|
||||
**GO at the lowest rung (H1): plain WireGuard, host-side, box-dials-out to a small public
|
||||
endpoint.** No coordinator, no relay, no mesh. Headscale (P5: 0.29.2 stood up throwaway) is a
|
||||
*separable fleet-management layer* — TLS control plane + DB + tailscaled-per-box + Tailscale-hosted
|
||||
DERP by default — not a traversal necessity for hub-and-spoke backup.
|
||||
- felhom.eu @ `bf099f6` (spike doc commit; this task's commit lands on top)
|
||||
- felhom-agent @ `4ba1b14` (v0.63.0, read-only grounding)
|
||||
|
||||
## Probe ladder results (both ends real: demo-felhom PVE host ⟷ throwaway Hetzner CX23)
|
||||
## Grounding citations used (for reviewer validation against Gitea)
|
||||
|
||||
- **P0** — the operator's line is **plain single-NAT with a fixed public IPv4**
|
||||
(`37.191.56.193`, catv.fixed.one.hu, AS21334), *not* CGNAT (no 100.64/10 hop; source IP at the
|
||||
server = egress IP). **Zero IPv6** (no address, no egress, 15-min RA capture: 0 packets).
|
||||
- **P1 (H0 v6-direct)** — N/A on this line; keep the production endpoint dual-stack anyway.
|
||||
- **P2 (H1)** — plain WG v4 up on the **first try**; RTT ~30–40 ms; **11.4-min fully-idle hold**
|
||||
(keepalive-only, NAT mapping unmoved, instant ping after); PBS 4.2.2 API through the tunnel with
|
||||
the **exact-cert pin intact** (`pin.go` model) + negative wrong-pin proof; `:8007` tunnel-only
|
||||
by firewall (public probe: unreachable).
|
||||
- **P3** — uplink is the bottleneck: direct 42 Mbit/s up / 200–230 down; tunnel upload ~39 Mbit/s
|
||||
(~5% overhead); PBS TLS benchmark through tunnel **4.94 MB/s = the full uplink**; **real 2 GiB
|
||||
worst-case (urandom) backup: 4.26 MiB/s sustained, 8 min, zero stalls**. First backup 50–100 GB
|
||||
≈ 3–6 h (overnight window). Oddity recorded: tunnel *download* (565 Mbit/s, 0 retrans) beat
|
||||
direct TCP (230, 285 retrans) — UDP encapsulation dodges whatever shapes inbound TCP.
|
||||
- **P4 (H2)** — outbound UDP fine on **51820 and 443**; no TCP/443 fallback machinery needed.
|
||||
- **P6** — plain-WG isolation is topological: per-peer `/32` AllowedIPs + forwarding off at the
|
||||
endpoint; only the PBS port admitted from the WG interface.
|
||||
- **P7** — hypothesis **refuted with evidence**: kernel WG link creation **succeeds** inside the
|
||||
unprivileged guest (no `/dev/net/tun` needed — that's confirmed absent and only blocks
|
||||
tailscaled). Host placement therefore stands on architecture: backups originate host-side, the
|
||||
tunnel must survive guest death (cloudflared precedent), the guest keeps zero infra creds.
|
||||
| Claim in the doc | Cited at |
|
||||
|---|---|
|
||||
| Backup = outbound push; PBS at LAN address | `01-topology-and-trust.md` §7; `felhom-agent/internal/pbs/client.go:25-26` (`Server` documented as `"192.168.0.180"`, port default 8007) |
|
||||
| Public edge HTTP-only, no public UDP | `felhom.eu/manifests/hub.yaml:233` (`ingressClassName: nginx-internal`) |
|
||||
| TLS pin above WG, unchanged through tunnel | `felhom-agent/internal/pbs/pin.go:17-35` (exact-cert `VerifyPeerCertificate`); spike P2b (positive + negative through-tunnel proof) |
|
||||
| Desired-state channel + generation gate | `felhom-agent/internal/hub/report.go:270` (`DesiredGeneration`), `report.go:289-296` (`WireDesiredState` + forward-compat siblings), `internal/desired/syncer.go:44-61` (fetch only on generation advance) |
|
||||
| IdentityBundle to extend | `felhom-agent/internal/escrow/identity.go:24-27` (`{TunnelToken, PBSToken}`) |
|
||||
| Agent already installs+enables host units | `felhom-agent/configs/felhom-agent.sudoers:16-18` (`install … /etc/systemd/system/*.mount`, `systemctl enable --now`) |
|
||||
| Hub drives external infra via held token | `felhom.eu/hub/internal/cloudflare/unblock.go` (CF token, WAF mutation) |
|
||||
| cloudflared = later adopter of the S3 pattern | `felhom-agent/internal/hub/cloudflared.go:9-12` (read-only prober, self-described "seam for the tunnel-management slice") |
|
||||
| All measurements | `documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md` (P0–P7) |
|
||||
|
||||
## Architecture recommendation (input to the next task — the production spec)
|
||||
## What the doc marks DECIDED (recorded, not re-litigated)
|
||||
|
||||
Host-side agent-managed WG service (cloudflared pattern); per-box keypair minted at install →
|
||||
pubkey to hub, private key 0600 in agent state dir + **joins the escrowed IdentityBundle** for DR;
|
||||
endpoint = small public dual-stack VM (€4–8/mo class), two shapes deferred to the spec: PBS-on-VM
|
||||
vs **VM-as-rendezvous-relay with DooPlex PBS dialing out too** (both ends outbound-only,
|
||||
ciphertext-only transit — also solves DooPlex's no-public-UDP). Trust model preserved: box only
|
||||
dials out; operator-inbound (B) and game-ingress (C) explicitly out of scope.
|
||||
D1 plain WG (no Headscale for traversal — separable future fleet layer); D2 host-side
|
||||
**agent-managed** `wg-felhom` (pilot for agent-managed host services; install-script placement
|
||||
rejected as one-shot; in-guest rejected on trust grounds — spike P7 shows it *works*, so the
|
||||
refutation is architectural); D3 one shared dumb hub-driven endpoint VM (WG + PBS, no agent);
|
||||
D4 hub = source of truth, `wireguard` block on the existing desired-state channel; D5 one
|
||||
datastore + per-customer namespaces; D6 PBS ON the VM — relay-through-DooPlex rejected
|
||||
(non-scaling through the ~40 Mbit home uplink, couples all DR to home uptime; DooPlex PBS =
|
||||
dev/demo only).
|
||||
|
||||
## Honest caveats
|
||||
## What the doc marks DEFERRED / OPEN (honest ledger, §7)
|
||||
|
||||
CGNAT traversal is argued from mechanism, **not proven on this line** (it's plain-NAT); second-ISP
|
||||
confirmation deferred to Peti's VM 110 (currently unreachable). Single ISP/evening; no multi-day
|
||||
hold; no home-IP-rotation event observed; Headscale assessed server-side only (no tailscaled on
|
||||
the production host); no restore-direction PBS run.
|
||||
- **Open validation (one):** true-CGNAT traversal — unmeasured (operator line = plain-NAT, spike
|
||||
P0); closure = mobile-hotspot smoke test, thorough = Peti VM 110; does not block S1/S2.
|
||||
- Endpoint peer-sync mechanism (push-API vs signed-manifest pull) — slice-1 design point.
|
||||
- WG key-rotation mechanics; endpoint scaling/sharding/HU-hosting; endpoint-VM self-DR depth.
|
||||
|
||||
## Cleanup (verified)
|
||||
## Slice roadmap set (each a future task)
|
||||
|
||||
demo-felhom: interface/keys/config/dataset gone (keys **shredded**), iperf3 purged, WG module
|
||||
unloaded, `/etc/wireguard` untouched-empty, guest probe link deleted — residue-free. Hetzner
|
||||
throwaway: WG + headscale + PBS torn down, secrets shredded, datastore deleted, box **powered
|
||||
off** (parked for the Peti follow-up or destruction; jarrs.eu box never touched).
|
||||
S1 endpoint provisioning + peer-sync surface → S2 hub schema + desired-state → S3 agent WG
|
||||
service (**risky/supervised**: sudoers + host service + escrow change) → S4 PBS-over-tunnel →
|
||||
S5 DR re-establishment (merges with DR-completeness) → S6 monitoring/hardening.
|
||||
**MVP = S1→S2→S3, then S4.**
|
||||
|
||||
## Repo changes (docs-only)
|
||||
## Author notes
|
||||
|
||||
- NEW `documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md` (P0–P7 verbatim + twelve
|
||||
§7 answers + recommendation).
|
||||
- `documentation/runbooks/day0-install.md` — the "LAN-only until Headscale/WireGuard lands"
|
||||
backlog line now points at the spike (decision made; spec is the remaining gate).
|
||||
- `CONTEXT.md` — transport decision recorded; next task = production connectivity spec.
|
||||
- `scripts/CHANGELOG.md` — docs entry.
|
||||
None — no §2 decision looked wrong while writing; the one place the task's own earlier framing
|
||||
was contradicted by evidence (guest-WG infeasibility) was already resolved by the spike and is
|
||||
recorded as an architectural (not technical) refutation in D2.
|
||||
|
||||
Reference in New Issue
Block a user