# Architecture Part 6 — Offsite Connectivity (the backup transport) > Status: **design-of-record** (2026-07-03). Records the settled offsite-backup-transport > decisions; grounded against felhom.eu @ `bf099f6` and felhom-agent @ `4ba1b14` (v0.63.0). > Evidence base: `documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md` (all > measurements cited below as "spike Pn"). This document ends in an implementation slice roadmap > (§8); **no slice is implemented yet** — each is a separate future task. Consistent with and > extending Part 01 (`01-topology-and-trust.md`); where this doc names a component's duty it uses > Part 01 §3 vocabulary. --- ## 1. Why (the gap) Offsite backup — the paid DR substrate (01 §8) — is structurally unreachable for a real customer today. Backup is an **outbound push to PBS** (01 §7: "Outbound only for control/report/backup"), but the only PBS lives at a **LAN address** — `internal/pbs/client.go:25-26` documents the `Server` field as literally `"192.168.0.180"`, port 8007. The operator's public edge is Cloudflare-Tunnel → `nginx-internal` (`felhom.eu/manifests/hub.yaml:233`, `ingressClassName: nginx-internal`) — HTTP(S) only; DooPlex has **no public IP and no public UDP**. So offsite backup currently works only because the demo box shares a LAN with the PBS host; a customer box cannot reach it at all. The DR crypto/identity side is done and drilled (`SPIKE-dr-recipe-2026-06-16.md`); **this transport is the last missing piece**, and the spike proved the cheapest transport suffices. --- ## 2. Decisions (settled — recorded, not re-litigated) | # | Decision | Rationale (one line) | Rejected alternative | |---|----------|----------------------|----------------------| | D1 | **Transport = plain WireGuard**, box-dials-outbound, `PersistentKeepalive=25` | Backup is hub-and-spoke to a *public* endpoint; outbound UDP + keepalive traverses any NAT tier — spike P2 (first-try up, 11.4-min idle hold), P4 (UDP 51820 *and* 443 pass) | Headscale/DERP **for traversal** — solves the p2p problem we don't have; a TLS control plane + DB + tailscaled-per-box + third-party DERP dependency for nothing (spike P5). Stays available as a *separable future fleet layer*. | | D2 | **Placement = customer host, AGENT-MANAGED** (`wg-felhom` systemd service reconciled from hub desired-state) | WG config is *ongoing reconciled desired-state* — the agent's job; the host-install script is one-shot and cannot react to hub changes (endpoint move, key rotation). Pattern exists: the agent already installs+enables host units via `configs/felhom-agent.sudoers:16-18` (`install … /etc/systemd/system/*.mount` + `systemctl enable --now`) | (a) install-script-owned static config — can't reconcile; (b) in-guest WG — *works* technically (spike P7: kernel WG needs no TUN) but violates 01 §3 (guest holds zero infra creds) and dies with the guest; backups originate host-side anyway. | | D3 | **Endpoint = ONE shared public dual-stack cloud VM** (Hetzner/EU class; HU optional later) running **WG server + the offsite PBS**. It is **dumb and hub-driven** — no agent on it; the hub is the only thing that mutates its WG peer list | Mirrors the proven "hub holds the credential and drives external infra" pattern (`hub/internal/cloudflare/unblock.go` — hub-held CF token mutating WAF rules). Zero-knowledge client-side encryption (01 §8) makes the VM's location privacy-neutral | An agent on the endpoint — a whole trust tier for a box that only needs a peer list and a datastore. | | D4 | **Hub = source of truth** for endpoint coordinates (DNS name, port, server pubkey) + the per-customer peer registry (customer pubkey → assigned tunnel `/32`), delivered over the **existing desired-state channel** | The channel is built for exactly this: heartbeat carries `DesiredGeneration` (`internal/hub/report.go:270`); the agent fetches the heavy state only on generation advance (`internal/desired/syncer.go:44-61`); `WireDesiredState` (`report.go:289-296`) already carries forward-compat sibling blocks (`RestoreDirective`, `StorageManifest`) — a `wireguard` block joins them | A parallel bespoke config channel. | | D5 | **PBS layout = one datastore, per-customer NAMESPACES**, each namespace under its own client-side encryption key (already escrowed — 01 §8 key custody) | PBS-native multi-tenancy; crypto isolation without datastore sprawl; the hub already holds "PBS namespace" in the customer record (01 §6) and `WireDesiredState.PBSNamespace` already exists on the wire (`report.go:294`) | Per-customer datastores — operational sprawl with no isolation gain over namespace+key. | | D6 | **Offsite PBS lives ON the cloud VM — NOT relayed through DooPlex** | Relaying funnels *every* customer's backup through the operator's one shared ~40 Mbit home uplink and couples all customer DR to home-connection uptime — non-scaling and a single point of failure. DC uplink is never the bottleneck (§5) | The spike's §4.4(b) rendezvous-relay shape — rejected by the operator for production. **DooPlex PBS → dev/demo only.** | --- ## 3. Architecture ### 3.1 Components & trust (extends 01 §3 / §5) | | Customer host | Endpoint VM | Hub | |---|---|---|---| | Runs | `wg-felhom` systemd service (agent-reconciled) + the existing agent | WG server + PBS (one datastore, per-customer namespaces) + a minimal hub-driven peer-sync surface | peer registry + endpoint coords in the customer/host record | | Holds | its own WG **private key** (0600, agent state dir) + PBS token + the pinned PBS fingerprint | WG server private key; the PBS ciphertext (zero-knowledge — 01 §10) | every box's WG **public** key + assigned `/32`; the endpoint-mutation credential; escrowed key material (unchanged, 01 §8) | | Mutates | only its own WG interface/service | nothing on its own — peer list changes arrive from the hub | the endpoint's peer list; the desired-state the agent consumes | | Never | accepts inbound (dials out only) | initiates into a box or the hub; holds plaintext | connects into a box (01 §4 — unchanged) | New trust-boundary rows (extends 01 §5): | Boundary | What crosses | Mechanism | Blast radius if breached | |---|---|---|---| | box ↔ endpoint VM | WG-encapsulated, TLS-pinned, client-side-encrypted PBS traffic | outbound UDP; per-peer `/32`; no inter-peer routing (§4.5) | that box's tunnel; ciphertext only | | hub ↔ endpoint VM | peer-list mutations (add/remove pubkey+`/32`) | hub-held credential, minimal surface (mechanism = slice-1 design point, §7) | the peer list — an attacker can *disconnect* backups, not read them | ### 3.2 Hub data model (conceptual fields — not SQL, not wire-final) - **Endpoint record** (one, for now): `dns_name`, `wg_port`, `server_pubkey`, `tunnel_subnet`, `pbs_tunnel_ip` (the endpoint's in-tunnel address the PBS client dials), plus the endpoint-mutation credential (out-of-band secret, never in desired-state). - **Per-host peer entry** (joins the customer/host record, 01 §6): `wg_pubkey` (box-generated, box-registered), `assigned_ip` (the box's tunnel `/32`, hub-assigned), `status`. - **On the wire:** a `wireguard` block on `WireDesiredState` (`report.go:289` — sibling to `Guests`/`RestoreDirective`, same forward-compat carriage rule as `report.go:284-288`: carried + cached by old agents, acted on by new ones): endpoint coords + this box's assignment. Delivery needs **zero new plumbing** — bump `DesiredGeneration`, the agent's existing syncer fetches (`syncer.go:44-61`). ### 3.3 Provisioning handshake (Day-0 join) Plugs into the existing Day-0 flow (`runbooks/day0-install.md`) after agent enrollment (01 §6 — "the agent enrolls first"); offsite stays retrofittable via the same path for boxes installed before this ships (the drilled `--rescope-acl` posture is unaffected). 1. **Box (agent):** generates the WG keypair; private key 0600 in the agent state dir, never leaves the box (except escrowed — §3.5). 2. **Box → hub:** registers `wg_pubkey` (new box-facing endpoint, slice-2; authenticated by the box's existing hub identity). 3. **Hub:** assigns the `/32` from the tunnel subnet, stores the peer entry, bumps `DesiredGeneration`. 4. **Hub → endpoint VM:** adds `{pubkey, /32}` to the WG server peer list (slice-1 surface). 5. **Box (agent):** next heartbeat sees the generation advance → fetches desired-state → writes the WG config → `install` + `systemctl enable --now wg-felhom` (the sudoers `*.mount` pattern of `felhom-agent.sudoers:16-18`, extended to this unit — slice-3). 6. **Box:** tunnel handshakes (outbound, keepalive); PBS storage is (re)pointed at `pbs_tunnel_ip:8007` (§3.4); first backup runs. Steps 3-4 and 5-6 are order-independent (WG tolerates either side appearing first); the reconcile loop retries until both halves exist. ### 3.4 PBS over the tunnel The PBS client config (`internal/pbs/client.go:24-31`) changes **only its `Server` value**: the endpoint's tunnel-internal address (`pbs_tunnel_ip`) instead of a LAN IP. The TLS exact-cert pin (`internal/pbs/pin.go:17-35`) sits **above** WG and is unchanged through the tunnel — spike-proven both ways (P2b: pin validated through the tunnel; wrong pin rejected). Per-customer namespace + per-customer escrowed encryption key exactly as today (D5); the endpoint sees ciphertext only (01 §10). ### 3.5 Key custody + DR re-establishment The WG private key joins the **escrowed IdentityBundle** (`internal/escrow/identity.go:24-27`, today `{TunnelToken, PBSToken}` → `+ WGPrivateKey`) — the same R-wrapped, zero-knowledge age blob the DR recipe already consumes. On host-loss re-provisioning (01 §9 restore mode): the box recovers the bundle with the customer's recovery code, restores the WG key, and re-establishes the tunnel; hub-side the peer entry still exists (or is re-added on re-registration — same handshake, §3.3). Revocation = hub deletes the peer at the endpoint: the box is off the offsite network immediately, auditable, no box cooperation needed. --- ## 4. Robustness (production details beyond the spike) - **4.1 Customer IP change = free, and explicitly NOT a DynDNS dependency.** The box dials out; WG roaming re-handshakes from the new source address automatically. Nothing anywhere references the customer's IP — the hub peer registry stores pubkey+`/32` only (§3.2). No customer-side DNS, ever. - **4.2 Endpoint referenced by DNS + a re-resolve watchdog.** The endpoint is `dns_name`, not a bare IP, so it can move. Trap to design around: `wg-quick` resolves `Endpoint=` **once** at service start — a later endpoint re-IP would strand running tunnels. The agent's reconcile loop therefore owns re-resolution: on tunnel-unhealthy (no recent handshake), re-resolve and re-apply `wg set … endpoint=` (slice-3). Endpoint moves are rare, hub-announced (coords in desired-state), and self-heal even without the watchdog on service restart. - **4.3 MTU/MSS.** WG costs 60 bytes on v4 (interface MTU 1420 — what the spike ran end-to-end under PBS TLS with zero fragmentation stalls, P3). The unit ships explicit `MTU 1420`; since only the box's own PBS client uses the tunnel (no forwarded flows), no MSS clamp is needed — noted here so nobody adds forwarding without one. - **4.4 Keepalive.** `PersistentKeepalive=25` — spike-proven to hold the NAT mapping through an 11.4-minute fully-idle window (P2) at ~150 B/s of overhead traffic. - **4.5 Isolation.** Per-peer `/32` `AllowedIPs`; IP forwarding stays **off** on the endpoint; its firewall admits, from the WG interface, only the PBS port — so a box can reach the PBS API and nothing else, and boxes cannot see each other **by topology** (spike P6). Public surface: SSH (operator) + the WG UDP port, nothing more. PBS tenancy on top: namespace + per-customer token + per-customer key (D5). - **4.6 Tunnel health → hub.** The tunnel is a storage dependency, so it reports like one — the storage-manifest model (01 §8: agent "continuously checks presence/reachability, and reports per-target status; a disconnected target → actionable notification") gains a tunnel-health input: no handshake within ~3 keepalive periods → the offsite target reports unreachable → the existing alerting path carries it. No new alarm channel. --- ## 5. The endpoint VM (the one new managed-infra piece) Runs three things, nothing else: the WG server (peer list = hub-written), PBS (one datastore, per-customer namespaces), and the minimal peer-sync surface the hub drives. **How** the hub mutates the peer list — a tiny authenticated push API on the VM vs the VM pulling a signed peer manifest from the hub — is deliberately **left as the slice-1 design point** (§7); the trust shape is fixed either way (hub holds the credential; the VM trusts only the hub; mirroring `hub/internal/cloudflare/unblock.go`). Firewall posture per §4.5. Dual-stack (spike P1: keep v6 available even though the pilot line had none). No agent, no controller, no customer plaintext. Bandwidth reality (why D6 is safe): the operator line measured 1 Gbit-class down / ~40 Mbit up (spike P3) — a customer backup is bound by *that customer's own uplink*, and a restore by their downlink; a DC-grade endpoint uplink is never the bottleneck, and customers never contend with the operator's home line (the rejected relay's flaw). Spike-earned deploy note for slice-1: a PBS datastore under `/root` fails (`backup` user cannot traverse `/root`) — create datastores under a world-traversable parent (`/srv`). --- ## 6. Trust-model conformance (01 §3/§4 preserved) - The box **only ever dials out** (UDP to the endpoint; the same posture as hub-poll and PBS-push, 01 §4/§7). No inbound listener is added to any customer box; the hub still never initiates into a box. - The endpoint VM cannot reach into boxes either: it terminates tunnels whose per-peer routes (§4.5) admit only box→PBS traffic; it holds no box credentials. - **Out of scope, explicitly:** (a) operator-inbound access to boxes — a *separate* future problem with its own trust decision; nothing here creates or presumes it; (b) game-server ingress — public players are not tunnel peers; that is a different ingress problem (public reachability), not a backup-transport one. --- ## 7. Open validation + open sub-decisions (honest ledger) **Open validation (one):** true-CGNAT traversal is argued from mechanism (outbound-initiated + keepalive holds a mapping on any NAT tier), **not yet measured** — the spike's line proved to be plain single-NAT with a public IPv4 (spike P0). Cheap closure: a mobile-hotspot smoke test (phone-tethered box → endpoint; mobile networks are true CGNAT); the deferred second-ISP vantage (Peti VM 110) remains the thorough confirmation. Do this before or during slice-3; it does not block slices 1-2. **Open sub-decisions (deferred by design):** - Endpoint peer-sync mechanism (push-API vs signed-manifest pull) — **slice-1 design point** (§5). - WG key rotation cadence/mechanics — the reconcile channel can carry it (new pubkey → hub → endpoint swap); design when fleet size warrants, possibly the moment the "separable fleet layer" (D1) conversation reopens. - Endpoint scaling/sharding (N customers per VM, second region, HU-hosted option) — capacity planning, not architecture; the peer registry already keys per-host. - Backup/DR **of the endpoint VM itself** (datastore redundancy, re-provision runbook) — slice-1 must at least write the re-provision runbook; PBS-side redundancy is a later economics call. --- ## 8. Implementation slice roadmap (each = a separate future task) | Slice | Scope | Done = | |---|---|---| | **S1 — endpoint provisioning + peer-sync surface** | provision the production VM (WG server, PBS, firewall, `/srv` datastore, namespaces); resolve the push-vs-pull peer-sync design point; re-provision runbook | hub can add/remove a peer on the live endpoint via its credential; runbook re-creates the VM from nothing | | **S2 — hub schema + desired-state** | endpoint coords + peer registry; `wireguard` block on `WireDesiredState`; box-facing pubkey-registration endpoint; generation bump on peer changes | a registered box's desired-state carries its assignment; hub UI shows the peer registry | | **S3 — agent WG service (the agent-managed-unit pilot)** | keygen + registration; reconcile `wg-felhom` from desired-state (sudoers additions on the `:16-18` pattern); re-resolve watchdog (§4.2); MTU 1420; IdentityBundle `+ WGPrivateKey`; tunnel-health into the report. **Risky/supervised class** (host service + sudoers + escrow change) | fresh box joins by §3.3 end-to-end; tunnel survives agent restart + endpoint re-IP; unhealthy tunnel alerts | | **S4 — PBS over the tunnel** | point the box's PBS storage at `pbs_tunnel_ip:8007`; verify pin-through-tunnel | real backup lands in the box's namespace over the tunnel (nearly free — spike-proven path) | | **S5 — DR re-establishment** | consume `WGPrivateKey` in restore mode; re-peer on re-provision | host-loss drill restores offsite connectivity with only the recovery code. *Merges with the DR-completeness arc.* | | **S6 — monitoring + isolation hardening** | endpoint-side alerting, per-peer firewall audit, CGNAT smoke-test closure if not yet done | tunnel-down and peer-anomaly both alert; isolation verified adversarially | **MVP that puts offsite backup in a real customer's hands = S1 → S2 → S3; then S4.** S5/S6 complete the story. The cloudflared service (`internal/hub/cloudflared.go:9-12` — today a read-only prober that names itself "the seam for the tunnel-management slice") adopts the S3-piloted agent-managed-service pattern in a later, separate slice.