docs: architecture Part 06 — offsite-connectivity design-of-record (slice roadmap S1-S6)
Records the settled transport decisions (plain WG, host-side agent-managed pilot, one hub-driven endpoint VM running WG+PBS, hub source-of-truth over WireDesiredState, one datastore + per-customer namespaces, relay-through-DooPlex rejected). Grounded at file:line vs felhom.eu@bf099f6 + felhom-agent@4ba1b14. Backlog line resolved to cite spike + design doc. Docs-only; slices are separate future tasks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
+13
@@ -3,6 +3,19 @@
|
||||
> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
|
||||
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
|
||||
|
||||
- **2026-07-03 — OFFSITE-CONNECTIVITY DESIGN-OF-RECORD WRITTEN (architecture Part 06)** —
|
||||
`documentation/architecture/06-offsite-connectivity.md`: records the six settled decisions
|
||||
(plain WG / host-side agent-managed `wg-felhom` as the agent-managed-unit PILOT / one shared
|
||||
hub-driven endpoint VM running WG+PBS, NO agent on it / hub = source of truth, `wireguard`
|
||||
block rides `WireDesiredState` on the existing DesiredGeneration channel / one datastore +
|
||||
per-customer namespaces / PBS ON the VM, relay-through-DooPlex REJECTED as non-scaling),
|
||||
the Day-0 join handshake, robustness set (WG-roaming-not-DynDNS, DNS re-resolve watchdog, MTU
|
||||
1420, per-/32 isolation, tunnel-health via storage-target model), trust conformance (outbound
|
||||
only; operator-inbound + game-ingress out of scope), the honest ledger (CGNAT still unmeasured
|
||||
— mobile-hotspot smoke closure; peer-sync push-vs-pull = slice-1 design point), and the S1–S6
|
||||
slice roadmap (**MVP = S1 endpoint + S2 hub schema/desired-state + S3 agent WG service, then S4
|
||||
PBS-over-tunnel**; S5 merges with DR-completeness). day0-install backlog line now cites spike +
|
||||
design doc. Docs-only; slices are separate future tasks.
|
||||
- **2026-07-03 — OFFSITE-BACKUP TRANSPORT DECIDED (connectivity spike, GO at the lowest rung)** —
|
||||
plain WireGuard, host-side, box-dials-out to a small public endpoint; **no Headscale/DERP needed
|
||||
for backup** (separable fleet layer, adoptable later). Live-proven on demo-felhom → throwaway
|
||||
|
||||
@@ -4,69 +4,59 @@
|
||||
|
||||
---
|
||||
|
||||
# REPORT — SPIKE: offsite-backup connectivity (WireGuard, home-NAT traversal) (2026-07-03)
|
||||
# REPORT — architecture Part 06: offsite-connectivity design-of-record (2026-07-03)
|
||||
|
||||
Docs-only run; evidence doc: `documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md`.
|
||||
No production code, no Headscale on the real hub, no agent change, no persistent felhom-pve config.
|
||||
Docs-only. Created `documentation/architecture/06-offsite-connectivity.md` (design-of-record for
|
||||
the offsite-backup transport); resolved the `day0-install.md` backlog line to cite spike + design
|
||||
doc; CONTEXT + scripts/CHANGELOG updated. No code, no version bump, no deploy, no endpoint
|
||||
provisioning.
|
||||
|
||||
## Verdict
|
||||
## Baselines (verified fresh-pulled before writing)
|
||||
|
||||
**GO at the lowest rung (H1): plain WireGuard, host-side, box-dials-out to a small public
|
||||
endpoint.** No coordinator, no relay, no mesh. Headscale (P5: 0.29.2 stood up throwaway) is a
|
||||
*separable fleet-management layer* — TLS control plane + DB + tailscaled-per-box + Tailscale-hosted
|
||||
DERP by default — not a traversal necessity for hub-and-spoke backup.
|
||||
- felhom.eu @ `bf099f6` (spike doc commit; this task's commit lands on top)
|
||||
- felhom-agent @ `4ba1b14` (v0.63.0, read-only grounding)
|
||||
|
||||
## Probe ladder results (both ends real: demo-felhom PVE host ⟷ throwaway Hetzner CX23)
|
||||
## Grounding citations used (for reviewer validation against Gitea)
|
||||
|
||||
- **P0** — the operator's line is **plain single-NAT with a fixed public IPv4**
|
||||
(`37.191.56.193`, catv.fixed.one.hu, AS21334), *not* CGNAT (no 100.64/10 hop; source IP at the
|
||||
server = egress IP). **Zero IPv6** (no address, no egress, 15-min RA capture: 0 packets).
|
||||
- **P1 (H0 v6-direct)** — N/A on this line; keep the production endpoint dual-stack anyway.
|
||||
- **P2 (H1)** — plain WG v4 up on the **first try**; RTT ~30–40 ms; **11.4-min fully-idle hold**
|
||||
(keepalive-only, NAT mapping unmoved, instant ping after); PBS 4.2.2 API through the tunnel with
|
||||
the **exact-cert pin intact** (`pin.go` model) + negative wrong-pin proof; `:8007` tunnel-only
|
||||
by firewall (public probe: unreachable).
|
||||
- **P3** — uplink is the bottleneck: direct 42 Mbit/s up / 200–230 down; tunnel upload ~39 Mbit/s
|
||||
(~5% overhead); PBS TLS benchmark through tunnel **4.94 MB/s = the full uplink**; **real 2 GiB
|
||||
worst-case (urandom) backup: 4.26 MiB/s sustained, 8 min, zero stalls**. First backup 50–100 GB
|
||||
≈ 3–6 h (overnight window). Oddity recorded: tunnel *download* (565 Mbit/s, 0 retrans) beat
|
||||
direct TCP (230, 285 retrans) — UDP encapsulation dodges whatever shapes inbound TCP.
|
||||
- **P4 (H2)** — outbound UDP fine on **51820 and 443**; no TCP/443 fallback machinery needed.
|
||||
- **P6** — plain-WG isolation is topological: per-peer `/32` AllowedIPs + forwarding off at the
|
||||
endpoint; only the PBS port admitted from the WG interface.
|
||||
- **P7** — hypothesis **refuted with evidence**: kernel WG link creation **succeeds** inside the
|
||||
unprivileged guest (no `/dev/net/tun` needed — that's confirmed absent and only blocks
|
||||
tailscaled). Host placement therefore stands on architecture: backups originate host-side, the
|
||||
tunnel must survive guest death (cloudflared precedent), the guest keeps zero infra creds.
|
||||
| Claim in the doc | Cited at |
|
||||
|---|---|
|
||||
| Backup = outbound push; PBS at LAN address | `01-topology-and-trust.md` §7; `felhom-agent/internal/pbs/client.go:25-26` (`Server` documented as `"192.168.0.180"`, port default 8007) |
|
||||
| Public edge HTTP-only, no public UDP | `felhom.eu/manifests/hub.yaml:233` (`ingressClassName: nginx-internal`) |
|
||||
| TLS pin above WG, unchanged through tunnel | `felhom-agent/internal/pbs/pin.go:17-35` (exact-cert `VerifyPeerCertificate`); spike P2b (positive + negative through-tunnel proof) |
|
||||
| Desired-state channel + generation gate | `felhom-agent/internal/hub/report.go:270` (`DesiredGeneration`), `report.go:289-296` (`WireDesiredState` + forward-compat siblings), `internal/desired/syncer.go:44-61` (fetch only on generation advance) |
|
||||
| IdentityBundle to extend | `felhom-agent/internal/escrow/identity.go:24-27` (`{TunnelToken, PBSToken}`) |
|
||||
| Agent already installs+enables host units | `felhom-agent/configs/felhom-agent.sudoers:16-18` (`install … /etc/systemd/system/*.mount`, `systemctl enable --now`) |
|
||||
| Hub drives external infra via held token | `felhom.eu/hub/internal/cloudflare/unblock.go` (CF token, WAF mutation) |
|
||||
| cloudflared = later adopter of the S3 pattern | `felhom-agent/internal/hub/cloudflared.go:9-12` (read-only prober, self-described "seam for the tunnel-management slice") |
|
||||
| All measurements | `documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md` (P0–P7) |
|
||||
|
||||
## Architecture recommendation (input to the next task — the production spec)
|
||||
## What the doc marks DECIDED (recorded, not re-litigated)
|
||||
|
||||
Host-side agent-managed WG service (cloudflared pattern); per-box keypair minted at install →
|
||||
pubkey to hub, private key 0600 in agent state dir + **joins the escrowed IdentityBundle** for DR;
|
||||
endpoint = small public dual-stack VM (€4–8/mo class), two shapes deferred to the spec: PBS-on-VM
|
||||
vs **VM-as-rendezvous-relay with DooPlex PBS dialing out too** (both ends outbound-only,
|
||||
ciphertext-only transit — also solves DooPlex's no-public-UDP). Trust model preserved: box only
|
||||
dials out; operator-inbound (B) and game-ingress (C) explicitly out of scope.
|
||||
D1 plain WG (no Headscale for traversal — separable future fleet layer); D2 host-side
|
||||
**agent-managed** `wg-felhom` (pilot for agent-managed host services; install-script placement
|
||||
rejected as one-shot; in-guest rejected on trust grounds — spike P7 shows it *works*, so the
|
||||
refutation is architectural); D3 one shared dumb hub-driven endpoint VM (WG + PBS, no agent);
|
||||
D4 hub = source of truth, `wireguard` block on the existing desired-state channel; D5 one
|
||||
datastore + per-customer namespaces; D6 PBS ON the VM — relay-through-DooPlex rejected
|
||||
(non-scaling through the ~40 Mbit home uplink, couples all DR to home uptime; DooPlex PBS =
|
||||
dev/demo only).
|
||||
|
||||
## Honest caveats
|
||||
## What the doc marks DEFERRED / OPEN (honest ledger, §7)
|
||||
|
||||
CGNAT traversal is argued from mechanism, **not proven on this line** (it's plain-NAT); second-ISP
|
||||
confirmation deferred to Peti's VM 110 (currently unreachable). Single ISP/evening; no multi-day
|
||||
hold; no home-IP-rotation event observed; Headscale assessed server-side only (no tailscaled on
|
||||
the production host); no restore-direction PBS run.
|
||||
- **Open validation (one):** true-CGNAT traversal — unmeasured (operator line = plain-NAT, spike
|
||||
P0); closure = mobile-hotspot smoke test, thorough = Peti VM 110; does not block S1/S2.
|
||||
- Endpoint peer-sync mechanism (push-API vs signed-manifest pull) — slice-1 design point.
|
||||
- WG key-rotation mechanics; endpoint scaling/sharding/HU-hosting; endpoint-VM self-DR depth.
|
||||
|
||||
## Cleanup (verified)
|
||||
## Slice roadmap set (each a future task)
|
||||
|
||||
demo-felhom: interface/keys/config/dataset gone (keys **shredded**), iperf3 purged, WG module
|
||||
unloaded, `/etc/wireguard` untouched-empty, guest probe link deleted — residue-free. Hetzner
|
||||
throwaway: WG + headscale + PBS torn down, secrets shredded, datastore deleted, box **powered
|
||||
off** (parked for the Peti follow-up or destruction; jarrs.eu box never touched).
|
||||
S1 endpoint provisioning + peer-sync surface → S2 hub schema + desired-state → S3 agent WG
|
||||
service (**risky/supervised**: sudoers + host service + escrow change) → S4 PBS-over-tunnel →
|
||||
S5 DR re-establishment (merges with DR-completeness) → S6 monitoring/hardening.
|
||||
**MVP = S1→S2→S3, then S4.**
|
||||
|
||||
## Repo changes (docs-only)
|
||||
## Author notes
|
||||
|
||||
- NEW `documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md` (P0–P7 verbatim + twelve
|
||||
§7 answers + recommendation).
|
||||
- `documentation/runbooks/day0-install.md` — the "LAN-only until Headscale/WireGuard lands"
|
||||
backlog line now points at the spike (decision made; spec is the remaining gate).
|
||||
- `CONTEXT.md` — transport decision recorded; next task = production connectivity spec.
|
||||
- `scripts/CHANGELOG.md` — docs entry.
|
||||
None — no §2 decision looked wrong while writing; the one place the task's own earlier framing
|
||||
was contradicted by evidence (guest-WG infeasibility) was already resolved by the spike and is
|
||||
recorded as an architectural (not technical) refutation in D2.
|
||||
|
||||
@@ -0,0 +1,216 @@
|
||||
# Architecture Part 6 — Offsite Connectivity (the backup transport)
|
||||
|
||||
> Status: **design-of-record** (2026-07-03). Records the settled offsite-backup-transport
|
||||
> decisions; grounded against felhom.eu @ `bf099f6` and felhom-agent @ `4ba1b14` (v0.63.0).
|
||||
> Evidence base: `documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md` (all
|
||||
> measurements cited below as "spike Pn"). This document ends in an implementation slice roadmap
|
||||
> (§8); **no slice is implemented yet** — each is a separate future task. Consistent with and
|
||||
> extending Part 01 (`01-topology-and-trust.md`); where this doc names a component's duty it uses
|
||||
> Part 01 §3 vocabulary.
|
||||
|
||||
---
|
||||
|
||||
## 1. Why (the gap)
|
||||
|
||||
Offsite backup — the paid DR substrate (01 §8) — is structurally unreachable for a real customer
|
||||
today. Backup is an **outbound push to PBS** (01 §7: "Outbound only for control/report/backup"),
|
||||
but the only PBS lives at a **LAN address** — `internal/pbs/client.go:25-26` documents the
|
||||
`Server` field as literally `"192.168.0.180"`, port 8007. The operator's public edge is
|
||||
Cloudflare-Tunnel → `nginx-internal` (`felhom.eu/manifests/hub.yaml:233`,
|
||||
`ingressClassName: nginx-internal`) — HTTP(S) only; DooPlex has **no public IP and no public UDP**.
|
||||
So offsite backup currently works only because the demo box shares a LAN with the PBS host; a
|
||||
customer box cannot reach it at all. The DR crypto/identity side is done and drilled
|
||||
(`SPIKE-dr-recipe-2026-06-16.md`); **this transport is the last missing piece**, and the spike
|
||||
proved the cheapest transport suffices.
|
||||
|
||||
---
|
||||
|
||||
## 2. Decisions (settled — recorded, not re-litigated)
|
||||
|
||||
| # | Decision | Rationale (one line) | Rejected alternative |
|
||||
|---|----------|----------------------|----------------------|
|
||||
| D1 | **Transport = plain WireGuard**, box-dials-outbound, `PersistentKeepalive=25` | Backup is hub-and-spoke to a *public* endpoint; outbound UDP + keepalive traverses any NAT tier — spike P2 (first-try up, 11.4-min idle hold), P4 (UDP 51820 *and* 443 pass) | Headscale/DERP **for traversal** — solves the p2p problem we don't have; a TLS control plane + DB + tailscaled-per-box + third-party DERP dependency for nothing (spike P5). Stays available as a *separable future fleet layer*. |
|
||||
| D2 | **Placement = customer host, AGENT-MANAGED** (`wg-felhom` systemd service reconciled from hub desired-state) | WG config is *ongoing reconciled desired-state* — the agent's job; the host-install script is one-shot and cannot react to hub changes (endpoint move, key rotation). Pattern exists: the agent already installs+enables host units via `configs/felhom-agent.sudoers:16-18` (`install … /etc/systemd/system/*.mount` + `systemctl enable --now`) | (a) install-script-owned static config — can't reconcile; (b) in-guest WG — *works* technically (spike P7: kernel WG needs no TUN) but violates 01 §3 (guest holds zero infra creds) and dies with the guest; backups originate host-side anyway. |
|
||||
| D3 | **Endpoint = ONE shared public dual-stack cloud VM** (Hetzner/EU class; HU optional later) running **WG server + the offsite PBS**. It is **dumb and hub-driven** — no agent on it; the hub is the only thing that mutates its WG peer list | Mirrors the proven "hub holds the credential and drives external infra" pattern (`hub/internal/cloudflare/unblock.go` — hub-held CF token mutating WAF rules). Zero-knowledge client-side encryption (01 §8) makes the VM's location privacy-neutral | An agent on the endpoint — a whole trust tier for a box that only needs a peer list and a datastore. |
|
||||
| D4 | **Hub = source of truth** for endpoint coordinates (DNS name, port, server pubkey) + the per-customer peer registry (customer pubkey → assigned tunnel `/32`), delivered over the **existing desired-state channel** | The channel is built for exactly this: heartbeat carries `DesiredGeneration` (`internal/hub/report.go:270`); the agent fetches the heavy state only on generation advance (`internal/desired/syncer.go:44-61`); `WireDesiredState` (`report.go:289-296`) already carries forward-compat sibling blocks (`RestoreDirective`, `StorageManifest`) — a `wireguard` block joins them | A parallel bespoke config channel. |
|
||||
| D5 | **PBS layout = one datastore, per-customer NAMESPACES**, each namespace under its own client-side encryption key (already escrowed — 01 §8 key custody) | PBS-native multi-tenancy; crypto isolation without datastore sprawl; the hub already holds "PBS namespace" in the customer record (01 §6) and `WireDesiredState.PBSNamespace` already exists on the wire (`report.go:294`) | Per-customer datastores — operational sprawl with no isolation gain over namespace+key. |
|
||||
| D6 | **Offsite PBS lives ON the cloud VM — NOT relayed through DooPlex** | Relaying funnels *every* customer's backup through the operator's one shared ~40 Mbit home uplink and couples all customer DR to home-connection uptime — non-scaling and a single point of failure. DC uplink is never the bottleneck (§5) | The spike's §4.4(b) rendezvous-relay shape — rejected by the operator for production. **DooPlex PBS → dev/demo only.** |
|
||||
|
||||
---
|
||||
|
||||
## 3. Architecture
|
||||
|
||||
### 3.1 Components & trust (extends 01 §3 / §5)
|
||||
|
||||
| | Customer host | Endpoint VM | Hub |
|
||||
|---|---|---|---|
|
||||
| Runs | `wg-felhom` systemd service (agent-reconciled) + the existing agent | WG server + PBS (one datastore, per-customer namespaces) + a minimal hub-driven peer-sync surface | peer registry + endpoint coords in the customer/host record |
|
||||
| Holds | its own WG **private key** (0600, agent state dir) + PBS token + the pinned PBS fingerprint | WG server private key; the PBS ciphertext (zero-knowledge — 01 §10) | every box's WG **public** key + assigned `/32`; the endpoint-mutation credential; escrowed key material (unchanged, 01 §8) |
|
||||
| Mutates | only its own WG interface/service | nothing on its own — peer list changes arrive from the hub | the endpoint's peer list; the desired-state the agent consumes |
|
||||
| Never | accepts inbound (dials out only) | initiates into a box or the hub; holds plaintext | connects into a box (01 §4 — unchanged) |
|
||||
|
||||
New trust-boundary rows (extends 01 §5):
|
||||
|
||||
| Boundary | What crosses | Mechanism | Blast radius if breached |
|
||||
|---|---|---|---|
|
||||
| box ↔ endpoint VM | WG-encapsulated, TLS-pinned, client-side-encrypted PBS traffic | outbound UDP; per-peer `/32`; no inter-peer routing (§4.5) | that box's tunnel; ciphertext only |
|
||||
| hub ↔ endpoint VM | peer-list mutations (add/remove pubkey+`/32`) | hub-held credential, minimal surface (mechanism = slice-1 design point, §7) | the peer list — an attacker can *disconnect* backups, not read them |
|
||||
|
||||
### 3.2 Hub data model (conceptual fields — not SQL, not wire-final)
|
||||
|
||||
- **Endpoint record** (one, for now): `dns_name`, `wg_port`, `server_pubkey`, `tunnel_subnet`,
|
||||
`pbs_tunnel_ip` (the endpoint's in-tunnel address the PBS client dials), plus the
|
||||
endpoint-mutation credential (out-of-band secret, never in desired-state).
|
||||
- **Per-host peer entry** (joins the customer/host record, 01 §6): `wg_pubkey` (box-generated,
|
||||
box-registered), `assigned_ip` (the box's tunnel `/32`, hub-assigned), `status`.
|
||||
- **On the wire:** a `wireguard` block on `WireDesiredState` (`report.go:289` — sibling to
|
||||
`Guests`/`RestoreDirective`, same forward-compat carriage rule as `report.go:284-288`: carried +
|
||||
cached by old agents, acted on by new ones): endpoint coords + this box's assignment. Delivery
|
||||
needs **zero new plumbing** — bump `DesiredGeneration`, the agent's existing syncer fetches
|
||||
(`syncer.go:44-61`).
|
||||
|
||||
### 3.3 Provisioning handshake (Day-0 join)
|
||||
|
||||
Plugs into the existing Day-0 flow (`runbooks/day0-install.md`) after agent enrollment (01 §6 —
|
||||
"the agent enrolls first"); offsite stays retrofittable via the same path for boxes installed
|
||||
before this ships (the drilled `--rescope-acl` posture is unaffected).
|
||||
|
||||
1. **Box (agent):** generates the WG keypair; private key 0600 in the agent state dir, never
|
||||
leaves the box (except escrowed — §3.5).
|
||||
2. **Box → hub:** registers `wg_pubkey` (new box-facing endpoint, slice-2; authenticated by the
|
||||
box's existing hub identity).
|
||||
3. **Hub:** assigns the `/32` from the tunnel subnet, stores the peer entry, bumps
|
||||
`DesiredGeneration`.
|
||||
4. **Hub → endpoint VM:** adds `{pubkey, /32}` to the WG server peer list (slice-1 surface).
|
||||
5. **Box (agent):** next heartbeat sees the generation advance → fetches desired-state → writes
|
||||
the WG config → `install` + `systemctl enable --now wg-felhom` (the sudoers `*.mount` pattern
|
||||
of `felhom-agent.sudoers:16-18`, extended to this unit — slice-3).
|
||||
6. **Box:** tunnel handshakes (outbound, keepalive); PBS storage is (re)pointed at
|
||||
`pbs_tunnel_ip:8007` (§3.4); first backup runs.
|
||||
|
||||
Steps 3-4 and 5-6 are order-independent (WG tolerates either side appearing first); the reconcile
|
||||
loop retries until both halves exist.
|
||||
|
||||
### 3.4 PBS over the tunnel
|
||||
|
||||
The PBS client config (`internal/pbs/client.go:24-31`) changes **only its `Server` value**: the
|
||||
endpoint's tunnel-internal address (`pbs_tunnel_ip`) instead of a LAN IP. The TLS exact-cert pin
|
||||
(`internal/pbs/pin.go:17-35`) sits **above** WG and is unchanged through the tunnel —
|
||||
spike-proven both ways (P2b: pin validated through the tunnel; wrong pin rejected). Per-customer
|
||||
namespace + per-customer escrowed encryption key exactly as today (D5); the endpoint sees
|
||||
ciphertext only (01 §10).
|
||||
|
||||
### 3.5 Key custody + DR re-establishment
|
||||
|
||||
The WG private key joins the **escrowed IdentityBundle**
|
||||
(`internal/escrow/identity.go:24-27`, today `{TunnelToken, PBSToken}` → `+ WGPrivateKey`) — the
|
||||
same R-wrapped, zero-knowledge age blob the DR recipe already consumes. On host-loss
|
||||
re-provisioning (01 §9 restore mode): the box recovers the bundle with the customer's recovery
|
||||
code, restores the WG key, and re-establishes the tunnel; hub-side the peer entry still exists
|
||||
(or is re-added on re-registration — same handshake, §3.3). Revocation = hub deletes the peer at
|
||||
the endpoint: the box is off the offsite network immediately, auditable, no box cooperation
|
||||
needed.
|
||||
|
||||
---
|
||||
|
||||
## 4. Robustness (production details beyond the spike)
|
||||
|
||||
- **4.1 Customer IP change = free, and explicitly NOT a DynDNS dependency.** The box dials out;
|
||||
WG roaming re-handshakes from the new source address automatically. Nothing anywhere references
|
||||
the customer's IP — the hub peer registry stores pubkey+`/32` only (§3.2). No customer-side DNS,
|
||||
ever.
|
||||
- **4.2 Endpoint referenced by DNS + a re-resolve watchdog.** The endpoint is `dns_name`, not a
|
||||
bare IP, so it can move. Trap to design around: `wg-quick` resolves `Endpoint=` **once** at
|
||||
service start — a later endpoint re-IP would strand running tunnels. The agent's reconcile loop
|
||||
therefore owns re-resolution: on tunnel-unhealthy (no recent handshake), re-resolve and re-apply
|
||||
`wg set … endpoint=` (slice-3). Endpoint moves are rare, hub-announced (coords in desired-state),
|
||||
and self-heal even without the watchdog on service restart.
|
||||
- **4.3 MTU/MSS.** WG costs 60 bytes on v4 (interface MTU 1420 — what the spike ran end-to-end
|
||||
under PBS TLS with zero fragmentation stalls, P3). The unit ships explicit `MTU 1420`; since
|
||||
only the box's own PBS client uses the tunnel (no forwarded flows), no MSS clamp is needed —
|
||||
noted here so nobody adds forwarding without one.
|
||||
- **4.4 Keepalive.** `PersistentKeepalive=25` — spike-proven to hold the NAT mapping through an
|
||||
11.4-minute fully-idle window (P2) at ~150 B/s of overhead traffic.
|
||||
- **4.5 Isolation.** Per-peer `/32` `AllowedIPs`; IP forwarding stays **off** on the endpoint; its
|
||||
firewall admits, from the WG interface, only the PBS port — so a box can reach the PBS API and
|
||||
nothing else, and boxes cannot see each other **by topology** (spike P6). Public surface: SSH
|
||||
(operator) + the WG UDP port, nothing more. PBS tenancy on top: namespace + per-customer token +
|
||||
per-customer key (D5).
|
||||
- **4.6 Tunnel health → hub.** The tunnel is a storage dependency, so it reports like one — the
|
||||
storage-manifest model (01 §8: agent "continuously checks presence/reachability, and reports
|
||||
per-target status; a disconnected target → actionable notification") gains a tunnel-health
|
||||
input: no handshake within ~3 keepalive periods → the offsite target reports unreachable → the
|
||||
existing alerting path carries it. No new alarm channel.
|
||||
|
||||
---
|
||||
|
||||
## 5. The endpoint VM (the one new managed-infra piece)
|
||||
|
||||
Runs three things, nothing else: the WG server (peer list = hub-written), PBS (one datastore,
|
||||
per-customer namespaces), and the minimal peer-sync surface the hub drives. **How** the hub
|
||||
mutates the peer list — a tiny authenticated push API on the VM vs the VM pulling a signed peer
|
||||
manifest from the hub — is deliberately **left as the slice-1 design point** (§7); the trust
|
||||
shape is fixed either way (hub holds the credential; the VM trusts only the hub; mirroring
|
||||
`hub/internal/cloudflare/unblock.go`). Firewall posture per §4.5. Dual-stack (spike P1: keep v6
|
||||
available even though the pilot line had none). No agent, no controller, no customer plaintext.
|
||||
|
||||
Bandwidth reality (why D6 is safe): the operator line measured 1 Gbit-class down / ~40 Mbit up
|
||||
(spike P3) — a customer backup is bound by *that customer's own uplink*, and a restore by their
|
||||
downlink; a DC-grade endpoint uplink is never the bottleneck, and customers never contend with
|
||||
the operator's home line (the rejected relay's flaw).
|
||||
|
||||
Spike-earned deploy note for slice-1: a PBS datastore under `/root` fails (`backup` user cannot
|
||||
traverse `/root`) — create datastores under a world-traversable parent (`/srv`).
|
||||
|
||||
---
|
||||
|
||||
## 6. Trust-model conformance (01 §3/§4 preserved)
|
||||
|
||||
- The box **only ever dials out** (UDP to the endpoint; the same posture as hub-poll and
|
||||
PBS-push, 01 §4/§7). No inbound listener is added to any customer box; the hub still never
|
||||
initiates into a box.
|
||||
- The endpoint VM cannot reach into boxes either: it terminates tunnels whose per-peer routes
|
||||
(§4.5) admit only box→PBS traffic; it holds no box credentials.
|
||||
- **Out of scope, explicitly:** (a) operator-inbound access to boxes — a *separate* future
|
||||
problem with its own trust decision; nothing here creates or presumes it; (b) game-server
|
||||
ingress — public players are not tunnel peers; that is a different ingress problem (public
|
||||
reachability), not a backup-transport one.
|
||||
|
||||
---
|
||||
|
||||
## 7. Open validation + open sub-decisions (honest ledger)
|
||||
|
||||
**Open validation (one):** true-CGNAT traversal is argued from mechanism (outbound-initiated +
|
||||
keepalive holds a mapping on any NAT tier), **not yet measured** — the spike's line proved to be
|
||||
plain single-NAT with a public IPv4 (spike P0). Cheap closure: a mobile-hotspot smoke test
|
||||
(phone-tethered box → endpoint; mobile networks are true CGNAT); the deferred second-ISP vantage
|
||||
(Peti VM 110) remains the thorough confirmation. Do this before or during slice-3; it does not
|
||||
block slices 1-2.
|
||||
|
||||
**Open sub-decisions (deferred by design):**
|
||||
- Endpoint peer-sync mechanism (push-API vs signed-manifest pull) — **slice-1 design point** (§5).
|
||||
- WG key rotation cadence/mechanics — the reconcile channel can carry it (new pubkey → hub →
|
||||
endpoint swap); design when fleet size warrants, possibly the moment the "separable fleet
|
||||
layer" (D1) conversation reopens.
|
||||
- Endpoint scaling/sharding (N customers per VM, second region, HU-hosted option) — capacity
|
||||
planning, not architecture; the peer registry already keys per-host.
|
||||
- Backup/DR **of the endpoint VM itself** (datastore redundancy, re-provision runbook) — slice-1
|
||||
must at least write the re-provision runbook; PBS-side redundancy is a later economics call.
|
||||
|
||||
---
|
||||
|
||||
## 8. Implementation slice roadmap (each = a separate future task)
|
||||
|
||||
| Slice | Scope | Done = |
|
||||
|---|---|---|
|
||||
| **S1 — endpoint provisioning + peer-sync surface** | provision the production VM (WG server, PBS, firewall, `/srv` datastore, namespaces); resolve the push-vs-pull peer-sync design point; re-provision runbook | hub can add/remove a peer on the live endpoint via its credential; runbook re-creates the VM from nothing |
|
||||
| **S2 — hub schema + desired-state** | endpoint coords + peer registry; `wireguard` block on `WireDesiredState`; box-facing pubkey-registration endpoint; generation bump on peer changes | a registered box's desired-state carries its assignment; hub UI shows the peer registry |
|
||||
| **S3 — agent WG service (the agent-managed-unit pilot)** | keygen + registration; reconcile `wg-felhom` from desired-state (sudoers additions on the `:16-18` pattern); re-resolve watchdog (§4.2); MTU 1420; IdentityBundle `+ WGPrivateKey`; tunnel-health into the report. **Risky/supervised class** (host service + sudoers + escrow change) | fresh box joins by §3.3 end-to-end; tunnel survives agent restart + endpoint re-IP; unhealthy tunnel alerts |
|
||||
| **S4 — PBS over the tunnel** | point the box's PBS storage at `pbs_tunnel_ip:8007`; verify pin-through-tunnel | real backup lands in the box's namespace over the tunnel (nearly free — spike-proven path) |
|
||||
| **S5 — DR re-establishment** | consume `WGPrivateKey` in restore mode; re-peer on re-provision | host-loss drill restores offsite connectivity with only the recovery code. *Merges with the DR-completeness arc.* |
|
||||
| **S6 — monitoring + isolation hardening** | endpoint-side alerting, per-peer firewall audit, CGNAT smoke-test closure if not yet done | tunnel-down and peer-anomaly both alert; isolation verified adversarially |
|
||||
|
||||
**MVP that puts offsite backup in a real customer's hands = S1 → S2 → S3; then S4.** S5/S6
|
||||
complete the story. The cloudflared service (`internal/hub/cloudflared.go:9-12` — today a
|
||||
read-only prober that names itself "the seam for the tunnel-management slice") adopts the
|
||||
S3-piloted agent-managed-service pattern in a later, separate slice.
|
||||
@@ -119,11 +119,12 @@ Notes:
|
||||
Felhom-pool guests — other guests on a shared box are invisible there. Always pick from
|
||||
`pct list` + `qm list` on the box (`documentation/runbooks/provisioning.md`).
|
||||
- Offsite backup (`felhom-pbs`) is NOT part of a go-live today: the PBS host is LAN-only. The
|
||||
transport decision is now MADE — plain WireGuard, host-side, box-dials-out to a small public
|
||||
endpoint (spike-proven: `documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md`);
|
||||
what remains is the production connectivity spec + endpoint VM. Go-live standard = **local
|
||||
backups only** (that is what the Part C command configures); offsite DR is retrofitted later
|
||||
via `--rescope-acl` / `--acl-storages` without reinstalling.
|
||||
transport is DECIDED and DESIGNED — plain WireGuard, host-side agent-managed, box-dials-out to
|
||||
a public endpoint VM running the offsite PBS. Evidence:
|
||||
`documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md`; design-of-record + slice
|
||||
roadmap: `documentation/architecture/06-offsite-connectivity.md` (MVP = slices S1→S2→S3, then
|
||||
S4). Go-live standard = **local backups only** (that is what the Part C command configures);
|
||||
offsite DR is retrofitted later via `--rescope-acl` / `--acl-storages` without reinstalling.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -1,5 +1,22 @@
|
||||
# Felhom scripts — Changelog
|
||||
|
||||
## docs — architecture Part 06: offsite connectivity design-of-record (2026-07-03)
|
||||
|
||||
`documentation/architecture/06-offsite-connectivity.md` — the settled offsite-backup-transport
|
||||
design, authored from the spike verdict + operator-resolved forks (recorded, not re-litigated):
|
||||
plain WG (D1), host-side **agent-managed** `wg-felhom` as the agent-managed-unit pilot on the
|
||||
sudoers `*.mount` install pattern (D2), one shared hub-driven endpoint VM running WG + the
|
||||
offsite PBS with no agent (D3, CF-token pattern), hub source-of-truth with a `wireguard` block
|
||||
riding the existing `WireDesiredState`/DesiredGeneration channel (D4), one datastore +
|
||||
per-customer namespaces (D5), and PBS **on** the VM — relay-through-DooPlex rejected as
|
||||
non-scaling (D6). Includes the Day-0 join handshake, the robustness set (NOT-DynDNS roaming,
|
||||
endpoint DNS re-resolve watchdog, MTU 1420, per-/32 topological isolation, tunnel-health through
|
||||
the storage-target reachability model), trust conformance, the honest open ledger (CGNAT
|
||||
unmeasured → mobile-hotspot smoke test; peer-sync push-vs-pull = slice-1 design point), and the
|
||||
S1–S6 slice roadmap (MVP = S1→S2→S3, then S4; S5 merges with DR-completeness). All claims cited
|
||||
at file:line against felhom.eu @ bf099f6 + felhom-agent @ 4ba1b14. `day0-install.md` backlog line
|
||||
now points at spike + design doc. Docs-only.
|
||||
|
||||
## docs — SPIKE: offsite-backup connectivity — plain WireGuard WINS the ladder; Headscale = separable fleet layer (2026-07-03)
|
||||
|
||||
`documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md` — the offsite-backup transport
|
||||
|
||||
Reference in New Issue
Block a user