felhom.eu/documentation/architecture/06-offsite-connectivity.md

Architecture Part 6 — Offsite Connectivity (the backup transport)

How to read this document. Two kinds of statement appear, and where this document marks them it marks them like this — the same wording as 07-backup-architecture.md:11-17, carried here on 2026-08-22 (R-376) so a reader meets one convention and not eight:

• [DESIGN] — a decision taken. Not derived from code; the code may not implement it yet.
• [FACT] — an observed property, carrying a file:line, a live command output or a citation.

Statements in this document are NOT yet all marked. Marking them wholesale is a large judgement exercise and a wrong mark is worse than none, so only what a session touches is marked (R-376). An unmarked statement therefore means "not yet classified", never "observed". That ambiguity is exactly what cost this project three sessions in August 2026: the hot/bulk placement decision sat unmarked beside a marked [FACT], and was read as an observation and reported as a defect.

Status: design-of-record (2026-07-03). Records the settled offsite-backup-transport decisions; grounded against felhom.eu @ bf099f6 and felhom-agent @ 4ba1b14 (v0.63.0). Evidence base: documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md (all measurements cited below as "spike Pn"). This document ends in an implementation slice roadmap (§8); no slice is implemented yet — each is a separate future task. Consistent with and extending Part 01 (01-topology-and-trust.md); where this doc names a component's duty it uses Part 01 §3 vocabulary.

---

1. Why (the gap)

Offsite backup — the paid DR substrate (01 §8) — is structurally unreachable for a real customer today. Backup is an outbound push to PBS (01 §7: "Outbound only for control/report/backup"), but the only PBS lives at a LAN address — internal/pbs/client.go:25-26 documents the Server field as literally "192.168.0.180", port 8007. The operator's public edge is Cloudflare-Tunnel → nginx-internal (felhom.eu/manifests/hub.yaml:233, ingressClassName: nginx-internal) — HTTP(S) only; DooPlex has no public IP and no public UDP. So offsite backup currently works only because the demo box shares a LAN with the PBS host; a customer box cannot reach it at all. The DR crypto/identity side is done and drilled (SPIKE-dr-recipe-2026-06-16.md); this transport is the last missing piece, and the spike proved the cheapest transport suffices.

STALE AS OF 2026-10-09 — the two transport facts in the paragraph above are no longer true, and the gap they describe is closed. Kept rather than rewritten, because the paragraph is the reason ep0 exists and the reasoning only makes sense with its premise visible.

MEASURED 2026-10-09, two channels. (1) DNS, from public resolvers (@1.1.1.1 and @8.8.8.8, not DooPlex's own, which answers the LAN address): hub.felhom.eu is a CNAME to dooplex.hopto.org → 37.191.56.193, which is not a Cloudflare address, and curl -sI https://hub.felhom.eu/ returns no cf-ray header. (2) The manifest: the hub is an ordinary k3s Ingress and manifests/hub.yaml names Cloudflare only in a setup comment about pointing DNS at the cluster. The control that makes this mean something: iso.felhom.eu resolves to 172.67.183.101 / 104.21.18.218 — real Cloudflare addresses — so the method does detect proxying when it is there.

So: DooPlex is reachable on a public address today (a dynamic-DNS name, No-IP dooplex.hopto.org, S13 in runbooks/total-loss-of-dooplex.md), and the hub's public edge is NOT a Cloudflare Tunnel — boxes reporting to hub.felhom.eu reach the home connection directly. Cloudflare is DNS for that name and nothing more. What Cloudflare does carry is the households' app traffic through their tunnels, which is a different path and is the one the system poster's "Known gaps" box is about. runbooks/total-loss-of-dooplex.md §9's "today a CNAME to dooplex.hopto.org" is correct and was confirmed by this measurement.

---

2. Decisions (settled — recorded, not re-litigated)

#	Decision	Rationale (one line)	Rejected alternative
D1	Transport = plain WireGuard, box-dials-outbound, PersistentKeepalive=25	Backup is hub-and-spoke to a public endpoint; outbound UDP + keepalive traverses any NAT tier — spike P2 (first-try up, 11.4-min idle hold), P4 (UDP 51820 and 443 pass)	Headscale/DERP for traversal — solves the p2p problem we don't have; a TLS control plane + DB + tailscaled-per-box + third-party DERP dependency for nothing (spike P5). Stays available as a separable future fleet layer.
D2	Placement = customer host, AGENT-MANAGED (wg-felhom systemd service reconciled from hub desired-state)	WG config is ongoing reconciled desired-state — the agent's job; the host-install script is one-shot and cannot react to hub changes (endpoint move, key rotation). Pattern exists: the agent already installs+enables host units via configs/felhom-agent.sudoers:16-18 (install … /etc/systemd/system/*.mount + systemctl enable --now)	(a) install-script-owned static config — can't reconcile; (b) in-guest WG — works technically (spike P7: kernel WG needs no TUN) but violates 01 §3 (guest holds zero infra creds) and dies with the guest; backups originate host-side anyway.
D3	Endpoint = ONE shared public dual-stack cloud VM (Hetzner/EU class; HU optional later) running WG server + the offsite PBS. It is dumb and hub-driven — no agent on it; the hub is the only thing that mutates its WG peer list	Mirrors the proven "hub holds the credential and drives external infra" pattern (hub/internal/cloudflare/unblock.go — hub-held CF token mutating WAF rules). Zero-knowledge client-side encryption (01 §8) makes the VM's location privacy-neutral	An agent on the endpoint — a whole trust tier for a box that only needs a peer list and a datastore.
D4	Hub = source of truth for endpoint coordinates (DNS name, port, server pubkey) + the per-customer peer registry (customer pubkey → assigned tunnel /32), delivered over the existing desired-state channel	The channel is built for exactly this: heartbeat carries DesiredGeneration (internal/hub/report.go:270); the agent fetches the heavy state only on generation advance (internal/desired/syncer.go:44-61); WireDesiredState (report.go:289-296) already carries forward-compat sibling blocks (RestoreDirective, StorageManifest) — a wireguard block joins them	A parallel bespoke config channel.
D5	PBS layout = one datastore, per-customer NAMESPACES, each namespace under its own client-side encryption key (already escrowed — 01 §8 key custody)	PBS-native multi-tenancy; crypto isolation without datastore sprawl; the hub already holds "PBS namespace" in the customer record (01 §6) and WireDesiredState.PBSNamespace already exists on the wire (report.go:294)	Per-customer datastores — operational sprawl with no isolation gain over namespace+key.
D6	Offsite PBS lives ON the cloud VM — NOT relayed through DooPlex	Relaying funnels every customer's backup through the operator's one shared ~40 Mbit home uplink and couples all customer DR to home-connection uptime — non-scaling and a single point of failure. DC uplink is never the bottleneck (§5)	The spike's §4.4(b) rendezvous-relay shape — rejected by the operator for production. DooPlex PBS → dev/demo only.

---

3. Architecture

3.1 Components & trust (extends 01 §3 / §5)

	Customer host	Endpoint VM	Hub
Runs	wg-felhom systemd service (agent-reconciled) + the existing agent	WG server + PBS (one datastore, per-customer namespaces) + a minimal hub-driven peer-sync surface	peer registry + endpoint coords in the customer/host record
Holds	its own WG private key (0600, agent state dir) + PBS token + the pinned PBS fingerprint	WG server private key; the PBS ciphertext (zero-knowledge — 01 §10)	every box's WG public key + assigned /32; the endpoint-mutation credential; escrowed key material (unchanged, 01 §8)
Mutates	only its own WG interface/service	nothing on its own — peer list changes arrive from the hub	the endpoint's peer list; the desired-state the agent consumes
Never	accepts inbound (dials out only)	initiates into a box or the hub; holds plaintext	connects into a box (01 §4 — unchanged)

New trust-boundary rows (extends 01 §5):

Boundary	What crosses	Mechanism	Blast radius if breached
box ↔ endpoint VM	WG-encapsulated, TLS-pinned, client-side-encrypted PBS traffic	outbound UDP; per-peer /32; no inter-peer routing (§4.5)	that box's tunnel; ciphertext only
hub ↔ endpoint VM	peer-list mutations (add/remove pubkey+/32)	hub-held credential, minimal surface (mechanism = slice-1 design point, §7)	the peer list — an attacker can disconnect backups, not read them

3.2 Hub data model (conceptual fields — not SQL, not wire-final)

• Endpoint record (one, for now): dns_name, wg_port, server_pubkey, tunnel_subnet, pbs_tunnel_ip (the endpoint's in-tunnel address the PBS client dials), plus the endpoint-mutation credential (out-of-band secret, never in desired-state).

• Per-host peer entry (joins the customer/host record, 01 §6): wg_pubkey (box-generated, box-registered), assigned_ip (the box's tunnel /32, hub-assigned), status.

• On the wire (S2-final — the shape S3 consumes; golden: hub/internal/api/testdata/desired-state-wireguard.golden.json, agent copy must stay byte-identical): a wireguard key on the served desired-state, merged at READ time — the stored desired_json stays a pure operator blob (the hub REJECTS an admin PUT containing a top-level wireguard key), hosts without a peer get byte-identical pass-through, and old agents drop the unknown key in encoding/json (live-proven on v0.63.0):

  "wireguard": {
    "endpoint": {
      "dns_name": "ep0.felhom.eu",
      "wg_port": 443,
      "server_pubkey": "<44-char base64>",
      "pbs_tunnel_ip": "10.77.0.1"
    },
    "pubkey": "<the box's registered pubkey>",
    "assigned_ip": "10.77.0.2/32"
  }
  

  Client-side AllowedIPs, PersistentKeepalive=25, and MTU 1280 are deliberately NOT wire fields — S3 agent constants derived from pbs_tunnel_ip + §4. Delivery needs zero new plumbing — registration/re-key/unbind bump DesiredGeneration, the agent's existing syncer fetches (syncer.go:44-61). Re-key replaces the pubkey IN PLACE and keeps the /32 (stable addressing); one bound peer per host is index-enforced.

3.3 Provisioning handshake (Day-0 join)

Plugs into the existing Day-0 flow (runbooks/day0-install.md) after agent enrollment (01 §6 — "the agent enrolls first"); offsite stays retrofittable via the same path for boxes installed before this ships (the drilled --rescope-acl posture is unaffected).

1. Box (agent): generates the WG keypair; private key 0600 in the agent state dir, never leaves the box (except escrowed — §3.5).
2. Box → hub: registers wg_pubkey (new box-facing endpoint, slice-2; authenticated by the box's existing hub identity).
3. Hub: assigns the /32 from the tunnel subnet, stores the peer entry, bumps DesiredGeneration.
4. Hub → endpoint VM: adds {pubkey, /32} to the WG server peer list (slice-1 surface).
5. Box (agent): next heartbeat sees the generation advance → fetches desired-state → writes the WG config → install + systemctl enable --now wg-felhom (the sudoers *.mount pattern of felhom-agent.sudoers:16-18, extended to this unit — slice-3).
6. Box: tunnel handshakes (outbound, keepalive); PBS storage is (re)pointed at pbs_tunnel_ip:8007 (§3.4); first backup runs.

Steps 3-4 and 5-6 are order-independent (WG tolerates either side appearing first); the reconcile loop retries until both halves exist.

3.4 PBS over the tunnel — SHIPPED 2026-07-04 (S4, agent v0.65-0.67)

The PBS client config changes only its Server value: the endpoint's tunnel-internal address (pbs_tunnel_ip, i.e. 10.77.0.1) instead of a LAN IP. The TLS exact-cert pin (internal/pbs/pin.go:17-35) sits above WG and is unchanged through the tunnel — spike-proven both ways (P2b: pin validated through the tunnel; wrong pin rejected). Per-customer namespace + per-customer box-born encryption key exactly as today (D5); the endpoint sees ciphertext only (01 §10).

Live-proven end-to-end 2026-07-04 (runbook §4a/§4b): a real vzdump of guest 9201 lands as ciphertext (root.pxar crypt-mode encrypt) in ns demo-felhom-01 on felhom-offsite over the tunnel; the box lists + verifies only its own namespace under a DatastoreBackup token (felhom@pbs!demo-felhom-01), cross-tenant access 403s. One agent addition was required and made (v0.67.0): the PBS client became namespace-aware — Snapshots passes ?ns=, Verify passes ns= — because the ns-unaware datastore-root calls 403 for a per-tenant token; a whole-datastore verify would have needed Datastore.Verify (~admin), breaking isolation. The confirmed minimal tenant ACL (DatastoreBackup on /datastore/<ds>/<ns> to BOTH the user and the token — PBS privsep = intersection) and its gotchas live in runbook §4a.

3.5 Key custody + DR re-establishment

The WG private key joins the escrowed IdentityBundle (internal/escrow/identity.go:24-27, today {TunnelToken, PBSToken} → + WGPrivateKey) — the same R-wrapped, zero-knowledge age blob the DR recipe already consumes. On host-loss re-provisioning (01 §9 restore mode): the box recovers the bundle with the customer's recovery code, restores the WG key, and re-establishes the tunnel; hub-side the peer entry still exists (or is re-added on re-registration — same handshake, §3.3). Revocation = hub deletes the peer at the endpoint: the box is off the offsite network immediately, auditable, no box cooperation needed.

Offsite escrow CREATED — S4 Part 5 DONE (2026-07-04, operator-present; runbook RUNBOOK-offsite-escrow-create). escrow-create -storage felhom-offsite -identity-bundle -directive -upload on felhom-pve minted a fresh R (operator-recorded off-transcript), wrapped the offsite client key K + the auto-injected WG private key + the offsite PBS token under it, and uploaded the opaque blobs to the hub. Independently verified in host_escrow (host demo-felhom-01: key_fingerprint = the offsite key b0:fe:2a…, identity blob 499 B, directive with the non-secret DR coords, created_at = now) — all with zero knowledge of R. [FACT, 2026-10-07 — R-105, 09 §3 decision 169] The directive is RETIRED: it was written only by this by-hand -directive flag (the customer's escrow wizard never sent one, so every wizard escrow stored {} over it), and nothing read it. The flag is gone (the agent release after 2026-10-07) and the hub no longer stores it; the re-enroll routes serve {}. The DR coordinates live in the DR recipe and tenantsync. tunnel_token intentionally empty (the Cloudflare edge lives in the guest/controller, re-provisioned separately in DR). S5 (DR consume) is now UNBLOCKED: R + the hub-stored blobs reconstruct K + the WG key (+ the offsite PBS token). Note: host_escrow is one slot per host (last-write-wins) — the offsite escrow is the one that matters for real DR (D6); the dev felhom-pbs key still lives on the box.

Revocation semantics, agent side (S3-final): the agent registers ONCE (key exists + no local marker). A wireguard block absent from a PRESENT desired-state = revocation → the agent disables the service, KEEPS its marker, and never re-registers on its own — revoked stays revoked until the operator re-adds the peer (the heartbeat report carries the box's pubkey as the recovery handle; re-add via POST /hosts/{id}/wg with the global key — the raw registry add does not bump the host's generation). The only automatic re-registration is the pubkey-mismatch (DR/re-key) path, backoff-bounded. Absent DATA (a failed fetch) is never a teardown signal. Rollout gate: wg_tunnel.enabled defaults FALSE agent-side until the production endpoint exists.

---

3.6 The ep0 datastore has a second copy — decision 70 (2026-10-03)

[DESIGN] A nightly PBS pull-sync copies felhom-offsite from ep0 to DooPlex's PBS over a read-only token. ep0's server snapshot never covered this volume and Hetzner has no volume snapshots (R-342). The copy is ciphertext per customer. [FACT] BUILT 2026-10-03: ep0 token root@pam!dooplex-sync (DatastoreReader only — the one change on ep0); ep0's PBS listens on wg0 only, so DooPlex reaches it through an SSH forward (felhom-ep0-pbs-tunnel.service, operator ruling the same day); DooPlex datastore ep0-copy, sync daily 05:00 with remove-vanished false, verify Saturdays, failures mailed via Resend. First pull 201 s / 12 GB / 4 of 4 snapshots. Restore route: runbooks/ep0-datastore-copy.md. Evidence audits/offsite-lock-build-2026-10-03/partF/.

4. Robustness (production details beyond the spike)

• 4.1 Customer IP change = free, and explicitly NOT a DynDNS dependency. The box dials out; WG roaming re-handshakes from the new source address automatically. Nothing anywhere references the customer's IP — the hub peer registry stores pubkey+/32 only (§3.2). No customer-side DNS, ever.
• 4.2 Endpoint referenced by DNS + a re-resolve watchdog — SHIPPED v0.66.0 (v4-pinned). The endpoint is dns_name, not a bare IP, so it can move. Trap: wg-quick resolves Endpoint= once at service start — a later endpoint re-IP would strand running tunnels. The agent now resolves the A record itself and writes the v4 LITERAL into the conf (internal/wgtunnel Resolver seam, LookupNetIP "ip4" → lowest address, deterministic fleet-wide), and a watchdog (Manager.Watchdog, loop-driven) re-resolves when the handshake age exceeds wg_tunnel.stale_after_seconds (default 180): IP changed → re-render + restart (re-IP recovery); IP same → no churn. Steady state does zero DNS + zero execs (the resolved IP is cached); a DNS failure keeps the last conf (never a teardown). This also closes the dual-stack trap below — pinning the A literal means the tunnel can never silently ride un-NATed IPv6. Endpoint moves are rare, hub-announced, and self-heal on service restart even without the watchdog. Dual-stack resolution trap (2026-07-04 CGNAT smoke test). The endpoint is dual-stack (A + AAAA). On a box with working IPv6, wg-quick resolves and pins the AAAA at bring-up and the tunnel rides un-NATed IPv6 — functionally fine (often better: no NAT), but it means the v4/NAT path is only exercised when v6 is absent. Two consequences: reasoning about "CGNAT traversal" applies to the v4 fallback only; and the re-resolve watchdog must not silently flip families mid-life in a way that strands a half-open v6 mapping. AI_ADDRCONFIG already gives the safe default — a v4-only box never sees the AAAA — but a v6-capable box would prefer it. The client MTU floor (§4.3) is family-agnostic (1280) so MTU correctness never depended on this; and the v4-pin above (v0.66.0) now resolves it outright — the agent writes the A literal, so the tunnel always takes the v4/NAT path regardless of the box's v6 capability. (Was tracked as an open determinism question through S3.1; closed by S4's v4-pin.)
• 4.3 MTU/MSS — DECIDED: fleet-wide client MTU 1280 (agent v0.65.0, S3.1). WG costs 60 bytes on v4 / 80 on IPv6. The unit ships explicit MTU 1280 — the IPv6-minimum link MTU (RFC 8200 guarantees every path carries ≥ 1280), so outer = 1280+60 = 1340 (v4) / 1280+80 = 1360 (v6), both fitting every realistic path (mobile ~1400, DS-Lite ~1452, PPPoE 1492, clean 1500). Because only the box's own PBS client uses the tunnel (locally-terminated TCP, no forwarded flows), the low interface MTU bounds both directions — it caps box→PBS directly and advertises MSS = MTU−40 to cap PBS→box — so no MSS clamp is needed and, critically, the endpoint's wg0 is left untouched (its sends are already capped by the box's advertised MSS). Zero live-endpoint risk; one client constant. 1280 is also family-agnostic, so MTU correctness does not depend on whether wg-quick resolves the endpoint's A or AAAA (§4.2). Noted so nobody adds forwarding without an MSS clamp. Why 1280 and not 1420 (2026-07-04 CGNAT smoke test). On a mobile-carrier uplink the outer path MTU measured ~1400 B (DF probe: 1400-byte packet passes, 1414 fails), so the previously shipped 1420 black-holed bulk data — ping and the WG handshake stayed healthy (small packets) while the PBS TLS page returned empty. Field data: MTU 1340 (= 1400 − 60) restored the full PBS page, 1360 still failed; 1280 sits safely below both. This is not a fixed-line-vs-mobile split: sub-1500 fixed-line paths — notably DS-Lite (~1452 AFTR path) and some PPPoE — would also have black-holed at 1420, so one conservative floor covers all customers. It "looked green, lost backups": the handshake and ping succeed at any MTU, so the failure is invisible until bulk TCP (the backup) flows — which is exactly S4. Rejected alternatives. Auto-probe path MTU / per-connection-type policy — a fragile moving part (a probe that can fail, and must re-run on every path change: ISP switch, mobile failover) to optimize throughput, which doesn't matter here (overnight backup, bound by the customer's home uplink, not by ~10% MTU overhead). MSS clamp — unnecessary without forwarded flows. A fixed conservative floor is strictly more robust. The interface MTU lives in the agent-rendered wg-felhom.conf (internal/wgtunnel clientMTU); revisiting it is a deliberate constant change, not an operational knob.
• 4.4 Keepalive. PersistentKeepalive=25 — spike-proven to hold the NAT mapping through an 11.4-minute fully-idle window (P2) at ~150 B/s of overhead traffic; further proven through a live mobile-carrier NAT for a 32-minute fully-idle soak, zero stalls (2026-07-04 CGNAT smoke test, §7).
• 4.5 Isolation. Per-peer /32 AllowedIPs; boxes cannot see each other (spike P6). PBS tenancy on top: namespace + per-customer token + per-customer key (D5). AMENDED 2026-07-05 (TASK H1 — OOB operator access). Forwarding is no longer blanket-off; it is ON but per-pair allow-listed. The endpoint runs net.ipv4.ip_forward=1 (sysctl.d) and a static forward posture: ct established,related accept; per (operator, box) pair ip saddr <operator/32> ip daddr <box/32> accept; and box↔box iifname wg0 oifname wg0 DROP is now an EXPLICIT rule (previously implicit under the absent capability), backed by the base-chain policy drop. These rules live in the endpoint's static nftables (NOT in felhom-peersync, which still manages only the peer list). Net effect: the operator peer reaches a box's felhom-sshd; boxes still cannot reach each other or the operator (only conntrack replies flow). The box side adds a second layer independent of the endpoint: a dedicated felhom-sshd on a claimed non-22 port, gated by the host-local inet felhom_oob belt (reachable only from the operator /32 over wg-felhom; the customer's stock sshd on :22 is never touched). Live-proven: operator→box SSH works; a dummy tunnel peer is dropped box↔box (counter); the operator /32 is rendered into the box's wg-felhom AllowedIPs so it survives the agent's self-heal ([OF-1]); PBS unaffected.
• 4.6 Tunnel health → hub. The tunnel is a storage dependency, so it reports like one: no handshake within ~3 keepalive periods → the offsite target reports unreachable → the existing alerting path carries it. No new alarm channel. EXTENDED 2026-07-05 (TASK H1). The agent's heartbeat now also carries an oob stanza (felhom_sshd_active, felhom_sshd_port, reachable, config_invalid, operator_peer_configured, operator_key_configured, wg_handshake_age_s) — the operator's "can I get into this box right now, and if not, why" signal. It reaches the hub over HTTPS even when felhom-sshd or the tunnel is DOWN (channel independence). The hub raises a transition-based oob_degraded/oob_recovered warning (felhom-sshd down while the operator peer is configured, OR config invalid).

---

5. The endpoint VM (the one new managed-infra piece)

Runs three things, nothing else: the WG server (peer list = hub-written), PBS (one datastore, per-customer namespaces), and the minimal peer-sync surface the hub drives. How the hub mutates the peer list — a tiny authenticated push API on the VM vs the VM pulling a signed peer manifest from the hub — is deliberately left as the slice-1 design point (§7); the trust shape is fixed either way (hub holds the credential; the VM trusts only the hub; mirroring hub/internal/cloudflare/unblock.go). Firewall posture per §4.5. Dual-stack (spike P1: keep v6 available even though the pilot line had none). No agent, no controller, no customer plaintext.

Bandwidth reality (why D6 is safe): the operator line measured 1 Gbit-class down / ~40 Mbit up (spike P3) — a customer backup is bound by that customer's own uplink, and a restore by their downlink; a DC-grade endpoint uplink is never the bottleneck, and customers never contend with the operator's home line (the rejected relay's flaw).

Spike-earned deploy note for slice-1: a PBS datastore under /root fails (backup user cannot traverse /root) — create datastores under a world-traversable parent (/srv).

---

6. Trust-model conformance (01 §3/§4 preserved)

• The box only ever dials out (UDP to the endpoint; the same posture as hub-poll and PBS-push, 01 §4/§7). No inbound listener is added to any customer box; the hub still never initiates into a box.
• The endpoint VM cannot reach into boxes either: it terminates tunnels whose per-peer routes (§4.5) admit only box→PBS traffic; it holds no box credentials.
• Out of scope, explicitly: (a) operator-inbound access to boxes — a separate future problem with its own trust decision; nothing here creates or presumes it; (b) game-server ingress — public players are not tunnel peers; that is a different ingress problem (public reachability), not a backup-transport one.

---

7. Open validation + open sub-decisions (honest ledger)

Open validation — CLOSED (2026-07-04, mobile-hotspot smoke test). A phone-tethered felhom-pve (agent v0.64.0) carried the live tunnel out through a mobile-carrier NAT and held the mapping across a 32-minute fully-idle soak on PersistentKeepalive=25 alone: 64 samples, max handshake age 125 s (normal ~120 s rekey cadence), zero stalls (no sample > 180 s), rx counter growing on keepalive traffic only, and a final in-tunnel ping after 32 min idle succeeding with no manual bounce. PBS login page reachable over the tunnel once MTU was corrected (§4.3). Two honest caveats, recorded not papered over: (1) this SIM was handed a public mobile IPv4 (84.225.188.131, Magyar Telekom), so the run exercised the phone's tether NAT + keepalive survival — a real single-NAT traversal — but did not reproduce true CGNAT 100.64/10 double-NAT; that stays a retest-on-a-CGNAT-SIM-when-available follow-up (low risk: mapping-hold is NAT-tier-agnostic by mechanism). (2) a dual-stack mobile uplink made wg-quick prefer the endpoint AAAA and ride un-NATed IPv6 until v4 was forced — functionally fine, but see §4.2. The deferred second-ISP vantage (Peti VM 110) is gone — Peti's box was RETIRED 2026-09-25 — so a second-ISP confirmation needs another venue; it gates nothing. Runbook: RUNBOOK-s3-cgnat-smoke.

Open sub-decisions (deferred by design):

• Endpoint peer-sync mechanism (push-API vs signed-manifest pull) — slice-1 design point (§5).
• WG key rotation cadence/mechanics — the reconcile channel can carry it (new pubkey → hub → endpoint swap); design when fleet size warrants, possibly the moment the "separable fleet layer" (D1) conversation reopens.
• Endpoint scaling/sharding (N customers per VM, second region, HU-hosted option) — capacity planning, not architecture; the peer registry already keys per-host.
• Backup/DR of the endpoint VM itself (datastore redundancy, re-provision runbook) — slice-1 must at least write the re-provision runbook; PBS-side redundancy is a later economics call.

---

8. Implementation slice roadmap (each = a separate future task)

Slice	Scope	Done =
S1 — endpoint provisioning + peer-sync surface — SHIPPED 2026-07-04 (dev endpoint; hub v0.32.1). Peer-sync design point RESOLVED: hub pushes over SSH (x/crypto/ssh, FixedHostKey pin + pinned-type HostKeyAlgorithms) to a forced-command reconcile script; runbook documentation/runbooks/offsite-endpoint.md. Done-criterion met live: hub add/remove → wg show on the endpoint; tunnel ep0.felhom.eu:443 → PBS through the wg0-only 8007 rule.	provision the production VM… (see runbook; production endpoint = a later re-run)	hub can add/remove a peer on the live endpoint via its credential; runbook re-creates the VM from nothing ✓
S2 — hub schema + desired-state — SHIPPED 2026-07-04 (hub v0.33.0). POST /hosts/{id}/wg (self-scoped) + merge-at-read wireguard block (§3.2 shape = the golden) + one-per-host index + re-key-in-place + /offsite UI page. Live-proven against the real felhom-pve record incl. old-agent (v0.63.0) tolerance.	endpoint coords + peer registry…	a registered box's desired-state carries its assignment ✓; hub UI shows the peer registry ✓
S3 — agent WG service (the agent-managed-unit pilot) — SHIPPED 2026-07-04 (agent v0.64.0; wg_tunnel.enabled DEFAULT FALSE until the production endpoint exists). internal/wgtunnel: keygen → marker-gated registration → wg-quick@wg-felhom reconciled from the served wireguard block (restart-not-reload; hash-gated zero-exec steady state); revocation completed §3.5: block absent from a PRESENT desired-state → disable + marker kept + never re-register (operator re-adds via the reported pubkey); absent DATA never tears down. FELHOM_WG sudoers (latest-handshakes is the ONLY wg read — dump leaks the private key); IdentityBundle.WGPrivateKey (pre-S3 blobs can't be retrofitted — S5 falls back to fresh-key re-register, /32 kept); report stanza = §4.6's tunnel-health input. Endpoint re-resolution: wg-quick resolves at unit (re)start; the manager restarts on conf change and the operator path covers re-IP (full §4.2 watchdog → S6 with tunnel-health alerting). Live on felhom-pve: up in 3 s, reboot-persistent, revocation drill, 30-min soak.	keygen + registration…	fresh box joins by §3.3 end-to-end ✓; tunnel survives agent restart + host reboot ✓; unhealthy-tunnel ALERTING → S6 (stanza shipped)
S4 — PBS over the tunnel — SHIPPED 2026-07-04 (agent v0.65-0.67; runbook §4a/§4b). Endpoint per-customer tenancy (ns + privsep DatastoreBackup token, dual-grant user+token on /datastore/<ds>/<ns>, cross-tenant 403); box felhom-offsite PVE storage Server=10.77.0.1, box-born encryption key, token-secret WARN fix, local_backup_target retarget; agent v0.66.0 v4-pin + re-resolve watchdog + FELHOM_WG Critical, v0.67.0 namespace-aware PBS client.	point the box's PBS storage at pbs_tunnel_ip:8007	real vzdump of 9201 lands as ciphertext in ns over the tunnel ✓; ns-scoped verify=ok under the box's own token ✓; WARN gone ✓; restore decrypts with the box-born key ✓. Follow-ups: restore-test scheduler WAN deadline + scratch VM.Allocate RESOLVED S4.1 (agent v0.68.0) — tier-aware restore-task timeout (pbs→120m, local→10m); the offsite restore-test now runs unattended on the agent-token path (pass:true, teardown clean). ⚠️ That claim SILENTLY STOPPED BEING TRUE and was restored 2026-07-27 (R-85, agent v0.104.0) — for the second time in this doc, by the same mechanism: a correct fix whose INPUT changed underneath it. S4.1 was verified while local_backup_target was felhom-pbs, so "the scheduler's one tier" and "the offsite tier" were the same thing. The R-82 target split retargeted it to "local", and from then on buildRestoreTestScheduler built its only runner on the LOCAL tier: the offsite tier's archives were never candidates and it was never restore-tested again, unattended or otherwise, while this note still said it was. R-85 rotates across every configured tier (oldest-proven first) and makes a failure audible. The lesson is the same one the timeout taught: a mechanism keyed on "the configured target" stops holding the moment there is more than one target. The VM.Allocate item was a phantom — the earlier 403 was the 10m-timeout consequence (guest not yet in /pool/felhom); no ACL change. Remaining: Tier-1-local-vs-Tier-2-offsite target split SHIPPED 2026-07-26 (R-82) — agent v0.97.0 gave BackupConfig per-target tiers (backup_targets[], each with its OWN cadence + retention), controller v0.174.0 reconciles them into ONE quiesce window, hub v0.76.0 judges each tier against its own threshold, host-install 1.20.0 defaults a fresh box to local-daily + offsite-weekly. Live: local 24h + felhom-pbs 168h on both demo boxes.

⚠️ S4.1's tier-aware timeout silently REGRESSED and was restored 2026-07-26 (agent v0.100.0) — the mechanism was never removed, its INPUT changed. S4.1 derived the tier with storageTier(BackupTarget()), which was correct while local_backup_target was felhom-pbs. The target split above retargeted it to "local", so a felhom-pbs: archive was thereafter classified local and got the 10-minute bound again — reproducing the S4.1 symptom exactly (wait expires mid-restore → teardown fires at a not-yet-pool-associated guest → the same 403 missing privilege VM.Allocate, and the same phantom reading of it). v0.100.0 derives the tier from the archive's own storage (restoreTierForArchive), which is input-independent. The lesson is not "add a timeout" — it was already there — but that a fix keyed on "the configured target" stops holding the moment more than one target exists. | | S5 — DR re-establishment — SAFE HALVES SHIPPED 2026-07-04 (agent v0.69.0); destructive drill operator-gated. Part 1: wgtunnel.InstallRecoveredKey (escrow-recovered WG key → same identity/pubkey//32, no keygen; create-only; --selftest=identity-consume -install-wg-key). Part 2: internal/dr consumes the host_loss restore_directive into an inspectable RestorePlan (per-guest restore coords+sizing, per-drive durable_id→mount) via the AddConsumer seam — derive-and-surface, execute nothing. Re-attach wrong-disk safety = ResolveStorageDevice (match resolves, mismatch REFUSED — unit-proven). | consume WGPrivateKey in restore mode | the tunnel re-establishes from the recovered key (same /32) ✓ (unit); the host_loss directive yields the correct PLAN ✓ (unit). Operator-gated: the destructive in-place 9201 restore (destroy→restore-from-offsite→re-attach-by-durable_id) is operator-present + STOP-gated — the operator runs the R-consume steps (CC never does). Pre-flight green (9201 offsite-restorable, S4.1). | | S6 — monitoring + isolation hardening | endpoint-side alerting, per-peer firewall audit; CGNAT smoke-test closure DONE 2026-07-04 (§7) — mobile-carrier NAT traversal + 32-min keepalive hold proven; §4.3 MTU decision RESOLVED (client MTU 1280, agent v0.65.0, §4.3); residual = true-CGNAT-SIM retest (low risk) | tunnel-down and peer-anomaly both alert; isolation verified adversarially |

MVP that puts offsite backup in a real customer's hands = S1 → S2 → S3; then S4. S5/S6 complete the story. The cloudflared service (internal/hub/cloudflared.go:9-12 — today a read-only prober that names itself "the seam for the tunnel-management slice") adopts the S3-piloted agent-managed-service pattern in a later, separate slice.