felhom.eu/documentation/architecture/06-offsite-connectivity.md

# Architecture Part 6 — Offsite Connectivity (the backup transport)

> Status: **design-of-record** (2026-07-03). Records the settled offsite-backup-transport
> decisions; grounded against felhom.eu @ `bf099f6` and felhom-agent @ `4ba1b14` (v0.63.0).
> Evidence base: `documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md` (all
> measurements cited below as "spike Pn"). This document ends in an implementation slice roadmap
> (§8); **no slice is implemented yet** — each is a separate future task. Consistent with and
> extending Part 01 (`01-topology-and-trust.md`); where this doc names a component's duty it uses
> Part 01 §3 vocabulary.

---

## 1. Why (the gap)

Offsite backup — the paid DR substrate (01 §8) — is structurally unreachable for a real customer
today. Backup is an **outbound push to PBS** (01 §7: "Outbound only for control/report/backup"),
but the only PBS lives at a **LAN address** — `internal/pbs/client.go:25-26` documents the
`Server` field as literally `"192.168.0.180"`, port 8007. The operator's public edge is
Cloudflare-Tunnel → `nginx-internal` (`felhom.eu/manifests/hub.yaml:233`,
`ingressClassName: nginx-internal`) — HTTP(S) only; DooPlex has **no public IP and no public UDP**.
So offsite backup currently works only because the demo box shares a LAN with the PBS host; a
customer box cannot reach it at all. The DR crypto/identity side is done and drilled
(`SPIKE-dr-recipe-2026-06-16.md`); **this transport is the last missing piece**, and the spike
proved the cheapest transport suffices.

---

## 2. Decisions (settled — recorded, not re-litigated)

| # | Decision | Rationale (one line) | Rejected alternative |
|---|----------|----------------------|----------------------|
| D1 | **Transport = plain WireGuard**, box-dials-outbound, `PersistentKeepalive=25` | Backup is hub-and-spoke to a *public* endpoint; outbound UDP + keepalive traverses any NAT tier — spike P2 (first-try up, 11.4-min idle hold), P4 (UDP 51820 *and* 443 pass) | Headscale/DERP **for traversal** — solves the p2p problem we don't have; a TLS control plane + DB + tailscaled-per-box + third-party DERP dependency for nothing (spike P5). Stays available as a *separable future fleet layer*. |
| D2 | **Placement = customer host, AGENT-MANAGED** (`wg-felhom` systemd service reconciled from hub desired-state) | WG config is *ongoing reconciled desired-state* — the agent's job; the host-install script is one-shot and cannot react to hub changes (endpoint move, key rotation). Pattern exists: the agent already installs+enables host units via `configs/felhom-agent.sudoers:16-18` (`install … /etc/systemd/system/*.mount` + `systemctl enable --now`) | (a) install-script-owned static config — can't reconcile; (b) in-guest WG — *works* technically (spike P7: kernel WG needs no TUN) but violates 01 §3 (guest holds zero infra creds) and dies with the guest; backups originate host-side anyway. |
| D3 | **Endpoint = ONE shared public dual-stack cloud VM** (Hetzner/EU class; HU optional later) running **WG server + the offsite PBS**. It is **dumb and hub-driven** — no agent on it; the hub is the only thing that mutates its WG peer list | Mirrors the proven "hub holds the credential and drives external infra" pattern (`hub/internal/cloudflare/unblock.go` — hub-held CF token mutating WAF rules). Zero-knowledge client-side encryption (01 §8) makes the VM's location privacy-neutral | An agent on the endpoint — a whole trust tier for a box that only needs a peer list and a datastore. |
| D4 | **Hub = source of truth** for endpoint coordinates (DNS name, port, server pubkey) + the per-customer peer registry (customer pubkey → assigned tunnel `/32`), delivered over the **existing desired-state channel** | The channel is built for exactly this: heartbeat carries `DesiredGeneration` (`internal/hub/report.go:270`); the agent fetches the heavy state only on generation advance (`internal/desired/syncer.go:44-61`); `WireDesiredState` (`report.go:289-296`) already carries forward-compat sibling blocks (`RestoreDirective`, `StorageManifest`) — a `wireguard` block joins them | A parallel bespoke config channel. |
| D5 | **PBS layout = one datastore, per-customer NAMESPACES**, each namespace under its own client-side encryption key (already escrowed — 01 §8 key custody) | PBS-native multi-tenancy; crypto isolation without datastore sprawl; the hub already holds "PBS namespace" in the customer record (01 §6) and `WireDesiredState.PBSNamespace` already exists on the wire (`report.go:294`) | Per-customer datastores — operational sprawl with no isolation gain over namespace+key. |
| D6 | **Offsite PBS lives ON the cloud VM — NOT relayed through DooPlex** | Relaying funnels *every* customer's backup through the operator's one shared ~40 Mbit home uplink and couples all customer DR to home-connection uptime — non-scaling and a single point of failure. DC uplink is never the bottleneck (§5) | The spike's §4.4(b) rendezvous-relay shape — rejected by the operator for production. **DooPlex PBS → dev/demo only.** |

---

## 3. Architecture

### 3.1 Components & trust (extends 01 §3 / §5)

| | Customer host | Endpoint VM | Hub |
|---|---|---|---|
| Runs | `wg-felhom` systemd service (agent-reconciled) + the existing agent | WG server + PBS (one datastore, per-customer namespaces) + a minimal hub-driven peer-sync surface | peer registry + endpoint coords in the customer/host record |
| Holds | its own WG **private key** (0600, agent state dir) + PBS token + the pinned PBS fingerprint | WG server private key; the PBS ciphertext (zero-knowledge — 01 §10) | every box's WG **public** key + assigned `/32`; the endpoint-mutation credential; escrowed key material (unchanged, 01 §8) |
| Mutates | only its own WG interface/service | nothing on its own — peer list changes arrive from the hub | the endpoint's peer list; the desired-state the agent consumes |
| Never | accepts inbound (dials out only) | initiates into a box or the hub; holds plaintext | connects into a box (01 §4 — unchanged) |

New trust-boundary rows (extends 01 §5):

| Boundary | What crosses | Mechanism | Blast radius if breached |
|---|---|---|---|
| box ↔ endpoint VM | WG-encapsulated, TLS-pinned, client-side-encrypted PBS traffic | outbound UDP; per-peer `/32`; no inter-peer routing (§4.5) | that box's tunnel; ciphertext only |
| hub ↔ endpoint VM | peer-list mutations (add/remove pubkey+`/32`) | hub-held credential, minimal surface (mechanism = slice-1 design point, §7) | the peer list — an attacker can *disconnect* backups, not read them |

### 3.2 Hub data model (conceptual fields — not SQL, not wire-final)

- **Endpoint record** (one, for now): `dns_name`, `wg_port`, `server_pubkey`, `tunnel_subnet`,
  `pbs_tunnel_ip` (the endpoint's in-tunnel address the PBS client dials), plus the
  endpoint-mutation credential (out-of-band secret, never in desired-state).
- **Per-host peer entry** (joins the customer/host record, 01 §6): `wg_pubkey` (box-generated,
  box-registered), `assigned_ip` (the box's tunnel `/32`, hub-assigned), `status`.
- **On the wire (S2-final — the shape S3 consumes; golden:
  `hub/internal/api/testdata/desired-state-wireguard.golden.json`, agent copy must stay
  byte-identical):** a `wireguard` key on the served desired-state, **merged at READ time** —
  the stored `desired_json` stays a pure operator blob (the hub REJECTS an admin PUT containing
  a top-level `wireguard` key), hosts without a peer get byte-identical pass-through, and old
  agents drop the unknown key in `encoding/json` (live-proven on v0.63.0):

  ```json
  "wireguard": {
    "endpoint": {
      "dns_name": "ep0.felhom.eu",
      "wg_port": 443,
      "server_pubkey": "<44-char base64>",
      "pbs_tunnel_ip": "10.77.0.1"
    },
    "pubkey": "<the box's registered pubkey>",
    "assigned_ip": "10.77.0.2/32"
  }
  ```

  Client-side `AllowedIPs`, `PersistentKeepalive=25`, and `MTU 1280` are deliberately NOT wire
  fields — S3 agent constants derived from `pbs_tunnel_ip` + §4. Delivery needs **zero new
  plumbing** — registration/re-key/unbind bump `DesiredGeneration`, the agent's existing syncer
  fetches (`syncer.go:44-61`). Re-key replaces the pubkey IN PLACE and keeps the `/32` (stable
  addressing); one bound peer per host is index-enforced.

### 3.3 Provisioning handshake (Day-0 join)

Plugs into the existing Day-0 flow (`runbooks/day0-install.md`) after agent enrollment (01 §6 —
"the agent enrolls first"); offsite stays retrofittable via the same path for boxes installed
before this ships (the drilled `--rescope-acl` posture is unaffected).

1. **Box (agent):** generates the WG keypair; private key 0600 in the agent state dir, never
   leaves the box (except escrowed — §3.5).
2. **Box → hub:** registers `wg_pubkey` (new box-facing endpoint, slice-2; authenticated by the
   box's existing hub identity).
3. **Hub:** assigns the `/32` from the tunnel subnet, stores the peer entry, bumps
   `DesiredGeneration`.
4. **Hub → endpoint VM:** adds `{pubkey, /32}` to the WG server peer list (slice-1 surface).
5. **Box (agent):** next heartbeat sees the generation advance → fetches desired-state → writes
   the WG config → `install` + `systemctl enable --now wg-felhom` (the sudoers `*.mount` pattern
   of `felhom-agent.sudoers:16-18`, extended to this unit — slice-3).
6. **Box:** tunnel handshakes (outbound, keepalive); PBS storage is (re)pointed at
   `pbs_tunnel_ip:8007` (§3.4); first backup runs.

Steps 3-4 and 5-6 are order-independent (WG tolerates either side appearing first); the reconcile
loop retries until both halves exist.

### 3.4 PBS over the tunnel — **SHIPPED 2026-07-04 (S4, agent v0.65-0.67)**

The PBS client config changes **only its `Server` value**: the endpoint's tunnel-internal address
(`pbs_tunnel_ip`, i.e. `10.77.0.1`) instead of a LAN IP. The TLS exact-cert pin
(`internal/pbs/pin.go:17-35`) sits **above** WG and is unchanged through the tunnel —
spike-proven both ways (P2b: pin validated through the tunnel; wrong pin rejected). Per-customer
namespace + per-customer box-born encryption key exactly as today (D5); the endpoint sees
ciphertext only (01 §10).

**Live-proven end-to-end 2026-07-04** (runbook §4a/§4b): a real vzdump of guest 9201 lands as
ciphertext (`root.pxar` crypt-mode `encrypt`) in ns `demo-felhom-01` on `felhom-offsite` over the
tunnel; the box lists + verifies **only its own namespace** under a `DatastoreBackup` token
(`felhom@pbs!demo-felhom-01`), cross-tenant access 403s. **One agent addition was required and made
(v0.67.0):** the PBS client became **namespace-aware** — `Snapshots` passes `?ns=`, `Verify` passes
`ns=` — because the ns-unaware datastore-root calls 403 for a per-tenant token; a whole-datastore
verify would have needed `Datastore.Verify` (~admin), breaking isolation. The confirmed minimal
tenant ACL (`DatastoreBackup` on `/datastore/<ds>/<ns>` to BOTH the user and the token — PBS privsep
= intersection) and its gotchas live in runbook §4a.

### 3.5 Key custody + DR re-establishment

The WG private key joins the **escrowed IdentityBundle**
(`internal/escrow/identity.go:24-27`, today `{TunnelToken, PBSToken}` → `+ WGPrivateKey`) — the
same R-wrapped, zero-knowledge age blob the DR recipe already consumes. On host-loss
re-provisioning (01 §9 restore mode): the box recovers the bundle with the customer's recovery
code, restores the WG key, and re-establishes the tunnel; hub-side the peer entry still exists
(or is re-added on re-registration — same handshake, §3.3). Revocation = hub deletes the peer at
the endpoint: the box is off the offsite network immediately, auditable, no box cooperation
needed.

**Offsite escrow CREATED — S4 Part 5 DONE (2026-07-04, operator-present; runbook
`RUNBOOK-offsite-escrow-create`).** `escrow-create -storage felhom-offsite -identity-bundle
-directive -upload` on felhom-pve minted a fresh R (operator-recorded off-transcript), wrapped the
offsite client key K + the auto-injected WG private key + the offsite PBS token under it, and
uploaded the opaque blobs to the hub. Independently verified in `host_escrow` (host `demo-felhom-01`:
`key_fingerprint` = the offsite key `b0:fe:2a…`, identity blob 499 B, directive with the non-secret
DR coords, `created_at` = now) — all with **zero knowledge of R**. `tunnel_token` intentionally empty
(the Cloudflare edge lives in the guest/controller, re-provisioned separately in DR). **S5 (DR
consume) is now UNBLOCKED:** R + the hub-stored blobs reconstruct K + the WG key (+ the offsite PBS
token). Note: `host_escrow` is one slot per host (last-write-wins) — the offsite escrow is the one
that matters for real DR (D6); the dev `felhom-pbs` key still lives on the box.

**Revocation semantics, agent side (S3-final):** the agent registers ONCE (key exists + no local
marker). A `wireguard` block absent from a PRESENT desired-state = revocation → the agent
disables the service, KEEPS its marker, and never re-registers on its own — revoked stays
revoked until the operator re-adds the peer (the heartbeat report carries the box's pubkey as
the recovery handle; **re-add via `POST /hosts/{id}/wg` with the global key** — the raw registry
add does not bump the host's generation). The only automatic re-registration is the
pubkey-mismatch (DR/re-key) path, backoff-bounded. Absent DATA (a failed fetch) is never a
teardown signal. **Rollout gate:** `wg_tunnel.enabled` defaults FALSE agent-side until the
production endpoint exists.

---

## 4. Robustness (production details beyond the spike)

- **4.1 Customer IP change = free, and explicitly NOT a DynDNS dependency.** The box dials out;
  WG roaming re-handshakes from the new source address automatically. Nothing anywhere references
  the customer's IP — the hub peer registry stores pubkey+`/32` only (§3.2). No customer-side DNS,
  ever.
- **4.2 Endpoint referenced by DNS + a re-resolve watchdog — SHIPPED v0.66.0 (v4-pinned).** The
  endpoint is `dns_name`, not a bare IP, so it can move. Trap: `wg-quick` resolves `Endpoint=`
  **once** at service start — a later endpoint re-IP would strand running tunnels. The agent now
  **resolves the A record itself and writes the v4 LITERAL into the conf** (`internal/wgtunnel`
  `Resolver` seam, `LookupNetIP "ip4"` → lowest address, deterministic fleet-wide), and a
  **watchdog** (`Manager.Watchdog`, loop-driven) re-resolves when the handshake age exceeds
  `wg_tunnel.stale_after_seconds` (default 180): IP changed → re-render + restart (re-IP recovery);
  IP same → no churn. Steady state does **zero DNS + zero execs** (the resolved IP is cached); a
  DNS failure keeps the last conf (never a teardown). This also **closes the dual-stack trap
  below** — pinning the A literal means the tunnel can never silently ride un-NATed IPv6. Endpoint
  moves are rare, hub-announced, and self-heal on service restart even without the watchdog.
  **Dual-stack resolution trap (2026-07-04 CGNAT smoke test).** The endpoint is dual-stack (A +
  AAAA). On a box with working IPv6, `wg-quick` resolves and pins the **AAAA** at bring-up and the
  tunnel rides **un-NATed IPv6** — functionally fine (often better: no NAT), but it means the
  v4/NAT path is only exercised when v6 is absent. Two consequences: reasoning about "CGNAT
  traversal" applies to the v4 fallback only; and the re-resolve watchdog must not silently flip
  families mid-life in a way that strands a half-open v6 mapping. `AI_ADDRCONFIG` already gives the
  safe default — a v4-only box never sees the AAAA — but a v6-capable box would prefer it. The
  client MTU floor (§4.3) is family-agnostic (1280) so MTU correctness never depended on this;
  and the **v4-pin above (v0.66.0) now resolves it outright** — the agent writes the A literal, so
  the tunnel always takes the v4/NAT path regardless of the box's v6 capability. (Was tracked as an
  open determinism question through S3.1; closed by S4's v4-pin.)
- **4.3 MTU/MSS — DECIDED: fleet-wide client MTU 1280 (agent v0.65.0, S3.1).** WG costs 60 bytes
  on v4 / 80 on IPv6. The unit ships explicit **`MTU 1280`** — the **IPv6-minimum link MTU** (RFC
  8200 guarantees every path carries ≥ 1280), so outer = 1280+60 = 1340 (v4) / 1280+80 = 1360 (v6),
  both fitting every realistic path (mobile ~1400, DS-Lite ~1452, PPPoE 1492, clean 1500). Because
  only the box's own PBS client uses the tunnel (locally-terminated TCP, no forwarded flows), the
  low **interface MTU bounds both directions** — it caps box→PBS directly and advertises MSS =
  MTU−40 to cap PBS→box — so **no MSS clamp is needed** and, critically, **the endpoint's `wg0` is
  left untouched** (its sends are already capped by the box's advertised MSS). Zero live-endpoint
  risk; one client constant. `1280` is also **family-agnostic**, so MTU correctness does **not**
  depend on whether `wg-quick` resolves the endpoint's A or AAAA (§4.2). Noted so nobody adds
  forwarding without an MSS clamp.
  **Why 1280 and not 1420 (2026-07-04 CGNAT smoke test).** On a mobile-carrier uplink the outer
  path MTU measured **~1400 B** (DF probe: 1400-byte packet passes, 1414 fails), so the previously
  shipped **1420 black-holed bulk data** — ping and the WG handshake stayed healthy (small packets)
  while the PBS TLS page returned **empty**. Field data: `MTU 1340` (= 1400 − 60) restored the full
  PBS page, `1360` still failed; `1280` sits safely below both. This is **not a fixed-line-vs-mobile
  split**: sub-1500 fixed-line paths — notably **DS-Lite (~1452 AFTR path)** and some PPPoE — would
  also have black-holed at 1420, so one conservative floor covers all customers. It "looked green,
  lost backups": the handshake and ping succeed at any MTU, so the failure is invisible until bulk
  TCP (the backup) flows — which is exactly S4.
  **Rejected alternatives.** *Auto-probe path MTU / per-connection-type policy* — a fragile moving
  part (a probe that can fail, and must re-run on every path change: ISP switch, mobile failover) to
  optimize **throughput**, which doesn't matter here (overnight backup, bound by the customer's home
  uplink, not by ~10% MTU overhead). *MSS clamp* — unnecessary without forwarded flows. A fixed
  conservative floor is strictly more robust. The interface MTU lives in the agent-rendered
  `wg-felhom.conf` (`internal/wgtunnel` `clientMTU`); revisiting it is a deliberate constant change,
  not an operational knob.
- **4.4 Keepalive.** `PersistentKeepalive=25` — spike-proven to hold the NAT mapping through an
  11.4-minute fully-idle window (P2) at ~150 B/s of overhead traffic; **further proven through a
  live mobile-carrier NAT for a 32-minute fully-idle soak, zero stalls** (2026-07-04 CGNAT smoke
  test, §7).
- **4.5 Isolation.** Per-peer `/32` `AllowedIPs`; boxes cannot see each other (spike P6). PBS tenancy
  on top: namespace + per-customer token + per-customer key (D5).
  **AMENDED 2026-07-05 (TASK H1 — OOB operator access).** Forwarding is no longer blanket-**off**; it
  is **ON but per-pair allow-listed**. The endpoint runs `net.ipv4.ip_forward=1` (sysctl.d) and a
  static forward posture: `ct established,related accept`; **per (operator, box) pair** `ip saddr
  <operator/32> ip daddr <box/32> accept`; and **box↔box `iifname wg0 oifname wg0` DROP is now an
  EXPLICIT rule** (previously implicit under the absent capability), backed by the base-chain `policy
  drop`. These rules live in the endpoint's static nftables (NOT in `felhom-peersync`, which still
  manages only the peer *list*). Net effect: the operator peer reaches a box's `felhom-sshd`; boxes
  still cannot reach each other or the operator (only conntrack replies flow). The box side adds a
  second layer independent of the endpoint: a dedicated `felhom-sshd` on a claimed non-22 port, gated
  by the host-local `inet felhom_oob` belt (reachable only from the operator `/32` over `wg-felhom`;
  the customer's stock sshd on :22 is never touched). Live-proven: operator→box SSH works; a dummy
  tunnel peer is dropped box↔box (counter); the operator `/32` is **rendered** into the box's
  `wg-felhom` AllowedIPs so it survives the agent's self-heal ([OF-1]); PBS unaffected.
- **4.6 Tunnel health → hub.** The tunnel is a storage dependency, so it reports like one: no handshake
  within ~3 keepalive periods → the offsite target reports unreachable → the existing alerting path
  carries it. No new alarm channel.
  **EXTENDED 2026-07-05 (TASK H1).** The agent's heartbeat now also carries an **`oob` stanza**
  (`felhom_sshd_active`, `felhom_sshd_port`, `reachable`, `config_invalid`, `operator_peer_configured`,
  `operator_key_configured`, `wg_handshake_age_s`) — the operator's "can I get into this box right
  now, and if not, why" signal. It reaches the hub over HTTPS even when `felhom-sshd` or the tunnel is
  DOWN (channel independence). The hub raises a transition-based **`oob_degraded`/`oob_recovered`**
  warning (felhom-sshd down while the operator peer is configured, OR config invalid).

---

## 5. The endpoint VM (the one new managed-infra piece)

Runs three things, nothing else: the WG server (peer list = hub-written), PBS (one datastore,
per-customer namespaces), and the minimal peer-sync surface the hub drives. **How** the hub
mutates the peer list — a tiny authenticated push API on the VM vs the VM pulling a signed peer
manifest from the hub — is deliberately **left as the slice-1 design point** (§7); the trust
shape is fixed either way (hub holds the credential; the VM trusts only the hub; mirroring
`hub/internal/cloudflare/unblock.go`). Firewall posture per §4.5. Dual-stack (spike P1: keep v6
available even though the pilot line had none). No agent, no controller, no customer plaintext.

Bandwidth reality (why D6 is safe): the operator line measured 1 Gbit-class down / ~40 Mbit up
(spike P3) — a customer backup is bound by *that customer's own uplink*, and a restore by their
downlink; a DC-grade endpoint uplink is never the bottleneck, and customers never contend with
the operator's home line (the rejected relay's flaw).

Spike-earned deploy note for slice-1: a PBS datastore under `/root` fails (`backup` user cannot
traverse `/root`) — create datastores under a world-traversable parent (`/srv`).

---

## 6. Trust-model conformance (01 §3/§4 preserved)

- The box **only ever dials out** (UDP to the endpoint; the same posture as hub-poll and
  PBS-push, 01 §4/§7). No inbound listener is added to any customer box; the hub still never
  initiates into a box.
- The endpoint VM cannot reach into boxes either: it terminates tunnels whose per-peer routes
  (§4.5) admit only box→PBS traffic; it holds no box credentials.
- **Out of scope, explicitly:** (a) operator-inbound access to boxes — a *separate* future
  problem with its own trust decision; nothing here creates or presumes it; (b) game-server
  ingress — public players are not tunnel peers; that is a different ingress problem (public
  reachability), not a backup-transport one.

---

## 7. Open validation + open sub-decisions (honest ledger)

**Open validation — CLOSED (2026-07-04, mobile-hotspot smoke test).** A phone-tethered
felhom-pve (agent v0.64.0) carried the live tunnel out through a mobile-carrier NAT and **held the
mapping across a 32-minute fully-idle soak** on `PersistentKeepalive=25` alone: 64 samples, max
handshake age 125 s (normal ~120 s rekey cadence), **zero stalls** (no sample > 180 s), rx counter
growing on keepalive traffic only, and a final in-tunnel ping after 32 min idle succeeding with **no
manual bounce**. PBS login page reachable over the tunnel once MTU was corrected (§4.3). Two honest
caveats, recorded not papered over: **(1)** this SIM was handed a **public mobile IPv4**
(`84.225.188.131`, Magyar Telekom), so the run exercised the phone's tether NAT + keepalive survival
— a real single-NAT traversal — but did **not** reproduce true CGNAT `100.64/10` double-NAT; that
stays a *retest-on-a-CGNAT-SIM-when-available* follow-up (low risk: mapping-hold is NAT-tier-agnostic
by mechanism). **(2)** a dual-stack mobile uplink made `wg-quick` prefer the endpoint **AAAA and ride
un-NATed IPv6** until v4 was forced — functionally fine, but see §4.2. The deferred second-ISP
vantage (Peti VM 110) remains the thorough confirmation but no longer gates anything. Runbook:
`RUNBOOK-s3-cgnat-smoke`.

**Open sub-decisions (deferred by design):**
- Endpoint peer-sync mechanism (push-API vs signed-manifest pull) — **slice-1 design point** (§5).
- WG key rotation cadence/mechanics — the reconcile channel can carry it (new pubkey → hub →
  endpoint swap); design when fleet size warrants, possibly the moment the "separable fleet
  layer" (D1) conversation reopens.
- Endpoint scaling/sharding (N customers per VM, second region, HU-hosted option) — capacity
  planning, not architecture; the peer registry already keys per-host.
- Backup/DR **of the endpoint VM itself** (datastore redundancy, re-provision runbook) — slice-1
  must at least write the re-provision runbook; PBS-side redundancy is a later economics call.

---

## 8. Implementation slice roadmap (each = a separate future task)

| Slice | Scope | Done = |
|---|---|---|
| **S1 — endpoint provisioning + peer-sync surface** — **SHIPPED 2026-07-04** (dev endpoint; hub v0.32.1). Peer-sync design point RESOLVED: **hub pushes over SSH** (`x/crypto/ssh`, `FixedHostKey` pin + pinned-type `HostKeyAlgorithms`) to a forced-command reconcile script; runbook `documentation/runbooks/offsite-endpoint.md`. Done-criterion met live: hub add/remove → `wg show` on the endpoint; tunnel `ep0.felhom.eu:443` → PBS through the wg0-only 8007 rule. | ~~provision the production VM…~~ (see runbook; production endpoint = a later re-run) | hub can add/remove a peer on the live endpoint via its credential; runbook re-creates the VM from nothing ✓ |
| **S2 — hub schema + desired-state** — **SHIPPED 2026-07-04** (hub v0.33.0). `POST /hosts/{id}/wg` (self-scoped) + merge-at-read `wireguard` block (§3.2 shape = the golden) + one-per-host index + re-key-in-place + `/offsite` UI page. Live-proven against the real felhom-pve record incl. old-agent (v0.63.0) tolerance. | ~~endpoint coords + peer registry…~~ | a registered box's desired-state carries its assignment ✓; hub UI shows the peer registry ✓ |
| **S3 — agent WG service (the agent-managed-unit pilot)** — **SHIPPED 2026-07-04** (agent v0.64.0; `wg_tunnel.enabled` DEFAULT FALSE until the production endpoint exists). `internal/wgtunnel`: keygen → marker-gated registration → `wg-quick@wg-felhom` reconciled from the served `wireguard` block (restart-not-reload; hash-gated zero-exec steady state); **revocation completed §3.5**: block absent from a PRESENT desired-state → disable + marker kept + never re-register (operator re-adds via the reported pubkey); absent DATA never tears down. `FELHOM_WG` sudoers (latest-handshakes is the ONLY wg read — `dump` leaks the private key); `IdentityBundle.WGPrivateKey` (pre-S3 blobs can't be retrofitted — S5 falls back to fresh-key re-register, /32 kept); report stanza = §4.6's tunnel-health input. Endpoint re-resolution: wg-quick resolves at unit (re)start; the manager restarts on conf change and the operator path covers re-IP (full §4.2 watchdog → S6 with tunnel-health alerting). Live on felhom-pve: up in 3 s, reboot-persistent, revocation drill, 30-min soak. | ~~keygen + registration…~~ | fresh box joins by §3.3 end-to-end ✓; tunnel survives agent restart + host reboot ✓; unhealthy-tunnel ALERTING → S6 (stanza shipped) |
| **S4 — PBS over the tunnel — SHIPPED 2026-07-04** (agent v0.65-0.67; runbook §4a/§4b). Endpoint per-customer tenancy (ns + privsep `DatastoreBackup` token, dual-grant user+token on `/datastore/<ds>/<ns>`, cross-tenant 403); box `felhom-offsite` PVE storage `Server=10.77.0.1`, box-born encryption key, token-secret WARN fix, `local_backup_target` retarget; agent **v0.66.0** v4-pin + re-resolve watchdog + FELHOM_WG Critical, **v0.67.0** namespace-aware PBS client. | ~~point the box's PBS storage at `pbs_tunnel_ip:8007`~~ | real vzdump of 9201 lands as ciphertext in ns over the tunnel ✓; ns-scoped verify=ok under the box's own token ✓; WARN gone ✓; restore decrypts with the box-born key ✓. Follow-ups: ~~restore-test scheduler WAN deadline + scratch `VM.Allocate`~~ **RESOLVED S4.1 (agent v0.68.0)** — tier-aware restore-task timeout (pbs→120m, local→10m); the offsite restore-test now runs unattended on the agent-token path (`pass:true`, teardown clean). **⚠️ That claim SILENTLY STOPPED BEING TRUE and was restored 2026-07-27 (R-85, agent v0.104.0) — for the second time in this doc, by the same mechanism: a correct fix whose INPUT changed underneath it.** S4.1 was verified while `local_backup_target` **was** `felhom-pbs`, so "the scheduler's one tier" and "the offsite tier" were the same thing. The R-82 target split retargeted it to `"local"`, and from then on `buildRestoreTestScheduler` built its only runner on the LOCAL tier: the offsite tier's archives were never candidates and it was **never restore-tested again**, unattended or otherwise, while this note still said it was. R-85 rotates across every configured tier (oldest-proven first) and makes a failure audible. The lesson is the same one the timeout taught: **a mechanism keyed on "the configured target" stops holding the moment there is more than one target.** The `VM.Allocate` item was a **phantom** — the earlier 403 was the 10m-timeout consequence (guest not yet in `/pool/felhom`); no ACL change. ~~Remaining: Tier-1-local-vs-Tier-2-offsite target split~~ **SHIPPED 2026-07-26 (R-82)** — agent v0.97.0 gave `BackupConfig` per-target tiers (`backup_targets[]`, each with its OWN cadence + retention), controller v0.174.0 reconciles them into ONE quiesce window, hub v0.76.0 judges each tier against its own threshold, host-install 1.20.0 defaults a fresh box to local-daily + offsite-weekly. Live: local 24h + `felhom-pbs` 168h on both demo boxes.

> ⚠️ **S4.1's tier-aware timeout silently REGRESSED and was restored 2026-07-26 (agent v0.100.0) — the mechanism was never removed, its INPUT changed.** S4.1 derived the tier with `storageTier(BackupTarget())`, which was correct while `local_backup_target` **was** `felhom-pbs`. The target split above retargeted it to `"local"`, so a `felhom-pbs:` archive was thereafter classified **local** and got the 10-minute bound again — reproducing the S4.1 symptom exactly (wait expires mid-restore → teardown fires at a not-yet-pool-associated guest → the same `403 missing privilege VM.Allocate`, and the same **phantom** reading of it). v0.100.0 derives the tier from **the archive's own storage** (`restoreTierForArchive`), which is input-independent. The lesson is not "add a timeout" — it was already there — but that a fix keyed on *"the configured target"* stops holding the moment more than one target exists. |
| **S5 — DR re-establishment — SAFE HALVES SHIPPED 2026-07-04 (agent v0.69.0); destructive drill operator-gated.** Part 1: `wgtunnel.InstallRecoveredKey` (escrow-recovered WG key → same identity/pubkey/`/32`, no keygen; create-only; `--selftest=identity-consume -install-wg-key`). Part 2: `internal/dr` consumes the host_loss `restore_directive` into an inspectable RestorePlan (per-guest restore coords+sizing, per-drive durable_id→mount) via the AddConsumer seam — derive-and-surface, execute nothing. Re-attach wrong-disk safety = `ResolveStorageDevice` (match resolves, mismatch REFUSED — unit-proven). | ~~consume `WGPrivateKey` in restore mode~~ | the tunnel re-establishes from the recovered key (same `/32`) ✓ (unit); the host_loss directive yields the correct PLAN ✓ (unit). **Operator-gated:** the destructive in-place 9201 restore (destroy→restore-from-offsite→re-attach-by-durable_id) is operator-present + STOP-gated — the operator runs the R-consume steps (CC never does). Pre-flight green (9201 offsite-restorable, S4.1). |
| **S6 — monitoring + isolation hardening** | endpoint-side alerting, per-peer firewall audit; ~~CGNAT smoke-test closure~~ **DONE 2026-07-04 (§7)** — mobile-carrier NAT traversal + 32-min keepalive hold proven; ~~§4.3 MTU decision~~ **RESOLVED (client MTU 1280, agent v0.65.0, §4.3)**; residual = true-CGNAT-SIM retest (low risk) | tunnel-down and peer-anomaly both alert; isolation verified adversarially |

**MVP that puts offsite backup in a real customer's hands = S1 → S2 → S3; then S4.** S5/S6
complete the story. The cloudflared service (`internal/hub/cloudflared.go:9-12` — today a
read-only prober that names itself "the seam for the tunnel-management slice") adopts the
S3-piloted agent-managed-service pattern in a later, separate slice.