felhom.eu/documentation/architecture/06-offsite-connectivity.md

Architecture Part 6 — Offsite Connectivity (the backup transport)

Status: design-of-record (2026-07-03). Records the settled offsite-backup-transport decisions; grounded against felhom.eu @ bf099f6 and felhom-agent @ 4ba1b14 (v0.63.0). Evidence base: documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md (all measurements cited below as "spike Pn"). This document ends in an implementation slice roadmap (§8); no slice is implemented yet — each is a separate future task. Consistent with and extending Part 01 (01-topology-and-trust.md); where this doc names a component's duty it uses Part 01 §3 vocabulary.

---

1. Why (the gap)

Offsite backup — the paid DR substrate (01 §8) — is structurally unreachable for a real customer today. Backup is an outbound push to PBS (01 §7: "Outbound only for control/report/backup"), but the only PBS lives at a LAN address — internal/pbs/client.go:25-26 documents the Server field as literally "192.168.0.180", port 8007. The operator's public edge is Cloudflare-Tunnel → nginx-internal (felhom.eu/manifests/hub.yaml:233, ingressClassName: nginx-internal) — HTTP(S) only; DooPlex has no public IP and no public UDP. So offsite backup currently works only because the demo box shares a LAN with the PBS host; a customer box cannot reach it at all. The DR crypto/identity side is done and drilled (SPIKE-dr-recipe-2026-06-16.md); this transport is the last missing piece, and the spike proved the cheapest transport suffices.

---

2. Decisions (settled — recorded, not re-litigated)

#	Decision	Rationale (one line)	Rejected alternative
D1	Transport = plain WireGuard, box-dials-outbound, PersistentKeepalive=25	Backup is hub-and-spoke to a public endpoint; outbound UDP + keepalive traverses any NAT tier — spike P2 (first-try up, 11.4-min idle hold), P4 (UDP 51820 and 443 pass)	Headscale/DERP for traversal — solves the p2p problem we don't have; a TLS control plane + DB + tailscaled-per-box + third-party DERP dependency for nothing (spike P5). Stays available as a separable future fleet layer.
D2	Placement = customer host, AGENT-MANAGED (wg-felhom systemd service reconciled from hub desired-state)	WG config is ongoing reconciled desired-state — the agent's job; the host-install script is one-shot and cannot react to hub changes (endpoint move, key rotation). Pattern exists: the agent already installs+enables host units via configs/felhom-agent.sudoers:16-18 (install … /etc/systemd/system/*.mount + systemctl enable --now)	(a) install-script-owned static config — can't reconcile; (b) in-guest WG — works technically (spike P7: kernel WG needs no TUN) but violates 01 §3 (guest holds zero infra creds) and dies with the guest; backups originate host-side anyway.
D3	Endpoint = ONE shared public dual-stack cloud VM (Hetzner/EU class; HU optional later) running WG server + the offsite PBS. It is dumb and hub-driven — no agent on it; the hub is the only thing that mutates its WG peer list	Mirrors the proven "hub holds the credential and drives external infra" pattern (hub/internal/cloudflare/unblock.go — hub-held CF token mutating WAF rules). Zero-knowledge client-side encryption (01 §8) makes the VM's location privacy-neutral	An agent on the endpoint — a whole trust tier for a box that only needs a peer list and a datastore.
D4	Hub = source of truth for endpoint coordinates (DNS name, port, server pubkey) + the per-customer peer registry (customer pubkey → assigned tunnel /32), delivered over the existing desired-state channel	The channel is built for exactly this: heartbeat carries DesiredGeneration (internal/hub/report.go:270); the agent fetches the heavy state only on generation advance (internal/desired/syncer.go:44-61); WireDesiredState (report.go:289-296) already carries forward-compat sibling blocks (RestoreDirective, StorageManifest) — a wireguard block joins them	A parallel bespoke config channel.
D5	PBS layout = one datastore, per-customer NAMESPACES, each namespace under its own client-side encryption key (already escrowed — 01 §8 key custody)	PBS-native multi-tenancy; crypto isolation without datastore sprawl; the hub already holds "PBS namespace" in the customer record (01 §6) and WireDesiredState.PBSNamespace already exists on the wire (report.go:294)	Per-customer datastores — operational sprawl with no isolation gain over namespace+key.
D6	Offsite PBS lives ON the cloud VM — NOT relayed through DooPlex	Relaying funnels every customer's backup through the operator's one shared ~40 Mbit home uplink and couples all customer DR to home-connection uptime — non-scaling and a single point of failure. DC uplink is never the bottleneck (§5)	The spike's §4.4(b) rendezvous-relay shape — rejected by the operator for production. DooPlex PBS → dev/demo only.

---

3. Architecture

3.1 Components & trust (extends 01 §3 / §5)

	Customer host	Endpoint VM	Hub
Runs	wg-felhom systemd service (agent-reconciled) + the existing agent	WG server + PBS (one datastore, per-customer namespaces) + a minimal hub-driven peer-sync surface	peer registry + endpoint coords in the customer/host record
Holds	its own WG private key (0600, agent state dir) + PBS token + the pinned PBS fingerprint	WG server private key; the PBS ciphertext (zero-knowledge — 01 §10)	every box's WG public key + assigned /32; the endpoint-mutation credential; escrowed key material (unchanged, 01 §8)
Mutates	only its own WG interface/service	nothing on its own — peer list changes arrive from the hub	the endpoint's peer list; the desired-state the agent consumes
Never	accepts inbound (dials out only)	initiates into a box or the hub; holds plaintext	connects into a box (01 §4 — unchanged)

New trust-boundary rows (extends 01 §5):

Boundary	What crosses	Mechanism	Blast radius if breached
box ↔ endpoint VM	WG-encapsulated, TLS-pinned, client-side-encrypted PBS traffic	outbound UDP; per-peer /32; no inter-peer routing (§4.5)	that box's tunnel; ciphertext only
hub ↔ endpoint VM	peer-list mutations (add/remove pubkey+/32)	hub-held credential, minimal surface (mechanism = slice-1 design point, §7)	the peer list — an attacker can disconnect backups, not read them

3.2 Hub data model (conceptual fields — not SQL, not wire-final)

• Endpoint record (one, for now): dns_name, wg_port, server_pubkey, tunnel_subnet, pbs_tunnel_ip (the endpoint's in-tunnel address the PBS client dials), plus the endpoint-mutation credential (out-of-band secret, never in desired-state).

• Per-host peer entry (joins the customer/host record, 01 §6): wg_pubkey (box-generated, box-registered), assigned_ip (the box's tunnel /32, hub-assigned), status.

• On the wire (S2-final — the shape S3 consumes; golden: hub/internal/api/testdata/desired-state-wireguard.golden.json, agent copy must stay byte-identical): a wireguard key on the served desired-state, merged at READ time — the stored desired_json stays a pure operator blob (the hub REJECTS an admin PUT containing a top-level wireguard key), hosts without a peer get byte-identical pass-through, and old agents drop the unknown key in encoding/json (live-proven on v0.63.0):

  "wireguard": {
    "endpoint": {
      "dns_name": "ep0.felhom.eu",
      "wg_port": 443,
      "server_pubkey": "<44-char base64>",
      "pbs_tunnel_ip": "10.77.0.1"
    },
    "pubkey": "<the box's registered pubkey>",
    "assigned_ip": "10.77.0.2/32"
  }
  

  Client-side AllowedIPs, PersistentKeepalive=25, and MTU 1280 are deliberately NOT wire fields — S3 agent constants derived from pbs_tunnel_ip + §4. Delivery needs zero new plumbing — registration/re-key/unbind bump DesiredGeneration, the agent's existing syncer fetches (syncer.go:44-61). Re-key replaces the pubkey IN PLACE and keeps the /32 (stable addressing); one bound peer per host is index-enforced.

3.3 Provisioning handshake (Day-0 join)

Plugs into the existing Day-0 flow (runbooks/day0-install.md) after agent enrollment (01 §6 — "the agent enrolls first"); offsite stays retrofittable via the same path for boxes installed before this ships (the drilled --rescope-acl posture is unaffected).

1. Box (agent): generates the WG keypair; private key 0600 in the agent state dir, never leaves the box (except escrowed — §3.5).
2. Box → hub: registers wg_pubkey (new box-facing endpoint, slice-2; authenticated by the box's existing hub identity).
3. Hub: assigns the /32 from the tunnel subnet, stores the peer entry, bumps DesiredGeneration.
4. Hub → endpoint VM: adds {pubkey, /32} to the WG server peer list (slice-1 surface).
5. Box (agent): next heartbeat sees the generation advance → fetches desired-state → writes the WG config → install + systemctl enable --now wg-felhom (the sudoers *.mount pattern of felhom-agent.sudoers:16-18, extended to this unit — slice-3).
6. Box: tunnel handshakes (outbound, keepalive); PBS storage is (re)pointed at pbs_tunnel_ip:8007 (§3.4); first backup runs.

Steps 3-4 and 5-6 are order-independent (WG tolerates either side appearing first); the reconcile loop retries until both halves exist.

3.4 PBS over the tunnel — SHIPPED 2026-07-04 (S4, agent v0.65-0.67)

The PBS client config changes only its Server value: the endpoint's tunnel-internal address (pbs_tunnel_ip, i.e. 10.77.0.1) instead of a LAN IP. The TLS exact-cert pin (internal/pbs/pin.go:17-35) sits above WG and is unchanged through the tunnel — spike-proven both ways (P2b: pin validated through the tunnel; wrong pin rejected). Per-customer namespace + per-customer box-born encryption key exactly as today (D5); the endpoint sees ciphertext only (01 §10).

Live-proven end-to-end 2026-07-04 (runbook §4a/§4b): a real vzdump of guest 9201 lands as ciphertext (root.pxar crypt-mode encrypt) in ns demo-felhom-01 on felhom-offsite over the tunnel; the box lists + verifies only its own namespace under a DatastoreBackup token (felhom@pbs!demo-felhom-01), cross-tenant access 403s. One agent addition was required and made (v0.67.0): the PBS client became namespace-aware — Snapshots passes ?ns=, Verify passes ns= — because the ns-unaware datastore-root calls 403 for a per-tenant token; a whole-datastore verify would have needed Datastore.Verify (~admin), breaking isolation. The confirmed minimal tenant ACL (DatastoreBackup on /datastore/<ds>/<ns> to BOTH the user and the token — PBS privsep = intersection) and its gotchas live in runbook §4a.

3.5 Key custody + DR re-establishment

The WG private key joins the escrowed IdentityBundle (internal/escrow/identity.go:24-27, today {TunnelToken, PBSToken} → + WGPrivateKey) — the same R-wrapped, zero-knowledge age blob the DR recipe already consumes. On host-loss re-provisioning (01 §9 restore mode): the box recovers the bundle with the customer's recovery code, restores the WG key, and re-establishes the tunnel; hub-side the peer entry still exists (or is re-added on re-registration — same handshake, §3.3). Revocation = hub deletes the peer at the endpoint: the box is off the offsite network immediately, auditable, no box cooperation needed.

Offsite escrow CREATED — S4 Part 5 DONE (2026-07-04, operator-present; runbook RUNBOOK-offsite-escrow-create). escrow-create -storage felhom-offsite -identity-bundle -directive -upload on felhom-pve minted a fresh R (operator-recorded off-transcript), wrapped the offsite client key K + the auto-injected WG private key + the offsite PBS token under it, and uploaded the opaque blobs to the hub. Independently verified in host_escrow (host demo-felhom-01: key_fingerprint = the offsite key b0:fe:2a…, identity blob 499 B, directive with the non-secret DR coords, created_at = now) — all with zero knowledge of R. tunnel_token intentionally empty (the Cloudflare edge lives in the guest/controller, re-provisioned separately in DR). S5 (DR consume) is now UNBLOCKED: R + the hub-stored blobs reconstruct K + the WG key (+ the offsite PBS token). Note: host_escrow is one slot per host (last-write-wins) — the offsite escrow is the one that matters for real DR (D6); the dev felhom-pbs key still lives on the box.

Revocation semantics, agent side (S3-final): the agent registers ONCE (key exists + no local marker). A wireguard block absent from a PRESENT desired-state = revocation → the agent disables the service, KEEPS its marker, and never re-registers on its own — revoked stays revoked until the operator re-adds the peer (the heartbeat report carries the box's pubkey as the recovery handle; re-add via POST /hosts/{id}/wg with the global key — the raw registry add does not bump the host's generation). The only automatic re-registration is the pubkey-mismatch (DR/re-key) path, backoff-bounded. Absent DATA (a failed fetch) is never a teardown signal. Rollout gate: wg_tunnel.enabled defaults FALSE agent-side until the production endpoint exists.

---

4. Robustness (production details beyond the spike)

• 4.1 Customer IP change = free, and explicitly NOT a DynDNS dependency. The box dials out; WG roaming re-handshakes from the new source address automatically. Nothing anywhere references the customer's IP — the hub peer registry stores pubkey+/32 only (§3.2). No customer-side DNS, ever.
• 4.2 Endpoint referenced by DNS + a re-resolve watchdog — SHIPPED v0.66.0 (v4-pinned). The endpoint is dns_name, not a bare IP, so it can move. Trap: wg-quick resolves Endpoint= once at service start — a later endpoint re-IP would strand running tunnels. The agent now resolves the A record itself and writes the v4 LITERAL into the conf (internal/wgtunnel Resolver seam, LookupNetIP "ip4" → lowest address, deterministic fleet-wide), and a watchdog (Manager.Watchdog, loop-driven) re-resolves when the handshake age exceeds wg_tunnel.stale_after_seconds (default 180): IP changed → re-render + restart (re-IP recovery); IP same → no churn. Steady state does zero DNS + zero execs (the resolved IP is cached); a DNS failure keeps the last conf (never a teardown). This also closes the dual-stack trap below — pinning the A literal means the tunnel can never silently ride un-NATed IPv6. Endpoint moves are rare, hub-announced, and self-heal on service restart even without the watchdog. Dual-stack resolution trap (2026-07-04 CGNAT smoke test). The endpoint is dual-stack (A + AAAA). On a box with working IPv6, wg-quick resolves and pins the AAAA at bring-up and the tunnel rides un-NATed IPv6 — functionally fine (often better: no NAT), but it means the v4/NAT path is only exercised when v6 is absent. Two consequences: reasoning about "CGNAT traversal" applies to the v4 fallback only; and the re-resolve watchdog must not silently flip families mid-life in a way that strands a half-open v6 mapping. AI_ADDRCONFIG already gives the safe default — a v4-only box never sees the AAAA — but a v6-capable box would prefer it. The client MTU floor (§4.3) is family-agnostic (1280) so MTU correctness never depended on this; and the v4-pin above (v0.66.0) now resolves it outright — the agent writes the A literal, so the tunnel always takes the v4/NAT path regardless of the box's v6 capability. (Was tracked as an open determinism question through S3.1; closed by S4's v4-pin.)
• 4.3 MTU/MSS — DECIDED: fleet-wide client MTU 1280 (agent v0.65.0, S3.1). WG costs 60 bytes on v4 / 80 on IPv6. The unit ships explicit MTU 1280 — the IPv6-minimum link MTU (RFC 8200 guarantees every path carries ≥ 1280), so outer = 1280+60 = 1340 (v4) / 1280+80 = 1360 (v6), both fitting every realistic path (mobile ~1400, DS-Lite ~1452, PPPoE 1492, clean 1500). Because only the box's own PBS client uses the tunnel (locally-terminated TCP, no forwarded flows), the low interface MTU bounds both directions — it caps box→PBS directly and advertises MSS = MTU−40 to cap PBS→box — so no MSS clamp is needed and, critically, the endpoint's wg0 is left untouched (its sends are already capped by the box's advertised MSS). Zero live-endpoint risk; one client constant. 1280 is also family-agnostic, so MTU correctness does not depend on whether wg-quick resolves the endpoint's A or AAAA (§4.2). Noted so nobody adds forwarding without an MSS clamp. Why 1280 and not 1420 (2026-07-04 CGNAT smoke test). On a mobile-carrier uplink the outer path MTU measured ~1400 B (DF probe: 1400-byte packet passes, 1414 fails), so the previously shipped 1420 black-holed bulk data — ping and the WG handshake stayed healthy (small packets) while the PBS TLS page returned empty. Field data: MTU 1340 (= 1400 − 60) restored the full PBS page, 1360 still failed; 1280 sits safely below both. This is not a fixed-line-vs-mobile split: sub-1500 fixed-line paths — notably DS-Lite (~1452 AFTR path) and some PPPoE — would also have black-holed at 1420, so one conservative floor covers all customers. It "looked green, lost backups": the handshake and ping succeed at any MTU, so the failure is invisible until bulk TCP (the backup) flows — which is exactly S4. Rejected alternatives. Auto-probe path MTU / per-connection-type policy — a fragile moving part (a probe that can fail, and must re-run on every path change: ISP switch, mobile failover) to optimize throughput, which doesn't matter here (overnight backup, bound by the customer's home uplink, not by ~10% MTU overhead). MSS clamp — unnecessary without forwarded flows. A fixed conservative floor is strictly more robust. The interface MTU lives in the agent-rendered wg-felhom.conf (internal/wgtunnel clientMTU); revisiting it is a deliberate constant change, not an operational knob.
• 4.4 Keepalive. PersistentKeepalive=25 — spike-proven to hold the NAT mapping through an 11.4-minute fully-idle window (P2) at ~150 B/s of overhead traffic; further proven through a live mobile-carrier NAT for a 32-minute fully-idle soak, zero stalls (2026-07-04 CGNAT smoke test, §7).
• 4.5 Isolation. Per-peer /32 AllowedIPs; boxes cannot see each other (spike P6). PBS tenancy on top: namespace + per-customer token + per-customer key (D5). AMENDED 2026-07-05 (TASK H1 — OOB operator access). Forwarding is no longer blanket-off; it is ON but per-pair allow-listed. The endpoint runs net.ipv4.ip_forward=1 (sysctl.d) and a static forward posture: ct established,related accept; per (operator, box) pair ip saddr <operator/32> ip daddr <box/32> accept; and box↔box iifname wg0 oifname wg0 DROP is now an EXPLICIT rule (previously implicit under the absent capability), backed by the base-chain policy drop. These rules live in the endpoint's static nftables (NOT in felhom-peersync, which still manages only the peer list). Net effect: the operator peer reaches a box's felhom-sshd; boxes still cannot reach each other or the operator (only conntrack replies flow). The box side adds a second layer independent of the endpoint: a dedicated felhom-sshd on a claimed non-22 port, gated by the host-local inet felhom_oob belt (reachable only from the operator /32 over wg-felhom; the customer's stock sshd on :22 is never touched). Live-proven: operator→box SSH works; a dummy tunnel peer is dropped box↔box (counter); the operator /32 is rendered into the box's wg-felhom AllowedIPs so it survives the agent's self-heal ([OF-1]); PBS unaffected.
• 4.6 Tunnel health → hub. The tunnel is a storage dependency, so it reports like one: no handshake within ~3 keepalive periods → the offsite target reports unreachable → the existing alerting path carries it. No new alarm channel. EXTENDED 2026-07-05 (TASK H1). The agent's heartbeat now also carries an oob stanza (felhom_sshd_active, felhom_sshd_port, reachable, config_invalid, operator_peer_configured, operator_key_configured, wg_handshake_age_s) — the operator's "can I get into this box right now, and if not, why" signal. It reaches the hub over HTTPS even when felhom-sshd or the tunnel is DOWN (channel independence). The hub raises a transition-based oob_degraded/oob_recovered warning (felhom-sshd down while the operator peer is configured, OR config invalid).

---

5. The endpoint VM (the one new managed-infra piece)

Runs three things, nothing else: the WG server (peer list = hub-written), PBS (one datastore, per-customer namespaces), and the minimal peer-sync surface the hub drives. How the hub mutates the peer list — a tiny authenticated push API on the VM vs the VM pulling a signed peer manifest from the hub — is deliberately left as the slice-1 design point (§7); the trust shape is fixed either way (hub holds the credential; the VM trusts only the hub; mirroring hub/internal/cloudflare/unblock.go). Firewall posture per §4.5. Dual-stack (spike P1: keep v6 available even though the pilot line had none). No agent, no controller, no customer plaintext.

Bandwidth reality (why D6 is safe): the operator line measured 1 Gbit-class down / ~40 Mbit up (spike P3) — a customer backup is bound by that customer's own uplink, and a restore by their downlink; a DC-grade endpoint uplink is never the bottleneck, and customers never contend with the operator's home line (the rejected relay's flaw).

Spike-earned deploy note for slice-1: a PBS datastore under /root fails (backup user cannot traverse /root) — create datastores under a world-traversable parent (/srv).

---

6. Trust-model conformance (01 §3/§4 preserved)

• The box only ever dials out (UDP to the endpoint; the same posture as hub-poll and PBS-push, 01 §4/§7). No inbound listener is added to any customer box; the hub still never initiates into a box.
• The endpoint VM cannot reach into boxes either: it terminates tunnels whose per-peer routes (§4.5) admit only box→PBS traffic; it holds no box credentials.
• Out of scope, explicitly: (a) operator-inbound access to boxes — a separate future problem with its own trust decision; nothing here creates or presumes it; (b) game-server ingress — public players are not tunnel peers; that is a different ingress problem (public reachability), not a backup-transport one.

---

7. Open validation + open sub-decisions (honest ledger)

Open validation — CLOSED (2026-07-04, mobile-hotspot smoke test). A phone-tethered felhom-pve (agent v0.64.0) carried the live tunnel out through a mobile-carrier NAT and held the mapping across a 32-minute fully-idle soak on PersistentKeepalive=25 alone: 64 samples, max handshake age 125 s (normal ~120 s rekey cadence), zero stalls (no sample > 180 s), rx counter growing on keepalive traffic only, and a final in-tunnel ping after 32 min idle succeeding with no manual bounce. PBS login page reachable over the tunnel once MTU was corrected (§4.3). Two honest caveats, recorded not papered over: (1) this SIM was handed a public mobile IPv4 (84.225.188.131, Magyar Telekom), so the run exercised the phone's tether NAT + keepalive survival — a real single-NAT traversal — but did not reproduce true CGNAT 100.64/10 double-NAT; that stays a retest-on-a-CGNAT-SIM-when-available follow-up (low risk: mapping-hold is NAT-tier-agnostic by mechanism). (2) a dual-stack mobile uplink made wg-quick prefer the endpoint AAAA and ride un-NATed IPv6 until v4 was forced — functionally fine, but see §4.2. The deferred second-ISP vantage (Peti VM 110) remains the thorough confirmation but no longer gates anything. Runbook: RUNBOOK-s3-cgnat-smoke.

Open sub-decisions (deferred by design):

• Endpoint peer-sync mechanism (push-API vs signed-manifest pull) — slice-1 design point (§5).
• WG key rotation cadence/mechanics — the reconcile channel can carry it (new pubkey → hub → endpoint swap); design when fleet size warrants, possibly the moment the "separable fleet layer" (D1) conversation reopens.
• Endpoint scaling/sharding (N customers per VM, second region, HU-hosted option) — capacity planning, not architecture; the peer registry already keys per-host.
• Backup/DR of the endpoint VM itself (datastore redundancy, re-provision runbook) — slice-1 must at least write the re-provision runbook; PBS-side redundancy is a later economics call.

---

8. Implementation slice roadmap (each = a separate future task)

Slice	Scope	Done =
S1 — endpoint provisioning + peer-sync surface — SHIPPED 2026-07-04 (dev endpoint; hub v0.32.1). Peer-sync design point RESOLVED: hub pushes over SSH (x/crypto/ssh, FixedHostKey pin + pinned-type HostKeyAlgorithms) to a forced-command reconcile script; runbook documentation/runbooks/offsite-endpoint.md. Done-criterion met live: hub add/remove → wg show on the endpoint; tunnel ep0.felhom.eu:443 → PBS through the wg0-only 8007 rule.	provision the production VM… (see runbook; production endpoint = a later re-run)	hub can add/remove a peer on the live endpoint via its credential; runbook re-creates the VM from nothing ✓
S2 — hub schema + desired-state — SHIPPED 2026-07-04 (hub v0.33.0). POST /hosts/{id}/wg (self-scoped) + merge-at-read wireguard block (§3.2 shape = the golden) + one-per-host index + re-key-in-place + /offsite UI page. Live-proven against the real felhom-pve record incl. old-agent (v0.63.0) tolerance.	endpoint coords + peer registry…	a registered box's desired-state carries its assignment ✓; hub UI shows the peer registry ✓
S3 — agent WG service (the agent-managed-unit pilot) — SHIPPED 2026-07-04 (agent v0.64.0; wg_tunnel.enabled DEFAULT FALSE until the production endpoint exists). internal/wgtunnel: keygen → marker-gated registration → wg-quick@wg-felhom reconciled from the served wireguard block (restart-not-reload; hash-gated zero-exec steady state); revocation completed §3.5: block absent from a PRESENT desired-state → disable + marker kept + never re-register (operator re-adds via the reported pubkey); absent DATA never tears down. FELHOM_WG sudoers (latest-handshakes is the ONLY wg read — dump leaks the private key); IdentityBundle.WGPrivateKey (pre-S3 blobs can't be retrofitted — S5 falls back to fresh-key re-register, /32 kept); report stanza = §4.6's tunnel-health input. Endpoint re-resolution: wg-quick resolves at unit (re)start; the manager restarts on conf change and the operator path covers re-IP (full §4.2 watchdog → S6 with tunnel-health alerting). Live on felhom-pve: up in 3 s, reboot-persistent, revocation drill, 30-min soak.	keygen + registration…	fresh box joins by §3.3 end-to-end ✓; tunnel survives agent restart + host reboot ✓; unhealthy-tunnel ALERTING → S6 (stanza shipped)
S4 — PBS over the tunnel — SHIPPED 2026-07-04 (agent v0.65-0.67; runbook §4a/§4b). Endpoint per-customer tenancy (ns + privsep DatastoreBackup token, dual-grant user+token on /datastore/<ds>/<ns>, cross-tenant 403); box felhom-offsite PVE storage Server=10.77.0.1, box-born encryption key, token-secret WARN fix, local_backup_target retarget; agent v0.66.0 v4-pin + re-resolve watchdog + FELHOM_WG Critical, v0.67.0 namespace-aware PBS client.	point the box's PBS storage at pbs_tunnel_ip:8007	real vzdump of 9201 lands as ciphertext in ns over the tunnel ✓; ns-scoped verify=ok under the box's own token ✓; WARN gone ✓; restore decrypts with the box-born key ✓. Follow-ups: restore-test scheduler WAN deadline + scratch VM.Allocate RESOLVED S4.1 (agent v0.68.0) — tier-aware restore-task timeout (pbs→120m, local→10m); the offsite restore-test now runs unattended on the agent-token path (pass:true, teardown clean). ⚠️ That claim SILENTLY STOPPED BEING TRUE and was restored 2026-07-27 (R-85, agent v0.104.0) — for the second time in this doc, by the same mechanism: a correct fix whose INPUT changed underneath it. S4.1 was verified while local_backup_target was felhom-pbs, so "the scheduler's one tier" and "the offsite tier" were the same thing. The R-82 target split retargeted it to "local", and from then on buildRestoreTestScheduler built its only runner on the LOCAL tier: the offsite tier's archives were never candidates and it was never restore-tested again, unattended or otherwise, while this note still said it was. R-85 rotates across every configured tier (oldest-proven first) and makes a failure audible. The lesson is the same one the timeout taught: a mechanism keyed on "the configured target" stops holding the moment there is more than one target. The VM.Allocate item was a phantom — the earlier 403 was the 10m-timeout consequence (guest not yet in /pool/felhom); no ACL change. Remaining: Tier-1-local-vs-Tier-2-offsite target split SHIPPED 2026-07-26 (R-82) — agent v0.97.0 gave BackupConfig per-target tiers (backup_targets[], each with its OWN cadence + retention), controller v0.174.0 reconciles them into ONE quiesce window, hub v0.76.0 judges each tier against its own threshold, host-install 1.20.0 defaults a fresh box to local-daily + offsite-weekly. Live: local 24h + felhom-pbs 168h on both demo boxes.

⚠️ S4.1's tier-aware timeout silently REGRESSED and was restored 2026-07-26 (agent v0.100.0) — the mechanism was never removed, its INPUT changed. S4.1 derived the tier with storageTier(BackupTarget()), which was correct while local_backup_target was felhom-pbs. The target split above retargeted it to "local", so a felhom-pbs: archive was thereafter classified local and got the 10-minute bound again — reproducing the S4.1 symptom exactly (wait expires mid-restore → teardown fires at a not-yet-pool-associated guest → the same 403 missing privilege VM.Allocate, and the same phantom reading of it). v0.100.0 derives the tier from the archive's own storage (restoreTierForArchive), which is input-independent. The lesson is not "add a timeout" — it was already there — but that a fix keyed on "the configured target" stops holding the moment more than one target exists. | | S5 — DR re-establishment — SAFE HALVES SHIPPED 2026-07-04 (agent v0.69.0); destructive drill operator-gated. Part 1: wgtunnel.InstallRecoveredKey (escrow-recovered WG key → same identity/pubkey//32, no keygen; create-only; --selftest=identity-consume -install-wg-key). Part 2: internal/dr consumes the host_loss restore_directive into an inspectable RestorePlan (per-guest restore coords+sizing, per-drive durable_id→mount) via the AddConsumer seam — derive-and-surface, execute nothing. Re-attach wrong-disk safety = ResolveStorageDevice (match resolves, mismatch REFUSED — unit-proven). | consume WGPrivateKey in restore mode | the tunnel re-establishes from the recovered key (same /32) ✓ (unit); the host_loss directive yields the correct PLAN ✓ (unit). Operator-gated: the destructive in-place 9201 restore (destroy→restore-from-offsite→re-attach-by-durable_id) is operator-present + STOP-gated — the operator runs the R-consume steps (CC never does). Pre-flight green (9201 offsite-restorable, S4.1). | | S6 — monitoring + isolation hardening | endpoint-side alerting, per-peer firewall audit; CGNAT smoke-test closure DONE 2026-07-04 (§7) — mobile-carrier NAT traversal + 32-min keepalive hold proven; §4.3 MTU decision RESOLVED (client MTU 1280, agent v0.65.0, §4.3); residual = true-CGNAT-SIM retest (low risk) | tunnel-down and peer-anomaly both alert; isolation verified adversarially |

MVP that puts offsite backup in a real customer's hands = S1 → S2 → S3; then S4. S5/S6 complete the story. The cloudflared service (internal/hub/cloudflared.go:9-12 — today a read-only prober that names itself "the seam for the tunnel-management slice") adopts the S3-piloted agent-managed-service pattern in a later, separate slice.