From a7f954dde7c5b24e5e0450a894ab8b64c4b55611 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sat, 4 Jul 2026 14:02:48 +0200 Subject: [PATCH] =?UTF-8?q?docs(offsite):=20resolve=20=C2=A74.3=20MTU=20op?= =?UTF-8?q?en=20decision=20=E2=86=92=20client=20MTU=201280=20(S3.1)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The CGNAT smoke test's OPEN DECISION is made: fleet-wide, permanent, family-agnostic client MTU 1280 (RFC 8200 IPv6-minimum floor), implemented in felhom-agent v0.65.0. - §4.3: OPEN DECISION → made decision + rationale + rejected alternatives (auto-probe/per-type = fragility for a non-metric; MSS-clamp = unneeded). "ships MTU 1420" → 1280; corrected "fixed-line unaffected" (DS-Lite ~1452, PPPoE would also black-hole at 1420 — one floor covers all). - §4.2: 1280 is family-agnostic so MTU correctness is decoupled from v4/v6 resolution; v4-pin noted as a separate optional determinism item. - §3.2 wire-field note + §8 S6 row: MTU 1420 → 1280; S6 residual now only the true-CGNAT-SIM retest. - REPORT.md overwritten. Docs-only; the constant change is in felhom-agent v0.65.0. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 --- REPORT.md | 49 +++++++---------- .../architecture/06-offsite-connectivity.md | 52 +++++++++++++------ 2 files changed, 55 insertions(+), 46 deletions(-) diff --git a/REPORT.md b/REPORT.md index a28ea98..11246b2 100644 --- a/REPORT.md +++ b/REPORT.md @@ -4,36 +4,27 @@ --- -# REPORT — S3 appendix: CGNAT smoke test (operational validation, docs-only in this repo) (2026-07-04) +# REPORT — S3.1 doc close-out: offsite-tunnel MTU decision resolved (docs-only) (2026-07-04) -Ran the deferred S3 appendix: the one open transport caveat in `06-offsite-connectivity.md` §7 — -does plain-WireGuard (box dials out + `PersistentKeepalive=25`) establish and **hold** a mapping -through a mobile-carrier NAT? Executed live on **felhom-pve** (`demo-felhom`, agent v0.64.0) with a -USB-tethered Xiaomi Mi 11i. No code touched; the only host changes were runtime `ip`/`dhclient` -one-shots, all reverted to the Phase-0 baseline at the end. +Docs-only companion to the S3.1 implementation (the constant change lives in **felhom-agent +v0.65.0** — see that repo's REPORT). Resolves the OPEN DECISION the 2026-07-04 CGNAT smoke test +left in `documentation/architecture/06-offsite-connectivity.md` §4.3. -**Method (surgical, SSH-safe):** kept the box on its wired default; steered **only** ep0's `/32` -out the phone so just the tunnel's outer UDP traversed the carrier NAT. Neutralized the phone's -RA-provided IPv6 + pinned ep0→v4 so the tunnel took the **v4/NAT** path (not un-NATed v6). 32-min -idle soak sampling handshake age every 30 s. Full restore verified against the captured baseline. +**Decision recorded:** fleet-wide client **MTU 1280** (RFC 8200 IPv6-minimum floor; outer 1340 v4 / +1360 v6 fits every realistic path), permanent + family-agnostic, client-only (endpoint `wg0` +untouched — interface MTU caps box→PBS, advertised MSS caps PBS→box). -**Findings:** -- **Uplink:** phone got a **public mobile IPv4** (`84.225.188.131`, Magyar Telekom) — a real - single-NAT traversal (phone tether NAT), **not** true CGNAT `100.64/10`. Recorded as such. -- **Handshake/PBS:** established over the mobile NAT path (endpoint v4 `167.233.158.164:443`); - in-tunnel ping 4/4 ~57 ms; PBS login page reachable **after** the MTU fix. -- **MTU (actionable):** outer path MTU ~**1400 B** → shipped `wg-felhom` MTU 1420 **black-holed - bulk data** (ping/handshake healthy, PBS page empty). MTU **1340** (= 1400 − 60) restored it; - 1360 failed. Recorded in §4.3 with an **OPEN DECISION**: permanent fleet-wide lower MTU vs - connection-type-dependent (1420 fixed-line / ≤1340 mobile, or MSS clamp). -- **Soak:** 64 samples, max handshake age 125 s (normal rekey), **zero stalls**, final ping after - 32 min idle with no bounce → **mapping held on keepalive alone.** -- **v6 trap:** a dual-stack mobile uplink made `wg-quick` prefer the endpoint AAAA and ride - un-NATed IPv6 until v4 was forced (recorded in §4.2). +**Edits to `06-offsite-connectivity.md`:** +- **§4.3** — replaced "OPEN DECISION (not yet made)" with the made decision + rationale + the + rejected alternatives (auto-probe/per-connection-type = fragility for a non-metric; MSS-clamp = + unneeded without forwarding). Updated "the unit ships explicit MTU 1420" → 1280. +- **§4.3** — corrected the over-general "fixed-line customers are unaffected": sub-1500 fixed-line + paths (DS-Lite ~1452, some PPPoE) would also have black-holed at 1420; one 1280 floor covers all. +- **§4.2** — noted the 1280 floor is chosen family-agnostic *specifically so* MTU correctness does + not depend on the v4/v6 resolution outcome; the deterministic v4-pin stays a separate, optional + future determinism note (not built here). +- **§3.2 / §8 S6 row** — stale "MTU 1420" wire-field note → 1280; S6 residual list drops the + now-resolved MTU decision, keeping only the low-risk true-CGNAT-SIM retest. -**Doc edits (this commit):** `06-offsite-connectivity.md` — §7 open-validation **CLOSED**; -§4.3 MTU mobile finding + open decision; §4.4 keepalive 32-min mobile hold; §4.2 dual-stack v6 -trap; §8 S6 row updated. Runbook: `RUNBOOK-s3-cgnat-smoke`. - -**Still open:** true-CGNAT-SIM retest (deferred, low risk — "when able"); the §4.3 MTU -permanent-vs-per-connection decision; S4 PBS-over-tunnel; S5 DR consume; S6 tunnel-health alerting. +**Not changed:** no code, no endpoint, no hub/controller, no JSON golden (MTU is never on the wire). +One docs-only commit. diff --git a/documentation/architecture/06-offsite-connectivity.md b/documentation/architecture/06-offsite-connectivity.md index 4c69a27..049f85f 100644 --- a/documentation/architecture/06-offsite-connectivity.md +++ b/documentation/architecture/06-offsite-connectivity.md @@ -83,7 +83,7 @@ New trust-boundary rows (extends 01 §5): } ``` - Client-side `AllowedIPs`, `PersistentKeepalive=25`, and `MTU 1420` are deliberately NOT wire + Client-side `AllowedIPs`, `PersistentKeepalive=25`, and `MTU 1280` are deliberately NOT wire fields — S3 agent constants derived from `pbs_tunnel_ip` + §4. Delivery needs **zero new plumbing** — registration/re-key/unbind bump `DesiredGeneration`, the agent's existing syncer fetches (`syncer.go:44-61`). Re-key replaces the pubkey IN PLACE and keeps the `/32` (stable @@ -161,21 +161,39 @@ production endpoint exists. v4/NAT path is only exercised when v6 is absent. Two consequences: reasoning about "CGNAT traversal" applies to the v4 fallback only; and the re-resolve watchdog must not silently flip families mid-life in a way that strands a half-open v6 mapping. `AI_ADDRCONFIG` already gives the - safe default — a v4-only box never sees the AAAA — but a v6-capable box will prefer it. -- **4.3 MTU/MSS.** WG costs 60 bytes on v4 (interface MTU 1420 — what the spike ran end-to-end - under PBS TLS with zero fragmentation stalls, P3). The unit ships explicit `MTU 1420`; since - only the box's own PBS client uses the tunnel (no forwarded flows), no MSS clamp is needed — - noted here so nobody adds forwarding without one. - **Mobile-path finding (2026-07-04 CGNAT smoke test).** On a mobile-carrier uplink the outer - path MTU measured **~1400 B** (DF probe: 1400-byte packet passes, 1414 fails), so the shipped - 1420 **black-holed bulk data**: ping and the WG handshake stayed healthy (small packets) while - the PBS TLS page returned **empty**. Lowering `wg-felhom` to **MTU 1340** (= 1400 − 60, WG's v4 - overhead) restored the full PBS page; 1360 still failed. Fixed-line customers are unaffected (the - spike's wired line carried 1420 clean). **OPEN DECISION (not yet made):** a lower MTU - **permanent fleet-wide** (simplest; costs a little throughput on good links) vs - **connection-type-dependent** (1420 fixed-line / ≤ 1340 mobile-CGNAT, or an MSS clamp on the - tunnel). If it becomes agent-driven (auto-probe path MTU, or per-box policy), that's a separate - TASK — the interface MTU is set in the agent-rendered `wg-felhom.conf`. + safe default — a v4-only box never sees the AAAA — but a v6-capable box will prefer it. The + client MTU floor (§4.3) is deliberately chosen **family-agnostic (1280)** *so that* MTU + correctness does not depend on this resolution outcome. A **deterministic v4-pin** — the agent + resolving ep0's A record and writing the literal IP into the conf (a natural extension of this + watchdog) — remains an **open, separate determinism question**; it is **not required for MTU + correctness** and is tracked on its own, not built by S3.1. +- **4.3 MTU/MSS — DECIDED: fleet-wide client MTU 1280 (agent v0.65.0, S3.1).** WG costs 60 bytes + on v4 / 80 on IPv6. The unit ships explicit **`MTU 1280`** — the **IPv6-minimum link MTU** (RFC + 8200 guarantees every path carries ≥ 1280), so outer = 1280+60 = 1340 (v4) / 1280+80 = 1360 (v6), + both fitting every realistic path (mobile ~1400, DS-Lite ~1452, PPPoE 1492, clean 1500). Because + only the box's own PBS client uses the tunnel (locally-terminated TCP, no forwarded flows), the + low **interface MTU bounds both directions** — it caps box→PBS directly and advertises MSS = + MTU−40 to cap PBS→box — so **no MSS clamp is needed** and, critically, **the endpoint's `wg0` is + left untouched** (its sends are already capped by the box's advertised MSS). Zero live-endpoint + risk; one client constant. `1280` is also **family-agnostic**, so MTU correctness does **not** + depend on whether `wg-quick` resolves the endpoint's A or AAAA (§4.2). Noted so nobody adds + forwarding without an MSS clamp. + **Why 1280 and not 1420 (2026-07-04 CGNAT smoke test).** On a mobile-carrier uplink the outer + path MTU measured **~1400 B** (DF probe: 1400-byte packet passes, 1414 fails), so the previously + shipped **1420 black-holed bulk data** — ping and the WG handshake stayed healthy (small packets) + while the PBS TLS page returned **empty**. Field data: `MTU 1340` (= 1400 − 60) restored the full + PBS page, `1360` still failed; `1280` sits safely below both. This is **not a fixed-line-vs-mobile + split**: sub-1500 fixed-line paths — notably **DS-Lite (~1452 AFTR path)** and some PPPoE — would + also have black-holed at 1420, so one conservative floor covers all customers. It "looked green, + lost backups": the handshake and ping succeed at any MTU, so the failure is invisible until bulk + TCP (the backup) flows — which is exactly S4. + **Rejected alternatives.** *Auto-probe path MTU / per-connection-type policy* — a fragile moving + part (a probe that can fail, and must re-run on every path change: ISP switch, mobile failover) to + optimize **throughput**, which doesn't matter here (overnight backup, bound by the customer's home + uplink, not by ~10% MTU overhead). *MSS clamp* — unnecessary without forwarded flows. A fixed + conservative floor is strictly more robust. The interface MTU lives in the agent-rendered + `wg-felhom.conf` (`internal/wgtunnel` `clientMTU`); revisiting it is a deliberate constant change, + not an operational knob. - **4.4 Keepalive.** `PersistentKeepalive=25` — spike-proven to hold the NAT mapping through an 11.4-minute fully-idle window (P2) at ~150 B/s of overhead traffic; **further proven through a live mobile-carrier NAT for a 32-minute fully-idle soak, zero stalls** (2026-07-04 CGNAT smoke @@ -265,7 +283,7 @@ vantage (Peti VM 110) remains the thorough confirmation but no longer gates anyt | **S3 — agent WG service (the agent-managed-unit pilot)** — **SHIPPED 2026-07-04** (agent v0.64.0; `wg_tunnel.enabled` DEFAULT FALSE until the production endpoint exists). `internal/wgtunnel`: keygen → marker-gated registration → `wg-quick@wg-felhom` reconciled from the served `wireguard` block (restart-not-reload; hash-gated zero-exec steady state); **revocation completed §3.5**: block absent from a PRESENT desired-state → disable + marker kept + never re-register (operator re-adds via the reported pubkey); absent DATA never tears down. `FELHOM_WG` sudoers (latest-handshakes is the ONLY wg read — `dump` leaks the private key); `IdentityBundle.WGPrivateKey` (pre-S3 blobs can't be retrofitted — S5 falls back to fresh-key re-register, /32 kept); report stanza = §4.6's tunnel-health input. Endpoint re-resolution: wg-quick resolves at unit (re)start; the manager restarts on conf change and the operator path covers re-IP (full §4.2 watchdog → S6 with tunnel-health alerting). Live on felhom-pve: up in 3 s, reboot-persistent, revocation drill, 30-min soak. | ~~keygen + registration…~~ | fresh box joins by §3.3 end-to-end ✓; tunnel survives agent restart + host reboot ✓; unhealthy-tunnel ALERTING → S6 (stanza shipped) | | **S4 — PBS over the tunnel** | point the box's PBS storage at `pbs_tunnel_ip:8007`; verify pin-through-tunnel | real backup lands in the box's namespace over the tunnel (nearly free — spike-proven path) | | **S5 — DR re-establishment** | consume `WGPrivateKey` in restore mode; re-peer on re-provision | host-loss drill restores offsite connectivity with only the recovery code. *Merges with the DR-completeness arc.* | -| **S6 — monitoring + isolation hardening** | endpoint-side alerting, per-peer firewall audit; ~~CGNAT smoke-test closure~~ **DONE 2026-07-04 (§7)** — mobile-carrier NAT traversal + 32-min keepalive hold proven; residual = true-CGNAT-SIM retest (low risk) + the §4.3 MTU decision | tunnel-down and peer-anomaly both alert; isolation verified adversarially | +| **S6 — monitoring + isolation hardening** | endpoint-side alerting, per-peer firewall audit; ~~CGNAT smoke-test closure~~ **DONE 2026-07-04 (§7)** — mobile-carrier NAT traversal + 32-min keepalive hold proven; ~~§4.3 MTU decision~~ **RESOLVED (client MTU 1280, agent v0.65.0, §4.3)**; residual = true-CGNAT-SIM retest (low risk) | tunnel-down and peer-anomaly both alert; isolation verified adversarially | **MVP that puts offsite backup in a real customer's hands = S1 → S2 → S3; then S4.** S5/S6 complete the story. The cloudflared service (`internal/hub/cloudflared.go:9-12` — today a