docs(offsite): resolve §4.3 MTU open decision → client MTU 1280 (S3.1)
The CGNAT smoke test's OPEN DECISION is made: fleet-wide, permanent, family-agnostic client MTU 1280 (RFC 8200 IPv6-minimum floor), implemented in felhom-agent v0.65.0. - §4.3: OPEN DECISION → made decision + rationale + rejected alternatives (auto-probe/per-type = fragility for a non-metric; MSS-clamp = unneeded). "ships MTU 1420" → 1280; corrected "fixed-line unaffected" (DS-Lite ~1452, PPPoE would also black-hole at 1420 — one floor covers all). - §4.2: 1280 is family-agnostic so MTU correctness is decoupled from v4/v6 resolution; v4-pin noted as a separate optional determinism item. - §3.2 wire-field note + §8 S6 row: MTU 1420 → 1280; S6 residual now only the true-CGNAT-SIM retest. - REPORT.md overwritten. Docs-only; the constant change is in felhom-agent v0.65.0. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
@@ -4,36 +4,27 @@
|
||||
|
||||
---
|
||||
|
||||
# REPORT — S3 appendix: CGNAT smoke test (operational validation, docs-only in this repo) (2026-07-04)
|
||||
# REPORT — S3.1 doc close-out: offsite-tunnel MTU decision resolved (docs-only) (2026-07-04)
|
||||
|
||||
Ran the deferred S3 appendix: the one open transport caveat in `06-offsite-connectivity.md` §7 —
|
||||
does plain-WireGuard (box dials out + `PersistentKeepalive=25`) establish and **hold** a mapping
|
||||
through a mobile-carrier NAT? Executed live on **felhom-pve** (`demo-felhom`, agent v0.64.0) with a
|
||||
USB-tethered Xiaomi Mi 11i. No code touched; the only host changes were runtime `ip`/`dhclient`
|
||||
one-shots, all reverted to the Phase-0 baseline at the end.
|
||||
Docs-only companion to the S3.1 implementation (the constant change lives in **felhom-agent
|
||||
v0.65.0** — see that repo's REPORT). Resolves the OPEN DECISION the 2026-07-04 CGNAT smoke test
|
||||
left in `documentation/architecture/06-offsite-connectivity.md` §4.3.
|
||||
|
||||
**Method (surgical, SSH-safe):** kept the box on its wired default; steered **only** ep0's `/32`
|
||||
out the phone so just the tunnel's outer UDP traversed the carrier NAT. Neutralized the phone's
|
||||
RA-provided IPv6 + pinned ep0→v4 so the tunnel took the **v4/NAT** path (not un-NATed v6). 32-min
|
||||
idle soak sampling handshake age every 30 s. Full restore verified against the captured baseline.
|
||||
**Decision recorded:** fleet-wide client **MTU 1280** (RFC 8200 IPv6-minimum floor; outer 1340 v4 /
|
||||
1360 v6 fits every realistic path), permanent + family-agnostic, client-only (endpoint `wg0`
|
||||
untouched — interface MTU caps box→PBS, advertised MSS caps PBS→box).
|
||||
|
||||
**Findings:**
|
||||
- **Uplink:** phone got a **public mobile IPv4** (`84.225.188.131`, Magyar Telekom) — a real
|
||||
single-NAT traversal (phone tether NAT), **not** true CGNAT `100.64/10`. Recorded as such.
|
||||
- **Handshake/PBS:** established over the mobile NAT path (endpoint v4 `167.233.158.164:443`);
|
||||
in-tunnel ping 4/4 ~57 ms; PBS login page reachable **after** the MTU fix.
|
||||
- **MTU (actionable):** outer path MTU ~**1400 B** → shipped `wg-felhom` MTU 1420 **black-holed
|
||||
bulk data** (ping/handshake healthy, PBS page empty). MTU **1340** (= 1400 − 60) restored it;
|
||||
1360 failed. Recorded in §4.3 with an **OPEN DECISION**: permanent fleet-wide lower MTU vs
|
||||
connection-type-dependent (1420 fixed-line / ≤1340 mobile, or MSS clamp).
|
||||
- **Soak:** 64 samples, max handshake age 125 s (normal rekey), **zero stalls**, final ping after
|
||||
32 min idle with no bounce → **mapping held on keepalive alone.**
|
||||
- **v6 trap:** a dual-stack mobile uplink made `wg-quick` prefer the endpoint AAAA and ride
|
||||
un-NATed IPv6 until v4 was forced (recorded in §4.2).
|
||||
**Edits to `06-offsite-connectivity.md`:**
|
||||
- **§4.3** — replaced "OPEN DECISION (not yet made)" with the made decision + rationale + the
|
||||
rejected alternatives (auto-probe/per-connection-type = fragility for a non-metric; MSS-clamp =
|
||||
unneeded without forwarding). Updated "the unit ships explicit MTU 1420" → 1280.
|
||||
- **§4.3** — corrected the over-general "fixed-line customers are unaffected": sub-1500 fixed-line
|
||||
paths (DS-Lite ~1452, some PPPoE) would also have black-holed at 1420; one 1280 floor covers all.
|
||||
- **§4.2** — noted the 1280 floor is chosen family-agnostic *specifically so* MTU correctness does
|
||||
not depend on the v4/v6 resolution outcome; the deterministic v4-pin stays a separate, optional
|
||||
future determinism note (not built here).
|
||||
- **§3.2 / §8 S6 row** — stale "MTU 1420" wire-field note → 1280; S6 residual list drops the
|
||||
now-resolved MTU decision, keeping only the low-risk true-CGNAT-SIM retest.
|
||||
|
||||
**Doc edits (this commit):** `06-offsite-connectivity.md` — §7 open-validation **CLOSED**;
|
||||
§4.3 MTU mobile finding + open decision; §4.4 keepalive 32-min mobile hold; §4.2 dual-stack v6
|
||||
trap; §8 S6 row updated. Runbook: `RUNBOOK-s3-cgnat-smoke`.
|
||||
|
||||
**Still open:** true-CGNAT-SIM retest (deferred, low risk — "when able"); the §4.3 MTU
|
||||
permanent-vs-per-connection decision; S4 PBS-over-tunnel; S5 DR consume; S6 tunnel-health alerting.
|
||||
**Not changed:** no code, no endpoint, no hub/controller, no JSON golden (MTU is never on the wire).
|
||||
One docs-only commit.
|
||||
|
||||
@@ -83,7 +83,7 @@ New trust-boundary rows (extends 01 §5):
|
||||
}
|
||||
```
|
||||
|
||||
Client-side `AllowedIPs`, `PersistentKeepalive=25`, and `MTU 1420` are deliberately NOT wire
|
||||
Client-side `AllowedIPs`, `PersistentKeepalive=25`, and `MTU 1280` are deliberately NOT wire
|
||||
fields — S3 agent constants derived from `pbs_tunnel_ip` + §4. Delivery needs **zero new
|
||||
plumbing** — registration/re-key/unbind bump `DesiredGeneration`, the agent's existing syncer
|
||||
fetches (`syncer.go:44-61`). Re-key replaces the pubkey IN PLACE and keeps the `/32` (stable
|
||||
@@ -161,21 +161,39 @@ production endpoint exists.
|
||||
v4/NAT path is only exercised when v6 is absent. Two consequences: reasoning about "CGNAT
|
||||
traversal" applies to the v4 fallback only; and the re-resolve watchdog must not silently flip
|
||||
families mid-life in a way that strands a half-open v6 mapping. `AI_ADDRCONFIG` already gives the
|
||||
safe default — a v4-only box never sees the AAAA — but a v6-capable box will prefer it.
|
||||
- **4.3 MTU/MSS.** WG costs 60 bytes on v4 (interface MTU 1420 — what the spike ran end-to-end
|
||||
under PBS TLS with zero fragmentation stalls, P3). The unit ships explicit `MTU 1420`; since
|
||||
only the box's own PBS client uses the tunnel (no forwarded flows), no MSS clamp is needed —
|
||||
noted here so nobody adds forwarding without one.
|
||||
**Mobile-path finding (2026-07-04 CGNAT smoke test).** On a mobile-carrier uplink the outer
|
||||
path MTU measured **~1400 B** (DF probe: 1400-byte packet passes, 1414 fails), so the shipped
|
||||
1420 **black-holed bulk data**: ping and the WG handshake stayed healthy (small packets) while
|
||||
the PBS TLS page returned **empty**. Lowering `wg-felhom` to **MTU 1340** (= 1400 − 60, WG's v4
|
||||
overhead) restored the full PBS page; 1360 still failed. Fixed-line customers are unaffected (the
|
||||
spike's wired line carried 1420 clean). **OPEN DECISION (not yet made):** a lower MTU
|
||||
**permanent fleet-wide** (simplest; costs a little throughput on good links) vs
|
||||
**connection-type-dependent** (1420 fixed-line / ≤ 1340 mobile-CGNAT, or an MSS clamp on the
|
||||
tunnel). If it becomes agent-driven (auto-probe path MTU, or per-box policy), that's a separate
|
||||
TASK — the interface MTU is set in the agent-rendered `wg-felhom.conf`.
|
||||
safe default — a v4-only box never sees the AAAA — but a v6-capable box will prefer it. The
|
||||
client MTU floor (§4.3) is deliberately chosen **family-agnostic (1280)** *so that* MTU
|
||||
correctness does not depend on this resolution outcome. A **deterministic v4-pin** — the agent
|
||||
resolving ep0's A record and writing the literal IP into the conf (a natural extension of this
|
||||
watchdog) — remains an **open, separate determinism question**; it is **not required for MTU
|
||||
correctness** and is tracked on its own, not built by S3.1.
|
||||
- **4.3 MTU/MSS — DECIDED: fleet-wide client MTU 1280 (agent v0.65.0, S3.1).** WG costs 60 bytes
|
||||
on v4 / 80 on IPv6. The unit ships explicit **`MTU 1280`** — the **IPv6-minimum link MTU** (RFC
|
||||
8200 guarantees every path carries ≥ 1280), so outer = 1280+60 = 1340 (v4) / 1280+80 = 1360 (v6),
|
||||
both fitting every realistic path (mobile ~1400, DS-Lite ~1452, PPPoE 1492, clean 1500). Because
|
||||
only the box's own PBS client uses the tunnel (locally-terminated TCP, no forwarded flows), the
|
||||
low **interface MTU bounds both directions** — it caps box→PBS directly and advertises MSS =
|
||||
MTU−40 to cap PBS→box — so **no MSS clamp is needed** and, critically, **the endpoint's `wg0` is
|
||||
left untouched** (its sends are already capped by the box's advertised MSS). Zero live-endpoint
|
||||
risk; one client constant. `1280` is also **family-agnostic**, so MTU correctness does **not**
|
||||
depend on whether `wg-quick` resolves the endpoint's A or AAAA (§4.2). Noted so nobody adds
|
||||
forwarding without an MSS clamp.
|
||||
**Why 1280 and not 1420 (2026-07-04 CGNAT smoke test).** On a mobile-carrier uplink the outer
|
||||
path MTU measured **~1400 B** (DF probe: 1400-byte packet passes, 1414 fails), so the previously
|
||||
shipped **1420 black-holed bulk data** — ping and the WG handshake stayed healthy (small packets)
|
||||
while the PBS TLS page returned **empty**. Field data: `MTU 1340` (= 1400 − 60) restored the full
|
||||
PBS page, `1360` still failed; `1280` sits safely below both. This is **not a fixed-line-vs-mobile
|
||||
split**: sub-1500 fixed-line paths — notably **DS-Lite (~1452 AFTR path)** and some PPPoE — would
|
||||
also have black-holed at 1420, so one conservative floor covers all customers. It "looked green,
|
||||
lost backups": the handshake and ping succeed at any MTU, so the failure is invisible until bulk
|
||||
TCP (the backup) flows — which is exactly S4.
|
||||
**Rejected alternatives.** *Auto-probe path MTU / per-connection-type policy* — a fragile moving
|
||||
part (a probe that can fail, and must re-run on every path change: ISP switch, mobile failover) to
|
||||
optimize **throughput**, which doesn't matter here (overnight backup, bound by the customer's home
|
||||
uplink, not by ~10% MTU overhead). *MSS clamp* — unnecessary without forwarded flows. A fixed
|
||||
conservative floor is strictly more robust. The interface MTU lives in the agent-rendered
|
||||
`wg-felhom.conf` (`internal/wgtunnel` `clientMTU`); revisiting it is a deliberate constant change,
|
||||
not an operational knob.
|
||||
- **4.4 Keepalive.** `PersistentKeepalive=25` — spike-proven to hold the NAT mapping through an
|
||||
11.4-minute fully-idle window (P2) at ~150 B/s of overhead traffic; **further proven through a
|
||||
live mobile-carrier NAT for a 32-minute fully-idle soak, zero stalls** (2026-07-04 CGNAT smoke
|
||||
@@ -265,7 +283,7 @@ vantage (Peti VM 110) remains the thorough confirmation but no longer gates anyt
|
||||
| **S3 — agent WG service (the agent-managed-unit pilot)** — **SHIPPED 2026-07-04** (agent v0.64.0; `wg_tunnel.enabled` DEFAULT FALSE until the production endpoint exists). `internal/wgtunnel`: keygen → marker-gated registration → `wg-quick@wg-felhom` reconciled from the served `wireguard` block (restart-not-reload; hash-gated zero-exec steady state); **revocation completed §3.5**: block absent from a PRESENT desired-state → disable + marker kept + never re-register (operator re-adds via the reported pubkey); absent DATA never tears down. `FELHOM_WG` sudoers (latest-handshakes is the ONLY wg read — `dump` leaks the private key); `IdentityBundle.WGPrivateKey` (pre-S3 blobs can't be retrofitted — S5 falls back to fresh-key re-register, /32 kept); report stanza = §4.6's tunnel-health input. Endpoint re-resolution: wg-quick resolves at unit (re)start; the manager restarts on conf change and the operator path covers re-IP (full §4.2 watchdog → S6 with tunnel-health alerting). Live on felhom-pve: up in 3 s, reboot-persistent, revocation drill, 30-min soak. | ~~keygen + registration…~~ | fresh box joins by §3.3 end-to-end ✓; tunnel survives agent restart + host reboot ✓; unhealthy-tunnel ALERTING → S6 (stanza shipped) |
|
||||
| **S4 — PBS over the tunnel** | point the box's PBS storage at `pbs_tunnel_ip:8007`; verify pin-through-tunnel | real backup lands in the box's namespace over the tunnel (nearly free — spike-proven path) |
|
||||
| **S5 — DR re-establishment** | consume `WGPrivateKey` in restore mode; re-peer on re-provision | host-loss drill restores offsite connectivity with only the recovery code. *Merges with the DR-completeness arc.* |
|
||||
| **S6 — monitoring + isolation hardening** | endpoint-side alerting, per-peer firewall audit; ~~CGNAT smoke-test closure~~ **DONE 2026-07-04 (§7)** — mobile-carrier NAT traversal + 32-min keepalive hold proven; residual = true-CGNAT-SIM retest (low risk) + the §4.3 MTU decision | tunnel-down and peer-anomaly both alert; isolation verified adversarially |
|
||||
| **S6 — monitoring + isolation hardening** | endpoint-side alerting, per-peer firewall audit; ~~CGNAT smoke-test closure~~ **DONE 2026-07-04 (§7)** — mobile-carrier NAT traversal + 32-min keepalive hold proven; ~~§4.3 MTU decision~~ **RESOLVED (client MTU 1280, agent v0.65.0, §4.3)**; residual = true-CGNAT-SIM retest (low risk) | tunnel-down and peer-anomaly both alert; isolation verified adversarially |
|
||||
|
||||
**MVP that puts offsite backup in a real customer's hands = S1 → S2 → S3; then S4.** S5/S6
|
||||
complete the story. The cloudflared service (`internal/hub/cloudflared.go:9-12` — today a
|
||||
|
||||
Reference in New Issue
Block a user