docs(offsite): resolve §4.3 MTU open decision → client MTU 1280 (S3.1)

The CGNAT smoke test's OPEN DECISION is made: fleet-wide, permanent,
family-agnostic client MTU 1280 (RFC 8200 IPv6-minimum floor), implemented in
felhom-agent v0.65.0.

- §4.3: OPEN DECISION → made decision + rationale + rejected alternatives
  (auto-probe/per-type = fragility for a non-metric; MSS-clamp = unneeded).
  "ships MTU 1420" → 1280; corrected "fixed-line unaffected" (DS-Lite ~1452,
  PPPoE would also black-hole at 1420 — one floor covers all).
- §4.2: 1280 is family-agnostic so MTU correctness is decoupled from v4/v6
  resolution; v4-pin noted as a separate optional determinism item.
- §3.2 wire-field note + §8 S6 row: MTU 1420 → 1280; S6 residual now only the
  true-CGNAT-SIM retest.
- REPORT.md overwritten.

Docs-only; the constant change is in felhom-agent v0.65.0.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
2026-07-04 14:02:48 +02:00
parent 2028785f5e
commit a7f954dde7
2 changed files with 55 additions and 46 deletions
+20 -29
View File
@@ -4,36 +4,27 @@
---
# REPORT — S3 appendix: CGNAT smoke test (operational validation, docs-only in this repo) (2026-07-04)
# REPORT — S3.1 doc close-out: offsite-tunnel MTU decision resolved (docs-only) (2026-07-04)
Ran the deferred S3 appendix: the one open transport caveat in `06-offsite-connectivity.md` §7 —
does plain-WireGuard (box dials out + `PersistentKeepalive=25`) establish and **hold** a mapping
through a mobile-carrier NAT? Executed live on **felhom-pve** (`demo-felhom`, agent v0.64.0) with a
USB-tethered Xiaomi Mi 11i. No code touched; the only host changes were runtime `ip`/`dhclient`
one-shots, all reverted to the Phase-0 baseline at the end.
Docs-only companion to the S3.1 implementation (the constant change lives in **felhom-agent
v0.65.0** — see that repo's REPORT). Resolves the OPEN DECISION the 2026-07-04 CGNAT smoke test
left in `documentation/architecture/06-offsite-connectivity.md` §4.3.
**Method (surgical, SSH-safe):** kept the box on its wired default; steered **only** ep0's `/32`
out the phone so just the tunnel's outer UDP traversed the carrier NAT. Neutralized the phone's
RA-provided IPv6 + pinned ep0→v4 so the tunnel took the **v4/NAT** path (not un-NATed v6). 32-min
idle soak sampling handshake age every 30 s. Full restore verified against the captured baseline.
**Decision recorded:** fleet-wide client **MTU 1280** (RFC 8200 IPv6-minimum floor; outer 1340 v4 /
1360 v6 fits every realistic path), permanent + family-agnostic, client-only (endpoint `wg0`
untouched — interface MTU caps box→PBS, advertised MSS caps PBS→box).
**Findings:**
- **Uplink:** phone got a **public mobile IPv4** (`84.225.188.131`, Magyar Telekom) — a real
single-NAT traversal (phone tether NAT), **not** true CGNAT `100.64/10`. Recorded as such.
- **Handshake/PBS:** established over the mobile NAT path (endpoint v4 `167.233.158.164:443`);
in-tunnel ping 4/4 ~57 ms; PBS login page reachable **after** the MTU fix.
- **MTU (actionable):** outer path MTU ~**1400 B** → shipped `wg-felhom` MTU 1420 **black-holed
bulk data** (ping/handshake healthy, PBS page empty). MTU **1340** (= 1400 60) restored it;
1360 failed. Recorded in §4.3 with an **OPEN DECISION**: permanent fleet-wide lower MTU vs
connection-type-dependent (1420 fixed-line / ≤1340 mobile, or MSS clamp).
- **Soak:** 64 samples, max handshake age 125 s (normal rekey), **zero stalls**, final ping after
32 min idle with no bounce → **mapping held on keepalive alone.**
- **v6 trap:** a dual-stack mobile uplink made `wg-quick` prefer the endpoint AAAA and ride
un-NATed IPv6 until v4 was forced (recorded in §4.2).
**Edits to `06-offsite-connectivity.md`:**
- **§4.3** — replaced "OPEN DECISION (not yet made)" with the made decision + rationale + the
rejected alternatives (auto-probe/per-connection-type = fragility for a non-metric; MSS-clamp =
unneeded without forwarding). Updated "the unit ships explicit MTU 1420" → 1280.
- **§4.3** — corrected the over-general "fixed-line customers are unaffected": sub-1500 fixed-line
paths (DS-Lite ~1452, some PPPoE) would also have black-holed at 1420; one 1280 floor covers all.
- **§4.2** — noted the 1280 floor is chosen family-agnostic *specifically so* MTU correctness does
not depend on the v4/v6 resolution outcome; the deterministic v4-pin stays a separate, optional
future determinism note (not built here).
- **§3.2 / §8 S6 row** — stale "MTU 1420" wire-field note → 1280; S6 residual list drops the
now-resolved MTU decision, keeping only the low-risk true-CGNAT-SIM retest.
**Doc edits (this commit):** `06-offsite-connectivity.md` — §7 open-validation **CLOSED**;
§4.3 MTU mobile finding + open decision; §4.4 keepalive 32-min mobile hold; §4.2 dual-stack v6
trap; §8 S6 row updated. Runbook: `RUNBOOK-s3-cgnat-smoke`.
**Still open:** true-CGNAT-SIM retest (deferred, low risk — "when able"); the §4.3 MTU
permanent-vs-per-connection decision; S4 PBS-over-tunnel; S5 DR consume; S6 tunnel-health alerting.
**Not changed:** no code, no endpoint, no hub/controller, no JSON golden (MTU is never on the wire).
One docs-only commit.
@@ -83,7 +83,7 @@ New trust-boundary rows (extends 01 §5):
}
```
Client-side `AllowedIPs`, `PersistentKeepalive=25`, and `MTU 1420` are deliberately NOT wire
Client-side `AllowedIPs`, `PersistentKeepalive=25`, and `MTU 1280` are deliberately NOT wire
fields — S3 agent constants derived from `pbs_tunnel_ip` + §4. Delivery needs **zero new
plumbing** — registration/re-key/unbind bump `DesiredGeneration`, the agent's existing syncer
fetches (`syncer.go:44-61`). Re-key replaces the pubkey IN PLACE and keeps the `/32` (stable
@@ -161,21 +161,39 @@ production endpoint exists.
v4/NAT path is only exercised when v6 is absent. Two consequences: reasoning about "CGNAT
traversal" applies to the v4 fallback only; and the re-resolve watchdog must not silently flip
families mid-life in a way that strands a half-open v6 mapping. `AI_ADDRCONFIG` already gives the
safe default — a v4-only box never sees the AAAA — but a v6-capable box will prefer it.
- **4.3 MTU/MSS.** WG costs 60 bytes on v4 (interface MTU 1420 — what the spike ran end-to-end
under PBS TLS with zero fragmentation stalls, P3). The unit ships explicit `MTU 1420`; since
only the box's own PBS client uses the tunnel (no forwarded flows), no MSS clamp is needed —
noted here so nobody adds forwarding without one.
**Mobile-path finding (2026-07-04 CGNAT smoke test).** On a mobile-carrier uplink the outer
path MTU measured **~1400 B** (DF probe: 1400-byte packet passes, 1414 fails), so the shipped
1420 **black-holed bulk data**: ping and the WG handshake stayed healthy (small packets) while
the PBS TLS page returned **empty**. Lowering `wg-felhom` to **MTU 1340** (= 1400 60, WG's v4
overhead) restored the full PBS page; 1360 still failed. Fixed-line customers are unaffected (the
spike's wired line carried 1420 clean). **OPEN DECISION (not yet made):** a lower MTU
**permanent fleet-wide** (simplest; costs a little throughput on good links) vs
**connection-type-dependent** (1420 fixed-line / ≤ 1340 mobile-CGNAT, or an MSS clamp on the
tunnel). If it becomes agent-driven (auto-probe path MTU, or per-box policy), that's a separate
TASK — the interface MTU is set in the agent-rendered `wg-felhom.conf`.
safe default — a v4-only box never sees the AAAA — but a v6-capable box will prefer it. The
client MTU floor (§4.3) is deliberately chosen **family-agnostic (1280)** *so that* MTU
correctness does not depend on this resolution outcome. A **deterministic v4-pin** — the agent
resolving ep0's A record and writing the literal IP into the conf (a natural extension of this
watchdog) — remains an **open, separate determinism question**; it is **not required for MTU
correctness** and is tracked on its own, not built by S3.1.
- **4.3 MTU/MSS — DECIDED: fleet-wide client MTU 1280 (agent v0.65.0, S3.1).** WG costs 60 bytes
on v4 / 80 on IPv6. The unit ships explicit **`MTU 1280`** — the **IPv6-minimum link MTU** (RFC
8200 guarantees every path carries ≥ 1280), so outer = 1280+60 = 1340 (v4) / 1280+80 = 1360 (v6),
both fitting every realistic path (mobile ~1400, DS-Lite ~1452, PPPoE 1492, clean 1500). Because
only the box's own PBS client uses the tunnel (locally-terminated TCP, no forwarded flows), the
low **interface MTU bounds both directions** — it caps box→PBS directly and advertises MSS =
MTU40 to cap PBS→box — so **no MSS clamp is needed** and, critically, **the endpoint's `wg0` is
left untouched** (its sends are already capped by the box's advertised MSS). Zero live-endpoint
risk; one client constant. `1280` is also **family-agnostic**, so MTU correctness does **not**
depend on whether `wg-quick` resolves the endpoint's A or AAAA (§4.2). Noted so nobody adds
forwarding without an MSS clamp.
**Why 1280 and not 1420 (2026-07-04 CGNAT smoke test).** On a mobile-carrier uplink the outer
path MTU measured **~1400 B** (DF probe: 1400-byte packet passes, 1414 fails), so the previously
shipped **1420 black-holed bulk data** — ping and the WG handshake stayed healthy (small packets)
while the PBS TLS page returned **empty**. Field data: `MTU 1340` (= 1400 60) restored the full
PBS page, `1360` still failed; `1280` sits safely below both. This is **not a fixed-line-vs-mobile
split**: sub-1500 fixed-line paths — notably **DS-Lite (~1452 AFTR path)** and some PPPoE — would
also have black-holed at 1420, so one conservative floor covers all customers. It "looked green,
lost backups": the handshake and ping succeed at any MTU, so the failure is invisible until bulk
TCP (the backup) flows — which is exactly S4.
**Rejected alternatives.** *Auto-probe path MTU / per-connection-type policy* — a fragile moving
part (a probe that can fail, and must re-run on every path change: ISP switch, mobile failover) to
optimize **throughput**, which doesn't matter here (overnight backup, bound by the customer's home
uplink, not by ~10% MTU overhead). *MSS clamp* — unnecessary without forwarded flows. A fixed
conservative floor is strictly more robust. The interface MTU lives in the agent-rendered
`wg-felhom.conf` (`internal/wgtunnel` `clientMTU`); revisiting it is a deliberate constant change,
not an operational knob.
- **4.4 Keepalive.** `PersistentKeepalive=25` — spike-proven to hold the NAT mapping through an
11.4-minute fully-idle window (P2) at ~150 B/s of overhead traffic; **further proven through a
live mobile-carrier NAT for a 32-minute fully-idle soak, zero stalls** (2026-07-04 CGNAT smoke
@@ -265,7 +283,7 @@ vantage (Peti VM 110) remains the thorough confirmation but no longer gates anyt
| **S3 — agent WG service (the agent-managed-unit pilot)** — **SHIPPED 2026-07-04** (agent v0.64.0; `wg_tunnel.enabled` DEFAULT FALSE until the production endpoint exists). `internal/wgtunnel`: keygen → marker-gated registration → `wg-quick@wg-felhom` reconciled from the served `wireguard` block (restart-not-reload; hash-gated zero-exec steady state); **revocation completed §3.5**: block absent from a PRESENT desired-state → disable + marker kept + never re-register (operator re-adds via the reported pubkey); absent DATA never tears down. `FELHOM_WG` sudoers (latest-handshakes is the ONLY wg read — `dump` leaks the private key); `IdentityBundle.WGPrivateKey` (pre-S3 blobs can't be retrofitted — S5 falls back to fresh-key re-register, /32 kept); report stanza = §4.6's tunnel-health input. Endpoint re-resolution: wg-quick resolves at unit (re)start; the manager restarts on conf change and the operator path covers re-IP (full §4.2 watchdog → S6 with tunnel-health alerting). Live on felhom-pve: up in 3 s, reboot-persistent, revocation drill, 30-min soak. | ~~keygen + registration…~~ | fresh box joins by §3.3 end-to-end ✓; tunnel survives agent restart + host reboot ✓; unhealthy-tunnel ALERTING → S6 (stanza shipped) |
| **S4 — PBS over the tunnel** | point the box's PBS storage at `pbs_tunnel_ip:8007`; verify pin-through-tunnel | real backup lands in the box's namespace over the tunnel (nearly free — spike-proven path) |
| **S5 — DR re-establishment** | consume `WGPrivateKey` in restore mode; re-peer on re-provision | host-loss drill restores offsite connectivity with only the recovery code. *Merges with the DR-completeness arc.* |
| **S6 — monitoring + isolation hardening** | endpoint-side alerting, per-peer firewall audit; ~~CGNAT smoke-test closure~~ **DONE 2026-07-04 (§7)** — mobile-carrier NAT traversal + 32-min keepalive hold proven; residual = true-CGNAT-SIM retest (low risk) + the §4.3 MTU decision | tunnel-down and peer-anomaly both alert; isolation verified adversarially |
| **S6 — monitoring + isolation hardening** | endpoint-side alerting, per-peer firewall audit; ~~CGNAT smoke-test closure~~ **DONE 2026-07-04 (§7)** — mobile-carrier NAT traversal + 32-min keepalive hold proven; ~~§4.3 MTU decision~~ **RESOLVED (client MTU 1280, agent v0.65.0, §4.3)**; residual = true-CGNAT-SIM retest (low risk) | tunnel-down and peer-anomaly both alert; isolation verified adversarially |
**MVP that puts offsite backup in a real customer's hands = S1 → S2 → S3; then S4.** S5/S6
complete the story. The cloudflared service (`internal/hub/cloudflared.go:9-12` — today a