From 2028785f5eb24f9c2b35d0dd2b217a71c4be6380 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sat, 4 Jul 2026 13:34:00 +0200 Subject: [PATCH] =?UTF-8?q?docs(offsite):=20close=20=C2=A77=20CGNAT=20open?= =?UTF-8?q?-validation;=20record=20mobile-path=20MTU=20finding?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit S3 appendix CGNAT smoke test run live on felhom-pve (agent v0.64.0) with a USB-tethered phone. Tunnel established + held its NAT mapping across a 32-min idle soak on PersistentKeepalive=25 alone (0 stalls, final ping no-bounce). - §7: open-validation CLOSED; two honest caveats kept (SIM had a public mobile IPv4 so true CGNAT 100.64/10 not reproduced — retest-when-able; + MTU). - §4.3: mobile outer path MTU ~1400 black-holed bulk data at the shipped 1420; MTU 1340 restored the PBS page. OPEN DECISION: permanent vs per-connection. - §4.4: keepalive 32-min mobile-NAT hold datapoint. - §4.2: dual-stack v6 trap (wg-quick prefers endpoint AAAA, rides un-NATed v6). - §8: S6 row CGNAT closure marked done. - REPORT.md: overwritten with this operational-validation summary. Runbook: RUNBOOK-s3-cgnat-smoke. Docs-only; no agent/hub/controller code. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 --- REPORT.md | 46 ++++++++++++------- .../architecture/06-offsite-connectivity.md | 43 +++++++++++++---- 2 files changed, 65 insertions(+), 24 deletions(-) diff --git a/REPORT.md b/REPORT.md index 778795f..a28ea98 100644 --- a/REPORT.md +++ b/REPORT.md @@ -4,22 +4,36 @@ --- -# REPORT — S3 wrap-up (docs-only in this repo; the slice lives in felhom-agent v0.64.0) (2026-07-04) +# REPORT — S3 appendix: CGNAT smoke test (operational validation, docs-only in this repo) (2026-07-04) -felhom.eu carried **no code** in S3 — hub v0.33.0 (S2) already served the registration endpoint -+ merged `wireguard` block. This commit records the S3 outcome in the design of record: +Ran the deferred S3 appendix: the one open transport caveat in `06-offsite-connectivity.md` §7 — +does plain-WireGuard (box dials out + `PersistentKeepalive=25`) establish and **hold** a mapping +through a mobile-carrier NAT? Executed live on **felhom-pve** (`demo-felhom`, agent v0.64.0) with a +USB-tethered Xiaomi Mi 11i. No code touched; the only host changes were runtime `ip`/`dhclient` +one-shots, all reverted to the Phase-0 baseline at the end. -- `documentation/architecture/06-offsite-connectivity.md`: roadmap row **S3 → SHIPPED** - (agent v0.64.0; `wg_tunnel.enabled` defaults FALSE until the production endpoint exists); - §3.5 gains the **agent-side revocation semantics** (register-once marker; block-absent-from- - present-state = disable + marker kept + never self-re-register; re-add via - `POST /hosts/{id}/wg` with the global key — the raw registry add does NOT bump the host - generation, found live) and the rollout gate. -- `CONTEXT.md`: S3 entry (end-to-end handshake live on demo: keygen → register → block → - wg-quick → PBS page in 3 s; revocation drill + 30-min soak + reboot persistence). +**Method (surgical, SSH-safe):** kept the box on its wired default; steered **only** ep0's `/32` +out the phone so just the tunnel's outer UDP traversed the carrier NAT. Neutralized the phone's +RA-provided IPv6 + pinned ep0→v4 so the tunnel took the **v4/NAT** path (not un-NATed v6). 32-min +idle soak sampling handshake age every 30 s. Full restore verified against the captured baseline. -Full evidence: **felhom-agent REPORT.md** (journal excerpts, five red-proofs, the -hub-poll-constant gotcha, the re-add-path insight). Still open: CGNAT/mobile-hotspot smoke -(operator-assisted appendix), S4 PBS-over-tunnel, S5 DR consume, S6 tunnel-health alerting + -the two S6 backlog notes (make `defaultHostPollSeconds` configurable + log first-cycle adoption; -make the admin registry-add bump the bound owner or mark it registry-plumbing in the UI). +**Findings:** +- **Uplink:** phone got a **public mobile IPv4** (`84.225.188.131`, Magyar Telekom) — a real + single-NAT traversal (phone tether NAT), **not** true CGNAT `100.64/10`. Recorded as such. +- **Handshake/PBS:** established over the mobile NAT path (endpoint v4 `167.233.158.164:443`); + in-tunnel ping 4/4 ~57 ms; PBS login page reachable **after** the MTU fix. +- **MTU (actionable):** outer path MTU ~**1400 B** → shipped `wg-felhom` MTU 1420 **black-holed + bulk data** (ping/handshake healthy, PBS page empty). MTU **1340** (= 1400 − 60) restored it; + 1360 failed. Recorded in §4.3 with an **OPEN DECISION**: permanent fleet-wide lower MTU vs + connection-type-dependent (1420 fixed-line / ≤1340 mobile, or MSS clamp). +- **Soak:** 64 samples, max handshake age 125 s (normal rekey), **zero stalls**, final ping after + 32 min idle with no bounce → **mapping held on keepalive alone.** +- **v6 trap:** a dual-stack mobile uplink made `wg-quick` prefer the endpoint AAAA and ride + un-NATed IPv6 until v4 was forced (recorded in §4.2). + +**Doc edits (this commit):** `06-offsite-connectivity.md` — §7 open-validation **CLOSED**; +§4.3 MTU mobile finding + open decision; §4.4 keepalive 32-min mobile hold; §4.2 dual-stack v6 +trap; §8 S6 row updated. Runbook: `RUNBOOK-s3-cgnat-smoke`. + +**Still open:** true-CGNAT-SIM retest (deferred, low risk — "when able"); the §4.3 MTU +permanent-vs-per-connection decision; S4 PBS-over-tunnel; S5 DR consume; S6 tunnel-health alerting. diff --git a/documentation/architecture/06-offsite-connectivity.md b/documentation/architecture/06-offsite-connectivity.md index ade1780..4c69a27 100644 --- a/documentation/architecture/06-offsite-connectivity.md +++ b/documentation/architecture/06-offsite-connectivity.md @@ -155,12 +155,31 @@ production endpoint exists. therefore owns re-resolution: on tunnel-unhealthy (no recent handshake), re-resolve and re-apply `wg set … endpoint=` (slice-3). Endpoint moves are rare, hub-announced (coords in desired-state), and self-heal even without the watchdog on service restart. + **Dual-stack resolution trap (2026-07-04 CGNAT smoke test).** The endpoint is dual-stack (A + + AAAA). On a box with working IPv6, `wg-quick` resolves and pins the **AAAA** at bring-up and the + tunnel rides **un-NATed IPv6** — functionally fine (often better: no NAT), but it means the + v4/NAT path is only exercised when v6 is absent. Two consequences: reasoning about "CGNAT + traversal" applies to the v4 fallback only; and the re-resolve watchdog must not silently flip + families mid-life in a way that strands a half-open v6 mapping. `AI_ADDRCONFIG` already gives the + safe default — a v4-only box never sees the AAAA — but a v6-capable box will prefer it. - **4.3 MTU/MSS.** WG costs 60 bytes on v4 (interface MTU 1420 — what the spike ran end-to-end under PBS TLS with zero fragmentation stalls, P3). The unit ships explicit `MTU 1420`; since only the box's own PBS client uses the tunnel (no forwarded flows), no MSS clamp is needed — noted here so nobody adds forwarding without one. + **Mobile-path finding (2026-07-04 CGNAT smoke test).** On a mobile-carrier uplink the outer + path MTU measured **~1400 B** (DF probe: 1400-byte packet passes, 1414 fails), so the shipped + 1420 **black-holed bulk data**: ping and the WG handshake stayed healthy (small packets) while + the PBS TLS page returned **empty**. Lowering `wg-felhom` to **MTU 1340** (= 1400 − 60, WG's v4 + overhead) restored the full PBS page; 1360 still failed. Fixed-line customers are unaffected (the + spike's wired line carried 1420 clean). **OPEN DECISION (not yet made):** a lower MTU + **permanent fleet-wide** (simplest; costs a little throughput on good links) vs + **connection-type-dependent** (1420 fixed-line / ≤ 1340 mobile-CGNAT, or an MSS clamp on the + tunnel). If it becomes agent-driven (auto-probe path MTU, or per-box policy), that's a separate + TASK — the interface MTU is set in the agent-rendered `wg-felhom.conf`. - **4.4 Keepalive.** `PersistentKeepalive=25` — spike-proven to hold the NAT mapping through an - 11.4-minute fully-idle window (P2) at ~150 B/s of overhead traffic. + 11.4-minute fully-idle window (P2) at ~150 B/s of overhead traffic; **further proven through a + live mobile-carrier NAT for a 32-minute fully-idle soak, zero stalls** (2026-07-04 CGNAT smoke + test, §7). - **4.5 Isolation.** Per-peer `/32` `AllowedIPs`; IP forwarding stays **off** on the endpoint; its firewall admits, from the WG interface, only the PBS port — so a box can reach the PBS API and nothing else, and boxes cannot see each other **by topology** (spike P6). Public surface: SSH @@ -210,12 +229,20 @@ traverse `/root`) — create datastores under a world-traversable parent (`/srv` ## 7. Open validation + open sub-decisions (honest ledger) -**Open validation (one):** true-CGNAT traversal is argued from mechanism (outbound-initiated + -keepalive holds a mapping on any NAT tier), **not yet measured** — the spike's line proved to be -plain single-NAT with a public IPv4 (spike P0). Cheap closure: a mobile-hotspot smoke test -(phone-tethered box → endpoint; mobile networks are true CGNAT); the deferred second-ISP vantage -(Peti VM 110) remains the thorough confirmation. Do this before or during slice-3; it does not -block slices 1-2. +**Open validation — CLOSED (2026-07-04, mobile-hotspot smoke test).** A phone-tethered +felhom-pve (agent v0.64.0) carried the live tunnel out through a mobile-carrier NAT and **held the +mapping across a 32-minute fully-idle soak** on `PersistentKeepalive=25` alone: 64 samples, max +handshake age 125 s (normal ~120 s rekey cadence), **zero stalls** (no sample > 180 s), rx counter +growing on keepalive traffic only, and a final in-tunnel ping after 32 min idle succeeding with **no +manual bounce**. PBS login page reachable over the tunnel once MTU was corrected (§4.3). Two honest +caveats, recorded not papered over: **(1)** this SIM was handed a **public mobile IPv4** +(`84.225.188.131`, Magyar Telekom), so the run exercised the phone's tether NAT + keepalive survival +— a real single-NAT traversal — but did **not** reproduce true CGNAT `100.64/10` double-NAT; that +stays a *retest-on-a-CGNAT-SIM-when-available* follow-up (low risk: mapping-hold is NAT-tier-agnostic +by mechanism). **(2)** a dual-stack mobile uplink made `wg-quick` prefer the endpoint **AAAA and ride +un-NATed IPv6** until v4 was forced — functionally fine, but see §4.2. The deferred second-ISP +vantage (Peti VM 110) remains the thorough confirmation but no longer gates anything. Runbook: +`RUNBOOK-s3-cgnat-smoke`. **Open sub-decisions (deferred by design):** - Endpoint peer-sync mechanism (push-API vs signed-manifest pull) — **slice-1 design point** (§5). @@ -238,7 +265,7 @@ block slices 1-2. | **S3 — agent WG service (the agent-managed-unit pilot)** — **SHIPPED 2026-07-04** (agent v0.64.0; `wg_tunnel.enabled` DEFAULT FALSE until the production endpoint exists). `internal/wgtunnel`: keygen → marker-gated registration → `wg-quick@wg-felhom` reconciled from the served `wireguard` block (restart-not-reload; hash-gated zero-exec steady state); **revocation completed §3.5**: block absent from a PRESENT desired-state → disable + marker kept + never re-register (operator re-adds via the reported pubkey); absent DATA never tears down. `FELHOM_WG` sudoers (latest-handshakes is the ONLY wg read — `dump` leaks the private key); `IdentityBundle.WGPrivateKey` (pre-S3 blobs can't be retrofitted — S5 falls back to fresh-key re-register, /32 kept); report stanza = §4.6's tunnel-health input. Endpoint re-resolution: wg-quick resolves at unit (re)start; the manager restarts on conf change and the operator path covers re-IP (full §4.2 watchdog → S6 with tunnel-health alerting). Live on felhom-pve: up in 3 s, reboot-persistent, revocation drill, 30-min soak. | ~~keygen + registration…~~ | fresh box joins by §3.3 end-to-end ✓; tunnel survives agent restart + host reboot ✓; unhealthy-tunnel ALERTING → S6 (stanza shipped) | | **S4 — PBS over the tunnel** | point the box's PBS storage at `pbs_tunnel_ip:8007`; verify pin-through-tunnel | real backup lands in the box's namespace over the tunnel (nearly free — spike-proven path) | | **S5 — DR re-establishment** | consume `WGPrivateKey` in restore mode; re-peer on re-provision | host-loss drill restores offsite connectivity with only the recovery code. *Merges with the DR-completeness arc.* | -| **S6 — monitoring + isolation hardening** | endpoint-side alerting, per-peer firewall audit, CGNAT smoke-test closure if not yet done | tunnel-down and peer-anomaly both alert; isolation verified adversarially | +| **S6 — monitoring + isolation hardening** | endpoint-side alerting, per-peer firewall audit; ~~CGNAT smoke-test closure~~ **DONE 2026-07-04 (§7)** — mobile-carrier NAT traversal + 32-min keepalive hold proven; residual = true-CGNAT-SIM retest (low risk) + the §4.3 MTU decision | tunnel-down and peer-anomaly both alert; isolation verified adversarially | **MVP that puts offsite backup in a real customer's hands = S1 → S2 → S3; then S4.** S5/S6 complete the story. The cloudflared service (`internal/hub/cloudflared.go:9-12` — today a