docs(offsite): close §7 CGNAT open-validation; record mobile-path MTU finding
S3 appendix CGNAT smoke test run live on felhom-pve (agent v0.64.0) with a USB-tethered phone. Tunnel established + held its NAT mapping across a 32-min idle soak on PersistentKeepalive=25 alone (0 stalls, final ping no-bounce). - §7: open-validation CLOSED; two honest caveats kept (SIM had a public mobile IPv4 so true CGNAT 100.64/10 not reproduced — retest-when-able; + MTU). - §4.3: mobile outer path MTU ~1400 black-holed bulk data at the shipped 1420; MTU 1340 restored the PBS page. OPEN DECISION: permanent vs per-connection. - §4.4: keepalive 32-min mobile-NAT hold datapoint. - §4.2: dual-stack v6 trap (wg-quick prefers endpoint AAAA, rides un-NATed v6). - §8: S6 row CGNAT closure marked done. - REPORT.md: overwritten with this operational-validation summary. Runbook: RUNBOOK-s3-cgnat-smoke. Docs-only; no agent/hub/controller code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
@@ -4,22 +4,36 @@
|
||||
|
||||
---
|
||||
|
||||
# REPORT — S3 wrap-up (docs-only in this repo; the slice lives in felhom-agent v0.64.0) (2026-07-04)
|
||||
# REPORT — S3 appendix: CGNAT smoke test (operational validation, docs-only in this repo) (2026-07-04)
|
||||
|
||||
felhom.eu carried **no code** in S3 — hub v0.33.0 (S2) already served the registration endpoint
|
||||
+ merged `wireguard` block. This commit records the S3 outcome in the design of record:
|
||||
Ran the deferred S3 appendix: the one open transport caveat in `06-offsite-connectivity.md` §7 —
|
||||
does plain-WireGuard (box dials out + `PersistentKeepalive=25`) establish and **hold** a mapping
|
||||
through a mobile-carrier NAT? Executed live on **felhom-pve** (`demo-felhom`, agent v0.64.0) with a
|
||||
USB-tethered Xiaomi Mi 11i. No code touched; the only host changes were runtime `ip`/`dhclient`
|
||||
one-shots, all reverted to the Phase-0 baseline at the end.
|
||||
|
||||
- `documentation/architecture/06-offsite-connectivity.md`: roadmap row **S3 → SHIPPED**
|
||||
(agent v0.64.0; `wg_tunnel.enabled` defaults FALSE until the production endpoint exists);
|
||||
§3.5 gains the **agent-side revocation semantics** (register-once marker; block-absent-from-
|
||||
present-state = disable + marker kept + never self-re-register; re-add via
|
||||
`POST /hosts/{id}/wg` with the global key — the raw registry add does NOT bump the host
|
||||
generation, found live) and the rollout gate.
|
||||
- `CONTEXT.md`: S3 entry (end-to-end handshake live on demo: keygen → register → block →
|
||||
wg-quick → PBS page in 3 s; revocation drill + 30-min soak + reboot persistence).
|
||||
**Method (surgical, SSH-safe):** kept the box on its wired default; steered **only** ep0's `/32`
|
||||
out the phone so just the tunnel's outer UDP traversed the carrier NAT. Neutralized the phone's
|
||||
RA-provided IPv6 + pinned ep0→v4 so the tunnel took the **v4/NAT** path (not un-NATed v6). 32-min
|
||||
idle soak sampling handshake age every 30 s. Full restore verified against the captured baseline.
|
||||
|
||||
Full evidence: **felhom-agent REPORT.md** (journal excerpts, five red-proofs, the
|
||||
hub-poll-constant gotcha, the re-add-path insight). Still open: CGNAT/mobile-hotspot smoke
|
||||
(operator-assisted appendix), S4 PBS-over-tunnel, S5 DR consume, S6 tunnel-health alerting +
|
||||
the two S6 backlog notes (make `defaultHostPollSeconds` configurable + log first-cycle adoption;
|
||||
make the admin registry-add bump the bound owner or mark it registry-plumbing in the UI).
|
||||
**Findings:**
|
||||
- **Uplink:** phone got a **public mobile IPv4** (`84.225.188.131`, Magyar Telekom) — a real
|
||||
single-NAT traversal (phone tether NAT), **not** true CGNAT `100.64/10`. Recorded as such.
|
||||
- **Handshake/PBS:** established over the mobile NAT path (endpoint v4 `167.233.158.164:443`);
|
||||
in-tunnel ping 4/4 ~57 ms; PBS login page reachable **after** the MTU fix.
|
||||
- **MTU (actionable):** outer path MTU ~**1400 B** → shipped `wg-felhom` MTU 1420 **black-holed
|
||||
bulk data** (ping/handshake healthy, PBS page empty). MTU **1340** (= 1400 − 60) restored it;
|
||||
1360 failed. Recorded in §4.3 with an **OPEN DECISION**: permanent fleet-wide lower MTU vs
|
||||
connection-type-dependent (1420 fixed-line / ≤1340 mobile, or MSS clamp).
|
||||
- **Soak:** 64 samples, max handshake age 125 s (normal rekey), **zero stalls**, final ping after
|
||||
32 min idle with no bounce → **mapping held on keepalive alone.**
|
||||
- **v6 trap:** a dual-stack mobile uplink made `wg-quick` prefer the endpoint AAAA and ride
|
||||
un-NATed IPv6 until v4 was forced (recorded in §4.2).
|
||||
|
||||
**Doc edits (this commit):** `06-offsite-connectivity.md` — §7 open-validation **CLOSED**;
|
||||
§4.3 MTU mobile finding + open decision; §4.4 keepalive 32-min mobile hold; §4.2 dual-stack v6
|
||||
trap; §8 S6 row updated. Runbook: `RUNBOOK-s3-cgnat-smoke`.
|
||||
|
||||
**Still open:** true-CGNAT-SIM retest (deferred, low risk — "when able"); the §4.3 MTU
|
||||
permanent-vs-per-connection decision; S4 PBS-over-tunnel; S5 DR consume; S6 tunnel-health alerting.
|
||||
|
||||
@@ -155,12 +155,31 @@ production endpoint exists.
|
||||
therefore owns re-resolution: on tunnel-unhealthy (no recent handshake), re-resolve and re-apply
|
||||
`wg set … endpoint=` (slice-3). Endpoint moves are rare, hub-announced (coords in desired-state),
|
||||
and self-heal even without the watchdog on service restart.
|
||||
**Dual-stack resolution trap (2026-07-04 CGNAT smoke test).** The endpoint is dual-stack (A +
|
||||
AAAA). On a box with working IPv6, `wg-quick` resolves and pins the **AAAA** at bring-up and the
|
||||
tunnel rides **un-NATed IPv6** — functionally fine (often better: no NAT), but it means the
|
||||
v4/NAT path is only exercised when v6 is absent. Two consequences: reasoning about "CGNAT
|
||||
traversal" applies to the v4 fallback only; and the re-resolve watchdog must not silently flip
|
||||
families mid-life in a way that strands a half-open v6 mapping. `AI_ADDRCONFIG` already gives the
|
||||
safe default — a v4-only box never sees the AAAA — but a v6-capable box will prefer it.
|
||||
- **4.3 MTU/MSS.** WG costs 60 bytes on v4 (interface MTU 1420 — what the spike ran end-to-end
|
||||
under PBS TLS with zero fragmentation stalls, P3). The unit ships explicit `MTU 1420`; since
|
||||
only the box's own PBS client uses the tunnel (no forwarded flows), no MSS clamp is needed —
|
||||
noted here so nobody adds forwarding without one.
|
||||
**Mobile-path finding (2026-07-04 CGNAT smoke test).** On a mobile-carrier uplink the outer
|
||||
path MTU measured **~1400 B** (DF probe: 1400-byte packet passes, 1414 fails), so the shipped
|
||||
1420 **black-holed bulk data**: ping and the WG handshake stayed healthy (small packets) while
|
||||
the PBS TLS page returned **empty**. Lowering `wg-felhom` to **MTU 1340** (= 1400 − 60, WG's v4
|
||||
overhead) restored the full PBS page; 1360 still failed. Fixed-line customers are unaffected (the
|
||||
spike's wired line carried 1420 clean). **OPEN DECISION (not yet made):** a lower MTU
|
||||
**permanent fleet-wide** (simplest; costs a little throughput on good links) vs
|
||||
**connection-type-dependent** (1420 fixed-line / ≤ 1340 mobile-CGNAT, or an MSS clamp on the
|
||||
tunnel). If it becomes agent-driven (auto-probe path MTU, or per-box policy), that's a separate
|
||||
TASK — the interface MTU is set in the agent-rendered `wg-felhom.conf`.
|
||||
- **4.4 Keepalive.** `PersistentKeepalive=25` — spike-proven to hold the NAT mapping through an
|
||||
11.4-minute fully-idle window (P2) at ~150 B/s of overhead traffic.
|
||||
11.4-minute fully-idle window (P2) at ~150 B/s of overhead traffic; **further proven through a
|
||||
live mobile-carrier NAT for a 32-minute fully-idle soak, zero stalls** (2026-07-04 CGNAT smoke
|
||||
test, §7).
|
||||
- **4.5 Isolation.** Per-peer `/32` `AllowedIPs`; IP forwarding stays **off** on the endpoint; its
|
||||
firewall admits, from the WG interface, only the PBS port — so a box can reach the PBS API and
|
||||
nothing else, and boxes cannot see each other **by topology** (spike P6). Public surface: SSH
|
||||
@@ -210,12 +229,20 @@ traverse `/root`) — create datastores under a world-traversable parent (`/srv`
|
||||
|
||||
## 7. Open validation + open sub-decisions (honest ledger)
|
||||
|
||||
**Open validation (one):** true-CGNAT traversal is argued from mechanism (outbound-initiated +
|
||||
keepalive holds a mapping on any NAT tier), **not yet measured** — the spike's line proved to be
|
||||
plain single-NAT with a public IPv4 (spike P0). Cheap closure: a mobile-hotspot smoke test
|
||||
(phone-tethered box → endpoint; mobile networks are true CGNAT); the deferred second-ISP vantage
|
||||
(Peti VM 110) remains the thorough confirmation. Do this before or during slice-3; it does not
|
||||
block slices 1-2.
|
||||
**Open validation — CLOSED (2026-07-04, mobile-hotspot smoke test).** A phone-tethered
|
||||
felhom-pve (agent v0.64.0) carried the live tunnel out through a mobile-carrier NAT and **held the
|
||||
mapping across a 32-minute fully-idle soak** on `PersistentKeepalive=25` alone: 64 samples, max
|
||||
handshake age 125 s (normal ~120 s rekey cadence), **zero stalls** (no sample > 180 s), rx counter
|
||||
growing on keepalive traffic only, and a final in-tunnel ping after 32 min idle succeeding with **no
|
||||
manual bounce**. PBS login page reachable over the tunnel once MTU was corrected (§4.3). Two honest
|
||||
caveats, recorded not papered over: **(1)** this SIM was handed a **public mobile IPv4**
|
||||
(`84.225.188.131`, Magyar Telekom), so the run exercised the phone's tether NAT + keepalive survival
|
||||
— a real single-NAT traversal — but did **not** reproduce true CGNAT `100.64/10` double-NAT; that
|
||||
stays a *retest-on-a-CGNAT-SIM-when-available* follow-up (low risk: mapping-hold is NAT-tier-agnostic
|
||||
by mechanism). **(2)** a dual-stack mobile uplink made `wg-quick` prefer the endpoint **AAAA and ride
|
||||
un-NATed IPv6** until v4 was forced — functionally fine, but see §4.2. The deferred second-ISP
|
||||
vantage (Peti VM 110) remains the thorough confirmation but no longer gates anything. Runbook:
|
||||
`RUNBOOK-s3-cgnat-smoke`.
|
||||
|
||||
**Open sub-decisions (deferred by design):**
|
||||
- Endpoint peer-sync mechanism (push-API vs signed-manifest pull) — **slice-1 design point** (§5).
|
||||
@@ -238,7 +265,7 @@ block slices 1-2.
|
||||
| **S3 — agent WG service (the agent-managed-unit pilot)** — **SHIPPED 2026-07-04** (agent v0.64.0; `wg_tunnel.enabled` DEFAULT FALSE until the production endpoint exists). `internal/wgtunnel`: keygen → marker-gated registration → `wg-quick@wg-felhom` reconciled from the served `wireguard` block (restart-not-reload; hash-gated zero-exec steady state); **revocation completed §3.5**: block absent from a PRESENT desired-state → disable + marker kept + never re-register (operator re-adds via the reported pubkey); absent DATA never tears down. `FELHOM_WG` sudoers (latest-handshakes is the ONLY wg read — `dump` leaks the private key); `IdentityBundle.WGPrivateKey` (pre-S3 blobs can't be retrofitted — S5 falls back to fresh-key re-register, /32 kept); report stanza = §4.6's tunnel-health input. Endpoint re-resolution: wg-quick resolves at unit (re)start; the manager restarts on conf change and the operator path covers re-IP (full §4.2 watchdog → S6 with tunnel-health alerting). Live on felhom-pve: up in 3 s, reboot-persistent, revocation drill, 30-min soak. | ~~keygen + registration…~~ | fresh box joins by §3.3 end-to-end ✓; tunnel survives agent restart + host reboot ✓; unhealthy-tunnel ALERTING → S6 (stanza shipped) |
|
||||
| **S4 — PBS over the tunnel** | point the box's PBS storage at `pbs_tunnel_ip:8007`; verify pin-through-tunnel | real backup lands in the box's namespace over the tunnel (nearly free — spike-proven path) |
|
||||
| **S5 — DR re-establishment** | consume `WGPrivateKey` in restore mode; re-peer on re-provision | host-loss drill restores offsite connectivity with only the recovery code. *Merges with the DR-completeness arc.* |
|
||||
| **S6 — monitoring + isolation hardening** | endpoint-side alerting, per-peer firewall audit, CGNAT smoke-test closure if not yet done | tunnel-down and peer-anomaly both alert; isolation verified adversarially |
|
||||
| **S6 — monitoring + isolation hardening** | endpoint-side alerting, per-peer firewall audit; ~~CGNAT smoke-test closure~~ **DONE 2026-07-04 (§7)** — mobile-carrier NAT traversal + 32-min keepalive hold proven; residual = true-CGNAT-SIM retest (low risk) + the §4.3 MTU decision | tunnel-down and peer-anomaly both alert; isolation verified adversarially |
|
||||
|
||||
**MVP that puts offsite backup in a real customer's hands = S1 → S2 → S3; then S4.** S5/S6
|
||||
complete the story. The cloudflared service (`internal/hub/cloudflared.go:9-12` — today a
|
||||
|
||||
Reference in New Issue
Block a user