diff --git a/CONTEXT.md b/CONTEXT.md index 042593b..1f8a08b 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -5,6 +5,18 @@ ## Current +- **v0.65.0** (2026-07-04, live on felhom-pve) — **S3.1 offsite-tunnel client MTU 1420 → 1280**: + resolves `06 §4.3`'s OPEN DECISION left by the CGNAT smoke test. 1420 **silently black-holed bulk + TCP** on sub-~1480 paths (mobile ~1400, DS-Lite ~1452) — handshake+ping healthy, PBS TLS page + (and at S4 the backup itself) drops. New `const clientMTU = 1280` (RFC 8200 IPv6-minimum floor; + outer 1340 v4 / 1360 v6 fits every realistic path), **permanent + fleet-wide + family-agnostic**. + **Client-only by construction** — interface MTU caps box→PBS, advertised MSS caps PBS→box, so the + endpoint's `wg0` is untouched (zero live-endpoint risk). Golden pins exact `MTU = 1280` + (red-proofed vs a 1420 flip); no wire/JSON change. Live: agent re-rendered on restart (hash-gated + apply), conf + live iface both 1280, PBS page loads at 1280 (no regression on wired). OPEN: + true-CGNAT-SIM retest (low risk); publish 0.65.0 + Day-0 vouch (operator); S4 PBS-over-tunnel. + Rollback: `felhom-agent.bak-0.64.0` on the box. The v4-pin (§4.2 determinism) is a separate, + optional future note — NOT needed for MTU correctness. - **v0.64.0** (2026-07-04, live on felhom-pve) — **S3 offsite WG tunnel**: new `internal/wgtunnel` (keygen 0600/0700, marker-gated one-shot registration, agent-managed `wg-quick@wg-felhom` from the hub's desired-state `wireguard` block via the new `desired.Syncer.AddConsumer` seam, diff --git a/REPORT.md b/REPORT.md index 0954705..297d5db 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,135 +1,71 @@ -# REPORT — S3: agent WG tunnel — keygen + registration + wg-quick@wg-felhom + escrow join (v0.64.0) +# REPORT — S3.1: offsite-tunnel client MTU 1420 → 1280 (v0.65.0) **Date:** 2026-07-04 · **Class:** implementation + live validation on felhom-pve (the expendable -demo box; Peti's box untouched by construction — `wg_tunnel.enabled` defaults FALSE). **Design of -record:** `felhom.eu/documentation/architecture/06-offsite-connectivity.md` row S3 (SHIPPED). +demo box; Peti's box untouched; endpoint untouched by construction). **Design of record:** +`felhom.eu/documentation/architecture/06-offsite-connectivity.md` §4.3 (OPEN DECISION → resolved). ## 1. Baselines & commits -Start: felhom-agent @ `4ba1b14` (v0.63.0); felhom.eu @ `4fe895e` (hub v0.33.0, code read-only). -Commits pushed to `main`: +Start: felhom-agent @ `03edb40` (v0.64.0); felhom.eu @ `2028785` (docs). Commits pushed to `main`: -1. `0daae92` wgtunnel keygen + hub wire (WireWireguard/report stanza/RegisterWG) + byte-identical golden copy -2. `fb24896` manager state machine + loop + desired raw-consumer seam -3. `e2b6c63` FELHOM_WG sudoers + capabilities + config (DEFAULT OFF) + main wiring + escrow join -4. (docs commit) CHANGELOG v0.64.0 + CONTEXT + this REPORT +- **`312fd5e`** — `wgtunnel: client MTU 1420 → 1280` (manager.go const + render, manager_test.go + golden + red-proof note, report.go stale comment, CHANGELOG v0.65.0). +- **``** — CONTEXT + this REPORT. +- felhom.eu docs (§4.3 decision + §4.2/§4.3 wording + S6 row): separate one commit — see felhom.eu REPORT. -## 2. Files +## 2. What changed & why -- NEW `internal/wgtunnel/{key,manager,loop}.go` + tests (key vectors, state-machine Groups A/B) -- `internal/hub/{report,client,collect}.go` (+`wg_contract_test.go`, `wg_client_test.go`, - NEW `testdata/desired-state-wireguard.golden.json` — **byte-identical hub copy**, diff-verified) -- `internal/desired/syncer.go` (`AddConsumer` + panic containment) + `consumer_test.go` -- `internal/escrow/identity.go` (`WGPrivateKey` + `AttachWGKey`) + `wgkey_test.go` -- `internal/config/config.go` (`WGTunnelConfig`, **Enabled default FALSE**) -- `cmd/felhom-agent/main.go` (daemon wiring, escrow-create auto-inject, `--selftest=wgtunnel`) -- `configs/felhom-agent.sudoers` (`Cmnd_Alias FELHOM_WG`), `internal/capability/manifest.go` (6 entries) +The 2026-07-04 CGNAT smoke test found the shipped interface MTU **1420** silently **black-holes +bulk TCP** on any path below ~1480 B (mobile ~1400, DS-Lite ~1452): the WG handshake and ping stay +healthy (small packets) while the PBS TLS page — and, at S4, the backup itself — drops. Resolved +`§4.3`'s OPEN DECISION with a **permanent, fleet-wide, family-agnostic client MTU of 1280** (RFC +8200 IPv6-minimum link MTU; outer = 1280+60 v4 / +80 v6, fits every realistic path). -## 3. Tests + the five §10 red-proof outcomes +- `internal/wgtunnel/manager.go`: new `const clientMTU = 1280` (single home, doc-referenced); + `renderConf` emits `MTU = %d` from it. +- **Client-only by construction:** interface MTU caps box→PBS; advertised MSS (=MTU−40) caps + PBS→box. The endpoint's `wg0` is **untouched** → zero live-endpoint risk, one constant changed. +- Rejected (recorded in §4.3): auto-probe / per-connection-type (fragile moving part for a + non-metric); MSS-clamp (no forwarded flows). No wire/JSON-golden change (MTU is client-derived). -`go build ./... && go vet ./... && go test ./...` — all green (20 pkgs; 18 with tests). -`visudo -cf` on felhom-pve: `parsed OK` (gated BEFORE install; binary restarted after). +## 3. Tests — render golden + red-proof (§10) -Red-proofs, each run → FAILED as required → reverted: -- (a) marker gate dropped → `re-registration after revocation: 5 calls` -- (b) hash comparison dropped → `steady-state tick ran execs: [install…, systemctl restart…]` -- (c) clamp removed → **honest adjustment discovered mid-proof:** x/crypto's X25519 clamps the - scalar internally (RFC 7748), so the DERIVED pubkey is clamp-invariant and the spec's - fixed-vector red-proof cannot fail. The clamp's real observable property is the **stored key - file's canonical form** (external `wg pubkey < private.key` must agree with what the agent - registered) — asserted by `TestEnsureKey_StoredKeyIsClamped`, which DOES fail with the clamp - removed (`stored key is not clamped: byte0=10101010`). Both pubkey vectors were generated with - the real `wg pubkey` (provenance in the test). -- (d) teardown-on-absent-data → `absent-data tick ran execs: [systemctl disable…]×3` -- (e) escrow auto-inject removed → `bundle key = ""` -Plus: the log-scan test greps the captured buffer for the actual private key (padded + unpadded) -— absent; the pubkey IS logged (intended). +- `go build ./... && go vet ./... && go test ./...` — **all green** (local Go 1.26). +- `TestRenderConf_Golden` pins the exact bytes incl. `MTU = 1280` (not a "contains MTU" check). +- **Red-proof:** flipped `const clientMTU` → 1420 → the golden **FAILED** on the MTU line + (`got MTU = 1420 / want MTU = 1280`); restored to 1280 → **PASS**. The assertion is non-vacuous. -## 4. Deploy + live validation (felhom-pve; journal excerpts verbatim; keys never logged by construction — the session-log grep for the key material was run and clean) +## 4. Deploy + live wired smoke (§13) — felhom-pve, agent v0.65.0 -Deploy order per spec: sudoers first (`visudo -cf` → install → `parsed OK`), binary -0.63.0→0.64.0 (backup kept), config: `wg_tunnel.enabled=true, interval_seconds=60`. +Built `-X main.version=0.65.0` on 180; deployed with `felhom-agent.bak-0.64.0` backup; `--version` +confirms **0.65.0**. Agent re-rendered + re-applied the conf on restart (hash-gated apply caught +the MTU change): -- **Poll-interval lever (the spec's step-0 diagnosis, learned live):** agent-side - `poll_seconds=120` was adopted for exactly ONE cycle — the hub envelope's - `poll_interval_seconds` is a hub-side CONSTANT (`defaultHostPollSeconds = 900`, - `hub/internal/api/handler.go`) silently adopted on the first cycle (the "changed" log fires - only on later cycles). Hub code is out of S3 scope, so validation ran on 900 s beats with - background watchers. Agent config restored to `poll_seconds=900` + the diagnosis SIGQUIT - restart documented below. **Observation for the hub backlog: make the constant configurable.** -- **Step 1 — bring-up (one shot, 3 s):** - `07:15:48 wgtunnel: generated new WG keypair pubkey=yV5hFF…TCg=` → - `07:15:49 wgtunnel: registered with hub … assigned_ip=10.77.0.2/32 existed=false generation=5 sync=ok` → - `07:15:49 desired: updated from hub generation=5` → - `07:15:50 wgtunnel: tunnel conf applied endpoint=ep0.felhom.eu:443 … action=enable`. - Perms verified: key 0600 in 0700 dir (felhom-agent), installed conf 0600 root. -- **Step 2 — data path:** `wg-quick@wg-felhom` active; in-tunnel ping 10.77.0.1 0% loss ~32 ms; - `curl https://10.77.0.1:8007/` = PBS login page. Hub `/admin/wg/peers` shows the peer BOUND to - `demo-felhom-01`. Report stanza in the hub's stored report_json: - `{"pubkey":"yV5hFF…","registered":true,"active":true,"last_handshake_age_s":28,"assigned_ip":"10.77.0.2/32"}`. - All 6 `wg-*` capability entries probe `status:ok`. -- **Agent-restart tolerance (bonus, via the diagnosis SIGQUIT):** on restart — key kept, NO - re-registration (marker gate), tunnel stayed active, desired refetched clean. -- **Step 3 — host reboot:** `wg-quick@wg-felhom` active from boot (unit persistence, before/without - the agent), tunnel pings, guest 9201 running, agent's first tick ran ZERO wgtunnel execs. -- **Step 4 — revocation drill:** admin DELETE 05:57:30Z → next poll - `08:10:52 desired: updated generation=6` + - `WARN wgtunnel: … hub revoked this peer; disabling the tunnel (marker kept; NO re-registration)` - → service inactive, marker present. Over the following 2 polls: **0** `wgtunnel: registered` - lines, **0** wg-related sudo execs (the load-bearing negatives — verbatim counts in §5). - Re-add (via the REGISTRATION endpoint — see §5's path insight) → next poll: - `08:55:52 desired: updated generation=8` + `wgtunnel: tunnel conf applied … action=enable`, - tunnel pings, **0 registration calls since the re-add** — same key, same /32, marker untouched. - Config lever restored (`poll_seconds` back to 900; restart clean — no keygen/register lines). -- **Step 5 — escrow:** `--selftest=escrow-create -storage felhom-pbs` (as root — the PBS key file - is root-readable only, the documented BUNDLE-slice reality) → - `escrow: identity bundle: +wg_private_key` (field name only), `identity=true`, blob 450 B - age-wrapped, self-verify OK. **Deviation from the spec's "blob uploaded":** run WITHOUT - `--upload` — uploading would overwrite the operator's real drill escrow with one whose R was - displayed into THIS session log (R was sed-redacted, but an unrecorded-R escrow is - unrecoverable and clobbering the prior blob breaks the operator's held R). The upload path is - pre-S3 code, unchanged by this slice — no coverage lost. -- **Step 6 — soak:** 30-min idle → handshake age **104 s** (<180 s; keepalive holding), active. +1. **Rendered MTU:** `grep MTU /etc/wireguard/wg-felhom.conf` → `MTU = 1280`; live iface + `ip link show wg-felhom` → `mtu 1280`. ✅ +2. **Tunnel:** `wg-quick@wg-felhom` active; handshake age 43 s; `ping 10.77.0.1` 3/3, ~33 ms. ✅ +3. **PBS page (no regression on the good path):** `curl -sk https://10.77.0.1:8007/` → full + `…` login page at MTU 1280. ✅ +4. Journal clean — no capability degradation, no errors. -## 5. Revocation-drill negatives (verbatim) +Scenario B (no regression at 1280 on the working wired path) **proven live**. The constrained-path +fix itself is established by arithmetic + prior field data (1280 < the 1340 that already worked in +the smoke test; 1420 was the value that failed) — the optional phone re-tether re-proof was **not +run** (not required per §13; skippable). -Over the 31 minutes following the teardown (05:57:30Z DELETE → 06:10:52Z teardown → checked -06:42Z), with **2 heartbeats landed hub-side** in the window (count from `host_reports`): +## 5. Not yet live-validated / open -``` -'wgtunnel: registered' lines since teardown: 0 -'desired: updated' lines since teardown: 0 ← correct: generation never advanced (nothing - changed hub-side), so heartbeats were cached - no-ops — the polls happened, the fetch didn't -wg-related sudo execs since teardown: 0 -wg-quick@wg-felhom: inactive (stayed down) -``` +- **True-CGNAT-SIM retest** — low risk, deferred ("when able"); the smoke SIM had a public mobile + IPv4, so 100.64/10 double-NAT was not reproduced (unchanged by this task). +- **PBS backup over the tunnel (S4)** — the next slice; this fix makes its bulk-TCP floor safe. +- **Optional constrained-path re-proof** (phone tether at MTU 1280) — skipped; available if extra + confidence wanted. -**Re-add path insight (found live, load-bearing for operator docs):** re-adding via the S1 -registry endpoint (`POST /admin/wg/peers` with `host_id`) binds the peer but does NOT bump the -host's generation — the agent never learns the block returned. The correct operator re-add is -the S2 **registration endpoint with the global key** (`POST /hosts/{id}/wg`), which allocates, -bumps (gen 8 in the drill), and syncs in one step. The drill was redone through it. Backlog -note for S6: either make the admin registry add bump the bound owner (symmetric with the S2 -delete-bump) or mark it registry-plumbing in the UI. +## 6. Observations -## 6. NOT yet live-validated — awaiting later work - -- **CGNAT/mobile-hotspot smoke** — the operator-assisted appendix (physical hotspot needed); - deferred non-blocking per spec. On failure it re-opens the DERP/Headscale fallback, not S3's code. -- PBS backup over the tunnel (**S4** — this slice deliberately left backup config untouched). -- DR consume of the escrowed WG key (**S5**). -- Tunnel-health surfacing/alerting in the hub UI (**S6** — the stanza ships now). -- Endpoint re-IP re-resolve watchdog beyond restart-time resolution (S6, noted in 06-doc S3 row). -- Gitea publish + Day-0 manifest vouch of 0.64.0 (**operator follow-ups** — publish-agent.sh + - the password-gated UI). - -## 7. Observations (noticed, not acted on) - -- The hub's `defaultHostPollSeconds = 900` constant silently overrides any agent-side cadence on - cycle 1 with no log line (first-cycle adoption is unlogged) — worth a hub-side config knob + - an INFO log for the first adoption. -- The pre-existing `pbs: cannot read token secret` WARN (non-root vs `/etc/pve/priv/...`) still - fires each cycle — S4's tunnel-target PBS with per-customer tokens resolves it structurally. -- `git add -A` hygiene: the sudoers deploy leaves `/root/felhom-agent.sudoers.bak-pre064` + - `agent.json.bak-pre064` on felhom-pve (intentional rollback artifacts, listed for the operator). +- The v4-pin (agent resolves ep0's A record + writes the literal, per §4.2's watchdog) is a + reasonable **later determinism** improvement but is **not needed for MTU correctness** — 1280 is + safe on both families' overhead. Recorded in the doc as a separate future note; green-lightable + independently whenever the re-resolve watchdog work happens. +- OPERATOR follow-up (unchanged from S3): publish 0.65.0 to Gitea + vouch in the Day-0 manifest + (password-gated UI — CC cannot). Rollback artifact now on the box: `felhom-agent.bak-0.64.0`.