docs: REPORT + CONTEXT for S3.1 MTU 1280 (v0.65.0 live on felhom-pve)
Deploy + wired smoke recorded: agent re-rendered conf on restart, live iface MTU 1280, PBS page loads (no regression). Red-proof outcome recorded. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
+12
@@ -5,6 +5,18 @@
|
|||||||
|
|
||||||
## Current
|
## Current
|
||||||
|
|
||||||
|
- **v0.65.0** (2026-07-04, live on felhom-pve) — **S3.1 offsite-tunnel client MTU 1420 → 1280**:
|
||||||
|
resolves `06 §4.3`'s OPEN DECISION left by the CGNAT smoke test. 1420 **silently black-holed bulk
|
||||||
|
TCP** on sub-~1480 paths (mobile ~1400, DS-Lite ~1452) — handshake+ping healthy, PBS TLS page
|
||||||
|
(and at S4 the backup itself) drops. New `const clientMTU = 1280` (RFC 8200 IPv6-minimum floor;
|
||||||
|
outer 1340 v4 / 1360 v6 fits every realistic path), **permanent + fleet-wide + family-agnostic**.
|
||||||
|
**Client-only by construction** — interface MTU caps box→PBS, advertised MSS caps PBS→box, so the
|
||||||
|
endpoint's `wg0` is untouched (zero live-endpoint risk). Golden pins exact `MTU = 1280`
|
||||||
|
(red-proofed vs a 1420 flip); no wire/JSON change. Live: agent re-rendered on restart (hash-gated
|
||||||
|
apply), conf + live iface both 1280, PBS page loads at 1280 (no regression on wired). OPEN:
|
||||||
|
true-CGNAT-SIM retest (low risk); publish 0.65.0 + Day-0 vouch (operator); S4 PBS-over-tunnel.
|
||||||
|
Rollback: `felhom-agent.bak-0.64.0` on the box. The v4-pin (§4.2 determinism) is a separate,
|
||||||
|
optional future note — NOT needed for MTU correctness.
|
||||||
- **v0.64.0** (2026-07-04, live on felhom-pve) — **S3 offsite WG tunnel**: new `internal/wgtunnel`
|
- **v0.64.0** (2026-07-04, live on felhom-pve) — **S3 offsite WG tunnel**: new `internal/wgtunnel`
|
||||||
(keygen 0600/0700, marker-gated one-shot registration, agent-managed `wg-quick@wg-felhom` from
|
(keygen 0600/0700, marker-gated one-shot registration, agent-managed `wg-quick@wg-felhom` from
|
||||||
the hub's desired-state `wireguard` block via the new `desired.Syncer.AddConsumer` seam,
|
the hub's desired-state `wireguard` block via the new `desired.Syncer.AddConsumer` seam,
|
||||||
|
|||||||
@@ -1,135 +1,71 @@
|
|||||||
# REPORT — S3: agent WG tunnel — keygen + registration + wg-quick@wg-felhom + escrow join (v0.64.0)
|
# REPORT — S3.1: offsite-tunnel client MTU 1420 → 1280 (v0.65.0)
|
||||||
|
|
||||||
**Date:** 2026-07-04 · **Class:** implementation + live validation on felhom-pve (the expendable
|
**Date:** 2026-07-04 · **Class:** implementation + live validation on felhom-pve (the expendable
|
||||||
demo box; Peti's box untouched by construction — `wg_tunnel.enabled` defaults FALSE). **Design of
|
demo box; Peti's box untouched; endpoint untouched by construction). **Design of record:**
|
||||||
record:** `felhom.eu/documentation/architecture/06-offsite-connectivity.md` row S3 (SHIPPED).
|
`felhom.eu/documentation/architecture/06-offsite-connectivity.md` §4.3 (OPEN DECISION → resolved).
|
||||||
|
|
||||||
## 1. Baselines & commits
|
## 1. Baselines & commits
|
||||||
|
|
||||||
Start: felhom-agent @ `4ba1b14` (v0.63.0); felhom.eu @ `4fe895e` (hub v0.33.0, code read-only).
|
Start: felhom-agent @ `03edb40` (v0.64.0); felhom.eu @ `2028785` (docs). Commits pushed to `main`:
|
||||||
Commits pushed to `main`:
|
|
||||||
|
|
||||||
1. `0daae92` wgtunnel keygen + hub wire (WireWireguard/report stanza/RegisterWG) + byte-identical golden copy
|
- **`312fd5e`** — `wgtunnel: client MTU 1420 → 1280` (manager.go const + render, manager_test.go
|
||||||
2. `fb24896` manager state machine + loop + desired raw-consumer seam
|
golden + red-proof note, report.go stale comment, CHANGELOG v0.65.0).
|
||||||
3. `e2b6c63` FELHOM_WG sudoers + capabilities + config (DEFAULT OFF) + main wiring + escrow join
|
- **`<this commit>`** — CONTEXT + this REPORT.
|
||||||
4. (docs commit) CHANGELOG v0.64.0 + CONTEXT + this REPORT
|
- felhom.eu docs (§4.3 decision + §4.2/§4.3 wording + S6 row): separate one commit — see felhom.eu REPORT.
|
||||||
|
|
||||||
## 2. Files
|
## 2. What changed & why
|
||||||
|
|
||||||
- NEW `internal/wgtunnel/{key,manager,loop}.go` + tests (key vectors, state-machine Groups A/B)
|
The 2026-07-04 CGNAT smoke test found the shipped interface MTU **1420** silently **black-holes
|
||||||
- `internal/hub/{report,client,collect}.go` (+`wg_contract_test.go`, `wg_client_test.go`,
|
bulk TCP** on any path below ~1480 B (mobile ~1400, DS-Lite ~1452): the WG handshake and ping stay
|
||||||
NEW `testdata/desired-state-wireguard.golden.json` — **byte-identical hub copy**, diff-verified)
|
healthy (small packets) while the PBS TLS page — and, at S4, the backup itself — drops. Resolved
|
||||||
- `internal/desired/syncer.go` (`AddConsumer` + panic containment) + `consumer_test.go`
|
`§4.3`'s OPEN DECISION with a **permanent, fleet-wide, family-agnostic client MTU of 1280** (RFC
|
||||||
- `internal/escrow/identity.go` (`WGPrivateKey` + `AttachWGKey`) + `wgkey_test.go`
|
8200 IPv6-minimum link MTU; outer = 1280+60 v4 / +80 v6, fits every realistic path).
|
||||||
- `internal/config/config.go` (`WGTunnelConfig`, **Enabled default FALSE**)
|
|
||||||
- `cmd/felhom-agent/main.go` (daemon wiring, escrow-create auto-inject, `--selftest=wgtunnel`)
|
|
||||||
- `configs/felhom-agent.sudoers` (`Cmnd_Alias FELHOM_WG`), `internal/capability/manifest.go` (6 entries)
|
|
||||||
|
|
||||||
## 3. Tests + the five §10 red-proof outcomes
|
- `internal/wgtunnel/manager.go`: new `const clientMTU = 1280` (single home, doc-referenced);
|
||||||
|
`renderConf` emits `MTU = %d` from it.
|
||||||
|
- **Client-only by construction:** interface MTU caps box→PBS; advertised MSS (=MTU−40) caps
|
||||||
|
PBS→box. The endpoint's `wg0` is **untouched** → zero live-endpoint risk, one constant changed.
|
||||||
|
- Rejected (recorded in §4.3): auto-probe / per-connection-type (fragile moving part for a
|
||||||
|
non-metric); MSS-clamp (no forwarded flows). No wire/JSON-golden change (MTU is client-derived).
|
||||||
|
|
||||||
`go build ./... && go vet ./... && go test ./...` — all green (20 pkgs; 18 with tests).
|
## 3. Tests — render golden + red-proof (§10)
|
||||||
`visudo -cf` on felhom-pve: `parsed OK` (gated BEFORE install; binary restarted after).
|
|
||||||
|
|
||||||
Red-proofs, each run → FAILED as required → reverted:
|
- `go build ./... && go vet ./... && go test ./...` — **all green** (local Go 1.26).
|
||||||
- (a) marker gate dropped → `re-registration after revocation: 5 calls`
|
- `TestRenderConf_Golden` pins the exact bytes incl. `MTU = 1280` (not a "contains MTU" check).
|
||||||
- (b) hash comparison dropped → `steady-state tick ran execs: [install…, systemctl restart…]`
|
- **Red-proof:** flipped `const clientMTU` → 1420 → the golden **FAILED** on the MTU line
|
||||||
- (c) clamp removed → **honest adjustment discovered mid-proof:** x/crypto's X25519 clamps the
|
(`got MTU = 1420 / want MTU = 1280`); restored to 1280 → **PASS**. The assertion is non-vacuous.
|
||||||
scalar internally (RFC 7748), so the DERIVED pubkey is clamp-invariant and the spec's
|
|
||||||
fixed-vector red-proof cannot fail. The clamp's real observable property is the **stored key
|
|
||||||
file's canonical form** (external `wg pubkey < private.key` must agree with what the agent
|
|
||||||
registered) — asserted by `TestEnsureKey_StoredKeyIsClamped`, which DOES fail with the clamp
|
|
||||||
removed (`stored key is not clamped: byte0=10101010`). Both pubkey vectors were generated with
|
|
||||||
the real `wg pubkey` (provenance in the test).
|
|
||||||
- (d) teardown-on-absent-data → `absent-data tick ran execs: [systemctl disable…]×3`
|
|
||||||
- (e) escrow auto-inject removed → `bundle key = ""`
|
|
||||||
Plus: the log-scan test greps the captured buffer for the actual private key (padded + unpadded)
|
|
||||||
— absent; the pubkey IS logged (intended).
|
|
||||||
|
|
||||||
## 4. Deploy + live validation (felhom-pve; journal excerpts verbatim; keys never logged by construction — the session-log grep for the key material was run and clean)
|
## 4. Deploy + live wired smoke (§13) — felhom-pve, agent v0.65.0
|
||||||
|
|
||||||
Deploy order per spec: sudoers first (`visudo -cf` → install → `parsed OK`), binary
|
Built `-X main.version=0.65.0` on 180; deployed with `felhom-agent.bak-0.64.0` backup; `--version`
|
||||||
0.63.0→0.64.0 (backup kept), config: `wg_tunnel.enabled=true, interval_seconds=60`.
|
confirms **0.65.0**. Agent re-rendered + re-applied the conf on restart (hash-gated apply caught
|
||||||
|
the MTU change):
|
||||||
|
|
||||||
- **Poll-interval lever (the spec's step-0 diagnosis, learned live):** agent-side
|
1. **Rendered MTU:** `grep MTU /etc/wireguard/wg-felhom.conf` → `MTU = 1280`; live iface
|
||||||
`poll_seconds=120` was adopted for exactly ONE cycle — the hub envelope's
|
`ip link show wg-felhom` → `mtu 1280`. ✅
|
||||||
`poll_interval_seconds` is a hub-side CONSTANT (`defaultHostPollSeconds = 900`,
|
2. **Tunnel:** `wg-quick@wg-felhom` active; handshake age 43 s; `ping 10.77.0.1` 3/3, ~33 ms. ✅
|
||||||
`hub/internal/api/handler.go`) silently adopted on the first cycle (the "changed" log fires
|
3. **PBS page (no regression on the good path):** `curl -sk https://10.77.0.1:8007/` → full
|
||||||
only on later cycles). Hub code is out of S3 scope, so validation ran on 900 s beats with
|
`<!DOCTYPE html>…` login page at MTU 1280. ✅
|
||||||
background watchers. Agent config restored to `poll_seconds=900` + the diagnosis SIGQUIT
|
4. Journal clean — no capability degradation, no errors.
|
||||||
restart documented below. **Observation for the hub backlog: make the constant configurable.**
|
|
||||||
- **Step 1 — bring-up (one shot, 3 s):**
|
|
||||||
`07:15:48 wgtunnel: generated new WG keypair pubkey=yV5hFF…TCg=` →
|
|
||||||
`07:15:49 wgtunnel: registered with hub … assigned_ip=10.77.0.2/32 existed=false generation=5 sync=ok` →
|
|
||||||
`07:15:49 desired: updated from hub generation=5` →
|
|
||||||
`07:15:50 wgtunnel: tunnel conf applied endpoint=ep0.felhom.eu:443 … action=enable`.
|
|
||||||
Perms verified: key 0600 in 0700 dir (felhom-agent), installed conf 0600 root.
|
|
||||||
- **Step 2 — data path:** `wg-quick@wg-felhom` active; in-tunnel ping 10.77.0.1 0% loss ~32 ms;
|
|
||||||
`curl https://10.77.0.1:8007/` = PBS login page. Hub `/admin/wg/peers` shows the peer BOUND to
|
|
||||||
`demo-felhom-01`. Report stanza in the hub's stored report_json:
|
|
||||||
`{"pubkey":"yV5hFF…","registered":true,"active":true,"last_handshake_age_s":28,"assigned_ip":"10.77.0.2/32"}`.
|
|
||||||
All 6 `wg-*` capability entries probe `status:ok`.
|
|
||||||
- **Agent-restart tolerance (bonus, via the diagnosis SIGQUIT):** on restart — key kept, NO
|
|
||||||
re-registration (marker gate), tunnel stayed active, desired refetched clean.
|
|
||||||
- **Step 3 — host reboot:** `wg-quick@wg-felhom` active from boot (unit persistence, before/without
|
|
||||||
the agent), tunnel pings, guest 9201 running, agent's first tick ran ZERO wgtunnel execs.
|
|
||||||
- **Step 4 — revocation drill:** admin DELETE 05:57:30Z → next poll
|
|
||||||
`08:10:52 desired: updated generation=6` +
|
|
||||||
`WARN wgtunnel: … hub revoked this peer; disabling the tunnel (marker kept; NO re-registration)`
|
|
||||||
→ service inactive, marker present. Over the following 2 polls: **0** `wgtunnel: registered`
|
|
||||||
lines, **0** wg-related sudo execs (the load-bearing negatives — verbatim counts in §5).
|
|
||||||
Re-add (via the REGISTRATION endpoint — see §5's path insight) → next poll:
|
|
||||||
`08:55:52 desired: updated generation=8` + `wgtunnel: tunnel conf applied … action=enable`,
|
|
||||||
tunnel pings, **0 registration calls since the re-add** — same key, same /32, marker untouched.
|
|
||||||
Config lever restored (`poll_seconds` back to 900; restart clean — no keygen/register lines).
|
|
||||||
- **Step 5 — escrow:** `--selftest=escrow-create -storage felhom-pbs` (as root — the PBS key file
|
|
||||||
is root-readable only, the documented BUNDLE-slice reality) →
|
|
||||||
`escrow: identity bundle: +wg_private_key` (field name only), `identity=true`, blob 450 B
|
|
||||||
age-wrapped, self-verify OK. **Deviation from the spec's "blob uploaded":** run WITHOUT
|
|
||||||
`--upload` — uploading would overwrite the operator's real drill escrow with one whose R was
|
|
||||||
displayed into THIS session log (R was sed-redacted, but an unrecorded-R escrow is
|
|
||||||
unrecoverable and clobbering the prior blob breaks the operator's held R). The upload path is
|
|
||||||
pre-S3 code, unchanged by this slice — no coverage lost.
|
|
||||||
- **Step 6 — soak:** 30-min idle → handshake age **104 s** (<180 s; keepalive holding), active.
|
|
||||||
|
|
||||||
## 5. Revocation-drill negatives (verbatim)
|
Scenario B (no regression at 1280 on the working wired path) **proven live**. The constrained-path
|
||||||
|
fix itself is established by arithmetic + prior field data (1280 < the 1340 that already worked in
|
||||||
|
the smoke test; 1420 was the value that failed) — the optional phone re-tether re-proof was **not
|
||||||
|
run** (not required per §13; skippable).
|
||||||
|
|
||||||
Over the 31 minutes following the teardown (05:57:30Z DELETE → 06:10:52Z teardown → checked
|
## 5. Not yet live-validated / open
|
||||||
06:42Z), with **2 heartbeats landed hub-side** in the window (count from `host_reports`):
|
|
||||||
|
|
||||||
```
|
- **True-CGNAT-SIM retest** — low risk, deferred ("when able"); the smoke SIM had a public mobile
|
||||||
'wgtunnel: registered' lines since teardown: 0
|
IPv4, so 100.64/10 double-NAT was not reproduced (unchanged by this task).
|
||||||
'desired: updated' lines since teardown: 0 ← correct: generation never advanced (nothing
|
- **PBS backup over the tunnel (S4)** — the next slice; this fix makes its bulk-TCP floor safe.
|
||||||
changed hub-side), so heartbeats were cached
|
- **Optional constrained-path re-proof** (phone tether at MTU 1280) — skipped; available if extra
|
||||||
no-ops — the polls happened, the fetch didn't
|
confidence wanted.
|
||||||
wg-related sudo execs since teardown: 0
|
|
||||||
wg-quick@wg-felhom: inactive (stayed down)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Re-add path insight (found live, load-bearing for operator docs):** re-adding via the S1
|
## 6. Observations
|
||||||
registry endpoint (`POST /admin/wg/peers` with `host_id`) binds the peer but does NOT bump the
|
|
||||||
host's generation — the agent never learns the block returned. The correct operator re-add is
|
|
||||||
the S2 **registration endpoint with the global key** (`POST /hosts/{id}/wg`), which allocates,
|
|
||||||
bumps (gen 8 in the drill), and syncs in one step. The drill was redone through it. Backlog
|
|
||||||
note for S6: either make the admin registry add bump the bound owner (symmetric with the S2
|
|
||||||
delete-bump) or mark it registry-plumbing in the UI.
|
|
||||||
|
|
||||||
## 6. NOT yet live-validated — awaiting later work
|
- The v4-pin (agent resolves ep0's A record + writes the literal, per §4.2's watchdog) is a
|
||||||
|
reasonable **later determinism** improvement but is **not needed for MTU correctness** — 1280 is
|
||||||
- **CGNAT/mobile-hotspot smoke** — the operator-assisted appendix (physical hotspot needed);
|
safe on both families' overhead. Recorded in the doc as a separate future note; green-lightable
|
||||||
deferred non-blocking per spec. On failure it re-opens the DERP/Headscale fallback, not S3's code.
|
independently whenever the re-resolve watchdog work happens.
|
||||||
- PBS backup over the tunnel (**S4** — this slice deliberately left backup config untouched).
|
- OPERATOR follow-up (unchanged from S3): publish 0.65.0 to Gitea + vouch in the Day-0 manifest
|
||||||
- DR consume of the escrowed WG key (**S5**).
|
(password-gated UI — CC cannot). Rollback artifact now on the box: `felhom-agent.bak-0.64.0`.
|
||||||
- Tunnel-health surfacing/alerting in the hub UI (**S6** — the stanza ships now).
|
|
||||||
- Endpoint re-IP re-resolve watchdog beyond restart-time resolution (S6, noted in 06-doc S3 row).
|
|
||||||
- Gitea publish + Day-0 manifest vouch of 0.64.0 (**operator follow-ups** — publish-agent.sh +
|
|
||||||
the password-gated UI).
|
|
||||||
|
|
||||||
## 7. Observations (noticed, not acted on)
|
|
||||||
|
|
||||||
- The hub's `defaultHostPollSeconds = 900` constant silently overrides any agent-side cadence on
|
|
||||||
cycle 1 with no log line (first-cycle adoption is unlogged) — worth a hub-side config knob +
|
|
||||||
an INFO log for the first adoption.
|
|
||||||
- The pre-existing `pbs: cannot read token secret` WARN (non-root vs `/etc/pve/priv/...`) still
|
|
||||||
fires each cycle — S4's tunnel-target PBS with per-customer tokens resolves it structurally.
|
|
||||||
- `git add -A` hygiene: the sudoers deploy leaves `/root/felhom-agent.sudoers.bak-pre064` +
|
|
||||||
`agent.json.bak-pre064` on felhom-pve (intentional rollback artifacts, listed for the operator).
|
|
||||||
|
|||||||
Reference in New Issue
Block a user