diff --git a/documentation/tests/unattended-test-campaign-2026-06-22-8443-diagnosis.md b/documentation/tests/unattended-test-campaign-2026-06-22-8443-diagnosis.md new file mode 100644 index 0000000..d7a7a55 --- /dev/null +++ b/documentation/tests/unattended-test-campaign-2026-06-22-8443-diagnosis.md @@ -0,0 +1,165 @@ +# 8443 diagnosis — controller→agent local-API unreachability ROOT CAUSE — 2026-06-22 + +> **Status: read-only diagnosis (no fix applied).** Follow-up to +> `unattended-test-campaign-2026-06-22-findings.md` finding #1. The task rejected the campaign's +> "rebind agent to 0.0.0.0" guess (the bridge-IP bind is deliberate defense-in-depth) and asked to +> establish the real cause from evidence. **It is established and verified.** + +## TL;DR verdict + +**Root cause = a connection/`http.Transport` leak in the *controller* (client side), not a network, +firewall, agent-bind, or endpoint problem.** `felhom-controller`'s `agentClient()` +(`internal/web/agent_disk_handlers.go:43`) calls `agentapi.New(...)` **on every agent API call**, and +`agentapi.New` (`internal/agentapi/client.go:83`) builds a **fresh `&http.Client{Transport: +&http.Transport{TLSClientConfig: …}}`** each time. That bare Transport has **`IdleConnTimeout: 0`** +(idle keep-alive connections never expire) and is discarded after the single call **without +`CloseIdleConnections()`**. Because the agent keeps connections alive, every call **leaks one idle +ESTABLISHED socket** to `192.168.0.162:8443`. They accumulate monotonically over controller uptime +until the **ephemeral source-port range** for the `(container → 192.168.0.162:8443)` tuple is +exhausted → `connect: cannot assign requested address` (**EADDRNOTAVAIL**). + +This is **H5 (ephemeral-port / socket exhaustion)**, with a precise code cause. **H1, H2, H3, H4, H6 +are ruled out with positive evidence below.** + +Why `:8006` always worked and `:8443` "failed": the controller **only ever dials `:8443`** (the agent). +It never connects to pveproxy `:8006`, so that destination never leaks and never exhausts. The failure +was never port-specific in the network sense — it was **destination-specific exhaustion** driven by the +leak. + +Why it "broke persistently" in the campaign but **works now**: the leak accumulates with controller +**uptime**. The campaign hit it at the controller's ~5-day uptime (pool exhausted). The Phase-7 guest +reboot restarted the controller with a fresh socket table → it works again, and is **already +re-accumulating** (measured below). This is an accumulating-state regression cleared by restart, NOT a +static misconfiguration — exactly why the campaign's static theory (agent bind) didn't fit its own data. + +--- + +## Evidence + +### Step 1 — agent listener ground truth (host) +``` +ss -tlnp 'sport = :8443' → LISTEN 192.168.0.162:8443 users:(("felhom-agent",pid=315505,fd=11)) +ss -tlnp 'sport = :8006' → LISTEN *:8006 users:(("pveproxy"…)) +ip -br addr → vmbr0 UP 192.168.0.162/24 ; veth9201i0@if2 UP ; (no IPv6 global) +``` +Agent = a single **IPv4** socket bound to `192.168.0.162` (vmbr0), one process. pveproxy binds wildcard +`*:8006`. Healthy listener. (The bind-to-bridge-IP is the intended defense-in-depth, and is fine.) + +### Step 2 — controller configured endpoint (redacted) +`bootstrap.json local_api`: `endpoint: 192.168.0.162:8443`, `token: «REDACTED»`, +`fingerprint: «REDACTED»`. The endpoint **is** the vmbr0 bridge IP + correct port — matches the error +string. **H2 (endpoint mismatch) ruled out.** + +### Step 3 — connect matrix (the localizer) — ALL SUCCEED post-reboot +Raw `socket.connect()` (AF_INET) to `192.168.0.162`, exact errno per layer × port: + +| layer | :8443 | :8006 | +|---|---|---| +| controller **container netns** (pid 1917) | **OK** (local 172.17.0.2) | OK | +| guest root netns | OK (local 192.168.0.141) | OK | +| host | OK (local 192.168.0.162) | OK | + +`nsenter -t 1917 -n ip route get 192.168.0.162` → `via 172.17.0.1 dev eth0 src 172.17.0.2 … cache` +(resolves cleanly, source assignable). Container has eth0 `172.17.0.2/16` + eth1 `172.18.0.8/16`, no +global IPv6. **At low uptime the TCP path is fully healthy — H3 (route/source) and H4 (address-family) +ruled out.** Live re-test of the real app path also succeeds now: controller `/api/disks` returns the +full disk JSON; in-container `curl https://192.168.0.162:8443/` → **404** (TCP+TLS OK), `:8006` → 200. +**H6 (agent socket truth) ruled out** — agent accepts past TCP+TLS. + +### Step 3b/core — the leak (the decisive artifact) +Controller container, **47 minutes** after the Phase-7 reboot: +``` +ss -s (container netns): TCP estab 194, closed 1049, timewait 9 +estab → 192.168.0.162:8443: 194 (essentially ALL established sockets go to the agent) +ip_local_port_range: 32768 60999 (= 28231 ephemeral ports) +``` +Monotonic growth (same netns, 30 s apart): **196 → 198**, later samples **206**. Rate ≈ 4/min ≈ +**~5 800/day**. Two-sided + idle confirmation: +``` +HOST (agent) side, estab on :8443 from 192.168.0.141 (guest): 206 (Recv-Q/Send-Q = 0 → idle) +controller netns estab→8443: 206 +controller process open fds: 217 (206 = leaked agent sockets) +``` +**Projection:** ~5 800 leaked sockets/day ÷ 28 231 ports ⇒ exhaustion in **~4–5 days** of controller +uptime → EADDRNOTAVAIL. Matches the campaign hitting it at ~5-day uptime exactly. + +### Step 4 — guest-side nat/filter (H1, the campaign's prime suspect) — RULED OUT +Inside guest 9201: +``` +iptables-save | grep -E '8443|8006' → NONE (no port-specific rule) +filter REJECT/DROP → only standard docker inter-bridge isolation + (-A DOCKER ! -i br-X -o br-X -j DROP), not outbound-to-LAN +docker subnets → 172.17/18/19/20/21.0.0/16 — none overlap 192.168.0.0/24 +``` +No rule treats 8443 differently from 8006; no subnet overlap. **H1 ruled out** (and the path works now, +which alone disproves a static guest block). + +### Step 5 — tcpdump death-point — N/A at current uptime +The failure is **not reproducible at low uptime** (the path works — the SYN leaves and connects). A +capture now would only show successful handshakes. The "death point" only appears once the ephemeral +pool is exhausted, at which point `connect()` fails **locally** (no SYN emitted) — consistent with the +EADDRNOTAVAIL semantics. No capture taken (nothing to capture); the leak measurement is the decisive +artifact instead. + +### Step 6 — temporal verdict +**Regression that accumulates with controller uptime; reset by restart.** Definitive evidence: failed +persistently during the campaign at the controller's ~5-day uptime (6/6 EADDRNOTAVAIL); after the +Phase-7 guest reboot (controller `Up 47 minutes`) the path works and the leak is at 194→206 and +climbing. The current container's logs only span 47 min (post-reboot), so the pre-reboot failures are +gone with the old container — but the accumulate→reset mechanism is directly measured, not inferred. It +"works after every restart, breaks after ~5 days." + +### Step 7 — agent socket accept — confirmed reachable +`curl -vk https://192.168.0.162:8443/` from in-container connects at TCP+TLS and returns HTTP 404; the +pinned-TLS app path (`/api/disks`) returns real data. The agent is NOT the problem. + +--- + +## Hypothesis scorecard + +| | Hypothesis | Verdict | Deciding evidence | +|---|---|---|---| +| H1 | guest-side docker/nat block on 8443 | ❌ ruled out | no 8443/8006 rule; path works now | +| H2 | configured-endpoint mismatch | ❌ ruled out | endpoint = `192.168.0.162:8443` (the bridge IP) | +| H3 | source/route selection failure | ❌ ruled out | `ip route get` resolves, src assignable, raw connect OK | +| H4 | address-family / IPv6 | ❌ ruled out | agent + container are IPv4-only; no global v6 | +| **H5** | **ephemeral-port / socket exhaustion** | ✅ **CONFIRMED** | 206 leaked idle ESTABLISHED→8443 in 47 min, growing ~5.8k/day vs 28 231 ports ⇒ exhausts in ~5 days; matches campaign timing | +| H6 | agent listener truth | ❌ ruled out | single healthy v4 socket; accepts TCP+TLS; returns data | + +**Precise mechanism within H5:** `agentClient()` → `agentapi.New()` per call → new `http.Transport` +with `IdleConnTimeout: 0`, discarded without `CloseIdleConnections()`; agent keep-alive ⇒ one leaked +idle ESTABLISHED socket per call. + +--- + +## Proposed fix direction (controller-side; NOT an agent rebind) + +The agent's bind to `192.168.0.162` (bridge IP) is correct and must stay. The fix is entirely in the +controller's client lifecycle: + +1. **Build the `agentapi.Client` once and reuse it** (it is stateless config — endpoint/token/ + fingerprint, all from `cfg.LocalAPI`). Construct it at server init, store it on `Server`, and have + `agentClient()` return the shared instance. A single reused Transport pools/reuses connections (≈2 + idle conns), eliminating the leak. **Preferred.** +2. **If a per-call client is kept for any reason**, the Transport must be tamed and closed: set + `IdleConnTimeout` (e.g. 30–90 s) + `MaxIdleConnsPerHost`, and `defer client.CloseIdleConnections()` + after use. (Reuse (#1) is cleaner and also avoids the per-call TLS handshake cost.) +3. Optionally add `MaxConnsPerHost`/keep-alive tuning on the shared Transport as belt-and-suspenders. + +These are pure `felhom-controller` changes (then build + bump + redeploy per the controller workflow); +no agent or firewall change. + +### Separate hardening gap (not the cause — note for later) +The defense-in-depth control the agent's own config comment calls for — *"a host firewall rule should +limit [8443] to the guest bridge subnet"* — is **absent**: `pve-firewall` is disabled and there is no +iptables rule scoping 8443. Close this once connectivity is fixed (a host rule allowing +`192.168.0.0/24`→`:8443` and dropping others), independent of the leak fix. + +--- + +## Note on the campaign finding #1 +`unattended-test-campaign-2026-06-22-findings.md` finding #1 correctly **observed** the symptom and +**correctly rejected** rebinding to 0.0.0.0 in this follow-up, but its first-pass theory (agent +bound to a specific IP ⇒ unreachable) was wrong — disproved by `:8006` reachability to the same IP and +by the path working at low uptime. The real cause is the controller connection leak above. No code or +config was changed by this diagnosis.