felhom.eu/documentation/tests/unattended-test-campaign-2026-06-22-8443-diagnosis.md

8443 diagnosis — controller→agent local-API unreachability ROOT CAUSE — 2026-06-22

Status: read-only diagnosis (no fix applied). Follow-up to unattended-test-campaign-2026-06-22-findings.md finding #1. The task rejected the campaign's "rebind agent to 0.0.0.0" guess (the bridge-IP bind is deliberate defense-in-depth) and asked to establish the real cause from evidence. It is established and verified.

TL;DR verdict

Root cause = a connection/http.Transport leak in the controller (client side), not a network, firewall, agent-bind, or endpoint problem. felhom-controller's agentClient() (internal/web/agent_disk_handlers.go:43) calls agentapi.New(...) on every agent API call, and agentapi.New (internal/agentapi/client.go:83) builds a fresh &http.Client{Transport: &http.Transport{TLSClientConfig: …}} each time. That bare Transport has IdleConnTimeout: 0 (idle keep-alive connections never expire) and is discarded after the single call without CloseIdleConnections(). Because the agent keeps connections alive, every call leaks one idle ESTABLISHED socket to 192.168.0.162:8443. They accumulate monotonically over controller uptime until the ephemeral source-port range for the (container → 192.168.0.162:8443) tuple is exhausted → connect: cannot assign requested address (EADDRNOTAVAIL).

This is H5 (ephemeral-port / socket exhaustion), with a precise code cause. H1, H2, H3, H4, H6 are ruled out with positive evidence below.

Why :8006 always worked and :8443 "failed": the controller only ever dials :8443 (the agent). It never connects to pveproxy :8006, so that destination never leaks and never exhausts. The failure was never port-specific in the network sense — it was destination-specific exhaustion driven by the leak.

Why it "broke persistently" in the campaign but works now: the leak accumulates with controller uptime. The campaign hit it at the controller's ~5-day uptime (pool exhausted). The Phase-7 guest reboot restarted the controller with a fresh socket table → it works again, and is already re-accumulating (measured below). This is an accumulating-state regression cleared by restart, NOT a static misconfiguration — exactly why the campaign's static theory (agent bind) didn't fit its own data.

---

Evidence

Step 1 — agent listener ground truth (host)

ss -tlnp 'sport = :8443'  → LISTEN 192.168.0.162:8443  users:(("felhom-agent",pid=315505,fd=11))
ss -tlnp 'sport = :8006'  → LISTEN *:8006               users:(("pveproxy"…))
ip -br addr               → vmbr0 UP 192.168.0.162/24 ; veth9201i0@if2 UP ; (no IPv6 global)

Agent = a single IPv4 socket bound to 192.168.0.162 (vmbr0), one process. pveproxy binds wildcard *:8006. Healthy listener. (The bind-to-bridge-IP is the intended defense-in-depth, and is fine.)

Step 2 — controller configured endpoint (redacted)

bootstrap.json local_api: endpoint: 192.168.0.162:8443, token: «REDACTED», fingerprint: «REDACTED». The endpoint is the vmbr0 bridge IP + correct port — matches the error string. H2 (endpoint mismatch) ruled out.

Step 3 — connect matrix (the localizer) — ALL SUCCEED post-reboot

Raw socket.connect() (AF_INET) to 192.168.0.162, exact errno per layer × port:

layer	:8443	:8006
controller container netns (pid 1917)	OK (local 172.17.0.2)	OK
guest root netns	OK (local 192.168.0.141)	OK
host	OK (local 192.168.0.162)	OK

nsenter -t 1917 -n ip route get 192.168.0.162 → via 172.17.0.1 dev eth0 src 172.17.0.2 … cache (resolves cleanly, source assignable). Container has eth0 172.17.0.2/16 + eth1 172.18.0.8/16, no global IPv6. At low uptime the TCP path is fully healthy — H3 (route/source) and H4 (address-family) ruled out. Live re-test of the real app path also succeeds now: controller /api/disks returns the full disk JSON; in-container curl https://192.168.0.162:8443/ → 404 (TCP+TLS OK), :8006 → 200. H6 (agent socket truth) ruled out — agent accepts past TCP+TLS.

Step 3b/core — the leak (the decisive artifact)

Controller container, 47 minutes after the Phase-7 reboot:

ss -s (container netns):  TCP estab 194, closed 1049, timewait 9
estab → 192.168.0.162:8443: 194  (essentially ALL established sockets go to the agent)
ip_local_port_range:       32768  60999   (= 28231 ephemeral ports)

Monotonic growth (same netns, 30 s apart): 196 → 198, later samples 206. Rate ≈ 4/min ≈ ~5 800/day. Two-sided + idle confirmation:

HOST (agent) side, estab on :8443 from 192.168.0.141 (guest):  206   (Recv-Q/Send-Q = 0 → idle)
controller netns estab→8443:                                    206
controller process open fds:                                    217   (206 = leaked agent sockets)

Projection: ~5 800 leaked sockets/day ÷ 28 231 ports ⇒ exhaustion in ~4–5 days of controller uptime → EADDRNOTAVAIL. Matches the campaign hitting it at ~5-day uptime exactly.

Step 4 — guest-side nat/filter (H1, the campaign's prime suspect) — RULED OUT

Inside guest 9201:

iptables-save | grep -E '8443|8006'  → NONE (no port-specific rule)
filter REJECT/DROP                   → only standard docker inter-bridge isolation
                                        (-A DOCKER ! -i br-X -o br-X -j DROP), not outbound-to-LAN
docker subnets                       → 172.17/18/19/20/21.0.0/16 — none overlap 192.168.0.0/24

No rule treats 8443 differently from 8006; no subnet overlap. H1 ruled out (and the path works now, which alone disproves a static guest block).

Step 5 — tcpdump death-point — N/A at current uptime

The failure is not reproducible at low uptime (the path works — the SYN leaves and connects). A capture now would only show successful handshakes. The "death point" only appears once the ephemeral pool is exhausted, at which point connect() fails locally (no SYN emitted) — consistent with the EADDRNOTAVAIL semantics. No capture taken (nothing to capture); the leak measurement is the decisive artifact instead.

Step 6 — temporal verdict

Regression that accumulates with controller uptime; reset by restart. Definitive evidence: failed persistently during the campaign at the controller's ~5-day uptime (6/6 EADDRNOTAVAIL); after the Phase-7 guest reboot (controller Up 47 minutes) the path works and the leak is at 194→206 and climbing. The current container's logs only span 47 min (post-reboot), so the pre-reboot failures are gone with the old container — but the accumulate→reset mechanism is directly measured, not inferred. It "works after every restart, breaks after ~5 days."

Step 7 — agent socket accept — confirmed reachable

curl -vk https://192.168.0.162:8443/ from in-container connects at TCP+TLS and returns HTTP 404; the pinned-TLS app path (/api/disks) returns real data. The agent is NOT the problem.

---

Hypothesis scorecard

	Hypothesis	Verdict	Deciding evidence
H1	guest-side docker/nat block on 8443	❌ ruled out	no 8443/8006 rule; path works now
H2	configured-endpoint mismatch	❌ ruled out	endpoint = 192.168.0.162:8443 (the bridge IP)
H3	source/route selection failure	❌ ruled out	ip route get resolves, src assignable, raw connect OK
H4	address-family / IPv6	❌ ruled out	agent + container are IPv4-only; no global v6
H5	ephemeral-port / socket exhaustion	✅ CONFIRMED	206 leaked idle ESTABLISHED→8443 in 47 min, growing ~5.8k/day vs 28 231 ports ⇒ exhausts in ~5 days; matches campaign timing
H6	agent listener truth	❌ ruled out	single healthy v4 socket; accepts TCP+TLS; returns data

Precise mechanism within H5: agentClient() → agentapi.New() per call → new http.Transport with IdleConnTimeout: 0, discarded without CloseIdleConnections(); agent keep-alive ⇒ one leaked idle ESTABLISHED socket per call.

---

Proposed fix direction (controller-side; NOT an agent rebind)

The agent's bind to 192.168.0.162 (bridge IP) is correct and must stay. The fix is entirely in the controller's client lifecycle:

1. Build the agentapi.Client once and reuse it (it is stateless config — endpoint/token/ fingerprint, all from cfg.LocalAPI). Construct it at server init, store it on Server, and have agentClient() return the shared instance. A single reused Transport pools/reuses connections (≈2 idle conns), eliminating the leak. Preferred.
2. If a per-call client is kept for any reason, the Transport must be tamed and closed: set IdleConnTimeout (e.g. 30–90 s) + MaxIdleConnsPerHost, and defer client.CloseIdleConnections() after use. (Reuse (#1) is cleaner and also avoids the per-call TLS handshake cost.)
3. Optionally add MaxConnsPerHost/keep-alive tuning on the shared Transport as belt-and-suspenders.

These are pure felhom-controller changes (then build + bump + redeploy per the controller workflow); no agent or firewall change.

Separate hardening gap (not the cause — note for later)

The defense-in-depth control the agent's own config comment calls for — "a host firewall rule should limit [8443] to the guest bridge subnet" — is absent: pve-firewall is disabled and there is no iptables rule scoping 8443. Close this once connectivity is fixed (a host rule allowing 192.168.0.0/24→:8443 and dropping others), independent of the leak fix.

---

Note on the campaign finding #1

unattended-test-campaign-2026-06-22-findings.md finding #1 correctly observed the symptom and correctly rejected rebinding to 0.0.0.0 in this follow-up, but its first-pass theory (agent bound to a specific IP ⇒ unreachable) was wrong — disproved by :8006 reachability to the same IP and by the path working at low uptime. The real cause is the controller connection leak above. No code or config was changed by this diagnosis.