# 8443 diagnosis — controller→agent local-API unreachability ROOT CAUSE — 2026-06-22

> **Status: read-only diagnosis (no fix applied).** Follow-up to
> `unattended-test-campaign-2026-06-22-findings.md` finding #1. The task rejected the campaign's
> "rebind agent to 0.0.0.0" guess (the bridge-IP bind is deliberate defense-in-depth) and asked to
> establish the real cause from evidence. **It is established and verified.**

## TL;DR verdict

**Root cause = a connection/`http.Transport` leak in the *controller* (client side), not a network,
firewall, agent-bind, or endpoint problem.** `felhom-controller`'s `agentClient()`
(`internal/web/agent_disk_handlers.go:43`) calls `agentapi.New(...)` **on every agent API call**, and
`agentapi.New` (`internal/agentapi/client.go:83`) builds a **fresh `&http.Client{Transport:
&http.Transport{TLSClientConfig: …}}`** each time. That bare Transport has **`IdleConnTimeout: 0`**
(idle keep-alive connections never expire) and is discarded after the single call **without
`CloseIdleConnections()`**. Because the agent keeps connections alive, every call **leaks one idle
ESTABLISHED socket** to `192.168.0.162:8443`. They accumulate monotonically over controller uptime
until the **ephemeral source-port range** for the `(container → 192.168.0.162:8443)` tuple is
exhausted → `connect: cannot assign requested address` (**EADDRNOTAVAIL**).

This is **H5 (ephemeral-port / socket exhaustion)**, with a precise code cause. **H1, H2, H3, H4, H6
are ruled out with positive evidence below.**

Why `:8006` always worked and `:8443` "failed": the controller **only ever dials `:8443`** (the agent).
It never connects to pveproxy `:8006`, so that destination never leaks and never exhausts. The failure
was never port-specific in the network sense — it was **destination-specific exhaustion** driven by the
leak.

Why it "broke persistently" in the campaign but **works now**: the leak accumulates with controller
**uptime**. The campaign hit it at the controller's ~5-day uptime (pool exhausted). The Phase-7 guest
reboot restarted the controller with a fresh socket table → it works again, and is **already
re-accumulating** (measured below). This is an accumulating-state regression cleared by restart, NOT a
static misconfiguration — exactly why the campaign's static theory (agent bind) didn't fit its own data.

---

## Evidence

### Step 1 — agent listener ground truth (host)
```
ss -tlnp 'sport = :8443'  → LISTEN 192.168.0.162:8443  users:(("felhom-agent",pid=315505,fd=11))
ss -tlnp 'sport = :8006'  → LISTEN *:8006               users:(("pveproxy"…))
ip -br addr               → vmbr0 UP 192.168.0.162/24 ; veth9201i0@if2 UP ; (no IPv6 global)
```
Agent = a single **IPv4** socket bound to `192.168.0.162` (vmbr0), one process. pveproxy binds wildcard
`*:8006`. Healthy listener. (The bind-to-bridge-IP is the intended defense-in-depth, and is fine.)

### Step 2 — controller configured endpoint (redacted)
`bootstrap.json local_api`: `endpoint: 192.168.0.162:8443`, `token: «REDACTED»`,
`fingerprint: «REDACTED»`. The endpoint **is** the vmbr0 bridge IP + correct port — matches the error
string. **H2 (endpoint mismatch) ruled out.**

### Step 3 — connect matrix (the localizer) — ALL SUCCEED post-reboot
Raw `socket.connect()` (AF_INET) to `192.168.0.162`, exact errno per layer × port:

| layer | :8443 | :8006 |
|---|---|---|
| controller **container netns** (pid 1917) | **OK** (local 172.17.0.2) | OK |
| guest root netns | OK (local 192.168.0.141) | OK |
| host | OK (local 192.168.0.162) | OK |

`nsenter -t 1917 -n ip route get 192.168.0.162` → `via 172.17.0.1 dev eth0 src 172.17.0.2 … cache`
(resolves cleanly, source assignable). Container has eth0 `172.17.0.2/16` + eth1 `172.18.0.8/16`, no
global IPv6. **At low uptime the TCP path is fully healthy — H3 (route/source) and H4 (address-family)
ruled out.** Live re-test of the real app path also succeeds now: controller `/api/disks` returns the
full disk JSON; in-container `curl https://192.168.0.162:8443/` → **404** (TCP+TLS OK), `:8006` → 200.
**H6 (agent socket truth) ruled out** — agent accepts past TCP+TLS.

### Step 3b/core — the leak (the decisive artifact)
Controller container, **47 minutes** after the Phase-7 reboot:
```
ss -s (container netns):  TCP estab 194, closed 1049, timewait 9
estab → 192.168.0.162:8443: 194  (essentially ALL established sockets go to the agent)
ip_local_port_range:       32768  60999   (= 28231 ephemeral ports)
```
Monotonic growth (same netns, 30 s apart): **196 → 198**, later samples **206**. Rate ≈ 4/min ≈
**~5 800/day**. Two-sided + idle confirmation:
```
HOST (agent) side, estab on :8443 from 192.168.0.141 (guest):  206   (Recv-Q/Send-Q = 0 → idle)
controller netns estab→8443:                                    206
controller process open fds:                                    217   (206 = leaked agent sockets)
```
**Projection:** ~5 800 leaked sockets/day ÷ 28 231 ports ⇒ exhaustion in **~4–5 days** of controller
uptime → EADDRNOTAVAIL. Matches the campaign hitting it at ~5-day uptime exactly.

### Step 4 — guest-side nat/filter (H1, the campaign's prime suspect) — RULED OUT
Inside guest 9201:
```
iptables-save | grep -E '8443|8006'  → NONE (no port-specific rule)
filter REJECT/DROP                   → only standard docker inter-bridge isolation
                                        (-A DOCKER ! -i br-X -o br-X -j DROP), not outbound-to-LAN
docker subnets                       → 172.17/18/19/20/21.0.0/16 — none overlap 192.168.0.0/24
```
No rule treats 8443 differently from 8006; no subnet overlap. **H1 ruled out** (and the path works now,
which alone disproves a static guest block).

### Step 5 — tcpdump death-point — N/A at current uptime
The failure is **not reproducible at low uptime** (the path works — the SYN leaves and connects). A
capture now would only show successful handshakes. The "death point" only appears once the ephemeral
pool is exhausted, at which point `connect()` fails **locally** (no SYN emitted) — consistent with the
EADDRNOTAVAIL semantics. No capture taken (nothing to capture); the leak measurement is the decisive
artifact instead.

### Step 6 — temporal verdict
**Regression that accumulates with controller uptime; reset by restart.** Definitive evidence: failed
persistently during the campaign at the controller's ~5-day uptime (6/6 EADDRNOTAVAIL); after the
Phase-7 guest reboot (controller `Up 47 minutes`) the path works and the leak is at 194→206 and
climbing. The current container's logs only span 47 min (post-reboot), so the pre-reboot failures are
gone with the old container — but the accumulate→reset mechanism is directly measured, not inferred. It
"works after every restart, breaks after ~5 days."

### Step 7 — agent socket accept — confirmed reachable
`curl -vk https://192.168.0.162:8443/` from in-container connects at TCP+TLS and returns HTTP 404; the
pinned-TLS app path (`/api/disks`) returns real data. The agent is NOT the problem.

---

## Hypothesis scorecard

| | Hypothesis | Verdict | Deciding evidence |
|---|---|---|---|
| H1 | guest-side docker/nat block on 8443 | ❌ ruled out | no 8443/8006 rule; path works now |
| H2 | configured-endpoint mismatch | ❌ ruled out | endpoint = `192.168.0.162:8443` (the bridge IP) |
| H3 | source/route selection failure | ❌ ruled out | `ip route get` resolves, src assignable, raw connect OK |
| H4 | address-family / IPv6 | ❌ ruled out | agent + container are IPv4-only; no global v6 |
| **H5** | **ephemeral-port / socket exhaustion** | ✅ **CONFIRMED** | 206 leaked idle ESTABLISHED→8443 in 47 min, growing ~5.8k/day vs 28 231 ports ⇒ exhausts in ~5 days; matches campaign timing |
| H6 | agent listener truth | ❌ ruled out | single healthy v4 socket; accepts TCP+TLS; returns data |

**Precise mechanism within H5:** `agentClient()` → `agentapi.New()` per call → new `http.Transport`
with `IdleConnTimeout: 0`, discarded without `CloseIdleConnections()`; agent keep-alive ⇒ one leaked
idle ESTABLISHED socket per call.

---

## Proposed fix direction (controller-side; NOT an agent rebind)

The agent's bind to `192.168.0.162` (bridge IP) is correct and must stay. The fix is entirely in the
controller's client lifecycle:

1. **Build the `agentapi.Client` once and reuse it** (it is stateless config — endpoint/token/
   fingerprint, all from `cfg.LocalAPI`). Construct it at server init, store it on `Server`, and have
   `agentClient()` return the shared instance. A single reused Transport pools/reuses connections (≈2
   idle conns), eliminating the leak. **Preferred.**
2. **If a per-call client is kept for any reason**, the Transport must be tamed and closed: set
   `IdleConnTimeout` (e.g. 30–90 s) + `MaxIdleConnsPerHost`, and `defer client.CloseIdleConnections()`
   after use. (Reuse (#1) is cleaner and also avoids the per-call TLS handshake cost.)
3. Optionally add `MaxConnsPerHost`/keep-alive tuning on the shared Transport as belt-and-suspenders.

These are pure `felhom-controller` changes (then build + bump + redeploy per the controller workflow);
no agent or firewall change.

### Separate hardening gap (not the cause — note for later)
The defense-in-depth control the agent's own config comment calls for — *"a host firewall rule should
limit [8443] to the guest bridge subnet"* — is **absent**: `pve-firewall` is disabled and there is no
iptables rule scoping 8443. Close this once connectivity is fixed (a host rule allowing
`192.168.0.0/24`→`:8443` and dropping others), independent of the leak fix.

---

## Note on the campaign finding #1
`unattended-test-campaign-2026-06-22-findings.md` finding #1 correctly **observed** the symptom and
**correctly rejected** rebinding to 0.0.0.0 in this follow-up, but its first-pass theory (agent
bound to a specific IP ⇒ unreachable) was wrong — disproved by `:8006` reachability to the same IP and
by the path working at low uptime. The real cause is the controller connection leak above. No code or
config was changed by this diagnosis.
