felhom-controller/REPORT.md

REPORT — controller v0.74.0: fix the controller→agent connection leak

Baseline → target

felhom-controller main v0.73.0 → v0.74.0. Controller-only change; agent/hub/firewall untouched.

Problem

Server.agentClient() built a fresh agentapi.Client (hence a fresh bare http.Transport, IdleConnTimeout:0) on every agent API call and discarded it without closing idle connections. The agent's keep-alive left one idle ESTABLISHED socket per call to 192.168.0.162:8443; they accumulated (~5.8k/day) until the ephemeral source-port range for that tuple exhausted → connect: cannot assign requested address (EADDRNOTAVAIL), disabling storage UI, host-metrics, and whole-guest backup after ~5 days of controller uptime. (:8006/pveproxy was immune — the controller never dials it.) Diagnosis: felhom.eu/documentation/tests/unattended-test-campaign-2026-06-22-8443-diagnosis.md.

Change (commit 2a5b88f)

• internal/web/server.go — Server gains agentCli *agentapi.Client, agentCliErr error, agentCliOnce sync.Once (+ the agentapi import).
• internal/web/agent_disk_handlers.go — agentClient() memoizes the build via agentCliOnce and returns one shared client (cfg.LocalAPI is static per process; a config-apply self-restarts the controller). The empty-endpoint "not configured" guard stays OUTSIDE the Once. Signature and all 19 call sites unchanged; *agentapi.Client/*http.Client are concurrency-safe so no extra locking.
• internal/agentapi/client.go — New Transport hardened: MaxIdleConns:4, MaxIdleConnsPerHost:2, IdleConnTimeout:90s (was bare, IdleConnTimeout:0). Added optional Client.Close() (CloseIdleConnections) hygiene helper.

Tests (green; both red-proofed)

• go build ./... && go vet ./... && go test ./... — all green.
• T1 TestAgentClient_ReusesSameInstance (web) — two agentClient() calls return the identical pointer; TestAgentClient_UnconfiguredErrors — empty endpoint still errors. Red-proof: reverting to per-call agentapi.New → pointers differ → FAIL (shown, reverted).
• T2 TestNew_TransportIdlePoolBounded (agentapi, white-box) — IdleConnTimeout>0 AND MaxIdleConnsPerHost>0. Red-proof: bare Transport → IdleConnTimeout==0 → FAIL (shown, reverted).

Deploy

Built+pushed gitea.dooplex.hu/admin/felhom-controller:0.74.0 on 192.168.0.180 (digest sha256:2e85376e…), deployed to guest 9201 via the bootstrap path (docker pull → pin /etc/felhom-controller-image → restart felhom-controller-bootstrap.service). docker inspect: image=:0.74.0 status=running health=healthy.

Process note: the first build packaged stale source (the build server's ~/git/felhom-controller was at v0.73.0 — the required git -C ~/git/felhom-controller pull step had been skipped, so the image was the old code mislabeled :0.74.0; the live test still leaked). Pulled the source to 2a5b88f and rebuilt — second image (sha256:2e85376e…) is the real fix.

LIVE acceptance — leak is gone (the real proof)

Burst of 120 agent calls (alternating /api/disks + /api/host-metrics) against the controller in guest 9201, sampling idle ESTABLISHED sockets to 192.168.0.162:8443 from the controller's netns (nsenter -t <pid> -n ss -tn state established dst 192.168.0.162:8443 | grep -c 192.168):

stage	pre-fix image (stale build)	fixed image (2e85376e)
before	8	1
after 40 calls	48	2
after 80 calls	90	2
after 120 calls	132	2

Fixed: flat at 2 (= MaxIdleConnsPerHost), independent of call count. Pre-fix grew ~1/call. Agent endpoints still return real data (/api/disks lists disks, /api/host-metrics returns host CPU/mem).

Not touched / separate open item

The agent, its bridge-IP ListenAddr bind (deliberate defense-in-depth), and all firewall rules were not changed (controller-only fix). Separate open item (NOT addressed here): the defense-in-depth host firewall rule scoping :8443 to the guest bridge subnet is still absent (pve-firewall disabled, no 8443 rule) — close it independently of this fix.