65 lines
4.1 KiB
Markdown
65 lines
4.1 KiB
Markdown
# REPORT — controller v0.74.0: fix the controller→agent connection leak
|
|
|
|
## Baseline → target
|
|
`felhom-controller` `main` v0.73.0 → **v0.74.0**. Controller-only change; agent/hub/firewall untouched.
|
|
|
|
## Problem
|
|
`Server.agentClient()` built a fresh `agentapi.Client` (hence a fresh bare `http.Transport`,
|
|
`IdleConnTimeout:0`) on **every** agent API call and discarded it without closing idle connections.
|
|
The agent's keep-alive left one idle `ESTABLISHED` socket per call to `192.168.0.162:8443`; they
|
|
accumulated (~5.8k/day) until the ephemeral source-port range for that tuple exhausted →
|
|
`connect: cannot assign requested address` (EADDRNOTAVAIL), disabling storage UI, host-metrics, and
|
|
whole-guest backup after ~5 days of controller uptime. (`:8006`/pveproxy was immune — the controller
|
|
never dials it.) Diagnosis:
|
|
`felhom.eu/documentation/tests/unattended-test-campaign-2026-06-22-8443-diagnosis.md`.
|
|
|
|
## Change (commit `2a5b88f`)
|
|
- `internal/web/server.go` — `Server` gains `agentCli *agentapi.Client`, `agentCliErr error`,
|
|
`agentCliOnce sync.Once` (+ the `agentapi` import).
|
|
- `internal/web/agent_disk_handlers.go` — `agentClient()` memoizes the build via `agentCliOnce` and
|
|
**returns one shared client** (cfg.LocalAPI is static per process; a config-apply self-restarts the
|
|
controller). The empty-endpoint "not configured" guard stays OUTSIDE the `Once`. Signature and all
|
|
19 call sites unchanged; `*agentapi.Client`/`*http.Client` are concurrency-safe so no extra locking.
|
|
- `internal/agentapi/client.go` — `New` Transport hardened: `MaxIdleConns:4`,
|
|
`MaxIdleConnsPerHost:2`, `IdleConnTimeout:90s` (was bare, `IdleConnTimeout:0`). Added optional
|
|
`Client.Close()` (CloseIdleConnections) hygiene helper.
|
|
|
|
## Tests (green; both red-proofed)
|
|
- `go build ./... && go vet ./... && go test ./...` — all green.
|
|
- **T1** `TestAgentClient_ReusesSameInstance` (web) — two `agentClient()` calls return the identical
|
|
pointer; `TestAgentClient_UnconfiguredErrors` — empty endpoint still errors.
|
|
Red-proof: reverting to per-call `agentapi.New` → pointers differ → FAIL (shown, reverted).
|
|
- **T2** `TestNew_TransportIdlePoolBounded` (agentapi, white-box) — `IdleConnTimeout>0` AND
|
|
`MaxIdleConnsPerHost>0`. Red-proof: bare Transport → `IdleConnTimeout==0` → FAIL (shown, reverted).
|
|
|
|
## Deploy
|
|
Built+pushed `gitea.dooplex.hu/admin/felhom-controller:0.74.0` on 192.168.0.180 (digest
|
|
`sha256:2e85376e…`), deployed to guest 9201 via the bootstrap path (`docker pull` → pin
|
|
`/etc/felhom-controller-image` → restart `felhom-controller-bootstrap.service`). `docker inspect`:
|
|
`image=:0.74.0 status=running health=healthy`.
|
|
> Process note: the first build packaged stale source (the build server's `~/git/felhom-controller`
|
|
> was at v0.73.0 — the required `git -C ~/git/felhom-controller pull` step had been skipped, so the
|
|
> image was the old code mislabeled `:0.74.0`; the live test still leaked). Pulled the source to
|
|
> `2a5b88f` and rebuilt — second image (`sha256:2e85376e…`) is the real fix.
|
|
|
|
## LIVE acceptance — leak is gone (the real proof)
|
|
Burst of **120 agent calls** (alternating `/api/disks` + `/api/host-metrics`) against the controller in
|
|
guest 9201, sampling idle `ESTABLISHED` sockets to `192.168.0.162:8443` from the controller's netns
|
|
(`nsenter -t <pid> -n ss -tn state established dst 192.168.0.162:8443 | grep -c 192.168`):
|
|
|
|
| stage | pre-fix image (stale build) | **fixed image (2e85376e)** |
|
|
|---|---|---|
|
|
| before | 8 | **1** |
|
|
| after 40 calls | 48 | **2** |
|
|
| after 80 calls | 90 | **2** |
|
|
| after 120 calls | 132 | **2** |
|
|
|
|
**Fixed: flat at 2 (= MaxIdleConnsPerHost), independent of call count.** Pre-fix grew ~1/call. Agent
|
|
endpoints still return real data (`/api/disks` lists disks, `/api/host-metrics` returns host CPU/mem).
|
|
|
|
## Not touched / separate open item
|
|
The agent, its bridge-IP `ListenAddr` bind (deliberate defense-in-depth), and all firewall rules were
|
|
**not** changed (controller-only fix). **Separate open item (NOT addressed here):** the defense-in-depth
|
|
host firewall rule scoping `:8443` to the guest bridge subnet is still absent (`pve-firewall` disabled,
|
|
no 8443 rule) — close it independently of this fix.
|