felhom.eu/REPORT-agent-transport-leak.md

REPORT — R-344: the agent's leaked PBS connections, fixed and proven on both boxes (2026-08-20)

Outcome: the fix works, measured three independent ways, and ep0 is back to its baseline of 17 file descriptors from 415. Both demo boxes now run agent 0.130.0. Nothing is published — that is the one decision left, filed as R-347.

1. Confirmed baselines

repo	in	out
felhom-agent	f17ed11 — v0.129.0	ede49b6 — 0.130.0, UNRELEASED
felhom.eu	9299f85	docs + register only

Both clean and equal to origin/main before each build. Agent version in: 0.129.0 on both boxes. Out: 0.130.0 on both, confirmed from the hub, not from the boxes' own --version.

2. The diff

file	symbol	change
internal/httpx/transport.go	new package	DefaultIdleConnTimeout = 90s; NewTransport(tlsCfg, idle) — fresh transport per call, <= 0 means use the default, never "no timeout"
internal/httpx/transport_test.go	new	zero/negative → default; the constant is read off http.DefaultTransport; freshness; TLS config preserved
internal/pbs/client.go	Config, NewClient	IdleConnTimeout field (tests only); transport via httpx
internal/pbs/client_leak_test.go	new	Scenarios A/C + the production-default pin
internal/hub/client.go	NewClient	via httpx — consistency only, did not contribute to the leak
internal/proxmox/client.go	NewClient	same
cmd/felhom-agent/main.go	version	0.92.1 → 0.130.0 (ldflags default)
REUSE.md, CHANGELOG.md	—	httpx.NewTransport entry; the release note

Commit ede49b6 on main. grep '&http.Transport{' now matches only httpx itself.

A check made before trusting the fix: doBody already reads the body to completion and closes it, so the connection genuinely reaches the idle pool. Had it not, the idle timeout would have been the wrong fix entirely.

3. Tests and the red-proofs

go build ./... && go vet ./... && go test ./... → 30 packages, 0 failures. python3 scripts/agent_gates.py → all 4 gates OK.

The leak test counts connections server-side and models what pbsTargetsFromPVE does — build a client, use it once, drop it. It deliberately does not assert err == nil or that a field holds a value; both were true of the leaking code (the R-224 lesson).

red-proof	mutation	seen failing with
1 — the fix	remove IdleConnTimeout	"abandoned pbs.Clients: after 5s the server still holds 5 open connection(s), want 0 (5 dialled in total)" — the count is in the message, so it cannot be a timeout with another cause
2 — the worse fix	DisableKeepAlives: true	the leak test PASSES. Caught only by TestPBSClient_KeepAliveStillReuses: "3 sequential requests over 3 connection(s), want 1"

Red-proof 2 is the load-bearing one: Scenario A alone would have accepted a fix that made the problem worse — no leak, at the price of a fresh dial for every one of ~40,000 daily requests. Both mutations reverted, tree re-verified clean.

4. §4 quoted, beside the results

P1 — "(i) ep0's fd count falls by ≈194 within seconds … (ii) … ≈194 sockets convert ESTAB → CLOSE-WAIT … (iii) neither — the count barely moves. Then the ownership attribution is wrong and the finding must be withdrawn." P2 — "demo-hp (fixed): ≈ 0 … demo-felhom (control, untouched): ≈ 4 per hour → ≈ 16 over 4 h." P3 — "ep0's overall leak rate should fall from ≈ 200/day to ≈ 100/day while one box is fixed, and to ≈ 0/day after Part 5."

5. P1 — outcome (i), in one second

	ep0 fd	ESTAB	demo-felhom	demo-hp	CLOSE-WAIT
T-0 09:15:46Z	415	398	199	199	0
T+1s 09:15:47Z	216	199	199	0	0
T+60s 09:16:42Z	218	201	199	2	0

Outcome (ii) did not occur, so it gets no register row. Not one socket converted to CLOSE-WAIT: ep0 reaps on peer FIN correctly. That also means the 543 CLOSE-WAIT at the 2026-08-18 wedge has some other explanation and is not evidence of a second defect on the protected machine — a finding in the negative, worth the sixty seconds it cost.

Ownership is now proven a third independent way: what dies with the process, agreeing with ss -tnp and with the access-log user agent. At 133 s uptime the fixed box held 0 connections.

6. P2 — divergence

Window 09:15:47Z → 10:17:52Z = 1.03 h. You closed the ≥4 h window early, so no daily rate is extrapolated and none is needed.

box	agent	start	end	delta	per hour	predicted
demo-felhom CONTROL	0.129.0	199	203	+4	3.87	≈4
demo-hp FIXED	0.130.0	0	0	+0	0.00	≈0

The assumption-free statement. ep0's access log counts the opportunities: each box made exactly 4 /snapshots and 4 /version calls in the window.

control: 4 cycles → 4 leaks. fixed: 4 cycles → 0 leaks.

Positive observable (standing rule 3): a zero leak is equally consistent with "the agent stopped working" — it did not; its four cycles are in ep0's log. The boxes' other traffic is near-identical (libwww-perl 924 vs 926, proxmox-backup-client 898 vs 898), so the only difference between them is the binary. Poisson alone gives P(0 | λ=4) = 1.8%, which is suggestive rather than conclusive and is not relied on alone.

7. P3 — the second box, and the backlog clearing itself

demo-felhom upgraded 10:18:56Z on your word.

	ep0 fd	ESTAB	CLOSE-WAIT
T-0 10:18:55Z	220	203	0
T+2s	17	0	0
settled 10:30–10:35Z	17–19	0–2	0

17 is precisely ep0's t0 baseline (fd 17, ESTAB 0, 2026-08-18 09:51:22Z), and it returns to 17 between poll cycles — the "healthy proxy near 20 fds" the incident document named. Predicted ≈0/day residual; observed the baseline itself.

A correction, made within the hour it was written. My STOP 1 report and the first CHANGELOG draft said "does not clear the 388 descriptors already stuck on ep0 — those persist until that proxy restarts." Wrong. They were held on both sides; restarting the agents released every one. ep0 was read-only throughout and its proxy PID never changed (551655). Corrected in the CHANGELOG, the audit document and R-344 rather than quietly edited.

8. Fleet sanity

Hub reports 0.130.0 on both boxes. No floor held line (0.130.0 > golden MinAgent 0.129.0). No pbsdr_box_unreachable / offsite_box_unreachable during any window. Positive observable rather than the absent one: the PBS-DR gauge kept refreshing (3.7% full (3.7 GB of 97.9 GB)) and host reports kept landing from both boxes throughout.

9. What is NOT done

• Not published. No package, no tag, no manifest or floor field touched, no self-update staged. A box installed from the current image still ships the leaking agent — R-347, your call.
• The CHANGELOG heading is ## UNRELEASED — v0.130.0 candidate. The release-complete gate convicted on ## v0.130.0 because there is no tag and no package, and it was right to. I did not use --no-verify; I made the heading stop claiming a release that has not happened. It flips to ## v0.130.0 in the same commit as the tag.
• Poll rate unchanged (R-336, re-scoped). pbsTargetsFromPVE not refactored; no CloseIdleConnections added. ep0 not touched.
• The 388 descriptors ARE cleared — see §7. This is the one item the prompt expected to remain outstanding, and it did not.

10. Register

• R-344 — updated with the fix, P1's outcome named, and P2/P3's numbers. Left OPEN, because a fix on two boxes by hand is not delivered.
• R-336 — re-scoped. Its new next-step cell, verbatim: "NEW ACCEPTANCE CRITERION, since the old one is void: the fd count is NOT the observable for this row any more — that belongs to R-344 and is already satisfied. Measure the REQUEST RATE at ep0's access log, and state the projected rate at the target customer count." The row now records explicitly that its old next-step would have "fixed" nothing while looking like a failed fix, and re-scopes it to what it is: ~85,000 requests/day to a weekly-write DR endpoint, ≈25 requests/second at fifty customers against a CX33.
• R-347 (new) — the delivery gap. Owner: Viktor decides, CC executes.
• R-348 (new) — an agent restart blanks the reported backup list for up to ~18 h, and the Store comment calls backups "unaffected". Blinds no alarm — checked, not assumed: the hub's backupEvidenceLookback scans 7 days for exactly this case, and pbs_snapshots stayed populated.
• No P1(ii) row, because outcome (ii) did not occur.
• R-346 — this run anchored on the measured t0 (fd 17 at 2026-08-18 09:51:22Z), never on a systemd timestamp, so the 5 h 56 m discrepancy did not touch these numbers.

12. Observations

• The closure refactor is not worth doing — recommend leaving it. With the idle timeout restored an abandoned client's connection is gone in 90 s, so the standing population is bounded at about one connection per box instead of growing without limit. Caching clients would add cache-invalidation questions (a storage's fingerprint, token or namespace can change under it) for no observable gain.
• Three other http.Transport defaults are still missing and were left alone: MaxIdleConns, TLSHandshakeTimeout (0 = no limit; DefaultTransport uses 10 s) and ExpectContinueTimeout. None accumulates, and every client bounds its request with http.Client.Timeout. TLSHandshakeTimeout is the only one with a plausible failure mode — a stalled handshake over the tunnel, bounded today only by the outer 30 s. Not changed, because widening the diff would have made this measurement unattributable. Worth a look on its own terms; not a defect.
• A measurement error of mine, recorded because it nearly cost four hours. The first P2 sampler reported both per-box columns as 0 while the totals were right: ss prints [::ffff:10.77.0.2]:port and my pattern expected 10.77.0.2:. Caught 15 minutes in, because a 0/0 split cannot sum to 199. Fixed, then one sample proved by hand before committing the window — which is what should have happened first.