felhom.eu/documentation/audits/SPIKE-ep0-established-connections-2026-08-20.md

SPIKE — who holds ep0's connections open, and does the leak track the poll rate?

Run: 2026-08-20 08:01–08:35 UTC, read-only throughout. Phase C (the one mutation) has NOT been run — it is held at STOP 1 for the operator's window decision. Evidence: evidence-ep0-established-connections-2026-08-20/, one file per step. Supersedes the plan in SPIKE-ep0-established-connections.md (issued 2026-08-18, never run).

Headline, and it is not the headline the spike expected. The leak is ours. Every one of the 388 leaked descriptors on ep0 is an established connection held open by felhom-agent on the customer boxes — one per PBS poll cycle, never closed on either side. pvestatd and proxmox-backup-client made 162,404 requests in the same window and leaked zero. R-336's premise — that cutting the ~85,000/day poll rate is the leak fix — does not survive this measurement. The poll rate is a separate (real) problem; it is not what is consuming descriptors.

---

Q0 — R-341's first dated check: has the fd slope changed since the PBS 4.2.5-1 upgrade?

Answer: NO. Unchanged, and now measured over a window 88× longer than the one it replaces.

Precondition passed. MainPID = 551655, ps -o lstart = Tue Aug 18 09:51:04 2026, NRestarts = 0 on both proxmox-backup-proxy and proxmox-backup. The proxy generation that carries t0 is the one that was read. The elapsed-window measurement is valid.

	value
t0	fd 17, ESTAB 0, 2026-08-18 09:51:22Z, PID 551655
reading A	fd 405, ESTAB 388, CLOSE-WAIT 0, 2026-08-20 08:02:13Z, PID 551655
elapsed	166,251 s = 46.181 h = 1.9242 d
delta	+388 fd
rate	201.6 fd/day (8.40/h)
Poisson on n=388	±19.7 counts → ±10.2/day (1σ), ±20.5/day (2σ) → 181.2 … 222.1 /day
limits	soft/hard 65536 / 65536 — the drop-in survived the upgrade
listener	LISTEN Recv-Q 0 Send-Q 1024 — accept queue empty, nothing wedged

Verdict against the pre-registration, quoted from the spike prompt §2 so it is visible beside the result:

"Predicted range: roughly 370–450 … A reading anywhere in 330–490 confirms an unchanged rate … In range → unchanged rate. This is the EXPECTED result, consistent with the changelog containing no mechanism to change it. It is not a failed upgrade."

Observed: 388. In range, and close to the centre. The rate is unchanged since the upgrade. This is the expected result and it is recorded as such — not as a success and not as a failure of the upgrade, which was run for rehearsal value and never claimed to fix anything.

For comparison, on the identical method: 183/day (31 min, pre-upgrade), 200/day (5.64 h, pre-upgrade), 225/day (32 min, post-upgrade). This window carries 97× the descriptor count of the 31-minute window, so its uncertainty is ±5% rather than ±50%.

Composition — the split, which is what says which leak it is

t0        fd  17   ESTAB   0   CLOSE-WAIT 0
reading A fd 405   ESTAB 388   CLOSE-WAIT 0          17 + 388 = 405, exactly

CLOSE-WAIT is absent from the ss histogram entirely — 0, not merely flat. 100% of the growth is established connections on :8007. This closes the question the incident's own correction block opened: the mechanism is connections the proxy never reaps, and CLOSE-WAIT is not merely the minority half, it is not present at all on this proxy generation.

Runway

From fd 405 at 201.6/day to the 65536 soft limit: ≈ 323 days, i.e. around 2027-07-09. Under a year. A deadline, not a comfort — unchanged in character from what R-341 recorded.

---

Q1 — who is at the far end?

Answer: exactly the two demo boxes, in a dead-even split. No third peer. Outcome (c) is excluded.

At 08:02:39Z, ss -tn state established '( sport = :8007 )', peer histogram:

peer	connections	identity
10.77.0.2	194	demo-felhom (host felhom-pve) — token felhom@pbs!demo-felhom
10.77.0.3	194	demo-hp (host felhom-host) — token felhom@pbs!demo-hp

The identity is not inferred from the address: ep0's own access log carries the API token name on every line (::ffff:10.77.0.3 - felhom@pbs!demo-hp …), and the mapping is one-to-one across the whole file.

194 / 194 is not approximately even — it is exactly even, which is itself a finding: whatever opens these is a per-box periodic task running the same schedule on both, not traffic proportional to anything the boxes differ in.

Descriptor-type breakdown (1.3)

lsof -np 551655, which distinguishes sockets from files, pipes and eventfds so a "leak" that is not sockets gets caught here:

type	count
IPv6 (TCP)	390
REG (log files, rrd journal, datastore lock)	32
unix	7
a_inode (epoll ×2, eventfd ×1)	3
DIR	2
CHR (/dev/null)	1

lsof … | grep -c TCP = 390 = 388 established + 1 listener + 1 other. The independent /proc/<pid>/fd link-target histogram agrees: 397 sockets (390 TCP + 7 unix), 9 regular/anon fds. The leak is sockets, and the sockets are TCP on :8007. Nothing else is growing.

1.4 — persistence: are they long-lived, or churn?

Full 4-tuples diffed between reading A (08:02:39Z) and reading B (08:33:42Z), 1,843 s apart:

	count
sockets in reading A	388
sockets in reading B	392
in BOTH — long-lived	388
only in A — closed during the interval	0
only in B — new arrivals	4 (2 per box)

Not one socket closed in 31 minutes. There is no churn to separate from the leak: every established connection on this proxy is the leak. The 4 arrivals over 1,843 s = 187.5 fd/day, consistent with the elapsed-window 201.6/day within Poisson on n=4.

Idle age and liveness. All 392 sockets carry a TCP keepalive timer and all 392 report retrans=0 — the far end is answering keepalive probes. These are not half-open sockets whose peer went away; they are mutually held, live, idle connections. That, on its own, points at outcome (a) and Part 2 confirms it.

---

Q2 — do the boxes hold them too?

Answer: outcome (a) — both sides hold every connection, and the two counts match exactly. Twice.

instant (UTC)	ep0 ESTAB on :8007	demo-felhom ESTAB to 10.77.0.1:8007	demo-hp	sum
reading A — ep0 08:02:39, boxes 08:04:37/38	388	194	194	388
reading B — ep0 08:33:42, boxes 08:34:01/02	392	196	196	392

Two independent paired readings, both exact. And the confirmation is tighter than the totals: the four new sockets that appeared on ep0 during the persistence window carry the same four source ports the boxes report as new —

ep0 new:            10.77.0.2:45620  10.77.0.2:53226  10.77.0.3:37554  10.77.0.3:46214
demo-felhom new:    10.77.0.2:45620  10.77.0.2:53226
demo-hp new:        10.77.0.3:37554  10.77.0.3:46214

Both boxes also closed zero sockets in the same interval (A=194, B=196, closed=0, new=2, on each).

This is not a hedge between two outcomes. Outcome (b) — half-open sockets ep0 never noticed — is excluded by both the matching counts and the retrans=0 keepalives. Outcome (c) is excluded by Q1.

And the owner has a name

ss -tnp on the boxes attributes 194 of 194 on each, to a single PID:

box	PID	process
demo-felhom	2596329	/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json
demo-hp	3199936	/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json

Zero are held by pvestatd. Zero by proxmox-backup-client. Both agents have been running continuously since 2026-08-12 (they did not restart); the count restarted at the proxy's 08-18 restart because that restart closed the old sockets on both sides.

Each agent's own descriptor table is 208 fds, of which 196 are these connections — so the leak is bilateral. The agents' RLIMIT_NOFILE is 524287, so they are in no danger; that is a reason it stayed invisible, not a reason it is harmless.

---

The user-agent attribution (1.5) — the access log settles it independently

Over the same 46.18 h window, from ep0's own access.log.1 + access.log:

user agent	endpoint	requests	req/day	descriptors leaked
libwww-perl/6.78	GET /admin/datastore (pvestatd)	81,192	42,195	0
proxmox-backup-client/1.0	GET …/felhom-offsite/status	80,061	41,607	0
libwww-perl/6.78	GET …/felhom-offsite/snapshots	1,131	588	0
Go-http-client/1.1	GET …/felhom-offsite/snapshots	387	201	388
Go-http-client/1.1	GET /version	371	193	(same connections)
Go-http-client/1.1	POST …/verify + task status	53	28	(same connections)
	total, both boxes	163,215	84,822	388

Go-http-client/1.1 is felhom-agent — the same process ss names on the box side, reached by a completely independent route. 387 agent /snapshots calls against 388 leaked sockets: one per call, within one.

162,404 requests from the two Proxmox pollers leaked nothing. They are 99.5% of the traffic and 0% of the leak.

Cadence reconciliation — the numbers close

source	cadence	cycles in 166,251 s
agent hub-report poll (poll_seconds: 900, read from each box's agent.json)	15 min	184.7
pbs.VerifyLoop (DefaultVerifyCadence = 6h)	6 h	7.7
	sum	192.4

Observed leaked per box: 194. Observed POST …/verify per box in the window: 8 (predicted 7.7). Observed GET /version per box: ~185.5 (predicted 184.7). Mean interval between leaked descriptors per box: 857 s. Every count reconciles against a cadence read from configuration, not fitted to the data.

Mechanism — named from source, and it is a two-line Go footgun

felhom-agent/internal/pbs/client.go:56-60:

http: &http.Client{
    Timeout:   timeout,
    Transport: &http.Transport{TLSClientConfig: tlsCfg},
},

A hand-rolled http.Transport takes the zero value for IdleConnTimeout, which in Go means no limit — idle keep-alive connections are never closed. (http.DefaultTransport sets 90 s; a composite literal does not inherit it.) That alone would be harmless if one client were reused. It is not: felhom-agent/cmd/felhom-agent/main.go:1486-1509 — pbsTargetsFromPVE — says so in its own doc comment:

"returns a pbs.Targets closure that, each cycle, discovers the pbs storages from the PVE config and builds a fingerprint-pinned, token-authed client for each"

So each cycle constructs a new Transport, performs one request, and leaves the connection idle in that transport's pool forever. The transport then becomes unreachable, but Go's persistConn read-loop goroutine keeps the socket alive — an unreachable http.Transport does not close its connections. grep across the whole felhom-agent repo for CloseIdleConnections, IdleConnTimeout, MaxIdleConns: no matches at all.

The same composite-literal pattern appears in two more clients — internal/hub/client.go:53 and internal/proxmox/client.go:69. Those are built once at start-up rather than per cycle, so they do not leak by this route; they are recorded because the pattern is one refactor away from doing so. Filed as R-344.

---

Control-window cleanliness (1.6)

The elapsed window is doing double duty as Phase C's control, so it has to be shown undisturbed. It was.

check	finding
ep0 host reboot	none — booted 2026-08-03 11:12:57, up 17 d
proxy / API daemon restart	none — NRestarts = 0 on both units, PID 551655 throughout
box reboots	none — both booted 2026-08-10 09:2xZ, up 10 d
agent restarts	none — both ActiveEnterTimestamp 2026-08-12
request-rate gaps	none. Hourly totals across the window: 3,536–3,553 every hour, no hour missing
HTTP errors	none. Every response in the window is 200, except 8 × 101 (expected protocol upgrades)
tunnel flaps	no evidence — WireGuard handshakes for 10.77.0.2, .3, .250 all seconds old; and a flap would have shown as a request gap or errors, and neither exists
a backup inside the window?	NO. The last offsite protocol upgrades were 2026-08-18 03:57 and 03:58Z — the incident re-runs, before t0 at 09:51:22Z. Both boxes' pvesm list felhom-pbs confirms the newest snapshots are 2026-08-18T03:57:43Z / 03:58:43Z. The tier's cadence_seconds is 604800 (weekly), read from each box's agent.json, so the next offsite run is due ~2026-08-25
other non-poll activity	yes, and it is routine, quantified, and does not contaminate. Server-side verify jobs ran on ep0 on a ~6 h cadence (8 per box in the window), and restore-test reads ran on 2026-08-19 at 04:45–04:50 and 06:52–06:54Z (8 reader upgrades total). Together with snapshot/version/task-status calls this is ~1,963 of 163,215 requests = 1.2% of the window's traffic, and at most 8 of the 388 descriptors could be attributed to reader connections (≤2%). The verify cycle is already counted inside the 194/box reconciliation above — it is part of the measured leak, not a contaminant of it

Verdict: the control window is clean. No correction is applied and none is needed.

Two nuances recorded rather than smoothed over.

1. systemctl show proxmox-backup-proxy -p ActiveEnterTimestamp reads 2026-08-18 03:54:54Z, not 09:51:04Z, while NRestarts = 0 and MainPID did change. The upgrade therefore re-exec'd the proxy rather than restarting the unit — systemd never saw a stop. Anyone using ActiveEnterTimestamp as the "when did this proxy generation start" anchor would get a figure 5 h 56 m too early and compute a rate ~12% low (388 fd over 52.1 h instead of 46.2 h = 178.7/day instead of 201.6/day). Use ps -o lstart= -p $MainPID, as R-341's own command does.
2. ep0 carries a fourth WireGuard peer, 10.77.0.4, whose last handshake was 2026-08-13 (7 d ago). It is drill-r50-0a4f9a, a drill host, and its dormant peer entry is already recorded in architecture/_recovery-inventory-2026-07-28.md. Outside the window; not a contaminant; not a new finding.

Hub view. No pbsdr_box_unreachable / offsite_box_unreachable and no *_recovered event was logged. The positive observable rather than the absent one: the PBS-DR gauge refreshes on schedule (PBS-DR box refreshed: 3.7% full (3.7 GB of 97.9 GB), every ~16 min through the run) and host reports land from both boxes. Honest limit: the hub pod is 39 h old (redeployed for v0.106.0 at 2026-08-18 19:29), so hub-side log evidence covers 36.6 of the window's 46.2 hours. For the first 9.6 h the evidence is ep0's own access log — which shows no gap and no error — and NRestarts = 0. Hub log timestamps render CEST.

---

Q3 — is the leak rate proportional to the request rate?

NOT ANSWERED BY MEASUREMENT. Phase C has not been run; it is held at STOP 1.

What exists instead is a pre-registered prediction, written and committed before any Phase C number can exist: evidence-ep0-established-connections-2026-08-20/phaseC-prediction.txt. Its substance:

• Stopping pvestatd on demo-hp removes both pvestatd-driven streams from that box (the PVE PBS storage plugin issues the libwww-perl datastore list and shells out to proxmox-backup-client for the status call, both on pvestatd's cycle) — −98.8% of demo-hp's requests, −49.4% of the fleet total. If ep0's request count does not fall by close to half, the independent variable did not move and no ratio may be computed.
• H1 (what Parts 1–2 predict): NOT proportional. demo-hp's felhom-agent keeps running, so both boxes keep leaking. Expect ~201.6 fd/day, unchanged — ~84 descriptors in a 10 h window (2σ 66–102) — split symmetrically ~42/~42 between the two peers.
• H2 (the hypothesis Q3 was written to test): proportional. Expect ~101 fd/day — ~42 descriptors in 10 h (2σ 29–55) — split strongly asymmetrically ~42/~0.

H1 and H2 do not overlap at 2σ for a window of 10 h or more. The peer split is the sharpest discriminator and it is free: it needs no second box and no arithmetic.

A result matching H2 falsifies the Parts 1–2 attribution, and the prediction file says so in those words, so the outcome cannot be rationalised in either direction afterwards.

---

Which fix the evidence points at

No design here, per the spike-first gate — only the target. The evidence points at felhom-agent/internal/pbs/client.go's HTTP transport lifetime, not at the poll rate. One pbs.Client is constructed per cycle by pbsTargetsFromPVE, each carrying a fresh http.Transport whose IdleConnTimeout is the zero value, and nothing ever closes it; the observed leak is exactly one socket per agent /snapshots call, on both sides, on both boxes, and it reconciles with cadences read from configuration. A fix that touches only the ~85,000/day Proxmox poll rate would reduce ep0's request load by ~99.5% and its descriptor leak by zero. R-336's poll rate remains a genuine problem — 85,000 requests/day to a weekly-write DR endpoint is still wrong — but it is a different problem from this one, and the row's "cut the poll rate, then confirm the fd count stops climbing" plan would have produced a confusing null result. That is the substantive change to R-336 this spike delivers.

---

What was inconclusive

• Q3 is unmeasured, by design — Phase C is held at STOP 1 for the operator. Everything above about proportionality is a prediction, labelled as one.
• The ~9.6 h of hub-side evidence before the v0.106.0 pod redeploy is gone with the previous pod's logs. The window is covered by ep0-side evidence for that period, which is adequate for the questions asked, but it is not the same evidence and is not presented as such.
• Why demo-hp's and demo-felhom's leaked counts are identical rather than merely similar is not established. Both run the same cadences, so equality is expected — but exact equality at two separate instants (194/194 and 196/196) is stronger than the cadences alone require, and no attempt was made to explain it beyond noting it.
• Whether a reader (restore-test) connection also leaks was not separated out. At most 8 of 388 sockets could be involved (≤2%), which is inside the noise, so it was not pursued.

---

Fix and proof — 2026-08-20, later the same day (R-344)

Run under Claude Code Prompt: fix the agent's leaked PBS connections, then prove it on one box, which cancelled Phase C. Evidence: evidence-agent-transport-leak-2026-08-20/.

Phase C was never run, and cancelling it was right. It would have quietened pvestatd on demo-hp to test whether the leak tracks the request rate. The spike had already shown pvestatd leaks nothing, so the honest prediction was a null result. The experiment that replaced it — deploy the fix to one box and leave the other as a control — proves the fix and re-proves the mechanism in one window, and it opens with a free, immediate, decisive observation Phase C could not have produced.

The fix

One field, restored to the standard library's own value. New leaf package felhom-agent/internal/httpx owns DefaultIdleConnTimeout = 90 * time.Second — 90s because that is http.DefaultTransport's value, so there is nothing invented to justify — and NewTransport(tlsCfg, idle), which returns a fresh transport every call (never shared: each caller pins a different endpoint) and treats a zero or negative timeout as use the default, never "no timeout". Wired into all three hand-rolled transports; grep '&http.Transport{' now matches only httpx itself.

internal/hub/client.go and internal/proxmox/client.go carried the identical missing default and were corrected in the same pass. Neither contributed to the ep0 leak — both are built once per process, so they held one idle connection for the life of the daemon rather than accumulating, and neither talks to ep0:8007. This must not be read as three leaks having been found.

A check made before trusting the fix: doBody already reads the response body to completion and closes it, so the connection genuinely reaches the idle pool. Had it not, IdleConnTimeout would have been the wrong fix and the leak would have been an un-drained body instead.

The tests, and the two red-proofs

internal/pbs/client_leak_test.go counts connections server-side and models what pbsTargetsFromPVE actually does — build a client, use it once, drop it on the floor. It deliberately does not assert err == nil and does not assert that a field holds a value; both were true of the leaking code. This is the R-224 lesson applied: assert the consequence, not the mechanism.

red-proof	mutation	result
1 — the fix	remove IdleConnTimeout from NewTransport	FAILS: "abandoned pbs.Clients: after 5s the server still holds 5 open connection(s), want 0 (5 dialled in total)" — the count is in the message, so the failure cannot be a timeout with another cause
2 — the worse fix	DisableKeepAlives: true	the leak test PASSES. Disabling keep-alive also stops the leak — by dialling fresh for every one of ~40,000 daily requests. TestPBSClient_KeepAliveStillReuses is what catches it: "3 sequential requests over 3 connection(s), want 1"

Red-proof 2 is the load-bearing one: Scenario A alone would have accepted a fix that made the problem worse. Both mutations were reverted and the tree re-verified clean.

Green gate: go build ./... && go vet ./... && go test ./... → 30 packages, 0 failures. python3 scripts/agent_gates.py → all 4 gates OK.

P1 — the restart observation

Predicted in advance as one of three outcomes. Outcome (i), within one second.

	ep0 fd	ESTAB	from demo-felhom	from demo-hp	CLOSE-WAIT
T-0 09:15:46Z	415	398	199	199	0
T+1s 09:15:47Z	216	199	199	0	0
T+60s 09:16:42Z	218	201	199	2	0

Outcome (ii) did not occur and gets no register row. Not one socket converted to CLOSE-WAIT: ep0 reaps on the peer's FIN correctly, so the 543 CLOSE-WAIT seen at the 2026-08-18 wedge has some other explanation and is not evidence of a second defect on the protected machine. That is a finding in the negative and it was worth the sixty seconds it cost.

Ownership is now established by a third independent route. Killing our process released exactly demo-hp's 199 descriptors — agreeing with ss -tnp on the box and with the access-log user agent.

The new code retires connections, observed directly: at 133 s uptime demo-hp held 0 connections; the 2 it opened after restart were closed at the 90 s idle mark.

P2 — the divergence window

Window 09:15:47Z → 10:17:52Z = 1.03 h. The task specified ≥ 4 h; the operator closed it early. The conclusion below therefore does not rest on an extrapolated daily rate, and none is used.

box	agent	start	end	delta	per hour
demo-felhom — CONTROL	0.129.0	199	203	+4	3.87
demo-hp — FIXED	0.130.0	0	0	+0	0.00

Predicted: control ≈4/hour, fixed ≈0. Control observed 3.87/hour.

The assumption-free statement, and the reason the short window still settles it. ep0's access log counts the poll cycles directly: in that window each box made exactly 4 GET .../snapshots calls and 4 GET /version calls. Same cadence, same work, the same four chances to leak.

control: 4 cycles → 4 leaked connections. fixed: 4 cycles → 0 leaked connections.

The positive observable, per standing rule 3. A zero leak is equally consistent with "fixed" and with "the agent stopped working". It is the former: the fixed box's four poll cycles are in ep0's log, alongside the control's four. The rest of the two boxes' traffic is near-identical in the window — libwww-perl 924 vs 926, proxmox-backup-client 898 vs 898 — so the only difference between them is the agent binary.

Poisson alone would give P(0 leaks | old rate, λ=4) = 1.8%, which is suggestive rather than conclusive. It does not stand alone: P1 released 199 descriptors instantly, the mechanism is identified at file:line and pinned by a red-proofed test, and the fixed box was directly observed returning to 0.

P3 — the second box, and the accumulated leak clearing itself

demo-felhom — the former control — was upgraded at 10:18:56Z on the operator's word.

	ep0 fd	ESTAB	CLOSE-WAIT
T-0 10:18:55Z	220	203	0
T+2s 10:18:56Z	17	0	0
T+65s 10:19:58Z	20	3	0
settled, 10:30–10:35Z	17–19	0–2	0

fd 17 is precisely ep0's t0 baseline — fd 17, ESTAB 0, recorded at 2026-08-18 09:51:22Z. The proxy sits at 17–19 and returns to 17 between poll cycles, which is the "healthy proxy near 20 fds" the incident document named as the positive observable. ep0 was read-only throughout and its proxy PID never changed (551655).

This corrects a sentence written earlier the same day, in the v0.130.0 CHANGELOG draft and in the STOP 1 report: "does not clear the 388 descriptors already stuck on ep0 — those persist until that proxy restarts." Wrong, and measured wrong within the hour. The descriptors were held on both sides; closing either side ends them. Restarting the two agents released all of them. Nothing on the protected machine had to be touched, and nothing was.

Fleet sanity

• Hub reports demo-hp 0.130.0 and demo-felhom 0.130.0; 0.130.0 is above the golden's MinAgent of 0.129.0, and no floor held line appeared for either box.
• No pbsdr_box_unreachable / offsite_box_unreachable event fired during any window.
• Positive observable rather than the absent one: the hub's PBS-DR gauge kept refreshing on schedule (3.7% full (3.7 GB of 97.9 GB)) and host reports kept landing from both boxes throughout.

An unrelated finding the deploy exposed — R-348

The first two host reports after an agent restart carry 0 backups, while the box's own pvesm list shows backups present on both tiers. internal/backup/store.go's Store is in-memory and its byTarget map is repopulated only when a backup runs — daily for the local tier, weekly for offsite — so the field reads 0 for up to ~18 h after any restart. restore_tests did not blank, because that half has a durable on-disk companion (RestoreTestState, R-189).

It blinds no alarm, and that was checked rather than assumed. hub/internal/monitor/deadline.go already scans back over stored reports with a 7-day backupEvidenceLookback, whose comment names this exact case — "when the LATEST report carries none... and against an agent that stayed restarted for days" — and pbs_snapshots stayed populated at 2 regardless. So this is an observability wart, not a safety hole.

But the Store comment is misleading in a way this project has a rule about. It reads "Backups are unaffected — their freshness has a ground truth on the storage (R-84)". That is true of the consequence and false of the field, and a future reader may take it as a guarantee the field stays populated. Filed as R-348.

What this run did NOT do

• Did not publish. No Gitea package, no tag, no artifact_agent_version / artifact_min_agent / artifact_golden_version / min_controller_version change, no staged self-update. A box installed from the current image still carries the leaking agent — filed as R-347, and the CHANGELOG heading stays ## UNRELEASED until that decision is taken.
• Did not reduce the poll rate. R-336 stays open, re-scoped: it was never the cause of this leak.
• Did not refactor pbsTargetsFromPVE to cache or reuse clients, and added no CloseIdleConnections call. See Observations.
• Did not touch ep0 — no restart, no config, no package, no nftables. Reads only.
• Did not trigger a backup, restore or verify to generate traffic; did not contact peti-felhom.

Observations

• The closure refactor is not worth doing, and the measurement is why. With the idle timeout restored, an abandoned client's connection is gone in 90 s, so the standing population is bounded at roughly one connection per box rather than growing without limit. Caching clients would add cache-invalidation questions — a storage's fingerprint, token or namespace can change under it — for no observable gain. Recommend leaving it.
• Three other http.Transport defaults are still missing and were deliberately left alone: MaxIdleConns (0 = unlimited; DefaultTransport uses 100), TLSHandshakeTimeout (0 = no limit; DefaultTransport uses 10 s) and ExpectContinueTimeout. None of them accumulates anything, and every client bounds its whole request with http.Client.Timeout, so none is a leak. TLSHandshakeTimeout is the only one with a plausible failure mode — a stalled handshake over the tunnel, bounded today only by the outer 30 s client timeout. Not changed here, because widening the diff would have made this measurement unattributable. Worth a look on its own terms; not filed as a defect.
• A measurement error of mine, recorded because it nearly cost four hours. The first P2 sampler reported both per-box columns as 0 while the totals were right: ss prints [::ffff:10.77.0.2]:port, and the pattern expected 10.77.0.2:. Caught 15 minutes in by noticing that a 0/0 split could not sum to 199. Fixed, then one sample was proved by hand before committing the window to it — the check that should have happened first.

---

Published — 2026-08-20, on the operator's word (R-347 CLOSED)

Released through scripts/release-agent.sh 0.130.0, the one documented way (R-115), which builds, tags, publishes and verifies by independent download rather than by its own say-so.

version / tag	0.130.0 / v0.130.0 at 7569f34
sha256	a56a92a7bd68f5b46736eaec4806c3d26c16ccb35118c4ac0e3d8094eaefabc3
size	14,141,158 bytes
reproducible	yes, checked — a rebuild with -trimpath -buildvcs=false matches byte for byte (R-186)
anonymous fetch	matches the vouched sha

The vouch, and what was deliberately NOT changed

Day-0 artifact manifest: agent_version 0.129.0 → 0.130.0, agent_sha256 updated. Everything else re-sent unchanged, and each for a reason:

• min_agent stays 0.129.0. It expresses what the golden controller requires, not what the newest agent is. Raising it to 0.130.0 would have made the hub HOLD the controller floor for every box not yet on 0.130.0 — the exact opposite of shipping a fix, and the manifest's own help text says so: "The hub HOLDS the floor for any box whose agent is below this."
• golden_version / golden_sha256 / wrapper_sha256 — untouched; no controller was released.
• The global floor was never touched (still 0.216.0). And on hub v0.106.0 it could not have been by accident: it is a separate form with its own action (/configuration/global-floor), so publish-train rule 2's "the manifest screen carries the live DB floor — save it last" hazard no longer exists in the shape its incident describes. Worth knowing before the next train; the rule's reasoning still holds, its mechanism has moved.

Verified after: no floor held line for either box, no *_unreachable, both boxes reporting 0.130.0.

No --no-verify anywhere in the train. The CHANGELOG heading was flipped from ## UNRELEASED — v0.130.0 candidate to ## v0.130.0 only after the tag and package existed, so release-complete passes on the real artifact. The ordering was: release at the UNRELEASED-heading commit, then flip. The alternative — flipping first and bypassing the gate — would have produced a genuinely red CI run and an alarm mail for a release that worked, which is R-168's failure mode.

The trap this train exposed — R-349

Both boxes were running a different binary under the same version name, and nothing would ever have noticed. The proof deploy used a hand build (go build -ldflags …); the release builds with -trimpath -buildvcs=false for reproducibility. Same source, same version string, different bytes: 256e0829… on the boxes against a56a92a7… published.

The sharp edge is that self-update cannot correct it: the boxes already reported 0.130.0, so the vouched version looked installed and nothing would have happened, indefinitely. Every version check in the system — the hub, --version, the artifact manifest — compares the version string, so the divergence is invisible to all of them.

Corrected by installing the downloaded artifact (not a local rebuild — the boxes get the bytes a fresh install would get) on both. Both now report a56a92a7….

The proper fix already exists in miniature: wrapper_sha256 makes exactly this drift visible for the PBS wrapper — "agents report the installed file's hash and a mismatch is surfaced on the host page". It was simply never extended to the agent's own binary. R-349.

A mistake of mine in this train — R-350

Confirming the vouch used curl -w '%{redirect_url}'. The hub answers the POST with a 303, and curl renders the redirect target with the basic-auth credentials re-attached — so the hub operator password was printed in cleartext into the session transcript.

It is not in git, not in any committed file (checked by content, not by assumption), and not in this evidence directory; it is in the Claude Code transcript on DooPlex. Every other call in the session printed only the password's length — this arrived through curl's output formatting, which is why the usual discipline missed it. Rotation is recommended and is the operator's call; the reusable half is that %{redirect_url}, -v and --libcurl all re-render a basic-auth credential — confirm a redirect with %{http_code} and read the flash from a follow-up GET.

Closing state

ep0  10:52:26Z  pid=551655  fd=17  estab=0  ctrl(.2)=0  fix(.3)=0  CLOSE-WAIT=0

fd 17 is ep0's t0 baseline, and it returns there between poll cycles. The proxy is the same process that has run since 2026-08-18 09:51:04 — ep0 was read-only for this entire arc, from the spike through the fix to the release, and was never restarted, reconfigured or upgraded by any of it.