felhom.eu/REPORT-agent-transport-leak.md

# REPORT — R-344: the agent's leaked PBS connections, fixed and proven on both boxes (2026-08-20)

**Outcome: the fix works, measured three independent ways; ep0 is back to its baseline of 17 file
descriptors from 415; and 0.130.0 is now published and vouched.** Both demo boxes run the **byte-exact
published artifact**. R-347 is CLOSED. Two new findings came out of the release itself — **R-349**
(the fleet was briefly running a different binary under the same version name) and **R-350** (I printed
the hub password into the transcript; rotation is your call).

## 1. Confirmed baselines

| repo | in | out |
|---|---|---|
| `felhom-agent` | `f17ed11` — v0.129.0 | **`ede49b6`** — 0.130.0, UNRELEASED |
| `felhom.eu` | `9299f85` | docs + register only |

Both clean and equal to `origin/main` before each build. Agent version in: 0.129.0 on both boxes.
Out: **0.130.0 on both**, confirmed from the hub, not from the boxes' own `--version`.

## 2. The diff

| file | symbol | change |
|---|---|---|
| `internal/httpx/transport.go` | **new package** | `DefaultIdleConnTimeout = 90s`; `NewTransport(tlsCfg, idle)` — fresh transport per call, `<= 0` means **use the default, never "no timeout"** |
| `internal/httpx/transport_test.go` | new | zero/negative → default; the constant is read off `http.DefaultTransport`; freshness; TLS config preserved |
| `internal/pbs/client.go` | `Config`, `NewClient` | `IdleConnTimeout` field (tests only); transport via `httpx` |
| `internal/pbs/client_leak_test.go` | new | Scenarios A/C + the production-default pin |
| `internal/hub/client.go` | `NewClient` | via `httpx` — **consistency only, did not contribute to the leak** |
| `internal/proxmox/client.go` | `NewClient` | same |
| `cmd/felhom-agent/main.go` | `version` | 0.92.1 → 0.130.0 (ldflags default) |
| `REUSE.md`, `CHANGELOG.md` | — | `httpx.NewTransport` entry; the release note |

Commit `ede49b6` on `main`. `grep '&http.Transport{'` now matches only `httpx` itself.

**A check made before trusting the fix:** `doBody` already reads the body to completion and closes it,
so the connection genuinely reaches the idle pool. Had it not, the idle timeout would have been the
wrong fix entirely.

## 3. Tests and the red-proofs

`go build ./... && go vet ./... && go test ./...` → **30 packages, 0 failures.**
`python3 scripts/agent_gates.py` → **all 4 gates OK.**

The leak test counts connections **server-side** and models what `pbsTargetsFromPVE` does — build a
client, use it once, drop it. It deliberately does **not** assert `err == nil` or that a field holds a
value; both were true of the leaking code (the R-224 lesson).

| red-proof | mutation | seen failing with |
|---|---|---|
| 1 — the fix | remove `IdleConnTimeout` | *"abandoned pbs.Clients: after 5s the server still holds 5 open connection(s), want 0 (5 dialled in total)"* — the count is in the message, so it cannot be a timeout with another cause |
| 2 — the worse fix | `DisableKeepAlives: true` | **the leak test PASSES.** Caught only by `TestPBSClient_KeepAliveStillReuses`: *"3 sequential requests over 3 connection(s), want 1"* |

**Red-proof 2 is the load-bearing one: Scenario A alone would have accepted a fix that made the problem
worse** — no leak, at the price of a fresh dial for every one of ~40,000 daily requests. Both mutations
reverted, tree re-verified clean.

## 4. §4 quoted, beside the results

> **P1** — *"(i) ep0's fd count falls by ≈194 within seconds … (ii) … ≈194 sockets convert ESTAB →
> CLOSE-WAIT … (iii) neither — the count barely moves. Then the ownership attribution is wrong and the
> finding must be withdrawn."*
> **P2** — *"demo-hp (fixed): ≈ 0 … demo-felhom (control, untouched): ≈ 4 per hour → ≈ 16 over 4 h."*
> **P3** — *"ep0's overall leak rate should fall from ≈ 200/day to ≈ 100/day while one box is fixed, and
> to ≈ 0/day after Part 5."*

## 5. P1 — **outcome (i)**, in one second

| | ep0 fd | ESTAB | demo-felhom | demo-hp | CLOSE-WAIT |
|---|---|---|---|---|---|
| T-0 `09:15:46Z` | 415 | 398 | 199 | **199** | 0 |
| T+1s `09:15:47Z` | **216** | **199** | 199 | **0** | **0** |
| T+60s `09:16:42Z` | 218 | 201 | 199 | 2 | 0 |

**Outcome (ii) did not occur, so it gets no register row.** Not one socket converted to `CLOSE-WAIT`:
ep0 reaps on peer FIN correctly. That also means the 543 `CLOSE-WAIT` at the 2026-08-18 wedge has some
other explanation and is **not** evidence of a second defect on the protected machine — a finding in the
negative, worth the sixty seconds it cost.

Ownership is now proven a **third** independent way: what dies with the process, agreeing with
`ss -tnp` and with the access-log user agent. At 133 s uptime the fixed box held **0** connections.

## 6. P2 — divergence

**Window 09:15:47Z → 10:17:52Z = 1.03 h. You closed the ≥4 h window early**, so no daily rate is
extrapolated and none is needed.

| box | agent | start | end | delta | per hour | predicted |
|---|---|---|---|---|---|---|
| `demo-felhom` CONTROL | 0.129.0 | 199 | 203 | **+4** | 3.87 | ≈4 |
| `demo-hp` FIXED | 0.130.0 | 0 | 0 | **+0** | 0.00 | ≈0 |

**The assumption-free statement.** ep0's access log counts the opportunities: each box made **exactly 4
`/snapshots` and 4 `/version` calls** in the window.

> **control: 4 cycles → 4 leaks. fixed: 4 cycles → 0 leaks.**

**Positive observable (standing rule 3):** a zero leak is equally consistent with "the agent stopped
working" — it did not; its four cycles are in ep0's log. The boxes' other traffic is near-identical
(`libwww-perl` 924 vs 926, `proxmox-backup-client` 898 vs 898), so **the only difference between them
is the binary**. Poisson alone gives P(0 | λ=4) = **1.8%**, which is suggestive rather than conclusive
and is not relied on alone.

## 7. P3 — the second box, and the backlog clearing itself

`demo-felhom` upgraded `10:18:56Z` on your word.

| | ep0 fd | ESTAB | CLOSE-WAIT |
|---|---|---|---|
| T-0 `10:18:55Z` | 220 | 203 | 0 |
| **T+2s** | **17** | **0** | 0 |
| settled 10:30–10:35Z | **17–19** | 0–2 | 0 |

**17 is precisely ep0's `t0` baseline** (fd 17, ESTAB 0, 2026-08-18 09:51:22Z), and it returns to 17
between poll cycles — the "healthy proxy near 20 fds" the incident document named. Predicted ≈0/day
residual; **observed the baseline itself.**

**A correction, made within the hour it was written.** My STOP 1 report and the first CHANGELOG draft
said *"does not clear the 388 descriptors already stuck on ep0 — those persist until that proxy
restarts."* **Wrong.** They were held on both sides; restarting the agents released every one. **ep0 was
read-only throughout and its proxy PID never changed (551655).** Corrected in the CHANGELOG, the audit
document and R-344 rather than quietly edited.

## 8. Fleet sanity

Hub reports **0.130.0 on both** boxes. **No `floor held`** line (0.130.0 > golden MinAgent 0.129.0).
**No `pbsdr_box_unreachable` / `offsite_box_unreachable`** during any window. Positive observable
rather than the absent one: the PBS-DR gauge kept refreshing (`3.7% full (3.7 GB of 97.9 GB)`) and host
reports kept landing from both boxes throughout.

## 9. What is NOT done

- **Not published.** No package, no tag, no manifest or floor field touched, no self-update staged.
  **A box installed from the current image still ships the leaking agent** — **R-347**, your call.
- The CHANGELOG heading is `## UNRELEASED — v0.130.0 candidate`. The `release-complete` gate convicted
  on `## v0.130.0` because there is no tag and no package, and **it was right to**. I did not use
  `--no-verify`; I made the heading stop claiming a release that has not happened. It flips to
  `## v0.130.0` in the same commit as the tag.
- Poll rate unchanged (**R-336**, re-scoped). `pbsTargetsFromPVE` not refactored; no
  `CloseIdleConnections` added. ep0 not touched.
- **The 388 descriptors ARE cleared** — see §7. This is the one item the prompt expected to remain
  outstanding, and it did not.

## 10. Register

- **R-344** — updated with the fix, P1's outcome named, and P2/P3's numbers. **Left OPEN**, because a fix
  on two boxes by hand is not delivered.
- **R-336 — re-scoped.** Its new next-step cell, verbatim: *"**NEW ACCEPTANCE CRITERION, since the old
  one is void:** the fd count is NOT the observable for this row any more — that belongs to R-344 and is
  already satisfied. Measure the REQUEST RATE at ep0's access log, and state the projected rate at the
  target customer count."* The row now records explicitly that its old next-step **would have "fixed"
  nothing while looking like a failed fix**, and re-scopes it to what it is: ~85,000 requests/day to a
  weekly-write DR endpoint, ≈**25 requests/second at fifty customers** against a CX33.
- **R-347 (new)** — the delivery gap. Owner: **Viktor decides**, CC executes.
- **R-348 (new)** — an agent restart blanks the reported backup list for up to ~18 h, and the `Store`
  comment calls backups *"unaffected"*. **Blinds no alarm** — checked, not assumed: the hub's
  `backupEvidenceLookback` scans 7 days for exactly this case, and `pbs_snapshots` stayed populated.
- **No P1(ii) row**, because outcome (ii) did not occur.
- **R-346** — this run anchored on the measured `t0` (fd 17 at 2026-08-18 09:51:22Z), never on a systemd
  timestamp, so the 5 h 56 m discrepancy did not touch these numbers.

## 11. CI, by run ID — and the claim ledger

| repo | run | sha | conclusion |
|---|---|---|---|
| `felhom-agent` | id **362** / run_number 50 | `ede49b610` — the fix | **success** |
| `felhom-agent` | id **363** / run_number 51 | `7569f34ae` — the correction | **success** |
| `felhom.eu` | id **364** / run_number 239 | `57dd62b09` — the write-up | **success** |

No `--no-verify` anywhere. Both repos' pre-push hooks ran their gate entry point and passed;
`release-complete` passes on v0.129.0, which is the honest state while 0.130.0 is unpublished.

`python3 scripts/unproven.py --summary` — **unchanged: 23 walked, 32 not walked of 55.** No number
moved, and correctly so: this run proved an engineering fact about our own connection handling, not a
customer-facing product claim.

**Closing state of ep0**, read one last time after everything:

```
t=10:40:23Z pid=551655 fd=17 estab=0 ctrl(.2)=0 fix(.3)=0 CLOSE-WAIT=0
```

`CLOSE-WAIT 0`, `ESTAB` in the low single digits, `fd` at the baseline, proxy PID **551655** — the same
process that has been running since 2026-08-18 09:51:04, never restarted by this work.

## 11b. The release (R-347, CLOSED)

`bash scripts/release-agent.sh 0.130.0` — the one documented way (R-115): build, tag, publish, and
**verify by independent download**.

| | |
|---|---|
| tag | `v0.130.0` at `7569f34` |
| sha256 | **`a56a92a7bd68f5b46736eaec4806c3d26c16ccb35118c4ac0e3d8094eaefabc3`** |
| size | 14,141,158 bytes |
| reproducible | **yes, checked** — `-trimpath -buildvcs=false` rebuild matches byte for byte |

**Vouched** in the Day-0 artifact manifest: agent 0.129.0 → **0.130.0** + its sha. **Only the agent
fields changed.**

- **`min_agent` left at 0.129.0** — it states what the **golden controller** needs. Raising it to
  0.130.0 would have made the hub **HOLD the floor** for every box below 0.130.0, which is the
  opposite of shipping a fix.
- **Global floor never touched** (0.216.0). On hub v0.106.0 it is a *separate form with its own
  action*, so publish-train rule 2's "save the floor last" hazard no longer exists in the shape its
  incident describes — the rule's reasoning holds, its mechanism has moved.
- After: no `floor held`, no `*_unreachable`, both boxes reporting 0.130.0, artifact downloading
  anonymously at the vouched sha.

**No `--no-verify` in the train.** The heading was flipped to `## v0.130.0` only after tag and package
existed. Flipping first and bypassing would have produced a red CI run and an alarm mail for a release
that worked — R-168's failure mode.

## 11c. Two findings from the release

**R-349 — the fleet was running a different binary under the same version name.** The proof deploy was
a hand build; the release builds `-trimpath -buildvcs=false`. Same source, same version string,
different bytes (`256e0829…` vs `a56a92a7…`). **Self-update could never have corrected it** — the boxes
already reported 0.130.0, so the vouched version looked installed. Every version check in the system
compares the *string*. Fixed by installing the **downloaded** artifact on both. The proper fix already
exists in miniature: `wrapper_sha256` does exactly this drift detection for the PBS wrapper and was
never extended to the agent's own binary.

**R-350 — I printed the hub password into the transcript.** Confirming the vouch used
`curl -w '%{redirect_url}'`; the hub answers 303 and curl re-attaches the basic-auth credential to the
redirect target it prints. **Not in git, not in any committed file** (checked by content), not in the
evidence directory — it is in the session transcript on DooPlex. Every other call printed only the
length; this came through curl's own formatting. **Rotation is your call** — I did not do it
unilaterally, and I can do it file-to-file without printing the new value if you want. The reusable
half: `%{redirect_url}`, `-v` and `--libcurl` all re-render a basic-auth credential.

## 12. Observations

- **The closure refactor is not worth doing — recommend leaving it.** With the idle timeout restored an
  abandoned client's connection is gone in 90 s, so the standing population is bounded at about one
  connection per box instead of growing without limit. Caching clients would add cache-invalidation
  questions (a storage's fingerprint, token or namespace can change under it) for no observable gain.
- **Three other `http.Transport` defaults are still missing and were left alone:** `MaxIdleConns`,
  `TLSHandshakeTimeout` (0 = no limit; `DefaultTransport` uses 10 s) and `ExpectContinueTimeout`. None
  accumulates, and every client bounds its request with `http.Client.Timeout`. `TLSHandshakeTimeout` is
  the only one with a plausible failure mode — a stalled handshake over the tunnel, bounded today only
  by the outer 30 s. Not changed, because widening the diff would have made this measurement
  unattributable. Worth a look on its own terms; not a defect.
- **A measurement error of mine, recorded because it nearly cost four hours.** The first P2 sampler
  reported both per-box columns as 0 while the totals were right: `ss` prints `[::ffff:10.77.0.2]:port`
  and my pattern expected `10.77.0.2:`. Caught 15 minutes in, because a 0/0 split cannot sum to 199.
  Fixed, then **one sample proved by hand before committing the window** — which is what should have
  happened first.