felhom.eu/documentation/audits/SPIKE-ep0-established-connections-2026-08-20.md

# SPIKE — who holds ep0's connections open, and does the leak track the poll rate?

**Run:** 2026-08-20 08:01–08:35 UTC, read-only throughout. **Phase C (the one mutation) has NOT been run** — it is
held at STOP 1 for the operator's window decision.
**Evidence:** `evidence-ep0-established-connections-2026-08-20/`, one file per step.
**Supersedes the plan in** `SPIKE-ep0-established-connections.md` (issued 2026-08-18, never run).

> **Headline, and it is not the headline the spike expected.**
> The leak is **ours**. Every one of the 388 leaked descriptors on ep0 is an established connection held
> open by **`felhom-agent`** on the customer boxes — one per PBS poll cycle, never closed on either side.
> `pvestatd` and `proxmox-backup-client` made **162,404 requests** in the same window and leaked **zero**.
> R-336's premise — that cutting the ~85,000/day poll rate is the leak fix — **does not survive this
> measurement.** The poll rate is a separate (real) problem; it is not what is consuming descriptors.

---

## Q0 — R-341's first dated check: has the fd slope changed since the PBS 4.2.5-1 upgrade?

### Answer: NO. Unchanged, and now measured over a window 88× longer than the one it replaces.

**Precondition passed.** `MainPID = 551655`, `ps -o lstart` = `Tue Aug 18 09:51:04 2026`, `NRestarts = 0` on
both `proxmox-backup-proxy` and `proxmox-backup`. The proxy generation that carries `t0` is the one that was
read. The elapsed-window measurement is valid.

| | value |
|---|---|
| `t0` | fd **17**, ESTAB 0, 2026-08-18 **09:51:22Z**, PID 551655 |
| reading A | fd **405**, ESTAB **388**, CLOSE-WAIT **0**, 2026-08-20 **08:02:13Z**, PID 551655 |
| elapsed | **166,251 s** = 46.181 h = 1.9242 d |
| delta | **+388 fd** |
| rate | **201.6 fd/day** (8.40/h) |
| Poisson on n=388 | ±19.7 counts → **±10.2/day (1σ)**, ±20.5/day (2σ) → **181.2 … 222.1 /day** |
| limits | soft/hard **65536 / 65536** — the drop-in survived the upgrade |
| listener | `LISTEN Recv-Q 0 Send-Q 1024` — accept queue empty, nothing wedged |

**Verdict against the pre-registration**, quoted from the spike prompt §2 so it is visible beside the result:

> *"Predicted range: roughly **370–450** … A reading anywhere in **330–490** confirms an unchanged rate …
> **In range** → unchanged rate. This is the EXPECTED result, consistent with the changelog containing no
> mechanism to change it. It is not a failed upgrade."*

**Observed: 388.** In range, and close to the centre. The rate is **unchanged** since the upgrade. This is the
expected result and it is recorded as such — not as a success and not as a failure of the upgrade, which was
run for rehearsal value and never claimed to fix anything.

For comparison, on the identical method: **183/day** (31 min, pre-upgrade), **200/day** (5.64 h, pre-upgrade),
**225/day** (32 min, post-upgrade). This window carries **97× the descriptor count** of the 31-minute window,
so its uncertainty is ±5% rather than ±50%.

### Composition — the split, which is what says *which* leak it is

```
t0        fd  17   ESTAB   0   CLOSE-WAIT 0
reading A fd 405   ESTAB 388   CLOSE-WAIT 0          17 + 388 = 405, exactly
```

**`CLOSE-WAIT` is absent from the `ss` histogram entirely — 0, not merely flat.** 100% of the growth is
**established** connections on `:8007`. This closes the question the incident's own correction block opened:
the mechanism is *connections the proxy never reaps*, and `CLOSE-WAIT` is not merely the minority half, it is
**not present at all** on this proxy generation.

### Runway

From fd 405 at 201.6/day to the 65536 soft limit: **≈ 323 days**, i.e. around **2027-07-09**. Under a year.
A deadline, not a comfort — unchanged in character from what R-341 recorded.

---

## Q1 — who is at the far end?

### Answer: exactly the two demo boxes, in a dead-even split. No third peer. Outcome (c) is excluded.

At 08:02:39Z, `ss -tn state established '( sport = :8007 )'`, peer histogram:

| peer | connections | identity |
|---|---|---|
| `10.77.0.2` | **194** | `demo-felhom` (host `felhom-pve`) — token `felhom@pbs!demo-felhom` |
| `10.77.0.3` | **194** | `demo-hp` (host `felhom-host`) — token `felhom@pbs!demo-hp` |

The identity is not inferred from the address: ep0's own access log carries the API token name on every line
(`::ffff:10.77.0.3 - felhom@pbs!demo-hp …`), and the mapping is one-to-one across the whole file.

**194 / 194 is not approximately even — it is exactly even**, which is itself a finding: whatever opens these
is a per-box periodic task running the same schedule on both, not traffic proportional to anything the boxes
differ in.

### Descriptor-type breakdown (1.3)

`lsof -np 551655`, which distinguishes sockets from files, pipes and eventfds so a "leak" that is not sockets
gets caught here:

| type | count |
|---|---|
| **IPv6 (TCP)** | **390** |
| REG (log files, rrd journal, datastore lock) | 32 |
| unix | 7 |
| a_inode (epoll ×2, eventfd ×1) | 3 |
| DIR | 2 |
| CHR (`/dev/null`) | 1 |

`lsof … | grep -c TCP` = **390** = 388 established + 1 listener + 1 other. The independent `/proc/<pid>/fd`
link-target histogram agrees: **397 sockets** (390 TCP + 7 unix), 9 regular/anon fds. **The leak is sockets,
and the sockets are TCP on :8007.** Nothing else is growing.

### 1.4 — persistence: are they long-lived, or churn?

Full 4-tuples diffed between reading A (08:02:39Z) and reading B (08:33:42Z), **1,843 s apart**:

| | count |
|---|---|
| sockets in reading A | 388 |
| sockets in reading B | 392 |
| **in BOTH — long-lived** | **388** |
| **only in A — closed during the interval** | **0** |
| only in B — new arrivals | **4** (2 per box) |

**Not one socket closed in 31 minutes.** There is no churn to separate from the leak: every established
connection on this proxy is the leak. The 4 arrivals over 1,843 s = 187.5 fd/day, consistent with the
elapsed-window 201.6/day within Poisson on n=4.

**Idle age and liveness.** All 392 sockets carry a TCP keepalive timer and **all 392 report `retrans=0`** —
the far end is answering keepalive probes. These are not half-open sockets whose peer went away; they are
mutually held, live, idle connections. That, on its own, points at outcome **(a)** and Part 2 confirms it.

---

## Q2 — do the boxes hold them too?

### Answer: **outcome (a)** — both sides hold every connection, and the two counts match exactly. Twice.

| instant (UTC) | ep0 ESTAB on `:8007` | `demo-felhom` ESTAB to `10.77.0.1:8007` | `demo-hp` | sum |
|---|---|---|---|---|
| reading A — ep0 08:02:39, boxes 08:04:37/38 | **388** | **194** | **194** | **388** |
| reading B — ep0 08:33:42, boxes 08:34:01/02 | **392** | **196** | **196** | **392** |

Two independent paired readings, both exact. And the confirmation is tighter than the totals: the **four new
sockets** that appeared on ep0 during the persistence window carry **the same four source ports** the boxes
report as new —

```
ep0 new:            10.77.0.2:45620  10.77.0.2:53226  10.77.0.3:37554  10.77.0.3:46214
demo-felhom new:    10.77.0.2:45620  10.77.0.2:53226
demo-hp new:        10.77.0.3:37554  10.77.0.3:46214
```

Both boxes also closed **zero** sockets in the same interval (A=194, B=196, closed=0, new=2, on each).

**This is not a hedge between two outcomes.** Outcome (b) — half-open sockets ep0 never noticed — is
excluded by both the matching counts and the `retrans=0` keepalives. Outcome (c) is excluded by Q1.

### And the owner has a name

`ss -tnp` on the boxes attributes **194 of 194** on each, to a single PID:

| box | PID | process |
|---|---|---|
| `demo-felhom` | 2596329 | `/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json` |
| `demo-hp` | 3199936 | `/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json` |

**Zero are held by `pvestatd`. Zero by `proxmox-backup-client`.** Both agents have been running continuously
since 2026-08-12 (they did not restart); the count restarted at the proxy's 08-18 restart because that
restart closed the old sockets on both sides.

Each agent's own descriptor table is **208 fds, of which 196 are these connections** — so the leak is
bilateral. The agents' `RLIMIT_NOFILE` is 524287, so they are in no danger; that is a reason it stayed
invisible, not a reason it is harmless.

---

## The user-agent attribution (1.5) — the access log settles it independently

Over the same 46.18 h window, from ep0's own `access.log.1` + `access.log`:

| user agent | endpoint | requests | req/day | **descriptors leaked** |
|---|---|---|---|---|
| `libwww-perl/6.78` | `GET /admin/datastore` (pvestatd) | 81,192 | 42,195 | **0** |
| `proxmox-backup-client/1.0` | `GET …/felhom-offsite/status` | 80,061 | 41,607 | **0** |
| `libwww-perl/6.78` | `GET …/felhom-offsite/snapshots` | 1,131 | 588 | **0** |
| **`Go-http-client/1.1`** | **`GET …/felhom-offsite/snapshots`** | **387** | **201** | **388** |
| `Go-http-client/1.1` | `GET /version` | 371 | 193 | (same connections) |
| `Go-http-client/1.1` | `POST …/verify` + task status | 53 | 28 | (same connections) |
| | **total, both boxes** | **163,215** | **84,822** | **388** |

`Go-http-client/1.1` is `felhom-agent` — the same process `ss` names on the box side, reached by a completely
independent route. **387 agent `/snapshots` calls against 388 leaked sockets: one per call, within one.**

**162,404 requests from the two Proxmox pollers leaked nothing.** They are 99.5% of the traffic and 0% of the
leak.

### Cadence reconciliation — the numbers close

| source | cadence | cycles in 166,251 s |
|---|---|---|
| agent hub-report poll (`poll_seconds: 900`, read from each box's `agent.json`) | 15 min | 184.7 |
| `pbs.VerifyLoop` (`DefaultVerifyCadence = 6h`) | 6 h | 7.7 |
| | **sum** | **192.4** |

Observed leaked per box: **194**. Observed `POST …/verify` per box in the window: **8** (predicted 7.7).
Observed `GET /version` per box: ~185.5 (predicted 184.7). Mean interval between leaked descriptors per box:
**857 s**. Every count reconciles against a cadence read from configuration, not fitted to the data.

### Mechanism — named from source, and it is a two-line Go footgun

`felhom-agent/internal/pbs/client.go:56-60`:

```go
http: &http.Client{
    Timeout:   timeout,
    Transport: &http.Transport{TLSClientConfig: tlsCfg},
},
```

A hand-rolled `http.Transport` takes the **zero value** for `IdleConnTimeout`, which in Go means *no limit* —
idle keep-alive connections are never closed. (`http.DefaultTransport` sets 90 s; a composite literal does
not inherit it.) That alone would be harmless if one client were reused. It is not:
`felhom-agent/cmd/felhom-agent/main.go:1486-1509` — `pbsTargetsFromPVE` — says so in its own doc comment:

> *"returns a `pbs.Targets` closure that, **each cycle**, discovers the pbs storages from the PVE config and
> **builds a fingerprint-pinned, token-authed client for each**"*

So each cycle constructs a **new** `Transport`, performs one request, and leaves the connection idle in that
transport's pool forever. The transport then becomes unreachable, but Go's `persistConn` read-loop goroutine
keeps the socket alive — an unreachable `http.Transport` does not close its connections. `grep` across the
whole `felhom-agent` repo for `CloseIdleConnections`, `IdleConnTimeout`, `MaxIdleConns`: **no matches at all.**

**The same composite-literal pattern appears in two more clients** — `internal/hub/client.go:53` and
`internal/proxmox/client.go:69`. Those are built once at start-up rather than per cycle, so they do not leak
by this route; they are recorded because the pattern is one refactor away from doing so. Filed as **R-344**.

---

## Control-window cleanliness (1.6)

The elapsed window is doing double duty as Phase C's control, so it has to be shown undisturbed. It was.

| check | finding |
|---|---|
| **ep0 host reboot** | none — booted 2026-08-03 11:12:57, up 17 d |
| **proxy / API daemon restart** | none — `NRestarts = 0` on both units, PID 551655 throughout |
| **box reboots** | none — both booted 2026-08-10 09:2xZ, up 10 d |
| **agent restarts** | none — both `ActiveEnterTimestamp` 2026-08-12 |
| **request-rate gaps** | **none.** Hourly totals across the window: 3,536–3,553 every hour, no hour missing |
| **HTTP errors** | **none.** Every response in the window is `200`, except 8 × `101` (expected protocol upgrades) |
| **tunnel flaps** | no evidence — WireGuard handshakes for `10.77.0.2`, `.3`, `.250` all seconds old; and a flap would have shown as a request gap or errors, and neither exists |
| **a backup inside the window?** | **NO.** The last offsite protocol upgrades were 2026-08-18 **03:57 and 03:58Z** — the incident re-runs, **before `t0` at 09:51:22Z**. Both boxes' `pvesm list felhom-pbs` confirms the newest snapshots are `2026-08-18T03:57:43Z` / `03:58:43Z`. The tier's `cadence_seconds` is **604800** (weekly), read from each box's `agent.json`, so the next offsite run is due ~2026-08-25 |
| **other non-poll activity** | **yes, and it is routine, quantified, and does not contaminate.** Server-side verify jobs ran on ep0 on a ~6 h cadence (8 per box in the window), and restore-test reads ran on 2026-08-19 at 04:45–04:50 and 06:52–06:54Z (8 reader upgrades total). Together with snapshot/version/task-status calls this is **~1,963 of 163,215 requests = 1.2%** of the window's traffic, and at most 8 of the 388 descriptors could be attributed to reader connections (≤2%). The verify cycle is already counted **inside** the 194/box reconciliation above — it is part of the measured leak, not a contaminant of it |

**Verdict: the control window is clean.** No correction is applied and none is needed.

**Two nuances recorded rather than smoothed over.**

1. `systemctl show proxmox-backup-proxy -p ActiveEnterTimestamp` reads **2026-08-18 03:54:54Z**, not 09:51:04Z,
   while `NRestarts = 0` and `MainPID` did change. The upgrade therefore **re-exec'd** the proxy rather than
   restarting the unit — systemd never saw a stop. Anyone using `ActiveEnterTimestamp` as the "when did this
   proxy generation start" anchor would get a figure **5 h 56 m too early** and compute a rate ~12% low (388 fd over 52.1 h instead of 46.2 h = 178.7/day instead of 201.6/day).
   Use `ps -o lstart= -p $MainPID`, as R-341's own command does.
2. ep0 carries a fourth WireGuard peer, `10.77.0.4`, whose last handshake was 2026-08-13 (7 d ago). It is
   `drill-r50-0a4f9a`, a drill host, and its dormant peer entry is already recorded in
   `architecture/_recovery-inventory-2026-07-28.md`. Outside the window; not a contaminant; not a new finding.

**Hub view.** No `pbsdr_box_unreachable` / `offsite_box_unreachable` and no `*_recovered` event was logged.
The positive observable rather than the absent one: the PBS-DR gauge refreshes on schedule
(`PBS-DR box refreshed: 3.7% full (3.7 GB of 97.9 GB)`, every ~16 min through the run) and host reports land
from both boxes. **Honest limit:** the hub pod is 39 h old (redeployed for v0.106.0 at 2026-08-18 19:29), so
hub-side log evidence covers **36.6 of the window's 46.2 hours**. For the first 9.6 h the evidence is ep0's
own access log — which shows no gap and no error — and `NRestarts = 0`. Hub log timestamps render **CEST**.

---

## Q3 — is the leak rate proportional to the request rate?

### NOT ANSWERED BY MEASUREMENT. Phase C has not been run; it is held at STOP 1.

What exists instead is a **pre-registered prediction**, written and committed before any Phase C number can
exist: `evidence-ep0-established-connections-2026-08-20/phaseC-prediction.txt`. Its substance:

- Stopping `pvestatd` on `demo-hp` removes **both** pvestatd-driven streams from that box (the PVE PBS storage
  plugin issues the `libwww-perl` datastore list *and* shells out to `proxmox-backup-client` for the status
  call, both on pvestatd's cycle) — **−98.8% of demo-hp's requests, −49.4% of the fleet total**. If ep0's
  request count does not fall by close to half, the independent variable did not move and no ratio may be
  computed.
- **H1 (what Parts 1–2 predict): NOT proportional.** demo-hp's `felhom-agent` keeps running, so both boxes
  keep leaking. Expect **~201.6 fd/day, unchanged — ~84 descriptors in a 10 h window (2σ 66–102) — split
  symmetrically ~42/~42 between the two peers.**
- **H2 (the hypothesis Q3 was written to test): proportional.** Expect **~101 fd/day — ~42 descriptors in
  10 h (2σ 29–55) — split strongly asymmetrically ~42/~0.**

H1 and H2 do not overlap at 2σ for a window of 10 h or more. The **peer split is the sharpest discriminator**
and it is free: it needs no second box and no arithmetic.

**A result matching H2 falsifies the Parts 1–2 attribution**, and the prediction file says so in those words,
so the outcome cannot be rationalised in either direction afterwards.

---

## Which fix the evidence points at

**No design here, per the spike-first gate — only the target.** The evidence points at
`felhom-agent/internal/pbs/client.go`'s HTTP transport lifetime, not at the poll rate. One `pbs.Client` is
constructed per cycle by `pbsTargetsFromPVE`, each carrying a fresh `http.Transport` whose `IdleConnTimeout`
is the zero value, and nothing ever closes it; the observed leak is exactly one socket per agent
`/snapshots` call, on both sides, on both boxes, and it reconciles with cadences read from configuration. A
fix that touches only the ~85,000/day Proxmox poll rate would reduce ep0's request load by ~99.5% and its
descriptor leak by **zero**. R-336's poll rate remains a genuine problem — 85,000 requests/day to a
weekly-write DR endpoint is still wrong — but **it is a different problem from this one, and the row's
"cut the poll rate, then confirm the fd count stops climbing" plan would have produced a confusing null
result.** That is the substantive change to R-336 this spike delivers.

---

## What was inconclusive

- **Q3 is unmeasured**, by design — Phase C is held at STOP 1 for the operator. Everything above about
  proportionality is a prediction, labelled as one.
- **The ~9.6 h of hub-side evidence before the v0.106.0 pod redeploy is gone** with the previous pod's logs.
  The window is covered by ep0-side evidence for that period, which is adequate for the questions asked, but
  it is not the same evidence and is not presented as such.
- **Why `demo-hp`'s and `demo-felhom`'s leaked counts are *identical* rather than merely similar** is not
  established. Both run the same cadences, so equality is expected — but exact equality at two separate
  instants (194/194 and 196/196) is stronger than the cadences alone require, and no attempt was made to
  explain it beyond noting it.
- **Whether a reader (restore-test) connection also leaks** was not separated out. At most 8 of 388 sockets
  could be involved (≤2%), which is inside the noise, so it was not pursued.

---

# Fix and proof — 2026-08-20, later the same day (R-344)

Run under `Claude Code Prompt: fix the agent's leaked PBS connections, then prove it on one box`,
which **cancelled Phase C**. Evidence: `evidence-agent-transport-leak-2026-08-20/`.

**Phase C was never run, and cancelling it was right.** It would have quietened `pvestatd` on `demo-hp`
to test whether the leak tracks the request rate. The spike had already shown `pvestatd` leaks nothing,
so the honest prediction was a null result. The experiment that replaced it — deploy the fix to one box
and leave the other as a control — proves the fix and re-proves the mechanism in one window, and it
opens with a free, immediate, decisive observation Phase C could not have produced.

## The fix

One field, restored to the standard library's own value. New leaf package `felhom-agent/internal/httpx`
owns `DefaultIdleConnTimeout = 90 * time.Second` — 90s because that is `http.DefaultTransport`'s value,
so there is nothing invented to justify — and `NewTransport(tlsCfg, idle)`, which returns a **fresh**
transport every call (never shared: each caller pins a different endpoint) and treats a zero or negative
timeout as **use the default, never "no timeout"**. Wired into all three hand-rolled transports;
`grep '&http.Transport{'` now matches only `httpx` itself.

`internal/hub/client.go` and `internal/proxmox/client.go` carried the identical missing default and were
corrected in the same pass. **Neither contributed to the ep0 leak** — both are built once per process,
so they held one idle connection for the life of the daemon rather than accumulating, and neither talks
to `ep0:8007`. This must not be read as three leaks having been found.

**A check made before trusting the fix:** `doBody` already reads the response body to completion and
closes it, so the connection genuinely reaches the idle pool. Had it not, `IdleConnTimeout` would have
been the wrong fix and the leak would have been an un-drained body instead.

### The tests, and the two red-proofs

`internal/pbs/client_leak_test.go` counts connections **server-side** and models what
`pbsTargetsFromPVE` actually does — build a client, use it once, drop it on the floor. It deliberately
does not assert `err == nil` and does not assert that a field holds a value; **both were true of the
leaking code**. This is the R-224 lesson applied: assert the consequence, not the mechanism.

| red-proof | mutation | result |
|---|---|---|
| 1 — the fix | remove `IdleConnTimeout` from `NewTransport` | **FAILS**: *"abandoned pbs.Clients: after 5s the server still holds 5 open connection(s), want 0 (5 dialled in total)"* — the count is in the message, so the failure cannot be a timeout with another cause |
| 2 — the worse fix | `DisableKeepAlives: true` | **the leak test PASSES.** Disabling keep-alive also stops the leak — by dialling fresh for every one of ~40,000 daily requests. `TestPBSClient_KeepAliveStillReuses` is what catches it: *"3 sequential requests over 3 connection(s), want 1"* |

Red-proof 2 is the load-bearing one: **Scenario A alone would have accepted a fix that made the problem
worse.** Both mutations were reverted and the tree re-verified clean.

Green gate: `go build ./... && go vet ./... && go test ./...` → 30 packages, 0 failures.
`python3 scripts/agent_gates.py` → all 4 gates OK.

## P1 — the restart observation

Predicted in advance as one of three outcomes. **Outcome (i), within one second.**

| | ep0 fd | ESTAB | from `demo-felhom` | from `demo-hp` | CLOSE-WAIT |
|---|---|---|---|---|---|
| T-0 `09:15:46Z` | 415 | 398 | 199 | **199** | 0 |
| T+1s `09:15:47Z` | **216** | **199** | 199 | **0** | **0** |
| T+60s `09:16:42Z` | 218 | 201 | 199 | 2 | 0 |

**Outcome (ii) did not occur and gets no register row.** Not one socket converted to `CLOSE-WAIT`: ep0
reaps on the peer's FIN correctly, so the 543 `CLOSE-WAIT` seen at the 2026-08-18 wedge has some other
explanation and is not evidence of a second defect on the protected machine. That is a finding in the
negative and it was worth the sixty seconds it cost.

**Ownership is now established by a third independent route.** Killing our process released exactly
`demo-hp`'s 199 descriptors — agreeing with `ss -tnp` on the box and with the access-log user agent.

**The new code retires connections, observed directly:** at 133 s uptime `demo-hp` held **0**
connections; the 2 it opened after restart were closed at the 90 s idle mark.

## P2 — the divergence window

**Window 09:15:47Z → 10:17:52Z = 1.03 h. The task specified ≥ 4 h; the operator closed it early.**
The conclusion below therefore does not rest on an extrapolated daily rate, and none is used.

| box | agent | start | end | delta | per hour |
|---|---|---|---|---|---|
| `demo-felhom` — CONTROL | 0.129.0 | 199 | 203 | **+4** | 3.87 |
| `demo-hp` — FIXED | 0.130.0 | 0 | 0 | **+0** | 0.00 |

Predicted: control ≈4/hour, fixed ≈0. **Control observed 3.87/hour.**

**The assumption-free statement, and the reason the short window still settles it.** ep0's access log
counts the poll cycles directly: in that window **each box made exactly 4 `GET .../snapshots` calls and
4 `GET /version` calls**. Same cadence, same work, the same four chances to leak.

> **control: 4 cycles → 4 leaked connections. fixed: 4 cycles → 0 leaked connections.**

**The positive observable, per standing rule 3.** A zero leak is equally consistent with "fixed" and
with "the agent stopped working". It is the former: the fixed box's four poll cycles are in ep0's log,
alongside the control's four. The rest of the two boxes' traffic is near-identical in the window —
`libwww-perl` 924 vs 926, `proxmox-backup-client` 898 vs 898 — so **the only difference between them is
the agent binary**.

Poisson alone would give P(0 leaks | old rate, λ=4) = **1.8%**, which is suggestive rather than
conclusive. It does not stand alone: P1 released 199 descriptors instantly, the mechanism is identified
at `file:line` and pinned by a red-proofed test, and the fixed box was directly observed returning to 0.

## P3 — the second box, and the accumulated leak clearing itself

`demo-felhom` — the former control — was upgraded at `10:18:56Z` on the operator's word.

| | ep0 fd | ESTAB | CLOSE-WAIT |
|---|---|---|---|
| T-0 `10:18:55Z` | 220 | 203 | 0 |
| **T+2s `10:18:56Z`** | **17** | **0** | **0** |
| T+65s `10:19:58Z` | 20 | 3 | 0 |
| settled, 10:30–10:35Z | **17–19** | 0–2 | 0 |

**fd 17 is precisely ep0's `t0` baseline** — fd 17, ESTAB 0, recorded at 2026-08-18 09:51:22Z. The proxy
sits at 17–19 and returns to 17 between poll cycles, which is the "healthy proxy near 20 fds" the
incident document named as the positive observable. **ep0 was read-only throughout and its proxy PID
never changed (551655).**

**This corrects a sentence written earlier the same day**, in the v0.130.0 CHANGELOG draft and in the
STOP 1 report: *"does not clear the 388 descriptors already stuck on ep0 — those persist until that
proxy restarts."* **Wrong, and measured wrong within the hour.** The descriptors were held on *both*
sides; closing either side ends them. Restarting the two agents released all of them. Nothing on the
protected machine had to be touched, and nothing was.

## Fleet sanity

- Hub reports `demo-hp` **0.130.0** and `demo-felhom` **0.130.0**; 0.130.0 is above the golden's
  MinAgent of 0.129.0, and **no `floor held` line appeared** for either box.
- **No `pbsdr_box_unreachable` / `offsite_box_unreachable` event** fired during any window.
- Positive observable rather than the absent one: the hub's PBS-DR gauge kept refreshing on schedule
  (`3.7% full (3.7 GB of 97.9 GB)`) and host reports kept landing from both boxes throughout.

## An unrelated finding the deploy exposed — R-348

The **first two host reports after an agent restart carry `0 backups`**, while the box's own
`pvesm list` shows backups present on both tiers. `internal/backup/store.go`'s `Store` is in-memory and
its `byTarget` map is repopulated only when a backup **runs** — daily for the local tier, weekly for
offsite — so the field reads 0 for up to ~18 h after any restart. `restore_tests` did **not** blank,
because that half has a durable on-disk companion (`RestoreTestState`, R-189).

**It blinds no alarm, and that was checked rather than assumed.** `hub/internal/monitor/deadline.go`
already scans back over stored reports with a 7-day `backupEvidenceLookback`, whose comment names this
exact case — *"when the LATEST report carries none... and against an agent that stayed restarted for
days"* — and `pbs_snapshots` stayed populated at 2 regardless. So this is an observability wart, not a
safety hole.

**But the `Store` comment is misleading in a way this project has a rule about.** It reads *"Backups are
unaffected — their freshness has a ground truth on the storage (R-84)"*. That is true of the
**consequence** and false of the **field**, and a future reader may take it as a guarantee the field
stays populated. Filed as R-348.

## What this run did NOT do

- **Did not publish.** No Gitea package, no tag, no `artifact_agent_version` / `artifact_min_agent` /
  `artifact_golden_version` / `min_controller_version` change, no staged self-update. **A box installed
  from the current image still carries the leaking agent** — filed as **R-347**, and the CHANGELOG
  heading stays `## UNRELEASED` until that decision is taken.
- **Did not reduce the poll rate.** R-336 stays open, **re-scoped**: it was never the cause of this leak.
- **Did not refactor `pbsTargetsFromPVE`** to cache or reuse clients, and added no
  `CloseIdleConnections` call. See Observations.
- **Did not touch ep0** — no restart, no config, no package, no nftables. Reads only.
- Did not trigger a backup, restore or verify to generate traffic; did not contact `peti-felhom`.

## Observations

- **The closure refactor is not worth doing, and the measurement is why.** With the idle timeout
  restored, an abandoned client's connection is gone in 90 s, so the standing population is bounded at
  roughly one connection per box rather than growing without limit. Caching clients would add
  cache-invalidation questions — a storage's fingerprint, token or namespace can change under it — for
  no observable gain. Recommend leaving it.
- **Three other `http.Transport` defaults are still missing** and were deliberately left alone:
  `MaxIdleConns` (0 = unlimited; `DefaultTransport` uses 100), `TLSHandshakeTimeout` (0 = no limit;
  `DefaultTransport` uses 10 s) and `ExpectContinueTimeout`. None of them accumulates anything, and every
  client bounds its whole request with `http.Client.Timeout`, so none is a leak. `TLSHandshakeTimeout`
  is the only one with a plausible failure mode — a stalled handshake over the tunnel, bounded today
  only by the outer 30 s client timeout. **Not changed here, because widening the diff would have made
  this measurement unattributable.** Worth a look on its own terms; not filed as a defect.
- **A measurement error of mine, recorded because it nearly cost four hours.** The first P2 sampler
  reported both per-box columns as 0 while the totals were right: `ss` prints
  `[::ffff:10.77.0.2]:port`, and the pattern expected `10.77.0.2:`. Caught 15 minutes in by noticing
  that a 0/0 split could not sum to 199. Fixed, then **one sample was proved by hand before committing
  the window to it** — the check that should have happened first.

---

# Published — 2026-08-20, on the operator's word (R-347 CLOSED)

Released through `scripts/release-agent.sh 0.130.0`, the one documented way (R-115), which builds,
tags, publishes and **verifies by independent download** rather than by its own say-so.

| | |
|---|---|
| version / tag | **0.130.0** / `v0.130.0` at `7569f34` |
| sha256 | **`a56a92a7bd68f5b46736eaec4806c3d26c16ccb35118c4ac0e3d8094eaefabc3`** |
| size | 14,141,158 bytes |
| reproducible | **yes, checked** — a rebuild with `-trimpath -buildvcs=false` matches byte for byte (R-186) |
| anonymous fetch | matches the vouched sha |

## The vouch, and what was deliberately NOT changed

Day-0 artifact manifest: `agent_version` 0.129.0 → **0.130.0**, `agent_sha256` updated. **Everything
else re-sent unchanged**, and each for a reason:

- **`min_agent` stays 0.129.0.** It expresses what the **golden controller** requires, not what the
  newest agent is. Raising it to 0.130.0 would have made the hub **HOLD the controller floor** for
  every box not yet on 0.130.0 — the exact opposite of shipping a fix, and the manifest's own help
  text says so: *"The hub HOLDS the floor for any box whose agent is below this."*
- **`golden_version` / `golden_sha256` / `wrapper_sha256`** — untouched; no controller was released.
- **The global floor was never touched** (still 0.216.0). **And on hub v0.106.0 it could not have been
  by accident:** it is a *separate form with its own action* (`/configuration/global-floor`), so
  publish-train rule 2's "the manifest screen carries the live DB floor — save it last" hazard no
  longer exists in the shape its incident describes. Worth knowing before the next train; the rule's
  reasoning still holds, its mechanism has moved.

Verified after: no `floor held` line for either box, no `*_unreachable`, both boxes reporting 0.130.0.

**No `--no-verify` anywhere in the train.** The CHANGELOG heading was flipped from
`## UNRELEASED — v0.130.0 candidate` to `## v0.130.0` **only after** the tag and package existed, so
`release-complete` passes on the real artifact. The ordering was: release at the UNRELEASED-heading
commit, then flip. The alternative — flipping first and bypassing the gate — would have produced a
genuinely red CI run and an alarm mail for a release that worked, which is R-168's failure mode.

## The trap this train exposed — R-349

**Both boxes were running a different binary under the same version name, and nothing would ever have
noticed.** The proof deploy used a hand build (`go build -ldflags …`); the release builds with
`-trimpath -buildvcs=false` for reproducibility. Same source, same version string, **different bytes**:
`256e0829…` on the boxes against `a56a92a7…` published.

The sharp edge is that **self-update cannot correct it**: the boxes already reported `0.130.0`, so the
vouched version looked installed and nothing would have happened, indefinitely. Every version check in
the system — the hub, `--version`, the artifact manifest — compares the version **string**, so the
divergence is invisible to all of them.

Corrected by installing the **downloaded** artifact (not a local rebuild — the boxes get the bytes a
fresh install would get) on both. Both now report `a56a92a7…`.

**The proper fix already exists in miniature:** `wrapper_sha256` makes exactly this drift visible for
the PBS wrapper — *"agents report the installed file's hash and a mismatch is surfaced on the host
page"*. It was simply never extended to the agent's own binary. R-349.

## A mistake of mine in this train — R-350

Confirming the vouch used `curl -w '%{redirect_url}'`. The hub answers the POST with a **303**, and
curl renders the redirect target **with the basic-auth credentials re-attached** — so the hub operator
password was printed in cleartext into the session transcript.

It is **not** in git, not in any committed file (checked by content, not by assumption), and not in
this evidence directory; it is in the Claude Code transcript on DooPlex. Every other call in the
session printed only the password's length — this arrived through curl's output formatting, which is
why the usual discipline missed it. Rotation is recommended and is the operator's call; the reusable
half is that **`%{redirect_url}`, `-v` and `--libcurl` all re-render a basic-auth credential** —
confirm a redirect with `%{http_code}` and read the flash from a follow-up GET.

## Closing state

```
ep0  10:52:26Z  pid=551655  fd=17  estab=0  ctrl(.2)=0  fix(.3)=0  CLOSE-WAIT=0
```

**fd 17 is ep0's `t0` baseline**, and it returns there between poll cycles. The proxy is the same
process that has run since 2026-08-18 09:51:04 — **ep0 was read-only for this entire arc**, from the
spike through the fix to the release, and was never restarted, reconfigured or upgraded by any of it.