Files
felhom.eu/documentation/audits/evidence-ep0-established-connections-2026-08-20/phaseC-prediction.txt
T
admin 19672e685e
gates / gates (push) Successful in 14s
SPIKE ep0 connections: the leak is felhom-agent's, not the poll rate
R-341's first dated check, taken at +46.2 h: fd 17 -> 405 over 166,251 s
= 201.6/day. Pre-registered range was 370-450; observed 388. UNCHANGED,
as predicted. CLOSE-WAIT is 0 -- absent entirely, not merely flat.

Q1: exactly two peers, 194 each, no third party.
Q2: outcome (a). ep0 388 = 194 + 194 on the boxes, twice, and the four
    new sockets carry the same source ports on both sides. 0 closed in 31 min.

The finding: all 388 are held by felhom-agent. pvestatd and
proxmox-backup-client made 162,404 requests and leaked zero. Mechanism is
a per-cycle http.Transport with a zero-value IdleConnTimeout that nothing
ever closes (internal/pbs/client.go:56, main.go:1486). R-336's premise
does not survive this -- cutting the poll rate would have fixed nothing.

Q3 NOT measured: Phase C held at STOP 1, prediction pre-registered first.

Part 0 captures the due-checks gate's first conviction on a real overdue
date (rc=1, names R-341, sole failure among 10 gates). Row cleared at
Part 4, after the result was recorded in R-341, not to make a push work.

New: R-344 (the transport leak), R-345 (hub/Makefile pushes :latest),
R-346 (ActiveEnterTimestamp reads 5h56m early -- NRestarts is still 0).
2026-08-20 10:41:37 +02:00

63 lines
4.0 KiB
Plaintext

PHASE C PREDICTION — written 2026-08-20 ~08:40Z, BEFORE any Phase C measurement exists.
Same discipline as evidence-ep0-pbs-upgrade-2026-08-18/stop1-ruling.txt: committed to git
before the mutation is performed, so the result cannot be rationalised afterwards.
-------------------------------------------------------------------------------
WHAT PARTS 1-2 ALREADY ESTABLISHED (measured, not assumed)
-------------------------------------------------------------------------------
Q2 outcome : (a) — both sides hold every connection. 194+194 = 388 = ep0 total,
and the 4 new sockets that appeared during the 31-min persistence
window carry the SAME source ports on ep0 and on the boxes.
Owner on the box side : felhom-agent, 100% (194/194 on each box, one PID each).
pvestatd and proxmox-backup-client hold ZERO.
Elapsed-window rate : 201.6 fd/day (388 fd over 166251 s), 2s = 181.2..222.1.
Request split : libwww-perl (pvestatd) 81192 req -> 0 descriptors leaked
proxmox-backup-client 80061 req -> 0 descriptors leaked
Go-http-client (agent) 811 req -> 388 descriptors leaked
(387 agent GET /snapshots calls vs 388 leaked sockets: 1:1 within 1)
-------------------------------------------------------------------------------
THE INDEPENDENT VARIABLE Phase C ACTUALLY MOVES
-------------------------------------------------------------------------------
Stopping pvestatd on demo-hp removes BOTH pvestatd-driven streams from that box (the PVE PBS
storage plugin issues the libwww-perl datastore list AND shells out to proxmox-backup-client
for the status call, both on pvestatd's cycle). Expected request effect:
demo-hp requests 81612 per 46.18 h -> ~982 per 46.18 h (-98.8%)
BOTH boxes, per day ~84822 -> ~42900 (-49.4%)
So the request rate roughly HALVES, as the spike intends. IF the ep0 request count does NOT
fall by close to half, the independent variable did not move and no ratio may be computed.
-------------------------------------------------------------------------------
PREDICTIONS, ONE PER HYPOTHESIS. For a 10-hour window (scale linearly for other lengths).
-------------------------------------------------------------------------------
H1 NOT PROPORTIONAL — the leak tracks the AGENT's cycle, not the request rate.
This is what Parts 1-2 predict, and it is the prediction this run commits to.
demo-hp's felhom-agent keeps running throughout, so both boxes keep leaking.
expected rate : ~201.6 fd/day, UNCHANGED
expected count : 84 descriptors in 10 h (2s band 66..102)
expected split : ~42 from 10.77.0.2 and ~42 from 10.77.0.3 — SYMMETRIC.
The symmetry is the sharpest single discriminator: under H1 the quietened box keeps
leaking at the same rate as the control box.
H2 PROPORTIONAL to total request rate (the hypothesis Q3 was written to test).
expected rate : ~101 fd/day
expected count : 42 descriptors in 10 h (2s band 29..55)
expected split : ~42 from 10.77.0.2, ~0 from 10.77.0.3 — STRONGLY ASYMMETRIC.
H1 and H2 do not overlap at 2 sigma for a window of 10 h or more. A 6-hour window gives
50 vs 25 (2s bands 36..64 vs 15..35) — separable but tighter; prefer the overnight run.
Under Q2 outcome (b) (half-open, ep0 only) the prediction would have been H2-like, and under
(c) Phase C would not have been run at all. Both are recorded here for completeness; the
measurement already excluded them.
-------------------------------------------------------------------------------
WHAT WOULD FALSIFY THE PARTS 1-2 ATTRIBUTION
-------------------------------------------------------------------------------
A result matching H2 — demo-hp's contribution going to ~zero while pvestatd is stopped — would
mean the socket ownership reading is not the whole story, and the attribution above must be
withdrawn rather than defended. Say so plainly if it happens.