SPIKE ep0 connections: the leak is felhom-agent's, not the poll rate
gates / gates (push) Successful in 14s

R-341's first dated check, taken at +46.2 h: fd 17 -> 405 over 166,251 s
= 201.6/day. Pre-registered range was 370-450; observed 388. UNCHANGED,
as predicted. CLOSE-WAIT is 0 -- absent entirely, not merely flat.

Q1: exactly two peers, 194 each, no third party.
Q2: outcome (a). ep0 388 = 194 + 194 on the boxes, twice, and the four
    new sockets carry the same source ports on both sides. 0 closed in 31 min.

The finding: all 388 are held by felhom-agent. pvestatd and
proxmox-backup-client made 162,404 requests and leaked zero. Mechanism is
a per-cycle http.Transport with a zero-value IdleConnTimeout that nothing
ever closes (internal/pbs/client.go:56, main.go:1486). R-336's premise
does not survive this -- cutting the poll rate would have fixed nothing.

Q3 NOT measured: Phase C held at STOP 1, prediction pre-registered first.

Part 0 captures the due-checks gate's first conviction on a real overdue
date (rc=1, names R-341, sole failure among 10 gates). Row cleared at
Part 4, after the result was recorded in R-341, not to make a push work.

New: R-344 (the transport leak), R-345 (hub/Makefile pushes :latest),
R-346 (ActiveEnterTimestamp reads 5h56m early -- NRestarts is still 0).
This commit is contained in:
admin committed 2026-08-20 10:41:37 +02:00
1 parent 848368ec38
commit 19672e685e
22 files changed
+1330 -8

No files matched your search

+162
View File
@@ -0,0 +1,162 @@
# REPORT — SPIKE: who holds ep0's connections open (2026-08-20)
**Status: STOP 1 reached. Parts 0, 1 and 2 are complete. Phase C (Part 3) has NOT been run — it is your
decision, below.** Everything done in this session was **read-only**. No machine was changed.
---
## ⛔ The decision waiting on you
Phase C wants one overnight window on `demo-hp`. **The measurement found something that changes which
mutation is worth running**, so there are two versions of it. Both are one mutation on a Tier-0
disposable box, both arm a dead-man timer first, both are unattended.
| | **Option A — stop `pvestatd`** (what the spike prompt specifies) | **Option B — stop `felhom-agent`** (what the evidence now points at) |
|---|---|---|
| what it tests | the original Q3: does the leak track the request rate? | does the leak track **our agent's** cycle? |
| predicted result | **null** — leak unchanged at ~201.6/day, ~84 descriptors in 10 h, split evenly ~42/~42 | leak **halves** — ~101/day, ~42 descriptors in 10 h, split ~42 from `demo-felhom` and **~0** from `demo-hp` |
| what it buys | **falsification.** If the leak *did* halve, the whole Part-1/2 attribution is wrong and must be withdrawn | **confirmation by a second, independent route.** A near-zero contribution from the quietened box is decisive |
| cost if the attribution is right | a confirmed null — real evidence, but no new information beyond what Parts 1–2 already show | the sharpest possible confirmation |
| risk | none beyond the window: `pvestatd` is PVE's stats daemon; the box keeps running, backups are not due until ~08-25 | slightly higher: the agent is our own product on the box. It would stop reporting to the hub for the window, and the hub's staleness watch may notice |
**I would run Option A, tonight.** Two reasons. It is the mutation the prompt authorises, and standing
rule 2 says exactly one mutation exists in this run — substituting one is my call to propose, not to
make. More importantly, **A is the falsification test and B is the confirmation test**, and the
attribution is already confirmed twice over (socket ownership on the boxes, and the access-log user
agent, by completely independent routes). A test that can prove me wrong is worth more right now than a
third test that can only agree with me.
**What I need from you: "A tonight", "B tonight", "at the weekend", or "skip it".** If you pick B I will
need you to say so explicitly, because it is a second mutation the prompt does not authorise.
---
## What I did, and what it found
### Part 0 — the dated-check gate's first real conviction, captured before anything else
The 2026-08-19 row was one day overdue. Every conviction this gate had produced before came from a
fixture or a `FELHOM_GATE_TODAY` override; **this is its first firing on a real overdue date in the live
register**, and it behaved correctly:
- **exit code 1** — a verdict, not a crash (2) and not a pass (0);
- the message **names R-341 and the days overdue** (`R-341 due 2026-08-19 1 day(s) OVERDUE`), so the
exit code is not doing the work alone — which is the failure mode this gate's own red-proof fell into
on 18 August;
- inside `repo_gates.py` it is the **only** conviction: 9 gates OK, `CONVICTED: due-checks`.
Captured verbatim in `part0-due-checks-gate.txt` and `part0-repo-gates.txt`. The row was **not** cleared
to make the push work — it was cleared at Part 4, after the measurement existed and its result was
recorded in R-341, which is the sanctioned order.
### Q0 — R-341's first dated check: the slope is UNCHANGED
Precondition passed: PID still **551655**, `NRestarts=0`, so the elapsed window is valid.
**fd 17 → 405 over 166,251 s (46.18 h) = 201.6 fd/day**, Poisson 2σ 181.2–222.1.
The prediction was pre-registered before the reading: **370–450**. **Observed 388.**
Verdict: **unchanged**, the expected result, and not a failed upgrade.
This settles the check on a window **88× longer** and a descriptor count **97× larger** than the
30-minute windows the original answer rested on. Uncertainty drops from roughly ±50% to ±5%.
**Composition:** ESTAB 0 → 388, and **CLOSE-WAIT is 0 — absent from the histogram entirely.** The
incident document's original emphasis on `CLOSE-WAIT` is not merely the minority story; on this proxy
generation that state does not occur at all.
Runway to the 65536 ceiling: **~323 days (~2027-07-09)**.
### Q1 — who is at the far end: exactly the two demo boxes, 194 each
No third peer. Outcome (c) excluded. The identity is read from the API token name on every access-log
line, not inferred from the address. `lsof` confirms the leak is sockets and nothing else: 390 of 405
descriptors are TCP.
**Persistence (31-minute diff of full 4-tuples): 388 in both readings, 0 closed, 4 new.** Not one socket
closed. All carry keepalive timers with `retrans=0` — the far ends are answering, so these are not
half-open sockets.
### Q2 — outcome **(a)**, confirmed twice, exactly
| instant | ep0 | `demo-felhom` | `demo-hp` | sum |
|---|---|---|---|---|
| 08:02:39 / 08:04:37Z | **388** | 194 | 194 | **388** |
| 08:33:42 / 08:34:01Z | **392** | 196 | 196 | **392** |
And the four sockets that appeared between the readings carry **the same four source ports** on ep0 and
on the boxes. Both sides hold every connection.
### The finding nobody predicted: the leak is ours
`ss -tnp` on the boxes names the owner of **194 of 194** on each: **`felhom-agent`**, one PID per box.
Zero are held by `pvestatd`. Zero by `proxmox-backup-client`.
ep0's access log says the same thing by a completely independent route:
| who | requests in the window | descriptors leaked |
|---|---|---|
| `libwww-perl` (pvestatd) | 81,192 | **0** |
| `proxmox-backup-client` | 80,061 | **0** |
| `Go-http-client` (**our agent**) | 811 (of which **387** `/snapshots` calls) | **388** |
**One leaked socket per agent `/snapshots` call, within one.** 99.5% of the traffic produces 0% of the
leak.
**Mechanism, named from source** — `felhom-agent/internal/pbs/client.go:56-60` builds
`&http.Transport{TLSClientConfig: tlsCfg}` as a composite literal, so `IdleConnTimeout` is the zero value
(= no limit; `http.DefaultTransport` sets 90 s and a literal does not inherit it), and
`cmd/felhom-agent/main.go:1486` builds **a fresh client every cycle**, as its own doc comment states.
`CloseIdleConnections`, `IdleConnTimeout` and `MaxIdleConns` appear **nowhere in the agent repo**.
Cadences reconcile without fitting: 900 s hub poll (184.7 cycles) + 6 h verify cadence (7.7) = 192.4
predicted against **194 observed per box**.
**No fix is proposed** — the spike-first gate forbids it, and the spec is a separate task. Filed as
**R-344**, with the open design questions listed rather than pre-answered.
### Control window: clean
No reboots (ep0 up 17 d, boxes up 10 d), no daemon restarts (`NRestarts=0`), no agent restarts, **no
request-rate gap** (3,536–3,553 per hour, every hour), **no HTTP errors at all** (every response 200
except 8 expected 101s), no tunnel flap. **No backup ran inside the window** — the last offsite protocol
upgrades were 08-18 03:57/03:58Z, *before* `t0`, and the tier is weekly with the next run due ~08-25, so
**tonight's window is also clear of one.** Routine verify jobs and restore-test reads did occur; they are
1.2% of traffic and are already counted inside the 194/box reconciliation.
**Hub:** no `*_unreachable` or `*_recovered` event; the PBS-DR gauge refreshes on schedule and host
reports land from both boxes. **Honest limit:** the hub pod is 39 h old, so hub-side logs cover 36.6 of
the window's 46.2 hours; the first 9.6 h rests on ep0's own evidence.
---
## Register changes
- **R-341** — first check recorded (taken at **+46.2 h, not +24 h**; the delay was pure elapsed time and
the longer window is stated as a **better** measurement, not a degraded one). **2026-08-19 row removed**
from `DUE-CHECKS`; 2026-08-25 kept, with the Phase-C perturbation quantified against it (**~3%** shift
on the 7-day slope even under the hypothesis we expect to be false — the reading stays usable).
- **R-336** — mechanism named; **premise corrected and re-ranked**. Its recorded next step would have
produced a null result and read as a failed fix. The poll rate is now a scaling/cost item; the leak fix
is R-344. **Q3's proportionality verdict is explicitly NOT recorded** — Phase C has not run.
- **R-340** — noted which of its wanted observations this run already produced, so the health-op task
reuses them rather than measuring a protected machine a third time. Only the loopback probe is still
owed.
- **R-344 (new)** — the agent's per-cycle transport leak. READY (S).
- **R-345 (new)** — `hub/Makefile` lines 21–22 tag and push `:latest`, which two rule files forbid.
READY (XS).
- **R-346 (new)** — `ActiveEnterTimestamp` reads 5 h 56 m early for this proxy generation (the upgrade
re-exec'd rather than restarted, so `NRestarts` is still 0). Anchoring a slope on it gives ~12% low.
READY (XS).
## Deliverables
- `documentation/audits/SPIKE-ep0-established-connections-2026-08-20.md` — the findings.
- `documentation/audits/evidence-ep0-established-connections-2026-08-20/` — 11 raw evidence files,
including the **pre-registered Phase C prediction**, committed before the mutation exists.
- `documentation/backlog/OPEN-ITEMS.md`, `STATUS.md` — as above.
## What is inconclusive
**Q3 is unmeasured**, by design. Everything stated about proportionality is a labelled prediction. Also
unexplained: why the two boxes' leaked counts are *exactly* equal at two separate instants rather than
merely close. And whether restore-test reader connections leak too was not separated out (≤2% of the
total, inside the noise).
+13 -3
View File
@@ -1,7 +1,7 @@
# STATUS — what works, what's broken, what's next
**Updated 2026-08-18 (afternoon — a listening socket that served nobody, a rehearsal upgrade that changed
nothing, and a new install that is finally current).**
**Updated 2026-08-20 (morning — we looked at who was actually holding the off-site box's connections
open, and it turned out to be us).**
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority; this page restates
> part of it in plain words, and **nothing may exist only here**. **Items, not paragraphs. One screen.**
@@ -116,6 +116,14 @@ record with no machine** — created 13 August, no host, no backups, nothing to
alarm. **Corrected later the same morning: my first estimate of how fast it leaks was too
optimistic by about 2.5×** — measured properly it is under a year to the new ceiling, not two years.
A deadline, not a comfort.
- **The leak is OURS, not Proxmox's, and asking fewer questions would not have fixed it** (R-344, found
2026-08-20). We finally looked at who was on the other end of the stuck connections. Every single one
belongs to **our own agent** on the two demo machines — it opens a connection to the off-site box on
each 15-minute cycle and never closes it, and neither does the box. The two Proxmox pollers that make
99.5% of the traffic leak **nothing at all**. So the plan recorded under R-336 — turn the question
rate down, then watch the count stop climbing — **would have changed nothing and looked like a failed
fix.** The count still says about a year to the ceiling. **Nothing is broken today**; this is a
deadline we now know where to aim at. *(register: R-344, and R-336 re-ranked)*
- **The off-site box was updated, and it did not help — as expected** (R-341). On your ruling we
installed the newer backup software for the practice, having first read its release notes and found
**nothing** about the fault we have. The update went cleanly and everything works, but the leak
@@ -155,7 +163,9 @@ record with no machine** — created 13 August, no host, no backups, nothing to
## Working on next
`demo-hp` is yours this evening — **this session did not touch it**. After that: the three remaining
**One decision is waiting for you, at the top of `REPORT-spike-ep0-connections.md`:** whether to run the
overnight measurement window on `demo-hp` tonight, and which of two versions of it. Nothing was changed on
any machine in this session — every reading was read-only. After that: the three remaining
R-264 readers, now that one has been built and we know what one costs; R-317 (one line in the agent);
R-327 (decide what the naming claim's status should be); then the 2026-08-09 batch (R-279 … R-292),
still untriaged against everything since.
@@ -0,0 +1,323 @@
# SPIKE — who holds ep0's connections open, and does the leak track the poll rate?
**Run:** 2026-08-20 08:01–08:35 UTC, read-only throughout. **Phase C (the one mutation) has NOT been run** — it is
held at STOP 1 for the operator's window decision.
**Evidence:** `evidence-ep0-established-connections-2026-08-20/`, one file per step.
**Supersedes the plan in** `SPIKE-ep0-established-connections.md` (issued 2026-08-18, never run).
> **Headline, and it is not the headline the spike expected.**
> The leak is **ours**. Every one of the 388 leaked descriptors on ep0 is an established connection held
> open by **`felhom-agent`** on the customer boxes — one per PBS poll cycle, never closed on either side.
> `pvestatd` and `proxmox-backup-client` made **162,404 requests** in the same window and leaked **zero**.
> R-336's premise — that cutting the ~85,000/day poll rate is the leak fix — **does not survive this
> measurement.** The poll rate is a separate (real) problem; it is not what is consuming descriptors.
---
## Q0 — R-341's first dated check: has the fd slope changed since the PBS 4.2.5-1 upgrade?
### Answer: NO. Unchanged, and now measured over a window 88× longer than the one it replaces.
**Precondition passed.** `MainPID = 551655`, `ps -o lstart` = `Tue Aug 18 09:51:04 2026`, `NRestarts = 0` on
both `proxmox-backup-proxy` and `proxmox-backup`. The proxy generation that carries `t0` is the one that was
read. The elapsed-window measurement is valid.
| | value |
|---|---|
| `t0` | fd **17**, ESTAB 0, 2026-08-18 **09:51:22Z**, PID 551655 |
| reading A | fd **405**, ESTAB **388**, CLOSE-WAIT **0**, 2026-08-20 **08:02:13Z**, PID 551655 |
| elapsed | **166,251 s** = 46.181 h = 1.9242 d |
| delta | **+388 fd** |
| rate | **201.6 fd/day** (8.40/h) |
| Poisson on n=388 | ±19.7 counts → **±10.2/day (1σ)**, ±20.5/day (2σ) → **181.2 … 222.1 /day** |
| limits | soft/hard **65536 / 65536** — the drop-in survived the upgrade |
| listener | `LISTEN Recv-Q 0 Send-Q 1024` — accept queue empty, nothing wedged |
**Verdict against the pre-registration**, quoted from the spike prompt §2 so it is visible beside the result:
> *"Predicted range: roughly **370–450** … A reading anywhere in **330–490** confirms an unchanged rate …
> **In range** → unchanged rate. This is the EXPECTED result, consistent with the changelog containing no
> mechanism to change it. It is not a failed upgrade."*
**Observed: 388.** In range, and close to the centre. The rate is **unchanged** since the upgrade. This is the
expected result and it is recorded as such — not as a success and not as a failure of the upgrade, which was
run for rehearsal value and never claimed to fix anything.
For comparison, on the identical method: **183/day** (31 min, pre-upgrade), **200/day** (5.64 h, pre-upgrade),
**225/day** (32 min, post-upgrade). This window carries **97× the descriptor count** of the 31-minute window,
so its uncertainty is ±5% rather than ±50%.
### Composition — the split, which is what says *which* leak it is
```
t0 fd 17 ESTAB 0 CLOSE-WAIT 0
reading A fd 405 ESTAB 388 CLOSE-WAIT 0 17 + 388 = 405, exactly
```
**`CLOSE-WAIT` is absent from the `ss` histogram entirely — 0, not merely flat.** 100% of the growth is
**established** connections on `:8007`. This closes the question the incident's own correction block opened:
the mechanism is *connections the proxy never reaps*, and `CLOSE-WAIT` is not merely the minority half, it is
**not present at all** on this proxy generation.
### Runway
From fd 405 at 201.6/day to the 65536 soft limit: **≈ 323 days**, i.e. around **2027-07-09**. Under a year.
A deadline, not a comfort — unchanged in character from what R-341 recorded.
---
## Q1 — who is at the far end?
### Answer: exactly the two demo boxes, in a dead-even split. No third peer. Outcome (c) is excluded.
At 08:02:39Z, `ss -tn state established '( sport = :8007 )'`, peer histogram:
| peer | connections | identity |
|---|---|---|
| `10.77.0.2` | **194** | `demo-felhom` (host `felhom-pve`) — token `felhom@pbs!demo-felhom` |
| `10.77.0.3` | **194** | `demo-hp` (host `felhom-host`) — token `felhom@pbs!demo-hp` |
The identity is not inferred from the address: ep0's own access log carries the API token name on every line
(`::ffff:10.77.0.3 - felhom@pbs!demo-hp …`), and the mapping is one-to-one across the whole file.
**194 / 194 is not approximately even — it is exactly even**, which is itself a finding: whatever opens these
is a per-box periodic task running the same schedule on both, not traffic proportional to anything the boxes
differ in.
### Descriptor-type breakdown (1.3)
`lsof -np 551655`, which distinguishes sockets from files, pipes and eventfds so a "leak" that is not sockets
gets caught here:
| type | count |
|---|---|
| **IPv6 (TCP)** | **390** |
| REG (log files, rrd journal, datastore lock) | 32 |
| unix | 7 |
| a_inode (epoll ×2, eventfd ×1) | 3 |
| DIR | 2 |
| CHR (`/dev/null`) | 1 |
`lsof … | grep -c TCP` = **390** = 388 established + 1 listener + 1 other. The independent `/proc/<pid>/fd`
link-target histogram agrees: **397 sockets** (390 TCP + 7 unix), 9 regular/anon fds. **The leak is sockets,
and the sockets are TCP on :8007.** Nothing else is growing.
### 1.4 — persistence: are they long-lived, or churn?
Full 4-tuples diffed between reading A (08:02:39Z) and reading B (08:33:42Z), **1,843 s apart**:
| | count |
|---|---|
| sockets in reading A | 388 |
| sockets in reading B | 392 |
| **in BOTH — long-lived** | **388** |
| **only in A — closed during the interval** | **0** |
| only in B — new arrivals | **4** (2 per box) |
**Not one socket closed in 31 minutes.** There is no churn to separate from the leak: every established
connection on this proxy is the leak. The 4 arrivals over 1,843 s = 187.5 fd/day, consistent with the
elapsed-window 201.6/day within Poisson on n=4.
**Idle age and liveness.** All 392 sockets carry a TCP keepalive timer and **all 392 report `retrans=0`** —
the far end is answering keepalive probes. These are not half-open sockets whose peer went away; they are
mutually held, live, idle connections. That, on its own, points at outcome **(a)** and Part 2 confirms it.
---
## Q2 — do the boxes hold them too?
### Answer: **outcome (a)** — both sides hold every connection, and the two counts match exactly. Twice.
| instant (UTC) | ep0 ESTAB on `:8007` | `demo-felhom` ESTAB to `10.77.0.1:8007` | `demo-hp` | sum |
|---|---|---|---|---|
| reading A — ep0 08:02:39, boxes 08:04:37/38 | **388** | **194** | **194** | **388** |
| reading B — ep0 08:33:42, boxes 08:34:01/02 | **392** | **196** | **196** | **392** |
Two independent paired readings, both exact. And the confirmation is tighter than the totals: the **four new
sockets** that appeared on ep0 during the persistence window carry **the same four source ports** the boxes
report as new —
```
ep0 new: 10.77.0.2:45620 10.77.0.2:53226 10.77.0.3:37554 10.77.0.3:46214
demo-felhom new: 10.77.0.2:45620 10.77.0.2:53226
demo-hp new: 10.77.0.3:37554 10.77.0.3:46214
```
Both boxes also closed **zero** sockets in the same interval (A=194, B=196, closed=0, new=2, on each).
**This is not a hedge between two outcomes.** Outcome (b) — half-open sockets ep0 never noticed — is
excluded by both the matching counts and the `retrans=0` keepalives. Outcome (c) is excluded by Q1.
### And the owner has a name
`ss -tnp` on the boxes attributes **194 of 194** on each, to a single PID:
| box | PID | process |
|---|---|---|
| `demo-felhom` | 2596329 | `/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json` |
| `demo-hp` | 3199936 | `/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json` |
**Zero are held by `pvestatd`. Zero by `proxmox-backup-client`.** Both agents have been running continuously
since 2026-08-12 (they did not restart); the count restarted at the proxy's 08-18 restart because that
restart closed the old sockets on both sides.
Each agent's own descriptor table is **208 fds, of which 196 are these connections** — so the leak is
bilateral. The agents' `RLIMIT_NOFILE` is 524287, so they are in no danger; that is a reason it stayed
invisible, not a reason it is harmless.
---
## The user-agent attribution (1.5) — the access log settles it independently
Over the same 46.18 h window, from ep0's own `access.log.1` + `access.log`:
| user agent | endpoint | requests | req/day | **descriptors leaked** |
|---|---|---|---|---|
| `libwww-perl/6.78` | `GET /admin/datastore` (pvestatd) | 81,192 | 42,195 | **0** |
| `proxmox-backup-client/1.0` | `GET …/felhom-offsite/status` | 80,061 | 41,607 | **0** |
| `libwww-perl/6.78` | `GET …/felhom-offsite/snapshots` | 1,131 | 588 | **0** |
| **`Go-http-client/1.1`** | **`GET …/felhom-offsite/snapshots`** | **387** | **201** | **388** |
| `Go-http-client/1.1` | `GET /version` | 371 | 193 | (same connections) |
| `Go-http-client/1.1` | `POST …/verify` + task status | 53 | 28 | (same connections) |
| | **total, both boxes** | **163,215** | **84,822** | **388** |
`Go-http-client/1.1` is `felhom-agent` — the same process `ss` names on the box side, reached by a completely
independent route. **387 agent `/snapshots` calls against 388 leaked sockets: one per call, within one.**
**162,404 requests from the two Proxmox pollers leaked nothing.** They are 99.5% of the traffic and 0% of the
leak.
### Cadence reconciliation — the numbers close
| source | cadence | cycles in 166,251 s |
|---|---|---|
| agent hub-report poll (`poll_seconds: 900`, read from each box's `agent.json`) | 15 min | 184.7 |
| `pbs.VerifyLoop` (`DefaultVerifyCadence = 6h`) | 6 h | 7.7 |
| | **sum** | **192.4** |
Observed leaked per box: **194**. Observed `POST …/verify` per box in the window: **8** (predicted 7.7).
Observed `GET /version` per box: ~185.5 (predicted 184.7). Mean interval between leaked descriptors per box:
**857 s**. Every count reconciles against a cadence read from configuration, not fitted to the data.
### Mechanism — named from source, and it is a two-line Go footgun
`felhom-agent/internal/pbs/client.go:56-60`:
```go
http: &http.Client{
Timeout: timeout,
Transport: &http.Transport{TLSClientConfig: tlsCfg},
},
```
A hand-rolled `http.Transport` takes the **zero value** for `IdleConnTimeout`, which in Go means *no limit* —
idle keep-alive connections are never closed. (`http.DefaultTransport` sets 90 s; a composite literal does
not inherit it.) That alone would be harmless if one client were reused. It is not:
`felhom-agent/cmd/felhom-agent/main.go:1486-1509` — `pbsTargetsFromPVE` — says so in its own doc comment:
> *"returns a `pbs.Targets` closure that, **each cycle**, discovers the pbs storages from the PVE config and
> **builds a fingerprint-pinned, token-authed client for each**"*
So each cycle constructs a **new** `Transport`, performs one request, and leaves the connection idle in that
transport's pool forever. The transport then becomes unreachable, but Go's `persistConn` read-loop goroutine
keeps the socket alive — an unreachable `http.Transport` does not close its connections. `grep` across the
whole `felhom-agent` repo for `CloseIdleConnections`, `IdleConnTimeout`, `MaxIdleConns`: **no matches at all.**
**The same composite-literal pattern appears in two more clients** — `internal/hub/client.go:53` and
`internal/proxmox/client.go:69`. Those are built once at start-up rather than per cycle, so they do not leak
by this route; they are recorded because the pattern is one refactor away from doing so. Filed as **R-344**.
---
## Control-window cleanliness (1.6)
The elapsed window is doing double duty as Phase C's control, so it has to be shown undisturbed. It was.
| check | finding |
|---|---|
| **ep0 host reboot** | none — booted 2026-08-03 11:12:57, up 17 d |
| **proxy / API daemon restart** | none — `NRestarts = 0` on both units, PID 551655 throughout |
| **box reboots** | none — both booted 2026-08-10 09:2xZ, up 10 d |
| **agent restarts** | none — both `ActiveEnterTimestamp` 2026-08-12 |
| **request-rate gaps** | **none.** Hourly totals across the window: 3,536–3,553 every hour, no hour missing |
| **HTTP errors** | **none.** Every response in the window is `200`, except 8 × `101` (expected protocol upgrades) |
| **tunnel flaps** | no evidence — WireGuard handshakes for `10.77.0.2`, `.3`, `.250` all seconds old; and a flap would have shown as a request gap or errors, and neither exists |
| **a backup inside the window?** | **NO.** The last offsite protocol upgrades were 2026-08-18 **03:57 and 03:58Z** — the incident re-runs, **before `t0` at 09:51:22Z**. Both boxes' `pvesm list felhom-pbs` confirms the newest snapshots are `2026-08-18T03:57:43Z` / `03:58:43Z`. The tier's `cadence_seconds` is **604800** (weekly), read from each box's `agent.json`, so the next offsite run is due ~2026-08-25 |
| **other non-poll activity** | **yes, and it is routine, quantified, and does not contaminate.** Server-side verify jobs ran on ep0 on a ~6 h cadence (8 per box in the window), and restore-test reads ran on 2026-08-19 at 04:45–04:50 and 06:52–06:54Z (8 reader upgrades total). Together with snapshot/version/task-status calls this is **~1,963 of 163,215 requests = 1.2%** of the window's traffic, and at most 8 of the 388 descriptors could be attributed to reader connections (≤2%). The verify cycle is already counted **inside** the 194/box reconciliation above — it is part of the measured leak, not a contaminant of it |
**Verdict: the control window is clean.** No correction is applied and none is needed.
**Two nuances recorded rather than smoothed over.**
1. `systemctl show proxmox-backup-proxy -p ActiveEnterTimestamp` reads **2026-08-18 03:54:54Z**, not 09:51:04Z,
while `NRestarts = 0` and `MainPID` did change. The upgrade therefore **re-exec'd** the proxy rather than
restarting the unit — systemd never saw a stop. Anyone using `ActiveEnterTimestamp` as the "when did this
proxy generation start" anchor would get a figure **5 h 56 m too early** and compute a rate ~12% low (388 fd over 52.1 h instead of 46.2 h = 178.7/day instead of 201.6/day).
Use `ps -o lstart= -p $MainPID`, as R-341's own command does.
2. ep0 carries a fourth WireGuard peer, `10.77.0.4`, whose last handshake was 2026-08-13 (7 d ago). It is
`drill-r50-0a4f9a`, a drill host, and its dormant peer entry is already recorded in
`architecture/_recovery-inventory-2026-07-28.md`. Outside the window; not a contaminant; not a new finding.
**Hub view.** No `pbsdr_box_unreachable` / `offsite_box_unreachable` and no `*_recovered` event was logged.
The positive observable rather than the absent one: the PBS-DR gauge refreshes on schedule
(`PBS-DR box refreshed: 3.7% full (3.7 GB of 97.9 GB)`, every ~16 min through the run) and host reports land
from both boxes. **Honest limit:** the hub pod is 39 h old (redeployed for v0.106.0 at 2026-08-18 19:29), so
hub-side log evidence covers **36.6 of the window's 46.2 hours**. For the first 9.6 h the evidence is ep0's
own access log — which shows no gap and no error — and `NRestarts = 0`. Hub log timestamps render **CEST**.
---
## Q3 — is the leak rate proportional to the request rate?
### NOT ANSWERED BY MEASUREMENT. Phase C has not been run; it is held at STOP 1.
What exists instead is a **pre-registered prediction**, written and committed before any Phase C number can
exist: `evidence-ep0-established-connections-2026-08-20/phaseC-prediction.txt`. Its substance:
- Stopping `pvestatd` on `demo-hp` removes **both** pvestatd-driven streams from that box (the PVE PBS storage
plugin issues the `libwww-perl` datastore list *and* shells out to `proxmox-backup-client` for the status
call, both on pvestatd's cycle) — **−98.8% of demo-hp's requests, −49.4% of the fleet total**. If ep0's
request count does not fall by close to half, the independent variable did not move and no ratio may be
computed.
- **H1 (what Parts 1–2 predict): NOT proportional.** demo-hp's `felhom-agent` keeps running, so both boxes
keep leaking. Expect **~201.6 fd/day, unchanged — ~84 descriptors in a 10 h window (2σ 66–102) — split
symmetrically ~42/~42 between the two peers.**
- **H2 (the hypothesis Q3 was written to test): proportional.** Expect **~101 fd/day — ~42 descriptors in
10 h (2σ 29–55) — split strongly asymmetrically ~42/~0.**
H1 and H2 do not overlap at 2σ for a window of 10 h or more. The **peer split is the sharpest discriminator**
and it is free: it needs no second box and no arithmetic.
**A result matching H2 falsifies the Parts 1–2 attribution**, and the prediction file says so in those words,
so the outcome cannot be rationalised in either direction afterwards.
---
## Which fix the evidence points at
**No design here, per the spike-first gate — only the target.** The evidence points at
`felhom-agent/internal/pbs/client.go`'s HTTP transport lifetime, not at the poll rate. One `pbs.Client` is
constructed per cycle by `pbsTargetsFromPVE`, each carrying a fresh `http.Transport` whose `IdleConnTimeout`
is the zero value, and nothing ever closes it; the observed leak is exactly one socket per agent
`/snapshots` call, on both sides, on both boxes, and it reconciles with cadences read from configuration. A
fix that touches only the ~85,000/day Proxmox poll rate would reduce ep0's request load by ~99.5% and its
descriptor leak by **zero**. R-336's poll rate remains a genuine problem — 85,000 requests/day to a
weekly-write DR endpoint is still wrong — but **it is a different problem from this one, and the row's
"cut the poll rate, then confirm the fd count stops climbing" plan would have produced a confusing null
result.** That is the substantive change to R-336 this spike delivers.
---
## What was inconclusive
- **Q3 is unmeasured**, by design — Phase C is held at STOP 1 for the operator. Everything above about
proportionality is a prediction, labelled as one.
- **The ~9.6 h of hub-side evidence before the v0.106.0 pod redeploy is gone** with the previous pod's logs.
The window is covered by ep0-side evidence for that period, which is adequate for the questions asked, but
it is not the same evidence and is not presented as such.
- **Why `demo-hp`'s and `demo-felhom`'s leaked counts are *identical* rather than merely similar** is not
established. Both run the same cadences, so equality is expected — but exact equality at two separate
instants (194/194 and 196/196) is stronger than the cadences alone require, and no attempt was made to
explain it beyond noting it.
- **Whether a reader (restore-test) connection also leaks** was not separated out. At most 8 of 388 sockets
could be involved (≤2%), which is inside the noise, so it was not pursued.
@@ -0,0 +1,16 @@
=== part0-due-checks-gate.txt ===
captured: 2026-08-20T10:01:45+02:00 (DooPlex, CEST)
utc: 2026-08-20T08:01:45+00:00
repo HEAD: 848368ec38a7f9a49c6beb3a68b5620c822edc86
cmd: python3 scripts/due_checks_gate.py
--- stdout+stderr ---
DUE-CHECKS GATE FAILED: 1 dated check(s) are due or overdue as of 2026-08-20 (UTC).
R-341 due 2026-08-19 1 day(s) OVERDUE
measure: ep0 proxy fd count + ESTAB/CLOSE-WAIT split; PID must still be 551655
the command and its preconditions are in the R-341 row of documentation/backlog/OPEN-ITEMS.md
Take the measurement, record the result in that R-row, then remove the row from the
DUE-CHECKS block. Moving the date instead is allowed — state the reason in the R-row.
NOTE: this gate fires on a PUSH, not on the date; it may be later than the date.
rc=1
@@ -0,0 +1,141 @@
=== part0-repo-gates.txt ===
captured: 2026-08-20T08:01:52+00:00 UTC
cmd: python3 scripts/repo_gates.py
--- stdout+stderr ---
repo_gates (felhom.eu) — 10 gate(s)
==============================================================================
== gate: site (site_gates.py)
==============================================================================
site gates OK — BOM, emoji=0, nav/footer consistent, analytics present, no CDN, no legacy tokens, no <style>, cache-busted assets
==============================================================================
== gate: hostinstall (hostinstall_gates.py)
==============================================================================
ok: SCRIPT_VERSION=1.28.0
ok: header has no version literal
ok: hub carries no host-install version literal (241 .go/.html files scanned, 6 shapes checked)
ok: age is in the installed package set
ok: felhom-pbs-apply is fetched from the agent repo
ok: felhom-pbs-apply installed 0755 to /usr/local/sbin
ok: uninstall removes felhom-pbs-apply
ok: rendered agent.json defaults wg_tunnel.enabled=true
ok: byo assert no longer forbids wg_tunnel
ok: PVE_STORAGES default contains felhom-pbs
ok: no raw/branch/ ref in the installer — every run-time fetch is pinned
ok: fetch_raw pins the agent configs to the vouched agent version
ok: manifest: /scripts/ syncs from an installer tag
ok: manifest: the website still tracks main (a copy edit must not need a release)
ok: every arm that resolves the backup target also grants on it (2 resolution(s), 3 grant(s))
ok: the offsite tier arms no client-side prune (keep_last=0; retention is ep0's prune jobs)
hostinstall gates: ALL PASS
==============================================================================
== gate: hub-confirm (hub_confirm_gate.py)
==============================================================================
hub confirm gate OK — no native confirm()/prompt() in hub templates
==============================================================================
== gate: manifest-bearer (manifest_bearer_gate.py)
==============================================================================
manifests/felhom.secret.yaml:39 KNOWN-BACKLOG committed secret 65cee3c4...7a86 (secrets.md de-git backlog; not this gate's failure)
manifest bearer gate OK - no bearer-shaped literals in manifests/
==============================================================================
== gate: reuse-refs (reuse_refs_check.py /mnt/5_hdd/felhom.eu/git/felhom.eu)
==============================================================================
note [felhom.eu] line 75: api/handler.go (resolved by suffix → hub/internal/api/handler.go)
OK [felhom.eu]: 66 cited paths — exact 65, suffix 1, ambiguous 0, cross-repo 0, FAILED 0 (siblings searched: app-catalog-felhom.eu, felhom-agent, felhom-controller)
==============================================================================
== gate: instructions (instructions_gate.py /mnt/5_hdd/felhom.eu/git/felhom.eu)
==============================================================================
instructions_gate: /mnt/5_hdd/felhom.eu/git/felhom.eu
CLAUDE.md effective lines : 128 (ceiling 200)
version literals : 0
TEMPORARY blocks : 0
rule files : 4 (4 path-scoped)
workspace file : SYMLINK -> felhom.eu/documentation/runbooks/workspace-CLAUDE.md (resolves to the versioned copy)
memory index : 158 lines (ceiling 200), 20238 bytes (ceiling 25600)
memory index content : 33 version literal(s), 4 host address(es), 0 expired statement(s) [WARN only]
memory topic files : 125 indexed, 0 orphaned, 40 archived
register citations : 10 cited, 342 register items known
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:26: version literal '1.25.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- [ISO train v1.25.0](iso-train-v1.25.0-2026-07-23.md) — R-71 build-gate (golden≥floor), r
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:47: version literal '1.2.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- [Offsite pool-box aggregate](offsite-pool-box-aggregate-2026-07-17.md) — R-5 hub 0.64/0.
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:50: version literal '0.90.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- [Guest RAM resize + fast-tick](guest-ram-resize-fasttick-2026-07-17.md) — R-24 controlle
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:50: version literal '0.143.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- [Guest RAM resize + fast-tick](guest-ram-resize-fasttick-2026-07-17.md) — R-24 controlle
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:59: version literal '0.142.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- [Offsite continuity](offsite-continuity-2026-07-17.md) — ctrl 0.142.0 orphan guard + hub
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:59: version literal '0.60.0' — the fleet is not uniform, so it is stale within a day. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- [Offsite continuity](offsite-continuity-2026-07-17.md) — ctrl 0.142.0 orphan guard + hub
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md: ... and 27 more version literal(s) — full list from the tally counts above.
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:42: host address '192.168.0.0' — operations/nodes.md is the single home for these. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- [Tailscale N100 location-independent](tailscale-n100-location-independent-2026-07-19.md)
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:46: host address '167.233.158.164' — operations/nodes.md is the single home for these. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- [Offsite PBS box RAM ceiling](offsite-pbs-box-ram-ceiling-2026-07-27.md) — **root SSH =
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:46: host address '10.77.0.1' — operations/nodes.md is the single home for these. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- [Offsite PBS box RAM ceiling](offsite-pbs-box-ram-ceiling-2026-07-27.md) — **root SSH =
WARNING: /mnt/5_hdd/felhom.eu/git/.claude-memory/MEMORY.md:139: host address '192.168.0.180' — operations/nodes.md is the single home for these. Not a failure: Claude writes this file between sessions, so this warning is aimed at the model that will next edit it, not at whoever is pushing.
- **CC runs ON DooPlex** (192.168.0.180) — repos `/mnt/5_hdd/felhom.eu/git/felhom-*`, buil
instructions_gate: OK (11 warning(s))
==============================================================================
== gate: golden-currency (golden_currency_gate.py)
==============================================================================
newest released controller : 0.216.0 (## v0.216.0 — one physical disk, one verdict (2026-08-14, R-335))
newest golden baked : 0.216.0 (documentation/tests/golden-0.216.0-2026-08-18)
golden currency gate OK — the newest released controller has a golden (NOTE: this checks the BAKE, not the vouch — see the module docstring)
==============================================================================
== gate: wire-contract (wire_contract_gate.py)
==============================================================================
wire-contract gate — 190 tag(s) checked across 4 declared wire(s); 80 skipped (generic / opaque / allowlisted)
wire-contract gate OK — every emitted field is at least decodable by its receiver
(BLIND SPOTS: generic tag names skipped; name-reachability is not use; only the
declared ROOTS are covered — hub desired-state and the agent local API are not.)
==============================================================================
== gate: hub-copy (hub_copy_gate.py)
==============================================================================
hub-copy gate OK — 95 hub file(s) scanned for 2 retired name(s); 4 customer surface(s) scanned for 4 retrieval stem(s), 0 registered claim(s), none unregistered
drift: controller gate's STEMS match the shared list (4 stem(s))
(BLIND SPOT: this checks the WORDS in the four declared customer surfaces. It cannot
tell whether a true-looking sentence is wired to a predicate that is actually true —
that is what render tests are for. And it does not read the operator's screens.)
==============================================================================
== gate: due-checks (due_checks_gate.py)
==============================================================================
DUE-CHECKS GATE FAILED: 1 dated check(s) are due or overdue as of 2026-08-20 (UTC).
R-341 due 2026-08-19 1 day(s) OVERDUE
measure: ep0 proxy fd count + ESTAB/CLOSE-WAIT split; PID must still be 551655
the command and its preconditions are in the R-341 row of documentation/backlog/OPEN-ITEMS.md
Take the measurement, record the result in that R-row, then remove the row from the
DUE-CHECKS block. Moving the date instead is allowed — state the reason in the R-row.
NOTE: this gate fires on a PUSH, not on the date; it may be later than the date.
==============================================================================
== summary
==============================================================================
site OK (exit 0)
hostinstall OK (exit 0)
hub-confirm OK (exit 0)
manifest-bearer OK (exit 0)
reuse-refs OK (exit 0)
instructions OK (exit 0)
golden-currency OK (exit 0)
wire-contract OK (exit 0)
hub-copy OK (exit 0)
due-checks FAILED (exit 1)
CONVICTED: due-checks
rc=1
@@ -0,0 +1,7 @@
=== Part 4 verification: due-checks gate AFTER clearing the 2026-08-19 row ===
captured: 2026-08-20T08:39:48+00:00 UTC
due-checks gate OK — 1 dated check(s) pending, none due yet.
today (UTC): 2026-08-20
nearest: R-341 due 2026-08-25 (in 5 day(s)) — same, +7 d. Anchor the elapsed time on `ps -o lstart= -p $MainPID`, NO
(fires on the next PUSH after a date passes, not on the date itself — by design)
rc=0
@@ -0,0 +1,33 @@
=== repo_gates.py — pre-push, AFTER Part 4 ===
captured: 2026-08-20T08:41:20+00:00 UTC
==============================================================================
hub-copy gate OK — 95 hub file(s) scanned for 2 retired name(s); 4 customer surface(s) scanned for 4 retrieval stem(s), 0 registered claim(s), none unregistered
drift: controller gate's STEMS match the shared list (4 stem(s))
(BLIND SPOT: this checks the WORDS in the four declared customer surfaces. It cannot
tell whether a true-looking sentence is wired to a predicate that is actually true —
that is what render tests are for. And it does not read the operator's screens.)
==============================================================================
== gate: due-checks (due_checks_gate.py)
==============================================================================
due-checks gate OK — 1 dated check(s) pending, none due yet.
today (UTC): 2026-08-20
nearest: R-341 due 2026-08-25 (in 5 day(s)) — same, +7 d. Anchor the elapsed time on `ps -o lstart= -p $MainPID`, NO
(fires on the next PUSH after a date passes, not on the date itself — by design)
==============================================================================
== summary
==============================================================================
site OK (exit 0)
hostinstall OK (exit 0)
hub-confirm OK (exit 0)
manifest-bearer OK (exit 0)
reuse-refs OK (exit 0)
instructions OK (exit 0)
golden-currency OK (exit 0)
wire-contract OK (exit 0)
hub-copy OK (exit 0)
due-checks OK (exit 0)
all felhom.eu gates OK
rc=0
@@ -0,0 +1,62 @@
PHASE C PREDICTION — written 2026-08-20 ~08:40Z, BEFORE any Phase C measurement exists.
Same discipline as evidence-ep0-pbs-upgrade-2026-08-18/stop1-ruling.txt: committed to git
before the mutation is performed, so the result cannot be rationalised afterwards.
-------------------------------------------------------------------------------
WHAT PARTS 1-2 ALREADY ESTABLISHED (measured, not assumed)
-------------------------------------------------------------------------------
Q2 outcome : (a) — both sides hold every connection. 194+194 = 388 = ep0 total,
and the 4 new sockets that appeared during the 31-min persistence
window carry the SAME source ports on ep0 and on the boxes.
Owner on the box side : felhom-agent, 100% (194/194 on each box, one PID each).
pvestatd and proxmox-backup-client hold ZERO.
Elapsed-window rate : 201.6 fd/day (388 fd over 166251 s), 2s = 181.2..222.1.
Request split : libwww-perl (pvestatd) 81192 req -> 0 descriptors leaked
proxmox-backup-client 80061 req -> 0 descriptors leaked
Go-http-client (agent) 811 req -> 388 descriptors leaked
(387 agent GET /snapshots calls vs 388 leaked sockets: 1:1 within 1)
-------------------------------------------------------------------------------
THE INDEPENDENT VARIABLE Phase C ACTUALLY MOVES
-------------------------------------------------------------------------------
Stopping pvestatd on demo-hp removes BOTH pvestatd-driven streams from that box (the PVE PBS
storage plugin issues the libwww-perl datastore list AND shells out to proxmox-backup-client
for the status call, both on pvestatd's cycle). Expected request effect:
demo-hp requests 81612 per 46.18 h -> ~982 per 46.18 h (-98.8%)
BOTH boxes, per day ~84822 -> ~42900 (-49.4%)
So the request rate roughly HALVES, as the spike intends. IF the ep0 request count does NOT
fall by close to half, the independent variable did not move and no ratio may be computed.
-------------------------------------------------------------------------------
PREDICTIONS, ONE PER HYPOTHESIS. For a 10-hour window (scale linearly for other lengths).
-------------------------------------------------------------------------------
H1 NOT PROPORTIONAL — the leak tracks the AGENT's cycle, not the request rate.
This is what Parts 1-2 predict, and it is the prediction this run commits to.
demo-hp's felhom-agent keeps running throughout, so both boxes keep leaking.
expected rate : ~201.6 fd/day, UNCHANGED
expected count : 84 descriptors in 10 h (2s band 66..102)
expected split : ~42 from 10.77.0.2 and ~42 from 10.77.0.3 — SYMMETRIC.
The symmetry is the sharpest single discriminator: under H1 the quietened box keeps
leaking at the same rate as the control box.
H2 PROPORTIONAL to total request rate (the hypothesis Q3 was written to test).
expected rate : ~101 fd/day
expected count : 42 descriptors in 10 h (2s band 29..55)
expected split : ~42 from 10.77.0.2, ~0 from 10.77.0.3 — STRONGLY ASYMMETRIC.
H1 and H2 do not overlap at 2 sigma for a window of 10 h or more. A 6-hour window gives
50 vs 25 (2s bands 36..64 vs 15..35) — separable but tighter; prefer the overnight run.
Under Q2 outcome (b) (half-open, ep0 only) the prediction would have been H2-like, and under
(c) Phase C would not have been run at all. Both are recorded here for completeness; the
measurement already excluded them.
-------------------------------------------------------------------------------
WHAT WOULD FALSIFY THE PARTS 1-2 ATTRIBUTION
-------------------------------------------------------------------------------
A result matching H2 — demo-hp's contribution going to ~zero while pvestatd is stopped — would
mean the socket ownership reading is not the whole story, and the attribution above must be
withdrawn rather than defended. Say so plainly if it happens.
@@ -0,0 +1,32 @@
ELAPSED-WINDOW SLOPE — proxy PID 551655, started 2026-08-18 09:51:04 UTC, NOT restarted since
(method identical to evidence-ep0-pbs-upgrade-2026-08-18/step2-slope-computation.txt)
t0 2026-08-18T09:51:22Z fd=17 (ESTAB 0)
tA 2026-08-20T08:02:13Z fd=405 (ESTAB 388, CLOSE-WAIT 0)
interval 166251 s = 46.181 h = 1.9242 d
delta 388 fd
= 8.40 fd/hour = 201.6 fd/DAY
Poisson uncertainty on n=388: sqrt(n)=19.7 counts
1 sigma on the rate: +/- 10.2/day -> 191.4 .. 211.9
2 sigma on the rate: +/- 20.5/day -> 181.2 .. 222.1
PRE-REGISTERED PREDICTION (spike prompt section 2, written before any number existed):
predicted leaked-descriptor count 370-450; confirm-band 330-490
at 183/day tightest before-figure -> expected 352 fd
at 200/day 5.6 h before-figure -> expected 385 fd
at 225/day 32-min after-figure -> expected 433 fd
OBSERVED: 388
VERDICT: IN RANGE (370-450). The rate is UNCHANGED since the 4.2.5-1 upgrade.
COMPOSITION — the split, not just the total:
fd0 17 with ESTAB 0 -> fdA 405 with ESTAB 388. 17+388 = 405 = fdA exactly.
CLOSE-WAIT is ABSENT from the ss histogram entirely (0), not merely flat.
=> 100% of the descriptor growth is ESTABLISHED connections on :8007.
RUNWAY to the 65536 soft limit from fd=405 at this rate:
323 days = 0.88 years (approx 2027-07-09)
BEFORE-figures for comparison (same measurement method, PID 542065 and 551655):
183/day (31 min window, 2026-08-18) 200/day (5.64 h window, 2026-08-18) 225/day (32 min, post-upgrade)
This window is 88x longer than the 31-min window and 97x its descriptor count.
@@ -0,0 +1,23 @@
=== ep0 reading A — 1.1 precondition + R-341 count ===
ep0 date: 2026-08-20T08:02:13+00:00 / UTC: 2026-08-20T08:02:13+00:00
epoch: 1787212933
PID=551655
--- proxy start (lstart) ---
Tue Aug 18 09:51:04 2026
--- proxy start (systemd ActiveEnterTimestamp) ---
Tue 2026-08-18 03:54:54 UTC
--- host uptime ---
up 2 weeks, 2 days, 20 hours, 49 minutes
boot: 2026-08-03 11:12:57
--- fd count ---
405
--- limits ---
Max open files 65536 65536 files
--- listener ---
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 1024 *:8007 *:*
--- socket state histogram (all states, sport 8007) ---
388 ESTAB
1 LISTEN
--- pbs version ---
proxmox-backup-server 4.2.5-1 running version: 4.2.5
@@ -0,0 +1,59 @@
=== 1.2 ownership — who is at the far end (reading A) ===
UTC: 2026-08-20T08:02:39+00:00 epoch=1787212959
--- peer address histogram (established, sport 8007) ---
194 [::ffff:10.77.0.3]
194 [::ffff:10.77.0.2]
--- with process ---
Recv-Q Send-Q Local Address:Port Peer Address:Port Process
0 0 [::ffff:10.77.0.1]:8007 [::ffff:10.77.0.3]:37112 users:(("proxmox-backup-",pid=551655,fd=247))
0 0 [::ffff:10.77.0.1]:8007 [::ffff:10.77.0.2]:40330 users:(("proxmox-backup-",pid=551655,fd=390))
0 0 [::ffff:10.77.0.1]:8007 [::ffff:10.77.0.3]:38018 users:(("proxmox-backup-",pid=551655,fd=213))
0 0 [::ffff:10.77.0.1]:8007 [::ffff:10.77.0.3]:56006 users:(("proxmox-backup-",pid=551655,fd=255))
--- full 4-tuple + timers (SAVED FOR DIFF) ---
389 /tmp/ep0-sockA.txt
--- sample (first 10) ---
Recv-Q Send-Q Local Address:Port Peer Address:Port Process
0 0 [::ffff:10.77.0.1]:8007 [::ffff:10.77.0.3]:37112 users:(("proxmox-backup-",pid=551655,fd=247)) timer:(keepalive,1min13sec,0)
0 0 [::ffff:10.77.0.1]:8007 [::ffff:10.77.0.2]:40330 users:(("proxmox-backup-",pid=551655,fd=390)) timer:(keepalive,51sec,0)
0 0 [::ffff:10.77.0.1]:8007 [::ffff:10.77.0.3]:38018 users:(("proxmox-backup-",pid=551655,fd=213)) timer:(keepalive,30sec,0)
0 0 [::ffff:10.77.0.1]:8007 [::ffff:10.77.0.3]:56006 users:(("proxmox-backup-",pid=551655,fd=255)) timer:(keepalive,28sec,0)
0 0 [::ffff:10.77.0.1]:8007 [::ffff:10.77.0.2]:32844 users:(("proxmox-backup-",pid=551655,fd=136)) timer:(keepalive,12sec,0)
0 0 [::ffff:10.77.0.1]:8007 [::ffff:10.77.0.2]:58280 users:(("proxmox-backup-",pid=551655,fd=261)) timer:(keepalive,,0)
0 0 [::ffff:10.77.0.1]:8007 [::ffff:10.77.0.2]:33480 users:(("proxmox-backup-",pid=551655,fd=252)) timer:(keepalive,24sec,0)
0 0 [::ffff:10.77.0.1]:8007 [::ffff:10.77.0.3]:38610 users:(("proxmox-backup-",pid=551655,fd=96)) timer:(keepalive,18sec,0)
0 0 [::ffff:10.77.0.1]:8007 [::ffff:10.77.0.2]:55858 users:(("proxmox-backup-",pid=551655,fd=398)) timer:(keepalive,6.044sec,0)
0 0 [::ffff:10.77.0.1]:8007 [::ffff:10.77.0.2]:37514 users:(("proxmox-backup-",pid=551655,fd=22)) timer:(keepalive,1min3sec,0)
--- idle-age distribution: parse timer(keepalive,<age>) ---
19 timer:(keepalive,30sec,
19 timer:(keepalive,18sec,
18 timer:(keepalive,38sec,
18 timer:(keepalive,14sec,
16 timer:(keepalive,26sec,
16 timer:(keepalive,22sec,
16 timer:(keepalive,10sec,
13 timer:(keepalive,34sec,
12 timer:(keepalive,1min13sec,
11 timer:(keepalive,42sec,
10 timer:(keepalive,59sec,
10 timer:(keepalive,16sec,
10 timer:(keepalive,1.980sec,
9 timer:(keepalive,6.080sec,
9 timer:(keepalive,55sec,
9 timer:(keepalive,47sec,
9 timer:(keepalive,32sec,
9 timer:(keepalive,24sec,
9 timer:(keepalive,1min5sec,
9 timer:(keepalive,1min3sec,
--- wireguard peers ---
wg0 KNaFXHiY9l08UhPO4gxUBFQz2WF3RYWyUGL8k+hzDVo= 37.191.56.193:43407
yNVpWt9Ekid9p459/VLHbmCDHIi8xTUYHbgazUp5S00= 37.191.56.193:62827
sJzw0h9w8IIeM9/BG/a1xetxEDGJQEUFMSXP2EQpyDk= 37.191.56.193:52231
kdhOHMyADRpaTaFmh8ypwsWngvBpIyMyon80DJK+00M= 37.191.56.193:46129
wg0 KNaFXHiY9l08UhPO4gxUBFQz2WF3RYWyUGL8k+hzDVo= 1787212855
wg0 yNVpWt9Ekid9p459/VLHbmCDHIi8xTUYHbgazUp5S00= 1787212952
wg0 sJzw0h9w8IIeM9/BG/a1xetxEDGJQEUFMSXP2EQpyDk= 1787212944
wg0 kdhOHMyADRpaTaFmh8ypwsWngvBpIyMyon80DJK+00M= 1786601749
@@ -0,0 +1,46 @@
=== 1.3 descriptor type breakdown (Proxmox support's own diagnostic) ===
UTC: 2026-08-20T08:02:59+00:00
PID=551655
--- lsof TYPE histogram ---
390 IPv6
32 REG
7 unix
3 a_inode
2 DIR
1 CHR
--- lsof TCP count ---
390
--- /proc fd link-target class histogram (independent of lsof) ---
397 socket
2 anon_inode:[eventpoll]
1 anon_inode:[eventfd]
1 0
1 /var/log/proxmox-backup/api/auth.log
1 /var/log/proxmox-backup/api/access.log
1 /var/lib/proxmox-backup/rrdb/rrd.journal
1 /mnt/pbs-datastore/.lock
1 /dev/null
=== 1.6 control-window cleanliness ===
--- access log files ---
total 55744
drwxr-xr-x 2 backup backup 4096 Aug 20 00:00 .
drwxr-xr-x 4 root root 4096 Jul 3 19:33 ..
-rw-r--r-- 1 backup backup 4023740 Aug 20 08:02 access.log
-rw-r--r-- 1 backup backup 43115482 Aug 19 23:59 access.log.1
-rw-r--r-- 1 backup backup 700249 Jul 24 00:00 access.log.10.zst
-rw-r--r-- 1 backup backup 724750 Jul 19 00:00 access.log.11.zst
-rw-r--r-- 1 backup backup 612227 Jul 15 00:00 access.log.12.zst
-rw-r--r-- 1 backup backup 959113 Aug 20 00:00 access.log.2.zst
-rw-r--r-- 1 backup backup 975898 Aug 16 00:00 access.log.3.zst
-rw-r--r-- 1 backup backup 1274332 Aug 13 00:00 access.log.4.zst
-rw-r--r-- 1 backup backup 1078875 Aug 9 00:00 access.log.5.zst
-rw-r--r-- 1 backup backup 850578 Aug 6 00:00 access.log.6.zst
-rw-r--r-- 1 backup backup 857058 Aug 3 00:00 access.log.7.zst
-rw-r--r-- 1 backup backup 986392 Jul 31 00:00 access.log.8.zst
-rw-r--r-- 1 backup backup 705209 Jul 28 00:00 access.log.9.zst
-rw-r--r-- 1 backup backup 164293 Aug 18 03:55 auth.log
--- log format sample ---
::ffff:10.77.0.3 - felhom@pbs!demo-hp [20/08/2026:08:02:52 +0000] "GET /api2/json/admin/datastore/felhom-offsite/status" 200 67 proxmox-backup-client/1.0
::ffff:10.77.0.3 - felhom@pbs!demo-hp [20/08/2026:08:02:54 +0000] "GET /api2/json/admin/datastore" 200 110 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [20/08/2026:08:02:54 +0000] "GET /api2/json/admin/datastore/felhom-offsite/status" 200 67 proxmox-backup-client/1.0
@@ -0,0 +1,31 @@
=== ep0 reading B (1.4 persistence) ===
UTC: 2026-08-20T08:33:42+00:00 epoch=1787214822
PID=551655 (precondition: must still be 551655)
fd count: 409
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 1024 *:8007 *:*
--- state histogram ---
392 ESTAB
1 LISTEN
--- peer histogram ---
196 10.77.0.2
196 10.77.0.3
=== 1.4 DIFF of full 4-tuples between reading A and reading B ===
sockets in reading A : 388
sockets in reading B : 392
in BOTH (long-lived = LEAK) : 388
only in A (closed since) : 0
only in B (opened since = new): 4
--- the 'only in A' set (did ANY socket close?) ---
--- the 'only in B' set (new arrivals in the interval) ---
[::ffff:10.77.0.1]:8007 [::ffff:10.77.0.2]:45620
[::ffff:10.77.0.1]:8007 [::ffff:10.77.0.2]:53226
[::ffff:10.77.0.1]:8007 [::ffff:10.77.0.3]:37554
[::ffff:10.77.0.1]:8007 [::ffff:10.77.0.3]:46214
--- keepalive-timer distribution over the LONG-LIVED set (reading B) ---
392 timer:(keepalive)
--- retransmit counter across ALL sockets (0 = peer is answering keepalives) ---
392 retrans=0
@@ -0,0 +1,113 @@
=== date range per log file ===
--- access.log ---
20/08/2026:00:00:04 +0000
20/08/2026:08:03:25 +0000
--- access.log.1 ---
16/08/2026:00:00:00 +0000
19/08/2026:23:59:56 +0000
--- access.log.2.zst ---
13/08/2026:00:00:04 +0000
16/08/2026:00:00:00 +0000
--- access.log.3.zst ---
09/08/2026:00:00:00 +0000
12/08/2026:23:59:58 +0000
=== 1.5 ATTRIBUTION — who makes the requests, by source IP + token + user agent ===
--- window slice: 18/08 from 09:51 onward, 19/08 all, 20/08 to now ---
5 10.77.0.3 [18/08/2026:12:52:31 libwww-perl/6.78
5 10.77.0.3 [16/08/2026:18:52:31 libwww-perl/6.78
5 10.77.0.3 [16/08/2026:12:52:31 libwww-perl/6.78
5 10.77.0.2 [20/08/2026:08:02:09 libwww-perl/6.78
5 10.77.0.2 [20/08/2026:07:52:09 libwww-perl/6.78
5 10.77.0.2 [20/08/2026:07:47:09 libwww-perl/6.78
5 10.77.0.2 [20/08/2026:07:42:09 libwww-perl/6.78
5 10.77.0.2 [20/08/2026:06:32:09 libwww-perl/6.78
5 10.77.0.2 [20/08/2026:05:27:09 libwww-perl/6.78
5 10.77.0.2 [20/08/2026:05:22:09 libwww-perl/6.78
5 10.77.0.2 [20/08/2026:05:17:09 libwww-perl/6.78
5 10.77.0.2 [20/08/2026:04:12:09 libwww-perl/6.78
5 10.77.0.2 [18/08/2026:03:54:55 libwww-perl/6.78
5 10.77.0.2 [15/08/2026:22:34:57 libwww-perl/6.78
5 10.77.0.2 [15/08/2026:12:54:56 libwww-perl/6.78
5 10.77.0.2 [15/08/2026:09:29:56 libwww-perl/6.78
5 10.77.0.2 [15/08/2026:08:04:56 libwww-perl/6.78
5 10.77.0.2 [14/08/2026:23:04:56 libwww-perl/6.78
5 10.77.0.2 [14/08/2026:19:24:56 libwww-perl/6.78
5 10.77.0.2 [14/08/2026:18:29:56 libwww-perl/6.78
=== request TOTALS in the elapsed window (18/08 09:51Z -> now) ===
3538 17/08/2026 13
3538 17/08/2026 14
3538 17/08/2026 15
3548 17/08/2026 16
3538 17/08/2026 17
673 17/08/2026 18
335 18/08/2026 03
3546 18/08/2026 04
3538 18/08/2026 05
3546 18/08/2026 06
3536 18/08/2026 07
3538 18/08/2026 08
3546 18/08/2026 09
3550 18/08/2026 10
3546 18/08/2026 11
3543 18/08/2026 12
3540 18/08/2026 13
3540 18/08/2026 14
3540 18/08/2026 15
3548 18/08/2026 16
3540 18/08/2026 17
3551 18/08/2026 18
3540 18/08/2026 19
3540 18/08/2026 20
3540 18/08/2026 21
3550 18/08/2026 22
3540 18/08/2026 23
3550 19/08/2026 00
3540 19/08/2026 01
3540 19/08/2026 02
3540 19/08/2026 03
3553 19/08/2026 04
3540 19/08/2026 05
3552 19/08/2026 06
3540 19/08/2026 07
3540 19/08/2026 08
3540 19/08/2026 09
3546 19/08/2026 10
3540 19/08/2026 11
3546 19/08/2026 12
3540 19/08/2026 13
3540 19/08/2026 14
3540 19/08/2026 15
3548 19/08/2026 16
3540 19/08/2026 17
3548 19/08/2026 18
3540 19/08/2026 19
3540 19/08/2026 20
3540 19/08/2026 21
3548 19/08/2026 22
3540 19/08/2026 23
3548 20/08/2026 00
3540 20/08/2026 01
3540 20/08/2026 02
3540 20/08/2026 03
3546 20/08/2026 04
3540 20/08/2026 05
3546 20/08/2026 06
3539 20/08/2026 07
209 20/08/2026 08
=== 1.6 BACKUP ACTIVITY in the window — protocol upgrades / writes ===
28 "POST /api2/json/admin/datastore/felhom-offsite/verify" 200 139
28 "POST /api2/json/admin/datastore/felhom-offsite/verify" 200 131
4 "GET //api2/json/reader?backup-id=9201&backup-time=1787025523&backup-type=ct&debug=true&ns=demo-hp&store=felhom-offsite" 101 0
4 "GET //api2/json/reader?backup-id=9201&backup-time=1787025463&backup-type=ct&debug=true&ns=demo-felhom&store=felhom-offsite" 101 0
4 "GET //api2/json/reader?backup-id=9201&backup-time=1786476553&backup-type=ct&debug=true&ns=demo-hp&store=felhom-offsite" 101 0
1 "PUT /api2/json/admin/datastore/felhom-offsite/notes?backup-id=9201&backup-time=1787025523&backup-type=ct&notes=felhom+local-api&ns=demo-hp" 200 13
1 "PUT /api2/json/admin/datastore/felhom-offsite/notes?backup-id=9201&backup-time=1787025463&backup-type=ct&notes=felhom+local-api&ns=demo-felhom" 200 13
1 "POST /api2/json/admin/datastore/felhom-offsite/upload-backup-log?backup-id=9201&backup-time=1787025523&backup-type=ct&ns=demo-hp" 200 13
1 "POST /api2/json/admin/datastore/felhom-offsite/upload-backup-log?backup-id=9201&backup-time=1787025463&backup-type=ct&ns=demo-felhom" 200 13
1 "GET //api2/json/backup?backup-id=9201&backup-time=1787025523&backup-type=ct&benchmark=false&debug=true&ns=demo-hp&store=felhom-offsite" 101 0
1 "GET //api2/json/backup?backup-id=9201&backup-time=1787025463&backup-type=ct&benchmark=false&debug=true&ns=demo-felhom&store=felhom-offsite" 101 0
--- explicit: any line mentioning backup/upload/chunk/finish ---
2
@@ -0,0 +1,34 @@
ATTRIBUTION — who makes the requests, and who leaks the descriptors
window 2026-08-18 ~09:51Z -> 2026-08-20 08:03Z (166251 s = 46.18 h)
user agent endpoint requests req/day
libwww-perl/6.78 pvestatd: GET /admin/datastore 81192 42195
proxmox-backup-client/1.0 GET .../felhom-offsite/status 80061 41607
libwww-perl/6.78 GET .../felhom-offsite/snapshots 1131 588
Go-http-client/1.1 GET .../felhom-offsite/snapshots 387 201
Go-http-client/1.1 GET /version 371 193
Go-http-client/1.1 POST .../verify + task status 53 28
TOTAL (both boxes, all agents) 163215 84822
LEAKED DESCRIPTORS OVER THE SAME WINDOW: 388 (all ESTABLISHED on :8007)
requests by libwww-perl + proxmox-backup-client : 162404 descriptors leaked: 0
requests by Go-http-client/1.1 (felhom-agent) : 811 descriptors leaked: 388
Go-http-client GET .../snapshots calls : 387
leaked ESTAB sockets : 388 -> ratio 1.0026 (ONE per call, within 1)
Per box, at the instant of reading A:
ep0 ESTAB from 10.77.0.2 (demo-felhom) = 194 ; from 10.77.0.3 (demo-hp) = 194
demo-felhom ESTAB to 10.77.0.1:8007 = 194 ; demo-hp = 194
194 + 194 = 388 = ep0 total. Both sides agree EXACTLY -> Q2 outcome (a).
CADENCE RECONCILIATION (why 194 and not 185):
hub report poll poll_seconds=900 -> 184.7 cycles in the window
pbs VerifyLoop DefaultVerifyCadence=6h -> 7.7 cycles in the window
sum -> 192.4 observed leaked/box: 194
observed POST .../verify per box in the window: 8 (matches 7.7)
observed GET /version per box: ~185.5 (matches the 184.7 hub cycles)
Mean interval between leaked descriptors, per box:
857 s = 14.28 min (900 s poll + a 6 h verify cycle superposed)
@@ -0,0 +1,69 @@
=== 1.5 ATTRIBUTION (clean) — source IP + token + endpoint + user agent ===
window: 18/08 09:51Z -> 20/08 08:0xZ, from access.log.1 + access.log
40598 10.77.0.3 felhom@pbs!demo-hp /api2/json/admin/datastore" 200 libwww-perl/6.78
40594 10.77.0.2 felhom@pbs!demo-felhom /api2/json/admin/datastore" 200 libwww-perl/6.78
40032 10.77.0.3 felhom@pbs!demo-hp /api2/json/admin/datastore/felhom-offsite/status" 200 proxmox-backup-client/1.0
40029 10.77.0.2 felhom@pbs!demo-felhom /api2/json/admin/datastore/felhom-offsite/status" 200 proxmox-backup-client/1.0
566 10.77.0.3 felhom@pbs!demo-hp /api2/json/admin/datastore/felhom-offsite/snapshots libwww-perl/6.78
565 10.77.0.2 felhom@pbs!demo-felhom /api2/json/admin/datastore/felhom-offsite/snapshots libwww-perl/6.78
194 10.77.0.2 felhom@pbs!demo-felhom /api2/json/admin/datastore/felhom-offsite/snapshots Go-http-client/1.1
193 10.77.0.3 felhom@pbs!demo-hp /api2/json/admin/datastore/felhom-offsite/snapshots Go-http-client/1.1
186 10.77.0.2 felhom@pbs!demo-felhom /api2/json/version" 200 Go-http-client/1.1
185 10.77.0.3 felhom@pbs!demo-hp /api2/json/version" 200 Go-http-client/1.1
8 10.77.0.3 felhom@pbs!demo-hp /api2/json/admin/datastore/felhom-offsite/verify" 200 Go-http-client/1.1
8 10.77.0.2 felhom@pbs!demo-felhom /api2/json/admin/datastore/felhom-offsite/verify" 200 Go-http-client/1.1
5 10.77.0.3 felhom@pbs!demo-hp /api2/json/nodes/felhom-hetzner/tasks/UPID:felhom-hetzner:00086AE7:07B20B30:00000003:6A84A9FD:verify:felhom%5Cx2doffsite%5Cx3ans-demo%5Cx2dhp:felhom@pbs%21demo-hp:/status" 200 Go-http-client/1.1
5 10.77.0.3 felhom@pbs!demo-hp /api2/json/nodes/felhom-hetzner/tasks/UPID:felhom-hetzner:00086AE7:07B20B30:00000001:6A84559D:verify:felhom%5Cx2doffsite%5Cx3ans-demo%5Cx2dhp:felhom@pbs%21demo-hp:/status" 200 Go-http-client/1.1
4 10.77.0.3 felhom@pbs!demo-hp /api2/json/nodes/felhom-hetzner/tasks/UPID:felhom-hetzner:00086AE7:07B20B30:00000006:6A84FE5D:verify:felhom%5Cx2doffsite%5Cx3ans-demo%5Cx2dhp:felhom@pbs%21demo-hp:/status" 200 Go-http-client/1.1
4 10.77.0.3 felhom@pbs!demo-hp //api2/json/reader proxmox-backup-client/1.0
4 10.77.0.2 felhom@pbs!demo-felhom /api2/json/nodes/felhom-hetzner/tasks/UPID:felhom-hetzner:00086AE7:07B20B30:0000000C:6A8534EF:verify:felhom%5Cx2doffsite%5Cx3ans-demo%5Cx2dfelhom:felhom@pbs%21demo-felhom:/status" 200 Go-http-client/1.1
4 10.77.0.2 felhom@pbs!demo-felhom /api2/json/nodes/felhom-hetzner/tasks/UPID:felhom-hetzner:00086AE7:07B20B30:00000004:6A84E08F:verify:felhom%5Cx2doffsite%5Cx3ans-demo%5Cx2dfelhom:felhom@pbs%21demo-felhom:/status" 200 Go-http-client/1.1
4 10.77.0.2 felhom@pbs!demo-felhom /api2/json/nodes/felhom-hetzner/tasks/UPID:felhom-hetzner:00086AE7:07B20B30:00000000:6A8437CF:verify:felhom%5Cx2doffsite%5Cx3ans-demo%5Cx2dfelhom:felhom@pbs%21demo-felhom:/status" 200 Go-http-client/1.1
4 10.77.0.2 felhom@pbs!demo-felhom //api2/json/reader proxmox-backup-client/1.0
3 10.77.0.3 felhom@pbs!demo-hp /api2/json/nodes/felhom-hetzner/tasks/UPID:felhom-hetzner:00086AE7:07B20B30:00000011:6A8552BD:verify:felhom%5Cx2doffsite%5Cx3ans-demo%5Cx2dhp:felhom@pbs%21demo-hp:/status" 200 Go-http-client/1.1
2 10.77.0.3 felhom@pbs!demo-hp /api2/json/nodes/felhom-hetzner/tasks/UPID:felhom-hetzner:00086AE7:07B20B30:0000001D:6A86A43D:verify:felhom%5Cx2doffsite%5Cx3ans-demo%5Cx2dhp:felhom@pbs%21demo-hp:/status" 200 Go-http-client/1.1
2 10.77.0.3 felhom@pbs!demo-hp /api2/json/nodes/felhom-hetzner/tasks/UPID:felhom-hetzner:00086AE7:07B20B30:00000019:6A864FDD:verify:felhom%5Cx2doffsite%5Cx3ans-demo%5Cx2dhp:felhom@pbs%21demo-hp:/status" 200 Go-http-client/1.1
2 10.77.0.3 felhom@pbs!demo-hp /api2/json/nodes/felhom-hetzner/tasks/UPID:felhom-hetzner:00086AE7:07B20B30:00000016:6A85FB7D:verify:felhom%5Cx2doffsite%5Cx3ans-demo%5Cx2dhp:felhom@pbs%21demo-hp:/status" 200 Go-http-client/1.1
2 10.77.0.3 felhom@pbs!demo-hp /api2/json/nodes/felhom-hetzner/tasks/UPID:felhom-hetzner:00086AE7:07B20B30:00000014:6A85A71D:verify:felhom%5Cx2doffsite%5Cx3ans-demo%5Cx2dhp:felhom@pbs%21demo-hp:/status" 200 Go-http-client/1.1
=== per-box request totals in that same window ===
81603 10.77.0.2
81612 10.77.0.3
=== which token belongs to which IP ===
10.77.0.2 felhom@pbs!demo-felhom
10.77.0.3 felhom@pbs!demo-hp
=== 1.6 verify-job timestamps (are they inside the window?) ===
1 17/08/2026:12:52:45 +0000
1 17/08/2026:16:45:35 +0000
1 18/08/2026:04:45:35 +0000
1 18/08/2026:06:52:45 +0000
1 18/08/2026:10:45:35 +0000
1 18/08/2026:12:52:45 +0000
1 18/08/2026:16:45:35 +0000
1 18/08/2026:18:52:45 +0000
1 18/08/2026:22:45:35 +0000
1 19/08/2026:00:52:45 +0000
1 19/08/2026:04:45:35 +0000
1 19/08/2026:06:52:45 +0000
1 19/08/2026:10:45:35 +0000
1 19/08/2026:12:52:45 +0000
1 19/08/2026:16:45:35 +0000
1 19/08/2026:18:52:45 +0000
1 19/08/2026:22:45:35 +0000
1 20/08/2026:00:52:45 +0000
1 20/08/2026:04:45:35 +0000
1 20/08/2026:06:52:45 +0000
=== 1.6 reader / backup protocol upgrades with timestamps ===
1 18/08/2026 03:57
1 18/08/2026 03:58
3 19/08/2026 04:45
1 19/08/2026 04:50
3 19/08/2026 06:52
1 19/08/2026 06:54
=== 1.6 any HTTP error (non-2xx) in the window ===
8 101
163207 200
@@ -0,0 +1,31 @@
=== 1.6 window cleanliness — reboots / restarts / flaps ===
--- box uptimes + agent + pvestatd ---
[felhom-pve]
hostname: demo-felhom
boot: 2026-08-10 09:25:35
uptime: up 1 week, 3 days, 1 hour, 8 minutes
wg handshake: f3d1ZI7/2/r+FWz8eo8Wl2HVygTZK9wBIN2M3dNHsgA= 1787214780
wg transfer: wg-felhom f3d1ZI7/2/r+FWz8eo8Wl2HVygTZK9wBIN2M3dNHsgA= 6720170264 3106980548
[demo-hp]
hostname: felhom-host
boot: 2026-08-10 09:28:18
uptime: up 1 week, 3 days, 1 hour, 6 minutes
wg handshake: f3d1ZI7/2/r+FWz8eo8Wl2HVygTZK9wBIN2M3dNHsgA= 1787214869
wg transfer: wg-felhom f3d1ZI7/2/r+FWz8eo8Wl2HVygTZK9wBIN2M3dNHsgA= 8576925788 3139271384
--- ep0 host/daemon restart evidence ---
boot: 2026-08-03 11:12:57
proxy ActiveEnter: Tue 2026-08-18 03:54:54 UTC
proxy NRestarts: 0
api NRestarts: 0
wg peers + allowed-ips:
wg0 KNaFXHiY9l08UhPO4gxUBFQz2WF3RYWyUGL8k+hzDVo= 10.77.0.2/32
wg0 yNVpWt9Ekid9p459/VLHbmCDHIi8xTUYHbgazUp5S00= 10.77.0.250/32
wg0 sJzw0h9w8IIeM9/BG/a1xetxEDGJQEUFMSXP2EQpyDk= 10.77.0.3/32
wg0 kdhOHMyADRpaTaFmh8ypwsWngvBpIyMyon80DJK+00M= 10.77.0.4/32
wg handshakes (epoch):
wg0 KNaFXHiY9l08UhPO4gxUBFQz2WF3RYWyUGL8k+hzDVo= 1787214780
wg0 yNVpWt9Ekid9p459/VLHbmCDHIi8xTUYHbgazUp5S00= 1787214829
wg0 sJzw0h9w8IIeM9/BG/a1xetxEDGJQEUFMSXP2EQpyDk= 1787214869
wg0 kdhOHMyADRpaTaFmh8ypwsWngvBpIyMyon80DJK+00M= 1786601749
now epoch: 1787214870
@@ -0,0 +1,7 @@
=== locate hub pod ===
felhom-system contact-mailer-5bb869b85b-pqtjt 1/1 Running 0 52d
felhom-system felhom-webpage-7969454965-gcv5n 3/3 Running 226 (6d3h ago) 7d2h
felhom-system filebrowser-59ff87cd88-662kz 1/1 Running 0 70d
felhom-system hub-654bbc8fbc-9wld9 1/1 Running 0 39h
felhom-system umami-7cd7f95cd8-89wrn 1/1 Running 1 (70d ago) 74d
felhom-system umami-db-5fd98f59c5-xhwfr 1/1 Running 0 70d
@@ -0,0 +1,25 @@
=== hub pod: hub-654bbc8fbc-9wld9 (felhom-system), age 39h, restarts 0 ===
NOTE: pod is 39h old, so it covers 08-19 ~17:30Z onward only; the window starts 08-18 09:51Z.
--- reachability / unreachable / recovered events in pod log ---
2026/08/18 19:29:07 [INFO] Offsite pool-box checker initialized: box=611714 fill warn=80% crit=90%, oversub warn=2.00x, unreachable after 3 consecutive failed reads, refresh 15m0s
2026/08/18 19:29:07 [INFO] PBS-DR box checker initialized: fill warn=80% crit=90%, unreachable after 3 consecutive failed reads, refresh 15m0s
(empty above = no unreachable/recovered event logged)
--- PBS-DR / offsite gauge lines (positive observable) ---
2026/08/20 09:34:31 [INFO] PBS-DR box refreshed: 3.7% full (3.7 GB of 97.9 GB)
2026/08/20 09:46:32 [INFO] Offsite pool-box refreshed: 0.2% full (2.4 GB of 1.00 TB), Σ shared quota 150 GB, oversub 0.15x
2026/08/20 09:50:30 [INFO] PBS-DR box refreshed: 3.7% full (3.7 GB of 97.9 GB)
2026/08/20 10:02:31 [INFO] Offsite pool-box refreshed: 0.2% full (2.4 GB of 1.00 TB), Σ shared quota 150 GB, oversub 0.15x
2026/08/20 10:06:31 [INFO] PBS-DR box refreshed: 3.7% full (3.7 GB of 97.9 GB)
2026/08/20 10:17:31 [INFO] Offsite pool-box refreshed: 0.2% full (2.4 GB of 1.00 TB), Σ shared quota 150 GB, oversub 0.15x
2026/08/20 10:22:31 [INFO] PBS-DR box refreshed: 3.7% full (3.7 GB of 97.9 GB)
2026/08/20 10:32:32 [INFO] Offsite pool-box refreshed: 0.2% full (2.4 GB of 1.00 TB), Σ shared quota 150 GB, oversub 0.15x
--- host reports from the two boxes (last few) ---
2026/08/20 10:22:35 [INFO] host-report from demo-hp-bb76ea (1 guests, 5 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 14944 bytes)
2026/08/20 10:22:35 [INFO] DR-recipe host-half stored for customer demo-hp (host demo-hp-bb76ea, v1)
2026/08/20 10:30:33 [INFO] host-report from demo-felhom-8363b5 (1 guests, 4 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 14340 bytes)
2026/08/20 10:30:33 [INFO] DR-recipe host-half stored for customer demo-felhom (host demo-felhom-8363b5, v1)
2026/08/20 10:31:34 [INFO] DR-recipe app-half stored for customer demo-hp (v1)
2026/08/20 10:31:34 [INFO] Received report from demo-hp (3145 bytes)
@@ -0,0 +1,42 @@
##################### felhom-pve #####################
host: demo-felhom date: 2026-08-20T10:04:37+02:00 UTC: 2026-08-20T08:04:37+00:00 epoch=1787213077
--- state histogram, dport 8007 ---
194 ESTAB
--- total established to 8007 ---
194
--- WHICH PROCESS holds them (histogram) ---
194 users:(("felhom-agent"
--- per-PID ---
194 pid=2596329
--- name the PIDs ---
pid=2596329 felhom-agent /usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json
--- sample 5 with timers ---
Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
0 0 10.77.0.2:32844 10.77.0.1:8007 users:(("felhom-agent",pid=2596329,fd=69)) timer:(keepalive,5.776sec,0)
0 0 10.77.0.2:32894 10.77.0.1:8007 users:(("felhom-agent",pid=2596329,fd=197)) timer:(keepalive,3.216sec,0)
0 0 10.77.0.2:33098 10.77.0.1:8007 users:(("felhom-agent",pid=2596329,fd=190)) timer:(keepalive,3.728sec,0)
0 0 10.77.0.2:33108 10.77.0.1:8007 users:(("felhom-agent",pid=2596329,fd=194)) timer:(keepalive,8.336sec,0)
0 0 10.77.0.2:33256 10.77.0.1:8007 users:(("felhom-agent",pid=2596329,fd=139)) timer:(keepalive,1.680sec,0)
--- save full for diff ---
195 /tmp/box-sockA.txt
##################### demo-hp #####################
host: felhom-host date: 2026-08-20T10:04:38+02:00 UTC: 2026-08-20T08:04:38+00:00 epoch=1787213078
--- state histogram, dport 8007 ---
194 ESTAB
--- total established to 8007 ---
194
--- WHICH PROCESS holds them (histogram) ---
194 users:(("felhom-agent"
--- per-PID ---
194 pid=3199936
--- name the PIDs ---
pid=3199936 felhom-agent /usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json
--- sample 5 with timers ---
Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
0 0 10.77.0.3:32980 10.77.0.1:8007 users:(("felhom-agent",pid=3199936,fd=106)) timer:(keepalive,12sec,0)
0 0 10.77.0.3:33490 10.77.0.1:8007 users:(("felhom-agent",pid=3199936,fd=61)) timer:(keepalive,4.018sec,0)
0 0 10.77.0.3:34054 10.77.0.1:8007 users:(("felhom-agent",pid=3199936,fd=70)) timer:(keepalive,433ms,0)
0 0 10.77.0.3:34174 10.77.0.1:8007 users:(("felhom-agent",pid=3199936,fd=140)) timer:(keepalive,12sec,0)
0 0 10.77.0.3:34348 10.77.0.1:8007 users:(("felhom-agent",pid=3199936,fd=20)) timer:(keepalive,8.113sec,0)
--- save full for diff ---
195 /tmp/box-sockA.txt
@@ -0,0 +1,54 @@
##################### felhom-pve #####################
host: demo-felhom UTC: 2026-08-20T08:34:01+00:00 epoch=1787214841
--- state histogram, dport 8007 ---
196 ESTAB
--- owner ---
196 users:(("felhom-agent"
--- persistence diff A vs B (source port is the key) ---
A=194 B=196 both=194 closed=0 new=2
closed set:
new set:
10.77.0.2:45620
10.77.0.2:53226
--- agent uptime (has it restarted? if so the count would have reset) ---
Wed 2026-08-12 18:45:28 CEST
2596329
--- agent fd total ---
agent fds: 208
Max open files 524287 524288 files
--- pvestatd state (Part 3 precondition) ---
active
1215
--- backup schedule: next offsite run? ---
"target_id": "felhom-pbs" "cadence_seconds": 604800 "keep_last": 0
--- last PBS snapshots on this box (pvesm) ---
Volid Format Type Size VMID
felhom-pbs:backup/ct/9201/2026-08-11T04:54:07Z pbs-ct backup 3978122771 9201
felhom-pbs:backup/ct/9201/2026-08-18T03:57:43Z pbs-ct backup 4104429602 9201
##################### demo-hp #####################
host: felhom-host UTC: 2026-08-20T08:34:02+00:00 epoch=1787214842
--- state histogram, dport 8007 ---
196 ESTAB
--- owner ---
196 users:(("felhom-agent"
--- persistence diff A vs B (source port is the key) ---
A=194 B=196 both=194 closed=0 new=2
closed set:
new set:
10.77.0.3:37554
10.77.0.3:46214
--- agent uptime (has it restarted? if so the count would have reset) ---
Wed 2026-08-12 20:52:29 CEST
3199936
--- agent fd total ---
agent fds: 208
Max open files 524287 524288 files
--- pvestatd state (Part 3 precondition) ---
active
1298
--- backup schedule: next offsite run? ---
"target_id": "felhom-pbs" "cadence_seconds": 604800 "keep_last": 0
--- last PBS snapshots on this box (pvesm) ---
Volid Format Type Size VMID
felhom-pbs:backup/ct/9201/2026-08-11T19:29:13Z pbs-ct backup 4067039591 9201
felhom-pbs:backup/ct/9201/2026-08-18T03:58:43Z pbs-ct backup 4287958060 9201
+7 -5
View File
@@ -636,16 +636,19 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server`
| **R-333** | **Two disk-health questions the deploy raised and did NOT act on.** **(a) The 55/60 °C bands are SPINNING-DISK bands applied to NVMe.** They were adopted unchanged from the operator's Prometheus config so the two systems cannot disagree — a deliberate, stated decision — but **measured on demo-hp 2026-08-14 the healthy Toshiba KXG50PNV1T02 NVMe idles at 53 °C, two degrees below Figyelmeztetés and seven below Hiba**, and NVMe routinely exceeds 60 °C under load with no fault whatever. As it stands a healthy customer NVMe under sustained write can be reported as **Hiba** — the single worst outcome this feature can produce. **(b) The agent runs bare `smartctl -a -j` with no `-n standby`** (`felhom-agent/internal/storage/hostops.go:368`), so every poll WAKES a spun-down drive; going 6h → hourly multiplies that by six. demo-hp is all-flash so the cadence measurement could not reveal it, and it was recorded rather than acted on per the task's own instruction. Mitigating datum from the fixture: the failing drive logged only **3375 load cycles in 60505 hours** (~one per 18h), i.e. that duty cycle barely spins down at all | **READY (S each) — NEW 2026-08-14** | — | (a) split the temperature bands by device class, or drop them for NVMe and rely on `critical_warning`; (b) add `-n standby` to the agent's smartctl invocation (an agent change, so fold it into R-330's session) | Viktor decides (a); CC does (b) |
| **R-334** | **CLOSED 2026-08-18 — golden 0.216.0 baked, published and VOUCHED; CI green by run id.** ~~WAIVER + open item: controller v0.215.0 is released and deployed, and NO golden carries it.~~ Convicted by `golden_currency_gate.py` on the 2026-08-14 push: newest released controller **0.215.0**, newest golden bake **0.214.0** (`documentation/tests/golden-0.214.0-2026-08-12`). **A machine installed right now receives 0.214.0** — i.e. a brand-new box would ship WITHOUT the R-328 severity fix and would keep emailing nobody about a failing disk. The running fleet is unaffected (demo-hp guest 9201 is on 0.215.0 and healthy); this is purely the day-0 install path. **Not baked in this session deliberately:** the task scoped deployment to demo-hp only, and the second half of the fix — vouching the bake in the hub's day-0 artifact manifest — is **operator-password-gated, so CC cannot complete it**; a baked-but-unvouched golden is worse than none. **The push was made with `git push --no-verify` and it is stated here and in the session report**, per `.claude/rules/gates.md` — the gate has no waiver parser, so recording a waiver does not clear it. **STILL OPEN and now one version WIDER, 2026-08-18:** the newest released controller is **0.216.0** (v0.215.0's own follow-up fix, R-335) and the newest bake is still **0.214.0**, so a new install now misses *two* releases. Re-convicted on this date's documentation-only push, which was likewise made with `--no-verify`; the gate reads `felhom-controller/CHANGELOG.md` and `documentation/tests/golden-*`, **neither of which that session touched** — the conviction is inherited, not caused | **CLOSED 2026-08-18.** Baked from `RUNBOOK-manual-build.md` §4.0+§4.1 in the DooPlex drill VM and published: **`GOLDEN_VERSION=0.216.0`**, **`GOLDEN_SHA256=ac004dc90d8cefccc5448377892f9cff3a4c3e1e27d0e11129120e38ac31c34b`**, 656,970,239 bytes at `…/generic/felhom-golden/0.216.0/golden.tar.zst`. **The published bytes were verified, not just the script's print** — the artifact was downloaded back out of Gitea and hashed, and it matches. **Vouched by the operator, all THREE fields together**, confirmed by reading the hub's own store rather than the save: `artifact_golden_version=0.216.0`, `artifact_agent_version=0.129.0`, `artifact_min_agent=0.129.0` (2026-08-18 11:00:59–11:01:00), and the hub's recorded sha256 matches the downloaded artifact. The R-216 shape was checked on the machine: `MinAgent` 0.129.0 is **equal to**, not above, the newest **published** agent. **`golden_currency_gate.py` rc=0 and `repo_gates.py --fast` rc=0 — all nine gates — and CI is GREEN BY RUN ID: run **353**, `head_sha 7d81681d6`, conclusion `success`** (the two prior runs 351/352 on this same afternoon were red on exactly this row, which is the contrast). That push needed **no `--no-verify`** — the first of the day that did not. Evidence: `documentation/tests/golden-0.216.0-2026-08-18/`, report `REPORT-golden-0.216.0.md`. **Closed with the run id quoted deliberately**: this row was re-confirmed once and widened once, and closing it on a local green a third time would have left the same ambiguity | — | Bake a golden on **0.216.0** per `runbooks/RUNBOOK-manual-build.md` §4.1, then vouch it — a THREE-field change (`golden_version` + `agent_version` + `min_agent`). Until then every NEW install lacks the severity fix | CC bakes; **Viktor vouches** |
| **R-335** | **One physical disk was walked TWICE per run, and the second walk sustained it against itself.** Found on live hardware ~2h after the v0.215.0 deploy, **by noticing the release's own positive observable disagreed with its own persisted artefact**: the hourly check logged *"3 disk(s) evaluated"* while `disk-health-state.json` held **two** records. Cause: demo-hp's `c11-scratch` and `felhom-backup` are the same physical NVMe (`/dev/nvme0n1`) and resolve to the same `diskKey`. **Not cosmetic** — `RunDiskHealthCheck` writes a disk's new record before the next entry reads it, so the SECOND copy consumed the FIRST copy's write as its prior: the disk **sustained against itself and reached Hiba on a FIRST sighting**, defeating truth-table row 6 — the exact rule separating a one-hour benign excursion from a false critical — and would have emitted **two identical events** for one drive. **Latent, not active, on demo-hp** (all three entries healthy, zero counters), but any aliased disk developing a single pending sector would have gone straight to Hiba. **This is the shape standing rule 3 warns about: an absent alarm was not evidence — the two artefacts had to be read AGAINST each other** | **CLOSED — controller v0.216.0, 2026-08-14.** Each `diskKey` is evaluated once per run; both entries stay marked `seen` so neither looks like a disappeared disk, and the card still renders both storage rows (the dedup is about state and alerts, not display). Pinned by `TestDiskCheck_SameDiskTwiceIsEvaluatedOnce`; companion red-proof run and reverted — deleting the guard makes the first sighting emit `Kind:2` (Hiba-from-sectors) at 8 sectors | — | — | CC |
| **R-336** | **The offsite DR endpoint is polled about once per second, and that is what turned a slow leak into an outage.** ep0's PBS proxy served **~85,000 requests/day** — a flat **3,538/hour**, every hour, from two boxes: `74,445 GET /api2/json/admin/datastore` (`libwww-perl`, i.e. PVE's `pvestatd`) and `73,171 GET /admin/datastore/felhom-offsite/status` (`proxmox-backup-client`). Two pollers asking substantially the same question at the same rate. On 2026-08-18 this walked a connection leak in the proxy to its 1024-fd soft limit in **14 days**, wedging the offsite tier for 9½ hours (`audits/INCIDENT-ep0-pbs-fd-exhaustion-2026-08-18.md`). The `LimitNOFILE=65536` drop-in applied that morning raises the ceiling **but does not fix the leak** — it converts a fortnightly outage into a multi-year one, which is mitigation, not a fix. A DR endpoint that is written to weekly does not need to be asked about every second | **READY (M) — NEW 2026-08-18** | — | **CORRECTED 2026-08-18 (evening) — the easy lever named here does not exist.** This cell used to read *"PVE storage status is the prime suspect, and its interval is tunable"*. **The first half is right and the second half is false.** `pvestatd` stats EVERY configured storage on each 10-second cycle, and Proxmox staff have stated the interval is not designed to be configurable — so there is no knob to turn down. The only lever PVE actually offers is disabling the storage entry (`pvesm set <id> --disable 1`) around the backup window, and that is **substantially more than a tuning knob**: it collides with `felhom-agent/internal/pbsdr/manager.go`'s health model, where an inactive-but-existing entry drives the consume-the-one-time-secret recovery path. So the fix is a design question (does the hub still need a 15-minute fill reading at all, given R-339 now reports reachability separately?), not a config edit. **Doc-only correction — no agent code was changed.** The remaining step is unchanged: cut the poll rate by whatever means survives that question, then confirm the fd count between restarts stops climbing — the positive observable, per standing rule 3. **Baseline measured 2026-08-18, and the FIRST measurement published was WRONG.** The initial "~85/day, matching the ~73/day implied by the failure" came from a single 17-minute window whose delta was **one descriptor** — a sample of one cannot carry a daily rate, and the agreement that made it feel solid was coincidence. **Re-measured over two independent windows the same morning: 183/day (31 min) and 200/day (5.6 h)** — ~2.6x the published figure, putting the runway to the 65536 ceiling at **~357 days, not the ~2 years first claimed**. **And the named mechanism is the minority one:** across that window `CLOSE-WAIT` held flat at 1 while `ESTAB` grew 45→49 — *all* the growth was established connections, and at the wedge the split was 1011 ESTAB / 543 CLOSE-WAIT. **The fix must target connections the proxy never reaps, not just `CLOSE-WAIT` sockets.** The PBS 4.2.5-1 upgrade (2026-08-18) did NOT change the slope and was never expected to — see R-341 | CC |
| **R-336** | **The offsite DR endpoint is polled about once per second, and that is what turned a slow leak into an outage.** ep0's PBS proxy served **~85,000 requests/day** — a flat **3,538/hour**, every hour, from two boxes: `74,445 GET /api2/json/admin/datastore` (`libwww-perl`, i.e. PVE's `pvestatd`) and `73,171 GET /admin/datastore/felhom-offsite/status` (`proxmox-backup-client`). Two pollers asking substantially the same question at the same rate. On 2026-08-18 this walked a connection leak in the proxy to its 1024-fd soft limit in **14 days**, wedging the offsite tier for 9½ hours (`audits/INCIDENT-ep0-pbs-fd-exhaustion-2026-08-18.md`). The `LimitNOFILE=65536` drop-in applied that morning raises the ceiling **but does not fix the leak** — it converts a fortnightly outage into a multi-year one, which is mitigation, not a fix. A DR endpoint that is written to weekly does not need to be asked about every second | **READY (M) — NEW 2026-08-18** | — | **CORRECTED 2026-08-18 (evening) — the easy lever named here does not exist.** This cell used to read *"PVE storage status is the prime suspect, and its interval is tunable"*. **The first half is right and the second half is false.** `pvestatd` stats EVERY configured storage on each 10-second cycle, and Proxmox staff have stated the interval is not designed to be configurable — so there is no knob to turn down. The only lever PVE actually offers is disabling the storage entry (`pvesm set <id> --disable 1`) around the backup window, and that is **substantially more than a tuning knob**: it collides with `felhom-agent/internal/pbsdr/manager.go`'s health model, where an inactive-but-existing entry drives the consume-the-one-time-secret recovery path. So the fix is a design question (does the hub still need a 15-minute fill reading at all, given R-339 now reports reachability separately?), not a config edit. **Doc-only correction — no agent code was changed.** The remaining step is unchanged: cut the poll rate by whatever means survives that question, then confirm the fd count between restarts stops climbing — the positive observable, per standing rule 3. **Baseline measured 2026-08-18, and the FIRST measurement published was WRONG.** The initial "~85/day, matching the ~73/day implied by the failure" came from a single 17-minute window whose delta was **one descriptor** — a sample of one cannot carry a daily rate, and the agreement that made it feel solid was coincidence. **Re-measured over two independent windows the same morning: 183/day (31 min) and 200/day (5.6 h)** — ~2.6x the published figure, putting the runway to the 65536 ceiling at **~357 days, not the ~2 years first claimed**. **And the named mechanism is the minority one:** across that window `CLOSE-WAIT` held flat at 1 while `ESTAB` grew 45→49 — *all* the growth was established connections, and at the wedge the split was 1011 ESTAB / 543 CLOSE-WAIT. **The fix must target connections the proxy never reaps, not just `CLOSE-WAIT` sockets.** The PBS 4.2.5-1 upgrade (2026-08-18) did NOT change the slope and was never expected to — see R-341 **SPIKE 2026-08-20 — THE PREMISE OF THIS ROW DOES NOT SURVIVE MEASUREMENT, and that is a change in what the row IS, not new evidence on it.** `audits/SPIKE-ep0-established-connections-2026-08-20.md`. **The leak is OURS, and the poll rate is not what feeds it.** Every one of the 388 leaked descriptors is an ESTABLISHED connection held open by **`felhom-agent`** on the boxes — 194 on each, `ss -tnp` naming a single PID per box, and **zero** held by `pvestatd` or `proxmox-backup-client`. Confirmed independently from ep0's access log over the same 46.18 h window: `libwww-perl` (pvestatd) **81,192 requests -> 0 descriptors**, `proxmox-backup-client` **80,061 requests -> 0 descriptors**, `Go-http-client/1.1` (the agent) **387 `/snapshots` calls -> 388 sockets — one per call, within one**. So **162,404 requests, 99.5% of the traffic, produce 0% of the leak.** **Mechanism, named from source:** `felhom-agent/internal/pbs/client.go:56-60` builds `&http.Transport{TLSClientConfig: tlsCfg}` — a composite literal, so `IdleConnTimeout` is the zero value = **no limit** (`http.DefaultTransport` sets 90 s; a literal does not inherit it) — and `cmd/felhom-agent/main.go:1486` (`pbsTargetsFromPVE`) builds **a fresh client every cycle**, as its own doc comment states. Each cycle therefore strands one idle keep-alive connection in a transport nothing ever closes; `CloseIdleConnections`/`IdleConnTimeout`/`MaxIdleConns` appear **nowhere** in the agent repo. Cadences reconcile without fitting: 900 s hub poll (184.7 cycles) + 6 h `DefaultVerifyCadence` (7.7 cycles) = 192.4 predicted vs **194 observed per box**. **CONSEQUENCE — RE-RANK.** The remaining step recorded above ("cut the poll rate, then confirm the fd count stops climbing") **would have produced a null result and read as a failed fix.** Cutting the Proxmox poll rate removes ~99.5% of ep0's reqLine truncated
| **R-337** | **`/backup/status` lagged a completed backup by minutes on one box and not the other — and it RESOLVED ITSELF, which is why this is WATCHING and not a defect.** During the R-336 recovery on 2026-08-18, `demo-hp`'s snapshot landed on ep0 at **03:58:43Z** (complete manifest; the host's own task index says `OK`) — yet `GET /backup/status` was **still serving the superseded 03:27:00Z failure at ~04:03Z**, four-plus minutes later. `demo-felhom` showed its new result within ~40 s of completion. **The lag cleared on its own:** demo-hp's 04:07:35Z host report carries `felhom-pbs success=true, 4.29 GB`, and the hub is green for both boxes. **The first draft of this row claimed the success was "still reported as failed" — that was written before the next report arrived and it was wrong; the corrected claim is a several-minute skew between the two boxes, not a stuck value.** It is recorded because a status field that can trail its own artifact by minutes will, during an incident, be read as a second failure — this session nearly did — and because the asymmetry between the two boxes is unexplained | **WATCHING — NEW 2026-08-18** | another observation, ideally during an incident rather than constructed | **Do not open a fix on this as written.** First establish the intended refresh path for `/backup/status` after an out-of-schedule run; only if the skew is not simply collection cadence is there anything to pin. If it is cadence, close this row and say so | CC |
| **R-338** | **`demo-hp` is not on the R-50 island at all, and `operations/nodes.md` states that it is.** The page records both fleet boxes as island-migrated 2026-07-25. True of `felhom-pve`; **false of `demo-hp`**, whose `agent.json` has `listen_addr: 192.168.0.87:8443` — the customer LAN address — and **no `island_bridge`/`island_guest_addr` keys at all**, whose guest 9201 has `net0` only (no `eth1`), and whose `vmbr9` exists with **zero members**. The controller's `controller.yaml` points at the LAN address, so the box works; this is inventory drift, not breakage. **Two costs.** A session trusting the page addresses the wrong endpoint — that happened on 2026-08-18 and the resulting timeout was briefly read as a fault. And the agent's local API is **bound to the customer LAN on this box** rather than to a point-to-point island, which is the exposure R-50 was built to remove — so a documented security property is claimed for a box that does not have it | **READY (S) — NEW 2026-08-18** | — | Decide which is true: migrate `demo-hp` to the island, or correct `nodes.md`. Leaving both is the one option that keeps the doc lying | Viktor decides; CC executes |
| **R-341** | **Does the fd slope change after the PBS 4.2.5 upgrade? — two dated checks, and the answer is expected to be NO.** ep0 was upgraded 4.2.2-1 → 4.2.5-1 on 2026-08-18 09:51Z on the operator's ruling, **for rehearsal value, not as a fix**: the full changelog range was read (128 lines, all three entries) and swept for connection-handling vocabulary, and it contains **no mechanism** by which descriptor reaping would change — the single keyword hit was `S3 … honor the node's proxy settings`, HTTP-proxy config for S3, not the PBS proxy daemon. **The 32-minute post-upgrade window is indistinguishable from the before window** (+5 fd/1919 s = 225/day vs +4 fd/1885 s = 183/day; the two differ by ONE descriptor and Poisson uncertainty on such counts is ±2, so both are consistent with one unchanged rate — the higher after-figure is noise, not a regression). **Thirty minutes cannot settle it in either direction and this row exists so nobody pretends it did.** New `t0` = **fd 17 at 2026-08-18 09:51:22Z, proxy PID 551655**; before-rate to beat = **183–200/day**. **Interpretation fixed in advance** (`evidence-ep0-pbs-upgrade-2026-08-18/stop1-ruling.txt`, written before any numbers existed): unchanged = EXPECTED, not a failed upgrade; changed = a SURPRISE needing explanation, not a confirmation | **WATCHING — NEW 2026-08-18** | elapsed time only | **Two dated checks, both CC:** **+24 h — 2026-08-19 ~10:00Z** and **+7 d — 2026-08-25 ~10:00Z**. Command (the incident's own positive observable): `ssh root@<ep0> 'PID=$(systemctl show proxmox-backup-proxy -p MainPID --value); ls /proc/$PID/fd \| wc -l; ss -lnt "( sport = :8007 )"; ss -tn state all "( sport = :8007 )" \| awk "NR>1{print \$1}" \| sort \| uniq -c'`. **Record the ESTAB/CLOSE-WAIT split, not just the total** — the split is what says which leak it is. If PID ≠ 551655 the window is void: something restarted the proxy and the count began again | CC on both dates |
| **R-341** | **Does the fd slope change after the PBS 4.2.5 upgrade? — two dated checks, and the answer is expected to be NO.** ep0 was upgraded 4.2.2-1 → 4.2.5-1 on 2026-08-18 09:51Z on the operator's ruling, **for rehearsal value, not as a fix**: the full changelog range was read (128 lines, all three entries) and swept for connection-handling vocabulary, and it contains **no mechanism** by which descriptor reaping would change — the single keyword hit was `S3 … honor the node's proxy settings`, HTTP-proxy config for S3, not the PBS proxy daemon. **The 32-minute post-upgrade window is indistinguishable from the before window** (+5 fd/1919 s = 225/day vs +4 fd/1885 s = 183/day; the two differ by ONE descriptor and Poisson uncertainty on such counts is ±2, so both are consistent with one unchanged rate — the higher after-figure is noise, not a regression). **Thirty minutes cannot settle it in either direction and this row exists so nobody pretends it did.** New `t0` = **fd 17 at 2026-08-18 09:51:22Z, proxy PID 551655**; before-rate to beat = **183–200/day**. **Interpretation fixed in advance** (`evidence-ep0-pbs-upgrade-2026-08-18/stop1-ruling.txt`, written before any numbers existed): unchanged = EXPECTED, not a failed upgrade; changed = a SURPRISE needing explanation, not a confirmation | **WATCHING — FIRST CHECK TAKEN 2026-08-20: the rate is UNCHANGED, exactly as pre-registered. Second check still due 2026-08-25** | elapsed time only | **Two dated checks, both CC:** **+24 h — 2026-08-19 ~10:00Z** and **+7 d — 2026-08-25 ~10:00Z**. Command (the incident's own positive observable): `ssh root@<ep0> 'PID=$(systemctl show proxmox-backup-proxy -p MainPID --value); ls /proc/$PID/fd \| wc -l; ss -lnt "( sport = :8007 )"; ss -tn state all "( sport = :8007 )" \| awk "NR>1{print \$1}" \| sort \| uniq -c'`. **Record the ESTAB/CLOSE-WAIT split, not just the total** — the split is what says which leak it is. If PID ≠ 551655 the window is void: something restarted the proxy and the count began again **FIRST CHECK — TAKEN 2026-08-20 08:02:13Z, and it was taken at +46.2 h, NOT +24 h.** No session ran between 18 and 20 August, so the 2026-08-19 date passed as pure elapsed time; the delay is stated here rather than backfilled. **The longer window is a BETTER measurement, not a degraded one** — the leaked count is **388** against the 4 and 5 descriptors the original answer rested on, roughly **97x**, so the uncertainty falls from about +/-50% to about +/-5%. **Precondition PASSED:** PID still **551655**, `ps -o lstart` 2026-08-18 09:51:04, `NRestarts=0` on both units. **Result:** fd **17 -> 405** over **166,251 s** = **201.6 fd/day**, Poisson +/-10.2/day (1s), 2s band **181.2-222.1**. Pre-registered range was **370-450** (confirm-band 330-490); **observed 388**, near the centre. **VERDICT: unchanged — the EXPECTED result, and not a failed upgrade.** **Composition:** ESTAB 0 -> 388, **CLOSE-WAIT 0 — absent from the histogram entirely**, so 100% of the growth is established connections and `CLOSE-WAIT` is not merely the minority half. Runway from fd 405 at 201.6/day to the 65536 ceiling: **~323 days (~2027-07-09)**. Full working: `audits/SPIKE-ep0-established-connections-2026-08-20.md` + `evidence-ep0-established-connections-2026-08-20/step1-slope-computation.txt`. **MEASUREMENT TRAP for the 2026-08-25 check, found on this run — see R-346:** the anchor must be `ps -o lstart= -p $MainPID`, NOT `systemctl show -p ActiveEnterTimestamp`, which reads 03:54:54Z for this generation (the upgrade re-exec'd the proxy; systemd never saw a stop, `NRestarts` is still 0) and would put the rate ~15% low. **PERTURBATION NOTE for the 2026-08-25 check:** this spike's Phase C quietens `pvestatd` on demo-hp for one overnight window inside that 7-day interval. Quantified so nobody reads the shortfall as a change in the leak: even if the leak were fully proportional to the request rate (which Parts 1-2 predict it is NOT), a 10 h half-rate window inside 168 h shifts the 7-day slope by **~3%**, far inside the +/-10% Poisson band on a ~1,400-descriptor count. **The +7 d reading remains usable.** | CC on both dates |
| **R-342** | **The ep0 snapshot covers less than it looks like it covers, and the next person will assume otherwise.** Quoting `audits/evidence-ep0-pbs-upgrade-2026-08-18/stop2-snapshot.txt` verbatim: *"covers — the 38 GB system disk /dev/sda (root), i.e. the PBS packages, unit files, /etc/systemd drop-ins, nftables and wg config. DOES NOT — /mnt/pbs-datastore. That is /dev/sdb, a separate 100 GB VOLUME, and Hetzner server snapshots do not include attached volumes. The backup data is therefore NOT protected by this snapshot."* Snapshot **421440873** (`felhom-hetzner-20260818`, 15.06 GB, Available) was taken as the rollback for the 4.2.2→4.2.5 PBS upgrade. **Rolling it back restores software state, not the datastore.** That was *acceptable for that change* — a package install writes no datastore content — and the file says so. **The problem is what happens next:** this fact lives in an evidence file nobody will open again, and a snapshot named as "the rollback" reads as protecting everything on the box. ep0 holds the only off-premises copy of a real customer's data | **READY (S) — NEW 2026-08-18** | — | **Decide the safeguard for any future ep0 procedure that could touch `/mnt/pbs-datastore` — it does not exist and has not been designed.** Candidates: a Hetzner **Volume** snapshot (a different object from the server snapshot), a PBS-level sync to a second location, or an explicit written acceptance that the datastore is unprotected for the duration. **Nothing may be added to a runbook implying a safeguard exists until one does** | **Viktor decides; CC executes** — a risk-to-customer-data question |
| **R-343** | **The managed controller floor was raised 0.214.0 → 0.216.0 — and it was NOT the no-op it was expected to be: it moved a live box nine seconds later.** Raised by the operator 2026-08-18 **12:36:58Z**, in a **separate save after** the artifact vouch. **Read back from the store, not the form** (`hub_settings.min_controller_version`, `GetGlobalMinControllerVersion` — `hub/internal/store/store.go:1751`): `min_controller_version = 0.216.0`, updated 12:36:58. **WHY IT HAD BEEN BEHIND — this was NOT drift, and describing it as "two releases behind" without this context reads as a defect it was not.** `publish-train-rules.md` rule 1 is *manifest before floor*, and rule 2 requires the floor field to be filled **LAST, in a separate save**, because the DB row overrides the env floor and **acts immediately on the next report cycle**. That rule was earned: on the 2026-07-11 publish train the floor was saved together with the manifest, acted at once, and pushed controller 0.113.0 onto Peti's box **~9 minutes ahead of agent 0.81.0** — the exact forbidden skew, benign only because that box had no NAS shares. R-120's row records the same deliberate choice (*"Floor untouched per publish-train rule 2"*). So the floor sitting at 0.214.0 was **policy being followed**, not neglect. **What it was functionally while it sat there:** not a live problem — every reporting box was at or above it — but a **safety net set two versions low**. The floor is what drags a box forward if it ever falls behind (restored from an old backup, reinstalled, long offline), and at 0.214.0 it would have pulled such a box only to two versions back, missing R-328's severity fix and R-335's follow-up. **THE MEASURED BLAST RADIUS — five reads, and the third and fifth are the findings.** **(1) Floor:** `0.216.0` @ 12:36:58Z, from the store. **(2) Per-customer overrides:** **zero** — all five `customer_configs` rows carry an empty `min_controller_version`, so nothing hides behind a lower override and the global applies to everyone. **(3) Every box's controller version:** `demo-felhom` **0.216.0**, `demo-hp` **0.216.0** — both AT the floor **now**; `drill-r50` **0.213.0** (status `blocked`, last report 2026-08-12, powered off/reverted) and `peti-felhom` **0.115.0** (host row DELETED) are **below** it but are **not reporting boxes**. **(4) Directives/holds:** no `managed floor HELD` line exists; the hub logged `[INFO] Global controller-version floor set to "0.216.0"`. **(5) THE FINDING — a controller DID auto-update after the raise.** `demo-felhom` had been on **0.214.0** since 2026-08-12 16:44 and the hub recorded `controller_updated — Controller frissítve: 0.214.0 → 0.216.0` at **12:37:07Z**, then `controller_started (0.216.0)` at 12:37:12Z — **nine seconds after the raise**, exactly the immediate action rule 2 documents. `demo-hp` was already on 0.216.0 (hand-deployed 2026-08-14 08:31) and did not move. **No error, warning or critical event followed** — the update completed and the controller came back up. **So the change was real, not inert: "every reporting box is at or above the floor" is true BECAUSE of the raise, not independently of it.** **Why it is safe by construction**, cited rather than asserted: `ResolveManagedFloor` (`hub/internal/store/store.go:2068`) sets `Held` and clears the floor entirely when `Floor > GoldenVersion` (the R-216 shape) — floor 0.216.0 **equals** golden 0.216.0, so that guard does not trip — and holds per-box when the box's agent is below the manifest's `MinAgent`, or unknown, or unparseable; both boxes report agent 0.129.0 against `MinAgent` 0.129.0, so the floor was served rather than held. That second guard remains armed for any box reporting with an old agent. **No ISO rebuild is required:** the golden is fetched at first boot from the hub's manifest, which is 0.216.0 — at the floor, not below it — and rule 5's `assert_golden_ge_floor` is a **build-time** gate for *future* builds (`scripts/iso/build-felhom-iso.sh:77`, called at **:267**) which **fails open with a warning** when its inputs are absent (`:78-82`: `if [[ -z "$golden" \| \| -z "$floor" ]]` → `log_warn "… UNENFORCED …"` → `return 0`), both confirmed in the script. **`peti-felhom` was NOT contacted and needs no contact** — from the PETI row: its host row was deleted 2026-07-15 and *"a report from a deleted host 401s and is not persisted"*, so it cannot receive a floor directive at all and the raise cannot reach it | **OPEN — NEW 2026-08-18.** Deliberately NOT closed: the task's closing condition was *all five reads clean, no directive served*, and read 5 shows a live box updated. It went cleanly and is the floor working as designed — but a change recorded as a no-op when it moved a customer box is exactly the kind of record that misleads later | — | **Confirm the 0.216.0 update on `demo-felhom` is healthy in normal operation** (it reported and restarted clean, but it has not yet run a full bacLine truncated
| **R-344** | **`felhom-agent` leaks one TCP connection to PBS per poll cycle, forever, on both sides — and it is the whole of the ep0 descriptor leak.** Found by the 2026-08-20 connections spike (`audits/SPIKE-ep0-established-connections-2026-08-20.md`), which was looking for a Proxmox poll-rate problem and found ours instead. **Two defects compounding.** **(a)** `internal/pbs/client.go:56-60` builds `&http.Transport{TLSClientConfig: tlsCfg}` as a composite literal, so **`IdleConnTimeout` is the zero value = no limit** — `http.DefaultTransport` sets 90 s and a literal does not inherit it. **(b)** `cmd/felhom-agent/main.go:1486` (`pbsTargetsFromPVE`) builds **a fresh `pbs.Client` every cycle**, as its own doc comment states, so each cycle strands one idle keep-alive connection in a transport that is then unreachable — and an unreachable `http.Transport` does **not** close its connections; the `persistConn` read-loop goroutine keeps the socket alive. `CloseIdleConnections` / `IdleConnTimeout` / `MaxIdleConns` appear **nowhere in the repo** (grep: no matches). **Measured live, both sides, twice:** ep0 held 388 ESTAB (194 from each box) at 08:02:39Z and 392 at 08:33:42Z; the boxes held 194+194 and 196+196 at the same instants, and the four new sockets carried **the same four source ports** on both sides. **Zero sockets closed in 31 minutes**, and all carried keepalive timers with `retrans=0` — mutually held live idle connections, not half-open ones. One socket per agent `/snapshots` call (387 calls vs 388 sockets). Rate **201.6/day** across the fleet; ep0 reaches its 65536 ceiling in **~323 days**. **It is bilateral and the box side is the under-watched half:** each agent holds **196 of its 208** descriptors in these sockets. Its limit is 524287, so the boxes are in no danger *today* — which is why this stayed invisible, not why it is harmless. | **READY (S) — NEW 2026-08-20** | none — it is a self-contained change in `felhom-agent` | **No fix is proposed here: the spike-first gate forbids it and the spec is a separate task.** What the spec must settle, and none of it is decided: whether the per-cycle client construction is the thing to remove or the transport is the thing to share; what `IdleConnTimeout` should be given the 900 s poll and the 6 h verify cadence; whether `internal/hub/client.go:53` and `internal/proxmox/client.go:69` — **the same composite-literal pattern, built once at start-up so not leaking by this route today** — should be changed in the same pass or left alone with a test pinning why. **The proof obligation is the fd count, not the diff:** per standing rule 3 the positive observable is ep0's ESTAB count going FLAT between proxy restarts, measured over a window long enough to matter — a green test suite proves nothing here, and a 30-minute window proves nothing here either (that error is already recorded twice in R-336 and R-341). | CC |
| **R-345** | **`hub/Makefile` tags and pushes `:latest`, which the project's own rules forbid in two places.** Lines 21-22 of `docker-push`: `docker tag $(IMAGE):$(VERSION) $(IMAGE):latest` then `docker push $(IMAGE):latest`. `.claude/rules/hub.md:35` says *"Pin explicit versions, never `:latest`"* and `.claude/rules/manifests.md:15` repeats it. Verified present on `848368ec3`. Small, and the deployed manifests do pin a version, so nothing is currently broken by it — but a documented command that performs the prohibited action is a trap for whoever next reads the Makefile as the reference for how to publish, and a floating `:latest` on the registry is exactly the thing an emergency `kubectl set image` reaches for. Noticed while running the 2026-08-20 connections spike; unfiled until now. | **READY (XS) — NEW 2026-08-20** | — | Delete the two lines, or keep them behind an explicit opt-in target that says in a comment why it exists. Check whether a stale `:latest` tag already sits on `gitea.dooplex.hu/admin/felhom-hub` before deciding — an existing floating tag is the more dangerous half. | CC |
| **R-346** | **`ActiveEnterTimestamp` answers a different question than the one a slope measurement asks, and on ep0 right now it is wrong by 5 h 56 m.** Found while taking R-341's first dated check. ep0's `proxmox-backup-proxy` has `MainPID=551655` started **2026-08-18 09:51:04Z** (`ps -o lstart=`), but `systemctl show -p ActiveEnterTimestamp` reads **03:54:54Z** and `NRestarts` reads **0** — because the 4.2.5-1 upgrade **re-exec'd** the daemon rather than restarting the unit, so systemd never observed a stop. Anyone anchoring "when did this proxy generation start" on `ActiveEnterTimestamp` would divide 388 descriptors by 52.1 h instead of 46.2 h and report **178/day instead of 201.6/day — ~12% low** — while every field consulted looks healthy and consistent. **This is the workspace rule's own case, in a new place:** ask of a timestamp *what exactly must have happened for this to be set?* Here the answer is "the unit entered active", which is not "this process started". `NRestarts=0` is the tell, and it reads like reassurance. | **READY (XS) — NEW 2026-08-20** | — | R-341's command already uses `ps -o lstart= -p $MainPID` and is correct; the risk is a future reader "improving" it to a systemd property. Add the reason as a comment beside that command in the R-341 row (done), and check whether any other slope or uptime check in the repo or in `scripts/felhom-tenantsync.sh` anchors on a systemd timestamp where it means a process start. | CC |
| **R-339** | **The hub was SILENT when it lost sight of the off-site stores — and a 9 h 37 m outage proved it.** Both box checkers (`OffsiteBoxChecker` over the Hetzner API, `PBSDRBoxChecker` over ep0's `usage` op) held their last snapshot and returned quietly on a failed fetch. That is **correct for a fill signal** — a missing reading must never be mistaken for 0%, which is why degraded data drives no band transition — but the consequence was that a completely dead off-site endpoint and a healthy one were **indistinguishable on the operator channel**. During the 2026-08-18 ep0 incident the hub said nothing for the entire outage; the only mails came from the boxes' own backup failures, and **only because the WEEKLY offsite run happened to fall inside the window**. Two days earlier, nothing would have fired at all | **SHIPPED — hub v0.106.0, 2026-08-18.** Reachability is now a second, independent signal: consecutive failed fetch windows counted per checker, `pbsdr_box_unreachable` / `offsite_box_unreachable` (severity `warning`) past a default **3 windows (≈30–45 min)**, with paired `*_recovered` all-clears wired into `recoveredPairedDownTypes` — necessary because both recoveries are severity `info` and `severityNotifies` drops `info`. Threshold tunable via `alerting.box_unreachable_windows`. **The fill logic is untouched**: no threshold, throttle, band or escalate-once behaviour changed. Evidence: `internal/monitor/box_reachability_test.go` (Scenarios A–F) + `internal/notify/dispatcher_box_reachability_test.go` (the cross-package wiring, asserting an actual operator mail), plus three companion red-proofs each seen failing with a message naming the right cause | — | **PROVEN-LIVE still owed.** No real or constructed outage has exercised the emit path end to end, and one cannot be manufactured without making ep0 or the Hetzner API unreachable — ep0 is Tier 2 protected, so that is forbidden. The honest route is a constructed outage against a scratch hub instance with the tenantsync client pointed at a blackholed address. **Do not close this row on the unit tests** | CC |
| **R-340** | **The new reachability check does not touch the surface that actually failed.** R-339 reports when the hub cannot READ ep0 — but the read it performs is the `usage` op, which is `proxmox-backup-manager` plus `df` over SSH, and therefore rides the **local API daemon**. The 2026-08-18 incident explicitly CLEARED that daemon: `proxmox-backup.service` was healthy throughout, and it was the **HTTPS proxy on 8007** that was wedged with a full accept queue. **So R-339's check would have returned green for all 9 h 37 m of that outage.** It closes the case where ep0 is unreachable *as a host*; it does not close the case that actually happened. This is not a defect in R-339 — it is the honest boundary of what it watches, recorded so a future reader does not mistake a green box gauge for a working off-site tier | **READY (M) — NEW 2026-08-18** | a tenantsync endpoint-script version bump (the op is added on ep0, so it needs the same version-gated rollout `ErrUsageUnsupported` already models) | Add a **health op** to `scripts/felhom-tenantsync.sh` that probes `https://127.0.0.1:8007/` on ep0 and reports the proxy's fd count and listen-queue depth, then surface it as a third signal. **Overlaps the connections spike (R-336's remaining half): both want the same observations from ep0, so whichever runs SECOND must reuse the first's evidence rather than re-measuring a protected machine twice** | CC |
| **R-340** | **The new reachability check does not touch the surface that actually failed.** R-339 reports when the hub cannot READ ep0 — but the read it performs is the `usage` op, which is `proxmox-backup-manager` plus `df` over SSH, and therefore rides the **local API daemon**. The 2026-08-18 incident explicitly CLEARED that daemon: `proxmox-backup.service` was healthy throughout, and it was the **HTTPS proxy on 8007** that was wedged with a full accept queue. **So R-339's check would have returned green for all 9 h 37 m of that outage.** It closes the case where ep0 is unreachable *as a host*; it does not close the case that actually happened. This is not a defect in R-339 — it is the honest boundary of what it watches, recorded so a future reader does not mistake a green box gauge for a working off-site tier | **READY (M) — NEW 2026-08-18** | a tenantsync endpoint-script version bump (the op is added on ep0, so it needs the same version-gated rollout `ErrUsageUnsupported` already models) | Add a **health op** to `scripts/felhom-tenantsync.sh` that probes `https://127.0.0.1:8007/` on ep0 and reports the proxy's fd count and listen-queue depth, then surface it as a third signal. **Overlaps the connections spike (R-336's remaining half): both want the same observations from ep0, so whichever runs SECOND must reuse the first's evidence rather than re-measuring a protected machine twice** **REUSE, per this row's own instruction — the connections spike ran FIRST (2026-08-20) and already produced most of what the health op wants; do not re-measure a protected machine a third time.** Available in `audits/evidence-ep0-established-connections-2026-08-20/`: the proxy **fd count** and its type breakdown (`lsof` + `/proc/<pid>/fd`), the **listen-queue depth** (`ss -lnt` — `Recv-Q 0`, `Send-Q 1024`), the **ESTAB/CLOSE-WAIT split**, the **per-peer** connection histogram, a **31-minute persistence diff** of full 4-tuples, and a **46.18 h** slope with Poisson bounds. What the health op would still add beyond these: a loopback `GET https://127.0.0.1:8007/` probe — the observation that distinguished "process problem" from "network problem" on 2026-08-18 and the one thing this spike did NOT take, because it is the surface R-339 cannot see. **And this spike sharpens what the op should report:** a rising **ESTAB** count is the live signal (CLOSE-WAIT was **0**, not merely flat), and per **R-344** the fd ceiling that matters may be the **agent's**, not only ep0's. | CC |
<!-- DUE-CHECKS-BEGIN — machine-readable. Parsed by scripts/due_checks_gate.py.
One row per dated check. The R-number must have a row above. Dates are UTC.
@@ -655,6 +658,5 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server`
the R-row. Duplicating them here would create the second source this design avoids. -->
| item | due (UTC) | what to measure |
|---|---|---|
| R-341 | 2026-08-19 | ep0 proxy fd count + ESTAB/CLOSE-WAIT split; PID must still be 551655 |
| R-341 | 2026-08-25 | same, +7 d |
| R-341 | 2026-08-25 | same, +7 d. Anchor the elapsed time on `ps -o lstart= -p $MainPID`, NOT `ActiveEnterTimestamp` (R-346). A Phase-C perturbation sits inside this interval and is quantified in the R-341 row |
<!-- DUE-CHECKS-END -->