felhom.eu/REPORT.md

REPORT — RUNBOOK ep0: read the PBS changelog, then decide whether to upgrade (2026-08-18, midday)

Outcome: changelog read → no connection-handling fix in the range; operator ruled to upgrade anyway for rehearsal value; upgraded 4.2.2-1 → 4.2.5-1 cleanly; the fd slope did not change, which is the predicted result. Two dated checks filed as R-341.

Both STOPs cleared by the operator. No code changed in any repo; documentation/ only. Evidence: documentation/audits/evidence-ep0-pbs-upgrade-2026-08-18/.

---

1. Baselines re-confirmed on the machine

Not from the audit file — from dpkg -l and apt-cache policy, per the runbook.

Installed	proxmox-backup-server / -client 4.2.2-1
Candidate	4.2.5-1 (4.2.3-1, 4.2.4-1 also available)
Proxy PID / started	542065, 03:54:53Z, unrestarted since the incident
Effective open files	65536 / 65536 — the morning's drop-in in force
Recv-Q / loopback	0 / 200 in 11 ms
Datastore	3.7 G of 98 G, 4%
felhom.eu main @ start	435e044 — matches the runbook's stated baseline

2. The before-slope — and a correction I owe the morning's report

Two independent windows on the same proxy generation:

window	from → to	delta	rate
31 min	09:18:21Z fd=62 → 09:49:46Z fd=66	+4	183/day
5.64 h	04:11:36Z fd=19 → 09:49:46Z fd=66	+47	200/day

This morning's incident note said ~85/day and "≈2 years of runway". Both were wrong. They were extrapolated from a single 17-minute window whose delta was one descriptor — a sample of one cannot carry a daily rate, and the agreement with the historical ~73/day that made it feel solid was coincidence. The real rate is ~185–200/day, ~2.6× what I published, and the runway is ~357 days, not two years. Corrected in the incident document and in R-336 rather than left standing.

The mechanism I named was also the minority one. CLOSE-WAIT held flat at 1 across the window while ESTAB grew 45 → 49 — all the growth was established connections. At the wedge the split was 1011 ESTAB / 543 CLOSE-WAIT, so ESTAB dominated there too. R-336's fix must target connections the proxy never reaps, not just CLOSE-WAIT sockets.

3. The changelog, verbatim — the run's primary deliverable

All three entries between 4.2.2-1 and 4.2.5-1 read in full (128 lines), then swept for connection|file descriptor|fd|accept(|close_wait|keep-alive|socket|EMFILE|nofile|leak|proxy|listen| backlog|hyper|tokio.

Exactly one hit, and it is a false positive:

* S3: config: allow editing the use-node-config flag that controls whether requests S3 endpoints honor the node's proxy settings or not

HTTP-proxy configuration for S3 requests — not the proxmox-backup-proxy daemon.

What the range does contain. 4.2.5-1 — a security release hardening client-supplied manifests:

* backup: harden the handling of client supplied backup manifests: - only accept archive names that are plain file names carrying a server side type extension. A crafted name in a manifest could previously make a sync job read or write outside of the snapshot directory, running as the unprivileged 'backup' user. - keep an uploaded manifest in memory and only persist it on backup finish, checking that every archive it lists was really uploaded during that session and that the checksums match

plus a sync/push chunk-reuse fix and a subscription-key architecture check. 4.2.4-1 — S3 rate limits, a file-locking user-lookup cache, the new proxmox-enterprise-support-keyring dependency, docs. 4.2.3-1 — UI/journal work, an LDAP search-filter escape, tape and timezone fixes.

Nothing addresses descriptor lifetime or connection reaping. My recommendation was: do not upgrade for this reason.

4. STOP 1 — the ruling

Operator (Viktor) ruled: upgrade anyway, for rehearsal value — "see how that works for us, we need practice with that too". Legitimate and recorded as such: this was a practice run of the upgrade procedure on a Tier-2 protected machine, not a fix for the leak.

The interpretation was fixed in writing before any numbers existed (stop1-ruling.txt): unchanged slope = expected, not a failed upgrade; changed slope = a surprise needing explanation, not a confirmation. That file was written at the ruling, not afterwards, so neither outcome could be rationalised into a success.

5. STOP 2 — the snapshot, and what it does not cover

Snapshot 421440873 felhom-hetzner-20260818, 15.06 GB, status Available (complete, not merely started), server #147604682, project 15217960.

It covers /dev/sda only. /mnt/pbs-datastore is /dev/sdb, a separate 100 GB Volume, and Hetzner server snapshots exclude attached volumes — so this is a rollback for the software state (packages, unit files, the LimitNOFILE drop-ins, nftables, wg) and not a backup of the backup data. Fine for a package install that writes no datastore content; it must not be remembered as datastore protection. Taken on a running server, deliberately: powering off ep0 to guard a userspace package install would take the only off-premises copy offline.

6. The upgrade and its verification

Simulated first (-s): 0 to remove, so the abort condition never triggered. Then apt-get install --only-upgrade -y proxmox-backup-server, 09:51:00→09:51:06Z, exit 0. Upgraded server/client/docs to 4.2.5-1 plus one new dependency, proxmox-enterprise-support-keyring 1.1 — which the 4.2.4-1 changelog had declared, a small real consistency check between what I read and what apt did.

check	result
installed	server / client / docs 4.2.5-1
daemons	proxmox-backup-proxy active running, proxmox-backup active running
proxy restarted	542065 → 551655 @ 09:51:04
effective open files	65536 / 65536 — survived the new package
drop-ins on disk	both present, unmodified
Recv-Q	0
loopback	200 in 12 ms
felhom-pve over tunnel	200 in 0.103 s, felhom-pbs active
demo-hp over tunnel	200 in 0.096 s, felhom-pbs active
hub gauge, post-upgrade	11:59:31 [INFO] PBS-DR box refreshed: 3.7% full (3.7 GB of 97.9 GB)

proxmox-backup-manager version now reads 4.2.5-1 running version: 4.2.5. This independently settles the morning's confusion: that string was never reporting a stale daemon, and now that installed and running genuinely match, both halves agree.

One false alarm, mine. I queried systemctl is-active proxmox-backup-api and got inactive. That unit does not exist — systemctl cat returns "No files found for proxmox-backup-api.service". The real pair is proxmox-backup-proxy.service ("API Proxy Server") and proxmox-backup.service ("API Server"), both active. A bad query, not a fault — recorded because for as long as it took to check, it looked exactly like one.

No backup, restore or verify was triggered to "prove" the endpoint, per the runbook: the reads above answer it without mutating a protected datastore.

7. The after-slope — unchanged, as predicted

	window	delta	rate
before (PID 542065)	09:18:21Z fd=62 → 09:49:46Z fd=66	+4 / 1885 s	183/day
after (PID 551655)	09:51:22Z fd=17 → 10:23:21Z fd=22	+5 / 1919 s	225/day

These are not distinguishable. The windows differ by one descriptor; Poisson uncertainty on n=4 is ±2 and on n=5 is ±2.2, so both are consistent with a single unchanged rate. The after-figure being numerically higher is noise, not a regression — and certainly not an improvement. Composition repeats the pattern: ESTAB 0 → 5, CLOSE-WAIT 0 → 1.

Thirty minutes cannot settle this in either direction, and nothing here claims it does.

8. Register

• R-341 filed, WATCHING — the two dated checks: +24 h (2026-08-19 ~10:00Z) and +7 d (2026-08-25 ~10:00Z), against new t0 fd=17 @ 09:51:22Z, PID 551655, with the exact command and the instruction to record the ESTAB/CLOSE-WAIT split, not just the total — the split is what identifies which leak it is. If the PID has changed, the window is void.
• R-336 updated and STILL OPEN. Its wrong 85/day baseline is corrected to 183–200/day, the mechanism is re-pointed at ESTAB, and the upgrade's null result is recorded. It stays open on its own merits: an upgrade that had fixed the leak still would not make ~85,000 requests/day to a weekly-write DR endpoint correct.
• documentation/runbooks/offsite-endpoint.md — no change made, correctly: it states no PBS version literal, only package names, so there was nothing to update (and per docs.md, version literals do not belong in a current-state doc anyway).

9. Teardown

This run provisioned nothing. No .deb was downloaded — apt-get changelog served the text directly, so the runbook's /tmp cleanup was never needed. The Hetzner snapshot is retained as the rollback; deleting it is an operator decision, and it costs €0.018161/GB/month.

10. Observations, not acted on

• pvesm status reports felhom-pbs with Total/Used/Available all 0 on both boxes while status reads active. Consistent before and after the upgrade, and the hub's own gauge reads the real 3.7 GB / 97.9 GB, so nothing is broken — but the PVE-side numbers are not usable as a capacity signal. Likely the namespace-scoped token lacking datastore-level audit. Not filed; noted here.
• 8 other packages are held back (8 not upgraded), untouched deliberately — full-upgrade on this machine is forbidden by the runbook and would change the kernel and WireGuard alongside the thing under test.
• The morning's --no-verify situation is unchanged: golden-currency still convicts on the inherited R-334 (controller 0.216.0 vs golden 0.214.0), untouched by this run.