admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
3e50902a985fa60d16e91b9ab55ce7e888d8e76e
felhom.eu/documentation/audits/evidence-ep0-pbs-upgrade-2026-08-18/step1-state.txt
T
admin 3e50902a98
gates / gates (push) Failing after 14s
Details
RUNBOOK ep0: PBS 4.2.2-1 -> 4.2.5-1, slope unchanged as predicted (R-341) 
Both STOPs cleared by the operator. No code changed; documentation only.

STEP 3 (the run's primary deliverable): the full changelog range 4.2.2-1 ->
4.2.5-1 was read (128 lines, all three entries) and swept for
connection-handling vocabulary. Exactly one keyword hit, a false positive
("S3 ... honor the node's proxy settings" = HTTP proxy config for S3, not the
PBS proxy daemon). 4.2.5-1 is a manifest-hardening security release; 4.2.4-1
is S3 rate limits and a locking cache; 4.2.3-1 is UI/LDAP/tape. NOTHING
addresses descriptor lifetime or connection reaping. Recommendation was: do
not upgrade for this reason.

STOP 1: operator ruled to upgrade anyway for rehearsal value. Recorded as a
practice run, not a fix -- and the interpretation was fixed IN WRITING BEFORE
any numbers existed (stop1-ruling.txt): unchanged = expected; changed =
surprise. Neither outcome could then be rationalised into a success.

STOP 2: Hetzner snapshot 421440873, Available. Documented that it covers
/dev/sda ONLY -- /mnt/pbs-datastore is a separate Volume and is NOT in it, so
it is a software rollback and not a backup of the backup data.

UPGRADE: simulated first (0 to remove), then installed 09:51:00->09:51:06Z,
exit 0. Verified: 4.2.5-1 installed, both daemons active, effective open
files still 65536 (the drop-in survived the new package), Recv-Q 0, loopback
200, 200 from BOTH boxes over the tunnel with felhom-pbs active, and the hub
gauge refreshed post-upgrade at 11:59:31.

SLOPE: before +4 fd/1885 s = 183/day; after +5 fd/1919 s = 225/day. NOT
distinguishable -- one descriptor apart, Poisson +/-2 on such counts. The
higher after-figure is noise, not a regression and not an improvement. 30
minutes cannot settle it; R-341 files the +24 h and +7 d checks.

CORRECTIONS to this morning's own report, both published rather than quietly
fixed:
  - the "~85/day, ~2 years of runway" figures were WRONG. They came from a
    single 17-minute window with a delta of ONE descriptor. Real rate is
    183-200/day over two independent windows; runway ~357 days, not 2 years.
  - the leak was attributed to CLOSE-WAIT. It is mostly ESTAB: CLOSE-WAIT held
    flat at 1 while ESTAB grew 45->49, and at the wedge it was 1011 ESTAB vs
    543 CLOSE-WAIT. R-336's fix must target unreaped connections.
  - "proxmox-backup-api" reported inactive during verification; that unit does
    not exist. Bad query, not a fault, written down because it looked like one.

R-336 stays open: even a fixed leak would not make ~85k requests/day to a
weekly-write DR endpoint correct.

golden-currency still convicts (inherited R-334, controller 0.216.0 vs golden
0.214.0, untouched by this run), so this push is --no-verify per
.claude/rules/gates.md.
2026-08-18 12:26:53 +02:00

106 lines
5.2 KiB
Plaintext
Raw Blame History

### capture-time
2026-08-18T09:18:21+00:00
### dpkg -l (INSTALLED — the truth)
ii proxmox-backup-client 4.2.2-1 amd64 Proxmox Backup Client tools
ii proxmox-backup-server 4.2.2-1 amd64 Proxmox Backup Server daemon with tools and GUI
### apt-cache policy proxmox-backup-server
proxmox-backup-server:
Installed: 4.2.2-1
Candidate: 4.2.5-1
Version table:
4.2.5-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.2.4-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.2.3-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
*** 4.2.2-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
100 /var/lib/dpkg/status
4.2.1-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.2.0-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.1.13-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.1.12-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.1.11-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.1.10-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.1.9-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.1.8-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.1.7-2 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.1.6-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.1.5-2 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.1.5-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.1.4-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.1.2-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.1.1-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.1.0-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.0.22-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.0.21-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.0.20-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.0.19-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.0.18-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.0.17-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.0.16-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.0.15-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.0.14-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.0.13-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.0.12-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.0.11-4 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.0.11-2 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.0.10-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.0.9-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.0.8-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.0.7-1 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
4.0.6-2 500
500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
### proxmox-backup-manager version (available THEN running — misleading, see incident)
proxmox-backup-server 4.2.5-1 running version: 4.2.2
### proxy MainPID
542065
### fd count
62
### limits
Max open files 65536 65536 files
### listen queue
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 1024 *:8007 *:*
### close-wait count
1
### proxy start time
Tue Aug 18 03:54:53 2026
### datastore
Filesystem Size Used Avail Use% Mounted on
/dev/sdb 98G 3.7G 95G 4% /mnt/pbs-datastore
Reference in New Issue View Git Blame Copy Permalink