RUNBOOK ep0: PBS 4.2.2-1 -> 4.2.5-1, slope unchanged as predicted (R-341)
gates / gates (push) Failing after 14s

Both STOPs cleared by the operator. No code changed; documentation only.

STEP 3 (the run's primary deliverable): the full changelog range 4.2.2-1 ->
4.2.5-1 was read (128 lines, all three entries) and swept for
connection-handling vocabulary. Exactly one keyword hit, a false positive
("S3 ... honor the node's proxy settings" = HTTP proxy config for S3, not the
PBS proxy daemon). 4.2.5-1 is a manifest-hardening security release; 4.2.4-1
is S3 rate limits and a locking cache; 4.2.3-1 is UI/LDAP/tape. NOTHING
addresses descriptor lifetime or connection reaping. Recommendation was: do
not upgrade for this reason.

STOP 1: operator ruled to upgrade anyway for rehearsal value. Recorded as a
practice run, not a fix -- and the interpretation was fixed IN WRITING BEFORE
any numbers existed (stop1-ruling.txt): unchanged = expected; changed =
surprise. Neither outcome could then be rationalised into a success.

STOP 2: Hetzner snapshot 421440873, Available. Documented that it covers
/dev/sda ONLY -- /mnt/pbs-datastore is a separate Volume and is NOT in it, so
it is a software rollback and not a backup of the backup data.

UPGRADE: simulated first (0 to remove), then installed 09:51:00->09:51:06Z,
exit 0. Verified: 4.2.5-1 installed, both daemons active, effective open
files still 65536 (the drop-in survived the new package), Recv-Q 0, loopback
200, 200 from BOTH boxes over the tunnel with felhom-pbs active, and the hub
gauge refreshed post-upgrade at 11:59:31.

SLOPE: before +4 fd/1885 s = 183/day; after +5 fd/1919 s = 225/day. NOT
distinguishable -- one descriptor apart, Poisson +/-2 on such counts. The
higher after-figure is noise, not a regression and not an improvement. 30
minutes cannot settle it; R-341 files the +24 h and +7 d checks.

CORRECTIONS to this morning's own report, both published rather than quietly
fixed:
  - the "~85/day, ~2 years of runway" figures were WRONG. They came from a
    single 17-minute window with a delta of ONE descriptor. Real rate is
    183-200/day over two independent windows; runway ~357 days, not 2 years.
  - the leak was attributed to CLOSE-WAIT. It is mostly ESTAB: CLOSE-WAIT held
    flat at 1 while ESTAB grew 45->49, and at the wedge it was 1011 ESTAB vs
    543 CLOSE-WAIT. R-336's fix must target unreaped connections.
  - "proxmox-backup-api" reported inactive during verification; that unit does
    not exist. Bad query, not a fault, written down because it looked like one.

R-336 stays open: even a fixed leak would not make ~85k requests/day to a
weekly-write DR endpoint correct.

golden-currency still convicts (inherited R-334, controller 0.216.0 vs golden
0.214.0, untouched by this run), so this push is --no-verify per
.claude/rules/gates.md.
This commit is contained in:
2026-08-18 12:26:53 +02:00
parent 435e044cf1
commit 3e50902a98
22 changed files with 1026 additions and 113 deletions
+11 -4
View File
@@ -1,6 +1,6 @@
# STATUS — what works, what's broken, what's next
**Updated 2026-08-18 (early — a listening socket that served nobody; off-site backups were down for
**Updated 2026-08-18 (midday — a listening socket that served nobody, then a rehearsal upgrade that changed nothing; off-site backups were down for
9½ hours overnight and are back).**
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority; this page restates
@@ -50,8 +50,8 @@ record with no machine** — created 13 August, no host, no backups, nothing to
were re-run the same morning and are on the off-site box**. **Nothing was lost and nothing was
skipped:** the daily copies on the machines themselves were never affected, and the off-site copy is
weekly, so exactly one attempt fell in the window. The underlying cause — we ask that box a question
about once a second, all day — is filed and **not yet fixed**; the raised ceiling buys years, not a
cure.
about once a second, all day — is filed and **not yet fixed**; the raised ceiling buys **under a
year** on the corrected measurement, not a cure.
- **The removal now genuinely reverses the installation** (R-316, `installer-v1.28.0` published). The
second reinstall used to hit our own leftover; it was watched failing on the cycle that actually
@@ -86,7 +86,14 @@ record with no machine** — created 13 August, no host, no backups, nothing to
for a box we actually write to once a week. That volume is what turned a slow internal leak into
last night's outage in a fortnight. The higher ceiling makes it rare, not impossible — **the leak
itself is untouched.** The honest health check is the resource count climbing, not the absence of an
alarm.
alarm. **Corrected later the same morning: my first estimate of how fast it leaks was too
optimistic by about 2.5×** — measured properly it is under a year to the new ceiling, not two years.
A deadline, not a comfort.
- **The off-site box was updated, and it did not help — as expected** (R-341). On your ruling we
installed the newer backup software for the practice, having first read its release notes and found
**nothing** about the fault we have. The update went cleanly and everything works, but the leak
behaves exactly as before, which is the result the release notes predicted. **Two dated checks are
booked — 19 August and 25 August** — because half an hour of watching cannot honestly settle it.
- **One machine's status took several minutes to admit a backup had worked** (R-337). `demo-hp` was
still showing this morning's failure for four minutes after the copy was safely on the off-site box;
`demo-felhom` updated in under a minute. **It corrected itself** and both machines now read