admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

INCIDENT + registers: ep0's PBS proxy served nobody for 9.5h (R-336..R-338)
gates / gates (push) Failing after 12s
Details

Browse Source 
Two whole_guest_backup_failed alerts at 04:30 and 04:32 CEST were one
incident, and not on either customer box: ep0's proxmox-backup-proxy was
active, holding its listening socket, and accepting nothing.

Root cause: accept() returning EMFILE. The process held exactly 1024 fds
-- its systemd-default soft RLIMIT_NOFILE -- of which 1016 were sockets
and 547 connections sat in CLOSE-WAIT. The 1024-deep accept backlog had
overflowed (Recv-Q 1025), so every client timed out. It was wedged from
its own loopback too, which is what moved this from a network problem to
a process problem.

Fed by ~85k requests/day (a flat 3,538/hour) against an endpoint written
to weekly, that leak reached the ceiling in 14 days of uptime.

Fix: LimitNOFILE=65536 drop-ins for both PBS units, restart, verified from
both boxes (200 in ~0.1s, felhom-pbs active), then re-drove the missed
backups through the product path -- POST /backup?target=felhom-pbs on each
agent's local API, not a hand-run vzdump.

  demo-felhom  ct/9201/2026-08-18T03:57:43Z  4.10 GB  36.4s
  demo-hp      ct/9201/2026-08-18T03:58:43Z  4.29 GB  41.5s

Both host reports now carry felhom-pbs success=true, so the hub is green on
the evidence rather than on a restart having been performed. No data lost,
no backup skipped: the daily local tier was never affected and the PBS tier
is weekly, so the window cost exactly one attempt.

Evidence copied off ep0 BEFORE the restart, per standing rule 5.

Filed: R-336 (the ~1 req/s poll rate is the real defect; the raised ceiling
is mitigation, not a cure), R-337 (a status endpoint that trailed its own
artifact by minutes then caught up -- WATCHING, downgraded from the defect
I first wrote, because it self-corrected), R-338 (demo-hp is not on the
R-50 island at all and nodes.md says it is; its local API is bound to the
customer LAN).

R-334 updated: still open, now one version wider (controller 0.216.0 vs
golden 0.214.0). golden-currency is the only failing gate and is inherited
-- it reads files this session did not touch -- so this push used
--no-verify, stated per .claude/rules/gates.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016p1PTCzb8rF5G9Aa1qhpBN
This commit is contained in:
authentik Default Admin
2026-08-18 06:09:38 +02:00
parent ea16a21bff
commit ebfd0967c1
7 changed files with 559 additions and 339 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/audits/evidence-ep0-fd-2026-08-18/access.log.last200
+200
View File
@@ -0,0 +1,200 @@
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:08:50 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:08:52 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:08:53 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:08:55 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:08:56 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:08:56 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:08:57 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:09:02 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:09:05 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:09:06 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:09:07 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:09:10 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:09:12 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:09:12 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:09:15 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:09:22 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:09:25 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:09:26 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:09:27 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:09:30 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:09:32 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:09:33 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:09:34 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:09:35 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:09:36 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:09:37 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:09:42 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:09:46 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:09:50 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:09:52 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:09:52 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:09:54 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:09:55 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:09:55 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:09:56 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:09:56 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:09:56 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:09:57 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:10:02 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:10:05 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:10:06 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:10:07 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:10:10 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:10:12 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:10:13 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:10:16 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:10:22 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:10:25 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:10:26 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:10:27 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:10:30 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:10:32 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:10:32 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:10:34 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:10:35 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:10:36 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:10:37 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:10:42 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:10:45 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:10:50 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:10:52 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:10:53 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:10:55 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:10:55 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:10:56 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:10:57 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:11:02 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:11:06 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:11:06 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:11:07 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:11:10 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:11:12 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:11:12 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:11:15 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:11:22 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:11:25 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:11:26 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:11:27 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:11:30 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:11:32 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:11:33 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:11:34 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:11:34 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:11:35 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:11:35 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:11:36 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:11:36 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:11:37 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:11:42 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:11:45 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:11:50 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:11:52 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:11:52 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:11:55 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:11:55 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:11:56 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:11:57 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:12:02 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:12:05 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:12:06 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:12:07 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:12:10 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:12:12 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:12:13 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:12:15 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:12:22 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:12:26 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:12:26 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:12:27 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:12:30 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:12:32 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:12:32 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:12:34 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:12:35 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:12:36 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:12:37 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:12:42 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:12:45 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:12:50 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:12:52 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:12:52 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:12:55 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:12:56 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:12:56 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:12:57 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:13:02 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:13:05 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:13:06 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:13:07 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:13:10 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:13:12 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:13:12 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:13:15 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:13:22 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:13:25 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:13:26 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:13:27 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:13:30 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:13:32 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:13:32 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:13:34 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:13:35 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:13:36 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:13:37 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:13:42 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:13:46 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:13:50 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:13:52 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:13:52 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:13:55 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:13:55 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:13:56 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:13:57 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:14:02 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:14:05 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:14:06 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:14:07 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:14:10 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:14:12 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:14:12 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:14:16 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:14:23 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:14:25 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:14:26 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:14:27 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:14:30 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:14:32 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:14:32 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:14:34 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:14:35 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:14:36 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:14:37 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:14:42 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:14:45 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:14:50 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:14:52 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:14:52 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:14:54 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:14:55 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:14:55 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:14:56 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:14:56 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:14:56 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:14:57 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:15:03 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:15:05 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:15:06 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:15:07 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:15:10 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:15:12 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:15:12 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:15:15 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:15:22 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.3 - felhom@pbs!demo-hp [17/08/2026:18:15:25 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:15:26 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:15:27 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:15:30 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:15:31 +0000] "GET /api2/json/admin/datastore" 400 88 libwww-perl/6.78
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:15:32 +0000] "GET /api2/json/version" 200 92 Go-http-client/1.1
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:15:32 +0000] "GET /api2/json/admin/datastore/felhom-offsite/snapshots?ns=demo-felhom" 400 27 Go-http-client/1.1
documentation/audits/evidence-ep0-fd-2026-08-18/post-fix-state.txt
+28
View File
@@ -0,0 +1,28 @@
### post-fix Tue Aug 18 04:00:41 UTC 2026
proxy_pid=542065 fds=18
Max open files 65536 65536 files
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 1024 *:8007 *:*
1 ESTAB
1 LISTEN
### snapshots
-- demo-felhom
2026-08-04T04:49:44Z
2026-08-11T04:54:07Z
2026-08-18T03:57:43Z
-- demo-hp
2026-08-04T19:24:16Z
2026-08-11T19:29:13Z
2026-08-18T03:58:43Z
### dropins
# 2026-08-18: proxy wedged after 14d uptime — accept() returned EMFILE at the
# systemd default soft limit of 1024 fds (1016 sockets, 547 in CLOSE-WAIT),
# the 1024-deep accept backlog overflowed, and every PBS client timed out.
# ~85k requests/day against this endpoint; 1024 is not a sane ceiling for it.
[Service]
LimitNOFILE=65536
# 2026-08-18: raised alongside the proxy (see its 20-nofile.conf). This daemon
# was NOT the one that exhausted fds (it held 15), but the systemd default 1024
# is not a sane ceiling for a PBS backend either.
[Service]
LimitNOFILE=65536
documentation/audits/evidence-ep0-fd-2026-08-18/pre-restart-state.txt
+28
View File
@@ -0,0 +1,28 @@
### date
Tue Aug 18 03:54:48 UTC 2026
### ss -lnt 8007
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 1025 1024 *:8007 *:*
### socket states
543 CLOSE-WAIT
1011 ESTAB
1 LISTEN
### fd count
1024
### fd types
1016 socket:anon
3 anon_inode:anon
1 0
1 /var/log/proxmox-backup/api/auth.log
1 /var/log/proxmox-backup/api/access.log
1 /var/lib/proxmox-backup/rrdb/rrd.journal-6a834fcb
1 /mnt/pbs-datastore/.lock
1 /dev/null
### limits
Max open files 1024 524288 files
### versions
proxmox-backup-server 4.2.5-1 running version: 4.2.2
### uptime
03:54:48 up 14 days, 16:41, 1 user, load average: 1.76, 1.88, 1.87
### last served request
::ffff:10.77.0.2 - felhom@pbs!demo-felhom [17/08/2026:18:15:32 +0000] "GET /api2/json/admin/datastore/felhom-offsite/snapshots?ns=demo-felhom" 400 27 Go-http-client/1.1