admin
ebfd0967c1
gates / gates (push) Failing after 12s
Two whole_guest_backup_failed alerts at 04:30 and 04:32 CEST were one incident, and not on either customer box: ep0's proxmox-backup-proxy was active, holding its listening socket, and accepting nothing. Root cause: accept() returning EMFILE. The process held exactly 1024 fds -- its systemd-default soft RLIMIT_NOFILE -- of which 1016 were sockets and 547 connections sat in CLOSE-WAIT. The 1024-deep accept backlog had overflowed (Recv-Q 1025), so every client timed out. It was wedged from its own loopback too, which is what moved this from a network problem to a process problem. Fed by ~85k requests/day (a flat 3,538/hour) against an endpoint written to weekly, that leak reached the ceiling in 14 days of uptime. Fix: LimitNOFILE=65536 drop-ins for both PBS units, restart, verified from both boxes (200 in ~0.1s, felhom-pbs active), then re-drove the missed backups through the product path -- POST /backup?target=felhom-pbs on each agent's local API, not a hand-run vzdump. demo-felhom ct/9201/2026-08-18T03:57:43Z 4.10 GB 36.4s demo-hp ct/9201/2026-08-18T03:58:43Z 4.29 GB 41.5s Both host reports now carry felhom-pbs success=true, so the hub is green on the evidence rather than on a restart having been performed. No data lost, no backup skipped: the daily local tier was never affected and the PBS tier is weekly, so the window cost exactly one attempt. Evidence copied off ep0 BEFORE the restart, per standing rule 5. Filed: R-336 (the ~1 req/s poll rate is the real defect; the raised ceiling is mitigation, not a cure), R-337 (a status endpoint that trailed its own artifact by minutes then caught up -- WATCHING, downgraded from the defect I first wrote, because it self-corrected), R-338 (demo-hp is not on the R-50 island at all and nodes.md says it is; its local API is bound to the customer LAN). R-334 updated: still open, now one version wider (controller 0.216.0 vs golden 0.214.0). golden-currency is the only failing gate and is inherited -- it reads files this session did not touch -- so this push used --no-verify, stated per .claude/rules/gates.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016p1PTCzb8rF5G9Aa1qhpBN
120 lines
7.1 KiB
Markdown
120 lines
7.1 KiB
Markdown
# REPORT — a listening socket that served nobody (2026-08-18, early)
|
|
|
|
**Trigger:** two `whole_guest_backup_failed` alert mails, 04:30 and 04:32 CEST.
|
|
**Outcome:** root cause found on **ep0**, fixed, both missed backups re-driven and landed.
|
|
**No repo code changed** — this was an operational run. Documentation, register and evidence only.
|
|
|
|
---
|
|
|
|
## 1. What was wrong
|
|
|
|
Both alerts were the same incident and neither was on a customer box. `demo-felhom` and `demo-hp`
|
|
each failed their `felhom-pbs` tier with `Can't connect to 10.77.0.1:8007 (Connection timed out)` —
|
|
the one thing they share, the Hetzner offsite PBS endpoint.
|
|
|
|
On ep0, `proxmox-backup-proxy` was `active`, held its listening socket, and **served nobody**:
|
|
|
|
```
|
|
ss -lnt '( sport = :8007 )' → LISTEN Recv-Q 1025 Send-Q 1024
|
|
ls /proc/<proxy>/fd | wc -l → 1024 # == its soft RLIMIT_NOFILE
|
|
```
|
|
|
|
`Send-Q` on a listener is the accept backlog; `Recv-Q` is the queue depth. At 1025 against 1024 the
|
|
queue had overflowed, because `accept()` was returning `EMFILE` on every call. **1016 of the 1024
|
|
descriptors were sockets and 547 connections sat in `CLOSE-WAIT`** — a connection leak, fed by
|
|
~85,000 requests/day, that reached the ceiling after 14 days of uptime. Last request served:
|
|
2026-08-17 18:15:32 UTC. Offsite DR was therefore down **≈ 9 h 37 m**.
|
|
|
|
**The observation that settled it:** the daemon was wedged from its own loopback too —
|
|
`curl https://127.0.0.1:8007/` on ep0 timed out. A listener that cannot serve `127.0.0.1` has no
|
|
network left to blame, and that check is cheap enough to make early.
|
|
|
|
Full record, including everything that was ruled out first (tunnel, nftables, disk, dead daemon):
|
|
**`documentation/audits/INCIDENT-ep0-pbs-fd-exhaustion-2026-08-18.md`**.
|
|
|
|
## 2. What was done
|
|
|
|
1. `LimitNOFILE=65536` drop-ins for `proxmox-backup-proxy.service` and `proxmox-backup.service`, each
|
|
carrying its reason inline. **The API daemon was not implicated** (15 fds) and its drop-in says so
|
|
— a later reader must not mistake it for a second culprit.
|
|
2. Restarted both. After: soft limit 65536, fds back to 18, `Recv-Q 0`, loopback `200`.
|
|
3. **Verified from the customer side, not only from ep0** — both boxes got `200` in ~0.1 s and
|
|
`pvesm status` read `felhom-pbs pbs active`.
|
|
4. Re-drove the missed backups **through the product path** — `POST /backup?target=felhom-pbs` on
|
|
each agent's local API, issued from inside the guest's controller container with the controller's
|
|
own credentials, i.e. the same call the scheduler makes. Not a hand-run `vzdump`.
|
|
|
|
**Result:** `demo-felhom` → `ct/9201/2026-08-18T03:57:43Z` (4.10 GB, 36.4 s);
|
|
`demo-hp` → `ct/9201/2026-08-18T03:58:43Z` (4.29 GB, 41.5 s). Both directories carry a full manifest
|
|
on ep0, and both hosts log the re-run vzdump as `OK`. **And the hub agrees** — both boxes' next host
|
|
reports carry `felhom-pbs success=true` (04:00:33Z and 04:07:35Z), so the operator view went green on
|
|
the evidence rather than on the fact that a restart was performed.
|
|
**No data was lost and no backup was skipped** — the daily local
|
|
tier was never affected (it completed on both boxes at 05:00 and 05:02), and the PBS tier is weekly,
|
|
so the outage window cost exactly one attempt, which was re-driven the same morning.
|
|
|
|
Evidence copied off ep0 **before** the restart, per standing rule 5:
|
|
`documentation/audits/evidence-ep0-fd-2026-08-18/` — pre-restart state, post-fix state, access-log tail.
|
|
|
|
## 3. What I got wrong, and corrected
|
|
|
|
- **`proxmox-backup-manager version` prints *available* then *running*.** It read
|
|
`4.2.5-1 running version: 4.2.2`, which looks exactly like a daemon left behind by a package
|
|
upgrade. It is not: `dpkg -l` shows **4.2.2-1 installed**, 4.2.5-1 merely available in the repo,
|
|
and the on-disk binary is dated 2026-06-18. I restarted the API daemon on that mistaken reading;
|
|
harmless, and its drop-in is a genuine improvement, but it was not needed.
|
|
- **I addressed `demo-hp`'s agent at the island address** because `operations/nodes.md` says that box
|
|
is island-migrated. It is not (**R-338**), and the resulting timeout was briefly read as a fault.
|
|
|
|
## 4. Findings filed — R-336, R-337, R-338
|
|
|
|
All three are in `documentation/backlog/OPEN-ITEMS.md` with numbers, per the registers-first rule.
|
|
|
|
- **R-336 — the poll rate is the real defect.** ~1 request/second against a DR endpoint written to
|
|
weekly. `LimitNOFILE` raises the ceiling; **it does not fix the leak**, it converts a fortnightly
|
|
outage into a multi-year one.
|
|
- **R-337 — a status endpoint that trailed its own artifact, then caught up. WATCHING, not a defect.**
|
|
`demo-hp`'s `/backup/status` was still serving the superseded 03:27:00Z failure at ~04:03Z while the
|
|
snapshot sat on ep0 and the host logged `OK`; `demo-felhom` updated within ~40 s. **I filed this as
|
|
a defect and that was premature** — the next host report (04:07:35Z) carried the success and the
|
|
skew cleared with no intervention. Rewritten as WATCHING, with the explicit instruction not to open
|
|
a fix until someone establishes whether this is just collection cadence. Recorded at all because
|
|
during the recovery it read as a second failure, and it was not one.
|
|
- **R-338 — `demo-hp` is not on the R-50 island and `nodes.md` says it is.** No `island_bridge` keys,
|
|
guest has no `eth1`, `vmbr9` has zero members, and the agent's local API is bound to the customer
|
|
LAN — the exposure R-50 existed to remove.
|
|
|
|
**Not done, deliberately:** PBS 4.2.5-1 was not applied. Upgrading a production offsite endpoint was
|
|
outside what this run was authorised to do, and its changelog should be read for the connection-
|
|
handling leak first.
|
|
|
|
## 5. Gates — one pre-existing conviction, and a stated bypass
|
|
|
|
`python3 scripts/repo_gates.py --fast` → **rc=1, CONVICTED: golden-currency.** Eight of nine gates
|
|
pass. The conviction is **R-334, inherited and not caused here**: newest released controller
|
|
**0.216.0**, newest golden bake **0.214.0**, so a new install misses two releases. The gate reads
|
|
`felhom-controller/CHANGELOG.md` and `documentation/tests/golden-*` — **this session touched
|
|
neither**, and its whole diff is documentation. Baking is possible; **vouching is
|
|
operator-password-gated, and a baked-but-unvouched golden is worse than none**, so it is not a
|
|
one-sided job CC can finish.
|
|
|
|
**This push therefore used `git push --no-verify`, stated here per `.claude/rules/gates.md`.**
|
|
R-334 is updated in the register with the new numbers rather than left reading 0.215.0.
|
|
|
|
*(Noted for accuracy: the first gate run was piped to `tail`, which returned `rc=0` — `tail`'s exit
|
|
code, not the gate's. It was re-run unpiped to read the real `rc=1`. That is standing rule 1's trap
|
|
in its smaller form, and the number reported above is the unpiped one.)*
|
|
|
|
## 6. What to watch
|
|
|
|
The positive observable is the descriptor count, not the absence of an alert — an empty alert queue
|
|
is equally consistent with "healthy" and "wedged again":
|
|
|
|
```bash
|
|
ssh root@<ep0> 'PID=$(systemctl show proxmox-backup-proxy -p MainPID --value); \
|
|
ls /proc/$PID/fd | wc -l; ss -lnt "( sport = :8007 )"'
|
|
```
|
|
|
|
Healthy is ~20 fds and `Recv-Q 0`. **A count climbing between restarts means R-336's leak is still
|
|
live.**
|