INCIDENT + registers: ep0's PBS proxy served nobody for 9.5h (R-336..R-338)

Two whole_guest_backup_failed alerts at 04:30 and 04:32 CEST were one
incident, and not on either customer box: ep0's proxmox-backup-proxy was
active, holding its listening socket, and accepting nothing.

Root cause: accept() returning EMFILE. The process held exactly 1024 fds
-- its systemd-default soft RLIMIT_NOFILE -- of which 1016 were sockets
and 547 connections sat in CLOSE-WAIT. The 1024-deep accept backlog had
overflowed (Recv-Q 1025), so every client timed out. It was wedged from
its own loopback too, which is what moved this from a network problem to
a process problem.

Fed by ~85k requests/day (a flat 3,538/hour) against an endpoint written
to weekly, that leak reached the ceiling in 14 days of uptime.

Fix: LimitNOFILE=65536 drop-ins for both PBS units, restart, verified from
both boxes (200 in ~0.1s, felhom-pbs active), then re-drove the missed
backups through the product path -- POST /backup?target=felhom-pbs on each
agent's local API, not a hand-run vzdump.

  demo-felhom  ct/9201/2026-08-18T03:57:43Z  4.10 GB  36.4s
  demo-hp      ct/9201/2026-08-18T03:58:43Z  4.29 GB  41.5s

Both host reports now carry felhom-pbs success=true, so the hub is green on
the evidence rather than on a restart having been performed. No data lost,
no backup skipped: the daily local tier was never affected and the PBS tier
is weekly, so the window cost exactly one attempt.

Evidence copied off ep0 BEFORE the restart, per standing rule 5.

Filed: R-336 (the ~1 req/s poll rate is the real defect; the raised ceiling
is mitigation, not a cure), R-337 (a status endpoint that trailed its own
artifact by minutes then caught up -- WATCHING, downgraded from the defect
I first wrote, because it self-corrected), R-338 (demo-hp is not on the
R-50 island at all and nodes.md says it is; its local API is bound to the
customer LAN).

R-334 updated: still open, now one version wider (controller 0.216.0 vs
golden 0.214.0). golden-currency is the only failing gate and is inherited
-- it reads files this session did not touch -- so this push used
--no-verify, stated per .claude/rules/gates.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016p1PTCzb8rF5G9Aa1qhpBN

STATUS.md

@@ -1,6 +1,7 @@
 # STATUS — what works, what's broken, what's next
 
-**Updated 2026-08-13 (late — the third name, a machine told to be quiet, and a picture you can query).**
+**Updated 2026-08-18 (early — a listening socket that served nobody; off-site backups were down for
+9½ hours overnight and are back).**
 
 > **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority; this page restates
 > part of it in plain words, and **nothing may exist only here**. **Items, not paragraphs. One screen.**
@@ -42,6 +43,16 @@ record with no machine** — created 13 August, no host, no backups, nothing to
 
 ## Shipped
 
+- **Last night's two backup alarms were real, and are fixed** (R-336). Both machines failed their
+  off-site backup at 04:30; **neither machine was at fault**. The off-site box in Germany had run out
+  of one internal resource and, while looking perfectly healthy from outside, was accepting no
+  connections at all — for 9½ hours. Restarted, given a ceiling 64× higher, and **both missed backups
+  were re-run the same morning and are on the off-site box**. **Nothing was lost and nothing was
+  skipped:** the daily copies on the machines themselves were never affected, and the off-site copy is
+  weekly, so exactly one attempt fell in the window. The underlying cause — we ask that box a question
+  about once a second, all day — is filed and **not yet fixed**; the raised ceiling buys years, not a
+  cure.
+
 - **The removal now genuinely reverses the installation** (R-316, `installer-v1.28.0` published). The
   second reinstall used to hit our own leftover; it was watched failing on the cycle that actually
   fails, then watched passing.
@@ -71,6 +82,22 @@ record with no machine** — created 13 August, no host, no backups, nothing to
 
 ## Broken, or knowingly incomplete
 
+- **We interrogate the off-site box about once a second** (R-336). Roughly 85,000 questions a day,
+  for a box we actually write to once a week. That volume is what turned a slow internal leak into
+  last night's outage in a fortnight. The higher ceiling makes it rare, not impossible — **the leak
+  itself is untouched.** The honest health check is the resource count climbing, not the absence of an
+  alarm.
+- **One machine's status took several minutes to admit a backup had worked** (R-337). `demo-hp` was
+  still showing this morning's failure for four minutes after the copy was safely on the off-site box;
+  `demo-felhom` updated in under a minute. **It corrected itself** and both machines now read
+  correctly, so this is a note to watch, **not something broken** — but during the repair it looked
+  briefly like a second fault, which is the reason it is written down.
+- **`demo-hp` is not set up the way our own notes say it is** (R-338). Our inventory records both
+  machines as moved onto the isolated internal link last July. `demo-felhom` was; **`demo-hp` was
+  not** — its control channel is still bound to the ordinary home network, which is exactly what that
+  change existed to stop. The machine works; the page is wrong, and it misled this session by an hour.
+  Your call which one to correct.
+
 - **Peti's machine has no recovery route at all** — see the `PETI` row. **This is a real machine
   belonging to a real person**, not one of ours and not a record: it reported to the hub for four and a
   half months and has been silent since 15 July, when its host record was deleted. There is no key, no