Repository navigation
On the shared listen fd one worker takes nearly every connection, always the same one #329
Description
On the shared listen fd — the default outside Linux and FreeBSD, and the second Linux CI run through TRUE_ASYNC_SERVER_SHARED_LISTEN_FD=1 — connections opened one after another go almost entirely to one worker, in a fixed direction. #240 settled that the server promises nothing about which worker accepts; these numbers are of a different kind than #240's single [1,0,8] under taskset -c 0, so they are recorded here rather than by reopening it.
Measured on 2026-09-29
Linux (WSL2, 16 CPUs, idle), 2 workers, 32 connections opened one after another, final split first:second:
| Probe | Shared listen fd | SO_REUSEPORT |
|---|---|---|
WebSocket, no rooms (ws_open + delay(50)) |
9:23, 6:26, 7:25, 5:27, 11:21 | 15:17, 17:15, 22:10, 12:20, 21:11 |
| WebSocket, rooms + subscribe | 6:26, 5:27, 2:30, 3:29, 4:28 | 15:17, 10:22, 16:16, 16:16, 16:16 |
| HTTP keep-alive, request per connection | 11:21, 15:17, 17:15, 19:13 ×2, 22:10, 25:7, 26:6, 28:4, 9:23 | — |
websocket/065's spread helper (publishes between rounds) |
ends at 0:32 on 2 to 5 runs of 20; otherwise the first worker holds 1 to 3 | first round already split |
On the shared fd every WebSocket run gives the majority to the second worker; on SO_REUSEPORT the direction varies. The cause is not traced. What would establish it: the same probe with 3 workers (does the last one always win), and strace -f -tt -e trace=epoll_wait,accept4 recording, per connection, whose epoll_wait returns first and whose accept4 gets EAGAIN.
Consequence
On macOS, where the shared fd is the default, setWorkers(N) serves sequential clients with essentially one worker. websocket/064–066, which need a subscriber on every worker, now skip on the shared fd (they failed there three times in two runs of PR #328).
Question
Does "the server promises nothing about the accept split" (#240) still hold with this measurement, or is an even-enough split on the shared fd a goal? No fix is proposed before the cause is traced.
Activity
test(websocket): skip the room retry tests on the shared listen fd (#329
EdmondDantes commentedon Oct 20, 2026
Traced on 2026-10-20: the skew is Linux's wake order on a shared listening socket, not a choice the server makes.
Each worker's event loop registers the shared fd in its own epoll instance without
EPOLLEXCLUSIVE.ep_ptable_queue_proc(fs/eventpoll.c) then callsadd_wait_queue, and__add_wait_queue(include/linux/wait.h) doeslist_addat the head of the socket's wait queue. A new connection wakes the entries from the head, so the epoll registered last is woken first. On an idle machine its thread reachesaccept4first, and every other worker getsEAGAIN. The worker that started last therefore takes almost every connection. That matches "every shared-fd run gave the majority to the second worker" above.Measured on this machine (WSL2, Linux), main + #429, 3 workers, 32 sequential connections, one request each, 50 ms apart, connections per worker sorted:
TRUE_ASYNC_SERVER_SHARED_LISTEN_FD=1EPOLLEXCLUSIVEwould not spread the load either.add_wait_queue_exclusiveappends at the tail and wakes one waiter, so the first-registered worker would win instead. Under load the order loosens, because the woken worker may be busy, which is why the keep-alive rows above vary.Linux and FreeBSD use SO_REUSEPORT by default, which hashes connections across the workers' sockets. The shared fd is the default only where SO_REUSEPORT does not balance: macOS, and Windows per #375. Kernel wake order there is not traced; macOS kqueue has not been checked.
EdmondDantes commentedon Oct 20, 2026
Closed by Edmond's decision (2026-10-20): the skew is Linux's wake order on a shared listening socket (traced above), not a server defect. Linux and FreeBSD use SO_REUSEPORT by default. The shared fd is the default only where SO_REUSEPORT does not balance (macOS, and Windows per #375); reopen if a measurement there shows it matters.