Repository navigation
Abort the websocket connection a Close() leaves open - #10033
Conversation
Close() cancels the connection loop and relies on the cancelled receive to abort the connection. When the cancellation lands before the first receive, while the Open handlers run, the loop ends without receiving and the connection stays open after Close() returns, until the next Connect() replaces it. The loop now aborts the last connection it opened when it ends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Review of a875f3c
The fix is correct and as small as it can be. I traced HandleConnection and Close() for every way the loop can end:
- Normal
Close(): the loop is blocked inReceiveAsync. Cancelling it already aborts the socket, so the newclient?.Abort()does nothing. Close()during the error back-off (connectionCts.Token.WaitHandle.WaitOne): the failed client is already faulted or aborted.Abort()on it does nothing.Close()betweenConnectAsyncand the firstReceiveAsync(the bug): the loop now aborts the connection before_taskConnectcompletes.Close()waits on that task, so the socket is aborted beforeClose()returns and beforeOnCloseruns. This also covers aClose()called from inside anOpen/Messagehandler on the connection thread, which left the socket open the same way before.Close()gives up after 5 s: aborting the loop's own local, not_client, is right. A laterConnect()may already have stored a new connection in_client.
The only visible difference for callers is that Closed handlers now see State Aborted, where they used to see Open in the racy case. I checked the brokerages that subscribe to Closed (for example Binance's handler, which stops its session and keep-alive timer). None of them reads the socket state there.
Evidence
- CI is green on this commit.
- I reproduced the test both ways. With only
Brokerages/WebSocketClientWrapper.csreverted to master,CloseWhileTheOpenHandlerRunsClosesTheConnectionfails withExpected: False, But was: True(IsOpenafterClose()). With the fix it passes 3 times in 3. The test holds theOpenhandler untilClose()has cancelled the loop, so it hits the interleaving every run. - I also ran a brokerage's reconnect tests against this build: 12 runs of the suite, 6 at a time. Its check that the server never has two connections open passed every time. Against the released
QuantConnect.Brokeragesthat check is flaky. - No end-to-end run is described. This path only runs when a brokerage's live connection races a
Close()against a reconnect, which a live run can't trigger on demand. The deterministic test reaches that exact interleaving, so I accept it. The description could say so in one line.
Other repositories
Brokerages that use WebSocketClientWrapper get the fix with the next QuantConnect.Brokerages package. None needs a code change.
Risk
I set risk:high. The diff is small, backwards compatible, and does nothing outside the racy path. But it changes WebSocketClientWrapper, which every websocket brokerage uses for its live connection, so its blast radius is wide.
|
Added the line on end-to-end testing to the description: a live run can't trigger the race on demand, and the test reaches the exact interleaving every run. No code change. 🤖 Generated with Claude Code |
There was a problem hiding this comment.
Review of a875f3c
The code is the same as when I approved it. The only change is the description: it now says why there's no end-to-end run (a live run can't trigger the race on demand, and the test reaches that exact interleaving every run). That answers the one note in my last review. CI is still green on this commit.
I'm approving again. The brokerage pull request on the same branch isn't ready yet, but that's because it's waiting for this fix to be merged and released. Holding this one back would block both. This fix is correct on its own.
Other repositories
Brokerages that use WebSocketClientWrapper get the fix with the next QuantConnect.Brokerages package. None needs a code change.
Risk
risk:high stays. The diff is small and backwards compatible, but it changes WebSocketClientWrapper, which every websocket brokerage uses for its live connection, so its blast radius is wide.
Problem
WebSocketClientWrapper.Close()cancels the connection loop and relies on the cancelledReceiveAsyncto abort the connection. If the cancellation lands afterConnectAsyncbut before the firstReceiveAsync, while theOpenhandlers run, the loop sees the cancellation, skips the receive and ends. Nothing aborts the connection:Close()callsCloseOutputAsyncwith the token it just cancelled, so no close frame goes out._clientis only disposed by the nextHandleConnection.So after
Close()returns,IsOpenis stilltrueand the server still has the connection open. It stays that way until the nextConnect(), or for good if none follows (shutdown). It's easy to hit when a reconnect, from the loop or from the server dropping the connection, overlaps aClose()from another thread.Close()waits on_lockerwhileConnectAsyncholds it, so it gets the lock just asOnOpenstarts.Change
HandleConnectionkeeps the lastClientWebSocketit created, and aborts it once the loop ends. The loop only ends once it's cancelled, so this runs only onClose(). By then, in every other case, the connection is already aborted or closed, andAbort()does nothing. It uses the loop's own local, not the_clientfield: ifClose()gave up waiting (5 s), a laterConnect()may already have put a new connection in_client. Nothing else changes.Tests
WebSocketClientWrapperTests.CloseWhileTheOpenHandlerRunsClosesTheConnection: a local websocket server, and anOpenhandler that holds the connection thread untilClose()has been called and for 1 s after it. That hits the interleaving every time. The test checks that afterClose(),IsOpenisfalseand the server sees its connection end.IsOpenisTrueafterClose().Full suite, run the way CI runs it (
dotnet build /p:Configuration=Release ... QuantConnect.Lean.sln, thendotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "TestCategory!=TravisExclude&TestCategory!=ResearchRegressionTests"):Passed: 39253, Failed: 0, Skipped: 100.Also tested with a brokerage whose reconnect path closes the socket from another thread while the loop reconnects. With the released
QuantConnect.Brokerages, its test that checks the server never has two connections open fails 1 run in 6 here (2 connections open at once). With this build it passed 15 runs in 15.No end-to-end run: this path only runs when a live connection's reconnect races a
Close(), which a live run can't trigger on demand. The test above reaches that exact interleaving every run.🤖 Generated with Claude Code