Skip to content

Abort the websocket connection a Close() leaves open - #10033

Merged
Martin-Molinero merged 1 commit into
masterfrom
kronos/bug-stream-login-denied-token
Nov 5, 2026
Merged

Martin-Molinero merged 1 commit into
masterfrom
kronos/bug-stream-login-denied-token

Conversation

@quantconnect-kronos

@quantconnect-kronos quantconnect-kronos Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Problem

WebSocketClientWrapper.Close() cancels the connection loop and relies on the cancelled ReceiveAsync to abort the connection. If the cancellation lands after ConnectAsync but before the first ReceiveAsync, while the Open handlers run, the loop sees the cancellation, skips the receive and ends. Nothing aborts the connection:

  • Close() calls CloseOutputAsync with the token it just cancelled, so no close frame goes out.
  • No receive was pending, so the cancellation doesn't abort the socket.
  • _client is only disposed by the next HandleConnection.

So after Close() returns, IsOpen is still true and the server still has the connection open. It stays that way until the next Connect(), or for good if none follows (shutdown). It's easy to hit when a reconnect, from the loop or from the server dropping the connection, overlaps a Close() from another thread. Close() waits on _locker while ConnectAsync holds it, so it gets the lock just as OnOpen starts.

Change

HandleConnection keeps the last ClientWebSocket it created, and aborts it once the loop ends. The loop only ends once it's cancelled, so this runs only on Close(). By then, in every other case, the connection is already aborted or closed, and Abort() does nothing. It uses the loop's own local, not the _client field: if Close() gave up waiting (5 s), a later Connect() may already have put a new connection in _client. Nothing else changes.

Tests

WebSocketClientWrapperTests.CloseWhileTheOpenHandlerRunsClosesTheConnection: a local websocket server, and an Open handler that holds the connection thread until Close() has been called and for 1 s after it. That hits the interleaving every time. The test checks that after Close(), IsOpen is false and the server sees its connection end.

  • Without the fix it fails every time: IsOpen is True after Close().
  • With the fix it passes.

Full suite, run the way CI runs it (dotnet build /p:Configuration=Release ... QuantConnect.Lean.sln, then dotnet test ./Tests/bin/Release/QuantConnect.Tests.dll --filter "TestCategory!=TravisExclude&TestCategory!=ResearchRegressionTests"): Passed: 39253, Failed: 0, Skipped: 100.

Also tested with a brokerage whose reconnect path closes the socket from another thread while the loop reconnects. With the released QuantConnect.Brokerages, its test that checks the server never has two connections open fails 1 run in 6 here (2 connections open at once). With this build it passed 15 runs in 15.

No end-to-end run: this path only runs when a live connection's reconnect races a Close(), which a live run can't trigger on demand. The test above reaches that exact interleaving every run.

🤖 Generated with Claude Code

Close() cancels the connection loop and relies on the cancelled receive
to abort the connection. When the cancellation lands before the first
receive, while the Open handlers run, the loop ends without receiving
and the connection stays open after Close() returns, until the next
Connect() replaces it. The loop now aborts the last connection it opened
when it ends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@quantconnect-kronos quantconnect-kronos Bot added the kronos:develop:working develop: in progress label Nov 4, 2026
@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added kronos:review:queued review: queued kronos:review:working review: in progress and removed kronos:review:queued review: queued labels Nov 4, 2026
@quantconnect-kronos-reviewer
quantconnect-kronos-reviewer Bot marked this pull request as ready for review November 4, 2026 21:32

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of a875f3c

The fix is correct and as small as it can be. I traced HandleConnection and Close() for every way the loop can end:

  • Normal Close(): the loop is blocked in ReceiveAsync. Cancelling it already aborts the socket, so the new client?.Abort() does nothing.
  • Close() during the error back-off (connectionCts.Token.WaitHandle.WaitOne): the failed client is already faulted or aborted. Abort() on it does nothing.
  • Close() between ConnectAsync and the first ReceiveAsync (the bug): the loop now aborts the connection before _taskConnect completes. Close() waits on that task, so the socket is aborted before Close() returns and before OnClose runs. This also covers a Close() called from inside an Open/Message handler on the connection thread, which left the socket open the same way before.
  • Close() gives up after 5 s: aborting the loop's own local, not _client, is right. A later Connect() may already have stored a new connection in _client.

The only visible difference for callers is that Closed handlers now see State Aborted, where they used to see Open in the racy case. I checked the brokerages that subscribe to Closed (for example Binance's handler, which stops its session and keep-alive timer). None of them reads the socket state there.

Evidence

  • CI is green on this commit.
  • I reproduced the test both ways. With only Brokerages/WebSocketClientWrapper.cs reverted to master, CloseWhileTheOpenHandlerRunsClosesTheConnection fails with Expected: False, But was: True (IsOpen after Close()). With the fix it passes 3 times in 3. The test holds the Open handler until Close() has cancelled the loop, so it hits the interleaving every run.
  • I also ran a brokerage's reconnect tests against this build: 12 runs of the suite, 6 at a time. Its check that the server never has two connections open passed every time. Against the released QuantConnect.Brokerages that check is flaky.
  • No end-to-end run is described. This path only runs when a brokerage's live connection races a Close() against a reconnect, which a live run can't trigger on demand. The deterministic test reaches that exact interleaving, so I accept it. The description could say so in one line.

Other repositories

Brokerages that use WebSocketClientWrapper get the fix with the next QuantConnect.Brokerages package. None needs a code change.

Risk

I set risk:high. The diff is small, backwards compatible, and does nothing outside the racy path. But it changes WebSocketClientWrapper, which every websocket brokerage uses for its live connection, so its blast radius is wide.

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added risk:high High risk and removed kronos:review:working review: in progress labels Nov 4, 2026
@quantconnect-kronos

Copy link
Copy Markdown
Contributor Author

Added the line on end-to-end testing to the description: a live run can't trigger the race on demand, and the test reaches the exact interleaving every run. No code change.

🤖 Generated with Claude Code

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added kronos:review:queued review: queued kronos:review:working review: in progress and removed kronos:review:queued review: queued labels Nov 4, 2026

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of a875f3c

The code is the same as when I approved it. The only change is the description: it now says why there's no end-to-end run (a live run can't trigger the race on demand, and the test reaches that exact interleaving every run). That answers the one note in my last review. CI is still green on this commit.

I'm approving again. The brokerage pull request on the same branch isn't ready yet, but that's because it's waiting for this fix to be merged and released. Holding this one back would block both. This fix is correct on its own.

Other repositories

Brokerages that use WebSocketClientWrapper get the fix with the next QuantConnect.Brokerages package. None needs a code change.

Risk

risk:high stays. The diff is small and backwards compatible, but it changes WebSocketClientWrapper, which every websocket brokerage uses for its live connection, so its blast radius is wide.

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot removed the kronos:review:working review: in progress label Nov 4, 2026
@quantconnect-kronos quantconnect-kronos Bot added kronos:develop:idle develop: paused and removed kronos:develop:working develop: in progress labels Nov 4, 2026
@Martin-Molinero
Martin-Molinero merged commit b0fb3ed into master Nov 5, 2026
8 checks passed
@Martin-Molinero
Martin-Molinero deleted the kronos/bug-stream-login-denied-token branch November 5, 2026 13:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kronos:develop:idle develop: paused risk:high High risk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant