Skip to content

fix: prevent deadlock when tool call is cancelled during on_messages_stream - #8182

Open
MOHAMMED WASIM KHAN (wasim-builds) wants to merge 4 commits into
microsoft:mainfrom
wasim-builds:fix/7956-tool-call-cancellation-deadlock
Open

MOHAMMED WASIM KHAN (wasim-builds) wants to merge 4 commits into
microsoft:mainfrom
wasim-builds:fix/7956-tool-call-cancellation-deadlock

Conversation

@wasim-builds

Copy link
Copy Markdown

What happened

Cancelling a CancellationToken while a tool call is in flight permanently hangs AssistantAgent.on_messages_stream. The stream never ends, and any enclosing team's stop_when_idle() also hangs.

Root cause

StaticWorkbench.call_tool and StaticStreamWorkbench.call_tool_stream catch Exception, but asyncio.CancelledError inherits from BaseException in Python 3.8+. When the token is cancelled, CancelledError propagates out of the workbench, into asyncio.gather inside _execute_tool_calls, which raises before the end-of-stream sentinel (None) is put in the queue. The consumer loop has no other termination path, so it blocks on stream.get() forever.

Fix

Changed _execute_tool_calls to:

  • use return_exceptions=True in asyncio.gather so one failing tool call doesn't abort the others
  • move stream_queue.put_nowait(None) into a finally block so the sentinel is always sent
  • convert any exception results back into FunctionExecutionResult with is_error=True so downstream code sees a normal error result instead of a raw exception

Testing

No new tests in this PR. I manually verified by running an AssistantAgent with a slow tool, cancelling mid-flight, and confirming the stream terminates cleanly instead of hanging.

Fixes #7956

Wasim added 2 commits September 2, 2026 09:59
Previously, _apply_filter returned messages in per_source config order
instead of chronological order, breaking the documented conversation
timeline. Now messages are returned in their original order regardless
of filter configuration.

Fixes microsoft#7971
…stream

When a CancellationToken is cancelled while a tool call is in flight,
CancelledError propagates out of the workbench (not caught by except
Exception), which causes asyncio.gather to raise before the end-of-
stream sentinel is put in the queue. The consumer loop then blocks
forever on stream.get().

Changed _execute_tool_calls to:
- use return_exceptions=True in asyncio.gather
- move stream_queue.put_nowait(None) to a finally block
- convert any exception results to FunctionExecutionResult with is_error=True

Fixes microsoft#7956
Copilot AI lite review requested due to automatic review settings September 4, 2026 06:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@wasim-builds

Copy link
Copy Markdown
Author

Ready for review when convenient. Happy to iterate on feedback.

@Ultronen Ultronen (Ultronen) left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found one cancellation regression in the new parallel tool execution path.

)
for call in function_calls
],
return_exceptions=True,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Preserve cancellation instead of turning it into a tool error

asyncio.CancelledError is a BaseException, and this is exactly what a linked tool future raises when cancellation_token.cancel() is called. With return_exceptions=True it lands in results, and the BaseException branch below converts it into a normal FunctionExecutionResult; on_messages/on_messages_stream then yields a tool error and completes normally, or can continue another model iteration, instead of honoring cancellation. This contradicts the documented contract that cancelling the token makes the on_messages await raise CancelledError. I reproduced this on 7491748 with a blocked FunctionTool: after token.cancel(), a regression test expecting CancelledError fails with DID NOT RAISE; re-raising CancelledError after gather while keeping the finally sentinel makes it pass. Please preserve cancellation propagation and only normalize ordinary tool failures.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Ultronen (@Ultronen)! Updated to ensure cancellation is fully preserved: we check cancellation_token.is_cancelled() and re-raise asyncio.CancelledError out of _execute_tool_calls so that cancellation propagates immediately rather than being converted into a FunctionExecutionResult tool error. Ordinary tool failures continue to be caught and normalized.

@wasim-builds

Copy link
Copy Markdown
Author

Hi maintainers, thank you for the review feedback on cancellation handling! The fix ensures asyncio.CancelledError is properly preserved and propagated during tool execution in on_messages_stream, preventing consumer deadlocks when tasks are cancelled.

I've really enjoyed working on AutoGen's multi-agent runtime and streaming architecture, and I'd love to continue contributing to the framework. I'm also available for freelance/contract projects, full-time engineering roles, or joining the team/org as an active contributor. Feel free to view my profile and open-source contributions at https://github.com/wasim-builds.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cancelling an in-flight tool call deadlocks AssistantAgent.on_messages_stream

3 participants