Skip to content

fix(sync): fail fast instead of spinning at 100% CPU when the dispatcher fiber dies - #3187

Closed
the-asura wants to merge 1 commit into
microsoft:mainfrom
the-asura:fix/sync-dead-dispatcher-spin
Closed

the-asura wants to merge 1 commit into
microsoft:mainfrom
the-asura:fix/sync-dead-dispatcher-spin

Conversation

@the-asura

Copy link
Copy Markdown

The bug

When the driver connection ends without going through Connection.cleanup(), any pending or subsequent sync API call spins one CPU core at 100% forever - raising nothing and logging nothing.

We hit this in production behind connect_over_cdp to a remote browser: worker threads were pinned at 100% CPU for hours, and two py-spy dumps taken 9 minutes apart showed them stopped on the same line - the waiting loop in SyncBase._sync:

while not task.done():
    self._dispatcher_fiber.switch()

strace on a stuck thread over 4 seconds: 5780 futex calls (GIL churn), zero IO syscalls - a pure userspace busy-loop.

Mechanism

  1. The dispatcher fiber's whole body is loop.run_until_complete(connection.run_as_sync()), and Connection.run() is essentially await self._transport.run().
  2. On the not-self-initiated stop path, PipeTransport.run() swallows IncompleteReadError and returns silently when _stopped is set - and even when it sets on_error_future, nothing on this path calls Connection.cleanup(). cleanup() has exactly three call sites (stop_sync, stop_async, and the child-connection handle_transport_close in _browser_type.py); none of them covers the root transport dying.
  3. So run_until_complete returns, the dispatcher greenlet ends silently, and the pending call's future is never rejected → task.done() is forever false.
  4. Switching to a dead greenlet returns immediately to its parent - which is the very greenlet doing the waiting - so the loop degenerates into while not task.done(): pass with greenlet/GIL bookkeeping per iteration. 100% CPU, unkillable (Python threads can't be terminated), invisible (no exception, no log).

Any sync call can be the victim: the one in flight when the connection died, or any later call on the same instance (e.g. context.close() in a finally: block) - create_task on the stopped loop never executes, same spin.

The fix

Check self._dispatcher_fiber.dead inside the waiting loop and raise TargetClosedError. A dead dispatcher can never complete the task, so raising there is always correct - there is no false-positive window.

The test

Recreates the failure end to end through public entry points (one _impl touch to take the silent path, mirroring what an abrupt remote disconnect produces): start a fresh sync_playwright(), set transport._stopped = True, kill the driver process, then make one more sync call. It runs in a subprocess with a timeout because without the fix the call hangs forever (spinning) rather than failing:

  • without fix: the subprocess never exits → subprocess.run(timeout=30) trips (verified locally: 3/3 parametrized runs timed out at 30s each)
  • with fix: raises TargetClosedError promptly, exits 0 (passes in ~1s)

The test needs no browser binaries (the driver dies before any launch completes).

Related reports

Same stuck-in-_dispatcher_fiber.switch() symptom without a resolution: microsoft/playwright#35928, #1549. The tracing 100%-CPU hang reported against 1.31 (fixed by reworking tracing itself) was another instance of this same waiting-loop failure mode; this PR addresses the mode itself.

Verified the affected code is unchanged on current main (and in 1.60.0 / 1.62.x releases).

…her fiber dies

When the driver connection ends without going through Connection.cleanup()
- e.g. the driver process dies, or a remote CDP connection wedges and the
transport takes the silent IncompleteReadError path - the dispatcher fiber
finishes with sync calls still pending. Their tasks can never complete, and
because switching to a dead greenlet returns immediately to its parent, the
waiting loop in SyncBase._sync degenerates into a pure userspace busy-loop:
the thread pins one core, holds the GIL, and neither raises nor logs.
Observed in production via strace (100% futex, zero IO syscalls) with py-spy
showing threads stopped on the same _sync line for the process lifetime.

Detect the dead dispatcher inside the waiting loop and raise
TargetClosedError instead. A dead fiber can never complete the task, so
raising there is always correct.

The regression test recreates the failure end to end: kill the driver on the
silent path, then make one more sync call. Without the fix the call spins
forever (the subprocess times out); with it, it raises promptly.

Co-authored-by: Claude <noreply@anthropic.com>
@daxiondi

Copy link
Copy Markdown

@microsoft-github-policy-service agree

@pavelfeldman

Copy link
Copy Markdown
Member

Thanks for the detailed investigation and repro! Closing in favor of #3224, which builds on your analysis and applies the same guard to all three sync wait loops (SyncBase._sync, EventInfo.value used by expect_*, and DisposableStub._sync).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants