Skip to content

fix(rtc): keep AVSynchronizer sending video after reset() mid-frame - #871

Open
ANDROIDHASSAN wants to merge 1 commit into
livekit:mainfrom
ANDROIDHASSAN:fix/av-sync-reset-mid-frame
Open

ANDROIDHASSAN wants to merge 1 commit into
livekit:mainfrom
ANDROIDHASSAN:fix/av-sync-reset-mid-frame

Conversation

@ANDROIDHASSAN

Copy link
Copy Markdown
Contributor

AVSynchronizer._capture_video sends each frame inside async with self._fps_controller, and most of its time is spent in wait_next_process(), sleeping until the frame's slot. AVSynchronizer.reset() clears the controller's schedule (_next_frame_time = None). Called during that sleep, it made the controller's after_process() hit assert self._next_frame_time is not None, which ended the capture task. Nothing logs that, and the video queue is bounded (one frame at 10 fps with the default 100 ms), so every later push() of a video frame waits forever and wait_for_playout() never returns.

A script with fake audio and video sources, 10 fps: push two frames, call reset() 30 ms later (while the task waits for the second frame's slot), push a third:

task done: True exception: AssertionError('wait_next_process must be called first')
frames captured: 2 of 3

after_process() now starts a new schedule from the current time when the controller was reset after wait_next_process(), the same thing wait_next_process() does for the first frame after a reset.

Tests in tests/rtc/test_synchronizer.py, with fake sources:

  • test_av_synchronizer_keeps_sending_after_a_reset_mid_frame: the fake video source calls reset() from capture_frame, which always falls between wait_next_process() and after_process(). All three frames must be captured and wait_for_playout() must return. On main it times out (the task died with the AssertionError).
  • test_fps_controller_schedules_from_now_after_a_reset_mid_frame: on main the AssertionError; now the next frame is one interval after the call.

tests/rtc and livekit-rtc/tests here, without the FFI library: the same 17 failures and 17 errors before and after, from the missing native library; 131 passed before, 133 now. ruff check, ruff format --check (0.15.4) pass; mypy (1.19.1) on livekit-rtc reports the same 4 errors in jupyter.py before and after.

Not changed: any other exception in _capture_video, for example from VideoSource.capture_frame after the source is closed, still ends the task without a log and leaves push() waiting.

The capture task waits for each frame's slot inside the FPS controller.
reset() during that wait cleared the schedule, so after_process() hit its
assert, the task ended silently, and pushes blocked once the video queue
filled. after_process() now starts a new schedule from the current time.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant