Repository navigation
quest: plan native audio decode backpressure #8380
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
02149dc
quest: plan native audio decode backpressure
kixelated 15cd12a
quest: state the native backpressure gate's playhead and current path
kixelated 3c6708c
quest: the native gate waits before decoding and reads the shared delay
kixelated 762ab20
Merge main into quest-plan/native-audio-backpressure
kixelated fd940ae
quest: the stretch model requires native backpressure; correct three …
kixelated File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,63 @@ | ||
| # [M] Native playback decodes audio only when the ring has room | ||
|
|
||
| ## Goal | ||
|
|
||
| Native playback (`moq_audio::decode::Consumer` feeding `moq play`'s sink) | ||
| decodes an audio frame when the browser's `Backpressure` would: while playing, | ||
| once the playhead is within the ring's floor of it, and, live, once the media | ||
| queued behind it covers the part of the target held encoded, so a backlog past | ||
| the target still reaches the ring to be caught up. The rest of the target | ||
| delay stays encoded upstream, so native memory follows the bytes actually | ||
| received and the ring never grows with the delay. Once this lands, Rust splits | ||
| the target between decoded PCM and encoded media the way JS does, and | ||
| `moq play` holds no decoded PCM outside the ring. | ||
|
|
||
| ## Plan | ||
|
|
||
| Planned 2026-11-09 (maintainer, /quest-complete follow-up of | ||
| [moq-dev/moq#8362](https://github.com/moq-dev/moq/pull/8362): "quest-plan | ||
| backpressure like JS does. So we don't decode audio until there's space in | ||
| the ring buffer"). ✅ m1, in this epic (rejected: m2, m3). | ||
|
|
||
| The browser already works this way: `Backpressure` in | ||
| `js/watch/src/audio/buffer.ts` decides when the decode loop may decode a frame, | ||
| holding the ring at its floor of PCM and the hold encoded in the container | ||
| consumer. Native decodes promptly instead: `Consumer::read` decodes each packet | ||
| as it is read, and the PCM waits in `play_audio`'s queue in | ||
|
pullfrog[bot] marked this conversation as resolved.
Resolved
|
||
| `rs/moq-cli/src/play/media.rs`. #8362 keeps a delay past the ring's budget in | ||
| sync by parking that PCM there until it is due. This quest replaces that queue | ||
| with the gate, so the hold stays encoded. | ||
|
coderabbitai[bot] marked this conversation as resolved.
Resolved
|
||
|
|
||
| Mirror the browser's rules, including the stalled refill that starts playback | ||
| with the whole target ahead of the playhead, rather than inventing new ones. | ||
| Share names and tests with the JS side where the two can match. | ||
|
|
||
| The gate waits before decoding, never before reading. `Consumer::read` stamps | ||
| each packet's arrival for the jitter estimate (`target.observe` in | ||
| `rs/moq-audio/src/decode/consumer.rs`), and `Consumer::delay()` documents that | ||
| reads paced to the speaker make the estimate measure the pacing. Keep reading | ||
| and observing encoded packets as they arrive and hold them encoded, as | ||
| `js/watch` reads a frame and only then awaits `ring.wait()`. | ||
|
|
||
| The gate has two inputs, and `play_audio` owns both, so the consumer needs | ||
| them passed in: | ||
|
|
||
| - The playhead the JS worklet reports. Native can derive it from | ||
| `Sink::buffered()` (`rs/moq-audio/src/playback/sink.rs`) and the last | ||
| write's timestamp. | ||
| - The shared playout delay: the catalog `delay` plus the target, the largest | ||
| among the subscribed renditions, which the browser passes as | ||
| `sync.out.delay` and `play_audio` already holds the sink to through | ||
| `presentation.due`. `Consumer::delay()` is this track's jitter target alone; | ||
| a gate on it would decode at a depth the sink has already outrun and could | ||
| not replace #8362's hold. | ||
|
|
||
| Keep the gate inside moq-audio's consumer if that stays additive; if it needs | ||
| a public API change, it moves to m2. | ||
|
|
||
| Public API: additive if the gate stays inside `moq_audio::decode::Consumer`. | ||
| Wire: none. | ||
|
|
||
| ## Related | ||
|
|
||
| - [Stretch model](/quest/m1/audio-jitter-target/stretch.md) - requires this gate, then changes the stalled refill mirrored here to a small cushion in both players | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Script executed:
Repository: moq-dev/moq
Length of output: 11383
Script executed:
Repository: moq-dev/moq
Length of output: 23469
Do not treat every p50 rise as a regression.
The replay p50 values are negative. A rise toward zero can move audio closer to the model’s shared-clock resting point. Judge re-measured changes against clock alignment, not only the previous value.