From 02149dc5afae166d8bf8680eda18018eadd01980 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Mon, 9 Nov 2026 11:25:37 -0800 Subject: [PATCH 1/4] quest: plan native audio decode backpressure Co-Authored-By: Claude Opus 5.5 --- quest/m1/audio-jitter-target/README.md | 1 + .../native-backpressure.md | 37 +++++++++++++++++++ 2 files changed, 38 insertions(+) create mode 100644 quest/m1/audio-jitter-target/native-backpressure.md diff --git a/quest/m1/audio-jitter-target/README.md b/quest/m1/audio-jitter-target/README.md index b025813de04..929688579c7 100644 --- a/quest/m1/audio-jitter-target/README.md +++ b/quest/m1/audio-jitter-target/README.md @@ -64,6 +64,7 @@ native playback against its budgets. - [Native A/V lag](/quest/m1/audio-jitter-target/native-av-lag.md) - `moq play` reports how far audio trails its presentation clock, and native stops expanding into the gap between arrivals if it settles audio late - [Native rendition delay](/quest/m1/audio-jitter-target/native-rendition-delay.md) - `moq play` holds each subscribed rendition's catalog `delay` plus its target, as the browser does - [Chromium Opus stalled share](/quest/m1/audio-jitter-target/opus-stalled-share.md) - the Chromium Opus browser rows stay under the 0.01 stalled-share budget every night +- [Native backpressure](/quest/m1/audio-jitter-target/native-backpressure.md) - native playback decodes audio only when the ring has room, holding the rest of the target encoded as the browser does ## Closes diff --git a/quest/m1/audio-jitter-target/native-backpressure.md b/quest/m1/audio-jitter-target/native-backpressure.md new file mode 100644 index 00000000000..ff97d8f60e9 --- /dev/null +++ b/quest/m1/audio-jitter-target/native-backpressure.md @@ -0,0 +1,37 @@ +# [M] Native playback decodes audio only when the ring has room + +## Goal + +Native playback (`moq_audio::decode::Consumer` feeding `moq play`'s sink) +decodes an audio frame only when the ring has room below its floor. The rest +of the target delay stays encoded upstream, as in the browser, so native +memory follows the bytes actually received and the ring never grows with the +delay. Rust and JS split the target between decoded PCM and encoded media the +same way, and `moq play` holds no decoded PCM outside the ring. + +## Plan + +Planned 2026-11-09 (maintainer, /quest-complete follow-up of +[moq-dev/moq#8362](https://github.com/moq-dev/moq/pull/8362): "quest-plan +backpressure like JS does. So we don't decode audio until there's space in +the ring buffer"). ✅ m1, in this epic (rejected: m2, m3). + +The browser already works this way: `Backpressure` in +`js/watch/src/audio/buffer.ts` decides when the decode loop may decode a frame, +holding the ring at its floor of PCM and the hold encoded in the container +consumer. Native decodes promptly instead. #8362 keeps a delay past the ring's +budget in sync by parking decoded PCM in a `moq play` queue until it is due. +This quest replaces that queue with the gate, so the hold stays encoded. + +Mirror the browser's rules, including the stalled refill that starts playback +with the whole target ahead of the playhead, rather than inventing new ones. +Share names and tests with the JS side where the two can match. Keep the gate +inside moq-audio's consumer if it fits there, so the change stays additive; if +it needs a public API change, it moves to m2. + +Public API: additive if the gate stays inside `moq_audio::decode::Consumer`. +Wire: none. + +## Related + +- [Native stretch](/quest/m1/audio-jitter-target/native-stretch.md) - the native time stretch this gate feeds From 15cd12a373e8a9293d189f9d4ad0cbf60743332c Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Mon, 9 Nov 2026 11:30:22 -0800 Subject: [PATCH 2/4] quest: state the native backpressure gate's playhead and current path Rust does not split the target yet, so the Goal now says that happens once this lands. The Plan names where native decodes and queues PCM today, and that the gate's playhead comes from the sink, which the consumer does not hold. Co-Authored-By: Claude Haiku 5.5 --- .../native-backpressure.md | 21 ++++++++++++------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/quest/m1/audio-jitter-target/native-backpressure.md b/quest/m1/audio-jitter-target/native-backpressure.md index ff97d8f60e9..4c2f2164a70 100644 --- a/quest/m1/audio-jitter-target/native-backpressure.md +++ b/quest/m1/audio-jitter-target/native-backpressure.md @@ -6,8 +6,8 @@ Native playback (`moq_audio::decode::Consumer` feeding `moq play`'s sink) decodes an audio frame only when the ring has room below its floor. The rest of the target delay stays encoded upstream, as in the browser, so native memory follows the bytes actually received and the ring never grows with the -delay. Rust and JS split the target between decoded PCM and encoded media the -same way, and `moq play` holds no decoded PCM outside the ring. +delay. Once this lands, Rust splits the target between decoded PCM and encoded +media the way JS does, and `moq play` holds no decoded PCM outside the ring. ## Plan @@ -19,15 +19,20 @@ the ring buffer"). ✅ m1, in this epic (rejected: m2, m3). The browser already works this way: `Backpressure` in `js/watch/src/audio/buffer.ts` decides when the decode loop may decode a frame, holding the ring at its floor of PCM and the hold encoded in the container -consumer. Native decodes promptly instead. #8362 keeps a delay past the ring's -budget in sync by parking decoded PCM in a `moq play` queue until it is due. -This quest replaces that queue with the gate, so the hold stays encoded. +consumer. Native decodes promptly instead: `Consumer::read` decodes each packet +as it is read, and the PCM waits in `play_audio`'s queue in +`rs/moq-cli/src/play/media.rs`. #8362 keeps a delay past the ring's budget in +sync by parking that PCM there until it is due. This quest replaces that queue +with the gate, so the hold stays encoded. Mirror the browser's rules, including the stalled refill that starts playback with the whole target ahead of the playhead, rather than inventing new ones. -Share names and tests with the JS side where the two can match. Keep the gate -inside moq-audio's consumer if it fits there, so the change stays additive; if -it needs a public API change, it moves to m2. +Share names and tests with the JS side where the two can match. The gate's +input is the playhead the JS worklet reports. Native can derive it from +`Sink::buffered()` (`rs/moq-audio/src/playback/sink.rs`) and the last write's +timestamp, but `play_audio` owns that sink, not the consumer, so the consumer +needs it passed in. Keep the gate inside moq-audio's consumer if that stays +additive; if it needs a public API change, it moves to m2. Public API: additive if the gate stays inside `moq_audio::decode::Consumer`. Wire: none. From 3c6708cd111f232e1700f4c2b6af63ddfcbf2889 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 9 Nov 2026 15:33:20 -0800 Subject: [PATCH 3/4] quest: the native gate waits before decoding and reads the shared delay Co-Authored-By: Claude --- .../native-backpressure.md | 43 ++++++++++++++----- 1 file changed, 32 insertions(+), 11 deletions(-) diff --git a/quest/m1/audio-jitter-target/native-backpressure.md b/quest/m1/audio-jitter-target/native-backpressure.md index 4c2f2164a70..4de6656be4e 100644 --- a/quest/m1/audio-jitter-target/native-backpressure.md +++ b/quest/m1/audio-jitter-target/native-backpressure.md @@ -3,11 +3,14 @@ ## Goal Native playback (`moq_audio::decode::Consumer` feeding `moq play`'s sink) -decodes an audio frame only when the ring has room below its floor. The rest -of the target delay stays encoded upstream, as in the browser, so native -memory follows the bytes actually received and the ring never grows with the -delay. Once this lands, Rust splits the target between decoded PCM and encoded -media the way JS does, and `moq play` holds no decoded PCM outside the ring. +decodes an audio frame when the browser's `Backpressure` would: while playing, +once the playhead is within the ring's floor of it, and, live, once the media +queued behind it covers the part of the target held encoded, so a backlog past +the target still reaches the ring to be caught up. The rest of the target +delay stays encoded upstream, so native memory follows the bytes actually +received and the ring never grows with the delay. Once this lands, Rust splits +the target between decoded PCM and encoded media the way JS does, and +`moq play` holds no decoded PCM outside the ring. ## Plan @@ -27,12 +30,30 @@ with the gate, so the hold stays encoded. Mirror the browser's rules, including the stalled refill that starts playback with the whole target ahead of the playhead, rather than inventing new ones. -Share names and tests with the JS side where the two can match. The gate's -input is the playhead the JS worklet reports. Native can derive it from -`Sink::buffered()` (`rs/moq-audio/src/playback/sink.rs`) and the last write's -timestamp, but `play_audio` owns that sink, not the consumer, so the consumer -needs it passed in. Keep the gate inside moq-audio's consumer if that stays -additive; if it needs a public API change, it moves to m2. +Share names and tests with the JS side where the two can match. + +The gate waits before decoding, never before reading. `Consumer::read` stamps +each packet's arrival for the jitter estimate (`target.observe` in +`rs/moq-audio/src/decode/consumer.rs`), and `Consumer::delay()` documents that +reads paced to the speaker make the estimate measure the pacing. Keep reading +and observing encoded packets as they arrive and hold them encoded, as +`js/watch` reads a frame and only then awaits `ring.wait()`. + +The gate has two inputs, and `play_audio` owns both, so the consumer needs +them passed in: + +- The playhead the JS worklet reports. Native can derive it from + `Sink::buffered()` (`rs/moq-audio/src/playback/sink.rs`) and the last + write's timestamp. +- The shared playout delay: the catalog `delay` plus the target, the largest + among the subscribed renditions, which the browser passes as + `sync.out.delay` and `play_audio` already holds the sink to through + `presentation.due`. `Consumer::delay()` is this track's jitter target alone; + a gate on it would decode at a depth the sink has already outrun and could + not replace #8362's hold. + +Keep the gate inside moq-audio's consumer if that stays additive; if it needs +a public API change, it moves to m2. Public API: additive if the gate stays inside `moq_audio::decode::Consumer`. Wire: none. From fd940aed65c0c5428765153303237adb332b9470 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 9 Nov 2026 15:55:00 -0800 Subject: [PATCH 4/4] quest: the stretch model requires native backpressure; correct three plan statements Co-Authored-By: Claude --- quest/m1/audio-jitter-target/grading.md | 13 ++++---- quest/m1/audio-jitter-target/native-av-lag.md | 7 +++-- .../native-backpressure.md | 2 +- quest/m1/audio-jitter-target/stretch.md | 30 +++++++++---------- 4 files changed, 27 insertions(+), 25 deletions(-) diff --git a/quest/m1/audio-jitter-target/grading.md b/quest/m1/audio-jitter-target/grading.md index a92db0201bf..07c7f60a16c 100644 --- a/quest/m1/audio-jitter-target/grading.md +++ b/quest/m1/audio-jitter-target/grading.md @@ -13,7 +13,9 @@ before the model lands, `test/audio-quality` grades: regression. - Dropped audio is the player's own count (`trimmed_ms`, `player_skips`). Speeding up is not a skip. -- No gate is fixed at zero underruns or zero stalls. +- No row's budget is fixed at zero underruns or zero stalls. Push-to-talk's + own zero gate, in `ptt-analyze.ts`, goes with + [Push-to-talk latency](/quest/m1/push-to-talk-latency.md), not here. - A congestion row, where the path stalls for longer than the target and then delivers the backlog, reports the audio dropped and how long audio takes to return to the clock. @@ -56,10 +58,11 @@ conformance corpus measures the target. Recommendations: -- Replay rows stay exact. Rewrite the note to say which direction is a - regression: a steady-state p50 lag that rises, or audio the player - trimmed. A p95 lag that rises through a stall, or an underrun count that - moves, is not one by itself. +- Replay rows stay exact, so any change to one still fails until it is + re-measured. Rewrite the note to say which re-measured moves are + regressions to fix rather than budget: a steady-state p50 lag that rises, + or audio the player trimmed. A p95 lag that rises through a stall, or an + underrun count that moves, is re-measured in. - Add the congestion case twice: a shaper profile that queues until packets drop, per the repository's impairment rule, and a replay trace where media trickles in late after a stall, which the audit of #8360 asked for. diff --git a/quest/m1/audio-jitter-target/native-av-lag.md b/quest/m1/audio-jitter-target/native-av-lag.md index 0f583476ee9..3da047e0a0c 100644 --- a/quest/m1/audio-jitter-target/native-av-lag.md +++ b/quest/m1/audio-jitter-target/native-av-lag.md @@ -12,9 +12,10 @@ playback decision does not change here. Decided 2026-11-09 (maintainer, /quest-complete on [moq-dev/moq#8361](https://github.com/moq-dev/moq/pull/8361)): ✅ keep the PR's metrics half and revert its playback changes (rejected: land its stretch -keyed to the lag at arrival). That stretch sped up into a thin buffer, taking -AAC `step` from 0 to 614 underruns a minute, and removed native's only -slow-down. It was built on the rule the +keyed to the lag at arrival). That stretch held audio on a clock that goes +stale after a path-delay step, taking AAC `step` from 0 to 614 underruns a +minute, where `main`'s slow-down had hidden the stale clock by settling +behind it. It was built on the rule the [epic](/quest/m1/audio-jitter-target/README.md) replaced the same day, so the PR's recorded decisions "accept AAC step underruns" and "p50 within one pitch period" go with it. diff --git a/quest/m1/audio-jitter-target/native-backpressure.md b/quest/m1/audio-jitter-target/native-backpressure.md index 4de6656be4e..8856b759e61 100644 --- a/quest/m1/audio-jitter-target/native-backpressure.md +++ b/quest/m1/audio-jitter-target/native-backpressure.md @@ -60,4 +60,4 @@ Wire: none. ## Related -- [Native stretch](/quest/m1/audio-jitter-target/native-stretch.md) - the native time stretch this gate feeds +- [Stretch model](/quest/m1/audio-jitter-target/stretch.md) - requires this gate, then changes the stalled refill mirrored here to a small cushion in both players diff --git a/quest/m1/audio-jitter-target/stretch.md b/quest/m1/audio-jitter-target/stretch.md index 05f10a3c15f..c8395c11e11 100644 --- a/quest/m1/audio-jitter-target/stretch.md +++ b/quest/m1/audio-jitter-target/stretch.md @@ -52,9 +52,10 @@ Decisions for this quest: both are worth taking. - Order: ✅ after native backpressure (rejected: this first; one quest for both). The level this rule reads is everything received ahead of the - playhead, decoded or still encoded, and the backlog while audio trails - sits encoded upstream in the browser. One mirrored rule needs both players - to split the buffer the same way. + playhead, decoded or still encoded. The browser keeps the part of the + target above its ring's floor encoded, and a buffered player its whole + lookahead, while native decodes everything at once. One mirrored rule needs + both players to split the buffer the same way. - Clock step: ✅ deferred (rejected: plan it now; drop it). The shared clock's reference only moves earlier, so after a lasting path-delay step audio would settle behind a stale clock. Report what the `step` rows' lag @@ -72,10 +73,15 @@ What the audit of `main` found (2026-11-09, `10772522a5`): expands, fast-accelerates at 20 ms of excess, stays silent after an underrun until the whole target refills, parks on a target rise, and turns the stretch off in buffered mode. -- [moq-dev/moq#8361](https://github.com/moq-dev/moq/pull/8361) sped up into a - thin buffer and took AAC `step` from 0 to 614 underruns a minute. A - speed-up is admitted only while the buffer stays above the target by the - lag it removes. +- [moq-dev/moq#8361](https://github.com/moq-dev/moq/pull/8361) held native + audio on the clock and took AAC `step` from 0 to 614 underruns a minute. + Its own analysis: the clock's reference never moves later, so after the 5 + to 60 ms step the clock-aligned depth sits about 55 ms short of the + estimator's target, less than AAC's 160 ms PES spacing, and `main` had + hidden that by settling a burst behind. It already admitted a speed-up only + inside the excess, so that rule alone prevents nothing here. Under this + model the slow-down lets audio settle behind the stale clock rather than + underrun, which is the lag the deferred clock decision is about. Recommendations: @@ -117,18 +123,10 @@ decision now runs a filter. Public API: none expected. Wire: none. -## Waiting - -- moq-dev/moq: [moq-dev/moq#8380](https://github.com/moq-dev/moq/pull/8380) - merges, planning native backpressure - (`/quest/m1/audio-jitter-target/native-backpressure.md`). Then list that - quest under Required and delete this item. It asks native to mirror the - browser's stalled refill to the full target; this quest then changes the - refill in both. - ## Required - [The audio harness grades the stretch model](/quest/m1/audio-jitter-target/grading.md) - the gates that pushed toward zero underruns and zero lag are gone before this is measured against them +- [Native backpressure](/quest/m1/audio-jitter-target/native-backpressure.md) - native holds its ring at a floor and the rest of the target encoded, as the browser does, so one rule reads the same buffer in both ## Closes