Repository navigation
js/watch: time-stretch the audio ring onto its target - #8058
Conversation
Quest-Claim: 2df820f42d974f08
Port the shared NetEq stretch from doc/concept/audio-jitter.md into the browser's AudioRingBuffer, so live playback converges by playing slightly faster or slower instead of running dry or skipping ahead. Every replay row, relay-mic-rtt40 included, now plays with zero player underruns. Expand departs from NetEq in both players and the spec records it: below the target it ignores the cooldown and repeats every whole period that fits at the lower threshold. NetEq-exact expand left relay-mic-rtt40 at 2.1 underruns a minute. A declared end's tail drains unstretched. The ring re-anchors only at the fast threshold or three quarters of its headroom, as native does, and counts each re-anchor as audio stats `skips`, shown in the stats panel and graded as `player_skips`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The quest's Goal becomes no audible glitch in steady state rather than zero underruns, so the listening check picks between fast expand and NetEq's exact expand. FAST_EXPAND selects it in both players; the budgets note, spec, and quests no longer hard-code a zero target. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Staging runs the read index up to a 30 ms window ahead of the playhead, so a successor truncating inside it lost its start to the predecessor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Per the maintainer's answer on #8058, start from NetEq: revert the stronger expand (every window, every whole period, lower threshold) to NetEq's preemptive expand in both players, with the spec text and corpus vectors it changed. Checking the port against decision_logic.cc found it had dropped NetEq's GetMaxDelayMs term from the high limit. Restore it in both players: the jitter estimator reports `lateness`, how late the slowest in-order frame of its two-second window arrived against the fastest, and the stretch's band above the target widens by it, as do the browser ring's write re-anchor and the CLI's incoming trim at the fast threshold. A fixed target ignores the arrivals, so its lateness is zero. The estimator corpus gains a lateness series and the stretch corpus three vectors. relay-mic-rtt40 falls from 2.1 to 1.0 player underruns a minute; the replay budgets are re-set to what NetEq measures, and the quest Goal from zero to that rate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…om/moq-dev/moq into feat/audio-jitter-stretch # Conflicts: # doc/concept/audio-jitter.md # js/watch/src/audio/stretch.ts # quest/m1/audio-jitter-target/native-stretch.md # quest/m1/audio-jitter-target/watch.md # rs/moq-audio/src/stretch.rs # test/audio-quality/budgets.json
|
Design change: both players now use NetEq's exact preemptive expand, and the decision band gets back the NetEq maximum-delay term the port had dropped.
(Written by Claude Opus 5.5) |
|
Coordinator note from
(Written by Claude Opus 5.5 with high effort) |
An emergency re-anchor moved the read index but kept the partly rendered stretch window, so the old PCM played on the new timeline and the playhead misreported it. Clear it, as moq-audio's trim discards staged output first. Dropping the startup backlog's staged window shifts each replay's later window phase, so the replay ceilings are re-measured: relay-mic-rtt40's two run-dries now each end mid-quantum and count twice (1.4 a minute), and the lag heuristic reads one or two more accelerates as skips. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Move the post-merge Nightly measurement into browser-ceilings.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lock The stretch compared the ring's current depth with the target, and that depth drains between arrivals because the next frame has not landed, not because audio is ahead of the shared clock. Expanding into every trough, with a band widened by the arrivals' lateness, settled audio a burst behind video: A/V lag p50 rose from 1 to 289 ms on bursty-aac. The ring now decides on its depth right after the latest write, less what the stretch has removed since, which is how far the playhead trails the newest arrival. Any excess accelerates, one past 20 ms fast-accelerates, and the ring never expands, so it rests on its floor right after each arrival, where the trim left it. The decode gate releases a live frame by where its output ends, so the stretch cannot draw frames through a hold. The lateness band is reverted in both players, with the estimator's lateness and its corpus fields. Native keeps NetEq's decision. The harness records the shared clock and reports av_lag_ms, budgeted exactly on every replay row, and unit tests hold that a ring never expands, rests on its floor under flushes, and underruns rather than settling later when its target is short of the spread. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… test types its playhead Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated
left a comment
There was a problem hiding this comment.
Automated review by review (OpenAI)
Reviewed 683d964 against aabd6d7. No new actionable findings.
The previously reported batching-induced audio lag is addressed in the revised browser controller: it steers from post-write depth, never expands, and gates live decoding by the output tail. Paced/batched replay regressions and A/V-lag ceilings now guard against settling later to reduce underruns. This follows the recorded maintainer decision to keep the clocks separate and accept congestion underruns. The prior staged-silence rewind is also corrected; native changes are limited to draining declared-end tails unstretched.
Verification limits: GitHub-only static review of the current diff and relevant ring/decode-gate context, compared with earlier review concerns. No tests, replays, listening, or browser/device runs were performed. Reported lag/underrun measurements are not independently verified; browser ceiling measurement and native A/V-lag measurement remain explicit follow-ups.
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes
I read the full 41-file diff at 683d9649 and re-derived the delta since the prior pullfrog head 03f8e9e5d, which is the 2026-11-08 rework of the controller.
- Browser controller: the ring now decides on
peak(its length right after the latest write, less what the stretch has removed since), never expands, fast-accelerates at 20 ms of excess, and otherwise removes only a pitch period that fits. The lateness band, the fast-expand variant, and the estimator'slatenessare reverted in both players, matching the recorded maintainer decision. - Ring staging:
#stagemoves full windows of live PCM throughStretchand stops at declared silences or a declared end's tail;timestampandlengthaccount for staged output,truncateunstages,#skipdrops the staged window with the audio it skips, and#peakis maintained across writes, drops, and resets. The declared-silence catch-up is forward-only and drops only silence. - Decode gate:
Backpressurereleases a live frame by where its output ends, so a tail past the floor is not accelerated away;insert's doc now states that PCM behind the read index is dropped. - Stretch and stats:
stretch.tsdecides on excess and runs without allocating in the render path; the corpus test grades accelerate and fast seams and holds where native expands.skipsis wired through the workletState,Audio.Stats, the stats panel, andplayer_skips_per_min. - Harness: replay and browser samples gain
clock, summaries gainav_lag_ms_p50/p95, and every replay row is re-baselined exactly withav_lagandplayer_skipswhile the chromium and native rows are untouched. - Native, docs, quests, packaging: a declared end's tail drains unstretched natively with a regression test; the concept/library docs and the epic's quest tree are updated (
watch.mddeleted,browser-ceilings.mdadded); the WebRTC notices ship with@moq/watch.
The four findings from the prior review are addressed at this head (forward-only catch-up with its regression test, no render-path subarray, the documented out-of-order drop, and the removed stale budgets sentence). I ran the changed js/watch audio suites (ring-buffer, stretch, playout, replay, render-worklet, decoder) and the harness analyze/probe suites, all green, and Check and Quest pass on this head. The replay budgets match the body's table; the browser rows remain deferred to browser-ceilings.md as the body states.
deepseek-v4.1-flash-1017 (free via Pullfrog for OSS) | 𝕏
…-target/watch # Conflicts: # test/audio-quality/budgets.json
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated
left a comment
There was a problem hiding this comment.
Automated review by review (OpenAI)
Reviewed head 77fce72 incrementally from 683d964, separating merged-main changes. No new actionable findings.
The new native A/V-lag follow-up makes the previously deferred measurement explicit: sample the presentation clock alongside the audio playhead, grade lag before changing native expansion, retain publisher/device drift compensation, and recalibrate ceilings if the decision changes. Its metric semantics match the existing browser schema. The previously reviewed browser batching/rewind fixes are unchanged; browser and native measurements remain future verification, not evidence supplied by this plan.
Verification limits: GitHub-only source/diff review, including native metrics and the shared sample schema. No builds, replays, listening or browser/device measurements performed.
…etching players main landed the browser stretch (#8058) with a browser-only player_skips count. Both players now report the same three counters: skips, the received audio they trimmed, and what the stretch accelerated away. moq play adds its skip count, and the browser ring counts trimmed and accelerated audio on the stretching ring. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Completes quest/m1/audio-jitter-target/watch.md and deletes it. The post-merge Nightly measurement moves to the new Browser ceilings quest under the same epic.
Summary
The browser's audio ring now catches up to its target by time-stretching what it plays, with the seams of WebRTC NetEq's accelerate as
doc/concept/audio-jitter.mdspecifies them and moq-dev/moq#7856 shipped natively. Wheremaindrops buffered audio to land back on the target, this plays it slightly faster. It never slows down. Audio stays on the shared clock video plays from, and underruns stay wheremainhas them: this PR spends no latency to avoid them.js/watch/src/audio/stretch.tsports the shared spec: the 4 kHz pitch search, NetEq's admission thresholds, and the crossfade seam.stretch.test.tsruns its seams against the shared PCM corpus.AudioRingBufferstages each 30 ms window of received PCM through the stretch on its way to the render quantum. The replay harness, the render worklet, and the tests all run this one path. Buffered playback, declared silences, and the tail before a declared end play unstretched.main's trim left it. Between arrivals the ring drains because the next frame hasn't landed, not because audio is ahead of the clock, so it never expands into that.Backpressurereleases a live frame once the playhead is within the floor of where that frame's output ends. Released by where it starts, its tail landed past the floor, and accelerating that away drew the next frame out sooner, playing straight through a hold.skip. The counter is in audioStats, the stats panel, and the harness (player_skips).truncate()unstages a staged window it lands inside, so the successor replaces the predecessor's unplayed audio. An emergency re-anchor drops the staged window along with the rest of the audio it skips, asmoq-audio's trim already does.sync.now(), andanalyze.tsreportsav_lag_ms(p50, p95): how far the audio playhead trails the shared clock while playing. Every replay row budgets it exactly as measured, so a change that settles audio later fails--enforce.moq-audiokeepsmain's NetEq decision and adds one thing: a declared end's tail drains unstretched, as the browser's does.@moq/watchasWEBRTC-LICENSEandWEBRTC-PATENTS.Replay rows
just test audio-quality --replays --enforce(deterministic) onffd6e03bbc, the merge oforigin/mainat7d91a5b634. Every row reads exactly what it read on683d964924. Themaincolumn ismainmeasured with the same A/V lag metric. A/V lag is in ms: negative means audio is ahead of the clock.main's on every row, and p95 is below it on every row. The ring rests one frame ahead of the newest arrival's playout, asmain's trim does on most rows, so audio leads the clock by 19 to 23 ms at p50. It no longer swings behind it the waymaindoes when a trim lands late (relay-bbbp95 10.5 → -4.2).relay-mic-rtt40rose by 2.4 a minute, andrelay-mic's tap counts 2 short quanta a minute, 2 ms of audio. That is the cost of playing slightly earlier, and it is accepted.relay-bbbdrops nothing now.local-aac's remaining 100.6 ms/min drops nothing either. Two receive stalls, 14 and 15.3 s in, leave audio 159 and 132 ms behind the clock, and the fast accelerate that brings it back within half a second steps the playhead's lag the wayskip_aheadsreads as a skip.player_skipsis 0 on every row.Browser rows were not re-run. No headless browser was launched. The chromium rows and the
*.browser.tslanes are left to CI and the nightly runner, and Browser ceilings re-measures them,av_lag_msincluded, once this lands.Tests that hold the line
ring-buffer.test.ts: a ring short of its floor plays only what it holds and runs dry (no expand). Under 100 ms flushes with a 120 ms floor, the depth right after each flush rests within one pitch period of the floor. With a 60 ms floor, every flush lands on an empty ring and underruns rather than the ring growing.replay.test.ts: at"auto", evenly paced and five-frame-flush traces play with audio never behind the shared clock and at most 25 ms ahead. A 40 ms target against 80 ms of spread underruns at least once a flush, with median lag at most one frame. Run against the previous head, the flush test fails (audio 68.9 ms behind) and so does the short-target one (36 ms behind at median, 898 underruns where this has 1200).stretch.test.ts:decidenever slows down, holds wherever native would expand, and accelerates only by a period that fits in the excess.analyze.test.ts:av_lag_msreads the playhead against the shared clock and leaves stalled samples out.budgets.json: exactav_lag_ms_p50andav_lag_ms_p95ceilings on every replay row.Public API
@moq/watch: audioout.statsgainsskips: number(additive). The stats panel shows it as "Skips".clock, and summaries gainav_lag_ms_p50andav_lag_ms_p95.doc/concept/audio-jitter.mdgains a "Browser decision" section. Native's decision is unchanged.Decisions
Decided 2026-11-06 (maintainer,
/quest-complete): "Is the goal wrong?"budgets.json's rule.Decided 2026-11-06 (maintainer, on this PR): "I think start with NetEq. 0 underruns is pretty aggressive? That being said, if the test is just +40ms RTT, are we sure our NetEq port is correct?" That settled the expand: NetEq-exact. The fast variant, its spec departure, and its corpus vectors were reverted.
Decided 2026-11-07 (maintainer,
/quest-unblock): no maintainer listening run; the Goal is judged by the replay and Nightly measurements alone. The quest's Goal drops the live MacBook Air rows and their stats-panel criterion.Decided 2026-11-07 (maintainer,
/quest-complete): ✅ fix moq-dev/moq#8071's replay regression onmainfirst, then re-run this PR's replay rows on a merge ofmain(rejected: land on top of the regression). moq-dev/moq#8218 fixed it.Decided 2026-11-07 (maintainer,
/quest-complete): ✅ land it, then measure (rejected: hold this PR until seven Nightlies re-measure the chromium ceilings).watch.mdis deleted, and the measurement is the new[S]questquest/m1/audio-jitter-target/browser-ceilings.md.Decided 2026-11-08 (maintainer, on this PR), on round 1's finding that audio settled behind video (lag p50 1 → 289 ms on
bursty-aac, -18 → 128 onlocal-aac, -22 → 154 onrelay-bbb): "fix it. obvious adding latency fixes underruns duh. Keep the audio and video clocks separate but at steady state they should be self-aligned if we correctly set the delay/latency. Update/add tests to make sure we NEVER increase latency to chase this dumb 0 underrun dragon. It's a terrible goal, we expect some underrun during congestion."main's, and A/V lag ceilings and tests guard it.How the call was applied: the ring decides on its depth right after each arrival and never expands. The lateness band this PR had added is reverted in both players, along with the estimator's
latenessin both languages and its corpus fields. The quest README records the decision.Choices made without asking
relay-bbb16.1 → 0,relay-mic6.1 → 0,local-aac12.1 → 4,relay-mic-rtt408.2 → 3.4), but only by playing about 20 ms later thanmainon those four rows. That is latency bought for underruns, so it is not taken. It is a one-line change if lag nearer 0 is wanted.maindrops it, so lag trails the clock for about 100 ms after each flush (median 9 ms, max 36) wheremainstays at 0. Underruns are the same (1200 against 1200)."auto"never picks such a target (it picks 120 ms there), and the replay test bounds the lag below one frame at median.main's decision, expand included. Only this PR's lateness band is reverted there. Native still expands on its buffered depth, so it may settle behind its clock the way the browser did, but native rows don't report A/V lag yet.native-stretch.mdnotes it as worth measuring.Validation
On
ffd6e03bbc, the merge oforigin/mainat7d91a5b634, after moq-dev/moq#8257 made replay budgets exact in both directions:quest gate just check: passes.just test audio-quality --replays --enforce: passes, every row exactly at its budget. The one conflict,bursty-aac'ssilence_share(mainre-measured 0.01, this branch 0.012), keeps this branch's value, which is what it measures.quest check --base origin/main: clean.On head
683d964924(merge oforigin/mainataabd6d7e21):quest gate nix develop --command just ci: passes. That covers 1714 Rust tests and the@moq/watch(576) and@moq/hang(549) bun tests.c5461c18fc, withclockpatched into its replay, the new lag tests fail. The flush trace plays 68.9 ms behind the clock, and the short target runs 36 ms behind at median with 898 underruns.Follow-ups
[M], under the same m1 epic:moq play --metricswrites the presentation clock so native rows gradeav_lag_ms, then native stops expanding into the gap between arrivals if it settles audio late, keeping its drift compensation.skip_aheadscannot tell a fast accelerate from a jump at its 250 ms sampling. Once the browser rows budgetplayer_skips, they could stop budgetingskip_aheads, which would stay for native.AudioContext.getOutputTimestamp(), as the quest's 2026-11-05 decision recommends.Models
Size: L
Implementer: Claude Code, claude-opus-5-5, unknown effort
Implementer: Claude Code, claude-opus-5-5, high effort
Implementer: Claude Code, claude-opus-5-5, high effort (iterate, S)
Implementer: Claude Code, claude-opus-5-5, xhigh effort (iterate, M)
Implementer: Claude Code, claude-opus-5-5, high effort (iterate, S)
Review
Rewritten by
quest reviewafter each round.01a1bbcd-79ce-7671-99a7-79ef90aa621f, round 3The only new branch-owned changes document a native A/V-lag follow-up; source-only review found no new issues or changes to the previously reviewed runtime behavior.🤖 Generated with Claude Code
(Written by Claude Opus 5.5 with high effort)