Skip to content

test(audio-quality): real-microphone baseline for time stretch, from the kixtop playout run - #7679

Merged
kixelated merged 10 commits into
mainfrom
quest/m1/audio-jitter-target/watch
Nov 3, 2026
Merged

kixelated merged 10 commits into
mainfrom
quest/m1/audio-jitter-target/watch

Conversation

@kixelated

@kixelated kixelated commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Works quest/m1/audio-jitter-target/watch.md. This is the measurement checkpoint for the browser's measured playout target. It holds the 2026-11-03 manual run on the maintainer's MacBook Air (kixtop), a real-microphone relay-mic trace, and a CI baseline that a time-stretch change will move deterministically. The quest is rewritten to the time-stretch work that closes its Goal.

Decisions

  • Goal (2026-11-03, maintainer): zero underruns after convergence is unreachable at QUANTILE = 0.95 without concealment.

    • Restate the Goal to the measured rate
    • Raise the percentile
    • ✅ Pull time stretch into m1 as this quest's fix
  • Implement here, or land as a checkpoint? I'm landing this as the checkpoint, for three reasons:

    • The stretch is blocked. The m2 Time stretch quest Required A/V clock, the clock the stretch converges toward, and A/V clock itself waits on Capped budgets (seventh nightly due about 2026-11-04).
    • The traces need deep expansion, not a gentle trim. A frame past the cushion needs a local slowdown of 11–31% at the median and up to 38–71% at worst, for up to 44–81 ms. That is NetEq-style preemptive expand by pitch-period repetition, plus accelerate, plus a listening check. The old Plan's "a few percent per quantum" bound cannot meet the Goal.
    • With the tests in both Rust and JS that you described, that is past [M].
  • Safari +40 ms (2026-11-03, maintainer): ✅ the by-hand Safari baseline moves out of this PR into the quest's Waiting section, on kixtop with Network Link Conditioner or sudo dnctl/pfctl, so this checkpoint lands without it.

  • Stretch design (2026-11-03, maintainer): ✅ copy what WebRTC's NetEq does. watch.md's Plan now mirrors accelerate and preemptive expand: whole pitch periods found by autocorrelation, removed or repeated with a cross-fade. It cites time_stretch.cc/.h, accelerate.cc, preemptive_expand.cc and decision_logic.cc/.h at WebRTC a057d00e46, with NetEq's thresholds as the starting point:

    • a 0.9 correlation (0.5 for fast accelerate);
    • 2.5–15 ms pitch lags at 4 kHz;
    • the low limit at the target, and the high limit at the target plus the arrival history's maximum delay plus 20 ms;
    • at most one stretch every 5 ticks of 10 ms.

    The Plan also records where it must depart from NetEq. NetEq conceals a missing packet with Expand, which is outside this quest, and at NetEq's cadence preemptive expand slows playback by about a quarter at most. That covers the median late frame (11–31%) but not the 38–71% tail, so the tail is decided from replay numbers after the port.

  • Tree: I folded the m2 Time stretch quest into this one rather than moving it. Moving it would leave a Watch ↔ Time stretch dependency cycle, and folding keeps one quest on this claim. Watch now Requires A/V clock, inherited from Time stretch, and is [L].

Changes

Since the last review on this PR

  • Baseline replay row. test/audio-quality/traces/relay-mic-rtt40.json is the real-microphone Chrome row at +40 ms RTT (180 s, minimum RTT 47 ms), converted by the README's recipe. It is budgeted exactly as measured: 7.9 player underruns/min.
  • player_underruns metric. This is the player's own counter, the one the stats panel shows: audio.out.stats.underruns in the page probe, and ring.underruns yielded by js/watch/src/audio/replay.ts.
    • Why it's needed: the output tap files each run-dry re-stall as stalled time, so the existing underruns key read 0.3/min on this row against 7.9 from the player.
    • Every replay row budgets it exactly as measured. Browser rows report it unbudgeted (their ceilings come only from nightly runs), and native rows leave it unset.
    • It is the first column of the grade table.
  • Tests.
    • The js/watch replay counts a single run-dry as one underrun, and a clean trace as none.
    • The analyzer counts the counter's rise across the graded window.
    • The probe test's player mock gains its stats signal.
  • Quest tree. quest/m2/watch-audio-time-stretch.md is folded into watch.md and deleted. Its links in the epic README, av-clock.md, watch-worker.md, and the m1 and m2 indexes now point to Watch.

Earlier on this PR

  • --mic-broadcast <name> for just test audio-quality-record --lanes relay-mic records a hand-published broadcast.
  • relay-mic is re-recorded from the MacBook Air's built-in microphone through Chrome 156, with its replay ceilings re-measured (target p95 60 → 80 ms, silence 0 → 0.001).

Baseline (the 2026-11-03 run)

Setup

  • Publisher: <moq-publish> in Chrome 156, capturing the built-in microphone.
  • Watchers: the harness page in a second Chrome 156 profile, and in Safari 26.6.2.
  • Added RTT: a userspace UDP relay on the laptop adding 20 ms each way, reached through Chrome's --host-resolver-rules. Safari can't be redirected that way and negotiated WebSocket, so its rows ran without added RTT.
  • Counting: "Underruns" is the stats panel's counter after the first 30 s.

Results

Row Transport RTT min Underruns after 30 s Skip-aheads after 30 s Auto delay p50 / max
Chrome, direct, 20 ms groups WebTransport 5 ms 9 in 88 s (6.1/min) 0 40 / 60 ms
Chrome, +40 ms, 20 ms groups WebTransport 47 ms 26 in 149 s (10.5/min) 1 (43 ms) 100 / 120 ms
Chrome, +40 ms, 100 ms groups WebTransport 47 ms 9 in 149 s (3.6/min) 0 100 / 120 ms
Safari, direct, 20 ms groups WebSocket 5 ms 5 in 118 s (2.5/min) 0 120 / 140 ms
Safari, direct, 100 ms groups WebSocket 5 ms 12 in 119 s (6.1/min) 0 120 / 140 ms

Cause. Replayed, each row's arrivals reproduce its live re-stalls and short quanta almost exactly. Each run-dry is one frame arriving later than the cushion the 0.95 quantile sizes, which happens in about 5% of 500 ms intervals.

Encoder. Lag is p50 0.9 ms and p99 1.1 ms, and drift is under 0.04 ms/s at both group sizes. The moq-dev/moq#3477 lag did not recur.

Replays on this head (--enforce, within budget)

Row Player underruns/min
relay-mic-rtt40 7.9
relay-mic 6.1
local-aac 12.1
relay-bbb 12.1
bursty-aac 4.4

Choices made without asking

  • Fold, not move: the quest is folded into Watch rather than moved, as described above.
  • Native out of the Goal: time stretch stays browser-only, as the m2 quest scoped it. Native parity in rs/moq-audio is suggested as a follow-up rather than added to this Goal.
  • Metric scope: player_underruns is budgeted only on the deterministic replay rows. Browser ceilings follow the nightly re-measure rule in the budgets note.
  • Baseline trace: the 20 ms-group Chrome +40 ms capture (the publisher's default) is the checked-in baseline, not the 100 ms one.

Review

  • Codex gpt-6.1-sol, medium effort, head 92876c9525501b2fc715eb1cd72380800bdb4cd3 against origin/main (7b4f531650), round 2 of the current session. 0 findings.
  • The session's round 1 on 17bee77d6a, and earlier rounds on 35633722db and 25da08bac3, also had 0 findings.

Checks

  • just check origin/main passes on the final head: biome, js check and test, markdown, gh, quest, shell, text, toml, nix, justfile.
  • quest check --base origin/main passes (577 documents).
  • just test audio-quality --replays --enforce is within budget.
  • @moq/audio-quality's CLI tests time out at 5 s when run bare on a host at load 30+. They pass under quest gate.

Suggested next step: /quest-merge.

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

kixelated and others added 2 commits November 2, 2026 22:48
The watch quest's manual proof re-records `relay-mic` with a real
microphone, but the recorder always published Chromium's fake capture
device. `--mic-broadcast <name>` records a broadcast already publishing
on the public relay's anon path instead.

Rewrites the quest to what remains: the replay rows re-measured on
today's main (unchanged), the nightly browser rows' steady-state stalls
as evidence for the manual run, and a Waiting item for that run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title js/watch: prove the measured playout target in a real browser test(audio-quality): record relay-mic from a real microphone; watch quest waits on the manual run Nov 3, 2026
kixelated and others added 2 commits November 3, 2026 10:27
…kixtop run

The manual run on a MacBook Air: Chrome and Safari playing the built-in
microphone through the public relay, Chrome also at +40 ms RTT, with the
20 ms and 100 ms groupDuration A/B and the encoder's lag. relay-mic is
re-recorded from that microphone and its replay ceilings re-measured.

Every row underruns in steady state; replay of the captured arrivals
reproduces it, and the cause is the estimator's 0.95 quantile with no
concealment, so the quest now waits on a decision about its Goal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title test(audio-quality): record relay-mic from a real microphone; watch quest waits on the manual run test(audio-quality): real-microphone relay-mic and the kixtop playout run Nov 3, 2026
kixelated and others added 4 commits November 3, 2026 11:42
The maintainer pulled time stretch into m1 as the Watch quest's fix
(2026-11-03), so the 2026-11-03 kixtop measurements become its baseline:

- relay-mic-rtt40 replays the real-microphone Chrome row at +40 ms RTT.
- player_underruns reads the player's own counter (the stats panel's) in
  the page probe and the replay, since the output tap files a run-dry
  re-stall as stalled time; every replay row budgets it as measured, so
  a time-stretch change shows up deterministically in CI.
- The m2 Time stretch quest folds into Watch, with the stretch depth the
  traces need.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title test(audio-quality): real-microphone relay-mic and the kixtop playout run test(audio-quality): real-microphone baseline for time stretch, from the kixtop playout run Nov 3, 2026
kixelated and others added 2 commits November 3, 2026 12:37
The maintainer asked to copy what WebRTC's NetEq does: accelerate and
preemptive expand by removing or repeating whole pitch periods found by
autocorrelation, with NetEq's thresholds as the starting point. The
Safari +40 ms baseline stays in the quest's Waiting, naming kixtop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated added a commit that referenced this pull request Nov 3, 2026
Maintainer on #7679, 2026-11-03: "m1 please copy what NetEQ does".
A sibling of Watch in the audio jitter target epic, sharing one algorithm and
its test vectors. Also drop proto-safe-names' Required link: #7498 merged and
deleted catalog-proto-ids.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated added a commit that referenced this pull request Nov 3, 2026
Main still keeps the browser stretch in m2 Time stretch; open #7679
moves it into Watch. Cross-link both, and record the native m1 decision in the
epic's boundaries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 3, 2026 20:49
@coderabbitai

coderabbitai Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 5 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 92791bd9-0376-49a4-ad17-ba1723a9416f
Commits

Reviewing files that changed from the base of the PR and between 7b4f531 and 92876c9.

Files selected for processing: 22
  • js/watch/src/audio/replay.test.ts
  • js/watch/src/audio/replay.ts
  • quest/m1/README.md
  • quest/m1/audio-jitter-target/README.md
  • quest/m1/audio-jitter-target/watch.md
  • quest/m1/av-clock.md
  • quest/m2/README.md
  • quest/m2/watch-audio-time-stretch.md
  • quest/m2/watch-worker.md
  • test/audio-quality/README.md
  • test/audio-quality/budgets.json
  • test/audio-quality/clients/js/grade.ts
  • test/audio-quality/clients/js/replay.ts
  • test/audio-quality/clients/js/src/analyze.test.ts
  • test/audio-quality/clients/js/src/analyze.ts
  • test/audio-quality/clients/js/src/probe.test.ts
  • test/audio-quality/clients/js/src/probe.ts
  • test/audio-quality/clients/js/src/schema.ts
  • test/audio-quality/record.sh
  • test/audio-quality/traces/relay-mic-rtt40.json
  • test/audio-quality/traces/relay-mic.json
  • test/justfile
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.

@kixelated
kixelated added this pull request to the merge queue Nov 3, 2026

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed head 92876c9 against base 7b4f531.

No actionable findings in the measurement checkpoint. The player's underrun counter reaches replay and browser samples, the analyzer grades its rise over the measured window, and replay budgets enforce the new metric while leaving the browser and native ceilings unchanged. The time-stretch work remains future work; the Safari +40 ms baseline is explicitly waiting.

Static GitHub-only review of the replay, recorder, metrics, budgets, traces and quest dependencies. I did not run tests or replays, or independently reproduce the microphone and browser measurements.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ No critical issues — one minor suggestion inline.

Reviewed changes

  • New player underrun metric: probe.ts reads audio.out.stats.underruns into Sample.playerUnderruns, analyze.ts grades the counter's rise across the window, schema.ts declares METRICS.player_underruns, and grade.ts shows it first. Browser rows report it unbudgeted; native rows leave it unset.
  • Replay counter: Quantum.underruns now carries AudioRingBuffer.underruns, with a test that one run-dry counts once and a clean trace counts none. The player's stats accumulate the same ring counter across rings, so the replay value has the same meaning.
  • Real-microphone baseline: relay-mic is re-recorded from a real microphone (ceilings re-measured to target 80 ms and silence 0.001), and relay-mic-rtt40 adds the Chrome +40 ms live row as a deterministic baseline at 7.9 player underruns/min. record.sh --mic-broadcast documents how the former is re-recorded.
  • Quest tree: the m2 Time stretch quest is folded into the m1 Watch quest, which is re-scoped to NetEq-style accelerate/preemptive expand, with the baseline, a Safari waiting item, and the A/V clock requirement carried over.

I replayed all five traces with replay.ts, reduced them with analyze.ts, and graded with grade.ts --enforce: within budget, and the new row reproduces its checked-in values exactly. The js/watch replay tests and the harness analyzer, probe, and CLI tests pass. I also confirmed the new row's 23 player underruns all occur after 52 s, so the 780 ms tune-in spike in target_ms_p95 does not inflate the baseline.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using deepseek-v4.1-flash-1017 (free via Pullfrog for OSS) | 𝕏

--source "$PUBLIC/anon, <moq-publish> in headless Chromium capturing its fake audio device, recorded $DATE" \
--description "A browser publisher through the public relay: getUserMedia, the capture worklet, and WebCodecs Opus, with the relay's round trip in between."
description="A browser publisher through the public relay: getUserMedia, the capture worklet, and WebCodecs Opus, with the relay's round trip in between."
if [[ -n "$MIC_BROADCAST" ]]; then

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--mic-broadcast is parsed for every invocation but only read in the relay-mic branch, so --lanes relay-bbb --mic-broadcast foo silently records the fake device instead of failing. Error when the flag is set without relay-mic among LANES, so a hand-run cannot check in a trace that is not the one asked for.

Merged via the queue into main with commit 90762dd Nov 3, 2026
14 checks passed
@kixelated
kixelated deleted the quest/m1/audio-jitter-target/watch branch November 3, 2026 22:02
kixelated added a commit that referenced this pull request Nov 4, 2026
Take #7679's deletion of m2 Time stretch and its epic README text, keeping
the Native stretch sentence; repoint Native stretch at Watch. Per the Merge
order table, drop edges to quests main finished: moq-crypto (#7702),
resume-model-7542, and mesh-fresh-port (#7688).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant