audio: make ASR dtype configurable - #113
Draft
barvhaim wants to merge 1 commit into
Draft
Conversation
Add a first-class asr_dtype config field so the precision the ASR
weights load in is tunable per checkpoint instead of hardcoded.
Previously ASRTranscriber.load() always requested float16, which
crashes any encoder with BatchNorm layers ("Expected weight to have
type Float but got Half") because BatchNorm will not promote a
float16 weight against float32 features. The only escape was to
smuggle torch_dtype through asr_pipeline_kwargs, which is opaque and
undocumented.
asr_dtype accepts auto/float16/bfloat16/float32 and is validated in
GraniteSwitchConfig.__init__ so a typo fails at compose time rather
than inside a vLLM worker. None/"auto" preserves today's behavior:
float16 on CUDA, float32 elsewhere. asr_pipeline_kwargs still merges
last and therefore still wins, so existing checkpoints that use the
torch_dtype workaround are unaffected.
Also exposed as --asr-dtype on the compose CLI (implies
--enable-audio) and documented in docs/AUDIO.md.
Separately, reduce comment and docstring volume across the audio
modules and their tests. Much of it restated the code or duplicated
docs/AUDIO.md; what remains is the comments carrying information the
code cannot, such as why kwargs.update must come last and why
requires_raw_input_tokens is set.
Note tests/vllm/test_audio_processor.py: three transcriber stubs did
not accept the new dtype kwarg and failed once processor.py started
passing it. Fixed, plus two tests asserting the dtype actually
reaches get_transcriber.
Signed-off-by: BAR HAIM <barha@il.ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a first-class asr_dtype config field so the precision the ASR weights load in is tunable per checkpoint instead of hardcoded.