Omni#4379
Open
dkalinowski wants to merge 10 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds initial “Omni” pipeline support to OVMS GenAI serving, enabling audio input (Chat Completions + Responses) and audio output generation/streaming integration, along with supporting utilities, docs, and demo clients.
Changes:
- Introduces a new OMNI pipeline type (proto + C++ enums) and routes initialization to a new legacy Omni servable/executor/initializer.
- Adds audio input decoding to the LLM input-processing pipeline and extends OpenAI-compatible request parsing for
input_audioand outputmodalities/audio. - Refactors
audio_utilsinto a dedicated namespace and provides non-resampling decode helper used by Omni audio input.
Reviewed changes
Copilot reviewed 34 out of 36 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| src/test/audio/text2speech_test.cpp | Updates tests to use ovms::audio_utils namespace. |
| src/test/audio/audio_utils_test.cpp | Updates tests to use ovms::audio_utils namespace. |
| src/llm/servable_initializer.hpp | Adds PipelineType::OMNI. |
| src/llm/servable_initializer.cpp | Detects OMNI models and initializes Omni legacy servable. |
| src/llm/omni_model/legacy/servable.hpp | Declares Omni legacy servable/context/properties. |
| src/llm/omni_model/legacy/servable.cpp | Implements Omni request lifecycle, audio response injection, streaming handling. |
| src/llm/omni_model/legacy/servable_initializer.hpp | Declares Omni legacy servable initializer. |
| src/llm/omni_model/legacy/servable_initializer.cpp | Builds ov::genai::OmniPipeline + tokenizer and config. |
| src/llm/omni_model/legacy/legacy_executor.hpp | Declares Omni legacy executor + wrapper thread. |
| src/llm/omni_model/legacy/legacy_executor.cpp | Implements Omni generation execution + speech streaming callback. |
| src/llm/llm_calculator.proto | Adds OMNI to pipeline enum for graph options. |
| src/llm/io_processing/input_processors/image_decoding_processor.cpp | Preserves multimodal part ordering when non-image parts exist (e.g., audio). |
| src/llm/io_processing/input_processors/audio_decoding_processor.hpp | Adds new processor to extract input_audio into tensors. |
| src/llm/io_processing/input_processors/audio_decoding_processor.cpp | Implements base64 decode + audio parsing to ov::Tensor. |
| src/llm/io_processing/input_processor.cpp | Wires AudioDecodingProcessor into pipeline for Omni chat requests. |
| src/llm/io_processing/input_processing_config.hpp | Adds isOmni config flag. |
| src/llm/BUILD | Adds Omni + audio decoding sources and deps to Bazel targets. |
| src/llm/apis/openai_responses.cpp | Supports input_audio items and serializes streamed audio delta events. |
| src/llm/apis/openai_request.hpp | Adds audio output request fields to request model. |
| src/llm/apis/openai_completions.cpp | Validates input_audio content parts for chat completions. |
| src/llm/apis/openai_api_handler.cpp | Parses modalities + audio output config. |
| src/audio/text_to_speech/t2s_calculator.cc | Qualifies audio utils calls with ovms::audio_utils. |
| src/audio/speech_to_text/s2t_calculator.cc | Qualifies audio utils calls with ovms::audio_utils. |
| src/audio/audio_utils.hpp | Moves audio helpers into ovms::audio_utils and adds readWithoutResample. |
| src/audio/audio_utils.cpp | Implements readWithoutResample and wraps implementation in namespace. |
| OMNI_PLAN.md | Documents design/plan and API mapping for Omni integration. |
| OMNI_AUDIO_OUTPUT.md | Documents audio output implementation and streaming event format. |
| demos/omni/omni_voice_chat.py | Adds a voice-to-voice demo client (record + stream + playback). |
| demos/omni/omni_stream_playback.py | Adds streaming playback demo (text then audio deltas). |
| demos/omni/omni_responses_client.py | Adds minimal Responses API audio-input demo client. |
| demos/omni/omni_full_client.py | Adds full multimodal demo (audio+image → text+audio). |
| demos/omni/omni_client.py | Adds minimal Chat Completions audio-input demo client. |
| demos/omni/omni_audio_output_client.py | Adds Chat Completions audio-output demo client. |
| demos/omni/.gitignore | Ignores generated WAVs in demos. |
Comment on lines
+33
to
+35
| bool OmniModelLegacyExecutor::hasRequests() { | ||
| return (requests.size() > 0); | ||
| } |
Comment on lines
+37
to
+39
| size_t OmniModelLegacyExecutor::requestsQueueSize() { | ||
| return requests.size(); | ||
| } |
Comment on lines
+41
to
+44
| void OmniModelLegacyExecutor::processRequest() { | ||
| OVMS_PROFILE_FUNCTION(); | ||
| auto& requestExecutionContext = requests.front(); | ||
| if (requestExecutionContext->clientDisconnected) { |
Comment on lines
+48
to
+52
| SPDLOG_INFO("isOmni: {}, isVLM: {}, isChatPath: {}", context.config.isOmni, context.config.isVLM, isChatPath); | ||
| if (context.config.isOmni && isChatPath) { | ||
| SPDLOG_INFO("Adding AudioDecodingProcessor to the pipeline"); | ||
| processors.emplace_back(std::make_unique<AudioDecodingProcessor>()); | ||
| } |
Comment on lines
+319
to
+322
| const uint64_t n = wav.totalPCMFrameCount; | ||
| std::vector<int16_t> pcm16(n * wav.channels); | ||
| drwav_read_pcm_frames_s16(&wav, n, pcm16.data()); | ||
| drwav_uninit(&wav); |
Comment on lines
+339
to
+343
| if (mp3.channels != 1 && mp3.channels != 2) { | ||
| drmp3_uninit(&mp3); | ||
| throw std::runtime_error("MP3 audio must be mono or stereo"); | ||
| } | ||
| constexpr size_t CHUNK_FRAMES = 1152; |
Comment on lines
+257
to
+265
| if (responseDoc.HasMember("choices") && responseDoc["choices"].IsArray() && responseDoc["choices"].Size() > 0) { | ||
| // Chat Completions format: inject into choices[0].message.audio | ||
| auto& message = responseDoc["choices"][0]["message"]; | ||
| rapidjson::Value audioObj(rapidjson::kObjectType); | ||
| audioObj.AddMember("data", rapidjson::Value(audioBase64.c_str(), alloc), alloc); | ||
| std::string transcript = message.HasMember("content") && message["content"].IsString() ? message["content"].GetString() : ""; | ||
| audioObj.AddMember("transcript", rapidjson::Value(transcript.c_str(), alloc), alloc); | ||
| message.AddMember("audio", audioObj, alloc); | ||
| } else if (responseDoc.HasMember("output") && responseDoc["output"].IsArray()) { |
Comment on lines
+458
to
+462
| // Existence of talker model indicates omni pipeline | ||
| bool isOmni = std::filesystem::exists(parsedModelsPathFs / "openvino_talker_model.xml"); | ||
| // Existence of embeddings models indicates we are dealing with VLM pipeline | ||
| bool isVLM = (std::filesystem::exists(parsedModelsPathFs / "openvino_text_embeddings_model.xml") && | ||
| std::filesystem::exists(parsedModelsPathFs / "openvino_vision_embeddings_model.bin")); | ||
| std::filesystem::exists(parsedModelsPathFs / "openvino_vision_embeddings_model.bin")) && !isOmni; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CVS-189193