External 4230 cache dir fix#4378
Open
rasapala wants to merge 7 commits into
Open
Conversation
The continuous batching servable initializer constructs the GenAI ContinuousBatchingPipeline directly and never applied the server-level --cache_dir (ServerSettings.cacheDir). Unlike the non-CB path, which applies it via ModelInstance::setCacheOptions, the CB path left model compilation caching disabled unless the user duplicated the value into the node's plugin_config as CACHE_DIR. As a result, .blob/.cl_cache artifacts were never persisted and every restart fully recompiled the model. Inject the global cache_dir into the pipeline plugin config before constructing the pipeline. An explicit CACHE_DIR in the node's plugin_config remains authoritative. Adds a regression test (LLMNodeOptionsCacheDirPropagation) covering both propagation of the global value and precedence of an explicit node value. Fixes #4230 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
LLMNodeOptionsCacheDirPropagation only asserts that --cache_dir lands in properties->pluginConfig, which doesn't prove OpenVINO Core actually persists compiled-model cache artifacts -- the actual symptom in #4230 (log said "cache enabled", nothing was ever written to disk). Addresses feedback on the issue: #4230 (comment) LLMNodeOptionsCacheDirWritesCacheArtifacts constructs a real ContinuousBatchingPipeline against a temp --cache_dir and asserts at least one cache file actually lands there. Verified locally on Windows (MSVC) against facebook/opt-125m: [ OK ] LLMOptionsHttpTest.LLMNodeOptionsCacheDirWritesCacheArtifacts (707 ms)
Extract the global --cache_dir propagation into a shared GenAiServableInitializer::applyGlobalCacheDir helper and call it from all GenAI initializers. The continuous batching path already applied it (and VLM_CB shares that same initializer), but the legacy LM and legacy VLM paths construct their pipelines directly and never applied the server-level cache_dir. Routing every path through one helper covers all four pipeline types (LM/VLM x CB/legacy) and removes the duplicated inline block from the CB initializer. In the cache_dir tests, replace the manual end-of-test Config restore with a GlobalCacheDirGuard RAII helper so a failed ASSERT mid-test can no longer leak the modified --cache_dir singleton into subsequent tests in the suite. The guard also removes the temporary cache directory on scope exit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR addresses a regression where the server-level --cache_dir was not propagated into OpenVINO GenAI pipeline plugin configuration for LLM/VLM servables, preventing compiled-model cache artifacts from being persisted across restarts (issue #4230).
Changes:
- Introduces a shared
GenAiServableInitializer::applyGlobalCacheDir()helper to inject the global--cache_dirintopluginConfigwhen the node doesn’t explicitly setCACHE_DIR. - Applies this helper in GenAI servable initializers (continuous batching + legacy).
- Adds/extends unit tests and updates model cache documentation.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| src/test/llm/llmnode_test.cpp | Adds regression tests for --cache_dir propagation and attempts an end-to-end cache artifact check. |
| src/llm/servable_initializer.hpp | Declares applyGlobalCacheDir() helper for GenAI initializers. |
| src/llm/servable_initializer.cpp | Implements applyGlobalCacheDir() and adds required includes. |
| src/llm/language_model/continuous_batching/servable_initializer.cpp | Calls applyGlobalCacheDir() before constructing the CB pipeline. |
| src/llm/language_model/legacy/servable_initializer.cpp | Calls applyGlobalCacheDir() before constructing the legacy pipeline. |
| src/llm/visual_language_model/legacy/servable_initializer.cpp | Calls applyGlobalCacheDir() before constructing the legacy VLM pipeline. |
| docs/model_cache.md | Documents how --cache_dir interacts with GenAI/LLM node plugin_config overrides. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🛠 Summary
Issue #4230
🧪 Checklist
``