Propagate global --cache_dir to continuous batching pipeline (#4230)

[exzile]

Summary

Fixes #4230.

The continuous batching servable initializer constructs the GenAI ContinuousBatchingPipeline directly and never applied the server-level --cache_dir (ServerSettings.cacheDir). Unlike the non-CB path — which applies it via ModelInstance::setCacheOptions (ieCore.set_property(ov::cache_dir(...))) — the CB path left model-compilation caching disabled unless the user manually duplicated the value into the node's plugin_config as CACHE_DIR. As a result, .blob/.cl_cache artifacts were never persisted and every server restart fully recompiled the model.

Fix

In ContinuousBatchingServableInitializer::initialize, inject the global cache_dir into the pipeline pluginConfig (keyed by ov::cache_dir.name() == CACHE_DIR) right after parsing the node plugin_config and before constructing the pipeline. An explicit CACHE_DIR in the node's plugin_config remains authoritative.

const std::string& globalCacheDir = Config::instance().cacheDir();
if (!globalCacheDir.empty()) {
    if (properties->pluginConfig.find(ov::cache_dir.name()) == properties->pluginConfig.end()) {
        properties->pluginConfig[ov::cache_dir.name()] = globalCacheDir;
    } // else: explicit node CACHE_DIR wins
}

This also applies to the VLM continuous-batching servable, which reuses this initializer.

Testing

Added a regression test LLMNodeOptionsCacheDirPropagation (run for both the CB and VLM fixtures) covering:

1. The global --cache_dir is propagated into pluginConfig["CACHE_DIR"] when the node does not set it.
2. An explicit CACHE_DIR in the node plugin_config takes precedence over the global value.

Built and ran locally on Windows (MSVC) against a real facebook/opt-125m continuous-batching pipeline:

[       OK ] LLMOptionsHttpTest.LLMNodeOptionsCheckPluginConfig
[       OK ] LLMOptionsHttpTest.LLMNodeOptionsCacheDirPropagation
[  PASSED  ] 2 tests.

🤖 Generated with Claude Code

[exzile]

End-to-end verification on GPU

Beyond the unit test, I verified the actual caching behavior on an Intel Arc Pro B70 dGPU (EXPORT_IMPORT capable) serving facebook/opt-125m via continuous batching with device: "GPU" and a --cache_dir pointing at an initially-empty directory:

Run 1 (empty cache):

• Log confirms the fix path: servable_initializer.cpp] Applying global cache_dir to continuous batching pipeline: .../cachedir
• Cache dir went from empty → *.cl_cache files + a 130 MB .blob (compiled model)
• Pipeline init → AVAILABLE: ~1.6 s

Run 2 (restart, populated cache):

• Same fix path fires
• No recompile: blob is reused (file unchanged), no new artifacts written
• Pipeline init → AVAILABLE: ~0.24 s (~6.7× faster cold start)

Without the change the cache directory stays empty on the CB path and every restart recompiles. With it, blobs persist and are reused across restarts as intended.

[Copilot]

Pull request overview

This PR restores the server-level --cache_dir behavior for GenAI servables by propagating it into the OpenVINO GenAI pipeline pluginConfig (unless explicitly overridden by node plugin_config), ensuring compiled-model cache artifacts can persist across restarts for LLM/VLM pipelines.

Changes:

• Add GenAiServableInitializer::applyGlobalCacheDir() to inject the global --cache_dir into GenAI pipeline plugin properties when CACHE_DIR is not explicitly set per node.
• Call applyGlobalCacheDir() from LM/VLM initializers (legacy + continuous batching) before constructing GenAI pipelines.
• Add regression tests for cache-dir propagation (and an end-to-end cache artifact write check), plus documentation clarifying behavior.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/test/llm/llmnode_test.cpp	Adds tests verifying --cache_dir propagation and precedence, plus an end-to-end artifact write assertion.
src/llm/visual_language_model/legacy/servable_initializer.cpp	Applies global cache dir before creating ov::genai::VLMPipeline.
src/llm/servable_initializer.hpp	Declares applyGlobalCacheDir() helper on the shared initializer base.
src/llm/servable_initializer.cpp	Implements applyGlobalCacheDir() using Config::instance().cacheDir() and ov::cache_dir.name().
src/llm/language_model/legacy/servable_initializer.cpp	Applies global cache dir before creating ov::genai::LLMPipeline.
src/llm/language_model/continuous_batching/servable_initializer.cpp	Applies global cache dir before creating ov::genai::ContinuousBatchingPipeline.
docs/model_cache.md	Documents cache-dir behavior for GenAI/LLM pipelines and precedence rules.