Skip to content

[SYCL] Keep fast kernel cache entries when the kernel name is already registered - #23083

Open
Agusx1211 wants to merge 1 commit into
intel:syclfrom
Agusx1211:fast-kernel-cache-multi-device
Open

[SYCL] Keep fast kernel cache entries when the kernel name is already registered#23083
Agusx1211 wants to merge 1 commit into
intel:syclfrom
Agusx1211:fast-kernel-cache-multi-device

Conversation

@Agusx1211

@Agusx1211 Agusx1211 commented Sep 2, 2026

Copy link
Copy Markdown

KernelProgramCache::saveKernel registers the kernel name in MFastKernelCache with

MFastKernelCache.try_emplace(std::string(KernelName), FastKernelSubcacheWrapper(KernelSubcache, getURContext()));

The wrapper temporary is built before the call and destroyed after it, whether it was inserted or not. When the name is already registered (every fast cache miss after the first one for that kernel) the temporary is destroyed intact, and ~FastKernelSubcacheWrapper erases every entry of the current context from the kernel's subcache.

With a single device this goes unnoticed, the entry that was just erased is added right back. With two or more devices in one context it makes the fast cache useless: a launch on device B erases the entry for device A, the next launch on A misses, erases B, and so on. Every one of those misses goes through the slow path and appends a (std::string kernel name, device) pair to MProgramToFastKernelCacheKeyMap, which is never trimmed while in-memory cache eviction is off (the default). On a llama.cpp server running Qwen3.8-27B tensor-split across two Intel Arc Pro B70s (oneAPI 2026.1, one context, about a thousand launches per decode step) this leaked around 3 MB/s while generating tokens and got the process OOM killed roughly every 2.5 hours.

The fix constructs the wrapper only when it is going to be stored; the lookup and the insert happen under MFastKernelCacheMutex, which saveKernel already holds. A unit test launches the same kernel alternately on two mock devices of one context and checks that the subcache keeps one entry per device and that tryToGetKernelFast hits for both.

How it was validated: on the shipped 2026.1.0 libsycl.so.9 first, by patching the equivalent branch in the compiled saveKernel (the je that skips the inlined destructor when the wrapper was moved-from, turned into a jmp). saveKernel calls during a 200 token generation went from about 4000 (breakpoint count under gdb, one per launch) to 0, RSS growth per 600 token generation went from +90..+280 MiB to within +-10 MiB, and single stream decode went from about 26 to about 31 tokens/s because the launches now hit the fast cache.

Then with this branch built from source (Release, GCC 13, buildbot/configure.py --disable-jit --disable-preview-lib), running KernelAndProgramTests-Non_Preview_Tests --gtest_filter='MultipleDevice*':

  • with the fix: 4 tests pass, including the new MultipleDeviceFastCacheTest.EntriesKeptForAllDevices.
  • with kernel_program_cache.hpp from the parent commit swapped back in: the new test fails with Subcache.Entries.size() being 1 instead of 2 (only the last device launched is left) and tryToGetKernelFast missing for device 0, which is the thrash described above. The three existing ProgramRetain cases still pass.
  • with the fix restored: 4 tests pass again.

@Agusx1211
Agusx1211 requested a review from a team as a code owner September 2, 2026 08:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant