[SYCL] Keep fast kernel cache entries when the kernel name is already registered - #23083
Open
Agusx1211 wants to merge 1 commit into
Open
[SYCL] Keep fast kernel cache entries when the kernel name is already registered#23083Agusx1211 wants to merge 1 commit into
Agusx1211 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
KernelProgramCache::saveKernelregisters the kernel name inMFastKernelCachewithThe wrapper temporary is built before the call and destroyed after it, whether it was inserted or not. When the name is already registered (every fast cache miss after the first one for that kernel) the temporary is destroyed intact, and
~FastKernelSubcacheWrappererases every entry of the current context from the kernel's subcache.With a single device this goes unnoticed, the entry that was just erased is added right back. With two or more devices in one context it makes the fast cache useless: a launch on device B erases the entry for device A, the next launch on A misses, erases B, and so on. Every one of those misses goes through the slow path and appends a
(std::string kernel name, device)pair toMProgramToFastKernelCacheKeyMap, which is never trimmed while in-memory cache eviction is off (the default). On a llama.cpp server running Qwen3.8-27B tensor-split across two Intel Arc Pro B70s (oneAPI 2026.1, one context, about a thousand launches per decode step) this leaked around 3 MB/s while generating tokens and got the process OOM killed roughly every 2.5 hours.The fix constructs the wrapper only when it is going to be stored; the lookup and the insert happen under
MFastKernelCacheMutex, whichsaveKernelalready holds. A unit test launches the same kernel alternately on two mock devices of one context and checks that the subcache keeps one entry per device and thattryToGetKernelFasthits for both.How it was validated: on the shipped 2026.1.0
libsycl.so.9first, by patching the equivalent branch in the compiledsaveKernel(thejethat skips the inlined destructor when the wrapper was moved-from, turned into ajmp).saveKernelcalls during a 200 token generation went from about 4000 (breakpoint count under gdb, one per launch) to 0, RSS growth per 600 token generation went from +90..+280 MiB to within +-10 MiB, and single stream decode went from about 26 to about 31 tokens/s because the launches now hit the fast cache.Then with this branch built from source (Release, GCC 13,
buildbot/configure.py --disable-jit --disable-preview-lib), runningKernelAndProgramTests-Non_Preview_Tests --gtest_filter='MultipleDevice*':MultipleDeviceFastCacheTest.EntriesKeptForAllDevices.kernel_program_cache.hppfrom the parent commit swapped back in: the new test fails withSubcache.Entries.size()being 1 instead of 2 (only the last device launched is left) andtryToGetKernelFastmissing for device 0, which is the thrash described above. The three existingProgramRetaincases still pass.