fix: prevent concurrent access to non-thread-safe maps in SketchBufferAggregatorHelper - #20090
Open
zhang-arvin wants to merge 1 commit into
Open
Conversation
…rAggregatorHelper The ParallelCombiner and StreamingMergeSortedGrouper are designed for concurrent read/write from different threads. However, SketchBufferAggregatorHelper used IdentityHashMap and Int2ObjectOpenHashMap, which are not thread-safe. Concurrent access to these maps by the writing thread (via aggregate()) and the reading thread (via get()) could corrupt the internal map structure, causing ArrayIndexOutOfBoundsException. This fix replaces IdentityHashMap with ConcurrentHashMap and Int2ObjectOpenHashMap with ConcurrentHashMap to ensure thread safety when used with ParallelCombiner. Fixes apache#18040
FrankChen021
left a comment
Member
There was a problem hiding this comment.
| Severity | Findings |
|---|---|
| P0 | 0 |
| P1 | 1 |
| P2 | 0 |
| P3 | 0 |
| Total | 1 |
Reviewed 1 of 1 changed files.
This is an automated review by Codex GPT-5.6-Luna(max)
| private final int maxIntermediateSize; | ||
| private final IdentityHashMap<ByteBuffer, Int2ObjectMap<Union>> unions = new IdentityHashMap<>(); | ||
| private final IdentityHashMap<ByteBuffer, WritableMemory> memCache = new IdentityHashMap<>(); | ||
| private final ConcurrentHashMap<ByteBuffer, ConcurrentHashMap<Integer, Union>> unions = new ConcurrentHashMap<>(); |
Member
There was a problem hiding this comment.
[P1] Do not key caches by mutable ByteBuffers
ConcurrentHashMap uses ByteBuffer.equals/hashCode, which depend on the buffer's remaining contents. Union updates mutate those contents in place, changing the key hash after insertion. Subsequent unions.get(buf) calls can miss and get() returns SketchHolder.EMPTY; distinct equal-content buffers can also alias. Preserve identity/stable-key semantics while making access concurrent.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #18040 - Runtime Exception when executing a query twice in a short duration.
Root Cause
The
ParallelCombinerandStreamingMergeSortedGrouperare designed for concurrent read/write from different threads. However,SketchBufferAggregatorHelperusedIdentityHashMapandInt2ObjectOpenHashMap, which are not thread-safe.Concurrent access to these maps by:
aggregate()→getOrCreateUnion())get())could corrupt the internal map structure, causing
ArrayIndexOutOfBoundsExceptionduring sketches aggregation (e.g.,Index 180 out of bounds for length 129).Fix
Replaced
IdentityHashMapwithConcurrentHashMapandInt2ObjectOpenHashMapwithConcurrentHashMapto ensure thread safety when used withParallelCombiner.Changes
extensions-core/datasketches/.../SketchBufferAggregatorHelper.java: Replace non-thread-safe maps withConcurrentHashMapKey Features/Changes
SketchBufferAggregatorHelperConcurrentHashMap.computeIfAbsentfor atomic map initializationVerification
The existing tests for
ParallelCombinerandStreamingMergeSortedGroupercover the concurrent read/write pattern. The fix ensures that Datasketches-based aggregators work correctly under parallel combining.