Repository navigation
Conversation
AntiO2
marked this pull request as ready for review
October 11, 2026 09:01
AntiO2
marked this pull request as draft
October 11, 2026 09:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Concurrent CDC deletion writers can publish a new tail before its slot count and initialized records, or overwrite another block’s slot count. An isolated 8-writer / 100,000-delete reproduction loses records even after GC. Timestamp-order assumptions independently omit older deletes at finite snapshots and discard future deletes during GC; import also incorrectly treats timestamps as duplicate identities across different rows.
Publish immutable deletion blocks, their initialized counts, and the bitmap state through one version-pointer CAS. Appends copy only the incomplete head block and share the immutable base bitmap; readers retain epoch protection without a tile mutex. GC rebuilds the future chain by testing every record’s timestamp, retries if publication changed, and never detaches links visible to an old reader. Imports preserve different rows regardless of timestamp and publish historical bitmap folds atomically.
Retirement uses a multi-producer queue. A try-only guard coordinates bookkeeping without blocking reads or publication; publishers periodically reclaim old epochs so sustained ingestion does not require a global idle interval. Epoch entry and reclamation scans now fence participation against protected-pointer loads.
The established production concurrency is bucket-parallel processing within one request, with a shared timestamp. Stream ordering is not being claimed broken. Out-of-order timestamp tests cover an additional API boundary, not a proven normal-stream execution.
Validation:
Same-timestamp 8-writer regression: unmodified master fails in the first round; atomic publication passes all 20 rounds. This isolates the publication race without any timestamp disorder.
Capacity 10,240, scalar: all 34 TileVisibility and 21 RGVisibility tests passed on the final revision, including the new same-timestamp regression.
Capacity 256, AVX2: all 33 TileVisibility and 21 RGVisibility tests passed.
AddressSanitizer/UndefinedBehaviorSanitizer, capacity 10,240: 13 selected tests repeated 20 times passed, including concurrent append/import/read/partial GC, with no sanitizer errors.
Jemalloc, capacity 10,240: 17 selected tests repeated 20 times passed.
Server JNI reproduction with the runtime’s jemalloc preload: ten rounds each returned exactly 100,000 deletions before and after GC.
Integrated 15-minute replay completed 10 partial-live Storage GC rewrites, then updated the surviving keys; all table populations returned to baseline after 17,702,769 row events and 177,973 commits. It physically deleted 28 old files / 125,261,498 bytes. This functional run used immutable publication, not the diagnostic mutex.
This remains a draft pending sustained allocation/throughput checks. The earlier tile-wide mutex was a diagnostic implementation and has been removed; its pilot data is not final performance evidence. The CAS path adds allocation and retry work, which must be measured under sustained replay.
Closes #1411
Additional combined live validation after the separate sink same-key coalescing fix (pixelsdb/pixels-sink#22): HyBench SF100 original Proto replay ingested 22,945,933 rows and fully committed all 452,616 admitted transactions. Two GC cycles and three partial-live group rewrites completed. After draining, savingaccount/checkingaccount each had 30,200,000 active versions, customer 30,000,000 and company 200,000, exactly matching their initial populations; insert-growing tables are reported separately. Sink shutdown had no error entries. This is functional integration evidence, not a steady-state or isolated performance comparison. The repeated formal matrix has resumed. Results: gengdy1545/retina-graph#34.