perf(rowids): count RangeWithHoles holes in O(1) - #9242
Open
jackylee-ch wants to merge 1 commit into
Open
jackylee-ch wants to merge 1 commit into
jackylee-ch wants to merge 1 commit into
Conversation
`U64Segment::len()` computes the hole count for a `RangeWithHoles` segment with `holes.iter().count()`, which allocates a boxed `dyn DoubleEndedIterator` and steps it once per hole. `U64Segment::get()` already uses `holes.len()` for the same quantity, and lance-format#6716 introduced that form there while fixing an O(N*K) row-id lookup; `len()` was left on the counting form. The two are equivalent unconditionally: every `EncodedU64Array` variant implements `iter()` as `cloned()`/`map()` over its backing `Vec`, so the iterator yields exactly `len()` items. `RowIdSequence::get()` calls `segment.len()` on every lookup, and `RowIdSequence::slice()` and `U64Segment::mask()` call it once and four times per invocation respectively. `RangeWithHoles` is selected while holes stay under a thirty-second of the span, so a segment covering a million slots can legitimately carry tens of thousands of holes, each one costing a virtual call per lookup. ## Testing `cargo test --release -p lance-table rowids` (113 passed). `test_segments` already asserts `values.len() == segment.len()` over the `RangeWithHoles` cases; perturbing the arm fails it along with five other tests, so the line is covered. Also `cargo fmt --all -- --check` and `cargo clippy --release -p lance-table --all-features --all-targets -- -D warnings`.
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
lance-gatekeeper
Bot
added
the
K-approved
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
U64Segment::len()computes the hole count for aRangeWithHolessegment withholes.iter().count(), which allocates a boxeddyn DoubleEndedIteratorand steps it once per hole.U64Segment::get()already usesholes.len()for the same quantity — #6716 introduced that form there while fixing an O(N*K) row-id lookup, andlen()was left behind. The two are equivalent unconditionally: everyEncodedU64Arrayvariant implementsiter()ascloned()/map()over its backingVec, so it yields exactlylen()items.RowIdSequence::get()callssegment.len()on every lookup.RangeWithHolesis selected while holes stay under a thirty-second of the span, so a segment covering a million slots can legitimately carry tens of thousands of them.Testing
cargo test --release -p lance-table rowids(113 passed).test_segmentsalready assertsvalues.len() == segment.len()over theRangeWithHolescases; perturbing the arm fails it and five other tests. Alsocargo fmt --all -- --checkandcargo clippy --release -p lance-table --all-features --all-targets -- -D warnings.