feat(datafusion): report measured widths for variable-width scan columns - #9477
Open
vivek-bharathan wants to merge 3 commits into
Open
vivek-bharathan wants to merge 3 commits into
vivek-bharathan wants to merge 3 commits into
Conversation
vivek-bharathan
force-pushed
the
vb/scan-byte-size-widths
branch
2 times, most recently
from
September 24, 2026 00:50
6bf3474 to
dfcc70e
Compare
vivek-bharathan
marked this pull request as ready for review
September 24, 2026 00:51
vivek-bharathan
force-pushed
the
vb/scan-byte-size-widths
branch
from
September 24, 2026 02:33
dfcc70e to
36e48f4
Compare
vivek-bharathan
force-pushed
the
vb/scan-byte-size-widths
branch
2 times, most recently
from
September 24, 2026 04:54
90bd275 to
130598c
Compare
Lance scans reported no byte size, so DataFusion's join guards fell back to row counts and broadcast wide tables. Scan, filtered read and take now report rows times a per-row width, over the columns a schema can actually measure: the fixed-width primitives, booleans at a bit a row, nulls at nothing, and structs or fixed-size lists of those. The width is floored at 8 bytes, which reproduces the 128 Ki row cap at DataFusion's default threshold ratio.
…timate A schema fixes no per-row width for strings, binary, lists, maps or dictionaries, so scans reported no byte size at all for any row holding one and DataFusion fell back to row counts. Seed those columns from lance_encoding's estimate_bytes_per_row plus the Arrow buffers around them: 64 bytes and a 4-byte offset for a string, five items for a list. A width is now always reported. Blob payloads stay excluded, and the guard is decided from what a node emits rather than from a projection's blob mode: the public output schema drops the blob marker, and a take or row-stream read carries columns it never projected. It covers v1 blobs as well as v2, and recurses through lists and maps. Dictionaries cost their keys at every nesting depth rather than the values they decode to. The four merge_insert plan snapshots move from CollectLeft to Partitioned. That is the correct plan rather than a regression: the 32,768-row uuid-hex key target is roughly 1.6 MiB, over DataFusion's 1 MiB collect threshold, and it was broadcast before only because no byte size existed and the row count fell under the 128 Ki cap.
vivek-bharathan
force-pushed
the
vb/scan-byte-size-widths
branch
from
September 24, 2026 18:24
130598c to
3866988
Compare
A schema fixes no per-row width for strings, binary, lists, maps or dictionaries, so scans report no byte size at all for any row containing one and DataFusion falls back to row counts. Measure what those columns actually cost on the batches a scan has already decoded, using get_array_memory_size -- the same quantity DataFusion reads from the other side of a join. A completed scan records a width per column into the session cache, keyed by dataset version, and plan construction folds what earlier scans measured into the Lance read nodes. A cold cache reports nothing, exactly as before. TakeExec is excluded: it carries payload columns it does not declare, so a width over its output schema would under-report what it holds.
vivek-bharathan
force-pushed
the
vb/scan-byte-size-widths
branch
from
September 24, 2026 21:08
3866988 to
7fd514c
Compare
Contributor
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The completed-scan measurement now reaches variable-width structs and fixed-size lists, while wholly fixed-width layouts retain schema-based sizing. The previously reported join-size underestimates are addressed, and the focused width regressions pass.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A schema fixes no per-row width for strings, binary, lists, maps or
dictionaries, so scans report no byte size at all for any row containing one
and DataFusion falls back to row counts.
Measure what those columns actually cost on the batches a scan has already
decoded, using get_array_memory_size -- the same quantity DataFusion reads
from the other side of a join. A completed scan records a width per column
into the session cache, keyed by dataset version, and plan construction folds
what earlier scans measured into the Lance read nodes.