Skip to content

feat(datafusion): report measured widths for variable-width scan columns - #9477

Open
vivek-bharathan wants to merge 3 commits into
lance-format:mainfrom
vivek-bharathan:vb/scan-byte-size-widths
Open

vivek-bharathan wants to merge 3 commits into
lance-format:mainfrom
vivek-bharathan:vb/scan-byte-size-widths

Conversation

@vivek-bharathan

@vivek-bharathan vivek-bharathan commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

A schema fixes no per-row width for strings, binary, lists, maps or
dictionaries, so scans report no byte size at all for any row containing one
and DataFusion falls back to row counts.

Measure what those columns actually cost on the batches a scan has already
decoded, using get_array_memory_size -- the same quantity DataFusion reads
from the other side of a join. A completed scan records a width per column
into the session cache, keyed by dataset version, and plan construction folds
what earlier scans measured into the Lance read nodes.

@github-actions github-actions Bot added A-python Python bindings enhancement New feature or request labels Sep 22, 2026
@vivek-bharathan
vivek-bharathan force-pushed the vb/scan-byte-size-widths branch 2 times, most recently from 6bf3474 to dfcc70e Compare September 24, 2026 00:50
@vivek-bharathan
vivek-bharathan marked this pull request as ready for review September 24, 2026 00:51
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Sep 24, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Sep 24, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Sep 24, 2026
@vivek-bharathan
vivek-bharathan force-pushed the vb/scan-byte-size-widths branch 2 times, most recently from 90bd275 to 130598c Compare September 24, 2026 04:54
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Sep 24, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Sep 24, 2026
Lance scans reported no byte size, so DataFusion's join guards fell back to row
counts and broadcast wide tables. Scan, filtered read and take now report rows
times a per-row width, over the columns a schema can actually measure: the
fixed-width primitives, booleans at a bit a row, nulls at nothing, and structs
or fixed-size lists of those. The width is floored at 8 bytes, which reproduces
the 128 Ki row cap at DataFusion's default threshold ratio.
…timate

A schema fixes no per-row width for strings, binary, lists, maps or
dictionaries, so scans reported no byte size at all for any row holding one
and DataFusion fell back to row counts.

Seed those columns from lance_encoding's estimate_bytes_per_row plus the Arrow
buffers around them: 64 bytes and a 4-byte offset for a string, five items for
a list. A width is now always reported.

Blob payloads stay excluded, and the guard is decided from what a node emits
rather than from a projection's blob mode: the public output schema drops the
blob marker, and a take or row-stream read carries columns it never projected.
It covers v1 blobs as well as v2, and recurses through lists and maps.

Dictionaries cost their keys at every nesting depth rather than the values they
decode to.

The four merge_insert plan snapshots move from CollectLeft to Partitioned.
That is the correct plan rather than a regression: the 32,768-row uuid-hex key
target is roughly 1.6 MiB, over DataFusion's 1 MiB collect threshold, and it
was broadcast before only because no byte size existed and the row count fell
under the 128 Ki cap.
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Sep 24, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Sep 24, 2026
@lance-gatekeeper lance-gatekeeper Bot added K-changes Latest Gatekeeper recommendation requests changes. and removed K-changes Latest Gatekeeper recommendation requests changes. labels Sep 24, 2026
A schema fixes no per-row width for strings, binary, lists, maps or
dictionaries, so scans report no byte size at all for any row containing one
and DataFusion falls back to row counts.

Measure what those columns actually cost on the batches a scan has already
decoded, using get_array_memory_size -- the same quantity DataFusion reads
from the other side of a join. A completed scan records a width per column
into the session cache, keyed by dataset version, and plan construction folds
what earlier scans measured into the Lance read nodes.

A cold cache reports nothing, exactly as before. TakeExec is excluded: it
carries payload columns it does not declare, so a width over its output
schema would under-report what it holds.
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Sep 24, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

The completed-scan measurement now reaches variable-width structs and fixed-size lists, while wholly fixed-width layouts retain schema-based sizing. The previously reported join-size underestimates are addressed, and the focused width regressions pass.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 24, 2026
@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. and removed K-approved Latest Gatekeeper recommendation permits acceptance. labels Sep 25, 2026
@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. and removed K-approved Latest Gatekeeper recommendation permits acceptance. labels Sep 25, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-python Python bindings enhancement New feature or request K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant