Skip to content

fix(encoding): reject mismatched miniblock buffer sizes - #9237

Open
lance-gatefixer[bot] wants to merge 1 commit into
mainfrom
gatekeeper/fix-8947-1
Open

lance-gatefixer[bot] wants to merge 1 commit into
mainfrom
gatekeeper/fix-8947-1

Conversation

@lance-gatefixer

Copy link
Copy Markdown
Contributor

Summary

  • derive the checked total represented by mini-block chunk metadata before building byte-prefix sums
  • reject metadata/data buffer size mismatches as a contextual corrupt-file error in every build profile
  • cover both under-reported and over-reported data buffer sizes

Root cause

build_chunk_index selected the prefix-sum storage width from data_buf_size while accumulating independent sizes decoded from the metadata words. Their equality was enforced only by a debug assertion, so debug builds panicked and release builds could retain offsets beyond the buffer or choose an undersized prefix representation.

The historical producer path in the reported run was removed by the column-stitching revert, but the reader-side integrity boundary remained unchecked.

Validation

  • cargo test -p lance-encoding test_rejects_mismatched_miniblock_data_size -- --nocapture
  • cargo test -p lance-encoding --lib (1819 passed, 5 ignored)
  • cargo test -p lance --lib index::vector::ivf::tests::test_create_index_nulls::case_5_ivf_hnsw_sq::index_version_1_IndexFileVersion__Legacy -- --exact --nocapture
  • cargo fmt --all -- --check
  • cargo clippy --all --tests --benches -- -D warnings

Fixes #8947

@github-actions github-actions Bot added bug Something isn't working A-encoding Encoding, IO, file reader/writer labels Sep 15, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

This closes a real reader-side integrity gap: mini-block metadata must now cover exactly the declared data buffer before offsets are cached, so malformed pages fail with a contextual corrupt-file error instead of panicking in debug builds or scheduling invalid ranges in release builds. The validation is localized to the parsing boundary and preserves compatibility with the stable file formats.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-encoding Encoding, IO, file reader/writer bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: build_chunk_index passes a data_buf_size that disagrees with its chunk deltas

0 participants