Repository navigation
fix: [branch-1.2] match Spark nullness for missing nested Parquet fields (#7085) - #7313
Merged
Merged
Conversation
…lds (apache#7085) * fix: match Spark nullness for missing nested Parquet fields * fix: fall back for name-mapped nested Iceberg scans * fix: decline Iceberg scans when table properties cannot be read * docs: migrate nested-field notes to consolidated scan guides * docs: describe list and map nulling for all-missing nested leaves (cherry picked from commit c4a0b1f)
andygrove
added
bug
comphead
approved these changes
Oct 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Backport of #7085 to
branch-1.2. #7085 closes #7030 onmain.Rationale for this change
When every requested nested leaf is missing from a Parquet file, the native scan can return non-null containers where Spark returns NULL. That changes
IS NULL, filters andcountresults, silently, on the default configuration (#7030, priority:critical, correctness).branch-1.2has the bug, so it needs the fix before 1.2.0 ships. #7085 also makes native scans of name-mapped nested Iceberg tables fall back to Spark.branch-1.2is the newest release branch, so no newer branch needs a backport. #7085 carries nobackport-1.1label, so nobranch-1.1backport is part of this.What changes are included in this PR?
git cherry-pick -x c4a0b1f2bcof the squash commit of #7085, with no adaptations. It applied without conflicts. The only change outside tests, docs and the scan code is a comment inoperator.proto; no proto field is added or renumbered. See #7085 for the full description.How are these changes tested?
On this branch, with a
libcometbuilt from the pick:cargo check --all-targets,cargo fmt --check, and the 351parquetunit tests indatafusion-cometpass../mvnw test-compilepasses.CometNativeReaderSuite,IcebergReflectionSuiteandCometIcebergNativeSuitepass: 255 succeeded, 0 failed, 3 canceled. The canceled tests are version-conditional (issue #4136, an unknown partition transform, and an output-ordering fallback); the new tests all ran.parquet_support.rsreverted tobranch-1.2,issue #7030: missing nested fields - legacy trueinCometNativeReaderSuitefails, so the branch has the bug and the test catches it.I did not run the Spark SQL test suites or the Iceberg suites across versions that #7085 ran in CI.