Repository navigation
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
What changes are included in this PR?
Java writes a partition field whose name isn't a valid Avro name under a sanitized name, e.g.
user.id_bucketasuser_x2Eid_bucket. iceberg-rust matched partition fields by name, so these values read as null with no error. Filtered scans then skipped the affected files, and anything else readingDataFile::partition()saw null too.The manifest reader now matches partition fields by
field-id, which the spec requires on every Avro field and which Java and PyIceberg use. Once per manifest, it renames the partition type's fields to the writer's Avro names, so the existing projection works unchanged. Only the partition record needs this, since it's the one part of a manifest whose field names come from the user.This is the follow-up from the #3354 review (comment), listed in #3262.
Behavior changes:
main. Java reads these as null, but no writer I know of omits IDs only on partition fields, so I kept the current behavior.Are these changes tested?
Yes:
test_parse_manifest_written_by_java_with_sanitized_partition_namereads V1 and V2 manifests written by iceberg-java 1.12.0, added totestdata/manifests/with the program that regenerates them. It fails onmain.test_parse_manifest_matches_partition_fields_by_field_idcovers swapped IDs, a name match with the wrong ID, and fields without IDs.test_read_manifest_entryexpected[1, None], because its fixture's Avro name for field 1001 differs from the spec's. It now reads[1, 1925], which matches PyIceberg's test on the same fixture.I broke each part of the fix in turn, including the V1 path, and confirmed a test fails each time. I also checked that a Java-written table filtered on a sanitized partition column returns its rows (none on
main).AI Disclosure
Drafted with AI assistance (Grok and Claude). I fully own the fix strategy and its correctness.