Skip to content

fix(spec): match manifest partition fields by field ID - #3566

Open
ewoodbury wants to merge 2 commits into
apache:mainfrom
ewoodbury:fix/3565-partition-field-id
Open

ewoodbury wants to merge 2 commits into
apache:mainfrom
ewoodbury:fix/3565-partition-field-id

Conversation

@ewoodbury

@ewoodbury ewoodbury commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

What changes are included in this PR?

Java writes a partition field whose name isn't a valid Avro name under a sanitized name, e.g. user.id_bucket as user_x2Eid_bucket. iceberg-rust matched partition fields by name, so these values read as null with no error. Filtered scans then skipped the affected files, and anything else reading DataFile::partition() saw null too.

The manifest reader now matches partition fields by field-id, which the spec requires on every Avro field and which Java and PyIceberg use. Once per manifest, it renames the partition type's fields to the writer's Avro names, so the existing projection works unchanged. Only the partition record needs this, since it's the one part of a manifest whose field names come from the user.

This is the follow-up from the #3354 review (comment), listed in #3262.

Behavior changes:

  • When the writer's partition fields have field IDs, they match by ID. A field with the right name but a different ID now reads as null, as in Java.
  • When they have no field IDs, they still match by name, as on main. Java reads these as null, but no writer I know of omits IDs only on partition fields, so I kept the current behavior.

Are these changes tested?

Yes:

  • test_parse_manifest_written_by_java_with_sanitized_partition_name reads V1 and V2 manifests written by iceberg-java 1.12.0, added to testdata/manifests/ with the program that regenerates them. It fails on main.
  • test_parse_manifest_matches_partition_fields_by_field_id covers swapped IDs, a name match with the wrong ID, and fields without IDs.
  • The Python test test_read_manifest_entry expected [1, None], because its fixture's Avro name for field 1001 differs from the spec's. It now reads [1, 1925], which matches PyIceberg's test on the same fixture.

I broke each part of the fix in turn, including the V1 path, and confirmed a test fails each time. I also checked that a Java-written table filtered on a sanitized partition column returns its rows (none on main).

AI Disclosure

Drafted with AI assistance (Grok and Claude). I fully own the fix strategy and its correctness.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Partition values under sanitized Avro names read as null, so filtered scans skip their files

1 participant