Partition values under sanitized Avro names read as null, so filtered scans skip their files #3565

Apache Iceberg Rust version

main (6f23a2a)

Describe the bug

Iceberg partition field names don't have to be valid Avro names, which must match [A-Za-z_][A-Za-z0-9_]*. When one isn't, Java writes the field to the manifest under a sanitized name and keeps the original in the iceberg-field-name property (TypeToSchema.java#L116-L128). A partition on a nested column always hits this: Java and Spark name bucket(16, user.id) user.id_bucket, and the manifest stores it as user_x2Eid_bucket. Columns with -, a space, or a leading digit in their names do too.

Manifest::parse_avro matches partition fields to the spec by name (RawLiteral::project_by_name). Partition fields are optional, so a field that doesn't match reads as null with no error. The Avro field also has the partition field's field-id, which the spec requires (spec.md#L1750), but the reader ignores it. The spec says manifests must use valid column projection (spec.md#L700), and projection matches columns by field ID (spec.md#L439). Java and PyIceberg (resolver.py) both match these fields by ID.

Scan planning then evaluates partition filters against null, so a filter on the partition's source column skips every file and the scan returns no rows. Unfiltered scans still return all rows. I hit this on a Spark table bucketed on a nested user ID field.

This was raised in the #3354 review (comment). It's older than that PR: the reader-schema resolution it replaced also matched by name.

To Reproduce

Write a table with iceberg-java 1.12.0:

Types.StructType userType =
    Types.StructType.of(Types.NestedField.optional(3, "id", Types.LongType.get()));
Schema schema = new Schema(
    Types.NestedField.optional(1, "event", Types.StringType.get()),
    Types.NestedField.optional(2, "user", userType));
PartitionSpec spec = PartitionSpec.builderFor(schema).bucket("user.id", 16).build();
// Create the table with HadoopTables and append four rows with user.id = 1..4, one file each.

The manifest's partition field is:

{"name": "user_x2Eid_bucket", "type": ["null", "int"], "default": null, "iceberg-field-name": "user.id_bucket", "field-id": 1000}

Java returns the matching row for user.id = 3. Reading the same table with iceberg-rust:

let table = StaticTable::from_metadata_file(&metadata, ident, FileIO::new_with_fs()).await?;
let scan = table
    .scan()
    .with_filter(Reference::new("user.id").equal_to(Datum::long(3)))
    .build()?;

Every planned task's partition is Struct { fields: [None] }.

Expected behavior

Partition fields are matched by field-id, with the name as a fallback for writers that don't store field IDs. The partition values read as written, and user.id = 3 returns one row.

Writing these names is a separate bug, tracked in #2535: iceberg-rust writes user.id_bucket as the Avro name as is, and then can't read its own manifest (Invalid field name).

Willingness to contribute

I can contribute a fix for this bug independently