[DG47]

Closes #4120

Rationale for this change

string_view and binary_view columns are accepted on write, and appending them to an unpartitioned table works. Appending them to a partitioned table fails in _determine_partitions, because PyArrow has no group by, filter or take kernels for view types, and the pyiceberg-core bucket and truncate transforms do not accept them either. pa.table(polars_df) produces exactly this shape, since Polars exports strings and binaries as view types over the PyCapsule interface.

Depending on the spec, the append fails with Keys of type string_view, Function 'array_filter' has no kernel matching input types (string_view, bool), or Unsupported data type for bucket transform: Utf8View.

This change casts view types, including ones nested in structs, lists and maps, to large_string / large_binary before the table is split into partitions. Tables without view types are not cast. The data files are written the same as for large_string / large_binary input.

One remaining gap is outside this repo: truncate on a binary column already fails for large_binary input in pyiceberg-core (apache/iceberg-rust#3119), and now binary_view input hits that same error.

Are these changes tested?

Yes:

• test_determine_partitions_with_view_types: identity, bucket and truncate partitions on a string_view column, with binary_view and nested view columns alongside.
• test_append_view_types_to_partitioned_table: end-to-end append to a partitioned in-memory table, then read back.

Both fail on main and pass with this change, on pyarrow 25.0.1 and on the minimum supported pyarrow 18.1.0.

Are there any user-facing changes?

Yes. Appending, overwriting, and dynamically overwriting partitioned tables with Arrow view type columns now works where it used to raise.

[Fokko]

Thanks @DG47 this makes sense to me. Instead of building this recursive function ourself, we can also implement this using the PyArrowSchemaVisitor:

iceberg-python/pyiceberg/io/pyarrow.py

Line 1327 in 0cc389b

class PyArrowSchemaVisitor(Generic[T], ABC):

[DG47]

Good idea, done in 75c2fe0. The recursion is gone; _determine_partitions now calls visit_pyarrow(arrow_table.schema, _ConvertViewTypes()), and the visitor lives next to _ConvertToLargeTypes. Unlike that one it keeps the list flavour, nested field names, nullability and metadata, so the cast is a pure view-to-large rewrite. One wrinkle: visit_pyarrow unwraps dictionary types before calling primitive, so the visitor re-applies the dictionary encoding in field, list and map to leave dictionary columns untouched.

[Fokko]

Thanks for moving this to the visitor @DG47 🙌

[kevinjqliu]

LGTM! Thanks for the PR @DG47

btw i pushed a small test (66200730f) for polars Categorical and List(Categorical), since those go through the _keep_dictionary path which wasn't covered.

polars exports:

• Categorical → dictionary<string_view> (source)
• List(Categorical) → large_list<dictionary<string_view>>

[kevinjqliu]

😭 feels like we're constantly writing custom code to workaround pyarrow