Skip to content

fix: read partition source columns through PartitionExpr children - #92

Draft
NoahKusaba wants to merge 3 commits into
apache:mainfrom
NoahKusaba:fix/partition-expr-children
Draft

NoahKusaba wants to merge 3 commits into
apache:mainfrom
NoahKusaba:fix/partition-expr-children

Conversation

@NoahKusaba

@NoahKusaba NoahKusaba commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

PartitionExpr read its source columns by position in the input batch, but reported no children. When ProjectionPushdown merged the partition projection into the projection below it, it had no column references to rewrite, so the positions kept pointing at the old layout:

INSERT INTO t SELECT b AS id, a AS val FROM source  -- t is partitioned by identity(id)

partitioned the rows by a instead of b. Files landed in the wrong partitions, and partition pruning then skipped matching rows.

What changes are included in this PR?

  • Sources are children. PartitionExpr has one child per partition field: the top-level column holding that field's source. The optimizer rewrites them like any other column reference, and EXPLAIN shows what each field reads:

    iceberg_partition_values(id: identity(b@1))
    
  • Transforms are evaluated in place. The expression applies the transforms itself instead of going through iceberg's positional PartitionValueCalculator. A nested source is read by descending from its top-level column, nulling values under a null parent struct as iceberg's RecordBatchProjector did.

  • The public API can't build a broken expression. PartitionExpr::try_new(spec, schema, sources) now takes the sources, so a codec passes the decoded children instead of assuming a table-shaped input. Only project_with_partition, which checks that its input matches the table schema, derives them from the table (crate-private try_new_over_table). with_new_children rejects the wrong number of children.

  • Spec and schema errors from try_new are iceberg DataInvalid errors. A missing source field was previously Unexpected.

Are these changes tested?

Yes.

  • test_insert_into_partitioned_through_reordering_projection (integration) runs the INSERT above. It snapshots the optimized plan, checks the partitions recorded in the manifests and reads back with a filter on the partition column. On main the manifests record partitions {1, 2} instead of {100, 200}.
  • Unit tests cover remapping the children, rebuilding from the retained spec, schema and children, equality depending on the children, the child-count check, each try_new error by kind and message, and a nested source under a null parent struct (unchanged behavior, previously untested).

cargo fmt --all -- --check, cargo clippy --workspace --locked --all-targets -- -D warnings and cargo test --workspace --locked all pass locally.

Are there any user-facing changes?

  • Breaking: PartitionExpr::try_new takes a third argument, sources: Vec<Arc<dyn PhysicalExpr>>. A codec that rebuilds the expression should pass the decoded children.
  • EXPLAIN output for partitioned inserts uses the format above.
  • The try_new error kinds above.

#19 relaxes the input schema check in project_with_partition. A plan-time check of the source types is left for a follow-up, so that it can use the same rule.

AI Disclosure

  • Used Claude Code to find the cause, write the fix and tests and draft this description. I reviewed the change and reproduced the failure on main.

PartitionExpr read its source columns by position in the input batch but
reported no children. When ProjectionPushdown merged the partition
projection into the projection below it, nothing rewrote those positions,
so partition values were computed from the wrong columns. Files landed in
the wrong partitions and partition pruning skipped them.

PartitionExpr now has one child per partition field, the top-level column
holding its source, and evaluates the transforms itself, descending into
structs for nested sources and nulling values under a null parent. The
optimizer rewrites the children like any other column reference.

The public `try_new` now requires these sources, so a codec must pass the
decoded children; only `project_with_partition`, which checks that its
input is laid out like the table, builds them from the table layout. Spec
and schema errors are reported as iceberg DataInvalid errors.

Closes apache#35
Also build source paths by pushing positions on the way down rather than
prepending them, and say why Hash leaves the sources out.
@NoahKusaba
NoahKusaba force-pushed the fix/partition-expr-children branch from 4e98604 to 2552390 Compare October 29, 2026 19:33
Hash the sources along with the spec and schema ids, so expressions that
differ only in their inputs no longer collide. The rebuild test now passes
freshly built children, as a codec would, so it fails if sources are
compared by pointer. Clarify in AGENTS.md that PartitionExpr reads the
batch only through its children.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Partitioned INSERT computes partition values from the wrong columns when the input plan ends in a projection

1 participant