Repository navigation
fix: read partition source columns through PartitionExpr children - #92
Draft
NoahKusaba wants to merge 3 commits into
Draft
NoahKusaba wants to merge 3 commits into
NoahKusaba wants to merge 3 commits into
Conversation
PartitionExpr read its source columns by position in the input batch but reported no children. When ProjectionPushdown merged the partition projection into the projection below it, nothing rewrote those positions, so partition values were computed from the wrong columns. Files landed in the wrong partitions and partition pruning skipped them. PartitionExpr now has one child per partition field, the top-level column holding its source, and evaluates the transforms itself, descending into structs for nested sources and nulling values under a null parent. The optimizer rewrites the children like any other column reference. The public `try_new` now requires these sources, so a codec must pass the decoded children; only `project_with_partition`, which checks that its input is laid out like the table, builds them from the table layout. Spec and schema errors are reported as iceberg DataInvalid errors. Closes apache#35
Also build source paths by pushing positions on the way down rather than prepending them, and say why Hash leaves the sources out.
NoahKusaba
force-pushed
the
fix/partition-expr-children
branch
from
October 29, 2026 19:33
4e98604 to
2552390
Compare
Hash the sources along with the spec and schema ids, so expressions that differ only in their inputs no longer collide. The rebuild test now passes freshly built children, as a codec would, so it fails if sources are compared by pointer. Clarify in AGENTS.md that PartitionExpr reads the batch only through its children.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
PartitionExprread its source columns by position in the input batch, but reported no children. WhenProjectionPushdownmerged the partition projection into the projection below it, it had no column references to rewrite, so the positions kept pointing at the old layout:partitioned the rows by
ainstead ofb. Files landed in the wrong partitions, and partition pruning then skipped matching rows.What changes are included in this PR?
Sources are children.
PartitionExprhas one child per partition field: the top-level column holding that field's source. The optimizer rewrites them like any other column reference, and EXPLAIN shows what each field reads:Transforms are evaluated in place. The expression applies the transforms itself instead of going through iceberg's positional
PartitionValueCalculator. A nested source is read by descending from its top-level column, nulling values under a null parent struct as iceberg'sRecordBatchProjectordid.The public API can't build a broken expression.
PartitionExpr::try_new(spec, schema, sources)now takes the sources, so a codec passes the decoded children instead of assuming a table-shaped input. Onlyproject_with_partition, which checks that its input matches the table schema, derives them from the table (crate-privatetry_new_over_table).with_new_childrenrejects the wrong number of children.Spec and schema errors from
try_neware icebergDataInvaliderrors. A missing source field was previouslyUnexpected.Are these changes tested?
Yes.
test_insert_into_partitioned_through_reordering_projection(integration) runs theINSERTabove. It snapshots the optimized plan, checks the partitions recorded in the manifests and reads back with a filter on the partition column. Onmainthe manifests record partitions{1, 2}instead of{100, 200}.try_newerror by kind and message, and a nested source under a null parent struct (unchanged behavior, previously untested).cargo fmt --all -- --check,cargo clippy --workspace --locked --all-targets -- -D warningsandcargo test --workspace --lockedall pass locally.Are there any user-facing changes?
PartitionExpr::try_newtakes a third argument,sources: Vec<Arc<dyn PhysicalExpr>>. A codec that rebuilds the expression should pass the decoded children.try_newerror kinds above.#19 relaxes the input schema check in
project_with_partition. A plan-time check of the source types is left for a follow-up, so that it can use the same rule.AI Disclosure
main.