fix: preserve projection and cast target field metadata

[adriangb]

Which issue does this PR close?

• Closes Projection field metadata is lost during physical plan optimization #24721

Rationale for this change

This is an alternative to #24670 by @gene-bordegaray, opened as a draft to show the diff and CI for the smaller shape rather than to compete with it. The projection commit here is his work, taken verbatim.

Field metadata on a ProjectionExec's output schema can silently disappear when the physical optimizer removes or rewrites projections, so query results lose declared field metadata and metadata-reading expressions return wrong answers. That is #24721, and @gene-bordegaray diagnosed it in three parts (identity-projection removal, collapsing across a metadata boundary, and make_with_child rederiving the schema).

Fixing those exposes a second, pre-existing bug. The logical Expr::Cast/Expr::TryCast carry a FieldRef target so a cast can express a destination that is more than a DataType — for example an extension type produced by a TypePlanner. cast_output_field ignored that field's metadata entirely and always inherited the source's, so Expr::to_field() disagreed with the physical CastExpr, which already treats a non-synthesized target field as authoritative. The disagreement was masked because the optimizer rederives a projection's schema from its expressions, repairing the logical schema on the way through. Once projections preserve metadata faithfully, the repair stops and a cast to an extension type loses it:

SELECT arrow_metadata(CAST(raw AS UUID), 'ARROW:extension:name')
-- 'arrow.uuid' before, NULL after

#24670 currently addresses that by introducing a CastTarget enum (stacked from #24725), which changes the public Cast.field/TryCast.field types and fails the semver check, blocking a 55.1 backport. This PR takes the target's metadata when it carries any and otherwise inherits the source's, which resolves the same divergence with no API change.

What changes are included in this PR?

Two commits, each green on its own:

1. preserve projection field metadata during physical optimization — @gene-bordegaray's physical-plan/src/projection.rs from fix: preserve projection metadata during optimization #24670, unmodified, applied to current main.
2. use the cast target's metadata when it carries any — 20 lines in expr/src/expr_schema.rs.

No public types change, so cargo-semver-checks stays clean.

The rule is deliberately narrower than the three-part sentinel (name.is_empty() && is_nullable && metadata.is_empty()) used by the physical cast and by #23169. That sentinel reclassifies a protobuf-roundtripped CAST(x AS Utf8) with nullable = false as an explicit target and drops its source metadata; an emptiness check cannot change behaviour for any cast whose target carries no metadata, which seems the safer property for a patch release.

Deliberately not included, to keep this reviewable and backportable:

• Stripping ARROW:extension:name/:metadata when a cast changes the storage type (Propagation of metadata through casts can strips extension type from the destination field and can result in invalid extension types #22079). Real bug, user-visible behaviour change, belongs on its own.
• The protobuf gap where to_proto writes field.metadata() and from_proto rebuilds from arrow_type + nullable only (Physical-plan serialization drops ProjectionExec metadata #24695), already tracked separately by @gene-bordegaray.
• Any change to how the cast target is represented. Whether that should be a CastTarget enum, the existing sentinel, or dropping source-metadata inheritance entirely is a real discussion, but it is a design question for the next major rather than a blocker on a correctness fix.

What is the testing strategy for this PR?

The existing datafusion/sqllogictest/test_files/cast_extension_type_metadata.slt is the regression test: with commit 1 alone it reproduces the CI failure @gabotechs reported on #24670, byte for byte and at the same line, and commit 2 makes it pass.

[Diff] (-expected|+actual)
-   00010203040506070809000102030506 arrow.uuid
+   00010203040506070809000102030506 NULL
at datafusion/sqllogictest/test_files/cast_extension_type_metadata.slt:36

Commit 2 also adds expr_schema::tests::test_cast_output_field_metadata, covering both directions of the rule for Cast and TryCast. Reverting the rule while keeping the test makes it fail, so it is load-bearing rather than decorative.

Locally: the full sqllogictest suite (504/504 files) and the datafusion-expr, -physical-expr, -physical-plan, -sql, -proto and -optimizer unit tests (5495 passed, 0 failed) are green.

Are there any user-facing changes?

Yes, both are bug fixes to schemas that were already wrong:

• Field metadata declared by a projection survives physical optimization.
• A cast whose target field carries metadata reports that metadata on to_field(), matching what the physical plan already produces. A plain CAST(expr AS type) synthesizes a target with no metadata, so its behaviour is unchanged.

No breaking changes to public APIs.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 89.47368% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.58%. Comparing base (a274959) to head (7d650aa).

Files with missing lines	Patch %	Lines
datafusion/physical-plan/src/projection.rs	84.15%	4 Missing and 12 partials ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #24831    +/-   ##
========================================
  Coverage   81.58%   81.58%            
========================================
  Files        1123     1123            
  Lines      406610   406748   +138     
  Branches   406610   406748   +138     
========================================
+ Hits       331719   331836   +117     
- Misses      55453    55460     +7     
- Partials    19438    19452    +14     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.