refactor(proto): destructure FileScanConfig and MemorySourceConfig serde hooks

[buraksenn]

Which issue does this PR close?

• Closes Destructure proto hooks for FileScanConfig and MemorySourceConfig #24624.

Rationale for this change

Proto hooks that access fields individually can silently omit newly added state. Exhaustive destructuring makes such omissions compile errors.

FileScanConfig is shared by all file sources, so a field omitted here is lost for Parquet, CSV, JSON, Arrow, and Avro.

What changes are included in this PR?

• Exhaustively destructure FileScanConfig and MemorySourceConfig in their encoders.
• Exhaustively destructure their protobuf nodes in the decoders.
• Document fields that are serialized indirectly or reconstructed during decoding.
• Add an optional preserve_order field so explicit values survive file scan round trips while older payloads retain their ordering-derived behavior.
• Reject default serialization of non-default expression adapter factories instead of silently discarding their behavior, and update the custom serialization example accordingly.
• Decode projected MemorySourceConfig sort information against the projected schema without applying projection twice.

The protobuf change is additive and backward compatible.

Are these changes tested?

Added or updated coverage for:

• explicit preserve_order values and legacy payload behavior
• default and custom expression adapter factories
• projected memory-source sort information, projection, fetch, and display settings

Are there any user-facing changes?

File scans now preserve an explicit preserve_order value across protobuf round trips. Default serialization of plans containing custom expression adapter factories now returns an explicit error rather than silently dropping the factory.

Projected memory sources now preserve their sort information across round trips. The new protobuf field is additive, and PhysicalExprAdapterFactory gains a default method, so existing implementations remain source compatible.

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-datasource v55.0.0 (current)
       Built [  42.709s] (current)
     Parsing datafusion-datasource v55.0.0 (current)
      Parsed [   0.030s] (current)
    Building datafusion-datasource v55.0.0 (baseline)
       Built [  42.743s] (baseline)
     Parsing datafusion-datasource v55.0.0 (baseline)
      Parsed [   0.030s] (baseline)
    Checking datafusion-datasource v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.313s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  87.531s] datafusion-datasource
    Building datafusion-physical-expr-adapter v55.0.0 (current)
       Built [  32.062s] (current)
     Parsing datafusion-physical-expr-adapter v55.0.0 (current)
      Parsed [   0.010s] (current)
    Building datafusion-physical-expr-adapter v55.0.0 (baseline)
       Built [  32.030s] (baseline)
     Parsing datafusion-physical-expr-adapter v55.0.0 (baseline)
      Parsed [   0.010s] (baseline)
    Checking datafusion-physical-expr-adapter v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.102s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  65.338s] datafusion-physical-expr-adapter
    Building datafusion-proto v55.0.0 (current)
       Built [  53.030s] (current)
     Parsing datafusion-proto v55.0.0 (current)
      Parsed [   0.017s] (current)
    Building datafusion-proto v55.0.0 (baseline)
       Built [  53.245s] (baseline)
     Parsing datafusion-proto v55.0.0 (baseline)
      Parsed [   0.018s] (baseline)
    Checking datafusion-proto v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.143s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 107.858s] datafusion-proto
    Building datafusion-proto-models v55.0.0 (current)
       Built [  24.837s] (current)
     Parsing datafusion-proto-models v55.0.0 (current)
      Parsed [   0.126s] (current)
    Building datafusion-proto-models v55.0.0 (baseline)
       Built [  24.755s] (baseline)
     Parsing datafusion-proto-models v55.0.0 (baseline)
      Parsed [   0.126s] (baseline)
    Checking datafusion-proto-models v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   2.203s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---

Description:
A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field FileScanExecConf.preserve_order in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1965
  field FileScanExecConf.preserve_order in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1965

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  53.321s] datafusion-proto-models

[codecov-commenter]

Codecov Report

❌ Patch coverage is 82.11382% with 44 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.52%. Comparing base (9ef3d47) to head (b7c55a0).
⚠️ Report is 10 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/proto/src/physical_plan/mod.rs	64.58%	10 Missing and 7 partials ⚠️
datafusion/proto-models/src/generated/pbjson.rs	0.00%	13 Missing ⚠️
datafusion/datasource/src/memory.rs	87.50%	1 Missing and 7 partials ⚠️
...atafusion/datasource/src/file_scan_config/proto.rs	94.78%	0 Missing and 6 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24813      +/-   ##
==========================================
- Coverage   81.52%   81.52%   -0.01%     
==========================================
  Files        1123     1123              
  Lines      405970   406349     +379     
  Branches   405970   406349     +379     
==========================================
+ Hits       330983   331277     +294     
- Misses      55626    55689      +63     
- Partials    19361    19383      +22     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.