fix: fall back for concat_ws with array arguments instead of failing natively

[peterxcli]

Which issue does this PR close?

Closes #5675.

Rationale for this change

Spark's ConcatWs accepts array<string> arguments after the separator and flattens their elements into the strings to join (concat_ws(',', array('a','b'), 'c') = a,b,c, null elements skipped). CometConcatWs.getSupportLevel never inspected child types and lowered every non-foldable call to DataFusion's concat_ws, which accepts only string arguments, so concat_ws(',', arr, s) failed at native execution with Input was List(...) which is not a supported datatype for concat_ws function where Spark returns a result.

What changes are included in this PR?

• CometConcatWs mixes in CodegenDispatchFallback and returns Unsupported (with a documented reason) whenever any argument has an ArrayType. These calls now run through the JVM codegen dispatcher (Spark's own ConcatWs.doGenCode inside the Comet pipeline) and fall back to Spark with that reason only when spark.comet.exec.scalaUDF.codegen.enabled=false. The NULL-separator shortcut and the native path for plain string arguments are unchanged.
• expressions.md: concat_ws is now Hybrid (what GenerateDocs emits for a CodegenDispatchFallback serde) with a note; the compatibility page picks up the new unsupported reason automatically.
• Follow-up (not in this PR): a Spark-compatible native concat_ws kernel that flattens list arguments (skipping null elements like Spark) so these calls can leave the dispatcher.

How are these changes tested?

• New CometStringExpressionSuite test concat_ws with array<string> arguments: over a Parquet table with an array<string> column it runs concat_ws(',', arr, s), concat_ws(',', s, arr), concat_ws(',', arr), concat_ws('-', arr, s, arr), concat_ws(',', split(s, ' ')) and concat_ws(',', split(s, ' '), arr) and asserts Spark's answer with the whole plan in Comet and the dispatcher actually running; with the dispatcher disabled it asserts the Spark answer plus the serde's fallback reason; it also asserts a NULL separator still runs natively and that plain string arguments keep the native path.
• sql-tests/expressions/string/concat_ws.sql: a new array<string> table and eight query blocks (column, mixed, single, repeated, split-produced and literal array arguments, NULL separator), all asserting Spark-identical results.
• Ran CometStringExpressionSuite and CometSqlFileTestSuite concat_ws (Spark 4.1 profile): 35/35 passed. Native code is untouched.

[sunchao]

Summary

concat_ws with array arguments now uses Spark's generated implementation through the existing JVM codegen dispatcher instead of reaching a native function that rejects lists. Ordinary string-column calls and the literal-NULL separator shortcut retain their native paths. I found no actionable P1/P2 issue in the reviewed changes.

Correctness and compatibility

I checked the maintained Spark 3.5 and 4.0 implementations, including array flattening, null arrays and elements, empty inputs, argument order and separator handling. The new input-shape guard and shared fallback path preserve those semantics. The SQL cases exercise the results, while the Scala assertions verify dispatcher use, disabled-dispatcher fallback and the unchanged string-only path.

Validation and CI

The new array regression and concat_ws.sql passed in the Spark 3.5 Linux job and Spark 4.0 macOS job. These results are from CI merge db59f299ef692c7276285bfb16f9400243d8e33a, whose relevant implementation and test sources match this head, not a local bare-head execution. At 14:11 UTC, 63 checks had succeeded, eight were skipped and two were still running. I did not run a local Spark build or benchmark.

Performance

The array case pays the existing cached codegen dispatch and Spark concatenation costs. The patch does not add a second array-flattening implementation or new work to the ordinary string-column path. This is a correctness fallback, not a measured speedup. Native array support remains separately tracked in #5675.

Design

Reusing CodegenDispatchFallback keeps support detection, dispatcher admission and disabled-dispatcher behavior in the established flow. The shared unsupported reason also feeds the generated compatibility documentation, while the hand-edited expression list accurately describes the hybrid implementation.

Abstraction & complexity

The change is limited to the existing trait, a shared reason and an array-type guard. The extra Scala coverage is justified by the dispatcher counters and fallback assertions that the SQL cases cannot express directly. I do not see an additional abstraction or simpler alternative needed before merge.

[sunchao]

Merged, thanks @peterxcli !

Maybe we should also implement native array support as separate follow-up items.

[peterxcli]

Maybe we should also implement native array support as separate follow-up items.

filed #5687, thanks for the suggestion!