[andygrove]

Which issue does this PR close?

No tracking issue. This is a dependency upgrade.

Rationale for this change

Comet pins arrow-java at 18.3.0. 19.0.0 is the latest release and the version Spark 4.2.0 ships. Comet shades Arrow, so it does not have to match Spark's version, but staying current keeps us on the upstream serialization and C Data fixes (for example GH-343, offset buffers for empty and nested arrays in IPC streams, which the Python UDF runner writes).

The upgrade is not just a version change. Arrow Java 19 changes two behaviors that Comet depended on.

What changes are included in this PR?

• pom.xml: arrow.version 18.3.0 to 19.0.0.
• ArrowImporter: Arrow Java 19 no longer closes the source ArrowArray struct after ArrayImporter.importArray moves its contents into the importer's copy (fix: Unsupported types for SinglePartition should fallback to Spark #765). ArrowImporter relied on that, so every imported column would leak its ArrowArray struct (128 bytes) in CometArrowImportAllocator per batch. It now closes the source itself, guarded by isClosed(). The error path comment is updated to match.
• Utils.scala, UtilsSuite.scala: BitVectorHelper.getValidityBufferSize is replaced by getValidityBufferSizeFromCount. The two names never exist in the same release, so this cannot compile against 18.3.
• UtilsSuite: setValueCount(0) on a large variable-width vector now allocates an 8-byte offset buffer. The empty-vector test needs a vector with no offset buffer, so it uses an unallocated one.

apache/arrow-java#88 (C Data import ignores ArrowArray.offset) is still open in 19.0.0, so the workarounds in ffi.md stay.

How are these changes tested?

On the default Spark 4.1 profile, 18 suites that exercise Arrow ran: the Arrow writers and streams, NativeUtil, UtilsSuite, CometBroadcastValueSuite, native shuffle, the in-memory cache, Parquet read, the UDF bridge, codegen dispatch, the Python runner and the Arrow Python UDF suites. Result: 440 passed, 0 failed, 9 canceled. Without the ArrowImporter change, NativeUtilSuite (4 tests) and CometBroadcastValueSuite (1 test) fail with leaked allocator memory, and the empty-vector UtilsSuite test fails without its edit. The same three suites pass on 18.3.0 before the upgrade.

I also built the shaded jar and checked it: only org/apache/arrow/c/** stays unshaded, as configured, and the compression service file is relocated.

Not covered by this local run, so CI should decide:

• The 9 canceled tests are the native Arrow Python UDF tests, which skip themselves because my native build lacked the python-udf feature. The PyArrow end-to-end path was not run.
• Spark 3.4, 3.5, 4.0 and 4.2 profiles, and the Iceberg suites.