[VL] Take Arrow out of the bundle for Spark 4.x and drop dead arrow-dataset

[jackylee-ch]

What changes are proposed in this pull request?

Two independent, self-contained changes that shrink the Velox bundle and cut down
Arrow version conflicts on Spark 4.x.

1) Spark 4.x reuses the Arrow that Spark itself ships

Spark 4.x already ships an Arrow new enough for Gluten, so Gluten no longer needs to
carry its own copy inside the bundle. Two new properties drive this per Spark profile:

property	3.3/3.4/3.5	4.0/4.1
arrow.deps.scope	compile	provided
spark.arrow.exclusion.groupId	org.apache.arrow	none

• arrow.deps.scope flips Gluten's Arrow dependencies from compile (shipped inside
  the bundle) to provided (compile-only; the Spark distribution owns Arrow at runtime).
• spark.arrow.exclusion.groupId parameterizes the <exclusion> groupId on the Spark
  dependencies in dependencyManagement. Setting it to none makes those exclusions
  stop matching, so Spark's own Arrow flows through transitively — including into the
  gluten-ut test classpath, which provided scope alone cannot reach.

Because provided scope is not transitive, backends-velox (which imports
org.apache.arrow.{memory,vector} directly) re-declares those Arrow artifacts at
${arrow.deps.scope} so it keeps its own compile classpath.

arrow.version now matches what each Spark actually ships: spark-4.0 → 18.1.0,
spark-4.1 → 18.3.0. Spark 3.3/3.4/3.5 keep 15.0.0 at compile, unchanged.

arrow-c-data stays compile/bundled on every Spark version: Spark ships neither
arrow-c-data nor its libarrow_cdata_jni, and its JNI symbols bind to the original
package names, so it can be neither provided nor relocated. Since org.apache.arrow.c.*
reaches into org.apache.arrow.util.* (Preconditions, AutoCloseables, Collections2)
from its constructors — references that live in the constant pool, not in any public
signature — org.apache.arrow.util.** is now also excluded from relocation in
package/pom.xml. Otherwise, once Arrow is no longer bundled, those shaded call sites
would have no target on the classpath and throw ClassNotFoundException.
dev/check-arrow-c-shading.sh is extended to scan constant pools (not just method
signatures) to guard this.

package/pom.xml needs no change to its shading logic: under provided the Arrow
artifacts never enter the shade artifactSet, so the org.apache.arrow relocation is a
no-op on 4.x while still applying on 3.x.

Resulting bundle sizes:

bundle	before	after
spark-4.1	138.6 MB	56.4 MB compressed (600.6 → 299.6 MB uncompressed)
spark-3.5	138.6 MB	68.6 MB compressed

The only Arrow left in the spark-4.1 bundle is arrow-c-data — 35 classes plus ~0.9 MB
of libarrow_cdata_jni.

2) Drop dead arrow-dataset

ArrowNativeMemoryPool / ArrowReservationListener were the only main-source users of
arrow-dataset, and have had no callers since #12130 removed the Arrow-CSV scan path.
This PR removes both classes, the arrow-dataset dependency, and the three
backends-velox/.../fs/ tests, which exercised Arrow's own FileSystemDatasetFactory
rather than any Gluten code.

Note for reviewers

On Spark 4.x the runtime Arrow allocator changes from arrow-memory-unsafe to
arrow-memory-netty: Gluten's default arrow-memory.artifact is still
arrow-memory-unsafe, but under provided scope it is no longer on the runtime
classpath, while Spark 4.x ships arrow-memory-netty. Arrow's CheckAllocator therefore
selects the Netty allocator. Both expose off-heap ArrowBuf with a stable
memoryAddress(), so JNI address hand-off to native is unaffected.

How was this patch tested?

• Existing Spark 4.0 / 4.1 Velox CI — the gluten-ut/spark40 and gluten-ut/spark41
  suites plus the TPC-H / TPC-DS gluten-it runs in velox_backend_x86.yml — exercises
  the provided classpath end to end (Spark's own Arrow on the runtime classpath).
• Verified the bundled arrow-c-data classes reference the unshaded Apache Arrow API in
  both method signatures and constant pools via the extended
  dev/check-arrow-c-shading.sh.
• Confirmed the resulting bundle sizes above for spark-3.5 and spark-4.1.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude claude-opus-4.7

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[jackylee-ch]

@zhouyuan @zhztheplayer PTAL

[Copilot]

Pull request overview

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Reduce Velox bundle size and Spark 4.x Arrow conflicts by relying on Spark-provided Arrow at runtime and removing unused Arrow Dataset integration.

Changes:

• Switch Spark 4.x builds to use Arrow dependencies as provided and allow Spark’s transitive Arrow to flow through by parameterizing Arrow exclusions.
• Remove dead arrow-dataset usage (dependencies, Arrow memory pool classes, and dataset-based filesystem tests).
• Extend shading verification to detect shaded Arrow references in constant pools and keep org.apache.arrow.util.** unshaded for Arrow C-Data.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
pom.xml	Add arrow.deps.scope / spark.arrow.exclusion.groupId properties and Spark 4.x profile overrides for provided Arrow.
package/pom.xml	Update shading relocation excludes to keep org.apache.arrow.util.** unshaded alongside C-Data/memory/vector.
gluten-arrow/pom.xml	Scope Arrow deps via ${arrow.deps.scope}; remove arrow-dataset; prevent arrow-c-data from reintroducing compile-scope Arrow deps.
dev/check-arrow-c-shading.sh	Enhance verification to scan constant pools for shaded Arrow references.
backends-velox/pom.xml	Re-declare Arrow deps so they remain available on compile classpath when upstream Arrow is provided.
gluten-arrow/src/main/java/org/apache/gluten/memory/arrow/pool/ArrowReservationListener.java	Remove unused listener tied to arrow-dataset.
gluten-arrow/src/main/java/org/apache/gluten/memory/arrow/pool/ArrowNativeMemoryPool.java	Remove unused memory-pool wrapper tied to arrow-dataset.
backends-velox/src/test/java/org/apache/gluten/fs/TestNativeDataset.java	Remove dataset-based test base class (Arrow dataset).
backends-velox/src/test/java/org/apache/gluten/fs/TestDataset.java	Remove dataset-based test utilities (Arrow dataset).
backends-velox/src/test/java/org/apache/gluten/fs/CsvWriteSupport.java	Remove CSV temp writer used only by Arrow dataset tests.
backends-velox/src/test/java/org/apache/gluten/fs/ArrowFilesystemTest.java	Remove Arrow FileSystemDatasetFactory test (not exercising Gluten code).

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Suppressed comments (2)

dev/check-arrow-c-shading.sh:111

• The script claims to validate org/apache/arrow/c/** constant pools, but the unzip pattern only extracts org/apache/arrow/c/* (non-recursive). That will skip any classes under org/apache/arrow/c/jni/ (which are also explicitly excluded from relocation in package/pom.xml), so shaded constant-pool references there would go undetected. Recommendation: extract and scan both org/apache/arrow/c/* and org/apache/arrow/c/jni/* (or use a recursive pattern) and update the existence check accordingly.

unzip -qo "$JAR" 'org/apache/arrow/c/*' -d "$WORKDIR/all" 2>/dev/null || true
if compgen -G "$WORKDIR/all/org/apache/arrow/c/*.class" > /dev/null; then
  refs=$(grep -rahoE "${SHADE_SLASHES}/org/apache/arrow/[a-zA-Z0-9_$/-]+" \
    "$WORKDIR/all/org/apache/arrow/c" 2>/dev/null | sort -u || true)
  if [[ -n "$refs" ]]; then
    echo "  FAIL org/apache/arrow/c/** — calls into gluten-shaded Arrow:"
    echo "$refs" | sed 's/^/    /'
    failures=$((failures + 1))

backends-velox/pom.xml:137

• This PR duplicates a fairly large, exclusion-heavy set of Arrow dependency declarations that are also present in gluten-arrow/pom.xml. That creates a drift risk (e.g., future exclusions/scope tweaks applied to one module but not the other). Recommendation: centralize these Arrow dependency definitions (including exclusions and scopes) in a shared place such as parent dependencyManagement (or a dedicated Maven profile/BOM-style module), and have both modules reference the managed dependencies without re-stating the full blocks.

    <!--
      Re-declared here because `provided` scope is not transitive: when
      ${arrow.deps.scope} is `provided` (Spark 4.x), gluten-arrow's Arrow
      dependencies do not reach this module's compile classpath on their own.
      Under `compile` (Spark 3.x) these are redundant but harmless.

      The allocator implementation (${arrow-memory.artifact}) is the exception:
      it is never compiled against, only discovered on the classpath at runtime,
      so it uses ${arrow-memory.scope} (runtime on Spark 3.x, provided on 4.x)
      rather than ${arrow.deps.scope}.
    -->
    <dependency>
      <groupId>org.apache.arrow</groupId>
      <artifactId>${arrow-memory.artifact}</artifactId>
      <version>${arrow.version}</version>
      <scope>${arrow-memory.scope}</scope>
      <exclusions>
        <exclusion>
          <groupId>io.netty</groupId>
          <artifactId>netty-common</artifactId>
        </exclusion>
        <exclusion>
          <groupId>io.netty</groupId>
          <artifactId>netty-buffer</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.arrow</groupId>
      <artifactId>arrow-memory-core</artifactId>
      <version>${arrow.version}</version>
      <scope>${arrow.deps.scope}</scope>
      <exclusions>
        <exclusion>
          <groupId>io.netty</groupId>
          <artifactId>netty-common</artifactId>
        </exclusion>
        <exclusion>
          <groupId>io.netty</groupId>
          <artifactId>netty-buffer</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.arrow</groupId>
      <artifactId>arrow-vector</artifactId>
      <version>${arrow.version}</version>
      <scope>${arrow.deps.scope}</scope>
      <exclusions>
        <exclusion>
          <groupId>io.netty</groupId>
          <artifactId>netty-common</artifactId>
        </exclusion>
        <exclusion>
          <groupId>io.netty</groupId>
          <artifactId>netty-buffer</artifactId>

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Suppressed comments (2)

dev/check-arrow-c-shading.sh:107

• The constant-pool scan only extracts org/apache/arrow/c/*, so it can miss shaded references in subpackages like org/apache/arrow/c/jni/*. Since the diagnostic text claims to validate org/apache/arrow/c/**, the unzip pattern and existence check should be recursive (at least one level) so the guard can't pass vacuously.

mkdir -p "$WORKDIR/all"
unzip -qo "$JAR" 'org/apache/arrow/c/*' -d "$WORKDIR/all" 2>/dev/null || true
if compgen -G "$WORKDIR/all/org/apache/arrow/c/*.class" > /dev/null; then
  refs=$(grep -rahoE "${SHADE_SLASHES}/org/apache/arrow/[a-zA-Z0-9_$/-]+" \
    "$WORKDIR/all/org/apache/arrow/c" 2>/dev/null | sort -u || true)

dev/check-arrow-c-shading.sh:61

• SHADE_SLASHES is intended to convert the dotted shade package (e.g. org.apache.gluten.shaded) into a slashed JVM internal-name prefix, but ${SHADE_PACKAGE//.//} removes dots instead of replacing them with /. That makes the constant-pool scan regex never match and can let shaded references slip through undetected.

This issue also appears on line 103 of the same file.

# Dotted form for javap signatures, slashed form for JVM internal names in
# constant pools. `.` is escaped so the dotted form is a literal regex.
SHADE_PACKAGE="${2:-org.apache.gluten.shaded}"
SHADE_DOTS_RE="${SHADE_PACKAGE//./\\.}"
SHADE_SLASHES="${SHADE_PACKAGE//.//}"

[zhouyuan]

@jackylee-ch in Gluten there is an extra profile to allow users to pick arrow-memory-netty based allocator (https://github.com/apache/gluten/blob/main/pom.xml#L1478-L1487), will this patch block this usage?

[zhouyuan]

is there a way to verify Gluten is using the arrow jar from spark distribution?

[github-actions]

Run Gluten Clickhouse CI on x86

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

[github-actions]

Run Gluten Clickhouse CI on x86

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Suppressed comments (1)

dev/check-arrow-c-shading.sh:127

• The constant-pool scan can be skipped unintentionally: the existence check relies on the ** glob, but the script never enables globstar (shopt -s globstar). On a default Bash config, compgen -G "$WORKDIR/.../**/*.class" won’t match nested classes (e.g., org/apache/arrow/c/jni/*.class), so shaded constant-pool references could slip through without failing the build.

if compgen -G "$WORKDIR/all/org/apache/arrow/c/**/*.class" > /dev/null ||
   compgen -G "$WORKDIR/all/org/apache/arrow/c/*.class" > /dev/null; then
  refs=$(grep -rahoE "${SHADE_SLASHES}/org/apache/arrow/[a-zA-Z0-9_$/-]+" \
    "$WORKDIR/all/org/apache/arrow/c" 2>/dev/null | sort -u || true)

[jackylee-ch]

No — -Parrow-netty still overrides ${arrow-memory.artifact} to arrow-memory-netty. On Spark 3.x it bundles netty at runtime; on 4.x both allocators resolve to provided since Spark ships netty natively.

[github-actions]

Run Gluten Clickhouse CI on x86

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Suppressed comments (1)

dev/check-arrow-c-shading.sh:135

• The existence check for extracted Arrow C-Data classes relies on a ** glob (compgen -G .../**/*.class) but the script never enables globstar, so Bash treats ** literally. This can cause the constant-pool scan to be skipped when classes are only present under org/apache/arrow/c/jni/ (or any nested path), reducing the effectiveness of the new guard.

if compgen -G "$WORKDIR/all/org/apache/arrow/c/**/*.class" > /dev/null ||
   compgen -G "$WORKDIR/all/org/apache/arrow/c/*.class" > /dev/null; then