Skip to content

Fix native SIGSEGV when hold() races with release() on NativeMemoryManager - #12714

Open
yikf wants to merge 1 commit into
apache:mainfrom
yikf:fix-nmm-hold-use-after-release
Open

Fix native SIGSEGV when hold() races with release() on NativeMemoryManager#12714
yikf wants to merge 1 commit into
apache:mainfrom
yikf:fix-nmm-hold-use-after-release

Conversation

@yikf

@yikf yikf commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

What changes are proposed in this pull request?

NativeMemoryManager.hold() called the native hold(handle) JNI method without checking whether the manager had already been released. When a task tears down its runtime (release() frees the native handle) while another thread is still closing an output iterator via ColumnarBatchOutIterator.close0() -> memoryManager().hold(), hold() dereferences a freed handle and crashes the JVM with a SIGSEGV (SEGV_MAPERR = use-after-free).

Surfaced as a flaky native crash in CI running GlutenSparkScriptTransformationSuite (its TRANSFORM ... USING tests close the columnar output iterator on a feed thread concurrently with task teardown):

From hs_err_pid*.log :

# C  [libgluten.so+0x4fe59e]  Java_org_apache_gluten_memory_NativeMemoryManagerJniWrapper_hold+0x1e
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR)

Current thread: JavaThread "Thread-SparkScriptTransformationWriterThread-Feed"
C  [libgluten.so+0x4fe59e]  Java_org_apache_gluten_memory_NativeMemoryManagerJniWrapper_hold+0x1e
j  org.apache.gluten.memory.NativeMemoryManager$Impl.hold()V
j  org.apache.gluten.vectorized.ColumnarBatchOutIterator.close0()V
j  org.apache.gluten.iterator.ClosableIterator.close()V
...
j  org.apache.spark.sql.execution.BaseScriptTransformationWriterThread.run()V

How was this patch tested?

flaky test, existed test to verify.

Was this patch authored or co-authored using generative AI tooling?

Yes, AI-assisted, Generated-by: Claude claude-opus-4-8.

Copilot AI lite review requested due to automatic review settings August 6, 2026 08:27
@yikf

yikf commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

@zhztheplayer could you please take a look if you have time, thanks.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a race in NativeMemoryManager where hold() could call into native code after release() had freed the underlying native handle, leading to a JVM crash (SIGSEGV) under concurrent task teardown and iterator close.

Changes:

  • Add an instance-level lock to make hold() and release() mutually exclusive around native JNI calls.
  • Make hold() a safe no-op when the memory manager has already been released, preventing use-after-free.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@zhztheplayer

zhztheplayer commented Aug 6, 2026

Copy link
Copy Markdown
Member

@yikf Thanks. I thought .hold() should never be called after .release() is called (Because all ColumnarBatchOutIterator instances should be closed before the Runtime is destroyed at the end of the Spark task). Is there a way to repeat the failure model from CI locally or in test?

@yikf

yikf commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @zhztheplayer. You're right that in the normal flow the iterator is closed before the Runtime is destroyed — but the ordering guarantee only holds within the task thread, and script transformation breaks it by closing the iterator from a second thread.

resIter.close() (which calls hold()) can be triggered from two places which are recycler and hasNext, both funneled through IteratorCompleter.tryComplete() and guarded by a CAS so it runs exactly once:

  • the TaskResources recycler (priority 100), run on the task thread at task completion;
  • IteratorCompleter.hasNext returning false, run on whatever thread is driving the iterator.

In the normal single-threaded path both run on the task thread, and releaseAll() executes resources by descending priority, so the iterator recycler (100) holds before the Runtime (30) releases — hold() always precedes release().

SparkScriptTransformationExec.processIterator starts BaseScriptTransformationWriterThread (a setDaemon(true) "Feed" thread) and never joins it; that thread drains the Gluten input iterator via iter.foreach. The race:

1. Feed thread: hasNext hits end-of-stream → tryComplete() wins the CAS (marks completed) but gets descheduled before running resIter.close().
2. Task thread: main thread finishes reading the script's stdout → task completes → the recycler's tryComplete() finds the CAS already taken and no-ops, so the task thread never holds and proceeds straight to Runtime.release() → nmm.release(), freeing the native handle.
3. Feed thread: resumes, runs resIter.close() → ColumnarBatchOutIterator.close0() → hold() on the now-freed handle → SIGSEGV.

So hold() runs exactly once, but on the feed thread, with no happens-before against the task thread's release(). The hs_err stack matches — hold() under BaseScriptTransformationWriterThread.run().

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f01a868459e, pid=7037, tid=7720
#
# JRE version: OpenJDK Runtime Environment (Red_Hat-17.0.6.0.9-0.3.ea.el8) (17.0.6+9) (build 17.0.6-ea+9-LTS)
# Java VM: OpenJDK 64-Bit Server VM (Red_Hat-17.0.6.0.9-0.3.ea.el8) (17.0.6-ea+9-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# C  [libgluten.so+0x4fe59e]  Java_org_apache_gluten_memory_NativeMemoryManagerJniWrapper_hold+0x1e
#
# Core dump will be written. Default location: Core dumps may be processed with "/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %h %d" (or dumping to /__w/gluten/gluten/gluten-ut/spark40/core.7037)
#
# If you would like to submit a bug report, please visit:
#   https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20Linux%208&component=java-17-openjdk
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  S U M M A R Y ------------

Command Line: -Dlog4j.configurationFile=file:src/test/resources/log4j2.properties -Dbasedir=/__w/gluten/gluten/gluten-ut/spark40 -Dspark.test.home=/opt/shims/spark40/spark_home/ -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false -Dio.netty.tryReflectionSetAccessible=true -Dfile.encoding=UTF-8 org.scalatest.tools.Runner -R /__w/gluten/gluten/gluten-ut/spark40/target/scala-2.13/classes /__w/gluten/gluten/gluten-ut/spark40/target/scala-2.13/test-classes -l org.apache.spark.tags.ExtendedSQLTest org.apache.spark.tags.SlowHiveTest org.apache.gluten.tags.UDFTest org.apache.gluten.tags.SkipTest -w org.apache.spark.sql.execution -w org.apache.spark.sql.catalyst -w org.apache.spark.sql.errors -w org.apache.spark.sql.extension -o -u /__w/gluten/gluten/gluten-ut/spark40/target/surefire-reports/.

Host: AMD EPYC 9V74 80-Core Processor, 4 cores, 15G, CentOS Stream release 8
Time: Thu Aug  6 05:15:16 2026 UTC elapsed time: 22.784636 seconds (0d 0h 0m 22s)

---------------  T H R E A D  ---------------

Current thread (0x00007f021007c970):  JavaThread "Thread-SparkScriptTransformationWriterThread-Feed" daemon [_thread_in_native, id=7720, stack(0x00007f0180ed1000,0x00007f0180fd2000)]

Stack: [0x00007f0180ed1000,0x00007f0180fd2000],  sp=0x00007f0180fcff60,  free space=1019k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libgluten.so+0x4fe59e]  Java_org_apache_gluten_memory_NativeMemoryManagerJniWrapper_hold+0x1e
j  org.apache.gluten.memory.NativeMemoryManager$Impl.hold()V+4
j  org.apache.gluten.vectorized.ColumnarBatchOutIterator.close0()V+9
j  org.apache.gluten.iterator.ClosableIterator.close()V+13
j  org.apache.gluten.backendsapi.velox.VeloxIteratorApi.$anonfun$genFinalStageIterator$2(Lscala/Function1;Lorg/apache/gluten/metrics/IteratorMetricsJniWrapper;Lorg/apache/gluten/vectorized/ColumnarBatchOutIterator;)V+13
j  org.apache.gluten.backendsapi.velox.VeloxIteratorApi$$Lambda$7598+0x0000000802b4afc8.apply$mcV$sp()V+12
J 18857 c1 org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.hasNext()Z (23 bytes) @ 0x00007f0226a9bed4 [0x00007f0226a9b780+0x0000000000000754]
J 18316 c1 org.apache.gluten.iterator.IteratorsV1$PayloadCloser.hasNext()Z (14 bytes) @ 0x00007f0225f5a3f4 [0x00007f0225f5a2c0+0x0000000000000134]
j  org.apache.gluten.iterator.IteratorsV1$LifeTimeAccumulator.hasNext()Z+4
j  org.apache.gluten.execution.VeloxColumnarToRowExec$$anon$1.hasNext()Z+4
J 15005 c2 scala.collection.Iterator$$anon$10.hasNext()Z (76 bytes) @ 0x00007f022d0418d0 [0x00007f022d0417c0+0x0000000000000110]
J 18859 c1 org.apache.gluten.iterator.IteratorsV1$InvocationFlowProtection.hasNext()Z (122 bytes) @ 0x00007f0226a9d26c [0x00007f0226a9cc80+0x00000000000005ec]
J 18857 c1 org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.hasNext()Z (23 bytes) @ 0x00007f0226a9b894 [0x00007f0226a9b780+0x0000000000000114]
J 963 c2 scala.collection.Iterator$$anon$9.hasNext()Z (10 bytes) @ 0x00007f022c94b1e8 [0x00007f022c94b1a0+0x0000000000000048]
J 10777 c2 scala.collection.IterableOnceOps.foreach(Lscala/Function1;)V (36 bytes) @ 0x00007f022ce7bf70 [0x00007f022ce7bea0+0x00000000000000d0]
J 6133 c1 scala.collection.AbstractIterator.foreach(Lscala/Function1;)V (6 bytes) @ 0x00007f0225794dec [0x00007f0225794d40+0x00000000000000ac]
j  org.apache.spark.sql.execution.BaseScriptTransformationWriterThread.processRowsWithoutSerde()V+21
j  org.apache.spark.sql.execution.SparkScriptTransformationWriterThread.processRows()V+1
j  org.apache.spark.sql.execution.BaseScriptTransformationWriterThread.$anonfun$run$1(Lorg/apache/spark/sql/execution/BaseScriptTransformationWriterThread;)V+13
j  org.apache.spark.sql.execution.BaseScriptTransformationWriterThread$$Lambda$8729+0x0000000802dd1da8.apply$mcV$sp()V+4
J 14838 c2 scala.runtime.java8.JFunction0$mcV$sp.apply()Ljava/lang/Object; (10 bytes) @ 0x00007f022d02585c [0x00007f022d025820+0x000000000000003c]
j  org.apache.spark.util.Utils$.logUncaughtExceptions(Lscala/Function0;)Ljava/lang/Object;+1
j  org.apache.spark.sql.execution.BaseScriptTransformationWriterThread.run()V+9
v  ~StubRoutines::call_stub
V  [libjvm.so+0x843dba]  JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*)+0x32a
".//spark-test-spark40-group2-test-log/gluten-ut/spark40/hs_err_pid7037.log" 2057L, 177036B

@yikf

yikf commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

btw, i found this case at other PR ci pipeline, https://github.com/apache/gluten/actions/runs/31071855879/job/92523319304?pr=12697

@yikf

yikf commented Aug 10, 2026

Copy link
Copy Markdown
Contributor Author

@zhztheplayer friendly re-ping, please take a look again if you have time.

@zhztheplayer

Copy link
Copy Markdown
Member

@yikf Would you help check the failed CI?

@yikf

yikf commented Aug 10, 2026

Copy link
Copy Markdown
Contributor Author

@zhztheplayer It also failed due to similar issues. I submitted a PR: #12740

@yikf
yikf force-pushed the fix-nmm-hold-use-after-release branch from b8927e8 to 2fcd5db Compare August 11, 2026 07:25
Copilot AI review requested due to automatic review settings August 12, 2026 11:45
@yikf
yikf force-pushed the fix-nmm-hold-use-after-release branch from 127a8b2 to e407538 Compare August 12, 2026 11:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

@zhztheplayer

zhztheplayer commented Aug 12, 2026

Copy link
Copy Markdown
Member

@yikf Thanks. Based on your context, this looks more like a Spark issue. As a broader bug might become the feed thread cannot access the SparkContext because it is stopped by the task thread, no matter Gluten is enabled or not.

If that's the case, we shouldn't handle the race silently but rather throw an error. What do you think?

@yikf

yikf commented Aug 13, 2026

Copy link
Copy Markdown
Contributor Author

@zhztheplayer You're right — this is essentially a Spark issue. In vanilla Spark that's relatively benign, but with Gluten, it escalates from a Java-level error into a native coredump — more severe than the plain-Spark case.

Given that, I think either handling is reasonable — silently no-op'ing or throwing. I've switched to throwing so the broken teardown ordering is surfaced rather than hidden. Could you take another look?

@yikf
yikf force-pushed the fix-nmm-hold-use-after-release branch from e407538 to 0fb5ceb Compare August 13, 2026 03:11
Copilot AI review requested due to automatic review settings August 13, 2026 03:11

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

gluten-arrow/src/main/scala/org/apache/gluten/memory/NativeMemoryManager.scala:90

  • To fully prevent the SIGSEGV described in the PR, release() also needs to be mutually exclusive with hold() at the JNI boundary. Otherwise a thread can enter hold() (before released=true) and race with NativeMemoryManagerJniWrapper.release(handle). Wrapping the native release call with the same synchronized monitor used by hold() prevents concurrent native hold/release on the same handle.
      NativeMemoryManagerJniWrapper.hold(handle)
    }
    override def getHandle(): Long = handle
    override def release(): Unit = {
      if (!released.compareAndSet(false, true)) {

Comment on lines +79 to +87
override def hold(): Unit = {
// hold() must run before release(). Reaching here after release means a broken teardown
// ordering, so surface it instead of dereferencing a freed native handle silently.
if (released.get()) {
throw new GlutenException(
s"Cannot hold memory manager instance that has already been released: $handle")
}
NativeMemoryManagerJniWrapper.hold(handle)
}
@yikf
yikf force-pushed the fix-nmm-hold-use-after-release branch from 0fb5ceb to 423c946 Compare August 13, 2026 09:53
@yikf
yikf force-pushed the fix-nmm-hold-use-after-release branch from 423c946 to a6d7253 Compare August 13, 2026 09:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants