Repository navigation
test_io failed with ASAN failures (heap-use-after-free) on AMD64 Arch Linux Asan 3.13 #154175
Description
AMD64 Arch Linux Asan 3.13: https://buildbot.python.org/#/builders/1464/builds/1125
== CPython 3.13.14+ (heads/3.13:303d0acfcdcc, Jul 19 2026, 12:37:17) [GCC 16.1.1 20260625]
== Linux-7.0.14-arch1-1-x86_64-with-glibc2.43 little-endian
== Python build: release ASAN
== cwd: /buildbot/buildarea/3.13.pablogsal-arch-x86_64.asan/build/build/test_python_worker_1123020æ
== CPU count: 2
== encodings: locale=UTF-8 FS=utf-8
== resources: all,-network,-urlfetch
== sanitizers: address
== ASAN_OPTIONS='detect_leaks=0:allocator_may_return_null=1:handle_segv=0'
(...)
FAIL: test_daemon_threads_shutdown_stderr_deadlock (test.test_io.CMiscIOTest.test_daemon_threads_shutdown_stderr_deadlock)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/buildbot/buildarea/3.13.pablogsal-arch-x86_64.asan/build/Lib/test/test_io.py", line 4755, in test_daemon_threads_shutdown_stderr_deadlock
self.check_daemon_threads_shutdown_deadlock('stderr')
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/buildbot/buildarea/3.13.pablogsal-arch-x86_64.asan/build/Lib/test/test_io.py", line 4745, in check_daemon_threads_shutdown_deadlock
self.assertFalse(err.strip('.!'))
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
AssertionError: '=================================================================\n
==1136209==ERROR: AddressSanitizer: heap-use-after-free on address 0x7c9453c52a58 at pc 0x555f594278c6 bp 0x7b64519fe080 sp 0x7b64519fe070\n
READ of size 8 at 0x7c9453c52a58 thread T1\n
' is not false
It seems like the test fails randomly since July 14.
Activity
[-]test_io failed with ASAN failures on AMD64 Arch Linux Asan 3.13[/-][+]test_io failed with ASAN failures (heap-use-after-free) on AMD64 Arch Linux Asan 3.13[/+]BHUVANSH855 commentedon Jul 25, 2026
I investigated this on the
3.13branch.So far I've found that:
test_daemon_threads_shutdown_stdout_deadlockandtest_daemon_threads_shutdown_stderr_deadlockpass locally.-F) also passes consistently on my release build.Lib/test/test_io.pyorModules/_iothat appear related.Since the buildbot log only contains the beginning of the AddressSanitizer report, I'm currently missing the stack trace showing where the use-after-free originates.
Would it make sense to investigate this further only with an ASAN build, or is there another area of the runtime that is already suspected?
cmaloney commentedon Aug 29, 2026
Have two potential improvements here:
_Py_eval_breaker_bit_is_setaccesseststateand, in 3.13, a finalizing main thread may free thetstateresulting in the use after free. For main this was fixed in race condition in threading when interpreter finalized while daemon thread runs (thread sanitizer identified) #124878 and suppressed in TSAN for 3.13. We could add an ASAN suppression to parallel the TSAN one but my leaning is just moving under the GIL a couple lines earlier.cpython/Python/ceval_gil.c
Line 266 in 87c9dfb
time.sleep(0.5)to get the I/O stack blocked in a particular way which isn't great. There are tests intest_threadingto make sure that an interpreter with daemon threads shuts down okay so I'd like to change thistest_ioto use a mocked Raw I/O which can usethreading.Eventto trigger the locking contention explicitly. That applies frommaindown to 3.13.encukou commentedon Sep 9, 2026
Well, since the
HangThreadchanges weren't backported, daemon threads in 3.13 feel like a lost cause. If we handle the danglingtstatethat indrop_gil, won't that just move the use-after-free to the nexttake_gil?cmaloney commentedon Sep 14, 2026
I think the
drop_gil_implcall is what enables the tstate go from "definitely allocated" to "maybe allocated" and why we hit the ASAN failure. There are cases wheretstatecould be passed as NULL but I think the failure if we were hitting those would be a bit different.Daemon threads are used by quite a few projects... we can definitely just silence this particular test but not sure how that will be for SRE type work in deployments.
[3.13] gh-154175: Fetch _PY_GIL_DROP_REQUEST_BIT under GIL (#157013)
gh-154175: Remove time.sleep in test_io daemon test (#157019)