Repository navigation
fix(python): drain abandoned callback dispatch before interpreter exit - #2802
Merged
Merged
Conversation
A timed-out `execute_sync` async callback leaves its private-loop dispatch running on Tokio's blocking pool. Dropping a `JoinHandle` cannot cancel a *blocking* task, so that pool thread stays parked in `result_rx.recv()` with the GIL released. The atexit hook joined the private-loop worker — which hands the cancelled result to that thread at the very end of the hook — but never waited for the thread itself, because the dispatch was never enrolled in the result-delivery barrier. Its re-attach then raced `Py_FinalizeEx`, and losing that race aborts the process on CPython < 3.13: `take_gil` sees the thread must exit and calls `PyThread_exit_thread()`, whose `pthread_exit` forced unwind crosses Rust frames that never rethrow it, so glibc kills the process with "FATAL: exception not rethrown" (SIGABRT). Enroll the dispatch through a shared `enroll_python_work()` guard and hold it across its Python work and its own `Py`/session drops (a last session clone can run `PyPrivateAsyncLoop::drop`, which attaches), so the exit hook drains it with the GIL released before finalization. Past the boundary the task skips Python entirely, as result deliveries already did. Joining the private-loop worker proves the worker finished, not that the thread waiting on it has — record that in the knowledge bundle as TM-PY-030 variant 4. Observed as a flaky SIGABRT in `test_interpreter_exit_does_not_crash[abandoned]` on the Python 3.9 CI leg.
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
bashkit | 04370d9 | Commit Preview URL Branch Preview URL |
Oct 31 2026, 09:38 AM |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
A timed-out async callback no longer leaves a native thread owing Python work
when the interpreter starts shutting down.
execute_syncdispatches private-loopcallbacks on Tokio's blocking pool; that dispatch now enrolls in the same
atexitbarrier the result deliveries already used, so the exit hook drains it(GIL released) before CPython finalization. Past the boundary the task skips
Python entirely instead of attaching.
No API or behavior change for callers — the only observable difference is that
interpreter exit no longer aborts in this window.
Why
test_interpreter_exit_does_not_crash[abandoned]failed on the Python 3.9 CIleg with
SIGABRT/FATAL: exception not rethrown(run 41132921244),
turning main's Python workflow red.
The abandoned callback is the trigger. A timeout drops the dispatch future, but
dropping a
JoinHandlecannot cancel a blocking task, so the pool threadstays parked in
result_rx.recv()with the GIL released. The exit hook joinedthe private-loop worker — which hands that thread the cancelled result at the
very end of the hook — but nothing waited for the thread itself, because the
dispatch was never enrolled in the delivery barrier. Its re-attach then raced
Py_FinalizeEx. Losing that race is fatal on CPython < 3.13:take_gilseesthe thread must exit and calls
PyThread_exit_thread(), whosepthread_exitforced unwind crosses Rust frames that never rethrow it, so glibc kills the
process.
Root cause in one line: joining the private-loop worker proves the worker
finished, not that the thread waiting on it has. Recorded as TM-PY-030
variant 4.
Before / After
Diagnostic on CPython 3.9.25 showing what the exit hook accounts for. An
atexithandler registered beforeimport bashkitruns after bashkit's ownhook (atexit is LIFO), so it observes the state the hook left behind:
The hook waits out the cancelled callback and joins
bashkit-py-loop(t=0.353),but
tokio-rt-worker— the parked dispatch thread that must still re-attach —was never part of the barrier. That is the thread that aborted in CI.
Validation, same interpreter (3.9.25) and the exact CI invocation:
pytest tests/(3.9)1 failed, 963 passed, 5 skipped(CI)test_teardown_determinism.py× 25 roundscargo clippy -p bashkit-python --all-targets -D warningscargo fmt --check,ruff check/format,check_okf.pyHonest caveat on the regression test: the abort is a genuine race with a very
narrow window, and it did not reproduce locally — 62 runs of the CI script
across plain, CPU-pinned and GIL-starved variants all passed, and re-running the
failed CI job passed too. The new
abandoned-slow-cancelcase therefore does notreliably fail pre-fix; what it does add is coverage of the precise window the
diagnostic above proves exists (worker and dispatch thread both still in flight
at interpreter exit), which no existing case exercised. The fix itself is
structural rather than timing-dependent: every native task that may attach is now
enrolled, so there is no window left to lose.
Risk
atexitwait for work that must complete anyway, withthe GIL released. Enrollment is atomic against the boundary closing (the flag
is set under the
DELIVERIESlock), so a task either enrolls and is drained orskips Python.
worker the hook already joins first, so every result sender is gone by the
time it waits. A callback that blocks without awaiting still bounds teardown by
the existing worker join, not by this barrier. The
bg_thread_runnerfallbackis not reachable from this call site (a fresh blocking thread has no running
asyncio loop). Confirmed empirically: 25 rounds, no hang.
Checklist
Generated by Claude Code