Skip to content

fix(python): drain abandoned callback dispatch before interpreter exit - #2802

Merged
chaliy merged 1 commit into
mainfrom
eve/clever-mccarthy-2fczv6
Oct 31, 2026
Merged

chaliy merged 1 commit into
mainfrom
eve/clever-mccarthy-2fczv6

Conversation

@chaliy

@chaliy chaliy commented Oct 31, 2026

Copy link
Copy Markdown
Contributor

What changed

A timed-out async callback no longer leaves a native thread owing Python work
when the interpreter starts shutting down. execute_sync dispatches private-loop
callbacks on Tokio's blocking pool; that dispatch now enrolls in the same
atexit barrier the result deliveries already used, so the exit hook drains it
(GIL released) before CPython finalization. Past the boundary the task skips
Python entirely instead of attaching.

No API or behavior change for callers — the only observable difference is that
interpreter exit no longer aborts in this window.

Why

test_interpreter_exit_does_not_crash[abandoned] failed on the Python 3.9 CI
leg with SIGABRT / FATAL: exception not rethrown
(run 41132921244),
turning main's Python workflow red.

The abandoned callback is the trigger. A timeout drops the dispatch future, but
dropping a JoinHandle cannot cancel a blocking task, so the pool thread
stays parked in result_rx.recv() with the GIL released. The exit hook joined
the private-loop worker — which hands that thread the cancelled result at the
very end of the hook — but nothing waited for the thread itself, because the
dispatch was never enrolled in the delivery barrier. Its re-attach then raced
Py_FinalizeEx. Losing that race is fatal on CPython < 3.13: take_gil sees
the thread must exit and calls PyThread_exit_thread(), whose pthread_exit
forced unwind crosses Rust frames that never rethrow it, so glibc kills the
process.

Root cause in one line: joining the private-loop worker proves the worker
finished, not that the thread waiting on it has.
Recorded as TM-PY-030
variant 4.

Before / After

Diagnostic on CPython 3.9.25 showing what the exit hook accounts for. An
atexit handler registered before import bashkit runs after bashkit's own
hook (atexit is LIFO), so it observes the state the hook left behind:

EXEC_RETURNED  t=0.052  threads=['bashkit-py-loop', 'python', 'tokio-rt-worker']
CANCEL_SEEN    t=0.052  threads=['bashkit-py-loop', 'python', 'tokio-rt-worker']
AFTER_BASHKIT  t=0.353  threads=['python', 'tokio-rt-worker']

The hook waits out the cancelled callback and joins bashkit-py-loop (t=0.353),
but tokio-rt-worker — the parked dispatch thread that must still re-attach —
was never part of the barrier. That is the thread that aborted in CI.

Validation, same interpreter (3.9.25) and the exact CI invocation:

Before After
pytest tests/ (3.9) 1 failed, 963 passed, 5 skipped (CI) 965 passed, 5 skipped
test_teardown_determinism.py × 25 rounds — 0/25 failed (213 s, no hang)
teardown + async callbacks on 3.13 (same abi3 wheel) — 37 passed
cargo clippy -p bashkit-python --all-targets -D warnings — clean
cargo fmt --check, ruff check/format, check_okf.py — clean

Honest caveat on the regression test: the abort is a genuine race with a very
narrow window, and it did not reproduce locally — 62 runs of the CI script
across plain, CPU-pinned and GIL-starved variants all passed, and re-running the
failed CI job passed too. The new abandoned-slow-cancel case therefore does not
reliably fail pre-fix; what it does add is coverage of the precise window the
diagnostic above proves exists (worker and dispatch thread both still in flight
at interpreter exit), which no existing case exercised. The fix itself is
structural rather than timing-dependent: every native task that may attach is now
enrolled, so there is no window left to lose.

Risk

  • Low
  • The change only makes atexit wait for work that must complete anyway, with
    the GIL released. Enrollment is atomic against the boundary closing (the flag
    is set under the DELIVERIES lock), so a task either enrolls and is drained or
    skips Python.
  • New-hang analysis: the barrier can only wait on dispatches whose private-loop
    worker the hook already joins first, so every result sender is gone by the
    time it waits. A callback that blocks without awaiting still bounds teardown by
    the existing worker join, not by this barrier. The bg_thread_runner fallback
    is not reachable from this call site (a fresh blocking thread has no running
    asyncio loop). Confirmed empirically: 25 rounds, no hang.

Checklist

  • Tests added or updated
  • Backward compatibility considered

Generated by Claude Code

A timed-out `execute_sync` async callback leaves its private-loop dispatch
running on Tokio's blocking pool. Dropping a `JoinHandle` cannot cancel a
*blocking* task, so that pool thread stays parked in `result_rx.recv()` with
the GIL released. The atexit hook joined the private-loop worker — which hands
the cancelled result to that thread at the very end of the hook — but never
waited for the thread itself, because the dispatch was never enrolled in the
result-delivery barrier. Its re-attach then raced `Py_FinalizeEx`, and losing
that race aborts the process on CPython < 3.13: `take_gil` sees the thread must
exit and calls `PyThread_exit_thread()`, whose `pthread_exit` forced unwind
crosses Rust frames that never rethrow it, so glibc kills the process with
"FATAL: exception not rethrown" (SIGABRT).

Enroll the dispatch through a shared `enroll_python_work()` guard and hold it
across its Python work and its own `Py`/session drops (a last session clone can
run `PyPrivateAsyncLoop::drop`, which attaches), so the exit hook drains it with
the GIL released before finalization. Past the boundary the task skips Python
entirely, as result deliveries already did.

Joining the private-loop worker proves the worker finished, not that the thread
waiting on it has — record that in the knowledge bundle as TM-PY-030 variant 4.

Observed as a flaky SIGABRT in
`test_interpreter_exit_does_not_crash[abandoned]` on the Python 3.9 CI leg.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
bashkit 04370d9 Commit Preview URL

Branch Preview URL
Oct 31 2026, 09:38 AM

@chaliy
chaliy merged commit 59bcc3e into main Oct 31, 2026
38 checks passed
@chaliy
chaliy deleted the eve/clever-mccarthy-2fczv6 branch October 31, 2026 10:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant