Repository navigation
[Feature]: Bound the test run when an SDK deadlock hangs an e2e test #875
Description
What would you like?
An e2e test that reaches an SDK deadlock through DurableFunctionTestRunner should fail and let the pytest process exit within a bounded time.
Today it does not. The sequence is:
DurableFunctionTestRunner.run()raisesTimeoutErrorafterexecution_timeout(runner.py:912-921). The test fails and pytest prints the failure summary.- The SDK handler runs in a
ThreadPoolExecutornameddex-handler(execution.py:564-574). On a deadlock, that worker thread stays blocked on a checkpointCompletionEvent. DurableFunctionTestRunner.close()shuts down the emulator's own executor and registry (runner.py:758-760). It has no handle on the SDK'sdex-handlerpool. So the blocked worker is not released.- Python joins
ThreadPoolExecutorworkers at interpreter exit. So the pytest process stays alive after printing the summary. - The
buildjob in.github/workflows/ci.ymlsets notimeout-minutes. GitHub's default is 360 minutes. So one regression holds a CodeBuild runner for 6 hours and delays the red status by that long.
Observed on #872: the e2e test for #871, run against the code before the fix, printed 1 failed in 30.19s and then stayed alive for over 4 minutes until killed. All 9 e2e files under packages/aws-durable-execution-sdk-python/tests/e2e/ that call runner.run() share this exposure. The Codex finding on #872 (arf_v1_w5mqrgwi23fcuefmbeyubiaerm) proposed a per-test subprocess wrapper. That fixes one file, and it also drops the handler path from test:cov. A repo-wide guard is the right place.
Possible Implementation
Options, smallest first. A and B can both be done.
- A. Job timeout. Add
timeout-minutes(for example 30) to thebuildjob inci.yml. One line. Bounds every hang in CI. Gives no thread dump and does not help a local run. - B. pytest-timeout. Add
pytest-timeoutto the test environment dependencies in the rootpyproject.tomland settimeout(for example 120) andtimeout_method = "thread"in[tool.pytest.ini_options]. The thread method dumps every thread's stack and then callsos._exit, so a blocked non-daemon worker cannot keep the process alive. Bounds every test, local and CI, and shows where the handler thread is blocked. - C. Runner-side abort. Give
DurableFunctionTestRunner.close()a way to release the SDK handler thread. Python cannot kill a thread, so this needs a cooperative abort path inExecutionStatethat unblocks pendingCompletionEvents with an error. Largest change, and the only option that leaves the process in a clean state; it is a design change to the SDK and would need a design note (CONTRIBUTING rule 3).
Recommendation: B, plus A as a backstop.
Is this a breaking change?
No. Test infrastructure only.
Does this require an RFC?
No for A and B. C would need a design note.
Additional Context
- fix: send an oversized first operation in a batch #872 (fix for [Bug]: Step result larger than 750 KB is never checkpointed; execution hangs #871), review thread with the Codex finding.
- Verify a candidate by checking out
state.pyfrom the test-only commit of fix: send an oversized first operation in a batch #872 (8f2729ea) and runningtests/e2e/large_step_result_int_test.py: the run must end with a failure within the configured bound and the process must exit.
Activity