Skip to content

[Feature]: Bound the test run when an SDK deadlock hangs an e2e test #875

Open

Description

@hln33

What would you like?

An e2e test that reaches an SDK deadlock through DurableFunctionTestRunner should fail and let the pytest process exit within a bounded time.

Today it does not. The sequence is:

  1. DurableFunctionTestRunner.run() raises TimeoutError after execution_timeout (runner.py:912-921). The test fails and pytest prints the failure summary.
  2. The SDK handler runs in a ThreadPoolExecutor named dex-handler (execution.py:564-574). On a deadlock, that worker thread stays blocked on a checkpoint CompletionEvent.
  3. DurableFunctionTestRunner.close() shuts down the emulator's own executor and registry (runner.py:758-760). It has no handle on the SDK's dex-handler pool. So the blocked worker is not released.
  4. Python joins ThreadPoolExecutor workers at interpreter exit. So the pytest process stays alive after printing the summary.
  5. The build job in .github/workflows/ci.yml sets no timeout-minutes. GitHub's default is 360 minutes. So one regression holds a CodeBuild runner for 6 hours and delays the red status by that long.

Observed on #872: the e2e test for #871, run against the code before the fix, printed 1 failed in 30.19s and then stayed alive for over 4 minutes until killed. All 9 e2e files under packages/aws-durable-execution-sdk-python/tests/e2e/ that call runner.run() share this exposure. The Codex finding on #872 (arf_v1_w5mqrgwi23fcuefmbeyubiaerm) proposed a per-test subprocess wrapper. That fixes one file, and it also drops the handler path from test:cov. A repo-wide guard is the right place.

Possible Implementation

Options, smallest first. A and B can both be done.

  • A. Job timeout. Add timeout-minutes (for example 30) to the build job in ci.yml. One line. Bounds every hang in CI. Gives no thread dump and does not help a local run.
  • B. pytest-timeout. Add pytest-timeout to the test environment dependencies in the root pyproject.toml and set timeout (for example 120) and timeout_method = "thread" in [tool.pytest.ini_options]. The thread method dumps every thread's stack and then calls os._exit, so a blocked non-daemon worker cannot keep the process alive. Bounds every test, local and CI, and shows where the handler thread is blocked.
  • C. Runner-side abort. Give DurableFunctionTestRunner.close() a way to release the SDK handler thread. Python cannot kill a thread, so this needs a cooperative abort path in ExecutionState that unblocks pending CompletionEvents with an error. Largest change, and the only option that leaves the process in a clean state; it is a design change to the SDK and would need a design note (CONTRIBUTING rule 3).

Recommendation: B, plus A as a backstop.

Is this a breaking change?

No. Test infrastructure only.

Does this require an RFC?

No for A and B. C would need a design note.

Additional Context

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions