Repository navigation
[Bug]: Step result larger than 750 KB is never checkpointed; execution hangs #871
Description
Expected Behavior
A step whose result is at or under the 1 MB checkpoint payload limit is checkpointed and the execution completes.
Actual Behavior
A step result larger than 750 KB is never sent to the CheckpointDurableExecution API. In the local test runner the execution never completes. On Lambda the invocation times out at 900 s, and the next invocation runs the step's function again. No error is raised.
Steps to Reproduce
- Deploy or run locally:
@durable_execution def handler(_event, ctx: DurableContext) -> str: return ctx.step(lambda _sc: "y" * (800 * 1024), name="big_step")
- Invoke it.
- Observe the hang (local) or
Task timed outfollowed by a second run of the step's function (Lambda). A 745 KB result completes in under 2 s.
SDK Version
3.0.0 (also main at b78ea00a)
Python Version
3.13
Is this a regression?
No
Last Working Version
None. Present since checkpoint batching (48a1c41b, 1.0.0). Results over 750 KB were above the Lambda checkpoint payload limit until 2026-10-13.
Additional Context
Also reproduced on Python 3.14, x86_64 and arm64. Cause in state.py: an operation larger than CheckpointBatcherConfig.max_batch_size_bytes (750 KB) goes to _overflow_queue, and _drain_overflow puts it back even when the batch is empty, so it is never sent. _wait_for_first_operation already adds an oversized first operation from _checkpoint_queue. Applying the same rule in _drain_overflow fixes it; all existing tests pass. Results over 1 MB then raise the CheckpointDurableExecution payload size error instead of hanging. Map and parallel branches with large results hang the same way. AT_MOST_ONCE_PER_RETRY is not affected.
Activity
test: run an 800 KB step result end to end
test: cover an oversized operation in the overflow queue
fix: send an oversized first operation in a batch
test: run an 800 KB step result end to end
hln33 commentedon Oct 28, 2026
Thanks for flagging this @dpatlut. We are currently working on a fix with priority and will release a new patch version soon.
Fix is being implemented here - #872
hln33 commentedon Oct 28, 2026
Why the other SDKs do not hang
Every SDK groups queued checkpoint updates into one
CheckpointDurableExecutionrequest. Every SDK caps one request at 750 KB. The cap only controls how updates are grouped. The service applies its own limit to each payload, and that limit is 1 MB.An update larger than 750 KB can never fit under the cap. So the SDK can send it only if it follows this rule: an empty request accepts the next update, whatever its size. The rule must hold on every path that adds an update to a request. If one path lacks it, an oversized update that reaches that path stays queued forever.
Python: one path lacks the rule
_overflow_queue. Seestate.py#L1437-L1442._drain_overflow. It callsbatch.fits(size)and does not check whether the batch is empty. Seestate.py#L1459-L1470.0 + 800 KB > 750 KB. SofitsreturnsFalse. The update goes back into_overflow_queue, and the drain stops._wait_for_first_operationdoes follow the rule. Seestate.py#L1490-L1492. It reads only_checkpoint_queue, so it never sees the update in_overflow_queue._wait_for_first_operationfor a new update that never arrives.So the update is never sent, and the execution hangs.
JS, Java, Go: one loop, with the rule
Each of these SDKs builds a request in one loop. An update that does not fit is not moved anywhere. It stays at the head of the same queue. The next request is built by the same loop. That loop stops only when the size or count limit is exceeded and the request already holds at least one update. So the first update of every request is always added, including an update over 750 KB.
(currentSize + nextItem.sizeBytes > MAX_PAYLOAD_SIZE || …) && batch.length > 0—checkpoint-manager.ts#L507-L524this.queue.processQueueruns again —#L583-L586!flushingItems.isEmpty() && (!canFit || …)—ApiRequestDelayedBatcher.java#L214-L222flushingQueue. The outerwhilebuilds the next request —#L205len(batch) > 0 && (size+p.size > checkpointBatchLimitBytes || …)—checkpoint.go#L582-L591cp.queue.flushcallstakeBatchLockedagain —#L514-L522So in these SDKs, an 800 KB step result goes out as a request of its own. The service accepts it, because 800 KB is under its 1 MB limit. A result over 1 MB is also sent. The service then rejects it with the payload-size error, so the SDK does not hang.
Therefore Python's
_drain_overflowis the only place, across the SDKs, where an empty request can refuse an update. The fix proposed in the issue adds the empty-batch rule to_drain_overflow. That gives Python the same behavior as the other four SDKs.hln33 commentedon Oct 29, 2026
Verified that newest patch version successfully released and contains the expected fix: https://pypi.org/project/aws-durable-execution-sdk-python/3.0.1/#files