[Bug]: OTel re-exports already completed STEP spans after invocation retry #857

Expected Behavior

Replaying an already checkpointed terminal STEP must not trigger another SDK-generated operation or attempt span in either OTel view. Span delivery remains best-effort.

Apply this without persisting spans or export acknowledgements, adding checkpoint operations, or changing stored operation results.

Actual Behavior

When a step completes and its checkpoint is committed, the invocation can still fail afterward. On the retry, replaying that completed step can emit onOperationEnd again and export a second Execution-view span with the same trace ID and span ID but different timing. The step body is correctly skipped; the duplicate is in telemetry.

Please avoid creating OTel spans for a STEP that was already terminal in the invocation's input checkpoint, including when its ID appears in UpdatedOperationIds. Preserve telemetry for steps that actually execute or finish during the current invocation and for externally completed operations observed after suspension.

Steps to Reproduce and Evidence

1. Execute a step and wait for its successful checkpoint.
2. Fail the invocation with a retryable invocation error after the step returns.
3. Retry with the completed step in the initial execution state and its ID in UpdatedOperationIds.
4. Observe that the step body runs once, but the Execution OTel plugin exports the logical step span twice.

This was reproduced locally through DurableExecutor and the public context.step API, using a controlled checkpoint client and the real OTel in-memory exporter. Both view variants were tested. Execution view reused the span ID with different start time and duration; Invocation view emitted an additional continuation span with a new ID. The latter avoids an ID collision but still creates a segment for a step that had already finished. A subsequent replay with an empty UpdatedOperationIds emitted no additional step span.

The test deliberately controlled checkpoint timestamps; they are not AWS latency measurements. This investigation did not run a new Java cloud reproduction. The relevant behavior is present in Java commit 64339d4f0e071cd841bdaf43c95a35324a3132bb and was reproduced against 4b1aa82b0d898fea495cda45760ce3737fbb341e.

Why it happens

UpdatedOperationIds describes changes since the previous successful invocation, so it can include a step already completed and observed during an invocation that later failed. It is not proof that the operation completed externally while suspended.

• DurableExecutionInput documents that boundary.
• BaseDurableOperation.execute fires onOperationEnd for a terminal replay when its ID is in that set.
• ExecutionOtelPlugin.onOperationEnd materializes another span using the deterministic operation ID. The first invocation can use an earlier locally captured start time; replay has only the checkpoint timestamps. Its invocation link also changes.

For a behavioral reference, the JavaScript step handler returns or raises the stored terminal outcome before entering the execution hook path.

Requested Behavior and Acceptance Criteria

• Replaying a STEP that was already SUCCEEDED or FAILED in the invocation's input checkpoint must not create another operation or attempt span in either OTel view. Its presence in UpdatedOperationIds must not change this behavior. In particular, Invocation view should not manufacture a new continuation segment for that completed step.
• OTel-specific suppression or a correction to completion provenance are both possible implementations. This issue does not require removing notifications from all generic plugins; keep the distinction between an operation already terminal at invocation entry and one newly completed during the current invocation.
• Apply the decision by STEP operation type, including applicable STEP subtypes; do not filter only one subtype name.
• Replay must still return the stored result or raise the stored failure without running the step body again.
• Actual step attempts and retries must retain their current hooks, attempt spans, completion spans, and error handling. A step resumed from a nonterminal checkpoint that completes during this invocation must still emit its completion span. Do not apply a blanket isReplay filter.
• Preserve completion observation and telemetry for externally completed waits, chained invokes, and callbacks. An operation that was unfinished when the invocation suspended still needs its completion represented when it is first observed.
• Add focused tests through the public SDK API for successful and failed terminal steps followed by invocation retry. Include a normal suspend control and a sequence with step A before a successful suspension and step B before an invocation failure: A must remain represented once, and replay must not re-export B.
• Update the lifecycle/OTel documentation to describe this as suppression of SDK-generated replay emissions, not an end-to-end exactly-once export guarantee.

Delivery limitation

Without persisting the span and delivery state, the SDK cannot guarantee both loss-free telemetry and exactly-once export across crashes. If a crash occurs after a step checkpoint commits but before its completion hook or export runs, suppressing terminal-step replay can leave that span missing. This proposal explicitly accepts that window. Exporter/network retries and backend ingestion deduplication remain separate concerns; this issue does not add durable span storage or promise identical span copies across invocations.

SDK Version / Java Version

Controlled public-API reproduction: Java source 4b1aa82b0d898fea495cda45760ce3737fbb341e, Java 17. The cited paths were also present at 64339d4f0e071cd841bdaf43c95a35324a3132bb. Rechecked the relevant replay and span-rendering paths on current main 1a501400d17532a30e4627f06ea516e0bed7b0cd on 2026-10-30; the behavior is still present in source.

Is this a regression?

Unknown. No working version was established.

Related Work

#850 covers telemetry behavior at hard invocation timeouts. This issue is narrower: suppressing a completed STEP's reconstruction during a later invocation, while explicitly accepting the crash-before-export loss window.

Companion Python implementation: aws/aws-durable-execution-sdk-python#892. That issue includes a new cloud reproduction of the shared retry-input behavior; its measured durations are Python evidence, not Java measurements.