[Bug]: Stop or execution timeout during an active step leaves terminal status missing from traces #848

Summary

When a durable execution is stopped or reaches its execution timeout while a step is still running, the invocation can survive long enough to reach its invocation-end hook. The next checkpoint is rejected, but the SDK/plugin contract reports only a retry disposition. Plugins are left without the execution's authoritative STOPPED or TIMED_OUT outcome, even though no further invocation follows.

This spans the core SDK, the shared plugin lifecycle interface, OTel consumers, and the existing telemetry-exporter plugin. Custom plugins that depend on the same contract for execution completion have the same information limitation. The existing error and hook payloads do not provide enough information for a status-mapping-only fix.

Expected Behavior

When an invocation survives the external termination and reaches finalization, the SDK/plugin integration should be able to discover and expose the authoritative terminal execution status and end time through the shared lifecycle contract. Both OTel Execution and Invocation views should make that outcome visible; the Execution view should export the terminal Workflow span with durable.execution.status=STOPPED or TIMED_OUT.

Telemetry exporters should be able to emit the confirmed terminal execution record under the applicable emission policy. Custom plugins should receive enough information to distinguish execution termination from a retryable invocation failure without parsing an ambiguous checkpoint error.

Execution outcome and invocation disposition should remain distinct. A terminal execution must not be represented solely as RETRYING in traces or RUNNING in telemetry records. A retry disposition does not guarantee that another invocation will run.

Actual Behavior

The supplied cloud report describes this sequence:

1. A step runs for approximately 60 seconds.
2. The execution is stopped approximately 20 seconds into the running step, or reaches an ExecutionTimeout of 30 seconds.
3. The current Lambda invocation continues and the step runs for the full 60 seconds.
4. Its next checkpoint fails with InvalidParameterValueException: Invalid checkpoint token.
5. The plugin ends Invocation with OTel status UNSET and durable.invocation.status=RETRYING.
6. No further invocation follows because the durable execution has already ended.
7. No span carries durable.execution.status; the Execution view never exports Workflow.

Continuing the current invocation after StopDurableExecution is documented platform behavior. This issue concerns the missing terminal telemetry, not a requirement to forcibly interrupt running user code.

Impact Scope

The supplied cloud report demonstrates the OTel symptom in both trace views. The additional non-OTel effects below are derived from source inspection at the reported revision; they have not been separately reproduced in a deployed environment.

A concrete affected consumer is TelemetryPlugin in the telemetry-exporter module. Its status mapping converts RETRYING to RUNNING, and its emission gates recognize only SUCCEEDED / FAILED as terminal:

The configured telemetry exporters consume records from this shared producer. The effect therefore reaches destinations such as CloudWatch Logs, S3, DynamoDB, EventBridge, SQS, and HTTP: stored execution views can remain RUNNING or have no completion record, duration/completion reports can be incomplete or misleading, and consumers receive no record identifying STOPPED/TIMED_OUT to drive terminal notifications. Changing the destination does not recover the missing service outcome. These are consequences of the producer/contract gap, not separately established defects in every destination adapter.

Custom instrumentation plugins that rely solely on invocation-end status to infer execution completion, failure, cancellation, or timeout have the same information limitation. Examples include execution-level metrics, audit records, status synchronization, and terminal notifications. This is a conditional impact assessment, not a claim that specific third-party plugins were reproduced as failing. Features that only count invocations, measure invocation-local work, attach correlation fields, or flush unconditionally at the surviving invocation's end do not directly require the missing execution outcome.

This issue does not establish incorrect service execution status/history, nor a failure of every plugin hook. The active invocation reaching finalization is essential to the proposed in-process recovery path; termination while no invocation is running or after runtime shutdown remains a separate coverage limitation.

Steps to Reproduce

1. Deploy a handler with a named durable step whose body takes 60 seconds to complete. A sleep/delay inside that step is a test fixture, not a durable-wait implementation.
2. Set the Lambda function/invocation timeout above 60 seconds (for example, 120 seconds), and enable an OTel plugin with sampling and export enabled.
3. Run each trace view separately.
4. For the stop case, configure an execution timeout above the test duration, start an execution, and call StopDurableExecution approximately 20 seconds after the step starts.
5. For the timeout case, configure the durable execution timeout to 30 seconds, start an execution, and do not issue a stop.
6. Wait for the running invocation to finish, inspect service execution status/history, and inspect exported spans correlated by execution ARN.
7. Compare the service's STOPPED/TIMED_OUT outcome with the final Invocation attributes and missing Workflow.
8. For the non-OTel regression, repeat with the telemetry-exporter plugin, sampling/export enabled, and each of ON_COMPLETE, ON_FAILURE, ON_CHANGE. Capture emitted execution records and compare their presence, status, error meaning, and timing with service status/history. These additional checks are proposed regression coverage; the supplied cloud report covers OTel.

SDK Version

Reported main revision: 64339d4f (Java).

Java Version

Not specified in the supplied report.

Is this a regression?

Unknown; no last-known-good version was established.

Source Analysis

The core classifies this checkpoint rejection as retryable, and the executor consequently reports InvocationStatus.RETRYING to the end hook. Both OTel views intentionally leave RETRYING as UNSET because the hook cannot distinguish a stopped/timed-out execution from an invocation failure that may be retried. Workflow is materialized only for SUCCEEDED or FAILED.

• Invalid-token classification

• Executor maps the error to RETRYING

• Execution plugin explicitly documents the ambiguity

• Invocation plugin's equivalent handling and Workflow gate

• Telemetry retry-to-RUNNING mapping

• Telemetry terminal gates and invocation-end record generation

• Telemetry end time and duration are derived from invocation finalization

An invalid checkpoint token does not uniquely identify STOPPED or TIMED_OUT: it can also mean a stale/superseded invocation while the execution remains active. The surviving invocation provides an opportunity to retrieve the real outcome, not proof that the current error already contains it.

Implementation Direction and Acceptance Criteria

• Obtain authoritative terminal execution information through an appropriate service signal or a bounded lookup such as GetDurableExecution, and expose status and authoritative end time (when available) through the shared SDK/plugin lifecycle contract independently of invocation retry disposition.
• In both trace views, export the confirmed STOPPED/TIMED_OUT execution status; export the terminal Workflow once with the authoritative execution end time when available.
• Update the telemetry-exporter consumer as well. For sampled-in executions with export enabled, ON_COMPLETE and ON_CHANGE should emit the confirmed terminal execution record. Use the authoritative execution end time when available; do not substitute invocation-finalization time as though it were the service termination time.
• Define and test the ON_FAILURE policy for STOPPED and TIMED_OUT explicitly; do not implicitly equate external cancellation with FAILED. Coordinate telemetry record types/schema and status-dependent exporter mappings (such as OTLP severity) so they preserve the confirmed outcome.
• Preserve existing behavior for genuine retryable invocation failures and stale-token rejection while the execution is still active. Do not infer terminal status solely from the invalid-token message.
• Define and test the fallback when status retrieval fails or is unavailable: preserve uncertainty without fabricating a terminal outcome or blocking invocation finalization indefinitely.
• Add focused core/plugin tests and deployed regression coverage for active-step stop and active-step execution timeout. Use a generic recording plugin to verify the shared lifecycle payload independently of OTel. Cover both OTel views and all telemetry emission modes with a capturing exporter, and compare exported records/spans with service status/history.
• Add a nonterminal invalid-token control case and a successful retry/resume case, preserving replay behavior and ensuring completed work is not repeated.
• Document the limitation for termination while no invocation is running or after runtime shutdown; an in-process finalization lookup alone cannot cover those cases. Coordinate any shared conformance requirement changes separately.

Additional Context

Reported environment: AWS Lambda in us-west-2; traces inspected in Dash0. The original report covers JavaScript, Python, and Java, in both Execution and Invocation views.

The cloud observations above come from the supplied OTel report. The linked core, plugin, and telemetry-exporter implementations were reviewed at the reported revision. The non-OTel impact is source-derived; no additional cloud reproduction or Dash0 retrieval was performed for this description update.

Related but different: #705 fixed invalid-token message matching. Preserve the corrected retryable classification for stale tokens belonging to an execution that is still active. #706 handles checkpoint responses with a missing token; this report concerns a rejected checkpoint after external stop or execution timeout.

Cross-SDK Tracking

• [Bug]: Stop or execution timeout during an active step leaves terminal status missing from traces aws-durable-execution-sdk-js#1127
• [Bug]: Stop or execution timeout during an active step leaves terminal status missing from traces aws-durable-execution-sdk-python#882