Skip to content

Fix skipped mutable side effect replay recovery - #3164

Draft
Quinn-With-Two-Ns wants to merge 1 commit into
mainfrom
fix/mutable-side-effect-replay-3163
Draft

Quinn-With-Two-Ns wants to merge 1 commit into
mainfrom
fix/mutable-side-effect-replay-3163

Conversation

@Quinn-With-Two-Ns

Copy link
Copy Markdown
Contributor

What changed?

  • Track replaying mutable-side-effect invocations separately from the command queue and resolve skipped invocations after their workflow task's command events have been applied.
  • Reuse the existing cancellation/cached-result transition, preserving callback workflow time without sweeping unrelated commands.
  • Introduce MUTABLE_SIDE_EFFECT_REPLAY_RECOVERY, disabled by default for its first release. Register opt-in usage at the mutableSideEffect call site and honor recorded flags during replay.
  • Add cache-eviction, callback-time, flag-registration, and negative nondeterminism-detection tests. Retain only the commandless-task and callback-time history fixtures.

Skip-count calculation and marker matching are unchanged.

Why?

Refs #3163.

A skipped mutable side effect has no corresponding marker event. During full replay, its internal placeholder can remain pending when there is no later command event to trigger cleanup, and an empty RecordMarker command can reach the server.

Unconditionally notifying the pending invocation at the next task start changes the workflow time observed by its continuation and can break replay. The added timing regression covers that case; flagged cleanup instead runs at the completed task's boundary, before the next task advances time or delivers signals.

Breaking changes?

None to public APIs. The new SDK flag remains disabled by default (Enabled: pending) for rollout compatibility; tests opt in explicitly. Unflagged histories retain their existing replay behavior.

Server PR

None required.

Validation

  • ./gradlew --offline spotlessApply
  • Retained state-machine and mutable-side-effect tests: 14 passed.
    ./gradlew --offline :temporal-sdk:test \
      --tests io.temporal.internal.statemachines.MutableSideEffectStateMachineTest \
      --tests io.temporal.workflow.MutableSideEffectReplayTest \
      --tests io.temporal.workflow.MutableSideEffectTest
  • Real-server configuration: 9 passed.
    ./gradlew --offline -PtestServer=dev-server :temporal-sdk:test \
      --tests io.temporal.workflow.MutableSideEffectReplayTest \
      --tests io.temporal.workflow.MutableSideEffectTest
  • One-off compatibility validation generated 20 histories with the unmodified baseline SDK (adf0fc8) on the test and real servers. Baseline replay passed first, followed by 40 patched replay cases comparing exact callback values, order, and workflow times with opt-in off/on. The bulk corpus and generator are intentionally not included in this PR.
  • Broader focused state-machine, side-effect, versioning, query, sticky-worker, and SDK-flag tests passed during development. The full repository suite has not been run.

Resolve pending mutable side effect invocations at the original workflow-task boundary without changing legacy replay timing. Gate recovery behind a first-release-disabled SDK flag and register opt-in use at the mutableSideEffect call site.

Add cache-eviction, callback-time, and nondeterminism detection replay coverage for issue #3163.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant