Repository navigation
fix(sdk): throw when getting a skipped parallel branch #799
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
hippi345
wants to merge
11
commits into
aws:main
Choose a base branch
from
hippi345:fix/752-skipped-branch-get
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
482ab3e
fix(sdk): throw when getting a skipped parallel branch
hippi345 0b805a4
docs(sdk): restore waitForOperationCompletion javadoc
hippi345 1141151
Merge branch 'main' into fix/752-skipped-branch-get
zhongkechen aed55d3
fix(sdk): keep skipped parallel branches stable for anyOf and get
hippi345 03cd312
Merge branch 'main' into fix/752-skipped-branch-get
hippi345 774bf9a
Merge branch 'main' into fix/752-skipped-branch-get
zhongkechen a7d38ac
fix(sdk): settle parallel branches registered after early completion
hippi345 2f3c402
Merge branch 'main' into fix/752-skipped-branch-get
zhongkechen 789c974
Merge branch 'main' into fix/752-skipped-branch-get
zhongkechen d5344e5
fix(sdk): address skipped-branch review findings
hippi345 25f49db
Merge branch 'main' into fix/752-skipped-branch-get
zhongkechen File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
216 changes: 216 additions & 0 deletions
216
...n-tests/src/test/java/software/amazon/lambda/durable/ParallelSkippedBranchReviewTest.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,216 @@ | ||
| // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
| package software.amazon.lambda.durable; | ||
|
|
||
| import static org.junit.jupiter.api.Assertions.assertEquals; | ||
| import static org.junit.jupiter.api.Assertions.assertThrows; | ||
| import static org.junit.jupiter.api.Assertions.assertTimeoutPreemptively; | ||
| import static org.junit.jupiter.api.Assertions.assertTrue; | ||
|
|
||
| import java.time.Duration; | ||
| import java.util.List; | ||
| import java.util.concurrent.CountDownLatch; | ||
| import java.util.concurrent.TimeUnit; | ||
| import java.util.concurrent.atomic.AtomicBoolean; | ||
| import org.junit.jupiter.api.Test; | ||
| import software.amazon.awssdk.services.lambda.model.OperationStatus; | ||
| import software.amazon.lambda.durable.config.CompletionConfig; | ||
| import software.amazon.lambda.durable.config.ParallelConfig; | ||
| import software.amazon.lambda.durable.exception.ParallelBranchSkippedException; | ||
| import software.amazon.lambda.durable.model.ExecutionStatus; | ||
| import software.amazon.lambda.durable.model.ParallelResult; | ||
| import software.amazon.lambda.durable.serde.JacksonSerDes; | ||
| import software.amazon.lambda.durable.serde.SerDes; | ||
| import software.amazon.lambda.durable.testing.LocalDurableTestRunner; | ||
|
|
||
| /** | ||
| * Repros for the three review findings on skipped parallel branches: {@code anyOf} replay, the completion-snapshot | ||
| * race, and nested skip exceptions classified as failures. | ||
| */ | ||
| class ParallelSkippedBranchReviewTest { | ||
|
|
||
| private static final Duration DEADLINE = Duration.ofSeconds(20); | ||
|
|
||
| @Test | ||
| void anyOfKeepsTheRunningWinnerWhenTheOtherBranchIsSkippedOnReplay() { | ||
| var runner = LocalDurableTestRunner.create(String.class, (input, context) -> { | ||
| var handlerThread = Thread.currentThread(); | ||
| var parallel = context.parallel( | ||
| "first-successful", | ||
| ParallelConfig.builder() | ||
| .maxConcurrency(1) | ||
| .completionConfig(CompletionConfig.firstSuccessful()) | ||
| .build()); | ||
| var winner = parallel.branch("winner", String.class, ctx -> { | ||
| waitUntilParked(handlerThread); | ||
| return "winner"; | ||
| }); | ||
| var skipped = parallel.branch("skipped", String.class, ctx -> "skipped"); | ||
| var first = String.valueOf(DurableFuture.anyOf(skipped, winner)); | ||
| context.wait("pause", Duration.ofSeconds(1)); | ||
| var second = String.valueOf(DurableFuture.anyOf(skipped, winner)); | ||
| return first + "," + second; | ||
| }); | ||
|
|
||
| var pending = assertTimeoutPreemptively(DEADLINE, () -> runner.run("input")); | ||
| assertEquals(ExecutionStatus.PENDING, pending.getStatus()); | ||
|
|
||
| runner.advanceTime(); | ||
| var replay = assertTimeoutPreemptively(DEADLINE, () -> runner.run("input")); | ||
| assertEquals(ExecutionStatus.SUCCEEDED, replay.getStatus()); | ||
| assertEquals("winner,winner", replay.getResult(String.class)); | ||
| } | ||
|
|
||
| @Test | ||
| void branchGetKeepsTheSameOutcomeWhenItFinishesDuringResultSerialization() { | ||
| var branchRunning = new CountDownLatch(1); | ||
| var allowBranch = new CountDownLatch(1); | ||
| var serDes = new ParallelResultSerializationGate(allowBranch); | ||
| var config = DurableConfig.builder().withSerDes(serDes).build(); | ||
| var runner = LocalDurableTestRunner.create( | ||
| String.class, | ||
| (input, context) -> { | ||
| var parallel = context.parallel( | ||
| "first-successful", | ||
| ParallelConfig.builder() | ||
| .completionConfig(CompletionConfig.firstSuccessful()) | ||
| .build()); | ||
| var slow = parallel.branch("branch-b", String.class, ctx -> { | ||
| branchRunning.countDown(); | ||
| await(allowBranch, "release branch-b"); | ||
| return "B"; | ||
| }); | ||
| parallel.branch("branch-a", String.class, ctx -> { | ||
| await(branchRunning, "branch-b to start"); | ||
| return "A"; | ||
| }); | ||
| var result = parallel.get(); | ||
| return "after-parallel|" + observe(slow) + "|" + result.statuses(); | ||
| }, | ||
| config); | ||
|
|
||
| try { | ||
| var execution = assertTimeoutPreemptively(DEADLINE, () -> runner.runUntilComplete("input")); | ||
| assertEquals(ExecutionStatus.SUCCEEDED, execution.getStatus()); | ||
| assertEquals("after-parallel|SKIPPED:branch-b|[SKIPPED, SUCCEEDED]", execution.getResult(String.class)); | ||
| } finally { | ||
| allowBranch.countDown(); | ||
| } | ||
| } | ||
|
|
||
| @Test | ||
| void skippedBranchStaysSkippedOnReplayWhenParallelResultIsAlreadyCheckpointed() { | ||
| var runner = LocalDurableTestRunner.create(String.class, (input, context) -> { | ||
| var parallel = context.parallel( | ||
| "first-successful", | ||
| ParallelConfig.builder() | ||
| .completionConfig(CompletionConfig.firstSuccessful()) | ||
| .build()); | ||
| parallel.branch("branch-a", String.class, ctx -> "A"); | ||
| var skipped = parallel.branch("branch-b", String.class, ctx -> "B"); | ||
| var result = parallel.get(); | ||
| context.step("record", String.class, stepCtx -> observe(skipped) + "|" + result.statuses()); | ||
| return "done"; | ||
| }); | ||
|
|
||
| var first = assertTimeoutPreemptively(DEADLINE, () -> runner.runUntilComplete("input")); | ||
| assertEquals(ExecutionStatus.SUCCEEDED, first.getStatus()); | ||
|
|
||
| var replay = assertTimeoutPreemptively(DEADLINE, () -> runner.run("input")); | ||
| assertEquals(ExecutionStatus.SUCCEEDED, replay.getStatus()); | ||
| assertEquals("done", replay.getResult(String.class)); | ||
| } | ||
|
|
||
| @Test | ||
| void nestedSkipExceptionStaysAFailedOuterBranch() { | ||
| var runner = LocalDurableTestRunner.create(String.class, (input, context) -> { | ||
| var outer = context.parallel("outer", ParallelConfig.builder().build()); | ||
| var outerBranch = outer.branch("outer-branch", String.class, ctx -> { | ||
| var inner = ctx.parallel( | ||
| "inner", | ||
| ParallelConfig.builder() | ||
| .maxConcurrency(1) | ||
| .completionConfig(CompletionConfig.firstSuccessful()) | ||
| .build()); | ||
| inner.branch("inner-winner", String.class, innerCtx -> "ok"); | ||
| var skipped = inner.branch("inner-skipped", String.class, innerCtx -> "no"); | ||
| inner.get(); | ||
| skipped.get(); | ||
| return "unreachable"; | ||
| }); | ||
|
|
||
| var result = outer.get(); | ||
| assertEquals(List.of(ParallelResult.Status.FAILED), result.statuses()); | ||
| assertEquals(1, result.failed()); | ||
| assertEquals(0, result.skipped()); | ||
| var thrown = assertThrows(ParallelBranchSkippedException.class, outerBranch::get); | ||
| assertEquals("inner-skipped", thrown.getBranchName()); | ||
| return "done"; | ||
| }); | ||
|
|
||
| var first = assertTimeoutPreemptively(DEADLINE, () -> runner.runUntilComplete("input")); | ||
| assertEquals(ExecutionStatus.SUCCEEDED, first.getStatus()); | ||
| assertEquals("done", first.getResult(String.class)); | ||
| var checkpoint = runner.getOperation("outer-branch"); | ||
| assertEquals(OperationStatus.FAILED, checkpoint.getStatus()); | ||
| assertTrue(checkpoint.getContextDetails().error().errorMessage().contains("inner-skipped")); | ||
|
|
||
| var replay = assertTimeoutPreemptively(DEADLINE, () -> runner.run("input")); | ||
| assertEquals(ExecutionStatus.SUCCEEDED, replay.getStatus()); | ||
| assertEquals("done", replay.getResult(String.class)); | ||
| } | ||
|
|
||
| private static void waitUntilParked(Thread handlerThread) { | ||
| var deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(10); | ||
| while (handlerThread.getState() != Thread.State.WAITING | ||
| && handlerThread.getState() != Thread.State.TIMED_WAITING) { | ||
| if (System.nanoTime() >= deadline) { | ||
| throw new IllegalStateException("timed out waiting for anyOf to park"); | ||
| } | ||
| Thread.onSpinWait(); | ||
| } | ||
| } | ||
|
|
||
| private static void await(CountDownLatch latch, String description) { | ||
| try { | ||
| if (!latch.await(10, TimeUnit.SECONDS)) { | ||
| throw new IllegalStateException("timed out waiting for " + description); | ||
| } | ||
| } catch (InterruptedException e) { | ||
| Thread.currentThread().interrupt(); | ||
| throw new IllegalStateException("interrupted while waiting for " + description, e); | ||
| } | ||
| } | ||
|
|
||
| private static String observe(DurableFuture<?> future) { | ||
| try { | ||
| return String.valueOf(future.get()); | ||
| } catch (ParallelBranchSkippedException e) { | ||
| return "SKIPPED:" + e.getBranchName(); | ||
| } | ||
| } | ||
|
|
||
| /** Blocks inside {@link ParallelResult} serialization so a branch can finish in that window. */ | ||
| private static final class ParallelResultSerializationGate implements SerDes { | ||
| private final JacksonSerDes delegate = new JacksonSerDes(); | ||
| private final CountDownLatch allowBranch; | ||
| private final AtomicBoolean gated = new AtomicBoolean(); | ||
|
|
||
| private ParallelResultSerializationGate(CountDownLatch allowBranch) { | ||
| this.allowBranch = allowBranch; | ||
| } | ||
|
|
||
| @Override | ||
| public String serialize(Object value) { | ||
| if (value instanceof ParallelResult && gated.compareAndSet(false, true)) { | ||
| allowBranch.countDown(); | ||
| } | ||
| return delegate.serialize(value); | ||
| } | ||
|
|
||
| @Override | ||
| public <T> T deserialize(String data, TypeToken<T> typeToken) { | ||
| return delegate.deserialize(data, typeToken); | ||
| } | ||
| } | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Codex AI review · Finding
arf_v1_ju3x4rei2iqoutsmdl2i2q7egz[P2] Make the skipped-branch fixture deterministic
This parallel uses unlimited concurrency, so
branch-bcan succeed or win instead of being skipped. In that valid scheduling outcome,observe(skipped)calls a normal branch'sget()from a STEP thread, andvalidateCurrentThreadCanWaitForDurableOperation()throwsIllegalStateException. The step retries/fails and the test intermittently rejects correct execution. Establish a deterministic skipped branch and observe/assert its outcome on the context thread, outside the checkpointed step body, so replay also evaluates the assertion.To suppress repeats, reply
/ai verdict deferred,/ai verdict false-positive, or/ai verdict out-of-scopeon the first nonblank line. Add rationale on later lines. A new/ai verdict acceptedallows future repeats again.