Local activity retries build a `Failure` chain the SDK cannot deserialize

[andrewspinks]

A LA that continually times out builds a Failure chain that eventually causes the sdk to error.

Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message had too many levels
of nesting.  May be malicious.  Use setRecursionLimit() to increase the recursion depth limit.
    at com.google.protobuf.InvalidProtocolBufferException.recursionLimitExceeded(InvalidProtocolBufferException.java:133)
    at com.google.protobuf.CodedInputStream.checkRecursionLimit(CodedInputStream.java:177)
    at com.google.protobuf.CodedInputStream$StreamDecoder.readMessage(CodedInputStream.java:2328)
    at io.temporal.api.failure.v1.Failure$Builder.mergeFrom(Failure.java:1371)
    at io.temporal.api.failure.v1.Failure$Builder.mergeFrom(Failure.java:1018)

Expected Behavior

The local activity exhausts its retry policy and the failure propagates to the workflow.

Actual Behavior

The workflow becomes unreadable to its own worker and is stuck permanently.

Steps to Reproduce the Problem

A workflow like:

public class LocalActivityFailureNesting {

  static final String TASK_QUEUE = "LocalActivityFailureNesting";

  @ActivityInterface
  public interface Activities {
    @ActivityMethod
    void alwaysTimesOut();
  }

  public static class ActivitiesImpl implements Activities {
    @Override
    public void alwaysTimesOut() {
      try {
        // Exceeds startToCloseTimeout, so every attempt times out and is retried.
        Thread.sleep(1000);
      } catch (InterruptedException e) {
        Thread.currentThread().interrupt();
      }
    }
  }

  @WorkflowInterface
  public interface NestingWorkflow {
    @WorkflowMethod
    void run();
  }

  public static class NestingWorkflowImpl implements NestingWorkflow {
    private final Activities activities =
        Workflow.newLocalActivityStub(
            Activities.class,
            LocalActivityOptions.newBuilder()
                .setStartToCloseTimeout(Duration.ofMillis(200))
                .setLocalRetryThreshold(Duration.ofMillis(30))
                .setRetryOptions(
                    RetryOptions.newBuilder()
                        .setInitialInterval(Duration.ofMillis(1))
                        .setBackoffCoefficient(2.0)
                        .setMaximumInterval(Duration.ofMillis(100))
                        .build())
                .build());

    @Override
    public void run() {
      activities.alwaysTimesOut();
    }
  }
}

Will trigger the error after 53 retries.

Working example: https://github.com/andrewspinks/samples-java/blob/main/core/src/main/java/io/temporal/samples/localactivityfailurenesting/LocalActivityFailureNesting.java

Specifications

Version: 1.37.0