Skip to content

feat: retry sandbox create on MUTATION_LOCK_TIMEOUT in the SDK and CLI #5697

Open

Description

@FrostGod

User Story

I run OpenShell density trials on EKS (1,200 sandboxes created from one controller). When the gateway is saturated, sandbox creates fail with a transient UNAVAILABLE that the server marks as retryable, and I need the CLI and SDK to retry them so a burst of creates completes without every caller writing its own retry loop.

Problem Statement

After #4696, a create that waits more than 10 s for a sandbox mutation lock or the SSH identity step fails with UNAVAILABLE. The status carries ErrorInfo.reason = MUTATION_LOCK_TIMEOUT and RetryInfo.retry_delay = 1s. The server rejects instead of queuing so saturation stays visible; the maintainers chose to keep retry out of the server (see the discussion on #4696).

No client acts on that signal today:

  • openshell sandbox create prints the status and exits non-zero.
  • The Rust SDK (openshell-sdk) retries only streams. SdkError::retry_delay() decodes the hint, but SdkError::retryable() returns false for every RPC error, so callers can't tell this case apart from a permanent failure.

Two server behaviors make a naive retry unreliable:

  • When failed-create cleanup can't finish, the gateway rebuilds the status with an appended message and drops the ErrorInfo and RetryInfo details. The code stays UNAVAILABLE, but the reason and hint are lost.
  • A failed create can leave the record in Deleting until cleanup or reconciliation finishes, so an immediate retry with the same name can return AlreadyExists.

Impact / Why This Matters

During burst creation, some creates fail with a transient error that would succeed one second later. Each automation (density harnesses, agent orchestrators, CI) has to rediscover the reason code, parse RetryInfo, and handle the AlreadyExists race on its own. Without that, large fan-out jobs drop a fraction of their sandboxes under load.

Proposed Design

  • The SDK classifies UNAVAILABLE with ErrorInfo.reason = MUTATION_LOCK_TIMEOUT as retryable, and creates retry it with bounded attempts or a total deadline. Each wait is at least RetryInfo.retry_delay, with jitter so a burst does not retry in lockstep.
  • The CLI uses that retry for sandbox create by default. A flag or setting controls the retry budget or turns retry off.
  • A retry that hits AlreadyExists for a sandbox the client just failed to create, while that record is still Deleting, waits for cleanup to finish and then retries. It does not report a name conflict to the user.
  • The gateway keeps ErrorInfo and RetryInfo on MUTATION_LOCK_TIMEOUT failures even when failed-create cleanup appends to the message.
  • Other UNAVAILABLE errors and all other codes keep today's behavior.

Suggested UX (if applicable)

openshell sandbox create --name worker-17 --no-tty --detach
# gateway busy (MUTATION_LOCK_TIMEOUT); retrying in 1.2s (attempt 2/5)
# sandbox worker-17 created

openshell sandbox create --name worker-17 --create-retry-timeout 60s
openshell sandbox create --name worker-17 --no-create-retry
let sandbox = client
    .create_sandbox(request)
    .retry(RetryPolicy::lock_contention().max_elapsed(Duration::from_secs(60)))
    .await?;

Acceptance Criteria

  • SdkError exposes whether a failure is a retryable lock-contention timeout, based on ErrorInfo.reason.
  • SDK and CLI create retries wait at least the server's retry_delay per attempt, add jitter, and stop at a configurable budget.
  • A same-name retry after a failed create does not surface AlreadyExists while the earlier record is still being cleaned up.
  • MUTATION_LOCK_TIMEOUT failures keep ErrorInfo and RetryInfo after failed-create cleanup modifies the message.
  • Errors other than MUTATION_LOCK_TIMEOUT are not retried.
  • Tests cover the retry, the delay floor, budget exhaustion, and the AlreadyExists cleanup race.

Alternatives Considered

  • Retry inside the gateway. Rejected on fix(server): scope the SSH identity lock per sandbox and bound data-pool lock holders #4696: under saturation it re-queues creates and hides the timeout signal.
  • Leave retry to every caller. This is today's behavior. It duplicates reason-code parsing and the AlreadyExists handling in each integration, and most callers get it wrong or skip it.
  • A gateway interceptor or middleware. These run server-side and have the same drawback as gateway retry. They cannot wait on the client's behalf without holding the request open.

Agent Investigation

Activity

  1. added
    state:acceptedA maintainer decided OpenShell should pursue this issue
    on Nov 18, 2026
  2. yjjnls commented on Nov 18, 2026

    @yjjnls

    Adding an operator data point in support of the proposed design — we hit the failure family this issue targets from the opposite side: not lock contention, but a version-fencing mismatch that turned every conditional write from one component into a spurious conflict, and the eventual fix was exactly the "treat the conflict as a refresh signal and retry" semantics proposed here.

    We run a multi-tenant LLM gateway control plane where configuration rows are guarded by a monotonic version counter (version fencing): writers send their last-read version, and a mismatch is rejected with a 409-style conflict. Two findings from a production incident there that map directly onto this issue's acceptance criteria:

    1. Spurious conflicts have a bootstrap/lineage failure mode, and only refresh-retry fixes it. A config-sync component created rows at the default version line (1) while a long-lived management component still held versions from the pre-existing lineage (29) in memory. Every subsequent conditional write from that component was rejected as conflicting even though nothing concurrent had modified the rows — the same user-visible symptom as false RESOURCE_VERSION_CONFLICTs (see #5227): reliability looks broken, but the real defect is stale client-side state. Our stopgap was advancing the stored version past the stale lineage to restore fencing monotonicity; the durable fix we converged on is: on conflict, the client re-reads the current version and retries the conditional write with the refreshed value, bounded attempts, with the retry carrying fresh state rather than replaying the old write. Blind retries are useless in this shape — they fail identically every time.

    2. Retry classification must key on structured reason codes, and those codes must survive error re-wrapping. This issue already requires classifying UNAVAILABLE by ErrorInfo.reason = MUTATION_LOCK_TIMEOUT rather than by code alone — we strongly endorse that, from a related scar: an intermediary in our stack re-wrapped error responses (appending cleanup context to the message) and dropped the structured details, which silently disabled every client's retryable/not-retryable classification and turned transient failures into terminal ones. Keeping ErrorInfo/RetryInfo intact when compensate_failed_create appends to the message (acceptance criterion 4) is the difference between a retry policy that works and one that quietly never fires.

    Two additional semantics points from operating retry loops at this scale:

    • Distinguish the three retry shapes. Lock-timeout is "retry blind" (no state changed; wait ≥ retry_delay, add jitter). The AlreadyExists-while-Deleting race is "refresh then proceed" (wait for the record to leave Deleting; retrying immediately just fails again — treating it as a name conflict, as the CLI does today, is the worst presentation of the three). Version conflicts are "retry with refreshed state". A single retry policy should expose all three rather than collapsing them, since the correct wait predicate differs.
    • Bounded budget with jitter is not optional at fan-out scale. Our first-generation consistency checker used naive retries at default per-request rate limits across a few hundred objects; concurrent runs stacked into 5–9 minute cycles against a 3-minute budget and writers collided with each other. The jitter requirement in the proposed design is what keeps a saturated gateway from converting visible saturation into lockstep retry waves.

    We verified this issue is accepted and currently unassigned with no linked PR. If it helps, we can exercise a candidate SDK/CLI implementation against a burst-create harness (saturated gateway, mixed fast/slow cleanup paths) and report retry-budget and AlreadyExists-race behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:acceptedA maintainer decided OpenShell should pursue this issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions