Repository navigation
Description
User Story
I run OpenShell density trials on EKS (1,200 sandboxes created from one controller). When the gateway is saturated, sandbox creates fail with a transient UNAVAILABLE that the server marks as retryable, and I need the CLI and SDK to retry them so a burst of creates completes without every caller writing its own retry loop.
Problem Statement
After #4696, a create that waits more than 10 s for a sandbox mutation lock or the SSH identity step fails with UNAVAILABLE. The status carries ErrorInfo.reason = MUTATION_LOCK_TIMEOUT and RetryInfo.retry_delay = 1s. The server rejects instead of queuing so saturation stays visible; the maintainers chose to keep retry out of the server (see the discussion on #4696).
No client acts on that signal today:
openshell sandbox createprints the status and exits non-zero.- The Rust SDK (
openshell-sdk) retries only streams.SdkError::retry_delay()decodes the hint, butSdkError::retryable()returnsfalsefor every RPC error, so callers can't tell this case apart from a permanent failure.
Two server behaviors make a naive retry unreliable:
- When failed-create cleanup can't finish, the gateway rebuilds the status with an appended message and drops the
ErrorInfoandRetryInfodetails. The code staysUNAVAILABLE, but the reason and hint are lost. - A failed create can leave the record in
Deletinguntil cleanup or reconciliation finishes, so an immediate retry with the same name can returnAlreadyExists.
Impact / Why This Matters
During burst creation, some creates fail with a transient error that would succeed one second later. Each automation (density harnesses, agent orchestrators, CI) has to rediscover the reason code, parse RetryInfo, and handle the AlreadyExists race on its own. Without that, large fan-out jobs drop a fraction of their sandboxes under load.
Proposed Design
- The SDK classifies
UNAVAILABLEwithErrorInfo.reason = MUTATION_LOCK_TIMEOUTas retryable, and creates retry it with bounded attempts or a total deadline. Each wait is at leastRetryInfo.retry_delay, with jitter so a burst does not retry in lockstep. - The CLI uses that retry for
sandbox createby default. A flag or setting controls the retry budget or turns retry off. - A retry that hits
AlreadyExistsfor a sandbox the client just failed to create, while that record is stillDeleting, waits for cleanup to finish and then retries. It does not report a name conflict to the user. - The gateway keeps
ErrorInfoandRetryInfoonMUTATION_LOCK_TIMEOUTfailures even when failed-create cleanup appends to the message. - Other
UNAVAILABLEerrors and all other codes keep today's behavior.
Suggested UX (if applicable)
openshell sandbox create --name worker-17 --no-tty --detach
# gateway busy (MUTATION_LOCK_TIMEOUT); retrying in 1.2s (attempt 2/5)
# sandbox worker-17 created
openshell sandbox create --name worker-17 --create-retry-timeout 60s
openshell sandbox create --name worker-17 --no-create-retrylet sandbox = client
.create_sandbox(request)
.retry(RetryPolicy::lock_contention().max_elapsed(Duration::from_secs(60)))
.await?;Acceptance Criteria
-
SdkErrorexposes whether a failure is a retryable lock-contention timeout, based onErrorInfo.reason. - SDK and CLI create retries wait at least the server's
retry_delayper attempt, add jitter, and stop at a configurable budget. - A same-name retry after a failed create does not surface
AlreadyExistswhile the earlier record is still being cleaned up. -
MUTATION_LOCK_TIMEOUTfailures keepErrorInfoandRetryInfoafter failed-create cleanup modifies the message. - Errors other than
MUTATION_LOCK_TIMEOUTare not retried. - Tests cover the retry, the delay floor, budget exhaustion, and the
AlreadyExistscleanup race.
Alternatives Considered
- Retry inside the gateway. Rejected on fix(server): scope the SSH identity lock per sandbox and bound data-pool lock holders #4696: under saturation it re-queues creates and hides the timeout signal.
- Leave retry to every caller. This is today's behavior. It duplicates reason-code parsing and the
AlreadyExistshandling in each integration, and most callers get it wrong or skip it. - A gateway interceptor or middleware. These run server-side and have the same drawback as gateway retry. They cannot wait on the client's behalf without holding the request open.
Agent Investigation
- Status built in
crates/openshell-server/src/grpc/mod.rs(MUTATION_LOCK_TIMEOUT, 1 sRetryInfo). - Details dropped in
compensate_failed_createandcompensate_failed_create_inner(crates/openshell-server/src/compute/mod.rs), which returnStatus::new(original.code(), ...)on partial cleanup. - CLI create in
crates/openshell-cli/src/run.rshandles onlyAlreadyExistsspecially. - SDK:
SdkError::retry_delay()andSdkError::retryable()incrates/openshell-sdk/src/error.rs. - Related: fix(server): scope the SSH identity lock per sandbox and bound data-pool lock holders #4686, fix(server): scope the SSH identity lock per sandbox and bound data-pool lock holders #4696.
Activity
yjjnls commentedon Nov 18, 2026
Adding an operator data point in support of the proposed design — we hit the failure family this issue targets from the opposite side: not lock contention, but a version-fencing mismatch that turned every conditional write from one component into a spurious conflict, and the eventual fix was exactly the "treat the conflict as a refresh signal and retry" semantics proposed here.
We run a multi-tenant LLM gateway control plane where configuration rows are guarded by a monotonic version counter (version fencing): writers send their last-read version, and a mismatch is rejected with a 409-style conflict. Two findings from a production incident there that map directly onto this issue's acceptance criteria:
1. Spurious conflicts have a bootstrap/lineage failure mode, and only refresh-retry fixes it. A config-sync component created rows at the default version line (1) while a long-lived management component still held versions from the pre-existing lineage (29) in memory. Every subsequent conditional write from that component was rejected as conflicting even though nothing concurrent had modified the rows — the same user-visible symptom as false
RESOURCE_VERSION_CONFLICTs (see #5227): reliability looks broken, but the real defect is stale client-side state. Our stopgap was advancing the stored version past the stale lineage to restore fencing monotonicity; the durable fix we converged on is: on conflict, the client re-reads the current version and retries the conditional write with the refreshed value, bounded attempts, with the retry carrying fresh state rather than replaying the old write. Blind retries are useless in this shape — they fail identically every time.2. Retry classification must key on structured reason codes, and those codes must survive error re-wrapping. This issue already requires classifying
UNAVAILABLEbyErrorInfo.reason = MUTATION_LOCK_TIMEOUTrather than by code alone — we strongly endorse that, from a related scar: an intermediary in our stack re-wrapped error responses (appending cleanup context to the message) and dropped the structured details, which silently disabled every client's retryable/not-retryable classification and turned transient failures into terminal ones. KeepingErrorInfo/RetryInfointact whencompensate_failed_createappends to the message (acceptance criterion 4) is the difference between a retry policy that works and one that quietly never fires.Two additional semantics points from operating retry loops at this scale:
retry_delay, add jitter). TheAlreadyExists-while-Deletingrace is "refresh then proceed" (wait for the record to leaveDeleting; retrying immediately just fails again — treating it as a name conflict, as the CLI does today, is the worst presentation of the three). Version conflicts are "retry with refreshed state". A single retry policy should expose all three rather than collapsing them, since the correct wait predicate differs.We verified this issue is accepted and currently unassigned with no linked PR. If it helps, we can exercise a candidate SDK/CLI implementation against a burst-create harness (saturated gateway, mixed fast/slow cleanup paths) and report retry-budget and
AlreadyExists-race behavior.