Repository navigation
Description
User Story
I use OpenShell, through OpenClaw Enterprise, to run Kubernetes-backed agent sandboxes and change their network policy while they run. I directly encountered live policy updates failing on an idle sandbox: UpdateConfig with expected_resource_version returned RESOURCE_VERSION_CONFLICT although no client had modified the sandbox. As a result, an approved network rule was never enforced.
Problem Statement
On a running, unchanged sandbox, the gateway rewrites the stored sandbox object and increments metadata.resource_version on every compute reconciliation pass. With the Kubernetes driver, this happens about three times a minute: each Listing sandboxes from Kubernetes pass (about every 30 s) and each reconcile.sandboxes sweep (about every 60 s).
The rewritten object is semantically identical to the previous one. I decoded consecutive stored payloads with openshell.v1.Sandbox from proto/openshell.proto. The only field that differs is metadata.resource_version; the raw bytes also differ because map entries (labels and annotations) are serialized in a different order.
A client must read the sandbox version and then send it back as expected_resource_version. Any reconciliation pass between those two steps makes the precondition stale, even though nothing about the sandbox changed.
Impact / Why This Matters
expected_resource_version is meant to protect against concurrent writers. Here it rejects updates with no concurrent change, so it gives no concurrency protection, only false conflicts. In this case the client read the version at 00:49:38.38 and wrote it back 0.4 s later. A reconciliation pass at 00:49:38.505 invalidated it:
00:49:38.360 GetSandbox OK
00:49:38.382 GetSandboxPolicyStatus OK <- client reads resource_version
00:49:38.505 openshell_driver_kubernetes::driver: Listing sandboxes from Kubernetes
00:49:38.792 UpdateConfig grpc-status 10 (ABORTED, RESOURCE_VERSION_CONFLICT)
The conflict is also costly to recover from. The failed request's durable mutation admission is left unresolved (success: null). Every retry with the same request_id is therefore rejected with FAILED_PRECONDITION REQUEST_OUTCOME_UNCERTAIN; the source comments say this is intentional. The client must reconcile and start a new request, and the policy is not enforced in the meantime. A fresh request a few minutes later succeeded.
The workaround is to re-read and retry with a new request_id until no reconciliation pass lands in the window. The frequency of these false conflicts scales with the number of sandboxes and the reconciliation rate.
Two smaller observability gaps made this hard to diagnose:
- The gateway logs these failed mutations at INFO as
http.response.status_code=200with onlyrpc.grpc.status_code, without the status message or error-info reason. - An
ABORTEDconflict that the atomic policy write rejected before any effect still leaves an unresolved admission, so it reads as "outcome uncertain" on retry.
Acceptance Criteria
- A reconciliation pass that finds no change does not rewrite the sandbox object or advance
metadata.resource_version. - For an idle sandbox,
GetSandboxreturns the sameresource_versionacross several reconciliation intervals. -
UpdateConfigwithexpected_resource_versionread fromGetSandboxsucceeds when no other writer changed the sandbox in between, regardless of reconciliation timing. - A rejected mutation's status message and reason appear in gateway logs.
Reproduction Steps
I observed this through OpenClaw Enterprise. The OpenShell-only steps below follow the same path but I have not run them separately.
- Start a gateway with the Kubernetes compute driver and SQLite storage, and create a sandbox. Leave it idle.
- Call
GetSandboxevery 5 s for 90 s and recordmetadata.resource_version. It increases about three times a minute, matching theListing sandboxes from Kubernetesandreconcile.sandboxeslog lines, with no other client writes.- Observed: 315 at 01:00:38, then 316 (01:00:38.593), 317 (01:01:01.816), 318 (01:01:08.574) and 319 (01:01:39.479).
- Read
resource_version, wait until just after a reconciliation pass, then callUpdateConfigwith a changed policy and thatexpected_resource_version. It fails withABORTEDRESOURCE_VERSION_CONFLICT. - Retry step 3 with the same
request_id. It fails withFAILED_PRECONDITIONREQUEST_OUTCOME_UNCERTAIN.
Environment
- OpenShell gateway 0.1.6 (
ghcr.io/nvidia/openshell/gateway:0.1.6@sha256:f0b36dba21d282c52e498c25289658cfd6cd4f343b0069fde5609e25b002cdea, tag commitac7f615c), with the Kubernetes compute driver. - Storage: SQLite (
--db-url sqlite:/var/openshell/openshell.db). - Cluster: k3d (k3s) on Fedora 44, x86_64.
- Not tested on
main. None of the 53 commits after v0.1.6 appear to address this.mutation_replay.rskeeps the same unresolved-admission behavior.
Logs
Stored sandbox row sampled from the gateway database every 5 s on an idle sandbox:
01:00:38 {"rv":315,"updated":"01:00:08.520"}
01:00:43 {"rv":316,"updated":"01:00:38.593"}
01:01:03 {"rv":317,"updated":"01:01:01.816"}
01:01:08 {"rv":318,"updated":"01:01:08.574"}
01:01:44 {"rv":319,"updated":"01:01:39.479"}
Gateway log over the same interval:
01:00:01.680 reconcile{otel.name="reconcile.sandboxes"}
01:00:08.514 openshell_driver_kubernetes::driver: Listing sandboxes from Kubernetes
01:00:38.588 openshell_driver_kubernetes::driver: Listing sandboxes from Kubernetes
01:01:01.769 reconcile{otel.name="reconcile.sandboxes"}
01:01:08.568 openshell_driver_kubernetes::driver: Listing sandboxes from Kubernetes
01:01:39.126 openshell_driver_kubernetes::driver: Listing sandboxes from Kubernetes
Semantic diff of stored Sandbox payloads at resource versions 313 and 319, decoded with openshell.v1.Sandbox and with map keys sorted:
sandbox.metadata.resource_version: "312" -> "318"
Mutation admission left by the failed request (hashes elided):
{"format_version":1,"payload_hash":"<hash>","workspace_id":"7b663fd6-…","target_id":null,"success":null,"completed_at_ms":null}Related: #4062 concerns policy revisions that do not advance the resource version. This issue is the reverse: non-policy no-op writes that do advance it.
Activity
yjjnls commentedon Nov 18, 2026
We operate a state-consistency checker that reconciles desired-vs-actual configuration across a few hundred objects on a fixed cron, and we went through precisely the failure mode this issue describes: a reconciler whose no-op passes still rewrote stored objects, advanced version counters, and manufactured conflicts for every concurrent conditional writer. Sharing what it took to shut it down, mapped onto the acceptance criteria here.
The false update is a serialization artifact, not a state change — and diffing raw bytes guarantees it. Our reconciler's first implementation compared raw serialized payloads and wrote back whenever bytes differed. The "differences" were map serialization order and fields the reconciler did not own — the exact phenomenon in this issue's semantic diff (only
resource_versiondiffers; raw bytes differ in label/annotation ordering). Byte-level comparison cannot be salvaged by sampling or window tuning; the fix is canonical comparison: decode to the typed object, sort map keys, exclude non-owned/volatile fields, and write only when semantic content differs. That is acceptance criterion 1, and in our experience it is the only durable fix — shrinking the read-modify-write window just rescales the collision rate instead of removing its source.Two-run arbitration: never write on the first observation of drift. The second rule we shipped: a drift detected in one pass must be re-confirmed on a second pass (or after a fresh read) before any write is issued. This is what separates real change from transient reads, in-flight mutations, and idempotent-retry ghosts — and it is what prevents the write-storm failure mode, where a reconciler that writes on every uncertain observation generates the very conflicts that trigger more retries. The false-conflict rate has a simple shape: roughly (client read-modify-write window) × (background write rate) × object count. Your measured ~3 no-op writes/min per sandbox means windows collide constantly across a fleet even for a 0.4 s window; eliminating no-op writes collapses the rate to actual-change-only.
Scale data point. Our checker's per-object polling form (point GETs at default per-client rate limits, ~4–6 reads per object across a few hundred objects) took ~3 minutes for a single pass, and concurrent passes stacked to 5–9 minutes against a 3-minute cron budget. After the rewrite — list-based reads, bidirectional set-diff on canonicalized state, two-run confirmation, and write only on confirmed semantic drift — steady-state passes converge to a ~7 s terminal state with effectively zero writes when nothing changed. The reconciler went from a source of version churn to something a conditional writer can share a version space with, which is exactly the property criterion 2 ("same
resource_versionacross several reconciliation intervals") asserts.The unresolved-admission trap deserves its own fix. The
success: null→REQUEST_OUTCOME_UNCERTAINbehavior on same-request_idretries matches a scar of ours: a rejected conditional write left the outcome record unresolved, so every retry was rejected as uncertain and the client had to run a full reconciliation protocol for a mutation that provably never took effect. AnABORTEDconflict rejected before any durable effect is precisely the "definitely not applied" case, and recording the outcome at reject time (not only at success time) lets clients retry safely instead of forking a new request id and losing the audit trail. We would rank that alongside the no-op-write fix in user-visible impact, because it determines whether recovery from the remaining conflicts is one retry or a protocol.One endorsement on observability (criterion 4): logging only the grpc status code with an HTTP 200-style success line is a diagnosis tax we have paid repeatedly — status message and error-info reason in gateway logs are what make conflicts attributable without byte-level log forensics.