Skip to content

bug(compute): reconciliation rewrites unchanged sandboxes and advances resource_version, causing false UpdateConfig conflicts #5227

Open

Description

@mrunalp

User Story

I use OpenShell, through OpenClaw Enterprise, to run Kubernetes-backed agent sandboxes and change their network policy while they run. I directly encountered live policy updates failing on an idle sandbox: UpdateConfig with expected_resource_version returned RESOURCE_VERSION_CONFLICT although no client had modified the sandbox. As a result, an approved network rule was never enforced.

Problem Statement

On a running, unchanged sandbox, the gateway rewrites the stored sandbox object and increments metadata.resource_version on every compute reconciliation pass. With the Kubernetes driver, this happens about three times a minute: each Listing sandboxes from Kubernetes pass (about every 30 s) and each reconcile.sandboxes sweep (about every 60 s).

The rewritten object is semantically identical to the previous one. I decoded consecutive stored payloads with openshell.v1.Sandbox from proto/openshell.proto. The only field that differs is metadata.resource_version; the raw bytes also differ because map entries (labels and annotations) are serialized in a different order.

A client must read the sandbox version and then send it back as expected_resource_version. Any reconciliation pass between those two steps makes the precondition stale, even though nothing about the sandbox changed.

Impact / Why This Matters

expected_resource_version is meant to protect against concurrent writers. Here it rejects updates with no concurrent change, so it gives no concurrency protection, only false conflicts. In this case the client read the version at 00:49:38.38 and wrote it back 0.4 s later. A reconciliation pass at 00:49:38.505 invalidated it:

00:49:38.360  GetSandbox                 OK
00:49:38.382  GetSandboxPolicyStatus     OK    <- client reads resource_version
00:49:38.505  openshell_driver_kubernetes::driver: Listing sandboxes from Kubernetes
00:49:38.792  UpdateConfig               grpc-status 10 (ABORTED, RESOURCE_VERSION_CONFLICT)

The conflict is also costly to recover from. The failed request's durable mutation admission is left unresolved (success: null). Every retry with the same request_id is therefore rejected with FAILED_PRECONDITION REQUEST_OUTCOME_UNCERTAIN; the source comments say this is intentional. The client must reconcile and start a new request, and the policy is not enforced in the meantime. A fresh request a few minutes later succeeded.

The workaround is to re-read and retry with a new request_id until no reconciliation pass lands in the window. The frequency of these false conflicts scales with the number of sandboxes and the reconciliation rate.

Two smaller observability gaps made this hard to diagnose:

  • The gateway logs these failed mutations at INFO as http.response.status_code=200 with only rpc.grpc.status_code, without the status message or error-info reason.
  • An ABORTED conflict that the atomic policy write rejected before any effect still leaves an unresolved admission, so it reads as "outcome uncertain" on retry.

Acceptance Criteria

  • A reconciliation pass that finds no change does not rewrite the sandbox object or advance metadata.resource_version.
  • For an idle sandbox, GetSandbox returns the same resource_version across several reconciliation intervals.
  • UpdateConfig with expected_resource_version read from GetSandbox succeeds when no other writer changed the sandbox in between, regardless of reconciliation timing.
  • A rejected mutation's status message and reason appear in gateway logs.

Reproduction Steps

I observed this through OpenClaw Enterprise. The OpenShell-only steps below follow the same path but I have not run them separately.

  1. Start a gateway with the Kubernetes compute driver and SQLite storage, and create a sandbox. Leave it idle.
  2. Call GetSandbox every 5 s for 90 s and record metadata.resource_version. It increases about three times a minute, matching the Listing sandboxes from Kubernetes and reconcile.sandboxes log lines, with no other client writes.
    • Observed: 315 at 01:00:38, then 316 (01:00:38.593), 317 (01:01:01.816), 318 (01:01:08.574) and 319 (01:01:39.479).
  3. Read resource_version, wait until just after a reconciliation pass, then call UpdateConfig with a changed policy and that expected_resource_version. It fails with ABORTED RESOURCE_VERSION_CONFLICT.
  4. Retry step 3 with the same request_id. It fails with FAILED_PRECONDITION REQUEST_OUTCOME_UNCERTAIN.

Environment

  • OpenShell gateway 0.1.6 (ghcr.io/nvidia/openshell/gateway:0.1.6@sha256:f0b36dba21d282c52e498c25289658cfd6cd4f343b0069fde5609e25b002cdea, tag commit ac7f615c), with the Kubernetes compute driver.
  • Storage: SQLite (--db-url sqlite:/var/openshell/openshell.db).
  • Cluster: k3d (k3s) on Fedora 44, x86_64.
  • Not tested on main. None of the 53 commits after v0.1.6 appear to address this. mutation_replay.rs keeps the same unresolved-admission behavior.

Logs

Stored sandbox row sampled from the gateway database every 5 s on an idle sandbox:

01:00:38 {"rv":315,"updated":"01:00:08.520"}
01:00:43 {"rv":316,"updated":"01:00:38.593"}
01:01:03 {"rv":317,"updated":"01:01:01.816"}
01:01:08 {"rv":318,"updated":"01:01:08.574"}
01:01:44 {"rv":319,"updated":"01:01:39.479"}

Gateway log over the same interval:

01:00:01.680  reconcile{otel.name="reconcile.sandboxes"}
01:00:08.514  openshell_driver_kubernetes::driver: Listing sandboxes from Kubernetes
01:00:38.588  openshell_driver_kubernetes::driver: Listing sandboxes from Kubernetes
01:01:01.769  reconcile{otel.name="reconcile.sandboxes"}
01:01:08.568  openshell_driver_kubernetes::driver: Listing sandboxes from Kubernetes
01:01:39.126  openshell_driver_kubernetes::driver: Listing sandboxes from Kubernetes

Semantic diff of stored Sandbox payloads at resource versions 313 and 319, decoded with openshell.v1.Sandbox and with map keys sorted:

sandbox.metadata.resource_version: "312" -> "318"

Mutation admission left by the failed request (hashes elided):

{"format_version":1,"payload_hash":"<hash>","workspace_id":"7b663fd6-…","target_id":null,"success":null,"completed_at_ms":null}

Related: #4062 concerns policy revisions that do not advance the resource version. This issue is the reverse: non-policy no-op writes that do advance it.

Activity

  1. added
    area:platformSDKs, CLIs, APIs, gateway infrastructure, persistence, configuration, and session management.
    topic:apiPublic APIs, gRPC/protobuf, SDK interfaces, versioning, compatibility, and API design standards.
    topic:observabilityLogs, metrics, traces, telemetry, dashboards, event correlation, and operational data export.
    topic:reliabilityHigh availability, recovery, retries, graceful degradation, and correctness across restarts.
    on Nov 15, 2026
  2. yjjnls commented on Nov 18, 2026

    @yjjnls

    We operate a state-consistency checker that reconciles desired-vs-actual configuration across a few hundred objects on a fixed cron, and we went through precisely the failure mode this issue describes: a reconciler whose no-op passes still rewrote stored objects, advanced version counters, and manufactured conflicts for every concurrent conditional writer. Sharing what it took to shut it down, mapped onto the acceptance criteria here.

    The false update is a serialization artifact, not a state change — and diffing raw bytes guarantees it. Our reconciler's first implementation compared raw serialized payloads and wrote back whenever bytes differed. The "differences" were map serialization order and fields the reconciler did not own — the exact phenomenon in this issue's semantic diff (only resource_version differs; raw bytes differ in label/annotation ordering). Byte-level comparison cannot be salvaged by sampling or window tuning; the fix is canonical comparison: decode to the typed object, sort map keys, exclude non-owned/volatile fields, and write only when semantic content differs. That is acceptance criterion 1, and in our experience it is the only durable fix — shrinking the read-modify-write window just rescales the collision rate instead of removing its source.

    Two-run arbitration: never write on the first observation of drift. The second rule we shipped: a drift detected in one pass must be re-confirmed on a second pass (or after a fresh read) before any write is issued. This is what separates real change from transient reads, in-flight mutations, and idempotent-retry ghosts — and it is what prevents the write-storm failure mode, where a reconciler that writes on every uncertain observation generates the very conflicts that trigger more retries. The false-conflict rate has a simple shape: roughly (client read-modify-write window) × (background write rate) × object count. Your measured ~3 no-op writes/min per sandbox means windows collide constantly across a fleet even for a 0.4 s window; eliminating no-op writes collapses the rate to actual-change-only.

    Scale data point. Our checker's per-object polling form (point GETs at default per-client rate limits, ~4–6 reads per object across a few hundred objects) took ~3 minutes for a single pass, and concurrent passes stacked to 5–9 minutes against a 3-minute cron budget. After the rewrite — list-based reads, bidirectional set-diff on canonicalized state, two-run confirmation, and write only on confirmed semantic drift — steady-state passes converge to a ~7 s terminal state with effectively zero writes when nothing changed. The reconciler went from a source of version churn to something a conditional writer can share a version space with, which is exactly the property criterion 2 ("same resource_version across several reconciliation intervals") asserts.

    The unresolved-admission trap deserves its own fix. The success: null → REQUEST_OUTCOME_UNCERTAIN behavior on same-request_id retries matches a scar of ours: a rejected conditional write left the outcome record unresolved, so every retry was rejected as uncertain and the client had to run a full reconciliation protocol for a mutation that provably never took effect. An ABORTED conflict rejected before any durable effect is precisely the "definitely not applied" case, and recording the outcome at reject time (not only at success time) lets clients retry safely instead of forking a new request id and losing the audit trail. We would rank that alongside the no-op-write fix in user-visible impact, because it determines whether recovery from the remaining conflicts is one retry or a protocol.

    One endorsement on observability (criterion 4): logging only the grpc status code with an HTTP 200-style success line is a diagnosis tax we have paid repeatedly — status message and error-info reason in gateway logs are what make conflicts attributable without byte-level log forensics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:platformSDKs, CLIs, APIs, gateway infrastructure, persistence, configuration, and session management.topic:apiPublic APIs, gRPC/protobuf, SDK interfaces, versioning, compatibility, and API design standards.topic:observabilityLogs, metrics, traces, telemetry, dashboards, event correlation, and operational data export.topic:reliabilityHigh availability, recovery, retries, graceful degradation, and correctness across restarts.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions