Skip to content

Supervisor fails a request in flight when a middleware server asks it to reconnect (graceful GOAWAY) #5710

Open

Description

@tomorrowx-git

User Story

I run OpenShell 0.1.7 on Kubernetes with the Kubernetes driver. My sandboxes' HTTP and MCP traffic goes through a supervisor middleware, which I run as several instances behind a Kubernetes Service for capacity and availability.

When a middleware instance asks a supervisor to reconnect, by a graceful stop during a rolling update or by a maximum connection age, I see a request from that sandbox fail.

This affects every rolling update of the middleware, and it puts a cost on moving sandboxes between middleware instances.

Problem Statement

When a supervisor middleware's gRPC server closes a connection gracefully (HTTP/2 GOAWAY, with 5 seconds allowed for calls in flight), a request that the sandbox has in progress fails. It is not completed on the old connection and not retried on a new one.

The sandbox's client receives 403 "middleware_failed" or 502 "response_delivery_failed". Requests after it succeed on the new connection. With a busy sandbox this is about one failed request for each reconnect.

Impact / Why This Matters

The middleware attachment fails closed, so the agent in the sandbox sees a refused request it did nothing to cause. The 502 response says the upstream request may have completed, so a client that retries may repeat a side effect.

Measured on OpenShell 0.1.7, with sandboxes sending MCP tools/call requests continuously and the middleware set to a maximum connection age:

  • 32 sandboxes on 4 middleware instances, connection age 5 minutes, 15 minutes: 93 of 3,795,509 requests failed (54 with 502, 39 with 403).
  • 16 sandboxes on 2 instances, same settings: 46 of 1,904,765 failed.
  • 8 sandboxes on 4 instances, connection age 1 minute, 6 minutes: 69 of 1,215,394 failed.
  • A rolling update of the 4 instances under the same kind of load: 22 of 385,056 failed, about one for each sandbox attached to each instance as it closed.

There is no workaround on the middleware side: the server already waits 5 seconds for calls in flight. The choices today are to never ask a supervisor to reconnect, which leaves each sandbox on the instance it first reached, so instances added later take no existing load, or to have every agent retry failed requests itself.

Acceptance Criteria

  • A sandbox sending requests continuously sees no failed request when its middleware server sends GOAWAY and allows calls in flight to finish.
  • Requests issued after the GOAWAY go out on a new connection without an error reaching the sandbox.
  • A rolling update of a middleware that drains gracefully completes with no failed sandbox requests.

Reproduction Steps

  1. Deploy OpenShell 0.1.7 with the Kubernetes driver and register a supervisor middleware in the gateway configuration (openshell.supervisor.middleware with grpc_endpoint, TLS, timeout 2s). The middleware's gRPC server must be able to close a connection gracefully: GOAWAY, then at least 5 seconds for calls in flight.
  2. Create a sandbox with a policy that attaches the middleware to an HTTP endpoint the sandbox may call.
  3. From inside the sandbox, send requests to that endpoint continuously from two concurrent clients.
  4. While they run, have the middleware server close the supervisor's connection gracefully, either through a maximum connection age or by stopping the server gracefully.
  5. One of the sandbox's requests fails with 403 middleware_failed or 502 response_delivery_failed. The requests that follow succeed.

With one sandbox, two concurrent clients and a connection age of 5 minutes, a 7-minute run had exactly one failed request, at the moment of the reconnect.

Suggested UX (if applicable)

No response

Environment

  • OpenShell: 0.1.7 (openshell --version; Helm chart 0.1.7; gateway and supervisor images 0.1.7)
  • OS: Kubernetes 1.37 on Amazon EKS, Amazon Linux 2023 nodes
  • OpenShell deployment mode and runtime: Kubernetes driver, one supervisor pod for each sandbox; middleware registered over TLS with gateway-signed bearer tokens, timeout 2s
  • Middleware: a gRPC server (grpc-java 1.85.1 on Netty), 2 to 4 instances behind a ClusterIP Service

Logs

Responses received by the client in the sandbox (cut at 160 characters by my test client):

HTTP 403 {"error":"middleware_failed","detail":"Request could not be processed by configured middleware","policy":"<policy>","layer":"l7","method":"POST","path":"/mcp" ...

HTTP 502 {"error":"response_delivery_failed","detail":"The upstream request may have completed, but OpenShell could not deliver its response. Retrying may repeat upstrea ...

In the sandbox's log (openshell logs) I found no warning or error line for the failed request: the only sign of it is the response the client received.

Activity

  1. tomorrowx-git commented on Nov 18, 2026

    @tomorrowx-git
    Author

    A related observation from the same deployment. Tell me if you would rather have it as a separate issue.

    When a middleware instance disappears without closing its connections, the sandboxes connected to it have their requests refused for much longer than it takes a replacement to be ready.

    I force-deleted one of four middleware instances (kubectl delete pod --grace-period=0 --force) while 32 sandboxes were sending requests continuously, two concurrent clients each. Seven sandboxes were connected to that instance. Every request from those seven failed for 28 seconds: 14 failures every 2 seconds, each after waiting out the 2s middleware timeout, as 403 middleware_failed (199) or 502 response_delivery_failed (11). Then their supervisors opened new connections and requests succeeded again. The other 25 sandboxes saw no failures. A replacement instance was ready behind the same Service 5 seconds after the kill.

    In the 4-minute run 210 of 874,966 requests failed, all from those seven sandboxes in that 28-second window.

    What I would hope for: a supervisor that finds its middleware connection dead reconnects through the configured endpoint within about one middleware timeout, so a lost instance costs its sandboxes a request or two, not half a minute.

    Same environment as above: OpenShell 0.1.7, Kubernetes driver, middleware over TLS behind a ClusterIP Service, timeout 2s.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:triage-neededOpened without agent diagnostics and needs triage

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions