Repository navigation
sdk-http-vertx: response writes ignore Vert.x backpressure #614
Description
Activity
I took a run at this and shipped the parts that are safe to land in #636 (the request-side companion + a latent bug it surfaced). Findings on the main response-side proposal:
-
Implementing the bounded window as proposed surfaces a blocker in
RequestProcessorImpl: its output subscription hard-assertsn == Long.MAX_VALUE("We don't support backpressure here because writing to the output stream is driven by code"). Any boundedrequest(n)from the adapter trips the assert under-ea, and with assertions off the state machine's output emission ignores demand anyway - so the adapter-side fix needs the invocation output pipeline to become demand-aware first. That's a design call I didn't want to make unilaterally in sdk-core. -
StaticResponseRequestProcessorre-emits the whole body on everyrequest()(RS rules 1.7/2.12). With today'srequest(Long.MAX_VALUE)-once subscribers it's invisible, but E2E with a bounded-demand subscriber duplicated the discovery response ~1000x (2.39 MB body, runtime rejects withtrailing characters at line 1 column 1180). Fixed in Emit static responses exactly once regardless of subscriber demand shape #636 - worth landing regardless of how the response side gets designed, since it bites the first bounded subscriber anywhere.
Happy to do the response-side follow-up once there's a direction for the
RequestProcessorImploutput demand semantics.-
- added a commit that references this issue
on Jul 14, 2026 - added a commit that references this issue
on Jul 14, 2026 Possibly related data point: bisecting a Restate SDK bump in our project (2.9.0 → 2.9.4), we found
:powerlink:test green at 2.9.0/2.9.1 and consistently broken at 2.9.2/2.9.3/2.9.4. Three tests fail identically
in RestateExtension.beforeAll → RestateRunner.start, before any test body runs:ApiException: createDeployment call failed with: 500 - {"message":"[META0003] ... h2 pool connection error: stream closed because of a broken pipe"}Reproducible in both directions by toggling only restateVersion, isolated to the 2.9.1 → 2.9.2 release, which
shipped #636. We confirmed sdk-java-http pulls in sdk-http-vertx transitively, so HttpResponseFlowAdapter is in
the runtime path — but we haven't traced the exact mechanism server-side, so flagging the correlation rather
than claiming a confirmed root cause. Environment: macOS + Docker Desktop, Testcontainers
(host.testcontainers.internal reverse proxy), sdk-testing's RestateRunner registering a deployment against a
real Restate server container.
Problem
HttpResponseFlowAdapterinsdk-http-vertxignores Vert.x's write-side backpressure on the HTTP/2 response stream:onSubscribepulls unboundedly:onNextwrites each slice unconditionally:HttpServerResponse.writeQueueFull(), nodrainHandler(...)to resume.The Reactive-Streams contract and Vert.x's own
WriteStreamdocs both expect callers to throttle production whenwriteQueueFull()returnstrue. The adapter throws away that signal.Observed evidence
While investigating a separate e2e issue, instrumentation on the SDK side recorded
writeQueueFull=trueon the very first response writes under a 50-concurrentctx.run× 10×64 KiB workload (Restate runtime as the peer). That confirms the writeQueue does cross Vert.x's high-watermark threshold in realistic Restate workloads — the adapter just keeps writing past it.(We did not observe SDK-side heap growth or
autoReadtoggling in that test, so this issue is a code-correctness / future-proofing fix rather than the cause of the failure we were chasing.)Proposal
Apply standard Reactive-Streams + Vert.x backpressure to
HttpResponseFlowAdapter:request(Long.MAX_VALUE)with a bounded initial request (e.g.request(N)).onNext, after writing, checkhttpServerResponse.writeQueueFull():subscription.request(...); instead installhttpServerResponse.drainHandler(v -> subscription.request(M))to resume.Pumphelper implements.Companion (lower priority, same class of bug)
The request side has an analogous unbounded enqueue:
HttpRequestFlowAdapter.handleIncomingBufferpushes incoming buffers into anArrayDeque<ByteBuffer>without an upper bound. Worth bounding while we're in the same module.Files
sdk-http-vertx/src/main/java/dev/restate/sdk/http/vertx/HttpResponseFlowAdapter.javasdk-http-vertx/src/main/java/dev/restate/sdk/http/vertx/HttpRequestFlowAdapter.java(companion)