Repository navigation
test(e2e): pass mock LLM handlers to startModelRecorder - #3264
Merged
Merged
Conversation
The egress and mounted-skills mocks put a second server with an unbuffered httputil.ReverseProxy behind startModelRecorder. Once the streamed response starts, that server closes the request body. The proxy transport then reads the body once more, gets ErrBodyReadAfterClose, and closes the upstream connection mid-stream. The agent sees unexpected EOF, and the egress test times out. startModelRecorder now takes an http.Handler and calls it in process with the buffered body. The two mocks lose their second server, and the other callers wrap the mock URL with mockLLMProxy. Signed-off-by: Danielle Mustillo <danielle.mustillo@solo.io>
danielle-mustillo
force-pushed
the
danielle-mustillo/e2e-egress-full-duplex
branch
from
October 19, 2026 07:05
ed6e860 to
d465730
Compare
danielle-mustillo
marked this pull request as ready for review
October 19, 2026 07:47
EItanya
approved these changes
Oct 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The
test-e2e (microvm)job fails at random on egress tests withunexpected EOF(job, job). The egress and mounted-skills mocks put a second server with an unbufferedhttputil.ReverseProxybehindstartModelRecorder. Once the streamed response starts, that server closes the request body. The proxy transport then reads the body once more past Content-Length, getsErrBodyReadAfterClose, and closes the upstream connection mid-stream.What changed
startModelRecordertakes anhttp.Handlerand calls it in process with the buffered body, so no proxy reads a body that the server can close.mockLLMProxy.TestModelRecorderBodyOutlivesResponseStartchecks that the upstream handler can read the body after the response starts.Scope
use of closed network connectionandunexpected EOFflake only.live fixed did not become ALLOWEDflake in the claude variant.Tradeoffs
EnableFullDuplexin each mock also fixes the race, but each new mock has to remember it. Removing the second hop removes the bug class.Blast Radius
Test code only. Every e2e test that uses
startModelRecorderchanges, but the request path is the same minus one local hop.Verification
TestModelRecorderBodyOutlivesResponseStartpasses, 50 of 50 runs. It fails withhttp: invalid Read on closed Bodyif the recorder forwards the unbuffered body.test-e2e (microvm)on this PR passed 3 of 3 attempts.