Skip to content

test(e2e): pass mock LLM handlers to startModelRecorder - #3264

Merged
EItanya merged 1 commit into
mainfrom
danielle-mustillo/e2e-egress-full-duplex
Oct 19, 2026
Merged

EItanya merged 1 commit into
mainfrom
danielle-mustillo/e2e-egress-full-duplex

Conversation

@danielle-mustillo

@danielle-mustillo danielle-mustillo commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Why

The test-e2e (microvm) job fails at random on egress tests with unexpected EOF (job, job). The egress and mounted-skills mocks put a second server with an unbuffered httputil.ReverseProxy behind startModelRecorder. Once the streamed response starts, that server closes the request body. The proxy transport then reads the body once more past Content-Length, gets ErrBodyReadAfterClose, and closes the upstream connection mid-stream.

What changed

  • startModelRecorder takes an http.Handler and calls it in process with the buffered body, so no proxy reads a body that the server can close.
  • The egress and mounted-skills mocks pass their routing handler directly and lose their second server. Other callers wrap the mock URL with mockLLMProxy.
  • TestModelRecorderBodyOutlivesResponseStart checks that the upstream handler can read the body after the response starts.

Scope

  • Fixes the use of closed network connection and unexpected EOF flake only.
  • Does not fix the separate live fixed did not become ALLOWED flake in the claude variant.

Tradeoffs

  • Calling EnableFullDuplex in each mock also fixes the race, but each new mock has to remember it. Removing the second hop removes the bug class.

Blast Radius

Test code only. Every e2e test that uses startModelRecorder changes, but the request path is the same minus one local hop.

Verification

  • TestModelRecorderBodyOutlivesResponseStart passes, 50 of 50 runs. It fails with http: invalid Read on closed Body if the recorder forwards the unbuffered body.
  • A local repro of the old chain with a 50 ms delay before body EOF truncated 200 of 200 responses.
  • test-e2e (microvm) on this PR passed 3 of 3 attempts.

@github-actions github-actions Bot added the testing Additional testing required label Oct 19, 2026
The egress and mounted-skills mocks put a second server with an
unbuffered httputil.ReverseProxy behind startModelRecorder. Once the
streamed response starts, that server closes the request body. The proxy
transport then reads the body once more, gets ErrBodyReadAfterClose, and
closes the upstream connection mid-stream. The agent sees unexpected EOF,
and the egress test times out.

startModelRecorder now takes an http.Handler and calls it in process with
the buffered body. The two mocks lose their second server, and the other
callers wrap the mock URL with mockLLMProxy.

Signed-off-by: Danielle Mustillo <danielle.mustillo@solo.io>
@danielle-mustillo
danielle-mustillo force-pushed the danielle-mustillo/e2e-egress-full-duplex branch from ed6e860 to d465730 Compare October 19, 2026 07:05
@danielle-mustillo danielle-mustillo changed the title test(e2e): enable full duplex in the egress skill mock proxy test(e2e): pass mock LLM handlers to startModelRecorder Oct 19, 2026
@github-actions github-actions Bot added testing Additional testing required and removed testing Additional testing required labels Oct 19, 2026
@danielle-mustillo
danielle-mustillo marked this pull request as ready for review October 19, 2026 07:47
@danielle-mustillo
danielle-mustillo requested a review from a team as a code owner October 19, 2026 07:47
@github-actions github-actions Bot added testing Additional testing required and removed testing Additional testing required labels Oct 19, 2026
@EItanya
EItanya added this pull request to the merge queue Oct 19, 2026
Merged via the queue into main with commit 26292bf Oct 19, 2026
89 checks passed
@EItanya
EItanya deleted the danielle-mustillo/e2e-egress-full-duplex branch October 19, 2026 13:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing Additional testing required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants