Skip to content

feat: Translate inference requests and replies in the forward proxy - #1934

Open
huang195 wants to merge 10 commits into
rossoctl:mainfrom
huang195:feat/translate-hub-1911-pP2
Open

huang195 wants to merge 10 commits into
rossoctl:mainfrom
huang195:feat/translate-hub-1911-pP2

Conversation

@huang195

@huang195 huang195 commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Part of #1911.

What changed

  • Pipeline calls (core/pipeline/destination.go). A WritesDestination plugin can now call pctx.Translate(source, target) and pctx.SetUpstreamHeaders(...). They have the same gates as SetRedirectPath.
  • Translation registry and golden tests (core/translate/). Request and Reply convert between the Anthropic Messages protocol and chat completions. They use the P1 encoders and decoders. Golden files cover each request rule and each reply shape (JSON, SSE, errors, Bob's three stream shapes).
  • Forward proxy (core/listener/forwardproxy/translate.go, server.go). When a plugin asks for a translation, the proxy converts the request after the request plugins and sets the fixed upstream headers. It converts the reply before the agent reads it, as JSON or as SSE. A JSON reply is converted whole before its status is sent, so an error in a 2xx body gets the status of that error. A 2xx reply over 10 MiB gets a 502 api_error. If it cannot convert the request, it refuses with a 502 api_error in the agent's protocol. A resend (Resender) gets the same conversion.
  • Capture (core/model/capture.go, core/session/archive/, core/sessionapi/). For a translated row, the capture keeps the converted request and the converted reply. It also keeps the server's raw reply in Upstream. The archive writes it, and GET .../raw returns it as upstream. agentop's raw view shows it as a "server's reply before translation" line.
  • .pre-commit-config.yaml. trailing-whitespace and end-of-file-fixer now skip \.golden$. This is not in the plan's T4 file list, but it is necessary. Without it, the fixers change the exact golden bytes, and the golden tests fail.
  • A forward-proxy deny invocation (translation_refused). The plan does not ask for it. It is necessary, because recordOutboundRejectIn records no row for a request that has no invocations. egressAllowRefusal does the same.

No router reads these calls yet. PR P3 adds the api, auth_header and headers server fields and the router code that calls Translate. The tests in this PR use a test plugin.

How it was tested

  • Unit tests: core/pipeline/translate_test.go, core/translate/golden_test.go, reader_test.go, registry_test.go.
  • Listener tests: core/listener/forwardproxy/translate_test.go sends requests through the proxy to a fake chat-completions server. It covers JSON and SSE replies, error replies, refusals, a resend, and the capture of each attempt.
  • Capture tests: core/session/archive/capture_test.go, core/sessionapi/archive_test.go, ingest_test.go, and cmd/agentop/tui/detail_raw_test.go.
  • The plan's gate passed on this branch's tip (go vet, gofmt -l, and the tests of each changed module).

Size

  • Counted lines: 2517 (added + removed).
  • Exempt lines: 220 (the .golden files that the golden tests' update flag writes).

The counted lines are over the 2,000-line limit by 517. The plan estimated 1,900. The listener tests (translate_test.go, 721 lines) are larger than the estimate. The golden test inputs (.json and .sse, about 170 lines) are not exempt, because they are inputs, not expected output. A split would put the capture work (T9) in a PR without the listener that fills it, so this PR keeps them together.

Review findings that were not fixed

All are minor.

  • T6, core/pipeline/destination.go SetUpstreamHeaders: a call with a nil or empty map is accepted and stores nil. Then UpstreamHeaders() returns nil, the same as when no call occurred. This does no harm, because the listener loops over the map. The doc comment does not say this.
  • T6, core/pipeline/destination.go: after a later Redirect by another WritesDestination plugin, the translate target and the upstream headers stay set and apply to the new host. SetRedirectPath behaves the same way, as the plan's "Gates as SetRedirectPath" says. No test covers a second Redirect after a Translate.
  • T4, core/translate/convert.go: Converted.Stream (set from req.Stream) has no test. The request goldens compare only Dropped and Body. T8 uses this field to select the reply encoder (decision 4). One assertion in TestRequestGolden on rule12_stream.json would cover it.
  • T4, core/translate/convert.go Reply: the fallback for a missing direction (a 502 api_error body in the source protocol, or nil for an unknown source) has no test. TestRequest_NoDirection covers only Request. One case, for example Reply(Chat, Anthropic, ...), would cover it.
  • T8: decision 19 is wired, but no test checks its output. In resendIfAsked (core/listener/forwardproxy/server.go), the refused attempt row gets pctx.ResponseBody = tr.replyBody(resp, head). TestTranslate_ResendSeesRawReply runs this line, but no assertion checks that the stored 403 row holds the Anthropic-shaped error. The reviewer checked the argument order against translate.Reply by hand, and it is correct.
  • T8: three small branches have no test: (a) the refusal for a nil pctx.Body (502 api_error "the request body was not buffered..."); (b) a Resender changes the request so that it cannot be converted, and the current answer goes back unchanged; (c) applyUpstreamHeaders runs again on the resent request.
  • T9, core/listener/forwardproxy/server.go: the rc creation moved before the response-phase plugins, so that wrapReply can fill rc.Upstream. As a result, the pctx.Extensions.Inference != nil gate is now checked before the response plugins run, not after. This agrees with recordCapturedRequest, which also keeps the request capture only for inference at request time. The reviewer found no regression. The commit and the code do not state the change in timing.
  • Refused attempt of a translated resend, resendIfAsked in core/listener/forwardproxy/server.go (near line 1340): pctx.ResponseCapture.Body is the raw chat head, and Upstream stays nil. The row for that attempt (pctx.ResponseBody) records the Anthropic form from tr.replyBody. The model.ResponseCapture doc says that for a translated reply, Body is the converted reply and Upstream is the raw reply. Because Upstream is nil here, agentop's raw view labels the chat bytes as the plain response, and it shows no "server's reply before translation" line. Decision 19 says only that the capture keeps the raw head. To agree with the field docs, put head in Upstream and the converted bytes in Body, or write in a comment that this case is an exception.
  • pipeline.Context.Translate (core/pipeline/destination.go:173) checks only that source and target are not empty and not equal. It does not check that the registry knows them. For an unknown source, startTranslation refuses the request, but translate.ErrorBody(src, ...) returns nil, so the 502 has BodyType application/json and an empty body. Today no caller can do this: the router comes in P3 and gives only known names. A check in Translate (translate.Lookup), or a fallback body in startTranslation, would prevent an empty 502.
  • Review fix rounds (7eef63e29, 920e4a986): the "over the limit" case in TestTranslate_NotStreamedErrorStatus still passes when the limit branch in convertWhole is removed, because the chat decoder's limit gives the same 502. A case with a non-2xx reply over 10 MiB would pin the branch.
  • Review fix round 920e4a986: the ttfbMs change (the time is taken before wrapReply) has no test.
  • Review fix round 7eef63e29: a translated reply over 10 MiB is a 502 row with no proxy_error kind and no warning log. bufferResponseBody gives both for an untranslated reply.
  • core/translate/chat/reply.go chunkError: an upstream type that is not an Anthropic type gives api_error, also when code gives a status. For example, LiteLLM's "type":"None","code":"429" should give rate_limit_error.
  • core/translate/chat/reply.go: a reply drops redacted_thinking blocks with no trace, but a request keeps them. (Reviewer suggestion.)
  • core/translate/chat/reply.go: strconv.Atoi(string(e.Code)) does not parse a string code such as "429", so the status is 500. (Reviewer suggestion.)
  • core/listener/forwardproxy/translate.go applyUpstreamHeaders: a plugin-set Host, Content-Length or Transfer-Encoding header has no effect, and the SetUpstreamHeaders doc does not say this. (Reviewer nit.)
  • core/translate/chat/request.go textParts: a document block in the top-level system is dropped and named in Dropped. In a user turn, the same block is refused with errDocument. The cache_control of a dropped system block is lost, and Dropped does not name it. Anthropic accepts only text blocks in system. (Delta-review suggestion.)
  • core/translate/pivot/pivot.go: the comment System []Block // text blocks is not true. The decoder keeps each block kind in System. (Delta-review nit.)

For a human after review

These steps need a live agent, a real server or a person's decision. They are not tasks in this workflow.

  1. Needs T1-T11 (all PRs). Run Claude Code (claude -p, with tools) against a chat-completions server through the router for at least 3 tool-call turns. A local Ollama or llama-server works when its api map has only chat-completions (agentop server add --api chat-completions=/v1/chat/completions …). Put the session id and its turn count in the PR P3 body.
  2. Needs T7. Check that Claude Code's /context still works with the local 404 for count_tokens (rule 6). If it does not, report what Claude Code does. Do not guess a token count.
  3. Needs T3, T8 (and feat: Router server URL path prefix, protocol probe at server add, and IBM Bob defaults #1912 for Bob). Check that Claude Code accepts a __thought__ tool id with + or / and sends it back unchanged. If Claude Code refuses or changes it, report what it does. Do not encode the id without a decision on that report.
  4. Needs T3. On Bob (feat: Router server URL path prefix, protocol probe at server add, and IBM Bob defaults #1912's list), check the rule 11 fields not yet tested there (tool_choice, parallel_tool_calls, stop, temperature, top_p), and that Bob accepts role: tool messages whose content is a list of text parts (decision 8).
  5. Needs P1-P3 opened. Post nothing outside the repository except the PRs themselves. Issue comments on feat: Translation hub with Anthropic Messages to and from chat completions, and the api, auth_header and headers server fields #1911 or feat: Inference translation hub, so any agent can run on any inference server (first: Claude Code on IBM Bob) #1910 are a person's call.

Assisted-By: Claude (Anthropic AI) noreply@anthropic.com

@huang195
huang195 requested a review from a team as a code owner November 17, 2026 21:20
@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Too many files!

This PR contains 109 files, which is 9 over the limit of 100.

To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch.

Upgrade to a paid plan to raise the limit.

This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: a10c2221-6e08-4280-a57c-49af82e27d7b
Commits

Reviewing files that changed from the base of the PR and between 02e489e and 47285e6.

Files selected for processing: 109

  • .pre-commit-config.yaml
  • cmd/agentop/tui/detail_raw.go
  • cmd/agentop/tui/detail_raw_test.go
  • core/listener/forwardproxy/server.go
  • core/listener/forwardproxy/translate.go
  • core/listener/forwardproxy/translate_test.go
  • core/model/capture.go
  • core/pipeline/context.go
  • core/pipeline/destination.go
  • core/pipeline/translate_test.go
  • core/session/archive/archive.go
  • core/session/archive/capture.go
  • core/session/archive/capture_test.go
  • core/sessionapi/archive_test.go
  • core/sessionapi/ingest.go
  • core/sessionapi/ingest_test.go
  • core/sessionapi/raw.go
  • core/translate/chat/reply.go
  • core/translate/chat/reply_test.go
  • core/translate/chat/request.go
  • core/translate/chat/request_test.go
  • core/translate/convert.go
  • core/translate/golden_test.go
  • core/translate/reader.go
  • core/translate/reader_test.go
  • core/translate/registry.go
  • core/translate/registry_test.go
  • core/translate/testdata/reply/bob_shape1.json
  • core/translate/testdata/reply/bob_shape1.json.to-json.golden
  • core/translate/testdata/reply/bob_shape1.sse
  • core/translate/testdata/reply/bob_shape1.sse.to-sse.golden
  • core/translate/testdata/reply/bob_shape2.json
  • core/translate/testdata/reply/bob_shape2.json.to-json.golden
  • core/translate/testdata/reply/bob_shape2.sse
  • core/translate/testdata/reply/bob_shape2.sse.to-sse.golden
  • core/translate/testdata/reply/bob_shape3.json
  • core/translate/testdata/reply/bob_shape3.json.to-json.golden
  • core/translate/testdata/reply/bob_shape3.sse
  • core/translate/testdata/reply/bob_shape3.sse.to-sse.golden
  • core/translate/testdata/reply/chunk_error.json
  • core/translate/testdata/reply/chunk_error.json.to-json.golden
  • core/translate/testdata/reply/chunk_error.sse
  • core/translate/testdata/reply/chunk_error.sse.to-sse.golden
  • core/translate/testdata/reply/error_403_litellm.json
  • core/translate/testdata/reply/error_403_litellm.json.to-json.golden
  • core/translate/testdata/reply/error_403_litellm.json.to-sse.golden
  • core/translate/testdata/reply/error_404_bob.json
  • core/translate/testdata/reply/error_404_bob.json.to-json.golden
  • core/translate/testdata/reply/error_404_bob.json.to-sse.golden
  • core/translate/testdata/reply/error_429_openai.json
  • core/translate/testdata/reply/error_429_openai.json.to-json.golden
  • core/translate/testdata/reply/error_429_openai.json.to-sse.golden
  • core/translate/testdata/reply/stream_two_tools.sse
  • core/translate/testdata/reply/stream_two_tools.sse.to-sse.golden
  • core/translate/testdata/reply/two_tools.json
  • core/translate/testdata/reply/two_tools.json.to-json.golden
  • core/translate/testdata/request/full_multiturn.golden
  • core/translate/testdata/request/full_multiturn.json
  • core/translate/testdata/request/rule11_unknown_fields.golden
  • core/translate/testdata/request/rule11_unknown_fields.json
  • core/translate/testdata/request/rule12_stream.golden
  • core/translate/testdata/request/rule12_stream.json
  • core/translate/testdata/request/rule13_system_blocks.golden
  • core/translate/testdata/request/rule13_system_blocks.json
  • core/translate/testdata/request/rule13_system_nontext.golden
  • core/translate/testdata/request/rule13_system_nontext.json
  • core/translate/testdata/request/rule13_system_only_nontext.golden
  • core/translate/testdata/request/rule13_system_only_nontext.json
  • core/translate/testdata/request/rule13_system_string.golden
  • core/translate/testdata/request/rule13_system_string.json
  • core/translate/testdata/request/rule13a_mid_system.golden
  • core/translate/testdata/request/rule13a_mid_system.json
  • core/translate/testdata/request/rule14_document.golden
  • core/translate/testdata/request/rule14_document.json
  • core/translate/testdata/request/rule14_image.golden
  • core/translate/testdata/request/rule14_image.json
  • core/translate/testdata/request/rule14_text.golden
  • core/translate/testdata/request/rule14_text.json
  • core/translate/testdata/request/rule14_tool_result.golden
  • core/translate/testdata/request/rule14_tool_result.json
  • core/translate/testdata/request/rule15_assistant.golden
  • core/translate/testdata/request/rule15_assistant.json
  • core/translate/testdata/request/rule16_server_tool.golden
  • core/translate/testdata/request/rule16_server_tool.json
  • core/translate/testdata/request/rule16_tools.golden
  • core/translate/testdata/request/rule16_tools.json
  • core/translate/testdata/request/rule17_any_no_parallel.golden
  • core/translate/testdata/request/rule17_any_no_parallel.json
  • core/translate/testdata/request/rule17_auto.golden
  • core/translate/testdata/request/rule17_auto.json
  • core/translate/testdata/request/rule17_none.golden
  • core/translate/testdata/request/rule17_none.json
  • core/translate/testdata/request/rule17_tool.golden
  • core/translate/testdata/request/rule17_tool.json
  • core/translate/testdata/request/rule18_sampling.golden
  • core/translate/testdata/request/rule18_sampling.json
  • core/translate/testdata/request/rule19_effort_low.golden
  • core/translate/testdata/request/rule19_effort_low.json
  • core/translate/testdata/request/rule19_effort_xhigh.golden
  • core/translate/testdata/request/rule19_effort_xhigh.json
  • core/translate/testdata/request/rule19_thinking_adaptive.golden
  • core/translate/testdata/request/rule19_thinking_adaptive.json
  • core/translate/testdata/request/rule19_thinking_disabled.golden
  • core/translate/testdata/request/rule19_thinking_disabled.json
  • core/translate/testdata/request/rule19_thinking_enabled.golden
  • core/translate/testdata/request/rule19_thinking_enabled.json
  • core/translate/testdata/request/rule20a_thought_id.golden
  • core/translate/testdata/request/rule20a_thought_id.json
  • docs/session-api.md

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@clawgenti clawgenti left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This review covers only the six commits after #1933 (318f2bc5a..fac202741): the Translate / SetUpstreamHeaders pipeline calls, the core/translate registry and golden tests, the forward-proxy conversion and resend wiring, and the capture of the translated request and the raw server reply. The listener wiring looks right to me. The conversion runs after every request plugin and before the request row, ownEncoding makes the transport decode gzip before conversion, the reply is wrapped before any response plugin or recorder reads it, and the resend path converts the changed request again. I ran the translate, pipeline, forwardproxy, sessionapi and agentop TUI tests on the head, and they pass. I found one blocking issue, in the ingest decoder (inline). The minor items your PR body lists already (no check of unknown protocols in Translate, nil Upstream for a 0-byte reply, the refused attempt's Upstream) are fair to defer, and I do not repeat them here.

Author: huang195 (MEMBER — maintainer)
Areas reviewed: Go (core/pipeline, core/translate, core/listener/forwardproxy, core/model, core/session/archive, core/sessionapi, cmd/agentop/tui), docs, pre-commit YAML
Agent/IDE config (.claude/.vscode): none
Commits: 6 commits in this PR's own range (14 including #1933), all signed-off: yes
CI status: passing

Comment thread core/sessionapi/ingest.go Outdated
Dropped bool `json:"dropped"`
Original []byte `json:"original"`
Forwarded []byte `json:"forwarded"`
Translated []byte `json:"translated"`

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

must-fix: The new typed fields refuse the whole ingest line when a producer sends the wrong JSON type. This covers translated here and upstream / upstreamTruncated in ingestResponse. On base these keys are unknown and ignored, so the same line is accepted.

I probed it with a test in the existing newIngestRig harness. It sent a request line with "translated": 123 and a response line with "upstream": 5, "upstreamTruncated": "x":

  • head fac2027: Lines:0 Refused:{bad_json:2}, 0 events stored
  • base d527246: Lines:2 Refused:{bad_json:0}, 2 events stored

A non-base64 string such as "translated": "{...}" fails the same way. The row is lost, not just its capture. This is the same class as the non-string source that 97f1a9d fixed with LenientString ("one key of the wrong type does not refuse the whole line"). Praxis (praxis/filters/src/record/line.rs) does not emit these keys yet, so the format is new now and the PR that makes praxis emit them is the one that can trip on this.

Suggested fix: decode these three fields leniently, for example with a small lenientBytes type (base64 string → bytes, anything else → nil) and a lenient bool, or with json.RawMessage and a best-effort decode. A bad value then drops the field and keeps the line. Please add a test like TestIngest_NonStringSourceKeepsTheRow that covers the three keys.

@clawgenti clawgenti left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Reviewed the forward-proxy translation hub: the pctx.Translate(source, target, path) plugin API, the Anthropic Messages ↔ chat-completions conversion through the protocol-neutral pivot form, the bounded 32 KiB-read replyReader streaming, and the session-archive recording of both the translated request and the pre-translation upstream reply. The engineering here is very solid — path validation (no query/fragment/..), refusal bodies in the agent's own protocol, resend re-conversion, Content-Length/Encoding stripped on rewritten replies, non-stream tool-input JSON validated before marshal, and extensive golden + integration tests. No must-fix findings; one suggestion and one nit below.

Author: huang195 (MEMBER — maintainer)
Areas reviewed: Go (forwardproxy, translate/pivot/anthropic/chat, pipeline, sessionapi, session/archive, agentop TUI), docs, pre-commit config, testdata
Agent/IDE config (.claude/.vscode): none
Commits: 14 commits, all signed-off: yes
CI status: passing

Generated by RepoMan github-pr-review@9672454 (https://github.com/rossoctl/agent-skills/blob/96724547c162978f58d845b9a8331a679bf39444/skills/github-pr-review/SKILL.md).

return nil
}
d.buf = append(d.buf, p...)
if !d.stream {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion — In Feed, the non-2xx path caps d.buf at maxErrorBody (1 MiB), but the 2xx paths have no cap: a non-stream reply accumulates the whole body in d.buf until End(), and a stream whose data: line never terminates grows d.buf across every Feed (each 32 KiB read appends; only complete lines are trimmed). A misbehaving or malicious inference upstream can therefore balloon proxy memory on exactly the replies the proxy is trusted to relay.

Suggested fix: cap accumulation in the 2xx paths the same way (e.g. fail with a 502 api_error past maxErrorBody or MaxResponseCapture), so the asymmetry with the error path is gone. Why it matters now: this is the forward proxy's untrusted-input boundary. (一句话:上游再热情,缓冲也要有底线 😄)

func applyUpstreamHeaders(r *http.Request, h http.Header) {
for k, v := range h {
r.Header[http.CanonicalHeaderKey(k)] = append([]string(nil), v...)
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit — applyUpstreamHeaders writes directly into r.Header, so plugin-supplied Host, Content-Length and Transfer-Encoding are silently ignored by Go's client transport (they come from r.Host/req.ContentLength, and Header copies of the latter two are excluded on write). Worth a doc line on SetUpstreamHeaders (or an explicit rejection) so plugin authors don't set headers that quietly do nothing. (小提醒:有些 header 只是看起来上车了 😄)

@clawgenti clawgenti left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This review covers the two commits since my last review (f72e755, ac583b7) and checks again the items still open from it. Both new commits do what they say. A not-streamed translated reply is now converted whole before its status goes out, so an error in a 2xx body gets its own status. The chat decoder caps a 2xx body and a stream line at 10 MiB. Upstream is non-nil for an empty reply, and ttfbMs is taken before wrapReply. I probed the main case of ac583b7 through the proxy: a 2xx JSON reply over 10 MiB to a stream request gives the agent a 200 SSE event: error with the over-limit api_error. This also covers my earlier suggestion on chat/reply.go:84.

One must-fix from the last review is still open, in the ingest decoder (inline). I ran the same probe again on ac583b7: a request line with "translated": 123 and a response line with "upstream": 5, "upstreamTruncated": "x" store 0 events at head and 2 at base d527246.

Also, before merge: #1933 is merged now, and main has da98da43 (tool_use stop reason, not-streamed tool ids), so the branch conflicts with main in core/translate/chat/reply_test.go. Only the tests conflict, and both sides keep their tests. I resolved the conflict locally by keeping both sides. Then go vet and the tests of ./translate/... and ./listener/forwardproxy/ pass on that merge, and the goldens still match. Please rebase on main, so that the diff shows only this PR.

Tests on ac583b7: translate/..., pipeline, listener/forwardproxy, sessionapi and model pass, and gofmt -l is clean. In session/archive, TestWriteSession_FailedEraseKeepsNumbering fails, but it fails the same way on base d527246. It looks like an environment problem in my sandbox, because I run as root, and it is not caused by this PR.

Author: huang195 (MEMBER — maintainer)
Areas reviewed: Go (core/listener/forwardproxy, core/translate/chat, core/translate convert, core/sessionapi ingest), merge with current main
Agent/IDE config (.claude/.vscode): none
Commits: 16 commits (8 in this PR's own range), all signed-off: yes
CI status: DCO passing (CodeRabbit skipped: too many files)

Comment thread core/sessionapi/ingest.go Outdated
Dropped bool `json:"dropped"`
Original []byte `json:"original"`
Forwarded []byte `json:"forwarded"`
Translated []byte `json:"translated"`

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

must-fix (still open from the last review, see the thread above at fac2027): Translated, and Upstream / UpstreamTruncated in ingestResponse, are still strict typed fields. If one of these keys has the wrong JSON type, decodeIngestLine refuses the whole line as bad_json. Base ignores these keys, so it accepts the same line.

I ran the probe again on ac583b7 with newIngestRig. It sent a request line with "translated": 123 and a response line with "upstream": 5, "upstreamTruncated": "x". Head stored 0 events, and base d527246 stored 2. The fix is the same as before. Decode these three fields leniently, for example with a lenientBytes type (a base64 string gives bytes, any other value gives nil) and a lenient bool, as LenientString does for source. Then add a test like TestIngest_NonStringSourceKeepsTheRow that covers the three keys.

@clawgenti clawgenti left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

你好呀!👋 Long time no see — and you show up carrying an entire translation hub. 好家伙,这哪是 PR,这是搬了一座翻译局来 😄

Credit first: the pivot design is the right call. A protocol-neutral form with per-protocol decoders/encoders and Extra stashing is exactly what keeps this from becoming an N² matrix of pairwise converters, and the determinism work (rule 9a — stable bytes so a moved cache marker changes only the marker's bytes, pinned by TestDeterministic) shows real care for prompt-cache hit rates. The byte-split golden tests (TestReplyReader_EveryByteSplit, decodeBytewise) are the tests this kind of streaming decoder actually needs, and the resend/capture plumbing (raw reply to the Resender, per-attempt Translated bodies, four-body capture for debugging conversions from the archive) is thought through end to end. Docs plan with numbered rules that the code comments cite back to is a lovely touch.

No must-fix findings. Two suggestions below; everything else checked clean — DCO on all 16 commits, no .claude//.vscode/ changes, no new dependencies, and the forward-proxy tests cover the refusal, resend, gzip, error-status and capture paths.

Author: huang195 (MEMBER — maintainer)
Areas reviewed: Go (translate pivot/codecs, forward proxy, pipeline, session archive/api), docs, pre-commit config
Agent/IDE config (.claude/.vscode): none
Commits: 16 commits, all signed-off: yes
CI status: DCO passing (CodeRabbit skipping)

Generated by RepoMan github-pr-review@9672454 (https://github.com/rossoctl/agent-skills/blob/96724547c162978f58d845b9a8331a679bf39444/skills/github-pr-review/SKILL.md).

// shortcut: a redacted_thinking entry is not sent, because the
// pivot events have no field for its data; add one when a
// server sends it.
if tb.Type != "thinking" {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion — The reply path drops redacted_thinking blocks (here, and the unsigned-entry skip at the streamDelta thinking_blocks loop), while the request path deliberately carries them through (BlockRedactedThinking → LiteLLM thinking_blocks redacted_thinking with Data). That asymmetry means a round trip loses the block: a server that emits a redacted-thinking block in its reply produces an Anthropic reply with no corresponding block, and the agent's next request no longer carries what Anthropic's API expects to see passed back verbatim. The shortcut comment says to add it when a server sends one — fair — but a server that does will silently lose data with no trace. Suggested fix: either give the pivot reply events a field for it now (the request side already models Data), or at least surface it the way the request encoder does — as a named entry in a dropped-style record the operator can see, rather than a silent continue. 慢工出细活,就差这最后一块拼图啦 🙂

// chunkError converts an {"error":…} chunk or body at status 2xx.
func (d *decoder) chunkError(e *chatError) {
status := http.StatusInternalServerError
if n, err := strconv.Atoi(string(e.Code)); err == nil && n >= 400 && n < 600 {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion — strconv.Atoi(string(e.Code)) only parses a JSON number: OpenAI- and LiteLLM-shaped error bodies frequently send "code": "429" as a string, and string(json.RawMessage) of "429" is "\"429\"" — Atoi fails on the quotes and the chunk falls back to status 500 with (usually) a rate-limit message attached, so a client can't back off correctly. Suggested fix: unquote first, e.g. try json.Unmarshal(e.Code, &s) into a string when the number parse fails, then Atoi(s) — a two-line change that keeps both shapes working. 小问题,不碍事 — the rest of the error mapping is spot on.

@clawgenti clawgenti left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add Translate and TranslateTarget. A routing plugin uses them to ask the
listener to convert a redirected request to another protocol and to send
it to a new path. Translate does not change Path and records nothing.

Add SetUpstreamHeaders and UpstreamHeaders. The listener sets these
headers after all plugins run. The call keeps a clone of the headers.

All the new calls use the gates of SetRedirectPath. SetRedirectPath now
uses the same gate and path check helpers.

Part of rossoctl#1911.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
Add the compiled-in protocol registry (rule H5) in core/translate:
Lookup, Names, Decodes and Target (decision 12). The responses
protocol is registered with only Name and EncodeError.

Add the conversion helpers: Request, Reply and ErrorBody. Add
NewReplyReader, which converts a reply as it reads it and does not
wait for the whole stream (rule 10).

Add golden tests from hand-written inputs: one request for each rule
from 11 to 19 and for rule 20a, a full multi-turn request, the three
Bob reply shapes, two tool calls, and the error replies, streamed and
not streamed. A test splits each reply at every byte. TestDeterministic
checks rule 9a.

Exclude golden files from the whitespace fixers of pre-commit. They
hold exact output bytes, and an SSE reply ends in a blank line.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
When a plugin calls pctx.Translate, the forward proxy now converts the
request after the last request plugin and converts the reply as it
reads it, before a reply plugin or a recorder reads it.

- The request goes to the target path with no query string. The
  request row keeps the path of the client.
- The proxy records modify/translated, with "dropped" when the
  target cannot carry some fields.
- A refused conversion gives an error in the protocol of the agent.
  Nothing goes upstream.
- The headers from SetUpstreamHeaders replace the headers of the
  same name, also on a resend.
- The transport decodes gzip for a translated request.
- A Resender reads the raw answer. The proxy converts the changed
  request again before it sends it.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
For a translated request, the lossless capture now keeps four bodies
(rule 8a of rossoctl#1911), so you can find a conversion bug from the archive:

- RequestCapture.Translated is the converted body as sent upstream.
- ResponseCapture.Upstream is the server's reply before the conversion.
  Body stays the converted reply, as the agent got it.

The archive, GET raw (translated, upstream and their Base64 forms) and
the ingest line carry both fields. The archive counts them in the
capture queue's size. agentop's raw view shows them with their own
labels, not as a plugin rewrite. A resend records the new converted
body on a copy of the capture.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
TestTranslate_ResendSeesRawReply now sets a substitute model on the
resend. The test makes sure that the refused row keeps the chat body
it sent and that the resent row has the new converted body.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
Add tests for each resend branch of a translated request:
- The row of the refused attempt has the refusal in Anthropic form
  (decision 19).
- The upstream headers are set again on the resent request.
- A changed request that cannot be converted is not sent again. The
  agent gets the refusal.
- A refused translation records a denied row.

When the changed request cannot be converted, the answer now goes back
with the inference parse of the request that was sent. Before, the row
named the model that the Resender wrote, which the upstream did not get.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
Fixes review: a not-streamed 2xx reply that converts to an error reached the agent as HTTP 200
Fixes review: the chat decoder buffer and the not-streamed reply had no size limit
Fixes review: blocks of unknown kinds were dropped with no record in Dropped
Fixes review: a 0-byte translated reply left the capture's Upstream nil
Files:
- core/listener/forwardproxy/translate.go
- core/listener/forwardproxy/translate_test.go
- core/translate/chat/reply.go
- core/translate/chat/reply_test.go
- core/translate/chat/request.go
- core/translate/chat/request_test.go
- core/translate/convert.go
- core/translate/golden_test.go

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
Fixes review: a 2xx JSON reply to a stream request had no buffer limit in the chat decoder
Fixes review: the limit check ran again after the decoder had stopped
Fixes review: the listener's over-limit test did not reach the limit branch
Fixes review: ttfbMs of a not-streamed translated reply included the time to read the body
Files:
- core/listener/forwardproxy/server.go
- core/listener/forwardproxy/translate_test.go
- core/translate/chat/reply.go
- core/translate/chat/reply_test.go
- core/translate/chat/request.go
- core/translate/convert.go

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
…line

Fixes review: a wrong JSON type in `translated`, `upstream` or
`upstreamTruncated` refused the whole ingest line as bad_json.
Files:
- core/sessionapi/ingest.go
- core/sessionapi/ingest_test.go

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>

@clawgenti clawgenti left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This review covers the commit since my last review (eeec211) and the rebase on main. It also checks again the must-fix from my two earlier reviews.

Earlier must-fixes

  • Resolved: the ingest decoder refused a whole line when translated, upstream or upstreamTruncated had the wrong JSON type (core/sessionapi/ingest.go). lenientBytes and lenientBool now drop the key and keep the line. I ran my earlier probe again on eeec211, a request line with "translated": 123 and a response line with "upstream": 5, "upstreamTruncated": "x". It stores 2 events and both captures. Arrays, objects and null behave the same way. TestIngest_BadTranslatedHopKeysKeepTheRow covers the three keys.
  • Resolved: the rebase. The branch now sits on main (02e489e9, the #1933 merge), and the diff shows only this PR's 9 commits. In the range-diff against ac583b7, 7eef63e and 920e4a9 differ only in reply_test.go context.

New in this round: one must-fix and one suggestion (inline). The must-fix is in a path next to a fix that 7eef63e claims. That commit says it fixes "blocks of unknown kinds were dropped with no record in Dropped", and that is true for user and assistant turns and for tool results. A top-level system array goes through textParts instead, and there any block that is not text turns into an empty text part with no record. I do not repeat the minor items that your PR body already lists as not fixed.

Tests on eeec211: go vet and gofmt -l are clean. translate/..., pipeline, listener/forwardproxy, sessionapi, model and cmd/agentop/tui pass, and the first four also pass with -race. In session/archive, TestWriteSession_FailedEraseKeepsNumbering fails, but it fails the same way on main. I run as root in my sandbox, and this PR does not cause it.

Author: huang195 (MEMBER — maintainer)
Areas reviewed: Go (core/sessionapi ingest, core/translate/chat request and reply, core/listener/forwardproxy translate), rebase range-diff
Agent/IDE config (.claude/.vscode): none
Commits: 9 commits, all signed-off: yes
CI status: passing (Rust praxis, agentop and README demo jobs still pending; CodeRabbit skipped)

}

// encoder collects the names of the dropped fields (decision 11).
// encoder collects the names of the dropped fields and blocks (decision 11).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

must-fix: 7eef63e says it fixes "blocks of unknown kinds were dropped with no record in Dropped", and this doc line now says the encoder collects dropped blocks. A top-level system array does not go through user(), though. It goes through textParts (line 174), which makes every block a text part from b.Text and does not look at b.Kind. So any block in system that is not text is sent as an empty text part, and nothing goes into Dropped.

I probed translate.Request(Anthropic, Chat, …) on eeec211:

  • system: [text "hi", image (base64), document (text)] gives {"role":"system","content":[{"type":"text","text":"hi"},{"type":"text","text":""},{"type":"text","text":""}]} and dropped=[]. In a user turn, the same document is refused with errDocument, and the image is sent as an image_url part.
  • system: [text, search_result{…}] gives an empty text part, and dropped=[content source title]. These are the block's extra fields, and search_result itself is not named. The same block in a user turn, or in a mid-conversation system message (rule 13a, through user()), records search_result.

Anthropic accepts only text blocks in system, so this input is off-spec. But the claimed fix does not cover this case, and the result is an empty part that the operator cannot see. Suggested fix: in textParts, record e.dropped[string(b.Kind)] and skip every block that is not pivot.BlockText, or reuse the user() rules (a document refuses, any other kind is dropped). Then add a request golden like rule13_system_blocks with an image block and an unknown block in system.

{"error body at 200", replyFixture(t, "chunk_error.json"), http.StatusInternalServerError},
{"not JSON", "not json", http.StatusBadGateway},
// Valid JSON: only the limit makes it an error.
{"over the limit", replyFixture(t, "bob_shape1.json") + strings.Repeat(" ", maxBodySize), http.StatusBadGateway},

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: The comment says "only the limit makes it an error", but this case still passes when the len(raw) > maxBodySize branch in convertWhole is removed. The chat decoder's own maxReplyBuffer gives the same 502 api_error. Your PR body lists this. I checked that the case it proposes pins the branch: a 500 reply whose body is {"error":{"message":"boom","type":"server_error"}} plus maxBodySize spaces gives a 502 at head. With the branch removed, it gives 500 "boom", because the non-2xx decoder keeps only 1 MiB and stops there, and the test fails. Please add it as a fourth case, and say in the comment that the 2xx case passes through both limits.

Fixes review: a system block that is not text was sent as an empty text part, with no record in Dropped.
Files:
- core/translate/chat/request.go
- core/translate/testdata/request/rule13_system_nontext.{json,golden}
- core/translate/testdata/request/rule13_system_only_nontext.{json,golden}

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-ai-review Request automated AI code review from clawgenti

Projects

Status: New/ToDo

Development

Successfully merging this pull request may close these issues.

2 participants