Repository navigation
Conversation
|
Important Review skippedToo many files! This PR contains 129 files, which is 29 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry.
|
clawgenti
left a comment
There was a problem hiding this comment.
Summary
I reviewed the five commits of this PR (fb23601e2..369b0d2fe): the api, auth_header and headers server fields and their checks, the router's routing plan (rules 4 to 7, C1 to C3), inference.ProtocolFor, the exact-name headers redaction, the agentop server add flags and listing, and the docs. The routing logic matches the issue. The plan runs only after a redirect takes effect, a translated request leaves pctx.Path alone so inference-parser keeps the Anthropic dialect, the refusal and count_tokens answers settle the pin and send nothing upstream, and setAuth follows the C2 table. I ran the inferencerouter, routerconfig, parse/inference, redact and forwardproxy tests on 369b0d2, and they pass.
One blocking item: this branch still carries the ingest decoder issue from #1934 (inline). It goes away when #1934's fix lands and this branch is rebased on it. I add two suggestions found in this PR's own code. I do not repeat the deferred items that the PR body already lists (the serverList index panic, the case-only duplicate header names, the $ check on the other flags, and the others).
Author: huang195 (MEMBER — maintainer)
Areas reviewed: Go (core/plugins/inferencerouter, routerconfig, core/parse/inference, core/redact, core/listener/forwardproxy tests, cmd/agentop), docs
Agent/IDE config (.claude/.vscode): none
Commits: 5 commits in this PR's own range (19 including #1933 and #1934), all signed-off: yes
CI status: passing (Verify Action Pinning pending)
| Dropped bool `json:"dropped"` | ||
| Original []byte `json:"original"` | ||
| Forwarded []byte `json:"forwarded"` | ||
| Translated []byte `json:"translated"` |
There was a problem hiding this comment.
must-fix (carried from #1934, not fixed on this branch): the new typed fields translated here and upstream / upstreamTruncated in ingestResponse make the whole ingest line fail when a producer sends the wrong JSON type. On base these keys are unknown, so they are ignored and the line is accepted. My probe on #1934's head showed Lines:0 Refused:{bad_json:2} for a request line with "translated": 123 and a response line with "upstream": 5, "upstreamTruncated": "x". Base gave Lines:2 Refused:{bad_json:0}. This file is the same at 369b0d2, so merging this PR into main ships the regression. See the review thread on #1934 for the fix (for example json.RawMessage fields that are decoded one at a time, and that drop only the bad field). After #1934 is fixed, rebase this branch on it.
| - `helper` (string, required) — the same for a request without tools, such as a title. | ||
| - `api` (map) — protocols the server serves, by protocol name, to the path it serves each at. Empty: requests go in the agent's protocol, to the agent's path. Valid keys: `anthropic-messages`, `chat-completions` and `responses`. Each path must be absolute, with no query, fragment or `..` segment. See **Servers that serve another protocol**. | ||
| - `auth_header` (string) — header that carries `key`. `authorization` sends `Bearer <key>`; any other name sends the raw key. Empty: the agent's header, or the target protocol's usual one for a translated request. | ||
| - `headers` (map) — fixed headers set on every request to this server after the plugins run. Values may use `${ENV_VAR}`. |
There was a problem hiding this comment.
suggestion: tell users to quote headers values that are written by hand. The YAML block goes through yamlNodeToJSON (n.Decode(&any)) before routerconfig.Decode reads it, so YAML picks the scalar type. I probed it on 369b0d2 with config.PluginEntry → routerconfig.Decode:
X-Api-Version: 2andX-Flag: true→json: cannot unmarshal number/bool into Go struct field Server.servers.headers of type string. The whole router config is refused at load.Anthropic-Version: 2023-06-01→ accepted, but the value becomes"2023-06-01T00:00:00Z", and that string goes upstream without any warning.
agentop server add --header is safe, because ScalarValue tags each value !!str and the values are written quoted. The ${ENV_VAR} hand-edit route that this line recommends is not safe. As a minimum, add "quote each value (YAML reads 2023-06-01 as a date)" here and in the Headers field description. A refusal of a non-string scalar cannot catch the date case, because the conversion happens before the router decodes the config.
| return fmt.Errorf("api: unknown protocol %q; valid protocols: %s", k, strings.Join(APIProtocols, ", ")) | ||
| } | ||
| p := api[k] | ||
| if !strings.HasPrefix(p, "/") || strings.ContainsAny(p, "?#") || slices.Contains(strings.Split(p, "/"), "..") { |
There was a problem hiding this comment.
suggestion: CheckAPI does not check that a path belongs to its key's protocol. For planPath, SetRedirectPath also sets pctx.Path, and inference-parser picks the reply dialect from pctx.Path (endpointPath → ReadResponse/NewStream). An entry such as anthropic-messages: /anthropic/messages (no /v1/messages ending) is accepted. The parser then reads the Anthropic reply as OpenAI chat, so the session row loses its usage, cost and content. The request itself still works, so nothing tells the user. You already have the rule in inference.ProtocolFor, but routerconfig stays free of core/parse. Two cheap options: copy the three suffix checks here (ProtocolFor(path) == key, held equal by a test the way APIProtocols is), or write in the docs that the path must end as the protocol's usual path does.
Add Translate and TranslateTarget. A routing plugin uses them to ask the listener to convert a redirected request to another protocol and to send it to a new path. Translate does not change Path and records nothing. Add SetUpstreamHeaders and UpstreamHeaders. The listener sets these headers after all plugins run. The call keeps a clone of the headers. All the new calls use the gates of SetRedirectPath. SetRedirectPath now uses the same gate and path check helpers. Part of rossoctl#1911. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>
Add the compiled-in protocol registry (rule H5) in core/translate: Lookup, Names, Decodes and Target (decision 12). The responses protocol is registered with only Name and EncodeError. Add the conversion helpers: Request, Reply and ErrorBody. Add NewReplyReader, which converts a reply as it reads it and does not wait for the whole stream (rule 10). Add golden tests from hand-written inputs: one request for each rule from 11 to 19 and for rule 20a, a full multi-turn request, the three Bob reply shapes, two tool calls, and the error replies, streamed and not streamed. A test splits each reply at every byte. TestDeterministic checks rule 9a. Exclude golden files from the whitespace fixers of pre-commit. They hold exact output bytes, and an SSE reply ends in a blank line. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>
When a plugin calls pctx.Translate, the forward proxy now converts the request after the last request plugin and converts the reply as it reads it, before a reply plugin or a recorder reads it. - The request goes to the target path with no query string. The request row keeps the path of the client. - The proxy records modify/translated, with "dropped" when the target cannot carry some fields. - A refused conversion gives an error in the protocol of the agent. Nothing goes upstream. - The headers from SetUpstreamHeaders replace the headers of the same name, also on a resend. - The transport decodes gzip for a translated request. - A Resender reads the raw answer. The proxy converts the changed request again before it sends it. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>
For a translated request, the lossless capture now keeps four bodies (rule 8a of rossoctl#1911), so you can find a conversion bug from the archive: - RequestCapture.Translated is the converted body as sent upstream. - ResponseCapture.Upstream is the server's reply before the conversion. Body stays the converted reply, as the agent got it. The archive, GET raw (translated, upstream and their Base64 forms) and the ingest line carry both fields. The archive counts them in the capture queue's size. agentop's raw view shows them with their own labels, not as a plugin rewrite. A resend records the new converted body on a copy of the capture. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>
TestTranslate_ResendSeesRawReply now sets a substitute model on the resend. The test makes sure that the refused row keeps the chat body it sent and that the resent row has the new converted body. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>
Add tests for each resend branch of a translated request: - The row of the refused attempt has the refusal in Anthropic form (decision 19). - The upstream headers are set again on the resent request. - A changed request that cannot be converted is not sent again. The agent gets the refusal. - A refused translation records a denied row. When the changed request cannot be converted, the answer now goes back with the inference parse of the request that was sent. Before, the row named the model that the Resender wrote, which the upstream did not get. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>
Fixes review: a not-streamed 2xx reply that converts to an error reached the agent as HTTP 200 Fixes review: the chat decoder buffer and the not-streamed reply had no size limit Fixes review: blocks of unknown kinds were dropped with no record in Dropped Fixes review: a 0-byte translated reply left the capture's Upstream nil Files: - core/listener/forwardproxy/translate.go - core/listener/forwardproxy/translate_test.go - core/translate/chat/reply.go - core/translate/chat/reply_test.go - core/translate/chat/request.go - core/translate/chat/request_test.go - core/translate/convert.go - core/translate/golden_test.go Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>
Fixes review: a 2xx JSON reply to a stream request had no buffer limit in the chat decoder Fixes review: the limit check ran again after the decoder had stopped Fixes review: the listener's over-limit test did not reach the limit branch Fixes review: ttfbMs of a not-streamed translated reply included the time to read the body Files: - core/listener/forwardproxy/server.go - core/listener/forwardproxy/translate_test.go - core/translate/chat/reply.go - core/translate/chat/reply_test.go - core/translate/chat/request.go - core/translate/convert.go Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>
…line Fixes review: a wrong JSON type in `translated`, `upstream` or `upstreamTruncated` refused the whole ingest line as bad_json. Files: - core/sessionapi/ingest.go - core/sessionapi/ingest_test.go Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>
369b0d2 to
f0dd424
Compare
clawgenti
left a comment
There was a problem hiding this comment.
Summary
I reviewed again after the rebase on #1934's head (eeec21132). git range-diff shows that the five commits of this PR (1146c0c56..f0dd424cf) are unchanged from 369b0d2fe, which I reviewed before. The only new code is #1934's three fix commits: 7eef63e2 and 920e4a98 (the same as f72e7556 and ac583b70, rebased), and eeec2113, which fixes the ingest decoder.
The must-fix from my last review is resolved. translated, upstream and upstreamTruncated now decode through lenientBytes / lenientBool, so a wrong JSON type drops only that key and keeps the line. I confirmed this against both commits. The new TestIngest_BadTranslatedHopKeysKeepTheRow fails on 369b0d2 (stored 0 events, want 2 for each of 123, "{not base64}", an object and true) and passes on f0dd424, so base behavior is restored for those keys. Nothing else in the diff decodes producer input with new typed fields: RawCapture/RawBody are only encoded, the archive records are the archive's own format, and routerconfig.Decode refuses unknown fields on purpose.
On f0dd424, go vet and the tests of sessionapi, inferencerouter/..., parse/inference, redact, listener/forwardproxy, translate/..., pipeline and model pass. My two earlier suggestions still apply: quote hand-written headers values in YAML (docs/inference-router-plugin.md:22), and check that each api path matches its key's protocol (routerconfig/config.go:68). Neither one blocks this PR. Approving. It should merge after #1934, so that the stacked commits land once.
Author: huang195 (MEMBER — maintainer)
Areas reviewed: Go (core/sessionapi ingest decoder, core/plugins/inferencerouter, routerconfig, core/parse/inference, core/redact, core/listener/forwardproxy, cmd/agentop), docs, pre-commit YAML
Agent/IDE config (.claude/.vscode): none
Commits: 14 commits (5 in this PR's own range, 9 from #1934), all signed-off: yes
CI status: passing
Fixes review: a system block that is not text was sent as an empty text part, with no record in Dropped.
Files:
- core/translate/chat/request.go
- core/translate/testdata/request/rule13_system_nontext.{json,golden}
- core/translate/testdata/request/rule13_system_only_nontext.{json,golden}
Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
Add docs/translation-server.md (rule F1): how to run LiteLLM on loopback for a protocol pair that Cortex cannot translate, and how to add it with agentop server add. Add the api, auth_header and headers fields to the inference router page, and a section that gives the routing rules, the auth header table, the refused header names and the modify/translated record. Add an index row in docs/README.md. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>
Add three optional fields to an inference router server: - api: the protocols that the server serves, each to its path. A key must be a known protocol. A path must be absolute, with no query, fragment or ".." segment. - auth_header: the header that carries the key. Host, Content-Length and the hop-by-hop headers are refused. - headers: fixed headers for each request to the server. The auth headers, the server's own auth_header, Host, Content-Length and the hop-by-hop headers are refused. A value with CR, LF or NUL is refused. redact.JSON now hides each value under a field named exactly "headers" and keeps the names. Fields such as id_headers do not change. Refs rossoctl#1911 Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>
routerconfig.Server now holds maps (api, headers), so it is not comparable and maps.Equal does not compile on it. sameRouter now compares the servers with reflect.DeepEqual. Refs rossoctl#1911 Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>
agentop server add now writes the api, auth_header and headers server fields (rule C4). It checks each flag with the routerconfig checks before it asks for the key. It refuses a header value with $, because config.Load expands it. agentop server shows the api, the auth header and the header names under each server. It does not show header values. Refs rossoctl#1911 Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>
The inference router now reads the api, auth_header and headers fields of a server (rules 4 to 7 and C1 to C3 of rossoctl#1911): - When the protocol of the agent is a key of api, the router sends the request to the path of that key. - When the protocol of the agent can be decoded and is not a key, the router asks the listener to translate the request to the first protocol of the map that has an encoder. It does not change the path. - When no protocol of the map has an encoder, the router refuses the request with a 502 in the protocol of the agent. - count_tokens to a server with no anthropic-messages entry gets a local 404 not_found_error. - setAuth puts the key in auth_header, or in the usual header of the target protocol for a translated request, and removes the other auth headers of the agent. Otherwise setKey applies as before. - The fixed headers go to SetUpstreamHeaders. inference.ProtocolFor gives the protocol of a path with the rule of dialectFor, so that the parser and the router cannot disagree. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>
f0dd424 to
81494c5
Compare
This PR stacks on #1934 (#1934), PR P2 of 3. The branch
feat/translate-hub-1911-pP2is in the fork only, so GitHub cannot use it as the base here. The base ismain. Until #1934 merges, the diff below also shows the P2 commits. To see only this PR, read the commits afterfeat/translate-hub-1911-pP2(5 commits, from1146c0c56tof0dd424cf).Fixes #1911.
What changed
core/plugins/inferencerouter/routerconfig/config.go). An inference router server has three new optional fields:api: the protocols that the server serves, each with its path. A key must be a known protocol (anthropic-messages,chat-completions,responses). A path must be absolute, with no query, fragment or..segment.auth_header: the header that carries the key. Host, Content-Length and the hop-by-hop headers are refused.headers: fixed headers for each request to the server. The auth headers, the server's ownauth_header, Host, Content-Length and the hop-by-hop headers are refused. A value with CR, LF or NUL is refused.core/redact/redact.go).redact.JSONhides each value under a field named exactlyheadersand keeps the names.core/plugins/inferencerouter/plugin.go,api.go). Rules 4 to 7 and C1 to C3 of feat: Translation hub with Anthropic Messages to and from chat completions, and the api, auth_header and headers server fields #1911:api, the router sends the request to the path of that key.pctx.Translate) to translate the request to the first protocol of the map that has an encoder.docs/translation-server.md.count_tokensto a server with noanthropic-messagesentry gets a local 404not_found_error.auth_header, or in the usual header of the target protocol for a translated request. The other auth headers of the agent are removed. The fixed headers go topctx.SetUpstreamHeaders.core/parse/inference/dialect.go).inference.ProtocolForuses the same rule as the parser, so the parser and the router cannot disagree.cmd/agentop/cmd_server_write.go,cmd_server.go).agentop server addhas--api,--auth-headerand--header(rule C4). Each flag is checked with the routerconfig checks before agentop asks for the key. A header value with$is refused, becauseconfig.Loadexpands it.agentop servershows the api, the auth header and the header names under each server. It does not show header values.cmd/agentop/tui/server_route.go).routerconfig.Servernow holds maps, so it is not comparable.sameRouternow usesmaps.EqualFuncwithreflect.DeepEqual.TestSameRouter_SeesAHeadersChangecovers it.docs/translation-server.md(rule F1): how to run LiteLLM on loopback for a protocol pair that Cortex cannot translate.docs/inference-router-plugin.mdgives the new fields, the routing rules, the auth header table, the refused header names and themodify/translatedrecord.docs/README.mdhas an index row.How it was tested
369b0d2fe, and again onf0dd424cfafter the rebase on feat: Translate inference requests and replies in the forward proxy #1934 (eeec21132):corevet, gofmt, tests andgo mod tidy -diff;cmd/cortexvet;cmd/agentopvet, gofmt, tidy and sub-package tests; andscripts/dev/go-test-shards.sh -short cmd/agentop(1114 tests, exit 0).routerconfig/config_test.go(CheckAPI, CheckAuthHeader, CheckHeaders, Validate),inferencerouter/api_test.go(each routing rule, the auth header table, the 502 and the 404),forwardproxy/inferencerouter_test.go(router plus listener: path change, translation, refusal),parse/inference/dialect_test.go,redact/redact_test.go, and the agentopserver addandservertests.Size
Review findings that were not fixed
docs/translation-server.mdlines 4-7 say that "for each other pair" Cortex answers 502. The router page (rule 5) and the plan say that Cortex answers 502 only when the agent speaksanthropic-messagesand theapimap holds no protocol that Cortex can write. A chat-completions or Responses request goes upstream untranslated. Change the text to "for an Anthropic Messages request to a server that serves only another protocol".docs/translation-server.md(line 17) is for the one pair that Cortex already translates. Add one line that says why: the example shows the setup, or use it when Cortex's own translation refuses the request (for example a PDF gets a 400translation_refused).docs/inference-router-plugin.mdrule 4 quotes the 502 message without its last part,(see https://github.com/rossoctl/cortex/blob/main/docs/translation-server.md).A user who searches for the exact error text does not find it. Quote the full message, or mark the quote as shortened.serverUsagetext say that a replace changes "its URL, key and models". A replace also clearsapi,auth_headerandheaderswhen the new command does not give them again.api,auth_headerorheadersyet.CheckHeadersdoes not refuseUser-Agentanduser-agentin one map.Configurethen callsheaders.Setin Go map order, so the value that is kept is random and can change at each load. To fix: refuse such pairs inCheckHeaders(canonical-name check), or iterate overslices.Sorted(maps.Keys(s.Headers)).CheckAPI's unknown-key error starts withapi:, so it is not the plan's exact wording. The reason is good:Validatethen showsservers.bob.api: ...with no duplicate prefix. The T10 tests and the docs expect the prefix.TestCheckHeaderschecks only that a CR, LF or NUL value gives an error. Add an assertion that the error text does not contain the value.config_test.go, no blank line is between the end ofTestDecode_Sessionsandfunc TestCheckAPI.$in other flags (T10). The$check covers only--headervalues. Butconfig.Loadrunsos.Expandon the whole file (core/config/config.go:1319).--header 'X$HOME=1',--auth-header 'X$HOME'and--api chat-completions=/v1/$xpass, and Cortex then expands the$part when it loads the file. The same check would fit incheckProtocolFlags(cmd/agentop/cmd_server_write.go).serverListmatches lines by position (T10).cmd/agentop/cmd_server.gomatches each table line tonames[i]. If a cell has a newline (a hand-edited server name, main or helper word, agent name, or aresolvedWordserror text), there are more lines than names, the detail lines go under the wrong server, andc.Servers[names[i]]panics with an index out of range. Before this change, such a cell only made the output look wrong. To fix: write the detail lines from the name loop, or checki < len(names).api,auth_headerandheaders(T10).serverAddon an existing name writes a new map for the server. This also removes hand-writtenheaders: {X: ${VAR}}entries, which agentop refuses to write itself. The README says thatserver addis how a key is rotated. If you rotate the key of a chat-only server and do not give--apiagain, itsapimap is removed, so Claude Code/v1/messagesrequests then go untranslated to a server that cannot serve them. The confirm prompt says only "Replacing it gives it this URL, key and models". As a minimum, name the removed fields in that prompt.OnRequest, theSetUpstreamHeaderserror branch records denyredirect_failedafterpin.settle(name),setAuthandchooseModelhave run. The rest of the router records everyredirect_failedbefore the pin settles. This error cannot occur after a successfulRedirect, so only the code structure is affected. To keep a single deny path, callSetUpstreamHeadersnext toSetRedirectPath/Translate, and put its error incmp.Or(redirectErr, pathErr).routedrecord names the captured path (T7). When theapimap changed the path (planPath) or the request was translated (planTranslate),details["captured"]is still the capture-table path (for example/v1/messages). The upstream path ispl.path(for example/v1/chat/completions). For a translated request, the listener'smodify/translatedrecord names the correct path. ForplanPath, no record names the correct path.apimap hasanthropic-messagesat another path (DeepSeek:/anthropic/v1/messages),/v1/messages/count_tokensgetsplanAsIs, becauseProtocolForgives"". The request goes to/v1/messages/count_tokens, not to/anthropic/v1/messages/count_tokens. This agrees with rules 2 and 5 as written. A person must decide if it is correct.headerson the wire. No listener test sends a routerheadersentry (for exampleUser-Agent: bob-shell) through the forward proxy to the upstream.TestRouter_HeadersReachUpstreamHeadersstops atpctx.UpstreamHeaders(). P2 testedapplyUpstreamHeaderswith its own test plugin. Nothing tests the path from router config to listener to wire, or the resend path (server.go:1360) with router headers.For a human after review
These steps need a live agent, a real server or a person's decision. They are not tasks in this workflow.
claude -p, with tools) against a chat-completions server through the router for at least 3 tool-call turns. A local Ollama or llama-server works when itsapimap has onlychat-completions(agentop server add --api chat-completions=/v1/chat/completions …). Put the session id and its turn count in the PR P3 body./contextstill works with the local 404 forcount_tokens(rule 6). If it does not, report what Claude Code does. Do not guess a token count.__thought__tool id with+or/and sends it back unchanged. If Claude Code refuses or changes it, report what it does. Do not encode the id without a decision on that report.tool_choice,parallel_tool_calls,stop,temperature,top_p), and that Bob acceptsrole: toolmessages whosecontentis a list of text parts (decision 8).Assisted-By: Claude (Anthropic AI) noreply@anthropic.com