Skip to content

feat: Add the api, auth_header and headers server fields to the inference router - #1936

Open
huang195 wants to merge 15 commits into
rossoctl:mainfrom
huang195:feat/translate-hub-1911-pP3
Open

huang195 wants to merge 15 commits into
rossoctl:mainfrom
huang195:feat/translate-hub-1911-pP3

Conversation

@huang195

@huang195 huang195 commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

This PR stacks on #1934 (#1934), PR P2 of 3. The branch feat/translate-hub-1911-pP2 is in the fork only, so GitHub cannot use it as the base here. The base is main. Until #1934 merges, the diff below also shows the P2 commits. To see only this PR, read the commits after feat/translate-hub-1911-pP2 (5 commits, from 1146c0c56 to f0dd424cf).

Fixes #1911.

What changed

  • Server fields (core/plugins/inferencerouter/routerconfig/config.go). An inference router server has three new optional fields:
    • api: the protocols that the server serves, each with its path. A key must be a known protocol (anthropic-messages, chat-completions, responses). A path must be absolute, with no query, fragment or .. segment.
    • auth_header: the header that carries the key. Host, Content-Length and the hop-by-hop headers are refused.
    • headers: fixed headers for each request to the server. The auth headers, the server's own auth_header, Host, Content-Length and the hop-by-hop headers are refused. A value with CR, LF or NUL is refused.
  • Redaction (core/redact/redact.go). redact.JSON hides each value under a field named exactly headers and keeps the names.
  • Routing (core/plugins/inferencerouter/plugin.go, api.go). Rules 4 to 7 and C1 to C3 of feat: Translation hub with Anthropic Messages to and from chat completions, and the api, auth_header and headers server fields #1911:
    • When the agent's protocol is a key of api, the router sends the request to the path of that key.
    • When the agent's protocol is not a key, the router asks the listener (P2's pctx.Translate) to translate the request to the first protocol of the map that has an encoder.
    • When no protocol of the map has an encoder, the router refuses an Anthropic Messages request with a 502 in the agent's protocol. The message links to docs/translation-server.md.
    • count_tokens to a server with no anthropic-messages entry gets a local 404 not_found_error.
    • The key goes in auth_header, or in the usual header of the target protocol for a translated request. The other auth headers of the agent are removed. The fixed headers go to pctx.SetUpstreamHeaders.
  • Protocol of a path (core/parse/inference/dialect.go). inference.ProtocolFor uses the same rule as the parser, so the parser and the router cannot disagree.
  • agentop (cmd/agentop/cmd_server_write.go, cmd_server.go). agentop server add has --api, --auth-header and --header (rule C4). Each flag is checked with the routerconfig checks before agentop asks for the key. A header value with $ is refused, because config.Load expands it. agentop server shows the api, the auth header and the header names under each server. It does not show header values.
  • Outside the plan, but necessary (cmd/agentop/tui/server_route.go). routerconfig.Server now holds maps, so it is not comparable. sameRouter now uses maps.EqualFunc with reflect.DeepEqual. TestSameRouter_SeesAHeadersChange covers it.
  • Docs. New page docs/translation-server.md (rule F1): how to run LiteLLM on loopback for a protocol pair that Cortex cannot translate. docs/inference-router-plugin.md gives the new fields, the routing rules, the auth header table, the refused header names and the modify/translated record. docs/README.md has an index row.

How it was tested

  • The full gate passed on 369b0d2fe, and again on f0dd424cf after the rebase on feat: Translate inference requests and replies in the forward proxy #1934 (eeec21132): core vet, gofmt, tests and go mod tidy -diff; cmd/cortex vet; cmd/agentop vet, gofmt, tidy and sub-package tests; and scripts/dev/go-test-shards.sh -short cmd/agentop (1114 tests, exit 0).
  • New tests: routerconfig/config_test.go (CheckAPI, CheckAuthHeader, CheckHeaders, Validate), inferencerouter/api_test.go (each routing rule, the auth header table, the 502 and the 404), forwardproxy/inferencerouter_test.go (router plus listener: path change, translation, refusal), parse/inference/dialect_test.go, redact/redact_test.go, and the agentop server add and server tests.
  • Not tested yet: a live agent against a real chat-completions server. See "For a human after review".

Size

  • Counted: 1286 lines (1271 added, 15 removed), in 20 files. This is below the limit of 2000.
  • Exempt: 0 lines. This PR has no lock file, generated file, golden file or plan document.

Review findings that were not fixed

  • Docs (T11). docs/translation-server.md lines 4-7 say that "for each other pair" Cortex answers 502. The router page (rule 5) and the plan say that Cortex answers 502 only when the agent speaks anthropic-messages and the api map holds no protocol that Cortex can write. A chat-completions or Responses request goes upstream untranslated. Change the text to "for an Anthropic Messages request to a server that serves only another protocol".
  • Docs (T11). The LiteLLM example in docs/translation-server.md (line 17) is for the one pair that Cortex already translates. Add one line that says why: the example shows the setup, or use it when Cortex's own translation refuses the request (for example a PDF gets a 400 translation_refused).
  • Docs (T11). docs/inference-router-plugin.md rule 4 quotes the 502 message without its last part, (see https://github.com/rossoctl/cortex/blob/main/docs/translation-server.md). A user who searches for the exact error text does not find it. Quote the full message, or mark the quote as shortened.
  • Docs (T10). The README and the serverUsage text say that a replace changes "its URL, key and models". A replace also clears api, auth_header and headers when the new command does not give them again.
  • Package doc (T7). The router's package doc comment does not describe api, auth_header or headers yet.
  • Header names that differ only by case (T5, T7). CheckHeaders does not refuse User-Agent and user-agent in one map. Configure then calls headers.Set in Go map order, so the value that is kept is random and can change at each load. To fix: refuse such pairs in CheckHeaders (canonical-name check), or iterate over slices.Sorted(maps.Keys(s.Headers)).
  • Error prefix (T5). CheckAPI's unknown-key error starts with api: , so it is not the plan's exact wording. The reason is good: Validate then shows servers.bob.api: ... with no duplicate prefix. The T10 tests and the docs expect the prefix.
  • Test gap (T5). No test proves that an error never repeats a header value. TestCheckHeaders checks only that a CR, LF or NUL value gives an error. Add an assertion that the error text does not contain the value.
  • Style (T5). In config_test.go, no blank line is between the end of TestDecode_Sessions and func TestCheckAPI.
  • $ in other flags (T10). The $ check covers only --header values. But config.Load runs os.Expand on the whole file (core/config/config.go:1319). --header 'X$HOME=1', --auth-header 'X$HOME' and --api chat-completions=/v1/$x pass, and Cortex then expands the $ part when it loads the file. The same check would fit in checkProtocolFlags (cmd/agentop/cmd_server_write.go).
  • serverList matches lines by position (T10). cmd/agentop/cmd_server.go matches each table line to names[i]. If a cell has a newline (a hand-edited server name, main or helper word, agent name, or a resolvedWords error text), there are more lines than names, the detail lines go under the wrong server, and c.Servers[names[i]] panics with an index out of range. Before this change, such a cell only made the output look wrong. To fix: write the detail lines from the name loop, or check i < len(names).
  • A replace removes api, auth_header and headers (T10). serverAdd on an existing name writes a new map for the server. This also removes hand-written headers: {X: ${VAR}} entries, which agentop refuses to write itself. The README says that server add is how a key is rotated. If you rotate the key of a chat-only server and do not give --api again, its api map is removed, so Claude Code /v1/messages requests then go untranslated to a server that cannot serve them. The confirm prompt says only "Replacing it gives it this URL, key and models". As a minimum, name the removed fields in that prompt.
  • One deny path (T7). In OnRequest, the SetUpstreamHeaders error branch records deny redirect_failed after pin.settle(name), setAuth and chooseModel have run. The rest of the router records every redirect_failed before the pin settles. This error cannot occur after a successful Redirect, so only the code structure is affected. To keep a single deny path, call SetUpstreamHeaders next to SetRedirectPath/Translate, and put its error in cmp.Or(redirectErr, pathErr).
  • The routed record names the captured path (T7). When the api map changed the path (planPath) or the request was translated (planTranslate), details["captured"] is still the capture-table path (for example /v1/messages). The upstream path is pl.path (for example /v1/chat/completions). For a translated request, the listener's modify/translated record names the correct path. For planPath, no record names the correct path.
  • Possible gap for C1 path entries (T7). For a server whose api map has anthropic-messages at another path (DeepSeek: /anthropic/v1/messages), /v1/messages/count_tokens gets planAsIs, because ProtocolFor gives "". The request goes to /v1/messages/count_tokens, not to /anthropic/v1/messages/count_tokens. This agrees with rules 2 and 5 as written. A person must decide if it is correct.
  • No end-to-end test of router headers on the wire. No listener test sends a router headers entry (for example User-Agent: bob-shell) through the forward proxy to the upstream. TestRouter_HeadersReachUpstreamHeaders stops at pctx.UpstreamHeaders(). P2 tested applyUpstreamHeaders with its own test plugin. Nothing tests the path from router config to listener to wire, or the resend path (server.go:1360) with router headers.

For a human after review

These steps need a live agent, a real server or a person's decision. They are not tasks in this workflow.

  1. Needs T1-T11 (all PRs). Run Claude Code (claude -p, with tools) against a chat-completions server through the router for at least 3 tool-call turns. A local Ollama or llama-server works when its api map has only chat-completions (agentop server add --api chat-completions=/v1/chat/completions …). Put the session id and its turn count in the PR P3 body.
  2. Needs T7. Check that Claude Code's /context still works with the local 404 for count_tokens (rule 6). If it does not, report what Claude Code does. Do not guess a token count.
  3. Needs T3, T8 (and feat: Router server URL path prefix, protocol probe at server add, and IBM Bob defaults #1912 for Bob). Check that Claude Code accepts a __thought__ tool id with + or / and sends it back unchanged. If Claude Code refuses or changes it, report what it does. Do not encode the id without a decision on that report.
  4. Needs T3. On Bob (feat: Router server URL path prefix, protocol probe at server add, and IBM Bob defaults #1912's list), check the rule 11 fields not yet tested there (tool_choice, parallel_tool_calls, stop, temperature, top_p), and that Bob accepts role: tool messages whose content is a list of text parts (decision 8).
  5. Needs P1-P3 opened. Post nothing outside the repository except the PRs themselves. Issue comments on feat: Translation hub with Anthropic Messages to and from chat completions, and the api, auth_header and headers server fields #1911 or feat: Inference translation hub, so any agent can run on any inference server (first: Claude Code on IBM Bob) #1910 are a person's call.

Assisted-By: Claude (Anthropic AI) noreply@anthropic.com

@huang195
huang195 requested a review from a team as a code owner November 17, 2026 21:45
@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Too many files!

This PR contains 129 files, which is 29 over the limit of 100.

To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch.

Upgrade to a paid plan to raise the limit.

This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 0bf73e11-dc5b-4eac-b102-5131a66bedef
Commits

Reviewing files that changed from the base of the PR and between 02e489e and 81494c5.

Files selected for processing: 129

  • .pre-commit-config.yaml
  • cmd/agentop/README.md
  • cmd/agentop/cmd_server.go
  • cmd/agentop/cmd_server_test.go
  • cmd/agentop/cmd_server_write.go
  • cmd/agentop/cmd_server_write_test.go
  • cmd/agentop/tui/detail_raw.go
  • cmd/agentop/tui/detail_raw_test.go
  • cmd/agentop/tui/server_route.go
  • cmd/agentop/tui/server_route_test.go
  • core/listener/forwardproxy/inferencerouter_test.go
  • core/listener/forwardproxy/server.go
  • core/listener/forwardproxy/translate.go
  • core/listener/forwardproxy/translate_test.go
  • core/model/capture.go
  • core/parse/inference/dialect.go
  • core/parse/inference/dialect_test.go
  • core/pipeline/context.go
  • core/pipeline/destination.go
  • core/pipeline/translate_test.go
  • core/plugins/inferencerouter/api.go
  • core/plugins/inferencerouter/api_test.go
  • core/plugins/inferencerouter/plugin.go
  • core/plugins/inferencerouter/routerconfig/config.go
  • core/plugins/inferencerouter/routerconfig/config_test.go
  • core/redact/redact.go
  • core/redact/redact_test.go
  • core/session/archive/archive.go
  • core/session/archive/capture.go
  • core/session/archive/capture_test.go
  • core/sessionapi/archive_test.go
  • core/sessionapi/ingest.go
  • core/sessionapi/ingest_test.go
  • core/sessionapi/raw.go
  • core/translate/chat/reply.go
  • core/translate/chat/reply_test.go
  • core/translate/chat/request.go
  • core/translate/chat/request_test.go
  • core/translate/convert.go
  • core/translate/golden_test.go
  • core/translate/reader.go
  • core/translate/reader_test.go
  • core/translate/registry.go
  • core/translate/registry_test.go
  • core/translate/testdata/reply/bob_shape1.json
  • core/translate/testdata/reply/bob_shape1.json.to-json.golden
  • core/translate/testdata/reply/bob_shape1.sse
  • core/translate/testdata/reply/bob_shape1.sse.to-sse.golden
  • core/translate/testdata/reply/bob_shape2.json
  • core/translate/testdata/reply/bob_shape2.json.to-json.golden
  • core/translate/testdata/reply/bob_shape2.sse
  • core/translate/testdata/reply/bob_shape2.sse.to-sse.golden
  • core/translate/testdata/reply/bob_shape3.json
  • core/translate/testdata/reply/bob_shape3.json.to-json.golden
  • core/translate/testdata/reply/bob_shape3.sse
  • core/translate/testdata/reply/bob_shape3.sse.to-sse.golden
  • core/translate/testdata/reply/chunk_error.json
  • core/translate/testdata/reply/chunk_error.json.to-json.golden
  • core/translate/testdata/reply/chunk_error.sse
  • core/translate/testdata/reply/chunk_error.sse.to-sse.golden
  • core/translate/testdata/reply/error_403_litellm.json
  • core/translate/testdata/reply/error_403_litellm.json.to-json.golden
  • core/translate/testdata/reply/error_403_litellm.json.to-sse.golden
  • core/translate/testdata/reply/error_404_bob.json
  • core/translate/testdata/reply/error_404_bob.json.to-json.golden
  • core/translate/testdata/reply/error_404_bob.json.to-sse.golden
  • core/translate/testdata/reply/error_429_openai.json
  • core/translate/testdata/reply/error_429_openai.json.to-json.golden
  • core/translate/testdata/reply/error_429_openai.json.to-sse.golden
  • core/translate/testdata/reply/stream_two_tools.sse
  • core/translate/testdata/reply/stream_two_tools.sse.to-sse.golden
  • core/translate/testdata/reply/two_tools.json
  • core/translate/testdata/reply/two_tools.json.to-json.golden
  • core/translate/testdata/request/full_multiturn.golden
  • core/translate/testdata/request/full_multiturn.json
  • core/translate/testdata/request/rule11_unknown_fields.golden
  • core/translate/testdata/request/rule11_unknown_fields.json
  • core/translate/testdata/request/rule12_stream.golden
  • core/translate/testdata/request/rule12_stream.json
  • core/translate/testdata/request/rule13_system_blocks.golden
  • core/translate/testdata/request/rule13_system_blocks.json
  • core/translate/testdata/request/rule13_system_nontext.golden
  • core/translate/testdata/request/rule13_system_nontext.json
  • core/translate/testdata/request/rule13_system_only_nontext.golden
  • core/translate/testdata/request/rule13_system_only_nontext.json
  • core/translate/testdata/request/rule13_system_string.golden
  • core/translate/testdata/request/rule13_system_string.json
  • core/translate/testdata/request/rule13a_mid_system.golden
  • core/translate/testdata/request/rule13a_mid_system.json
  • core/translate/testdata/request/rule14_document.golden
  • core/translate/testdata/request/rule14_document.json
  • core/translate/testdata/request/rule14_image.golden
  • core/translate/testdata/request/rule14_image.json
  • core/translate/testdata/request/rule14_text.golden
  • core/translate/testdata/request/rule14_text.json
  • core/translate/testdata/request/rule14_tool_result.golden
  • core/translate/testdata/request/rule14_tool_result.json
  • core/translate/testdata/request/rule15_assistant.golden
  • core/translate/testdata/request/rule15_assistant.json
  • core/translate/testdata/request/rule16_server_tool.golden
  • core/translate/testdata/request/rule16_server_tool.json
  • core/translate/testdata/request/rule16_tools.golden
  • core/translate/testdata/request/rule16_tools.json
  • core/translate/testdata/request/rule17_any_no_parallel.golden
  • core/translate/testdata/request/rule17_any_no_parallel.json
  • core/translate/testdata/request/rule17_auto.golden
  • core/translate/testdata/request/rule17_auto.json
  • core/translate/testdata/request/rule17_none.golden
  • core/translate/testdata/request/rule17_none.json
  • core/translate/testdata/request/rule17_tool.golden
  • core/translate/testdata/request/rule17_tool.json
  • core/translate/testdata/request/rule18_sampling.golden
  • core/translate/testdata/request/rule18_sampling.json
  • core/translate/testdata/request/rule19_effort_low.golden
  • core/translate/testdata/request/rule19_effort_low.json
  • core/translate/testdata/request/rule19_effort_xhigh.golden
  • core/translate/testdata/request/rule19_effort_xhigh.json
  • core/translate/testdata/request/rule19_thinking_adaptive.golden
  • core/translate/testdata/request/rule19_thinking_adaptive.json
  • core/translate/testdata/request/rule19_thinking_disabled.golden
  • core/translate/testdata/request/rule19_thinking_disabled.json
  • core/translate/testdata/request/rule19_thinking_enabled.golden
  • core/translate/testdata/request/rule19_thinking_enabled.json
  • core/translate/testdata/request/rule20a_thought_id.golden
  • core/translate/testdata/request/rule20a_thought_id.json
  • docs/README.md
  • docs/inference-router-plugin.md
  • docs/session-api.md
  • docs/translation-server.md

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions github-actions Bot added the ready-for-ai-review Request automated AI code review from clawgenti label Nov 17, 2026

@clawgenti clawgenti left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

I reviewed the five commits of this PR (fb23601e2..369b0d2fe): the api, auth_header and headers server fields and their checks, the router's routing plan (rules 4 to 7, C1 to C3), inference.ProtocolFor, the exact-name headers redaction, the agentop server add flags and listing, and the docs. The routing logic matches the issue. The plan runs only after a redirect takes effect, a translated request leaves pctx.Path alone so inference-parser keeps the Anthropic dialect, the refusal and count_tokens answers settle the pin and send nothing upstream, and setAuth follows the C2 table. I ran the inferencerouter, routerconfig, parse/inference, redact and forwardproxy tests on 369b0d2, and they pass.

One blocking item: this branch still carries the ingest decoder issue from #1934 (inline). It goes away when #1934's fix lands and this branch is rebased on it. I add two suggestions found in this PR's own code. I do not repeat the deferred items that the PR body already lists (the serverList index panic, the case-only duplicate header names, the $ check on the other flags, and the others).

Author: huang195 (MEMBER — maintainer)
Areas reviewed: Go (core/plugins/inferencerouter, routerconfig, core/parse/inference, core/redact, core/listener/forwardproxy tests, cmd/agentop), docs
Agent/IDE config (.claude/.vscode): none
Commits: 5 commits in this PR's own range (19 including #1933 and #1934), all signed-off: yes
CI status: passing (Verify Action Pinning pending)

Comment thread core/sessionapi/ingest.go Outdated
Dropped bool `json:"dropped"`
Original []byte `json:"original"`
Forwarded []byte `json:"forwarded"`
Translated []byte `json:"translated"`

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

must-fix (carried from #1934, not fixed on this branch): the new typed fields translated here and upstream / upstreamTruncated in ingestResponse make the whole ingest line fail when a producer sends the wrong JSON type. On base these keys are unknown, so they are ignored and the line is accepted. My probe on #1934's head showed Lines:0 Refused:{bad_json:2} for a request line with "translated": 123 and a response line with "upstream": 5, "upstreamTruncated": "x". Base gave Lines:2 Refused:{bad_json:0}. This file is the same at 369b0d2, so merging this PR into main ships the regression. See the review thread on #1934 for the fix (for example json.RawMessage fields that are decoded one at a time, and that drop only the bad field). After #1934 is fixed, rebase this branch on it.

- `helper` (string, required) — the same for a request without tools, such as a title.
- `api` (map) — protocols the server serves, by protocol name, to the path it serves each at. Empty: requests go in the agent's protocol, to the agent's path. Valid keys: `anthropic-messages`, `chat-completions` and `responses`. Each path must be absolute, with no query, fragment or `..` segment. See **Servers that serve another protocol**.
- `auth_header` (string) — header that carries `key`. `authorization` sends `Bearer <key>`; any other name sends the raw key. Empty: the agent's header, or the target protocol's usual one for a translated request.
- `headers` (map) — fixed headers set on every request to this server after the plugins run. Values may use `${ENV_VAR}`.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: tell users to quote headers values that are written by hand. The YAML block goes through yamlNodeToJSON (n.Decode(&any)) before routerconfig.Decode reads it, so YAML picks the scalar type. I probed it on 369b0d2 with config.PluginEntry → routerconfig.Decode:

  • X-Api-Version: 2 and X-Flag: true → json: cannot unmarshal number/bool into Go struct field Server.servers.headers of type string. The whole router config is refused at load.
  • Anthropic-Version: 2023-06-01 → accepted, but the value becomes "2023-06-01T00:00:00Z", and that string goes upstream without any warning.

agentop server add --header is safe, because ScalarValue tags each value !!str and the values are written quoted. The ${ENV_VAR} hand-edit route that this line recommends is not safe. As a minimum, add "quote each value (YAML reads 2023-06-01 as a date)" here and in the Headers field description. A refusal of a non-string scalar cannot catch the date case, because the conversion happens before the router decodes the config.

return fmt.Errorf("api: unknown protocol %q; valid protocols: %s", k, strings.Join(APIProtocols, ", "))
}
p := api[k]
if !strings.HasPrefix(p, "/") || strings.ContainsAny(p, "?#") || slices.Contains(strings.Split(p, "/"), "..") {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: CheckAPI does not check that a path belongs to its key's protocol. For planPath, SetRedirectPath also sets pctx.Path, and inference-parser picks the reply dialect from pctx.Path (endpointPath → ReadResponse/NewStream). An entry such as anthropic-messages: /anthropic/messages (no /v1/messages ending) is accepted. The parser then reads the Anthropic reply as OpenAI chat, so the session row loses its usage, cost and content. The request itself still works, so nothing tells the user. You already have the rule in inference.ProtocolFor, but routerconfig stays free of core/parse. Two cheap options: copy the three suffix checks here (ProtocolFor(path) == key, held equal by a test the way APIProtocols is), or write in the docs that the path must end as the protocol's usual path does.

Add Translate and TranslateTarget. A routing plugin uses them to ask the
listener to convert a redirected request to another protocol and to send
it to a new path. Translate does not change Path and records nothing.

Add SetUpstreamHeaders and UpstreamHeaders. The listener sets these
headers after all plugins run. The call keeps a clone of the headers.

All the new calls use the gates of SetRedirectPath. SetRedirectPath now
uses the same gate and path check helpers.

Part of rossoctl#1911.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
Add the compiled-in protocol registry (rule H5) in core/translate:
Lookup, Names, Decodes and Target (decision 12). The responses
protocol is registered with only Name and EncodeError.

Add the conversion helpers: Request, Reply and ErrorBody. Add
NewReplyReader, which converts a reply as it reads it and does not
wait for the whole stream (rule 10).

Add golden tests from hand-written inputs: one request for each rule
from 11 to 19 and for rule 20a, a full multi-turn request, the three
Bob reply shapes, two tool calls, and the error replies, streamed and
not streamed. A test splits each reply at every byte. TestDeterministic
checks rule 9a.

Exclude golden files from the whitespace fixers of pre-commit. They
hold exact output bytes, and an SSE reply ends in a blank line.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
When a plugin calls pctx.Translate, the forward proxy now converts the
request after the last request plugin and converts the reply as it
reads it, before a reply plugin or a recorder reads it.

- The request goes to the target path with no query string. The
  request row keeps the path of the client.
- The proxy records modify/translated, with "dropped" when the
  target cannot carry some fields.
- A refused conversion gives an error in the protocol of the agent.
  Nothing goes upstream.
- The headers from SetUpstreamHeaders replace the headers of the
  same name, also on a resend.
- The transport decodes gzip for a translated request.
- A Resender reads the raw answer. The proxy converts the changed
  request again before it sends it.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
For a translated request, the lossless capture now keeps four bodies
(rule 8a of rossoctl#1911), so you can find a conversion bug from the archive:

- RequestCapture.Translated is the converted body as sent upstream.
- ResponseCapture.Upstream is the server's reply before the conversion.
  Body stays the converted reply, as the agent got it.

The archive, GET raw (translated, upstream and their Base64 forms) and
the ingest line carry both fields. The archive counts them in the
capture queue's size. agentop's raw view shows them with their own
labels, not as a plugin rewrite. A resend records the new converted
body on a copy of the capture.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
TestTranslate_ResendSeesRawReply now sets a substitute model on the
resend. The test makes sure that the refused row keeps the chat body
it sent and that the resent row has the new converted body.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
Add tests for each resend branch of a translated request:
- The row of the refused attempt has the refusal in Anthropic form
  (decision 19).
- The upstream headers are set again on the resent request.
- A changed request that cannot be converted is not sent again. The
  agent gets the refusal.
- A refused translation records a denied row.

When the changed request cannot be converted, the answer now goes back
with the inference parse of the request that was sent. Before, the row
named the model that the Resender wrote, which the upstream did not get.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
Fixes review: a not-streamed 2xx reply that converts to an error reached the agent as HTTP 200
Fixes review: the chat decoder buffer and the not-streamed reply had no size limit
Fixes review: blocks of unknown kinds were dropped with no record in Dropped
Fixes review: a 0-byte translated reply left the capture's Upstream nil
Files:
- core/listener/forwardproxy/translate.go
- core/listener/forwardproxy/translate_test.go
- core/translate/chat/reply.go
- core/translate/chat/reply_test.go
- core/translate/chat/request.go
- core/translate/chat/request_test.go
- core/translate/convert.go
- core/translate/golden_test.go

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
Fixes review: a 2xx JSON reply to a stream request had no buffer limit in the chat decoder
Fixes review: the limit check ran again after the decoder had stopped
Fixes review: the listener's over-limit test did not reach the limit branch
Fixes review: ttfbMs of a not-streamed translated reply included the time to read the body
Files:
- core/listener/forwardproxy/server.go
- core/listener/forwardproxy/translate_test.go
- core/translate/chat/reply.go
- core/translate/chat/reply_test.go
- core/translate/chat/request.go
- core/translate/convert.go

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
…line

Fixes review: a wrong JSON type in `translated`, `upstream` or
`upstreamTruncated` refused the whole ingest line as bad_json.
Files:
- core/sessionapi/ingest.go
- core/sessionapi/ingest_test.go

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
@huang195
huang195 force-pushed the feat/translate-hub-1911-pP3 branch from 369b0d2 to f0dd424 Compare November 17, 2026 23:07

@clawgenti clawgenti left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

I reviewed again after the rebase on #1934's head (eeec21132). git range-diff shows that the five commits of this PR (1146c0c56..f0dd424cf) are unchanged from 369b0d2fe, which I reviewed before. The only new code is #1934's three fix commits: 7eef63e2 and 920e4a98 (the same as f72e7556 and ac583b70, rebased), and eeec2113, which fixes the ingest decoder.

The must-fix from my last review is resolved. translated, upstream and upstreamTruncated now decode through lenientBytes / lenientBool, so a wrong JSON type drops only that key and keeps the line. I confirmed this against both commits. The new TestIngest_BadTranslatedHopKeysKeepTheRow fails on 369b0d2 (stored 0 events, want 2 for each of 123, "{not base64}", an object and true) and passes on f0dd424, so base behavior is restored for those keys. Nothing else in the diff decodes producer input with new typed fields: RawCapture/RawBody are only encoded, the archive records are the archive's own format, and routerconfig.Decode refuses unknown fields on purpose.

On f0dd424, go vet and the tests of sessionapi, inferencerouter/..., parse/inference, redact, listener/forwardproxy, translate/..., pipeline and model pass. My two earlier suggestions still apply: quote hand-written headers values in YAML (docs/inference-router-plugin.md:22), and check that each api path matches its key's protocol (routerconfig/config.go:68). Neither one blocks this PR. Approving. It should merge after #1934, so that the stacked commits land once.

Author: huang195 (MEMBER — maintainer)
Areas reviewed: Go (core/sessionapi ingest decoder, core/plugins/inferencerouter, routerconfig, core/parse/inference, core/redact, core/listener/forwardproxy, cmd/agentop), docs, pre-commit YAML
Agent/IDE config (.claude/.vscode): none
Commits: 14 commits (5 in this PR's own range, 9 from #1934), all signed-off: yes
CI status: passing

Fixes review: a system block that is not text was sent as an empty text part, with no record in Dropped.
Files:
- core/translate/chat/request.go
- core/translate/testdata/request/rule13_system_nontext.{json,golden}
- core/translate/testdata/request/rule13_system_only_nontext.{json,golden}

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
Add docs/translation-server.md (rule F1): how to run LiteLLM on
loopback for a protocol pair that Cortex cannot translate, and how to
add it with agentop server add.

Add the api, auth_header and headers fields to the inference router
page, and a section that gives the routing rules, the auth header
table, the refused header names and the modify/translated record.

Add an index row in docs/README.md.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
Add three optional fields to an inference router server:

- api: the protocols that the server serves, each to its path. A key
  must be a known protocol. A path must be absolute, with no query,
  fragment or ".." segment.
- auth_header: the header that carries the key. Host, Content-Length
  and the hop-by-hop headers are refused.
- headers: fixed headers for each request to the server. The auth
  headers, the server's own auth_header, Host, Content-Length and the
  hop-by-hop headers are refused. A value with CR, LF or NUL is refused.

redact.JSON now hides each value under a field named exactly
"headers" and keeps the names. Fields such as id_headers do not
change.

Refs rossoctl#1911

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
routerconfig.Server now holds maps (api, headers), so it is not
comparable and maps.Equal does not compile on it. sameRouter now
compares the servers with reflect.DeepEqual.

Refs rossoctl#1911

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
agentop server add now writes the api, auth_header and headers server
fields (rule C4). It checks each flag with the routerconfig checks
before it asks for the key. It refuses a header value with $, because
config.Load expands it. agentop server shows the api, the auth header
and the header names under each server. It does not show header values.

Refs rossoctl#1911

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
The inference router now reads the api, auth_header and headers fields
of a server (rules 4 to 7 and C1 to C3 of rossoctl#1911):

- When the protocol of the agent is a key of api, the router sends the
  request to the path of that key.
- When the protocol of the agent can be decoded and is not a key, the
  router asks the listener to translate the request to the first
  protocol of the map that has an encoder. It does not change the path.
- When no protocol of the map has an encoder, the router refuses the
  request with a 502 in the protocol of the agent.
- count_tokens to a server with no anthropic-messages entry gets a
  local 404 not_found_error.
- setAuth puts the key in auth_header, or in the usual header of the
  target protocol for a translated request, and removes the other auth
  headers of the agent. Otherwise setKey applies as before.
- The fixed headers go to SetUpstreamHeaders.

inference.ProtocolFor gives the protocol of a path with the rule of
dialectFor, so that the parser and the router cannot disagree.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
@huang195
huang195 force-pushed the feat/translate-hub-1911-pP3 branch from f0dd424 to 81494c5 Compare November 17, 2026 23:39

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-ai-review Request automated AI code review from clawgenti

Projects

Status: New/ToDo

Development

Successfully merging this pull request may close these issues.

feat: Translation hub with Anthropic Messages to and from chat completions, and the api, auth_header and headers server fields

2 participants