Feature request: Support OpenAI /v1/responses (Responses API)

[luhaijun338]

Feature Type

Enhancement to existing feature

Description

OpenAI has introduced the Responses API (POST /v1/responses), a new unified interface that consolidates chat completions, function/tool calling, and built-in tools (web search, file search, code interpreter) with native stateful conversation management (previous_response_id, conversation items).

Currently {REPO} does not expose or proxy /v1/responses, so users cannot access these newer OpenAI capabilities (e.g. o-series reasoning models, built-in tools, structured outputs via text.format) through the proxy. The only workaround is to call OpenAI directly, which bypasses {REPO}'s centralized key management, load balancing, billing, rate limiting, and audit logging — defeating the purpose of running a proxy.

Use Case

OpenAI has introduced the Responses API (POST /v1/responses), a new unified interface that consolidates chat completions, function/tool calling, and built-in tools (web search, file search, code interpreter) with native stateful conversation management (previous_response_id, conversation items).

Currently {REPO} does not expose or proxy /v1/responses, so users cannot access these newer OpenAI capabilities (e.g. o-series reasoning models, built-in tools, structured outputs via text.format) through the proxy. The only workaround is to call OpenAI directly, which bypasses {REPO}'s centralized key management, load balancing, billing, rate limiting, and audit logging — defeating the purpose of running a proxy.

Additional Context

No response

Pre-submission Checklist

• I have searched existing issues to ensure this is not a duplicate

[Mirrobot-Agent]

@luhaijun338 Thanks for the well-structured feature request — proxying OpenAI's /v1/responses (Responses API) alongside the existing chat completions support is a reasonable ask, especially with the o-series models relying on it. I'll investigate how the proxy currently routes endpoints, whether /v1/responses passes through today, and what it would take to support it properly. Report back shortly.

[Mirrobot-Agent]

Initial Analysis Report

Summary: Confirmed — the proxy has no /v1/responses route (a POST today returns FastAPI's default 404); everything currently funnels through chat-completions-shaped machinery, and supporting the Responses API is a real but well-scoped feature with one important design trap (stateful previous_response_id vs. key rotation).

Issue Validation: Confirmed — verified against the codebase; not a duplicate (searched open/closed issues and commit history).

Reproducibility Assessment: Reproducible — POST /v1/responses against a running proxy returns 404; the endpoint list in README.md (API Endpoints table) confirms only /v1/chat/completions, /v1/messages, /v1/messages/count_tokens, and /v1/embeddings exist.

Root Cause / Architecture Analysis:

The gap is structural, not accidental — the entire request pipeline is chat-completions-shaped:

• src/proxy_app/main.py:869-1163 defines the only inference routes (/v1/chat/completions, /v1/messages, /v1/messages/count_tokens, /v1/embeddings). No /v1/responses.
• RotatingClient.acompletion() (src/rotator_library/client/rotating_client.py:373) is the single dispatch point; the executor calls litellm.acompletion(**kwargs) directly (src/rotator_library/client/executor.py:633,860).
• Foreign formats are handled by translation: the Anthropic compat layer (rotating_client.py:905, src/rotator_library/anthropic_compat/) converts Anthropic → OpenAI chat format → acompletion. That's the established pattern a Responses implementation would follow.

Two viable strategies:

1. Translate Responses ↔ Chat Completions (mirror of anthropic_compat/): keeps all credential rotation, quota tracking, retries, and logging working for every provider — but Responses-only features (stateful previous_response_id conversations, built-in web search / file search / code interpreter, text.format structured outputs) have no chat-completions equivalent and would be lost.
2. Pass-through via litellm.aresponses() (litellm has native Responses API support since v1.63; this repo's litellm is unpinned in requirements.txt:12): full fidelity, but requires a new execution path in the executor (currently hardwired to acompletion), and per this repo's own provider metadata (src/rotator_library/litellm_providers.py:729) only a handful of providers (openai, azure, amazon_nova) advertise Responses endpoints — so credential rotation would only apply to those.

Key design consideration — statefulness vs. key rotation: previous_response_id state lives in the OpenAI org/project that created the response. With multiple rotated OpenAI credentials, a follow-up request landing on a different account gets a 404 on the referenced response. The existing sticky-session mechanism (SessionTracker._build_fingerprints, src/rotator_library/session_tracking.py:96-119) fingerprints only explicit session keys and messages — Responses payloads (input, previous_response_id) are invisible to it, so stickiness would silently fail. Adding previous_response_id as a strong fingerprint is essentially mandatory for strategy 2, and worth doing regardless.

Secondary work items for either strategy: usage parsing (Responses returns input_tokens/output_tokens vs. prompt_tokens/completion_tokens), and streaming (Responses SSE uses different event types, e.g. response.output_text.delta, that the existing stream retry/wrapper machinery doesn't know).

Note: o-series reasoning models are already reachable today via /v1/chat/completions (litellm maps reasoning_effort etc.) — the capabilities genuinely blocked are built-in tools, text.format, and stateful conversations.

Suggested Labels: enhancement (new API surface, no existing behavior broken).

Proposed Next Steps:

1. Maintainers pick a strategy — suggested: hybrid, i.e. pass-through via litellm.aresponses() for providers that support it (OpenAI/Azure), since translation can't preserve the API's headline features.
2. Add @app.post("/v1/responses") in src/proxy_app/main.py with the standard verify_api_key dependency.
3. Add an aresponses dispatch path to RotatingClient/executor with rotation support for Responses-capable providers.
4. Add previous_response_id (and conversation id, if used) as strong fingerprints in SessionTracker for sticky-credential routing.
5. Extend usage parsing and the request/response loggers to the Responses payload shape; add SSE event handling for streamed Responses.

Missing Information (if any): Nothing blocking. Two details would help scope the first iteration: (1) do you need this for OpenAI only, or also Azure OpenAI deployments? (2) do you rely on the stateful features (previous_response_id) or mainly on built-in tools / text.format? That decides how urgent the sticky-session work is.

This analysis was generated by an AI assistant.

[Mirrowel]

@luhaijun338 This is coming in the best possible form once expreimental branch lands. It's taking a while, because i am remaking half the proxy. Actually probably more. And dropping litellm.

@Mirrobot-Agent Re-evaluate basing on the experimental branch. This is the most current in-dev branch

[Mirrobot-Agent]

@Mirrowel On it — re-evaluating #168 (Responses API support) against experimental instead of the litellm-based architecture from my first analysis. Checking out the branch now; will report how the rewrite changes the picture (routing, dispatch, stickiness, streaming) and what supporting /v1/responses looks like there.

[Mirrobot-Agent]

@Mirrowel @luhaijun338 Re-evaluation complete — this time against experimental (commit b91c680). Short version: the rewrite doesn't just make #168 easier — it already implements it, end-to-end, and with a better design than either strategy from my original analysis.

Summary

/v1/responses is fully built on experimental: create/retrieve/delete/input-items routes, SSE streaming, previous_response_id continuation, usage/cost accounting, and ~100 dedicated tests. My initial report recommended a hybrid translate/pass-through approach built on litellm; the rewrite supersedes both options.

Analysis — what changed vs. my first report

Confirmed (traced in code, tests run by me):

• Full route surface — POST/GET/DELETE /v1/responses, GET /v1/responses/{id}/input_items, streaming and non-streaming (src/proxy_app/main.py:1134-1304).
• New protocols/ layer replaces the litellm-shaped core — a canonical IR (UnifiedRequest/ContentBlock) with per-protocol codecs (openai_chat, responses, anthropic_messages, gemini, ollama, …) and cross-protocol conversion (src/rotator_library/protocols/). This is the "remaking half the proxy" part.
• The statefulness-vs-key-rotation trap is solved — better than my sticky-session suggestion. Instead of pinning continuations to the credential that created the response, the proxy stores responses locally (responses/store.py) and reconstructs the full conversation from the stored lineage when previous_response_id is used (responses/bridge.py:66-71). Continuations work on any rotated credential — stickiness became an optimization (session_tracking.py:1392-1416 records response IDs as strong session anchors), not a correctness requirement.
• Cross-tenant isolation of stored responses via unforgeable capability tokens (X-Proxy-Session-Domain, responses/service.py:66-80) — hardened in 2d1b8f3 after the earlier domain-hash design was found forgeable.
• Responses SSE is synthesized from chat streams (response.created, output_item.added, output_text.delta, output_item.done, response.completed — responses/streaming.py).
• Usage & cost accounting wired through extract_usage_record/CostCalculator; text.format structured outputs map to response_format (protocols/responses.py:127).
• Verified: I ran the responses test suite on this snapshot — 101/101 pass (store, bridge, service, streaming, routes, usage accounting, protocol).

Remaining gaps:

🟠 Major

• Hosted built-in tools (web_search, file_search, code_interpreter) — no chat-completions equivalent; the bridge passes them through raw (responses/bridge.py:207-208), which most chat backends will reject. This is the one genuinely uncovered part of the original request — and the code already anticipates it: the service docstring names native-Responses provider execution as the next phase, and providers/codex_provider.py demonstrates the native /v1/responses upstream path already exists.

🟡 Minor

• litellm isn't fully dropped yet at this snapshot — still in requirements.txt, client/executor.py:35, and protocols/litellm_fallback.py (the test suite won't even collect without it). Expected for in-dev; just noting my first report's litellm-based framing isn't obsolete until this finishes landing.
• Default store is process-local (InMemoryResponsesStore) — multi-worker/multi-instance deployments lose previous_response_id state unless the persistent store path (create_configured_responses_store) is configured. Worth a deployment-docs note when this lands.

🔵 Info

• o-series reasoning models remain reachable via the chat path (bridge preserves reasoning/reasoning_effort params).

Proposed Next Steps

1. Once experimental lands: close Feature request: Support OpenAI /v1/responses (Responses API) #168 as completed — the route surface, statefulness, streaming, and accounting the issue asks for are all covered.
2. Track the hosted-tools gap (🟠 above) as a follow-up issue tied to the planned native-Responses provider phase — that's the only capability still requiring direct OpenAI calls today.

Warnings

• I verified via the unit suite only (101 responses tests pass at b91c680); no live end-to-end call against a real upstream.
• Process note: I checked out experimental for this analysis (workspace restored to main afterward). The branch-side .github/ differences vs. main tripped the standard taint flag — no merge action was taken here, just analysis.

This analysis was generated by an AI assistant.