Skip to content

Add OrcaRouter-Adaptive submission to leaderboard#43

Open
ZhenghuaBao wants to merge 2 commits into
RouteWorks:mainfrom
ZhenghuaBao:add-orcarouter-adaptive
Open

Add OrcaRouter-Adaptive submission to leaderboard#43
ZhenghuaBao wants to merge 2 commits into
RouteWorks:mainfrom
ZhenghuaBao:add-orcarouter-adaptive

Conversation

@ZhenghuaBao

@ZhenghuaBao ZhenghuaBao commented May 29, 2026

Copy link
Copy Markdown

Summary

Adds the OrcaRouter-Adaptive entry to the leaderboard data, mirroring the upstream RouterArena submission merged in PR #104 (predictions), PR #108 (README rank), and PR #110 (website link).

Submission

  • Router: OrcaRouter-Adaptive
  • Approach: LinUCB contextual bandit with embedding features
  • Model pool (10): Anthropic claude-haiku/sonnet-4, Google gemini-2.5-flash/-lite, OpenAI gpt-4o-mini/gpt-5-mini, DeepSeek chat/reasoner, Alibaba qwen3-235b/30b
  • Website: https://www.orcarouter.ai/
  • Type: closed-source (paper + GitHub may be added in a follow-up)

Bot-reported metrics

Metric Value
Arena Score 72.08
Accuracy 75.54
Cost / 1K Queries $1.00
Robustness 22.62
Optimal Selection
Optimal Cost
Optimal Accuracy
Latency

The three optimality columns and latency are dashed — the submitted prediction file did not include for_optimality=True entries, same convention as GPT-5 and Azure-Model-Router. May be filled in a follow-up submission.

Files touched

  • src/data/routerMetrics/leaderboard.json — new row, sorted by Arena Score (between Sqwish 75.27 and R2-Router 71.6)
  • src/data/routers.json — new metadata entry
  • src/data/routers.yaml — new metadata entry

No category_scores.json change — routerData.ts auto-synthesizes the entry from the leaderboard row.

Test plan

  • python -c "json.load(...)" on both edited JSON files — parses clean

ZhenghuaBao and others added 2 commits May 29, 2026 10:45
Mirrors the upstream RouterArena submission merged in PR #104 (predictions),
#108 (README), and #110 (link). LinUCB contextual bandit over a 10-model
pool; Arena 72.08, Accuracy 75.54, Cost/1K $1.00, Robustness 22.62. The
three optimality columns and latency are nulled — the submitted prediction
file did not include for_optimality entries, same convention as GPT-5 and
Azure-Model-Router.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@ZhenghuaBao

Copy link
Copy Markdown
Author

Hi @yl231,

Thank you for maintaining RouterArena. I would like to request that this PR be reviewed and merged so that our work can be displayed on the official RouterArena website.

Thanks for your time and consideration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant