Skip to content

Add BARouter submission#157

Open
zulk2002 wants to merge 1 commit into
RouteWorks:mainfrom
zulk2002:submit-barouter
Open

Add BARouter submission#157
zulk2002 wants to merge 1 commit into
RouteWorks:mainfrom
zulk2002:submit-barouter

Conversation

@zulk2002

@zulk2002 zulk2002 commented Jul 3, 2026

Copy link
Copy Markdown

BARouter: Buget Adaptive Router## BARouter Submission

This PR adds the BARouter prediction artifacts for RouterArena leaderboard evaluation.

Router

BARouter uses model pool:

  • qwen3-235b-a22b-instruct-2507
  • gpt-4.1
  • deepseek-v3.1

The submission includes final routing decisions and generated outputs only; no router implementation code or local predictor artifacts are included.

Files

  • router_inference/config/BARouter.json
  • router_inference/predictions/BARouter.json
  • router_inference/predictions/BARouter-robustness.json

Local Validation

I locally evaluated the full split and checked the submission files.

  • Full split: 8,400 regular entries + 1,618 optimality entries
  • Robustness split: 420 entries
  • All full entries include generated_result, accuracy, cost, and usable token usage
  • Abnormal entries: 0

Local full-split metrics:

  • RouterArena Score: 0.6714
  • Accuracy: 68.86%
  • Total Cost: $5.2636
  • Avg Cost per Query: $0.000627
  • Avg Cost per 1K Queries: $0.6266
  • Opt.Sel: 0.5862
  • Opt.Cost: 0.5919
  • Opt.Acc: 0.8793

@zulk2002

zulk2002 commented Jul 3, 2026

Copy link
Copy Markdown
Author

/evaluate

@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown

Router Evaluation Results

Router: BARouter
Dataset Split: full

RouterArena Metrics

Metric Value
RouterArena Score 0.6709
Accuracy 68.80%
Total Cost $5.263578
Avg Cost per Query $0.000627
Avg Cost per 1K Queries $0.6266
Number of Queries 8400
Abnormal Entries 0
Robustness Score 0.5238

Optimality Metrics

Metric Value
Opt.Sel (Optimal Selection) 0.5856
Opt.Cost (Cost Efficiency) 0.5903
Opt.Acc (Accuracy vs Optimal) 0.8791

Evaluation completed by RouterArena automated workflow

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant