We found a model-resolution issue affecting three of RouterArena's current top five submissions. After xAI retired grok-4-1-fast-reasoning on May 15, calls to that slug silently resolved to Grok 4.3 with low reasoning effort. RouterArena still recorded and priced those calls as Grok 4.1 Fast. The confirmed artifacts therefore combine Grok 4.3 outputs with Grok 4.1 Fast costs.
This is not a harmless alias change. It substitutes a stronger, substantially more expensive model:
The Artificial Analysis values are independent estimates captured on July 15, 2026. The problem is the combination: a materially stronger model's answers were scored at the retired model's much lower price.
Upstream change
xAI's May 15 retirement notice says that after 12:00 PM PT:
Requests to grok-4-1-fast-reasoning are redirected to grok-4.3 with low reasoning effort.
RouterArena's pinned model_cost.json still assigns the retired slug Grok 4.1 Fast's $0.20/$0.50 prices.
Affected leading submissions
We audited the prediction artifacts at commit 711bb4d. Where a file contains appended optimality runs, we kept the first non-optimality row for each of the 8,400 unique requests.
| Current rank |
Submission |
Date |
Calls selecting retired slug |
Evidence |
| 1 |
Cross-Router, PR #163 |
July 10 |
217 |
Confirmed: all 217 record provider=openrouter and model_used=x-ai/grok-4.3 |
| 2 |
vLLM-SR, PR #147 |
June 17 |
257 |
Confirmed: all 257 record provider=openrouter and model_used=x-ai/grok-4.3 |
| 4 |
Nadir-Tumbler, PR #159 |
July 4 |
3,571 |
The PR says these calls used xAI. They were made after xAI's documented redirect, but the saved rows lack model_used and invocation timestamps. |
Artifacts: Cross-Router, vLLM-SR, Nadir-Tumbler, and the leaderboard at the audited commit.
This appears accidental. The teams uploaded complete submission artifacts, and the OpenRouter metadata they preserved is what exposed the mismatch.
Naive repricing shows materiality, not the corrected ranking
Holding every submitted model choice and answer fixed, then billing the affected calls at Grok 4.3's price, gives:
| Submission |
Affected calls |
Submitted total |
Repriced total |
Submitted cost / 1K |
Repriced cost / 1K |
Submitted Arena |
Naively repriced Arena |
Mechanical rank* |
| Cross-Router |
217 |
$2.557320 |
$3.330657 |
$0.3044 |
$0.3965 |
76.121 |
75.753 |
#1 |
| vLLM-SR |
257 |
$2.537787 |
$3.509615 |
$0.3021 |
$0.4178 |
75.302 |
74.857 |
#3 |
| Nadir-Tumbler [unconfirmed] |
3,571 |
$0.887913 |
$3.368981 |
$0.1057 |
$0.4011 |
75.17 |
~73.34 |
~#5 |
* Mechanical rank if all three affected entries are repriced and every unaffected score stays fixed.
Nadir-Tumbler is not directly confirmed because its artifact lacks the actual-model field. Its committed artifact also sums to a different original cost than the leaderboard reports, so this row uses the committed artifact and the leaderboard's rounded accuracy. It is an estimate, not a reproducible correction to the published score.
The recorded cost rises 30.2% for Cross-Router, 38.3% for vLLM-SR, and about 279% for Nadir-Tumbler's committed artifact. These numbers establish that the mismatch is rank-material. However, they are not corrected router scores.
Why the existing scores cannot be repaired by changing the price column
These are cost-aware routers. Grok 4.3 looked like a strong, unusually cheap arm during training, calibration, or inference-time selection because it carried Grok 4.1 Fast's price. Had the router seen the real price, it could have selected Grok far less often or chosen different models entirely.
That changes both sides of the Arena score. The final policy might lose more than the naive calculation suggests, or it might recover part of the loss by finding good substitutes. The saved routes cannot tell us which counterfactual policy each system would have learned, nor what accuracy that policy would have achieved. Naive repricing is therefore neither a corrected score nor a reliable bound. The affected submissions need to be rerun with the actual model identity and price visible to the router.
Nadir-Tumbler
Nadir-Tumbler was submitted on July 4 and its PR says the Grok calls were made through xAI. xAI had retired the requested slug on May 15 and documents the same automatic redirect for its own API, so Grok 4.3 is the expected resolved model. Unlike the OpenRouter artifacts, however, these rows do not preserve an actual-model field or timestamps. We therefore distinguish this case from the two directly confirmed ones and ask the maintainers to verify it with the submission team.
The task-matched performance is consistent with that explanation. We formed an Azure-hosted Grok 4.1 comparison bank from Azure-Model-Router and Nadir's earlier Cascade submission. Both artifacts record Azure as the provider, so they were not subject to xAI's endpoint redirect. Across 2,042 matched, scorable requests, Nadir-Tumbler scores 67.11% against 64.27% for those Azure-hosted responses, a paired shift of 2.84 points (approximate 95% interval: 1.27 to 4.42 points), with 85.5% binary correctness agreement. Across 155 matched, scorable requests shared with the confirmed OpenRouter Grok 4.3 responses, Nadir-Tumbler is nearly identical: 63.94% versus 64.06%, a -0.12-point difference (approximate 95% interval: -4.73 to 4.49 points), with 91.6% correctness agreement.
Those comparisons use different prompt and token budgets, so they are corroborating evidence rather than model-identity proof. The primary evidence remains the July submission date, the xAI provider, and xAI's documented redirect.
There is also a separate accounting discrepancy worth checking during that rerun. The cost fields in the currently committed 8,400-row artifact sum to $0.887913 ($0.1057/1K), while the PR evaluation and leaderboard report $0.633785 ($0.0755/1K). RouterArena's compute_scores.py reproduces the former value from the committed file.
Requested remedy
Because the mismatch affects three current top-five entries and their corrected scores cannot be inferred from the saved choices, we suggest:
- Temporarily remove the current ranks for Cross-Router, vLLM-SR, and Nadir-Tumbler.
- Invite each team to rerun and resubmit with the actual model identity and current price visible to its routing procedure.
- Restore the entries after the normal RouterArena evaluation workflow scores the corrected submissions.
Small guardrails that would prevent a repeat
- Reject retired or redirected slugs during submission preflight.
- Save
requested_model, actual_model, provider, and invoked_at separately whenever the provider exposes them.
- Price the actual model rather than the requested alias.
- Snapshot the model-price table and alias map with each evaluation.
- Probe each provider/model pair once before a full run and verify the resolved identity.
OpenRouter exposes the actual model. Many providers do not. Where that field is unavailable, a large change in token counts, latency, or matched-task performance can be a warning for manual review, but should not be treated as proof by itself.
Thanks for maintaining RouterArena and for keeping its submissions inspectable. We are happy to help turn the preflight identity check and price snapshot into a small PR.
We found a model-resolution issue affecting three of RouterArena's current top five submissions. After xAI retired
grok-4-1-fast-reasoningon May 15, calls to that slug silently resolved to Grok 4.3 with low reasoning effort. RouterArena still recorded and priced those calls as Grok 4.1 Fast. The confirmed artifacts therefore combine Grok 4.3 outputs with Grok 4.1 Fast costs.This is not a harmless alias change. It substitutes a stronger, substantially more expensive model:
The Artificial Analysis values are independent estimates captured on July 15, 2026. The problem is the combination: a materially stronger model's answers were scored at the retired model's much lower price.
Upstream change
xAI's May 15 retirement notice says that after 12:00 PM PT:
RouterArena's pinned
model_cost.jsonstill assigns the retired slug Grok 4.1 Fast's $0.20/$0.50 prices.Affected leading submissions
We audited the prediction artifacts at commit
711bb4d. Where a file contains appended optimality runs, we kept the first non-optimality row for each of the 8,400 unique requests.provider=openrouterandmodel_used=x-ai/grok-4.3provider=openrouterandmodel_used=x-ai/grok-4.3model_usedand invocation timestamps.Artifacts: Cross-Router, vLLM-SR, Nadir-Tumbler, and the leaderboard at the audited commit.
This appears accidental. The teams uploaded complete submission artifacts, and the OpenRouter metadata they preserved is what exposed the mismatch.
Naive repricing shows materiality, not the corrected ranking
Holding every submitted model choice and answer fixed, then billing the affected calls at Grok 4.3's price, gives:
* Mechanical rank if all three affected entries are repriced and every unaffected score stays fixed.
Nadir-Tumbler is not directly confirmed because its artifact lacks the actual-model field. Its committed artifact also sums to a different original cost than the leaderboard reports, so this row uses the committed artifact and the leaderboard's rounded accuracy. It is an estimate, not a reproducible correction to the published score.
The recorded cost rises 30.2% for Cross-Router, 38.3% for vLLM-SR, and about 279% for Nadir-Tumbler's committed artifact. These numbers establish that the mismatch is rank-material. However, they are not corrected router scores.
Why the existing scores cannot be repaired by changing the price column
These are cost-aware routers. Grok 4.3 looked like a strong, unusually cheap arm during training, calibration, or inference-time selection because it carried Grok 4.1 Fast's price. Had the router seen the real price, it could have selected Grok far less often or chosen different models entirely.
That changes both sides of the Arena score. The final policy might lose more than the naive calculation suggests, or it might recover part of the loss by finding good substitutes. The saved routes cannot tell us which counterfactual policy each system would have learned, nor what accuracy that policy would have achieved. Naive repricing is therefore neither a corrected score nor a reliable bound. The affected submissions need to be rerun with the actual model identity and price visible to the router.
Nadir-Tumbler
Nadir-Tumbler was submitted on July 4 and its PR says the Grok calls were made through xAI. xAI had retired the requested slug on May 15 and documents the same automatic redirect for its own API, so Grok 4.3 is the expected resolved model. Unlike the OpenRouter artifacts, however, these rows do not preserve an actual-model field or timestamps. We therefore distinguish this case from the two directly confirmed ones and ask the maintainers to verify it with the submission team.
The task-matched performance is consistent with that explanation. We formed an Azure-hosted Grok 4.1 comparison bank from Azure-Model-Router and Nadir's earlier Cascade submission. Both artifacts record Azure as the provider, so they were not subject to xAI's endpoint redirect. Across 2,042 matched, scorable requests, Nadir-Tumbler scores 67.11% against 64.27% for those Azure-hosted responses, a paired shift of 2.84 points (approximate 95% interval: 1.27 to 4.42 points), with 85.5% binary correctness agreement. Across 155 matched, scorable requests shared with the confirmed OpenRouter Grok 4.3 responses, Nadir-Tumbler is nearly identical: 63.94% versus 64.06%, a -0.12-point difference (approximate 95% interval: -4.73 to 4.49 points), with 91.6% correctness agreement.
Those comparisons use different prompt and token budgets, so they are corroborating evidence rather than model-identity proof. The primary evidence remains the July submission date, the xAI provider, and xAI's documented redirect.
There is also a separate accounting discrepancy worth checking during that rerun. The
costfields in the currently committed 8,400-row artifact sum to $0.887913 ($0.1057/1K), while the PR evaluation and leaderboard report $0.633785 ($0.0755/1K). RouterArena'scompute_scores.pyreproduces the former value from the committed file.Requested remedy
Because the mismatch affects three current top-five entries and their corrected scores cannot be inferred from the saved choices, we suggest:
Small guardrails that would prevent a repeat
requested_model,actual_model,provider, andinvoked_atseparately whenever the provider exposes them.OpenRouter exposes the actual model. Many providers do not. Where that field is unavailable, a large change in token counts, latency, or matched-task performance can be a warning for manual review, but should not be treated as proof by itself.
Thanks for maintaining RouterArena and for keeping its submissions inspectable. We are happy to help turn the preflight identity check and price snapshot into a small PR.