Add chuzom-clean: benchmark-agnostic confidence-gated cascade router#160
Closed
ypollak2 wants to merge 2 commits into
Closed
Add chuzom-clean: benchmark-agnostic confidence-gated cascade router#160ypollak2 wants to merge 2 commits into
ypollak2 wants to merge 2 commits into
Conversation
Probe 2 cheapest models (qwen3-235b + deepseek-v4-flash); ship cheapest on answer-agreement, else escalate to deepseek-v3.2. No RouterArena prompt-template matching; the single agreement threshold is calibrated on self-generated data only. SHA-256 contamination audit: 0 overlap with RA full+robustness prompts. Shared evaluator files unchanged; all models already in model_cost.json.
Contributor
Author
|
/evaluate |
…plit rows Prior commit shipped generated_result:null placeholders, causing the evaluate workflow to fail check_config_prediction_files.py's --check-generated-result validation. Ran the already-decided routing model per row (no re-probing) via the OpenRouter API to produce real generated_answer/success/token_usage for every row. Validated locally against check_config_prediction_files.py (full + robustness, both pass).
Contributor
Author
|
/evaluate |
Router Evaluation ResultsRouter: RouterArena Metrics
Evaluation completed by RouterArena automated workflow |
Contributor
Author
|
Withdrawing this submission to continue development on the router; will resubmit once it clears our internal quality bar. Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
chuzom-clean is a benchmark-agnostic, confidence-gated cascade router. It supersedes the withdrawn PR #158 with a clean-break design that routes on live model behaviour, never on RouterArena prompt templates.
How it routes
qwen/qwen3-235b-a22b-2507anddeepseek/deepseek-v4-flash.qwen3-235b). Disagree → escalate todeepseek/deepseek-v3.2.The decision uses cross-model agreement (a label-free confidence signal); the single agreement threshold was frozen on self-generated data before any RA prompt was routed.
Model pool & routing share (full split)
Compliance
Context: None,\boxed, task-header strings, etc.) in the router source.llm_evaluation/metrics.py,router_inference/compare_router_accuracy.py,llm_inference/model_inference.pyare byte-identical tomain.model_cost/edits — all three models were already in the cost table./evaluate