feat(workflows): add model fallbacks to Agent, Evaluator, and Router - #7972
Merged
Merged
Conversation
Adds the type to the shared SubBlockType union, the block registry test allowlist, the tool-input exclusion set, and the docs generator's semantic type map.
The Agent block gains an ordered list of fallback models as an advanced field. The helpers normalize stored rows into execution candidates, decide which models a row may offer (credentials the block can actually supply), whether a row needs its own env-var key, and how the primary's tuning carries over: graded knobs only when the fallback declares the value, temperature and max output tokens clamped to the fallback's caps.
…t fails One handler invocation tries the primary then each fallback in order, so block retry wraps the whole chain. Falling through is as indiscriminate as retry: only a stop or a non-retryable failure ends it early. Messages are built once; hydration is cached per provider; a fallback that is blacklisted, not permitted, or cannot take the attachments is skipped. The models that failed are recorded on the block log and rendered in the trace as a Fell back from row.
…rows
Copilot and YAML writes are refused when a row names an unknown model,
sim-auto, a raw key instead of a {{ENV_VAR}} reference, or a tuning value
the model does not declare. Export keeps only reference keys, search never
rewrites a row key, and the canvas card summarizes the models.
Ordered rows (2nd choice, 3rd choice, ...) directly above Retry on fail. Each row picks a model the block can supply credentials for, an env-var reference for its key when a different provider needs one, and a tuning value only for the knobs the primary's setting cannot fill.
Executor: hydrate the primary before the secret registries settle and fork
again, re-forking per fallback provider, so file provenance stays in the
result registry; project the fall-through warn against the failed attempt's
error registry and reinstate it before a post-skip rethrow; prime a
streaming candidate's first chunk when another candidate follows, so a
tool-loop startup failure still falls back; never fall back on a
deep-research follow-up turn; strip the sim-auto identity preamble from a
named fallback's messages; treat a row key still in {{VAR}} form as no key;
stop starting candidates after an abort; record failed models on every exit,
clearing them when a retry succeeds, and only when the names project safely.
Helpers: row tuning applies only while its field is shown; the editor's row
transforms are pure functions; changing a row's model to another provider
drops the key reference; Bedrock honors NEXT_PUBLIC_BEDROCK_DEFAULT_CREDENTIALS.
Editor: legacy Combobox to match the block editor, a row's own model is never
disabled in its own picker, move controls only for two or more rows, the
shared dashed add-row button, Create Secret wording, non-reference keys never
rendered. Trace: one Failed model row per fallback. Export sanitizer uses the
strict whole-reference check.
Retry on fail used to wrap the whole fallback chain, so tries 3 with fallbacks B and C ran A, B, C three times over. A builder who lists fallbacks wants the selected model retried and the fallbacks tried once each after its last try fails, which is also how LiteLLM orders retries and fallbacks and how OpenRouter treats each model in its list. The executor keeps the retry policy. Each try is now told where it sits in it through the node metadata (`BlockNodeMetadata.retry`, with the executor's own `isFinalTry` judgment), and the Agent handler keeps the fallbacks out of the candidate list until the final try. Every earlier try runs the primary alone and lets the failure escape for the policy to replay. Blocks without fallbacks, and blocks with retry off, behave as before; other handlers ignore the field.
Review fixes: the executor judges the final try once per iteration; the
skip-warn is one helper; the fallback warn names the candidate position
rather than reusing `attempt`; the BlockLog.modelFallbacks doc matches the
final-try semantics; the viability check resolves a provider once through
the new providerRequiresFamilyCredentials; editor handlers read rows via a
ref so a keystroke in one row no longer re-renders every row, and tuning
options keep their identity across renders.
Two consistency fixes from the red-team pass: a sim-auto fallback takes
the projected system prompt rather than the raw input, and a row key is
honoured at runtime only when the block stored it as a whole {{NAME}}
reference, the one form the editor, the validator, and an export agree on.
Tests now cover the composed executor-plus-handler sequence (three tries on
the selected model, then each fallback once), the node-taking handler
signature, a fallback whose provider cannot take the attachments, the
per-provider hydration cache, a stop during a skipped candidate, the
un-primed stream when no candidate follows, a non-retryable failure on a
non-final try, a numeric tuning value, and an unresolved temperature.
A routed sim-auto primary that fails now shows in the trace as the auto identity rather than the pool model, which is the name applyAutoModelLabel exists to hide. Per-row tuning resolves against the model the builder configured, the one the editor showed the fields for, so a row's value applies under sim-auto whatever pool model was routed. A fallback on another provider no longer receives the primary's Azure, Vertex, or Bedrock fields, which only that family reads.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 36 files
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Fix all with cubic | Re-trigger cubic
A candidate stream that closes before its first chunk is a startup failure to fall through from, not an empty answer. The provider-decides tuning entry is stored as absence on the copilot path too, and a row key survives a same-provider model change only when it is a reference. The preview of a fallback list now gates its rows against the previewed version's selected model and settings rather than the live block's, and the preview value is typed instead of cast to any.
Contributor
Author
Contributor
Author
|
@cubic-dev-ai review this PR |
Contributor
@mzxchandra I have started the AI code review. It will take a few minutes to complete. |
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 36 files
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Fix all with cubic | Re-trigger cubic
The tuning doc read as if an unknown primary's values passed through. They do not: the block never shows a graded knob for a model outside the catalog, so a value stored under one is stale and the row's own field decides. A test now pins that.
Contributor
Author
Contributor
Author
|
@cubic-dev-ai review this PR |
Contributor
@mzxchandra I have started the AI code review. It will take a few minutes to complete. |
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 36 files
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Fix all with cubic | Re-trigger cubic
icecrasher321
changed the title
Collaborator
Collaborator
|
@cubic-dev-ai review this PR |
Contributor
@icecrasher321 I have started the AI code review. It will take a few minutes to complete. |
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 48 files
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Fix all with cubic | Re-trigger cubic
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds ordered fallback models to Agent, Evaluator, and Router, including legacy Router blocks. When a provider request fails, the block can try another model while preserving its prompts, inputs, and output contract.
NO_MATCHand response parsing outside model failover. A completed routing decision cannot trigger another model request.Behavior and limits
Testing
Checklist