Commit 2d87824
feat(workflows): add model fallbacks to Agent, Evaluator, and Router (#7972)
* feat(workflow-types): register the model-fallback-list subblock type
Adds the type to the shared SubBlockType union, the block registry test
allowlist, the tool-input exclusion set, and the docs generator's semantic
type map.
* feat(agent): add the fallbackModels subblock and its shared helpers
The Agent block gains an ordered list of fallback models as an advanced
field. The helpers normalize stored rows into execution candidates, decide
which models a row may offer (credentials the block can actually supply),
whether a row needs its own env-var key, and how the primary's tuning
carries over: graded knobs only when the fallback declares the value,
temperature and max output tokens clamped to the fallback's caps.
* feat(executor): walk fallback models when the agent's provider request fails
One handler invocation tries the primary then each fallback in order, so
block retry wraps the whole chain. Falling through is as indiscriminate as
retry: only a stop or a non-retryable failure ends it early. Messages are
built once; hydration is cached per provider; a fallback that is
blacklisted, not permitted, or cannot take the attachments is skipped. The
models that failed are recorded on the block log and rendered in the trace
as a Fell back from row.
* feat(workflows): validate, sanitize, index, and label fallback model rows
Copilot and YAML writes are refused when a row names an unknown model,
sim-auto, a raw key instead of a {{ENV_VAR}} reference, or a tuning value
the model does not declare. Export keeps only reference keys, search never
rewrites a row key, and the canvas card summarizes the models.
* feat(editor): fallback models list on the Agent block
Ordered rows (2nd choice, 3rd choice, ...) directly above Retry on fail.
Each row picks a model the block can supply credentials for, an env-var
reference for its key when a different provider needs one, and a tuning
value only for the knobs the primary's setting cannot fill.
* docs(agent): document fallback models and their interplay with retry
* fix(agent): harden fallback models after pre-landing review
Executor: hydrate the primary before the secret registries settle and fork
again, re-forking per fallback provider, so file provenance stays in the
result registry; project the fall-through warn against the failed attempt's
error registry and reinstate it before a post-skip rethrow; prime a
streaming candidate's first chunk when another candidate follows, so a
tool-loop startup failure still falls back; never fall back on a
deep-research follow-up turn; strip the sim-auto identity preamble from a
named fallback's messages; treat a row key still in {{VAR}} form as no key;
stop starting candidates after an abort; record failed models on every exit,
clearing them when a retry succeeds, and only when the names project safely.
Helpers: row tuning applies only while its field is shown; the editor's row
transforms are pure functions; changing a row's model to another provider
drops the key reference; Bedrock honors NEXT_PUBLIC_BEDROCK_DEFAULT_CREDENTIALS.
Editor: legacy Combobox to match the block editor, a row's own model is never
disabled in its own picker, move controls only for two or more rows, the
shared dashed add-row button, Create Secret wording, non-reference keys never
rendered. Trace: one Failed model row per fallback. Export sanitizer uses the
strict whole-reference check.
* fix(agent): retry the selected model before falling back
Retry on fail used to wrap the whole fallback chain, so tries 3 with
fallbacks B and C ran A, B, C three times over. A builder who lists
fallbacks wants the selected model retried and the fallbacks tried once
each after its last try fails, which is also how LiteLLM orders retries
and fallbacks and how OpenRouter treats each model in its list.
The executor keeps the retry policy. Each try is now told where it sits
in it through the node metadata (`BlockNodeMetadata.retry`, with the
executor's own `isFinalTry` judgment), and the Agent handler keeps the
fallbacks out of the candidate list until the final try. Every earlier
try runs the primary alone and lets the failure escape for the policy to
replay. Blocks without fallbacks, and blocks with retry off, behave as
before; other handlers ignore the field.
* fix(agent): address the pre-landing review of retry-then-fallback
Review fixes: the executor judges the final try once per iteration; the
skip-warn is one helper; the fallback warn names the candidate position
rather than reusing `attempt`; the BlockLog.modelFallbacks doc matches the
final-try semantics; the viability check resolves a provider once through
the new providerRequiresFamilyCredentials; editor handlers read rows via a
ref so a keystroke in one row no longer re-renders every row, and tuning
options keep their identity across renders.
Two consistency fixes from the red-team pass: a sim-auto fallback takes
the projected system prompt rather than the raw input, and a row key is
honoured at runtime only when the block stored it as a whole {{NAME}}
reference, the one form the editor, the validator, and an export agree on.
Tests now cover the composed executor-plus-handler sequence (three tries on
the selected model, then each fallback once), the node-taking handler
signature, a fallback whose provider cannot take the attachments, the
per-provider hydration cache, a stop during a skipped candidate, the
un-primed stream when no candidate follows, a non-retryable failure on a
non-final try, a numeric tuning value, and an unresolved temperature.
* fix(agent): close the adversarial review's fallback findings
A routed sim-auto primary that fails now shows in the trace as the auto
identity rather than the pool model, which is the name applyAutoModelLabel
exists to hide. Per-row tuning resolves against the model the builder
configured, the one the editor showed the fields for, so a row's value
applies under sim-auto whatever pool model was routed. A fallback on
another provider no longer receives the primary's Azure, Vertex, or
Bedrock fields, which only that family reads.
* docs(agent): name the fallback row cap and the empty tuning field
* fix(agent): address the first review round on the fallback models PR
A candidate stream that closes before its first chunk is a startup
failure to fall through from, not an empty answer. The provider-decides
tuning entry is stored as absence on the copilot path too, and a row key
survives a same-provider model change only when it is a reference. The
preview of a fallback list now gates its rows against the previewed
version's selected model and settings rather than the live block's, and
the preview value is typed instead of cast to any.
* docs(agent): say which side of a fallback the catalog may not know
The tuning doc read as if an unknown primary's values passed through. They
do not: the block never shows a graded knob for a model outside the
catalog, so a value stored under one is stale and the row's own field
decides. A test now pins that.
* feat(workflows): share model fallbacks across LLM blocks
---------
Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>1 parent 0b7c6ae commit 2d87824
48 files changed
Lines changed: 4221 additions & 58 deletions
File tree
- apps
- docs/content/docs/workflows/blocks
- sim
- app/workspace/[workspaceId]
- logs/components/log-details/components/trace-view
- w
- [workflowId]/components
- panel/components/editor/components/sub-block
- components
- model-fallback-list
- workflow-block
- components/preview/components/preview-workflow/components/block
- blocks
- blocks
- executor
- execution
- handlers
- agent
- evaluator
- router
- utils
- lib
- logs
- execution/trace-spans
- workflows
- blocks
- credentials
- editing
- search-replace
- subblocks
- tools
- packages/workflow-types/src
- scripts
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
28 | | - | |
| 28 | + | |
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
| |||
87 | 87 | | |
88 | 88 | | |
89 | 89 | | |
| 90 | + | |
| 91 | + | |
90 | 92 | | |
91 | 93 | | |
92 | 94 | | |
| |||
148 | 150 | | |
149 | 151 | | |
150 | 152 | | |
| 153 | + | |
151 | 154 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
35 | 41 | | |
36 | 42 | | |
37 | 43 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
38 | 44 | | |
39 | 45 | | |
40 | 46 | | |
| |||
Lines changed: 3 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
696 | 696 | | |
697 | 697 | | |
698 | 698 | | |
| 699 | + | |
| 700 | + | |
| 701 | + | |
699 | 702 | | |
700 | 703 | | |
701 | 704 | | |
| |||
Lines changed: 1 addition & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
| |||
Lines changed: 1 addition & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
0 commit comments