Skip to content

Commit 2d87824

Browse files
feat(workflows): add model fallbacks to Agent, Evaluator, and Router (#7972)
* feat(workflow-types): register the model-fallback-list subblock type Adds the type to the shared SubBlockType union, the block registry test allowlist, the tool-input exclusion set, and the docs generator's semantic type map. * feat(agent): add the fallbackModels subblock and its shared helpers The Agent block gains an ordered list of fallback models as an advanced field. The helpers normalize stored rows into execution candidates, decide which models a row may offer (credentials the block can actually supply), whether a row needs its own env-var key, and how the primary's tuning carries over: graded knobs only when the fallback declares the value, temperature and max output tokens clamped to the fallback's caps. * feat(executor): walk fallback models when the agent's provider request fails One handler invocation tries the primary then each fallback in order, so block retry wraps the whole chain. Falling through is as indiscriminate as retry: only a stop or a non-retryable failure ends it early. Messages are built once; hydration is cached per provider; a fallback that is blacklisted, not permitted, or cannot take the attachments is skipped. The models that failed are recorded on the block log and rendered in the trace as a Fell back from row. * feat(workflows): validate, sanitize, index, and label fallback model rows Copilot and YAML writes are refused when a row names an unknown model, sim-auto, a raw key instead of a {{ENV_VAR}} reference, or a tuning value the model does not declare. Export keeps only reference keys, search never rewrites a row key, and the canvas card summarizes the models. * feat(editor): fallback models list on the Agent block Ordered rows (2nd choice, 3rd choice, ...) directly above Retry on fail. Each row picks a model the block can supply credentials for, an env-var reference for its key when a different provider needs one, and a tuning value only for the knobs the primary's setting cannot fill. * docs(agent): document fallback models and their interplay with retry * fix(agent): harden fallback models after pre-landing review Executor: hydrate the primary before the secret registries settle and fork again, re-forking per fallback provider, so file provenance stays in the result registry; project the fall-through warn against the failed attempt's error registry and reinstate it before a post-skip rethrow; prime a streaming candidate's first chunk when another candidate follows, so a tool-loop startup failure still falls back; never fall back on a deep-research follow-up turn; strip the sim-auto identity preamble from a named fallback's messages; treat a row key still in {{VAR}} form as no key; stop starting candidates after an abort; record failed models on every exit, clearing them when a retry succeeds, and only when the names project safely. Helpers: row tuning applies only while its field is shown; the editor's row transforms are pure functions; changing a row's model to another provider drops the key reference; Bedrock honors NEXT_PUBLIC_BEDROCK_DEFAULT_CREDENTIALS. Editor: legacy Combobox to match the block editor, a row's own model is never disabled in its own picker, move controls only for two or more rows, the shared dashed add-row button, Create Secret wording, non-reference keys never rendered. Trace: one Failed model row per fallback. Export sanitizer uses the strict whole-reference check. * fix(agent): retry the selected model before falling back Retry on fail used to wrap the whole fallback chain, so tries 3 with fallbacks B and C ran A, B, C three times over. A builder who lists fallbacks wants the selected model retried and the fallbacks tried once each after its last try fails, which is also how LiteLLM orders retries and fallbacks and how OpenRouter treats each model in its list. The executor keeps the retry policy. Each try is now told where it sits in it through the node metadata (`BlockNodeMetadata.retry`, with the executor's own `isFinalTry` judgment), and the Agent handler keeps the fallbacks out of the candidate list until the final try. Every earlier try runs the primary alone and lets the failure escape for the policy to replay. Blocks without fallbacks, and blocks with retry off, behave as before; other handlers ignore the field. * fix(agent): address the pre-landing review of retry-then-fallback Review fixes: the executor judges the final try once per iteration; the skip-warn is one helper; the fallback warn names the candidate position rather than reusing `attempt`; the BlockLog.modelFallbacks doc matches the final-try semantics; the viability check resolves a provider once through the new providerRequiresFamilyCredentials; editor handlers read rows via a ref so a keystroke in one row no longer re-renders every row, and tuning options keep their identity across renders. Two consistency fixes from the red-team pass: a sim-auto fallback takes the projected system prompt rather than the raw input, and a row key is honoured at runtime only when the block stored it as a whole {{NAME}} reference, the one form the editor, the validator, and an export agree on. Tests now cover the composed executor-plus-handler sequence (three tries on the selected model, then each fallback once), the node-taking handler signature, a fallback whose provider cannot take the attachments, the per-provider hydration cache, a stop during a skipped candidate, the un-primed stream when no candidate follows, a non-retryable failure on a non-final try, a numeric tuning value, and an unresolved temperature. * fix(agent): close the adversarial review's fallback findings A routed sim-auto primary that fails now shows in the trace as the auto identity rather than the pool model, which is the name applyAutoModelLabel exists to hide. Per-row tuning resolves against the model the builder configured, the one the editor showed the fields for, so a row's value applies under sim-auto whatever pool model was routed. A fallback on another provider no longer receives the primary's Azure, Vertex, or Bedrock fields, which only that family reads. * docs(agent): name the fallback row cap and the empty tuning field * fix(agent): address the first review round on the fallback models PR A candidate stream that closes before its first chunk is a startup failure to fall through from, not an empty answer. The provider-decides tuning entry is stored as absence on the copilot path too, and a row key survives a same-provider model change only when it is a reference. The preview of a fallback list now gates its rows against the previewed version's selected model and settings rather than the live block's, and the preview value is typed instead of cast to any. * docs(agent): say which side of a fallback the catalog may not know The tuning doc read as if an unknown primary's values passed through. They do not: the block never shows a graded knob for a model outside the catalog, so a value stored under one is stale and the row's own field decides. A test now pins that. * feat(workflows): share model fallbacks across LLM blocks --------- Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>
1 parent 0b7c6ae commit 2d87824

48 files changed

Lines changed: 4221 additions & 58 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

apps/docs/content/docs/workflows/blocks/agent.mdx

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ Answer in two sentences, cite the doc you used, and never guess a price.
2525

2626
### Model
2727

28-
The model that runs the step. Defaults to `claude-sonnet-4-6`. Type or pick any model from OpenAI, Anthropic, Google, xAI, Groq, Cerebras, DeepSeek, Azure, AWS Bedrock, Google Vertex, or OpenRouter, or a local model through Ollama or VLLM.
28+
The model that runs the step. Defaults to `claude-sonnet-5`. Type or pick any model from OpenAI, Anthropic, Google, xAI, Groq, Cerebras, DeepSeek, Azure, AWS Bedrock, Google Vertex, or OpenRouter, or a local model through Ollama or VLLM.
2929

3030
For a custom cloud deployment, enter its provider prefix and model ID: `azure/my-deployment`, `azure-anthropic/my-deployment`, `bedrock/my-inference-profile`, or `vertex/my-gemini-model`. The prefix selects the provider and shows its credential fields even when the ID is absent from the catalog. Bedrock accepts full inference profile ARNs after `bedrock/`; Vertex uses the Gemini API and accepts Google model resource names. The deployment must support the selected provider's API. Custom IDs have no catalog pricing or token limits.
3131

@@ -87,6 +87,8 @@ Some settings live under advanced, or appear only for models that support them:
8787
- **Reasoning effort / Thinking level.** For models with extended reasoning, how much the model thinks before answering. Higher is more thorough but slower and costs more tokens.
8888
- **Prompt caching.** For Anthropic Claude models, reuses the system prompt and tool definitions between runs instead of re-reading them every time. Cached input costs a tenth of the normal rate, but writing the cache costs 1.25x, so leave it off for one-off runs and turn it on when the same agent runs repeatedly. The cache covers a prefix only if it reaches 1,024 tokens (2,048 on Haiku) — below that Anthropic ignores it and nothing changes. Entries expire after five minutes of no use.
8989
- **API key.** Your key for the chosen provider. Hidden on hosted Sim, which supplies one.
90+
- **Fallback models.** An ordered list of up to five models to try when the request to the selected model fails, whether the provider is overloaded, rate-limited, or down. Sim tries the 2nd choice, then the 3rd, and so on, once each, and `<agent.model>` reports the model that answered. On hosted Sim, hosted models use your workspace's BYOK or platform credentials; local and self-hosted installations may still require a key. A model that needs its own key takes it from a workspace environment variable you pick on the row; a model on the same provider as the selected model reuses the block's key. A stored row key stops applying when its key field is hidden. Providers that require family-specific credentials, such as Vertex, can only be fallbacks for a selected model of the same family. The Auto model cannot be a fallback. A fallback runs with the selected model's settings where its provider accepts them: temperature and max output tokens are clamped to the fallback's limits, and when the fallback has a reasoning effort, thinking level, or verbosity setting that the selected model's value does not fit, the row shows that field so you can pick a value for it; leave it empty and the provider's default applies.
91+
- **Retry on fail.** Retries the selected model after a failure, up to a maximum number of tries with a wait between them. When its tries run out, the fallback models are tried in order, once each, with no wait before the first of them. A fallback is never retried. A failure that happens after the model already called a tool runs that conversation again on the next try or the next model, so keep fallbacks and retry off for agents whose tools must not repeat.
9092

9193
OpenAI and Gemini cache automatically at no extra cost and need no setting; their discount is already reflected in what you are charged.
9294

@@ -148,4 +150,5 @@ The Agent reads the message from Start with `<start.input>` and returns a result
148150
<FAQ items={[
149151
{ question: "How does max output tokens work with Anthropic models?", answer: "The Agent block uses each Anthropic model's full max output token limit by default (for example, 64,000 tokens). You can override this with the Max Output Tokens setting. For non-streaming requests that exceed the SDK's internal threshold, the provider automatically uses internal streaming to avoid timeouts." },
150152
{ question: "Can I use the Agent block with a custom or self-hosted model?", answer: "Yes. Use any Ollama or VLLM-compatible model by typing the model name directly into the model combobox, as long as it exposes a compatible API endpoint." },
153+
{ question: "What happens when my model's provider is down?", answer: "Add fallback models under Additional fields. When the request to the selected model fails, Sim tries the 2nd choice, then the 3rd, in order, and the block succeeds if any of them answers. The log detail shows the model that answered and the models it fell back from. Turn on Retry on fail as well to give the selected model a few tries first; the fallbacks are tried once each after its last try fails." },
151154
]} />

apps/docs/content/docs/workflows/blocks/evaluator.mdx

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,12 @@ The content to score. Usually an earlier output like `<agent.content>`. Structur
3232

3333
The model that does the scoring, defaulting to `claude-sonnet-4-6`. Stronger reasoning models give more consistent scores. Type or pick any supported model. **Temperature** and a **System Prompt** are available under advanced, and on hosted Sim the API key is supplied for you.
3434

35+
### Fallback models
36+
37+
Under **Additional fields**, add up to five models to try in order when a model request fails. With **Retry on fail** enabled, the selected model exhausts its tries first; each fallback is then tried once. Every attempt uses the same content, metrics, and response schema. `<evaluator.model>` reports the model that answered.
38+
39+
Hosted models on hosted Sim use workspace BYOK or platform credentials. On local or self-hosted installations, a fallback on another provider may need a secret selected on its row; same-provider fallbacks reuse the selected model's key. The picker shows supported tuning fields when the selected model's settings cannot be inherited. Auto cannot be a fallback.
40+
3541
## Outputs
3642

3743
The Evaluator returns a number for each metric, read by the metric's lowercase name:

apps/docs/content/docs/workflows/blocks/router.mdx

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,12 @@ Each route is a **title** and a **description** of when to choose it ("Route her
3535

3636
The model that makes the decision, defaulting to `claude-sonnet-4-6`. Stronger reasoning models route more accurately; a faster, cheaper model is fine when the routes are clearly distinct. Type or pick any supported model, or a local one through Ollama or VLLM. On hosted Sim the API key is supplied for you.
3737

38+
### Fallback models
39+
40+
Under **Additional fields**, add up to five models to try in order when a model request fails. With **Retry on fail** enabled, the selected model exhausts its tries first; each fallback is then tried once. Every attempt uses the same context and route definitions. `<router.model>` reports the model that answered. This also works for existing legacy Router blocks.
41+
42+
Hosted models on hosted Sim use workspace BYOK or platform credentials. On local or self-hosted installations, a fallback on another provider may need a secret selected on its row; same-provider fallbacks reuse the selected model's key. Auto cannot be a fallback. A completed `NO_MATCH` decision still takes the error path; it does not trigger another model request.
43+
3844
## Outputs
3945

4046
| Output | What it is |

apps/sim/app/workspace/[workspaceId]/logs/components/log-details/components/trace-view/trace-view.tsx

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -696,6 +696,9 @@ const TraceDetailPane = memo(function TraceDetailPane({ span }: { span: TraceSpa
696696
if (span.tries !== undefined) metaEntries.push({ label: 'Tries', value: String(span.tries) })
697697
if (span.provider) metaEntries.push({ label: 'Provider', value: span.provider })
698698
if (span.model) metaEntries.push({ label: 'Model', value: span.model })
699+
for (const failedModel of span.modelFallbacks ?? []) {
700+
metaEntries.push({ label: 'Failed model', value: failedModel })
701+
}
699702
if (span.finishReason) metaEntries.push({ label: 'Finish reason', value: span.finishReason })
700703
const ttftFormatted = formatTtft(span.ttft)
701704
if (ttftFormatted) metaEntries.push({ label: 'TTFT', value: ttftFormatted })

apps/sim/app/workspace/[workspaceId]/w/[workflowId]/components/panel/components/editor/components/sub-block/components/index.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ export { LongInput } from './long-input'
1818
export { McpDynamicArgs } from './mcp-dynamic-args'
1919
export { McpServerSelector, McpToolSelector } from './mcp-server-modal'
2020
export { MessagesInput } from './messages-input'
21+
export { ModelFallbackList } from './model-fallback-list'
2122
export { maskSecretText, PASSWORD_MASKED_SUBBLOCK_TYPES } from './password-mask'
2223
export { ResponseFormat } from './response'
2324
export { ScheduleInfo } from './schedule-info'
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
export { ModelFallbackList } from './model-fallback-list'
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,275 @@
1+
/**
2+
* @vitest-environment jsdom
3+
*/
4+
import { act } from 'react'
5+
import { createRoot } from 'react-dom/client'
6+
import { renderToStaticMarkup } from 'react-dom/server'
7+
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
8+
import {
9+
getDeploymentShape,
10+
resetDeploymentShape,
11+
resolveDeploymentShape,
12+
seedDeploymentShape,
13+
} from '@/lib/core/config/deployment-shape'
14+
15+
const { subBlockValues, mockSetValue } = vi.hoisted(() => ({
16+
subBlockValues: {
17+
model: 'claude-sonnet-5' as string,
18+
fallbackModels: [] as Array<{
19+
id: string
20+
model: string
21+
apiKey?: string
22+
reasoningEffort?: string
23+
}>,
24+
},
25+
mockSetValue: vi.fn(),
26+
}))
27+
28+
vi.mock('next/navigation', () => ({
29+
useParams: () => ({ workspaceId: 'workspace-1' }),
30+
}))
31+
32+
vi.mock('@sim/emcn', () => ({
33+
Chip: ({
34+
children,
35+
disabled,
36+
'aria-label': ariaLabel,
37+
}: {
38+
children?: React.ReactNode
39+
disabled?: boolean
40+
'aria-label'?: string
41+
}) => (
42+
<button type='button' disabled={disabled} aria-label={ariaLabel}>
43+
{children}
44+
</button>
45+
),
46+
ChipCombobox: ({
47+
options,
48+
value,
49+
placeholder,
50+
}: {
51+
options: Array<{ value: string; label: string; disabled?: boolean }>
52+
value?: string
53+
placeholder?: string
54+
}) => (
55+
<div data-combobox={placeholder} data-value={value}>
56+
{options.map((option) => (
57+
<span key={option.value} data-disabled={option.disabled ? 'true' : undefined}>
58+
{option.label}
59+
</span>
60+
))}
61+
</div>
62+
),
63+
ChipDropdown: ({
64+
options,
65+
value,
66+
placeholder,
67+
}: {
68+
options: Array<{ value: string; label: string }>
69+
value?: string
70+
placeholder?: string
71+
}) => (
72+
<div data-combobox={placeholder} data-value={value}>
73+
{options.map((option) => (
74+
<span key={option.value}>{option.label}</span>
75+
))}
76+
</div>
77+
),
78+
Label: ({ children }: { children?: React.ReactNode }) => <span>{children}</span>,
79+
Tooltip: {
80+
Root: ({ children }: { children?: React.ReactNode }) => <>{children}</>,
81+
Trigger: ({ children }: { children?: React.ReactNode }) => <>{children}</>,
82+
Content: () => null,
83+
},
84+
}))
85+
86+
vi.mock('@sim/emcn/icons', () => ({
87+
ChevronDown: () => null,
88+
ChevronUp: () => null,
89+
Plus: () => null,
90+
Trash: () => null,
91+
}))
92+
93+
vi.mock(
94+
'@/app/workspace/[workspaceId]/w/[workflowId]/components/panel/components/editor/components/sub-block/hooks/use-sub-block-value',
95+
() => ({
96+
useSubBlockValue: (_blockId: string, subBlockId: string) => [
97+
subBlockId === 'model' || subBlockId === 'fallbackModels' ? subBlockValues[subBlockId] : null,
98+
mockSetValue,
99+
],
100+
})
101+
)
102+
103+
vi.mock('@/hooks/queries/environment', () => ({
104+
usePersonalEnvironment: () => ({ data: { PERSONAL_KEY: 'x' } }),
105+
useWorkspaceEnvironment: () => ({
106+
data: { workspace: { OPENROUTER_API_KEY: 'x' }, personal: {}, conflicts: [] },
107+
}),
108+
}))
109+
110+
vi.mock('@/hooks/use-permission-config', () => ({
111+
usePermissionConfig: () => ({ isModelUsable: (model: string) => model !== 'denied-model' }),
112+
}))
113+
114+
vi.mock('@/hooks/use-settings-navigation', () => ({
115+
useSettingsNavigation: () => ({ navigateToSettings: vi.fn() }),
116+
}))
117+
118+
vi.mock('@/lib/credentials/client-state', () => ({
119+
writePendingCredentialCreateRequest: vi.fn(),
120+
}))
121+
122+
vi.mock('@/stores/providers/store', () => ({
123+
useProvidersStore: (selector: (state: { providers: object }) => unknown) =>
124+
selector({ providers: {} }),
125+
}))
126+
127+
vi.mock('@/blocks/utils', () => ({
128+
shouldRequireApiKeyForModel: (model: string) =>
129+
model.startsWith('openrouter/') || (model.startsWith('gpt') && !getDeploymentShape().hosted),
130+
getModelOptions: () => [
131+
{ id: 'claude-sonnet-5', label: 'claude-sonnet-5' },
132+
{ id: 'gpt-5', label: 'gpt-5' },
133+
{ id: 'denied-model', label: 'denied-model' },
134+
{ id: 'openrouter/x', label: 'openrouter/x' },
135+
{ id: 'sim-auto', label: 'Auto' },
136+
],
137+
}))
138+
139+
vi.mock('@/lib/workflows/blocks/fallback-models', async (importOriginal) => {
140+
const actual = await importOriginal<typeof import('@/lib/workflows/blocks/fallback-models')>()
141+
return {
142+
...actual,
143+
isViableFallbackModel: (model: string, primary: string) =>
144+
model !== 'sim-auto' && model !== primary,
145+
getFallbackTuningKnobsToShow: (model: string) => (model === 'gpt-5' ? ['reasoningEffort'] : []),
146+
getTuningOptionsForModel: (model: string, knob: string) =>
147+
model === 'gpt-5' && knob === 'reasoningEffort' ? ['auto', 'low', 'high'] : null,
148+
}
149+
})
150+
151+
import { ModelFallbackList } from '@/app/workspace/[workspaceId]/w/[workflowId]/components/panel/components/editor/components/sub-block/components/model-fallback-list/model-fallback-list'
152+
153+
beforeEach(() => {
154+
vi.stubGlobal('IS_REACT_ACT_ENVIRONMENT', true)
155+
seedDeploymentShape({ ...resolveDeploymentShape(), hosted: true })
156+
})
157+
158+
afterEach(() => {
159+
resetDeploymentShape()
160+
vi.unstubAllGlobals()
161+
})
162+
163+
function render(extra: Partial<React.ComponentProps<typeof ModelFallbackList>> = {}) {
164+
return renderToStaticMarkup(
165+
<ModelFallbackList blockId='block-1' subBlockId='fallbackModels' disabled={false} {...extra} />
166+
)
167+
}
168+
169+
describe('ModelFallbackList', () => {
170+
beforeEach(() => {
171+
subBlockValues.model = 'claude-sonnet-5'
172+
subBlockValues.fallbackModels = []
173+
mockSetValue.mockReset()
174+
})
175+
176+
it('renders only the add affordance when nothing is configured', () => {
177+
const html = render()
178+
expect(html).toContain('Add fallback model')
179+
expect(html).not.toContain('choice')
180+
})
181+
182+
it('labels rows as ordinal choices and offers viable, permitted models', () => {
183+
subBlockValues.fallbackModels = [
184+
{ id: 'r1', model: 'gpt-5' },
185+
{ id: 'r2', model: '' },
186+
]
187+
const html = render()
188+
expect(html).toContain('2nd choice')
189+
expect(html).toContain('3rd choice')
190+
expect(html).not.toContain('Auto')
191+
expect(html).not.toContain('denied-model')
192+
/** The primary is never offered. A model another row holds is disabled there, never in its own row. */
193+
expect(html).not.toContain('>claude-sonnet-5<')
194+
expect(html.match(/data-disabled="true">gpt-5</g)).toHaveLength(1)
195+
expect(html.match(/>gpt-5</g)).toHaveLength(2)
196+
expect(html).toContain('aria-label="Move up"')
197+
})
198+
199+
it('renders no move controls for a single row and never shows a non-reference key', () => {
200+
subBlockValues.fallbackModels = [
201+
{ id: 'r1', model: 'openrouter/x', apiKey: 'sk-raw-through-socket' },
202+
]
203+
const html = render()
204+
expect(html).not.toContain('aria-label="Move up"')
205+
expect(html).not.toContain('sk-raw-through-socket')
206+
expect(html).toContain('data-combobox="Select a secret" data-value=""')
207+
})
208+
209+
it('asks for an environment variable only when the row model needs its own key', () => {
210+
subBlockValues.fallbackModels = [{ id: 'r1', model: 'gpt-5' }]
211+
expect(render()).not.toContain('data-combobox="Select a secret"')
212+
213+
subBlockValues.fallbackModels = [
214+
{ id: 'r1', model: 'openrouter/x', apiKey: '{{OPENROUTER_API_KEY}}' },
215+
]
216+
const html = render()
217+
expect(html).toContain('data-combobox="Select a secret"')
218+
expect(html).toContain('data-value="{{OPENROUTER_API_KEY}}"')
219+
expect(html).toContain('OPENROUTER_API_KEY')
220+
expect(html).toContain('Create Secret')
221+
})
222+
223+
it('updates key visibility when hosted context arrives after mount, without rewriting the rows', async () => {
224+
seedDeploymentShape({ ...resolveDeploymentShape(), hosted: false })
225+
subBlockValues.fallbackModels = [{ id: 'r1', model: 'gpt-5' }]
226+
const container = document.createElement('div')
227+
const root = createRoot(container)
228+
try {
229+
await act(async () => {
230+
root.render(<ModelFallbackList blockId='block-1' subBlockId='fallbackModels' />)
231+
})
232+
expect(container.querySelector('[data-combobox="Select a secret"]')).not.toBeNull()
233+
234+
await act(async () => {
235+
seedDeploymentShape({ ...resolveDeploymentShape(), hosted: true })
236+
})
237+
expect(container.querySelector('[data-combobox="Select a secret"]')).toBeNull()
238+
expect(mockSetValue).not.toHaveBeenCalled()
239+
} finally {
240+
await act(async () => root.unmount())
241+
}
242+
})
243+
244+
it('shows a tuning field only for the knobs the helper says need one', () => {
245+
subBlockValues.fallbackModels = [
246+
{ id: 'r1', model: 'gpt-5', reasoningEffort: 'low' },
247+
{ id: 'r2', model: 'openrouter/x' },
248+
]
249+
const html = render()
250+
expect(html).toContain('data-combobox="Select reasoning effort" data-value="low"')
251+
expect(html.match(/Select reasoning effort/g)).toHaveLength(1)
252+
expect(html).not.toContain('Thinking level')
253+
})
254+
255+
it('gates a preview against the previewed primary, not the live block', () => {
256+
/** The live block selects claude-sonnet-5; the previewed version selected gpt-5. */
257+
const html = render({
258+
isPreview: true,
259+
previewValue: [{ id: 'r1', model: 'openrouter/x' }],
260+
previewPrimary: { model: 'gpt-5' },
261+
})
262+
expect(html).not.toContain('>gpt-5<')
263+
expect(html).toContain('>claude-sonnet-5<')
264+
expect(html).not.toContain('Add fallback model')
265+
})
266+
267+
it('disables the add affordance at the cap', () => {
268+
subBlockValues.fallbackModels = Array.from({ length: 5 }, (_, i) => ({
269+
id: `r${i}`,
270+
model: `m-${i}`,
271+
}))
272+
const html = render()
273+
expect(html).toMatch(/<button[^>]*disabled=""[^>]*>Add fallback model/)
274+
})
275+
})

0 commit comments

Comments
 (0)