Skip to content

Commit 86c0930

Browse files
DavertMikclaude
andcommitted
feat: Decision helper modes, cover response body with timeout
Decision request timeout now also covers reading the response body, so a stalled body fails after the configured timeout. Decision helper gets a mode option: assert (default), report (never fails, results and errors added to step comment) and skip (no requests). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent b5d2241 commit 86c0930

4 files changed

Lines changed: 147 additions & 22 deletions

File tree

‎docs/assertions.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -459,6 +459,21 @@ expected page to satisfy "success message is shown" (12%) with confidence of 70%
459459

460460
`I.decideVisually` also sends a screenshot, so it needs a model with image input. It uses `visualModel`, which is [Clef](https://openrouter.ai/cloudflare/clef) by default. Visual decisions are experimental.
461461

462+
To turn decisions off without removing the helper, set its `mode`:
463+
464+
| Mode | Behavior |
465+
|---|---|
466+
| `assert` | default, fails the step when a statement is not confirmed |
467+
| `report` | requests the model but never fails, even on API errors; results and errors are added to the step as a comment |
468+
| `skip` | does not request the model, every decision passes |
469+
470+
```js
471+
helpers: {
472+
Playwright: { url: 'http://localhost' },
473+
Decision: { mode: process.env.CI ? 'assert' : 'skip' },
474+
}
475+
```
476+
462477
A failed decision is not retried by the [retryFailedStep](/plugins/retryFailedStep) plugin: asking again would cost another request and return the same answer. Only connection errors and timeouts are retried.
463478

464479
Use decision assertions for what a page means, and built-in assertions for exact values. `I.see('Total: $42.00')` is still the right check for a price.

‎lib/ai.js‎

Lines changed: 20 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -347,27 +347,33 @@ class DecisionAI {
347347
const controller = new AbortController()
348348
const timer = setTimeout(() => controller.abort(), this.config.timeout)
349349

350-
let response
350+
let result
351351
try {
352-
response = await this.fetchImpl(this.endpoint.url, {
353-
method: 'POST',
354-
headers: { Authorization: `Bearer ${apiKey}`, 'Content-Type': 'application/json' },
355-
body: JSON.stringify({ model, state, questions }),
356-
signal: controller.signal,
357-
})
352+
let response
353+
try {
354+
response = await this.fetchImpl(this.endpoint.url, {
355+
method: 'POST',
356+
headers: { Authorization: `Bearer ${apiKey}`, 'Content-Type': 'application/json' },
357+
body: JSON.stringify({ model, state, questions }),
358+
signal: controller.signal,
359+
})
360+
} catch (err) {
361+
throw new DecisionConnectionError(`Decision model ${model} request failed: ${err.message}`)
362+
}
363+
364+
if (!response.ok) {
365+
const body = await response.text().catch(() => '')
366+
throw new Error(`Decision model ${model} responded with ${response.status}: ${body}`)
367+
}
368+
369+
result = await response.json()
358370
} catch (err) {
359371
if (controller.signal.aborted) throw new DecisionConnectionError(`Decision model ${model} did not respond in ${this.config.timeout}ms`)
360-
throw new DecisionConnectionError(`Decision model ${model} request failed: ${err.message}`)
372+
throw err
361373
} finally {
362374
clearTimeout(timer)
363375
}
364376

365-
if (!response.ok) {
366-
const body = await response.text().catch(() => '')
367-
throw new Error(`Decision model ${model} responded with ${response.status}: ${body}`)
368-
}
369-
370-
const result = await response.json()
371377
debug('Decision response', result?.answers, result?.usage)
372378

373379
const answers = result?.answers || {}

‎lib/helper/Decision.js‎

Lines changed: 52 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import Helper from '@codeceptjs/helper'
55
import Container from '../container.js'
66
import Config from '../config.js'
77
import store from '../store.js'
8+
import output from '../output.js'
89
import AssertionFailedError from '../assert/error.js'
910
import { compactAriaSnapshot } from '../aria.js'
1011
import { minifyHtml } from '../html.js'
@@ -37,8 +38,7 @@ import { DecisionAI, DecisionConnectionError } from '../ai.js'
3738
*
3839
* ## Configuration
3940
*
40-
* Decision model is configured in the `ai.decisionModel` section of the config.
41-
* The helper itself has no options:
41+
* Decision model is configured in the `ai.decisionModel` section of the config:
4242
*
4343
* ```js
4444
* ai: {
@@ -63,18 +63,45 @@ import { DecisionAI, DecisionConnectionError } from '../ai.js'
6363
* * `timeout` (default: `15000`) - request timeout in ms.
6464
* * `maxLength` (default: `12000`) - maximal length of ARIA snapshot or HTML sent to the model.
6565
*
66+
* The helper has one option:
67+
*
68+
* * `mode` (default: `assert`) - how decisions are executed:
69+
* * `assert` - request the model and fail the step when a statement is not confirmed.
70+
* * `report` - request the model but never fail, even on API errors. Results and errors are added to the step as a comment.
71+
* * `skip` - do not request the model, every decision passes.
72+
*
73+
* ```js
74+
* Decision: { mode: process.env.CI ? 'assert' : 'skip' }
75+
* ```
76+
*
6677
* Decisions work without the `--ai` flag.
6778
* Browse all decision models at [OpenRouter](https://openrouter.ai/models?output_modalities=decisions).
6879
*
6980
* ## Methods
7081
*/
82+
const MODES = ['assert', 'report', 'skip']
83+
7184
class Decision extends Helper {
7285
constructor(config = {}) {
7386
super(config)
87+
this.mode = config.mode || 'assert'
88+
if (!MODES.includes(this.mode)) throw new Error(`Unknown Decision helper mode "${this.mode}", use one of: ${MODES.join(', ')}`)
7489
this.decisionAI = new DecisionAI(Config.get('ai', {}).decisionModel)
7590
this.options = this.decisionAI.config
7691
}
7792

93+
get isAsserting() {
94+
return this.mode === 'assert'
95+
}
96+
97+
get isReporting() {
98+
return this.mode === 'report'
99+
}
100+
101+
get isSkipping() {
102+
return this.mode === 'skip'
103+
}
104+
78105
/**
79106
* Asserts that a statement (or each of statements) is true for the current page.
80107
* The decision model receives page URL, title and ARIA snapshot.
@@ -87,9 +114,10 @@ class Decision extends Helper {
87114
* ```
88115
*
89116
* @param {string|string[]} statements statement or list of statements to verify.
90-
* @returns {Promise<number|number[]>} probability of each statement.
117+
* @returns {Promise<number|number[]|undefined>} probability of each statement, `undefined` in `skip` mode.
91118
*/
92119
async decide(statements) {
120+
if (this.isSkipping) return this._skip()
93121
return this._run(async () => {
94122
const state = await this._grabState()
95123
return this._assert(statements, this.options.model, state)
@@ -107,9 +135,10 @@ class Decision extends Helper {
107135
* ```
108136
*
109137
* @param {string|string[]} statements statement or list of statements to verify.
110-
* @returns {Promise<number|number[]>} probability of each statement.
138+
* @returns {Promise<number|number[]|undefined>} probability of each statement, `undefined` in `skip` mode.
111139
*/
112140
async decideVisually(statements) {
141+
if (this.isSkipping) return this._skip()
113142
return this._run(async () => {
114143
const state = await this._grabState()
115144
const screenshot = await this._grabScreenshot()
@@ -121,10 +150,19 @@ class Decision extends Helper {
121150
})
122151
}
123152

153+
_skip() {
154+
if (store.currentStep) store.currentStep.comment = ' (skipped)'
155+
}
156+
124157
async _run(fn) {
125158
try {
126159
return await fn()
127160
} catch (err) {
161+
if (this.isReporting) {
162+
if (store.currentStep) store.currentStep.comment = ` (error: ${err.message})`
163+
output.say(` ✖ ${err.message}`, 'yellow')
164+
return
165+
}
128166
if (!(err instanceof DecisionConnectionError)) err.isTerminal = true
129167
throw err
130168
}
@@ -137,12 +175,20 @@ class Decision extends Helper {
137175
const probabilities = await this.decisionAI.decide(model, state, list)
138176

139177
const failed = []
140-
list.forEach((statement, i) => {
178+
const results = list.map((statement, i) => {
141179
const passed = probabilities[i] >= this.options.confidence
142-
this.debugSection('Decision', `${passed ? '✔' : '✖'} ${statement} (${formatProbability(probabilities[i])})`)
180+
const result = `${passed ? '✔' : '✖'} ${statement} (${formatProbability(probabilities[i])})`
181+
this.debugSection('Decision', result)
143182
if (!passed) failed.push(`"${statement}" (${formatProbability(probabilities[i])})`)
183+
return result
144184
})
145185

186+
if (this.isReporting) {
187+
if (store.currentStep) store.currentStep.comment = `\n${results.join('\n')}`
188+
results.forEach(result => output.say(` ${result}`, result.startsWith('✔') ? 'green' : 'yellow'))
189+
return Array.isArray(statements) ? probabilities : probabilities[0]
190+
}
191+
146192
if (failed.length) {
147193
const err = new AssertionFailedError(
148194
{ statements: failed.join(', '), confidence: formatProbability(this.options.confidence) },

‎test/unit/helper/Decision_test.js‎

Lines changed: 60 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,9 @@ import Decision from '../../../lib/helper/Decision.js'
66
import store from '../../../lib/store.js'
77
import Config from '../../../lib/config.js'
88

9-
function createDecision(decisionModel) {
9+
function createDecision(decisionModel, config = {}) {
1010
Config.create({ ai: { decisionModel } })
11-
return new Decision({})
11+
return new Decision(config)
1212
}
1313

1414
function fakeFetch(answers, calls, status = 200) {
@@ -159,6 +159,20 @@ describe('Decision helper', () => {
159159
expect(err.isTerminal).to.be.undefined
160160
})
161161

162+
it('times out stalled response body', async () => {
163+
decision = createDecision({ apiKey: 'secret', timeout: 50 })
164+
decision._actingHelper = () => browser
165+
decision.decisionAI.fetchImpl = async (url, { signal }) => ({
166+
ok: true,
167+
status: 200,
168+
json: () => new Promise((_, reject) => signal.addEventListener('abort', () => reject(new Error('aborted')))),
169+
})
170+
171+
const err = await decision.decide('page is loaded').catch(e => e)
172+
expect(err.message).to.include('did not respond in 50ms')
173+
expect(err.isTerminal).to.be.undefined
174+
})
175+
162176
it('marks failed decisions as not retryable', async () => {
163177
decision.decisionAI.fetchImpl = fakeFetch(noul(0.1), calls)
164178
const err = await decision.decide('cart is empty').catch(e => e)
@@ -214,4 +228,48 @@ describe('Decision helper', () => {
214228
if (key) process.env.TYPESAFE_API_KEY = key
215229
}
216230
})
231+
232+
describe('mode', () => {
233+
afterEach(() => {
234+
store.currentStep = null
235+
})
236+
237+
it('skips decisions without requests', async () => {
238+
decision = createDecision({}, { mode: 'skip' })
239+
decision.decisionAI.fetchImpl = fakeFetch(noul(0.1), calls)
240+
store.currentStep = { comment: '' }
241+
242+
expect(await decision.decide('cart is empty')).to.be.undefined
243+
expect(await decision.decideVisually('sidebar is shown')).to.be.undefined
244+
expect(calls).to.be.empty
245+
expect(store.currentStep.comment).to.include('skipped')
246+
})
247+
248+
it('reports results in step comment without failing', async () => {
249+
decision = createDecision({ apiKey: 'secret' }, { mode: 'report' })
250+
decision._actingHelper = () => browser
251+
decision.decisionAI.fetchImpl = fakeFetch(noul(0.9, 0.1), calls)
252+
store.currentStep = { comment: '' }
253+
254+
const probabilities = await decision.decide(['form has fields', 'submit enabled'])
255+
expect(probabilities).to.eql([0.9, 0.1])
256+
expect(calls).to.have.length(1)
257+
expect(store.currentStep.comment).to.include('✔ form has fields (90%)')
258+
expect(store.currentStep.comment).to.include('✖ submit enabled (10%)')
259+
})
260+
261+
it('reports api errors in step comment without failing', async () => {
262+
decision = createDecision({ apiKey: 'secret' }, { mode: 'report' })
263+
decision._actingHelper = () => browser
264+
decision.decisionAI.fetchImpl = fakeFetch({}, calls, 500)
265+
store.currentStep = { comment: '' }
266+
267+
expect(await decision.decide('page is loaded')).to.be.undefined
268+
expect(store.currentStep.comment).to.include('responded with 500')
269+
})
270+
271+
it('validates mode', () => {
272+
expect(() => createDecision({}, { mode: 'soft' })).to.throw('Unknown Decision helper mode')
273+
})
274+
})
217275
})

0 commit comments

Comments
 (0)