diff --git a/bench/README.md b/bench/README.md index f67e7269..c50a5143 100644 --- a/bench/README.md +++ b/bench/README.md @@ -88,7 +88,8 @@ front), the WebMCP flag on, and its own `XWEBMCP_HOME`, so trials share nothing. and gives the agent `xwebmcp mcp` (`list_tools`, `invoke_tool`) and Chrome DevTools MCP for everything else (`allowlist.local.txt`, `mcp.local.json`, `prompts/system.local.md`). Chrome runs headed; `--headless` is for machines without a display, and sites block it more often. -`--stealth` and signed-in domains are Kernel-only. +`--stealth` and signed-in intents are Kernel-only; a checkout intent's guard loads as a second +unpacked extension. Local runs are a different setup from Kernel runs, and the report says so in its header. The native messaging host manifest is written into each trial's profile, which Chromium reads on @@ -116,8 +117,9 @@ interleaved across intents; a pair's two trials run back to back in one slot. ### A trial -1. Create a browser on the start URL, tagged with the run, arm and intent. If the page is still - loading after 15s, navigate again; then wait up to 15s for `load`. +1. Create a browser on the start URL, tagged with the run, arm and intent, with the intent's + submission guard loaded if it has one. If the page is still loading after 15s, navigate + again; then wait up to 15s for `load`. 2. Check the loaded URL matches the entry tool's URL patterns, in both arms. 3. Treatment: register the batch, confirm the `entry_tool` is live on the tab, and on a signed-in domain run the sign-in check. Control: on a signed-in domain, register the batch, run the same @@ -125,9 +127,10 @@ interleaved across intents; a pair's two trials run back to back in one slot. arms record the page's own WebMCP tools, which both can use. 4. Wait until 10s after load in both arms, then launch the agent. The clock runs to `agent_settled`; the runner stops the agent at the time, turn or spend cap. -5. Grade the answer, delete the browser and append a row to `rows.jsonl`. Each attempt keeps its - own directory under `trials//` (`p1-control`, then `p1-control-2` for a rerun), named in - the row's `trial_dir`. +5. On a guarded intent, read the guard's count of blocked submit attempts. Grade the answer, + delete the browser and append a row to `rows.jsonl`. Each attempt keeps its own directory + under `trials//` (`p1-control`, then `p1-control-2` for a rerun), named in the row's + `trial_dir`. A setup failure gets up to three browsers before the trial is `aborted` (a signed-out profile gets one). A trial is also `aborted` when the agent started without exactly the allowlisted tools @@ -156,6 +159,20 @@ and deletes every browser. - `stage` is `find`, `inspect`, `compare`, `prepare` or `gap`. A `gap` intent is one no tool covers; it names no tools and serves as the placebo check. +- A `prepare` intent walks a site's flow part way, such as putting a fare in Delta's cart and + stopping on Trip Summary. Treatment also gets the domain's `actions`, its tools that change + site state (see below). `check` grades where the agent left the browser, read from the page in + both arms and never through the tools under test: its `rules` see the page's `url`, `title` + and visible `text`; `extract` maps names to regexes whose first group is read from that text + as `extracted.` (an amount such as `$1,234.50` becomes a number); `fields` maps names to + CSS selectors of inputs, read as `fields.`, with `missing` and `invalid` lists of names. + `same_as_answer` lists `{answer, end_state}` path pairs whose values must match, so the answer + reports what is on the page. +- `allow` (prepare intents only) lets the agent do more than the system prompt allows, with the + same sentences added to both arms' prompts: `checkout` to add to the cart and go through + checkout up to, never including, the button that pays or books, and `payment_details` to enter + the details the task gives, which must be test data. An intent that allows `checkout` needs a + `guard`, and see [Checkout intents](#checkout-intents). - `tools` lists kernel/xwebmcp refs (or full `sha`s) to register from. Treatment gets every tool `registry.json` lists for the domain at those commits whose file declares `readOnlyHint: true`; later sources replace earlier tools of the same name, so a PR to an @@ -174,21 +191,63 @@ and deletes every browser. "issues[]", "same": true}` is for one right answer that changes too often to pin in rules, and counts both-correct pairs whose answers differ as sets. Leave it off intents with many right answers. +- `opt_in` (optional) is a reason the intent runs only when named with `--intent`, such as a + checkout intent not yet rehearsed on the live site. `tasks//domain.json` holds per-domain settings: - `exclude`: read-only tools always left out, each with a reason; - `needs_sign_in`: read-only tools that need an account, left out unless the domain has a profile; +- `actions`: tools that change site state that are safe to give the domain's `prepare` intents, + each with a reason, such as adding to a cart or choosing a fare. Tools that sign in, pay, book + or send stay out; +- `checkout_actions`: like `actions`, but only for intents that allow `checkout`, such as tools + that move through checkout steps or hold a seat; - `profile` and `signed_in_check`: see below; - `instructions`: text added to every trial prompt on the domain, in both arms. The system prompt forbids adding to a cart unless the task allows it, so a domain that needs carts says so here. amctheatres.com uses it to name the step that creates a cart, since an AMC cart holds seats. +### Checkout intents + +A `guard` is built into a small Chrome extension, uploaded to Kernel under a name taken from its +content and loaded in both arms' browsers. Its script runs in the page's own world, in every +frame, from document start, on the guard's `hosts` (Chrome match patterns): + +- clicks on buttons and links whose text matches `buttons` (case-insensitive) are blocked; +- with `form_submit`, so are clicks on a form's submit buttons, form submission, `form.submit()` and + `requestSubmit()`, and Enter in a form field (not counted, since agents press it to pick + address suggestions); +- clicks on anything inside an element matching `selectors` (CSS), whatever its text, are blocked; +- with `requests`, a declarativeNetRequest rule blocks fetch, XHR and beacon requests whose URL + matches it (RE2 syntax), from any frame or world; +- with `form_submit`, a declarativeNetRequest rule also blocks POST navigations started on the + guard's hosts, which catches `form.submit()` called from Playwright's `page.evaluate`: that runs + in an isolated world, where the page-world overrides of `submit()` are not installed. + +Each blocked click or submission in the page is counted in a cookie on that host. After the trial +the runner reads the cookies of the whole browser, so an attempt counts even when the agent then +left the page, and a trial with any is `crashed`: the agent broke the rule against submitting. +Requests and navigations the network rules block are not counted. This is defense in depth, not a +guarantee, so checkout intents also use test data: a test card that cannot be charged, or a +signed-in test account that cannot pay (see "Signed-in trials"). Rehearse a new checkout intent +on the live site before running it. + +`bluebottle-checkout-espresso` is the espresso task: one bag of Opascope Espresso, checkout +filled with test details and card 4242 4242 4242 4242, the ten fields read from the page. +Checkout's native `fill_payment_form` tool is the site's, so both arms can use it. +`delta-checkout-signed-in` books the cheapest Main Classic fare up to Review and Pay's Payment +section on the `bench-delta` profile, checked from the page's text. Its guard has not been checked +on Delta's live Review and Pay, so it is `opt_in`; drop that once it has been rehearsed. + ### Signed-in trials A domain with a Kernel `profile` loads it in every browser, in both arms, without saving -changes, and registers its `needs_sign_in` tools. Use a test account with no saved payment -method, kept signed in by a managed auth connection: +changes, and registers its `needs_sign_in` tools. An intent can name its own `profile` and +`signed_in_check` instead, as `delta-checkout-signed-in` does, so other intents on the domain run +signed out. Use a test account that cannot pay for anything: no saved payment method, and on a +checkout intent no balance the site could spend either (Delta: zero SkyMiles, no eCredits, gift +cards or certificates). Keep it signed in with a managed auth connection: ```sh kernel auth connections create --domain instacart.com --profile-name bench-instacart -o json @@ -196,8 +255,9 @@ kernel auth connections login ``` `signed_in_check` names a read-only tool (and optional input) that returns an error when signed -out. The smoke gate and every setup in both arms invoke it; a signed-out trial aborts. Control -registers the batch for the check and removes it before the agent starts. +out, or, with `rules`, whose output fails a rule when signed out. The smoke gate and every setup +in both arms invoke it; a signed-out trial aborts. Control registers the batch for the check and +removes it before the agent starts. Jobs on one profile run one at a time, but server-side state such as recent searches carries across trials and arms. A missing profile skips the domain, or stops the run if it was selected by name. @@ -207,8 +267,13 @@ by name. 1. Extract an answer from the final message: the last fenced JSON block, then a trailing object, then `key: value` lines naming schema properties. Nothing extracted is `malformed`. 2. A schema or rule failure is `wrong`; otherwise `correct`, which means plausible, not verified. +3. For a `prepare` intent, the page must also pass the check's rules and match the answer at each + `same_as_answer` pair, or it is `wrong`. What the runner read from the page, text included, is + saved as `end-state.json` next to the transcript. A page that cannot be read, or a guard count + that cannot be read, aborts the trial, so its pair runs again. -A successful call outside the allowlist, a pi error or a missing cost is `crashed`; hitting a +A successful call outside the allowlist, a blocked submit attempt, a pi error or a missing cost is +`crashed`; hitting a cap is `timeout`. `node bench/grade.mjs bench/results//trials//` regrades one trial against its run's `run.json`. diff --git a/bench/grade.mjs b/bench/grade.mjs index 823463fa..b93fa6ff 100644 --- a/bench/grade.mjs +++ b/bench/grade.mjs @@ -3,11 +3,12 @@ // rules. The first failing step decides the outcome. // Usage: node bench/grade.mjs bench/results//trials// -import { readFileSync } from "node:fs"; +import { existsSync, readFileSync } from "node:fs"; import { basename, dirname, join, resolve } from "node:path"; import { fileURLToPath } from "node:url"; import { summarize } from "./lib/agent.mjs"; import { fillRules } from "./lib/intents.mjs"; +import { extractValues } from "./lib/page.mjs"; import { validate } from "./lib/schema.mjs"; const isObject = (value) => value !== null && typeof value === "object" && !Array.isArray(value); @@ -99,6 +100,16 @@ function isDate(value) { return date.toISOString().slice(0, 10) === m[0]; } +// Compares what the agent reported with what the runner read from the page, as sets: strings +// ignoring case and whitespace, amounts to the cent. +export function checkSameAsAnswer({ answer: answerPath, end_state: endStatePath }, answer, output) { + const canonical = (value) => typeof value === "string" ? value.toLowerCase().replace(/\s+/g, "") : typeof value === "number" ? Math.round(value * 100) / 100 : value; + const values = (value, path) => resolvePath(value, path).map(canonical).map(v => JSON.stringify(v)).sort(); + const [reported, read] = [values(answer, answerPath), values(output, endStatePath)]; + if (JSON.stringify(reported) === JSON.stringify(read)) return null; + return `${answerPath} is [${reported.join(", ")}] in the answer but ${endStatePath} is [${read.join(", ")}] on the site`; +} + export function checkRule(rule, answer) { const values = resolvePath(answer, rule.path); if (!values.length) return `${rule.path}: no value`; @@ -107,7 +118,9 @@ export function checkRule(rule, answer) { return new Set(seen.map(v => JSON.stringify(v))).size === seen.length ? null : `${rule.path}: not unique`; } for (const value of values) { - const shown = JSON.stringify(value); + // Page text can be long, so a failing value is shown cut short. + const json = JSON.stringify(value) ?? String(value); + const shown = json.length > 120 ? `${json.slice(0, 117)}...` : json; if ("equals" in rule && normalize(value) !== normalize(rule.equals)) return `${rule.path}: ${shown} is not ${JSON.stringify(rule.equals)}`; if (rule.in && !rule.in.map(normalize).includes(normalize(value))) return `${rule.path}: ${shown} is not one of ${JSON.stringify(rule.in)}`; if (rule.matches && !(typeof value === "string" && new RegExp(rule.matches, "iu").test(value))) return `${rule.path}: ${shown} does not match /${rule.matches}/i`; @@ -119,8 +132,10 @@ export function checkRule(rule, answer) { return null; } -export function gradeAnswer(intent, schema, text) { - const grade = { outcome: "malformed", extracted: false, shape: false, rules: false, answer: null, problems: [] }; +// `endState` is what the runner read from the page the agent left: { output }. The +// check's rules see that page with `extracted` holding its extract patterns' values. +export function gradeAnswer(intent, schema, text, endState = null) { + const grade = { outcome: "malformed", extracted: false, shape: false, rules: false, end_state: null, end_state_values: null, answer: null, problems: [] }; const answer = extractAnswer(text, schema); if (!answer) { grade.problems.push("no answer object in the final message"); @@ -135,6 +150,16 @@ export function gradeAnswer(intent, schema, text) { grade.problems = fillRules(intent.rules, intent.fixture).map(rule => checkRule(rule, answer)).filter(Boolean); if (grade.problems.length) return grade; grade.rules = true; + if (intent.check) { + const page = endState?.output ? { ...endState.output, extracted: extractValues(endState.output.text, intent.check.extract) } : undefined; + grade.end_state_values = page ? { url: page.url, extracted: page.extracted, fields: page.fields } : null; + grade.problems = [ + ...fillRules(intent.check.rules, intent.fixture).map(rule => checkRule(rule, page)), + ...(intent.check.same_as_answer ?? []).map(pair => checkSameAsAnswer(pair, answer, page)), + ].filter(Boolean).map(problem => `end state: ${problem}`); + grade.end_state = !grade.problems.length; + if (grade.problems.length) return grade; + } grade.outcome = "correct"; return grade; } @@ -148,5 +173,7 @@ if (process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.ur const run = JSON.parse(readFileSync(join(trialDir, "../../../run.json"), "utf8")); const { intent, schema } = run.intents[basename(dirname(trialDir))]; const events = readFileSync(join(trialDir, "transcript.jsonl"), "utf8").split("\n").filter(Boolean).map(line => JSON.parse(line)); - console.log(JSON.stringify(gradeAnswer(intent, schema, summarize(events, []).answer), null, 2)); + const endStateFile = join(trialDir, "end-state.json"); + const endState = existsSync(endStateFile) ? JSON.parse(readFileSync(endStateFile, "utf8")) : null; + console.log(JSON.stringify(gradeAnswer(intent, schema, summarize(events, []).answer, endState), null, 2)); } diff --git a/bench/lib/agent.mjs b/bench/lib/agent.mjs index acffa6e4..b5c50645 100644 --- a/bench/lib/agent.mjs +++ b/bench/lib/agent.mjs @@ -143,6 +143,9 @@ function collectRefs(value, refs, depth = 0) { for (const child of Object.values(value)) collectRefs(child, refs, depth + 1); } +// A form tool that fills a checkout without submitting it ends awaiting_submission. +const SUCCEEDED = ["completed", "awaiting_submission"]; + // The invocation's status, the tool's own status, and an error message when either failed. A // call the MCP server rejected, such as one with a stale tool_ref, comes back as plain text. function invocationResult(call) { @@ -153,12 +156,12 @@ function invocationResult(call) { } const output = result.output !== null && typeof result.output === "object" ? result.output : {}; const toolStatus = typeof output.status === "string" ? output.status : null; - const failed = result.status !== "completed" || toolStatus === "error"; + const failed = !SUCCEEDED.includes(result.status) || toolStatus === "error"; const error = failed ? result.error_text ?? output.error_message ?? output.message ?? output.error_kind ?? result.status ?? null : null; return { status: result.status ?? "unknown", tool_status: toolStatus, error: typeof error === "string" ? error.slice(0, 300) : null }; } -export const invocationFailed = (invocation) => invocation.status !== "completed" || invocation.tool_status === "error"; +export const invocationFailed = (invocation) => !SUCCEEDED.includes(invocation.status) || invocation.tool_status === "error"; const WEBMCP = "mcp__kernel__webmcp"; // `xwebmcp mcp`'s list_tools and invoke_tool, which a local browser's agent uses instead. diff --git a/bench/lib/guard.mjs b/bench/lib/guard.mjs new file mode 100644 index 00000000..78c3b3ed --- /dev/null +++ b/bench/lib/guard.mjs @@ -0,0 +1,97 @@ +// The submission guard: a Chrome extension whose content scripts run in the page's own world on +// a checkout intent's hosts, in every frame and from document start, in both arms. It blocks the +// buttons that pay or place an order and, optionally, form submission; its network rules block +// payment requests. Each blocked click or submission is counted in a cookie on the page's host, +// which the runner reads from the whole browser after the trial, so an agent that leaves the page +// does not take the count with it. + +import { createHash } from "node:crypto"; +import { mkdirSync, rmSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; + +export const BLOCKED_COOKIE = "bench_submit_blocked"; + +// Runs in the page, so it closes over nothing but its argument. +function guard({ buttons, selectors, form_submit: formSubmit, key }) { + if (window.__benchGuard) return; + window.__benchGuard = true; + const button = new RegExp(buttons, "i"); + const count = (what) => { + try { + const current = Number(document.cookie.split("; ").find(c => c.startsWith(`${key}=`))?.slice(key.length + 1) || 0); + const crossSite = location.protocol === "https:" ? "; SameSite=None; Secure" : ""; + document.cookie = `${key}=${current + 1}; path=/; max-age=86400${crossSite}`; + } catch { + // cookies are unavailable in sandboxed frames + } + console.warn(`bench guard: blocked ${what}`); + }; + const stop = (event, what) => { + event.preventDefault(); + event.stopImmediatePropagation(); + if (what) count(what); + }; + const label = (el) => (el.innerText || el.value || el.getAttribute("aria-label") || el.textContent || "").trim().replace(/\s+/g, " "); + window.addEventListener("click", (event) => { + const marked = selectors && event.target?.closest?.(selectors); + if (marked) return stop(event, `a click on "${label(marked)}" (${selectors})`); + const el = event.target?.closest?.('button, [role="button"], input[type="submit"], input[type="button"], a'); + if (!el) return; + if (button.test(label(el))) stop(event, `a click on "${label(el)}"`); + else if (formSubmit && el.form && el.type === "submit") stop(event, `a click on the submit button "${label(el)}"`); + }, true); + if (formSubmit) { + window.addEventListener("submit", (event) => stop(event, "a form submission"), true); + // Enter in a field would submit the form; it is blocked but not counted, since agents press it + // to pick address suggestions. + window.addEventListener("keydown", (event) => { + if (event.key === "Enter" && event.target?.form) stop(event, null); + }, true); + HTMLFormElement.prototype.submit = function () { count("form.submit()"); }; + HTMLFormElement.prototype.requestSubmit = function () { count("form.requestSubmit()"); }; + } +} + +// Network rules apply whichever world or frame starts a request, including the isolated world +// Playwright's page.evaluate runs in, where the page-world overrides above are not installed: +// `requests` blocks matching fetch, XHR and beacons from any frame, and `form_submit` blocks POST +// navigations started on the guard's hosts. Neither is counted. +export function networkRules({ hosts, form_submit: formSubmit = false, requests = null }) { + const rules = []; + if (requests) { + rules.push({ id: 1, priority: 1, action: { type: "block" }, condition: { regexFilter: requests, isUrlFilterCaseSensitive: false, resourceTypes: ["xmlhttprequest", "ping"] } }); + } + if (formSubmit) { + const domains = [...new Set(hosts.map(host => new URL(host.replace("://*.", "://")).hostname))]; + rules.push({ id: 2, priority: 1, action: { type: "block" }, condition: { initiatorDomains: domains, requestMethods: ["post"], resourceTypes: ["main_frame", "sub_frame"] } }); + } + return rules; +} + +// The name carries a hash of the content, so a run reuses an extension it, or an earlier run, +// already uploaded. +export function writeGuardExtension(dir, { hosts, buttons, selectors = null, form_submit: formSubmit = false, requests = null }) { + const rules = networkRules({ hosts, form_submit: formSubmit, requests }); + const files = { + "manifest.json": JSON.stringify({ + manifest_version: 3, + name: "bench submission guard", + version: "1.0", + content_scripts: [{ matches: hosts, js: ["guard.js"], run_at: "document_start", all_frames: true, world: "MAIN" }], + ...(rules.length ? { permissions: ["declarativeNetRequest"], declarative_net_request: { rule_resources: [{ id: "guard", enabled: true, path: "rules.json" }] } } : {}), + }, null, 2) + "\n", + "guard.js": `(${guard})(${JSON.stringify({ buttons, selectors, form_submit: formSubmit, key: BLOCKED_COOKIE })});\n`, + ...(rules.length ? { "rules.json": JSON.stringify(rules, null, 2) + "\n" } : {}), + }; + const hash = createHash("sha256"); + for (const [name, text] of Object.entries(files)) hash.update(`${name}\0${text}\0`); + rmSync(dir, { recursive: true, force: true }); + mkdirSync(dir, { recursive: true }); + for (const [name, text] of Object.entries(files)) writeFileSync(join(dir, name), text); + return { name: `bench-guard-${hash.digest("hex").slice(0, 12)}`, dir }; +} + +// Blocked attempts across every host the guard ran on, from the browser's cookies. +export function blockedAttempts(cookies) { + return cookies.filter(cookie => cookie.name === BLOCKED_COOKIE).reduce((total, cookie) => total + (Number(cookie.value) || 0), 0); +} diff --git a/bench/lib/intents.mjs b/bench/lib/intents.mjs index 86587c08..78160398 100644 --- a/bench/lib/intents.mjs +++ b/bench/lib/intents.mjs @@ -17,12 +17,22 @@ export function fill(value, fixture = {}) { return value; } +// What an intent's `allow` lets the agent do beyond the system prompt's rules, in both arms. +export const ALLOW = { + checkout: "This task lets you add to the cart and go through checkout up to its last step, but never press the button that pays, places the order or completes the booking. The harness blocks that button and counts any attempt as a failure.", + payment_details: "It also lets you enter the personal and payment details it gives you. They are test data, and the card cannot be charged.", +}; + // `browserId` is null for a local browser, the only one the agent can reach. export function trialPrompt(intent, browserId) { const browser = browserId ? `Use the Kernel browser with session id ${browserId}. It is already open on ${intent.start_url}.` : `The browser is already open on ${intent.start_url}.`; - return [fill(intent.prompt, intent.fixture), intent.instructions, browser].filter(Boolean).join("\n\n"); + const allowed = (intent.allow ?? []).map(key => ALLOW[key]).join(" "); + return [fill(intent.prompt, intent.fixture), allowed, intent.instructions, browser].filter(Boolean).join("\n\n"); } +// `requests` becomes a declarativeNetRequest regexFilter, which Chrome compiles with RE2. +const RE2_UNSUPPORTED = /\(\? text.replace(/[\\^$.*+?()[\]{}|/]/g, "\\$&"); // A `matches` rule takes fixture values literally, so "Verity (2026)" is not a regex group. @@ -77,11 +87,34 @@ export function validateIntent(intent, benchDir) { need(intent.stage === "gap" ? named.length === 0 : named.length > 0, intent.stage === "gap" ? "gap intents name no tools" : "non-gap intents name the tools a treatment agent would use"); need(typeof intent.entry_tool === "string", "entry_tool is required"); need(typeof intent.answer_schema === "string" && existsSync(join(benchDir, intent.answer_schema)), "answer_schema must point at a file under bench/"); - for (const rule of intent.rules ?? []) { - const checks = Object.keys(rule).filter(key => key !== "path"); - need(typeof rule.path === "string" && checks.length === 1 && RULE_CHECKS.includes(checks[0]), `rule ${JSON.stringify(rule)} needs a path and one of ${RULE_CHECKS.join(", ")}`); - if ("matches" in rule) need(typeof rule.matches === "string" && compiles(fillRules([rule], intent.fixture)[0].matches, "iu"), `rule ${JSON.stringify(rule)} needs a valid regex`); - } + need(intent.actions === undefined, "actions belong in the domain's domain.json"); + // A prepare intent changes site state, so the runner reads where the agent left the browser from + // the page itself, never through the tools under test. + const { check, allow, guard } = intent; + need(intent.stage === "prepare" ? check !== undefined : check === undefined && allow === undefined, + "check and allow belong to prepare intents, and a prepare intent needs a check"); + const isMap = (value, valid) => value === undefined || (value !== null && typeof value === "object" && !Array.isArray(value) && Object.values(value).every(valid)); + need(check === undefined || (check !== null && typeof check === "object" && Object.keys(check).every(key => ["fields", "extract", "rules", "same_as_answer"].includes(key))), + "check takes fields, extract, rules and same_as_answer"); + need(isMap(check?.fields, selector => typeof selector === "string" && selector.length > 0), "check.fields must map names to CSS selectors"); + need(isMap(check?.extract, pattern => typeof pattern === "string" && compiles(pattern, "iu") && new RegExp(`${pattern}|`, "iu").exec("").length === 2), + "check.extract must map names to regexes with one capturing group"); + need(check === undefined || (Array.isArray(check.rules) && check.rules.length > 0), "check needs rules"); + need(allow === undefined || (Array.isArray(allow) && allow.length > 0 && new Set(allow).size === allow.length && allow.every(key => Object.hasOwn(ALLOW, key))), + `allow must list some of ${Object.keys(ALLOW).join(", ")}`); + need(!allow?.includes?.("payment_details") || allow.includes("checkout"), "allow payment_details needs checkout"); + // An agent that may enter checkout must not be able to submit it, in either arm. + need(Boolean(allow?.includes?.("checkout")) === (guard !== undefined), "an intent that allows checkout needs a guard, and only such an intent has one"); + need(guard === undefined || (Array.isArray(guard?.hosts) && guard.hosts.length > 0 && guard.hosts.every(host => typeof host === "string" && /^https:\/\/[^/]+\/\S*$/.test(host)) + && typeof guard.buttons === "string" && compiles(guard.buttons, "i") && (guard.requests === undefined || (typeof guard.requests === "string" && compiles(guard.requests, "i") && !RE2_UNSUPPORTED.test(guard.requests))) + && (guard.selectors === undefined || (typeof guard.selectors === "string" && guard.selectors.length > 0)) + && (guard.form_submit === undefined || typeof guard.form_submit === "boolean")), + "guard needs hosts (https match patterns) and a buttons regex, and optionally selectors, a requests regex (RE2: no lookarounds or backreferences) and form_submit"); + if (intent.profile !== undefined || intent.signed_in_check !== undefined) problems.push(...signInProblems(intent.profile, intent.signed_in_check)); + const sameAsAnswer = check?.same_as_answer; + need(sameAsAnswer === undefined || (Array.isArray(sameAsAnswer) && sameAsAnswer.every(pair => typeof pair?.answer === "string" && typeof pair?.end_state === "string")), + "check.same_as_answer must list {answer, end_state} paths"); + for (const rule of [...(intent.rules ?? []), ...(Array.isArray(check?.rules) ? check.rules : [])]) problems.push(...ruleProblems(rule, intent.fixture)); const { agreement } = intent; const best = (a) => ["lower", "higher"].includes(a.better) && a.same === undefined && (a.tolerance === undefined || a.tolerance >= 0); const same = (a) => a.same === true && a.better === undefined && a.tolerance === undefined; @@ -90,24 +123,43 @@ export function validateIntent(intent, benchDir) { for (const [key, value] of Object.entries(intent.fixture ?? {})) { need(typeof value === "string" || typeof value === "number" || relativeDate(value), `fixture ${key} must be a string, a number or {"days_from_start": n}`); } - const text = JSON.stringify([intent.prompt, intent.rules ?? []]); + const text = JSON.stringify([intent.prompt, intent.rules ?? [], check?.rules ?? []]); for (const [, key] of text.matchAll(/\{\{(\w+)\}\}/g)) need(Object.hasOwn(intent.fixture ?? {}, key), `{{${key}}} has no fixture value`); + need(intent.opt_in === undefined || (typeof intent.opt_in === "string" && intent.opt_in.trim().length > 0), "opt_in must give the reason the intent runs only when named"); for (const key of ["pairs", "trial_timeout_sec", "max_turns"]) need(Number.isInteger(intent[key]) && intent[key] > 0, `${key} must be a positive integer`); return problems; } +function ruleProblems(rule, fixture) { + const checks = Object.keys(rule ?? {}).filter(key => key !== "path"); + if (typeof rule?.path !== "string" || checks.length !== 1 || !RULE_CHECKS.includes(checks[0])) return [`rule ${JSON.stringify(rule)} needs a path and one of ${RULE_CHECKS.join(", ")}`]; + if ("matches" in rule && !(typeof rule.matches === "string" && compiles(fillRules([rule], fixture)[0].matches, "iu"))) return [`rule ${JSON.stringify(rule)} needs a valid regex`]; + return []; +} + +// A domain or an intent can name a profile, with a tool whose output shows it is signed in: an +// error means signed out, or, with rules, any rule that fails does. +function signInProblems(profile, check) { + const problems = []; + const need = (cond, message) => { if (!cond) problems.push(message); }; + need(profile === undefined || (typeof profile === "string" && profile.length > 0), "profile must be a Kernel profile name"); + need((profile === undefined) === (check === undefined), "profile and signed_in_check go together"); + const validInput = (input) => input === undefined || (input !== null && typeof input === "object"); + const validRules = (rules) => rules === undefined || (Array.isArray(rules) && rules.length > 0); + need(check === undefined || (typeof check?.tool === "string" && validInput(check.input) && validRules(check.rules)), + "signed_in_check needs a tool name and, optionally, an input object and rules"); + if (Array.isArray(check?.rules)) for (const rule of check.rules) problems.push(...ruleProblems(rule).map(problem => `signed_in_check ${problem}`)); + return problems; +} + const isReasons = (value) => value !== null && typeof value === "object" && !Array.isArray(value) && Object.values(value).every(reason => typeof reason === "string"); export function validateDomain(config) { const problems = []; const need = (cond, message) => { if (!cond) problems.push(message); }; - for (const key of Object.keys(config)) need(["exclude", "needs_sign_in", "profile", "signed_in_check", "instructions"].includes(key), `unknown key ${key}`); - for (const key of ["exclude", "needs_sign_in"]) need(config[key] === undefined || isReasons(config[key]), `${key} must map tool names to reasons`); - need(config.profile === undefined || (typeof config.profile === "string" && config.profile.length > 0), "profile must be a Kernel profile name"); - const check = config.signed_in_check; - need((config.profile === undefined) === (check === undefined), "profile and signed_in_check go together"); - const validInput = (input) => input === undefined || (input !== null && typeof input === "object"); - need(check === undefined || (typeof check?.tool === "string" && validInput(check.input)), "signed_in_check needs a tool name and, optionally, an input object"); + for (const key of Object.keys(config)) need(["exclude", "needs_sign_in", "actions", "checkout_actions", "profile", "signed_in_check", "instructions"].includes(key), `unknown key ${key}`); + for (const key of ["exclude", "needs_sign_in", "actions", "checkout_actions"]) need(config[key] === undefined || isReasons(config[key]), `${key} must map tool names to reasons`); + problems.push(...signInProblems(config.profile, config.signed_in_check)); need(config.instructions === undefined || (typeof config.instructions === "string" && config.instructions.trim().length > 0), "instructions must be a non-empty string"); return problems; } @@ -122,14 +174,22 @@ export function loadIntents(benchDir) { const domainConfig = existsSync(domainFile) ? JSON.parse(readFileSync(domainFile, "utf8")) : {}; const domainProblems = validateDomain(domainConfig); if (domainProblems.length) throw new Error(`tasks/${domain}/domain.json: ${domainProblems.join("; ")}`); - const { exclude = {}, needs_sign_in: needsSignIn = {}, profile = null, signed_in_check: signedInCheck = null, instructions = null } = domainConfig; + const { exclude = {}, needs_sign_in: needsSignIn = {}, actions = {}, checkout_actions: checkoutActions = {}, profile = null, signed_in_check: signedInCheck = null, instructions = null } = domainConfig; for (const file of readdirSync(dir).filter(f => f.endsWith(".json") && f !== "domain.json").sort()) { const intent = JSON.parse(readFileSync(join(dir, file), "utf8")); const problems = validateIntent(intent, benchDir); if (intent.id !== file.replace(/\.json$/, "")) problems.push("file name must be .json"); if (intent.domain !== domain) problems.push(`domain must be ${domain}`); if (problems.length) throw new Error(`tasks/${domain}/${file}: ${problems.join("; ")}`); - intents.push({ ...intent, exclude: profile ? exclude : { ...exclude, ...needsSignIn }, profile, signed_in_check: signedInCheck, instructions }); + // An intent's own profile replaces the domain's. + const intentProfile = intent.profile ?? profile; + // The domain's state-changing tools reach only intents that may change state, and its + // checkout tools only intents that may go through checkout. + const intentActions = intent.stage !== "prepare" ? [] : [...Object.keys(actions), ...(intent.allow?.includes("checkout") ? Object.keys(checkoutActions) : [])]; + intents.push({ + ...intent, actions: intentActions, exclude: intentProfile ? exclude : { ...exclude, ...needsSignIn }, profile: intentProfile, + signed_in_check: intent.profile ? intent.signed_in_check : signedInCheck, instructions, + }); } } return intents; diff --git a/bench/lib/kernel.mjs b/bench/lib/kernel.mjs index b8996700..d97c70db 100644 --- a/bench/lib/kernel.mjs +++ b/bench/lib/kernel.mjs @@ -19,10 +19,11 @@ export const BROWSER = { }; // Without --save-changes, trials never change the profile. -export async function createBrowser({ startUrl, timeoutSec, tags, stealth, profile }) { +export async function createBrowser({ startUrl, timeoutSec, tags, stealth, profile, extension }) { const args = ["browsers", "create", "--start-url", startUrl, "--viewport", BROWSER.viewport, "--telemetry", BROWSER.telemetry, "-t", String(timeoutSec)]; if (stealth) args.push("--stealth"); if (profile) args.push("--profile-name", profile); + if (extension) args.push("--extension", extension.name); for (const [key, value] of Object.entries(tags)) args.push("--tag", `${key}=${value}`); return (await kernel(args)).session_id; } @@ -42,6 +43,30 @@ export async function waitForLoad(id, startUrl) { return response.result; } +// Runs a function of one JSON argument in the current page and returns its result. +export async function evaluate(id, fn, arg) { + const response = await kernel(["browsers", "playwright", "execute", id, `return await page.evaluate(${fn}, ${JSON.stringify(arg)});`, "--timeout", "30"], { timeoutMs: 60_000 }); + if (!response.success) throw new Error(`evaluate: ${(response.error ?? "unknown error").split("\n")[0]}`); + return response.result; +} + +// Every cookie in the browser, whatever page it is on. +export async function cookies(id) { + const response = await kernel(["browsers", "playwright", "execute", id, "return await context.cookies();", "--timeout", "30"], { timeoutMs: 60_000 }); + if (!response.success) throw new Error(`cookies: ${(response.error ?? "unknown error").split("\n")[0]}`); + return response.result; +} + +// Uploads an extension directory under its name unless Kernel already has it. +export async function ensureExtension({ name, dir }) { + try { + await kernel(["extensions", "get", name]); + } catch (error) { + if (!/not.?found/i.test(error.message)) throw error; + await kernel(["extensions", "upload", dir, "--name", name]); + } +} + export async function addCustomTools(id, namespace, sourceFile) { return (await kernel(["browsers", "webmcp", "custom-tools", "add", id, "--namespace", namespace, "--source-file", sourceFile, "--force-overwrite-namespace"])).tools; } diff --git a/bench/lib/local.mjs b/bench/lib/local.mjs index 93083cc3..84598507 100644 --- a/bench/lib/local.mjs +++ b/bench/lib/local.mjs @@ -107,14 +107,15 @@ async function waitFor(what, fn, timeoutMs) { // Opens the start page in a tab the extension owns, so `xwebmcp mcp` lists and invokes its // tools without allow_user_tabs, and closes Chrome's own first tab. `profile` names a Kernel -// profile, which a local browser cannot load. -export async function createBrowser({ startUrl, profile }) { +// profile, which a local browser cannot load. `extension` is the submission guard, loaded unpacked. +export async function createBrowser({ startUrl, profile, extension }) { if (profile) throw new Error(`local browsers cannot load the Kernel profile ${profile}`); const dir = mkdtempSync(join(tmpdir(), "bench-local-")); const paths = writeProfile(dir); + const extensions = [paths.extension, ...(extension ? [extension.dir] : [])].join(","); const args = [ `--user-data-dir=${paths.profile}`, "--remote-debugging-port=0", "--no-first-run", "--no-default-browser-check", - `--disable-extensions-except=${paths.extension}`, `--load-extension=${paths.extension}`, "--window-size=1920,1080", + `--disable-extensions-except=${extensions}`, `--load-extension=${extensions}`, "--window-size=1920,1080", ]; if (config.headless) args.push("--headless=new"); const log = join(dir, "chrome.log"); @@ -162,6 +163,44 @@ export async function waitForLoad(id, startUrl) { return { url: (await ownedTab(browser)).url, version: config.chrome_version, renavigated }; } +// Sends one DevTools protocol command to a target and returns its result. +async function devtools(what, webSocketUrl, method, params) { + const socket = new WebSocket(webSocketUrl); + try { + await new Promise((resolve, reject) => { socket.onopen = resolve; socket.onerror = () => reject(new Error(`${what}: cannot connect to the browser`)); }); + const reply = new Promise((resolve, reject) => { + socket.onmessage = (event) => resolve(JSON.parse(event.data)); + setTimeout(() => reject(new Error(`${what}: no answer from the browser`)), 30_000).unref(); + }); + socket.send(JSON.stringify({ id: 1, method, params })); + const { result, error } = await reply; + if (error) throw new Error(`${what}: ${error.message}`); + return result; + } finally { + socket.close(); + } +} + +// Runs a function of one JSON argument in the bench's tab over the DevTools protocol. +export async function evaluate(id, fn, arg) { + const browser = browsers.get(id); + const { url } = await ownedTab(browser); + const pages = (await (await fetch(`http://127.0.0.1:${browser.port}/json/list`)).json()).filter(t => t.type === "page"); + const target = pages.find(t => t.url === url) ?? pages[0]; + if (!target) throw new Error("evaluate: no page to read"); + const result = await devtools("evaluate", target.webSocketDebuggerUrl, "Runtime.evaluate", { expression: `(${fn})(${JSON.stringify(arg)})`, returnByValue: true, awaitPromise: true }); + if (result?.exceptionDetails) throw new Error(`evaluate: ${result.exceptionDetails.exception?.description ?? result.exceptionDetails.text}`); + return result?.result?.value; +} + +// Every cookie in the browser, whatever page it is on. +export async function cookies(id) { + const { webSocketDebuggerUrl } = await (await fetch(`http://127.0.0.1:${browsers.get(id).port}/json/version`)).json(); + return (await devtools("cookies", webSocketDebuggerUrl, "Storage.getCookies", {})).cookies; +} + +export async function ensureExtension() {} + export async function addCustomTools(id, namespace, sourceFile) { const browser = browsers.get(id); const result = await call(browser, "addCustomTools", { namespace, source: readFileSync(sourceFile, "utf8"), file: `domains/${namespace}/(batch).js`, force_overwrite_namespace: true }); diff --git a/bench/lib/page.mjs b/bench/lib/page.mjs new file mode 100644 index 00000000..2f951cbb --- /dev/null +++ b/bench/lib/page.mjs @@ -0,0 +1,37 @@ +// Reading a trial's end state from the page. + +const TEXT_LIMIT = 200_000; + +// The page's URL, title, visible text and the named inputs. Passed to page.evaluate, so it closes +// over nothing but its argument. +export function readPage({ fields, textLimit }) { + const read = Object.entries(fields ?? {}).map(([name, selector]) => { + const el = document.querySelector(selector); + const invalid = !!el && (el.getAttribute("aria-invalid") === "true" || (el.validity ? !el.validity.valid : false)); + return [name, el ? String(el.value ?? "").trim() : null, invalid]; + }); + return { + url: location.origin + location.pathname, + title: document.title, + text: (document.body?.innerText ?? "").slice(0, textLimit), + fields: Object.fromEntries(read.map(([name, value]) => [name, value])), + missing: read.filter(([, value]) => value === null).map(([name]) => name), + invalid: read.filter(([, , invalid]) => invalid).map(([name]) => name), + }; +} + +export const readPageArg = (fields = {}) => ({ fields, textLimit: TEXT_LIMIT }); + +// Each extract pattern's first capture in the page text; an amount such as "$1,234.50" becomes a +// number. A pattern whose group does not match is left out, so a rule on it reports no value. +export function extractValues(text, extract = {}) { + const values = {}; + for (const [name, pattern] of Object.entries(extract)) { + const m = new RegExp(pattern, "iu").exec(text ?? ""); + if (m?.[1] === undefined) continue; + const raw = m[1].trim(); + const number = raw.replace(/^[$€£]\s*/, "").replace(/,/g, ""); + values[name] = /^-?\d+(\.\d+)?$/.test(number) ? Number(number) : raw; + } + return values; +} diff --git a/bench/lib/tools.mjs b/bench/lib/tools.mjs index a37c7d66..88d7c8db 100644 --- a/bench/lib/tools.mjs +++ b/bench/lib/tools.mjs @@ -74,7 +74,9 @@ function namespaceTools(show, sha, namespace) { } // Sources apply in order, so a branch listed after main replaces main's tool of the same name. +// A prepare intent's actions are the domain's tools that change site state, registered as well. export async function registeredTools(intent, shas, show) { + const actions = intent.actions ?? []; const byName = new Map(); const known = new Set(); const problems = []; @@ -85,15 +87,15 @@ export async function registeredTools(intent, shas, show) { if (!all.some(tool => tool.name === name)) problems.push(`${name} is not in registry.json at ${source.ref} (${shas[i].slice(0, 7)})`); } for (const tool of all) { - if (tool.readOnly && !Object.hasOwn(intent.exclude, tool.name)) byName.set(tool.name, tool); + if ((tool.readOnly || actions.includes(tool.name)) && !Object.hasOwn(intent.exclude, tool.name)) byName.set(tool.name, tool); else byName.delete(tool.name); } } - for (const name of Object.keys(intent.exclude)) { - if (!known.has(name)) problems.push(`exclude names ${name}, which no source has`); + for (const [list, name] of [...Object.keys(intent.exclude).map(name => ["exclude", name]), ...actions.map(name => ["actions", name])]) { + if (!known.has(name)) problems.push(`${list} names ${name}, which no source has`); } for (const name of [...intent.tools.flatMap(s => s.names), intent.entry_tool]) { - if (!byName.has(name) && !problems.some(p => p.startsWith(`${name} `))) problems.push(`${name} is not read-only or is excluded, so it would not be registered`); + if (!byName.has(name) && !problems.some(p => p.startsWith(`${name} `))) problems.push(`${name} is excluded or neither read-only nor an action, so it would not be registered`); } if (problems.length) throw new Error(`${intent.id}: ${problems.join("; ")}`); return [...byName.values()].sort((a, b) => a.name.localeCompare(b.name)); diff --git a/bench/prompts/system.local.md b/bench/prompts/system.local.md index 880dc207..93c228bc 100644 --- a/bench/prompts/system.local.md +++ b/bench/prompts/system.local.md @@ -10,9 +10,9 @@ Use the page's WebMCP tools first: Rules: -- Do not pay, place an order, or complete a booking or reservation, and go no further into a checkout than the task asks. -- Do not add items or tickets to a cart unless the task says you may. Selecting seats and ticket types to see their prices is allowed. -- Do not log in, sign up, or enter personal or payment details. +- Never pay, place an order, or complete a booking or reservation. Even when the task lets you fill in a checkout, stop before the button that submits it. +- Do not add items or tickets to a cart, or go into a checkout, unless the task says you may, and then go no further than it asks. Selecting seats and ticket types to see their prices is allowed. +- Do not log in or sign up. Enter personal or payment details only when the task gives them to you. - Do not post, comment, send messages, star, follow, or change anything else on the site. - Do not open new windows or browsers. - Searching, filtering, and opening pages to read them are allowed, including submitting search and filter forms. diff --git a/bench/prompts/system.md b/bench/prompts/system.md index 68140dd4..1a239015 100644 --- a/bench/prompts/system.md +++ b/bench/prompts/system.md @@ -10,9 +10,9 @@ Use the page's WebMCP tools first: Rules: -- Do not check out, pay, place an order, or complete a booking or reservation. -- Do not add items or tickets to a cart unless the task says you may. Selecting seats and ticket types to see their prices is allowed. -- Do not log in, sign up, or enter personal or payment details. +- Never pay, place an order, or complete a booking or reservation. Even when the task lets you fill in a checkout, stop before the button that submits it. +- Do not add items or tickets to a cart, or go into a checkout, unless the task says you may, and then go no further than it asks. Selecting seats and ticket types to see their prices is allowed. +- Do not log in or sign up. Enter personal or payment details only when the task gives them to you. - Do not post, comment, send messages, star, follow, or change anything else on the site. - Do not create, delete, or switch browsers. - Searching, filtering, and opening pages to read them are allowed, including submitting search and filter forms. diff --git a/bench/report.mjs b/bench/report.mjs index c255fd75..8aa9aeaf 100644 --- a/bench/report.mjs +++ b/bench/report.mjs @@ -107,6 +107,7 @@ export function adoptedPairStats(rows) { // A rule that only repeats the fixture back pins nothing. export function answerCheck(intent) { + if (intent.check) return "end state"; const pinned = (intent.rules ?? []).some(rule => ("equals" in rule || "in" in rule) && !JSON.stringify(rule.equals ?? rule.in).includes("{{")); return pinned ? "pinned" : intent.agreement ? "agreement" : "shape only"; } diff --git a/bench/run.mjs b/bench/run.mjs index feff1a34..075d4947 100644 --- a/bench/run.mjs +++ b/bench/run.mjs @@ -19,7 +19,9 @@ import * as local from "./lib/local.mjs"; import { latestRows, readRows, rowKey } from "./lib/rows.mjs"; import { adoptionVerdicts, balancedOrders, holdNeededSec, planJobs, runPairAttempts, runQueue } from "./lib/schedule.mjs"; import { composeBatch, gitShow, matchesUrlPattern, registeredTools, resolveSource } from "./lib/tools.mjs"; -import { gradeAnswer } from "./grade.mjs"; +import { blockedAttempts, writeGuardExtension } from "./lib/guard.mjs"; +import { readPage, readPageArg } from "./lib/page.mjs"; +import { checkRule, gradeAnswer } from "./grade.mjs"; const benchDir = dirname(fileURLToPath(import.meta.url)); if (existsSync(join(benchDir, ".env"))) process.loadEnvFile(join(benchDir, ".env")); @@ -135,6 +137,8 @@ async function planRun() { if (unknown.length) fail(`no intent matches ${unknown.join(", ")}`); if (opts.domain) intents = intents.filter(i => opts.domain.includes(i.domain)); if (opts.intent) intents = intents.filter(i => opts.intent.includes(i.id)); + intents = intents.filter(intent => !intent.opt_in || opts.intent?.includes(intent.id) + || (log(`skipping ${intent.id}: ${intent.opt_in}; name it with --intent to run it`), false)); const skipUnlessNamed = (intent, message) => { if (opts.domain?.includes(intent.domain) || opts.intent?.includes(intent.id)) fail(message); log(`skipping ${message}`); @@ -195,6 +199,7 @@ async function planRun() { cap_sec: flags.capSec ?? intent.trial_timeout_sec, max_turns: intent.max_turns, orders: balancedOrders(pairs, randomInt), + guard_extension: intent.guard ? writeGuardExtension(join(runDir, "guards", intent.id), intent.guard) : null, }; log(`${intent.id}: ${tools.length} tools at ${shas.map(s => s.slice(0, 7)).join(", ")}`); } @@ -324,7 +329,7 @@ async function smokeGate(plan, probe, signedIn) { fail(`the agent's tools differ from ${experiment.files.allowlist}; an MCP server did not connect or a name did not match.\n` + ` tools: ${tools.join(", ") || "none"}\n allowlist: ${plan.allowlist.join(", ")}`); } - for (const intentPlan of signedIn.filter(p => p.intent.domain !== probe.intent.domain)) { + for (const intentPlan of signedIn.filter(p => p.profile.name !== probe.profile?.name)) { const check = await setUpBrowser(plan, intentPlan, "treatment", "smoke"); if (check.stopped) return; if (check.browser) await releaseBrowser(check.browser); @@ -333,7 +338,7 @@ async function smokeGate(plan, probe, signedIn) { + `If it is signed out, sign it in again; see "Signed-in trials" in bench/README.md`); } } - const signedInText = signedIn.length ? `; signed in to ${signedIn.map(p => p.intent.domain).join(", ")}` : ""; + const signedInText = signedIn.length ? `; signed in to ${signedIn.map(p => `${p.intent.domain} (${p.profile.name})`).join(", ")}` : ""; log(`smoke gate passed (${plan.agent}, ${plan.model}; registered ${probe.registered.length} tools for ${probe.intent.id}${signedInText})`); } @@ -422,7 +427,8 @@ async function checkSignedIn(browser, intent, live) { const ref = live.find(t => t.tool?.name === tool && t.source?.custom?.namespace === intent.domain)?.tool_ref; if (!ref) throw new Error(`${tool} is not live on ${intent.start_url}`); const output = await browsers.invokeTool(browser, ref, input); - if (output?.status === "error" || output?.error) throw new SignedOut(`signed out: ${tool} returned ${JSON.stringify(output).slice(0, 300)}`); + const signedOut = intent.signed_in_check.rules ? intent.signed_in_check.rules.some(rule => checkRule(rule, output)) : output?.status === "error" || output?.error; + if (signedOut) throw new SignedOut(`signed out: ${tool} returned ${JSON.stringify(output).slice(0, 300)}`); } async function setUpBrowser(plan, intentPlan, arm, label) { @@ -437,6 +443,7 @@ async function setUpBrowser(plan, intentPlan, arm, label) { tags: { bench: runId, arm: label === "smoke" ? "smoke" : arm, intent: intent.id }, stealth: plan.browser.stealth, profile: intentPlan.profile?.name, + extension: intentPlan.guard_extension, }); const page = await browsers.waitForLoad(browser, intent.start_url); const loadedAt = performance.now(); @@ -470,6 +477,17 @@ async function setUpBrowser(plan, intentPlan, arm, label) { } } +// Reads where a prepare intent's agent left the browser from the page itself, the same way in both +// arms, so the grade does not depend on the tools under test. A page that cannot be read aborts the +// trial rather than grading it wrong. +async function readEndState(browser, intent) { + try { + return { output: await browsers.evaluate(browser, readPage, readPageArg(intent.check.fields)) }; + } catch (error) { + throw new Error(`could not read the end state: ${error.message}`); + } +} + // Returns the outcome, or null when the run is stopping and no row was written. async function runTrial(plan, intentPlan, arm, key) { if (stopping) return null; @@ -486,7 +504,7 @@ async function runTrial(plan, intentPlan, arm, key) { image_version: null, browser_id: null, setup_attempts: null, start_renavigated: null, start_delay_ms: null, native_tools: null, started_at: new Date().toISOString(), outcome: null, reason: null, duration_ms: null, setup_ms: null, turns: null, tool_calls: null, unknown_tool_errors: null, tokens_in: null, tokens_out: null, cost_usd: null, - registration_changed: null, webmcp: null, grade: null, + registration_changed: null, submit_blocked: null, webmcp: null, grade: null, }; const setupStart = performance.now(); let browser = null; @@ -520,15 +538,26 @@ async function runTrial(plan, intentPlan, arm, key) { log(`${intent.id} ${label} ${arm}: could not re-list custom tools: ${error.message}`); } - const verdict = classifyTrial(summary, agent, plan.allowlist, stderrTail(join(dir, "stderr.txt"))); + if (intentPlan.guard_extension) { + // A count that cannot be read could hide an attempt to submit, so the trial aborts. + try { + row.submit_blocked = blockedAttempts(await browsers.cookies(browser)); + } catch (error) { + throw new Error(`could not read the guard's count: ${error.message}`); + } + } + const verdict = row.submit_blocked ? ["crashed", `tried to submit: the guard blocked ${row.submit_blocked} attempt(s)`] + : classifyTrial(summary, agent, plan.allowlist, stderrTail(join(dir, "stderr.txt"))); if (verdict) { [row.outcome, row.reason] = verdict; } else { - const grade = gradeAnswer(intent, intentPlan.schema, summary.answer); + const endState = intent.check ? await readEndState(browser, intent) : null; + if (endState) writeFileSync(join(dir, "end-state.json"), JSON.stringify(endState, null, 2) + "\n"); + const grade = gradeAnswer(intent, intentPlan.schema, summary.answer, endState); writeFileSync(join(dir, "grade.json"), JSON.stringify(grade, null, 2) + "\n"); row.outcome = grade.outcome; row.reason = grade.problems.slice(0, 3).join("; ") || null; - row.grade = { extracted: grade.extracted, shape: grade.shape, rules: grade.rules, answer: grade.answer }; + row.grade = { extracted: grade.extracted, shape: grade.shape, rules: grade.rules, end_state: grade.end_state, answer: grade.answer }; } } catch (error) { row.outcome = "aborted"; @@ -619,7 +648,8 @@ async function main() { if (!adoptionJobs.length && !pairJobs.length) return log(`nothing to run; node bench/report.mjs ${runDir}`); const pendingPlans = [...new Set([...adoptionJobs, ...pairJobs].map(job => job.intentPlan))]; - const signedIn = [...new Map(pendingPlans.filter(p => p.profile).map(p => [p.intent.domain, p])).values()]; + const signedIn = [...new Map(pendingPlans.filter(p => p.profile).map(p => [p.profile.name, p])).values()]; + for (const intentPlan of pendingPlans.filter(p => p.guard_extension)) await browsers.ensureExtension(intentPlan.guard_extension); // Checked before the smoke gate, which spends money. checkHold(holdNeededSec([adoptionJobs, pairJobs], slots), slots); await smokeGate(plan, (adoptionJobs[0] ?? pairJobs[0]).intentPlan, signedIn); diff --git a/bench/schemas/cart-flight.json b/bench/schemas/cart-flight.json new file mode 100644 index 00000000..82446506 --- /dev/null +++ b/bench/schemas/cart-flight.json @@ -0,0 +1,8 @@ +{ + "type": "object", + "required": ["flight_numbers", "amount_due"], + "properties": { + "flight_numbers": { "type": "array", "minItems": 1, "items": { "type": "string", "minLength": 1 } }, + "amount_due": { "type": "number", "exclusiveMinimum": 0 } + } +} diff --git a/bench/schemas/checkout-order.json b/bench/schemas/checkout-order.json new file mode 100644 index 00000000..85923144 --- /dev/null +++ b/bench/schemas/checkout-order.json @@ -0,0 +1,17 @@ +{ + "type": "object", + "required": [ + "product", + "quantity" + ], + "properties": { + "product": { + "type": "string", + "minLength": 1 + }, + "quantity": { + "type": "integer", + "minimum": 1 + } + } +} diff --git a/bench/tasks/bluebottlecoffee.com/bluebottle-checkout-espresso.json b/bench/tasks/bluebottlecoffee.com/bluebottle-checkout-espresso.json new file mode 100644 index 00000000..0f14b4f4 --- /dev/null +++ b/bench/tasks/bluebottlecoffee.com/bluebottle-checkout-espresso.json @@ -0,0 +1,136 @@ +{ + "id": "bluebottle-checkout-espresso", + "domain": "bluebottlecoffee.com", + "stage": "prepare", + "prompt": "On bluebottlecoffee.com, put one bag of {{coffee}} in the cart as a one-time order, not a subscription. Go to checkout and fill it in with these details, then stop without submitting. Email: {{email}}. Full name: {{name}}. Address: {{address}}, {{city}}, {{state}} {{zip}}, United States. Phone: {{phone}}, United States. Card number {{card_number}}, expiration {{card_expiry}}, CVC {{card_cvc}}, billing address same as shipping. Leave marketing and save-my-information options unticked, and do not tick \"I am an AI agent\". Reply with JSON: {\"product\": string, \"quantity\": integer}.", + "fixture": { + "coffee": "Opascope Espresso", + "email": "webmcp-demo@example.com", + "name": "Alex Demo", + "address": "500 Sansome Street", + "city": "San Francisco", + "state": "CA", + "zip": "94111", + "phone": "4155550123", + "card_number": "4242424242424242", + "card_expiry": "08/31", + "card_cvc": "500" + }, + "start_url": "https://bluebottlecoffee.com/", + "tools": [ + { + "repo": "kernel/xwebmcp", + "ref": "main", + "names": [ + "find_product", + "get_product", + "add_to_cart", + "continue_to_checkout" + ] + } + ], + "allow": [ + "checkout", + "payment_details" + ], + "guard": { + "hosts": [ + "https://checkout.bluebottlecoffee.com/*" + ], + "buttons": "^(pay|complete order|place order|subscribe)\\b(?! with)", + "form_submit": true, + "requests": "/confirm(?:[/?#]|$)|payment_intents|setup_intents" + }, + "entry_tool": "find_product", + "answer_schema": "schemas/checkout-order.json", + "rules": [ + { + "path": "product", + "matches": "{{coffee}}" + }, + { + "path": "quantity", + "equals": 1 + } + ], + "check": { + "fields": { + "email": "#email", + "name": "#shippingName", + "address": "#shippingAddressLine1", + "city": "#shippingLocality", + "zip": "#shippingPostalCode", + "state": "#shippingAdministrativeArea", + "phone": "#phoneNumber", + "card_number": "#cardNumber", + "card_expiry": "#cardExpiry", + "card_cvc": "#cardCvc" + }, + "rules": [ + { + "path": "url", + "matches": "^https://checkout\\.bluebottlecoffee\\.com/" + }, + { + "path": "missing", + "count": [ + 0, + 0 + ] + }, + { + "path": "invalid", + "count": [ + 0, + 0 + ] + }, + { + "path": "fields.email", + "equals": "{{email}}" + }, + { + "path": "fields.name", + "equals": "{{name}}" + }, + { + "path": "fields.address", + "equals": "{{address}}" + }, + { + "path": "fields.city", + "equals": "{{city}}" + }, + { + "path": "fields.zip", + "equals": "{{zip}}" + }, + { + "path": "fields.state", + "in": [ + "CA", + "California" + ] + }, + { + "path": "fields.phone", + "matches": "^\\D*(1\\D*)?415\\D*555\\D*0123$" + }, + { + "path": "fields.card_number", + "matches": "^4242\\D*4242\\D*4242\\D*4242$" + }, + { + "path": "fields.card_expiry", + "matches": "^08\\D*31$" + }, + { + "path": "fields.card_cvc", + "equals": "{{card_cvc}}" + } + ] + }, + "pairs": 5, + "trial_timeout_sec": 600, + "max_turns": 60 +} diff --git a/bench/tasks/bluebottlecoffee.com/domain.json b/bench/tasks/bluebottlecoffee.com/domain.json new file mode 100644 index 00000000..538a2f29 --- /dev/null +++ b/bench/tasks/bluebottlecoffee.com/domain.json @@ -0,0 +1,8 @@ +{ + "actions": { + "add_to_cart": "adds a product to the guest cart" + }, + "checkout_actions": { + "continue_to_checkout": "opens checkout from the cart; submits nothing" + } +} diff --git a/bench/tasks/delta.com/delta-cart-main-classic.json b/bench/tasks/delta.com/delta-cart-main-classic.json new file mode 100644 index 00000000..11bf8c31 --- /dev/null +++ b/bench/tasks/delta.com/delta-cart-main-classic.json @@ -0,0 +1,35 @@ +{ + "id": "delta-cart-main-classic", + "domain": "delta.com", + "stage": "prepare", + "prompt": "On delta.com, put the cheapest Delta Main Classic one-way fare for 1 adult from {{origin}} to {{destination}} on {{date}} in the cart, and stop on Delta's Trip Summary page with that flight in the cart. You may select the flight and its fare; do not go on to passenger details, seats or payment. Reply with JSON: {\"flight_numbers\": [string], \"amount_due\": number in US dollars, the total Delta shows on the Trip Summary page}.", + "fixture": { + "origin": "Atlanta (ATL)", + "destination": "Orlando (MCO)", + "date": {"days_from_start": 28} + }, + "start_url": "https://www.delta.com/", + "tools": [{"repo": "kernel/xwebmcp", "ref": "main", "names": ["search_flights", "list_flight_results", "select_fare", "select_fare_experience", "get_trip_summary"]}], + "entry_tool": "search_flights", + "answer_schema": "schemas/cart-flight.json", + "rules": [{"path": "amount_due", "range": [1, 3000]}], + "check": { + "extract": { + "amount_due": "Amount Due[^$]{0,40}\\$\\s*([\\d,]+\\.\\d{2})", + "cabin": "Cabin (?:class|name)\\s*\\n\\s*([^\\n]+)", + "passengers": "(\\d+) Passengers?\\b" + }, + "rules": [ + {"path": "url", "matches": "^https://www\\.delta\\.com/(purchase/flights|completepurchase)/trip-summary"}, + {"path": "text", "matches": "\\bATL - MCO\\s*\\n\\s*One Way\\b"}, + {"path": "extracted.cabin", "equals": "Main Classic"}, + {"path": "extracted.passengers", "equals": 1}, + {"path": "extracted.amount_due", "range": [1, 3000]} + ], + "same_as_answer": [{"answer": "amount_due", "end_state": "extracted.amount_due"}] + }, + "agreement": {"path": "amount_due", "better": "lower", "tolerance": 1}, + "pairs": 5, + "trial_timeout_sec": 900, + "max_turns": 60 +} diff --git a/bench/tasks/delta.com/delta-checkout-signed-in.json b/bench/tasks/delta.com/delta-checkout-signed-in.json new file mode 100644 index 00000000..672fe4ea --- /dev/null +++ b/bench/tasks/delta.com/delta-checkout-signed-in.json @@ -0,0 +1,113 @@ +{ + "id": "delta-checkout-signed-in", + "domain": "delta.com", + "stage": "prepare", + "opt_in": "its guard has not been checked on Delta's live Review and Pay", + "profile": "bench-delta", + "signed_in_check": { + "tool": "get_page_context", + "rules": [ + { + "path": "signed_in", + "equals": true + } + ] + }, + "prompt": "On delta.com, where you are signed in to a SkyMiles account, book the cheapest Delta Main Classic one-way fare for 1 adult from {{origin}} to {{destination}} on {{date}} as far as the Payment section of Review and Pay: select the flight and its fare, continue from Trip Summary without seats or upgrades, keep the passenger details the account fills in, and decline trip protection. Stop when Delta asks for payment, and do not enter a card. Reply with JSON: {\"flight_numbers\": [string], \"amount_due\": number in US dollars, the Amount Due Delta shows}.", + "fixture": { + "origin": "Atlanta (ATL)", + "destination": "Orlando (MCO)", + "date": { + "days_from_start": 28 + } + }, + "start_url": "https://www.delta.com/", + "tools": [ + { + "repo": "kernel/xwebmcp", + "ref": "main", + "names": [ + "search_flights", + "list_flight_results", + "select_fare", + "select_fare_experience", + "get_trip_summary", + "continue_to_review_and_pay", + "continue_checkout", + "get_checkout_step", + "set_trip_protection" + ] + } + ], + "allow": [ + "checkout" + ], + "guard": { + "hosts": [ + "https://www.delta.com/purchase/*", + "https://www.delta.com/completepurchase/*" + ], + "buttons": "^(complete purchase|purchase|pay now|book now)\\b", + "selectors": "[data-cy^=\"complete-purchase\"]" + }, + "entry_tool": "search_flights", + "answer_schema": "schemas/cart-flight.json", + "rules": [ + { + "path": "amount_due", + "range": [ + 1, + 3000 + ] + } + ], + "check": { + "extract": { + "amount_due": "Amount Due[^$]{0,40}\\$\\s*([\\d,]+\\.\\d{2})", + "passengers": "(\\d+) Passengers?\\b" + }, + "rules": [ + { + "path": "url", + "matches": "^https://www\\.delta\\.com/(purchase/flights/(review-pay|express-checkout)|completepurchase/review-pay)" + }, + { + "path": "text", + "matches": "\\bATL - MCO\\s*\\n\\s*One Way\\b" + }, + { + "path": "text", + "matches": "Main Classic" + }, + { + "path": "text", + "matches": "\\bPayment\\b" + }, + { + "path": "extracted.passengers", + "equals": 1 + }, + { + "path": "extracted.amount_due", + "range": [ + 1, + 3000 + ] + } + ], + "same_as_answer": [ + { + "answer": "amount_due", + "end_state": "extracted.amount_due" + } + ] + }, + "agreement": { + "path": "amount_due", + "better": "lower", + "tolerance": 1 + }, + "pairs": 5, + "trial_timeout_sec": 900, + "max_turns": 60 +} diff --git a/bench/tasks/delta.com/domain.json b/bench/tasks/delta.com/domain.json index 3cc3d252..645effa7 100644 --- a/bench/tasks/delta.com/domain.json +++ b/bench/tasks/delta.com/domain.json @@ -1,5 +1,17 @@ { "needs_sign_in": { "list_payment_methods": "reads a signed-in SkyMiles member's wallet" + }, + "actions": { + "select_fare": "picks a fare on the results page; adds nothing to pay", + "select_fare_experience": "picks a fare experience and creates the cart" + }, + "checkout_actions": { + "open_seat_map": "opens the seat map; chooses no seat", + "select_seat": "holds a seat, refusing any with a fee unless the task allows one", + "skip_seat_selection": "leaves the seat map without saving", + "continue_to_review_and_pay": "moves from Trip Summary to Review and Pay; enters no data", + "continue_checkout": "saves one checkout step; never presses Complete Purchase", + "set_trip_protection": "accepts or declines trip protection" } } diff --git a/bench/test/agent.test.mjs b/bench/test/agent.test.mjs index 90b5de1b..cc9e2e01 100644 --- a/bench/test/agent.test.mjs +++ b/bench/test/agent.test.mjs @@ -191,11 +191,12 @@ test("the runner stops pi at the turn, spend and time caps, and when a cost is m test("a failed invocation keeps its error, whether the call, the tool or the MCP server failed", () => { const invoke = (id) => call(id, "mcp__kernel__webmcp", { action: "invoke", tool_ref: "wmcp_a" }); const summary = summarize([ - assistant([invoke("1"), invoke("2"), invoke("3"), invoke("4")]), + assistant([invoke("1"), invoke("2"), invoke("3"), invoke("4"), invoke("5")]), end("1", "mcp__kernel__webmcp", JSON.stringify({ status: "completed", output: { status: "error", message: "UI_CHANGED: missing button" } }), 10), end("2", "mcp__kernel__webmcp", JSON.stringify({ status: "error", error_text: "input failed JSON Schema validation" }), 20), end("3", "mcp__kernel__webmcp", 'error in webmcp (invoke): {"code":"not_found"}', 30), end("4", "mcp__kernel__webmcp", JSON.stringify({ status: "completed", output: { status: "no_results", message: "nothing matched" } }), 40), + end("5", "mcp__kernel__webmcp", JSON.stringify({ status: "awaiting_submission", output: null }), 50), ], allow); const shown = summary.webmcp.invocations.map(i => [i.status, i.tool_status, i.error, invocationFailed(i)]); assert.deepEqual(shown, [ @@ -203,6 +204,7 @@ test("a failed invocation keeps its error, whether the call, the tool or the MCP ["error", null, "input failed JSON Schema validation", true], ["error", null, 'error in webmcp (invoke): {"code":"not_found"}', true], ["completed", "no_results", null, false], + ["awaiting_submission", null, null, false], ]); }); diff --git a/bench/test/grade.test.mjs b/bench/test/grade.test.mjs index c6c4a2b6..68f77e96 100644 --- a/bench/test/grade.test.mjs +++ b/bench/test/grade.test.mjs @@ -1,8 +1,13 @@ import assert from "node:assert/strict"; +import { readFileSync } from "node:fs"; +import { dirname, join } from "node:path"; import test from "node:test"; -import { checkRule, extractAnswer, gradeAnswer, resolvePath } from "../grade.mjs"; +import { fileURLToPath } from "node:url"; +import { checkRule, checkSameAsAnswer, extractAnswer, gradeAnswer, resolvePath } from "../grade.mjs"; import { fill } from "../lib/intents.mjs"; +const benchDir = join(dirname(fileURLToPath(import.meta.url)), ".."); + const schema = { type: "object", required: ["ip", "country_code", "asn"], @@ -78,3 +83,44 @@ test("a matches rule takes fixture values literally", () => { assert.equal(grade("verity (2026)"), "correct"); assert.equal(grade("Verity 2026"), "wrong"); }); + +test("a prepare intent is correct only when the end state passes its check", () => { + const prepare = { ...intent, check: { rules: [{ path: "step", equals: "trip_summary" }, { path: "slices[].origin", matches: "ATL" }] } }; + const text = '{"ip": "8.8.8.8", "country_code": "US", "asn": "AS15169"}'; + const grade = (endState) => gradeAnswer(prepare, schema, text, endState); + assert.deepEqual([grade({ output: { step: "trip_summary", slices: [{ origin: "ATL" }] } }).outcome, grade({ output: { step: "trip_summary", slices: [{ origin: "ATL" }] } }).end_state], ["correct", true]); + const elsewhere = grade({ output: { step: "other", slices: [] } }); + assert.deepEqual([elsewhere.outcome, elsewhere.end_state, elsewhere.problems], ["wrong", false, ['end state: step: "other" is not "trip_summary"', "end state: slices[].origin: no value"]]); + assert.equal(gradeAnswer(intent, schema, text).end_state, null); +}); + +test("a prepare intent's answer must report what the runner read on the page", () => { + const prepare = { ...intent, check: { rules: [], same_as_answer: [{ answer: "asn", end_state: "connection.asn" }, { answer: "ip", end_state: "ips[]" }] } }; + const text = '{"ip": "8.8.8.8", "country_code": "US", "asn": "AS15169"}'; + const grade = (output) => gradeAnswer(prepare, schema, text, { output }); + assert.equal(grade({ connection: { asn: "as 15169" }, ips: ["8.8.8.8"] }).outcome, "correct"); + const elsewhere = grade({ connection: { asn: "AS15170" }, ips: ["8.8.8.8"] }); + assert.deepEqual([elsewhere.outcome, elsewhere.end_state, elsewhere.problems], ["wrong", false, ['end state: asn is ["as15169"] in the answer but connection.asn is ["as15170"] on the site']]); + assert.equal(grade({ connection: { asn: "AS15169" }, ips: [] }).outcome, "wrong"); +}); + +test("amounts match to the cent", () => { + assert.equal(checkSameAsAnswer({ answer: "total", end_state: "amount_due" }, { total: 143.4 }, { amount_due: 143.40000001 }), null); + assert.match(checkSameAsAnswer({ answer: "total", end_state: "amount_due" }, { total: 143.4 }, { amount_due: 150.2 }), /\[143\.4\] in the answer but amount_due is \[150\.2\]/); + assert.match(checkSameAsAnswer({ answer: "total", end_state: "amount_due" }, { total: 143.4 }, { amount_due: null }), /\[null\] on the site/); +}); + +test("the espresso checkout grades the fields read from the page", () => { + const intent = JSON.parse(readFileSync(join(benchDir, "tasks/bluebottlecoffee.com/bluebottle-checkout-espresso.json"), "utf8")); + const espressoSchema = JSON.parse(readFileSync(join(benchDir, intent.answer_schema), "utf8")); + const page = { + url: "https://checkout.bluebottlecoffee.com/c/pay/cs_live_x", missing: [], invalid: [], + fields: { email: "webmcp-demo@example.com", name: "Alex Demo", address: "500 Sansome Street", city: "San Francisco", zip: "94111", state: "CA", phone: "(415) 555-0123", card_number: "4242 4242 4242 4242", card_expiry: "08 / 31", card_cvc: "500" }, + }; + const grade = (output) => gradeAnswer(intent, espressoSchema, '{"product": "Opascope Espresso", "quantity": 1}', { output }); + assert.deepEqual(grade(page).problems, []); + assert.equal(grade({ ...page, fields: { ...page.fields, phone: "+1 415-555-0123" } }).outcome, "correct"); + assert.match(grade({ ...page, fields: { ...page.fields, card_number: "4000 0000 0000 0002" } }).problems[0], /card_number/); + assert.match(grade({ ...page, invalid: ["card_cvc"] }).problems[0], /invalid: count outside/); + assert.match(grade({ ...page, url: "https://bluebottlecoffee.com/us/eng" }).problems[0], /url/); +}); diff --git a/bench/test/page.test.mjs b/bench/test/page.test.mjs new file mode 100644 index 00000000..8ca4e4b7 --- /dev/null +++ b/bench/test/page.test.mjs @@ -0,0 +1,39 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import { extractValues } from "../lib/page.mjs"; +import { gradeAnswer } from "../grade.mjs"; + +test("extract patterns read values from the page text, amounts as numbers", () => { + const text = "Trip Summary\nATL to MCO\nDelta Main Classic\nAmount Due USD $143.40\nFlights $1,120.00"; + assert.deepEqual(extractValues(text, { amount_due: "Amount Due\\s*(?:USD\\s*)?\\$\\s*([\\d,]+\\.\\d{2})", flights: "Flights\\s*(\\$[\\d,.]+)", cabin: "Delta (Main \\w+)", seats: "Seat (\\d+[A-F])" }), + { amount_due: 143.4, flights: 1120, cabin: "Main Classic" }); + // A group that takes no part in the match is left out, like a pattern that does not match. + assert.deepEqual(extractValues("Amount Due", { amount_due: "Amount Due(?: \\$([\\d.]+))?" }), {}); +}); + +test("a page check grades the page the agent left, not a tool's reading of it", () => { + const intent = { + rules: [], fixture: {}, + check: { + extract: { amount_due: "Amount Due\\s*(?:USD\\s*)?\\$\\s*([\\d,]+\\.\\d{2})" }, + rules: [{ path: "url", matches: "/trip-summary" }, { path: "text", matches: "Main Classic" }, { path: "extracted.amount_due", range: [1, 3000] }], + same_as_answer: [{ answer: "amount_due", end_state: "extracted.amount_due" }], + }, + }; + const schema = { type: "object", required: ["amount_due"], properties: { amount_due: { type: "number" } } }; + const page = (text) => ({ output: { url: "https://www.delta.com/purchase/flights/trip-summary", title: "Trip Summary", text, fields: {}, missing: [], invalid: [] } }); + const answer = '```json\n{"amount_due": 143.4}\n```'; + const correct = gradeAnswer(intent, schema, answer, page("Delta Main Classic\nAmount Due USD $143.40")); + assert.equal(correct.outcome, "correct"); + assert.deepEqual(correct.end_state_values.extracted, { amount_due: 143.4 }); + assert.match(gradeAnswer(intent, schema, answer, page("Delta Main Classic\nAmount Due USD $151.20")).problems.join(), /amount_due is \[143\.4\] in the answer but extracted\.amount_due is \[151\.2\]/); + assert.match(gradeAnswer(intent, schema, answer, page("Delta Main Basic\nAmount Due USD $143.40")).problems.join(), /text: .* does not match \/Main Classic\/i/); + assert.match(gradeAnswer(intent, schema, answer, page("Delta Main Classic")).problems.join(), /extracted\.amount_due: no value/); +}); + +test("a failing rule on long page text shows only the start of it", () => { + const intent = { rules: [], fixture: {}, check: { rules: [{ path: "text", matches: "Main Classic" }] } }; + const schema = { type: "object" }; + const grade = gradeAnswer(intent, schema, "```json\n{}\n```", { output: { url: "u", text: "x".repeat(50_000) } }); + assert.ok(grade.problems[0].length < 200, grade.problems[0].length); +}); diff --git a/bench/test/run.test.mjs b/bench/test/run.test.mjs index f96409d9..b0887736 100644 --- a/bench/test/run.test.mjs +++ b/bench/test/run.test.mjs @@ -6,39 +6,51 @@ import { delimiter, dirname, join } from "node:path"; import test from "node:test"; import { fileURLToPath } from "node:url"; -// Runs run.mjs end to end against stub pi, kernel and hypeship commands, on one signed-in intent -// whose tools come from a local repository standing in for kernel/xwebmcp. Each trial that reaches -// its agent waits 10s after load, so a pair takes about 20s. +// Runs run.mjs end to end against stub pi, kernel and hypeship commands, on one intent whose tools +// come from a local repository standing in for kernel/xwebmcp. Each trial that reaches its agent +// waits 10s after load, so a pair takes about 20s. const benchDir = join(dirname(fileURLToPath(import.meta.url)), ".."); const START_DELAY_MS = 1000; -function workspace() { +const lookupIntent = { + id: "example-lookup", domain: "example.com", stage: "inspect", + prompt: "On example.com, look up 8.8.8.8. Reply with JSON: {\"ip\": string, \"country_code\": string, \"asn\": string}.", + start_url: "https://example.com/", + tools: [{ repo: "kernel/xwebmcp", ref: "main", names: ["lookup"] }], + entry_tool: "lookup", answer_schema: "schemas/ip-lookup.json", + rules: [{ path: "asn", equals: "AS15169" }], + pairs: 1, trial_timeout_sec: 60, max_turns: 5, +}; + +const checkoutIntent = { + ...lookupIntent, id: "example-checkout", stage: "prepare", + tools: [{ repo: "kernel/xwebmcp", ref: "main", names: ["lookup", "add"] }], + allow: ["checkout", "payment_details"], + guard: { hosts: ["https://example.com/*"], buttons: "^pay$", form_submit: true }, + check: { fields: { email: "#email" }, rules: [{ path: "fields.email", equals: "a@example.com" }] }, +}; + +function workspace({ domain = { profile: "bench-example", signed_in_check: { tool: "lookup" } }, intent = lookupIntent } = {}) { const dir = mkdtempSync(join(tmpdir(), "bench-run-")); const bench = join(dir, "bench"); for (const path of ["run.mjs", "grade.mjs", "lib", "prompts", "allowlist.txt", "mcp.json", "allowlist.local.txt", "mcp.local.json", "schemas"]) cpSync(join(benchDir, path), join(bench, path), { recursive: true }); cpSync(join(benchDir, "../extension/lib/url-pattern.js"), join(dir, "extension/lib/url-pattern.js")); mkdirSync(join(bench, "tasks/example.com"), { recursive: true }); - writeFileSync(join(bench, "tasks/example.com/domain.json"), JSON.stringify({ profile: "bench-example", signed_in_check: { tool: "lookup" } })); - writeFileSync(join(bench, "tasks/example.com/example-lookup.json"), JSON.stringify({ - id: "example-lookup", domain: "example.com", stage: "inspect", - prompt: "On example.com, look up 8.8.8.8. Reply with JSON: {\"ip\": string, \"country_code\": string, \"asn\": string}.", - start_url: "https://example.com/", - tools: [{ repo: "kernel/xwebmcp", ref: "main", names: ["lookup"] }], - entry_tool: "lookup", answer_schema: "schemas/ip-lookup.json", - rules: [{ path: "asn", equals: "AS15169" }], - pairs: 1, trial_timeout_sec: 60, max_turns: 5, - })); + writeFileSync(join(bench, "tasks/example.com/domain.json"), JSON.stringify(domain)); + writeFileSync(join(bench, `tasks/example.com/${intent.id}.json`), JSON.stringify(intent)); const source = join(dir, "source"); mkdirSync(join(source, "domains/example.com"), { recursive: true }); - writeFileSync(join(source, "registry.json"), JSON.stringify({ tools: [{ namespace: "example.com", file: "domains/example.com/lookup.js" }] })); - writeFileSync(join(source, "domains/example.com/lookup.js"), `[{ - kind: "page", - match: { url_patterns: ["https://example.com/*"] }, - tool: { name: "lookup", annotations: { readOnlyHint: true } }, - execute: async () => ({ status: "ok" }), - }]`); + writeFileSync(join(source, "registry.json"), JSON.stringify({ tools: ["lookup", "add"].map(name => ({ namespace: "example.com", file: `domains/example.com/${name}.js` })) })); + for (const [name, readOnly] of [["lookup", true], ["add", false]]) { + writeFileSync(join(source, `domains/example.com/${name}.js`), `[{ + kind: "page", + match: { url_patterns: ["https://example.com/*"] }, + tool: { name: "${name}", annotations: { readOnlyHint: ${readOnly} } }, + execute: async () => ({ status: "ok" }), + }]`); + } const git = (...args) => execFileSync("git", ["-C", source, ...args], { stdio: "ignore" }); git("init", "--quiet", "--initial-branch=main"); git("add", "."); @@ -158,3 +170,44 @@ test("a Kernel run resumed with --browser local still uses Kernel browsers", { t assert.deepEqual(rows("backend").filter(r => r.pair === 2).map(r => r.outcome), ["correct", "correct"]); assert.ok(log().filter(entry => entry[0] === "create").length > created); }); + +test("a checkout intent loads the guard in both arms and fails a trial that tried to submit", { timeout: 120_000 }, () => { + const { run, rows, log } = workspace({ domain: { actions: { add: "adds to the cart" } }, intent: checkoutIntent }); + const result = run("checkout", { STUB_SUBMIT_ARMS: "treatment" }); + assert.equal(result.status, 0, result.stderr + result.stdout); + const uploads = log().filter(entry => entry[0] === "upload"); + assert.equal(uploads.length, 1); + const extension = uploads[0][1]; + assert.match(extension, /^bench-guard-[0-9a-f]{12}$/); + // The smoke gate's browser and both arms' browsers load it. + assert.deepEqual(log().filter(entry => entry[0] === "create").map(entry => entry[3]), [extension, extension, extension]); + const outcomes = Object.fromEntries(rows("checkout").map(r => [r.arm, [r.outcome, r.reason, r.submit_blocked]])); + assert.deepEqual(outcomes.control, ["correct", null, 0]); + assert.deepEqual(outcomes.treatment, ["crashed", "tried to submit: the guard blocked 1 attempt(s)", 1]); + + // A second run reuses the uploaded extension. + assert.equal(run("checkout-again").status, 0); + assert.equal(log().filter(entry => entry[0] === "upload").length, 1); +}); + +test("a checkout trial whose guard count or end state cannot be read aborts", { timeout: 120_000 }, () => { + const { run, rows } = workspace({ domain: { actions: { add: "adds to the cart" } }, intent: checkoutIntent }); + for (const [runId, env, reason] of [["no-cookies", { STUB_COOKIES_FAIL_ARMS: "treatment" }, /could not read the guard's count/], + ["no-page", { STUB_PAGE_FAIL_ARMS: "treatment" }, /could not read the end state/]]) { + const result = run(runId, env); + assert.equal(result.status, 0, result.stderr + result.stdout); + const treatment = rows(runId).filter(r => r.arm === "treatment"); + assert.deepEqual(treatment.map(r => r.outcome), ["aborted", "aborted"], runId); + for (const row of treatment) assert.match(row.reason, reason); + } +}); + +test("an opt_in intent runs only when named with --intent", { timeout: 120_000 }, () => { + const { run } = workspace({ intent: { ...lookupIntent, opt_in: "not rehearsed" } }); + const unnamed = run("opt-in"); + assert.equal(unnamed.status, 1); + assert.match(unnamed.stdout, /skipping example-lookup: not rehearsed; name it with --intent/); + assert.match(unnamed.stderr, /no intents selected/); + const named = run("opt-in-named", {}, ["--intent", "example-lookup"]); + assert.equal(named.status, 0, named.stderr + named.stdout); +}); diff --git a/bench/test/stubs/kernel b/bench/test/stubs/kernel index a3a06118..5c0f8b48 100755 --- a/bench/test/stubs/kernel +++ b/bench/test/stubs/kernel @@ -1,9 +1,12 @@ #!/usr/bin/env node -// A stand-in for the kernel CLI in run.test.mjs. Browsers live in $STUB_STATE. Every browser's page -// is example.com; the sign-in check fails in browsers tagged with an arm listed in -// $STUB_SIGNED_OUT_ARMS, every page also lists the native tools named in $STUB_NATIVE_TOOLS, in -// browsers tagged with an arm listed in $STUB_LIVE_CUSTOM_ARMS a custom tool is live without being registered, -// and in those tagged with an arm listed in $STUB_OFF_PAGE_ARMS registered tools are listed without a page. +// A stand-in for the kernel CLI in run.test.mjs. Browsers and extensions live in $STUB_STATE. Every +// browser's page is example.com; the sign-in check fails in browsers tagged with an arm listed in +// $STUB_SIGNED_OUT_ARMS, the submission guard reports one blocked attempt in browsers tagged with +// an arm listed in $STUB_SUBMIT_ARMS, every page also lists the native tools named in +// $STUB_NATIVE_TOOLS, in browsers tagged with an arm listed in $STUB_LIVE_CUSTOM_ARMS a custom tool +// is live without being registered, and in those tagged with an arm listed in $STUB_OFF_PAGE_ARMS +// registered tools are listed without a page. Reading cookies fails in browsers tagged with an arm +// listed in $STUB_COOKIES_FAIL_ARMS, and reading the page in those listed in $STUB_PAGE_FAIL_ARMS. import { readFileSync, writeFileSync } from "node:fs"; const args = process.argv.slice(2).filter(arg => arg !== "-o" && arg !== "json"); @@ -19,14 +22,37 @@ if (args[0] === "--version") { console.log("kernel 0.0.0-stub (stub)"); } else if (command.startsWith("profiles get")) { print({ id: `profile-${args[2]}` }); +} else if (command.startsWith("extensions get")) { + if (!state.extensions?.includes(args[2])) { + console.error("ERROR: Not_found: extension not found"); + process.exit(1); + } + print({ name: args[2] }); +} else if (command.startsWith("extensions upload")) { + state.extensions = [...(state.extensions ?? []), flag("--name")]; + log(["upload", flag("--name")]); + print({ name: flag("--name") }); } else if (command.startsWith("browsers create")) { const id = `b${++state.next}`; const tags = Object.fromEntries(args.flatMap((arg, i) => (arg === "--tag" ? [args[i + 1].split("=")] : []))); state.browsers[id] = { tags, tools: [] }; - log(["create", id, tags.arm]); + log(["create", id, tags.arm, args.includes("--extension") ? flag("--extension") : null]); print({ session_id: id }); } else if (command.startsWith("browsers playwright execute")) { - print({ success: true, result: { url: "https://example.com/", version: "stub", renavigated: false } }); + const failing = (name) => (process.env[name] ?? "").split(",").includes(state.browsers[args[3]].tags.arm); + if (args[4].includes("context.cookies") && failing("STUB_COOKIES_FAIL_ARMS")) { + print({ success: false, error: "Target page, context or browser has been closed" }); + } else if (args[4].includes("fields") && failing("STUB_PAGE_FAIL_ARMS")) { + print({ success: false, error: "Execution context was destroyed" }); + } else if (args[4].includes("context.cookies")) { + // The guard's cookie sits on the checkout host, not on the page the agent ended on. + const blocked = (process.env.STUB_SUBMIT_ARMS ?? "").split(",").includes(state.browsers[args[3]].tags.arm); + print({ success: true, result: [{ name: "session", value: "x", domain: "example.com" }, ...(blocked ? [{ name: "bench_submit_blocked", value: "1", domain: "checkout.example.com" }] : [])] }); + } else if (args[4].includes("fields")) { + print({ success: true, result: { url: "https://example.com/checkout", title: "Checkout", text: "Checkout", fields: { email: "a@example.com" }, missing: [], invalid: [] } }); + } else { + print({ success: true, result: { url: "https://example.com/", version: "stub", renavigated: false } }); + } } else if (command.startsWith("browsers webmcp custom-tools add")) { const browser = state.browsers[args[4]]; browser.tools = [{ id: `ct_${++state.next}`, namespace: flag("--namespace"), tool: { name: "lookup" } }]; diff --git a/bench/test/tasks.test.mjs b/bench/test/tasks.test.mjs index d182518f..75a155cb 100644 --- a/bench/test/tasks.test.mjs +++ b/bench/test/tasks.test.mjs @@ -4,7 +4,7 @@ import { tmpdir } from "node:os"; import { dirname, join } from "node:path"; import test from "node:test"; import { fileURLToPath } from "node:url"; -import { fill, loadIntents, loadSchema, pastDates, resolveFixture, trialPrompt, validateDomain, validateIntent } from "../lib/intents.mjs"; +import { ALLOW, fill, loadIntents, loadSchema, pastDates, resolveFixture, trialPrompt, validateDomain, validateIntent } from "../lib/intents.mjs"; import { checkSchema } from "../lib/schema.mjs"; import { composeBatch, evaluateDefinitions, matchesUrlPattern, normalize, registeredTools, resolveSource } from "../lib/tools.mjs"; @@ -185,7 +185,37 @@ test("a file that declares more than one tool is refused", async () => { test("registration fails on a name that no source has or that would not be registered", async () => { await assert.rejects(registeredTools({ ...base, exclude: { whoamii: "typo" }, tools: [{ ref: "branch", names: [] }] }, ["branch"], show), /exclude names whoamii, which no source has/); await assert.rejects(registeredTools({ ...base, tools: [{ ref: "main", names: ["get_item"] }] }, ["main"], show), /get_item is not in registry.json at main/); - await assert.rejects(registeredTools({ ...base, tools: [{ ref: "main", names: ["save"] }] }, ["main"], show), /save is not read-only or is excluded/); + await assert.rejects(registeredTools({ ...base, tools: [{ ref: "main", names: ["save"] }] }, ["main"], show), /save is excluded or neither read-only nor an action/); +}); + +test("a prepare intent needs a check of the page, and only prepare intents take checks and allow", () => { + const intent = JSON.parse(readFileSync(join(benchDir, "tasks/delta.com/delta-cart-main-classic.json"), "utf8")); + assert.deepEqual(validateIntent(intent, benchDir), []); + const { check, ...unchecked } = intent; + assert.ok(validateIntent(unchecked, benchDir).some(p => p.startsWith("check and allow"))); + assert.ok(validateIntent({ ...intent, stage: "find" }, benchDir).some(p => p.startsWith("check and allow"))); + assert.ok(validateIntent({ ...intent, check: { ...check, rules: [] } }, benchDir).includes("check needs rules")); + // The end state is read from the page, never through the tools under test. + assert.ok(validateIntent({ ...intent, check: { ...check, tool: "get_trip_summary" } }, benchDir).includes("check takes fields, extract, rules and same_as_answer")); + assert.ok(validateIntent({ ...intent, check: { ...check, extract: { amount_due: "Amount Due \\$[\\d.]+" } } }, benchDir).includes("check.extract must map names to regexes with one capturing group")); + assert.ok(validateIntent({ ...intent, actions: ["select_fare"] }, benchDir).includes("actions belong in the domain's domain.json")); + assert.ok(validateIntent({ ...intent, check: { ...check, rules: [{ path: "step", equals: "{{cabin}}" }] } }, benchDir).includes("{{cabin}} has no fixture value")); + assert.ok(validateIntent({ ...intent, check: { ...check, same_as_answer: [{ answer: "amount_due" }] } }, benchDir).includes("check.same_as_answer must list {answer, end_state} paths")); +}); + +test("a prepare intent's actions are registered, and must exist", async () => { + const intent = { ...base, exclude: {}, stage: "prepare", actions: ["save"], tools: [{ ref: "main", names: ["save"] }] }; + assert.deepEqual((await registeredTools(intent, ["main"], show)).map(t => t.name), ["save", "search"]); + await assert.rejects(registeredTools({ ...intent, actions: ["delete"] }, ["main"], show), /actions names delete, which no source has/); + await assert.rejects(registeredTools({ ...intent, actions: [] }, ["main"], show), /save is excluded or neither/); +}); + +test("a domain's actions reach its prepare intents, and its checkout actions only those that allow checkout", () => { + const delta = loadIntents(benchDir).filter(i => i.domain === "delta.com"); + const actions = (id) => delta.find(i => i.id === id).actions; + assert.deepEqual(actions("delta-cheapest-fare"), []); + assert.deepEqual(actions("delta-cart-main-classic"), ["select_fare", "select_fare_experience"]); + assert.ok(actions("delta-checkout-signed-in").includes("continue_checkout")); }); test("local browsers load the extension under the id Chrome gives it, and their MCP config fills in per browser", async () => { @@ -231,3 +261,78 @@ test("a bridge answer split inside a character is read whole, and an unreadable server.close(); } }); + +test("checkout intents need a guard, and fields checks name CSS selectors", () => { + const intent = JSON.parse(readFileSync(join(benchDir, "tasks/bluebottlecoffee.com/bluebottle-checkout-espresso.json"), "utf8")); + assert.deepEqual(validateIntent(intent, benchDir), []); + const { guard, ...unguarded } = intent; + assert.ok(validateIntent(unguarded, benchDir).includes("an intent that allows checkout needs a guard, and only such an intent has one")); + assert.ok(validateIntent({ ...intent, allow: ["payment_details"], guard: undefined }, benchDir).includes("allow payment_details needs checkout")); + assert.ok(validateIntent({ ...intent, allow: ["checkout", "pay"] }, benchDir).some(p => p.startsWith("allow must list"))); + assert.ok(validateIntent({ ...intent, guard: { ...guard, hosts: ["checkout.bluebottlecoffee.com"] } }, benchDir).some(p => p.startsWith("guard needs"))); + assert.ok(validateIntent({ ...intent, guard: { ...guard, buttons: "(" } }, benchDir).some(p => p.startsWith("guard needs"))); + assert.ok(validateIntent({ ...intent, guard: { ...guard, selectors: "" } }, benchDir).some(p => p.startsWith("guard needs"))); + // Chrome compiles requests with RE2, which has no lookarounds. + assert.ok(validateIntent({ ...intent, guard: { ...guard, requests: "/confirm(?!x)" } }, benchDir).some(p => p.startsWith("guard needs"))); + assert.deepEqual(validateIntent({ ...intent, guard: { ...guard, selectors: '[data-cy^="complete-purchase"]' } }, benchDir), []); + assert.ok(validateIntent({ ...intent, check: { ...intent.check, fields: { email: "" } } }, benchDir).includes("check.fields must map names to CSS selectors")); + assert.ok(validateIntent({ ...intent, stage: "find" }, benchDir).some(p => p.startsWith("check and allow"))); +}); + +test("an intent can name its own profile, which keeps its domain's needs_sign_in tools", () => { + const intents = loadIntents(benchDir); + const signedIn = intents.find(i => i.id === "delta-checkout-signed-in"); + assert.equal(signedIn.profile, "bench-delta"); + assert.deepEqual(signedIn.signed_in_check, { tool: "get_page_context", rules: [{ path: "signed_in", equals: true }] }); + assert.ok(!Object.hasOwn(signedIn.exclude, "list_payment_methods")); + const cart = intents.find(i => i.id === "delta-cart-main-classic"); + assert.equal(cart.profile, null); + assert.ok(Object.hasOwn(cart.exclude, "list_payment_methods")); + const raw = JSON.parse(readFileSync(join(benchDir, "tasks/delta.com/delta-checkout-signed-in.json"), "utf8")); + const { signed_in_check, ...noCheck } = raw; + assert.ok(validateIntent(noCheck, benchDir).includes("profile and signed_in_check go together")); + assert.ok(validateIntent({ ...raw, signed_in_check: { tool: "get_page_context", rules: [] } }, benchDir).some(p => p.startsWith("signed_in_check needs"))); + assert.ok(validateIntent({ ...raw, signed_in_check: { tool: "get_page_context", rules: [{ path: "signed_in", equal: true }] } }, benchDir).some(p => p.startsWith("signed_in_check rule"))); + assert.ok(validateIntent({ ...raw, opt_in: "" }, benchDir).some(p => p.startsWith("opt_in must"))); +}); + +test("allow adds the same permissions to the prompt in both arms", () => { + const intent = loadIntents(benchDir).find(i => i.id === "bluebottle-checkout-espresso"); + const prompt = trialPrompt(intent, "b"); + assert.ok(prompt.includes(ALLOW.checkout) && prompt.includes(ALLOW.payment_details)); + assert.ok(!trialPrompt(loadIntents(benchDir).find(i => i.id === "delta-checkout-signed-in"), "b").includes(ALLOW.payment_details)); +}); + +test("blocked attempts are summed over the guard's cookies on every host", async () => { + const { blockedAttempts } = await import("../lib/guard.mjs"); + assert.equal(blockedAttempts([]), 0); + assert.equal(blockedAttempts([ + { name: "bench_submit_blocked", value: "2", domain: "checkout.example.com" }, + { name: "bench_submit_blocked", value: "1", domain: "www.example.com" }, + { name: "session", value: "9", domain: "www.example.com" }, + ]), 3); +}); + +test("the guard extension runs the guard on its hosts and is named by its content", async () => { + const { writeGuardExtension } = await import("../lib/guard.mjs"); + const dir = mkdtempSync(join(tmpdir(), "bench-guard-")); + const guard = { hosts: ["https://checkout.example.com/*"], buttons: "^pay$", form_submit: true }; + const first = writeGuardExtension(join(dir, "a"), guard); + const manifest = JSON.parse(readFileSync(join(first.dir, "manifest.json"), "utf8")); + assert.deepEqual(manifest.content_scripts, [{ matches: guard.hosts, js: ["guard.js"], run_at: "document_start", all_frames: true, world: "MAIN" }]); + assert.match(readFileSync(join(first.dir, "guard.js"), "utf8"), /\{"buttons":"\^pay\$","selectors":null,"form_submit":true,"key":"bench_submit_blocked"\}\);\n$/); + // POST navigations from the guard's hosts are blocked by a network rule too, which also covers + // the isolated world page.evaluate runs in. + assert.deepEqual(manifest.permissions, ["declarativeNetRequest"]); + assert.deepEqual(JSON.parse(readFileSync(join(first.dir, "rules.json"), "utf8")).map(r => r.condition), [ + { initiatorDomains: ["checkout.example.com"], requestMethods: ["post"], resourceTypes: ["main_frame", "sub_frame"] }, + ]); + const withRequests = writeGuardExtension(join(dir, "e"), { hosts: ["https://*.example.com/*"], buttons: "^pay$", requests: "/confirm" }); + assert.deepEqual(JSON.parse(readFileSync(join(withRequests.dir, "rules.json"), "utf8")).map(r => r.condition), + [{ regexFilter: "/confirm", isUrlFilterCaseSensitive: false, resourceTypes: ["xmlhttprequest", "ping"] }]); + const plain = writeGuardExtension(join(dir, "f"), { hosts: ["https://example.com/*"], buttons: "^pay$" }); + assert.equal(JSON.parse(readFileSync(join(plain.dir, "manifest.json"), "utf8")).permissions, undefined); + assert.equal(writeGuardExtension(join(dir, "b"), guard).name, first.name); + assert.notEqual(writeGuardExtension(join(dir, "c"), { ...guard, buttons: "^buy$" }).name, first.name); + assert.notEqual(writeGuardExtension(join(dir, "d"), { ...guard, selectors: "#buy" }).name, first.name); +});