bench: task-first intents and time per step

bench/README.md

@@ -1,7 +1,12 @@
 # bench
 
 Measures, per user intent on a site, how much more often and how much faster an agent succeeds
-when the domain's read-only xwebmcp tools are registered in its Kernel browser.
+when the domain's read-only xwebmcp tools are registered in its Kernel browser, and where on the
+site agents spend their time, so the tools can be built for the steps that need them.
+
+Intents describe what a user wants done on the site, written without looking at the tool list.
+Treatment gets whatever tools the domain has; a domain with none runs control alone, and the
+report's tool backlog lists the slow steps no tool covers.
 
 Each trial runs [pi](https://github.com/earendil-works/pi) (`pi -p --mode json`) against one
 Kernel browser. Both arms get the same model, prompt, kernel MCP server, tool allowlist and
@@ -120,15 +125,18 @@ interleaved across intents; a pair's two trials run back to back in one slot.
 1. Create a browser on the start URL, tagged with the run, arm and intent, with the intent's
    submission guard loaded if it has one. If the page is still loading after 15s, navigate
    again; then wait up to 15s for `load`.
-2. Check the loaded URL matches the entry tool's URL patterns, in both arms.
-3. Treatment: register the batch, confirm the `entry_tool` is live on the tab, and on a signed-in
+2. Check the loaded URL is on the domain (or matches the `entry_tool`'s URL patterns, when the
+   intent names one), in both arms.
+3. Treatment: register the batch, confirm every tool registered and, with an `entry_tool`, that it
+   is live on the tab, and on a signed-in
    domain run the sign-in check. Control: on a signed-in domain, register the batch, run the same
    check and remove the tools again; then fail if any custom tools are registered or live. Both
    arms record the page's own WebMCP tools, which both can use.
 4. Wait until 10s after load in both arms, then launch the agent. The clock runs to
    `agent_settled`; the runner stops the agent at the time, turn or spend cap.
-5. On a guarded intent, read the guard's count of blocked submit attempts. Grade the answer,
-   delete the browser and append a row to `rows.jsonl`. Each attempt keeps its own directory
+5. On a guarded intent, read the guard's count of blocked submit attempts. Grade the answer, read
+   the trial's page navigations from the browser's telemetry to split its time into steps (Kernel
+   browsers only), delete the browser and append a row to `rows.jsonl`. Each attempt keeps its own directory
    under `trials/<intent>/` (`p1-control`, then `p1-control-2` for a rerun), named in the row's
    `trial_dir`.
 
@@ -143,17 +151,20 @@ and deletes every browser.
 
 ```json
 {
-  "id": "fandango-theater-showtimes",
-  "domain": "fandango.com",
-  "stage": "inspect",
-  "prompt": "On fandango.com, list every film and every showtime at {{theater}} in {{city}} on {{date}}. Reply with JSON: ...",
-  "fixture": {"theater": "AMC 19th St. East 6", "city": "New York, NY", "date": {"days_from_start": 2}},
-  "start_url": "https://www.fandango.com/",
-  "tools": [{"repo": "kernel/xwebmcp", "ref": "hypeship/fandango-showtimes", "pr": 58, "names": ["search_theaters", "get_theater_showtimes"]}],
-  "entry_tool": "search_theaters",
-  "answer_schema": "schemas/theater-showtimes.json",
-  "rules": [{"path": "date", "equals": "{{date}}"}, {"path": "films[].title", "unique": true}],
-  "pairs": 5, "trial_timeout_sec": 600, "max_turns": 40
+  "id": "delta-cart-main-classic",
+  "domain": "delta.com",
+  "stage": "prepare",
+  "prompt": "On delta.com, put the cheapest Delta Main Classic one-way fare for 1 adult from {{origin}} to {{destination}} on {{date}} in the cart, and stop on Delta's Trip Summary page ... Reply with JSON: ...",
+  "fixture": {"origin": "Atlanta (ATL)", "destination": "Orlando (MCO)", "date": {"days_from_start": 28}},
+  "start_url": "https://www.delta.com/",
+  "answer_schema": "schemas/cart-flight.json",
+  "rules": [{"path": "amount_due", "range": [1, 3000]}],
+  "check": {
+    "extract": {"amount_due": "Amount Due\\s*(?:USD\\s*)?\\$\\s*([\\d,]+\\.\\d{2})"},
+    "rules": [{"path": "url", "matches": "/trip-summary"}, {"path": "text", "matches": "Main Classic"}],
+    "same_as_answer": [{"answer": "amount_due", "end_state": "extracted.amount_due"}]
+  },
+  "pairs": 5, "trial_timeout_sec": 900, "max_turns": 60
 }
 ```
 
@@ -168,18 +179,22 @@ and deletes every browser.
   CSS selectors of inputs, read as `fields.<name>`, with `missing` and `invalid` lists of names.
   `same_as_answer` lists `{answer, end_state}` path pairs whose values must match, so the answer
   reports what is on the page.
+- `steps` (optional) names the task's pages, `[{"name": "trip summary", "url_patterns":
+  ["https://www.delta.com/*/trip-summary*"]}]`, for the report's step tables. Without it a step is
+  the page's host and path, with id-like segments collapsed to `*`.
 - `allow` (prepare intents only) lets the agent do more than the system prompt allows, with the
   same sentences added to both arms' prompts: `checkout` to add to the cart and go through
   checkout up to, never including, the button that pays or books, and `payment_details` to enter
   the details the task gives, which must be test data. An intent that allows `checkout` needs a
   `guard`, and see [Checkout intents](#checkout-intents).
-- `tools` lists kernel/xwebmcp refs (or full `sha`s) to register from. Treatment gets every tool
-  `registry.json` lists for the domain at those commits whose file declares
-  `readOnlyHint: true`; later sources replace earlier tools of the same name, so a PR to an
-  existing domain lists main first. Tool files are evaluated in a sandboxed child process.
-  `names` are the tools a treatment agent would use, and the run fails at start if any would not
-  be registered. A `pr` whose branch is gone resolves to `refs/pull/<pr>/head`, and the report
-  marks it.
+- `tools` (optional, main by default) lists kernel/xwebmcp refs (or full `sha`s) to register
+  from, such as a PR's branch after main. Treatment gets every tool `registry.json` lists for the
+  domain at those commits whose file declares `readOnlyHint: true`; later sources replace earlier
+  tools of the same name. Tool files are evaluated in a sandboxed child process. `names`
+  (optional) are tools the intent expects to be used, and the run fails at start if any would not
+  be registered. `entry_tool` (optional) must be live on the start page. A `pr` whose branch is
+  gone resolves to `refs/pull/<pr>/head`, and the report marks it. Leave all three off a new
+  intent: it describes the user's task, and the report shows which tools it used.
 - `{{name}}` in `prompt` and `rules` comes from `fixture`. `{"days_from_start": n}` resolves to a
   date when the run starts. A fixed date that has passed skips the intent, or stops the run if
   the intent was selected by name; move dates tied to a film's run forward before each run.
@@ -295,6 +310,12 @@ scored:
   slower arm is `mixed` when it was correct in 2 fewer pairs, gave the worse `better` answer in
   2 more, or the arms' `same` answers differed in 2 pairs.
 
+Each intent with a step timeline also gets a steps table: agent seconds per trial on each page in
+each arm, how many treatment trials invoked a tool there, and which tools. Tasks on domains with no
+tools get their own table of control results and slowest steps. The tool backlog then lists, longest
+first, the steps where control spent at least 20s per trial and treatment invoked a tool in fewer
+than half its trials, or that have no tools at all.
+
 A tool failures table lists, per tool, the treatment invocations that did not complete or whose
 tool returned `status: "error"`, with their most common error messages. Then the worst label per
 PR, and per run: spend over every attempt including smoke gates, the label histogram (noting

bench/lib/agent.mjs

@@ -199,6 +199,8 @@ export function summarize(events, allow) {
   const refs = new Map();
   for (const call of calls.values()) collectRefs(call.text, refs);
   const webmcp = { listed: false, adopted: false, invocations: [] };
+  // When each call started, and which WebMCP tool it invoked, for the step timeline.
+  const timeline = [];
 
   for (const call of calls.values()) {
     const args = call.arguments ?? {};
@@ -208,18 +210,24 @@ export function summarize(events, allow) {
     if (isList(call)) {
       webmcp.listed = true;
     } else if (isInvoke(call)) {
+      const tool = refs.get(args.tool_ref) ?? args.tool_ref ?? null;
       webmcp.invocations.push({
         path: "mcp",
-        tool: refs.get(args.tool_ref) ?? args.tool_ref ?? null,
+        tool,
         ...invocationResult(call),
         duration_ms: call.done && call.startedAt !== undefined ? call.endedAt - call.startedAt : null,
       });
+      timeline.push({ start_ms: call.startedAt ?? null, invoke: tool ?? "unknown" });
+      continue;
     } else if (CODE_PATHS[call.name] && typeof args.code === "string") {
       if (/\blistTools\s*\(/.test(args.code)) webmcp.listed = true;
       if (/\binvokeTool\s*\(/.test(args.code)) {
         webmcp.invocations.push({ path: CODE_PATHS[call.name], tool: null, status: null, tool_status: null, error: null, duration_ms: null });
+        timeline.push({ start_ms: call.startedAt ?? null, invoke: `unknown (${CODE_PATHS[call.name]})` });
+        continue;
       }
     }
+    timeline.push({ start_ms: call.startedAt ?? null, invoke: null });
   }
   webmcp.adopted = webmcp.invocations.length > 0;
 
@@ -240,6 +248,7 @@ export function summarize(events, allow) {
     cost_usd: sum(u => u.cost?.total),
     outside_allowlist: [...outsideAllowlist],
     webmcp,
+    timeline: timeline.filter(call => call.start_ms !== null),
   };
 }
 

bench/lib/intents.mjs

@@ -1,6 +1,7 @@
 import { existsSync, readdirSync, readFileSync } from "node:fs";
 import { join } from "node:path";
 import { checkSchema } from "./schema.mjs";
+import { matchesUrlPattern } from "./tools.mjs";
 
 const STAGES = ["find", "inspect", "compare", "prepare", "gap"];
 const RULE_CHECKS = ["equals", "in", "matches", "range", "count", "time", "date", "unique"];
@@ -30,6 +31,15 @@ export function trialPrompt(intent, browserId) {
   return [fill(intent.prompt, intent.fixture), allowed, intent.instructions, browser].filter(Boolean).join("\n\n");
 }
 
+function validUrlPattern(pattern) {
+  try {
+    matchesUrlPattern("https://example.com/", pattern);
+    return typeof pattern === "string";
+  } catch {
+    return false;
+  }
+}
+
 // `requests` becomes a declarativeNetRequest regexFilter, which Chrome compiles with RE2.
 const RE2_UNSUPPORTED = /\(\?<?[=!]|\\[1-9]|\\k</;
 
@@ -75,19 +85,24 @@ export function validateIntent(intent, benchDir) {
   need(STAGES.includes(intent.stage), `stage must be one of ${STAGES.join(", ")}`);
   need(typeof intent.prompt === "string" && intent.prompt.length > 0, "prompt is required");
   need(/^https:\/\//.test(intent.start_url ?? ""), "start_url must be https");
-  need(Array.isArray(intent.tools) && intent.tools.length > 0, "tools must list at least one {repo, ref, names}");
+  // An intent describes the user's task; `tools` only says which commits to register the domain's
+  // tools from (main by default), and `names`, when given, are tools it expects to be used.
+  need(intent.tools === undefined || (Array.isArray(intent.tools) && intent.tools.length > 0), "tools, when given, must list at least one {repo, ref}");
   for (const source of intent.tools ?? []) {
     need(typeof source.repo === "string" && typeof source.ref === "string", "each tools entry needs repo and ref");
     // The runner evaluates tool files, so only this repository.
     need(source.repo === "kernel/xwebmcp", "tools repo must be kernel/xwebmcp");
     need(source.sha === undefined || /^[0-9a-f]{40}$/.test(source.sha), "tools sha, when pinned, must be a full commit sha");
-    need(Array.isArray(source.names), "each tools entry needs a names array");
+    need(source.names === undefined || (Array.isArray(source.names) && source.names.every(name => typeof name === "string")), "tools names, when given, must list tool names");
   }
   const named = (intent.tools ?? []).flatMap(source => source.names ?? []);
-  need(intent.stage === "gap" ? named.length === 0 : named.length > 0, intent.stage === "gap" ? "gap intents name no tools" : "non-gap intents name the tools a treatment agent would use");
-  need(typeof intent.entry_tool === "string", "entry_tool is required");
-  need(typeof intent.answer_schema === "string" && existsSync(join(benchDir, intent.answer_schema)), "answer_schema must point at a file under bench/");
+  need(intent.stage !== "gap" || named.length === 0, "gap intents name no tools");
+  need(intent.entry_tool === undefined || typeof intent.entry_tool === "string", "entry_tool, when given, must be a tool name");
   need(intent.actions === undefined, "actions belong in the domain's domain.json");
+  need(intent.steps === undefined || (Array.isArray(intent.steps) && intent.steps.length > 0 && intent.steps.every(step => typeof step?.name === "string" && step.name.length > 0
+    && Array.isArray(step.url_patterns) && step.url_patterns.length > 0 && step.url_patterns.every(validUrlPattern))),
+    "steps must list {name, url_patterns} with valid URL patterns");
+  need(typeof intent.answer_schema === "string" && existsSync(join(benchDir, intent.answer_schema)), "answer_schema must point at a file under bench/");
   // A prepare intent changes site state, so the runner reads where the agent left the browser from
   // the page itself, never through the tools under test.
   const { check, allow, guard } = intent;
@@ -187,7 +202,8 @@ export function loadIntents(benchDir) {
       // checkout tools only intents that may go through checkout.
       const intentActions = intent.stage !== "prepare" ? [] : [...Object.keys(actions), ...(intent.allow?.includes("checkout") ? Object.keys(checkoutActions) : [])];
       intents.push({
-        ...intent, actions: intentActions, exclude: intentProfile ? exclude : { ...exclude, ...needsSignIn }, profile: intentProfile,
+        ...intent, tools: intent.tools ?? [{ repo: "kernel/xwebmcp", ref: "main" }], actions: intentActions,
+        exclude: intentProfile ? exclude : { ...exclude, ...needsSignIn }, profile: intentProfile,
         signed_in_check: intent.profile ? intent.signed_in_check : signedInCheck, instructions,
       });
     }

bench/lib/kernel.mjs

@@ -57,6 +57,23 @@ export async function cookies(id) {
   return response.result;
 }
 
+// Top-frame navigations since `sinceMs` from the browser's telemetry, as {at, url} in epoch ms.
+// The API filters each page of events after paging, so `--all` stops at the first page with no
+// page event; this follows next_offset to the end instead.
+export async function navigations(id, sinceMs) {
+  const args = ["browsers", "telemetry", "events", id, "--categories", "page", "--since", new Date(sinceMs).toISOString(), "--limit", "100"];
+  const found = [];
+  for (let offset = null, pages = 0; pages < 1000; pages++) {
+    const response = await kernel(offset ? [...args, "--offset", offset] : args);
+    for (const { event } of response.events ?? []) {
+      if (event.type === "page_navigation" && !event.data?.parent_frame_id) found.push({ at: event.ts / 1000, url: event.data.url });
+    }
+    offset = response.next_offset;
+    if (!offset) break;
+  }
+  return found;
+}
+
 // Uploads an extension directory under its name unless Kernel already has it.
 export async function ensureExtension({ name, dir }) {
   try {
@@ -68,7 +85,7 @@ export async function ensureExtension({ name, dir }) {
 }
 
 export async function addCustomTools(id, namespace, sourceFile) {
-  return (await kernel(["browsers", "webmcp", "custom-tools", "add", id, "--namespace", namespace, "--source-file", sourceFile, "--force-overwrite-namespace"])).tools;
+  return (await kernel(["browsers", "webmcp", "custom-tools", "add", id, "--namespace", namespace, "--source-file", sourceFile, "--force-overwrite-namespace"])).tools ?? [];
 }
 
 export async function listCustomTools(id) {

bench/lib/local.mjs

@@ -211,7 +211,7 @@ export async function addCustomTools(id, namespace, sourceFile) {
   // setup then reports the entry tool as not live.
   const live = async () => (await call(browser, "listTools")).some(t => t.source?.custom?.namespace === namespace);
   await waitFor("tools", live, 10_000).catch(() => {});
-  return result.tools;
+  return result.tools ?? [];
 }
 
 export async function listCustomTools(id) {

bench/lib/page.mjs

@@ -1,4 +1,6 @@
-// Reading a trial's end state from the page.
+// Reading a trial's end state from the page, and splitting its agent time into steps by page.
+
+import { matchesUrlPattern } from "./tools.mjs";
 
 const TEXT_LIMIT = 200_000;
 
@@ -35,3 +37,42 @@ export function extractValues(text, extract = {}) {
   }
   return values;
 }
+
+// A step is a page of the task. An intent's `steps` name them by URL pattern; otherwise a step is
+// the host and path, with id-like segments (anything with a digit and 6+ characters, or a long
+// token) collapsed to *, so every trial's checkout page is the same step.
+export function stepName(url, steps = null) {
+  for (const step of steps ?? []) if (step.url_patterns.some(pattern => matchesUrlPattern(url, pattern))) return step.name;
+  let parsed;
+  try {
+    parsed = new URL(url);
+  } catch {
+    return url;
+  }
+  const path = parsed.pathname.split("/").map(segment => (/\d/.test(segment) && segment.length >= 6) || /^[\w-]{24,}$/.test(segment) ? "*" : segment).join("/");
+  return `${parsed.host}${path.replace(/\/$/, "") || ""}`;
+}
+
+const BROWSER_URL = /^(about|chrome|chrome-error|data|blob):/;
+
+// Agent time per step: from each top-frame navigation to the next, between the agent's launch and
+// the end of its run, with the calls the agent made while on that step. `navigations` are
+// [{ms, url}] relative to the launch; the last one before it is the page the agent started on.
+export function stepTimeline({ navigations, calls, durationMs, startUrl, steps = null }) {
+  const pages = navigations.filter(n => !BROWSER_URL.test(n.url)).sort((a, b) => a.ms - b.ms);
+  const before = pages.filter(n => n.ms <= 0).at(-1);
+  const segments = [{ start: 0, url: before?.url ?? startUrl }, ...pages.filter(n => n.ms > 0 && n.ms < durationMs).map(n => ({ start: n.ms, url: n.url }))];
+  const byStep = new Map();
+  for (const [i, segment] of segments.entries()) {
+    const end = segments[i + 1]?.start ?? durationMs;
+    const name = stepName(segment.url, steps);
+    const entry = byStep.get(name) ?? { step: name, ms: 0, calls: 0, invocations: [] };
+    entry.ms += end - segment.start;
+    for (const call of calls.filter(c => c.start_ms >= segment.start && c.start_ms < end)) {
+      entry.calls++;
+      if (call.invoke) entry.invocations.push(call.invoke);
+    }
+    byStep.set(name, entry);
+  }
+  return [...byStep.values()].map(entry => ({ ...entry, ms: Math.round(entry.ms) }));
+}