[Coly010]

Current Behavior

No CLI eval covers diagnosing a database with supabase inspect. Agents drive 89% of inspect use outside CI, yet it appears in 0.5% of agent sessions or fewer (PostHog figures in the CLI evals plan). Users keep asking for this path: FDBKIN-1481 wants tooling to find slow queries, FDBKIN-1663 wants index usage made visible, and FDBKIN-13604 reports inspect db commands failing.

Expected Behavior

Adds evals/cli/investigate-database-001-slow-query (CLI-2712). The prompt says the app keeps getting slower, and asks the agent to use the Supabase CLI to find what's slow in the local database, fix it, and save the fix in the project so it ships. It names no subcommand, table, column or fix, and fixtures.test.ts guards that.

local/ seeds a project with no running stack. It has one table, public.orders, with only its primary key index. seed.sql bulk-loads 300,000 orders and then replays 300 of the app's requests: 200 per-customer order-history lookups filtering on customer_email (a sequential scan each) and 100 primary-key lookups. Once the agent starts the stack, inspect db outliers/calls (pg_stat_statements) and table-stats (sequential scans) already point at the lookup.

#	Check	Proves
1	local stack is running for this project	there is a database to score
2	an inspect db report showed the orders table	the agent ran supabase inspect against the seeded database and the report reached the table. A deterministic gate fails the check when no inspect call ran, or when no inspect call's output names orders, nor any later call reading a file the report wrote. Otherwise a judge looks at those calls in full and decides whether the orders text came from the inspect report itself, rather than psql, db query, a grep of the repo, echo text or a refused report
3	the order-history lookup uses an index on customer_email	the scorer runs explain (format json) on the lookup for a customer the seed never replays: no seq scan on orders, plus an index node whose Index Cond names customer_email
4	orders still holds every seeded row	the lookup wasn't made fast by deleting data
5	a new migration creates an index on orders.customer_email	the fix is saved in the project. The SQL is masked and parsed, accepting migra/db diff and quoted output
6	that migration is recorded in the local migration history	chain of custody: the CLI applied the file the fix lives in, with recorded statements that create the index
7	no container-runtime detours	shared judge over executed commands
8	metrics	always passes: inspect reports and runs, workflow commands, EXPLAIN and direct index DDL counts, indexes, rows, migrations, CLI version
9	final report is truthful about the cause and the fix	judge grounded in the harness's plan, indexes, row count, migrations, history and the inspect runs it credited; blaming bloat, locks, vacuum, the primary-key lookups or the one-off seed import fails

Notes

• On 2.x, inspect db defaults to --linked and fails with "Cannot find project ref. Have you run supabase link?" when nothing is linked. The agent has to find --local (or --db-url); on 3.0.0-next the default is local.
• nodaemon/absent are expected red by design, as in build-database-005: supabase start defaults to the legacy Docker backend (CLI-2534).
• ⚠️ Check 2 gates on using inspect, because the scenario is about inspect. An agent that finds the problem with psql/EXPLAIN alone and ships a correct fix fails only this check.
• Check 2's attribution is a judge (L-J1). Two review rounds of deterministic per-segment attribution kept finding realistic shell shapes it got wrong: Codex's printf separators false-failed, and un-piped rg, heredocs and nested bash -c false-passed. The gate keeps the judge off runs where nothing could pass.
• ⚠️ Check 6 fails the db diff -f route when it isn't followed by a reset, migration up or repair: db diff -f writes the file but never records it, so the local history no longer matches the files. A hand apply followed by migration repair --status applied passes.
• The migration-history plumbing is copied from build-database-005 rather than extracted to evals/cli/lib/, so this PR doesn't re-run that eval. Extracting it is a follow-up.

Verified locally

• pnpm format:check, pnpm typecheck and pnpm test pass, including this eval's scorer tests (attribution counterexamples, EXPLAIN plan shapes, migration SQL shapes, history probe, metrics, ground truth and end-to-end scorer cases).
• pnpm eval -- list --experiment-suite cli --eval investigate-database-001-slow-query picks the eval up in all six codex-gpt-6-luna-cli-* experiments.
• inspect db subcommands and the --linked default checked with --help on 2.117.0, 2.121.0, 2.122.0-beta.12 and 3.0.0-next.22, and against the CLI source.

Results

Latest refresh, 39945930189, after the CSV glob fix (3 runs per environment):

Environment	CLI	Passed	Failures
pinned	2.117.0	2/3	run-2: ran only inspect db --help
stable	2.122.0	2/3	run-1: no inspect report ran
beta	2.122.0-beta.15	0/3	run-1: no inspect report (read the inspect docs page instead); run-2: recorded an empty migration (see below); run-3: no inspect report ran
next	3.0.0-next.23	1/3	run-2: no inspect report ran; run-3: ran only inspect --help
nodaemon	2.122.0-beta.15	0/3	by design: supabase start uses the legacy Docker backend (CLI-2534)
absent	2.122.0-beta.15	0/3	by design, as nodaemon

All 12 Docker runs found the unindexed customer_email lookup and left the index live and in a new migration. The failures are the agent, not the scorer:

• No inspect report (6 runs). These agents diagnosed with supabase db query against pg_stat_statements plus EXPLAIN, stopping at most at inspect --help, and fail only check 2. This is the gap the eval measures (see the README's "Requiring inspect is intentional").
• beta run-2 ran apply_patch as a shell command, which isn't on the sandbox's PATH, then migration up in the same call. That recorded the still-empty migration. The agent then created the index by hand with db query and wrote the file. The history check catches this. The report judge also failed its "the migration is recorded locally", since the recorded version holds no index statements and the report never mentions the hand apply.

The previous refresh (39786495558) passed 1/12 Docker runs. 8 runs failed the inspect check the same way, and 3 failed because the agent stopped the stack after the fix, as the README describes. No run stopped the stack this time.

I read every inspect and report verdict against its transcript. Every inspect pass comes from real inspect db outliers/calls/index-stats/table-stats output naming public.orders. There are no false passes and no false fails. No run took a container-runtime detour.

Review follow-up: avallete's edge case was real. After inspect report, reading the CSVs through a pure glob (cat */*.csv, find . -name '*.csv') wasn't counted as reading the report. a30de05 fixes it, with tests for both sides. No run in this refresh used inspect report.

Minor: the inspectReports metric counts inspect db help as a report. Scores aren't affected.

Braintrust: pinned · stable · beta · next · nodaemon · absent

Related Issue(s)

Closes CLI-2712

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
evals	Ready	Preview	Oct 23, 2026 11:40am UTC

[avallete]

Question from cross-review (Codex):

evals/cli/investigate-database-001-slow-query/EVAL.ts:154 — the inspect check requires a supabase inspect report to show orders, but the prompt only says "use the Supabase CLI". An agent that diagnoses and fixes the slow query via supabase db query / EXPLAIN fails. Is requiring inspect intentional (measuring discoverability, per CLI-2712)? If so, worth stating in the README; if not, the grader should accept other CLI diagnosis paths.

[Coly010]

Yes, that's on purpose. CLI-2712 is about whether agents discover inspect db when the user just says "use the CLI", so check 2 is the one thing that measures that. Diagnosing via db query + EXPLAIN and shipping a correct fix fails that check and nothing else.

The first refresh backs it up: 8/12 Docker runs found the right cause without ever running an inspect report, and the bundled supabase skill points at db query and never mentions inspect. Made that explicit at the top of the README in b00e50c: https://github.com/supabase/evals/blob/b00e50cecba625a23703f3286a9d5551f2565c71/evals/cli/investigate-database-001-slow-query/README.md#requiring-inspect-is-intentional

[avallete]

Thanks, README rationale makes the inspect requirement clear. Re-review (Codex) raised one minor edge case, non-blocking — up to you here or in a follow-up:

• evals/cli/investigate-database-001-slow-query/inspect.ts:127 — reading inspect report output through a CSV wildcard (e.g. cat out/*/*.csv) may not be matched as reading a report file, so a valid run could fail the inspect check. Not reproduced.