[Coly010]

Current Behavior

There's no eval for the loop agents spend most of their time in with database tests: the suite fails, find the cause, fix the migration, re-apply, re-run supabase test db. In a 1% PostHog device sample over Sep 15 to Oct 14, test db → db reset → test db appears in about 2,100 sessions, and 78% of them hit a non-zero exit. CLI-2710 covers writing tests; nothing covers fixing a red suite.

Expected Behavior

Adds evals/cli/resolve-database-006-fix-failing-pgtap-suite (CLI-2733). The prompt says the database tests broke after yesterday's discounts change and asks the agent to get them passing again and leave the database running. It names no command, file, column or function, and it's the same under every codex-gpt-6-luna-cli-* experiment that runs it.

The seeded project (local/) has two migrations and a two-file pgTAP suite. The second migration, yesterday's change, has two bugs that mask each other: a not null discount column with no default (one test file stops at its first insert), and a total that sums the discount instead of what's left (three totals wrong). Fixing only the first turns the insert error into a wrong total. Valid fixes are an in-place edit plus db reset, or a new fix migration plus migration up, with either a default of 0 or a nullable column and coalesce.

The harness pre-starts the stack (projectRunning: true), so both bugs are live when the agent starts and its first supabase test db prints Files=2, Tests=6, Result: FAIL. The agent has to fix the migration, get it into the database through the CLI and re-run the suite. An earlier revision started with the stack stopped; agents then fixed the migration by reading it before the first supabase start, and only 5 of 36 Docker runs ever saw a red suite (see Results).

#	Check	Proves
1	local stack is running for this project	the workspace's stack resolves and answers select 1; a guard that the agent left it running, since the harness started it
2	the whole suite passed under supabase test db	some test db summary (every one a call printed, in order) was Result: PASS for the whole suite (Files= of at least 2, or a run with no path or --db-url argument), and any later Result: FAIL was cleared by a later whole-suite PASS
3	the seeded tests pass against the local database	the scorer's own pristine copies of the seeded tests pass via psql on the live DB, so editing the tests can't help
4	the seeded tests pass on a database rebuilt from the migrations	the workspace's migrations replayed into a scratch DB give a green suite, so the fix lives in the files, not only in a hand-patched DB
5	the migration history matches the migration files	schema_migrations statements equal each file now, so the fix reached the DB through the CLI (not left unapplied or run with psql -f)
6	the seeded tests were left as they were	the agent fixed the bug, not the tests: every seeded statement is still there in order, and the only additions are pgTAP assertions (with the plan raised) or inserts of new test data
7	no container-runtime detours	shared lib judge
8	metrics	always passes: versions, stack backend, test db runs and results, testDbFailedFirst (did the agent see the suite red before fixing it), reset / migration up counts, migration file changes, pgProveDirect, cliDetours
9	final report is truthful about the tests and the fix	LLM judge grounded in checks 3–6 and every test db summary; it fails false claims, not omissions

Notes

• Adds a shared evals/cli/lib/pgtap.ts (TAP parsing, pg_prove summary, supabase test db run detection), generalised from investigate-database-002-rls-policy-tests, with its own tests and a line in evals/cli/lib/README.md. That eval's folder is untouched; moving it onto the shared module is a follow-up.
• Adding assertions to a seeded test file is allowed, so check 6 uses an allowlist rather than a byte comparison. Anything that could hide or disable a seeded assertion (todo/skip, DDL, set, update, do, inserts into pgTAP's own tables or for the seeded invoices) is counted as a modification. The scan covers the code outside literals and any literal pgTAP could run as SQL (one that starts with a command word), so lives_ok($$alter table …$$) is caught and a description like 'cannot delete a paid invoice' isn't. A description that itself starts with a command word is still a false positive; the README lists that and the residual (SQL assembled at run time or hidden behind E'' escapes).
• No check credits the pre-started state: a run that does nothing passes the stack, history and tests-unchanged guards and fails test db, live and rebuilt-database. nodaemon and absent skip the eval, since they can't pre-start a stack.
• The seeded project_id is sandbox-invoices-app, so the harness's leftover sweep (which matches the sandbox- prefix) cleans up a crashed run's stack.
• The scratch database has no Supabase baseline (no auth/storage), so the seed uses none. The history check can't tell a fix migration applied by hand and then marked applied with migration repair from migration up; the end state is the same. Both are in the README's known limitations.
• No framework changes.

Verified locally

• The seed against the real supabase/pg_prove:3.36 image and supabase/postgres 17: seeded Files=2, Tests=6, Result: FAIL; default only fixed Files=2, Tests=9, Result: FAIL; both fixed Files=2, Tests=9, Result: PASS.
• The scorer's probes (live suite, replay, history, tests-unchanged) against that container: the seeded state, an in-place fix plus reset, a fix migration plus migration up, a hand-patched DB, and a new fix migration applied with psql -f each pass or fail the expected checks. Re-applying the in-place fixed file with psql -f on the pre-started database errors on the existing column, so that path fails the live check as well as history.
• schema_migrations.statements format is identical on 2.117.0 and 3.0.0-next.23; next's test db prints the same plain pg_prove summary in agent mode.
• pnpm format:check, pnpm typecheck and the eval's tests pass. pnpm test stops at test:framework locally for lack of a judge key.
• Merged main after refactor(evals): share migration-history, psql and stack-lifecycle helpers in cli/lib #429 (shared migration-history and psql helpers in evals/cli/lib); the only conflict was evals/cli/lib/README.md, where both additions are kept.

Results

Three refreshes on the pre-started stack, 3 runs per environment each: 40332677144 at 4a35927 (the switch), 40334537287 at e609c87 (the stale results cleanup) and 40354332826 at 65e1fe5 (the review fixes).

Environment	CLI	4a35927	e609c87	65e1fe5	Failures
pinned	2.117.0	0/3	1/3	2/3	agent gaps, history check
stable	2.122.0	2/3	2/3	3/3	agent gaps, history check
beta	2.123.0-beta.2, then beta.3	3/3	3/3	2/3	agent gap, history check
next	3.0.0-next.24	2/3	1/3	0/3	agent gaps, history check
nodaemon		skipped	skipped	skipped
absent		skipped	skipped	skipped

In 33 of 36 runs the agent saw the suite red under supabase test db (testDbFailedFirst), fixed the migration, and ran it again until it printed Files=2, Tests=9, Result: PASS. On the stopped stack, 5 of 36 Docker runs did that.

I checked the commands and check notes of all 36 runs, and read every failing transcript. No false fails or false passes. All 15 failures are the history check, and in each one the agent put the fix into the database by hand:

• 12 runs edited the applied seeded migration and ran the fix SQL with psql or supabase db query, so the history still records the old statements.
• 3 runs ran migration up on the empty file migration new had just created, then applied the fix with psql, so the new migration is recorded with no statements.

The 21 passes all have a new fix migration whose recorded statements match the file, and the rebuilt database passes. 20 applied it with migration up. One (stable, e609c87 run 3) ran it with supabase db query --file and then migration repair --status applied, which ends in the same state; that path is listed in the README's known limitations. No run used db reset.

The stale nodaemon and absent rows for this eval are dropped from apps/web/src/data/cli-eval-results.json in e609c87. Merge refreshes only replace experiment/eval pairs that ran, so they would otherwise have stayed at 0/3.

Earlier refreshes on the stopped stack

Three refreshes at 4575eb4, 12dcdef and c4ba12d (nodaemon and absent were red by design then). They surfaced two scorer false fails, both fixed with a regression test built from the transcript:

• The report judge failed honest "fixed the file, couldn't start a stack" reports on nodaemon and absent, because probes that never ran were listed as failed. They're now listed as not run.
• The tests-unchanged check failed an agent that appended rows for a new invoice to the seeded inserts. Rows for new invoices appended to a seeded insert now pass; rows for seeded invoices still don't.

The last of those rounds was 12/12 on the four Docker arms, but only 5 of the 36 Docker runs across the three rounds saw a red test db: agents fixed the migration by reading it and let supabase start apply it. That's why the eval now starts with the stack running.

Braintrust (65e1fe5): pinned · stable · beta · next
Braintrust (e609c87): pinned · stable · beta · next
Braintrust (4a35927): pinned · stable · beta · next

Related Issue(s)

Closes CLI-2733

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
evals	Ready	Preview	Oct 26, 2026 4:05pm UTC

[avallete]

Cross-review (Codex gpt-6-sol, verified by Claude): grader false-pass/false-fail findings inline.