[Coly010]

Current Behavior

No CLI eval covers writing database tests and running them with supabase test db, or using those tests to find and fix a row level security bug. In PostHog (Sep 14 – Oct 13), test fails on about 40% of agent runs outside CI, against 25% for humans. Some of that is tests failing as intended, so it's a softer signal than login or branches.

Expected Behavior

Adds evals/cli/investigate-database-002-rls-policy-tests (CLI-2710). The prompt is a nervous pre-launch ask: everyone should only ever see their own notes, so write proper database tests for that, run them, and fix anything they turn up, then leave everything running. It names no command, test framework, policy, RLS, migration, auth role or path.

local/ seeds a notes-app project with one migration. It creates public.notes with row level security on and four correct owner policies. It also adds a leftover "Enable read access for authenticated users" policy, for select to authenticated using (true), which is the Dashboard template left behind. Permissive policies are OR-ed, so every signed-in user reads every note. Anon still sees nothing, so anon-only tests miss the bug, and only a cross-user read test catches it. There is no supabase/tests, seed.sql or app code, and no stack is running.

#	Check	Proves
1	local stack is running for this project	there's a database to probe (lib stack resolution, across relocated homes). When the agent's last stack command was a stop, the notes say so
2	each user can see only their own notes	the leak is closed in the running DB and the owner still sees their notes. Measured by a rolled-back probe with two seeded users plus anon
3	only a note's owner can change it	the fix didn't open or break writes. Same probe; forged inserts are counted as postgres, so owner-setting triggers and column grants pass
4	the notes policy fix is in an applied migration	chain of custody: every migration touching the notes access rules is in supabase_migrations.schema_migrations with the same statements, and replaying those rules from the seeded state (rolled back) closes the leak. A psql hand fix, a never-applied file, or a migration that misses the leaky policy all fail
5	the project's pgTAP tests pass against the local database	every supabase/tests/**/*.{sql,pg} passes on the fixed DB, judged the way pg_prove judges it, run from supabase/tests like test db
6	the tests fail when the seeded notes policies are put back	mutation test: a file is run with the seed's five policies put back, and in two twin runs with identical policies, where a scorer-owned function lets a signed-in user read only their own notes in one and every note in the other. It counts only when the same assertion passes in the owner-only twin and fails both in the open-reads twin and with the seed's policies put back. If that assertion never runs, both runs must stop at the same statement, and a per-row file must fail its plan the same way in both. A test that only reads the catalog (policies_are, policy_cmd_is, counts, a for all owner policy existing, using (true)) gets the same result in both twins, so it can't count. Nor can a file that fails a catalog assertion under the leak while skipping its cross-user one. A file that commits a mutation is detected and the policies are restored
7	the tests were run with supabase test db	an executed test db / db test printed a pg_prove summary (Result: PASS/FAIL, or Files=N, Tests=M>0). A call that also runs pg_prove by hand (e.g. supabase test db || pg_prove …) isn't credited, because the summary could be pg_prove's
8	no container-runtime detours	shared judge over executed commands
9	metrics	always passes; CLI version and channel, backend and runtime, test and catch counts, test db results, direct policy DDL, hand-run pg_prove, migration counts
10	final report is truthful about the tests and the fix	judge grounded in the harness's own probe results, history rows, per-file clean and mutated runs, and test db summaries

Notes

• Red by design on nodaemon and absent. Legacy supabase start and legacy test db (the supabase/pg_prove:3.36 container) need Docker, and the native path is the opt-in stack backend (CLI-2534). A run passes there only if the agent discovers the stack backend.
• ⚠️ A FAIL test db run counts as having run the tests; NOTESTS doesn't. An error the leak itself causes (a scalar subquery returning two rows) counts as a catch, because the tests after it never run and flip.
• ⚠️ Check 4's replay re-runs only the notes access-rule statements on top of the live table and functions. A migration whose policies depend on something made by hand would replay here but fail a real reset. The README's Known limitations lists this and the other blind spots: tests outside supabase/tests, and test db wrapped in scripts.
• Motivation also cites FDBKIN-8983 (a user built an external RLS test tool after a leak), FDBKIN-36404 (pgTAP RLS testing too hard) and FDBKIN-7467 (creating auth.users rows by hand for pgTAP tests).
• readRecordedPolicyMigrations closely mirrors build-database-005's history reader. Lifting both into evals/cli/lib/migrations.ts is a follow-up, so this PR doesn't touch (and re-run) 005.

Verified locally

• pnpm format:check, pnpm typecheck and pnpm test pass, including this eval's unit tests.
• pnpm eval -- list --experiment-suite cli --eval investigate-database-002-rls-policy-tests picks the eval up in all six codex-gpt-6-luna-cli-* experiments.
• The scorer's real probe, suite and mutation commands were run against public.ecr.aws/supabase/postgres:17.11.0.004:
  • fixed DB: all of checks 2–6 pass;
  • seeded leaky DB: checks 2 and 5 fail, and so does 6;
  • an anon-only suite fails check 6;
  • seven structural-only files aren't counted as a catch (policies_are, policy_cmd_is with policy_roles_are, a for all owner policy existing, a per-policy no_plan check that no policy is open, and mixes of them), while a two-user read test is;
  • three files that fail a catalog assertion under the leak while skipping their cross-user test (or erroring before it, or skipping it under a per-row plan) were counted as catches before the same-assertion rule and aren't now. Honest files that error, emit one test per row, or fail a plan(1) under the leak still count;
  • re-scoring the agent workspaces from all three earlier refreshes (54 runs) gives every one the same catch verdict as before, and every catch is a cross-user read test;
  • a test file that commits the mutation is detected and restored;
  • \i helper includes resolve;
  • an owner-setting insert trigger passes check 3;
  • for check 4, a recorded wrong-name drop policy if exists plus a hand fix fails, while a recorded DO-block drop passes;
  • pgtap is dropped again afterwards, and the policies are unchanged.
• The pg_prove PASS, FAIL and NOTESTS output that check 7 parses was captured from the real supabase/pg_prove:3.36 image. A transcript where supabase test db fails on Docker and a hand-run pg_prove prints PASS in the same call was credited before and isn't now. No agent in any refresh has run pg_prove by hand, so no earlier verdict changes.

Results

Latest refresh (run 39947817565, commit 9467a96, 3 runs per environment):

Environment	CLI	Passed	Failures
pinned	2.117.0	3/3
stable	2.122.0	3/3
beta	2.122.0-beta.15	3/3
next	3.0.0-next.23	3/3
nodaemon	2.122.0-beta.15	0/3	red by design (CLI-2534): start and test db hit the unreachable daemon, and no agent tried the stack backend. Every agent still wrote the tests and a fix migration, and reported truthfully that the tests couldn't run
absent	2.122.0-beta.15	0/3	as nodaemon

This refresh runs check 6 with the same-assertion rule and check 7 with the hand-run pg_prove guard. Every passing run caught the leak with a cross-user read test (e.g. "Alice can only read Alice notes" failed with the seeded policies put back and once every signed-in user could read notes, and passed when only owners could), put the fix in an applied migration (a new file, or the seeded file edited before start/db reset), and got a PASS from supabase test db. No run ran pg_prove by hand. I checked the passing transcripts in every environment and found no false pass.

• ⚠️ One next run checked npm view supabase version, then ran start, db reset and test db through npx --yes supabase@2.122.0 instead of the installed 3.0.0-next.23. Its result is valid, but that run exercised stable rather than next. metrics.cliOverride records it, and the report judge sees it.
• One next run ran policy DDL directly, only to put the leak back for a moment, confirm its tests fail, and drop it again.

The third refresh (run 39942440188, commit d2e0dc1) was the first under the twin runs and passed 11 of 12 on the Docker environments. The one failure was an agent gap: a stable run found the leaky policy, then called apply_patch as a shell command, couldn't find it and stopped to ask for it instead of editing another way. A second cross-review then showed check 6 could credit a file that failed a catalog assertion under the leak while skipping its cross-user test, and check 7 could credit a hand-run pg_prove in the same call as test db. Both are fixed in 9467a96.

The second refresh (run 39797703229, commit 66f062f) passed 12 of 12 on the Docker environments under the earlier check 6, which ran each file under two owner-only policy sets. A cross-review then showed that version could credit a structural-only file (a for all owner policy existing, or a per-policy no_plan check) as a catch. The twin runs replace it.

The first refresh (run 39792789442, commit 4504a46) passed 5 of 12 on the Docker environments. All 7 failures were one false fail. The agent had done the whole job (tests passing through test db, fix in a migration), then tidied up with supabase stop (once --no-backup), so every database check found nothing to probe. The prompt never asked to keep anything running. Fixed on this branch: the prompt now ends with "Leave everything running when you're done, I want to poke at it myself afterwards." When a run stops the stack anyway, check 1's notes, metrics.stackStoppedByAgent and the report judge's ground truth say so. The regression tests are built from those transcripts. The only other failure in that refresh was an agent gap: an absent report claimed 11 pgTAP checks for a file with 10 assertions under plan(11), and the judge caught it.

Braintrust experiments (latest refresh, compare view against the third refresh):

• pinned · stable · beta · next · nodaemon · absent
• All six from the latest refresh · all six from the third refresh · all six from the second refresh · all six from the first refresh

Related Issue(s)

Closes CLI-2710

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
evals	Ready	Preview	Oct 23, 2026 12:32pm UTC

[avallete]

Cross-review (Codex) finding — SIGNIFICANT, plausible (not reproduced):

evals/cli/investigate-database-002-rls-policy-tests/pgtap.ts:524 — the mutation verdict may credit a policy-shape assertion (e.g. checking which policies exist) as a behavioral RLS test, without any test exercising two users. The leakOnly path in catchVerdict (a test that fails only with the seeded policies restored and is absent under both safe variants) looks like the opening. Could you confirm a structural-only test file cannot reach caught: true?

[Coly010]

yeah this one was real, thanks for flagging it. I reproduced it against a real Postgres through the scorer's own modules and three structural-only files got caught: true:

1. a test that a for all owner policy using auth.uid() exists
2. that, plus no qual = 'true' policy and RLS on
3. a per-policy no_plan "no policy is open" file, through the leakOnly route like you said

(policies_are / policy_cmd_is / policy_roles_are files were already rejected)

Rather than patch each route, I swapped the two safe policy sets for two twin runs with identical policies, where a scorer-owned function decides whether a signed-in user can read other people's notes (owner-only vs select true). A file only counts now if it fails under the seeded leak and its results change between the twins, so anything that reads the catalog gets the same answer in both and can't count. The leakOnly route is gone.

All 7 structural repros now come back not caught, an honest two-user read test still is, and re-scoring the agent workspaces from the last two refreshes gives the same verdicts as before. Fix is in d2e0dc1, and the refresh after it (run 39942440188) passed 11 of 12 on the Docker environments, every catch through a cross-user read test. The one miss was an agent that stopped to ask for apply_patch before writing anything

The only things left that could tell the twins apart without reading notes as a user are a test that inspects the scorer's own function, or one that's random - I've put that in the README's limitations. Happy to dig further if you can think of another way round it!

[avallete]

Thanks for reproducing and the twin-run redesign, that closes the structural route nicely. Re-review (Codex) raised two minor edge cases, non-blocking and not reproduced — up to you here or in a follow-up:

• evals/cli/investigate-database-002-rls-policy-tests/test-db-runs.ts:53 — a failed supabase test db followed by pg_prove in the same tool call can have the pg_prove summary credited as a CLI test run.
• evals/cli/investigate-database-002-rls-policy-tests/pgtap.ts:545 — a file whose seeded-leak run fails only on a policy-shape assertion, while skipping the cross-user assertion that flips between the twins, might still pass the catch check.