[Coly010]

Current Behavior

The cli suite has no eval for moving storage objects between buckets on the local stack, or for the declarative bucket seeding in config.toml that decides what a fresh local setup holds. That is the design-phase storage journey in CLI-2713. Known traps:

• FDBKIN-25373: moving objects across buckets doesn't work.
• FDBKIN-5816: no CLI flag or command sets a bucket's visibility.
• FDBKIN-5351: declarative bucket and file seeding is under-documented.

PostHog (CLI evals plan): storage commands are about 80% agent-driven, and agent runs of them fail about 15% of the time against 11% for non-agent runs.

Expected Behavior

This PR adds evals/cli/build-storage-002-move-to-private-bucket (CLI-2713). The harness starts the local stack before the agent runs, and that start seeds all six files into the public media bucket. The prompt is a user whose customer invoices end up in the public storage bucket when they run Supabase locally, "and they're in there right now". It asks the agent to move them into a private bucket and make sure a fresh local setup doesn't put them back in the public one. So the agent has to move live objects across buckets and also fix the declarative seed.

The prompt names no command, flag, path, bucket name, config key or file name.

The seed (Ledgerly, an invoicing app):

• supabase/config.toml declares one bucket, media, with public = true and objects_path = "./storage/media".
• supabase/storage/media/avatars/*.svg: three avatars that src/lib/avatars.ts loads by public URL.
• supabase/storage/media/invoices/*.pdf: three invoices. All six files have distinct sha256s.
• README.md describes the layout without instructions.

CLI facts that shape it (verified on 2.117.0, 2.122.0, 2.123.0-beta.1 and 3.0.0-next.23):

• storage ls/cp/mv/rm need --experimental.
• On 2.x they default to the linked project and need --local.
• storage mv and ss→ss cp refuse to cross buckets. So a move is download, upload and rm; a config fix followed by db reset; or stop --no-backup and a fresh start.
• seed buckets creates and overwrites but never deletes, so on its own it leaves the originals in media. A plain stop then start keeps the database volume, so nothing is re-seeded.

#	Check	Proves
1	invoices are in a private bucket on the local stack	Each invoice's bytes (service-key download, sha256) sit in a bucket with public = false. A SQL update storage.objects set bucket_id moves the row but not the bytes, so it fails.
2	no invoice is publicly readable on the local stack	No invoice bytes are in a public bucket, and no invoice-holding object or original media/invoices/* key answers its public URL or the anon key with a 2xx. Service and anon control requests (GET /storage/v1/bucket = 200) prove both identities were accepted. If the agent scored on a managed stack it started itself, the original keys are also read on the harness's legacy stack, and a 2xx there fails.
3	avatars are still public where the app loads them	media/avatars/* still answer their public URLs with the seeded bytes, so making media private fails.
4	a fresh local setup keeps invoices private and avatars public	config.toml is parsed with smol-toml and each bucket's objects_path is hashed in the sandbox. No public bucket seeds an invoice, some private bucket seeds every invoice, and media stays public and seeds every avatar under its key. The listing mirrors the CLI: symlinked root and files followed, symlinked directories skipped, single-file paths supported.
5	no container-runtime detours	The shared detour judge, over executed commands only. Storage, seed, db reset, psql, curl and config edits are allowed. Writing into a storage container or the stack's storage directory counts as a detour.
6	metrics	Always passes. It records CLI, channel, Docker, stack, keys found and the identities used, buckets, invoice locations, storage command counts (including --experimental and --local), fresh-setup invoice buckets, stack stop, the harness's legacy stack when another stack was scored, and detours.
7	final report is truthful about the invoices	A judge with ground truth from the storage probe and the seed config. Keys are redacted everywhere the judges see them.

Notes

• Expected today: pinned, stable, beta and next can pass. The friction is --experimental, --local on 2.x, no cross-bucket mv/cp, and removing the originals from media. The eval needs Docker and a pre-started stack, so skipUnlessDockerless drops it from nodaemon and absent. Their rows from the earlier design are deleted from cli-eval-results.json, because --merge refreshes never drop rows they don't replace.
• Why the stack starts running. An earlier version had the agent start the stack (projectRunning: false). In that version 19 of 20 passes fixed the config before their first start, so no run moved a live object. With the stack pre-started, a config-only fix, seed buckets alone, and a plain stop and start all leave the invoices public, and checks 1–2 catch them.
• No "leave it running" sentence. The stack is already running and nothing in the task needs it stopped. An agent that stops it fails checks 1–3 by its own action, and the notes, metrics and report ground truth all name the stop.
• Shared lib:
  • evals/cli/lib/stack-keys.ts reads the stack's API keys under the same CLI homes that resolved it.
  • homePrefix in lib/stack.ts is now exported.
  • The stack start/stop reader is readStackLifecycle from lib/cli-invocations.ts, which came in with refactor(evals): share migration-history, psql and stack-lifecycle helpers in cli/lib #429.
  • Follow-up, not done here: build-functions-008 still carries a private copy of the key reader.
• Known limits (detailed in the README):
  • The fresh-setup check reads config and files statically rather than resetting the stack, because a reset would destroy the evidence.
  • Only the project's default stack is resolved. While the harness's stack is up, a stack the agent starts under --stack <name> or another CLI home goes unseen, so the harness's stack is scored instead.
  • The legacy-stack read covers only the original media/invoices/* keys, not invoice copies elsewhere on that stack.
  • The anon check covers the object GET endpoint, not signed URLs.

Verified locally

• pnpm format:check && pnpm typecheck && pnpm test
• pnpm --filter @supabase-evals/framework exec vitest run --config vitest.cli-lib.config.ts evals/cli/build-storage-002-move-to-private-bucket evals/cli/lib, plus the whole test:cli-lib suite
• pnpm --silent eval -- list --experiment-suite cli --eval build-storage-002-move-to-private-bucket, which lists pinned, stable, beta and next
• shell.test.ts runs the real listing and curl commands against a temp dir and a local HTTP server.
• An engineer review, then three eval-reviewer rounds with executed counterexamples. Round 3, on the pre-started stack, found that a managed stack started alongside the harness's legacy stack hid the leftover invoices and scored a false pass. Fixed, with a regression test.
  • Fixes: the seed listing no longer descends into symlinked directories, and a failed public-bucket listing fails check 4. The anon identity now has a control request. Deleted or mistyped invoice seeds fail check 4. A run that ends in supabase stop is now named as such.

No local stack was started; this PR's run-evals-changed refresh is the live run.

Results

Latest refresh: 40355625422 on a9bf4dc, with the stack pre-started; 3 runs per environment. nodaemon and absent skip the eval.

Environment	CLI	Passed	Failing runs
pinned	2.117.0	3/3	none
stable	2.122.0	3/3	none
beta	2.123.0-beta.3	3/3	none
next	3.0.0-next.24	3/3	none

How the passing runs got there:

• Every pass fixed config.toml (a private invoices bucket, PDFs moved out of media's objects_path) and moved the live objects too. In all 12 runs, the probe found the invoice bytes only in the private bucket and media holding just the 3 avatars.
• The live move went through the Storage API every time.
  • Some runs used the CLI (storage cp/mv/rm --experimental, often with seed buckets for the new bucket).
  • Others called the HTTP API directly with the service key, via curl, Node fetch or supabase-js. That route doesn't appear in storageCommands, which counts only CLI storage commands.
• No run moved rows with SQL or wrote into a container. Stable run 3's docker exec calls were read-only psql selects, and the detour judge passed all 12.
• No run stopped the stack: lastStackCommand is null or a redundant supabase start.
• I checked one passing transcript per environment for a false pass. In each, the probe's downloads and public reads match a real move.

Earlier refreshes on the pre-started stack:

• 40336100513 on a32fd2d: 11/12. Beta run 2 was an agent gap. It fixed only the seed config and left the live invoices public, and its report said so.
• 40353411193 on b1f4bd1: 11/12. Beta run 3 was a false fail.
  • Storage answers a missing object with HTTP 400 and a body whose statusCode is "404". The report judge failed a correct report for calling that a 404.
  • a9bf4dc tells the judge about that reply, with a regression test built from the transcript.

The earlier design, where the agent started the stack, passed 3/3 on all four Docker arms (40316873437). 19 of those 20 passes fixed the config before the first start and never moved a live object. On the pre-started stack, the config-only fix fails, as the first refresh's beta run 2 showed.

Scorer fixes along the way:

• b1f4bd1 (from review): a config field the CLI rejects fails check 4 instead of falling back to its default. Examples are a quoted public = "false", a non-boolean storage.enabled, a non-string objects_path, or a bucket that isn't a table. Every channel's supabase start refuses to load such a config, so check 4 could previously pass one that never starts.
• 0ab6236 (earlier design, 40315089602): the report judge failed four runs whose claims the ground truth confirmed. It now accepts a check summarised from a sample, or one whose output wasn't recorded, when the ground truth confirms the result.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
evals	Ready	Preview	Oct 26, 2026 4:15pm UTC

[avallete]

Cross-review (Codex gpt-6-sol, verified by Claude): grader false-pass/false-fail findings inline.