[EPIC] Point E2E at an ephemeral per-runner Supabase — retires the mutex, the 25% cancellation rate, and #567

[TortoiseWolfe]

⏳ The parity run is FREE, and the window closes 2026-08-12

Parity was the one step believed to need a cloud E2E run — impossible until the quota
refills on 2026-09-02. It is not. The baseline already exists, from the last green
cloud run before the quota died:

run	31048279017
SHA	de0f7f080c8d75949e4e6c89fdf66ab7d3da8029
when	2026-08-05T21:21:26Z — 66 min before the first 402
passed	1,807
skipped	194
flaky	0
jobs	24 (chromium + firefox + webkit), 0 non-success

So parity = run the full matrix of e2e-local.yml on de0f7f0 and match those three
numbers. Zero cloud quota. No waiting until September.

Compare counts, not green/green. A local stack that silently skipped every auth test
would also report "all green". 1,807 / 194 / 0 is falsifiable; "it passed" is not.

Deadline: the run's artifacts expire 2026-08-12T22:13Z. The counts above are recorded
here so they survive that, but the per-test detail needed to diagnose a mismatch does
not. If parity fails after the 12th there is nothing to diff against until September.

PARITY RUN 2 — 2026-08-07 (run 31191204546, widened matrix on de0f7f0)

Step 1 is done (#628). Two parity runs have happened; here is where it stands.

	run 1	run 2	baseline
passed	589	1757	1807
skipped	1402	215	194
flaky	0	1	0
failed	34	52	0

Run 1 reported 24 green jobs while Firefox and WebKit executed ZERO tests. Every
*-gen / *-msg project declares dependencies: ['setup'] and auth.setup.ts runs
in chromium; installing only the matrix browser left setup unable to launch, and
Playwright skips tests whose dependency failed rather than failing them. Fixed by
installing chromium alongside the matrix browser. Only the count comparison caught
this — green/green would have declared parity and silently dropped two thirds of
browser coverage. A per-shard tripwire now fails any shard that executes 0 tests.

Run 2: all three browsers run (chromium 589 / firefox 581 / webkit 587 passed).
Remaining delta is −50 passed, +21 skipped, 52 failed, 1 flaky, from three causes:

1. storage-state-auth-b.json missing — 9 failures. auth.setup.ts:307 writes it
   only when the User-B branch completes; payment-isolation.spec.ts:56 loads it. The
   secondary credentials ARE present in .env.local-supabase, so this is not missing
   env — the B sign-in itself needs tracing.
2. Realtime specs failing — live transaction counter, connection-status indicator,
   burst coalescing, subscription status changes. NOT publication membership: the local
   supabase_realtime publication already carries payment_results, subscriptions,
   conversations, messages, typing_indicators, conversation_members,
   user_connections. The realtime service config is the next thing to look at.
3. One shard was cancelled (webkit-gen 4/6), so 23 of 24 reported. The parity job
   correctly refuses a count match on a partial run.

Skips break down as: 87 "requires local Docker Supabase", 48 bare skip, 27 avatar
upload, 15 no-conversations, 3 Stripe keys. Several of those exist because CI ran
against cloud and should now be re-examined — some may be able to run here, which
would move numbers in the right direction.

Do not switch e2e.yml until the counts match. The gate is working; it has already
prevented one false declaration of parity.

What actually remains

1. Widen e2e-local.yml from its single chromium-gen --shard=1/6 to the full 24-job
   matrix. This is the only real work left.
2. Run it on de0f7f0. Compare to 1,807 / 194 / 0.
3. On match, switch e2e.yml's 7 injection sites and delete the contention workarounds —
   the repo-wide mutex, max-parallel: 2, the priming curl, the 50s stagger.

Why this is now urgent rather than tidy

Measured 2026-08-06 from the billing history (invoices dated the 2nd of each month, so the
cycle runs the 2nd → 2nd):

44 E2E runs consumed an entire month's quota in 3.9 days — 10 on Aug 2, 4 on Aug 3,
none on Aug 4, then 30 on Aug 5, with the first 402 arriving 66 minutes after that
burst. All three meters blew: egress, MAU and realtime.

That is ~2.3% of the monthly allowance per run, i.e. the free tier funds roughly 44
full E2E runs per month. At this repo's own documented throughput (~1 PR per ~100 min,
every merge adding a main run), two or three genuinely active days exhaust it. The suite
and the free tier are structurally incompatible — this is not a merge-pacing problem.

If e2e.yml still points at the cloud project on 2026-09-02, the new quota is gone
inside a week and the second exhaustion invites the Fair Use Policy, which the first did
not. That is the deadline that matters.

---

Every painful property of e2e.yml traces to one fact: all 24 matrix jobs
share a single cloud Supabase project. Two other workflows in this repo already
avoid it, and the plumbing to fix E2E is largely present.

What the shared project costs

Workaround in e2e.yml	Its own stated reason
repo-wide concurrency mutex, cancel-in-progress: false	"every E2E run shares one Supabase project"
max-parallel: 2	cap peak concurrent load on the free tier
"Prime Supabase connection pool" curl	free-tier cold start
up to 50s of deliberate sleep per job	free-tier rate limits
messaging runs 1 shard instead of 2	shards race each other's messages.delete()

Measured consequences: 68-minute median on success, 25% of runs cancelled
(5 of the last 20, hitting the 60-minute job cap while queued behind the mutex),
and the quota exhaustion that is #567 — which currently has production auth,
messaging and payments down.

None of that is Playwright being slow. It is queueing for one shared database.

The fix already exists here, twice

conformance.yml stands up a full local Supabase stack and finishes — stack
boot plus tests — in 2.5 minutes.

signup-mailer.yml:54-67 is the exact CI recipe: copy .env.local-supabase
→ .env, rewrite host.docker.internal → localhost, export
NEXT_PUBLIC_SUPABASE_URL=http://localhost:54321 and
SUPABASE_ADMIN_URL=http://localhost:54321, build with DISABLE_BASE_PATH=true,
npx serve out, run Playwright.

docker-compose.yml already defines nine services under
profiles: [supabase] — db, kong, auth, rest, realtime, storage, meta, studio,
mailpit.

Nothing in playwright.config.ts is cloud-specific, and
tests/e2e/global-setup.ts:112-115 already reads the indirection:

const supabaseUrl = process.env.SUPABASE_ADMIN_URL || process.env.NEXT_PUBLIC_SUPABASE_URL!;

The actual gap

global-setup.ts expects pre-seeded shared PRIMARY/SECONDARY/TERTIARY users.
scripts/setup-e2e-users.sh (→ seed-test-users.ts + seed-connections.ts)
already provisions exactly those against a local stack for pnpm dev:local. So
the gap is wiring, not capability.

The other blocker is e2e.yml:69-71, which pins the cloud project at workflow
level:

env:
  NEXT_PUBLIC_SUPABASE_URL: ${{ vars.NEXT_PUBLIC_SUPABASE_URL }}

What it buys

• The mutex becomes unnecessary. Nothing is shared, so nothing races. Runs go
  parallel and the 25% cancellation rate goes to zero.
• max-parallel: 2, the priming curl and the 50s sleeps all come out.
• Messaging can shard again, since no two jobs share a messages table.
• Supabase quota burn goes to zero, which retires CI runs its full 30-job E2E suite against production Supabase — 33 runs in two days exhausted three quotas and took prod down #567 rather than
  mitigating it — CI stops being able to take production down.
• Throughput stops being ~1 PR per run.

Scoping update 2026-08-06 — smaller than first estimated

Four of the five prerequisites already exist. Verified, not assumed:

1. Schema fidelity is already solved. This was listed below as the top risk. It is not a
risk — docker-compose.yml mounts the exact same monolithic migration prod uses:

- ./supabase/migrations/20251006_complete_monolithic_setup.sql:/docker-entrypoint-initdb.d/migrations/99999999999999_app_monolithic.sql:ro

Mounted as a file, not a directory, with a long comment explaining why: a directory bind
would shadow the 38 migrations the image bakes in, and the host dir also holds
999_drop_all_tables.sql ("WARNING: This script will DELETE EVERYTHING") which migrate.sh
would glob alphabetically. Someone already walked this minefield.

2. The env recipe exists — signup-mailer.yml:54-67, copy .env.local-supabase → .env,
rewrite host.docker.internal → localhost, export NEXT_PUBLIC_SUPABASE_URL and
SUPABASE_ADMIN_URL to localhost:54321.

3. The indirection is already in the test code — tests/e2e/global-setup.ts:112-115
reads SUPABASE_ADMIN_URL || NEXT_PUBLIC_SUPABASE_URL.

4. User seeding exists — scripts/setup-e2e-users.sh provisions the PRIMARY / SECONDARY /
TERTIARY users the global setup expects, and is already used by pnpm dev:local.

5. Proof of speed — conformance.yml brings the stack up and runs its tests in
2.5 minutes total.

So the actual remaining work is

• point e2e.yml at the local stack instead of vars.NEXT_PUBLIC_SUPABASE_URL (7 sites)
• run the seed script before the matrix
• prove parity: one full local run vs one full cloud run, same pass/fail set
• only then remove the mutex, max-parallel: 2, the priming curl and the 50s sleeps

Step 3 is the real work. Steps 1, 2 and 4 are mechanical.

And it can be developed right now

The local stack needs no cloud project, so this is buildable while #567 has production at
402 — unlike almost everything else in the backlog.

---

Risks worth naming up front

• Migration fidelity. The local stack must apply the same monolithic
  migration, or E2E passes against a schema prod does not have. conformance.yml
  already solves this; copy its approach rather than inventing one.
• Runner resources. Nine containers plus a browser matrix on one runner. May
  need the matrix trimmed or the stack shared per-job rather than per-shard.
• Things that genuinely need cloud stay on cloud — smoke.yml hits the live
  origin by design and must not move.
• This is a big change to the suite that guards everything else. It should
  land behind a parallel run (both configurations, compared) before the cloud
  path is removed.

Suggested shape

1. Prototype on a branch: one browser, one shard, local stack, prove green.
2. Compare a full local run against a full cloud run — same pass/fail set.
3. Flip the matrix; keep the cloud path behind workflow_dispatch for a while.
4. Remove the mutex, max-parallel: 2, the priming curl and the sleeps.
5. Re-shard messaging.

Filed as an epic because step 2 is the real work and steps 4–5 are only safe
after it.

---

Exit criterion: the production service-role key leaves CI

This epic cannot be closed while e2e.yml still injects the production secret.

Today the production SUPABASE_SERVICE_ROLE_KEY is injected at 7 sites in e2e.yml,
and that is the only remaining place — #577 removed it from ci.yml and
accessibility.yml. Those jobs run pnpm install and then an arbitrary third-party
dependency graph, so an RLS-bypassing credential is handed to that graph on every code PR.

The gate, which must return nothing:

grep -l 'secrets.SUPABASE_SERVICE_ROLE_KEY' .github/workflows/*.yml

Once it does, rotate the key once — coordinated with the 14 edge functions that
consume it, or payments, webhooks and transactional email break. Rotating before this
epic lands is a reset rather than a fix, because the key would immediately be re-injected
on the next PR. See #574 for the full reasoning.

The reference implementation is already in this repo

e2e.yml does not need anything invented. Two workflows already do exactly this:

conformance.yml:65	cp .env.local-supabase .env
conformance.yml:75	grep -E '^(NEXT_PUBLIC_SUPABASE_ANON_KEY|SUPABASE_SERVICE_ROLE_KEY)=' .env.local-supabase >> "$GITHUB_ENV"
signup-mailer.yml:57	same cp, then the host.docker.internal → localhost rewrite

.env.local-supabase is tracked in git — signup-mailer.yml:17 says plainly that these
are "the well-known PUBLIC Supabase demo keys." That is the whole trick: a local stack
needs no secret at all, so the production credential simply stops being required.

[TortoiseWolfe]

Prototype landed — PR #586. The thesis is proven, not assumed.

Run locally 2026-08-06 while the cloud project was returning HTTP 402:

local  /auth/v1/health -> 200          cloud -> 402
public tables          -> 17          identical to production
seed-test-users.ts     -> all four users created
global-setup           -> "Sign-in successful", auth state saved
mobile-check.spec.ts   -> 2 passed (16.4s)

global-setup is the exact gate that aborts on cloud with the exceed_*_quota message.
It passed against local. E2E could have been running throughout this entire outage.

Findings

• The host-derived localStorage key is a non-issue. Local produces
  sb-supabase-kong-auth-token, cloud produces sb-<ref>-auth-token — but
  tests/e2e/utils/test-user-factory.ts:591 already matches by
  startsWith('sb-') && endsWith('-auth-token'). Portable already; verified rather than
  assumed.
• test-user-factory.ts:90 emits a warning that is false locally — "Supabase rejects
  @example.com emails". Cloud-specific. Those users seeded and signed in fine against the
  local stack. Cosmetic, but it reads as a failure in a log.
• The URL differs by execution context, which the workflow must keep straight: a GitHub
  runner uses localhost:54321, a docker-network container uses supabase-kong:8000. This
  is exactly what signup-mailer.yml's sed host.docker.internal → localhost handles.

The schema check is a gate, not a comment

The workflow asserts >= 17 public tables and fails loudly. Without it, a migration that
silently stopped applying would let every test run against an empty database and report
green — #396's shape.

Remaining work on this epic

• Parity run — one full local matrix vs one full cloud matrix, same pass/fail set.
  This is the real work; everything else is mechanical.
• Point e2e.yml at the local stack (7 injection sites)
• Remove the mutex, max-parallel: 2, the priming curl, the 50s stagger
• Re-shard messaging (it runs single-shard only because shards raced each other's
  messages.delete() on shared data)
• Then rotate the service-role key once — see Both required checks are handed SUPABASE_SERVICE_ROLE_KEY, and every Supabase call in them is mocked #574

Parity cannot run until the cloud side is available for comparison, so this epic is
now blocked on #567 for its next step, though the prototype itself was not.