feat(#575): capture the cloud E2E baseline before it becomes unrecoverable

[TortoiseWolfe]

Groundwork for #575, landed separately because it is time-critical in a way the rest
of the port is not.

The problem

#575 moves E2E onto a per-runner ephemeral Supabase. That is only safe if the local suite
runs the same tests as the cloud suite did — and nothing in this repo could express
what "the same tests" means. There was no artifact to compare against.

What this captures

Run 31048279017,
SHA de0f7f0 — the last green cloud E2E run, 66 minutes before #567 exhausted the
quota:

tests	2001
expected	1807
skipped	194
flaky	0
shape	{chromium,firefox,webkit} × {gen 585, msg 6, msg-iso 76}

Why it had to happen today

• Source artifacts expire 2026-08-12T22:13Z.
• The next cloud run that could regenerate them is impossible until the quota refills
  2026-09-02.
• The obvious fallback does not work. The github reporter only annotates
  failures, so this green run left 6 check-run annotations, not 2001. Checked
  rather than assumed — an earlier review confidently proposed annotations as the
  recovery path, and it was wrong.

Identities, not counts

A suite can drop one test, gain another, and still total 2001.

That is not hypothetical here. 228 tests — all 76 *-msg-iso per browser — sit behind
test.skip(!fixture, 'isolation seed failed…'), and seedIsolatedAdmin
(tests/e2e/utils/test-user-factory.ts:2761-2772) returns null on two silent paths
before it reaches its loud throw. If a local stack cannot seed those fixtures, all 228
skip quietly and the run reports green. A count-based gate waves that straight through.

So scripts/e2e-parity-diff.mjs compares per-test identities, directionally:

transition	verdict
expected → skipped	coverage lost — fails
present in baseline, missing entirely	fails
expected → unexpected / flaky	fails
skipped → expected	gain — reported, allowed

The last row is deliberate: a local stack can legitimately run something the cloud project
could not. At least one case is already known —
debug/capture-decryption-logs.spec.ts skips on cloud and should run locally — so a small
gained set is expected, not a bug.

Verified by making it fail

Per the repo's own rule that a probe which cannot fail proves nothing:

• --selftest mutates the baseline four ways and asserts rejection each time, including
  the same-count-different-tests case.
• 12 node:test cases in scripts/__tests__/e2e-parity-diff.test.js, run by
  pnpm test:scripts in ci.yml. Suite: 134 pass, 0 fail.
• CLI exit codes checked without a pipe, since | tail eats them: 2 on no args, 0
  on the real report, 1 on a mutated one.
• The committed baseline is asserted to contain exactly 1807/194/2001, so a silent
  regeneration cannot slip past.

Round-tripped: the real 35.9 MB merged cloud report diffs clean against the manifest
derived from it.

Two corrections to the record

Both were things I had stated as fact earlier in #575 and got wrong:

1. The 194 skips are not uniform across browsers — 60 chromium / 70 firefox / 61
   webkit. Eleven are browser-keyed and port cleanly; the rest are environment-keyed (66
   admin-dashboard, 27 avatar upload, 54 payment) and could flip on a different backend in
   either direction.
2. e2e-local.yml does not exist at de0f7f0. "Run it on that SHA" needs a merge
   branch pairing the workflow with that commit's test code — noted here so the next step
   does not trip on it.

What this does not do

No workflow changes. e2e.yml is untouched and still points at cloud. This is only the
measuring stick — widening e2e-local.yml to the full 24-job matrix is the next PR, and it
now has something to be judged against.

Refs #575

🤖 Generated with Claude Code