Repository navigation
ci: batch short e2e apps (+re-rightsize), prewarm e2e caches, retry flaky Playwright tests - #25669
Open
sentry-junior[bot] wants to merge 13 commits into
Open
sentry-junior[bot] wants to merge 13 commits into
sentry-junior[bot] wants to merge 13 commits into
Conversation
189 of the 231 required E2E jobs finish in about 42 s (p50) on the 4 vCPU runner, so they pay for cores they don't use. They now run on the small profile. The 20 apps whose jobs take longer than about 90 s set `sentryTest.runner` to keep the 4 vCPU profile. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
…2E caches Every sampled first-attempt CI run failed at least one job, and CI never retried a single test. Playwright now retries a failed test once in CI. A test that passes on retry no longer fails the job, but it is listed in the job summary, and the weekly flaky test report now also counts it, so flakes stay visible. The flaky test detector keeps `--retries=0`. The required E2E apps on the 2 vCPU runner now run four per job, one after another, which cuts their jobs from 189 to 48. Setup, Playwright and the runner queue are paid once per four apps. Every app and variant still runs, a failed app does not stop the others, and the job fails if any app failed. Apps that set `sentryTest.runner` keep a job of their own. The E2E jobs share one cache volume, where the last job to finish overwrites what the others saved. A nightly job installs every test app into the pnpm store on both E2E cache tags and prunes packages no app uses, so later jobs fork a complete store that stays under the size limit. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
Contributor
Optional E2E testsSome optional checks were not successful. They do not block the merge. 1 failing, 63 successful. See the run.
Successful checks
|
pnpm dedupe --check fetches package metadata from npm, and PR runs of this job regularly take 8+ minutes, so the 10 minute limit cancels it. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
- Warm cache: disable pnpm's verify-deps install, drop `pnpm store prune` (it empties a store whose packages were copied across filesystems), start from an empty store when the volume is mostly full, and prove the store is complete with an offline reinstall of every app. - Upload Playwright traces for tests that passed on retry. - Count flaky tests per test in the weekly report via a warning annotation per test, also in jobs that failed. - Scope clean-up between batched apps to the previous app's processes (by an inherited env marker) and the containers created since it started. - Fail the matrix step if batching drops, duplicates or changes an entry. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
…entry-javascript into ci/flaky-retries-and-batched-e2e Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
`pnpm dedupe --check` reads the metadata of every dependency, about 300 MB from npm, and the job had no cache volume, so every run fetched it again (6+ min). It also ran a full `pnpm install` just to check that pnpm-lock.yaml is stable, which `--lockfile-only` does without downloading packages. Both checks only resolve, so the job now runs on the 2 vCPU runner with its own cache volume. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
The weekly report matched a test's passed-on-retry annotation to its failure annotations by test name only. Two apps of one batched E2E job can run tests with the same name, so a flake in one app marked the same test of the other app as passed on retry too. Match by path and name; the path names the app's directory. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
Contributor
size-limit report 📦Compared: 1737fc4 (PR test merge using develop at 3aafe9e) Against: 3aafe9e (develop) · base run #63239 ✅ Baseline matches: this diff contains only this PR's changes.
|
Single npm and Playwright downloads on Namespace runners sometimes crawl at kB/s for longer than a job's timeout, and none of pnpm, npm or Playwright give up on them. - pnpm: download the standalone binary directly, verify it against the registry's integrity, abort and retry slow transfers, and keep it in the Actions cache. This replaces pnpm/action-setup's two downloads. - Root pnpm install: bounded attempts, then a longer final one. - Playwright: bounded browser download attempts. Chromium only installs system dependencies when a library is missing; on the runner image those were just fonts, and no test compares pixels. - Node integration: each npm install attempt in a suite's setup hook gets its own timeout, and the base branch keeps the suites' npm cache in the Actions cache, per shard and week. - E2E: give node-express-static's and the Angular canary variants their own labels, and fail the matrix on duplicate labels. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
joshuarli
marked this pull request as ready for review
October 22, 2026 21:43
joshuarli
requested review from
Lms24,
andreiborza,
isaacs and
logaretm
and removed request for
a team
October 22, 2026 21:43
joshuarli
changed the title
joshuarli
enabled auto-merge (squash)
October 22, 2026 22:35
JPeer264
requested changes
Oct 23, 2026
JPeer264
left a comment
JPeer264
left a comment
Member
There was a problem hiding this comment.
Not sure if this should land as is. There are slow downloads, which should be fixed upstream. We should first only focus on caching properly.
Having 12minutes is already really bad. We aimed for having 8ish minutes per run.
Blocking this for now and I'm clarifying this internally.
Lms24
requested changes
Oct 23, 2026
Lms24
left a comment
Lms24
left a comment
Member
There was a problem hiding this comment.
This is not reviewable. It looks like this PR combines 4 changes and has a far too long and tedious description. Please split up the changes into individual PRs.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
i was able to cut CI down significantly with caching slow downloads, bounded retries (we download a bunch of stuff lol) to kill download flakiness, and best of all batched e2e jobs (+ a re-rightsizing)
no test coverage changes/reductions at all
here's the end result (compared to a few close commits (21f2cf1, 40eff80, f15e256, d476963) on master excluding skipped/optionally flaky jobs)
5b5026c)Baselines are push runs; the after run is a pull_request run with a colder cache tag. E2E app coverage is identical (232 apps both sides).
CI-only changes on top of #25667 (merged). None drops a test or a variant.
Retry flaky tests, and keep reporting them. Every first-attempt CI run I sampled failed at least one job, but CI never retried a single test (
retries: 0for E2E outside dev mode and for browser integration). Now Playwright retries a failed test once in CI. A test that only passes on retry doesn't fail the job, but it still shows up:Flaky tests passed on retrystep;Passed on retry: …warning annotation, and its failed attempt's traces are uploaded even though the job passed;weekly-flaky-tests.mjscounts flaky tests per test from those annotations, also in jobs where another test failed for good, in a new "Passed on retry" column. It now also fetches annotations for passing jobs that ran the flaky step, so it makes more API calls than before.test:detect-flakypasses--retries=0, so new tests still have to pass on every run. The apps that setretries: 0themselves (the Cloudflare, Hono and send-to-Sentry apps) keep it.Four short E2E apps per job. The median required E2E job runs about 55 s but waits about 77 s for a runner, and it repeats checkout, artifact download, Playwright install and prepare for every app. The apps on the 2 vCPU runner now run four per job, one after another, which cuts their jobs from 189 to 48 (231 → 90 E2E jobs in total). Details:
continue-on-errorsteps with the old time limits, its own directory and its own traces, so a failed app doesn't stop the others.Check E2E resultsthen fails the job.SENTRY_E2E_APPenv marker (inherited by detached servers too) and the containers created since it started.sentryTest.runnerkeep a job of their own.All required jobs passed or were skipped.getTestMatrix.mjsfails the matrix step if batching drops, duplicates or changes an entry (231 entries today), or if two entries share a label.Warm the E2E pnpm store nightly. All E2E jobs share one Namespace cache volume per tag, and the last job to finish overwrites what the others saved. So the store never holds every app's packages, and the downloads it misses are the ones that crawl and time out the builds. The new
warm-e2e-cache.ymlruns on both E2E tags:--ignore-scripts) against the tarballs of the latest develop build;pnpm store prune: the apps install outside the cache volume, so their files are copies, and prune would empty the store. Instead it starts from an empty store when the volume is ≥70% full (Namespace stops cloning a cache that outgrows its limit);--offlineto prove the store alone is complete, and fails if not;node_modules;This depends on Namespace spreading new generations across its nodes, so treat it as likely to help, not proven. The other half of the idea, a runner image with Playwright, Bun, Deno and SAM already installed, is a runner-profile setting in the Namespace dashboard, so it isn't in this PR.
prepare-e2e-testnow combines two smaller actions,setup-e2e-tests(runs once per job) andprepare-e2e-app(runs once per app).optional-e2e.ymlstill uses it unchanged.Faster
Check lockfile. The job had no cache volume, sopnpm dedupe --checkfetched the metadata of every dependency again on each run: about 300 MB from npm (1.2 GB unpacked), which took 6+ minutes and timed out #25667 three times. It also ran a fullpnpm installonly to check that pnpm-lock.yaml is stable. Now the job keeps pnpm's caches on its own volume (sentry-javascript-lockfile,-prfor PRs), checks the lockfile withpnpm install --lockfile-only, and runs on 2 vCPUs, because both checks only resolve. Locally, the two checks took 25 s cold and 4 s warm, and--lockfile-onlychanged pnpm-lock.yaml exactly like a full install for a changed range and for a new dependency. It also runs on CI changes now, so changes like this one get tested.Bounded retries, and the Actions cache for the slowest downloads. Single downloads from npm sometimes crawl at kB/s for longer than a job's timeout, and pnpm, npm and Playwright never give up on them. These were the most common infra failures left:
install-pnpmreplacespnpm/action-setup, which installed a bootstrap pnpm with npm and then downloaded a second 24 MB pnpm in every job. It downloads the standalone@pnpm/exebinary directly, checks it against the registry's integrity, aborts transfers under 256 kB/s so a retry gets a new connection, and keeps the binary in the Actions cache (~24 MB, saved outside PRs only).pnpm installand the Playwright browser download get bounded attempts (300 s, 300 s, then 600 s). apt is never killed midway.install-depsunlesslddfinds a missing library. On the runner image, the only packages it installed for Chromium were 9 font packages, which took 6–10 min on a cold cache, and no test compares pixels. WebKit and Firefox still install their dependencies.npm installattempt in a Node integration suite's setup hook gets a 50 s timeout (the hook limit goes from 120 s to 210 s), and the base branch's Node 24 jobs keep the suites' npm cache in the Actions cache, one entry per shard and ISO week. The weekly key means suite edits don't save extra copies, and versions the suites stopped using drop out. The repo's Actions cache is already at its 10 GB limit, mostly with staledependencies-*caches of the old workflow, so these are the only two new entries.node-express-static's variant and the two Angular canary variants had labels shared with other apps. They now have their own.via @joshuarli.
--
View Junior Session [Sentry]