ci(smoketests): cut about two minutes from the smoke tests - #846
Merged
Merged
Conversation
…iles Jest runs the describe blocks of one file one after another, so the blueprint file waited for its lifecycle build before starting the build-context, retrieval and network-policy builds, and was always the last file to finish. The independent build groups and the devbox from-blueprint group now live in separate files, which Jest runs in parallel. Test names are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Jest treats a --maxWorkers percentage above 100% as a plain count, so "800%" started 800 worker processes on a 4-CPU runner. Every run spent about 46 seconds starting them before the first test file began. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Same fix as the smoke tests: Jest reads "800%" as 800 workers. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
✅ Object Smoke Tests & Coverage ReportTest Results✅ All smoke tests passed Coverage Results
Coverage Requirement: 100% function coverage (all public methods must be called in smoke tests) ✅ All tests passed and all object methods are covered! View detailed coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Makes the smoke tests that run after every Runloop deploy finish about two minutes sooner. No test is removed or changed.
--maxWorkers=800%, meaning "8 workers per CPU". Jest only treats a percentage as relative to the CPU count when it is 100% or less. Anything larger is read as a plain number, so every run started 800 worker processes on a 4-CPU runner. The first test file never began until about 46 seconds after Jest started.32is what800%was meant to be on these runners, and it still gives every test file its own worker. The coverage workflow had the same setting and gets the same fix.describeblocks one after another. Onlytest.concurrenttests inside the same block overlap.object-oriented/blueprint.test.tstherefore waited for its lifecycle blueprint build before starting three unrelated groups of builds. That made it the last file to finish in every run, 40–60 seconds after the next one. Those groups now live inblueprint-build.test.ts. Likewise, the "devbox creation from blueprint and snapshot" group (about 2 minutes) moves fromdevbox.test.tstodevbox-from-blueprint.test.ts. Jest runs separate files in parallel. The outerdescribenames are kept, so all 272 full test names are unchanged.Motivation
The TypeScript smoke-test jobs are the slowest part of the post-deploy smoketest stage in
runloopai/runloop. In run 36460506602 they took about 5.5 minutes, while the Python smoketests took about 4.2 minutes. Where the "Run smoke tests" step (293s on the HTTP/2 leg) spent its time:object-oriented/blueprint.test.ts, the last file to finishobject-oriented/devbox.test.ts)The same pattern shows in the other runs since #843: a 47–48s startup gap every time, and
blueprint.test.tsfinishing 41–59s after the next file.Expected saving on that run: about 42s of startup plus about 80s of tail. The tail would drop from 243s to roughly the next-slowest file, about 160s: the lifecycle build in
blueprint.test.tsorexecutions.test.ts.Changes
.github/workflows/smoke-tests.yml,.github/workflows/sdk-coverage.yml:--maxWorkers=800%becomes--maxWorkers=32.tests/smoketests/object-oriented/blueprint-build.test.ts(new): the build-context, list/retrieval and network-policy groups, moved unchanged fromblueprint.test.ts.tests/smoketests/object-oriented/devbox-from-blueprint.test.ts(new): the from-blueprint/snapshot group, moved unchanged fromdevbox.test.ts.tests/smoketests/object-oriented/README.md: lists the new files.Testing
Ran Jest locally against an unreachable base URL, with every test filtered out, so nothing reached a real environment. With
--maxWorkers=800%it started 800 worker processes (counted withps) and took 12–13s to finish. With--maxWorkers=32it took 2.5s.Checked that the full test names are identical before and after the split: 272 each, using
jest --json.tsc --noEmitpasses, and Prettier is clean on the changed files.Not yet run against dev. To check the timing, dispatch this workflow on the branch:
gh workflow run smoke-tests.yml -R runloopai/api-client-ts -r ross/faster-smoketests -f environment=dev.Unit tests added
Integration tests added
Smoke Tests added/updated
Tested locally
Breaking Changes
None.
Checklist
🤖 Generated with Claude Code