Skip to content

ci(smoketests): cut about two minutes from the smoke tests - #846

Merged
ross-rl merged 3 commits into
mainfrom
ross/faster-smoketests
Sep 28, 2026
Merged

ross-rl merged 3 commits into
mainfrom
ross/faster-smoketests

Conversation

@ross-rl

@ross-rl ross-rl commented Sep 28, 2026

Copy link
Copy Markdown
Contributor

Description

Makes the smoke tests that run after every Runloop deploy finish about two minutes sooner. No test is removed or changed.

  • Start 32 Jest workers, not 800. The workflow passed --maxWorkers=800%, meaning "8 workers per CPU". Jest only treats a percentage as relative to the CPU count when it is 100% or less. Anything larger is read as a plain number, so every run started 800 worker processes on a 4-CPU runner. The first test file never began until about 46 seconds after Jest started. 32 is what 800% was meant to be on these runners, and it still gives every test file its own worker. The coverage workflow had the same setting and gets the same fix.
  • Run the slow blueprint and devbox groups in separate files. Jest runs a file's describe blocks one after another. Only test.concurrent tests inside the same block overlap. object-oriented/blueprint.test.ts therefore waited for its lifecycle blueprint build before starting three unrelated groups of builds. That made it the last file to finish in every run, 40–60 seconds after the next one. Those groups now live in blueprint-build.test.ts. Likewise, the "devbox creation from blueprint and snapshot" group (about 2 minutes) moves from devbox.test.ts to devbox-from-blueprint.test.ts. Jest runs separate files in parallel. The outer describe names are kept, so all 272 full test names are unchanged.

Motivation

The TypeScript smoke-test jobs are the slowest part of the post-deploy smoketest stage in runloopai/runloop. In run 36460506602 they took about 5.5 minutes, while the Python smoketests took about 4.2 minutes. Where the "Run smoke tests" step (293s on the HTTP/2 leg) spent its time:

Phase Time
Jest start until the first test file begins (800 workers) ~46s
object-oriented/blueprint.test.ts, the last file to finish 243s
Next-slowest file (object-oriented/devbox.test.ts) 184s

The same pattern shows in the other runs since #843: a 47–48s startup gap every time, and blueprint.test.ts finishing 41–59s after the next file.

Expected saving on that run: about 42s of startup plus about 80s of tail. The tail would drop from 243s to roughly the next-slowest file, about 160s: the lifecycle build in blueprint.test.ts or executions.test.ts.

Changes

  • .github/workflows/smoke-tests.yml, .github/workflows/sdk-coverage.yml: --maxWorkers=800% becomes --maxWorkers=32.
  • tests/smoketests/object-oriented/blueprint-build.test.ts (new): the build-context, list/retrieval and network-policy groups, moved unchanged from blueprint.test.ts.
  • tests/smoketests/object-oriented/devbox-from-blueprint.test.ts (new): the from-blueprint/snapshot group, moved unchanged from devbox.test.ts.
  • tests/smoketests/object-oriented/README.md: lists the new files.

Testing

  • Ran Jest locally against an unreachable base URL, with every test filtered out, so nothing reached a real environment. With --maxWorkers=800% it started 800 worker processes (counted with ps) and took 12–13s to finish. With --maxWorkers=32 it took 2.5s.

  • Checked that the full test names are identical before and after the split: 272 each, using jest --json.

  • tsc --noEmit passes, and Prettier is clean on the changed files.

  • Not yet run against dev. To check the timing, dispatch this workflow on the branch: gh workflow run smoke-tests.yml -R runloopai/api-client-ts -r ross/faster-smoketests -f environment=dev.

  • Unit tests added

  • Integration tests added

  • Smoke Tests added/updated

  • Tested locally

Breaking Changes

None.

Checklist

  • PR title follows Conventional Commits format
  • Documentation updated (if needed)
  • Breaking changes documented (if applicable)

🤖 Generated with Claude Code

ross-rl and others added 3 commits September 28, 2026 11:15
…iles

Jest runs the describe blocks of one file one after another, so the
blueprint file waited for its lifecycle build before starting the
build-context, retrieval and network-policy builds, and was always the
last file to finish. The independent build groups and the devbox
from-blueprint group now live in separate files, which Jest runs in
parallel. Test names are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Jest treats a --maxWorkers percentage above 100% as a plain count, so
"800%" started 800 worker processes on a 4-CPU runner. Every run spent
about 46 seconds starting them before the first test file began.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Same fix as the smoke tests: Jest reads "800%" as 800 workers.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

✅ Object Smoke Tests & Coverage Report

Test Results

✅ All smoke tests passed

Coverage Results

Metric Coverage Required Status
Functions 100% 100% ✅
Lines 89.27% - ℹ️
Branches 66.14% - ℹ️
Statements 87.91% - ℹ️

Coverage Requirement: 100% function coverage (all public methods must be called in smoke tests)

✅ All tests passed and all object methods are covered!

View detailed coverage report
File Functions Lines Branches
src/sdk.ts ✅ 100% 86.86% 71.87%
src/sdk/agent.ts ✅ 100% 100% 100%
src/sdk/axon.ts ✅ 100% 93.75% 100%
src/sdk/blueprint.ts ✅ 100% 100% 80%
src/sdk/devbox.ts ✅ 100% 92.17% 94.28%
src/sdk/eviction.ts ✅ 100% 91.66% 55.55%
src/sdk/execution-result.ts ✅ 100% 92.68% 70.83%
src/sdk/execution.ts ✅ 100% 93.1% 80%
src/sdk/gateway-config.ts ✅ 100% 100% 100%
src/sdk/mcp-config.ts ✅ 100% 100% 100%
src/sdk/network-policy.ts ✅ 100% 100% 100%
src/sdk/secret.ts ✅ 100% 100% 100%
src/sdk/snapshot.ts ✅ 100% 100% 100%
src/sdk/storage-object.ts ✅ 100% 80% 48.93%

📋 View workflow run

@ross-rl
ross-rl marked this pull request as ready for review September 28, 2026 19:25
@ross-rl
ross-rl merged commit e02c8e3 into main Sep 28, 2026
9 checks passed
@ross-rl
ross-rl deleted the ross/faster-smoketests branch September 28, 2026 19:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant