Repository navigation
"Run tests at cursor" takes exponentially longer with more pytest tests #21151
Description
Type: Bug
When using Python pytest to parametrize tests, the test explorer takes exponentially longer, while CPU is at ~100%, the more tests are created using parametize when using "Run tests at cursor" to start the tests (starting from the "Run tests" button in test explorer works fine).
Here is a minimal example:
import pytest
@pytest.mark.parametrize("value", range(10))
def test_test(value):
assert TrueWith range(10) my system (Apple M1 Pro 16 GB) takes ~1 second (measured by hand with stopwatch).
With range(100) it takes ~8 seconds
With range(150) it takes ~50 seconds
I'd expect it to take roughly as long as running it from test explorer or pytest <file> from command line - which is about 0.1 seconds (<1 second for range(1500)).
VS Code version: Code 1.76.2 (Universal) (ee2b180d582a7f601fa6ecfdad8d9fd269ab1884, 2023-03-14T17:54:09.061Z)
OS version: Darwin arm64 22.3.0
Modes:
Sandboxed: No
System Info
| Item | Value |
|---|---|
| CPUs | Apple M1 Pro (10 x 24) |
| GPU Status | 2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled metal: disabled_off multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_renderer: enabled_on video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: disabled_off |
| Load (avg) | 5, 43, 74 |
| Memory (System) | 16.00GB (0.07GB free) |
| Process Argv | --crash-reporter-id 3f533b57-abc7-4565-8781-b3146cb3a6b8 |
| Screen Reader | no |
| VM | 0% |
Extensions (33)
| Extension | Author (truncated) | Version |
|---|---|---|
| vscode-gcode-syntax | app | 0.7.7 |
| better-toml | bun | 0.3.2 |
| vscode-markdownlint | Dav | 0.49.0 |
| vscode-eslint | dba | 2.4.0 |
| EditorConfig | Edi | 0.16.4 |
| elm-ls-vscode | elm | 2.6.0 |
| json-tools | eri | 1.0.2 |
| prettier-vscode | esb | 9.10.4 |
| copilot | Git | 1.78.9758 |
| vscode-test-explorer | hbe | 2.21.1 |
| flux | inf | 1.0.4 |
| elixir-ls | Jak | 0.13.0 |
| vscode-edit-csv | jan | 0.7.4 |
| vscode-home-assistant | kee | 1.35.0 |
| vscode-sshfs | Kel | 1.26.0 |
| vscode-docker | ms- | 1.24.0 |
| vscode-language-pack-de | MS- | 1.76.2023030809 |
| flake8 | ms- | 2023.4.0 |
| isort | ms- | 2022.8.0 |
| python | ms- | 2023.4.1 |
| vscode-pylance | ms- | 2023.3.40 |
| jupyter | ms- | 2023.2.1200692131 |
| jupyter-keymap | ms- | 1.1.0 |
| jupyter-renderers | ms- | 1.0.15 |
| vscode-jupyter-cell-tags | ms- | 0.1.8 |
| vscode-jupyter-slideshow | ms- | 0.1.5 |
| remote-containers | ms- | 0.288.0 |
| test-adapter-converter | ms- | 0.1.7 |
| vscode-typescript-tslint-plugin | ms- | 1.3.4 |
| vscode-commons | red | 0.0.6 |
| vscode-yaml | red | 1.12.2 |
| rust-analyzer | rus | 0.3.1451 |
| markdown-all-in-one | yzh | 3.5.1 |
(6 theme extensions excluded)
A/B Experiments
vsliv368:30146709
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vswsl492cf:30256860
vslsvsres303:30308271
vserr242cf:30382550
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscoreces:30445986
pythondataviewer:30285071
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
cmake_vspar411:30581797
vsaa593cf:30376535
pythonvs932:30410667
cppdebug:30492333
vsclangdc:30486549
c4g48928:30535728
dsvsc012cf:30540253
pynewext54:30695312
azure-dev_surveyone:30548225
vsccc:30610678
nodejswelcome1:30587005
3biah626:30602489
pyind779:30671433
f6dab269:30613381
pythonsymbol12:30671437
2i9eh265:30646982
6233i204:30672705
vsccsb:30677849
pythonb192:30669360
functionswalk:30687959
pythonms35:30701012
Activity
vscodenpa commentedon Apr 1, 2023
Thanks for creating this issue! It looks like you may be using an old version of VS Code, the latest stable release is 1.77.0. Please try upgrading to the latest version and checking whether this issue remains.
Happy Coding!
farmio commentedon Apr 1, 2023
I just did the update to 1.77.0 and the issue persists.
Version: 1.77.0 (Universal)
Commit: 7f329fe6c66b0f86ae1574c2911b681ad5a45d63
Date: 2023-03-29T09:57:11.797Z
Electron: 19.1.11
Chromium: 102.0.5005.196
Node.js: 16.14.2
V8: 10.2.154.26-electron.0
OS: Darwin arm64 22.3.0
Sandboxed: No
connor4312 commentedon Apr 24, 2023
This takes a while for me, but I think this is on the Python side. Looking at the Output window, they run one process per parameterized "variant" (not sure of the correct term.) So that's 150 processes for
range(150). VS Code core is fine.connor4312 commentedon Apr 24, 2023
oh, actually, this is because we "include" all of them individually. Hmm. We could probably optimize that in VS Code. I'll put in a change there.
12 remaining items
bw4sz commentedon Dec 23, 2023
This seems really critical to me, I can see a 10x speed slowdown in using vscode versus running pytest. Is there a workaround? x-dist?
eleanorjboyd commentedon Jun 28, 2024
Hi Connor Peet (@connor4312), wanted to follow up here sorry its been a minute since we spoke on this. I am attempting to implement and I am still seeing the parameterized children test items get passed as part of the runTests request: TestRunRequest. I am trying it with this example:
which produces 10 parameterized tests and then the function node that is the parent of those but is runnable would be test_numbers. I checked and test_numbers has the same range as the children items (the 10 parameterized tests) but both the 10 parameterized tests + the test_numbers node show in the test run request.
Are there any things other than range which are checked to determine if the children should be coalesce? And am I right assuming that the children being combined means I should expect one test run node for the parent test_numbers and not 11?
Thanks!
connor4312 commentedon Jun 28, 2024
Which version of the Python extension are you using? On the current prerelease I'm seeing the same behavior that the parent node of parameterized tests don't have an associated range here
You can also check this by seeing if the "go to test" button in the Test Explorer view works for a given test
eleanorjboyd commentedon Jul 1, 2024
This was on a branch I made with edits to add a range for the parent item - Ill give the go to test button a look to confirm and if I am able to verify ill send over a link to the branch. Thanks!
eleanorjboyd commentedon Jul 1, 2024
Connor Peet (@connor4312) here's a video of what I am seeing, the issue being that all 4 (the 3 parameterized tests and single main test) are being included in the request. Here is a link to the branch: https://github.com/eleanorjboyd/vscode-python/tree/21151-param-gutter
param-gutter-all-showing.mov
connor4312 commentedon Jul 2, 2024
Over a call we noticed that Eleanor was testing the gutter click action where this logic hadn't been implemented. I made a PR on VS Code (linked) to do this. It's a little more complicated, but also generic.
testing: simplify tests when actioning gutter items
testing: simplify tests when actioning gutter items (#219809)
testing: simplify tests when actioning gutter items (microsoft#219809)
eleanorjboyd commentedon Sep 23, 2026
🤖 Thanks for the report and detailed reproduction. This was fixed in VS Code core by microsoft/vscode@6ab3a15 via microsoft/vscode#219809: gutter-triggered test runs now coalesce parameterized children into their runnable parent instead of including every child individually. This can be closed as completed.