Repository navigation
ci: shorten test runs with two workers and faster fixture shutdown - #5051
Merged
Merged
Conversation
Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 258e835c-9f3b-49d2-bd15-16fb97e12ca5
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Parallel execution still needs passing full-matrix CI results on the final rebased tree.
No open issues 🎉
What changed in this PR
Shortens Spec Kit’s test feedback without reducing the existing OS/Python matrix or test selection.
Changes:
- Runs the full suite with two pytest workers.
- Reduces test HTTP-server shutdown polling to 0.01 seconds.
- Adds test-only parallelization dependencies and audit snapshot entries.
| File | Description |
|---|---|
tests/specify_cli/integrations/test_installed_adapters.py |
Shortens server polling while preserving cleanup. |
pyproject.toml |
Adds pytest-xdist to the test extra. |
.github/workflows/test.yml |
Enables two workers in existing test jobs. |
.github/security-audit-requirements.txt |
Adds hashed dependency pins and provenance. |
Review effort: Max
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Shorten test feedback without reducing the existing OS/Python matrix or selecting fewer tests:
pytest-xdist>=3.0only to the test extra. Add hashed execnet 2.1.2 and pytest-xdist 3.8.0 entries and their provenance to the audit snapshot, without changing existing pins or audit behavior.These are four files, 16 insertions and two deletions. No production code, test exclusions, workflow restructuring, runner settings, or permissions change.
For a directional reference, the slowest test step in the final fork candidate was 833 seconds, versus a mean of 1,675.4 seconds for the slowest test step in each of five successful upstream main push runs sampled on October 20: 39487888924, 39470755550, 39438133070, 39433509775, and 39318274471. This is about 50% lower, not a controlled speedup estimate: commits, test inventories, runner images, and resolved dependencies differ. The metric excludes queue, setup, and retries and is not end-to-end workflow time.
Testing
uv run specify --helpuv sync && uv run pytestExisting hosted evidence: KSchlobohm/spec-kit#92, source
5a4ff5536876845cf894dc1518ec0ac52f916b3b, actual checkout mergea2ac24a3a6d42312dabcfc665a497631e3f4bd9c. Its four-file patch is reproduced here on upstream mainf90f7ddf4fa17b2f3bee427f27339cd4d61b0d1a, without changing the patch's scope or behavior.The final fork run passed
uv run pytest -n 2in all six cells. Each collected 12,787 tests and passed all 671 installed-adapter cases:The two macOS jobs initially failed to acquire runners and executed no steps. One user-authorized failed-jobs-only retry passed both; the original four Ubuntu/Windows passes were retained. There were no executed test failures.
Fork Ruff, markdownlint, shellcheck, the extension version guard, and both CodeQL analyses passed. Dependency audit passed, including the freshness check actually executing on changed dependency inputs and pip-audit reporting no known vulnerabilities.
Checks on this upstream-based worktree: Windows, Python 3.14.8, uv 0.13.2, pytest 9.1.1, pytest-xdist 3.8.0:
uv sync --extra test --python 3.14.8uv run specify --help.venv\Scripts\python -m pytest tests\test_security_workflow.py tests\specify_cli\integrations\test_installed_adapters.py -k 'test_security_workflow or test_registered_package_force_replacement_restores_sources_and_config_on_rollback or test_round9_optional_metadata' -n 2 -q.venv\Scripts\python -m pytest tests --collect-only -quvx ruff@0.15.0 check src testsuvx --from pip-audit==2.10.0 pip-audit --disable-pip --require-hashes -r .github\security-audit-requirements.txt --progress-spinner offgit diff --checkThose checks were run on upstream base
505781d7cabb5180b7098a7d8371c48456f9d4f6. Before publication, upstream gained two integration fixes. The same four-file change was rebased without conflicts ontof90f7ddf4fa17b2f3bee427f27339cd4d61b0d1a. The focused command above was rerun withor test_check_skips_optional_tools_for_external_ide_adapteradded to its-kselection: 83 passed. Collection was rerun with the same command: 12,915 collected.git diff --check origin/main HEADpassed.The final upstream tree has 128 more collected cases than the fork evidence. The full upstream-based suite has not been run locally; its normal PR CI is still needed. Local dependency resolution also differs from the hosted fork environment, so these focused results are not a repeated six-cell validation.
Earlier non-timing fixture probes passed against both original and changed fixtures for HTTP 200/404 responses, distinct roots/ports, and closed sockets/joined threads after normal and simulated-error exits. An earlier same-module Windows/Python 3.13 comparison passed the same 671 cases with two workers: summed worker teardown time fell from 203.53 to 5.61 seconds (baseline, polling treatment). This sum is not wall time; dependencies also drifted, so it does not establish a controlled whole-job gain.
No new permanent exact-YAML or matrix assertions are included. Existing success/failure tests, fixture probes, and hosted results provide observed evidence, not a permanent regression guard for worker count or performance. Sample-project testing is not applicable: the dependency addition is test-only and no shipped CLI or slash-command behavior changes.
AI Disclosure
AI disclosure: Prepared with GitHub Copilot using GPT-6.1 Sol (model ID: gpt-6.1-sol), operating autonomously within a user-directed scope. AI assistance covered code/CI edits, dependency snapshot verification, validation, evidence analysis, and drafting this description. Reasoning effort was not explicitly configured in the task.