Skip to content

ci: shorten test runs with two workers and faster fixture shutdown - #5051

Merged
KSchlobohm merged 3 commits into
mainfrom
kschlobohm-oct-21-upstream-ci-improvement
Oct 21, 2026
Merged

KSchlobohm merged 3 commits into
mainfrom
kschlobohm-oct-21-upstream-ci-improvement

Conversation

@KSchlobohm

Copy link
Copy Markdown
Contributor

Description

Shorten test feedback without reducing the existing OS/Python matrix or selecting fewer tests:

  • Run the full suite with two pytest-xdist workers in each existing Ubuntu, Windows, and macOS job, on Python 3.13 and 3.14. Leave the independent Ruff job unchanged.
  • Reduce the installed-adapter test HTTP server's shutdown polling interval to 0.01 seconds. Preserve its per-test temporary root, ephemeral port, and shutdown/socket-close/thread-join cleanup.
  • Add pytest-xdist>=3.0 only to the test extra. Add hashed execnet 2.1.2 and pytest-xdist 3.8.0 entries and their provenance to the audit snapshot, without changing existing pins or audit behavior.

These are four files, 16 insertions and two deletions. No production code, test exclusions, workflow restructuring, runner settings, or permissions change.

For a directional reference, the slowest test step in the final fork candidate was 833 seconds, versus a mean of 1,675.4 seconds for the slowest test step in each of five successful upstream main push runs sampled on October 20: 39487888924, 39470755550, 39438133070, 39433509775, and 39318274471. This is about 50% lower, not a controlled speedup estimate: commits, test inventories, runner images, and resolved dependencies differ. The metric excludes queue, setup, and retries and is not end-to-end workflow time.

Testing

  • Tested locally with uv run specify --help
  • Ran existing tests with uv sync && uv run pytest
  • Tested with a sample project (if applicable)

Existing hosted evidence: KSchlobohm/spec-kit#92, source 5a4ff5536876845cf894dc1518ec0ac52f916b3b, actual checkout merge a2ac24a3a6d42312dabcfc665a497631e3f4bd9c. Its four-file patch is reproduced here on upstream main f90f7ddf4fa17b2f3bee427f27339cd4d61b0d1a, without changing the patch's scope or behavior.

The final fork run passed uv run pytest -n 2 in all six cells. Each collected 12,787 tests and passed all 671 installed-adapter cases:

OS / Python Passed Skipped Test-step seconds
Ubuntu / 3.13 12,690 97 414
Ubuntu / 3.14 12,690 97 519
Windows / 3.13 12,144 643 813
Windows / 3.14 12,144 643 833
macOS / 3.13 12,686 101 549
macOS / 3.14 12,686 101 637

The two macOS jobs initially failed to acquire runners and executed no steps. One user-authorized failed-jobs-only retry passed both; the original four Ubuntu/Windows passes were retained. There were no executed test failures.

Fork Ruff, markdownlint, shellcheck, the extension version guard, and both CodeQL analyses passed. Dependency audit passed, including the freshness check actually executing on changed dependency inputs and pip-audit reporting no known vulnerabilities.

Checks on this upstream-based worktree: Windows, Python 3.14.8, uv 0.13.2, pytest 9.1.1, pytest-xdist 3.8.0:

Command Result
uv sync --extra test --python 3.14.8 Passed; isolated editable install resolves to this worktree
uv run specify --help Passed
.venv\Scripts\python -m pytest tests\test_security_workflow.py tests\specify_cli\integrations\test_installed_adapters.py -k 'test_security_workflow or test_registered_package_force_replacement_restores_sources_and_config_on_rollback or test_round9_optional_metadata' -n 2 -q 82 passed, including existing success and failure/rollback coverage
.venv\Scripts\python -m pytest tests --collect-only -q 12,904 collected
uvx ruff@0.15.0 check src tests Passed
uvx --from pip-audit==2.10.0 pip-audit --disable-pip --require-hashes -r .github\security-audit-requirements.txt --progress-spinner off Passed; no known vulnerabilities
git diff --check Passed

Those checks were run on upstream base 505781d7cabb5180b7098a7d8371c48456f9d4f6. Before publication, upstream gained two integration fixes. The same four-file change was rebased without conflicts onto f90f7ddf4fa17b2f3bee427f27339cd4d61b0d1a. The focused command above was rerun with or test_check_skips_optional_tools_for_external_ide_adapter added to its -k selection: 83 passed. Collection was rerun with the same command: 12,915 collected. git diff --check origin/main HEAD passed.

The final upstream tree has 128 more collected cases than the fork evidence. The full upstream-based suite has not been run locally; its normal PR CI is still needed. Local dependency resolution also differs from the hosted fork environment, so these focused results are not a repeated six-cell validation.

Earlier non-timing fixture probes passed against both original and changed fixtures for HTTP 200/404 responses, distinct roots/ports, and closed sockets/joined threads after normal and simulated-error exits. An earlier same-module Windows/Python 3.13 comparison passed the same 671 cases with two workers: summed worker teardown time fell from 203.53 to 5.61 seconds (baseline, polling treatment). This sum is not wall time; dependencies also drifted, so it does not establish a controlled whole-job gain.

No new permanent exact-YAML or matrix assertions are included. Existing success/failure tests, fixture probes, and hosted results provide observed evidence, not a permanent regression guard for worker count or performance. Sample-project testing is not applicable: the dependency addition is test-only and no shipped CLI or slash-command behavior changes.

AI Disclosure

  • I did not use AI assistance for this contribution
  • I did use AI assistance (fill in the disclosure below)

AI disclosure: Prepared with GitHub Copilot using GPT-6.1 Sol (model ID: gpt-6.1-sol), operating autonomously within a user-directed scope. AI assistance covered code/CI edits, dependency snapshot verification, validation, evidence analysis, and drafting this description. Reasoning effort was not explicitly configured in the task.

Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 258e835c-9f3b-49d2-bd15-16fb97e12ca5
@KSchlobohm
KSchlobohm requested a review from mnriem as a code owner October 21, 2026 14:05
Copilot AI max review requested due to automatic review settings October 21, 2026 14:05
Comment thread .github/security-audit-requirements.txt Outdated
Comment thread .github/security-audit-requirements.txt Outdated
KSchlobohm and others added 2 commits October 21, 2026 09:10
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
@KSchlobohm KSchlobohm self-assigned this Oct 21, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Parallel execution still needs passing full-matrix CI results on the final rebased tree.

No open issues 🎉

What changed in this PR

Shortens Spec Kit’s test feedback without reducing the existing OS/Python matrix or test selection.

Changes:

  • Runs the full suite with two pytest workers.
  • Reduces test HTTP-server shutdown polling to 0.01 seconds.
  • Adds test-only parallelization dependencies and audit snapshot entries.
File Description
tests/​specify_cli/​integrations/​test_installed_adapters.py Shortens server polling while preserving cleanup.
pyproject.toml Adds pytest-xdist to the test extra.
.github/​workflows/​test.yml Enables two workers in existing test jobs.
.github/​security-audit-requirements.txt Adds hashed dependency pins and provenance.

Review effort: Max

Copilot AI max review requested due to automatic review settings October 21, 2026 14:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The passing fork matrix covers a different test inventory, so suite-wide parallel execution still needs validation on the final upstream-based tree.

No open issues 🎉

Review effort: Max

@KSchlobohm
KSchlobohm merged commit be7bcf6 into main Oct 21, 2026
19 checks passed
@KSchlobohm
KSchlobohm deleted the kschlobohm-oct-21-upstream-ci-improvement branch October 21, 2026 14:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants