[zibi-pplx]

Summary

The Python tests job now runs its 1,978 tests on one pytest-xdist worker per CPU, four on its runner, instead of in one process. It was one of the jobs, each taking 4-5 min, that a pull request without Kind smoke waits on.

	One process	With this PR (CI run)
Python tests job	3.5 min	2m21s
pytest	163 s	84 s, 4 workers
Result	1,972 passed, 6 skipped	1,972 passed, 6 skipped

How. ci/scripts/run-python-tests.sh passes --numprocesses (one worker per CPU by default, SPACE_PYTHON_TEST_WORKERS to override, 0 for one process) and --dist loadfile.

• --dist loadfile keeps each file's tests on one worker. Tests in a file can share module state. The run's Postgres server has one database, in which test_prune_cluster_artifacts.py, the only file that uses it, creates fixed table names. With this, no two of those tests run at once.
• pytest-xdist 3.8.0, with its dependency execnet, joins the locked test group in pyproject.toml and uv.lock.

Spreading single tests instead of files would balance the workers better than loadfile's 84 s. With this change the job is no longer among the slowest of a pull request, so this PR keeps the safer grouping.

Stack

1. #1503: unblock the dependency checks; keep GitHub outages from failing PRs.
2. #1473: run the shell regression tests in parallel.
3. This PR: run the Python tests in parallel.
4. #1476: lint alone only the crates a change can affect.

Validation

• This PR's CI ran all 1,978 tests on 4 workers: 1,972 passed and 6 were skipped, the same as one process on main.
• Locally on macOS, 23 tests fail and 21 error on 4 workers. I checked one of those files, test_bench_sbx_balloon.py: it also hangs in one process (SPACE_PYTHON_TEST_WORKERS=0), so it does not run on macOS at all. The rest I did not re-run one at a time. Linux CI runs them all.

Severity and risk

Severity: does not apply; this is a CI speedup.

Risk:

• A test that depends on another file's test running first in the same process, or on a fixed port or path, could fail on a worker of its own. SPACE_PYTHON_TEST_WORKERS=0 runs one process again.
• One more dependency in the test group. It is pinned and locked like the others.
• To roll back, revert this PR.

AI assistance: Claude Code drafted the change and this description.

🤖 Generated with Claude Code

[coderabbitai]

Automatic approval withheld

Automatic approval is disabled by the configured approval policy for this pull request. Review comments and requests for changes continue normally.

Commit: 0f577d58ab09.

No actionable comments were generated in the recent review.

Walkthrough

The Python test dependency group now pins pytest-xdist to version 3.8.0. The test runner uses SPACE_PYTHON_TEST_WORKERS to set the worker count, with auto as the default and 0 for single-process execution. It passes --dist loadfile so tests from each file run on one worker.

Possibly related PRs

• perplexityai/SPACE#1473: This PR applies environment-controlled parallel test execution to a separate CI test runner.

Merge Risk: Minimal · previous review 6873d

The change only alters CI test scheduling, with dependency wiring and the single-process fallback intact; no concrete merge-blocking risk remains.

Security architecture risk: Low · up to 6873d

The workers remain within the existing CI job without changes to deployment privileges or database provisioning. File-level scheduling and isolated database schemas limit interference. Cancellation cleanup and concurrent-job isolation are not fully established, but no introduced security failure was identified.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — The demonstrated change affects execution concurrency within the existing Python CI job and access to its test database. The full comparison contains no IAM, secret, production service, or network-policy changes.

Trust Boundaries and Controls

• observed — The worker override is passed as one quoted pytest argument, not evaluated as shell code. Database resource creation remains in the parent shell before worker execution.

Resilience and Maintainability Implications

• observed — After successful container creation, an EXIT trap attempts forced removal on ordinary completion or subsequent failure. Cleanup errors are ignored, and the trap is installed after docker run. These pre-existing controls do not establish cleanup after forced termination or ownership isolation between concurrent invocations sharing a build identifier.

Pre-merge checks: 5

Check name	Status	Explanation
Title check		The title uses the required lowercase area prefix "ci:", starts the summary with lowercase "run", and accurately describes the test parallelization change.
Description Follows The Pr Template		The description contains all required template items: a substantive "## Summary" section, a "## Validation" section naming the CI run and reporting its result, a "## Severity and risk" section with "d…
Bug Fixes Name A Regression Test		The pull request changes only the Python CI runner and test dependencies (ci/scripts/run-python-tests.sh, pyproject.toml, and uv.lock). It does not fix application behavior under crates/, `shi…
Weakened Safeguards Are Justified		The diff adds pytest-xdist, worker options, and explanatory comments. It does not remove tests, assertions, lint checks, CI steps, or repository guards. The runner still invokes pytest on the reposito…
Public-Safe Content		The title, description, and reviewed diff contain no credentials, tokens, private keys, internal or private links, cloud IDs/ARNs, personal workstation paths, production logs, incident details, or und…

Full details: Description Follows The Pr Template

Explanation

The description contains all required template items: a substantive "## Summary" section, a "## Validation" section naming the CI run and reporting its result, a "## Severity and risk" section with "does not apply" severity and concrete risks, and an "AI assistance:" line stating that Claude Code drafted the change and description.

Full details: Bug Fixes Name A Regression Test

Explanation

The pull request changes only the Python CI runner and test dependencies (ci/scripts/run-python-tests.sh, pyproject.toml, and uv.lock). It does not fix application behavior under crates/, shim/, or ui/, so the regression-test requirement does not apply.

Full details: Weakened Safeguards Are Justified

Explanation

The diff adds pytest-xdist, worker options, and explanatory comments. It does not remove tests, assertions, lint checks, CI steps, or repository guards. The runner still invokes pytest on the repository root with --strict-config and --strict-markers; --dist loadfile preserves file-level test grouping, and the comment gives the rationale for parallel execution.

Full details: Public-Safe Content

Explanation

The title, description, and reviewed diff contain no credentials, tokens, private keys, internal or private links, cloud IDs/ARNs, personal workstation paths, production logs, incident details, or undisclosed vulnerabilities. The changed files add pytest-xdist configuration and comments in ci/scripts/run-python-tests.sh (lines 106-125), the dependency in pyproject.toml:34, and standard public PyPI package URLs in uv.lock. The description links to a public GitHub Actions run and does not disclose sensitive content.

Recent review info

Automatic approval policy

Commit: 0f577d58ab09.

Setting	Effective value	Source
Automatic approvals	Disabled	Override 4
Merge risk	Low	Override 2
Security risk	Not set	Global configuration

Matching overrides (numbered as in the resolved configuration):

• Override 2 — any author; paths: **, !ui/**; merge risk: Low.
• Override 4 — any author; paths: .coderabbit.yaml, .coderabbit.yml, coderabbit.yaml, coderabbit.yml, .coderabbit.config.ts, **/AGENTS.md, **/CLAUDE.md, .agents/**, .claude/**, .claude-plugin/**, CONTRIBUTING.md, docs/versioning.md, docs/rust/space-conventions.md, docs/rust/space-review-lessons.md, .github/**, ci/**, scripts/source_guards/**, scripts/check*, clippy.toml, deny.toml, .gitleaks.toml, .pre-commit-config.yaml, lefthook.yaml, rust-toolchain.toml, .cargo/**, supply-chain/**, pyproject.toml, uv.lock, **/package.json, **/package-lock.json, **/go.mod, **/go.sum, **/requirements*.txt, **/requirements*.in, proto/**, db/**, crates/l40_api/api/src/auth.rs, crates/l40_api/api/src/auth/**, crates/l40_api/api/src/middleware/auth*, crates/l40_api/api/src/middleware/auth/**, crates/l40_api/api/src/routes/authz.rs, crates/l10_foundation/bearer/**, crates/l10_foundation/rustls/**, crates/l10_foundation/guest/**, crates/l10_foundation/pii/**, shim/**, docker/**, templates/**, infra/space/aws/**, infra/space/cluster/**, infra/space/identity/**, infra/space/observability/**, infra/space/packer/**, infra/space/kind/charts/**, LICENSE, SECURITY.md; automatic approvals: disabled.

Any matching disabled override withholds approval, regardless of risk. Otherwise, the strictest explicitly set limit wins for each score. Limits omitted by all matching overrides inherit the global configuration.

Not set means no limit was configured after inheritance. Use @coderabbitai configuration to inspect the resolved settings and their sources.

Run configuration

• Source mode: Auto
• Repository: perplexityai/SPACE
• Branch: zibi/python-tests-parallel
• File: .coderabbit.yaml
• Commit: 6873d09
• Layers: Repository YAML; Enforced overrides: Organization
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 59d0c0c1-39cd-4017-9873-d63fb01bb339

Commits

Reviewing files that changed from the base of the PR and between 41e0ef0 and 6873d09.

Files ignored due to path filters: 1

• uv.lock is excluded by !**/*.lock, !uv.lock

Files selected for processing: 2

• ci/scripts/run-python-tests.sh
• pyproject.toml

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review.

[the-gigi-pplx]

Review

No findings.

Verified against b08bd967048aaa3c97a35545f1ac70824a9a9279.

What I ran. The pytest command from run-python-tests.sh on macOS with Python 3.11, once with --numprocesses 0 and once with 4 --dist loadfile. Postgres wasn't available, so that file's tests skipped, and I left out test_bench_sbx_balloon.py, which hangs here in one process too.

• Both runs failed the same 26 tests, so on this machine none of the failures comes from parallel runs.
• The 4-worker run took 64 s, against 227 s in one process.
• This answers the description's open macOS question: the failures predate this PR.

What I read.

• PRUNE_CLUSTER_ARTIFACTS_TEST_DATABASE_URL is read only by test_prune_cluster_artifacts.py, so --dist loadfile keeps the shared database to one worker.
• No conftest.py exists, and no session-scoped fixture or session hook would now run once per worker.
• The fixed ports and /tmp paths in the tests are strings passed to fakes, not sockets or files.

[the-gigi-pplx]

Approved. No findings; details in the review comment above.