Skip to content

Add deterministic parser performance baseline suite with isolated CI workflow - #54

Draft
jkowalleck with Copilot wants to merge 2 commits into
mainfrom
copilot/add-deterministic-performance-benchmarking
Draft

jkowalleck with Copilot wants to merge 2 commits into
mainfrom
copilot/add-deterministic-performance-benchmarking

Conversation

Copilot AI commented Jul 20, 2026 •

Copy link
Copy Markdown

This PR introduces a reproducible performance baseline for parser APIs using existing syntax fixtures, with machine-readable output for historical comparison. Perf coverage is isolated from regular test/release pipelines via a dedicated pytest marker and workflow split.

  • Perf benchmark module (tests/test_perf.py)

    • Adds a dedicated @pytest.mark.perf test that reuses valid_syntax.json / invalid_syntax.json.
    • Benchmarks explicit public parser entry points:
      • parse
      • is_valid_syntax
      • make_syntax_validator
      • RFC3987_SYNTAX_TERM_VALIDATORS[term]
      • exported is_valid_syntax_<term> validators (for fixture-backed terms)
    • Separates build/init timing from runtime parse timing.
    • Captures deterministic runtime metrics plus tracemalloc peak memory.
    • Emits structured JSON results (RFC3987_PERF_OUTPUT_JSON, default .pytest_perf/perf-results.json).
  • Deterministic loop controls

    • Adds env-driven loop/warmup configuration:
      • RFC3987_PERF_LOOPS (default 1 for CI safety)
      • RFC3987_PERF_WARMUP_LOOPS (default 1)
    • No random input generation; corpus and execution path are fixed.
  • Pytest marker registration

    • Registers perf marker in pyproject.toml to avoid unknown-marker warnings and enable clean selector-based execution.
  • CI pipeline separation

    • Adds dedicated workflow: .github/workflows/perf.yml
      • Triggers: workflow_dispatch + schedule (weekly)
      • Uploads benchmark artifacts (artifacts/perf/...) for baseline retention/comparison
    • Updates existing workflows to exclude perf tests:
      • python-tests.yml: pytest -m "not perf" ...
      • python-publish.yml: pytest -m "not perf" ...
    • Keeps perf runs out of release/build/deploy flows.
  • Docs / contributor guidance

    • Updates CONTRIBUTING.md with local perf execution, loop tuning, and JSON output path usage.
# Local perf run (default deterministic settings)
python -m pytest -m perf tests/test_perf.py

# Higher-volume baseline run with explicit output
RFC3987_PERF_LOOPS=1000 \
RFC3987_PERF_WARMUP_LOOPS=1 \
RFC3987_PERF_OUTPUT_JSON=.pytest_perf/perf-results.json \
python -m pytest -m perf tests/test_perf.py

Copilot AI changed the title [WIP] Add deterministic performance benchmarking infrastructure for parser Add deterministic parser performance baseline suite with isolated CI workflow Jul 20, 2026
Copilot AI requested a review from jkowalleck July 20, 2026 18:47
@jkowalleck jkowalleck added the QA label Jul 21, 2026
@jkowalleck
jkowalleck requested a review from Copilot July 21, 2026 18:09

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a deterministic, fixture-backed performance baseline suite for the RFC3987 parser/validator APIs, and isolates it from the normal CI/release pipelines via a dedicated perf pytest marker and a separate scheduled workflow that publishes machine-readable artifacts.

Changes:

  • Introduces tests/test_perf.py to benchmark parser and validator entry points with deterministic loop controls, build/init vs runtime split, and JSON output (including tracemalloc peak memory).
  • Registers the perf pytest marker and updates standard CI/test + publish workflows to exclude perf tests (-m "not perf").
  • Adds a dedicated .github/workflows/perf.yml workflow (manual + weekly schedule) to run perf tests and upload benchmark artifacts; updates contributor docs and ignores local perf output directory.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tests/test_perf.py Adds deterministic perf benchmark test producing structured JSON metrics for parser/validator APIs.
pyproject.toml Registers the perf pytest marker.
CONTRIBUTING.md Documents how to run perf benchmarks locally and clarifies default test invocation excludes perf.
.gitignore Ignores .pytest_perf/ benchmark output directory.
.github/workflows/python-tests.yml Excludes perf tests from the main CI test runs.
.github/workflows/python-publish.yml Excludes perf tests from the publish pipeline’s test step.
.github/workflows/perf.yml Adds an isolated, scheduled/manual perf workflow that uploads benchmark artifacts.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/test_perf.py
Comment on lines +104 to +108
def _measure_make_validator_build_init(term: str, valid_sample: str) -> float:
start = time.perf_counter()
validator = sut.make_syntax_validator(term)
assert validator(valid_sample) is True
return time.perf_counter() - start
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants