Add deterministic parser performance benchmarks and CI reporting by Copilot · Pull Request #37 · python-jsonschema/rfc3987-syntax2

Copilot · 2026-07-17T09:24:19Z

This adds a dedicated performance suite for the RFC 3987 parser helpers so parser construction cost, parse throughput, and memory usage can be compared over time. The benchmarks are deterministic, separately runnable, and wired into CI as report-only smoke/full jobs with machine-readable output.

Benchmark suite
- Adds tests/test_perf.py with a separate perf marker
- Measures build/init cost independently from parse cost
- Reports per-parser, per-dataset metrics for:
  - ms/op
  - ops/s
  - peak allocated memory
- Covers the public parser entry points explicitly:
  - shared parser APIs: parse, is_valid_syntax
  - term-specific validators: is_valid_syntax_*
Deterministic corpus + reporting
- Uses fixed average-case corpora for small / medium / large runs
- Captures runtime environment in JSON output:
  - Python version
  - platform
  - lark version
- Supports optional baseline comparison via prior JSON results and prints deltas in the summary table
Pytest configuration
- Registers the perf marker in pyproject.toml
- Keeps default test runs on the functional suite with -m "not perf"
CI strategy
- PRs run a lightweight perf smoke job and upload JSON artifacts
- scheduled/manual runs execute the full perf suite and upload results for comparison
- keeps perf report-only; no regression threshold enforcement is introduced yet

Example benchmark invocation:

RFC3987_PERF_MODE=full \
RFC3987_PERF_RESULTS_JSON=perf-results/current.json \
RFC3987_PERF_BASELINE_JSON=perf-results/baseline.json \
python -m pytest -m perf tests/test_perf.py -s

…-15 per rule - Create .github/workflows/python-perf.yml with perf-smoke (PRs) and perf-full (push to main/next, schedule, manual) jobs - Remove perf-smoke and perf-full jobs from python-tests.yml - Add -m "not perf" to python-publish.yml quick-test step - Expand RULE_SAMPLES in tests/test_perf.py from 2 to 10-15 valid samples per rule (406 total, all validated)

…nce of fragment

jkowalleck · 2026-07-18T09:18:32Z

 jobs:
  perf-smoke:
-    name: Performance Smoke (py${{ env.PYTHON_VERSION_DEFAULT }})
+    name: Performance Smoke (py3.14)


@copilot dont prnit my version here ...

Updated in commit 2e1d1c7: the job name no longer prints the Python version.

jkowalleck · 2026-07-18T09:18:35Z

          path: perf-results/
  perf-full:
-    name: Performance Full (py${{ env.PYTHON_VERSION_DEFAULT }})
+    name: Performance Full (py3.14)


@copilot dont prnit my version here ...

Updated in commit 2e1d1c7: the job name no longer prints the Python version.

jkowalleck · 2026-07-18T09:19:49Z

+  pull_request:
+  push:
+    branches: [ 'main', 'next' ]
+  schedule:


@copilot no scheduling needed

Updated in commit 2e1d1c7: removed the schedule trigger from this workflow.

jkowalleck · 2026-07-18T09:20:22Z

+          python -m pytest -m perf tests/test_perf.py -s
+      - name: Upload perf smoke results
+        if: always()
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02  # v4


@copilot have the version vomment the full version matching the git node id

Updated in commit 2e1d1c7: changed the pinned action version comment to the full matching tag v4.6.2 for this hash.

jkowalleck · 2026-07-18T09:21:00Z

+          mkdir -p perf-results
+          python -m pytest -m perf tests/test_perf.py -s
+      - name: Upload perf smoke results
+        if: always()


@copilot add a comemnt linking to the sction
ala # see https://github.com/actions/setup-python

Updated in commit 2e1d1c7: added a see comment linking to the actions/upload-artifact action.

jkowalleck · 2026-07-18T09:21:34Z

+          python -m pytest -m perf tests/test_perf.py -s
+      - name: Upload full perf results
+        if: always()
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02  # v4


@copilot add a comemnt linking to the sction
ala # see https://github.com/actions/setup-python

Updated in commit 2e1d1c7: added a see comment linking to the actions/upload-artifact action.

jkowalleck · 2026-07-18T09:21:46Z

+          python -m pytest -m perf tests/test_perf.py -s
+      - name: Upload full perf results
+        if: always()
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02  # v4


@copilot have the version vomment the full version matching the git node id

Updated in commit 2e1d1c7: changed the pinned action version comment to the full matching tag v4.6.2 for this hash.

jkowalleck · 2026-07-18T09:22:12Z

+          path: perf-results/
+  perf-full:
+    name: Performance Full (py3.14)
+    if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || github.event_name == 'push'


@copilot the condition is not needed

Updated in commit 2e1d1c7: removed the unnecessary job condition.

Initial plan

2d836e4

Copilot AI assigned Copilot and jkowalleck Jul 17, 2026

Copilot started work on behalf of jkowalleck July 17, 2026 09:24

View session

Copilot AI linked an issue Jul 17, 2026 that may be closed by this pull request

tests: add performance tests #36

Open

10 tasks

Add reproducible performance benchmarks

9a625a4

Copilot AI changed the title ~~[WIP] Add performance tests for all parsers~~ Add deterministic parser performance benchmarks and CI reporting Jul 17, 2026

Copilot AI requested a review from jkowalleck July 17, 2026 09:43

Copilot finished work on behalf of jkowalleck July 17, 2026 09:43

Copilot started work on behalf of jkowalleck July 17, 2026 13:48

View session

Copilot AI added 2 commits July 17, 2026 14:00

fix: replace double-@ iri sample with single-@ variant

11453b3

Copilot finished work on behalf of jkowalleck July 17, 2026 14:01

Copilot started work on behalf of jkowalleck July 17, 2026 14:04

View session

Copilot AI added 3 commits July 17, 2026 14:15

feat: incorporate JSON test data into perf tests and add new test cases

949f73b

feat: incorporate JSON test data into perf tests and add new test cases

dd5a496

clarify: add reason to absolute_iri entry explaining intentional abse…

219b491

…nce of fragment

Copilot finished work on behalf of jkowalleck July 17, 2026 14:20

Copilot started work on behalf of jkowalleck July 17, 2026 17:25

View session

Pin actions/upload-artifact to hash in python-perf.yml

a24baa1

Copilot finished work on behalf of jkowalleck July 17, 2026 17:26

Copilot started work on behalf of jkowalleck July 18, 2026 06:45

View session

Fix invalid env context in python-perf job names

01e82f1

Copilot finished work on behalf of jkowalleck July 18, 2026 07:00

jkowalleck requested changes Jul 18, 2026

View reviewed changes

Copilot started work on behalf of jkowalleck July 18, 2026 09:23

View session

jkowalleck added the QA label Jul 18, 2026

Apply PR feedback to python-perf workflow

2e1d1c7

Copilot finished work on behalf of jkowalleck July 18, 2026 09:25

Copilot AI requested a review from jkowalleck July 18, 2026 09:25

Copilot started work on behalf of jkowalleck July 18, 2026 09:33

View session

Remove smoke perf mode; keep single full perf CI job

c1d33a0

Copilot finished work on behalf of jkowalleck July 18, 2026 09:36

Conversation

Copilot AI commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented Jul 17, 2026 •

edited

Loading