Skip to content

Fix benchmark CI with compatible released pytest versions - #312

Merged
Nikhil Navakiran (nikhilNava) merged 2 commits into
microsoft:mainfrom
nikhilNava:copilot/fix-benchmark-pytest-compat
Oct 28, 2026
Merged

Nikhil Navakiran (nikhilNava) merged 2 commits into
microsoft:mainfrom
nikhilNava:copilot/fix-benchmark-pytest-compat

Conversation

@nikhilNava

@nikhilNava Nikhil Navakiran (nikhilNava) commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Affected PR: Reuse Azure Monitor offline storage for Agent365 #300 (failing benchmark job).
  • Pin the Performance workflow to released pytest==9.1.1 and pytest-benchmark==5.3.0 in both candidate and baseline installation steps.
  • Avoid pytest 9.2's removal of the private _nodeid attribute still accessed by pytest-benchmark 5.3.0. Without these pins, all four benchmarks fail during fixture setup before any performance measurements run.
  • Keep runtime dependencies, SDK code, and benchmark behavior unchanged. No unmerged upstream code or monkey patches are used.

This is independent of #300 and is based directly on main.

Validation

  • Ran all four existing performance benchmarks with the exact released pins: 4 passed; benchmark JSON generated successfully.
  • Local runtime: Windows, Python 3.14. The Performance CI workflow also passed on Ubuntu/Python 3.12: candidate benchmarks, baseline benchmarks, and the performance comparison all succeeded.
  • git diff --check passed.
  • Existing OpenTelemetry deprecation warnings and localhost exporter retry messages remain unchanged.

Pin pytest 9.1.1 and pytest-benchmark 5.3.0 for both candidate and baseline runs. Pytest 9.2 removes the private _nodeid attribute used by the released benchmark plugin, preventing all benchmark fixtures from initializing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 14661e89-86a3-4afe-a454-263eaeeba021
@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Performance comparison

Threshold: regressions >15.0% on gating scenarios fail the build. Higher ops/s is better; positive Δ means the PR is slower.

Scenario Gating Baseline (ops/s) Candidate (ops/s) Δ % Status
azure_monitor_log yes 21,928.9 21,934.2 -0.02% ✅
azure_monitor_span yes 164,203.6 162,022.0 +1.35% ✅
otel_log no 25,217.5 25,338.9 -0.48% ✅
otel_span no 47,180.9 47,580.5 -0.84% ✅

@nikhilNava Nikhil Navakiran (nikhilNava) added the skip-changelog Not required for changes to test files, github actions, etc. label Oct 28, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the performance-workflow dependency pins, though tests were not run during review and Ubuntu/Python 3.12 validation remains for CI.

No open issues 🎉

What changed in this PR

Pins the performance workflow’s test dependencies to avoid the reported pytest compatibility failure, without changing SDK runtime dependencies or benchmark behavior.

Changes:

  • Uses pytest==9.1.1 and pytest-benchmark==5.3.0 for both candidate and baseline runs.
  • Keeps both sides of the benchmark comparison on the same test-tool versions.

No actionable findings. Tests were not run during this review; Ubuntu/Python 3.12 validation remains for CI.

File Description
.github/​workflows/​performance.yml Pins pytest and pytest-benchmark in both installation steps.

Review effort: Balanced

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 14661e89-86a3-4afe-a454-263eaeeba021
@nikhilNava
Nikhil Navakiran (nikhilNava) merged commit 6009595 into microsoft:main Oct 28, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-changelog Not required for changes to test files, github actions, etc.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants