gh-158895: Teach libregrtest to output github job summaries by cmaloney · Pull Request #158896 · python/cpython

cmaloney · 2026-10-06T05:42:11Z

Update libregrtest to output GitHub Job Summary and Error Annotations for each failing job, test when run in Github Actions. Screenshots of the changes below.

Job Summaries: For each job which fails output a markdown file with details. Auto-expanded when there are few, collapsed when there are many. Only failures to reduce noise (there are a lot of jobs run)
Error Annotations: When a test fails emit a marker so Github can link the specific diff line to the job failure.
Introduce a "Reporter" concept to libregrtest which Github is one instance of to keep the integration from leaking around. Buildbot (calling github API), JUnit XML, etc. could be additional reporters in time.

This creates a number of UI adjustments which should make it a little easier to get around. These only happen if run in Github. Goal is to make triage a little bit simpler/faster but keep the full fidelity of the existing libregrtest output.

note: iOS and Android are excluded for now, as are other kinds of jobs like docs and mypy. Can expand support to those but would rather do as separate well scoped PRs.

Job Summaries

If you click on "checks" tab on a PR then the "tests" you get taken to the summary page (ex. https://github.com/python/cpython/actions/runs/38330186853). At the top is the workflow graph, if you scroll past that you see a job summary per failed job. If there are many failures they are "collapsed". Just a few will start expanded. The tracebacks are colored with Python traceback syntax highlighting.

Error Annotations

If you click on "Files Changed" in a PR there is a button to "Open Alerts Panel". This will show all the error annotations on all the files. Individual test failures are mapped back to the source line when possible, and that line in the Diff view will show an indicator that there is an error which can be expanded. Clicking on "show details" will take you to the specific job logs.

On individual test jobs all the annotations from that job are included at the top of the job page.

Unfortunately this does result in redundant printing of the assertion failures as we print both the libregrtest form and a form of the github annotation line is presented. I chose for this to be just after the existing failure message which I think reads reasonably.

edit: Rewrote with screenshots to show features.

Issue: Improve regrtest integration with Github UI for test failures #158895

Goal is to make it faster to triage individual test failures while keeping the full existing regrtest fidelity. Fairly minimal integration to hopefully save some human eyeball time. Setup so only the test summary process should output the summary, subprocesses have the env var which will cause output unset.

hugovk · 2026-10-06T05:50:12Z

Can you temporarily trigger a test failure so we can see it in action?

cmaloney · 2026-10-06T05:58:03Z

Yea, planning to pass + fail + iterate on failure to get it pretty then get back to passing and mark as ready for review :)

…rmatting code some

cmaloney · 2026-10-15T06:09:36Z

I think this is ready for review now. I updated the PR comment with full details.

My hope is in the longer term the Reporter could be used if we, for instance, want buildbot jobs to annotate the github commit (or if moving places other than github). With this PR there is the text, JUNIT_XML, and GitHub outputs. I'd like Reporter to work for JUNIT_XML but moving it is a separate project.

edvilme · 2026-10-15T14:25:43Z

+class Reporter:
+    """Report test failures, for example to a CI service. A no-op here."""
+
+    def run_started(self, runtests: RunTests) -> None:


Suggested change

def run_started(self, runtests: RunTests) -> None:

def test_started(self, runtests: RunTests) -> None:

Maybe, for parity with the other methods?

The "run" concept is different than a "test" or "test suite" hence the different name. Parallels RunTests vs. a TestResult

edvilme · 2026-10-15T14:32:03Z

+        if not failed:
+            return
+        ncases = sum(len(result.failed_test_cases) for result in failed)
+        blocks = [f"## {decolor(state)}: "


Suggested change

blocks = [f"## {decolor(state)}: "

blocks = [f"## 🛑 {decolor(state)}: "

Maybe it makes it a little easier to parse vs just markdown text?
Maybe also lowercase the state? I think it is all caps currently

If we output success as well as failure definitely, this only outputs failures to github summaries. I tried outputting both success and failures but we run 30+ test jobs and that just made a lot of summaries that needed to be browsed past and weren't particularly useful.

edvilme · 2026-10-15T14:44:34Z

+            filename, line = frames[0].groups()
+            exception = traceback[frames[-1].end():]
+        # Prefer the last frame in a test file, the assertion
+        for frame in frames:


Maybe I'm wrong but will this get executed only when the above comparison is True?

Correct.In general this tries to find a filename from the traceback but if we fail that's okay (it shouldn't block the test run from finishing). The if frames translates to "if the length of the list of frames is > 1".

The loop then finds the last frame with a name that starts with "test_" and uses that for the annotation. Definitely not a perfect metric but seems to work reasonably for getting back to the test filename.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

zware

I'm all for it :)

I have not reviewed every line in detail, but I have reviewed enough to know that it does what it says on the tin and I don't see any obvious simplifications.

My only concern is that I think one of the new tests is triggering false-positives in the summary; looking at the summary for this PR (which is passing all tests) includes quite a few entries. Unless those have built up over the lifetime of the PR (which I don't think is how it works?), there may be some cleanup to be done. However, I'd be ok with leaving that to a followup.

This reverts commit 81d4828. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cmaloney · 2026-10-23T18:05:03Z

I think the mixed was just that I added errors for giving a lightning talk while you were reviewing which were reporting in. Just removed those failures so we'll find out once CI finishes :)

ZeroIntensity

The demo was convincing. I really really want this :)

test_run_failed_module_live runs a nested regrtest that is expected to crash; unset GITHUB_STEP_SUMMARY so it doesn't land in the CI summary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

zware · 2026-10-23T18:29:52Z

I think the mixed was just that I added errors for giving a lightning talk while you were reviewing which were reporting in. Just removed those failures so we'll find out once CI finishes :)

Turns out it wasn't a new test, it was the old one that you found and fixed :)

The "Verify excluded tests still need exclusion" step expects each excluded test to fail; unset GITHUB_STEP_SUMMARY so those failures aren't annotated or added to the job summary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Lets a workflow or test turn off the GitHub reporter by setting the variable to an empty string instead of failing to open ''. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

bedevere-app Bot added the type-feature A feature request or enhancement label Oct 6, 2026

bedevere-app Bot mentioned this pull request Oct 6, 2026

Improve regrtest integration with Github UI for test failures #158895

Open

cmaloney added 2 commits October 5, 2026 23:09

Only output errors, too many jobs. Try coloring a bit more

8fcfa5d

Fail CI on linux for experimenting with output formatting

e6e09bd

edvilme reviewed Oct 6, 2026

View reviewed changes

Comment thread Lib/test/libregrtest/main.py Outdated

cmaloney added 4 commits October 5, 2026 23:42

Reformat failures to be failure first, easier visibility. Simplify fo…

54d123f

…rmatting code some

add error annotations

3468742

Add groups more of the time

ff7b75a

drop groups, error only on final failure

946c801

cmaloney changed the title ~~gh-158895: Teach librertest to output github job summaries~~ gh-158895: Teach libregrtest to output github job summaries Oct 6, 2026

cmaloney added 17 commits October 6, 2026 10:27

More structured error blocks

14394da

Fail more tests

6359876

Adjust and simplify a little

041e577

Annotate inline with test output

ba449b4

Move annotations before test output

59b1608

Have regrtest workers print annotations in better place

75c44b0

Full annotations from worker

1222c34

Remove redundant printing

287df00

Remove special summary case, more full output everywhere

53024ed

Better message in the annotations

c2d1d79

Tweak message

c3cca61

Move to simpler regex traceback info extraction

b125f6f

lots of review changes

3f320dd

Add comment

9b1ce48

Annotations after failure, better test_regrtest coverage

b561060

annotate env changed better. pgo runs

e506d7d

simplify, rely on env var everywhere. Stop using findspec

2ca705b

cmaloney added 7 commits October 12, 2026 21:52

Fix mypy

01ad16c

Refactor to use a reporter of which github is one

475549a

Personal review fixes

dc83a3c

Simplify, generlize reporter

589e6a7

Simplify tests, reduce comments

24e7820

Personal review fixes

6d2f432

Fix intentionally broken tests

f8189f7

cmaloney marked this pull request as ready for review October 15, 2026 06:07

bedevere-app Bot added the awaiting review label Oct 15, 2026

edvilme reviewed Oct 15, 2026

View reviewed changes

Re-add intentional failures for testing GitHub Actions output

81d4828

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

zware approved these changes Oct 23, 2026

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting review labels Oct 23, 2026

Remove intentional test failures used to test GitHub Actions output

116d8d3

This reverts commit 81d4828. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ZeroIntensity approved these changes Oct 23, 2026

View reviewed changes

Don't report test_asdasd crash to the GitHub job summary

eb988ce

test_run_failed_module_live runs a nested regrtest that is expected to crash; unset GITHUB_STEP_SUMMARY so it doesn't land in the CI summary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cmaloney requested a review from pablogsal as a code owner October 23, 2026 18:18

cmaloney and others added 2 commits October 23, 2026 11:36

Treat an empty GITHUB_STEP_SUMMARY as disabled

f900806

Lets a workflow or test turn off the GitHub reporter by setting the variable to an empty string instead of failing to open ''. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cmaloney requested review from DinoV and Yhg1s as code owners October 23, 2026 18:38

zware enabled auto-merge (squash) October 23, 2026 19:44

zware merged commit 7715c92 into python:main Oct 23, 2026
106 of 108 checks passed

bedevere-app Bot removed the awaiting merge label Oct 23, 2026

cmaloney deleted the regrtest-gha-summary branch October 23, 2026 20:12

JacobCoffee mentioned this pull request Oct 23, 2026

Add pytest plugin for gha annotations pypi/warehouse#20799

Draft

	def run_started(self, runtests: RunTests) -> None:
	def test_started(self, runtests: RunTests) -> None:

	blocks = [f"## {decolor(state)}: "
	blocks = [f"## 🛑 {decolor(state)}: "

Conversation

cmaloney commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Job Summaries

Error Annotations

Uh oh!

hugovk commented Oct 6, 2026

Uh oh!

cmaloney commented Oct 6, 2026

Uh oh!

Uh oh!

cmaloney commented Oct 15, 2026

Uh oh!

edvilme Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

cmaloney Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edvilme Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

cmaloney Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

edvilme Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

cmaloney Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zware left a comment

Choose a reason for hiding this comment

Uh oh!

cmaloney commented Oct 23, 2026

Uh oh!

ZeroIntensity left a comment

Choose a reason for hiding this comment

Uh oh!

zware commented Oct 23, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

cmaloney commented Oct 6, 2026 •

edited

Loading

cmaloney Oct 15, 2026 •

edited

Loading

cmaloney Oct 15, 2026 •

edited

Loading