Skip to content

Stop reuse self-tests posting fake CI notices - #1634

Open
Mikola Lysenko (mikolalysenko) wants to merge 1 commit into
mainfrom
ci-janitor/quiet-reuse-test-notices
Open

Mikola Lysenko (mikolalysenko) wants to merge 1 commit into
mainfrom
ci-janitor/quiet-reuse-test-notices

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Every lint-ecosystems run (PR, merge queue and main push) posts 10 fake notice annotations, for example on main's latest push run (check run 117423824414 at 7d3086f):

Could not verify merge-queue CI (TimeoutExpired); running all checks.
Could not compare with the previous PR head (CalledProcessError); running all checks.
… 10 in total

Nothing failed. The reuse scripts never run in that job. These lines come from the job's python3 -B -m unittest discover -s scripts/tests step: the fallback self-tests let ci-reuse-merge-group.py and ci-reuse-pr-head.py print their real ::notice:: workflow commands to stdout, and GitHub turns those lines into annotations. A local run of the suite prints 15 of them, and GitHub keeps 10 (its per-step cap).

The harm:

  • A reader can take the annotations to mean the merge-queue reuse check or the PR-head reuse check failed, which sends anyone debugging a CI-reuse decision down the wrong path.
  • They fill the step's notice cap, so a real notice from that step would be dropped.

CI is otherwise healthy this run: the merge queue is empty, nothing is queued or running, main is green, and the only failed run in 3 days (38806080761) was fixed by that PR's next push. So this falls under priority 4 (make CI more robust).

Root cause

test_api_failure_timeout_and_bad_payload_* in scripts/tests/test_ci_reuse_merge_group.py and scripts/tests/test_ci_reuse_pr_head.py call reusable_run on its error path without capturing stdout.

Fix

A small assert_falls_back helper in each test file. It runs reusable_run under contextlib.redirect_stdout, asserts that the result is None (unchanged), and also asserts that the ::notice:: fallback line was printed. The fallback is now checked, not just silenced. No production script or workflow changes.

Proof

  • python3 -B -m unittest discover -s scripts/tests: 347 tests, OK (6 skipped). Lines starting with :: in stdout went from 15 to 0.
  • Negative check: removing ::notice:: from ci-reuse-merge-group.py makes the new assertion fail 7 subtests.

Removed or moved tests

None. Every test still runs in the same place (lint-ecosystems).

https://claude.ai/code/session_01SeHhBxEvePf8ERTaTTenMK


Generated by Claude Code

The fallback tests for ci-reuse-merge-group.py and ci-reuse-pr-head.py
let the scripts print their real "::notice::Could not verify
merge-queue CI (...)" lines to stdout. lint-ecosystems runs those tests
in a GitHub job, so every run on PRs, the merge queue and main gained
ten fake annotations claiming the reuse check had failed. They fill the
step's ten-notice cap and send anyone debugging a reuse decision down
the wrong path.

Capture stdout in those cases and assert that the notice is printed.
The fallback path is now checked, not just silenced.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SeHhBxEvePf8ERTaTTenMK
@mikolalysenko Mikola Lysenko (mikolalysenko) added the ci-janitor Opened by the CI janitor routine (flakes, redundant tests, CI perf) label Oct 17, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 564c31a. Configure here.

@mikolalysenko Mikola Lysenko (mikolalysenko) added the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Oct 17, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Labeled Ready for review (burn-down agent).

  • Head: 564c31a
  • CI: 54/54 green (35 success, 19 skipped). No main-wide failures.
  • Bugbot: reviewed 564c31a, no findings. 0 open threads.
  • Mergeable with no conflicts. CHANGELOG.md is not touched.
  • Reviewer focus: only the two reuse self-tests under scripts/tests/ change; check they no longer emit real ::notice annotations.

Slack announcement not sent this run (the Slack send tool was unavailable). The next run will retry.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-janitor Opened by the CI janitor routine (flakes, redundant tests, CI perf) Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants