Repository navigation
Don't count a failing test run piped into tail as passed - #252
Merged
Merged
Conversation
`pytest -q | tail` exits with tail's 0 when tests fail. lcode read that as the checks passing, didn't run them, and reported them as passed. A run whose exit code isn't the check's now counts only if its output shows a test summary with nothing failing. Otherwise lcode runs the checks itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 6, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while reading the failed
build-cython-extattempts on 0.61.0's gate run. One attempt ran the project's checks itself aspython -m pytest -q 2>&1 | tail -40, while a test was failing. That command exits withtail's 0, so lcode took the checks as passed and never ran them.lcode -p --output jsonwould also report them as passed.What changes
When the model's run of the check has something after it that takes over the exit code (a pipe,
||,;,&or another line), the exit code no longer counts. The run counts as passed only if its output shows a test summary with nothing failing (18 passed, or unittest'sOK, and no1 failed,2 errors,FAILEDorERROR). Otherwise lcode runs the checks itself, as it does when the model hasn't run them.A run with only redirections or
&&after it (pytest -q 2>&1,pytest -q && echo done) keeps its exit code, so nothing changes there.Real test
A real model (qwen3.6-35b at 32K) on a small project with one failing test, asked to add a function and run the tests with exactly
python -m pytest -q 2>&1 | tail -5:In this test the model saw the failure in its own output and reported it either way. The fix matters when it doesn't, and for what lcode records: on 0.61.0 the run counted as checks that passed.
The new tests cover a failing run through
tailfrom start to finish withFakeOllama, and 20 command and output pairs. The full suite,ruff,basedpyrightandmkdocs build --strictpass.Left out on purpose
python -m pytest -q tests/test_a.pycontains the check's command, so it counts as the checks if it passes. Changing that would mean running a whole test suite again after many focused runs, which can take minutes on a big project, so it's a decision of its own.This changes when the checks run and what the model is told, so it goes through the next gate before it ships, with #250.
🤖 Generated with Claude Code