[nasser1941]

Why

On the 0.60.0 dev slice (32K, 75 attempts), all 34 automatic checks failed, and 28 of them told the model "your changes are done, but the project's checks fail… fix what your changes broke". The question was how to stop misleading the model on projects whose checks already fail, without losing the check on small projects where it's useful. The analysis needed no model run: its numbers come from the traces, checked against the three earlier full nights. Real-model tests are under How it was tested.

What the 34 checks were

What happened	Checks	Each took	Attempts that edited after the note
Runner missing or no tests: already not sent to the model	6	under 0.2 s	none
No test ran: Django, a unittest traceback with no Ran N test line	13	0.0–0.3 s	1 of 12
No test ran: pytest stopped while collecting, exit 2 (build-cython-ext, pytest's own repository)	6	0.7–1.4 s	1 of 6
Tests ran and some failed (matplotlib, xarray)	8	263–364 s	0 of 8
Killed at the 600 s limit (sympy)	1	600 s	0 of 1

After the "no test ran" notes the model spent 26 minutes (154 model calls), used git stash in 5 attempts to see for itself, and in 2 attempts kept editing. One downloaded a 244 MB database and added a skip to a test file the request had excluded.

Over four nights it's 48 of 103 checks with no test run, and in those 47 attempts only 3 edited anything after the note. The 28 checks where tests ran and failed led to no edits at all, and within a night every attempt at a task fails the same tests. So that note already works, and contained no regression.

What each option would have done

Option	No test ran (19)	Tests ran, some failed (8)	Added time on these checks	Cost on a small project
Run the check before the first change	16; not build-cython-ext, whose project doesn't exist until the model clones it	8, comparing failures by test name	+43 min on 8 attempts (+39% for them), to save at most the 8 min after those notes	Every request that changes files pays one run, passing or not: 0.02–0.23 s on 3 tests, about 3 min on lcode's own suite. In ask mode, an approval prompt before the first edit
Read the output: no test ran (this PR)	19	Left as they are	None, and up to 26 min saved	Nothing when tests ran: a regex over the output. When a run stops early, one look at the checkpoint store: 0.09 s on Django's 6,120 files
After a quick failure, put the old files back and run again	16	Skipped as too slow	Seconds	Seconds on a failure only, but it swaps the model's work out of the user's tree while it runs

What changes

• A run that stops before its first test isn't sent to the model (pytest exit 2–4 with no N passed/failed; unittest exit 1 with a traceback and no Ran N test). You get the reason and what to run. If the output names a file the request changed, shell commands included, it still goes to the model: that's what the check is for.
• Checks run from the project's root, not wherever the model's last cd left the shell. 3 of the 34 ran from tests/. Without this a fine project would "not start" after a cd tests. A command from your own checks setting runs in the folder you started lcode in.
• A check that hits the time limit isn't called a failure. The model is told that it doesn't say whether the change works, which still catches a hang the change caused.
• /status, the task sheet, the review prompt and lcode -p say "didn't run" for these instead of "failed (exit code 1)", also for no tests or no test runner. Traces keep the real exit code.

Notes for tests that ran and failed are unchanged, and so is the system prompt.

The cost, plainly

Removing the notes can lose a pass now and then. build-cython-ext__x9ttQdi had stopped with 1 failed, 17 passed on a test it called pre-existing, and the verifier needs that test to pass. The misleading note got it going again, and among the out-of-scope work it fixed that test. Without the note, that attempt most likely scores 0. That's 1 attempt in 47, on 1 night in 4. django-14792__yvwDw7W also ran its new test only because of the note and fixed it, with no effect on its reward.

Expect about 26 minutes less per night, and now and then one attempt fewer. The gate can't resolve a difference that small.

How it was tested

Real model (qwen3.6-35b at 32K) on this branch merged with main, which includes 0.61.0 and #252.

Harbor, one attempt each, on tasks where a run stops before its first test. All three reached the check. In each, the "stopped before running a test" message appeared, no note reached the model, and the request ended with the model's answer, with no model call after the check.

Attempt	What stopped the run	Notes to the model	Took	Verifier
django__django-12774	ImproperlyConfigured (unittest, no Ran N tests)	0	412 s	0: its fix also accepts multi-field constraints, and 2 PASS_TO_PASS tests fail
pytest-dev__pytest-10051	No module named 'xmlschema' (pytest exit 2)	0	211 s	0: it misses test_clear_for_call_stage; this task failed 4 of 5 attempts on the last three runs
build-cython-ext	Could not find a knots database file (pytest exit 2)	0	1476 s	1: it fixed KeyError: 'pos' itself, with no note

Speed limited this test, so it says nothing about scores or time; that's for the gate. The machine was short of RAM in the daytime. build-cython-ext needs about 10 minutes at healthy speed, so the attempt above ran at 43 tokens/s with its limit doubled (harbor run --agent-timeout-multiplier 2). Four more attempts didn't reach a check and say nothing about this change: one lost its reply stream when lcode update restarted Ollama, two ran out of build-cython-ext's 15 minutes at 14 and 46 tokens/s, and the repeat guard stopped one that kept running runtests.py from the wrong folder.

A real terminal with the real model, on three small projects in ask mode:

• cd tests first: with the status bar on tests/, .venv/bin/python -m pytest -q and python3 -m unittest discover -s tests ran from the root and passed, and the shell stayed in tests/. Neither command starts from inside tests/.
• A change that really breaks the project reached the model: a rename, where pytest stops while collecting and the error names calc.py, and a - b, where unittest reports FAILED (failures=2). The model fixed both, and the checks then passed.
• A project already stuck on a module that isn't installed: the warning appeared, the model's answer stood, and /status said didn't run. A rename in that same project still reached the model, because the error names calc.py.

These tests found one bug in this PR, now fixed: the message cut Django's error at 160 characters (DJANGO_SETTIN….), and a reason ending in a full stop got a second one (download_database()..). It now keeps 240 characters and one full stop.

Also:

• Tests with FakeOllama and make_agent, using real pytest and unittest runs. A syntax error, a new missing import, a removed name, a broken new test, a broken conftest.py, the same edit by shell command, and rm of a module all still reach the model.
• The real code run over all 103 recorded checks: 48 classified as no test ran, none naming a changed file. On the two newest runs (0.61.0): 19 checks, 8 no test ran, none naming a changed file.
• A scripted model on real Django, in the task's image, no network: a harmless change gives no note, the same after cd tests, and breaking django/conf/__init__.py gets a note.

This changes what the model is told, so it still goes through a gate night.

Left out on purpose

• A second look when the model's own last test run failed. It would have fired on 3 of the 47 attempts, and the other two got a note and changed nothing.
• Outside git, the "root" is still the current folder, so the check still follows the model's cd. That's an existing bug, not made worse.
• Slow suites are the bigger cost, at 53 of the night's 484 minutes, and this doesn't touch them.

Found while testing, not changed here

• A broken reply stream crashes lcode. chat_stream turns connection errors into OllamaError only when opening the request, so ChunkedEncodingError ends both lcode and lcode -p in a traceback. Restarting Ollama during a reply is enough.
• A narrowed run counts as the whole check. .venv/bin/python -m pytest -q tests/test_calc.py passing made /status say passed on the stuck project, and lcode skipped its own run (covers only looks for the check's text). That's 1 of 137 recorded attempts and 1 of the 4 terminal sessions, always after the model had seen the exact command.
• After a real failure, the model edited the tests in 2 of 3 terminal sessions although the request said not to. The note says "Fix what your changes broke" and doesn't mention the request's limits.

🤖 Generated with Claude Code