Skip to content

fix: keep follow-up reviews from repeating open findings - #64

Merged
frostming merged 2 commits into
mainfrom
fix/review-existing-threads
Oct 7, 2026
Merged

frostming merged 2 commits into
mainfrom
fix/review-existing-threads

Conversation

@frostming

@frostming frostming commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

After a new commit, Landing's automatic review posted inline comments that repeated findings already raised in open review threads. Each CI run starts from an empty SQLite database, so the agent has no model history of earlier reviews, and the review guidance never asked it to read the PR's existing threads.

The review guidance now has the agent read existing review threads, including resolved and outdated state, before publishing. It comments inline only on new findings, refers to still-applicable open findings from the review body by link, replies in a thread only when the candidate changes its conclusion or location, and keeps resolved threads settled unless new evidence changes them. The GitHub guide describes this behavior.

The long GitHub guidance constants are also wrapped with implicit string concatenation; their rendered text is byte-for-byte unchanged.

Validation

  • uv run ty check: passed.

  • uv run pre-commit run --all-files: passed.

  • uv run python -m pytest tests/test_github.py: 32 passed.

  • make docs-test: no issues.

  • Compared every *_GUIDANCE constant before and after wrapping: identical.

  • tests/test_mcp.py::test_cancellation_closes_mcp_subprocess failed once on ubuntu/3.12 with JSONDecodeError: the fixture MCP server creates its receipt before writing the record, so the test could read an empty file. The test now waits for the complete record; it passed in five consecutive local runs.

This changes prompt guidance only, so no test asserts its wording. Whether follow-up reviews actually stop repeating findings still needs observation on a real PR with an open Landing finding and a later commit.

AI assistance

Implemented with Claude Code (Claude Opus 5.5, claude-opus-5-5[1m]).

🤖 Generated with Claude Code

Each CI review starts from an empty database and the review guidance
never asked the agent to read the PR's existing review threads, so a
new commit received inline comments that repeated still-open findings.
Read existing threads before publishing, comment inline only on new
findings, refer to still-applicable ones by link, and keep resolved
threads settled unless new evidence changes them.

Also wrap the long GitHub guidance strings with implicit concatenation;
their rendered text is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking findings.

The guidance change stays inside the review path it describes: REVIEW_GUIDANCE renders only when no thread reply is required (expected_review requires not thread), so the new "reply in that thread only when…" sentence cannot override the confirmed-reply contract for thread follow-ups, and the existing-thread step is reachable with the prepared gh CLI. Comparing every *_GUIDANCE constant between e4fc599 and c6bae1b shows only REVIEW_GUIDANCE changed, so the wrapping claim holds.

The failing native leg is unrelated to this candidate. tests-and-type-check (ubuntu-latest, 3.12) failed in tests/test_mcp.py::test_cancellation_closes_mcp_subprocess: the test waits for the receipt file to exist and then parses it (tests/test_mcp.py:172-174), while the fixture server creates that file with open("a") before writing its JSON (tests/test_mcp.py:37-40), so an empty read is possible on the base revision too. This candidate changes two prompt-string files and one documentation sentence, does not touch that test or MCP execution, and the other three matrix legs plus a local five-run replay of the single test passed. Treat it as a separate follow-up if it recurs.

The MCP fixture server opens the receipt before writing its record, so
the cancellation test could see the file while it was still empty and
fail with JSONDecodeError. Wait until the record's line is complete.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking findings.

The only change since the previous review at c6bae1b is the receipt wait in tests/test_mcp.py, which repairs the flake that pass reported as a separate follow-up. tests-and-type-check passes on all four legs of run 37561856275 covering this head, and the single test passed on the merge checkout here. Waiting for a line terminator, rather than file creation, still requires a complete record and leaves the cancellation and process-exit assertions intact.

The review guidance added in c6bae1b is byte-identical at this head, so the earlier verification of its rendered text and of the review-only delivery path still applies. Whether the new instructions reduce repeated findings is only observable on a later review of a PR with an open Landing thread; that is not a blocking acceptance condition.

@frostming
frostming merged commit 8d2610c into main Oct 7, 2026
9 checks passed
@frostming
frostming deleted the fix/review-existing-threads branch October 7, 2026 02:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant