Repository navigation
feat(review): quest review records each round in the PR body #508
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
6b6713d
feat(review): quest review records each round in the PR body
kixelated 812522b
fix(review): bound each round's reply schema to the last round's find…
kixelated 9f33ced
fix(review): a record with open findings names no full SHA
kixelated 06265bb
test(review): narrow the PR body in the record test
kixelated 175b628
fix(review): find the Review section as Markdown parses it
kixelated 47f66bc
fix(review): refuse a record that a body's unclosed construct would s…
kixelated File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,95 @@ | ||
| // `quest review` rewrites the `## Review` section of its branch's open PR after each round, so the head | ||
| // it reviewed counts without an agent copying the round into the body by hand. | ||
| import { reviewSection } from "../core"; | ||
| import { type ClaimDependencies, repositoryPath, upstream } from "./claims"; | ||
| import { git } from "./git"; | ||
| import { requireRemote, string } from "./github"; | ||
| import { type Round, located, parseBase } from "./review"; | ||
|
|
||
| const HARNESSES: Record<Round["reviewer"], string> = { claude: "Claude Code", codex: "Codex" }; | ||
|
|
||
| /** | ||
| * Model text as one inline code span, so a mention, issue reference, or Markdown in it renders as text. | ||
| * The land gate counts every full SHA the section names, so no full SHA appears in it. | ||
| */ | ||
| function code(text: string): string { | ||
| const flat = text | ||
| .replace(/\s+/g, " ") | ||
| .trim() | ||
| .replace(/\b[0-9a-f]{40}\b/g, (sha) => sha.slice(0, 12)); | ||
| const fence = "`".repeat(Math.max(0, ...(flat.match(/`+/g) ?? []).map((run) => run.length)) + 1); | ||
| const pad = /^`|`$/.test(flat) ? " " : ""; | ||
| return `${fence}${pad}${flat}${pad}${fence}`; | ||
| } | ||
|
|
||
| const capitalized = (word: string) => word[0]!.toUpperCase() + word.slice(1); | ||
|
|
||
| /** The `## Review` section's text for `round`, without its heading. */ | ||
| export function reviewRecord(round: Round): string { | ||
| const lines = [ | ||
| "Rewritten by `quest review` after each round.", | ||
| "", | ||
| `- Reviewer: ${HARNESSES[round.reviewer]}, ${round.model}, ${round.effort} effort`, | ||
| `- Session: ${code(round.session)}, round ${round.round}`, | ||
| // The land gate counts a full SHA here as a review of that head, which an open finding has not passed. | ||
| `- Head: ${round.findings.length === 0 ? round.head : `${round.head.slice(0, 12)}, named in full once a round reports no findings`}`, | ||
| `- Base: ${round.base.ref} at ${round.base.sha.slice(0, 12)}`, | ||
| `- Score: ${round.score}/10, ${code(round.rationale)}`, | ||
| ]; | ||
| if (round.findings.length === 0) lines.push("- Findings: none"); | ||
| else lines.push("- Findings, open until the next round settles them:"); | ||
| for (const finding of round.findings) | ||
| lines.push(` - ${capitalized(finding.severity)}, ${code(located(finding))}: ${code(finding.title)}`); | ||
| const settled = round.earlier?.findings ?? []; | ||
| if (settled.length > 0) lines.push(`- The last round's findings, at ${round.earlier!.head.slice(0, 12)}:`); | ||
| for (const finding of settled) | ||
| lines.push( | ||
| ` - ${capitalized(finding.disposition)}: ${finding.severity}, ${code(located(finding))}: ${code(finding.title)}; ${code(finding.reason)}`, | ||
| ); | ||
| return lines.join("\n"); | ||
| } | ||
|
|
||
| /** `body` with `record` as its `## Review` section, which it gains when it has none; the rest stays as written. */ | ||
| export function withReview(body: string, record: string): string { | ||
| const section = reviewSection(body); | ||
| if (!section) { | ||
| const appended = `${body.trimEnd()}${body.trim() ? "\n\n" : ""}## Review\n\n${record}\n`; | ||
| // A body that ends inside an unclosed code fence or HTML comment would swallow the heading. | ||
| if (reviewSection(appended) === null) | ||
| throw new Error( | ||
| "the PR body ends inside an unclosed code fence or HTML comment; close it so the record can follow", | ||
| ); | ||
| return appended; | ||
| } | ||
| const { start, end } = section; | ||
| return `${body.slice(0, start)}\n\n${record}\n${end < body.length ? "\n" : ""}${body.slice(end)}`; | ||
| } | ||
|
|
||
| /** Where a round's record went: the PR whose body holds it, or why no body changed. */ | ||
| export type Recorded = { url: string } | { skipped: string }; | ||
|
|
||
| /** | ||
| * Record `round` in the `## Review` section of the open PR from its branch into its base, found among the | ||
| * upstream's PRs from the repository the base's remote pushes to, as claims find a branch's PR. The edit | ||
| * is the token's user's, which the land gate counts only when that user wrote the PR. | ||
| */ | ||
| export async function recordRound(root: string, round: Round, deps: ClaimDependencies): Promise<Recorded> { | ||
| // The checkout's top level, as the round reviewed it: a subdirectory would look for the quest tree below it. | ||
| const top = git(root, ["rev-parse", "--show-toplevel"]); | ||
| const { remote, branch: base } = parseBase(top, round.base.ref); | ||
| const up = await upstream(top, requireRemote(top, remote), deps); | ||
| if (up.github === null) return { skipped: `${remote} is not on GitHub` }; | ||
| const { client, repository, head } = up.github; | ||
| const from = `${head.owner}:${round.branch}`; | ||
| const query = `state=open&head=${encodeURIComponent(from)}&base=${encodeURIComponent(base)}&per_page=100`; | ||
| const [pull] = await client.all(`${repositoryPath(repository)}/pulls?${query}`); | ||
| if (pull === undefined) return { skipped: `no open PR from ${from} into ${base}` }; | ||
| const body = typeof pull.body === "string" ? pull.body : ""; | ||
| const next = withReview(body, reviewRecord(round)); | ||
| if (next !== body) | ||
| await client.request(`${repositoryPath(repository)}/pulls/${Number(pull.number)}`, { | ||
| method: "PATCH", | ||
| body: { body: next }, | ||
| }); | ||
| return { url: string(pull.html_url) }; | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fix the guidance on where to add context to the
## Reviewsection.The text says to add context "only after the record". The text gives the next-round rewrite as the reason.
withReviewreplaces the whole section, from after the heading to the next level-1 or level-2 heading. Text added after the record inside the section is deleted on the next round.An agent that follows this line loses its context. Tell agents to put context outside the
## Reviewsection, for example in another section or a PR comment. Only a record written by hand afterrecord skippedbelongs inside the section.