Runners publish the App's quest review check

cloudflare/GITHUB.md

@@ -14,7 +14,7 @@ and installation tokens stay inside the Worker; agents receive neither.
 | Contents: write       | Read quest files, write job branches and iterated PR heads on the upstream | Read for quest reads; write only for that publication or merge |
 | Issues: write         | Read native issues, publish approved comments, label merge jobs' PRs       | Read or write for that operation                               |
 | Pull requests: write  | Observe, open and update job PRs, explicitly merge an exact head           | Read or write for that operation                               |
-| Checks: read          | Observe checks on a commit                                                 | Check reads only                                               |
+| Checks: write         | Observe checks; record a runner's review as the `quest review` check run   | Read for check reads; write only to record a runner's review   |
 | Commit statuses: read | Observe legacy CI status contexts                                          | Check reads only                                               |
 
 The manifest subscribes to push, issues, issue comments, pull requests, reviews,
@@ -26,7 +26,7 @@ user permission, which Better Auth needs to sign in users whose email is private
 It grants no repository authority: the user token is used only for the sign-in
 profile and email reads and to list the [project picker](#project-enrollment)'s
 repositories, and never for repository writes or authority checks. No organization,
-administration, workflow, or checks-write permissions are requested. These installation grants
+administration, or workflow permissions are requested. These installation grants
 are necessarily broader than individual operations: minted tokens always name
 one repository ID and the particular permissions required. Tokens are never
 cached and are revoked after use; failed cleanup is logged without secrets and
@@ -217,7 +217,8 @@ landed (`src/github/review.ts`):
 - an approval on GitHub of that commit by a person with write access other than
   the pull request's author; reviews by Apps never count;
 - a completed, successful `quest review` check run on that commit, written by
-  this App. Only the App can write one, and GitHub binds it to the commit;
+  this App from a runner's review (see [Runner reviews](#runner-reviews)).
+  Only the App can write one, and GitHub binds it to the commit;
 - or the author's own `## Review` record, as the guide's default review policy
   counts it in every enrolled repository: the `## Review` section of the PR
   body names the commit's full SHA, the author has write access, and the body's
@@ -289,6 +290,35 @@ plan/start/bootstrap publication on the enrolled upstream without a fork,
 GitHub's signature on App-authored API commits, and pushing or resetting a job
 branch under the repository's branch rules.
 
+### Runner reviews
+
+A runner reviews the heads its jobs publish, so they land through the gate
+above without a person's approval. When a job delivers a head it published,
+other than to a pull request that stays a draft, the runner process runs
+`quest review` on that head with the other vendor's model: Codex for a job on
+an Anthropic account, Claude Code otherwise. It runs outside the agent's
+sandbox, in a checkout of the published commit the agent cannot reach, and
+reports the round over `POST /api/runner/review`. The agent never touches the
+report, so it can neither skip nor fake the review. A merge job publishes its
+fixes, has them reviewed, and lands only a head whose review found nothing.
+
+kix records the round as the App's `quest review` check run on the commit,
+with the reviewer, model, effort, and round as its summary and each finding as
+plain text: `success` when the round found nothing, `failure` otherwise. The
+agent fixes or refutes each finding and delivers again; the next round resumes
+the reviewer's session with its responses. kix records a report only from the
+job's current attempt, only on the head kix itself last published for the job
+and GitHub attributes to a bot, not one a merge job continued onto,
+and only from another vendor's model than the job's account runs, and writes it
+with `checks: write` on behalf of the job's requester, after rechecking their
+maintainer authority.
+
+Claude Code reviews with one of the runner's Claude Code accounts that is not
+limited. Runners hold no Codex accounts, so Codex reviews with the login of
+the user running `quest runner`. A runner that cannot review, for want of a
+reviewer or because the review failed, says so to the agent and leaves the
+head to a person's approval, as before.
+
 ## Webhooks and revocation
 
 `POST /webhooks/github` verifies `X-Hub-Signature-256` over the exact raw bytes
@@ -334,7 +364,10 @@ reconcile current provider state rather than replay issue/PR payloads as facts.
    PEM keys, installation tokens or `.dev.vars`.
 3. Install the public App on selected upstream repositories. Existing
    installations must accept permission increases before operations needing
-   those grants become available. GitHub's installation UI, not browser form
+   those grants become available. An App registered before runner reviews
+   holds `checks: read`: raise it to **Checks: write** in the App's settings,
+   and have each installation accept the change, or runners' reviews are not
+   recorded and their heads need a person's approval to land. GitHub's installation UI, not browser form
    claims, chooses repository access. Enroll each repository on `/projects`;
    jobs publish their branches there, with no fork setup.
 4. Run `bun run wrangler d1 migrations apply DB --remote`, then the normal

cloudflare/README.md

@@ -171,7 +171,11 @@ decision. The current attempt of a job on an issue or pull request reads it at
 current authority: an issue's text and comments, or a pull request's text,
 comments, reviews, unresolved review threads, and checks at its head. A new
 plan's attempt reads its `idea` and `context` there instead. [GITHUB.md](GITHUB.md#publication) describes the request, its
-validation, and how interrupted publications are reconciled.
+validation, and how interrupted publications are reconciled. After a delivery,
+the runner reviews the head the job published with the other vendor's model
+and reports the round at `POST /api/runner/review`, which kix records as the
+App's `quest review` check run on that commit
+([GITHUB.md](GITHUB.md#runner-reviews)).
 
 ### Raw output
 

cloudflare/dev/github.ts

@@ -108,6 +108,8 @@ export type Repo = {
 	rules?: Record<string, unknown>[];
 	/** The default branch's classic protection, as `GET /branches/main/protection` returns it; none when absent. */
 	protection?: Record<string, any>;
+	/** Check runs the App wrote through `POST /check-runs`, oldest first. */
+	checkRuns?: { name: string; head_sha: string; conclusion: string; app: number; output: Record<string, unknown> }[];
 	/** Rulesets by ID, as `GET /rulesets/<id>` shows them to the App. */
 	rulesets?: Record<number, { current_user_can_bypass: "always" | "pull_requests_only" | "never" }>;
 };
@@ -902,6 +904,19 @@ export function fakeGitHub(world: World) {
 			const found = repo.rulesets?.[Number(ruleset[1])];
 			return found ? json({ id: Number(ruleset[1]), ...found }) : notFound();
 		}
+		if (tail === "/check-runs" && method === "POST") {
+			if (!writes("checks")) return forbidden();
+			if (body.status !== "completed" || typeof body.head_sha !== "string" || !repo.commits[body.head_sha])
+				return json({ message: "Validation Failed" }, 422);
+			(repo.checkRuns ??= []).push({
+				name: body.name,
+				head_sha: body.head_sha,
+				conclusion: body.conclusion,
+				app: world.app.id,
+				output: body.output,
+			});
+			return json({ id: repo.checkRuns.length, name: body.name, head_sha: body.head_sha }, 201);
+		}
 		const runs = tail.match(/^\/commits\/([0-9a-f]{40})\/check-runs$/);
 		if (runs && method === "GET") {
 			if (!scope.permissions.checks) return forbidden();
@@ -918,7 +933,13 @@ export function fakeGitHub(world: World) {
 						app: { id: run.app ?? ACTIONS_APP },
 					})),
 			);
-			const check_runs = found.filter(
+			// The App's own runs, newest first as `filter=latest` keeps one per name.
+			const written = (repo.checkRuns ?? [])
+				.filter((run) => run.head_sha === runs[1])
+				.reverse()
+				.filter((run, i, all) => all.findIndex((other) => other.name === run.name) === i)
+				.map((run) => ({ ...run, status: "completed", app: { id: run.app } }));
+			const check_runs = [...written, ...found].filter(
 				(run) => (!name || run.name === name) && (!app || String(run.app.id) === app),
 			);
 			return json({ total_count: check_runs.length, check_runs });

cloudflare/github-app-manifest.json

@@ -12,7 +12,7 @@
 		"contents": "write",
 		"issues": "write",
 		"pull_requests": "write",
-		"checks": "read",
+		"checks": "write",
 		"statuses": "read",
 		"email_addresses": "read"
 	},

cloudflare/src/github/service.ts

@@ -2,7 +2,7 @@ import { createHash, createPrivateKey, sign } from "node:crypto";
 import { HOLD_PREFIX, isHoldLabel, latestHold, heldToken, check, isQuest, parse, ready } from "quest/core";
 import { cleanAutoMergeRefusal, immediateMerge } from "../../../src/github/merge";
 import { osiApproved } from "../licenses";
-import { unreviewed } from "./review";
+import { REVIEW_CHECK, unreviewed } from "./review";
 import { positiveId } from "./webhook";
 import {
 	activityQuery,
@@ -1765,6 +1765,40 @@ export class GitHubService {
 		);
 	}
 
+	/**
+	 * Write the App's `quest review` check run on `head`, a commit a job
+	 * published and GitHub attributes to a bot, from a runner's review of it, on behalf of the job's
+	 * requester. GitHub binds the run to the commit, and `unreviewed` counts
+	 * only this App's, so a passing one lets kix land exactly that head.
+	 */
+	writeReviewCheck(
+		repo: GitHubRepository,
+		userId: string,
+		input: {
+			head: string;
+			conclusion: "success" | "failure";
+			output: { title: string; summary: string; text?: string };
+		},
+	) {
+		return this.#authorized(repo, userId, { contents: "read", checks: "write" }, async (token) => {
+			// The job's head is someone else's commit after a merge job continues onto it; that one is not a runner's to pass.
+			const { committer } = await this.#api<{ committer: { type: string } | null }>(
+				token,
+				`${repositoryPath(repo)}/commits/${sha(input.head)}`,
+			);
+			if (committer?.type !== "Bot")
+				throw new GitHubError("kix records a runner's review only of a commit it published");
+			await this.#api(token, `${repositoryPath(repo)}/check-runs`, "POST", {
+				name: REVIEW_CHECK,
+				head_sha: sha(input.head),
+				status: "completed",
+				conclusion: input.conclusion,
+				completed_at: new Date().toISOString(),
+				output: input.output,
+			});
+		});
+	}
+
 	/**
 	 * Merge exactly `expectedHead` with the first method the repository
 	 * allows: squash, then a merge commit, then rebase, once a review counts on

cloudflare/src/jobs/review.ts

@@ -0,0 +1,121 @@
+// A runner's review of a head its job published. The runner process, outside
+// the agent's sandbox, runs `quest review` with the other vendor's model and
+// reports the round here; kix records it as the App's `quest review` check run
+// on that commit, which its land gate counts (`src/github/review.ts`). Every
+// field is untrusted input: the findings are model text, and only the job's
+// current attempt may report on the head kix itself last published for it.
+import { GitHubService } from "../github/service";
+import { id, text } from "./protocol";
+import * as store from "./store";
+import { JobError, type Runner } from "./store";
+
+/** The vendor whose model each reviewer CLI runs, in the provider names runner accounts report. */
+export const VENDOR = { claude: "anthropic", codex: "openai" } as const;
+type Harness = keyof typeof VENDOR;
+const SEVERITIES = ["high", "medium", "low"] as const;
+
+export type Finding = {
+	file: string;
+	line: number | null;
+	severity: (typeof SEVERITIES)[number];
+	title: string;
+	body: string;
+};
+export type ReviewReport = {
+	job: string;
+	attempt: number;
+	head: string;
+	reviewer: Harness;
+	model: string;
+	effort: string;
+	round: number;
+	findings: Finding[];
+};
+
+/** Bounds a report: GitHub keeps at most 65,535 characters of a check run's text. */
+export const REVIEW_LIMITS = { findings: 100, title: 300, body: 8000 };
+
+const count = (value: unknown, min: number) =>
+	typeof value === "number" && Number.isSafeInteger(value) && value >= min ? value : null;
+
+/** Parse a runner's review report, or throw a JobError the runner can read. */
+export function parseReview(body: Record<string, unknown>): ReviewReport {
+	const job = id(body.job);
+	const attempt = count(body.attempt, 1);
+	if (!job || !attempt) throw new JobError("Job and attempt required");
+	if (typeof body.head !== "string" || !/^[a-f0-9]{40}$/.test(body.head))
+		throw new JobError("The reviewed head must be a full SHA");
+	const reviewer = body.reviewer;
+	if (reviewer !== "claude" && reviewer !== "codex") throw new JobError("The reviewer is claude or codex");
+	const model = text(body.model, 100);
+	const effort = text(body.effort, 50);
+	const round = count(body.round, 1);
+	if (!model || !effort || !round) throw new JobError("The reviewer's model, effort, and round are required");
+	if (!Array.isArray(body.findings) || body.findings.length > REVIEW_LIMITS.findings)
+		throw new JobError(`At most ${REVIEW_LIMITS.findings} findings per review`);
+	const findings = body.findings.map((item: Record<string, unknown> | null): Finding => {
+		const file = text(item?.file, 1024);
+		const title = text(item?.title, REVIEW_LIMITS.title);
+		const detail = typeof item?.body === "string" && item.body.length <= REVIEW_LIMITS.body ? item.body : null;
+		const line = item?.line === null ? null : count(item?.line, 1);
+		const severity = SEVERITIES.find((s) => s === item?.severity);
+		if (!file || !title || detail === null || (item?.line !== null && line === null) || !severity)
+			throw new JobError("Each finding needs a file, line or null, severity, title, and body");
+		return { file, line, severity, title, body: detail };
+	});
+	return { job, attempt, head: body.head, reviewer, model, effort, round, findings };
+}
+
+/** Untrusted text inside a fence no run of backticks in it can close. */
+function fenced(value: string) {
+	const longest = Math.max(2, ...(value.match(/`+/g) ?? []).map((run) => run.length));
+	const fence = "`".repeat(longest + 1);
+	return `${fence}text\n${value.replace(/[\x00-\x08\x0b-\x1f\x7f]/g, "")}\n${fence}`;
+}
+
+/** The check run's output: a title, a summary naming the reviewer, and the findings as plain text. */
+export function checkOutput(report: ReviewReport) {
+	const n = report.findings.length;
+	const title = n ? `${n} finding${n === 1 ? "" : "s"}` : "No findings";
+	const summary = `A kix runner reviewed this commit with \`quest review\`: ${report.reviewer}, ${report.model} at ${report.effort} effort, round ${report.round}.`;
+	const findings = report.findings.map((f) =>
+		fenced(`${f.severity} ${f.line === null ? f.file : `${f.file}:${f.line}`} ${f.title}\n\n${f.body}`),
+	);
+	let text = "";
+	for (const [i, finding] of findings.entries()) {
+		const next = `${text}${text ? "\n\n" : ""}${finding}`;
+		// GitHub refuses a check run's text past 65,535 characters.
+		if (next.length > 60_000) {
+			text += `\n\n${findings.length - i} more not shown.`;
+			break;
+		}
+		text = next;
+	}
+	return { title, summary, ...(text && { text }) };
+}
+
+/**
+ * Record a review the job's current attempt reports as the App's
+ * `quest review` check run, passing only when the round found nothing. Only a
+ * head kix published for the job counts, the one its branch holds now, and
+ * only a review by another vendor's model than the job's account ran.
+ */
+export async function recordReview(env: Env, runner: Runner, report: ReviewReport) {
+	const job = await store.publishing(env.DB, runner, report.job, report.attempt);
+	if (job.pending !== null) throw new JobError("A publication of this job is still in flight; review its head after");
+	if (report.head !== job.head || job.head === job.revision)
+		throw new JobError("kix records a review only of the head this job last published");
+	const account = await env.DB.prepare("SELECT provider FROM runner_accounts WHERE runner=? AND id=?")
+		.bind(runner.id, job.account)
+		.first<{ provider: string }>();
+	if (!account) throw new JobError("kix no longer knows the account this job runs on, so it cannot tell its vendor");
+	if (account.provider === VENDOR[report.reviewer])
+		throw new JobError("A review counts only from another vendor's model than the one the job runs on");
+	const conclusion = report.findings.length ? "failure" : "success";
+	await new GitHubService(env).writeReviewCheck({ owner: job.owner, name: job.name, id: job.repository }, job.actor, {
+		head: report.head,
+		conclusion,
+		output: checkOutput(report),
+	});
+	return { conclusion };
+}

cloudflare/src/jobs/routes.ts

@@ -10,6 +10,7 @@ import * as output from "./output";
 import { formAnswer, parseAnswer, parsePrompt } from "./prompt";
 import { data, id, text } from "./protocol";
 import { HeadMoved, LIMITS, parsePublication, publish, pullOf, restart, validateContinuation } from "./publish";
+import { parseReview, recordReview } from "./review";
 import * as store from "./store";
 import { JobError } from "./store";
 
@@ -137,6 +138,17 @@ jobs.post("/api/runner/publish", async (c) => {
 	return c.json({ decision, cancel }, 202);
 });
 
+// The runner reports its own review of a head the job published, which kix
+// records as the App's `quest review` check run on that commit.
+jobs.post("/api/runner/review", async (c) => {
+	const runner = await machine(c);
+	if (!runner) return c.json({ error: "Runner token required" }, 401);
+	const length = Number(c.req.header("Content-Length"));
+	if (!Number.isSafeInteger(length) || length > 2 * 1024 * 1024)
+		return c.json({ error: "A Content-Length of at most 2 MiB is required" }, 413);
+	return c.json(await recordReview(c.env, runner, parseReview(await json(c))));
+});
+
 // The current attempt of a job on an issue or pull request reads its target
 // as the App, with the requester's current authority: runners hold no GitHub
 // credentials, and the job record keeps only the target's number. A new