[kixelated]

Runners now review the heads their jobs publish. kix records each review as the App's quest review check, so a runner's head can land through kix's land gate without a person's approval. This finishes Runner review checks, from moq-dev/quest#351's follow-ups.

Changes

• Runner (src/runner/review.ts, src/runner/job.ts): after deliver publishes a head, the runner reviews it before the tool returns. It skips heads the job did not publish and pull requests that stay drafts. The runner process runs quest review in-process, outside the agent's sandbox. It reviews a separate checkout of the published commit, beside the job's checkout, on a kix-review/<job> branch, so later rounds resume the same reviewer session. The agent receives the findings and answers them through a new responses argument on its next deliver.
  • A merge job now publishes its fixes, gets them reviewed, and only then sends a second, landing-only publication. It does not land when the review reports findings.
  • If the runner cannot review (no reviewer, or the review failed), it tells the agent and falls back to the existing gate, which pauses on a decision for a person's approval.
• Reviewer choice: a job on Anthropic's models is reviewed by Codex; any other job is reviewed by Claude Code.
  • Claude Code uses one of the runner's Claude Code accounts that is not rate-limited (CLAUDE_CONFIG_DIR), through the Claude Code CLI the runner pins.
  • Runners hold no Codex accounts, so Codex uses the login of the user running quest runner, as a local quest review does.
• kix (cloudflare/src/jobs/review.ts, POST /api/runner/review): validates the report and writes the App's quest review check run on that commit, success or failure, on behalf of the job's requester. The check text shows the findings inside a fenced block. Before writing, kix requires all of the following:
  • the report comes from the job's current attempt;
  • the head is the one kix last published for the job, and not the job's starting revision;
  • GitHub attributes the commit to a bot, which excludes another writer's head a merge job continued onto;
  • the reviewer is from another vendor than the job account's provider.
• App permission: checks: write replaces checks: read in the manifest. cloudflare/GITHUB.md documents runner reviews and the operator step: an already-registered App must be raised to Checks: write, and its installations must accept the change. The fake GitHub now models writing check runs.
• quest review (src/cli/review.ts): a ResponsesError subclass and an exported sessionFile, so the runner can show the agent a clear error and clean up a finished job's review session.
• Quests: deletes quest/m0/kix/review-check.md. Adds a live check of the real checks API to Operator setup.

Choices made without asking

• Which deliveries get a review: every delivery whose head the job published, except when its pull request stays a draft. That covers non-draft start and bootstrap deliveries, every iterate delivery, and a merge job's delivery before it lands. Plan pull requests are always drafts, so they are never reviewed. An iterate job cannot tell whether its pull request is a draft, so its deliveries are always reviewed.
• HTTPS route instead of a WebSocket message: the review report goes to POST /api/runner/review, like publication, so the GitHub write never holds up the runner hub's serialized messages.
• Codex credentials: runners hold no Codex accounts. A claude job is therefore reviewed with the Codex login of the user running quest runner (CODEX_HOME or Codex's default), using whichever codex is on that user's PATH. This matches a local quest review. Claude Code reviews use one of the runner's own Claude Code accounts that is not rate-limited, through the pinned CLI. Alternatives considered:
  • review-only Codex accounts in quest runner account add, kept out of job routing;
  • an opencode reviewer harness in quest review, which needs read-only sandboxing for opencode, since opencode loads plugins from the checkout.
• A finished delivery needs a passing review: a delivery whose review reported findings does not count as delivered. If the agent ends before answering, the job fails, and a usage limit still pauses it as limited.
• Only App-committed heads: kix refuses to record a review of a head GitHub does not attribute to a bot, such as another writer's head a merge job continued onto. The runner's response then tells the agent the head needs a person's approval.
• Pass/fail rule: a round passes when it reports no findings. A refuted finding passes only once the reviewer's next round accepts the refutation, as the guide's review loop already requires.

Review

Codex CLI (quest review), gpt-6.1-sol at medium effort, session 01a19f65, 3 rounds against origin/main:

• Round 1 on 9efdd40: 2 findings, both fixed in d663aa1.
  • src/runner/job.ts: a delivery whose review reported findings still counted as delivered, so the job completed. Fixed: it fails, or pauses as limited, until the agent answers the findings.
  • src/runner/review.ts: a stopped or cancelled attempt never cancelled or awaited its reviewer. Fixed: the attempt's abort signal kills the reviewer CLI, and the attempt awaits it before removing the checkout.
• Round 2 on d663aa1: 1 finding, fixed in 248723d.
  • src/cli/review.ts: an aborted turn rejected before the reviewer process had exited. Fixed: it rejects only on the child's close, sending SIGKILL after a 10 s grace period.
• Round 3 on 248723d: no findings.

The current head, 3fbf62e, only adds a conflict-free merge of origin/main: git diff origin/main...HEAD is identical to the diff of the reviewed head.

Local checks: just check and just test pass (943 CLI tests and 477 Worker tests).

Suggested follow-ups

• Codex reviewer credentials for runners: pin the Codex CLI in the runner package (QUEST_RUNNER_CODEX, or put it on the reviewer's PATH), after the review-harness quest confirms a version works with quest review's flags.
• Cancel a running review when its job is cancelled or stopped. Today the reviewer CLI runs to the end and kix then refuses the stale report.
• quest review runs git fetch and ls-remote synchronously. Inside the runner process that blocks the event loop, including the WebSocket, for the length of the fetch.

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 32 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 70e5bc03-9869-44f8-81a7-170d78bfdec6

Commits

Reviewing files that changed from the base of the PR and between acb8682 and 3fbf62e.

Files selected for processing: 19

• cloudflare/GITHUB.md
• cloudflare/README.md
• cloudflare/dev/github.ts
• cloudflare/github-app-manifest.json
• cloudflare/src/github/service.ts
• cloudflare/src/jobs/review.ts
• cloudflare/src/jobs/routes.ts
• cloudflare/test/publish.test.ts
• quest/m0/kix/README.md
• quest/m0/kix/operator-setup.md
• quest/m0/kix/review-check.md
• src/cli/review.ts
• src/runner/job.ts
• src/runner/prompt.ts
• src/runner/review.ts
• src/runner/runner.ts
• src/runner/workspace.ts
• test/review.test.ts
• test/runner.test.ts

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

CodeRabbit Verify

• Run CodeRabbit Verify to test the changes in this pull request.

[kixelated]

Two follow-ups from the /quest-spawn coordinator's evidence check. Both are relevant to this PR's change, so they belong here rather than in new quests:

1. Runner review blocks the event loop. reviewHead (src/runner/review.ts) calls review() in-process. review() runs git fetch (src/cli/review.ts:750) and git ls-remote --symref (:691) through gitOutput, which is execFileSync (src/cli/claims.ts:351). The review checkout's origin is the GitHub URL (workspace.ts:101), so each fetch blocks the runner's WebSocket and every other job for its duration. It also fetches into the shared bare clone outside locked(). Make those calls async (review() is already async), and run them under the workspace lock, or pre-fetch the base in reviewCheckout and pass --base.
2. Pin codex in the runner package. The runner wrapper in flake.nix pins claude, opencode and srt, but reviewerFor takes whatever codex is on PATH, so a runner user without one can't review Claude jobs. nixpkgs' codex 0.164.0 accepts every flag quest review passes: exec, exec review and exec resume with --ignore-rules, --ignore-user-config, --output-schema, --json, --base and -c (checked 2026-11-02). Add --set-default QUEST_RUNNER_CODEX ${getExe pkgs.codex}, and in reviewerFor put that binary's directory first on PATH, as for claude. The user still needs a Codex login.

(Written by Claude Opus 5.5)