Skip to content

quest: review harness check for m0, with more Claude Code plants - #426

Merged
kixelated merged 2 commits into
mainfrom
quest/m0/review-harness
Nov 3, 2026
Merged

quest: review harness check for m0, with more Claude Code plants#426
kixelated merged 2 commits into
mainfrom
quest/m0/review-harness

Conversation

@kixelated

@kixelated kixelated commented Nov 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Finishes the m0 run of quest/m0/review-harness.md and creates the m1 run, quest/m1/review-harness.md, which requires m0.

Summary

  • Models: no bump. With Claude Code 2.1.316 and codex-cli 0.166.0, claude-opus-5-5 is still Anthropic's default general model; only the premium Fable tier (claude-fable-5-1) is above it. gpt-6.1-sol is still Codex's first-listed "latest workhorse" in its model cache, next to the frontier gpt-6-astra. The quest's deep tier is gone: moq-dev/quest#361 dropped --deep, so there is nothing to compare there.

  • Live sandbox test passes with the installed CLIs (quest gate just review-sandbox: 7 passed, 1 skipped, 98s). The skipped one is the new Claude-only test running under Codex.

  • More Claude Code plants (test/live/review-sandbox.test.ts):

    • hooks in .claude/settings.local.json
    • an allow rule plus its additionalDirectories
    • an env that routes every Bash command through a planted CLAUDE_CODE_SHELL_PREFIX script
    • apiKeyHelper
    • statusLine

    Every plant except the status line fires in the control run, where Claude Code is asked to run the allowed command. A new test runs one turn with quest's reviewer flags (claudeFlags), asks for git status and then the allowed command, and checks that nothing ran and the allowed command is in permission_denials. A review alone can't show this, because the reviewer chooses its own commands.

  • Logins with no file to copy. If the user's config home has no login file, which is the case for a macOS keychain login, the test no longer throws. It now passes the user's environment to quest review, which strips it and falls back on the CLI's environment credentials (moq-dev/quest#339). The control run uses the same environment through a new reviewerEnvironment export. A logged-out error names CLAUDE_CODE_OAUTH_TOKEN or ANTHROPIC_API_KEY, or CODEX_API_KEY. The README section on just review-sandbox says the same.

  • Codex worktree hooks: nothing depends on them. The land fixer (src/cli/land-fix.ts) passes --ignore-user-config, which also skips the project layer (openai/codex#49333). quest agent doesn't use Codex hooks, and quest init installs none. This is recorded in the m1 quest's "check first" list.

Choices made without asking

  • For keychain logins, I used environment credentials instead of reading the keychain. Copying a keychain item into the disposable home can't be verified on Linux. A token refresh would also write the new token back to the keychain under another name and could sign the user out. claude setup-token and CODEX_API_KEY avoid both problems and go through the review's real credential path. I haven't run it on macOS, so the m1 run checks that first.
  • The status line has no control: Claude Code draws it only in interactive sessions. It stays planted so the test catches a release that also runs it in print mode, and m1 rechecks this.
  • The env plant is a shell prefix, not ANTHROPIC_BASE_URL. A redirected base URL would break the control's API call. The shell prefix still shows whether the env block was applied, and the reviewer really does run Bash.
  • src/cli/review.ts now exports Context, claudeFlags, and reviewerEnvironment, only for the live test. Runtime behavior is unchanged.

Notes

  • On this host, claude on PATH is a wrapper. When no routing variable is set, it sends Claude Code through a local subscription gateway, so the Claude half of the live run went through that gateway, not through a copied login file. Both paths reach the same flags, and the plants check isolation, not billing.

Checks

  • just check: passed
  • just test: passed (936 + 474 tests)
  • quest gate just review-sandbox: passed

Review

Codex CLI 0.166.0, gpt-6.1-sol, medium effort, via quest review --base origin/main (round 1, session 01a19f8e-dd52-7db0-99ad-62c13c6d3df2). It reviewed head 00bfa0a against base acb86820 and reported no findings.

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

kixelated and others added 2 commits November 2, 2026 19:42
@kixelated
chore(quest): claim quest/m0/review-harness.md
dd1bac6 
Quest-Claim: f7afab7ade1c227d
@kixelated @claude
test(review): plant more Claude Code settings, and run logins without…
00bfa0a 
… a file

The live sandbox test now plants Claude Code's local settings hooks, an
allow rule with its additional directory, an `env` shell prefix, an API key
helper, and a status line, and checks that quest's reviewer flags deny the
command the planted rule allows. A login with no file to copy, such as one
in the macOS keychain, runs on the credentials its CLI reads from the
environment, through quest review's own fallback.

Both pinned review models are still current. The m0 run of the recurring
review harness check is done; the m1 run carries its findings forward.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Review harness check quest: review harness check for m0, with more Claude Code plants Nov 3, 2026
@kixelated
kixelated marked this pull request as ready for review November 3, 2026 03:59
@coderabbitai

coderabbitai Bot commented Nov 3, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 58 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 084a125c-b020-43b7-9a6f-9c0d18d3e9f4
Commits

Reviewing files that changed from the base of the PR and between acb8682 and 00bfa0a.

Files selected for processing: 7
  • README.md
  • quest/m0/README.md
  • quest/m0/review-harness.md
  • quest/m1/README.md
  • quest/m1/review-harness.md
  • src/cli/review.ts
  • test/live/review-sandbox.test.ts
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.

@kixelated
kixelated added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit 0b6fa16 Nov 3, 2026
21 checks passed
@kixelated
kixelated deleted the quest/m0/review-harness branch November 3, 2026 15:56
kixelated added a commit that referenced this pull request Nov 3, 2026
@kixelated @claude
Merge origin/main into quest/m0/review-gateway
1e4ec9b 
Resolves src/cli/review.ts and test/review.test.ts against main's abort
signal (#421/#426): the routed and stored-login status checks pass it on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated mentioned this pull request Nov 3, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@kixelated