Skip to content

feat(review): add pr-review-cli, a review lane on run-claude-cli - #862

Merged
kyle-sexton merged 2 commits into
mainfrom
feat/pr-review-cli
Oct 23, 2026
Merged

kyle-sexton merged 2 commits into
mainfrom
feat/pr-review-cli

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

No related issue: second step of moving the Claude review lanes onto the run-claude-cli launcher (follows #861); claude-code-plugins' shadow and cutover callers consume it.

Summary

The two review lanes run on claude-code-action with one subscription token. This adds a reusable workflow that runs either lane on the run-claude-cli launcher: the funding role's API key first, its subscription token after a credit stop, a 401 or 402, or no key, and nothing after a 403. A summary mode lets a caller run it as a shadow beside the live lanes, posting nothing. pr-review.yml and pr-review-security.yml are unchanged.

Fix

  • New .github/workflows/pr-review-cli.yml, inputs lane: code|security and publish: comments|summary, three jobs on the two-job pattern intake-triage.yml uses:
    • review-key runs the key leg; review-subscription runs the token leg only on an exact handoff == 'true' or credential-present == 'false', on ubuntu-24.04, repeating every step. The repository variable REVIEWS_KEY_EXHAUSTED=true makes the key job report no key without a model call.
    • Both model jobs: contents: read, pull-requests: read, cache-mode: none, no id-token, no second secret; tools Read, Grep, Glob, Skill, Agent under --permission-mode dontAsk with Skill(review:<lane skill>) and Agent allowed and Bash disallowed; an init-message permission-mode check; leg limits of 11 and 13 minutes; background tasks off.
    • Both carry the same gate: pull_request events only, open, not a draft, head repository equal to the base, no Dependabot as actor or author, bots only by sender id.
    • Scope, full diff and PR context are ported from pr-review.yml, and both lanes now get all three prompt lines (FULL DIFF: ... is in, REVIEW SCOPE: incremental ... That change is in, PR CONTEXT: ... are in). The incremental note points at the FULL DIFF file in place of gh pr diff, since the model has no shell.
    • The review plugin is staged read-only into .lane/plugins/review from claude-code-plugins' default-branch tip. A head that holds .lane fails.
    • An extract step reads the review only from the launcher's scanned structured-output-file and its facts from the scanned log, checks shapes and caps, and raises the findings count to the comment and manifest-flag counts.
  • New .github/actions/publish-review (no model), run by the claude-review-status / security-review status job:
    • It checks every anchor against a fresh read of the PR's diff at the head; one bad anchor sends the batch to the summary.
    • It strips HTML comments, appends claude-lane: <lane> to each comment, and withholds everything on a blocking hit from claude-code-plugins' lib/publish-gate.mjs, which it reads at that repository's tip.
    • It posts one COMMENT review and skips comments already posted on the head.
    • It writes the marker and the claude-lane-review-complete notice as pr-review.yml does, with account=fallback after the subscription leg.
  • Summary mode posts nothing and never ends red. Every step continues on error, every job keeps at least 5 minutes over the sum of its steps' limits, and lane records use pr-review-shadow and pr-review-security-shadow. The status job writes a job summary and a scanned JSON artifact (findings, comments, turns, total_cost_usd, models, leg and wall time) named per lane and attempt.
  • actions-expression.cjs evaluates vars and secrets paths. The lane-outcome emitter test lists the two new lane jobs.

Verification

  • node --test .github/scripts/*.test.cjs: 610 pass, 0 fail.
    • pr-review-cli.test.cjs (16) covers: which job holds which secret; read-only model jobs; the gate turning away fork, draft, closed, other-event, Dependabot and unlisted-bot runs; the exact handoff compare; key and subscription steps equal; the exhausted-key variable; the never-red rule for summary, empty and unknown values (steps continue on error, timeouts add up with headroom); the prompt-line contract; the extract step on a good review and on every gap it checks; and run-claude-cli's readConfig/cliArgs accepting the launcher step's inputs for both lanes.
    • publish-review.test.cjs (13) covers: commentable lines against a hand-numbered patch; the batch rejection; the footer and HTML strip; the post, marker and notice text; the fallback account; the rerun skip; a withheld credential versus a warn-only label; each failure class in comments mode; the skip carry-forward; summary mode never failing or writing across failure inputs and a throwing API; and the shadow report.
  • Mutation check: dropping one step's continue-on-error, and letting summary mode throw, each fail the never-red tests.
  • run-claude-cli tests 72 pass; report-lane-outcome tests 74 pass. actionlint (only the repository's ignored $/ and cache-mode messages), zizmor, biome over .github/scripts, markdownlint, typos and gitleaks: clean.
  • Not provable here: whether the nested permissions and $/ references start cleanly in a caller. The first caller's own run is that check.

Related

🤖 Generated with Claude Code

A reusable workflow that runs the code or security review on the
run-claude-cli launcher with the two-job pattern: the role's API key in
review-key, its subscription token in review-subscription only on an
exact handoff or with no key (REVIEWS_KEY_EXHAUSTED=true skips the key's
model call). The model jobs are read-only with Read, Grep, Glob, Skill and
Agent under dontAsk, the review plugin from claude-code-plugins' default
tip, and the diff and context as files named by the FULL DIFF, REVIEW
SCOPE and PR CONTEXT prompt lines. A status job with no model publishes
the structured review through the new publish-review action: anchors
checked against the diff at the head, HTML comments stripped, a
claude-lane footer, the publish gate's credential scan, one COMMENT
review, the marker and the completion notice. Summary mode posts nothing
and never ends red. pr-review.yml and pr-review-security.yml are
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review October 23, 2026 02:47
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-23T02:49:24.190936Z b986353 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b986353771

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/pr-review-cli.yml
The status job receives the review as one environment variable, and
Linux refuses a string over 128 KiB, so a large valid review could stop
the publish step from starting. The schema now caps comments at 25 and
bodies at 3,000 characters, and the extract step calls a serialized
review over 100,000 bytes incomplete instead of passing it on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@claude

claude Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 3m 45s —— View job


Security review complete

  • Gather context (PR body, diff, prior review comments)
  • Read changed files: publish-review action/script, pr-review-cli.yml diff vs prior, test files
  • Apply skip gate / scope check
  • Review for security issues per criteria (logic flaws, authz, injection, credential handling, manifest changes)
  • Adversarial validation of candidate findings
  • Post final review

No security issues found in this PR at ad522df.

What I looked at

  • The ad522df commit itself: it fixes the environment-string-size issue Codex and @kyle-sexton flagged on b986353 (KEY_REVIEW/SUB_REVIEW could exceed Linux's environment-string limit). The fix lowers the schema caps (25 comments × 3,000 chars) and adds a hard 100,000-byte serialized-review ceiling in the extract step (pr-review-cli.yml#L1042-L1058, mirrored for the subscription leg), refusing to pass the review on rather than handing the publish step an environment it can't start with. The new test (pr-review-cli.test.cjs) exercises both the largest-passing ASCII review and a multi-byte-character review that exceeds the limit. This closes the gap correctly — it fails safe (treated as an incomplete review) rather than crashing the publish step.
  • .github/actions/publish-review/publish.cjs (new, not previously reviewed inline): verified the anchor check (anchorProblem) re-validates every inline comment's path/line/side against a fresh read of the PR's diff at the head before posting, so a prompt-injected or stale comment can't be anchored to arbitrary lines; one bad anchor correctly routes the whole batch to the summary. scan()/findSecret gating fails closed — if the publish gate can't be loaded, the step fails (publish-failed) rather than posting unscanned model output, in both comment and shadow modes. The WARN_ONLY exception ("password or secret assignment") only applies to text that already appears in the (public) diff the model reviewed, so it doesn't introduce a new disclosure path.
  • .github/actions/publish-review/action.yml, actions-expression.cjs (vars/secrets context support), and the lane-outcome-emitters.test.cjs update: no security-relevant behavior change; the new workflow's credential handling, trust gates, and read-only model-job scoping match the SECURITY MODEL header and are unchanged from b986353.
  • No .claude-plugin/plugin.json/marketplace.json changes in this PR, and no instruction/rules files (CLAUDE.md, AGENTS.md, skill/agent bodies) touched — both out of scope for this diff.

GitHub Actions hardening concerns (expression injection, trigger safety, permission scoping, pinning) are zizmor's lane; I deferred to it per this skill's instructions and didn't re-check those here.

claude-lane-summary: lane=claude-security-review sha=ad522dfbaf61fa0e839384471f2a8725411de27f run=39891824847
· branch

@claude

claude Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 3m 38s —— View job


Claude code review in progress

  • Gather context (PR diff, workflow files, PR body contract, REVIEW.md scope split)
  • Confirm the earlier environment-string-size finding (Codex / kyle-sexton) is fixed and tested in the latest commit
  • Hunt for additional correctness issues in publish-review action and pr-review-cli.yml
  • Verify candidates
  • Post findings as inline comments
  • Post summary

@github-actions

Copy link
Copy Markdown

claude-security-review has reviewed this pull request through ad522df; a later push is reviewed from there.

@claude

claude Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 6m 21s —— View job


Code review — pr-review-cli.yml / publish-review

  • Gather context (PR diff, changed files, existing review threads)
  • Run /review:code-review over the diff
  • Post findings as inline comments (none meet the bar — see below)
  • Post summary comment

Summary

The previously-flagged P2 (KEY_REVIEW/SUB_REVIEW could exceed Linux's ~128 KiB per-environment-string limit) is correctly fixed in ad522df: the schema now caps comments at 25 items and body at 3,000 chars (pr-review-cli.yml#L826-L838), and the extract step now refuses a serialized review over 100,000 bytes rather than handing the publish step an environment string it can't start with (pr-review-cli.yml#L2740 — mirrored at L3637 for the subscription leg). The fix is applied symmetrically to both the review-key and review-subscription legs, and pr-review-cli.test.cjs's "no larger than one environment string can hold" test exercises both the ASCII ceiling and a multi-byte-character review that should be refused. I did not have Bash access in this lane to re-run the suite myself, so the PR body's "610 pass, 0 fail" claim is author-claimed and unverified by me; the test code itself reads correctly for what it asserts.

I read publish.cjs's judge/placeComments/scan logic and the extract/scope steps closely and didn't find a new correctness bug: the leg-selection, failure-class attribution (including the credit-stop/auth handoff-reason carry-through to a subscription leg that also has no credential), anchor validation against a fresh diff read, and the credential-scan-before-publish ordering all check out against their tests.

One maintainability point worth raising, not blocking:

Suggestion — pr-review-cli.yml's review-key and review-subscription jobs duplicate ~650 lines of step definitions verbatim. Everything between "Isolate the runner" and "Extract the review" (L248-L880) is byte-for-byte repeated in the subscription job (L1145-L1777), differing only in the credential input and the fallback-reason. The repo already has a dedicated test ("the subscription job repeats the key job's steps...") whose entire job is to catch drift between the two copies — a signal that this duplication is a recognized risk, not an oversight. The file already factors shared logic into composite actions for apply-claude-managed-settings and check-author-trust; the same pattern (composite actions for "Scope the review", "Write the full diff", "Fetch the pull request description and closing issues", "Stage the review plugin", "Compose the review", each still running fresh inside its own job for the same isolation guarantees) would make the two legs structurally identical instead of test-detected-identical, per this repo's own docs/standards/simpler-code.md ("collapses boilerplate into a helper") and reference-dont-duplicate.md (T4: the shape is namable and stable across both legs).

That said, I note intake-triage.yml's claude/subscription jobs already use this exact same duplicated-job pattern, so this PR is following established repo precedent rather than introducing something new — which is why I'm not blocking on it. Worth a cross-cutting look if/when that pattern gets revisited.

No security findings reported here: this repo runs a dedicated security lane (pr-review-security.yml), so security-scope items are left to it per REVIEW.md.

claude-lane-summary: lane=claude-review sha=ad522dfbaf61fa0e839384471f2a8725411de27f run=39891824847

@github-actions

Copy link
Copy Markdown

claude-review has reviewed this pull request through ad522df; a later push is reviewed from there.

@kyle-sexton
kyle-sexton added this pull request to the merge queue Oct 23, 2026
Merged via the queue into main with commit 8778094 Oct 23, 2026
57 of 58 checks passed
@kyle-sexton
kyle-sexton deleted the feat/pr-review-cli branch October 23, 2026 03:08
kyle-sexton added a commit to melodic-software/standards that referenced this pull request Oct 23, 2026
…contract (#848)

Refs: melodic-software/claude-code-account-rotation#259

## Summary

Registers the runner-policy contract for ci-workflows v0.63.0
`pr-review-cli.yml` (`8778094736e414233110f3e0d6d9948628ba77b5`), so
claude-code-plugins can move its live security review lane onto it.

## Fix

- `policy.json`: a hosted-only contract, `fixedRunsOn:
["ubuntu-24.04"]`, the inputs `lane`, `publish`, `role`, `model-args`,
`allowed-bot-sender-ids`, `allowed-author-ids`, `docs-only-paths`,
`incremental-review` and `cli-version`, and both reviews-role secrets:
`CLAUDE_API_KEY` from `REVIEWS_CLAUDE_API_KEY` (tried first) and
`CLAUDE_CODE_OAUTH_TOKEN` from `REVIEWS_CLAUDE_CODE_OAUTH_TOKEN` (after
a key handoff).
- `README.md`: the v0.63.0 entry.
- `runner-policy.test.mjs`: pins the contract; the secret expressions
are escaped template literals.

Supersedes the key-only #847.

## Verification

- `node --test components/runner-policy/runner-policy.test.mjs`: 336
pass, 0 fail.
- `node components/runner-policy/runner-policy.mjs --root .`: passed.
- This branch's analyzer run against the claude-code-plugins security
caller that moves to `pr-review-cli.yml@8778094` with both secrets:
passed; the currently synced policy rejects the same caller with
`runner-target-contract`.
- biome and markdownlint clean on the changed files.

## Related

- melodic-software/ci-workflows#862
- #842 (precedent), #847 (closed, key-only)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant