Skip to content

Exclude agent reasoning from findings judge input - #54

Merged
MagMueller merged 1 commit into
mainfrom
codex/judge-without-reasoning
Sep 25, 2026
Merged

MagMueller merged 1 commit into
mainfrom
codex/judge-without-reasoning

Conversation

@MagMueller

@MagMueller MagMueller commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

The findings judge currently sees agent reasoning summaries as trajectory evidence and can penalize speculative drafts that were never delivered. Exclude top-level thinking: and reasoning: steps when constructing judge messages.

Original step numbers remain attached to tools and screenshots. Saved traces, final answers, output files, scoring rules and deterministic canary checks stay intact. Filter before the trajectory text budget; record adapter version 2.1.4. Existing results and charts are not recomputed.

Validation: 72 offline tests completed (69 passed, 3 optional integration tests skipped), including screenshot-reference and raw-evidence preservation; rubric revision review and the workflow's Ruff checks passed. No paid model or browser evaluations.


Summary by cubic

Prevents the findings judge from treating top-level thinking: and reasoning: summaries as trajectory evidence, so speculative drafts no longer affect scoring. Saved traces, tool and screenshot evidence, final answers, output files, and deterministic checks remain unchanged.

  • Preserves original step numbers for tool and screenshot references.
  • Applies the filtering before the trajectory text budget.
  • Records adapter version 2.1.4.
  • Adds coverage for reasoning exclusion and raw evidence preservation.

Written for commit fec2232. Summary will update on new commits.

Review in cubic

@MagMueller
MagMueller merged commit 1e0e2f1 into main Sep 25, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants