Skip to content

feat: add assertions schema to EvalCriteria for code-based grading - #4088

Open
melmennaoui wants to merge 1 commit into
mainfrom
feat/4082-assertions-schema
Open

feat: add assertions schema to EvalCriteria for code-based grading#4088
melmennaoui wants to merge 1 commit into
mainfrom
feat/4082-assertions-schema

Conversation

@melmennaoui

Copy link
Copy Markdown
Contributor

Summary

Add code-based assertion support to the evaluation criteria schema, enabling declarative grading checks that run against agent output without requiring an LLM judge.

Changes

pkg/session/session.go

  • Assertion structName, Type, Value fields for declarative checks. Supported types: contains, not_contains, equals, starts_with, ends_with, regex, cost_threshold, tool_called.
  • Assertions field on EvalCriteria — optional list of code-based assertions.
  • Verify field on EvalCriteria — optional shell script for post-agent outcome verification.
  • AssertionsCheck — result type with passed/total counts and per-assertion results.
  • AssertionResult — individual assertion outcome (name, type, passed, reason).
  • VerifyCheck — verify script result (passed, exit_code, output).
  • Added both to EvalResultChecks.

pkg/session/branch.go

  • Updated cloneEvalCriteria to deep-clone the Assertions slice.
  • Updated cloneEvalResultChecks to deep-clone AssertionsCheck and VerifyCheck.

pkg/session/session_test.go

  • Added test cases for JSON unmarshaling of assertions and verify fields.

Example eval JSON

{
  "evals": {
    "relevance": ["The agent created the file"],
    "assertions": [
      {"name": "mentions file", "type": "contains", "value": "hello.txt"},
      {"name": "used write tool", "type": "tool_called", "value": "write_file"},
      {"name": "cost under budget", "type": "cost_threshold", "value": "0.50"}
    ],
    "verify": "test -f /workspace/hello.txt"
  }
}

Closes #4082

Add code-based assertion support to the evaluation criteria schema:

- Assertion struct (Name, Type, Value) for declarative grading checks
  supporting types: contains, not_contains, equals, starts_with,
  ends_with, regex, cost_threshold, tool_called
- Verify field on EvalCriteria for post-agent shell script verification
- AssertionsCheck and VerifyCheck types on EvalResultChecks
- AssertionResult type for per-assertion outcomes
- Deep-clone support in cloneEvalCriteria and cloneEvalResultChecks
- Tests for JSON unmarshaling of the new fields

Closes #4082
@melmennaoui
melmennaoui requested a review from a team as a code owner August 31, 2026 15:30
@aheritier aheritier added area/sessions For features/issues/fixes related to session lifecycle (resume, persistence, export) kind/feat PR adds a new feature (maps to feat:). Use on PRs only. labels Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/sessions For features/issues/fixes related to session lifecycle (resume, persistence, export) kind/feat PR adds a new feature (maps to feat:). Use on PRs only.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add assertions schema to EvalCriteria for code-based grading

2 participants