Skip to content

feat: add verify script runner for post-agent outcome verification - #4089

Open
melmennaoui wants to merge 1 commit into
mainfrom
feat/4083-verify-script
Open

feat: add verify script runner for post-agent outcome verification#4089
melmennaoui wants to merge 1 commit into
mainfrom
feat/4083-verify-script

Conversation

@melmennaoui

Copy link
Copy Markdown
Contributor

Summary

Add a verify script runner to the evaluation package that executes a shell script via docker exec on the eval container after the agent completes. This enables filesystem-level and side-effect checks that cannot be expressed as text assertions on the response (e.g. "did the agent actually create the file?").

Changes

pkg/evaluation/verify.go (new)

  • runVerifyScript() — executes a shell script on a running container:
    • Zero exit code → pass; non-zero → fail
    • 60-second timeout prevents runaway scripts
    • stdout + stderr captured and merged
    • Output capped at 8 KB to bound memory
    • Empty script is a no-op pass (verify is optional)
  • verifyResult struct — Passed, ExitCode, Output
  • maxVerifyOutputBytes constant (8192)

pkg/evaluation/verify_test.go (new)

  • Empty script passes without running anything
  • Zero exit code passes and captures output
  • Non-zero exit code fails and captures output
  • Large output is capped at the limit
  • writeFakeExecRuntime test helper avoids needing a real Docker daemon

Usage (wired by #4086)

The eval JSON declares a verify script in the evals block:

{
  "evals": {
    "verify": "test -f /workspace/hello.txt && grep -q 'Hello' /workspace/hello.txt"
  }
}

After the agent run, the framework runs docker exec <container> sh -c '<script>' and records the result.

Closes #4083

@melmennaoui
melmennaoui requested a review from a team as a code owner August 31, 2026 15:54
@aheritier aheritier added the kind/feat PR adds a new feature (maps to feat:). Use on PRs only. label Aug 31, 2026
Add runVerifyScript() to the evaluation package. It executes a shell
script via `docker exec` on the eval container after the agent run
completes, enabling filesystem-level and side-effect checks that
cannot be expressed as text assertions on the response.

- Zero exit code → pass; non-zero → fail
- 60-second timeout prevents runaway scripts
- stdout + stderr captured, capped at 8 KB
- Empty script is a no-op pass (verify is optional)
- Fake-runtime test helper avoids a real Docker daemon

Closes #4083
@melmennaoui
melmennaoui force-pushed the feat/4083-verify-script branch from 5549027 to d028a9e Compare August 31, 2026 16:02
@aheritier aheritier added the area/testing Test infrastructure, CI/CD, test runners, evaluation label Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/testing Test infrastructure, CI/CD, test runners, evaluation kind/feat PR adds a new feature (maps to feat:). Use on PRs only.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add verify script for post-agent outcome verification

2 participants