Skip to content

fix: accept "tools": "" in scenario files like --tools "" - #49

Open
queso wants to merge 1 commit into
mainfrom
fix/scenario-empty-tools
Open

queso wants to merge 1 commit into
mainfrom
fix/scenario-empty-tools

Conversation

@queso

@queso queso commented Sep 30, 2026

Copy link
Copy Markdown
Owner

Closes #37.

Cause

The scenario loader read tools, at the top level and per case, with stringValue, which rejects "". Its error, string value must be a non-empty string, did not name the field. --tools "" never went through that check. After config load, "" was already handled correctly: effectiveTools uses ??, the claude-p runner checks === "", and the cache key uses the tools value in effect.

Fix

A toolsValue helper for the two tools fields accepts any string, including "". A non-string value now gets an error that names the field:

t.tools must be a string: a comma-separated tool list, "default", or "" for no tools

"" means no tools at every level. The order is: per-case tools, then --tools or top-level tools, then the mode default. Install delivery still rejects "".

README and SPEC now describe the tools field and this order.

Behavior kept as is

  • A per-case tools wins over --tools, as before. This is now documented.
  • validateRunnerSupport looks only at the tools value in effect, so "tools": "" on an artifact-mode scenario passes on a text-only runner. --tools "" already behaved this way.

Testing

  • bun run check: 165 tests pass, typecheck clean.
  • New tests failed first with the old error:
    • the issue's shape ("mode": "text", "tools": "") loads, and runs the same as --tools "" on a text-only runner
    • in artifact mode, "" sends no tools
    • a per-case "" overrides a top-level list
    • non-string values get errors that name the field
    • install delivery rejects "" from the file

🤖 Generated with Claude Code

The scenario loader parsed `tools` with the generic non-empty string
helper, so `"tools": ""` failed with "string value must be a non-empty
string" while `--tools ""` was accepted. Parse `tools` at both the top
level and per case with a helper that accepts "" and names the field in
its error. An explicit "" already flowed through effectiveTools, the
runner, and the cache key unchanged, so it now means no tools from every
source. Install delivery still rejects it.

Fixes #37

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick review — approve

The change adds file-grader support, JSON path assertions, strict template rendering, and skill installation with careful edge-case handling (null-prototype vars, own-property checks, reserved exit codes), backed by dedicated test coverage. No correctness, security, or performance risks surfaced in the cited code. Approving.

0 inline comment(s).

Verdict was approve, posted as a comment — this repo does not allow GitHub Actions to approve PRs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scenario JSON rejects "tools": "" while the CLI accepts --tools ""

1 participant