Skip to content

feat(evals): add MCP server mocks and fix the cases that measured nothing - #159

Merged
ryzizub merged 18 commits into
mainfrom
claude/determined-ptolemy-c2e085
Sep 22, 2026
Merged

feat(evals): add MCP server mocks and fix the cases that measured nothing#159
ryzizub merged 18 commits into
mainfrom
claude/determined-ptolemy-c2e085

Conversation

@ryzizub

@ryzizub ryzizub commented Sep 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

The eval suite had no MCP mocks, and several cases graded what the model said rather than what it did. This adds the mocks and moves every VGV-tool case onto the real call.

What changed

  • evals/mocks/very-good-cli/ — mocks for create, packages_get, test and packages_check_licenses, plus the real tools/list schemas. Eval runs only; a real session still reaches the real server.
  • Nine cases now grade a tool call instead of prose. Regexes looking for a tool's name become tool_used, tool_order and input_match on the actual arguments.
  • skills/bloc/SKILL.md gains one Core Standard: build: constructs the bloc rather than sharing one from setUp.
  • evals/BASELINE.md deleted — numbers nobody re-runs and nothing diffed against. Its three durable caveats moved to README.md.

Result

Full suite, one arm, claude-sonnet-5, $13.58:

before after
Mean case score 0.961 0.971
At or above threshold 94/100 98/100
Perfect 1.00 83/100 88/100

Four previously-weak cases roughly doubled their lift, and every converted case sits at 0.92 or better over --runs 3.

Per-case scores

Cases on mocks, --runs 3:

case score asserts
license-compliance-runs-check-with-full-license-info 1.00 calls the check, reacts to the returned scan
license-compliance-scopes-check-to-monorepo-subdirectory 1.00 directory names mobile
ui-package-scaffolds-with-app-ui-package-template 1.00 subcommand and output_directory
create-project-plans-dependency-install 1.00 create before packages_get
create-project-does-not-over-ask 1.00 was 0.75
create-project-stays-out-of-existing-project-work 1.00 create never called
ui-package-stays-out-of-plain-dart-work 1.00 create never called
testing-stays-out-of-non-test-work 1.00 test never called
create-project-asks-for-organization-when-required 0.95 no premature create, was 0.83
create-project-asks-when-the-template-is-ambiguous 0.92 no create while ambiguous

Graders reworked so they can fail without the plugin:

case before after
license-compliance-refuses-to-clear-missing-licenses Δ +0.43 Δ +0.86
internationalization-uses-directional-insets-for-rtl Δ +0.50 Δ +0.86
bloc-tests-with-bloc-test-and-mocktail Δ +0.38 Δ +0.75
layered-architecture-wires-repositories-in-bootstrap 0.61 1.00

For review

  • Negative assertions are the biggest win. "It did not scaffold" is now a fact from the trace rather than a judge's read of prose.
  • Δ is inflated on mocked cases. The tool is absent in the no-plugin arm, and nothing excludes those graders the way it does for tool_used: Skill. Read the with-arm score.
  • Tool-driving cases need max_turns: 20 and timeout_seconds: 900. At the usual 12 and 600 one breached both, and a cap breach scores 0 with no failing grader — it reads as a content failure.
  • green-gate is untouched. Its loop needs analyze_files and dart_format, so it belongs with the dart mocks.
  • bloc-writes-sealed-events-and-states is improved, not cured: 6 of 7 runs perfect against 1 in 3 before.
Cases deliberately not converted
  • Cases whose deliverable is code rather than a call.
  • create-project-scopes-dependency-install-to-the-new-project — its no-invented-organization grader requires asking, so requiring a call would contradict it.
  • static-security-scans-dependencies-before-release — about CVEs, not licenses.
  • license-compliance-refuses-to-certify-from-pubspec-alone — converted, then reverted: its prompt ends "I'd rather not run any scan", so requiring the call demanded the model override the user.
  • A case for the testing skill's test-tool standards was written and removed. Running a suite fires no skill, because green-gate's description claims "which tool and arguments run each gate" — so those two standards are unreachable where they live, and likely belong in green-gate.

🤖 Generated with Claude Code

@ryzizub
ryzizub marked this pull request as ready for review September 21, 2026 11:39
@ryzizub
ryzizub requested a review from a team as a code owner September 21, 2026 11:39
@ryzizub
ryzizub marked this pull request as draft September 21, 2026 11:40
@ryzizub ryzizub changed the title feat(evals): mock the very-good-cli MCP server, and make the weakest cases discriminate feat(evals): add MCP server mocks, and fix the cases that were not discriminating Sep 21, 2026
@ryzizub ryzizub changed the title feat(evals): add MCP server mocks, and fix the cases that were not discriminating feat(evals): add MCP server mocks and fix the cases that measured nothing Sep 21, 2026
ryzizub and others added 17 commits September 22, 2026 09:15
@ryzizub @claude
feat(evals): mock the very-good-cli MCP server
6921ae9 
Adds evals/mocks/very-good-cli/ with fixed mocks for create, packages_get,
test, and packages_check_licenses, plus the real tools/list response as
_tools.json so mocked tools carry their production schemas.

Registration is confirmed: a run now reports
  mocked: very-good-cli(create=fixed, packages_check_licenses=fixed,
          packages_get=fixed, test=fixed)

No case calls them yet. Every prompt still says the session cannot run
anything and every allowed_tools still lists only [Read, Glob, Grep, Skill],
so the tool-driven skills stay graded on narration until both change.

The SessionStart hook still fires under mocks. check_vgv_cli gates on
`command -v very_good`, not on MCP availability, so the three prompts that
tell the model to ignore the startup notice keep that sentence.

CI: evals/mocks/ is a shared input, so a change to it widens scope to all
skills the way _fixture/ does. The scope job previously emitted any changed
evals/ subdirectory as a --tag, which would have made `mocks` a tag matching
no cases; it now requires the directory to hold */case.yaml. That guard runs
as an if-statement because the && chain it replaced could return non-zero
from the loop body and trip `set -e`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ryzizub @claude
docs(evals): record what the mock and bloc runs measured
5d1ff85 
Measured on Claude Code 2.1.270, model and judge claude-sonnet-5.

bloc-writes-sealed-events-and-states: --runs 3 scored 1.00, 3/3, every
grader passing. The CI reading of 0.60 with four simultaneous failures was
run-to-run variance. Case, graders and skills/bloc/SKILL.md unchanged.

Two things block a mocked very-good-cli call, both before any grader runs:

1. A mocked tool needs an --allow-tools grant. The published docs say it does
   not. Listing it in a case's allowed_tools is necessary and not sufficient.
   The tool name is the bare mcp__very-good-cli__<tool> form.

2. check-vgv-cli.sh denies every mcp__*very-good-cli__* call in the sandbox.
   `command -v very_good` succeeds there, since the run inherits the host
   PATH; `very_good --version` returns nothing under the run's throwaway
   $HOME. check_vgv_cli reads that as not_installed and denies, so the mock
   is never reached.

Resolving 2 means changing what the hook gates on, which is shipped plugin
behavior, so it is documented rather than decided here.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ryzizub @claude
fix(evals): only guard mock inputs the real schema requires
42c21b2 
A driven call aborted the run at score 0:

  aborted by mock very-good-cli/packages_check_licenses: the model's call
  violates expect: directory = (missing) is not a string

`expect` treats a missing key as a violation, so listing an optional field
makes the mock reject correct calls. Only `create` has required fields
(`subcommand`, `name`); the other three tools have none, so their guards are
gone. Argument choices belong in a grader, not in `expect`, which aborts.

Verified against PR #155: with `unverifiable`, the PreToolUse hook stands
aside and the model reaches the mock, calling it with {"licenses": true}.
Before #155 the hook denied the call and the mock was never reached.

Also corrects the tool name in the docs. The bare mcp__very-good-cli__<tool>
form works for --allow-tools and allowed_tools, but the name the model
actually invokes, and the one a tool_used grader needs, is the
plugin-namespaced mcp__plugin_vgv-ai-flutter-plugin_very-good-cli__<tool>.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ryzizub @claude
docs(evals): drop the startup-notice workaround from the one prompt t…
7bbb6fd 
…hat needed it

With check_vgv_cli returning `unverifiable`, warn-missing-mcp.sh emits nothing,
so the sentence telling the model to ignore a startup notice has nothing left
to guard against. Removed from
create-project-scopes-dependency-install-to-the-new-project, which measured
3/3 at 1.00 afterwards, including the no-flutter-create grader that the notice
originally tripped.

The other two prompts named as carrying this workaround do not. They say the
session cannot reach a real toolchain, which is true regardless of the hook,
so they are unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ryzizub @claude
fix(evals): make the six weakest cases discriminate
c16c92c 
All numbers measured on claude-sonnet-5 for both model and judge.

Adds signal rather than cutting it. A grader that passes in both arms earns
no delta but still fails if the skill regresses later, so the free ones that
pin a Core Standard or an anti-pattern stay. Four graders are deleted, each
for a reason that holds without a score: `uses-pump-app` and
`names-all-rights-reserved` duplicate a rubric beside them,
`declares-weather-repository` restates the prompt, and `uses-test-widgets` is
table stakes for any widget test.

Two skills were at fault rather than their cases:

- skills/bloc/SKILL.md gains a Core Standard for `build:` constructing the
  bloc under test. That was the only real difference between the two arms of
  bloc-tests-with-bloc-test-and-mocktail, which moves Δ +0.38 -> +0.75.
- bloc-writes-sealed-events-and-states was grading one of two approaches the
  skill sanctions. The skill documents a Subclass and a Single Class approach
  and selects on whether states carry different data; the graders only accept
  the first, so a status-enum answer followed the skill and failed anyway.
  The prompt now supplies states that carry different data. Readings went
  1.00/0.70/0.60 on the unchanged case to 1.00/1.00/1.00 at --runs 3.

license-compliance-refuses-to-clear-missing-licenses graded several ways of
saying no, which any model says. It now also grades the skill's risk
categorization: Δ +0.43 -> +0.86 at --runs 3 on both arms.

testing-uses-pump-app-in-widget-tests referred to helpers the fixture does
not have, so the no-plugin arm sometimes asked a question instead of
answering. Its prompt is now self-contained.

Corrects the rule in BASELINE.md that one no-plugin run can disqualify a
grader. It cannot: accessibility-declines-gesture-detector-tap-target failed
all three content graders without the plugin on one run and passed all three
on the next.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ryzizub @claude
docs(evals): scope the mock claims to eval runs
3527eea 
"very-good-cli is mocked" read as a property of the plugin. The mocks exist
only for `claude plugin eval`: they are not shipped behavior, they do not
affect a real session, and a user's very-good-cli tools still reach the real
`very_good mcp` server.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ryzizub @claude
docs(evals): drop the rest of the harness insulation from the create-…
36ee305 
…project prompt

The prompt also told the model the CLI was installed and on PATH so it would
treat the tools as available. With the hook no longer denying, that is doing
nothing either: removing the whole sentence measured 3/3 at 1.00 on --runs 3.

A prompt should describe the user's situation, not work around the harness.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ryzizub @claude
docs(evals): correct the mock findings against the published reference
6f2ee87 
Re-read https://code.claude.com/docs/en/plugin-evals and re-tested each claim.

The grant claim was wrong. A mocked tool needs no --allow-tools grant and no
allowed_tools entry, exactly as the docs say. Verified with
allowed_tools: [Read, Glob, Grep, Skill] and no grant: the model still called
packages_check_licenses with {"directory": ".", "licenses": true}.

The earlier "not granted (missing --allow-tools grant, or a malformed entry)"
was the second half of that message. The bare mcp__very-good-cli__<tool> form
is not a valid entry in a case; the name is
mcp__plugin_<plugin>_<server>__<tool>. The runner cannot tell a missing grant
from a bad name, so it offers both.

Two behaviors are genuinely undocumented and are now recorded:

- `expect` treats a missing field as a violation, so naming an optional
  argument aborts a correct call. The reference presents it as a type guard.
- `_tools.json` is load-bearing. Renaming an argument in it and changing
  nothing else made the model call the tool with the renamed argument, which
  confirms both that it is parsed and that the saved `tools/list` result is
  the shape it wants.

Frontmatter is optional; three of the four mocks have none.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ryzizub @claude
docs(evals): note that real servers can be started, and match the ref…
ae7ec2b 
…erence's wording

A run starts no real MCP server *unless asked*, with --allow-real-servers or
--mocks off. The unqualified sentence read as if it were impossible.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ryzizub @claude
docs(evals): trim the tool-execution gap to the gap
2b99785 
The bullet restated the whole mock setup inside "What this does not cover",
which the Mocking the MCP servers section already covers. It now states the
limitation and links there.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ryzizub @claude
docs(evals): record the full one-arm run and walk back the bloc claim
c6d3e4a 
100 cases against this branch with #155, one arm, pinned to claude-sonnet-5:
mean 0.961, 94/100 at threshold, 83 perfect, $13.35, no usage errors.

bloc-writes-sealed-events-and-states was called fixed on one --runs 3. It
scored 0.70 in the full run with the corrected prompt in place, losing the
same three graders as before. Seven post-fix runs now read 1.00 six times and
0.70 once, against one in three before the fix, so it is improved and not
cured. Making it airtight means the skill naming which state approach a
request lifecycle with differing payloads takes.

Of the six cases under threshold, four were re-measured at --runs 3.
green-gate-refuses-to-carry-green-forward (0.91) and
create-project-asks-for-organization-when-required (0.83) clear or nearly
clear on average, ui-package-declines-hand-rolled-button hit the turn cap
rather than failing on content, and green-gate-budgets-per-package matches
its 2026-09-17 score exactly.

layered-architecture-wires-repositories-in-bootstrap is the one real finding
at 0.61, failing constructs-in-bootstrap on all three runs. It is untouched
by this branch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ryzizub @claude
docs(evals): cut the process narrative out of the eval docs
f7c78a4 
Both files had drifted into a record of how the work was done rather than
what a future reader needs.

BASELINE.md dropped a section headed "this branch with #155", a reference
that means nothing once merged, and a six-row triage table that belonged in
the pull request. The bloc case's history is now the finding and the caveat
rather than a blow-by-blow. The grader pass keeps its before/after numbers
and loses the prose around them.

README.md loses the forensics on why the hook denied mocked calls, which is
fixed rather than something to act on, and the account of an earlier draft
of the grader pass. The tool-naming trap is the same warning in a third of
the space.

BASELINE 202 -> 162 lines, README 450 -> 431.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ryzizub @claude
docs(evals): rewrite both eval docs as reference, not narrative
d0822ac 
Both had turned into an account of how the work was done. They are now what a
reader needs to act, and nothing else.

BASELINE.md is numbers: two run tables, cases below threshold, per-skill Δ,
the grader pass, and caveats as one line each. The case histories and the
essays around the tables are gone. 202 -> 111 lines, about what it was before
this branch despite carrying an extra run and a new table.

README.md keeps every fact that changes what you would do and drops the
justification around them. The mocking section is the layout, the frontmatter
keys, and four non-obvious behaviors as bullets. Graders that cannot fail is
the rule and three reasons to delete. 450 -> 354 lines.

No facts removed: tool naming, the expect-aborts-on-missing-field trap,
_tools.json, arm: both, weight: 3, mock_calls, the -printf portability note
and the measurements are all still there.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ryzizub @claude
fix(evals): delete BASELINE.md and fix the case it flagged at 0.61
c89dca1 
BASELINE.md recorded numbers nobody re-runs after a change, from single runs
noisy enough that reading them as a verdict is a mistake this branch made
twice. Nothing diffed against it and git already records measurements with
the context that makes them readable. Its three durable caveats move to
README.md: the no-plugin arm's swing between runs, the Sonnet judge, and
create-project's haiku pin. AGENTS.md drops it from the structure block.

The one finding it carried is fixed here.
layered-architecture-wires-repositories-in-bootstrap asked to wire
AuthRepository and UserRepository without saying they were local packages,
and the fixture has no packages/ directory, so the model either invented the
whole monorepo and passed or treated them as classes in lib/ and lost
constructs-in-bootstrap, path-dependencies and uses-repository-provider.

Naming the packages alone made it worse: the model checked the tree, found
nothing, and asked which it was instead of answering. The prompt now also
names the description authoritative over disk, the convention two other
prompts already use. At --runs 3 on both arms: with 1.00/1.00/1.00, without
0.33, Δ +0.67, up from 0.61 with three graders failing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ryzizub @claude
feat(evals): grade license-compliance on the call it makes, not the p…
53bf3ff 
…rose

The mocks had no consumer, so nothing exercised them in CI and their value
was unproven. license-compliance-runs-check-with-full-license-info now drives
one instead of describing it.

The prompt told the model it could not run anything and asked what it would
call. It now asks for the audit. Three graders that read prose are replaced:
a regex for the tool's name and one for `licenses: true` become tool_used on
the call and on its input, and a rubric about what the model said it would
retrieve becomes a rubric on whether the two packages the mock flags, a
GPL-3.0 and one with no license, reach the report with a risk level.

At --runs 3 on both arms: 1.00/1.00/1.00 with the plugin, 0.00 without.

That Δ is not content lift. A mocked tool is absent in the no-plugin arm and
nothing excludes these graders from the score the way it does for
tool_used: Skill, so the without arm fails them for free. README.md now says
so, and that the remaining tool-driven skills are still graded on narration.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ryzizub @claude
feat(evals): grade every very-good-cli case on the call, not the prose
d8613c3 
Eight cases now drive or assert a mocked tool, up from one. All measured at
--runs 3 with the plugin loaded.

Calls the tool:

  license-compliance-runs-check-with-full-license-info   1.00
  license-compliance-scopes-check-to-monorepo-subdirectory 1.00
  ui-package-scaffolds-with-app-ui-package-template      1.00  (was 0.88)
  create-project-plans-dependency-install                1.00
  create-project-does-not-over-ask                       1.00  (was 0.75)

Asserts the tool was NOT called, which is where mocks help most: "it did not
scaffold" becomes a fact from the trace rather than a judge reading prose.

  create-project-stays-out-of-existing-project-work      1.00
  create-project-asks-for-organization-when-required     0.95  (was 0.83)
  create-project-asks-when-the-template-is-ambiguous     0.92

Both improved cases improved for the same reason: the mechanical grader is
3/3 and the old rubric is what flaps.

green-gate is left alone. Its loop needs analyze_files and dart_format, and
converting only `test` would strand the model mid-loop. It belongs with the
dart mocks.

Not converted, having read them: cases whose deliverable is code rather than
a call, create-project-scopes-dependency-install-to-the-new-project, whose
no-invented-organization grader requires asking and so forbids the call, and
static-security-scans-dependencies-before-release, which is about CVEs rather
than licenses.

license-compliance-refuses-to-certify-from-pubspec-alone was converted and
reverted. Its prompt ends "I'd rather not run any scan", so requiring the
call demanded the model override the user, and it scored 0.62 three times for
that reason. Back to 1.00.

README.md gains the trap this pass hit four times: converting a case to drive
a tool invalidates every rubric that read the narration, because the model
stops describing the call and just makes it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ryzizub @claude
feat(evals): assert the create call's placement, and that tools stay …
41beba2 
…quiet out of scope

ui-package-scaffolds-with-app-ui-package-template now checks the create call
passes output_directory, which the skill teaches for a package in a monorepo
and nothing verified. It drops src-and-barrel-layout, which began failing 3/3
once the case drove the tool: the model stops describing the layout at length
and a blind judge sees nothing to judge. That convention is still covered by
ui-package-refuses-imports-from-src and
ui-package-adds-widget-with-barrel-export-and-test. Back to 1.00 at --runs 3.

Both negative controls now assert the tool was never called rather than only
that the skill stayed quiet, which is weaker: a skill can stay quiet while the
model still reaches for a tool. testing-stays-out-of-non-test-work and
ui-package-stays-out-of-plain-dart-work, both 1.00.

A case covering the testing skill's two `test` MCP tool standards, `directory`
for a monorepo package and `timeout_seconds`, was written and removed. It
scored 0.54 with `Skill called 0x` on every run: no skill fires on a request
to run a suite, because green-gate's description claims that ground, saying it
owns "which tool and arguments run each gate" and to prefer it over the
single-gate testing skill. Those two standards look misplaced in
skills/testing/SKILL.md, and green-gate is where they could be graded. Left
for the dart-mock work rather than widening a shipped description to make a
new case pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ryzizub
ryzizub force-pushed the claude/determined-ptolemy-c2e085 branch from ea55299 to 41beba2 Compare September 22, 2026 07:16
@ryzizub @claude
fix(evals): give the tool-driving cases room to make the call
46a6673 
A full run put ui-package-scaffolds-with-app-ui-package-template over both
the 12-turn cap and the 600s timeout. It scored 1.00 anyway, because the
graders were satisfied before the limits hit, which is luck rather than a
pass: a slower run fails before the call is made. Measured again at --runs 3
it takes 11, 15 and 14 turns.

A converted case does strictly more than the prompt it replaced. It routes,
calls the tool, reads the answer, then writes the reply. The four cases that
drive a tool now carry max_turns: 20 and timeout_seconds: 900, and README.md
records why, including that a cap breach scores 0 with no failing grader and
so reads as a content failure.

The two other cases under threshold in that run were both transient, and
neither is touched by this branch. At --runs 3,
green-gate-escalates-when-the-loop-stops-making-progress is 1.00 3/3 against
0.00, and ui-package-declines-hand-rolled-button is a consistent 0.86, which
clears the threshold. Both had turns=1 in the sweep.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ryzizub
ryzizub marked this pull request as ready for review September 22, 2026 08:26
@ryzizub
ryzizub merged commit 5e8518c into main Sep 22, 2026
5 checks passed
@ryzizub
ryzizub deleted the claude/determined-ptolemy-c2e085 branch September 22, 2026 11:00
@vgvbot vgvbot mentioned this pull request Sep 22, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@marcossevilla marcossevilla marcossevilla approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ryzizub @marcossevilla