Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
6921ae9
feat(evals): mock the very-good-cli MCP server
ryzizub Sep 18, 2026
5d1ff85
docs(evals): record what the mock and bloc runs measured
ryzizub Sep 18, 2026
42c21b2
fix(evals): only guard mock inputs the real schema requires
ryzizub Sep 18, 2026
7bbb6fd
docs(evals): drop the startup-notice workaround from the one prompt t…
ryzizub Sep 18, 2026
c16c92c
fix(evals): make the six weakest cases discriminate
ryzizub Sep 18, 2026
3527eea
docs(evals): scope the mock claims to eval runs
ryzizub Sep 18, 2026
36ee305
docs(evals): drop the rest of the harness insulation from the create-…
ryzizub Sep 18, 2026
6f2ee87
docs(evals): correct the mock findings against the published reference
ryzizub Sep 18, 2026
ae7ec2b
docs(evals): note that real servers can be started, and match the ref…
ryzizub Sep 18, 2026
2b99785
docs(evals): trim the tool-execution gap to the gap
ryzizub Sep 21, 2026
c6d3e4a
docs(evals): record the full one-arm run and walk back the bloc claim
ryzizub Sep 21, 2026
f7c78a4
docs(evals): cut the process narrative out of the eval docs
ryzizub Sep 21, 2026
d0822ac
docs(evals): rewrite both eval docs as reference, not narrative
ryzizub Sep 21, 2026
c89dca1
fix(evals): delete BASELINE.md and fix the case it flagged at 0.61
ryzizub Sep 21, 2026
53bf3ff
feat(evals): grade license-compliance on the call it makes, not the p…
ryzizub Sep 21, 2026
d8613c3
feat(evals): grade every very-good-cli case on the call, not the prose
ryzizub Sep 21, 2026
41beba2
feat(evals): assert the create call's placement, and that tools stay …
ryzizub Sep 21, 2026
46a6673
fix(evals): give the tool-driving cases room to make the call
ryzizub Sep 22, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions .github/workflows/evals.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -82,12 +82,14 @@ jobs:
echo "$CHANGED" | { grep -oE '^skills/[^/]+/' || true; } | cut -d/ -f2
echo "$CHANGED" | { grep -oE '^evals/[^_][^/]*/' || true; } | cut -d/ -f2
} | sort -u | while read -r s; do
[ -n "$s" ] && [ -d "evals/$s" ] && echo "$s"
if [ -n "$s" ] && compgen -G "evals/$s/*/case.yaml" >/dev/null; then
echo "$s"
fi
done
)

if echo "$CHANGED" | grep -qE '^evals/_fixture/'; then
echo "fixture changed — widening to all skills"
if echo "$CHANGED" | grep -qE '^evals/(_fixture|mocks)/'; then
echo "shared eval input changed — widening to all skills"
SKILLS=$(all_skills)
fi
fi
Expand Down
1 change: 0 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@ docs/ # Gitignored, local only
plan/ # Planning and design documents
evals/ # `claude plugin eval` suite — all 15 skills, 100 cases
README.md # Case format, grader reference, how to add a case
BASELINE.md # Last full two-arm measurement: per-skill Δ and routing
_fixture/
fixture.sh # The only copy of the neutral Flutter skeleton; every case symlinks here
<skill>/ # One directory per skill, named for the skill it covers
Expand Down
1 change: 1 addition & 0 deletions config/cspell.json
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
"Bienvenido",
"bypassable",
"Codex",
"compgen",
"CSPRNG",
"cupertino",
"Cupertino",
Expand Down
110 changes: 0 additions & 110 deletions evals/BASELINE.md

This file was deleted.

Loading
Loading