feat: grader file format with test-style assertions

README.md

@@ -3,15 +3,18 @@
 `promptdiff` is a small Bun CLI for testing whether an LLM prompt or skill
 change actually changes behavior — with any model, not just Claude.
 
-It has four commands:
+It has five commands:
 
 - `run`: one bounded model invocation with an agent file and optional inlined
   skill files.
 - `compare`: N-run baseline-vs-proposed comparison from a JSON scenario file,
-  with deterministic text or command graders (plus calibrated LLM judges).
+  with deterministic text, json, command, or grader-file graders (plus
+  calibrated LLM judges).
 - `measure`: N-run single-arm characterization — pass rates, no assertions.
 - `calibrate`: prove a judge grader against labeled rubric fixtures before
   compare/measure will let it grade anything.
+- `grade`: run one named grader from a grader file by hand, for example
+  inside a kept sandbox.
 
 Model access goes through pluggable runners. Two ship today:
 
@@ -334,6 +337,11 @@ Reading results:
 - Declare `"productionModel": "gpt-5.5"` and any arm testing a different
   model gets a warning on the summary — a pass on the wrong model validates
   prompt logic, not production behavior.
+- Runs whose grader reported [no artifact](#no-artifact) are left out of the
+  pass rate and counted on their own (`1/1 pass (100%), 1 no-artifact`),
+  with a `NOTE` naming each arm's count. An arm with no graded run at all
+  fails a target or regression assertion, since it has no pass rate to
+  compare.
 
 ### Raw per-run results
 
@@ -462,14 +470,24 @@ per-case pass rates, no delta and no assertions:
 ```
 
 The scenario file is the compare format — render vars, images, pricing,
-`productionModel`, and both grader types all apply — but only `skills` (or
+`productionModel`, and every grader type apply — but only `skills` (or
 `baselineSkills`) is required; no proposed arm. An explicit `[]` is valid and
 measures the agent with no skills. Don't fake this with
 identical compare arms: two identical arms at small n routinely produce
 verdicts like `FAIL: proposed regressed below baseline` out of pure sampling
 noise. `measure` exits 0 whenever the runs complete — a measurement has no
 pass/fail.
 
+A run where the agent wrote nothing is not a pass and not a reproduction.
+When the grader reports [no artifact](#no-artifact), the summary line counts
+it separately, and the pass rate covers only the graded runs:
+
+```
+reproduces-defect
+  4/4 pass (100%), 1 no-artifact | $0.4210
+  run 3 no artifact: 02_debating/edit_plan.draft.json does not exist in the grader's working directory
+```
+
 ## Comparing models
 
 The A/B variable does not have to be the skill text. Hold the prompt and
@@ -683,6 +701,113 @@ scenario so no tools are demanded, and script against the file:
 }
 ```
 
+### Grader files
+
+A grader file holds named graders for one artifact. Most real graders are
+semantic, so lead with `result.assert`: plain code for the condition, a
+message for when it fails, and the data it judged as context.
+
+```ts
+// grade.eval.ts
+import { grade } from "@theaiteam/promptdiff";
+
+type Plan = { a_roll: Array<{ source_in: number; source_out: number }> };
+const keptSeconds = (plan: Plan) => plan.a_roll.reduce((sum, s) => sum + (s.source_out - s.source_in), 0);
+
+export default grade("02_debating/edit_plan.draft.json", {
+  "product-named": ({ result }) => {
+    const kept: Plan["a_roll"] = result.json().a_roll;
+    result.assert(
+      kept.some((s) => s.source_in <= 8.0 && 8.0 <= s.source_out),
+      "no kept range covers t=8.0",
+      { ranges: kept },
+    );
+    result.shouldHave("cinema cutie");
+    result.shouldNotHave("lorem");
+  },
+  "still-edits": ({ result }) => {
+    const kept = keptSeconds(result.json());
+    result.assert(kept / 64.672 <= 0.9, `kept ${kept}s of 64.672s: keep-everything degeneration`);
+  },
+});
+```
+
+The scenario names the file and the grader:
+
+```json
+"grader": { "file": "./grade.eval.ts", "name": "product-named" }
+```
+
+`result` offers:
+
+- `result.assert(condition, message, context?)`: passes when `condition` is
+  truthy. On failure it prints `message` and `context` as JSON. Use it for
+  anything a substring cannot express: ranges, counts, ratios, structure.
+- `result.json()`: the artifact parsed as JSON. Invalid JSON fails the
+  grader with the parse error.
+- `result.text` and `result.path`: the raw contents and the path as given.
+- `result.shouldHave(str)`, `result.shouldNotHave(str)`,
+  `result.shouldMatch(regex)`: literal checks. A failure prints what the
+  artifact held instead (its start, or the text around a forbidden match).
+
+Every check returns whether it passed and never stops the grader, so one
+run reports every failure. An exception in the grader (a bug, or `json()` on
+invalid JSON) is one more failure; checks made before it still count. A
+grader that makes no checks fails.
+
+How it runs:
+
+- The artifact path resolves against the grader's working directory: the
+  run sandbox, or the grader's `cwd` inside it, the same place a command
+  grader runs. Absolute paths work too. To grade the run's final text output
+  instead of a file, pass `process.env.PROMPTDIFF_OUTPUT_FILE!`.
+- A file grader is a command grader underneath. The engine runs
+  `promptdiff grade --file <file> --name <name>` in the sandbox, with the
+  same exit-code contract: 0 pass, 1 failed checks (all printed to stderr),
+  77 [no artifact](#no-artifact). Run that command yourself inside a
+  `--keep-sandbox` directory to debug a grader without new model runs.
+- The grader's last stderr line (`grade "product-named" (...): 2 of 3 checks
+  failed`) becomes the run's message in the summary, followed by the
+  failure lines.
+- The file and the name are checked when the scenario loads, before any paid
+  run: a missing file, a default export that is not `grade(...)`, or an
+  unknown name fails with the names the file does define.
+- Optional fields: `"cwd"` and `"timeoutMs"`, as for command graders. Like
+  command graders, grader files need a tool-capable runner (claude-p) unless
+  the scenario sets `"mode": "text"`.
+- The import resolves to the running promptdiff under `promptdiff grade`, so
+  the file works without the package installed beside it. `import { grade }
+  from "promptdiff"` works too. Install `@theaiteam/promptdiff` as a dev
+  dependency for editor types.
+- `compare --cache` keys on the grader file's content, so editing a check
+  misses the cache. Files the grader imports are not hashed.
+
+[examples/07-grader-file](./examples/07-grader-file/) runs two semantic
+graders against sample schedules with no model run.
+
+### No artifact
+
+A run where the agent wrote nothing gives a grader nothing to judge. That
+is its own outcome, `no-artifact`, distinct from a failed check: counting it
+as a failure inflates reproduction rates, and a negative check
+(`shouldNotHave`) against a missing file would otherwise pass.
+
+- File graders report it on their own when the artifact path does not exist.
+- Command graders report it by exiting 77. The code is also in
+  `$PROMPTDIFF_NO_ARTIFACT_EXIT_CODE`:
+
+  ```sh
+  test -f draft.json || exit "$PROMPTDIFF_NO_ARTIFACT_EXIT_CODE"
+  ```
+
+  A command grader with `"expectExitCode": 77` keeps treating 77 as a pass.
+  Before this existed, exit 77 was an ordinary failure; it is still never a
+  pass.
+
+`measure` and `compare` exclude no-artifact runs from pass rates and show
+the count on the summary line (`4/4 pass (100%), 1 no-artifact`). Reports
+and receipts carry a `noArtifact` count when it is non-zero.
+
 ## Judge graders (calibrated)
 
 Some judgments cannot be expressed as string checks — "is this the
@@ -770,8 +895,9 @@ Lessons from production compare runs, for scenario authors:
 ## Security note
 
 Command graders execute arbitrary shell commands from scenario files (inside
-the per-run sandbox, but with your local permissions). Only run scenario files
-you trust.
+the per-run sandbox, but with your local permissions). Grader files are code
+too: loading a scenario imports each one to check its grader names. Only run
+scenario files you trust.
 
 ## Development
 

SPEC.md

@@ -244,6 +244,54 @@ output is also written into the sandbox and exposed to command graders as
 `$PROMPTDIFF_OUTPUT_FILE`, so completion-only runs can be command-graded
 (scenario `mode: "text"` keeps the tools demand at zero).
 
+**Grader files.** The command contract (run in the sandbox, exit code is the
+verdict, stderr is the diagnostics) stays the base layer; any executable in
+any language can grade. A grader file is a layer on top of it for the common
+case of checking one artifact from TS/JS. The file's default export is
+`grade(artifactPath, { name: ({ result }) => { ... } })`, and a scenario
+selects one grader with `{ "file": "./grade.eval.ts", "name": "..." }` (no
+`type` needed; `"type": "file"` is accepted). `result` offers
+`assert(condition, message, context?)` for semantic checks, `json()`, `text`,
+`path`, and the literal checks `shouldHave`, `shouldNotHave`, and
+`shouldMatch`. `assert` is the primary check in docs and examples on
+purpose: the strongest graders in use are semantic (range coverage,
+calibrated regex rulers), and substring sugar alone would push authors
+toward brittle string matching.
+
+The engine compiles a file grader to a command grader:
+`<bun> <promptdiff> grade --file <abs path> --name <name>`, run in the same
+sandbox cwd with the same timeout handling. `promptdiff grade` resolves the
+artifact against its cwd (the sandbox, or the grader's `cwd` inside it),
+reads it, runs the named grader, and collects every check rather than
+stopping at the first failure. It exits 0 when all checks pass, 1 when any
+fail (each failure on one stderr line with its context, then a one-line
+verdict that becomes the run's grade message), 77 when the artifact does not
+exist, and 2 when the file or name is unusable. A grader that makes no
+checks fails. An exception inside a grader is recorded as one more failure.
+Under `promptdiff grade`, the specifiers `@theaiteam/promptdiff` and
+`promptdiff` resolve to the running copy through a Bun runtime plugin, so a
+grader file needs no local install and cannot drift from the runner's
+version. Scenario loading checks each file and name in a child process
+(`promptdiff grade --list`) before any paid run. The baseline cache key
+includes the grader file's content hash (not the files it imports).
+
+**No artifact.** A grader that finds nothing to grade reports `no-artifact`,
+a third outcome beside pass and assertion-failed. File graders report it
+when the artifact path does not exist. Command graders report it by exiting
+77 (also exported as `$PROMPTDIFF_NO_ARTIFACT_EXIT_CODE`), unless their
+`expectExitCode` is 77, which keeps its old meaning. No existing command
+grader changes between pass and fail: an exit 77 was a failure before and is
+still not a pass. A no-artifact run is never a pass, so a negative check
+cannot pass on an agent that produced nothing, and it is not counted as a
+failure, so reproduction rates are not inflated by runs with no signal.
+`ArmSummary.noArtifact` counts these runs; `passRate` is passes over graded
+runs (`totalRuns - noArtifact`), and Fisher's p uses the graded counts.
+Summaries print `4/4 pass (100%), 1 no-artifact` and list each such run.
+`compare` adds a `NOTE` with both arms' no-artifact counts whenever either
+is non-zero, and fails target and regression assertions for an arm with no
+graded run at all, which has no pass rate to compare. An arm that produces
+the artifact less often is noted, not failed.
+
 LLM judges are implemented, WITH mandatory calibration, for judgments that
 cannot be expressed as local checks (semantic and style rubrics that regex
 both under- and over-catches). The design premise is that an uncalibrated
@@ -282,6 +330,8 @@ promptdiff          # Bun executable shim
 src/cli.ts          # command parsing and user-facing orchestration
 src/args.ts         # strict local flag parser
 src/prompt.ts       # frontmatter stripping and prompt assembly
+src/index.ts        # package entry for grader files (exports grade)
+src/grade.ts        # grader-file authoring API and check collection
 src/runner/         # provider-coupled runner code
 src/engine/         # compare loop, config loading, graders, sandbox lifecycle
 test/               # Bun tests
@@ -310,8 +360,10 @@ always runs fresh. Deleting the cache directory busts it.
   decisive effects at small N, but there is no statistical test yet.
 - Scenario authoring is manual JSON. A future tuning loop should generate these
   files from accepted findings.
-- Command graders run trusted local commands from scenario files. Do not run
-  untrusted scenario files.
+- Command graders run trusted local commands from scenario files, and loading
+  a scenario imports its grader files. Do not run untrusted scenario files.
+- The cache key hashes a grader file but not modules it imports; editing a
+  helper module alone does not bust the baseline cache.
 - The openai runner is a single completion per run: no tool use, so it can only
   answer text-graded questions. Tool-using evals on non-Claude models would
   need an agentic runner (e.g. wrapping another agent CLI).

examples/07-grader-file/README.md

@@ -0,0 +1,111 @@
+# Grader file
+
+A grader file replaces a hand-written grader script: one TS file, named
+graders, and a `result` object with checks. promptdiff reads the artifact,
+parses the JSON, runs every check, prints each failure with the data it was
+judged on, and turns the outcome into the command-grader exit code.
+
+The skill under test plans four meetings into one day and writes
+`schedule.json`. The two graders in [`grade.eval.ts`](./grade.eval.ts) are
+semantic: "no two meetings overlap" and "nothing lands in the lunch block"
+are time-range math, which no substring check can express. So both lead
+with `result.assert(condition, message, context)`, and `shouldHave` appears
+once, for the one literal thing worth checking (Priya's name).
+
+```ts
+export default grade("schedule.json", {
+  "lunch-stays-free": ({ result }) => {
+    const meetings: Meeting[] = result.json().meetings;
+    const clashes = meetings.filter((m) => minutes(m.start) < minutes("13:00") && minutes(m.end) > minutes("12:00"));
+    result.assert(clashes.length === 0, "a meeting overlaps the 12:00-13:00 lunch block", { clashes });
+  },
+});
+```
+
+The scenario references each grader by name:
+
+```json
+"grader": { "file": "./grade.eval.ts", "name": "lunch-stays-free" }
+```
+
+**Cost:** free for the grader runs below. The `measure` run (8 haiku runs
+in artifact mode) was not run for this README, so its cost is not measured.
+**Requires:** bun; the claude CLI for `measure`
+
+## Try the graders without a model run
+
+`promptdiff grade` is the command every file grader compiles to. It resolves
+the artifact path against the current directory, which inside a real run is
+the sandbox. The `samples/` directories hold hand-written schedules, so you
+can check a grader before paying for any runs.
+
+A clean schedule:
+
+```bash
+cd examples/07-grader-file/samples/pass
+../../../../promptdiff grade --file ../../grade.eval.ts --name no-overlaps
+```
+
+```
+grade "no-overlaps" (schedule.json): 6 of 6 checks passed
+```
+
+A schedule with two defects. Each grader reports its own, with the meetings
+that caused it:
+
+```bash
+cd examples/07-grader-file/samples/lunch-clash
+../../../../promptdiff grade --file ../../grade.eval.ts --name no-overlaps
+```
+
+```
+FAIL 1:1 with Priya starts before Design review ends | context: {"before":{"title":"Design review","start":"09:00","end":"10:00"},"after":{"title":"1:1 with Priya","start":"09:30","end":"10:00"}}
+grade "no-overlaps" (schedule.json): 1 of 6 checks failed
+```
+
+```bash
+../../../../promptdiff grade --file ../../grade.eval.ts --name lunch-stays-free
+```
+
+```
+FAIL a meeting overlaps the 12:00-13:00 lunch block | context: {"clashes":[{"title":"Hiring debrief","start":"12:00","end":"13:00"}]}
+grade "lunch-stays-free" (schedule.json): 1 of 1 checks failed
+```
+
+No schedule at all (run from `samples/`, which has no `schedule.json`):
+
+```bash
+cd examples/07-grader-file/samples
+../../../promptdiff grade --file ../grade.eval.ts --name no-overlaps; echo $?
+```
+
+```
+no artifact: schedule.json does not exist in the grader's working directory
+77
+```
+
+## Run it for real
+
+```bash
+./promptdiff measure --scenario ./examples/07-grader-file/scenario.json
+```
+
+This output was not captured for this README. Each scenario prints a line
+like `3/3 pass (100%), 1 no-artifact`.
+
+## What to notice
+
+- **Every failure is reported, not the first one.** A failed check records
+  itself and the grader keeps going, so one run shows every defect.
+- **No artifact is not a failure.** Exit 77 means the agent wrote nothing.
+  `measure` and `compare` leave those runs out of the pass rate and count
+  them on their own (`3/3 pass (100%), 1 no-artifact`), so a run that
+  produced nothing can neither pass a negative check nor count as a
+  reproduced defect.
+- **Context is the evidence.** The third argument to `result.assert` is
+  printed on failure. Pass the data the check looked at, and the summary
+  answers "why did run 2 fail" without a `--keep-sandbox` re-run.
+- The import is `@theaiteam/promptdiff`. Under `promptdiff grade` it
+  resolves to the running promptdiff, so the grader works without
+  installing the package next to it; install it as a dev dependency if you
+  want editor types.

examples/07-grader-file/agent.md

@@ -0,0 +1,7 @@
+---
+name: scheduler
+description: Example agent for promptdiff's grader-file example.
+---
+
+You are a scheduling assistant. You work in the current directory and write
+your results to files there. Follow the skill instructions exactly.