From aa0b119c8dacaf314099e88d4a42a1f2c72a9014 Mon Sep 17 00:00:00 2001 From: Adam Reznik Date: Tue, 3 Nov 2026 22:41:53 +0600 Subject: [PATCH 1/2] Add blind-test skill Co-Authored-By: Claude Opus 5.5 --- skills/blind-test/SKILL.md | 75 ++++++++++++++ skills/blind-test/agents/claude.yaml | 5 + skills/blind-test/agents/openai.yaml | 5 + skills/blind-test/references/build-tests.md | 45 +++++++++ skills/blind-test/references/calibration.md | 38 ++++++++ skills/blind-test/references/grading.md | 97 +++++++++++++++++++ skills/blind-test/references/key-format.md | 69 +++++++++++++ skills/blind-test/references/levels.md | 37 +++++++ skills/blind-test/references/report.md | 29 ++++++ skills/blind-test/references/runs.md | 85 ++++++++++++++++ skills/blind-test/references/target-design.md | 58 +++++++++++ skills/blind-test/scripts/blind-labels.mjs | 44 +++++++++ skills/blind-test/scripts/refcheck.mjs | 62 ++++++++++++ skills/blind-test/scripts/score.mjs | 77 +++++++++++++++ skills/blind-test/scripts/target-server.mjs | 60 ++++++++++++ skills/blind-test/templates/key.template.json | 37 +++++++ .../blind-test/templates/report.template.md | 43 ++++++++ 17 files changed, 866 insertions(+) create mode 100644 skills/blind-test/SKILL.md create mode 100644 skills/blind-test/agents/claude.yaml create mode 100644 skills/blind-test/agents/openai.yaml create mode 100644 skills/blind-test/references/build-tests.md create mode 100644 skills/blind-test/references/calibration.md create mode 100644 skills/blind-test/references/grading.md create mode 100644 skills/blind-test/references/key-format.md create mode 100644 skills/blind-test/references/levels.md create mode 100644 skills/blind-test/references/report.md create mode 100644 skills/blind-test/references/runs.md create mode 100644 skills/blind-test/references/target-design.md create mode 100644 skills/blind-test/scripts/blind-labels.mjs create mode 100644 skills/blind-test/scripts/refcheck.mjs create mode 100644 skills/blind-test/scripts/score.mjs create mode 100644 skills/blind-test/scripts/target-server.mjs create mode 100644 skills/blind-test/templates/key.template.json create mode 100644 skills/blind-test/templates/report.template.md diff --git a/skills/blind-test/SKILL.md b/skills/blind-test/SKILL.md new file mode 100644 index 000000000..49ff05d58 --- /dev/null +++ b/skills/blind-test/SKILL.md @@ -0,0 +1,75 @@ +--- +name: blind-test +description: Prove whether an agent skill, harness, prompt, rule file or plugin actually changes what an agent does, with a blind test scored against a hidden answer key. Builds a fictional target with planted problems and traps (legitimate decisions that look like mistakes), writes the key before any run, runs the same request with and without the skill in isolated agents, grades the outputs without knowing which arm produced them, and reports what changed, what didn't and what the numbers can honestly claim. It has two levels, a quick check by default and the full test with calibration and normalization when the numbers go public. Use when someone asks "does my skill work", "is it any good", "is it better than the free ones", "prove it before I publish", wants numbers for a skill listing, or is about to tune a skill by feel. +summary: Blind tests for agent skills, scored against a hidden key, with traps that catch false alarms. +version: 1.3.0 +tags: [agent-skills, evaluation, blind-test, ab-test, benchmarking, claude-code, prompts, agent-harness] +--- + +# Blind test + +A skill is a claim: "with me, the agent does X better." A blind test checks the claim the way a careful reviewer would. The agent gets a target and a request. The answer — a key of planted problems and traps — is written before any run and kept where the agents can't see it. Every output is graded against the key, without knowing which arm produced it. + +Two things make this different from asking a model to judge two outputs: + +- **The key is ground truth.** A finding either matches a planted problem or it doesn't. A judge's taste doesn't enter. +- **Traps measure false alarms.** A skill that makes the agent flag everything finds more problems and is worse. Traps are legitimate decisions that look like mistakes; reporting one as a failure counts against the arm. + +## 0. Before anything + +Find out, from the files first and the user second: + +1. **The claim.** One sentence per thing the skill says it changes: "finds data that disagrees with itself", "doesn't report 44 px as a WCAG AA failure", "builds lists that survive ten times the items". No claim, no test: a test without claims measures noise. +2. **The test type.** *Review* (the agent finds problems in something that exists) or *Build* (the agent makes something from a brief). Many skills need both. Read `references/build-tests.md` for Build. +3. **The arms.** Baseline (same agent, no skill) always. Treatment (with the skill). Optionally a competitor skill, installed in a sandbox (`references/runs.md`). When the change is the harness itself, the arms are one harness with and without a setting, or two harnesses on the same model (`references/runs.md`, Harnesses). +4. **The model.** Start with the smallest model the skill is meant for: a skill's effect shows most where the model is weakest, and runs are cheap. Confirm on a larger one only if the result is borderline. +5. **The level** (`references/levels.md`). A *quick check* shows a direction: one target, 6–8 K, 2–3 T, the key and the decision rule before any run, two runs per arm, one blind grader, about five agents. The *full test* adds calibration (two baseline pilot runs, four when the target needs its revision), the rewrite of every output into one format before grading and, before a public number, three or more runs per arm; in our runs on Haiku it started 13–16 agents and took 90–110 minutes. Take the full test when the request asks for proof or for numbers that go public, and the quick check when it asks whether a skill is any good. Otherwise name both with their cost and ask. If you can't ask (you run as a subagent, or were told to go ahead without questions), take the quick check and say so in the first line of the report. At either level, one run per arm is a demo, two show the spread, three or more come before a published number; record tokens per run. + +## 1. Read what applies + +| If | Read | +|---|---| +| Always | `references/levels.md`, `references/target-design.md`, `references/key-format.md`, `references/runs.md`, `references/grading.md` | +| The full test | `references/calibration.md` | +| The skill builds things | `references/build-tests.md` | +| The numbers go into a listing, a post or a README | `references/report.md` | + +## 2. The rules of a fair test + +1. **The key is written before the first run** and never edited to match an output. A real problem found by an agent but not planted goes into the key as *found beyond the key*, credited to that run only. +2. **Every claim has at least one planted problem that discriminates.** If the baseline would find it anyway, it doesn't test the skill: keep it as a control, which shows the target is fair. Pilot runs of the baseline decide which is which (`references/calibration.md`); before them, and in a quick check that skips them, the roles are a guess. +3. **Every claim that could cause false alarms has a trap.** A rule against uppercase labels needs a legitimate uppercase label; a rule about 24 px targets needs a 20 px button with spacing. +4. **The target is fictional.** No real brand, product or person. Real names leak the answer from training data and drag third parties into a published test. +5. **Agents see only the target, the request and (in the treatment arm) the skill.** The key, other runs, this skill's files and, when the test is about the rendered result, the target's source stay out of their reach. +6. **The request is identical, word for word**, in every arm. Only the skill changes. If the request contains folklore (a common wrong belief the skill corrects), it's the same folklore in every arm. +7. **Nothing the agents do touches anything real.** Targets run on localhost; every state-changing request is logged (`scripts/target-server.mjs` writes `posts.log`), and a non-empty log is itself a finding. Every file the test writes stays inside the test folder, the runs' scratch files and your own probes and logs included: a file left in `/tmp` or in a session scratchpad counts as a side effect. +8. **Grading is blind.** Outputs are relabelled before grading (`scripts/blind-labels.mjs`); the grader learns which arm is which after the scores are written. Run paths are stripped before the labels. In the full test every output is also rewritten into one neutral format after them, so a skill's format doesn't name its arm (`scripts/refcheck.mjs` checks the rewrite lost no place); a quick check keeps each output's own format and says it may give the arm away. +9. **Fixes go to classes, then the test runs again** — on the same target to confirm, and on a fresh one before you publish, so the skill isn't tuned to one key. + +## 3. Loop + +The full test runs every step. A quick check skips step 4, keeps two runs per arm in step 5 and skips the rewrite in step 6 (`references/levels.md`). + +1. **Claims → key.** Write the claims, then for each claim the planted problems (K), traps (T), and the folklore line for the request (`references/key-format.md`). Mark which K are controls (a strong baseline should find them) and which discriminate, as a first guess; calibration (step 4) settles it. Write the decision rule now, before any run: what result means the skill helps ("treatment finds at least two more discriminating K than baseline in every run, with no more traps flagged"), and the calibration rule with it. Keep `key.json` outside every folder an agent can read. +2. **Target.** Build it from the key (`references/target-design.md`): realistic, internally consistent, no comments or names that give a planted problem away. Serve it with `scripts/target-server.mjs` or hand over a folder copy. +3. **Check the target before any run.** Every K is actually present and observable; every T is actually legitimate; nothing else is accidentally broken (or, if it is, it's in the key as R, a real issue). +4. **Calibrate** (full test; `references/calibration.md`). Two pilot runs of the baseline, with the final prompt, graded against the key. Every K a pilot finds or half-finds becomes a control; the rest discriminate. Too few discriminating items means the target is too easy: revise it once, where the pilots' attention was thin, check it again and run two new pilots. Then freeze the key and write its sha256 next to the decision rule. Pilots don't count in the result. +5. **Runs.** Same request, isolated agents, one folder per run (`references/runs.md`). Confirm each treatment run actually used the skill; record tokens and time. +6. **Blind grading** (`references/grading.md`). Strip run paths from the outputs and relabel them; in the full test, rewrite each into one neutral format and check with `scripts/refcheck.mjs` that no place was lost. Then grade each against the key: found, partial, missed; traps reported as failures; folklore corrected or repeated; side effects. Then unblind. +7. **Score.** `scripts/score.mjs key.json grades.json` → the table per arm, and the classes only the treatment found. +8. **Fix and rerun.** Misses and false alarms become fixes to the skill, written as classes. Rerun the same arms. Before publishing, one more run on a target the skill was never tuned on. +9. **Report** (`references/report.md`): what was tested, the calibration, the table, what only the skill found, what it missed, and the honest limits. + +## 4. When the result is bad + +A treatment that finds less, or flags more traps, than the baseline is the most useful result this test produces. Report it as it is, find the rule that caused it (an imperative without its exception is the usual suspect), fix it, rerun. Never drop a trap or soften the key to rescue a number. + +## 5. Hand-over + +- The level, and for a quick check what it skipped and what the full test would add. +- The claims, and which held. +- The calibration (full test): what the pilots found, which items became controls, and any revision of the target. +- The table per arm: planted found, partial, missed; traps flagged; folklore; side effects; tokens. +- What only the treatment found, what only the baseline found, what neither did. +- The fixes made to the skill and the rerun result. +- The limits, in one sentence, next to any number that leaves the room. diff --git a/skills/blind-test/agents/claude.yaml b/skills/blind-test/agents/claude.yaml new file mode 100644 index 000000000..e7be8efee --- /dev/null +++ b/skills/blind-test/agents/claude.yaml @@ -0,0 +1,5 @@ +interface: + display_name: "Blind Test" + short_description: "Prove a skill works with a keyed blind test" + skill_file: "skills/blind-test/SKILL.md" + default_prompt: "Use $blind-test to check whether this skill actually changes what the agent does." diff --git a/skills/blind-test/agents/openai.yaml b/skills/blind-test/agents/openai.yaml new file mode 100644 index 000000000..e7be8efee --- /dev/null +++ b/skills/blind-test/agents/openai.yaml @@ -0,0 +1,5 @@ +interface: + display_name: "Blind Test" + short_description: "Prove a skill works with a keyed blind test" + skill_file: "skills/blind-test/SKILL.md" + default_prompt: "Use $blind-test to check whether this skill actually changes what the agent does." diff --git a/skills/blind-test/references/build-tests.md b/skills/blind-test/references/build-tests.md new file mode 100644 index 000000000..64ee5fe71 --- /dev/null +++ b/skills/blind-test/references/build-tests.md @@ -0,0 +1,45 @@ +# Build tests + +For skills that make things — screens, code, copy — the target is a brief and data, and the key is a set of checks on what gets built. + +## The brief + +- **Short, the way people actually ask.** A one-line brief tests the skill's judgment; a long one tests the agent's reading. Run the short one; add a detailed one only if the skill claims to follow specs. +- **No hints at the answer.** "Make sure numbers come from the data" turns a test of the skill into a test of reading. +- **Say where the work goes and how it's checked** (the build command, the file name), so every arm can finish. + +## Data variants + +The build is checked against more data than the brief shows. Prepare, for each data file: + +- **typical** — the one in the brief; +- **many** — about ten times the items; +- **few or calm** — one item, nothing wrong, an empty list; +- **awkward** — the longest real name, duplicate names, missing values. + +Swap them in after the run, without telling the agent. + +## Checks + +Split them by how they're verified, and write them before the runs: + +| Kind | Example | How | +|---|---|---| +| **Measured** | no page scroll at the target viewport; contrast failures; overflow on a phone | a script or a render, a number per run | +| **Checked by hand** | every number computed from data; a fact told once; problems first | a yes/no with the line or region that shows it | +| **Traps** | the build may legitimately use uppercase labels in a table, glass on a floating bar | flagged only if the skill's rules would wrongly forbid them and the build avoided them at a cost | + +Avoid checks every arm passes: they don't discriminate and they pad the score. + +For a measured check on a render, capture the original twice with the same command first: the difference between the two captures is the noise floor, and a build differs only where it goes above it. Give every capture a fresh browser profile: with one profile shared across captures, a page captured later can be caught at another moment of its animation. + +## Same conditions + +- Same brief, same data, same project template, same tools — including a simulator or browser if any arm gets one. A skill whose loop says "render and look" can't be judged on a run that wasn't allowed to render. +- If the brief invites a review round ("review what you built, then improve it"), it's in every arm's brief. +- **Served code is part of the target.** In a rebuild or clone test, the page's code reaches every run: a minified bundle still carries shader sources, constants and timings, and a run without the skill can read and copy them. Decide before the runs whether reading the code is part of the task, say so in the request for every arm, and name it in the limits. +- Screenshots of every arm at the same viewport, theme and data, for the side-by-side. + +## Report + +A table of checks per arm, the side-by-side screenshots, and the variants that broke each build. Don't call a build "better" from a screenshot alone: the measured and hand checks are the result; the screenshot is the illustration. diff --git a/skills/blind-test/references/calibration.md b/skills/blind-test/references/calibration.md new file mode 100644 index 000000000..1141092a4 --- /dev/null +++ b/skills/blind-test/references/calibration.md @@ -0,0 +1,38 @@ +# Calibration + +The key marks each planted problem `control` (a strong baseline finds it) or `discriminating` (only the skill should). That mark starts as a guess, and guesses about what a model finds on its own are often wrong: in one code-review round all 12 items predicted as discriminating were found by the baseline in every run, and the test could show nothing. Calibration turns the guess into a measurement before the runs that count. + +## When + +In the full test (`references/levels.md`): after the target check (step 3 of the loop) and before the first counted run. Write the calibration rule into the decision file before the first pilot: what turns an item into a control, how many discriminating items the decision rule needs, and what happens when there are fewer. + +## Pilots + +1. **Two pilot runs of the baseline arm**, with the exact prompt the final baseline runs will get, each in its own folder (`pilot-1`, `pilot-2`, apart from the final runs; `references/runs.md`). Same model, same tools, same isolation. +2. **Grade them against the key** with the rules in `references/grading.md`. They don't need to be blind: their arm is known and they don't count in the result. +3. **Set the roles.** Every K that either pilot finds or half-finds gets `"role": "control"`; the rest get `"role": "discriminating"`. `score.mjs` counts its Discriminating column from `role`. +4. **Count.** Enough discriminating items when the decision rule can still be met and each claim the rule leans on keeps at least one. A rule that asks for "two more discriminating K than the baseline" needs at least three, and 6 or more give it room. + +## When the target is too easy + +Too few discriminating items means the baseline already does what the target asks. Revise the target once: + +- Keep 5–8 of the found items as controls, unchanged. They show the target stays fair, and an item kept unchanged keeps the role it earned in the first pilots. +- Fix the other found items in the target's code, so they stop costing the runs reading time. +- Add new problems where the pilots' outputs show their attention was thin: files they never opened, numbers they never ran, assurances they took on faith, tests they read only for familiar shapes. Each new problem is real, observable in what the agents get, and free of tells (`references/target-design.md`). +- Whoever revises the target reads the pilots' outputs and the key and stays away from the skill under test, so the new problems come from what the baseline missed. +- Check the revised target (step 3), then run two new pilots and set the roles again. + +One revision only. If the second calibration still leaves too few discriminating items, run the test on the revised target with the calibrated key, and say in the report that the decision rule could not be met with that many items. Each further revision tunes the target to the pilots. + +## Freeze the key + +Once the roles are set, write the key's sha256 next to the decision rule. From then on nothing in the key changes. Corrections found while grading the pilots (a trap whose text reaches too far, a stray detail in an item) go in before the freeze, apply to both arms alike, and are listed in the calibration record. + +## What to keep + +`calibration/`: each pilot's prompt and output (with its run path stripped), the grades table, tokens and time, the outcome (which items became controls, which stayed discriminating) and, after a revision, the old-to-new item ids with what was fixed and what was added. + +## Limits it adds + +Discriminating items are the ones two baseline runs missed. The final baseline runs are new runs and can still find them; that is part of the result, and the report says so. Calibration costs two baseline runs, four with a revision. A quick check skips it, and its report then says the roles are guesses. diff --git a/skills/blind-test/references/grading.md b/skills/blind-test/references/grading.md new file mode 100644 index 000000000..5ac588f15 --- /dev/null +++ b/skills/blind-test/references/grading.md @@ -0,0 +1,97 @@ +# Grading + +Grade every output against the key, without knowing which arm wrote it. + +## Make it blind + +1. **Strip what names the run.** Outputs quote their own folder (`/…/runs/treatment-2/target/src/app.ts`), and the folder names the arm. In a copy of each result, replace the run's folder path with `.`, shortened forms too (`…/treatment-2/`), then search the copies for every run name. +2. `node scripts/blind-labels.mjs stripped/ graded/` copies each run's stripped result into `graded/` under a random label (`A`, `B`, `C`…) and writes the label → run map to `graded/.labels.json`. A leftover found after labelling means a new labelling into a new folder, so the map stays unseen. +3. Grade only the files in `graded/`, after normalizing them (below). Don't open `.labels.json` until every grade is written. +4. If someone else grades, give them the key and the normalized outputs only; the run folders stay with you. Take `role` and anything that describes the arms out of their copy of the key: a grader who knows which items discriminate reads those harder. + +Outputs often reveal their arm (a skill's vocabulary, its report format). Normalizing removes most of that. Blindness is worth it either way: it stops you from reading generously the run you hope wins. + +## Normalize + +Full test. A quick check skips the rewrite, grades the stripped and relabelled outputs as they are, and says in its report that a skill's format may have given its arm away (`references/levels.md`). + +A skill shapes its output: its section names, its severity scale, marks like "verified". A grader who recognises the format knows the arm. Before grading, rewrite every labelled output into one neutral form: one agent per label, each in its own folder with that one file, so no rewriter sees another output. + +```text +Your working folder is . Read and write only inside it, temporary files included. Don't create tasks, tickets, pull requests, reminders or messages, and don't hand work to other agents. + +