Describe eight examples chosen to break the contract - #213
Closed
abernier wants to merge 4 commits into
Closed
Conversation
The pilot #192 asked for before deciding how much of the gallery to rewrite: eight examples picked for what they would break, not for being easy to write. Six were empty, two already had a line. pass-through-portals the technique is a prop -- `localClippingEnabled` on the renderer, `clippingPlanes` on the material, `THREE.Plane` through the namespace import. None of it is listable, so the prose carries it alone. water-shader no notable drei API: the ocean is three-stdlib's, registered as a JSX element by `extend`. react-ellipsecurve the smallest source in the sample, 2.5 kB. ssgi-spheres-... the largest, 515 kB, most of it a vendored copy of a postprocessing library the example imports from. transparent-aesop-... described already, and thinly -- 47 characters that named the material and missed the render-order pinning that keeps the backdrop behind the glass. bubbles described already, and 147 characters of it, spent listing five effects that its tags already carry. glass-flower empty, and one of the five the eval baseline missed. ecctrl-fisheye empty, and the miss the baseline could not even answer with a name. Two of them found a shape the spec had not: an identifier reached through a namespace import (`new THREE.MeshPhysicalMaterial`) cannot go in `apis` any more than a prop can, and a default import renamed at the call site (`import Controller from "ecctrl"`) would only put a meaningless local name there. Both are prose cases, which is one more than #192 expected. UNDESCRIBED loses six, 39 to 33. OVERLONG loses `bubbles`, five to four. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YQmbpRu6oCNrLiUQvhz5x6
One of the two "already described" slots the pilot was asked to fill, and the answer it came back with is no. The line it replaced -- "PBR transmission and thickness to emulate glas." -- names the technique in 47 characters and is the demo author's own. The rewrite named the render-order pinning instead and dropped the thickness, which is as load-bearing here as the transmission is: one true detail traded for another, for a line three times longer. The eval agrees, in the only way it can: no question moved on it. The glass questions were already answered by `diamond-refraction` and by `aquarium`. So the typo is fixed and the sentence is the author's again. `apis` stays, because it adds a field rather than overwriting one -- which is the whole difference between the two halves of this work. The other slot, `bubbles`, is not touched: 147 characters is over the bound, and the rewrite there is what the ratchet requires, not a matter of taste. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YQmbpRu6oCNrLiUQvhz5x6
19/20, and the four questions that flipped do not all belong to the eight lines: `aquarium` was written before the pilot and takes one of them, which `baseline.md` half-predicted by recording an aborted pass that answered it correctly on the old index. The file says so rather than quoting the delta -- a number whose provenance is not written down is a number nobody can check later. The miss left in place is the part worth keeping. Four of the baseline's five misses were taken and `faucets-select-highlight` was not; it is still missed, by the same near neighbour, which is what separates an index that got better from a sample drawn from its own answers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YQmbpRu6oCNrLiUQvhz5x6
abernier
changed the title
Three of the seven changes in this stack are tools with an end: the skill, the eval harness and the runs it recorded. They exist to get 170 examples described and explained, and a tool that outlives its job becomes a document that rots while claiming to be one. Each now says so in its own header, where you cannot open it without reading its expiry -- the pattern `description-exceptions.mjs` already set with "the schema tightens when this file is empty". What survives is what runs on its own and fails loudly: `apis ⊆ imports`, the description bound, the backticked-identifier check. Plus `questions.json`, because twenty hand-written answers are the expensive half of an eval and the only half a future run would need. That leaves the contract homeless, since it lived in the skill. It moves to `schemas/pmndrs.schema.json`, onto the two fields it governs -- which is where the author of a new example meets it, in an editor tooltip, years after this stack is forgotten. The skill keeps the long form and the worked example; the schema keeps the rule. These land on the top branch rather than on the branches that introduced each file: the alternative is restacking five PRs to place four comments. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YQmbpRu6oCNrLiUQvhz5x6
This was referenced Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #198. Last of the seven in #192's stack, and the only one that writes
prose rather than the machinery for it.
Eight examples described under the contract from #196, picked for what they
would break: a technique carried by a prop, one with no notable drei API, the
smallest source in the gallery, the largest, two that already had a line, and
two of the misses the baseline in #197 recorded.
apislands on all eight;UNDESCRIBEDgoes 39 → 33 andOVERLONG5 → 4, in the same commits.15/20 → 19/20, of which +3 belong to these lines and +1 to
aquarium, whichwas written before the pilot. The provenance, the fifteen controls and the two
runs are in
bin/eval/pilot-eight.mdrather than in a delta quoted here.faucets-select-highlightwas left empty on purpose and is still missed, by thesame near neighbour as before. That is deliberate: a sample covering all five
baseline misses would have scored better and shown less.
Three things the sample broke, all of them findings rather than failures:
apis ⊆ importshas two more ways to be unsatisfiable than the spec named — anamespace import (
new THREE.MeshPhysicalMaterial) and a default importrenamed at the call site (
import Controller from "ecctrl"). Both are prosecases, like the prop case already was.
ssgi-spheres-with-rapier-physicsis 515 kB, of which 506 kB is a checked-inbuild the example imports from.
transparent-aesop-bottleswas rewritten and then reverted to its author's sentence — the rewrite
traded
thickness: 500for the render-order pinning, at three times thelength, and moved no question.
bubblesis the other half of that slot andstays rewritten: 147 characters is over the bound, which is not a matter of
taste.
The decision this unblocks is in
the issue:
neither "fill the 40" nor "rewrite the 170", but the ~81 that fail a test
written down in advance — empty, over the bound, carrying an identifier or a
URL, or under 40 characters — leaving the 89 that pass untouched.
🤖 Generated with Claude Code
https://claude.ai/code/session_01YQmbpRu6oCNrLiUQvhz5x6