Ask a model which example llms.txt sends it to

[abernier]

Human review answers "is this sentence true and well turned". It does not
answer the only question the index line exists for: does an agent asking
about refractive glass land on the right example? aquarium · #transmission passes review and is useless -- it shares its one tag with a
dozen other lines, carries no words to tell it from them, and is not a
refraction demo anyway.

So ask. Twenty realistic questions, each with the example a maintainer
would accept, written by hand -- that oracle is the expensive part and the
part no script can produce. For each, a model is handed the contents of
llms.txt and nothing else, names the one example it would open, and the
score is how often that name is in expected.

Today's index scores 15/20. All five misses are examples whose
description is empty: the reader either picks a near neighbour that does
carry words, or gives up on the index and starts writing code. Ten of the
twenty questions point at that population, so the pilot has room to move
the number; the other ten are the control.

This has to be measured now. The PRs in front of it change no description,
so the baseline they leave behind is still today's -- but the moment eight
descriptions are rewritten, the "before" number is gone for good.

Never in CI. It calls a model, so it is non-deterministic, costs money and
needs the network -- the same reasoning that keeps a model out of the bin/
scripts the build depends on. Nothing here is wired into a turbo task, a
workflow, or a package script.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com

---

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}

---

Closes #197. Position 3 of the stack, on top of #203.

Baseline: 15/20, recorded in bin/eval/baseline.md. openai/gpt-oss-120b via the Vercel AI Gateway, temperature 0, one call per question. Not a Claude number: the workspace is on the gateway's free tier, which returns 403 RestrictedModelsError for every frontier model. --model overrides it once the workspace has credits, and the script needs no change.

Two things the number depends on, both stated in baseline.md: a score is only comparable within one model, and the noise floor is ±1 rather than zero — temperature: 0 is the least noise on offer, not determinism.

pnpm check 5/5 · pnpm test:turbo 55/55.

🤖 Generated with Claude Code