evals/labelled/injection.jsonl holds 20 items and the injection_guard rail scores 20 of 20 on it. A set that size says little about precision and recall on real attacks.
Build a larger labelled set from the public benchmarks in the same JSONL shape (state.tool, state.text, label, note): InjecAgent (1,054 tool-output injections) and AgentDojo (97 tasks, 629 security cases). Then run the rail on it and add the precision and recall table to docs/benchmarks.md:
uv run s1a run injection_guard --slot jev --labelled-set evals/labelled/injection-public.jsonl
docs/roadmap.md § "Prompt-injection guard" has the plan and the two sources. Needs a Jev key (TYPESAFE_API_KEY or OPENROUTER_API_KEY).
evals/labelled/injection.jsonlholds 20 items and theinjection_guardrail scores 20 of 20 on it. A set that size says little about precision and recall on real attacks.Build a larger labelled set from the public benchmarks in the same JSONL shape (
state.tool,state.text,label,note): InjecAgent (1,054 tool-output injections) and AgentDojo (97 tasks, 629 security cases). Then run the rail on it and add the precision and recall table todocs/benchmarks.md:docs/roadmap.md§ "Prompt-injection guard" has the plan and the two sources. Needs a Jev key (TYPESAFE_API_KEYorOPENROUTER_API_KEY).