T-ENG05: parameterize eval targets, surface thresholds, add run-evidence runbook

[emmanuelgjr]

Plan ID: ENG-05            Ticket: T-ENG05            Wave: W4
Constraints honored: C4 (no threshold or severity authored — every value stays the profile author's, marked DRAFT — SME review required), C1 (no names added), C2 (webapp untouched)
Files touched: evals/ (26: 13 garak, 6 pyrit, laaf ×3, run_all.sh, ci template, README, THRESHOLDS.md, samples/README.md) + scripts/evals.test.mjs (2 new files)
Deliberately NOT changed: no probe/payload logic; no committed threshold default; no fabricated run committed (runbook + needs-human-run issue instead); no data/ or docs/
Human decisions/approvals needed: (1) run one sanitised pass per track → needs-human-run issue #39; (2) SME review of every threshold in evals/THRESHOLDS.md
Verify:
  $ node --test scripts/evals.test.mjs      → 11 tests, 11 pass
  $ npm test                                → 59 tests (58 pass + 1 pre-existing exports.test.mjs parallel flake, see below)
  $ node scripts/validate.js                → 0 error(s), 84 warning(s)  ✓ PASSED
  $ npm run stats:check                     → stats.json + README current
  $ bash -n run_all.sh / run_laaf.sh; python -m py_compile evals/**/*.py  → clean
Baseline before/after: 0 err / 84 warn → 0 err / 84 warn

The problem

evals/ shipped as configuration with no evidence any profile had ever run, and every runner had a concrete target compiled in — model_type: openai / model_name: gpt-4o-mini in all 13 Garak profiles, OpenAIChatTarget(deployment_name=…"gpt-4o-mini", endpoint=…"api.openai.com") in all 6 PyRIT scripts, gpt-4o-mini defaults in both runners and the CI template. An unconfigured invocation didn't fail — it silently tested OpenAI's gpt-4o-mini.

Targets — nothing baked in, and the runners refuse to guess

Track	Before	After
Garak ×13	model_type/model_name in the YAML	removed; usage shows required --model_type <type> --model_name <name>
PyRIT ×6	os.environ.get("OPENAI_MODEL", "gpt-4o-mini"), …"api.openai.com/v1"	_required_env("EVAL_MODEL_NAME"/"EVAL_ENDPOINT"/"OPENAI_API_KEY") — exits if unset
run_all.sh	${GARAK_MODEL_NAME:-gpt-4o-mini}, hand-listed 7 of 13 profiles	exit 2 unless both vars set; globs every *.yaml
run_laaf.sh	${LAAF_MODEL:-gpt-4o-mini}	exit 2 unless target set (LAAF_TARGET=mock needs no model/key)
ci/github-action.yml	`EVAL_MODEL_NAME: …

scripts/evals.test.mjs fences it: a regex for gpt-4o|api.openai.com|OPENAI_MODEL|… over every file in evals/ must find nothing.

Thresholds — surfaced, not authored (C4)

New evals/THRESHOLDS.md tabulates all 25 thresholds (13 Garak + 6 PyRIT + 6 LAAF) in one place. Each row quotes the reasoning the profile author already wrote and is marked DRAFT — SME review required; I set no value and changed no committed default. Each is overridable per run (Garak --eval_threshold, PyRIT EVAL_FAIL_THRESHOLD, LAAF LAAF_THRESHOLD_S1..S6). The test asserts the table equals the values actually in the files, so it can't drift, and that every row carries the DRAFT marker. An "Open questions for the reviewer" section records what I noticed but must not decide (e.g. the S2 reasoning anchors to an AIVSS score that T-ACC04 already flagged as itself unreviewed; LLM05 records no reasoning at all).

Run evidence — runbook, not fabrication

Per the hard rule "never fabricate results", no run is committed. evals/samples/README.md is the runbook: how to produce one sanitised run per track, a sanitisation checklist (strip credentials, endpoints, raw attack completions, PII), and a RUN.md provenance template. Producing a real run needs a credential, written authorisation, and budget — the needs-human-run issue #39.

Fixed in passing — the LAAF reporter had never run

Making laaf_crosswalk.py actually invokable surfaced that it carried an escaped-quote f-string (f"…{f\"…\"}…") that is a SyntaxError on every Python version — the reporter had never executed. Repaired, and its output forced to UTF-8 (write_text(..., encoding="utf-8") + a UTF-8 stdout write) so it doesn't crash a Windows console on the ✅/❌ status glyphs. Smoke-tested both to stdout and --out.

Observed, not changed (same as #37)

npm test runs suite files in parallel; generate.test.mjs rewrites data/entries/ while exports.test.mjs reads it, so OSCAL component definition carries at least one implemented requirement fails ~1 run in 3 and passes in isolation and on main. Pre-existing, unrelated to this ticket (my evals.test.mjs reads only static files and passed every run). Worth a follow-up to serialize those two suites.

🤖 Generated with Claude Code

https://claude.ai/code/session_0147wBugcuzLkswKPqgofcke

[emmanuelgjr]

CI note — two failures are the T-ENG03 bug on main, not this ticket

The first CI run showed three red jobs. One was mine and is fixed in 620877d; the other two are the latent main breakage that PR #37 fixes, now firing:

• Markdown lint — mine: evals/samples/README.md had a bare `````` fence (MD040). Fixed in 620877d (````text`).
• Generator reproducibility and Unit tests → "left no changes behind" — not T-ENG05. Both run node scripts/generate.js and then git diff --exit-code on docs/*.js. On main, the bundles carry a // Generated: <date> header, so CI regenerates 2026-08-28 → 2026-08-31 and the diff fails. This branch touches no generated file. It is exactly the day-bound failure described in T-ENG03: make the webapp data bundles deterministic #37, which removes that header.

Sequencing: #37 should merge first; then this branch rebases clean and all jobs go green. Until #37 lands, every PR cut from main fails these two jobs identically — worth confirming against #37's own checks. The T-ENG05 diff itself is limited to evals/ + scripts/evals.test.mjs; scripts/evals.test.mjs passed every local run.