docs(evals): document running an eval from a single question file

[MikeNitsenko]

Eval runs can now be scoped to a single question file rather than always
running the whole set. This updates /admin/ai/evals to match.

Note

The product change backing this is not live yet — please merge after it ships.

What changed on the page:

• Running an eval — the new Questions choice: All questions (the
  default) or one question file. Notes that it only appears when the branch's
  questions span more than one file, that each option shows its question
  count, and that switching branches resets it.
• Run list table — the new Questions column (file, or All questions).
• Reading the results — the run header repeats the scope.
• Questions tab note — the new File column.
• Authoring / Concepts — a run is the whole set or one file, which
  gives splitting questions across files a second purpose.

No new sections; five surgical edits to the existing page.

🤖 Generated with Claude Code

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
cubed3	🟢 Ready	View Preview	Sep 16, 2026, 7:28 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

[claude]

Claude finished @MikeNitsenko's task in 1m 25s —— View job

---

Docs-only change, reads correctly and matches page conventions — 2 low, both wording nits posted inline. Merge gate: author notes the backing product change isn't live yet.

Review details

• Read PR diff (git diff origin/master...HEAD)
• Read docs-mintlify/admin/ai/evals.mdx in full, not just the hunks
• Checked anchor, nav registration, and naming conventions
• Handled prior review threads
• Posted findings

Findings

#	Site	Severity	Finding
1	docs-mintlify/admin/ai/evals.mdx:86	low	"each option shows how many questions that file holds" omits the All questions option, which per the PR description also shows a count
2	docs-mintlify/admin/ai/evals.mdx:37	low	The scoping fact is stated four times (lines 25, 37, 83, 108); the Authoring sentence is the one that adds nothing

Both posted inline with suggestion blocks.

Verified clean

• [Running an eval](#running-an-eval) resolves — the heading at evals.mdx:77 generates exactly that slug.
• No docs.json change needed; admin/ai/evals is already registered (docs-mintlify/docs.json:403).
• Terminology matches docs-mintlify/CLAUDE.md: eval run, Run eval, Eval run column, Evals — no stray "Evaluation".
• No new sections, no H1, no committed images — consistent with the surgical-edit rule.
• Nothing security-, performance-, or test-relevant: single .mdx file, no code, no product code for this feature in the repo (Cloud-side), so the UI labels — Questions column position, the File column, the run-header scope — could not be verified against an implementation. They're taken on the author's word.

Optional, not posted inline

Limitations (evals.mdx:154) still reads "Very large question sets can be slow to run" with no pointer to the new mitigation. A trailing clause like "— scope a run to one question file to cut the time" would close the loop, but it's additive, so it's your call.

Review threads

The listing returned 0 of my own threads (gh list-review-threads cube-js cube 11906 came back with an empty nodes array and hasNextPage: false) — no prior threads. Nothing resolved, nothing re-affirmed; first round on this PR.

· branch [`mikhail/docs-evals-question-file`](https://github.com/cube-js/cube/tree/mikhail/docs-evals-question-file)

[claude]

Claude finished @MikeNitsenko's task in 1m 19s —— View job

---

Review: docs-only, 1 low finding — otherwise good to merge (after the product change ships)

Full review

• Read the diff (git diff origin/master...HEAD)
• Read the whole page, not just the hunks
• Checked anchors, nav, and docs-mintlify/CLAUDE.md conventions
• Handled prior review threads
• Posted findings

Finding (low) — docs-mintlify/admin/ai/evals.mdx:153

The Limitations bullet now says "scope the run to a single question file to cut it down", but line 84 states the Questions selector appears only when the branch's questions span more than one file. A reader with one large agents/eval_questions.yml — the layout line 33 calls "the simplest place to start" — sees no selector and can't act on the advice without first splitting the file. Suggested wording posted inline.

Checked and fine

• Anchor #running-an-eval at line 38 resolves against ## Running an eval (line 76).
• Terminology matches the Evals section of docs-mintlify/CLAUDE.md: "eval run", Run eval, Questions — no "Evaluation"/"Evaluations".
• No new page, no nav change needed (admin/ai/evals is already in docs.json:403).
• Five edits, all surgical; nothing padded, and the scope statement isn't repeated within a single surface — Concepts, the dialog, the run-list column, the run header and the Questions tab are five distinct UI surfaces.
• The per-file-count rewording in 5bb7b8f correctly avoids implying All questions carries a count.

Review threads

Listing returned 2 of my own threads. Resolved both (one withdrawn after MikeNitsenko refuted it against actual product behavior, with a reply on the thread noting the withdrawal; one fixed in 5bb7b8f). 0 re-affirmed. The new finding is not a duplicate of either.

Note on merge order

The PR body says the backing product change isn't live yet — nothing here changes that; the page describes UI that must ship first.

· branch [`mikhail/docs-evals-question-file`](https://github.com/cube-js/cube/tree/mikhail/docs-evals-question-file)

[claude]

Claude finished @MikeNitsenko's task in 1m 53s —— View job

---

Docs-only change, reads accurately — 0 blocking, 2 low (details)

Review details

• Read PR diff
• Read the full changed file in context
• Checked anchors, terminology, and table wording against docs-mintlify/CLAUDE.md
• Handled prior review threads
• Posted findings

Findings

Both low, both posted inline with suggestions:

Where	Finding
docs-mintlify/admin/ai/evals.mdx:106-108	Third enumeration of the scope values; :25 and :98 already give them. What's new on the run page is only where the scope appears.
docs-mintlify/admin/ai/evals.mdx:153-154	The remedy doesn't remove the limitation — splitting files doesn't make a full-set run faster, it only lets you run a subset while iterating. Reworded suggestion offered.

Checked and clean

• #running-an-eval resolves against ## Running an eval at :76.
• Terminology matches the Evals Terminology section of docs-mintlify/CLAUDE.md — "eval run", Run eval, Eval run column label; no "Evaluation"/"Evaluations" leaked in.
• The per-file count wording at :84-85 is right per the author's own check against product behavior — only file options carry a count, and the text now says "each file option".
• The selector-visibility condition (more than one file) is consistent with the Limitations advice after 6c9f918, and with the authoring section's one-file-to-many story at :33-38.
• No new sections, no page created — five surgical edits, in line with the "prefer surgical edits" rule.

One non-blocking observation, not posted inline: the page's premise is tracking accuracy across changes, and a single-file run's NN% (passed/total) isn't comparable with a full-set run's. The new Questions column at :98 is what lets a reader tell them apart, so this may not need saying — worth a sentence only if support questions show up.

The PR note that the backing product change isn't live yet is the merge-timing call, not a docs defect.

Review threads

Listing returned 1 thread of mine. Resolved 1 (PRRT_kwDOCOH19M6jEoxX, the prior :154 remedy-actionability finding — confirmed taken in 6c9f918 and the wording is in the file). Re-affirmed 0. Both findings this round are new root causes, so neither was skipped as a duplicate; the second shares a line with the resolved thread but not its concern.

· branch `mikhail/docs-evals-question-file`

[claude]

Claude finished @MikeNitsenko's task in 1m 24s —— View job

---

Review of #11906 — approve; 0 issues found (2 prior threads resolved as fixed)

Review details

• Read the diff (git diff origin/master...HEAD)
• Read the whole page, not just the hunks
• Checked against docs-mintlify/CLAUDE.md naming and style conventions
• Handled prior review threads
• Posted findings (none inline)

What I checked

Five edits to one page, all consistent with the page as a whole:

• docs-mintlify/admin/ai/evals.mdx:25 — the Eval (run) definition now admits both scopes, matching the docs-mintlify/CLAUDE.md taxonomy entry ("one execution of the agent against the question set") rather than contradicting it.
• docs-mintlify/admin/ai/evals.mdx:37 — the cross-link anchor #running-an-eval resolves to the ## Running an eval heading at line 76.
• docs-mintlify/admin/ai/evals.mdx:82 — the selector bullet states the three things a reader can't guess: when it appears, what the count refers to, and the branch-switch reset. The "only when the branch's questions span more than one file" condition and the Limitations remedy at line 152 agree — a reader on a single file is told to split.
• docs-mintlify/admin/ai/evals.mdx:98 — the Questions column slots into the existing run-list table without disturbing the others.
• Terminology: Evals / eval run / Question used per the Evals Terminology section; no "Evaluation", no new page, so no docs.json change needed.

No factual claim on the page contradicts another, and nothing here is code — no security, performance, or test surface.

Not raised

One thing I considered and dropped: the Limitations bullet at line 152 and the Concepts pointer at line 37 both mention splitting files plus scoping. It reads as redundancy against "say it once", but the two do different jobs — line 37 is a one-clause pointer, line 152 is the self-contained workaround that 6c9f918 deliberately made actionable on its own. Re-raising it would argue against a wording a human just chose.

Review threads

The listing returned 2 of my own threads, both unresolved. Resolved both, re-affirmed none:

• PRRT_kwDOCOH19M6jEubn (originalLine 108, scope values enumerated a third time) — fixed in d41b5a4; the header sentence no longer re-lists the values.
• PRRT_kwDOCOH19M6jEugh (originalLine 154, remedy didn't remove the stated limitation) — fixed in d41b5a4; line 152-153 now matches the suggested wording.

No human-opened threads on the PR.

Note on merge

The PR body says the backing product change isn't live yet. The page describes the selector in the present tense throughout, which is right once it ships — worth holding the merge until then, as the author already flagged.

· branch [`mikhail/docs-evals-question-file`](https://github.com/cube-js/cube/tree/mikhail/docs-evals-question-file)