Skip to content

fix(explorer): label each verifier's results and show check descriptions and penalties - #395

Open
polakamtejas wants to merge 3 commits into
mainfrom
fix/explorer-verifier-panel-labels
Open

polakamtejas wants to merge 3 commits into
mainfrom
fix/explorer-verifier-panel-labels

Conversation

@polakamtejas

@polakamtejas polakamtejas commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

What and why

The explorer's verifier tab renders verifiers that are not rubric, sandbox or aggregate shaped (env_outcome_verifier, for one) through a generic panel in task-instance-viewer.tsx. With a task that has two env_outcome_verifier steps (records on weighted_average, db_match on all_pass), that panel had three gaps:

  1. No verifier label. The summary row showed only {passed}/{total} checks passed and the score, so two verifiers rendered as two identical-looking blocks ("1/2 checks passed · Score: 0%" twice) with no way to tell which was which. The rubric, sandbox and aggregate cards all show the verifier id; this branch dropped it.
  2. Check descriptions hidden. Each check showed r.id and r.message, but env_outcome_verifier checks carry description (the shipped examples/mail-mcp/artifacts/check_mail/verify.py returns one), so their explanation never appeared.
  3. Penalties looked like ordinary checks. weighted_average (verifiers/scoring.py) treats a negative weight as a penalty: a failing check with weight -w subtracts w from the weighted sum, and a passing one adds nothing. Nothing marked such a check, so "1/2 checks passed" next to "Score: 0%" read as a contradiction.

What changed

In the generic branch only, no restructuring:

  • The summary row ends with the verifier id, styled like the other verifier cards (text-[10px] font-mono text-[var(--muted-foreground)]).
  • A check's text is r.message ?? r.description.
  • A check with a negative weight gets a small amber penalty −w badge whose tooltip says that when it fails weighted_average subtracts w from the weighted sum.
  • The inline result type gains optional description and weight, and message becomes optional, matching what verifiers write.

The aggregator is not persisted with an env_outcome_verifier's output, so the panel can't name it; the tooltip names weighted_average because it is the only aggregator that reads weights.

How it was tested

From src/agent_env/explorer/ui, on Node 20.11 (CI uses 22; npm prints an EBADENGINE warning for eslint-visitor-keys, nothing else):

  • npm ci: exit 0
  • npm run typecheck: exit 0
  • npm run lint: exit 0, 0 errors and 59 warnings, the same 59 as on main (none in the changed lines)
  • npm run test:smoke: exit 0, 15 ran, 2 skipped (the two allowed trajectory-fixture capabilities), 0 failed
  • npm run build:static: exit 0; the exported CSS includes the new bg-amber-500/10 and text-amber-600 classes

The change is JSX inside TaskInstanceViewer with no pure helper to test, so no smoke test was added (the smokes cover src/lib helpers). It was not checked in a browser.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no new actionable issues were found.

Summary

Adds verifier names, check descriptions, and negative-weight badges to the generic verifier panel.

  • The verifier panel labels each verifier and explains negative-weight checks.

Reviews (3) · Last reviewed commit: "Merge branch 'main' into fix/explorer-ve..." · Reviewed by Greptile

…ons and penalties

The verifier tab's generic panel (verifiers that are not rubric, sandbox or
aggregate shaped, such as env_outcome_verifier) now shows each verifier's id
in its summary row, falls back to a check's `description` when it has no
`message`, and marks a check with a negative `weight` as a penalty.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@polakamtejas
polakamtejas requested a review from a team as a code owner October 29, 2026 19:31
polakamtejas and others added 2 commits November 3, 2026 11:20
… does

weighted_average subtracts (1 - score) × |weight| for a negative-weight check, its score its own or else 1 or 0 from
its result, so a partly scored penalty subtracts less than its whole weight. The tooltip now gives that rule and what
this check subtracts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant