Repository navigation
fix(explorer): label each verifier's results and show check descriptions and penalties - #395
Open
polakamtejas wants to merge 3 commits into
Open
polakamtejas wants to merge 3 commits into
polakamtejas wants to merge 3 commits into
Conversation
…ons and penalties The verifier tab's generic panel (verifiers that are not rubric, sandbox or aggregate shaped, such as env_outcome_verifier) now shows each verifier's id in its summary row, falls back to a check's `description` when it has no `message`, and marks a check with a negative `weight` as a penalty. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… does weighted_average subtracts (1 - score) × |weight| for a negative-weight check, its score its own or else 1 or 0 from its result, so a partly scored penalty subtracts less than its whole weight. The tooltip now gives that rule and what this check subtracts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
The explorer's verifier tab renders verifiers that are not rubric, sandbox or aggregate shaped (
env_outcome_verifier, for one) through a generic panel intask-instance-viewer.tsx. With a task that has twoenv_outcome_verifiersteps (recordsonweighted_average,db_matchonall_pass), that panel had three gaps:{passed}/{total} checks passedand the score, so two verifiers rendered as two identical-looking blocks ("1/2 checks passed · Score: 0%" twice) with no way to tell which was which. The rubric, sandbox and aggregate cards all show the verifier id; this branch dropped it.r.idandr.message, butenv_outcome_verifierchecks carrydescription(the shippedexamples/mail-mcp/artifacts/check_mail/verify.pyreturns one), so their explanation never appeared.weighted_average(verifiers/scoring.py) treats a negativeweightas a penalty: a failing check with weight-wsubtractswfrom the weighted sum, and a passing one adds nothing. Nothing marked such a check, so "1/2 checks passed" next to "Score: 0%" read as a contradiction.What changed
In the generic branch only, no restructuring:
text-[10px] font-mono text-[var(--muted-foreground)]).r.message ?? r.description.weightgets a small amberpenalty −wbadge whose tooltip says that when it failsweighted_averagesubtractswfrom the weighted sum.descriptionandweight, andmessagebecomes optional, matching what verifiers write.The aggregator is not persisted with an
env_outcome_verifier's output, so the panel can't name it; the tooltip namesweighted_averagebecause it is the only aggregator that reads weights.How it was tested
From
src/agent_env/explorer/ui, on Node 20.11 (CI uses 22; npm prints anEBADENGINEwarning foreslint-visitor-keys, nothing else):npm ci: exit 0npm run typecheck: exit 0npm run lint: exit 0, 0 errors and 59 warnings, the same 59 as onmain(none in the changed lines)npm run test:smoke: exit 0, 15 ran, 2 skipped (the two allowed trajectory-fixture capabilities), 0 failednpm run build:static: exit 0; the exported CSS includes the newbg-amber-500/10andtext-amber-600classesThe change is JSX inside
TaskInstanceViewerwith no pure helper to test, so no smoke test was added (the smokes coversrc/libhelpers). It was not checked in a browser.🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge; no new actionable issues were found.Summary
Adds verifier names, check descriptions, and negative-weight badges to the generic verifier panel.
Reviews (3) · Last reviewed commit: "Merge branch 'main' into fix/explorer-ve..." · Reviewed by Greptile