fix: weight:0 un-gates criteria + configurable retry policy#34
Open
uipreliga wants to merge 1 commit into
Open
fix: weight:0 un-gates criteria + configurable retry policy#34uipreliga wants to merge 1 commit into
uipreliga wants to merge 1 commit into
Conversation
Three independent fixes plus code-review follow-ups: - weight: 0 now makes a criterion purely informational — excluded from both the weighted score AND the pass/fail gate (previously it could still flip a task to FAILURE). New BaseSuccessCriterion.is_gating drives the single-source gate (all_criteria_passed) and the `coder-eval evaluate` exit code. A validator rejects weight:0 combined with stop_when or suite_thresholds (arming a non-gating criterion for a pass/fail gate is incoherent). - RunLimits.retry (RetryPolicy) makes the built-in retry policy configurable per run via a single resolve_retry_config seam; a non-retryable error category is never made retryable by an override. Persisted error_details now reflect the run's actual policy. - reports.py failure-reason sampling uses pass_threshold, not score < 1.0. - Doc drift: dropped the non-existent `rephrase` mutation; corrected the stale "max_memory_mb NOT enforced" claim (docker --memory/--cpus/--pids-limit). - evalboard: /trends and /watchlist ship in the OSS edition (they derive purely from run artifacts); only /path-to-ga stays internal-gated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
uipreliga
requested review from
akshaylive,
bai-uipath and
tmatup
as code owners
July 21, 2026 23:34
|
Claude finished @uipreliga's task in 1m 59s —— View job Code Review In Progress
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
Three independent fixes surfaced by a prior review, plus the code-review follow-ups on top of them.
1.
weight: 0now makes a criterion informationalPreviously a
weight: 0criterion contributed nothing to the weighted score but could still single-handedly flip a task toFAILURE— the docs told authors to also setpass_threshold: 0, which was easy to miss.BaseSuccessCriterion.is_gating(weight > 0.0) is the single source of truth for "does a low score here fail the task".EvaluationResult.all_criteria_passedand thecoder-eval evaluateexit code now gate overis_gatingcriteria only. Aweight: 0miss no longer flipsfinal_statusor the exit code.weight: 0combined withstop_whenorsuite_thresholds— arming a non-gating criterion for a pass/fail gate (early-stop gate, or the across-row suite gate that drives the run exit code) is incoherent and is now caught at load time.2. Configurable retry policy
RunLimits.retry(RetryPolicy:max_retries/initial_delay/backoff_multiplier, all optional) exposed via-D run_limits.retry.max_retries=0to fail fast while debugging.errors/retry.py::resolve_retry_config. A category that is non-retryable in the built-in table (auth/billing) is never made retryable by a per-run override — retryability is a property of the error, not the run.error_details.is_retryable/retry_delay_secondsintask.jsonnow reflect the run's actual policy instead of the built-in defaults.3. Doc drift + evalboard
rephraseprompt mutation from the catalog.max_memory_mbNOT enforced" claim (it maps todocker run --memory;max_cpus/max_pids→--cpus/--pids-limit).reports.pyfailure-reason sampling usespass_threshold, notscore < 1.0./trendsand/watchlistnow ship in the OSS edition (they derive purely from run artifacts); only/path-to-gastays internal-gated.Test plan
make verifygreen — 3437 passed, 2 skipped, 90.90% coverage, ruff + pyright + custom CE lint clean.weight: 0un-gating semantics, thestop_when/suite_thresholdsrejections, retry-policy resolution + fail-fast,error_detailspolicy fidelity, and acoder-eval evaluateexit-code test (with fixture) proving an informational miss doesn't fail the run.Follow-ups (not in this PR)
Display-only sites (terminal PASS/FAIL label,
reports_html, evalboard result pill,checkerlog line) still label a below-thresholdweight: 0criterion as failed.final_statusand both exit codes are correct; a clean fix needs a coordinated gating marker onCriterionResult/CriterionSummary+ the evalboard DTO, so it's deferred to avoid partial header-vs-list contradictions.🤖 Generated with Claude Code