Skip to content

fix: weight:0 un-gates criteria + configurable retry policy#34

Open
uipreliga wants to merge 1 commit into
mainfrom
fix/weight-zero-gating-and-retry-policy
Open

fix: weight:0 un-gates criteria + configurable retry policy#34
uipreliga wants to merge 1 commit into
mainfrom
fix/weight-zero-gating-and-retry-policy

Conversation

@uipreliga

Copy link
Copy Markdown
Collaborator

Summary

Three independent fixes surfaced by a prior review, plus the code-review follow-ups on top of them.

1. weight: 0 now makes a criterion informational

Previously a weight: 0 criterion contributed nothing to the weighted score but could still single-handedly flip a task to FAILURE — the docs told authors to also set pass_threshold: 0, which was easy to miss.

  • New BaseSuccessCriterion.is_gating (weight > 0.0) is the single source of truth for "does a low score here fail the task".
  • EvaluationResult.all_criteria_passed and the coder-eval evaluate exit code now gate over is_gating criteria only. A weight: 0 miss no longer flips final_status or the exit code.
  • A validator rejects weight: 0 combined with stop_when or suite_thresholds — arming a non-gating criterion for a pass/fail gate (early-stop gate, or the across-row suite gate that drives the run exit code) is incoherent and is now caught at load time.

2. Configurable retry policy

  • New RunLimits.retry (RetryPolicy: max_retries / initial_delay / backoff_multiplier, all optional) exposed via -D run_limits.retry.max_retries=0 to fail fast while debugging.
  • All retry decisions resolve through one seam, errors/retry.py::resolve_retry_config. A category that is non-retryable in the built-in table (auth/billing) is never made retryable by a per-run override — retryability is a property of the error, not the run.
  • Persisted error_details.is_retryable / retry_delay_seconds in task.json now reflect the run's actual policy instead of the built-in defaults.

3. Doc drift + evalboard

  • Dropped the non-existent rephrase prompt mutation from the catalog.
  • Corrected the stale "max_memory_mb NOT enforced" claim (it maps to docker run --memory; max_cpus/max_pids--cpus/--pids-limit).
  • reports.py failure-reason sampling uses pass_threshold, not score < 1.0.
  • evalboard: /trends and /watchlist now ship in the OSS edition (they derive purely from run artifacts); only /path-to-ga stays internal-gated.

Test plan

  • make verify green — 3437 passed, 2 skipped, 90.90% coverage, ruff + pyright + custom CE lint clean.
  • New regression tests: weight: 0 un-gating semantics, the stop_when/suite_thresholds rejections, retry-policy resolution + fail-fast, error_details policy fidelity, and a coder-eval evaluate exit-code test (with fixture) proving an informational miss doesn't fail the run.

Follow-ups (not in this PR)

Display-only sites (terminal PASS/FAIL label, reports_html, evalboard result pill, checker log line) still label a below-threshold weight: 0 criterion as failed. final_status and both exit codes are correct; a clean fix needs a coordinated gating marker on CriterionResult/CriterionSummary + the evalboard DTO, so it's deferred to avoid partial header-vs-list contradictions.

🤖 Generated with Claude Code

Three independent fixes plus code-review follow-ups:

- weight: 0 now makes a criterion purely informational — excluded from
  both the weighted score AND the pass/fail gate (previously it could
  still flip a task to FAILURE). New BaseSuccessCriterion.is_gating drives
  the single-source gate (all_criteria_passed) and the `coder-eval
  evaluate` exit code. A validator rejects weight:0 combined with
  stop_when or suite_thresholds (arming a non-gating criterion for a
  pass/fail gate is incoherent).
- RunLimits.retry (RetryPolicy) makes the built-in retry policy
  configurable per run via a single resolve_retry_config seam; a
  non-retryable error category is never made retryable by an override.
  Persisted error_details now reflect the run's actual policy.
- reports.py failure-reason sampling uses pass_threshold, not score < 1.0.
- Doc drift: dropped the non-existent `rephrase` mutation; corrected the
  stale "max_memory_mb NOT enforced" claim (docker --memory/--cpus/--pids-limit).
- evalboard: /trends and /watchlist ship in the OSS edition (they derive
  purely from run artifacts); only /path-to-ga stays internal-gated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown

Claude finished @uipreliga's task in 1m 59s —— View job


Code Review In Progress

Review Tasks

  • Read review guidelines from .github/code_review.md
  • Read project conventions from CLAUDE.md
  • Get full PR diff
  • Review each changed file for context
  • Perform cross-file consistency checks
  • Analyze design decisions
  • Check for missing elements
  • Provide structured feedback

Starting review...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant