Skip to content

Add scorer result callbacks - #4

Open
jkudish wants to merge 1 commit into
pestphp:5.xfrom
jkudish:add-scorer-result-callbacks
Open

Add scorer result callbacks#4
jkudish wants to merge 1 commit into
pestphp:5.xfrom
jkudish:add-scorer-result-callbacks

Conversation

@jkudish

@jkudish jkudish commented Aug 8, 2026

Copy link
Copy Markdown

Hi Pest team 👋

Thanks for building the Evals plugin. I’ve been exploring it as the scoring foundation for a separate Pest benchmarking plugin, and ran into one small missing integration point: there currently isn’t a structured way for another plugin to observe scorer results from both Pest’s built-in eval expectations and custom scorers.

Without that seam, a benchmarking or reporting plugin has to either wrap only the scorers it controls or parse terminal output. This PR proposes a small, provider-neutral callback that keeps scoring, sampling, judging, and output ownership inside Pest Evals while allowing other tooling to record the resulting evidence.

What changed

  • Adds a public Scored event containing the original ScorerResult, threshold, input, output, expected value, sample position/count, and derived pass/fail result.
  • Adds pest()->evals()->afterScored(...) for registering callbacks.
  • Dispatches the event after each scorer returns and before verbose rendering or the threshold assertion.
  • Clears registered callbacks through the existing eval configuration reset.
use Pest\Evals\Events\Scored;

pest()->evals()->afterScored(
    function (Scored $event): void {
        // Record or report scorer evidence.
    },
);

Callbacks run synchronously in registration order and are scoped to the current Pest process. They are not invoked when scoring is disabled. The event includes raw evaluation data that may be sensitive, so consumers are responsible for redaction and storage policy. Callback exceptions propagate and fail the test rather than silently losing evidence.

Why this shape

The intent is to expose the smallest useful extension seam without adding reporting, persistence, or benchmarking concerns to Pest Evals itself. It also avoids terminal parsing and works consistently for conveniences such as toBeRelevant() and toBeSafe(), as well as toPassScorer().

I’m very open to adjusting the API or naming if there’s a pattern you’d prefer within Pest.

Verification

  • Passing and failing scorer results
  • Built-in toBeRelevant() scorer
  • Custom toPassScorer() scorer
  • Repeated samples with the correct sample position and total
  • Disabled scoring emits no event and does not invoke the scorer
  • Multiple callbacks retain registration order
  • Callback exceptions propagate before the threshold assertion
  • Passing at the exact threshold boundary
  • Configuration flush clears registered callbacks
  • 9 focused callback tests passed, 44 assertions
  • Complete suite: 146 passed, 337 assertions; 20 live-model evals skipped as designed
  • composer lint passed
  • composer test passed, including Pint, Rector, PHPStan level max, and the complete test suite
  • composer test:types passed as part of the full pipeline

This is my first contribution to Pest. I used AI tools to assist with exploration and review, but I designed and engineered the change, verified the behavior, and reviewed the final implementation myself.

Thanks for taking a look — and thanks again for all the work on Pest and the new Evals tooling!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant