Add scorer result callbacks - #4
Open
jkudish wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi Pest team 👋
Thanks for building the Evals plugin. I’ve been exploring it as the scoring foundation for a separate Pest benchmarking plugin, and ran into one small missing integration point: there currently isn’t a structured way for another plugin to observe scorer results from both Pest’s built-in eval expectations and custom scorers.
Without that seam, a benchmarking or reporting plugin has to either wrap only the scorers it controls or parse terminal output. This PR proposes a small, provider-neutral callback that keeps scoring, sampling, judging, and output ownership inside Pest Evals while allowing other tooling to record the resulting evidence.
What changed
Scoredevent containing the originalScorerResult, threshold, input, output, expected value, sample position/count, and derived pass/fail result.pest()->evals()->afterScored(...)for registering callbacks.Callbacks run synchronously in registration order and are scoped to the current Pest process. They are not invoked when scoring is disabled. The event includes raw evaluation data that may be sensitive, so consumers are responsible for redaction and storage policy. Callback exceptions propagate and fail the test rather than silently losing evidence.
Why this shape
The intent is to expose the smallest useful extension seam without adding reporting, persistence, or benchmarking concerns to Pest Evals itself. It also avoids terminal parsing and works consistently for conveniences such as
toBeRelevant()andtoBeSafe(), as well astoPassScorer().I’m very open to adjusting the API or naming if there’s a pattern you’d prefer within Pest.
Verification
toBeRelevant()scorertoPassScorer()scorercomposer lintpassedcomposer testpassed, including Pint, Rector, PHPStan level max, and the complete test suitecomposer test:typespassed as part of the full pipelineThis is my first contribution to Pest. I used AI tools to assist with exploration and review, but I designed and engineered the change, verified the behavior, and reviewed the final implementation myself.
Thanks for taking a look — and thanks again for all the work on Pest and the new Evals tooling!