feat(examples): add verified task examples - #223
Open
svonava wants to merge 5 commits into
Open
Conversation
Contributor
Author
|
@coderabbitai review |
Contributor
Author
|
Review update:\n\n- Fixed the provenance tests so the documented uv run workflow does not scan generated virtual-environment metadata.\n- Added all three uv.lock files and reran with the resolved SIE SDK and Ruff versions.\n- Verified 17 offline tests, artifact hashes, full-vector score recomputation, source offsets, Ruff lint, and Ruff format.\n\nNo model calls or recorded outputs were rewritten. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe PR adds three standalone, verified SIE examples for multimodal search, named-entity extraction, and primary-source reranking, with runners, validation logic, offline tests, project configuration, documentation, and recorded-artifact integrity checks. ChangesVerified SIE examples
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant CLI
participant run_search
participant SIEClient
participant Evaluator
CLI->>run_search: execute search
run_search->>SIEClient: encode query and images
SIEClient-->>run_search: return embedding responses
run_search->>Evaluator: compute cosine similarities
Evaluator-->>CLI: return verified ranked matches
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Comment |
Contributor
Author
|
@coderabbitai review |
✅ Action performedReview finished.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (1)
examples/named-entity-extraction/tests/test_example.py (1)
68-95: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winFailure-path coverage is limited to one branch of
validate_response.Only the "missing required anchors" branch is tested.
validate_responsealso raises on unrequested labels, out-of-bounds/mismatched offsets, and invalid/out-of-range scores (Lines 163-177 ofrun.py), none of which have dedicated tests. Consider adding tests for at least one or two of these to guard the fail-closed contract as the recorded artifacts evolve.As per path instructions, "Check that commands match the implementation and that tests cover failure paths."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/named-entity-extraction/tests/test_example.py` around lines 68 - 95, Expand failure-path coverage for validate_response beyond the existing missing-required-anchors case by adding dedicated tests for at least one or two other validation failures, such as unrequested labels, invalid or out-of-range offsets, or invalid scores. Each test should construct an otherwise representative response, assert the expected ValueError and message, and preserve the existing artifact-based test setup.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@examples/multimodal-search/tests/test_example.py`:
- Around line 25-93: Add failure-path tests to MultimodalSearchExampleTests
covering checksum tampering, incorrect image counts, a true
synthetic_or_generated_images flag, non-finite vectors passed to dense_vector,
and an unexpected top match from evaluate. Assert each relevant
load_and_verify_sources, _validate_sources_dict, dense_vector, or evaluate call
raises ValueError, using the existing fixtures and validation entry points
without weakening success-path assertions.
- Around line 9-11: Fix the Ruff E402 violation in the test module by avoiding
the runtime sys.path.insert followed by a top-level import run; use an import
approach that preserves access to the example’s run module without placing an
import after executable statements. Alternatively, add the targeted tests/* E402
per-file ignore in pyproject.toml if retaining the current import pattern.
In `@examples/named-entity-extraction/tests/test_example.py`:
- Around line 9-11: Fix Ruff E402 in the test module by adding a targeted
per-file ignore for this file in the nearest pyproject.toml, rather than
changing the runtime path setup or import order. Keep the existing ROOT,
sys.path.insert, and import run behavior unchanged.
In `@examples/rerank/run.py`:
- Around line 147-150: Reject boolean values in the score validation loop of
validate_response by checking for bool before the numeric and finite checks. In
examples/rerank/run.py lines 147-150, preserve the existing ValueError for
invalid scores; in examples/rerank/tests/test_example.py lines 38-58, change one
recorded score to True and assert that validate_response raises ValueError.
In `@examples/rerank/tests/test_example.py`:
- Around line 9-11: Update the module-level imports in test_example.py: import
importlib before modifying sys.path, then load the run module via importlib
after sys.path.insert executes, eliminating Ruff E402 while preserving the
existing module resolution.
---
Nitpick comments:
In `@examples/named-entity-extraction/tests/test_example.py`:
- Around line 68-95: Expand failure-path coverage for validate_response beyond
the existing missing-required-anchors case by adding dedicated tests for at
least one or two other validation failures, such as unrequested labels, invalid
or out-of-range offsets, or invalid scores. Each test should construct an
otherwise representative response, assert the expected ValueError and message,
and preserve the existing artifact-based test setup.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 70111323-ba00-4b7b-950c-65e59cd6e9a4
⛔ Files ignored due to path filters (41)
.coderabbit.yamlis excluded by none and included by noneexamples/multimodal-search/data/images/black-camera.pngis excluded by!**/*.png,!examples/**/data/**and included byexamples/**examples/multimodal-search/data/images/black-handbag.jpgis excluded by!**/*.jpg,!examples/**/data/**and included byexamples/**examples/multimodal-search/data/images/blue-running-sneaker.pngis excluded by!**/*.png,!examples/**/data/**and included byexamples/**examples/multimodal-search/data/images/green-backpack.pngis excluded by!**/*.png,!examples/**/data/**and included byexamples/**examples/multimodal-search/data/images/red-leather-handbag.pngis excluded by!**/*.png,!examples/**/data/**and included byexamples/**examples/multimodal-search/data/images/red-shoes.jpgis excluded by!**/*.jpg,!examples/**/data/**and included byexamples/**examples/multimodal-search/data/sources.jsonis excluded by!examples/**/data/**and included byexamples/**examples/multimodal-search/uv.lockis excluded by!**/*.lock,!examples/**/uv.lockand included byexamples/**examples/multimodal-search/verified-run/evaluation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/multimodal-search/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/multimodal-search/verified-run/raw/image-candidate-embeddings.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/multimodal-search/verified-run/raw/text-query-embedding.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/multimodal-search/verified-run/requests/image-candidates.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/multimodal-search/verified-run/requests/text-query.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/data/cases.jsonis excluded by!examples/**/data/**and included byexamples/**examples/named-entity-extraction/data/sources.jsonis excluded by!examples/**/data/**and included byexamples/**examples/named-entity-extraction/uv.lockis excluded by!**/*.lock,!examples/**/uv.lockand included byexamples/**examples/named-entity-extraction/verified-run/evaluation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/raw/cms-orthosis-documentation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/raw/ntsb-bearing-alert.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/raw/sec-restatement.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/raw/supreme-court-caption.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/requests/cms-orthosis-documentation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/requests/ntsb-bearing-alert.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/requests/sec-restatement.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/requests/supreme-court-caption.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/data/cases.jsonis excluded by!examples/**/data/**and included byexamples/**examples/rerank/data/sources.jsonis excluded by!examples/**/data/**and included byexamples/**examples/rerank/uv.lockis excluded by!**/*.lock,!examples/**/uv.lockand included byexamples/**examples/rerank/verified-run/evaluation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/raw/cms-orthosis-documentation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/raw/ntsb-bearing-alert.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/raw/sec-restatement.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/raw/supreme-court-arbitration.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/requests/cms-orthosis-documentation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/requests/ntsb-bearing-alert.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/requests/sec-restatement.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/requests/supreme-court-arbitration.jsonis excluded by!examples/**/verified-run/**and included byexamples/**
📒 Files selected for processing (16)
examples/README.mdexamples/multimodal-search/.gitignoreexamples/multimodal-search/README.mdexamples/multimodal-search/pyproject.tomlexamples/multimodal-search/run.pyexamples/multimodal-search/tests/test_example.pyexamples/named-entity-extraction/.gitignoreexamples/named-entity-extraction/README.mdexamples/named-entity-extraction/pyproject.tomlexamples/named-entity-extraction/run.pyexamples/named-entity-extraction/tests/test_example.pyexamples/rerank/.gitignoreexamples/rerank/README.mdexamples/rerank/pyproject.tomlexamples/rerank/run.pyexamples/rerank/tests/test_example.py
Contributor
Author
|
@coderabbitai full review |
✅ Action performedFull review finished. Your included review limit is currently reached under our Fair Usage Limits Policy. Your recent PR review activity is in the 95th percentile or higher among CodeRabbit users, so adaptive limits apply. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 49 minutes. |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@examples/named-entity-extraction/run.py`:
- Around line 173-177: Update the score validation around entity score
extraction to explicitly reject boolean values before numeric validation, while
preserving finite and 0–1 range checks for valid numbers. Add a regression test
in the named-entity extraction example tests confirming boolean scores are
rejected.
In `@examples/rerank/tests/test_example.py`:
- Around line 38-58: Add a failure-path test alongside
test_recorded_responses_pass_fail_closed_checks that mutates a recorded response
to create an invalid validation case, then assert run.validate_response raises
ValueError. Reuse the existing case-loading, artifact lookup, and
response-reading setup, and preserve the current success-path assertions.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 18dc32af-abcb-4276-b7b6-1954fd6befce
⛔ Files ignored due to path filters (41)
.coderabbit.yamlis excluded by none and included by noneexamples/multimodal-search/data/images/black-camera.pngis excluded by!**/*.png,!examples/**/data/**and included byexamples/**examples/multimodal-search/data/images/black-handbag.jpgis excluded by!**/*.jpg,!examples/**/data/**and included byexamples/**examples/multimodal-search/data/images/blue-running-sneaker.pngis excluded by!**/*.png,!examples/**/data/**and included byexamples/**examples/multimodal-search/data/images/green-backpack.pngis excluded by!**/*.png,!examples/**/data/**and included byexamples/**examples/multimodal-search/data/images/red-leather-handbag.pngis excluded by!**/*.png,!examples/**/data/**and included byexamples/**examples/multimodal-search/data/images/red-shoes.jpgis excluded by!**/*.jpg,!examples/**/data/**and included byexamples/**examples/multimodal-search/data/sources.jsonis excluded by!examples/**/data/**and included byexamples/**examples/multimodal-search/uv.lockis excluded by!**/*.lock,!examples/**/uv.lockand included byexamples/**examples/multimodal-search/verified-run/evaluation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/multimodal-search/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/multimodal-search/verified-run/raw/image-candidate-embeddings.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/multimodal-search/verified-run/raw/text-query-embedding.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/multimodal-search/verified-run/requests/image-candidates.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/multimodal-search/verified-run/requests/text-query.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/data/cases.jsonis excluded by!examples/**/data/**and included byexamples/**examples/named-entity-extraction/data/sources.jsonis excluded by!examples/**/data/**and included byexamples/**examples/named-entity-extraction/uv.lockis excluded by!**/*.lock,!examples/**/uv.lockand included byexamples/**examples/named-entity-extraction/verified-run/evaluation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/raw/cms-orthosis-documentation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/raw/ntsb-bearing-alert.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/raw/sec-restatement.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/raw/supreme-court-caption.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/requests/cms-orthosis-documentation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/requests/ntsb-bearing-alert.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/requests/sec-restatement.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/named-entity-extraction/verified-run/requests/supreme-court-caption.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/data/cases.jsonis excluded by!examples/**/data/**and included byexamples/**examples/rerank/data/sources.jsonis excluded by!examples/**/data/**and included byexamples/**examples/rerank/uv.lockis excluded by!**/*.lock,!examples/**/uv.lockand included byexamples/**examples/rerank/verified-run/evaluation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/raw/cms-orthosis-documentation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/raw/ntsb-bearing-alert.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/raw/sec-restatement.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/raw/supreme-court-arbitration.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/requests/cms-orthosis-documentation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/requests/ntsb-bearing-alert.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/requests/sec-restatement.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/rerank/verified-run/requests/supreme-court-arbitration.jsonis excluded by!examples/**/verified-run/**and included byexamples/**
📒 Files selected for processing (16)
examples/README.mdexamples/multimodal-search/.gitignoreexamples/multimodal-search/README.mdexamples/multimodal-search/pyproject.tomlexamples/multimodal-search/run.pyexamples/multimodal-search/tests/test_example.pyexamples/named-entity-extraction/.gitignoreexamples/named-entity-extraction/README.mdexamples/named-entity-extraction/pyproject.tomlexamples/named-entity-extraction/run.pyexamples/named-entity-extraction/tests/test_example.pyexamples/rerank/.gitignoreexamples/rerank/README.mdexamples/rerank/pyproject.tomlexamples/rerank/run.pyexamples/rerank/tests/test_example.py
There was a problem hiding this comment.
🧹 Nitpick comments (1)
examples/named-entity-extraction/tests/test_example.py (1)
84-91: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winCover non-finite scores in the regression tests.
The runner now rejects
NaNand infinities, but this test only exercises boolean scores. Add cases forfloat("nan"),float("inf"), andfloat("-inf").As per path instructions,
examples/**tests must cover failure paths.Proposed test addition
+ def test_non_finite_score_fails_closed(self) -> None: + cases, _ = run.load_and_verify_inputs() + case_id = "scotus_two_contracts" + response = run.read_json(ROOT / "verified-run" / "raw" / "supreme-court-caption.json") + + for score in (float("nan"), float("inf"), float("-inf")): + with self.subTest(score=score): + response["entities"][0]["score"] = score + with self.assertRaisesRegex(ValueError, "Invalid score"): + run.validate_response(case_id, cases["cases"][case_id], response)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/named-entity-extraction/tests/test_example.py` around lines 84 - 91, Add regression coverage in test_boolean_score_fails_closed for non-finite entity scores by exercising float("nan"), float("inf"), and float("-inf") through run.validate_response. Assert each case raises ValueError with the existing "Invalid score" message, while preserving the current boolean-score assertion.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@examples/named-entity-extraction/tests/test_example.py`:
- Around line 84-91: Add regression coverage in test_boolean_score_fails_closed
for non-finite entity scores by exercising float("nan"), float("inf"), and
float("-inf") through run.validate_response. Assert each case raises ValueError
with the existing "Invalid score" message, while preserving the current
boolean-score assertion.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: f08c07fc-6574-475e-905d-27e2211a8b27
📒 Files selected for processing (5)
examples/multimodal-search/tests/test_example.pyexamples/named-entity-extraction/run.pyexamples/named-entity-extraction/tests/test_example.pyexamples/rerank/run.pyexamples/rerank/tests/test_example.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds runnable examples for reranking, named entity extraction, and multimodal search.\n\nEach example includes primary-source inputs, exact recorded public SIE responses, checksummed evidence manifests, and offline fail-closed tests. The multimodal example includes the licensed source images and full vectors needed to recompute every displayed cosine score.\n\nValidation: 17 unit tests, Ruff, Ty, JSON and artifact hashes, and examples-only scope checks.
Summary by CodeRabbit
scorevalues in reranking and named-entity extraction..gitignorefiles to exclude common local runtime artifacts.