Replace fictional insurance claim with public FEMA appeal - #225
Open
svonava wants to merge 7 commits into
Open
Conversation
Contributor
Author
|
@coderabbitai review |
Contributor
Author
|
@coderabbitai review |
Contributor
Author
|
Review update:\n\n- Added SHA-256 hashes for every saved model output and derived artifact.\n- Added tests that verify the bundled FEMA PDFs against the source manifest and recompute the saved evaluation from review.json.\n- Updated the gallery entry so it describes the real FEMA appeal instead of the removed fictional claim.\n- Reran 8 locked offline tests plus Ruff lint and format.\n\nThe factual model outputs remain unchanged. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe insurance claims example now reviews FEMA Flood Insurance Appeal Decision B8 and related policy documents. Claim-packet preparation, vision processing, and image fixtures were removed; appeal fact extraction, policy retrieval, structured review generation, scope evaluation, and artifact hashing were added. ChangesFEMA appeal review
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant ReviewCLI
participant DefaultStage
participant GenerationStage
ReviewCLI->>DefaultStage: parse appeal and policy sources
DefaultStage->>DefaultStage: extract claim facts and retrieve policy chunks
ReviewCLI->>GenerationStage: start generation stage
GenerationStage->>GenerationStage: generate structured appeal review
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Comment |
Contributor
Author
|
@coderabbitai full review |
✅ Action performedFull review finished. Your included review limit is currently reached under our Fair Usage Limits Policy. Your recent PR review activity is in the 95th percentile or higher among CodeRabbit users, so adaptive limits apply. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 49 minutes. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
examples/insurance-claims-agent/tests/test_review.py (1)
69-91: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd a failure-path test for
evaluate_review.Every test here only feeds
evaluate_reviewfully-correct data and asserts all checks pass. There's no test that a check correctly reportspassed=Falseon bad input (wrong amount, missing scope keyword, missing finding category), so a logic bug in any individualCheck(wrong tolerance, inverted condition, wrong field name) wouldn't be caught.As per path instructions, "Check that commands match the implementation and that tests cover failure paths."
✅ Example negative-case test
def test_evaluation_fails_when_proof_of_loss_amount_is_wrong() -> None: review = { "route": "scope_review_required", "appeal_summary": { "proof_of_loss_amount": 1, # wrong "removal_estimate": 49500, "barge_estimate": 181832.94, "debris_cubic_yards_min": 12, "debris_cubic_yards_max": 15, }, "decision": { "covered_scope": "Remove flood-borne stones from underneath the insured building to its perimeter.", "excluded_scope": "Barge transport, handling, disposal, and yard removal.", "evidence_needed": "Other contractor estimates.", "prior_claim_check": "Proof of repairs from previous claims.", }, "findings": [ {"category": "covered_removal"}, {"category": "excluded_transport"}, {"category": "price_support"}, {"category": "prior_claim_overlap"}, ], } checks = {check.name: check for check in evaluate_review(review)} assert checks["proof-of-loss-amount"].passed is FalseAlso applies to: 96-105
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/insurance-claims-agent/tests/test_review.py` around lines 69 - 91, Add failure-path coverage for evaluate_review by introducing a negative-case test with otherwise valid review data and one deliberately invalid field, such as an incorrect proof_of_loss_amount. Collect checks by name and assert the corresponding check’s passed value is False; add analogous coverage for missing scope keywords or finding categories if needed to exercise each check’s failure behavior.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@examples/insurance-claims-agent/README.md`:
- Line 37: Update the sentence describing the verified-run directory to add a
comma after “2026,” preserving the existing date and surrounding wording.
---
Nitpick comments:
In `@examples/insurance-claims-agent/tests/test_review.py`:
- Around line 69-91: Add failure-path coverage for evaluate_review by
introducing a negative-case test with otherwise valid review data and one
deliberately invalid field, such as an incorrect proof_of_loss_amount. Collect
checks by name and assert the corresponding check’s passed value is False; add
analogous coverage for missing scope keywords or finding categories if needed to
exercise each check’s failure behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 5a290268-6b8c-4d52-a645-5ff63c8dc80b
⛔ Files ignored due to path filters (22)
.coderabbit.yamlis excluded by none and included by noneexamples/insurance-claims-agent/fixtures/SOURCES.mdis excluded by!examples/**/fixtures/**and included byexamples/**,examples/**/fixtures/SOURCES.mdexamples/insurance-claims-agent/fixtures/claim-note.txtis excluded by!examples/**/fixtures/**and included byexamples/**examples/insurance-claims-agent/fixtures/claim.jsonis excluded by!examples/**/fixtures/**and included byexamples/**examples/insurance-claims-agent/fixtures/sources/flooded-house-interior.jpgis excluded by!**/*.jpg,!examples/**/fixtures/**and included byexamples/**examples/insurance-claims-agent/fixtures/sources/nfip-appeal-b8.pdfis excluded by!**/*.pdf,!examples/**/fixtures/**and included byexamples/**examples/insurance-claims-agent/fixtures/sources/nfip-proof-of-loss.pdfis excluded by!**/*.pdf,!examples/**/fixtures/**and included byexamples/**examples/insurance-claims-agent/uv.lockis excluded by!**/*.lock,!examples/**/uv.lockand included byexamples/**examples/insurance-claims-agent/verified-run/README.mdis excluded by!examples/**/verified-run/**and included byexamples/**,examples/**/verified-run/README.mdexamples/insurance-claims-agent/verified-run/claim-facts.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/insurance-claims-agent/verified-run/evaluation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/insurance-claims-agent/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/insurance-claims-agent/verified-run/markdown/appeal_decision.mdis excluded by!examples/**/verified-run/**and included byexamples/**examples/insurance-claims-agent/verified-run/markdown/policy.mdis excluded by!examples/**/verified-run/**and included byexamples/**examples/insurance-claims-agent/verified-run/policy-evidence.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/insurance-claims-agent/verified-run/raw/appeal_decision-parse.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/insurance-claims-agent/verified-run/raw/claim-facts.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/insurance-claims-agent/verified-run/raw/policy-parse.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/insurance-claims-agent/verified-run/raw/policy-rerank.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/insurance-claims-agent/verified-run/raw/review-completion.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/insurance-claims-agent/verified-run/review.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/insurance-claims-agent/verified-run/source-manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**
📒 Files selected for processing (12)
examples/README.mdexamples/insurance-claims-agent/README.mdexamples/insurance-claims-agent/config.yamlexamples/insurance-claims-agent/insurance_claims/config.pyexamples/insurance-claims-agent/insurance_claims/evaluate.pyexamples/insurance-claims-agent/insurance_claims/fetch.pyexamples/insurance-claims-agent/insurance_claims/prepare.pyexamples/insurance-claims-agent/insurance_claims/review.pyexamples/insurance-claims-agent/pyproject.tomlexamples/insurance-claims-agent/tests/test_config.pyexamples/insurance-claims-agent/tests/test_prepare.pyexamples/insurance-claims-agent/tests/test_review.py
💤 Files with no reviewable changes (3)
- examples/insurance-claims-agent/tests/test_prepare.py
- examples/insurance-claims-agent/insurance_claims/prepare.py
- examples/insurance-claims-agent/insurance_claims/config.py
There was a problem hiding this comment.
🧹 Nitpick comments (1)
examples/insurance-claims-agent/tests/test_review.py (1)
107-118: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winExclusion set diverges from
evaluate.py's own artifact-hashing logic.This test excludes
README.mdandmanifest.jsonfromexpected_paths, butevaluate_run()inevaluate.pyonly excludesmanifest.jsonwhen buildingmanifest["artifacts"]. It currently passes only becauseverified-run/README.mdwasn't present when the recorded manifest's artifact list was generated. Ifevaluate_runis ever rerun againstverified-runto refresh the pinned manifest,README.mdwould get hashed intoartifactsand this test would start failing for a reason unrelated to a real regression.Consider sharing one exclusion set (e.g., a
_DOC_FILENAMES = {"README.md"}constant importable fromevaluate.py) between the production artifact-hashing loop and this test, or havingevaluate_runitself skip documentation files by name.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/insurance-claims-agent/tests/test_review.py` around lines 107 - 118, The artifact exclusion rules in test_verified_manifest_pins_every_recorded_artifact diverge from evaluate_run. Define and reuse a shared documentation exclusion set, such as _DOC_FILENAMES from evaluate.py, in both the production artifact-hashing loop and the test, ensuring README.md and manifest.json are handled consistently when regenerating the manifest.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@examples/insurance-claims-agent/tests/test_review.py`:
- Around line 107-118: The artifact exclusion rules in
test_verified_manifest_pins_every_recorded_artifact diverge from evaluate_run.
Define and reuse a shared documentation exclusion set, such as _DOC_FILENAMES
from evaluate.py, in both the production artifact-hashing loop and the test,
ensuring README.md and manifest.json are handled consistently when regenerating
the manifest.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: cf9e6b61-ace8-485b-8a6b-13576b305ad9
⛔ Files ignored due to path filters (22)
.coderabbit.yamlis excluded by none and included by noneexamples/insurance-claims-agent/fixtures/SOURCES.mdis excluded by!examples/**/fixtures/**and included byexamples/**,examples/**/fixtures/SOURCES.mdexamples/insurance-claims-agent/fixtures/claim-note.txtis excluded by!examples/**/fixtures/**and included byexamples/**examples/insurance-claims-agent/fixtures/claim.jsonis excluded by!examples/**/fixtures/**and included byexamples/**examples/insurance-claims-agent/fixtures/sources/flooded-house-interior.jpgis excluded by!**/*.jpg,!examples/**/fixtures/**and included byexamples/**examples/insurance-claims-agent/fixtures/sources/nfip-appeal-b8.pdfis excluded by!**/*.pdf,!examples/**/fixtures/**and included byexamples/**examples/insurance-claims-agent/fixtures/sources/nfip-proof-of-loss.pdfis excluded by!**/*.pdf,!examples/**/fixtures/**and included byexamples/**examples/insurance-claims-agent/uv.lockis excluded by!**/*.lock,!examples/**/uv.lockand included byexamples/**examples/insurance-claims-agent/verified-run/README.mdis excluded by!examples/**/verified-run/**and included byexamples/**,examples/**/verified-run/README.mdexamples/insurance-claims-agent/verified-run/claim-facts.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/insurance-claims-agent/verified-run/evaluation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/insurance-claims-agent/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/insurance-claims-agent/verified-run/markdown/appeal_decision.mdis excluded by!examples/**/verified-run/**and included byexamples/**examples/insurance-claims-agent/verified-run/markdown/policy.mdis excluded by!examples/**/verified-run/**and included byexamples/**examples/insurance-claims-agent/verified-run/policy-evidence.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/insurance-claims-agent/verified-run/raw/appeal_decision-parse.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/insurance-claims-agent/verified-run/raw/claim-facts.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/insurance-claims-agent/verified-run/raw/policy-parse.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/insurance-claims-agent/verified-run/raw/policy-rerank.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/insurance-claims-agent/verified-run/raw/review-completion.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/insurance-claims-agent/verified-run/review.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/insurance-claims-agent/verified-run/source-manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**
📒 Files selected for processing (12)
examples/README.mdexamples/insurance-claims-agent/README.mdexamples/insurance-claims-agent/config.yamlexamples/insurance-claims-agent/insurance_claims/config.pyexamples/insurance-claims-agent/insurance_claims/evaluate.pyexamples/insurance-claims-agent/insurance_claims/fetch.pyexamples/insurance-claims-agent/insurance_claims/prepare.pyexamples/insurance-claims-agent/insurance_claims/review.pyexamples/insurance-claims-agent/pyproject.tomlexamples/insurance-claims-agent/tests/test_config.pyexamples/insurance-claims-agent/tests/test_prepare.pyexamples/insurance-claims-agent/tests/test_review.py
💤 Files with no reviewable changes (3)
- examples/insurance-claims-agent/tests/test_prepare.py
- examples/insurance-claims-agent/insurance_claims/prepare.py
- examples/insurance-claims-agent/insurance_claims/config.py
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
examples/insurance-claims-agent/insurance_claims/evaluate.py (1)
24-96: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winReject unsupported finding categories.
The category check accepts the expected categories as a subset, so a model can add arbitrary unsupported findings and still pass evaluation. Require an exact category set, or explicitly validate/document an allowlist for additional categories; otherwise undocumented synthetic findings are certified as valid.
As per path instructions, undocumented synthetic results in runnable examples must be flagged.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/insurance-claims-agent/insurance_claims/evaluate.py` around lines 24 - 96, Update the “finding-categories” check in evaluate_review to require categories to exactly match the documented expected set, rejecting any unsupported or synthetic finding categories. Preserve the existing required category names and diagnostic output while replacing the subset-only validation with exact-set validation.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@examples/insurance-claims-agent/insurance_claims/evaluate.py`:
- Line 14: Update the artifact filtering associated with
ARTIFACT_EXCLUDED_FILENAMES and rglob("*") to compare each file’s path relative
to the run root, excluding only top-level README.md and manifest.json. Preserve
nested files with those names, and add a nested-name fixture and corresponding
assertion so the test verifies this behavior rather than duplicating the
production filtering rule.
In `@examples/insurance-claims-agent/tests/test_review.py`:
- Around line 120-121: Update the assertions around evaluate_review in
test_review.py to verify that the failed check names are exactly
{"proof-of-loss-amount"}, while preserving the existing assertion that this
check is not passed.
---
Outside diff comments:
In `@examples/insurance-claims-agent/insurance_claims/evaluate.py`:
- Around line 24-96: Update the “finding-categories” check in evaluate_review to
require categories to exactly match the documented expected set, rejecting any
unsupported or synthetic finding categories. Preserve the existing required
category names and diagnostic output while replacing the subset-only validation
with exact-set validation.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 21b8a795-5d5f-463b-bf6b-b579eb88eb75
📒 Files selected for processing (3)
examples/insurance-claims-agent/README.mdexamples/insurance-claims-agent/insurance_claims/evaluate.pyexamples/insurance-claims-agent/tests/test_review.py
🚧 Files skipped from review as they are similar to previous changes (1)
- examples/insurance-claims-agent/README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Verification
The example summarizes a completed public FEMA appeal. It does not decide a live claim.
Summary by CodeRabbit