Add Crucible to Mutation Testing#96
Conversation
|
Review changes with |
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
Reviewer's guide (collapsed on small PRs)Reviewer's GuideAdds Crucible to the Mutation Testing tools list in README in alphabetical order, with a concise description emphasizing adversarial test-hardening for AI-written code. Sequence diagram for Crucible adversarial test-hardening workflowsequenceDiagram
actor Developer
participant Crucible
participant TesterAgent
participant Mutmut
participant CriticAgent
participant Pytest
Developer->>Crucible: run crucible score
Crucible->>TesterAgent: generate_initial_tests
TesterAgent-->>Developer: tests_written
Crucible->>Mutmut: run_mutation_testing
Mutmut-->>Crucible: surviving_mutants
Crucible->>CriticAgent: generate_targeted_tests
CriticAgent-->>Developer: new_tests_for_survivors
Crucible->>Pytest: run_test_suite
Pytest-->>Crucible: mutation_score
Crucible-->>Developer: report_surviving_mutants_and_score
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've left some high level feedback:
- Consider shortening Crucible’s description to a single concise phrase (similar to other entries) and trimming the multi-sentence explanation to keep the README’s mutation testing list consistent in style and length.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- Consider shortening Crucible’s description to a single concise phrase (similar to other entries) and trimming the multi-sentence explanation to keep the README’s mutation testing list consistent in style and length.Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds a new entry for 'Crucible' to the list of Python mutation testing resources in the README.md. The reviewer suggested capitalizing 'mutmut' as 'Mutmut' for consistency with other entries in the list.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
|
|
||
| - [bough](https://github.com/CodeEnPlace/bough) - Bough is a polyglot incremental mutation tester. | ||
| - [Cosmic Ray](https://github.com/sixty-north/cosmic-ray) - makes small changes to your source code, running your test suite for each one. | ||
| - [Crucible](https://github.com/Jott2121/crucible) - Adversarial test-hardening for AI-written code, built on mutmut. A Tester agent writes tests, mutation testing names the survivors, and a Critic agent kills exactly those. |
There was a problem hiding this comment.
For consistency with the entry for Mutmut on line 130, please capitalize "mutmut" as "Mutmut".
| - [Crucible](https://github.com/Jott2121/crucible) - Adversarial test-hardening for AI-written code, built on mutmut. A Tester agent writes tests, mutation testing names the survivors, and a Critic agent kills exactly those. | |
| - [Crucible](https://github.com/Jott2121/crucible) - Adversarial test-hardening for AI-written code, built on Mutmut. A Tester agent writes tests, mutation testing names the survivors, and a Critic agent kills exactly those. |
![llamapreview[bot]](https://avatars.githubusercontent.com/in/1023541?s=60&v=4){kind=small}
There was a problem hiding this comment.
Auto Pull Request Review from LlamaPReview
Review Status: Automated Review Skipped
Dear contributor,
Thank you for your Pull Request. LlamaPReview has analyzed your changes and determined that this PR does not require an automated code review.
Analysis Result:
PR only contains documentation changes (1 files)
Technical Context:
Documentation changes typically include:
- Markdown/RST file updates
- API documentation
- Code comments
- README updates
- Documentation in /docs directory
- License and contribution files
We're continuously improving our PR analysis capabilities. Have thoughts on when and how LlamaPReview should perform automated reviews? Share your insights in our GitHub Discussions.
Best regards,
LlamaPReview Team
|
Tick the box to add this pull request to the merge queue (same as
|
1 participant
Adds Crucible to the Mutation Testing section (alphabetical, between Cosmic Ray and Mutatest).
What it is: adversarial test-hardening for AI-written code, built on top of mutmut rather than competing with it. A Tester agent writes tests, mutation testing names the surviving mutants, and a Critic agent is handed those named survivors and writes tests to kill exactly them. Every verdict is mechanical — pytest kills the mutant or it doesn't — so no model ever grades model output.
Why it belongs on this list: the coverage/fault-detection gap gets systematically worse when a model writes both the code and its tests, and mutation testing is the only cheap way to see it. On a real module with 7 passing tests and 97% line coverage, 25 of 71 injected defects survived.
mutmutpinned to 3.6.0crucible score) calls no model and needs no API keyHappy to reword the entry or move it if you'd prefer a different section or a shorter description.
Summary by Sourcery
Documentation: