fix: don't call methods on mock objects while recording - #329
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
MockDetector can return invalid null lookup results and cannot reliably detect mocks across isolated class loaders.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
This PR prevents AppMap recording from invoking methods on Mockito mocks and adds integration coverage.
Changes:
- Adds reflective Mockito detection with a
[mocked]placeholder. - Adds Mockito regression tests with agent and control runs.
- Adds standalone Gradle/Bats test-project configuration.
File summaries
| File | Summary |
|---|---|
agent/test/mockito/src/test/java/com/example/mockito/MockRecordingTest.java |
Tests Mockito stubbing and recording behavior. |
agent/test/mockito/src/main/java/com/example/mockito/Service.java |
Provides the instrumented test fixture. |
agent/test/mockito/src/main/java/com/example/mockito/Calculator.java |
Defines the mocked collaborator. |
agent/test/mockito/settings.gradle |
Defines the Gradle project. |
agent/test/mockito/mockito.bats |
Runs control and agent integration tests. |
agent/test/mockito/build.gradle |
Configures test tasks and dependencies. |
agent/test/mockito/appmap.yml |
Configures recorded packages. |
agent/test/mockito/.gitignore |
Ignores generated files. |
agent/src/main/java/com/appland/appmap/util/MockDetector.java |
Detects Mockito mocks reflectively. |
agent/src/main/java/com/appland/appmap/output/v1/Value.java |
Records placeholders for mocks. |
Review details
Suppressed comments (2)
agent/src/main/java/com/appland/appmap/util/MockDetector.java:47
getClassLoader() == nulldoes not prove an object cannot be a Mockito mock: Mockito's inline mock maker can instrument bootstrap-loaded concrete types such asjava.util.ArrayList. This early return causesValue.freeze()to calltoString()on those mocks, reintroducing the matcher-state corruption; use a fallback loader that can resolve Mockito and add a regression test for a bootstrap-loaded mock.
if (loader == null) {
// Bootstrap classes are never mocks.
return null;
agent/src/main/java/com/appland/appmap/util/MockDetector.java:54
- The API lookup is limited to the inspected object's class loader. With an isolated application loader, an inline mock can be an instance of an application class while Mockito is visible only from the test/context loader; this
Class.forNamethen fails, and the cachednullmakes every later freeze calltoString()on that mock. Please add a compatible context/system/agent-loader fallback instead of assuming the object loader can see Mockito.
Method mockingDetails = Class.forName("org.mockito.Mockito", false, loader)
.getMethod("mockingDetails", Object.class);
Method isMock = Class.forName("org.mockito.MockingDetails", false, loader)
.getMethod("isMock");
- Files reviewed: 10/10 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
0849838 to
20bc8b5
Compare
Recording a value called toString() on it. When the value is a mock,
that call is an invocation as far as the mocking framework is concerned,
and mocking frameworks keep thread-local state between one call and the
next. Mockito collects argument matchers on a stack and binds them to
the next invocation it sees, and when() applies to the most recent
invocation. A toString() from the agent lands in the middle of that
handshake, so Mockito attributes the matcher or the stubbing to the
agent's call instead of the test's.
The result is that the stubbing a test set up silently never applies,
and the stubbed method returns the Java type default. A test that does
when(service.getData()).thenReturn("expectedValue");
fails with "expected:<expectedValue> but was:<null>" under the agent and
passes without it. Where all arguments use matchers, Mockito reports
"2 matchers expected, 1 recorded" instead.
Recognize mocks and record a placeholder rather than asking them for a
value. Detection goes through the mocking framework's own API, because
the class name isn't a reliable signal: Mockito's inline mock maker
(the default since 5.0) mocks a class by retransforming it, so a mock
of com.example.Foo is an instance of com.example.Foo. The lookup is
resolved against the class loader of the object being inspected and
cached per class, so it costs nothing when no mocking framework is on
the class path.
The new test project has a control task that runs the same tests with
no agent attached, so a failure under the agent is attributable.
Assisted-by: Claude:claude-opus-5
20bc8b5 to
6bcd7dd
Compare
There was a problem hiding this comment.
🟢 Approval recommended
The functional change is well-contained and backed by targeted regression tests; remaining feedback is limited to a minor documentation correction.
Review details
- Files reviewed: 10/10 changed files
- Comments generated: 1
- Review effort level: Lite
Update comment to clarify Mockito's mock-detection behavior. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
🎉 This PR is included in version 1.30.2 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
Recording a value called toString() on it. When the value is a mock, that call is an invocation as far as the mocking framework is concerned, and mocking frameworks keep thread-local state between one call and the next. Mockito collects argument matchers on a stack and binds them to the next invocation it sees, and when() applies to the most recent invocation. A toString() from the agent lands in the middle of that handshake, so Mockito attributes the matcher or the stubbing to the agent's call instead of the test's.
The result is that the stubbing a test set up silently never applies, and the stubbed method returns the Java type default. A test that does
fails with "expected: but was:" under the agent and passes without it. Where all arguments use matchers, Mockito reports "2 matchers expected, 1 recorded" instead.
Recognize mocks and record a placeholder rather than asking them for a value. Detection goes through the mocking framework's own API, because the class name isn't a reliable signal: Mockito's inline mock maker (the default since 5.0) mocks a class by retransforming it, so a mock of com.example.Foo is an instance of com.example.Foo. The lookup is resolved against the class loader of the object being inspected and cached per class, so it costs nothing when no mocking framework is on the class path.
The new test project has a control task that runs the same tests with no agent attached, so a failure under the agent is attributable.
Assisted-by: Claude:claude-opus-5