Repository navigation
feat: keploy-junit5, a JUnit 5 extension for keploy mock record/replay - #188
Merged
Merged
Conversation
keploy mock record and replay judge each test of a suite once the test framework tells keploy where each test begins and ends. Go and Playwright suites can; JUnit suites could not. keploy-junit5 is that client for JUnit Jupiter: it claims keploy's run for the test JVM, and begins and ends each test on keploy's scope protocol v1, the wire go-sdk's keploytest speaks. keploy keeps every rule; the extension decides nothing. - JUnit loads it for the whole run (extension autodetection), so it wraps every test outside every other extension: the begin comes after the class's @BeforeAll and before any BeforeEach callback or @beforeeach method, the end after every @AfterEach method and AfterEach callback, with the test's outcome (a failed assumption is skipped). - A test keploy set aside is aborted with keploy's reason before any of its own methods runs; a test keploy fails at its end fails with keploy's text. - It fails closed: an answer it cannot trust fails its test, a failed claim fails every class, and a run it cannot wrap whole (registered only with @ExtendWith, JUnit's parallel execution, a @testfactory, a test begun a second time) fails with the reason. - Without keploy (KEPLOY_MOCK_AGENT unset or off) it does nothing. It is a Maven project of its own, not a module of the root pom, so the release workflow does not build or deploy it. It brings no dependency beyond the suite's own JUnit (the JSON of the wire is read by a small strict reader). Rows run on JDK 8 with JUnit 5.10, JDK 17 with JUnit 5.14 and JDK 21 with JUnit 6.1. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
An e2e job for keploy-junit5: e2e/lane.sh installs this commit's extension with Maven into a Maven repository of the lane's own, then runs a small JUnit 5 suite (e2e/shop: a shop whose tests call a provider over HTTP, a @nested class and a parameterized test among them) with Maven, offline, under keploy 3.8.106, the enterprise build that judges each test under KEPLOY_PROVEN=1. CI pins the binary by version and sha256, on JDK 17. The lane checks what a user reads: exit codes, Surefire's summary, each test's report and its XML report, and the files keploy wrote. - Without keploy the suite passes as plain JUnit, and nothing says keploy. - keploy mock record keeps each test's calls under its class and JUnit's name of it (shop.OrdersTest$Books::pricesABook(), shop.PaymentsTest::chargesACard(int)[1]) and commits 3 replayable tests. - keploy mock replay, with the provider stopped, passes all 3. - A call with no recorded answer fails its own test, keploy's miss added to its failure, and the other 2 pass. - A recorded call the test no longer makes fails that test at its end, with keploy's text. - A test quarantined in the set is skipped by JUnit, keploy's reason in Surefire's report, and the replay passes. The README now says what the lane showed a user needs: keploy.yml's mock.command and mock.name, and --local to keep the recording on disk. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
Rows for behaviour of keploy-junit5 that no row pinned yet: - The JSON reader refuses an object member name with no opening quote and a raw U+001F inside a string, and reads every digit (9 too) and a space inside a string. - An agent answer whose action is "" is no action: the test runs, and its end does not fail it. No change outside keploy-junit5/src/test. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
Under junit-platform-suite a test's unique ID is [engine:junit-platform-suite]/[suite:...]/[engine:junit-jupiter]/ [class:...]/[method:...]. The suite segment fell to the default case, so every plain @test run through a @suite failed with "cannot name ... as one test (a suite; ...): write it as a @test or a @ParameterizedTest", a wrong remedy for a test that is one. The suite segment, like the engine ones, is not the test: a test run through a suite now has the ID it has when run directly. Two suites that select one class run each of its tests twice in a JVM; the second run is refused as a rerun, with its reason. Rows run Named (a parameterized, a repeated, a @nested test and a @testfactory) through a suite, and one class through two suites, on every CI leg. The README says that only Jupiter tests are judged: the tests of every other JUnit Platform engine (Vintage, Spock, Kotest, jqwik, Cucumber) run unjudged, as without keploy. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
JUnit 5.13 added class templates: a @ParameterizedClass runs each of its tests once per invocation of the class, each in a BeforeEach/AfterEach pair of its own. Their unique IDs carry class-template, nested-class-template and class-template-invocation segments, which TestRef did not know, so every such test failed under keploy with a wrong reason: that JUnit runs it inside one BeforeEach/AfterEach pair, as a @testfactory's dynamic tests. Each is a test of its own now, named with the class's invocation index before the method, one per class template it runs in, outermost first: Prices::[2]price(), Prices::[2]tax(java.lang.String)[1] and Prices$Coupons::[2][1]coupon() for a @ParameterizedClass Coupons nested in another. The rows build on JUnit 5.10, which has no class templates, so the new row compiles its fixture with the JDK's compiler when JUnit has them, and runs it with keploy-junit5 loaded. On JUnit 5.10 it checks that JUnit is older than 5.13. At the old TestRef the row fails on JUnit 5.14.4 and 6.1.3. 84 rows pass on JDK 8 + 5.10.5, JDK 17 + 5.14.4 and JDK 21 + 6.1.3. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
keploy-junit5-release.yml publishes io.keploy:keploy-junit5 from a pushed tag keploy-junit5-vX.Y.Z. It runs junit5-sdk.yml first (the rows on every JDK and JUnit leg, and the e2e lane). It refuses a tag that is not keploy-junit5-vX.Y.Z with X.Y.Z a release version, and refuses to start, naming each one, when a secret it needs is not set. It then sets the pom's version to X.Y.Z for that run only and runs mvn -P release clean deploy: the jar with its sources and javadoc jars, each signed, deployed to Central. keploy-junit5 stays a project of its own with its own version: release.yml (v*.*.* tags) builds and deploys only the root's modules, so an agent release does not republish the extension, and the extension can ship a fix without an agent release. It signs and deploys with the key and the secret names release.yml uses: CENTRAL_USERNAME, CENTRAL_PASSWORD, MAVEN_GPG_PRIVATE_KEY and MAVEN_GPG_PASSPHRASE. The pom gains what Central requires (developers, scm) and the root pom's release profile, with the same plugin versions. Pull requests that touch keploy-junit5/ run the release build unsigned, with no deploy, and check that the three jars are built. junit5-sdk.yml gains workflow_call so the tag can run it. The README's Releasing section names the tag and the secrets. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
The publish job holds the signing key and the Central token, and the rows it waits on run in the same workflow run. Each actions/checkout and actions/setup-java in keploy-junit5-release.yml and junit5-sdk.yml now names the commit its v4 tag points to today (checkout v4.4.0, setup-java v4.9.1), so a moved tag cannot change what runs. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
Maven reads an scm connection as scm:<provider>:<url>. The ones copied from the root pom left the provider out; they now read scm:git:https://github.com/keploy/java-sdk.git and scm:git:ssh://git@github.com/keploy/java-sdk.git. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
keploy-junit5/, a JUnit 5 (Jupiter) extension that letskeploy mock recordandkeploy mock replayjudge a JUnit suite test by test. keploy judges each test once the framework tells it where the test
begins and ends; go-sdk's
keploytestand@keploy/playwrightdo that for Go and Playwright, and JUnithad nothing. The extension speaks the same wire (keploy's scope protocol v1: claim, begin, end). keploy
keeps every rule, and the extension decides nothing.
Setup is one line in
src/test/resources/junit-platform.properties(
junit.jupiter.extensions.autodetection.enabled=true), thenKEPLOY_PROVEN=1 keploy mock record|replay -c "mvn test". Without keploy (KEPLOY_MOCK_AGENTunset oroff) it does nothing.How it maps onto JUnit
@BeforeAll.before every other extension's BeforeEach callback and every
@BeforeEachmethod, and after theclass's
@BeforeAll.ExtensionContext.getExecutionException(): no exception is passed, an abort is skipped, and anythingelse is failed. When keploy fails the test at its end, the test fails with keploy's text; if it had
already failed, keploy's text is added to that failure as a suppressed exception.
<class binary name>::<JUnit's method name>, with the invocation index for parameterizedand repeated tests, e.g.
com.shop.OrdersTest::price(java.lang.String)[2]. Each invocation of a@ParameterizedClass(JUnit 5.13 and later) runs each test of the class as a test of its own, with theclass's invocation index before the method:
com.shop.OrdersTest::[2]price(java.lang.String).Fails closed
Each of these fails the affected tests or classes with the reason. No Jupiter test runs unjudged:
1 MiB);
KEPLOY_PROVEN=1);KEPLOY_MOCK_AGENTthat is not an http URL;@ExtendWithonly;@TestFactory, whose dynamic tests share one BeforeEach/AfterEach pair;@Suitethat selects the same class).A test run through a JUnit Platform
@Suitehas the ID it has when it runs directly: the suite and itsengine are not the test.
Limits
KEPLOY_TEST_PID. A process itstests start that also claims is refused by keploy.
keploy: JUnit 4 tests on the Vintage engine, and Spock, Kotest, jqwik and Cucumber tests. No other
engine calls a Jupiter extension.
Packaging and publishing
It is a Maven project of its own, not a module of the root pom, released on its own tag with its own
version:
release.yml(v*.*.*tags) builds and deploys only the root's modules and never this one, so anagent release does not republish the extension and the extension can ship a fix without an agent release.
It adds no runtime dependency beyond the suite's own JUnit (the JSON of the wire is read by a small strict
reader in the package). Its coordinates are
io.keploy:keploy-junit5..github/workflows/keploy-junit5-release.ymlpublishes to Maven Central from a pushed tagkeploy-junit5-vX.Y.Z. It runsjunit5-sdk.ymlfirst (the rows on every JDK and JUnit leg, and the e2elane). It refuses a tag that is not
keploy-junit5-vX.Y.Zwith X.Y.Z a release version with no leadingzeros, and refuses to start, naming each one, when a secret it needs is not set. It then sets the pom's
version to X.Y.Z for that run only, and runs
mvn -P release clean deploy: the jar with its sources andjavadoc jars, each signed, deployed by
central-publishing-maven-plugin. Every action in it and injunit5-sdk.yml, whose rows run in the same workflow run, is pinned by commit: the publish job holds thesigning key and the Central token.
releaseprofile (sources, javadoc, gpg, central-publishing, the same plugin versions). Its scmconnections name the
gitprovider (scm:git:...), which the root pom's leave out.release.ymlreads, under the same names:CENTRAL_USERNAMEandCENTRAL_PASSWORD(a Central Portal user token for theio.keploynamespace),MAVEN_GPG_PRIVATE_KEY(the ASCII-armored release key, id
8541784E4EC36FB8) andMAVEN_GPG_PASSPHRASE. The repository'ssecrets list today shows
MAVEN_USERNAME,MAVEN_PASSWORD,GPG_PRIVATE_KEYandGPG_PASSPHRASE, notthese four, and no organization secret is visible to it. Until the four are set (as repository or
organization secrets), both this workflow and
release.yml's Central deploy refuse to publish; this onenames the missing secrets.
keploy-junit5/run the release build as a dry run (thereleaseprofile,unsigned, no deploy) and check that the jar, sources jar and javadoc jar are built.
install it from this folder with
mvn install.Tests
84 rows (
mvn verifyinkeploy-junit5/):methods, outcomes, skip, keploy's fail, agent errors, claim failure, refusals, rerun, names, and tests
run through a JUnit Platform
@Suite);@ParameterizedClass, with a nested one inside it, compiled and run by the row itself on JUnit 5.13and later (the rows also build on 5.10, which has no class templates; there the row checks that JUnit is
older than 5.13).
The new
keploy-junit5workflow runs them on JDK 8 + JUnit 5.10.5, JDK 17 + 5.14.4 and JDK 21 + 6.1.3;all 84 pass on each of those three.
e2e (
keploy-junit5/e2e/lane.sh, CI jobe2e, JDK 17): the extension installed with Maven into thelane's own Maven repository, used by a small JUnit 5.14.4 suite whose tests call a provider over HTTP (a
@Nestedclass and a parameterized test among them), run with Maven offline under keploy 3.8.106 (theenterprise build, which judges each test under
KEPLOY_PROVEN=1; pinned by version and sha256).(
shop.OrdersTest$Books::pricesABook(),shop.PaymentsTest::chargesACard(int)[1]).The README also says what the lane showed a user needs:
keploy.yml'smock.commandandmock.name, and--localto keep the recording on disk.