Skip to content

feat: keploy-junit5, a JUnit 5 extension for keploy mock record/replay - #188

Merged
slayerjain merged 8 commits into
mainfrom
fix/sdk-jest-pytest-junit-next
Nov 2, 2026
Merged

slayerjain merged 8 commits into
mainfrom
fix/sdk-jest-pytest-junit-next

Conversation

@slayerjain

Copy link
Copy Markdown
Member

What

keploy-junit5/, a JUnit 5 (Jupiter) extension that lets keploy mock record and keploy mock replay
judge a JUnit suite test by test. keploy judges each test once the framework tells it where the test
begins and ends; go-sdk's keploytest and @keploy/playwright do that for Go and Playwright, and JUnit
had nothing. The extension speaks the same wire (keploy's scope protocol v1: claim, begin, end). keploy
keeps every rule, and the extension decides nothing.

Setup is one line in src/test/resources/junit-platform.properties
(junit.jupiter.extensions.autodetection.enabled=true), then
KEPLOY_PROVEN=1 keploy mock record|replay -c "mvn test". Without keploy (KEPLOY_MOCK_AGENT unset or
off) it does nothing.

How it maps onto JUnit

  • Claim: once per test JVM, before the first class's @BeforeAll.
  • Begin: a BeforeEach callback of the extension. Autodetection registers it at the root, so it runs
    before every other extension's BeforeEach callback and every @BeforeEach method, and after the
    class's @BeforeAll.
  • Skip: a test keploy set aside is aborted with keploy's reason, and none of its own methods run.
  • End: an AfterEach callback, which runs after every other one. The outcome comes from
    ExtensionContext.getExecutionException(): no exception is passed, an abort is skipped, and anything
    else is failed. When keploy fails the test at its end, the test fails with keploy's text; if it had
    already failed, keploy's text is added to that failure as a suppressed exception.
  • ID: <class binary name>::<JUnit's method name>, with the invocation index for parameterized
    and repeated tests, e.g. com.shop.OrdersTest::price(java.lang.String)[2]. Each invocation of a
    @ParameterizedClass (JUnit 5.13 and later) runs each test of the class as a test of its own, with the
    class's invocation index before the method: com.shop.OrdersTest::[2]price(java.lang.String).

Fails closed

Each of these fails the affected tests or classes with the reason. No Jupiter test runs unjudged:

  • an agent answer the extension cannot trust (non-2xx, no or wrong protocol version, unreadable, over
    1 MiB);
  • a failed claim, which fails every class (a plain 404 names KEPLOY_PROVEN=1);
  • a KEPLOY_MOCK_AGENT that is not an http URL;
  • registration with @ExtendWith only;
  • JUnit's parallel execution;
  • a @TestFactory, whose dynamic tests share one BeforeEach/AfterEach pair;
  • a test begun twice in one JVM (a rerun, or a second @Suite that selects the same class).

A test run through a JUnit Platform @Suite has the ID it has when it runs directly: the suite and its
engine are not the test.

Limits

  • One test JVM at a time: keploy refuses a second claim.
  • A JVM cannot change its own environment, so it does not export KEPLOY_TEST_PID. A process its
    tests start that also claims is refused by keploy.
  • Only Jupiter tests are judged. The tests of every other JUnit Platform engine run unjudged, as without
    keploy: JUnit 4 tests on the Vintage engine, and Spock, Kotest, jqwik and Cucumber tests. No other
    engine calls a Jupiter extension.

Packaging and publishing

It is a Maven project of its own, not a module of the root pom, released on its own tag with its own
version: release.yml (v*.*.* tags) builds and deploys only the root's modules and never this one, so an
agent release does not republish the extension and the extension can ship a fix without an agent release.
It adds no runtime dependency beyond the suite's own JUnit (the JSON of the wire is read by a small strict
reader in the package). Its coordinates are io.keploy:keploy-junit5.

  • .github/workflows/keploy-junit5-release.yml publishes to Maven Central from a pushed tag
    keploy-junit5-vX.Y.Z. It runs junit5-sdk.yml first (the rows on every JDK and JUnit leg, and the e2e
    lane). It refuses a tag that is not keploy-junit5-vX.Y.Z with X.Y.Z a release version with no leading
    zeros, and refuses to start, naming each one, when a secret it needs is not set. It then sets the pom's
    version to X.Y.Z for that run only, and runs mvn -P release clean deploy: the jar with its sources and
    javadoc jars, each signed, deployed by central-publishing-maven-plugin. Every action in it and in
    junit5-sdk.yml, whose rows run in the same workflow run, is pinned by commit: the publish job holds the
    signing key and the Central token.
  • The pom gains what Central requires and the root pom already has (developers, scm) and the root's
    release profile (sources, javadoc, gpg, central-publishing, the same plugin versions). Its scm
    connections name the git provider (scm:git:...), which the root pom's leave out.
  • What it needs: the repository secrets release.yml reads, under the same names: CENTRAL_USERNAME and
    CENTRAL_PASSWORD (a Central Portal user token for the io.keploy namespace), MAVEN_GPG_PRIVATE_KEY
    (the ASCII-armored release key, id 8541784E4EC36FB8) and MAVEN_GPG_PASSPHRASE. The repository's
    secrets list today shows MAVEN_USERNAME, MAVEN_PASSWORD, GPG_PRIVATE_KEY and GPG_PASSPHRASE, not
    these four, and no organization secret is visible to it. Until the four are set (as repository or
    organization secrets), both this workflow and release.yml's Central deploy refuse to publish; this one
    names the missing secrets.
  • Pull requests that touch keploy-junit5/ run the release build as a dry run (the release profile,
    unsigned, no deploy) and check that the jar, sources jar and javadoc jar are built.
  • Nothing is published by this PR, and keploy-junit5 is not on Maven Central yet, so the README says to
    install it from this folder with mvn install.

Tests

84 rows (mvn verify in keploy-junit5/):

  • JSON reader and writer;
  • the agent client against a fake agent;
  • the environment rules;
  • real JUnit runs of fixture suites through EngineTestKit (order around other extensions and lifecycle
    methods, outcomes, skip, keploy's fail, agent errors, claim failure, refusals, rerun, names, and tests
    run through a JUnit Platform @Suite);
  • a @ParameterizedClass, with a nested one inside it, compiled and run by the row itself on JUnit 5.13
    and later (the rows also build on 5.10, which has no class templates; there the row checks that JUnit is
    older than 5.13).

The new keploy-junit5 workflow runs them on JDK 8 + JUnit 5.10.5, JDK 17 + 5.14.4 and JDK 21 + 6.1.3;
all 84 pass on each of those three.

e2e (keploy-junit5/e2e/lane.sh, CI job e2e, JDK 17): the extension installed with Maven into the
lane's own Maven repository, used by a small JUnit 5.14.4 suite whose tests call a provider over HTTP (a
@Nested class and a parameterized test among them), run with Maven offline under keploy 3.8.106 (the
enterprise build, which judges each test under KEPLOY_PROVEN=1; pinned by version and sha256).

  • Without keploy the suite passes as plain JUnit, and nothing says keploy.
  • Record keeps each test's calls under its class and JUnit's name of it
    (shop.OrdersTest$Books::pricesABook(), shop.PaymentsTest::chargesACard(int)[1]).
  • Replay with the provider stopped passes all 3.
  • A call with no recorded answer fails only its own test, with keploy's miss added to its failure.
  • A recorded call the test no longer makes fails only that test, with keploy's text.
  • A quarantined test is skipped by JUnit, with keploy's reason in Surefire's report, and the replay passes.

The README also says what the lane showed a user needs: keploy.yml's mock.command and mock.name, and
--local to keep the recording on disk.

keploy mock record and replay judge each test of a suite once the test
framework tells keploy where each test begins and ends. Go and Playwright
suites can; JUnit suites could not. keploy-junit5 is that client for JUnit
Jupiter: it claims keploy's run for the test JVM, and begins and ends each
test on keploy's scope protocol v1, the wire go-sdk's keploytest speaks.
keploy keeps every rule; the extension decides nothing.

- JUnit loads it for the whole run (extension autodetection), so it wraps
  every test outside every other extension: the begin comes after the
  class's @BeforeAll and before any BeforeEach callback or @beforeeach
  method, the end after every @AfterEach method and AfterEach callback,
  with the test's outcome (a failed assumption is skipped).
- A test keploy set aside is aborted with keploy's reason before any of its
  own methods runs; a test keploy fails at its end fails with keploy's text.
- It fails closed: an answer it cannot trust fails its test, a failed claim
  fails every class, and a run it cannot wrap whole (registered only with
  @ExtendWith, JUnit's parallel execution, a @testfactory, a test begun a
  second time) fails with the reason.
- Without keploy (KEPLOY_MOCK_AGENT unset or off) it does nothing.

It is a Maven project of its own, not a module of the root pom, so the
release workflow does not build or deploy it. It brings no dependency
beyond the suite's own JUnit (the JSON of the wire is read by a small
strict reader). Rows run on JDK 8 with JUnit 5.10, JDK 17 with JUnit 5.14
and JDK 21 with JUnit 6.1.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
An e2e job for keploy-junit5: e2e/lane.sh installs this commit's extension
with Maven into a Maven repository of the lane's own, then runs a small
JUnit 5 suite (e2e/shop: a shop whose tests call a provider over HTTP, a
@nested class and a parameterized test among them) with Maven, offline,
under keploy 3.8.106, the enterprise build that judges each test under
KEPLOY_PROVEN=1. CI pins the binary by version and sha256, on JDK 17.

The lane checks what a user reads: exit codes, Surefire's summary, each
test's report and its XML report, and the files keploy wrote.
- Without keploy the suite passes as plain JUnit, and nothing says keploy.
- keploy mock record keeps each test's calls under its class and JUnit's
  name of it (shop.OrdersTest$Books::pricesABook(),
  shop.PaymentsTest::chargesACard(int)[1]) and commits 3 replayable tests.
- keploy mock replay, with the provider stopped, passes all 3.
- A call with no recorded answer fails its own test, keploy's miss added
  to its failure, and the other 2 pass.
- A recorded call the test no longer makes fails that test at its end,
  with keploy's text.
- A test quarantined in the set is skipped by JUnit, keploy's reason in
  Surefire's report, and the replay passes.

The README now says what the lane showed a user needs: keploy.yml's
mock.command and mock.name, and --local to keep the recording on disk.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
Rows for behaviour of keploy-junit5 that no row pinned yet:

- The JSON reader refuses an object member name with no opening quote
  and a raw U+001F inside a string, and reads every digit (9 too) and a
  space inside a string.
- An agent answer whose action is "" is no action: the test runs, and
  its end does not fail it.

No change outside keploy-junit5/src/test.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
Under junit-platform-suite a test's unique ID is
[engine:junit-platform-suite]/[suite:...]/[engine:junit-jupiter]/
[class:...]/[method:...]. The suite segment fell to the default case,
so every plain @test run through a @suite failed with "cannot name ...
as one test (a suite; ...): write it as a @test or a
@ParameterizedTest", a wrong remedy for a test that is one.

The suite segment, like the engine ones, is not the test: a test run
through a suite now has the ID it has when run directly. Two suites
that select one class run each of its tests twice in a JVM; the second
run is refused as a rerun, with its reason. Rows run Named (a
parameterized, a repeated, a @nested test and a @testfactory) through a
suite, and one class through two suites, on every CI leg.

The README says that only Jupiter tests are judged: the tests of every
other JUnit Platform engine (Vintage, Spock, Kotest, jqwik, Cucumber)
run unjudged, as without keploy.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
JUnit 5.13 added class templates: a @ParameterizedClass runs each of its
tests once per invocation of the class, each in a BeforeEach/AfterEach
pair of its own. Their unique IDs carry class-template,
nested-class-template and class-template-invocation segments, which
TestRef did not know, so every such test failed under keploy with a
wrong reason: that JUnit runs it inside one BeforeEach/AfterEach pair, as
a @testfactory's dynamic tests.

Each is a test of its own now, named with the class's invocation index
before the method, one per class template it runs in, outermost first:
Prices::[2]price(), Prices::[2]tax(java.lang.String)[1] and
Prices$Coupons::[2][1]coupon() for a @ParameterizedClass Coupons nested
in another.

The rows build on JUnit 5.10, which has no class templates, so the new
row compiles its fixture with the JDK's compiler when JUnit has them, and
runs it with keploy-junit5 loaded. On JUnit 5.10 it checks that JUnit is
older than 5.13. At the old TestRef the row fails on JUnit 5.14.4 and
6.1.3. 84 rows pass on JDK 8 + 5.10.5, JDK 17 + 5.14.4 and JDK 21 +
6.1.3.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
keploy-junit5-release.yml publishes io.keploy:keploy-junit5 from a
pushed tag keploy-junit5-vX.Y.Z. It runs junit5-sdk.yml first (the rows
on every JDK and JUnit leg, and the e2e lane). It refuses a tag that is
not keploy-junit5-vX.Y.Z with X.Y.Z a release version, and refuses to
start, naming each one, when a secret it needs is not set. It then sets
the pom's version to X.Y.Z for that run only and runs
mvn -P release clean deploy: the jar with its sources and javadoc jars,
each signed, deployed to Central.

keploy-junit5 stays a project of its own with its own version:
release.yml (v*.*.* tags) builds and deploys only the root's modules,
so an agent release does not republish the extension, and the
extension can ship a fix without an agent release. It signs and deploys
with the key and the secret names release.yml uses: CENTRAL_USERNAME,
CENTRAL_PASSWORD, MAVEN_GPG_PRIVATE_KEY and MAVEN_GPG_PASSPHRASE.

The pom gains what Central requires (developers, scm) and the root
pom's release profile, with the same plugin versions. Pull requests that
touch keploy-junit5/ run the release build unsigned, with no deploy,
and check that the three jars are built. junit5-sdk.yml gains
workflow_call so the tag can run it. The README's Releasing section
names the tag and the secrets.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
The publish job holds the signing key and the Central token, and the
rows it waits on run in the same workflow run. Each actions/checkout and
actions/setup-java in keploy-junit5-release.yml and junit5-sdk.yml now
names the commit its v4 tag points to today (checkout v4.4.0, setup-java
v4.9.1), so a moved tag cannot change what runs.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
Maven reads an scm connection as scm:<provider>:<url>. The ones copied
from the root pom left the provider out; they now read
scm:git:https://github.com/keploy/java-sdk.git and
scm:git:ssh://git@github.com/keploy/java-sdk.git.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
@slayerjain
slayerjain merged commit e8ec940 into main Nov 2, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant