diff --git a/.claude/skills/test/SKILL.md b/.claude/skills/test/SKILL.md index d7b28f6d260..54806497514 100644 --- a/.claude/skills/test/SKILL.md +++ b/.claude/skills/test/SKILL.md @@ -1,13 +1,14 @@ --- name: test -description: Run tests for a specific SDK module. Use when asked to "run tests", "test module", "run unit tests", "run system tests", "run e2e tests", or test a specific class. Auto-detects unit vs system tests. Supports interactive mode. +description: Run tests for a specific SDK module. Use when asked to "run tests", "test module", "run unit tests", "run integration tests", "run e2e tests", or test a specific class. Auto-detects unit tests vs integration test suites. Supports interactive mode. allowed-tools: Bash, Read, Glob, AskUserQuestion argument-hint: [interactive] [test-class-filter] --- # Run Tests -Run tests for a specific module. Auto-detects whether to run unit tests or system tests. +Run tests for a specific module. Auto-detects whether to run unit tests or an integration test +suite. ## Step 0: Check for Interactive Mode @@ -19,7 +20,7 @@ In interactive mode, use AskUserQuestion at decision points as described in the The argument can be either: - A **file path** (e.g., `@sentry/src/test/java/io/sentry/ScopesTest.kt`) -- A **module name** (e.g., `sentry-android-core`, `sentry-samples-spring-boot-4`) +- A **module name** (e.g., `sentry-android-core`, `sentry-spring-boot-4`) - A **module name + test filter** (e.g., `sentry ScopesTest`) Extract the module name and optional test class filter from the argument. @@ -30,10 +31,13 @@ Extract the module name and optional test class filter from the argument. | Signal | Test Type | |--------|-----------| -| Path contains `sentry-samples/` | System test | -| Module name starts with `sentry-samples-` | System test | +| Path contains `/src/integrationTest*/` or `/src/agentFixture/` | Integration test suite | +| Path contains `sentry-samples/`, or module name starts with `sentry-samples-` | None: samples have no tests | | Everything else | Unit test | +For a sample, tell the user that samples have no tests, and that the end-to-end tests for what +the sample shows live in the module it uses (see the `e2e_tests` rule). + ## Step 3a: Run Unit Tests Determine the Gradle test task: @@ -60,31 +64,23 @@ Without a filter: ./gradlew '::' --info ``` -## Step 3b: Run System Tests +## Step 3b: Run an Integration Test Suite -System tests require the Python-based test runner which manages a mock Sentry server and sample app lifecycle. +The suite's task is named after its source set directory: `src/integrationTestWebflux/...` runs +with `integrationTestWebflux`. The OpenTelemetry agent suite runs in two tasks, one per init mode, +so for `src/integrationTestOtelAgent/` or `src/agentFixture/` run both: -1. Ensure the Python venv exists: ```bash -test -d .venv || make setupPython +./gradlew '::integrationTestOtelAgent' '::integrationTestOtelAgentWithoutAutoInit' ``` -2. Extract the sample module name. For file paths like `sentry-samples//src/...`, the sample module is the directory name (e.g., `sentry-samples-spring`). +Otherwise, with or without a test class filter: -3. Run the system test: ```bash -.venv/bin/python test/system-test-runner.py test --module +./gradlew '::' --tests="**" ``` -This starts the mock Sentry server, starts the sample app (Spring Boot/Tomcat/CLI), runs tests via `./gradlew :sentry-samples::systemTest`, and cleans up afterwards. - -To run **every** system test instead of one module, use the Makefile targets — they also create the -venv for you: - -```bash -make systemTest # all system tests (--all) -make systemTestInteractive # pick the setups to run (--interactive) -``` +To run every Spring integration suite at once, use `./gradlew integrationTests`. ## Step 4: Report Results diff --git a/.cursor/rules/e2e_tests.mdc b/.cursor/rules/e2e_tests.mdc index 17e088774b5..0f154dbf54c 100644 --- a/.cursor/rules/e2e_tests.mdc +++ b/.cursor/rules/e2e_tests.mdc @@ -3,26 +3,82 @@ alwaysApply: false description: Java SDK End to End Tests --- -# Java SDK End to End Tests (System Tests) +# Java SDK End to End Tests -The samples in the `sentry-samples` directory are used to run end to end tests against them. +End-to-end tests live in the module whose code they test, not in `sentry-samples`. Samples have no +tests. A test runs the code the way an app would and asserts what reaches Sentry over the real HTTP +transport. -There is a python script (`system-test-runner.py`) that can be used to run one (using `--module SAMPLE_NAME`) or all (using `--all`) system tests. +## MockSentryServer -The script has an interactive mode (`-i`) which allows selection of test setups to execute, whether to run the tests or just prepare infrastructure for testing from IDE. +`MockSentryServer` (`sentry-test-support`) is an in-process HTTP server on a free port. Point the +SDK at `server.dsn`, then: -The tests run a mock Sentry server via `system-test-sentry-server.py`. Any system under test will then have a DSN set that reflects this local mock server like `http://502f25099c204a2fbf4cb16edc5975d1@localhost:8000/0`. -By using this local DSN, the system under test sends events to the local mock server. -The tests can then use `TestHelper` to assert envelopes that were received by the mock server. -`TestHelper` uses HTTP requests to retrieve the JSON payload of the received events and deserialize them back to objects for easier assertion. -Tests can then assert events, transactions, logs etc. similar to how they would appear in `beforeSend` and similar callbacks. +- `awaitEnvelope(timeoutMillis) { predicate }` polls for a matching envelope and fails with what it + received on timeout. +- `assertNoEnvelope(millis) { predicate }` checks that nothing matching arrives. +- `MockSentryServer { ... }` takes a hook that runs before each check while waiting, for example to + flush an OpenTelemetry tracer provider. -`TestHelper` has a lot of helper methods for asserting, e.g. by span name, log body etc. +`EnvelopeMatchers.kt` has the accessors and matchers: `event()`, `transaction()`, `logs()`, +`metrics()`, `profileChunk()`, `hasSpanWithOp`, `hasFlag`, `hasLogWithAttribute`, `hasMetric` and +so on. -The end to end tests either expect the system under test to either be running on a server or call `java -jar` to execute a CLI system under test. +Call `Sentry.flush()` before awaiting when Sentry runs in the test's JVM: logs and metrics are +batched. Envelopes that arrive late can outlive a server's `reset()`, so tests that share a server +match on values only they produce. -For Spring Boot, we spin up the Spring Boot server. The tests then send requests to that server and assert what is sent to Sentry. +## Where the tests are -End to end tests are also executed on CI using a matrix build, as defined in `.github/workflows/system-tests-backend.yml`. +- **Core and plain integrations** use their module's `src/test`. Examples: `SentryEndToEndTest` + (`sentry`), `SentryAppenderIntegrationTest` (`sentry-logback`, `sentry-log4j2`, loading the + appender from XML), `SentryHandlerIntegrationTest` (`sentry-jul`), + `ContinuousProfilerEndToEndTest` (`sentry-async-profiler`), `SentryKafkaEndToEndTest` + (`sentry-kafka`), `AgentlessEndToEndTest` and `OtlpEndToEndTest` (`sentry-opentelemetry-*`). +- **Spring Boot** (`sentry-spring-boot`, `sentry-spring-boot-jakarta`, `sentry-spring-boot-4`) has + one JVM test suite per flavor, because a flavor is a classpath: Sentry's auto-configuration and + OpenTelemetry mode switch on which classes are present. + - `integrationTest`: servlet app. + - `integrationTestWebflux`: reactive app. + - `integrationTestOtelNoAgent`: `sentry-opentelemetry-agentless-spring`. + - `integrationTestOtelAgent` and `integrationTestOtelAgentWithoutAutoInit`: the built agent. + - `integrationTestOtlp` (Boot 4): `sentry-opentelemetry-otlp-spring`. + - `integrationTestLog4j2` (Boot 3, 4): Log4j2 instead of Logback, for the auto-configured + Log4j2 appender. -Some of the samples are tested in multiple ways, e.g. with OpenTelemetry Agent auto init turned on and off. + Each suite has a base class with `@SpringBootTest(RANDOM_PORT)` and a `@DynamicPropertySource` + that sets `sentry.dsn`. Subclasses share one context and one server. +- **Plain Spring** (`sentry-spring`, `sentry-spring-jakarta`, `sentry-spring-7`) has an + `integrationTest` suite that starts the app in an embedded Tomcat, the way a WAR starts. Spring + Boot stays off its classpath, because `SentrySpringServletContainerInitializer` behaves + differently with it. + +## The OpenTelemetry agent + +The agent must not run in the test's JVM, because it would instrument the test's own HTTP client +and server. The Spring Boot modules have an `agentFixture` source set with the app. `AgentApp` +starts it once per test JVM in a child process, with `ChildJvmApp` and the agent jar passed through +`ChildJvmTestArguments`. The suite's task runs with the agent initializing Sentry, and +`integrationTestOtelAgentWithoutAutoInit` runs the same tests with the Spring Boot starter doing +it. Gradle runs the two in parallel. + +## Running them + +- A module's `check` runs its suites. `./gradlew integrationTests` runs every Spring suite. +- The `spring-boot-{2,3,4}-matrix.yml` workflows run them against each supported Spring Boot + version, by setting the version in `gradle/libs.versions.toml`. + +## Gotchas + +- Don't use the context's `TestRestTemplate`. It comes from the `RestTemplateBuilder` that Sentry + customizes, and that the OpenTelemetry starter instruments, so it adds or replaces trace headers + on the requests a test sends. Build one outside the context. +- Sentry exports OpenTelemetry spans in batches every 5 seconds. In-process, flush the tracer + provider from the server's hook. With the agent this isn't possible, so each test waits for a + batch and negative checks wait at least 6 seconds. +- Sentry is global, so a second live Spring context in an OpenTelemetry suite takes it over. Keep + one live context per JVM (`@DirtiesContext` when a test needs its own). +- The async profiler rejects a traces path with a `.` anywhere in its absolute form. Put the path + in `java.io.tmpdir`. +- Kafka uses an in-JVM broker (`@EmbeddedKafka` or spring-kafka-test's broker classes), so no + Docker is needed. diff --git a/.cursor/rules/new_module.mdc b/.cursor/rules/new_module.mdc index d499acf03a0..b9fdde3df6a 100644 --- a/.cursor/rules/new_module.mdc +++ b/.cursor/rules/new_module.mdc @@ -69,8 +69,8 @@ static { addIntegrationToSdkVersion("{IntegrationName}"); ``` -7. If adding a JVM sample, add E2E (system) tests, following the structure we have in the existing JVM examples. - The test should then be added to `test/system-test-runner.py` and `.github/workflows/system-tests-backend.yml`. +7. Samples have no tests. Add end-to-end coverage of what the sample shows to the module it uses, + against `MockSentryServer`, as described in the `e2e_tests` rule. `sentry-bom` and the root `build.gradle.kts` need no change — they iterate over subprojects. diff --git a/AGENTS.md b/AGENTS.md index badfba1b3e4..85e3de87c3b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -20,7 +20,7 @@ rule file in `.cursor/rules/`: | `continuous_profiling_jvm` | `sentry-async-profiler`, `IContinuousProfiler`, `ProfileChunk`, JFR files, `ProfileLifecycle` | | `opentelemetry` | `sentry-opentelemetry-*`, agent vs agentless, span processing, sampling, context propagation | | `new_module` | Adding a new integration or sample module | -| `e2e_tests` | System tests, sample applications, `system-test-runner.py`, mock Sentry server | +| `e2e_tests` | End-to-end tests, `MockSentryServer`, integration test suites, the OpenTelemetry agent fixture, `integrationTests` | Rules can be combined — a tracing scope issue may need both `scopes` and `opentelemetry`. There is no rule for Android profiling yet; read the `sentry-android-core` profiling code @@ -55,7 +55,7 @@ The project uses **Gradle** with Kotlin DSL. Key build files: ``` To run tests, use the `test` skill rather than composing the Gradle invocation by hand — it -resolves the per-module test task and the unit-test vs system-test split for you. +resolves the per-module test task, including integration test suites, for you. ### Android-Specific Commands ```bash @@ -109,8 +109,7 @@ The repository is organized into multiple modules: - **Other**: `sentry-spotlight`, `sentry-kotlin-extensions`, `sentry-android-distribution` ### Utility Modules -- **`sentry-test-support`** - Shared test utilities -- **`sentry-system-test-support`** - System testing infrastructure +- **`sentry-test-support`** - Shared test utilities, including `MockSentryServer` and envelope matchers - **`sentry-samples`** - Example applications - **`sentry-bom`** - Bill of Materials for dependency management @@ -158,7 +157,7 @@ PR description why the broad catch is necessary. ### Testing Requirements - Write comprehensive unit tests for new features - Android modules require both unit tests and instrumented tests where applicable -- System tests validate end-to-end functionality with sample applications +- End-to-end tests run in the module they test, against `MockSentryServer`. Samples have no tests - **Assertions**: For new unit tests, prefer [Google Truth](https://truth.dev/) (`com.google.common.truth.Truth.assertThat`) over `kotlin.test`/JUnit assertions for its readable, fluent API. Keep using `kotlin.test` for test structure (`@Test`, `assertFailsWith`). See `sentry/src/test/java/io/sentry/DsnTest.kt` for the style. Don't rewrite existing `kotlin.test` assertions solely to switch libraries. - Truth is wired into the `sentry` module. When adding Truth-based tests to another module, add `testImplementation(libs.google.truth)` to that module's `build.gradle.kts`.