[Feature]: Decouple the OTel plugin layer from application SDK dependencies

[zhongkechen]

What would you like?

Make the OpenTelemetry plugin support two explicit runtime contracts without requiring the ADOT/global path to package an application-side OpenTelemetry SDK:

1. ADOT/global mode: the plugin layer and Java agent own provider integration. Function code must not need opentelemetry-sdk or initialize another provider.
2. Standalone/explicit mode: the application deliberately supplies the OpenTelemetry SDK, processors, and exporter.

This should align with the Python OTel package model introduced by aws/aws-durable-execution-sdk-python#646 while using a Java/Maven-appropriate artifact design.

Current state

The proposed plugin layer in #621 contains only the thin OTel plugin JAR. However, application-loaded plugin classes still reference SDK types:

• InvocationOtelPlugin and ExecutionOtelPlugin expose SdkTracerProviderBuilder, store SdkTracerProvider, and call forceFlush().
• Their global constructors create DeterministicIdGenerator, which implements the SDK IdGenerator interface.
• OtelPluginSupport performs SDK-specific provider checks.
• The current Java conformance example explicitly packages opentelemetry-sdk, so it does not validate the thin plugin layer without an application SDK.

Consequently, packaging the layer without SDK dependencies relies on SDK classes being visible through agent/runtime classloader behavior that is not covered by an end-to-end test. Packaging another SDK in function code creates separate application-side and agent-side OTel classloader worlds and introduces version-skew and provider-ownership risks.

Option 1: Add a standalone SDK module (recommended)

Keep aws-durable-execution-sdk-java-plugin-otel as the layer/global artifact and add a second module, for example aws-durable-execution-sdk-java-plugin-otel-sdk.

The core/layer module would:

• keep application-loaded plugin classes free of io.opentelemetry.sdk.* references;
• use an SDK-free scoped-ID bridge around plugin-owned SpanBuilder.startSpan() calls;
• retain the Java-agent AutoConfigurationCustomizerProvider and deterministic SDK IdGenerator as agent-loaded classes with SDK/SPI dependencies in provided scope;
• use only the global API Tracer in ADOT mode and leave flushing/lifecycle ownership to the agent.

The standalone module would:

• depend on the core plugin and opentelemetry-sdk;
• expose typed factories or builders for explicit providers;
• install the deterministic ID generator on SdkTracerProviderBuilder;
• own explicit-provider flushing and shutdown behavior.

Example standalone API:

var plugin = OtelSdkPlugins.invocation(
        SdkTracerProvider.builder().addSpanProcessor(processor),
        config);

Advantages

• Maven dependency graphs accurately describe runtime ownership.
• Layer users cannot accidentally inherit an application SDK.
• SDK-linked classes cannot be loaded by application classpath scanners in global mode.
• Standalone usage remains type-safe and discoverable.

Disadvantages

• Adds a published artifact and module.
• Requires a small standalone factory API instead of the current builder constructors.

Option 2: Keep one artifact with strict class isolation

Keep a single aws-durable-execution-sdk-java-plugin-otel artifact, but isolate its classes by responsibility:

• core plugin packages use only OTel API types;
• an agent package contains the SDK IdGenerator and auto-configuration SPI implementation;
• an sdk package contains standalone factories that reference SdkTracerProviderBuilder;
• opentelemetry-sdk and auto-configuration SPI dependencies are non-transitive (provided or optional as appropriate);
• standalone users explicitly add opentelemetry-sdk; global users never load the SDK-linked packages.

Advantages

• Retains one artifact and one Lambda plugin layer JAR.
• Avoids another published module.

Disadvantages

• Maven cannot express package-level optional dependencies.
• Reflection or broad classpath scanning may load SDK-linked classes unexpectedly.
• Accidental SDK imports into core classes can silently regress the layer contract.
• Requires architectural or jdeps enforcement to preserve the boundary.

Shared implementation requirements

Regardless of the selected option:

• Extract scoped deterministic-ID state into an application-side class with no SDK dependency.
• Keep the agent-side IdGenerator bridge compatible across the application and agent classloaders.
• Remove SDK-specific provider checks and forceFlush() from global mode.
• Decide and document how opentelemetry-api becomes application-visible for layer-only discovery; do not shade duplicate OTel API packages into the plugin JAR.
• Update ci: publish Java OTel Lambda layer #621 documentation and runtime assumptions to match the selected contract.

Acceptance criteria

• A deployed ADOT/global conformance test passes when the function artifact contains no opentelemetry-sdk.
• The test inspects the function artifact to prevent SDK classes or JARs from being included accidentally.
• Java-agent extension discovery installs deterministic ID generation successfully from the plugin layer.
• Standalone/explicit mode passes when the application explicitly supplies the SDK and exporter.
• Global mode neither creates nor registers an application-owned SDK provider.
• An architecture or jdeps test prevents SDK references from application-loaded core plugin classes.
• Documentation clearly describes the supported ADOT/global and standalone dependency sets.

Is this a breaking change?

No. The OTel plugin has not been officially released.

Does this require an RFC?

Yes. This changes artifact boundaries, provider ownership, and the plugin-layer runtime contract.

Additional context

Related work:

• ci: publish Java OTel Lambda layer #621 publishes a thin OTel plugin Lambda layer.
• fix(otel): scope deterministic ID generation #627 scopes deterministic IDs, removes AUTO_OTLP, and makes provider ownership explicit.
• fix(otel): scope deterministic IDs to plugin tracers aws-durable-execution-sdk-python#646 removes base-package OTel dependencies and uses a standalone extra.