fix(otel): scope deterministic IDs to plugin tracers

[zhongkechen]

Summary

This PR fixes two independent defects that could place multiple root spans in
the same OpenTelemetry trace.

Issue 1: deterministic IDs leaked through the shared provider

DeterministicIdGenerator was installed on the resolved TracerProvider.
A provider is shared by every instrumentation scope that uses it, so unrelated
instrumentation could receive the active durable execution trace ID when it
created a root span. The unrelated span had no durable parent, but appeared as
another root of the durable workflow trace.

The generator also held the current execution trace ID as shared mutable state.
Concurrent plugin instances or executions could therefore replace or consume
one another's deterministic IDs.

Issue 2: the Workflow root reused the ambient Lambda/X-Ray trace ID

The parentless Workflow span derived its trace ID from _X_AMZN_TRACE_ID.
The ambient Lambda span and the plugin's Invocation span used that same trace,
but the Workflow span was deliberately created without a parent. The result
was two disconnected root trees sharing one trace ID:

Lambda/ambient root -> Invocation -> ...

Workflow root -> ...

The same collision occurred without ambient instrumentation when Workflow
and Invocation were independently created as roots with the deterministic
execution trace ID.

Why multiple roots are harmful

OpenTelemetry backends expect a trace to describe one causal tree. Multiple
roots turn it into a disconnected forest. Backend behavior varies, but the
potential consequences include:

• ambiguous trace names, entry points, status, and root-cause attribution;
• incorrect duration and latency summaries when the backend chooses one root
  or measures across unrelated root branches;
• broken critical-path analysis and misleading service or dependency maps;
• durable and unrelated library spans appearing to be causally related when
  they only share a leaked trace ID;
• independent root-based sampling decisions, which can retain one branch and
  discard another, leaving a partial trace.

Changes

Scope deterministic generation to plugin spans

• Install DeterministicIdGenerator on the SDK Tracer used by each durable
  plugin instead of mutating the shared TracerProvider.
• Use ContextVar-scoped overrides only around the plugin-owned
  start_span() call that needs a deterministic ID.
• Consume a deterministic span ID before returning it, preserving the previous
  one-shot safeguard for re-entrant generation.
• Keep execution state on each plugin instance and delegate all non-overridden
  generation to the provider's original generator.
• Implement is_trace_id_random() so newer OpenTelemetry SDKs set trace flags
  correctly for deterministic and fallback IDs.

Separate workflow and invocation trace identities

• Derive the durable Workflow trace ID only from the execution ARN and start
  time. It no longer reuses the ambient Lambda/X-Ray trace ID.
• Parent Invocation spans to the active ambient context in both GLOBAL and
  EXPLICIT modes, using extracted upstream context as a fallback.
• When no ambient or extracted parent exists, let the provider generate a fresh
  trace ID for the root Invocation span.

The resulting trace models are:

ExecutionOtelPlugin

Workflow trace:  Workflow -> Operation -> Attempt -> instrumented child spans
Ambient trace:   Lambda/ambient -> Invocation
                                      ^
                                      linked from durable operation spans

InvocationOtelPlugin

Workflow trace:  Workflow
                    ^
                    linked from durable operation spans

Ambient trace:   Lambda/ambient -> Invocation -> Operation -> Attempt

Each durable Workflow trace has one root. Each invocation joins its ambient
trace, or creates a provider-generated trace in which Invocation is the sole
root.

Do not fabricate continuation links

InvocationOtelPlugin previously reconstructed a deterministic trace ID and
span ID to link a continuation segment to a prior span for the same logical
operation. The prior SpanContext is not checkpointed, so the plugin cannot
know that the target span was actually created or exported, nor preserve its
real trace flags or trace state.

Continuation and retry segments now receive fresh span IDs and retain the real
link to the durable Workflow span, but do not emit a synthetic link to an
unobserved prior operation span.

Keep provider ownership explicit

• Retry global provider resolution at invocation start so a plugin constructed
  before ADOT setup can bind after the ProxyTracer resolves.
• Disable telemetry for the entire invocation when an SDK tracer is still
  unavailable, preventing partial traces, and retry on the next invocation.
• Remove AUTO_OTLP. Provider/exporter construction, sampling, propagation,
  and HTTP instrumentation remain application or ADOT responsibilities.
• Keep only GLOBAL and EXPLICIT provider modes.

Result

• Unrelated root spans keep the provider's normal ID generation and cannot
  enter the durable workflow trace accidentally.
• Workflow and ambient invocation trees use distinct trace IDs and no longer
  form disconnected roots in one trace.
• Durable workflow and initial operation IDs remain deterministic across
  replay, while continuation segments use fresh span IDs.
• Concurrent executions cannot overwrite or consume each other's scoped IDs.
• Proxy-provider initialization cannot produce partial durable traces.

Testing

• hatch run dev-otel:test (115 passed)
• OTel package type check (23 source files)
• Ruff lint and format checks
• focused generator, provider, and plugin integration tests against
  OpenTelemetry SDK 1.20.0 (39 passed)

Closes #644

[zhongkechen]

It's expected to be backward incompatible and the docs will be updated later.

[zhongkechen]

@wangyb-A is this included in your fix of attach/detach issue?

[github-actions]

Codex AI review

[P1] Do not reuse the plugin's previous Workflow context as ambient context. ExecutionOtelPlugin attaches its Workflow span but never detaches it; ending and resetting the span leaves its SpanContext valid. On warm reuse without an outer ADOT context, the next invocation is therefore parented to the previous execution's Workflow trace. Track and detach the attach token during reset, and add a two-invocation same-instance test.

[github-actions]

Codex AI review

[P1] Preserve the existing public provider configuration API. This change removes exported ProviderSource, ExporterConfig, and instrumentation helpers, as well as supported configuration fields such as provider_source and exporter_config. Existing applications will fail during import or plugin construction after upgrading. Retain deprecated compatibility symbols and map legacy settings to the new behavior, or defer removal to an explicitly breaking release with migration coverage.

[github-actions]

Codex AI review

Found two blocking regressions affecting trace isolation and public API compatibility. Tests do not cover warm reuse or legacy configuration imports.

Reviewed commit 05b0d0f11f18b9397b810c5900e45c7a6c302281. Workflow run

[github-actions]

Claude AI review

Review: fix(otel): scope deterministic IDs to plugin tracers (#646)

No blocking findings. The change is well-scoped and the test suite is updated comprehensively to match the new behavior. Reviewed against the base checkout plus the SHA-anchored diff; verification notes below.

What was verified

• Scoping correctness. Deterministic generation now installs on the plugin's Tracer (install_on_tracer) instead of the shared TracerProvider, and overrides are applied only inside use_ids(...) around plugin-owned start_span() calls. install_on_tracer is idempotent (isinstance(current_generator, cls) short-circuit) and _bind_sdk_tracer re-resolves a deferred global/proxy provider on each on_invocation_start. Covered by test_install_on_tracer_does_not_replace_provider_generator, test_unrelated_root_spans_keep_provider_id_generation, and the proxy-bind tests.
• Concurrency/determinism. The override moved from a module-level ContextVar + shared _execution_trace_id to a per-instance ContextVar[_IdOverride]. One-shot span-ID consumption, exception restore, nested-scope semantics, and thread/async isolation are all exercised. generate_trace_id/is_trace_id_random do not consume state, so their result is call-order independent within a scope.
• Trace-identity separation. _to_otel_trace_id is now purely execution_arn + execution_start_time derived (X-Ray env read removed). Workflow spans root the deterministic trace; Invocation joins the ambient/extracted trace via _invocation_parent_context(). Operation spans always have a valid parent, so their trace_id override is inert (parent trace wins) — consistent with the documented models.
• Continuation links. Fabricated SpanContext reconstruction is removed; continuation/retry segments get fresh span IDs and link only to the real Workflow span. Tests updated accordingly.
• API compatibility. All references to removed symbols (ProviderSource, ExporterConfig, AUTO_OTLP, enable_http_instrumentation, exporter_config, propagators, register_standalone_instrumentations, sampler/endpoint helpers) exist only in base-revision files the diff deletes/rewrites; __init__.py exports and both example handlers are updated to match. No external consumer left dangling.

Residual test risk (not defects)

• SDK internal reliance. The mechanism depends on opentelemetry.sdk.trace.Tracer.id_generator being a live, mutable per-tracer attribute read at start_span() time. This is an SDK implementation detail, and the dependency pin is opentelemetry-sdk>=1.20.0 with no upper bound; a future SDK that renames/removes this attribute would make install_on_tracer raise AttributeError inside on_invocation_start (no guard around tracer.id_generator). Behavior is only asserted against SDK 1.20.0. Consider a lightweight version guard or an upper bound if the team wants forward-compat safety.
• Start-time stability. Cross-invocation trace stitching now hinges on InvocationStartInfo.execution_start_time being identical across every invocation of a durable execution (it feeds the trace-ID timestamp bytes). This is a framework contract; if a resumed invocation ever reports a different start time, the Workflow trace would fragment. Worth an explicit test if that guarantee is not already enforced elsewhere.
• is_trace_id_random on older SDKs. The getattr-based fallback returns False when the fallback generator lacks the method — harmless on SDKs without the random-trace-flag feature, and correct (True) on newer SDKs where RandomIdGenerator provides it. No span-level assertion exercises the fallback-root-span random-flag path, but the generator method itself is tested.

Reviewed commit 05b0d0f11f18b9397b810c5900e45c7a6c302281. Workflow run