Skip to content

[Bug]: Factory wrappers bypass exclusive-group validation and allow both OTel views #891

Open

Description

@zhongkechen

Expected Behavior

Reject a configuration that enables both Execution and Invocation OTel views, including when a user-written factory delegates to one of the built-in factories. The error should name both views before any plugin lifecycle hooks or the user handler run. Registering the two built-in factories directly already produces this error at handler initialization.

Actual Behavior

A wrapper with only create_plugin(info) loses the factory class's registration metadata. The SDK accepts it alongside the other view and runs both plugins.

A local single-step execution completed successfully, with the step body called once, but exported 2 Workflow spans and 10 spans in total. There were 3 duplicate (trace_id, span_id) pairs: DurableExecutionRoot, step, and Workflow. The two attempt spans had different IDs in this reproduction. This is duplicate instrumentation, not duplicate execution of the step.

Steps to Reproduce

From this repository at the commit below, save the following as repro.py and run hatch run dev-otel:python repro.py. It uses the local test runner and an in-memory exporter; no AWS deployment is required.

import os
from collections import defaultdict

from aws_durable_execution_sdk_python.execution import durable_execution
from aws_durable_execution_sdk_python.exceptions import PluginLoadError
from aws_durable_execution_sdk_python_testing.runner import DurableFunctionTestRunner
from aws_durable_execution_sdk_python_otel import (
    ExecutionOtelPluginFactory,
    InvocationOtelPluginFactory,
    OtelPluginConfig,
)
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter

os.environ.pop("DURABLE_EXECUTION_PLUGINS", None)
os.environ.pop("_X_AMZN_TRACE_ID", None)  # Exercise the local fallback trace.
provider = TracerProvider()
exporter = InMemorySpanExporter()
provider.add_span_processor(SimpleSpanProcessor(exporter))
config = OtelPluginConfig(tracer_provider=provider, enrich_logger=False)
execution = ExecutionOtelPluginFactory(config)
invocation = InvocationOtelPluginFactory(config)

class WrappedFactory:
    def create_plugin(self, info):
        return execution.create_plugin(info)

calls = []
def work(_context):
    calls.append("step")
    return "saved"

def handler(_event, context):
    return context.step(work, name="step")

# Control: the same two factories are rejected when passed directly.
try:
    durable_execution(handler, plugins=[execution, invocation])
except PluginLoadError as error:
    print("direct factories rejected:", error)
else:
    raise AssertionError("Expected direct factories to be rejected")
assert not exporter.get_finished_spans()

# Reproduction: wrapping only the execution factory bypasses the check.
configured = durable_execution(handler, plugins=[WrappedFactory(), invocation])
with DurableFunctionTestRunner(handler=configured) as runner:
    result = runner.run(input="{}", execution_timeout=15)

spans = exporter.get_finished_spans()
by_id = defaultdict(list)
for span in spans:
    by_id[(span.context.trace_id, span.context.span_id)].append(span.name)
print("status:", result.status.value)
print("step calls:", len(calls))
print("Workflow spans:", sum(span.name == "Workflow" for span in spans))
print("total spans:", len(spans))
print("duplicate IDs:", sorted(names for names in by_id.values() if len(names) > 1))
provider.shutdown()

Observed output, omitting the full direct-registration error:

status: SUCCEEDED
step calls: 1
Workflow spans: 2
total spans: 10
duplicate IDs: [['DurableExecutionRoot', 'DurableExecutionRoot'], ['Workflow', 'Workflow'], ['step', 'step']]

The direct-registration error is a PluginLoadError naming ExecutionOtelPluginFactory and InvocationOtelPluginFactory and telling the caller to keep only one plugin from aws-durable-execution-otel-view.

SDK Version

Verified from source at commit 44b9f4d1c862b50041423e19fe6a250bbf7bd72f: core 3.0.0, OTel 2.0.0, testing 1.3.0 (source package version strings).

Python Version

Python 3.15.2 on macOS, using the repository's dev-otel Hatch environment.

Is this a regression?

Yes: the previously enforced one-view invariant regressed during the per-invocation factory migration in PR #737, merged 2026-10-22 at 20:41:47 UTC, commit 894c2b1e174bb324a3b5f12dccad1eaf472d4b03. The literal create_plugin(info) forwarding-factory API was introduced by that migration; the same snippet was not supported by the prior provider API.

Additional Context

This is a follow-up to the closed exclusivity issue #652: direct factory registration is protected, but composition through a user-written factory bypasses the protection.

The existing test_competing_views_rejected_before_hooks cases pass for explicit, environment, and mixed registration in both orders (6 passed); they do not cover a composition wrapper.

The missing protection belongs to generic exclusive-group handling, rather than a hardcoded check for OTel class names. A wrapper should not need to copy factory registration metadata for the returned plugins' mutually exclusive capabilities to remain enforceable. Any runtime conflict check should run before hooks and remain distinct from the ordinary plugin/factory error containment path. Regression coverage should include wrapping either or both factories, both registration orders, and one OTel view alongside an unrelated plugin.

Related SDK issues

Regression history

  • Original protection: PR #753, merged 2026-10-07 at 18:53:55 UTC, commit ef683592fbd3edb48c20b880d859c294bc0ae7a0, fixed [otel] Allow only one OTel plugin at a time: document and validate #652. The protection belonged to actual plugin classes/provider-declared plugin types, with enforcement of the returned type.
  • Reintroduction: PR #737 / 894c2b1e moved the group declaration from the OTel plugin classes to the factories and changed discovery to validate _factory_type(factory). Per-invocation construction no longer enforces exclusivity of the actual returned views.
  • Historical runtime verification at the immediate parent 531969b5c14f6abd35c9ef59b2c3b3355832ee84: the old provider API with a forwarding factory and the other OTel view rejects the conflict before the delegate is called. Declaring an unrelated unmarked plugin_type does not bypass the guard because the returned type is checked. At 894c2b1e, the new forwarding-factory configuration succeeds, exports two Workflow spans and ten spans total, and produces duplicate root/step/Workflow identities while calling the step once. Direct factories still reject.
  • The prior safeguard shipped in core 2.1.0 / OTel 1.1.0. The first GitHub release affected by the new factory-wrapper path is core 3.0.0 / OTel 2.0.0, published 2026-10-26 at 02:24:14 UTC. The release contains the migration commit; this is not inferred only from source version strings.

All dates in this history are UTC. This is a regression of the actual-view exclusivity guarantee during the API migration. It does not imply that the identical 3.x wrapper syntax was previously valid, or that every possible wrapper around a plugin instance was protected by the old API.

V2 verification: wrapper scope clarification

Matching historical core/OTel/testing sources from published release tags were executed with the local test runner, verifying the import origins and version strings:

Released versions Direct views Old provider factory forwarding creation, with real plugin_type Plugin instance forwarding all public hooks
Core 2.1.0 / OTel 1.1.0 / Testing 1.3.0 (1e1f0245) Rejected Rejected before factory call Accepted, duplicate spans
Core 2.0.1 / OTel 1.0.0 / Testing 1.2.1 (2313fcbf) Accepted, duplicate spans Accepted, duplicate spans Accepted, duplicate spans

The v2 wrapper is a custom ForwardingPlugin(DurableInstrumentationPlugin) whose concrete class has no registration marker and which explicitly forwards invocation, operation, user-function, and propagation hooks to its delegate. It is registered as plugins=[ForwardingPlugin(execution), invocation].

  • At 2.1.0, wrapping either view gives one business-step call, two Workflow records, ten total spans, and duplicate root/Workflow/step identities.
  • At 2.0.1, direct and wrapped cases give one business-step call, two Workflow records, eight total spans, and duplicate Workflow/step identities.
  • Declaring a broad plugin_type=DurableInstrumentationPlugin while returning an ExecutionOtelPlugin still fails the exact returned-type check in both versions; it does not bypass validation.
  • The v2.1 discovery path inspects the registered plugin's type and skips unmarked types, so it cannot see the OTel delegate behind the custom instance.

Therefore, 2.1.0 is not a universally wrapper-safe baseline. #753 fixed direct/old-provider registration but left the plugin-instance/hook-wrapper path open. The #737 regression history above applies specifically to the new 3.x factory-wrapper path; wrapper-related metadata loss did not first appear in 3.x.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugneeds-triageIssue needs triagepkg:pluginsPlugin interface, possibly impacting all plugins

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions