[Feature]: Add virtualParentId to plugin metadata and preserve virtual-context span hierarchy #855

What would you like?

Add optional virtualParentId to plugin operation metadata and update both Execution and Invocation OTel plugins to parent operations under their direct virtual context.

Virtual contexts already produce OTel spans, but have no durable CONTEXT checkpoint. Their operations use the nearest checkpointed ancestor as parent, so the virtual span and its operations appear as siblings. This operation parentage matches durable execution history. The proposed change deliberately makes tracing include the logical virtual-context hierarchy while leaving the history hierarchy unchanged.

Metadata contract

• virtualParentId is the stable operation ID of the direct owning parent context, only when that parent is virtual. Otherwise leave it null. Use the same ID representation as existing plugin metadata.
• It is not the nearest virtual ancestor through an intervening regular context. For virtual A -> regular B -> step C, B identifies A as its virtual parent; C leaves the new field absent and keeps its existing parent B. For virtual A -> virtual B -> step C, B identifies A and C identifies B.
• Preserve parentId. OTel selects virtualParentId when present, otherwise parentId. Step attempt spans remain children of their step operation.
• This is instrumentation metadata only: no virtual START/SUCCEED/FAIL checkpoints, additional durable operations or checkpoint requests, or serialized payload wrappers. Operation IDs, wire ParentId values, checkpoint payloads, validation, results, and replay semantics must remain unchanged.

Possible implementation

• Register operation-to-virtual-parent ownership when the core allocates/traverses the stable operation ID, before replay inspection or a cached-result early return. Do not rerun completed user code or fabricate lifecycle events to learn ancestry.
• Keep the registry invocation-local and safe for concurrent child contexts. Supply known ownership to applicable lifecycle hooks and checkpoint-derived replay/update/snapshot metadata, including waits without a user-function body.
• Completion snapshots can arrive before traversal establishes the virtual parent. Defer parent resolution where possible and prefer subsequently learned ownership over a raw backend parent. Retain completed parent SpanContexts until invocation cleanup for late children.
• If a virtual scope is never traversed and no source supplies its ancestry, leave the field absent and retain the documented existing-parent fallback. Flattened history and opaque IDs cannot reconstruct an unrecorded relationship; do not add persistence or guess.

Both OTel views must consume the field:

• Execution: children emitted before suspension and after resume must reference the same logical virtual parent identity as the eventual exported virtual span. Avoid dangling references to discarded placeholders.

• Invocation: resolve the actual virtual segment from the current invocation, including completion-only spans. Different invocations need distinct segment identities; never substitute an earlier invocation's segment.

• Add nullable String virtualParentId to applicable lifecycle and corresponding user-function records. Retain existing public record constructor signatures and PluginInfoConverter overloads by delegating with a null default; verify compatibility with existing compiled plugin consumers.

• Include replay completion paths that emit End without Start.

• Resolve current virtual Invocation span-ID reuse here or as an explicit prerequisite. Each invocation needs its own virtual segment identity. This field describes an operation's parent, not whether the operation itself is virtual; it cannot by itself fix re-entry classification.

Reproduction and acceptance criteria

Use the public API to run otel-virtual-context as a virtual child containing step otel-virtual-before, a five-second durable wait otel-virtual-wait, and step otel-virtual-after. Force suspension/resume and inspect raw exporter records in both views.

• Direct virtual-parent metadata and raw exported parent IDs are correct on initial execution and resume; attempts remain under their operations.
• Cover nested virtual contexts, a regular context inside a virtual context, a step inside that regular context, synchronous failure, and concurrent sibling contexts/FLAT branches.
• Cover completion without local START, known-parent updates arriving before traversal or after the parent body returns, and unknown-parent fallback separately.
• Verify zero added virtual checkpoints, unchanged operation IDs/wire parents/payloads, stable results and replayed failures, and no repeated completed-step side effects.
• Preserve existing metadata construction and plugins without the field. Document the new field, view-specific rules, fallback limits, and the deliberate difference from execution-history nesting.

Coordinate any shared conformance requirement separately; implement focused core/OTel tests and public-handler regressions in this SDK.

Scope and compatibility

Original virtual start-time recovery and general exactly-once export across later replays remain separate lifecycle work. Consistent parent identities are required here; adding this field must not be described as solving timestamp recovery or export deduplication.

Is this a breaking change? Make the metadata extension additive, preserving existing construction paths and fallback. Virtual-child OTel parentage intentionally changes for trace consumers; document it in release notes and decide the appropriate compatibility/versioning treatment.

Does this require an RFC? Yes: align this tracing hierarchy contract across JavaScript, Python, and Java before implementing the behavior change.

Evidence and related work

Source inspection and a local public-API probe used commit 4b1aa82b0d898fea495cda45760ce3737fbb341e. DurableContextImpl.getParentId returns the nonvirtual ancestor; PluginInfoConverter receives that same parent. Invocation span-ID selection uses a deterministic ID when isReplay is false, including current virtual starts on every invocation.

The LocalDurableTestRunner/InMemorySpanExporter probe confirmed flattened parents and repeated virtual trace/span IDs with different invocation parents. Both step bodies executed exactly once. This was local validation, not a cloud reproduction or a wall-clock five-second measurement. Implement against current main.

• #507 / PR #508 addressed deterministic span-ID reuse for step retries; virtual-context re-entry needs equivalent protection.
• JS #997 tracks the separate completed-virtual replay export problem.

Companion SDK issues

• JavaScript #1136
• Python #890