Skip to content

Add voice tracing - #48633

Open
knit (knitvoger) wants to merge 6 commits into
Azure:mainfrom
knitvoger:wujin/voice-trace-v2-20260817
Open

Add voice tracing#48633
knit (knitvoger) wants to merge 6 commits into
Azure:mainfrom
knitvoger:wujin/voice-trace-v2-20260817

Conversation

@knitvoger

@knitvoger knit (knitvoger) commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Description

Adds content-free OpenTelemetry observability to the typed Voice WebSocket relay, including connection and callback spans, duration and propagation-failure metrics, and application-owned TargetTurn spans for model and tool work. It also adds allowlisted W3C context propagation and explicit session termination outcomes.

Updates Voice close-code classification, lifecycle handling, samples, documentation, and API metadata, with expanded tests covering tracing, terminal behavior, cleanup, and transport failures.

Spec: voice_live_bridge/spec.md · Design: hosted_text_agent_and_voice_live_bridge.md

All SDK Contribution checklist:

  • The pull request does not introduce [breaking changes]
  • CHANGELOG is updated for new features, bug fixes or other significant changes.
  • I have read the contribution guidelines.

General Guidelines and Best Practices

  • Title of the pull request is clear and informative.
  • There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

  • Pull request includes test coverage for the included changes.

knitvo added 3 commits August 18, 2026 12:35
Add fail-open connection and callback telemetry plus application-declared target turns. Harden propagation, transport outcome classification, and the application-owned Voice sample lifecycle.
Classify WebSocket Going Away consistently as a clean close and preserve source-aware outcomes for unfinished turns in the basic Voice sample.
Copilot AI balanced review requested due to automatic review settings August 19, 2026 06:24
@github-actions github-actions Bot added Community Contribution Community members are working on the issue customer-reported Issues that are reported by GitHub users external to the Azure organization. Hosted Agents sdk/agentserver/* labels Aug 19, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution knit (@knitvoger)! We will review the pull request and get back to you soon.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
9 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR introduces content-free OpenTelemetry tracing/metrics for the typed Voice WebSocket relay, adds application-declared “target turn” spans for model/tool work, and tightens W3C propagation handling (allowlisted baggage, tracestate validation, and close-code semantics).

Changes:

  • Added SessionTermination and Session.start_target_turn(...) with TargetTurn.activate() / TargetTurn.complete(...) to let apps explicitly model target-decision traces.
  • Implemented Voice connection/callback OpenTelemetry spans + aggregate duration/propagation-failure metrics while filtering/sanitizing propagated context.
  • Updated samples and expanded test coverage around telemetry behavior, termination classification, and transport failure ordering.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
sdk/agentserver/azure-ai-agentserver-invocations/azure/ai/agentserver/invocations/voice/_voice_host.py Adds W3C extraction filtering/validation, connection+callback spans, termination committing, and default observability configuration behavior.
sdk/agentserver/azure-ai-agentserver-invocations/azure/ai/agentserver/invocations/voice/_session.py Adds SessionTermination, exposes Session.termination, and wires connection context + target-turn creation.
sdk/agentserver/azure-ai-agentserver-invocations/azure/ai/agentserver/invocations/voice/_turn.py New TargetTurn API for application-owned target-decision tracing and duration metrics.
sdk/agentserver/azure-ai-agentserver-invocations/azure/ai/agentserver/invocations/voice/_tracing.py New package-local tracing + metrics helpers (_SpanScope, duration histograms, propagation failure counter).
sdk/agentserver/azure-ai-agentserver-invocations/azure/ai/agentserver/invocations/voice/init.py Exposes new public surface: SessionTermination, TargetTurn* symbols.
sdk/agentserver/azure-ai-agentserver-invocations/azure/ai/agentserver/invocations/_constants.py Adds “going away” close code constant and normal-close-code set.
sdk/agentserver/azure-ai-agentserver-invocations/samples/basic_voice_agent/basic_voice_agent.py Updates sample to declare/complete target turns, cap concurrency, bound output retention, and classify outcomes.
sdk/agentserver/azure-ai-agentserver-invocations/samples/basic_voice_agent/README.md Documents target-turn tracing completion and sample outcome mapping.
sdk/agentserver/azure-ai-agentserver-invocations/README.md Updates Voice docs with tracing model, propagation constraints, and target-turn usage.
sdk/agentserver/azure-ai-agentserver-invocations/tests/voice/test_voice_transport_findings.py Updates propagation expectations and adds tracestate size-limit validation test.
sdk/agentserver/azure-ai-agentserver-invocations/tests/voice/test_voice_tracing.py New contract tests for spans/metrics, sanitization, termination semantics, and failure modes.
sdk/agentserver/azure-ai-agentserver-invocations/tests/voice/test_voice_session.py Updates session slot expectations for new fields.
sdk/agentserver/azure-ai-agentserver-invocations/tests/voice/test_basic_voice_agent.py Extends sample tests for outcome mapping, cancellation, and completion semantics.
sdk/agentserver/azure-ai-agentserver-invocations/pyproject.toml Promotes opentelemetry-api to runtime dependency and bumps dev OTel versions.
sdk/agentserver/azure-ai-agentserver-invocations/dev_requirements.txt Updates OTel API/SDK version constraints.
sdk/agentserver/azure-ai-agentserver-invocations/_version.py Bumps package version to 1.1.0b2.
sdk/agentserver/azure-ai-agentserver-invocations/api.md Updates API surface documentation for new tracing/termination APIs.
sdk/agentserver/azure-ai-agentserver-invocations/api.metadata.yml Updates API metadata hash and recorded python version.
sdk/agentserver/azure-ai-agentserver-invocations/CHANGELOG.md Adds 1.1.0b2 release notes describing new tracing/termination behavior.
.chronus/changes/wujin-voice-tracing-v2-2026-8-17.md Adds chronus change entry for the feature.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Copilot AI review requested due to automatic review settings August 19, 2026 07:00

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.

Suppressed comments (4)

sdk/agentserver/azure-ai-agentserver-invocations/azure/ai/agentserver/invocations/voice/_turn.py:97

  • TargetTurn.complete(..., output_item_count: int | None = None) is typed as optional, but _validate_completion effectively makes output_item_count mandatory for at least RESPONSE (must be provided and >= 1) and NONE (must be explicitly 0; None currently fails). Either make output_item_count required in the public signature (and docs/api.md) or adjust validation to treat output_item_count=None as 0 for NONE and keep the clearer error for RESPONSE.
    if normalized_outcome is TargetTurnOutcome.RESPONSE:
        if validated_response_id is None or validated_output_count is None or validated_output_count < 1:
            raise ValueError("response outcome requires a response_id and at least one output item")
    elif normalized_outcome is TargetTurnOutcome.NONE:
        if validated_response_id is not None or validated_output_count != 0:
            raise ValueError("none outcome requires no response_id and zero output items")
    elif validated_output_count is not None and validated_output_count > 0 and validated_response_id is None:
        raise ValueError("positive output_item_count requires a response_id")

sdk/agentserver/azure-ai-agentserver-invocations/azure/ai/agentserver/invocations/voice/_tracing.py:136

  • This relies on token.var being present and stable, which is an internal detail of how the OTel context implementation currently stores its token. Prefer using the public API (otel_context.detach(token)) and store only what detach needs (the token), to reduce the chance of breakage across OpenTelemetry versions.
def _attach_context(context: Any) -> tuple[Any, Any] | None:
    if context is None:
        return None
    previous = _current_context()
    if previous is None:
        return None
    try:
        return otel_context.attach(context), previous
    except BaseException:  # pylint: disable=broad-exception-caught
        return None


def _reset_context(attachment: tuple[Any, Any] | None) -> None:
    if attachment is None:
        return
    token, previous = attachment
    try:
        token.var.reset(token)
        return
    except BaseException:  # pylint: disable=broad-exception-caught
        pass
    try:
        token.var.set(previous)
    except BaseException:  # pylint: disable=broad-exception-caught
        pass

sdk/agentserver/azure-ai-agentserver-invocations/azure/ai/agentserver/invocations/voice/_voice_host.py:316

  • Returning None here disables all connection/callback tracing even when only the propagator failed to construct. Consider failing open by using base_context as the extracted context when _VOICE_TRACE_PROPAGATOR is None (still record the propagation failure), so local spans/metrics can remain available even if remote extraction is unavailable.
        base_context = _new_voice_context()
        if base_context is None or _VOICE_TRACE_PROPAGATOR is None:
            _record_propagation_failure("extraction_error")
            return None

sdk/agentserver/azure-ai-agentserver-invocations/tests/voice/test_voice_tracing.py:69

  • This fixture mutates global OpenTelemetry state (potentially replacing the tracer provider) and never restores the prior provider, which can cause order-dependent failures when running the full test suite. Prefer saving the previous provider and restoring it in teardown (e.g., via a yield fixture), or isolate these tests by using monkeypatching around trace.set_tracer_provider/metrics.set_meter_provider.
@pytest.fixture
def spans():
    """Capture spans without replacing a provider another test installed."""
    global _PROVIDER, _EXPORTER
    if _PROVIDER is None:
        existing = trace.get_tracer_provider()
        if hasattr(existing, "add_span_processor"):
            _PROVIDER = existing
        else:
            _PROVIDER = TracerProvider()
            trace.set_tracer_provider(_PROVIDER)
        _EXPORTER = InMemorySpanExporter()
        _PROVIDER.add_span_processor(SimpleSpanProcessor(_EXPORTER))
    _EXPORTER.clear()
    return _PROVIDER, _EXPORTER

Copilot AI review requested due to automatic review settings August 19, 2026 07:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.

Suppressed comments (6)

sdk/agentserver/azure-ai-agentserver-invocations/azure/ai/agentserver/invocations/voice/_turn.py:305

  • Span.set_status(...) is being called with StatusCode.ERROR. In the OpenTelemetry Python API, set_status typically expects a Status object (e.g., Status(StatusCode.ERROR)) rather than the enum value. As written, this call can be ignored (due to the broad exception handler) and you’ll silently lose error status on target-turn spans. Consider constructing/passing the proper Status type so error spans are correctly marked.
                        span.set_status(otel_trace.StatusCode.ERROR)

sdk/agentserver/azure-ai-agentserver-invocations/azure/ai/agentserver/invocations/voice/_tracing.py:271

  • Same Span.set_status(...) issue as in _turn.py: passing StatusCode.ERROR may be rejected by the OTel API and (because exceptions are swallowed) will result in spans not being marked as errored. Use the OpenTelemetry Status object (or the API’s supported status type) so callback/connection spans reliably surface error status.
            self._span.set_attribute("error.type", error_type)

sdk/agentserver/azure-ai-agentserver-invocations/azure/ai/agentserver/invocations/voice/_tracing.py:275

  • Same Span.set_status(...) issue as in _turn.py: passing StatusCode.ERROR may be rejected by the OTel API and (because exceptions are swallowed) will result in spans not being marked as errored. Use the OpenTelemetry Status object (or the API’s supported status type) so callback/connection spans reliably surface error status.
            self._span.set_status(otel_trace.StatusCode.ERROR)

sdk/agentserver/azure-ai-agentserver-invocations/azure/ai/agentserver/invocations/voice/_voice_host.py:332

  • When multiple traceparent headers are present, the code flags the request as invalid but still passes the duplicate traceparent values into the propagator. Depending on the propagator’s behavior, one of the duplicates could still be accepted, which undermines the ‘invalid’ classification and can allow header-spoofing/ambiguity. Consider sanitizing trace_headers in the len(traceparents) != 1 case (e.g., drop all traceparent headers before extraction, or only keep a single canonical one while still recording invalid) so extraction behavior matches the security intent.
        traceparents = [value for name, value in raw_headers if name.lower() == b"traceparent"]
        if not traceparents:
            failure = "missing"
        elif len(traceparents) != 1:
            failure = "invalid"
        trace_headers = raw_headers
        valid_tracestate = _has_valid_tracestate(raw_headers)
        if not valid_tracestate:
            failure = "invalid"
            trace_headers = [(name, value) for name, value in raw_headers if name.lower() != b"tracestate"]
        context = _VOICE_TRACE_PROPAGATOR.extract(
            carrier=trace_headers,
            context=base_context,
            getter=_VOICE_HEADER_GETTER,
        )

sdk/agentserver/azure-ai-agentserver-invocations/azure/ai/agentserver/invocations/voice/_voice_host.py:164

  • _configure_voice_observability accepts enable_sensitive_data but explicitly ignores it, always deriving the value from the environment variable. If callers (or InvocationAgentServerHost) pass enable_sensitive_data=True, it will have no effect, which is surprising for an argument in a public configuration path. Consider either honoring the explicit parameter (e.g., treat it as an opt-in that overrides/env-ORs with the env var) or removing the parameter from this adapter if it’s intentionally unsupported for Voice.
def _configure_voice_observability(
    *,
    connection_string: str | None = None,
    log_level: str | None = None,
    enable_sensitive_data: bool = False,
) -> None:
    del enable_sensitive_data
    configured = os.environ.get("OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT", "false")
    try:
        _CORE_CONFIGURE_OBSERVABILITY(
            connection_string=connection_string,
            log_level=log_level,
            enable_sensitive_data=configured.strip().lower() in {"1", "true"},
        )
    except BaseException:  # pylint: disable=broad-exception-caught
        pass

sdk/agentserver/azure-ai-agentserver-invocations/azure/ai/agentserver/invocations/voice/_tracing.py:136

  • Context reset is implemented via token.var.reset(token) (and a set(previous) fallback). OpenTelemetry’s context API provides otel_context.detach(token) for detaching an attachment token, which avoids relying on internal contextvars.Token shape/behavior and is the standard pairing for attach. Consider switching _reset_context to call otel_context.detach(token) (and only falling back to the lower-level reset/set approach if needed) to reduce the risk of context leaks across versions.
def _attach_context(context: Any) -> tuple[Any, Any] | None:
    if context is None:
        return None
    previous = _current_context()
    if previous is None:
        return None
    try:
        return otel_context.attach(context), previous
    except BaseException:  # pylint: disable=broad-exception-caught
        return None


def _reset_context(attachment: tuple[Any, Any] | None) -> None:
    if attachment is None:
        return
    token, previous = attachment
    try:
        token.var.reset(token)
        return
    except BaseException:  # pylint: disable=broad-exception-caught
        pass
    try:
        token.var.set(previous)
    except BaseException:  # pylint: disable=broad-exception-caught
        pass

Copilot AI review requested due to automatic review settings August 19, 2026 07:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 2 comments.

Suppressed comments (2)

sdk/agentserver/azure-ai-agentserver-invocations/pyproject.toml:1

  • The minimum OpenTelemetry versions are pinned to >=1.43.0 for both opentelemetry-api and opentelemetry-sdk. With my dependency knowledge cutoff (Aug 2025), 1.43.0 is not a known released version and could cause installation failures in environments that don’t have that release available. Consider lowering the minimum to the oldest released version that provides the required APIs, or using a bounded range aligned with the repo’s tested OpenTelemetry versions.
[project]

sdk/agentserver/azure-ai-agentserver-invocations/azure/ai/agentserver/invocations/voice/_voice_host.py:171

  • _configure_voice_observability currently swallows BaseException, which includes SystemExit and KeyboardInterrupt. That can interfere with process shutdown behavior and makes failures harder to diagnose. Prefer catching Exception here (and optionally logging a content-free warning) so only expected runtime failures are suppressed while critical process-control exceptions still propagate.
    except BaseException:  # pylint: disable=broad-exception-caught
        pass

Copilot AI review requested due to automatic review settings August 19, 2026 09:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.

Suppressed comments (2)

sdk/agentserver/azure-ai-agentserver-invocations/azure/ai/agentserver/invocations/voice/_tracing.py:136

  • otel_context.attach() returns an opaque token whose internal shape is not part of the public API; accessing token.var relies on OpenTelemetry internals and can break across OTel implementations/versions. Prefer storing the attach token and using the public opentelemetry.context.detach(token) to reset, instead of manipulating ContextVar internals.
def _attach_context(context: Any) -> tuple[Any, Any] | None:
    if context is None:
        return None
    previous = _current_context()
    if previous is None:
        return None
    try:
        return otel_context.attach(context), previous
    except BaseException:  # pylint: disable=broad-exception-caught
        return None


def _reset_context(attachment: tuple[Any, Any] | None) -> None:
    if attachment is None:
        return
    token, previous = attachment
    try:
        token.var.reset(token)
        return
    except BaseException:  # pylint: disable=broad-exception-caught
        pass
    try:
        token.var.set(previous)
    except BaseException:  # pylint: disable=broad-exception-caught
        pass

sdk/agentserver/azure-ai-agentserver-invocations/azure/ai/agentserver/invocations/voice/_tracing.py:233

  • The attribute key "azure.ai.agentserver.invocations_ws.close_code" is duplicated as a raw string here, while the codebase also defines InvocationsWSConstants.ATTR_SPAN_CLOSE_CODE. Using the shared constant (or a single module-level constant for span attribute keys) reduces the risk of drift/typos between places that set/read these attributes.
    def complete_connection(self, outcome: str, close_code: int) -> None:
        if self._completed:
            return
        self._completed = True
        self._set_attributes(
            {
                "azure.ai.agentserver.invocations_ws.close_code": close_code,
                "bridge.outcome": outcome,
            }
        )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Community Contribution Community members are working on the issue customer-reported Issues that are reported by GitHub users external to the Azure organization. Hosted Agents sdk/agentserver/*

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants