Skip to content

Build no per-trace sampling state when extracting a trace - #13043

Draft
dougqh wants to merge 5 commits into
dougqh/ptags-sampling-state-cas-stackedfrom
dougqh/shared-sampling-states
Draft

dougqh wants to merge 5 commits into
dougqh/ptags-sampling-state-cas-stackedfrom
dougqh/shared-sampling-states

Conversation

@dougqh

@dougqh dougqh commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

Extracting a trace no longer builds a per-trace sampling state, whatever the headers. Three steps, one commit each (plus tests):

  1. Shared states for simple decisions.
    • A decision with no ot member and no sampling rate is just a priority and a decision maker. Extracted priorities, manual keeps and forced keeps are almost always one of a few dozen such pairs.
    • States are immutable, so these now use preallocated shared states.
  2. The tracestate lives on the propagation tags, not in the state.
    • Only extraction writes a trace's tracestate, before the trace is shared, so it needn't be published together with the decision.
    • A tracestate change only replaces the state when its ot member changes.
  3. The inherited ot member lives on the trace.
    • The merge already builds a W3CTracestate holding the member unparsed. The trace now keeps that as its tracestate, and it parses the member on first read.
    • InheritedSamplingState becomes a shared marker that records only the decision and its pending checks.
    • The ot member is now read through PropagationTags.getOtelTraceState(SamplingState). A marker state throws if read directly, so a missed caller fails loudly instead of reading nothing.

Sampling states built per extraction (instrumented count):

scenario before after
Datadog-only 1 0
compound, tracestate without ot 2 0
compound sameVersion (our own injected headers, which carry ot) 2 0
compound withOtelMember 2 0

Motivation

TraceSegment work (APMLP-2049), phase 3: per-trace sampling state at extraction.

Allocation (PropagationScenariosBenchmark, JDK 17, -t 4, default sampler, -prof gc, B/op). These are 1-fork spot checks taken after each step; the same jar varies by about ±16 B from run to run.

step leaf sameVersion intermediate sameVersion
base (#13042) 1704.0 ± 0.1 3328.0 ± 0.2
shared states 1648.0 ± 0.1 3280.0 ± 0.1
tracestate on the tags 1648.0 ± 0.1 3264.0 ± 0.2
ot member on the trace 1608.0 ± 0.1 3208.0 ± 0.1

That's −96 and −120 B per extracted request. The withOtelMember arms move the same way but have ±20–90 B error.

No new per-trace cost: PTags stays at 88 B, because the tracestate field holds either the header value or the W3CTracestate. Shared inherited states are filled on first use. The cache of whether the ot member keeps its position, which W3C injection checks on every inject, moves to the W3CTracestate for inherited members.

Additional Notes

  • Deliberately not done: replacing the preformatted dm and ksr values with primitives. Formatting ksr at export would allocate a string per trace, where the static cache allocates nothing for a single-rate service.
  • Testing:
    • CompoundExtractionGoldenTest passes; full corpus: 0 of 10150 cases differ.
    • Propagation, sampling, OTLP and CoreSpanBuilderTest: 2992 pass.
    • Mutation checks on step 3: two were caught by existing tests. Two survived until the new tests in the last commit (a read under changed checks; locking the decision maker keeping pending checks).
    • dd-trace-core suite (4861 tests): only the known failures, the Docker-only TracerConnectionReliabilityTest and the order-dependent PendingTraceBufferTest flare dump.
  • Throughput not measured; it's part of the stack's interleaved run.
  • Rebase warning: Fix mismatched _dd.p.ksr under concurrent sampling decisions with different rates #12988 (merged to master) fixes the ksr cache race in PTagsFactory. This stack predates it, so a rebase onto master will conflict there.
  • Stacked on Replace the per-trace sampling state lock with compare-and-set #13042.

🤖 Generated with Claude Code

dougqh and others added 5 commits October 23, 2026 20:48
A decision with no tracestate, otel member or sampling rate is just a
priority and a decision maker, and almost every extracted priority and
manual or forced keep is one of a few dozen such pairs. States are
immutable, so those decisions now use preallocated shared states instead
of building one per trace.

Datadog-only extraction no longer builds a sampling state at all, and
compound extraction builds one instead of two.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only extraction replaces a trace's tracestate, before the trace is
shared, so it needn't be published together with the sampling decision.
The tracestate now lives only on the propagation tags, and the codec reads
it from there. A sampling state changes with the tracestate only when its
ot member does, so a tracestate without one keeps the trace's shared
state: compound and W3C extraction of such a tracestate build no sampling
state at all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ember

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An extracted trace with an ot tracestate member still built a sampling
state per trace to hold the member unparsed. The merge already builds a
W3CTracestate holding it, so the trace now keeps that as its tracestate
and it parses the member on first read. The inherited state records only
the decision and its pending checks, so it's shared like the others:
extraction with an ot member no longer builds a sampling state.

The member is now read through the propagation tags, which resolve an
inherited state against the member they hold.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dougqh dougqh added comp: core Tracer core tag: performance Performance related changes tag: no release notes Changes to exclude from release notes type: refactoring comp: context propagation Trace context propagation tag: ai generated Largely based on code generated by an AI or LLM labels Oct 24, 2026
@datadog-prod-us1-4

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.87 s 13.93 s [-1.0%; +0.3%] (no difference)
startup:insecure-bank:tracing:Agent 12.85 s 12.97 s [-1.6%; -0.1%] (maybe better)
startup:petclinic:appsec:Agent 17.25 s 17.04 s [+0.5%; +2.0%] (maybe worse)
startup:petclinic:iast:Agent 17.06 s 16.98 s [-0.3%; +1.3%] (no difference)
startup:petclinic:profiling:Agent 16.62 s 16.82 s [-2.2%; -0.1%] (maybe better)
startup:petclinic:sca:Agent 17.09 s 16.99 s [-0.2%; +1.4%] (no difference)
startup:petclinic:tracing:Agent 15.69 s 16.22 s [-7.5%; +1.0%] (no difference)

Commit: 9d907af7 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: context propagation Trace context propagation comp: core Tracer core tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes tag: performance Performance related changes type: refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant