test(sampling): widen sampling-rate tolerance to 5 std_dev

[bm1549]

Motivation

Test_SamplingRates.test_sampling_rates sends 2000 requests to a tracer sampling at 50% and checks the kept-trace count lands within a ±k·std_dev band of the expected count. The band was ±4 std_dev.

The kept count is Binomial(n, p), so a correctly-sampling tracer falls outside ±4 std_dev about 6e-5 of the time. This assertion runs ~6,500 times/day across all tracers and weblogs (per CI Visibility), so ±4 std_dev meant a correct tracer failed it purely by chance ~0.4 times/day, ~150/year. Real recent examples on default branches: apache-mod-8.1 (php), spring-boot-jetty and jersey-grizzly2 (java), three in the last 90 days, each a ~4σ tail landing on both sides of 50%.

The comment and message were also wrong. The comment called a ±4 std_dev band a "95% confidence interval" (95% is ~1.96 std_dev), and the message said the test "should fail ~5% of the time" when the real rate was ~0.006%.

Changes

Widen the band to ±5 std_dev and correct the comment and failure message.

band	per run	per day (~6.5k runs)	per year
±4 std_dev (before)	~6e-5	~0.4	~150
±5 std_dev (after)	~6e-7	~0.004	~1

TOTAL_REQUESTS (2000) and the delivery assertion (assert_all_traces_requests_forwarded) are unchanged.

This test is a backstop for severe sampling regressions, not the exact-correctness check. Per-decision correctness is already verified deterministically, using the same Knuth formula, by:

• Test_SamplingDecision.test_sampling_decision — every root span's decision must equal trace_should_be_kept(rate, trace_id).
• Test_SampleRateFunction.test_sample_rate_function — fixed (trace_id → expected decision) fixtures from tests/fixtures/sampling_rates.csv.

Those catch subtle drift; a persistent rate bias is still caught here. Widening the aggregate band doesn't lose that coverage.

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? — No, only tests/test_sampling_rates.py.
• A docker base image is modified? — No.
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed? — No.
  • Get a review from R&P team

🤖 Generated with Claude Code

[github-actions]

CODEOWNERS have been resolved as:

tests/test_sampling_rates.py                                            @DataDog/system-tests-core

[datadog-prod-us1-6]

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 4277cd2 | Docs | Datadog PR Page | Give us feedback!}