Skip to content

[DET-109][jobs] inject ol_env into ol spark run tags#11786

Draft
ajyu wants to merge 1 commit into
masterfrom
ansha/ol-env
Draft

[DET-109][jobs] inject ol_env into ol spark run tags#11786
ajyu wants to merge 1 commit into
masterfrom
ansha/ol-env

Conversation

@ajyu

@ajyu ajyu commented Jun 29, 2026

Copy link
Copy Markdown

Fixes DET-109

Adds _dd.ol_env:<env> to the spark.openlineage.run.tags set in
setupOpenLineage, alongside the existing _dd.ol_service / _dd.ol_app_id.
The value comes from Config.get().getEnv(), and the tag is omitted when env
is empty.

Motivation

Downstream (dd-source lineage-processor) uses the run environment as the
Spark application UGP node's namespace. A node's entity_id is
hash(name, type, namespace); today the OpenLineage path and the tracer-only
path (djm-span-processor) derive namespace differently, so the same Spark
application resolves to two different entities. Carrying env here lets both
paths key the namespace off the same value (env) and converge on one
entity_id. _dd.ol_service already aligns name; this aligns namespace.

When env is unset the tag is omitted so the consumer falls back to the
OpenLineage namespace (no empty-namespace).

Additional Notes

  • Consumer side
  • Existing setupOpenLineage tests assert via .contains(...), so the new tag
    doesn't affect them; added a test asserting _dd.ol_env:<env> is present when
    dd.env is set.

@ajyu ajyu changed the title [do][job] inject ol_env into ol spark run tags [DET-109][job] inject ol_env into ol spark run tags Jun 29, 2026
@ajyu ajyu changed the title [DET-109][job] inject ol_env into ol spark run tags [DET-109][jobs] inject ol_env into ol spark run tags Jun 29, 2026
@dd-octo-sts

dd-octo-sts Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.98 s 13.91 s [-0.5%; +1.4%] (no difference)
startup:insecure-bank:tracing:Agent 12.91 s 13.00 s [-1.4%; +0.0%] (no difference)
startup:petclinic:appsec:Agent 16.89 s 16.18 s [+0.1%; +8.8%] (maybe worse)
startup:petclinic:iast:Agent 16.86 s 16.93 s [-1.3%; +0.4%] (no difference)
startup:petclinic:profiling:Agent 16.82 s 16.43 s [-2.1%; +6.8%] (no difference)
startup:petclinic:sca:Agent 16.88 s 16.68 s [+0.2%; +2.2%] (maybe worse)
startup:petclinic:tracing:Agent 16.20 s 15.71 s [-1.2%; +7.4%] (no difference)

Commit: 8c07a2fe · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant