Repository navigation
Conversation
spec.pipelineConfig runs a YAML pipeline locally instead of fetching pipelineID from Remote Config. The operator renders it into an owned <worker>-pipeline ConfigMap, mounts it read-only, runs `run --pipeline-config=... --skip-key-validation`, falls back to the Worker UID for DD_OP_PIPELINE_ID, and rolls pods via a checksum/pipeline annotation. Remote Config workers render unchanged. pipelineID is now optional in the CRD schema; Go validation requires pipelineID or pipelineConfig on the Worker, and pipelineID on every BYOC pipeline. Demo subset: cleanup of the ConfigMap when local mode is turned off, reserved-name checks and envtest coverage follow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Switching a Worker back to Remote Config now removes its <worker>-pipeline ConfigMap if the Worker controls it. Envtest covers Remote Config -> local -> edit -> Remote Config: the ConfigMap appears, the checksum changes on edit, and switching back leaves no ConfigMap, pipeline args, volume, mount or checksum annotation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In local mode, user volumes named pipeline-config and mounts at, under or over /etc/observability-pipelines-worker/pipeline are rejected; Remote Config Workers keep those names. Envtest adds a local Worker without a pipelineID reaching Available with its UID as the pipeline ID, and an unowned <worker>-pipeline ConfigMap causing a conflict and surviving local mode being turned off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 813bbacde0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
| configPath := fieldPath.Child("pipelineConfig") | ||
| // sigs.k8s.io/yaml decodes only the first document, so later documents would be silently dropped. | ||
| if strings.Contains(*spec.PipelineConfig, "\n---") { |
There was a problem hiding this comment.
For valid YAML streams using CR-only line endings, such as sources: {}\r---\rsinks: {}, this check does not detect the second document. The subsequent unmarshal accepts the first mapping while silently ignoring the remainder, so an invalid multi-document pipelineConfig passes reconciliation and may make the Worker run only part of the supplied configuration or fail at startup. Use a document-aware decoder and require EOF after the first non-empty mapping rather than searching for one separator spelling.
Useful? React with 👍 / 👎.
❌ ErrorsYour PR has failed checks. Please review the issues below and take necessary action before merging. 🚦 1 Pipeline job failed
ℹ️ Info🎯 Code Coverage (details) Useful? React with 👍 / 👎 This comment will be updated automatically if new data arrives.🔗 Commit SHA: 0da64d4 | Docs | View more details | Give us feedback! |
The single-document check matched "\n---", so a stream using CR-only
line breaks (sources: {}\r---\rsinks: {}) passed and the second document
was silently dropped. Decode the first document with a stream decoder
and require EOF after it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What does this PR do?
Adds a local mode to
DatadogObservabilityPipelinesWorker: setspec.pipelineConfigto an Observability Pipelines pipeline in YAML and the Worker runs it instead of fetchingpipelineIDfrom Remote Config.When
pipelineConfigis set, the controller:<worker>-pipeline(keypipeline.yaml), mounted read-only at/etc/observability-pipelines-worker/pipeline;run --pipeline-config=/etc/observability-pipelines-worker/pipeline/pipeline.yaml --skip-key-validation;DD_OP_PIPELINE_IDtopipelineIDif given, otherwise the Worker UID;checksum/pipelinepod annotation so editing the pipeline rolls the pods (OPW does not reload the file).Switching back to Remote Config deletes the ConfigMap (only if the Worker controls it) and removes the args, volume, mount and annotation. Remote Config Workers render exactly as before, so they don't restart on operator upgrade.
API:
pipelineIDis now optional in the CRD schema (it lives on a struct shared withDatadogBYOCClustercomponents.pipelines[]). Validation moves to Go, no CEL: a Worker needspipelineIDorpipelineConfig;pipelineConfigmust be a single, non-empty YAML mapping; every BYOC pipeline still needspipelineID. In local mode, user volumes namedpipeline-configand mounts overlapping the pipeline directory are rejected.Motivation
BYOC APM ships Datadog-authored pipelines rather than Remote Config ones. This is the Worker half; a follow-up lets the
DatadogBYOCClustercreate a child Worker with the pipeline from its release (BYOCAPM-162, blocks BYOCAPM-163).Additional Notes
--pipeline-configand--skip-key-validation. Without--skip-key-validation, startup validates the pipeline ID against Remote Config and fails. The flag also skips the API key check at startup. OPW team sign-off is pending.pipelineIDand would panic on a Worker without one.Available=Falseuntil the spec is fixed.Minimum Agent Versions
N/A (no Agent change).
Describe your test plan
pipelineIDcheck.pipelineIDreachesAvailablewith its UID as the pipeline ID; Remote Config → local → edit (checksum changes) → Remote Config (ConfigMap, args, volume, mount and annotation all gone); an unowned<worker>-pipelineConfigMap reportsConflictand survives local mode being turned off.make generateclean,make lint,go test ./...scope,make integration-testspass.43d60b1e: a local-mode Worker withoutpipelineIDreachedAvailable; spans reached the BYOC cluster and the bridge check passed. PatchingpipelineConfigchanged the StatefulSetupdateRevisionand the rollout completed with all checks still passing.sampleprocessor) passesobservability-pipelines-worker validatebuilt from OPW43d60b1e.Checklist
enhancementqa/skip-qalabel🤖 Generated with Claude Code