[justin-lesko]

What does this PR do?

Adds a collectionMode field (poll|watch) to the event collection feature, mapped to the kubernetes_apiserver check's event_collection_mode option. "watch" streams events through a single persistent reflector-backed watch instead of re-opening a watch on every check run.

Motivation

Expose a new agent configuration.

Minimum Agent Versions

• Cluster Agent: 7.81.1

Describe your test plan

Tested on minikube (Kubernetes v1.37.0, containerd 2.3.4) with an operator image built from this branch. Each scenario was checked by running agent configcheck in the Cluster Agent, where the kubernetes_apiserver check runs.

Default (no collectionMode)

Supporting configuration

...
spec:
  global:
    ...
  # no features.eventCollection fields set
...

Proof

# CHECK USES THE CLUSTER AGENT IMAGE'S DEFAULT CONFIG
root@datadog-cluster-agent-7f56fd9b9-dpfxh:/# agent configcheck
=== kubernetes_apiserver check ===
Configuration provider: file
Configuration source: file:/etc/datadog-agent/conf.d/kubernetes_apiserver.d/conf.yaml.default
Config for instance ID: kubernetes_apiserver:d7eb902eea9f51f7
{}
~

# NO CHECK CONFIGMAP IS CREATED
$ kubectl -n system get configmap datadog-kube-apiserver-config
Error from server (NotFound): configmaps "datadog-kube-apiserver-config" not found

Outcome: PASS existing behavior is unchanged: no ConfigMap, and the check runs from the image default.

collectionMode: poll

Supporting configuration

...
spec:
  features:
    eventCollection:
      collectionMode: poll
...

Proof

# CHECK USES THE CLUSTER AGENT IMAGE'S DEFAULT CONFIG (SAME POD AS DEFAULT, NO ROLLOUT)
root@datadog-cluster-agent-7f56fd9b9-dpfxh:/# agent configcheck
=== kubernetes_apiserver check ===
Configuration provider: file
Configuration source: file:/etc/datadog-agent/conf.d/kubernetes_apiserver.d/conf.yaml.default
Config for instance ID: kubernetes_apiserver:d7eb902eea9f51f7
{}
~

# NO CHECK CONFIGMAP IS CREATED
$ kubectl -n system get configmap datadog-kube-apiserver-config
Error from server (NotFound): configmaps "datadog-kube-apiserver-config" not found

Outcome: PASS an explicit poll is treated like the default: no ConfigMap, and the Cluster Agent isn't restarted.

collectionMode: watch

Supporting configuration

...
spec:
  features:
    eventCollection:
      collectionMode: watch
...

Proof

# CHECK USES THE OPERATOR-MANAGED CONFIG WITH WATCH MODE
root@datadog-cluster-agent-59fb99cbf4-xbgpd:/# agent configcheck
=== kubernetes_apiserver check ===
Configuration provider: file
Configuration source: file:/etc/datadog-agent/conf.d/kubernetes_apiserver.d/kubernetes_apiserver.yaml
Config for instance ID: kubernetes_apiserver:33baea1590a247b8
event_collection_mode: watch
unbundle_events: false
~

# CONFIGMAP OMITS collected_event_types WHEN UNBUNDLING IS OFF
$ kubectl -n system get configmap datadog-kube-apiserver-config -o jsonpath='{.data.kubernetes_apiserver\.yaml}'
init_config: null
instances:
- event_collection_mode: watch
  unbundle_events: false

Outcome: PASS event_collection_mode: watch reaches the check and the Cluster Agent logs no config errors.

unbundleEvents only (no collectionMode)

Supporting configuration

...
spec:
  features:
    eventCollection:
      unbundleEvents: true
      collectedEventTypes:
        - kind: Pod
          reasons: ["Failed", "BackOff"]
        - kind: Node
          reasons: ["NodeNotReady"]
...

Proof

# UNBUNDLED CONFIG WITHOUT event_collection_mode
root@datadog-cluster-agent-74fbcf797-vnxqp:/# agent configcheck
=== kubernetes_apiserver check ===
Configuration provider: file
Configuration source: file:/etc/datadog-agent/conf.d/kubernetes_apiserver.d/kubernetes_apiserver.yaml
Config for instance ID: kubernetes_apiserver:ac3c90410994b66d
collected_event_types:
  - kind: Pod
    reasons:
      - Failed
      - BackOff
  - kind: Node
    reasons:
      - NodeNotReady
unbundle_events: true
~

Outcome: PASS the unbundled-events ConfigMap is the same as before this change, with no event_collection_mode key. An empty value would make the Cluster Agent reject the config.

collectionMode: watch with unbundleEvents

Supporting configuration

...
spec:
  features:
    eventCollection:
      collectionMode: watch
      unbundleEvents: true
      collectedEventTypes:
        - kind: Pod
          reasons: ["Failed", "BackOff"]
        - kind: Node
          reasons: ["NodeNotReady"]
...

Proof

# BOTH SETTINGS IN ONE CONFIG
root@datadog-cluster-agent-7cc4f96c9b-9c8kb:/# agent configcheck
=== kubernetes_apiserver check ===
Configuration provider: file
Configuration source: file:/etc/datadog-agent/conf.d/kubernetes_apiserver.d/kubernetes_apiserver.yaml
Config for instance ID: kubernetes_apiserver:7a5b73b4a601d6ef
collected_event_types:
  - kind: Pod
    reasons:
      - Failed
      - BackOff
  - kind: Node
    reasons:
      - NodeNotReady
event_collection_mode: watch
unbundle_events: true
~

Outcome: PASS watch mode and unbundled events are both in the same check config.

collectionMode: watch, Cluster Agent < 7.81.1

Supporting configuration

...
spec:
  features:
    eventCollection:
      collectionMode: watch
  override:
    clusterAgent:
      image:
        tag: 7.81.0 # minimum is 7.81.1
...

Proof

# OPERATOR LOGS INCLUDE ERROR
{"level":"INFO","ts":"2026-10-21T17:37:27.810Z","logger":"controllers.DatadogAgentInternal","msg":"features.eventCollection.collectionMode watch requires a newer Cluster Agent, ignoring and using the Cluster Agent default","namespace":"system","name":"datadog","reconcileID":"095d040f-0329-4e85-854a-1c1f0ed4d0e4","kind":"DatadogAgentInternal","version":"7.81.0","minVersion":"7.81.1-0"}

# CHECK FALLS BACK TO THE CLUSTER AGENT IMAGE'S DEFAULT CONFIG
root@datadog-cluster-agent-7795f6fdcb-q45cx:/# agent configcheck
=== kubernetes_apiserver check ===
Configuration provider: file
Configuration source: file:/etc/datadog-agent/conf.d/kubernetes_apiserver.d/conf.yaml.default
Config for instance ID: kubernetes_apiserver:d7eb902eea9f51f7
{}
~

# NO CHECK CONFIGMAP (THE ONE FROM THE PREVIOUS SCENARIO WAS CLEANED UP)
$ kubectl -n system get configmap datadog-kube-apiserver-config
Error from server (NotFound): configmaps "datadog-kube-apiserver-config" not found

Outcome: PASS watch is not enabled on an unsupported Cluster Agent, and the operator's logs include the configuration error.

collectionMode: watch, Cluster Agent = 7.81.1

Supporting configuration

...
spec:
  features:
    eventCollection:
      collectionMode: watch
  override:
    clusterAgent:
      image:
        tag: 7.81.1 # minimum version (backported in DataDog/datadog-agent#53329)
...

Proof

# CHECK USES WATCH MODE ON THE MINIMUM VERSION
root@datadog-cluster-agent-5bbdd55c6d-hmx7f:/# agent configcheck
=== kubernetes_apiserver check ===
Configuration provider: file
Configuration source: file:/etc/datadog-agent/conf.d/kubernetes_apiserver.d/kubernetes_apiserver.yaml
Config for instance ID: kubernetes_apiserver:33baea1590a247b8
event_collection_mode: watch
unbundle_events: false
~

# CHECK RUNS SUCCESSFULLY WITH WATCH MODE
root@datadog-cluster-agent-5bbdd55c6d-hmx7f:/# agent status
...
    kubernetes_apiserver
    --------------------
      Instance ID: kubernetes_apiserver [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/kubernetes_apiserver.d/kubernetes_apiserver.yaml[0]
      Total Runs: 3
...

Outcome: PASS the minimum-version boundary is inclusive, and the 7.81.1 check runs [OK] with watch mode.

Invalid collectionMode

Supporting configuration

...
spec:
  features:
    eventCollection:
      collectionMode: wach # invalid
...

Proof

# API SERVER REJECTS THE VALUE
$ kubectl -n system apply -f collection-mode-invalid.yaml
The DatadogAgent "datadog" is invalid: spec.features.eventCollection.collectionMode: Unsupported value: "wach": supported values: "poll", "watch"

Outcome: PASS CRD enum validation rejects invalid values before they reach the operator.

Checklist

• PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
• PR has a milestone or the qa/skip-qa label
• All commits are signed (see: signing commits)

[datadog-prod-us1-4]

 

🎯 Code Coverage (details)
• Patch Coverage: 83.33%
• Overall Coverage: 55.85% (+0.95%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: e7029ff | Docs | Give us feedback!}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-21T17:47:50.294684Z	665cfda	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 665cfdaad2

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Derive the gate from the effective Cluster Agent image

When image.name is a full tagged reference and image.tag is also present, the image resolver ignores image.tag, but GetAgentVersionFromImage gives it priority. For example, name: cluster-agent:7.81.1 with a stale tag: 7.81.0 deploys 7.81.1 while this gate silently disables the requested watch mode; the inverse can enable it on an unsupported image. Resolve the version using the same precedence as the actual image assembly.

Useful? React with 👍 / 👎.

[Copilot]

🟢 Approval recommended

Configuration wiring, version gating, generated schemas, and test coverage are consistent, with no blocking issues identified.

No open issues 🎉

2 files not reviewed

• api/datadoghq/v2alpha1/zz_generated.deepcopy.go: Generated file
• api/datadoghq/v2alpha1/zz_generated.openapi.go: Generated file

What changed in this PR

Exposes Kubernetes event collection mode through the Datadog Operator, enabling persistent watch collection on supported Cluster Agents while preserving default behavior.

Changes:

• Adds optional collectionMode: poll|watch configuration with schema validation.
• Generates watch configuration only for Cluster Agent 7.81.1+.
• Updates generated APIs, documentation, and configuration tests.

File	Description
pkg/​testutils/​builder.go	Adds collection-mode test helper.
internal/​controller/​datadogagent/​feature/​eventcollection/​feature.go	Wires collection mode with version gating.
internal/​controller/​datadogagent/​feature/​eventcollection/​feature_test.go	Covers modes, version boundaries, and unbundling.
internal/​controller/​datadogagent/​feature/​eventcollection/​configmap.go	Emits optional collection-mode configuration.
docs/​configuration.v2alpha1.md	Documents the new option.
docs/​configuration_public.md	Updates public configuration reference.
config/​crd/​bases/​v1/​datadoghq.com_datadogagents.yaml	Adds Agent schema validation.
config/​crd/​bases/​v1/​datadoghq.com_datadogagents_v2alpha1.json	Updates Agent JSON schema.
config/​crd/​bases/​v1/​datadoghq.com_datadogagentprofiles.yaml	Adds profile schema validation.
config/​crd/​bases/​v1/​datadoghq.com_datadogagentprofiles_v1alpha1.json	Updates profile JSON schema.
config/​crd/​bases/​v1/​datadoghq.com_datadogagentinternals.yaml	Adds internal resource schema validation.
config/​crd/​bases/​v1/​datadoghq.com_datadogagentinternals_v1alpha1.json	Updates internal resource JSON schema.
api/​datadoghq/​v2alpha1/​zz_generated.openapi.go	Exposes the field in OpenAPI.
api/​datadoghq/​v2alpha1/​zz_generated.deepcopy.go	Copies the optional field.
api/​datadoghq/​v2alpha1/​datadogagent_types.go	Defines and documents the API field.

_{Review effort: Balanced}

[maycmlee]

Just some suggestions