[AliDatadog]

What does this PR do?

Adds opt-in setting to check-operator upgrade: AGENT_SKIP_NOT_READY_NODES.

The check does not count nodes that are not Ready and whose oldest Agent pod is out of date. These nodes (Ready condition False or Unknown) usually cannot run the new pod, so today they hold the check until its timeout.

Two more settings tune the safety limits:

Variable	Default	Effect
AGENT_SKIP_NOT_READY_MAX_PCT	10	The most % of the scheduled nodes that the check can skip. Above it, the check skips none.
AGENT_SKIP_NOT_READY_MAX_NODE_LOOKUPS	300	The most nodes with an out-of-date Agent pod that the check looks up Above it, the check skips none. Values of 0 or less are ignored.

Motivation

The check runs as a Helm post-upgrade hook. On some internal clusters, significant % of the Agent pods sit on nodes that are Unknown (the kubelet stopped reporting). So the check waits much longer than the rollout of the healthy nodes:

• mudkip-d, 2026-10-21: the check ran from 18:58 to 19:44 UTC (46 min), according to its logs.
• A replay of the same rule from kubernetes_state.daemonset.* metrics matches the logs. It also shows that the gap stayed at 45–64 pods for 30 min, close to the number of pods on dead nodes.

Evidence: https://app.datadoghq.com/notebook/16020842

How it counts

The check removes nodes from Current (currentNumberScheduled):

• It counts nodes, not pods, one per DaemonSet. It looks only at the oldest pod on the node, like the controller.
• It ignores a pod only if it is both terminal and deleted. A pod stuck in Terminating on a dead node still counts. This is the usual case after the controller deletes an unavailable old pod.
• A pod is up to date if its pod-template-generation or its controller-revision-hash matches. The check reads the current hash from a pod on the current generation. If there is no such pod yet, or the pod belongs to another Agent DaemonSet with the same selector, the hash is unknown, so the check skips no node of that DaemonSet. Otherwise, after a template revert, it could skip pods that the status counts as updated.
• It does not count deleted nodes, because the status does not count them.
• It only counts a node that the current template targets, like nodeShouldRunDaemonPod: its nodeName, required node affinity and nodeSelector, and its NoSchedule/NoExecute taints (template tolerations plus the DaemonSet default tolerations). The pod on the node may come from an older template, for example before a profile label was added to the node. The status does not count these nodes either.
• So the setting only helps if the Agent tolerates the node.kubernetes.io/not-ready:NoSchedule and node.kubernetes.io/unreachable:NoSchedule taints that not-Ready nodes get (for example operator: Exists). Otherwise the status already leaves these nodes out.

flowchart TD
  A[Plain threshold reached?] -->|yes| P[Agent done]
  A -->|no, setting off| W[Wait]
  A -->|no, setting on| G{Passes with at most MAX_PCT of Current removed? Find the fewest nodes N that would pass}
  G -->|no| W
  G -->|yes| B[List out-of-date pods per Agent DaemonSet]
  B --> C{More than MAX_NODE_LOOKUPS nodes?}
  C -->|yes| W
  C -->|no| D[GET each node. Keep not-Ready nodes that the status counts and whose oldest pod is out of date]
  D --> E{Fewer than N, or more than MAX_PCT of Current?}
  E -->|yes| W
  E -->|no| F[Current -= skipped nodes. Check the threshold again]

Loading

Safety limits:

• If more than AGENT_SKIP_NOT_READY_MAX_PCT (default 10%) of the scheduled nodes would be skipped, the check skips none. A large number of not-Ready nodes is more likely an outage than background churn.
• If the check cannot pass even with AGENT_SKIP_NOT_READY_MAX_PCT of the nodes removed, it makes no API call. Early in a rollout, the out-of-date pod list would return almost every Agent pod.
• If more than AGENT_SKIP_NOT_READY_MAX_NODE_LOOKUPS (default 300) nodes run an out-of-date Agent pod, the check skips none.
• The check stops its API calls as soon as it has fewer candidates than the N nodes it needs, or more than the cap.
• The out-of-date and per-node pod lists are served from the API server watch cache (resourceVersion=0). Before Kubernetes 1.31, a list with a field selector otherwise reads every pod of the namespace from etcd.
• On any API error, the check logs the error and uses the plain threshold.
• When it skips, the check logs the skipped <DaemonSet>/<node> pairs (up to 20) and the adjusted node count.

Additional Notes

• New permissions needed when the setting is on: list on pods and daemonsets in the DatadogAgent namespace, and get on nodes (a ClusterRole, as nodes are cluster-scoped). The hook RBAC lives in the chart that deploys the check, not in this repo.
• Change size: Medium. It is opt-in, but it makes a rollout check looser when it is on, and it needs new read permissions.
• Known limits:
  • A node whose only Agent pod is Pending with no node assigned is not skipped. This is the strict side.
  • The check uses the DatadogAgent status, which the operator writes at reconcile time, together with live pods and nodes. Pods on dead nodes stay for a long time, so the risk of a mismatch is low.

Minimum Agent Versions

None.

Describe your test plan

Unit tests.

How to reproduce on a 5-node kind cluster

Goal: prove the upgrade check skips only not-Ready nodes that hold an out-of-date Agent pod, and that
each safety limit (AGENT_SKIP_NOT_READY_MAX_PCT, AGENT_SKIP_NOT_READY_MAX_NODE_LOOKUPS, missing
RBAC, broken rollout) keeps the check strict.

Prerequisites

• A kind cluster with 1 control-plane and 4 worker nodes (names below assume kind create cluster --name skip-notready).
• The operator-check image built from this PR, loaded in kind. Set CHECK_IMAGE to its name (default operator-check:skip-notready).
• The Datadog Operator installed with Helm in namespace datadog (the Operator itself is unchanged by this PR; any recent version works):

  kubectl create namespace datadog
  kubectl -n datadog create secret generic datadog-secret --from-literal=api-key=00000000000000000000000000000000
  helm install datadog-operator datadog/datadog-operator -n datadog --wait

Why the threshold settings below: the check runs as a Job like the Helm hook, with
AGENT_COMPLETION_MIN=0 (the default 10 would pass on a 5-node cluster without skipping anything) and
CHECK_TIMEOUT_MINUTES=2 so failing cases end quickly. With 5 Agent nodes, the default 10% cap allows
int(5 × 0.10) = 0 skipped nodes, so the passing cases raise AGENT_SKIP_NOT_READY_MAX_PCT.

Files

dda.yaml:

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
  namespace: datadog
spec:
  global:
    clusterName: skip-notready-qa
    kubelet:
      tlsVerify: false
    credentials:
      apiSecret:
        secretName: datadog-secret
        keyName: api-key
  override:
    nodeAgent:
      # Tolerate every taint, like production Agent DaemonSets. Without it, the DaemonSet status
      # drops the dead node (unreachable:NoSchedule taint) and there is nothing to skip.
      tolerations:
        - operator: Exists
      # The stuck Terminating pod on the dead node uses one unavailable slot. Production clusters get
      # more slots (10% of the nodes); on 5 nodes, allow 2 so the healthy nodes keep rolling.
      updateStrategy:
        type: RollingUpdate
        rollingUpdate:
          maxUnavailable: 2

check-rbac.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: check-operator
  namespace: datadog
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: check-operator
  namespace: datadog
rules:
  - apiGroups: ["datadoghq.com"]
    resources: ["datadogagents"]
    verbs: ["get"]
  - apiGroups: ["datadoghq.com"]
    resources: ["datadogagentinternals"]
    verbs: ["list"]
  # Only needed with AGENT_SKIP_NOT_READY_NODES=true.
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["list"]
  - apiGroups: ["apps"]
    resources: ["daemonsets"]
    verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: check-operator
  namespace: datadog
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: check-operator
subjects:
  - kind: ServiceAccount
    name: check-operator
    namespace: datadog
---
# Only needed with AGENT_SKIP_NOT_READY_NODES=true.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: check-operator-nodes
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: check-operator-nodes
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: check-operator-nodes
subjects:
  - kind: ServiceAccount
    name: check-operator
    namespace: datadog

run-check.sh (runs one check as a Job, prints PASSED/FAILED and its logs):

#!/usr/bin/env bash
# Usage: ./run-check.sh <case> <skip true|false> <max pct> <max node lookups> [completion min, default 0]
set -euo pipefail
NAME=$1 SKIP=$2 PCT=$3 LOOKUPS=$4 MIN=${5:-0}
IMAGE=${CHECK_IMAGE:-operator-check:skip-notready}
kubectl -n datadog delete job "check-$NAME" --ignore-not-found >/dev/null
cat <<YAML | kubectl apply -f - >/dev/null
apiVersion: batch/v1
kind: Job
metadata:
  name: check-$NAME
  namespace: datadog
spec:
  backoffLimit: 0
  template:
    spec:
      serviceAccountName: check-operator
      restartPolicy: Never
      containers:
        - name: check
          image: $IMAGE
          imagePullPolicy: IfNotPresent
          args: ["upgrade", "datadog"]
          env:
            - {name: CHECK_TIMEOUT_MINUTES, value: "2"}
            - {name: AGENT_COMPLETION_PCT, value: "95"}
            - {name: AGENT_COMPLETION_MIN, value: "$MIN"}
            - {name: AGENT_SKIP_NOT_READY_NODES, value: "$SKIP"}
            - {name: AGENT_SKIP_NOT_READY_MAX_PCT, value: "$PCT"}
            - {name: AGENT_SKIP_NOT_READY_MAX_NODE_LOOKUPS, value: "$LOOKUPS"}
YAML
# Wait for either outcome; the check times out after 2 minutes.
for _ in $(seq 1 100); do
  RESULT=$(kubectl -n datadog get job "check-$NAME" -o jsonpath='{.status.succeeded}/{.status.failed}')
  [[ $RESULT == 1/ ]] && { echo "=== $NAME: PASSED"; break; }
  [[ $RESULT == /1 ]] && { echo "=== $NAME: FAILED"; break; }
  sleep 3
done
kubectl -n datadog logs "job/check-$NAME" | grep -v "v2alpha1 is available"

Steps

1. Deploy the Agent

chmod +x run-check.sh
kubectl apply -f dda.yaml -f check-rbac.yaml
kubectl -n datadog rollout status ds/datadog-agent --timeout=5m
kubectl -n datadog get pods -l agent.datadoghq.com/component=agent -o wide   # 5 pods, one per node

Baseline: ./run-check.sh baseline true 25 300 → PASSED, nothing skipped.
The first log line must list the settings: skip not-Ready nodes: true, max node lookups: 300, max skipped pct: 25.

2. One dead node during a rollout

docker exec skip-notready-worker3 systemctl stop kubelet
# Wait ~40s until the node is Unknown with the unreachable:NoSchedule taint
kubectl get node skip-notready-worker3 -o jsonpath='{.status.conditions[?(@.type=="Ready")].status} {.spec.taints}{"\n"}'
kubectl -n datadog patch datadogagent datadog --type merge \
  -p '{"spec":{"override":{"nodeAgent":{"env":[{"name":"QA_ROLLOUT","value":"1"}]}}}}'
# Wait until the DDA status shows upToDate 4 / current 5
kubectl -n datadog get datadogagent datadog -o jsonpath='{.status.agent.upToDate}/{.status.agent.current}{"\n"}'

Expected state: 4 new pods; the old pod on worker3 stays Terminating forever (its kubelet is gone).

#	Command	Expected result	Expected log line
1	./run-check.sh 1-skip-off false 25 300	FAILED (timeout)	[Agent] nb pods: 5, nb updated pods: 4 on every poll
2	./run-check.sh 2-default-10pct true 10 300	FAILED	No skip line: 10% of 5 nodes is 0, so the check makes no extra API call
3	./run-check.sh 3-pct-25 true 25 300	PASSED on the first poll	[Agent] skipping 1 not-Ready nodes with an out-of-date Agent pod, counting 4 nodes instead of 5: datadog-agent/skip-notready-worker3, then reached threshold): 4, nb updated pods: 4
4	./run-check.sh 4-lookup-cap-0 true 25 0	FAILED	1 nodes have an out-of-date Agent pod, above the 0 node lookups cap: not skipping any
5	Missing node RBAC (below)	FAILED	unable to count not-Ready nodes, not skipping any: ... cannot get resource "nodes"

Case 5:

kubectl delete clusterrolebinding check-operator-nodes
./run-check.sh 5-no-node-rbac true 25 300
kubectl apply -f check-rbac.yaml   # restore

3. Two dead nodes: the skip cap

docker exec skip-notready-worker systemctl stop kubelet   # wait until Unknown
kubectl -n datadog patch datadogagent datadog --type merge -p '{"spec":{"override":{"nodeAgent":{
  "env":[{"name":"QA_ROLLOUT","value":"2"}],
  "updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"maxUnavailable":3}}}}}}'
# Wait until upToDate 3 / current 5, with 2 Terminating pods on worker and worker3

#	Command	Expected result	Expected log line
6	./run-check.sh 6-two-dead-cap-25 true 25 300 1	FAILED	2 not-Ready nodes have an out-of-date Agent pod, above the 1 cap (25% of scheduled nodes): not skipping any
7	./run-check.sh 7-two-dead-pct-40 true 40 300	PASSED	skipping 2 not-Ready nodes ... counting 3 nodes instead of 5, then reached threshold): 3, nb updated pods: 3

(Case 6 passes AGENT_COMPLETION_MIN=1 so the check gets past its pre-check and actually counts the 2 nodes.)

4. A broken rollout still blocks the check

docker exec skip-notready-worker systemctl start kubelet   # worker recovers; wait for upToDate 4 / current 5
kubectl -n datadog patch datadogagent datadog --type merge -p '{"spec":{"override":{"nodeAgent":{
  "image":{"name":"gcr.io/datadoghq/agent","tag":"0.0.0-does-not-exist"},
  "updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"maxUnavailable":2}}}}}}'
# Wait ~1 min: one pod in ErrImagePull, rollout stalled at upToDate 1 / current 5

#	Command	Expected result	Expected log line
8	./run-check.sh 8-broken-config-pct-100 true 100 300	FAILED	skipping 1 not-Ready nodes (only worker3), then still nb updated pods: 1. Ready nodes with old pods are never skipped, even with a 100% cap.

Cleanup

kind delete cluster --name skip-notready

Notes for the reviewer

• The Agent must tolerate the node.kubernetes.io/unreachable:NoSchedule taint (the DDA uses operator: Exists). Without it, the DaemonSet status already drops the dead node and there is nothing to skip.
• Every failing case ends with Error: unable to list DatadogAgentInternals: context deadline exceeded. That is how the existing check reports its timeout, not a new error.
• The stuck Terminating pod uses one maxUnavailable slot. If a cluster has more dead nodes than maxUnavailable, the DaemonSet rollout itself stalls, and this setting cannot help.

Results on kind (2026-10-23)

#	Scenario	Settings	Result
0	Healthy baseline	skip on, 25%	PASSED
1	1 dead node, 4/5 updated	skip off	FAILED (timeout)
2	same	skip on, 10% (0 nodes on 5)	FAILED
3	same	skip on, 25%	PASSED, skipping 1
4	same	25%, lookups 0	FAILED, lookup cap
5	same	25%, no get nodes RBAC	FAILED, error logged
6	2 dead nodes, 3/5 updated	25% (cap 1 node), min 1	FAILED, skip cap
7	same	40%	PASSED, skipping 2
8	broken image rollout, 1/5 updated	100%	FAILED, only the dead node skipped

Cases 0–8 ran on e8cd9e2a. Cases 1–4 ran again on ad7bf82f (affinity, ownership and API-load changes), with the same results, during a template revert: generation-3 pods counted as up to date by their revision hash, and only worker3 was skipped.

🤖 Generated with Claude Code

[datadog-datadog-prod-us1]

✨ Unblock PR with BitsAI

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 2 Pipeline jobs failed

DataDog/datadog-operator | build_and_test — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

validation | build — 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: ad7bf82 | Docs | View more details | Give us feedback!}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-22T14:57:36.358381Z	3a0fde1	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3a0fde1e8e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Match the DaemonSet controller's phase-aware pod ordering

When a node has multiple counted DaemonSet pods in different phases—for example, an older out-of-date Pending or terminating pod and a newer up-to-date Running pod—Kubernetes's updateDaemonSetStatus orders Running pods ahead of other phases before comparing creation times. This timestamp-only ordering can therefore report the node as outdated even though UpdatedNumberScheduled counts it as updated; the caller then subtracts it from Current without subtracting it from UpToDate, allowing the opt-in upgrade check to pass before the configured threshold is actually reached.

Useful? React with 👍 / 👎.

[gabedos]

If we are worried about performance and/or API server load from querying all "old" Agent pods as well as their node conditions... we could potentially just calculate the "notReadyDiscount" one time and re-use the same discount value for every future reconciliation loop? We can either do it at the very start of the rollout/check OR at the inflection point wherever the max allowed discount is pivotal in the decision to complete the job.