P3 - container-runtime/copilot-runtime-bazel-cache-container-oomkilled on catalog_service:copilot-runtime-bazel-cache,kube_cluster:dotcom-1-gc-cus-02,kube_deployment:copilot-runtime-bazel-cache,kube_namespace:copilot-runtime-bazel-ca... - 2026-10-23T20:02Z #5248

Alert Details

Source: [P3] [Triggered] container-runtime/copilot-runtime-bazel-cache-container-oomkilled on catalog_service:copilot-runtime-bazel-cache,kube_cluster:dotcom-1-gc-cus-02,kube_deployment:copilot-runtime-bazel-cache,kube_namespace:copilot-runtime-bazel-cache-production
Status: CRITICAL
Triggered at: 2026-10-23T20:02Z

This issue was created because the "create_github_issue_on_alert:true" tag was applied to the monitor. This issue was created by ISP.

%%%
At least one container in deployment copilot-runtime-bazel-cache (service copilot-runtime-bazel-cache, namespace copilot-runtime-bazel-cache-production) in cluster dotcom-1-gc-cus-02 reported an OOM-terminated state during the last 10 minutes. The container exceeded its memory limit and was terminated.

Investigate memory growth or a possible leak, or raise the memory limit. See also the memory-usage-vs-limits saturation check.

Dashboard: Kubernetes Workload Observability

Playbook: https://github.com/github/ops/blob/master/docs/playbooks/kubernetes/workload-alerts/oomkilled.md

NOTE: Fleet-wide platform-default check (priority 3). When this fires, a GitHub issue is opened against your service's Service Catalog owning team. For github/github deployments, the issue goes to the deployment's owner in fleetwide-aliases.yaml, or to monolith platform if the deployment isn't listed.

HOW TO OPT OUT

1. Pick ONE of the following exclusions:

   • !catalog_service:<your-service-slug> — excludes all your workloads (not for github/github: every monolith deployment shares catalog_service:github, so monolith teams must exclude their kube_deployment instead)
   • !kube_deployment:<your-deployment> — excludes one deployment
   • !kube_namespace:<your-namespace> — excludes an entire namespace

2. Open a PR in github/datadog-monitoring adding your chosen exclusion inside every {...} filter block in this monitor's query. The CPU monitors (usage-vs-limits and throttling) have FOUR filter blocks — the exclusion must appear in every one of them or the filter is inconsistent. Memory and error monitors have one block. Repeat on all six monitor families for a full opt-out.

Cluster-level opt-outs are not accepted — this is a per-service safety net.

Questions? #compute-platform-support

@webhook-isp-alert

max(last_10m):default_zero(sum:kubernetes_state.container.status_report.count.terminated{!kube_namespace:*staging*,!catalog_service:N/A,!kube_deployment:N/A,kube_deployment:*,catalog_service:*,!kube_cluster_profile:test,!kube_cluster_profile:akstest,!kube_cluster_profile:actionspoc,!kube_cluster_profile:review-lab,!kube_cluster_profile:proxima-review-lab,!kube_cluster_stamp:test*,reason:oomkilled} by {kube_namespace,kube_deployment,catalog_service,kube_cluster}) >= 1

The monitor was last triggered at Fri Oct 23 2026 20:02:19 UTC.

[Monitor Status] · [Edit Monitor] · [Related Logs]
%%%

Tags

alerting_stage:notifying,catalog_service:copilot-runtime-bazel-cache,create_github_issue_on_alert:true,fleet-wide,kube_cluster:dotcom-1-gc-cus-02,kube_deployment:copilot-runtime-bazel-cache,kube_namespace:copilot-runtime-bazel-cache-production,managed:true,moda_health_check:kubernetes.container_oom_killed,monitor,priority:3,priority:p3,reason:oomkilled,routing_mode:fleetwide,team:compute-platform

On-Call

(none)