Skip to content

Add a startupProbe to the A4X Max asapd-lite DaemonSet - #646

Open
jkru3 wants to merge 1 commit into
GoogleCloudPlatform:masterfrom
jkru3:asapd-lite-a4x-max-startup-probe
Open

jkru3 wants to merge 1 commit into
GoogleCloudPlatform:masterfrom
jkru3:asapd-lite-a4x-max-startup-probe

Conversation

@jkru3

@jkru3 jkru3 commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator

On first boot, asapd-lite configures all eight ConnectX-8 ports and sets up DOCA PCC before /healthz starts serving. This can run past the 60 second liveness window, so kubelet kills the container (exit code 143) and setup starts over. In a 16-node A4X Max test on Ubuntu 26.04 nodes, kubelet killed 9 of 12 asapd-lite pods this way. The GKE A4X Max guide has users kubectl apply this manifest directly, so they can hit the same restarts

  • Add a startupProbe on /healthz port 19540 with periodSeconds: 10 and failureThreshold: 60, which gives setup up to 10 minutes to finish
  • Keep the existing livenessProbe and readinessProbe as they are. Kubernetes holds them until the startup probe passes
  • Checked that the manifest parses and that the new probe is the only change. Not run on hardware with this exact file

On first boot, asapd-lite configures all eight ConnectX-8 ports and sets
up DOCA PCC before /healthz starts serving. This can run past the 60
second liveness window, so kubelet kills the container with exit code
143 and setup starts over. In a 16-node A4X Max test on Ubuntu 26.04
nodes, kubelet killed 9 of 12 asapd-lite pods this way

Add a startupProbe on the same /healthz endpoint that gives setup up to
10 minutes (60 checks, 10 seconds apart) to finish. The liveness and
readiness probes are unchanged and start once the startup probe passes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant