Repository navigation
Kubernetes: standardize object labeling - #3891
Merged
Merged
Conversation
Now all dstack-managed resources (jump pod, job pod, gateway, volume,
registry-auth secret, services) get the same set of labels:
* `app.kubernetes.io/name=dstack-{ssh-proxy|job|gateway|volume}`
* `app.kubernetes.io/instance={unique_generated_name}`
* `app.kubernetes.io/managed-by=dstack`
* `k8s.dstack.ai/project`
* `k8s.dstack.ai/name` (if applicable)
* `k8s.dstack.ai/user` (if applicable)
jvstme
approved these changes
May 21, 2026
un-def
added a commit
that referenced
this pull request
Nov 2, 2026
The jump pod was a bare pod, reused for as long as it existed. Once evicted, for example due to node pressure, it stayed in the Failed phase and was never recreated, so every job in the project failed until someone deleted the pod by hand. The jump pod is now managed by a ReplicaSet, which replaces the pod after an eviction or a node removal. Not a Deployment: its ReplicaSet selects pods by the pod-template-hash label and thus cannot adopt the existing pod, and rollouts and rollbacks are not needed anyway. * run_job() creates the ReplicaSet if it doesn't exist. The ReplicaSet adopts a running legacy pod, a finished one is deleted. A pod created before #3891 has different labels and is left as is until it's gone. * A service created before #3891 selects pods by the old labels. Once the old pod is gone, the service is recreated with the new selector and the same NodePort, so that ssh_proxy of running jobs stays valid. * update_provisioning_data() lists jump pods by label and picks a ready one. It falls back to the name for a pod created before #3891 and fails right away if that pod is finished, as it's never replaced. * Creating the namespace, the ReplicaSet, and the service ignores 409 to tolerate concurrent run_job() calls. Notes: * The Role now needs get/create on replicasets and list on pods. Without the update, Kubernetes provisioning fails with 403. * Once the legacy pod is gone, servers before this commit recreate a bare pod and the service on each run_job(). The ReplicaSet adopts and deletes the pod, and the NodePort changes if proxy_jump.port is not set. Downgrading requires deleting the ReplicaSet. Part-of: #4407
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Now all dstack-managed resources (jump pod, job pod, gateway, volume, registry-auth secret, services) get the same set of labels:
app.kubernetes.io/name=dstack-{ssh-proxy|job|gateway|volume}app.kubernetes.io/instance={unique_generated_name}app.kubernetes.io/managed-by=dstackk8s.dstack.ai/projectk8s.dstack.ai/name(if applicable)k8s.dstack.ai/user(if applicable)