[un-def]

The jump pod was a bare pod, reused for as long as it existed. Once evicted, for example due to node pressure, it stayed in the Failed phase and was never recreated, so every job in the project failed until someone deleted the pod by hand.

The jump pod is now managed by a ReplicaSet, which replaces the pod after an eviction or a node removal. Not a Deployment: its ReplicaSet selects pods by the pod-template-hash label and thus cannot adopt the existing pod, and rollouts and rollbacks are not needed anyway.

• run_job() creates the ReplicaSet if it doesn't exist. The ReplicaSet adopts a running legacy pod, a finished one is deleted. A pod created before Kubernetes: standardize object labeling #3891 has different labels and is left as is until it's gone.
• A service created before Kubernetes: standardize object labeling #3891 selects pods by the old labels. Once the old pod is gone, the service is recreated with the new selector and the same NodePort, so that ssh_proxy of running jobs stays valid.
• update_provisioning_data() lists jump pods by label and picks a ready one. It falls back to the name for a pod created before Kubernetes: standardize object labeling #3891 and fails right away if that pod is finished, as it's never replaced.
• Creating the namespace, the ReplicaSet, and the service ignores 409 to tolerate concurrent run_job() calls.

Notes:

• The Role now needs get/create on replicasets and list on pods. Without the update, Kubernetes provisioning fails with 403.
• Once the legacy pod is gone, servers before this commit recreate a bare pod and the service on each run_job(). The ReplicaSet adopts and deletes the pod, and the NodePort changes if proxy_jump.port is not set. Downgrading requires deleting the ReplicaSet.

Part-of: #4407