[Feature]: Support Python version upgrade for gateways and migrate from Python 3.10

Problem

We want to drop Python 3.10 support from dstack, but every gateway runs dstack on Python 3.10, and there is no way to change a gateway's Python version.

Gateways are provisioned on Ubuntu 22.04 and use the system python3 (3.10) for the blue/green venvs. This applies to both cloud VMs (cloud-init installs python3.10-venv) and Kubernetes gateway pods (ubuntu:22.04):

dstack/src/dstack/_internal/core/backends/base/compute.py

Lines 1273 to 1286 in 43874b1

    
           def get_gateway_user_data(authorized_key: str) -> str: 
        
               return get_cloud_config( 
        
                   package_update=True, 
        
                   packages=[ 
        
                       "nginx", 
        
                       "python3.10-venv", 
        
                   ], 
        
                   snap={"commands": [["install", "--classic", "certbot"]]}, 
        
                   runcmd=[ 
        
                       ["ln", "-s", "/snap/bin/certbot", "/usr/bin/certbot"], 
        
                       ["su", "ubuntu", "-c", " && ".join(get_dstack_gateway_commands())], 
        
                   ], 
        
                   ssh_authorized_keys=[authorized_key], 
        
               )

dstack/src/dstack/_internal/core/backends/base/compute.py

Lines 1392 to 1400 in 43874b1

    
           def get_dstack_gateway_commands() -> List[str]: 
        
               gateway_package, _ = get_dstack_gateway_package_and_target_version() 
        
               return [ 
        
                   "mkdir -p /home/ubuntu/dstack", 
        
                   "python3 -m venv /home/ubuntu/dstack/blue", 
        
                   "python3 -m venv /home/ubuntu/dstack/green", 
        
                   f"/home/ubuntu/dstack/blue/bin/pip install '{gateway_package}'", 
        
                   "sudo /home/ubuntu/dstack/blue/bin/python -m dstack._internal.proxy.gateway.systemd install --run", 
        
               ]

dstack/src/dstack/_internal/core/backends/kubernetes/compute.py

Line 1416 in 43874b1

# install packages

The blue/green update that the server runs on startup only reinstalls the package into the inactive venv with its existing interpreter, so a gateway stays on the Python it was provisioned with:

dstack/src/dstack/_internal/server/services/gateways/__init__.py

Lines 844 to 876 in 43874b1

    
           # Blue/green: install the new build into the currently inactive venv and flip to it 
        
           _GATEWAY_UPDATE_SCRIPT = """\ 
        
           set -e 
        
           gateway_package="$1" 
        
           build="$2" 
        
           root=/home/ubuntu/dstack 
        
           if [ -f "$root/version" ]; then 
        
             version=$(cat "$root/version") 
        
           else 
        
             version=blue 
        
           fi 
        
           if [ -n "$build" ]; then 
        
             current_build=$("$root/$version/bin/pip" show dstack | grep Version | awk '{print $2}') 
        
             if [ "$current_build" = "$build" ]; then 
        
               echo "The build $build is already installed. Skipping..." 
        
               exit 0 
        
             fi 
        
           fi 
        
           if [ "$version" = blue ]; then 
        
             version=green 
        
           else 
        
             version=blue 
        
           fi 
        
           # dstack-gateway is a pre-0.21.4 package that may still be installed and require a conflicting dstack version 
        
           "$root/$version/bin/pip" uninstall -y dstack-gateway dstack 
        
           "$root/$version/bin/pip" cache remove dstack 
        
           "$root/$version/bin/pip" install "$gateway_package" 
        
           sudo "$root/$version/bin/python" -m dstack._internal.proxy.gateway.systemd install 
        
           echo "$version" > "$root/version" 
        
           sudo systemctl daemon-reload 
        
           sudo systemctl restart dstack.gateway 
        
           echo "Update successfully completed" 
        
           """

Ubuntu 22.04 has no suitable newer Python in its default repos, so dropping 3.10 would break updates for all existing gateways.

Solution

Make the gateway Python version a server-side choice, and move existing gateways to it through the regular blue/green update. Since the update script ships with the server, the same release can migrate gateways and drop 3.10.

Install the gateway's Python with uv (uv python install, a prebuilt standalone CPython) instead of using the OS python3, so the server controls the gateway's Python version independently of the image. Target Python 3.12 initially.
Install uv to a fixed location under /home/ubuntu/dstack and always call it by that path, since ~/.local/bin is usually not on PATH in non-interactive SSH sessions.
Update script:
- Skip the update only if the active venv already has the target build and a Python at or above the supported minimum. Read both versions with a single bin/python -c call. Never skip master or URL builds, and don't skip if the check fails.
- Bootstrap uv if missing and install the target Python.
- Recreate the inactive venv from scratch with uv venv --managed-python --seed, then install dstack with uv pip install (--refresh-package dstack for master or URL builds). --managed-python keeps every gateway on the same interpreter build even if the image ships a matching system Python.
- Keep the systemd install, version flip and restart unchanged.
Provisioning (cloud-init and the Kubernetes pod): drop python3.10-venv and create the venvs and install dstack the same way.
Drop Python 3.10 from requires-python, classifiers and CI.

Properties:

Downtime is unchanged: the new venv is built while the old app serves, and the switch is the usual short systemctl restart.
Any failure before the flip leaves the old app running, and the server retries on the next start.
Downgrading the server keeps working: older servers' pip-based update script installs into the inactive venv, which is either the leftover 3.10 venv or a 3.12 venv with pip seeded.
Installs get faster with uv, and later Python bumps become a one-line change.

Gateways that the server doesn't update (DSTACK_SKIP_GATEWAY_UPDATE, or unreachable at server start) stay on 3.10 until a later update reaches them.

Workaround

None. Recreating gateways also gets Python 3.10.

Install gateway app with uv-managed Python 3.12 (#4439)

We drop Python 3.10 support for running dstack after #4439 released and gateways are updated. This could be released together but let's migrate gateways first just in case.

Navigation Menu

Repository navigation

[Feature]: Support Python version upgrade for gateways and migrate from Python 3.10 #4423

Problem

Solution

Workaround

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	def get_gateway_user_data(authorized_key: str) -> str:
	return get_cloud_config(
	package_update=True,
	packages=[
	"nginx",
	"python3.10-venv",
	],
	snap={"commands": [["install", "--classic", "certbot"]]},
	runcmd=[
	["ln", "-s", "/snap/bin/certbot", "/usr/bin/certbot"],
	["su", "ubuntu", "-c", " && ".join(get_dstack_gateway_commands())],
	],
	ssh_authorized_keys=[authorized_key],
	)

	def get_dstack_gateway_commands() -> List[str]:
	gateway_package, _ = get_dstack_gateway_package_and_target_version()
	return [
	"mkdir -p /home/ubuntu/dstack",
	"python3 -m venv /home/ubuntu/dstack/blue",
	"python3 -m venv /home/ubuntu/dstack/green",
	f"/home/ubuntu/dstack/blue/bin/pip install '{gateway_package}'",
	"sudo /home/ubuntu/dstack/blue/bin/python -m dstack._internal.proxy.gateway.systemd install --run",
	]

	# Blue/green: install the new build into the currently inactive venv and flip to it
	_GATEWAY_UPDATE_SCRIPT = """\
	set -e
	gateway_package="$1"
	build="$2"
	root=/home/ubuntu/dstack
	if [ -f "$root/version" ]; then
	version=$(cat "$root/version")
	else
	version=blue
	fi
	if [ -n "$build" ]; then
	current_build=$("$root/$version/bin/pip" show dstack \| grep Version \| awk '{print $2}')
	if [ "$current_build" = "$build" ]; then
	echo "The build $build is already installed. Skipping..."
	exit 0
	fi
	fi
	if [ "$version" = blue ]; then
	version=green
	else
	version=blue
	fi
	# dstack-gateway is a pre-0.21.4 package that may still be installed and require a conflicting dstack version
	"$root/$version/bin/pip" uninstall -y dstack-gateway dstack
	"$root/$version/bin/pip" cache remove dstack
	"$root/$version/bin/pip" install "$gateway_package"
	sudo "$root/$version/bin/python" -m dstack._internal.proxy.gateway.systemd install
	echo "$version" > "$root/version"
	sudo systemctl daemon-reload
	sudo systemctl restart dstack.gateway
	echo "Update successfully completed"
	"""

[Feature]: Support Python version upgrade for gateways and migrate from Python 3.10 #4423

Description

Problem

Solution

Workaround

Activity

r4victor commented on Oct 30, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions