Install gateway app with uv-managed Python 3.12

.github/workflows/build-artifacts.yml

@@ -85,19 +85,9 @@ jobs:
         with:
           name: frontend-build
           path: src/dstack/_internal/server/statics
-      - name: Run pytest on POSIX
-        if: matrix.os != 'windows-latest'
-        # Skip Postgres tests on macos since macos runner doesn't have Docker.
-        run: |
-          RUNPOSTGRES=""
-          if [ "${{ matrix.os }}" != "macos-latest" ]; then
-            RUNPOSTGRES="--runpostgres"
-          fi
-          uv run pytest -n auto src/tests --runui $RUNPOSTGRES
-      - name: Run pytest on Windows
-        if: matrix.os == 'windows-latest'
-        run: |
-          uv run pytest -n auto src/tests --runui --runpostgres
+      - name: Run pytest
+        # Skip Postgres and Docker tests on macos since macos runner doesn't have Docker.
+        run: uv run pytest -n auto src/tests ${{ matrix.os == 'macos-latest' && '--runui' || '--runall' }}
 
   runner-test:
     defaults:

AGENTS.md

@@ -36,7 +36,7 @@ Before touching a subsystem, read the relevant notes in `contributing/`: `ARCHIT
 - Preserve client/server backward compatibility when updating Pydantic models. New fields must be excluded from client requests when unset, for compatibility with older servers (see `core/compatibility/*.py`). If the allowed domain/type of an existing field is extended, server responses may need to be patched for older clients (see `server/compatibility/*.py`). No need to exclude new fields from server responses, since clients rely on `validate_extra_ignore`.
 
 ## Testing Guidelines
-- Default to `uv run pytest`. Use markers from `src/tests/conftest.py` like `--runpostgres` if need to include specific tests.
+- Default to `uv run pytest` for quick testing. Opt-in tests are skipped by default; enable them when the change touches what they cover, with the flags from `src/tests/conftest.py`: `--runpostgres`, `--rundocker`, `--runui`, or `--runall` for all of them.
 - Scope the run to the change: for trivial or localized edits, run only the affected test modules, `Test*` classes, or `-k` selection instead of the whole suite. Reserve the full suite for broad or cross-cutting changes.
 - Speed up large runs with `-n auto` (pytest-xdist), e.g. `uv run pytest -n auto`.
 - Group tests for the same unit (function/class) using `Test*` classes that mirror unit's name.

CONTRIBUTING.md

@@ -55,11 +55,15 @@ Then you can do:
 uv run pytest src/tests
 ```
 
-(Optionally) By default, tests run against SQLite.
-Use the `--runpostgres` flag to run the tests against Postgres as well:
+(Optionally) Some tests are skipped by default because they need extra setup. Enable them with these flags:
+
+* `--runpostgres` – run the tests against Postgres in addition to SQLite. Requires Docker.
+* `--rundocker` – run the tests that use Docker containers. Requires Docker.
+* `--runui` – run the tests of the bundled frontend. Requires the frontend build in `src/dstack/_internal/server/statics`.
+* `--runall` – all of the above.
 
 ```shell
-uv run pytest src/tests --runpostgres
+uv run pytest src/tests --runall
 ```
 
 Alternatively, you can run tests via [tox](https://tox.wiki/) inside an isolated environment ensuring that the `dstack` package itself is built correctly and all requirements are specified and correct. tox and [just](https://just.systems/) must be already installed.

contributing/PROXY.md

@@ -136,19 +136,21 @@ Users can create a gateway using the `dstack apply` command. The gateway YAML co
 Provisioning happens as follows:
 1. Launch a non-GPU instance (usually the smallest) with all ports exposed.
 2. Install Nginx, Certbot, and patch configs.
-3. Create blue-green virtual environments.
-4. Install the `dstack[gateway]` Python package.
-5. Run the systemd service `dstack.gateway.service`.
+3. Run the gateway install script (`core/backends/base/resources/gateway_install.sh`), which installs uv, Python, and the `dstack[gateway]` package and runs the systemd service `dstack.gateway.service`. The same script is used for updates, see below.
 
 ### Update
 
-The gateway has a "blue-green deployment"-like configuration: there are two virtual environments to be swapped on update. The systemd service uses the newly installed package after a restart.
+The gateway has a "blue-green deployment"-like configuration: the app runs from one of two virtual environments, `blue` or `green`, as recorded in `~/dstack/version`. The systemd service uses the newly installed package after a restart.
 
-The update process looks like this:
-1. Install the new package to the unused venv.
-2. Update scripts and systemd service config.
-3. Swap the active venv name in `~/dstack/version`.
-4. Restart the systemd service.
+On startup, the `dstack` server runs the install script on every gateway over SSH. The script:
+1. Exits if the active venv already has the target `dstack` version on the script's `python_version`. Unversioned packages (master builds, `DSTACK_GATEWAY_PACKAGE_URL`) are always reinstalled.
+2. Installs uv (`uv_version`) and Python (`python_version`) if missing. Both are pinned in the script.
+3. Recreates the unused venv and installs the new package into it.
+4. Updates scripts and systemd service config.
+5. Swaps the active venv name in `~/dstack/version`.
+6. Restarts the systemd service.
+
+Changing `python_version` migrates existing gateways to the new Python on their next update. The venvs are created with pip (`uv venv --seed`) because update scripts of older servers call pip directly, e.g. after a server downgrade.
 
 ## Gateway development
 

src/dstack/_internal/core/backends/base/compute.py

@@ -1,3 +1,4 @@
+import importlib.resources
 import os
 import random
 import re
@@ -1275,12 +1276,12 @@ def get_gateway_user_data(authorized_key: str) -> str:
         package_update=True,
         packages=[
             "nginx",
-            "python3.10-venv",
+            "wget",
         ],
         snap={"commands": [["install", "--classic", "certbot"]]},
         runcmd=[
             ["ln", "-s", "/snap/bin/certbot", "/usr/bin/certbot"],
-            ["su", "ubuntu", "-c", " && ".join(get_dstack_gateway_commands())],
+            ["su", "ubuntu", "-c", get_dstack_gateway_install_command()],
         ],
         ssh_authorized_keys=[authorized_key],
     )
@@ -1389,15 +1390,30 @@ def get_dstack_gateway_package_and_target_version() -> tuple[str, str | None]:
     return package, None
 
 
-def get_dstack_gateway_commands() -> List[str]:
-    gateway_package, _ = get_dstack_gateway_package_and_target_version()
-    return [
-        "mkdir -p /home/ubuntu/dstack",
-        "python3 -m venv /home/ubuntu/dstack/blue",
-        "python3 -m venv /home/ubuntu/dstack/green",
-        f"/home/ubuntu/dstack/blue/bin/pip install '{gateway_package}'",
-        "sudo /home/ubuntu/dstack/blue/bin/python -m dstack._internal.proxy.gateway.systemd install --run",
-    ]
+def get_dstack_gateway_install_command() -> str:
+    """
+    Returns a shell command that installs the gateway app on a fresh gateway host or updates it
+    on an existing one. Must be run as the `ubuntu` user.
+    """
+    gateway_package, target_version = get_dstack_gateway_package_and_target_version()
+    return shlex.join(
+        [
+            "/bin/sh",
+            "-c",
+            _get_dstack_gateway_install_script(),
+            "sh",  # $0
+            gateway_package,
+            target_version or "",
+        ]
+    )
+
+
+def _get_dstack_gateway_install_script() -> str:
+    return (
+        importlib.resources.files("dstack._internal.core.backends.base")
+        .joinpath("resources/gateway_install.sh")
+        .read_text()
+    )
 
 
 def merge_tags(

src/dstack/_internal/core/backends/base/resources/gateway_install.sh

@@ -0,0 +1,71 @@
+# Installs the dstack gateway app on a fresh gateway host or updates it on an existing one.
+# Usage: sh -c "$script" sh <gateway_package> <target_build>
+# <target_build> is empty for unversioned packages (master builds, custom URLs).
+#
+# Blue/green: the app runs from one of two venvs, blue or green, named in the version file.
+# The inactive venv is recreated from scratch, the dstack package is installed into it,
+# and the gateway service is restarted from it.
+#
+# Python and the venvs are managed by uv, which is installed by the script,
+# so the gateway doesn't depend on the OS Python. Changing python_version migrates existing
+# gateways to the new Python on their next update.
+#
+# On a fresh host there is no version file and no venvs. Blue is considered active but fails
+# the version check, and the script installs uv, Python, and the package into green, creates
+# an empty blue venv, writes the systemd service, and starts it.
+set -e
+gateway_package="$1"
+build="$2"
+gateway_dir=/home/ubuntu/dstack
+uv_version=0.13.1
+python_version=3.12
+uv="$gateway_dir/bin/uv"
+# To prevent the uv cache grow and avoid pruning the cache, re-download all deps, which takes a few seconds.
+export UV_NO_CACHE=1
+
+mkdir -p "$gateway_dir"
+cd "$gateway_dir"
+if [ -f "$gateway_dir/version" ]; then
+  active=$(cat "$gateway_dir/version")
+else
+  active=blue
+fi
+if [ -n "$build" ]; then
+  current=$("$gateway_dir/$active/bin/python" -c 'import sys, importlib.metadata as m; print("%d.%d %s" % (sys.version_info[0], sys.version_info[1], m.version("dstack")))' 2>/dev/null || true)
+  if [ "$current" = "$python_version $build" ]; then
+    echo "The build $build on Python $python_version is already installed. Skipping..."
+    exit 0
+  fi
+fi
+if [ "$active" = blue ]; then
+  version=green
+else
+  version=blue
+fi
+# The static binary from the uv release needs no Python to bootstrap and runs on any Linux.
+# GitHub is required anyway, as uv downloads Python builds from there.
+if [ "$("$uv" --version 2>/dev/null | cut -d ' ' -f 2)" != "$uv_version" ]; then
+  archive=$(mktemp)
+  wget -qO "$archive" "https://github.com/astral-sh/uv/releases/download/$uv_version/uv-$(uname -m)-unknown-linux-musl.tar.gz"
+  mkdir -p "$gateway_dir/bin"
+  tar -xzf "$archive" --strip-components=1 -C "$gateway_dir/bin"
+  rm -f "$archive"
+fi
+"$uv" python install "$python_version"
+# The venv may contain root-owned files, e.g. bytecode written when installing the systemd
+# service from it with sudo
+sudo rm -rf "${gateway_dir:?}/${version:?}"
+# --seed installs pip. If the dstack server is downgraded, the older server runs its own update
+# script, which installs into the inactive venv with its pip, so both venvs must exist.
+"$uv" venv --managed-python --seed --python "$python_version" "$gateway_dir/$version"
+if [ ! -d "$gateway_dir/$active" ]; then
+  "$uv" venv --managed-python --seed --python "$python_version" "$gateway_dir/$active"
+fi
+# Compile bytecode now, as otherwise the app compiles it on the first start, prolonging
+# the downtime during the restart
+"$uv" pip install --compile-bytecode --python "$gateway_dir/$version/bin/python" "$gateway_package"
+sudo "$gateway_dir/$version/bin/python" -m dstack._internal.proxy.gateway.systemd install
+echo "$version" > "$gateway_dir/version"
+sudo systemctl daemon-reload
+sudo systemctl restart dstack.gateway
+echo "Update successfully completed"

src/dstack/_internal/core/backends/kubernetes/compute.py

@@ -32,7 +32,7 @@
     generate_unique_name,
     generate_unique_volume_name,
     get_docker_commands,
-    get_dstack_gateway_commands,
+    get_dstack_gateway_install_command,
     merge_tags,
 )
 from dstack._internal.core.backends.base.offers import (
@@ -1423,8 +1423,6 @@ def _wait_for_load_balancer_address(
 
 def _get_gateway_commands(authorized_keys: List[str]) -> List[str]:
     authorized_keys_content = "\n".join(authorized_keys).strip()
-    gateway_commands = " && ".join(get_dstack_gateway_commands())
-    quoted_gateway_commands = shlex.quote(gateway_commands)
 
     commands = [
         # install packages
@@ -1451,7 +1449,7 @@ def _get_gateway_commands(authorized_keys: List[str]) -> List[str]:
         "rm -rf /etc/ssh/ssh_host_*",
         "ssh-keygen -A > /dev/null",
         # install gateway
-        f"su ubuntu -c {quoted_gateway_commands}",
+        f"su ubuntu -c {shlex.quote(get_dstack_gateway_install_command())}",
         # start docker-systemctl-replacement as an init replacement (PID 1), which
         # - starts and supervises enabled services (sshd, nginx, dstack.gateway)
         # - stops running services on SIGTERM (graceful shutdown)

src/dstack/_internal/server/services/gateways/__init__.py

@@ -1,7 +1,6 @@
 import asyncio
 import datetime
 import itertools
-import shlex
 import uuid
 from collections.abc import AsyncGenerator
 from contextlib import asynccontextmanager
@@ -16,7 +15,7 @@
 
 import dstack._internal.utils.random_names as random_names
 from dstack._internal.core.backends.base.compute import (
-    get_dstack_gateway_package_and_target_version,
+    get_dstack_gateway_install_command,
 )
 from dstack._internal.core.backends.features import (
     BACKENDS_WITH_GATEWAY_SUPPORT,
@@ -776,11 +775,11 @@ async def init_gateways(session: AsyncSession):
                 "Skipping gateway replicas update due to DSTACK_SKIP_GATEWAY_UPDATE env variable"
             )
         else:
-            gateway_package, target_version = get_dstack_gateway_package_and_target_version()
+            install_command = get_dstack_gateway_install_command()
 
             for gateway_replica, res in await gather_map_async(
                 gateway_replicas,
-                lambda r: _update_gateway_replica(r, gateway_package, target_version),
+                lambda r: _update_gateway_replica(r, install_command),
                 return_exceptions=True,
             ):
                 if isinstance(res, Exception):
@@ -802,10 +801,12 @@ async def init_gateways(session: AsyncSession):
                 )
 
 
+_GATEWAY_UPDATE_TIMEOUT = 600
+
+
 async def _update_gateway_replica(
     gateway_replica_model: GatewayReplicaModel,
-    gateway_package: str,
-    target_version: str | None,
+    install_command: str,
 ) -> bool:
     if gateway_replica_model.ip_address is None:
         logger.warning(
@@ -824,58 +825,13 @@ async def _update_gateway_replica(
     )
     logger.debug("Updating gateway replica %s", connection.ip_address)
 
-    command = (
-        "/bin/sh -c "
-        + shlex.quote(_GATEWAY_UPDATE_SCRIPT)
-        + " sh "  # $0 placeholder
-        + shlex.quote(gateway_package)
-        + " "
-        + shlex.quote(target_version or "")
-    )
-    stdout = await connection.tunnel.aexec(command, timeout=_GATEWAY_UPDATE_TIMEOUT)
+    stdout = await connection.tunnel.aexec(install_command, timeout=_GATEWAY_UPDATE_TIMEOUT)
     if "Update successfully completed" in stdout:
         logger.info("Gateway replica %s updated", connection.ip_address)
         return True
     return False
 
 
-_GATEWAY_UPDATE_TIMEOUT = 600
-
-# Blue/green: install the new build into the currently inactive venv and flip to it
-_GATEWAY_UPDATE_SCRIPT = """\
-set -e
-gateway_package="$1"
-build="$2"
-root=/home/ubuntu/dstack
-if [ -f "$root/version" ]; then
-  version=$(cat "$root/version")
-else
-  version=blue
-fi
-if [ -n "$build" ]; then
-  current_build=$("$root/$version/bin/pip" show dstack | grep Version | awk '{print $2}')
-  if [ "$current_build" = "$build" ]; then
-    echo "The build $build is already installed. Skipping..."
-    exit 0
-  fi
-fi
-if [ "$version" = blue ]; then
-  version=green
-else
-  version=blue
-fi
-# dstack-gateway is a pre-0.21.4 package that may still be installed and require a conflicting dstack version
-"$root/$version/bin/pip" uninstall -y dstack-gateway dstack
-"$root/$version/bin/pip" cache remove dstack
-"$root/$version/bin/pip" install "$gateway_package"
-sudo "$root/$version/bin/python" -m dstack._internal.proxy.gateway.systemd install
-echo "$version" > "$root/version"
-sudo systemctl daemon-reload
-sudo systemctl restart dstack.gateway
-echo "Update successfully completed"
-"""
-
-
 def _recently_updated(gateway_replica_model: GatewayReplicaModel) -> bool:
     return gateway_replica_model.app_updated_at.replace(
         tzinfo=datetime.timezone.utc

src/tests/_internal/core/backends/base/conftest.py

@@ -0,0 +1,4 @@
+from tests._internal.core.backends.base.gateway_host import (  # noqa: F401
+    gateway_host,
+    gateway_host_container,
+)