Feat: Add caching/cached variants of pip and conda installers

[cjac]

This PR introduces cached variants of the standard Python package installation scripts: pip-install-cached.sh and conda-install-cached.sh. These scripts are designed to significantly reduce cluster initialization time and improve reliability in constrained environments (e.g., behind proxies or with limited egress bandwidth).

Resolves #1408

Problem Solved

Traditional package installation on every node startup can become a bottleneck due to:

• Proxy Throttling: Strict proxies rate-limiting external downloads (PyPI/Conda).
• Large Payloads: Repeatedly downloading large ML libraries (e.g., TensorFlow, PyTorch).
• Startup Timeouts: Egress bottlenecks causing cluster creation to exceed timeout limits.

Solution Approach: "Check-Cache-or-Build"

The new scripts leverage Google Cloud Storage (GCS) to cache pre-packaged Python environments.

1. Cache Key Identification: A unique cache key is generated based on requested packages, OS, Dataproc version, and Architecture.
2. Cache Hit: If the environment archive exists in GCS, it is downloaded and unpacked directly to the target environment path, bypassing external downloads.
3. Cache Miss:
   • The script locks the cache key using a .building sentinel file in GCS to prevent concurrent builds from other nodes.
   • Installs packages standardly.
   • Packages the environment and uploads it to GCS for future reuse.
4. Locking/Waiting: If another node is building, subsequent nodes will wait for the cache file to appear rather than duplicating the build effort.

Key Features & Customizations

• Transparent Fallback: Falls back to standard installation if GCS is unreachable or errors occur during caching.
• Default Temp Bucket Usage: Automatically defaults to using the cluster's dataproc-temp-bucket if no specific CACHE_BUCKET is provided.
• Customization via Metadata:
  • CACHE_BUCKET: Override default storage bucket.
  • CACHE_KEY_OVERRIDE: Force a manual cache key name.
  • CACHE_TIMEOUT: Set cache validity period (Defaults to effectively infinite/10 years).
  • TARGET_ENV_PATH: Explicitly define destination environment directory.

Testing & Usage

Documentation has been added to python/README.md covering options and examples.

Example Usage:

gcloud dataproc clusters create my-cluster \
    --metadata 'PIP_PACKAGES=pandas scipy' \
    --initialization-actions gs://<my-bucket>/python/pip-install-cached.sh

[cjac]

/gcbrun

[gemini-code-assist]

Code Review

This pull request introduces two new initialization scripts, conda-install-cached.sh and pip-install-cached.sh, to support caching of Conda and PyPI package environments in GCS, along with corresponding documentation in README.md. The review feedback identifies several critical and high-severity issues, including the risk of caching system-wide directories (like /usr) when target environment paths are inferred, potential script crashes under set -e during date parsing or cache unpacking failures, and pathname expansion (globbing) bugs when passing unquoted package lists. The reviewer suggests adding safety checks to prevent system directory caching, wrapping unpacking commands in conditional blocks for graceful fallbacks, using Bash arrays to handle package lists safely, and removing unused helper functions.

[cjac]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces two new initialization scripts, conda-install-cached.sh and pip-install-cached.sh, along with documentation, to support caching Conda and PyPI environments in GCS for faster Dataproc cluster creation. Feedback on these changes highlights a race condition in concurrent lock acquisition for both scripts, which can be resolved by using gsutil with the x-goog-if-generation-match:0 header. Additionally, pip-install-cached.sh should be updated to install python3-pip instead of the deprecated python-pip, and its fallback behavior when TARGET_ENV_PATH is omitted needs to be addressed, as it currently fails on system-wide environments.

[cjac]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces two new initialization action scripts, conda-install-cached.sh and pip-install-cached.sh, along with updated documentation in README.md to support cached installation of Conda and PyPI packages on Dataproc clusters using Google Cloud Storage. The code review identified several critical and high-severity issues: first, copying the entire isolated virtual environment back to /usr in the pip script can corrupt the system Python interpreter via circular symlinks, which should be resolved by only copying the site-packages directory; second, the lock-waiting mechanism in both scripts can lead to infinite loops or indefinite hangs if a build sentinel file becomes stale; and finally, the pip script is missing the remove_old_backports function required to prevent package update failures on older Debian-based Dataproc images.

[cjac]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces two new initialization scripts, conda-install-cached.sh and pip-install-cached.sh, along with updated documentation in README.md to support installing Conda and PyPI packages with GCS-based environment caching. Feedback on pip-install-cached.sh highlights three key improvements: adding timeouts to curl to prevent hangs in air-gapped environments, copying executable binaries from the isolated virtual environment's bin/ directory to /usr/local/bin/ during merging, and verifying that DATAPROC_IMAGE_VERSION is non-empty before running bc to avoid script crashes.

[cjac]

/gcbrun

[cjac]

there are no gcbrun tests for the python/ directory so we'll skip dependence on the running of those tests.

[cjac]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces two new initialization-action scripts, pip-install-cached.sh and conda-install-cached.sh, along with updated documentation in README.md to support installing and caching Python and Conda environments in GCS. The review feedback identifies several critical and high-severity issues in the scripts: the virtualenv at ${isolated_env_path} is never initialized, failures in merge_isolated_env are silently ignored, copied scripts retain shebangs pointing to the isolated virtualenv instead of the system python, and both scripts lack a fallback for OS detection when lsb_release is missing on minimal images.

[gemini-code-assist]

The virtualenv at ${isolated_env_path} is never actually created. Although ensure_venv is called to install the system packages for venv, the virtualenv directory itself is never initialized. This will cause the script to fail on any cache miss when attempting to run pip from the non-existent virtualenv path. We should explicitly create the virtualenv after ensuring venv is available.

Suggested change

	ensure_venv
	original_env_path="${env_path}"
	env_path="${isolated_env_path}"
	ensure_venv
	python3 -m venv "${isolated_env_path}" || virtualenv "${isolated_env_path}"
	original_env_path="${env_path}"
	env_path="${isolated_env_path}"

[gemini-code-assist]

If merge_isolated_env fails during unpack_and_merge, the failure is silently ignored and the function returns 0 (success). This happens because merge_isolated_env is called inside an if statement, which disables set -e behavior for the entire function call, and the function then unconditionally returns 0. We should explicitly propagate the failure by returning 1 if merge_isolated_env fails.

Suggested change

	if gcs_cat "${tarball}" | tar -C "${target_path}" -xz; then
	if [[ "${is_isolated}" == "true" ]]; then
	merge_isolated_env "${target_path}"
	fi
	return 0
	fi
	if gcs_cat "${tarball}" | tar -C "${target_path}" -xz; then
	if [[ "${is_isolated}" == "true" ]]; then
	merge_isolated_env "${target_path}" || return 1
	fi
	return 0
	fi

[gemini-code-assist]

When copying executable scripts from the isolated virtualenv to /usr/local/bin/, their shebangs will still point to the virtualenv's python interpreter (e.g., #!/opt/pip-isolated-env/bin/python3). While this works as long as the virtualenv exists, it is cleaner and more robust to rewrite these shebangs to use the system python (/usr/bin/env python3), making the scripts independent of the virtualenv path.

Suggested change

	if [[ -d "${venv_path}/bin" ]]; then
	find "${venv_path}/bin" -type f ! -name "python*" ! -name "pip*" ! -name "easy_install*" ! -name "activate*" -exec cp -a {} /usr/local/bin/ \;
	fi
	if [[ -d "${venv_path}/bin" ]]; then
	find "${venv_path}/bin" -type f ! -name "python*" ! -name "pip*" ! -name "easy_install*" ! -name "activate*" -exec cp -a {} /usr/local/bin/ \\;
	# Rewrite shebangs of copied scripts to use system python
	for f in $(find "${venv_path}/bin" -type f ! -name "python*" ! -name "pip*" ! -name "easy_install*" ! -name "activate*" -exec basename {} \\;); do
	if head -n 1 "/usr/local/bin/${f}" | grep -q "python"; then
	sed -i "1s|^#!.*python.*|#!/usr/bin/env python3|" "/usr/local/bin/${f}"
	fi
	done
	fi

[gemini-code-assist]

On some minimal images (such as Rocky Linux), lsb_release is not installed by default. If lsb_release is missing, the command substitution will fail and cause the script to exit immediately due to set -e. We should add a fallback to /etc/os-release to make the OS detection more robust.

Suggested change

	readonly OS_NAME=$(lsb_release -is | tr '[:upper:]' '[:lower:]')
	readonly OS_NAME=$(if command -v lsb_release >/dev/null 2>&1; then lsb_release -is | tr '[:upper:]' '[:lower:]'; else . /etc/os-release; echo "${ID}" | tr '[:upper:]' '[:lower:]'; fi)

[gemini-code-assist]

On some minimal images (such as Rocky Linux), lsb_release is not installed by default. If lsb_release is missing, the command substitution will fail and cause the script to exit immediately due to set -e. We should add a fallback to /etc/os-release to make the OS detection more robust.

Suggested change

	readonly OS_NAME=$(lsb_release -is | tr '[:upper:]' '[:lower:]')
	readonly OS_NAME=$(if command -v lsb_release >/dev/null 2>&1; then lsb_release -is | tr '[:upper:]' '[:lower:]'; else . /etc/os-release; echo "${ID}" | tr '[:upper:]' '[:lower:]'; fi)

[cjac]

/gcbrun

[cjac]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces initialization actions and integration tests for installing PyPI and Conda packages with caching support in Google Cloud Storage (GCS) to speed up Dataproc cluster creations. Feedback on the changes highlights three main areas of improvement: ensuring the target environment directory is created before unpacking the Conda cache, handling potential network failures gracefully when querying Debian repositories in private clusters, and verifying that the conda binary is available in the PATH before attempting to configure channels.

[cjac]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces caching mechanisms for installing PyPI and Conda packages on Dataproc clusters using GCS, including the scripts pip-install-cached.sh and conda-install-cached.sh, documentation, and integration tests. Feedback suggests several improvements: using --system-site-packages in the isolated virtual environment to avoid redundant downloads, warning users against caching the base /opt/conda directory, sanitizing DATAPROC_IMAGE_VERSION to prevent bc parsing errors, and using run_command instead of assert_command in test tearDown methods to avoid masking setup failures.