test: share integration session, use spark.master=local, and align CI extras

[dborowitz]

Summary

Refactors the integration test suite and CI workflows to substantially reduce Dataproc Serverless session churn, VM footprint, and vulnerability to regional GCE stockouts (503 Failed to create main node pool), while expanding CI coverage across packaging extras ([client], [spark], and with/without [interactive]) and switching CI dependency installation to uv.

Metric	Before	After	Change
Collected integration test items	44 (42 active, 2 skipped)	11 (10 active, 1 skipped)	-75%
Remote Dataproc Serverless clusters created	42	5 (1 multi-node + 4 single-node)	-88%
Total GCE VMs provisioned per run	~130 VMs	7 VMs (3 + 4×1)	-94.6%
Leaked clusters per run	4 (ml-pipeline-session-* × 2 + terminated-recreate × 2)	0 (all wrapped in finally + 10m idle TTL)	-100%
End-to-end integration test runtime	~45–60+ min (often failing on stockouts)	~8 min (494s)	~85% faster

Key Changes

P0: Shared Session Fixture, Dead Parameterization Removal, and Leak Prevention

• Shared multi-node session fixture (tests/integration/conftest.py): Provides a session-scoped default multi-node ManagedSparkSession (_shared_connect_session / connect_session) shared across all non-destructive tests (test_create_spark_session_with_default_notebook_behavior, test_reuse_s8s_spark_session, test_sql_functions, test_sql_udf, test_sparksql_magic_with_managed_spark_session, test_dpip_install_success, test_dpip_install_failure), and an isolated_session_state fixture for tests that manage their own session lifecycle.
• Remove dead image_version=[None, "3.0"] fixture: Previously parameterized every test in test_session.py and test_magics.py without ever passing image_version to the session builder, doubling the entire integration suite.
• Safety TTLs & leak cleanup: Sets MANAGED_SPARK_CONNECT_TTL_SECONDS=1800 (30m) and MANAGED_SPARK_CONNECT_IDLE_TTL_SECONDS=600 (10m, the minimum allowed by ServerlessValidator) by default in conftest.py, and folds test_stop_named_session_with_terminate_true + finally: cleanup into test_session_reuse_with_custom_id so named sessions are never leaked for 24 hours.
• Subnetwork default & short-name normalization (tests/integration/conftest.py): Falls back to "default" when MANAGED_SPARK_CONNECT_SUBNET is unset or empty, and qualifies short subnetwork names (e.g. "default", "default-auto") to projects/{test_project}/regions/{test_region}/subnetworks/{test_subnet} while passing already-qualified projects/.../subnetworks/... URIs through unchanged.

P1: Single-Node Mode (spark.master=local) & Test Consolidation

• Use spark.master=local on isolated control-plane/lifecycle tests: Provisions a single 4-vCPU driver VM and 0 executor node pools for test_create_spark_session_with_session_template_and_user_provided_session_config, test_session_reuse_with_custom_id, and test_serverless_session_termination_and_deletion_lifecycle, while keeping the shared connect_session multi-node to test out-of-the-box defaults, remote UDFs, and %dpip artifact distribution.
• Right-size executor config override & fix SessionTemplate default version assertion: Changes .config("spark.executor.cores", "7") ... .config("spark.executor.cores", "16") to "8" $\rightarrow$ "4" with spark.master=local, and asserts session.runtime_config.version == ManagedSparkSession._DEFAULT_RUNTIME_VERSION on the created Session (fixing a precedence bug in the old assert ... == ... if image_version is None else image_version expression, which previously evaluated assert "3.0" because CreateSessionTemplate leaves session_template.runtime_config.version empty until a session is instantiated).
• Consolidate termination & deletion lifecycle tests: Combines test_stop_managed_session_with_terminate_false, test_get_or_create_spark_session_with_terminated_serverless_session, test_stop_spark_session_with_terminated_serverless_session, and test_stop_spark_session_with_deleted_serverless_session into test_serverless_session_termination_and_deletion_lifecycle (2 single-node sessions instead of 4 multi-node sessions).
• Drop local-only unit test duplicates from tests/integration/: Removes pure local CLI/regex validation tests that are already covered in tests/unit/.

P2: Realistic CI Packaging Lanes & uv Dependency Installation

• Switch all GitHub Actions workflows to uv (astral-sh/setup-uv): Replaces pip install with astral-sh/setup-uv + uv pip install --system across tests.yaml, integration-tests.yaml, pyink.yaml, and lint-readme.yaml (with ~/.cache/uv keyed on setup.py and requirements-test.txt), cutting dependency installation to ~2s for [client] and ~20s for [spark].
• Install .[client,interactive] in integration-tests.yaml: Tests against the lightweight pyspark-client + pyspark-connect runtime recommended in the README and ensures %%sparksql is not skipped in integration CI.
• Verify both without and with [interactive] in tests.yaml: Installs .[${{ matrix.spark-extra }}] first to run TestWithoutInteractiveExtra (expanded to test ManagedSparkSession.builder.getOrCreate(), get_tqdm_bar(), _repr_html_(), and stop() when IPython/sqlparse/traitlets/colabsqlviz/ipywidgets are absent), then installs .[${{ matrix.spark-extra }},interactive] for the full unit and local suites.
• Run test_create_local_spark_session in tests/local/test_sparksql_local.py: Moves the DATAPROC_WORKLOAD_TYPE=batch local classic Spark test into tests/local/ so it runs on every PR in the spark lane under Java 17/21, and isolates local Spark Connect proxy/environment state.

[gemini-code-assist]

Code Review

This pull request refactors the integration test suite for ManagedSparkSession, including significant cleanup of fixtures and test logic to improve test isolation and reliability. The changes include centralizing session state management in conftest.py, removing redundant test code, and updating tests to handle session lifecycle and cleanup more robustly. My review identified potential test isolation issues related to PySpark's global session state and unnecessary view creation in tests, which are addressed in the provided comments.

[dborowitz]

/gemini review

[gemini-code-assist]

Code Review

This pull request refactors the integration test suite for ManagedSparkSession, improving test isolation, session lifecycle management, and cleanup logic. It also updates environment variable handling in fixtures and adds robust error handling for session termination. My feedback highlights a potential risk in the os_environment fixture regarding os.environ.clear() and suggests a safer approach to restoring environment variables, as well as a recommendation to wrap the test_serverless_session_termination_and_deletion_lifecycle test in a try...finally block to ensure reliable resource cleanup.