feat!: Move PySpark dependencies into [client] and [spark] extras

[dborowitz]

pyspark is an ~800 MB install that most Spark Connect users do not need, while pyspark-client is a lightweight (~2 MB) pure-Python Spark Connect client. However, pyspark and pyspark-client write to the same pyspark module directory, so installing both in a single environment risks immediate version skew or broken imports if either package is later uninstalled. Because Python package extras are strictly additive, move the Spark dependency out of install_requires into two mutually exclusive extras:

• [client]: depends on pyspark-client~=4.0.0
• [spark]: depends on pyspark[connect]~=4.0.0
• no Spark extra: relies on whichever pyspark provider is already present in the environment

If pyspark is missing or incomplete (such as a leftover pyspark/ directory after uninstalling one of the two packages) when google.cloud.managed_spark_connect is imported, raise PySparkNotInstalledError recommending google-cloud-spark-connect[client,interactive] when inside an IPython shell, or google-cloud-spark-connect[client] otherwise. It subclasses both ManagedSparkConnectException and ImportError, so existing except ImportError guards keep working. As install_requires no longer pins pyspark, also reject a pyspark older than 4.0 on import with UnsupportedPySparkVersionError, a PySparkNotInstalledError subclass, recommending an in-place upgrade via [spark]. %%sparksql gives the same install hint when pyspark is missing.

Update README.md to recommend [client,interactive] as the primary installation command and [spark,interactive] when upgrading an existing dataproc-spark-connect environment in-place. Switch requirements-dev.txt to pyspark-client~=4.0.0, and document how to switch a dev environment to full pyspark. CI runs unit tests against each of the [client] and [spark] extras; the local Spark tests need a JVM, so they only run against [spark].

BREAKING CHANGE: pyspark[connect] is no longer installed by default. Install google-cloud-spark-connect[client] (or [client,interactive] for notebooks), google-cloud-spark-connect[spark], or provide pyspark / pyspark-client in the environment.
Release-As: 2.0.0

[gemini-code-assist]

Code Review

This pull request introduces interactive extras for the google-cloud-spark-connect package, including support for %dpip and %%sparksql magics in Jupyter notebooks, along with improved dependency management and installation options. The reviewer provided several constructive suggestions to improve memory management by using weak references for shell states, simplifying token identification in the SQL parser, and optimizing performance by caching dynamically resolved classes. All review comments were kept as they provide actionable improvements.