feat: Automatically load notebook extras on import with opt-out controls

[dborowitz]

Automatically initialize interactive notebook extras when importing google.cloud.managed_spark_connect inside an IPython kernel:

• Inject colabsqlviz's explore_dataframe() into IPython user_ns
• Load the %dpip line magic extension (google.cloud.managed_spark_magics)
• Load the %%sparksql cell magic extension (sparksql_magic)
• Add google-colabsqlviz>=0.3.0 and sparksql-magic>=0.0.3 to dependencies

Add opt-out and runtime configuration controls:

• Environment variable: MANAGED_SPARK_CONNECT_ENABLE_EXTRAS=false
• IPython traitlet: ManagedSparkConnect.enable_extras = False (supports both persistent file-based config and runtime toggling via %config, tracking and only undoing changes that managed_spark_connect itself performed).

Dependency & Footprint Justification

The new hard dependencies added by this PR (google-colabsqlviz, sparksql-magic) add just a few MiB of wheel downloads. In more detail:

1. Zero Version Conflicts or Package Mutations:
   All core data dependencies (pyspark, pandas>=2.0.0, pyarrow>=10.0.1, protobuf>=4.24.0, packaging>=20.0) are already satisfied by pyspark[connect] and google-api-core. Installing both libraries causes 0 upgrades or downgrades to existing packages.
2. sparksql-magic Has Zero Extra Transitive Cost:
   sparksql-magic (4.2 KiB wheel, 6.9 KiB extracted, 14.4 KiB with .pyc) only depends on pyspark and ipython—a strict subset of google-colabsqlviz and ipykernel. Adding it alongside google-colabsqlviz adds 0 additional transitive dependencies.
3. Negligible Footprint Relative to the ~840–930 MiB Base:
   In local notebook kernels (e.g., VS Code or Jupyter, which require ipykernel), the entire ipython stack is already present. Adding both libraries requires downloading just 3.59 MiB of wheels (~1.7% of the existing install size dominated by pyspark @ ~435 MiB wheel and pyarrow @ ~130 MiB):

Environment Scenario	Base Installed Size	Added Wheel Download	Added Extracted (no .pyc, e.g. uv)	Added Extracted + .pyc (pip)	Relative Increase (pip)	New Packages Added (Both Libraries)
Local Kernel + Widgets (ipykernel + ipywidgets)	~945 MiB	1.08 MiB	3.88 MiB	5.36 MiB	+0.6%	4 (google-colabsqlviz, sparksql-magic, anywidget, psygnal)
Minimal VS Code Kernel (ipykernel only)	~933 MiB	3.59 MiB	13.79 MiB	15.75 MiB	+1.7%	7 (above 4 + ipywidgets, jupyterlab-widgets, widgetsnbextension)
Bare Headless venv (no kernel installed)	~840 MiB	11.50 MiB	38.85 MiB	53.50 MiB	+6.4%	24 (above 7 + ipython stack)

(Note: Pure Python code across all 7 packages added in the minimal VS Code kernel scenario is < 1 MiB. The remaining extracted space is static frontend JS bundles/source maps in widgetsnbextension and anywidget [~11.5 MiB], psygnal's compiled mypyc .so binary [~1.3 MiB], and pip's .pyc bytecode cache duplicating embedded JS strings in colabsqlviz [~1.0 MiB].)

[gemini-code-assist]

Code Review

This pull request introduces Jupyter Notebook Extras to automatically load interactive conveniences like explore_dataframe(), %dpip, and %%sparksql when importing the package inside an IPython kernel. It includes configuration options to opt out of these extras, updates dependencies, and adds comprehensive tests. The review feedback highlights a potential memory leak in _ipython.py due to strong references to the IPython shell in _SHELL_STATES, recommending the use of weakref.WeakKeyDictionary. Additionally, a minor grammatical redundancy was pointed out in the README.md documentation.

[medb]

I feel like we may want to make all of these optional - we have customers that do not use notebooks w/ Spark Connect, for them these dependencies are not necessary.

[medb]

Did we decide to remove this dependency and rely on manually installed one, so it can work w/ pyspark-client as well?

[medb]

Did we measure latency of this call?