[SPARK-59553][CONNECT] Make SparkSession.addArtifact honor spark.jars.ivySettings

[qianlan717]

What changes were proposed in this pull request?

This PR makes SparkSession.addArtifact and SparkSession.addArtifacts honor the existing spark.jars.ivySettings Spark configuration when resolving ivy:// dependencies.

Spark resolves Ivy dependencies through a runtime dependency resolver configured from the driver SparkConf. Spark Connect adds a typed Maven dependency to the ordered artifact batch and resolves it on the server, where the Ivy settings file, repository network, and credentials are available. Support is negotiated through the serverSideMavenArtifacts.v1 capability. A new client falls back to the existing client-side resolution only when the server does not advertise that capability.

The server preserves the order of mixed local and Ivy artifacts and resolves the complete batch
before registering it. Server-side resolution rejects repositories supplied through ?repos=, uses an isolated Ivy cache per Spark session, and includes bounded concurrency, connection/read timeouts, an overall deadline, cancellation, quotas, and content-collision checks.

Why are the changes needed?

SparkSession.addArtifact currently ignores spark.jars.ivySettings. In Spark Connect, Maven dependencies are also resolved by the client, which often cannot read a server-local Ivy settings file or access the server's Maven network and credentials. As a result, an ivy:// dependency can fail through addArtifact even though the same server configuration works for other Maven dependency surfaces.

Does this PR introduce any user-facing change?

Yes. ivy:// dependencies passed to classic or Connect SparkSession.addArtifact(s) now use spark.jars.ivySettings when configured. Spark Connect resolves them on capable servers. New clients connected to older servers retain the previous client-side resolution behavior.

How was this patch tested?

Added tests covering:

• custom Ivy settings, repository policy, resolver timeouts, and cancellation;
• both classic SparkSession artifact APIs and per-session Ivy cache isolation;
• ordered mixed Connect artifacts, capability negotiation, old-server fallback, and propagation of capability RPC failures;
• server-side Maven resolution before artifact registration while preserving order.

The Python protobuf bindings were regenerated with dev/connect-gen-protos.sh. git diff --check, Scala parser checks, protobuf Java generation, and generated-binding smoke checks passed.

build/sbt common-utils/Test/compile could not run because Maven Central DNS resolution is
unavailable in the current development environment (Could not resolve host: repo1.maven.org).

Was this patch authored or co-authored using generative AI tooling?