[ZEPPELIN-6701] Support Spark 4.1 and 4.2

[pan3793]

What is this PR for?

Adds official Spark 4.1 and 4.2 support.

• Adds spark-4.1 and spark-4.2 build profiles, using Spark 4.1.3 and 4.2.0.
• Extends the supported Spark version range to 3.3-4.2; 4.3 is the unsupported-future boundary.
• Restructures the spark-test job as a matrix-driven job covering Spark, Scala, Java, and Python combinations.
• Runs Spark 4.1 and 4.2 tests with JDK 17 and Python 3.14.
• Updates the Python 3.14 test environment using constraints from Spark branch metadata.
• Updates the Spark build documentation.

What type of PR is it?

Feature

How should this be tested?

CI validates all supported combinations through the matrix-driven spark-test job. The Python 3.14 runs use the Spark-derived dependency constraints, including pandas>=2.2,<3.0, numpy>=1.21, pyarrow>=18, and grpcio>=1.76,<2.

Locally verified:

./mvnw package -pl spark/interpreter -am -DskipTests -Pspark-4.1 -Pspark-scala-2.13
./mvnw package -pl spark/interpreter -am -DskipTests -Pspark-4.2 -Pspark-scala-2.13
./mvnw test -pl spark/interpreter -am -Dtest=SparkVersionTest -DfailIfNoTests=false -Pspark-4.1 -Pspark-scala-2.13

Also verified the Python 3.14 environment solve and documentation publication build.

Screenshots

N/A

How this resolves?

Provides the Maven profiles, runtime version boundary, CI coverage, and documentation needed to test and build against Spark 4.1 and 4.2.

Questions

• Does the licenses file need to update? No
• Is there breaking changes for older versions? No
• Does this need documentation? Yes