[SPARK-59678][SQL] Fix pivot() failing on an array column with non-nullable elements

[jiwen624]

What changes were proposed in this pull request?

PivotTransformer checks and casts each pivot value against pivotColumn.dataType. This PR checks and casts against pivotColumn.dataType.asNullable instead, so a pivot value is matched to the pivot column by type, ignoring nullability.

Why are the changes needed?

pivot() on an array column fails at analysis when the array's element type is non-nullable, with an error that prints the same type on both sides:

scala> Seq(1.0d).toDF("v").selectExpr("v", "array(v) AS a").groupBy("v").pivot("a").count()
org.apache.spark.sql.AnalysisException: [PIVOT_VALUE_DATA_TYPE_MISMATCH] Invalid pivot value '[1.0]':
  value data type array<double> does not match pivot column data type array<double>

The pivot values are collected by Spark and turned into literals by Literal.apply, which builds ArrayType(elementType), i.e. containsNull = true. Cast.canCast then rejects the cast to the column's containsNull = false type. Nullability is irrelevant here — the value is a constant that is only compared against the column — and there is no workaround, since the user does not supply the literals.

Does this PR introduce any user-facing change?

Yes. Queries that failed with PIVOT_VALUE_DATA_TYPE_MISMATCH purely because of a nullability difference now succeed. No previously working query changes its result: the cast is only loosened on nullability flags, never on base types, so genuinely mismatched values are still rejected.

How was this patch tested?

Added UT cases.

Was this patch authored or co-authored using generative AI tooling?

Yes