[SPARK-58381][SQL] Gate the Arrow cache zero-copy write path on physical congruence with the cache schema

[viirya]

What changes were proposed in this pull request?

Replace the blocklist that decides zero-copy eligibility on the Arrow cache write path (ColumnarBatchToArrowCachedBatchIterator) with a per-column physical-congruence check against the cache schema, using ArrowUtils.isCompatibleWithDeclaredField (introduced by SPARK-58258 for the columnar Python UDF pass-through). Input vectors whose field trees are not physically congruent with the corresponding cache schema field now take the row-based conversion path, which rewrites the values through ArrowWriter under the cache schema. The containsCacheSchemaMismatch blocklist is removed.

Why are the changes needed?

The zero-copy path serializes the input vectors' buffers verbatim under the cache's own schema: serializeBatch writes only the record batch, and the read path reconstructs the schema from cacheSchema and loads the buffers positionally into it. The blocklist covered only large var-width vectors and interchange-shaped nanosecond timestamp / CalendarInterval vectors, so every other physical divergence was forwarded verbatim and silently reinterpreted under the canonical layout when the cached batch is read back -- the same defect class SPARK-58258 eliminated on the UDF read side, one lifecycle stage earlier. Shapes the blocklist missed include view and dictionary encodings, list offset widths (LargeList, ListView), tagged structs carrying extra children, map entry children in the wrong order, and timestamp units.

The corruption is concrete: an Arrow map vector whose entry-struct children sit in [value, key] order is tolerated by Arrow and read correctly by ArrowColumnVector (key/value are addressed by name), but its buffers are positional, so a cached {123: 7} comes back as {7: 123} -- silently, and for the lifetime of the cached relation. The new test demonstrates this: it fails on master with Array((7, 123)) did not equal Array((123, 7)) and passes with the gate.

In-tree columnar producers happened to be covered by the blocklist (the cache scan itself produces canonical vectors; Python UDF output's large var-width and interchange nanos/interval shapes were the blocked entries), so this is reachable today only from external Arrow-backed DSv2 columnar sources -- supportsColumnarInput accepts any of them. The congruence check closes the whole class rather than the enumerated instances, and keeps the write gate symmetric with the read-side gate from SPARK-58258.

A false reject is safe by construction: the row-based fallback re-encodes values under the cache schema (with its value-domain guards), so eligible batches lose only the zero-copy optimization, never correctness.

Does this PR introduce any user-facing change?

No. The Arrow cache serializer (SPARK-57268) is unreleased.

How was this patch tested?

New test in ArrowCachedBatchSerializerSuite that builds a spec-tolerated but non-canonical map vector (entry children [value, key]), verifies the source reads correctly, round-trips it through the serializer's columnar write and read paths, and asserts the values survive. Without the fix it fails with the keys and values swapped. Full ArrowCachedBatchSerializerSuite passes (72 tests), confirming canonical shapes remain zero-copy eligible.

Was this patch authored or co-authored using generative AI tooling?

Yes, this pull request and its description were written by Claude Code.