[SPARK-59599][SQL] Pre-size the ArrowConverters.serializeBatch output buffer to avoid repeated reallocation

[david-mollitor-db]

What changes were proposed in this pull request?

ArrowConverters.serializeBatch serializes a single ArrowRecordBatch into a byte[] through a
ByteArrayOutputStream created with no initial capacity, so it starts at the JDK default of 32
bytes and grows by repeatedly doubling its backing array (grow + Arrays.copyOf) as the batch
is written.

This pre-sizes the ByteArrayOutputStream to the batch's body length
(ArrowRecordBatch.computeBodyLength(), plus a small margin for the IPC message header), clamped
to the Int range:

val estimatedSize =
  (batch.computeBodyLength() + IPC_HEADER_SIZE_ESTIMATE).min(Int.MaxValue).max(0L).toInt
val out = new ByteArrayOutputStream(estimatedSize)

ArrowCachedBatchSerializer (the Arrow columnar cache serializer) contained a verbatim copy of the
same method. This PR widens ArrowConverters.serializeBatch to private[sql] and has
ArrowCachedBatchSerializer.serializeBatch delegate to it, so the columnar cache path gets the
same pre-sizing and the duplicated implementation is removed.

Why are the changes needed?

The batch is written to the stream incrementally, not as one block: MessageSerializer.serialize
writes the message metadata and then each of the batch's buffers (with alignment padding)
individually, and the WritableByteChannel returned by Channels.newChannel further splits each
write into chunks of at most 8 KB. So a fresh 32-byte ByteArrayOutputStream reallocates and
copies its backing array on the order of log2(batchSize / 32) times per batch as it grows,
producing that many short-lived intermediate arrays (GC pressure) plus the copies.

computeBodyLength() returns the exact serialized body size and is cheap: it is O(number of
buffers) integer arithmetic over the buffer layout (no scan of the payload, no allocation), which
is negligible next to serializing the batch body. Pre-sizing lets the buffer be allocated once at
the right size, eliminating the repeated grow-and-copy -- now on both the Arrow output path and the
Arrow columnar cache path.

This is an allocation micro-optimization; it is not expected to change throughput on its own.

Does this PR introduce any user-facing change?

No. The serialized output is byte-for-byte identical.

How was this patch tested?

Existing ArrowConvertersSuite and ArrowCachedBatchSerializerSuite pass. This is a
behavior-preserving change (buffer sizing plus de-duplication of an identical method), so no new
tests were added.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Isaac

This pull request and its description were written by Isaac.