[format] Close the codec probe stream in HadoopCompressionUtils

[jackylee-ch]

Purpose

getCompressionCodecByCompression probes the codec by opening an output stream and discards it unclosed. Hadoop leases a Compressor from CodecPool for that stream and returns it only on close(), so every call orphaned one lease — once per data file, so a long-lived csv/json/text writer leaks steadily. A native compressor also keeps its z_stream, released only via end(). The probe is kept; it surfaces a missing native library at selection time.

AbstractTextFileWriter.close() also drops a redundant flush(): close() already flushes, and since SingleFileWriter closes writer and stream separately, a throwing flush() lost the lease for good.

Tests

HadoopCompressionUtilsTest#testCodecProbeDoesNotLeaseACompressor asserts the leased count does not grow. DEFLATE, since GzipCodec.getCompressorType() is null without native zlib. Reverting the fix:

expected: 1
 but was: 5

paimon-format: 634 run, 0 failures.

Written with Claude Code; verification is mine.

[JingsongLi]

Good catch. codec.createOutputStream(...) leases a compressor from the CodecPool, and without closing the stream the lease is never returned — every probe permanently grew the leased count. Wrapping the probe in try-with-resources is the minimal fix, and the test asserting a stable CodecPool.getLeasedCompressorsCount across repeated calls pins the behavior precisely.

Dropping the writer.flush() in AbstractTextFileWriter.close() is also correct — close() flushes on its own.