[VL][TEST] Cover Parquet row-group sizing from Hadoop conf

[felipepessoto]

What changes are proposed in this pull request?

Adds focused Velox Parquet row-group coverage for parquet.block.size set on Spark's runtime Hadoop configuration, matching the configuration channel used by Delta's DeletionVectorsWithPredicatePushdownSuite setup.

The plain Parquet suite establishes two controls for a single-partition, 1M-row int64 write through the native Velox writer:

1. the default 128MB block size produces one row group;
2. runtime hadoopConfiguration.set("parquet.block.size", 1 MiB in bytes) produces multiple row groups.

The Delta suite applies the same runtime Hadoop configuration to writes with and without deletion vectors. Each test now first asserts that exactly one Parquet data file was produced, then asserts that file contains multiple row groups. This prevents multiple output files from being mistaken for multiple row groups.

backends-velox/src-delta/test is activated only by Maven's delta profile. The focused validation therefore uses -Pdelta together with the Velox, Spark, and Scala profiles so VeloxDeltaParquetRowGroupSuite is compiled and selected explicitly.

The earlier draft also tested spark.gluten.sql.columnar.parquet.write.blockSize. Even when configured as the valid capacity string 1MB, the current writer injection parses it to bytes and re-emits unitless parquet.block.size=1048576, which Velox rejects as an invalid capacity string. That separate production behavior was removed from this test-only patch so it cannot mask the runtime-Hadoop-conf coverage with a setup failure.

How was this patch tested?

Using the repository's focused runner with Spark 3.5, Scala 2.12, Velox, and Delta profiles:

./dev/run-scala-test.sh \
  -Pspark-3.5,scala-2.12,backends-velox,delta \
  -pl backends-velox \
  -s org.apache.spark.sql.execution.VeloxParquetRowGroupSuite \
  -t "native writer should respect parquet.block.size set on the runtime Hadoop conf"

./dev/run-scala-test.sh \
  -Pspark-3.5,scala-2.12,backends-velox,delta \
  -pl backends-velox \
  -s org.apache.gluten.execution.VeloxDeltaParquetRowGroupSuite \
  -t "delta write should respect parquet.block.size set on the runtime Hadoop conf"

./dev/run-scala-test.sh \
  -Pspark-3.5,scala-2.12,backends-velox,delta \
  -pl backends-velox \
  -s org.apache.gluten.execution.VeloxDeltaParquetRowGroupSuite \
  -t "delta write with deletion vectors should respect parquet.block.size on the runtime Hadoop conf"

All three focused tests passed. The two Delta tests were each discovered and executed as one selected test under -Pdelta; there are no intentional failures remaining on Spark 3.5.

Also checked the two changed files with the repository Spotless configuration and current CI license-header checker. A local Spark 4.1 compile was attempted, but this Windows worktree materializes src-delta40 symlinks as text files, so Scala compilation stops on those existing checkout artifacts; CI remains the authoritative Spark 4.1 validation.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: GitHub Copilot App (GPT-5.6 Sol)

[felipepessoto]

Closing this draft as superseded by #13036 and #13058.

The investigation and green CI here confirmed that parquet.block.size set through Spark’s runtime Hadoop configuration is propagated to Gluten’s native Parquet writer. The Delta fixture failures were therefore not caused by a missing configuration. They occur when the fixture is delivered as one Arrow batch, because Velox evaluates the byte-based flush threshold only after that batch has been written and cannot split it into multiple row groups.

#13036 and #13058 address the affected Delta fixtures deterministically with scoped native row-count limits and validate the resulting downstream tests in the full Delta Spark UT pipeline. The additional 1M-row regression suites in this PR are no longer necessary.