[GLUTEN-12743][CI] Make Delta DV row-group fixtures deterministic

[felipepessoto]

What changes are proposed in this pull request?

References #12743.

Patch Delta v4.2.0 after cloning so DeltaParquetFileFormatSuite.generateData scopes spark.gluten.sql.native.parquet.write.blockRows=10000 around its 20,000-row fixture write.

The fixture expects parquet.block.size=50 KiB to create multiple row groups. Gluten propagates that Hadoop setting to the native writer, as covered separately by #12969. However, Velox writes each input ColumnarBatch to Arrow before evaluating the buffered-byte flush policy. This fixture arrives in one batch, so even reducing the byte threshold to 5 KiB only closes the one row group after all 20,000 rows have already been written; it cannot split that batch. Velox passes its row-count limit to Arrow's maxRowGroupLength, which splits a record batch while writing. A 10,000-row limit therefore creates at least two row groups while preserving the native write path.

With the fixture precondition fixed, 9 tests pass and are removed from the known-failure baseline. The remaining 15 cases proceed beyond the row-group assertion but expose separate DV metadata/result mismatches, so they remain quarantined. This reduces the baseline from 707 to 698 entries. The source patch requires exactly one original fixture write and exactly one resulting row-count scope, so Delta source drift fails setup explicitly.

How was this patch tested?

• bash -n .github/workflows/util/delta-spark-ut/setup-delta.sh
• Executed the exact transformation against upstream Delta v4.2.0 and verified one original fixture write becomes one scoped withSQLConf block
• CI run 35127602529 with the scoped row-count limit passed the multi-row-group assertion; 9 cases passed and 15 continued to downstream DV metadata/result failures
• CI run 35143376122 with a 5 KiB parquet.block.size still failed the multi-row-group assertion, confirming byte-threshold changes cannot split this one-batch fixture
• Verified the baseline contains 698 non-comment entries: 9 target entries removed and 15 retained
• git diff --check

The corrected baseline is being validated by the full Delta Spark UT workflow.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: GitHub Copilot App 1.0.84-5

[Copilot]

🟡 Changes recommended

The setup guard does not verify that the row-cap setting was inserted adjacent to the expected block-size setting.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Makes Delta deletion-vector fixtures deterministic by enforcing multiple Parquet row groups and removing obsolete known failures.

Changes:

• Adds a 10,000-row Parquet row-group cap.
• Removes 24 resolved entries from the known-failure baseline.

File summaries

File	Description
.github/workflows/util/delta-spark-ut/setup-delta.sh	Patches Delta fixture configuration and validates the change.
.github/workflows/util/delta-spark-ut/known-failures.txt	Removes 24 affected test entries.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟢 Approval recommended

No unresolved blocking issues were identified.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟢 Approval recommended

No unresolved review issues remain.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

The setup script has three unresolved moderate issues involving scope, ref compatibility, and substitution matching.

Review details

Suppressed comments (3)

.github/workflows/util/delta-spark-ut/setup-delta.sh:192

• The guard and replacement operate over the whole source file rather than the generateData method. If a future Delta ref moves/removes the fixture assignment but leaves an identical assignment in another helper, the pre/post counts still pass and sed patches the unrelated helper, leaving this fixture at 50 KiB and silently restoring the DV failures. Scope both checks and the replacement to the generateData block so that this source drift fails setup.

ORIGINAL_BLOCK_SIZE_LINES=$(
  grep -Fxc '    hadoopConf().set("parquet.block.size", (1024 * 50).toString)' "$DPFFS" || true
)
if [ "$ORIGINAL_BLOCK_SIZE_LINES" -ne 1 ]; then
  echo "ERROR: expected exactly one original 50 KiB Parquet block-size line;" \

.github/workflows/util/delta-spark-ut/setup-delta.sh:191

• workflow_dispatch.delta_ref is documented to accept any Delta tag/branch/SHA (.github/workflows/delta_spark_ut.yml:70-75), but this unconditional exact-match guard only succeeds for a checkout containing the v4.2.0 50 KiB line. A run against another compatible ref now exits before tests, even if that ref already has an equivalent fixture with a different source shape. Scope this patch to the supported v4.2.0 ref or make the patch/baseline selection ref-specific instead of making the generic ref input fail here.

ORIGINAL_BLOCK_SIZE_LINES=$(
  grep -Fxc '    hadoopConf().set("parquet.block.size", (1024 * 50).toString)' "$DPFFS" || true
)
if [ "$ORIGINAL_BLOCK_SIZE_LINES" -ne 1 ]; then

.github/workflows/util/delta-spark-ut/setup-delta.sh:197

• This substitution is broader than the guard: it rewrites every line containing parquet.block.size and the 50 KiB expression, while the pre/post checks only count the exact four-space spelling. If the source gains another equivalent assignment with different indentation or trailing text, setup can rewrite both and still report one canonical 5 KiB line, silently patching unintended code instead of rejecting source drift. Match the complete expected assignment here so only the guarded line can be changed.

sed -i '/parquet.block.size/s/(1024 \* 50)/(1024 * 5)/' "$DPFFS"

• Files reviewed: 2/2 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

Scope the validation and replacement specifically to generateData to prevent silently patching an unrelated write.

Review details

Suppressed comments (1)

.github/workflows/util/delta-spark-ut/setup-delta.sh:190

• ORIGINAL_WRITES is counted over the entire suite, so it can be satisfied by an unrelated method while generateData no longer contains this exact line; sed would then patch the wrong write (or no fixture). The later ROW_CAP_SCOPES check is also global, so an existing scope elsewhere could make setup pass without changing generateData. Scope both checks and the replacement to the generateData block (including its block-size fixture context), otherwise future Delta source drift can silently reintroduce the one-row-group failures while these baseline entries are removed.

ORIGINAL_WRITES=$(
  grep -Fxc '    df.write.format("delta").mode("append").save(tablePath)' "$DPFFS" || true
)

• Files reviewed: 2/2 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

Scope the fixture-pattern uniqueness and replacement checks to prevent patching the wrong write.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

.github/workflows/util/delta-spark-ut/setup-delta.sh:190

• This uniqueness check is not scoped to generateData or to the fixture's parquet.block.size setup, and the sed replacement below uses the same unscoped write pattern. If a Delta ref keeps one identical write in another helper while changing the fixture write's shape/location, setup will patch the wrong write and the global row-cap count can still pass, silently leaving the multi-row-group fixture unpatched. Match the block-size setup and write as one expected fixture pattern, then validate the exact replacement in that same region.

• Files reviewed: 2/2 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

Strengthen the source-drift guard to validate the complete fixture-write sequence.

Get a fresh assessment by requesting another Copilot review.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Lite

[felipepessoto]

@zhouyuan, @philo-he, when you have a chance, could you take a look?

[zhouyuan]

@felipepessoto thanks for the continuous efforts on maintaining the delta CI!