What is the problem the feature request solves?
Since #5677, the Iceberg Spark test diffs run the upstream Iceberg suites with the native Iceberg writer on. A green run cannot tell a native write from a silent fallback:
CometIcebergNativeWrite has about 22 eligibility rules that fall back silently, and writes whose input is a local relation fall back unless spark.comet.exec.localTableScan.enabled is set.- The diffs set
spark.comet.explainFallback.enabled=true, but gradle does not forward Spark's log output into the CI job log, so the fallback reasons are not visible there.
- No upstream Iceberg test asserts which writer ran.
So the Iceberg jobs are weak evidence for the "diffs pass with the flags on" criterion in #5644. Comet's own suites do assert CometIcebergWriteExec is in the plan, but they run only on Spark 4.1 / Iceberg 1.11 per PR.
Describe the potential solution
Make native-write coverage measurable in the Iceberg jobs, for example:
- a listener installed by the diffs that counts
CometIcebergWriteExec vs IcebergWriteExec executions and fallback reasons, and writes them to a file the job uploads or prints as a summary, or
- forward the
CometExecRule fallback warnings through gradle's test logging so they can be grepped, or
- a strict mode (test-only config) that fails a write which falls back for any reason outside an expected list.
A per-version summary (native writes / total writes, top fallback reasons) would also show how much of the suite exercises the native path.
Additional context
Found in an audit of the native Iceberg write path before enabling it by default. Part of #5649. Related: #5644.
What is the problem the feature request solves?
Since #5677, the Iceberg Spark test diffs run the upstream Iceberg suites with the native Iceberg writer on. A green run cannot tell a native write from a silent fallback:
CometIcebergNativeWritehas about 22 eligibility rules that fall back silently, and writes whose input is a local relation fall back unlessspark.comet.exec.localTableScan.enabledis set.spark.comet.explainFallback.enabled=true, but gradle does not forward Spark's log output into the CI job log, so the fallback reasons are not visible there.So the Iceberg jobs are weak evidence for the "diffs pass with the flags on" criterion in #5644. Comet's own suites do assert
CometIcebergWriteExecis in the plan, but they run only on Spark 4.1 / Iceberg 1.11 per PR.Describe the potential solution
Make native-write coverage measurable in the Iceberg jobs, for example:
CometIcebergWriteExecvsIcebergWriteExecexecutions and fallback reasons, and writes them to a file the job uploads or prints as a summary, orCometExecRulefallback warnings through gradle's test logging so they can be grepped, orA per-version summary (native writes / total writes, top fallback reasons) would also show how much of the suite exercises the native path.
Additional context
Found in an audit of the native Iceberg write path before enabling it by default. Part of #5649. Related: #5644.