Corrected 2026-09-23. This issue originally said Spark 3.x wraps every non-fatal write failure in a "Writing job aborted" SparkException. That is Spark 3.3 and earlier behaviour, which Comet does not support. The description below is corrected. See #6153 for the source references.
Describe the bug
When an Iceberg write fails under the split-operator plan, IcebergCommitExec aborts the write and rethrows the original cause, on both the job-failure path and the commit-failure path.
On every Spark version Comet supports (3.4 through 4.2), Spark's own V2TableWriteExec.writeWithV2 behaves as follows:
- if
batchWrite.abort succeeds, it rethrows the original cause unchanged
- if
batchWrite.abort itself throws, it attaches the abort failure to the cause as suppressed and throws QueryExecutionErrors.writingJobFailedError(cause), a SparkException ("Writing job failed.") with the original failure as its cause
IcebergCommitExec matches the first case. In the second it attaches the abort failure as suppressed but still throws the raw cause, so when the abort fails the user sees a different exception type depending on whether the split plan is on.
Steps to reproduce
Make an Iceberg write fail (at commit, or in a task) with a BatchWrite whose abort also throws, with spark.comet.write.iceberg.splitOperator.enabled off and then on, and compare the thrown exception.
Expected behavior
The split plan throws the same exception as Spark's own path: the original cause when the abort succeeds, and writingJobFailedError(cause) when the abort fails. No version shim is needed, since the behaviour is the same on 3.4 through 4.2.
Additional context
Found in an audit of the native Iceberg write path. Part of #5649.
Describe the bug
When an Iceberg write fails under the split-operator plan,
IcebergCommitExecaborts the write and rethrows the original cause, on both the job-failure path and the commit-failure path.On every Spark version Comet supports (3.4 through 4.2), Spark's own
V2TableWriteExec.writeWithV2behaves as follows:batchWrite.abortsucceeds, it rethrows the original cause unchangedbatchWrite.abortitself throws, it attaches the abort failure to the cause as suppressed and throwsQueryExecutionErrors.writingJobFailedError(cause), aSparkException("Writing job failed.") with the original failure as its causeIcebergCommitExecmatches the first case. In the second it attaches the abort failure as suppressed but still throws the raw cause, so when the abort fails the user sees a different exception type depending on whether the split plan is on.Steps to reproduce
Make an Iceberg write fail (at commit, or in a task) with a
BatchWritewhoseabortalso throws, withspark.comet.write.iceberg.splitOperator.enabledoff and then on, and compare the thrown exception.Expected behavior
The split plan throws the same exception as Spark's own path: the original cause when the abort succeeds, and
writingJobFailedError(cause)when the abort fails. No version shim is needed, since the behaviour is the same on 3.4 through 4.2.Additional context
Found in an audit of the native Iceberg write path. Part of #5649.