Skip to content

IcebergWriteStrategy ignores spark.comet.enabled, so the split-operator write plan survives the kill switch #6142

Description

@andygrove

Describe the bug

IcebergWriteStrategy checks only spark.comet.write.iceberg.splitOperator.enabled (spark/src/main/scala/org/apache/comet/iceberg/IcebergWriteStrategy.scala:38), and the strategy is registered unconditionally (CometSparkSessionExtensions.scala:102). It does not check spark.comet.enabled or whether Comet is loaded.

Today the split flag defaults to false, so this has no effect. Once it defaults to true (#5644), a session with spark.comet.enabled=false would still plan Iceberg writes as Comet's IcebergWriteExec + IcebergCommitExec. The main kill switch would no longer restore Spark's own write plan, which is the first thing users try when they suspect Comet.

Steps to reproduce

spark.conf.set("spark.comet.enabled", "false")
spark.conf.set("spark.comet.write.iceberg.splitOperator.enabled", "true")
spark.sql("INSERT INTO iceberg_table SELECT ...").explain()
// plan contains IcebergCommitExec / IcebergWriteExec

Expected behavior

With spark.comet.enabled=false (or Comet not loaded), the strategy returns Nil and Spark plans its own AppendDataExec / ReplaceDataExec and so on. Gate it the same way CometRule gates conversion.

Additional context

This blocks flipping the split-operator default, the first step of the rollout in #5644. Found in an audit of the native Iceberg write path. Part of #5649.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions