Describe the bug
On a format-version 1 table, a partition spec can mix a live field with a void field whose source column has since been dropped (iceberg-java keeps a dropped V1 partition field as a void transform). The native Iceberg writer fails every task on such a spec, and the query does not fall back.
CometLocationGenerator::try_new (native/core/src/execution/operators/iceberg_partition_path.rs:66-70) skips resolving the partition type only when the spec is entirely void (is_unpartitioned()). For a mixed spec it calls PartitionSpec::partition_type(schema), which errors with "No column with source column id" because the void field's source column is gone. The partition-value computation hits the same resolution.
The fixes for #5691 and #5693 cover the all-void spec only.
This was found by reading the code and has not been reproduced yet.
Steps to reproduce
CREATE TABLE t (id INT, a STRING, b STRING) USING iceberg
PARTITIONED BY (a, b) TBLPROPERTIES ('format-version'='1');
ALTER TABLE t DROP PARTITION FIELD b;
ALTER TABLE t DROP COLUMN b;
-- with spark.comet.iceberg.write.enabled=true
INSERT INTO t VALUES (1, 'x');
Expected behavior
The insert succeeds, as it does with iceberg-java. Either the native writer resolves the partition type without the void field's source column (a void field never contributes a value), or the gate declines the native write when a void field's source column is missing from the schema.
Additional context
Found in an audit of the native Iceberg write path before enabling it by default. Part of #5649. Related: #5691, #5693.
Describe the bug
On a format-version 1 table, a partition spec can mix a live field with a
voidfield whose source column has since been dropped (iceberg-java keeps a dropped V1 partition field as avoidtransform). The native Iceberg writer fails every task on such a spec, and the query does not fall back.CometLocationGenerator::try_new(native/core/src/execution/operators/iceberg_partition_path.rs:66-70) skips resolving the partition type only when the spec is entirelyvoid(is_unpartitioned()). For a mixed spec it callsPartitionSpec::partition_type(schema), which errors with "No column with source column id" because thevoidfield's source column is gone. The partition-value computation hits the same resolution.The fixes for #5691 and #5693 cover the all-
voidspec only.This was found by reading the code and has not been reproduced yet.
Steps to reproduce
Expected behavior
The insert succeeds, as it does with iceberg-java. Either the native writer resolves the partition type without the
voidfield's source column (avoidfield never contributes a value), or the gate declines the native write when avoidfield's source column is missing from the schema.Additional context
Found in an audit of the native Iceberg write path before enabling it by default. Part of #5649. Related: #5691, #5693.