Skip to content

Native Iceberg write fails on a V1 spec that mixes a live field with a void field whose source column was dropped #6141

Description

@andygrove

Describe the bug

On a format-version 1 table, a partition spec can mix a live field with a void field whose source column has since been dropped (iceberg-java keeps a dropped V1 partition field as a void transform). The native Iceberg writer fails every task on such a spec, and the query does not fall back.

CometLocationGenerator::try_new (native/core/src/execution/operators/iceberg_partition_path.rs:66-70) skips resolving the partition type only when the spec is entirely void (is_unpartitioned()). For a mixed spec it calls PartitionSpec::partition_type(schema), which errors with "No column with source column id" because the void field's source column is gone. The partition-value computation hits the same resolution.

The fixes for #5691 and #5693 cover the all-void spec only.

This was found by reading the code and has not been reproduced yet.

Steps to reproduce

CREATE TABLE t (id INT, a STRING, b STRING) USING iceberg
  PARTITIONED BY (a, b) TBLPROPERTIES ('format-version'='1');
ALTER TABLE t DROP PARTITION FIELD b;
ALTER TABLE t DROP COLUMN b;
-- with spark.comet.iceberg.write.enabled=true
INSERT INTO t VALUES (1, 'x');

Expected behavior

The insert succeeds, as it does with iceberg-java. Either the native writer resolves the partition type without the void field's source column (a void field never contributes a value), or the gate declines the native write when a void field's source column is missing from the schema.

Additional context

Found in an audit of the native Iceberg write path before enabling it by default. Part of #5649. Related: #5691, #5693.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions