Skip to content

Native Iceberg write over-counts NaNs for nested float fields when the input batch is already sliced #6146

Description

@andygrove

Describe the bug

For float and double fields inside a list or map, the native Iceberg writer can over-count NaN values in the data file's metrics when the input batch is already a slice of a larger array.

The writer's RowSlicer handles slices it cuts itself. A batch that arrives already sliced (for example from GlobalLimitExec in INSERT ... SELECT ... LIMIT n) keeps its full child arrays through the schema cast (iceberg_write.rs:871-879), and iceberg-rust's NaN counter reads list.values(), so it also counts NaNs outside the batch's window.

Effect: wrong nan_value_counts for nested float/double fields. Spark cannot push predicates onto list elements, so this is mostly incorrect metadata rather than wrong query results. Found by reading the code; not reproduced.

Steps to reproduce

CREATE TABLE t (id INT, xs ARRAY<DOUBLE>) USING iceberg;
-- source where NaNs appear only in rows past the limit
INSERT INTO t SELECT id, xs FROM src ORDER BY id LIMIT 10;
SELECT readable_metrics FROM t.files;

Expected behavior

nan_value_counts counts only the elements of the rows written, matching iceberg-java. Compacting the list/map child arrays (or the whole batch) before the metrics are computed would fix it.

Additional context

Found in an audit of the native Iceberg write path. Part of #5649.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions