Skip to content

Comparison operators on arrays and structs with floating-point leaves do not match Spark for signed zero #6157

Description

@andygrove

Describe the bug

<=>, <, <=, > and >= on arrays and structs with FLOAT/DOUBLE leaves run natively and compare the nested values with Arrow's total order, so -0.0 and 0.0 are different values. Spark compares nested values with TypeUtils.getInterpretedOrdering, which bottoms out in SQLOrderingUtil.compareDoubles/compareFloats, where signed zeros are equal.

CometExecRule.normalize only wraps scalar float operands. #6073 fixes =, <> and IN for nested operands, but leaves <=> and the ordering operators on the raw comparator. #5507 covers ORDER BY and rank on nested float keys, and this is the same gap in the comparison predicates.

Steps to reproduce

On main at cccc08b, default Spark 4.1 profile, as a CometSqlFileTestSuite fixture:

CREATE TABLE t (a ARRAY<DOUBLE>, b ARRAY<DOUBLE>, s STRUCT<v: DOUBLE>, u STRUCT<v: DOUBLE>) USING parquet;
INSERT INTO t VALUES (
  array(CAST('-0.0' AS DOUBLE)), array(CAST('0.0' AS DOUBLE)),
  named_struct('v', CAST('-0.0' AS DOUBLE)), named_struct('v', CAST('0.0' AS DOUBLE)));

SELECT a <=> b FROM t;
SELECT a < b, a >= b, s <=> u, s < u FROM t;

Both queries run fully native (CometProject over CometNativeScan) and return different answers:

Query Spark Comet
a <=> b true false
a < b, a >= b, s <=> u, s < u false, true, true, false true, false, false, true

CAST('-0.0' AS DOUBLE) is a string cast, so the sign survives. CAST(-0.0 AS DOUBLE) would produce +0.0.

Expected behavior

The same answers as Spark: signed zeros inside arrays and structs compare equal for every comparison operator.

Additional context

Found while reviewing #6073. I only verified signed zero. NaN payloads and signs probably diverge too, since total_cmp orders a negative NaN below every other value while Spark treats every NaN as the largest value. The equivalence in #6073's nested_comparison.rs could likely be extended into an ordering comparator for these operators.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions