Describe the bug
<=>, <, <=, > and >= on arrays and structs with FLOAT/DOUBLE leaves run natively and compare the nested values with Arrow's total order, so -0.0 and 0.0 are different values. Spark compares nested values with TypeUtils.getInterpretedOrdering, which bottoms out in SQLOrderingUtil.compareDoubles/compareFloats, where signed zeros are equal.
CometExecRule.normalize only wraps scalar float operands. #6073 fixes =, <> and IN for nested operands, but leaves <=> and the ordering operators on the raw comparator. #5507 covers ORDER BY and rank on nested float keys, and this is the same gap in the comparison predicates.
Steps to reproduce
On main at cccc08b, default Spark 4.1 profile, as a CometSqlFileTestSuite fixture:
CREATE TABLE t (a ARRAY<DOUBLE>, b ARRAY<DOUBLE>, s STRUCT<v: DOUBLE>, u STRUCT<v: DOUBLE>) USING parquet;
INSERT INTO t VALUES (
array(CAST('-0.0' AS DOUBLE)), array(CAST('0.0' AS DOUBLE)),
named_struct('v', CAST('-0.0' AS DOUBLE)), named_struct('v', CAST('0.0' AS DOUBLE)));
SELECT a <=> b FROM t;
SELECT a < b, a >= b, s <=> u, s < u FROM t;
Both queries run fully native (CometProject over CometNativeScan) and return different answers:
| Query |
Spark |
Comet |
a <=> b |
true |
false |
a < b, a >= b, s <=> u, s < u |
false, true, true, false |
true, false, false, true |
CAST('-0.0' AS DOUBLE) is a string cast, so the sign survives. CAST(-0.0 AS DOUBLE) would produce +0.0.
Expected behavior
The same answers as Spark: signed zeros inside arrays and structs compare equal for every comparison operator.
Additional context
Found while reviewing #6073. I only verified signed zero. NaN payloads and signs probably diverge too, since total_cmp orders a negative NaN below every other value while Spark treats every NaN as the largest value. The equivalence in #6073's nested_comparison.rs could likely be extended into an ordering comparator for these operators.
Describe the bug
<=>,<,<=,>and>=on arrays and structs withFLOAT/DOUBLEleaves run natively and compare the nested values with Arrow's total order, so-0.0and0.0are different values. Spark compares nested values withTypeUtils.getInterpretedOrdering, which bottoms out inSQLOrderingUtil.compareDoubles/compareFloats, where signed zeros are equal.CometExecRule.normalizeonly wraps scalar float operands. #6073 fixes=,<>andINfor nested operands, but leaves<=>and the ordering operators on the raw comparator. #5507 coversORDER BYand rank on nested float keys, and this is the same gap in the comparison predicates.Steps to reproduce
On
mainat cccc08b, default Spark 4.1 profile, as aCometSqlFileTestSuitefixture:Both queries run fully native (
CometProjectoverCometNativeScan) and return different answers:a <=> btruefalsea < b, a >= b, s <=> u, s < ufalse, true, true, falsetrue, false, false, trueCAST('-0.0' AS DOUBLE)is a string cast, so the sign survives.CAST(-0.0 AS DOUBLE)would produce+0.0.Expected behavior
The same answers as Spark: signed zeros inside arrays and structs compare equal for every comparison operator.
Additional context
Found while reviewing #6073. I only verified signed zero. NaN payloads and signs probably diverge too, since
total_cmporders a negative NaN below every other value while Spark treats every NaN as the largest value. The equivalence in #6073'snested_comparison.rscould likely be extended into an ordering comparator for these operators.