fix(spark): correct mod ANSI zero divisor and negative zero handling - #23987
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #23987 +/- ##
==========================================
- Coverage 81.19% 81.19% -0.01%
==========================================
Files 1110 1110
Lines 388750 388879 +129
Branches 388750 388879 +129
==========================================
+ Hits 315657 315750 +93
- Misses 54506 54533 +27
- Partials 18587 18596 +9 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
115cecb to
612ea63
Compare
|
Thanks for checking, @kumarUjjawal. There is some implementation overlap, but the two PRs fix different functions. #23898 is scoped to pmod: it rewrites spark_pmod so that it no longer uses try_rem, while leaving mod behavior unchanged. This PR fixes mod by correcting try_rem and adds coverage in mod.slt, closing #23894. The shared overlap is the negative_zero/is_zero helpers. I will rebase and deduplicate those helpers after whichever PR lands first. |
0270100 to
faf495e
Compare
kosiew
left a comment
There was a problem hiding this comment.
Thanks for working on this. The changes look good overall, especially the handling of both positive and negative floating-point zero divisors and the added regression coverage. I have one non-blocking suggestion to extend the coverage to pmod as well.
| let neg = lt(&result, &zero)?; | ||
| let plus = zip(&neg, right, &zero)?; | ||
| let result = add(&plus, &result)?; | ||
| let result = try_rem(&result, right, enable_ansi_mode)?; |
There was a problem hiding this comment.
Nice work covering these cases for mod. Since pmod also reaches the updated try_rem helper twice, could we add similar coverage for that path too? It would be useful to include pmod SQLLogic or unit cases for ANSI float 0.0, legacy -0.0, and ANSI NULL % 0.0. That should help make sure the second remainder path stays protected as well.
There was a problem hiding this comment.
Good call — added coverage for the pmod path in the latest push:
- Unit tests mirroring the mod ones: ANSI float
0.0raises, legacy-0.0returns NULL, ANSI-0.0raises, and an ANSI NULL dividend with a zero divisor returns NULL without raising (both int and float). - SQLLogic cases in
spark/math/pmod.sltfor the same scenarios.
Also rebased onto the latest main.
In ANSI mode a floating-point zero divisor quietly returned NaN because Arrow's rem only reports division by zero for integer and decimal types, and a -0.0 divisor went unrecognised in both modes because Arrow's floating-point comparisons order totally, so -0.0 is distinct from 0.0. try_rem now detects zero divisors itself via an is_zero helper that counts -0.0 as zero, raises for a zero divisor of any numeric type in ANSI mode (masked by the validity of the dividend, since Spark's remainder is null intolerant), and substitutes NULL for zero divisors before calling Arrow's rem so the kernel never sees one. Closes apache#23894.
faf495e to
07126ed
Compare
|
Thanks @u70b3 |
…pache#23987) ## Which issue does this PR close? - Closes apache#23894 ## Rationale for this change `datafusion-spark`'s `mod` routes through the shared `try_rem` helper in `datafusion/spark/src/function/math/modulus.rs`, which has two gaps in its zero-divisor handling (both verified against Spark 3.5.8 – 4.2.0 in the issue): 1. **ANSI mode, floating-point divisor.** `try_rem` delegates to Arrow's `rem` kernel in ANSI mode, but Arrow only reports division by zero for integer and decimal types. Floating-point divisors follow IEEE 754 and quietly produce `NaN`, while Spark raises `REMAINDER_BY_ZERO` for a zero divisor of any numeric type: ```sql set datafusion.execution.enable_ansi_mode = true; SELECT mod(10.5::float8, 0.0::float8); -- NaN, Spark raises ``` 2. **`-0.0` divisor, both modes.** The legacy path nulls out zero divisors via `eq(right, 0)`, but Arrow's floating-point comparisons use a total order in which `-0.0` is distinct from `0.0`, so a `-0.0` divisor goes unrecognised. Spark's `isZero` is a numeric comparison and treats `-0.0` as zero: ```sql SELECT mod(10.5::float8, -0.0::float8); -- NaN, Spark returns NULL (legacy) ``` ## What changes are included in this PR? `try_rem` now detects zero divisors itself, mirroring the shape apache#23898 established for `pmod`: - A new `is_zero` helper counts `-0.0` as zero for the floating-point types (via a `negative_zero` companion, same as apache#23898). - In ANSI mode, any row with a zero divisor raises `ArrowError::DivideByZero` — the same error Arrow's `rem` already produces for integers today, so the message stays uniform across types. The check is masked by the validity of the dividend because Spark's remainder expressions are null intolerant: a NULL dividend short-circuits to NULL before the divisor is validated, so `mod(NULL, 0)` must return NULL rather than raise. - Both modes substitute NULL for zero divisors before calling Arrow's `rem`, so the kernel never sees a zero divisor: legacy mode gets NULLs, and ANSI mode has already raised on the rows that required it. Note on overlap with apache#23898: that PR rewrites `pmod` to no longer use `try_rem` and adds identical `is_zero`/`negative_zero` helpers. This PR is independent of it — `mod` is fixed either way — but whichever lands second should dedupe the helpers in a rebase. Until apache#23898 lands, `pmod` also picks up the `-0.0` and ANSI floating-point zero-divisor fixes through the shared helper. Out of scope: apache#23897 (reproducing Spark's exact ANSI error text) is a repository-wide error-message policy question, as noted in that issue. ## Are these changes tested? Yes: - New unit tests in `modulus.rs`: ANSI floating-point zero divisor raises; `-0.0` divisor returns NULL in legacy mode and raises in ANSI mode; a NULL dividend with a zero divisor returns NULL in ANSI mode (integer and float). - New sqllogictest cases in `spark/math/mod.slt` covering the same behavior at SQL level. - Verified `cargo test -p datafusion-spark`, the `spark/math/mod.slt` and `spark/math/pmod.slt` sqllogictests, `./dev/rust_lint.sh`, and the extended workspace suite (ci profile with `avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption`) — all green. ## Are there any user-facing changes? Only the bug fixes, and only for the Spark `mod`/`pmod` functions: in ANSI mode a floating-point zero divisor now raises instead of returning NaN, and a `-0.0` divisor is treated as zero in both modes (NULL in legacy mode, an error in ANSI mode), matching Spark. No API changes.
…o handling Audit `pmod` against Apache Spark 4.2.0 and fix three divergences. Spark evaluates `(r + n) % n` with Java arithmetic, which wraps around for `int` and `long` and promotes `byte` and `short` to `int`. Arrow's checked `add` reported an overflow instead, so `pmod(-1, -2147483648)` errored where Spark returns `2147483647`. Use `add_wrapping`, and widen the narrow integer types before the addition so Java's operand promotion is reproduced. Arrow's `rem` only reports division by zero for integer and decimal types, so in ANSI mode a floating point zero divisor quietly produced `NaN` where Spark raises. Check the divisor directly, masked by the validity of the dividend so the null intolerant short circuit is preserved and `pmod(NULL, 0)` stays NULL. Arrow's comparison kernels order floating point values totally, so `-0.0` was neither recognised as a zero divisor nor kept out of the negative branch. Treat `-0.0` as zero when testing the divisor, and select the adjusted value only where the remainder is negative so a `-0.0` result keeps its sign. `pmod` no longer shares `try_rem` with `mod`, whose behavior is unchanged. Rebase note: apache#23987 landed the same ANSI-zero-divisor and `-0.0` fixes for `mod` while this branch was open, by rewriting the shared `try_rem` and adding `negative_zero` and `is_zero`. This rebase keeps main's `try_rem` and its helpers rather than the versions developed here, so `mod` is untouched; the `is_zero` helper is identical either way. What remains specific to this branch is the Java overflow and operand promotion handling, the `-0.0` result sign, and the `pmod` test coverage. The two test suites are kept in full.
Which issue does this PR close?
Rationale for this change
datafusion-spark'smodroutes through the sharedtry_remhelper indatafusion/spark/src/function/math/modulus.rs, which has two gaps in itszero-divisor handling (both verified against Spark 3.5.8 – 4.2.0 in the issue):
ANSI mode, floating-point divisor.
try_remdelegates to Arrow'sremkernel in ANSI mode, but Arrow only reports division by zero for integer and
decimal types. Floating-point divisors follow IEEE 754 and quietly produce
NaN, while Spark raisesREMAINDER_BY_ZEROfor a zero divisor of anynumeric type:
-0.0divisor, both modes. The legacy path nulls out zero divisors viaeq(right, 0), but Arrow's floating-point comparisons use a total order inwhich
-0.0is distinct from0.0, so a-0.0divisor goes unrecognised.Spark's
isZerois a numeric comparison and treats-0.0as zero:What changes are included in this PR?
try_remnow detects zero divisors itself, mirroring the shape #23898established for
pmod:is_zerohelper counts-0.0as zero for the floating-point types(via a
negative_zerocompanion, same as fix(spark): correct pmod overflow, ANSI zero divisor and negative zero handling #23898).ArrowError::DivideByZero— the same error Arrow's
remalready produces for integers today, so themessage stays uniform across types. The check is masked by the validity of
the dividend because Spark's remainder expressions are null intolerant: a
NULL dividend short-circuits to NULL before the divisor is validated, so
mod(NULL, 0)must return NULL rather than raise.rem,so the kernel never sees a zero divisor: legacy mode gets NULLs, and ANSI
mode has already raised on the rows that required it.
Note on overlap with #23898: that PR rewrites
pmodto no longer usetry_remand adds identicalis_zero/negative_zerohelpers. This PR isindependent of it —
modis fixed either way — but whichever lands secondshould dedupe the helpers in a rebase. Until #23898 lands,
pmodalso picksup the
-0.0and ANSI floating-point zero-divisor fixes through the sharedhelper.
Out of scope: #23897 (reproducing Spark's exact ANSI error text) is a
repository-wide error-message policy question, as noted in that issue.
Are these changes tested?
Yes:
modulus.rs: ANSI floating-point zero divisor raises;-0.0divisor returns NULL in legacy mode and raises in ANSI mode; a NULLdividend with a zero divisor returns NULL in ANSI mode (integer and float).
spark/math/mod.sltcovering the same behavior atSQL level.
cargo test -p datafusion-spark, thespark/math/mod.sltandspark/math/pmod.sltsqllogictests,./dev/rust_lint.sh, and the extendedworkspace suite (ci profile with
avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption)— all green.
Are there any user-facing changes?
Only the bug fixes, and only for the Spark
mod/pmodfunctions: in ANSImode a floating-point zero divisor now raises instead of returning NaN, and a
-0.0divisor is treated as zero in both modes (NULL in legacy mode, an errorin ANSI mode), matching Spark. No API changes.