Skip to content

fix(spark): correct mod ANSI zero divisor and negative zero handling - #23987

Merged
kosiew merged 1 commit into
apache:mainfrom
u70b3:fix/spark-mod-zero-divisor
Aug 16, 2026
Merged

kosiew merged 1 commit into
apache:mainfrom
u70b3:fix/spark-mod-zero-divisor

Conversation

@u70b3

@u70b3 u70b3 commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

datafusion-spark's mod routes through the shared try_rem helper in
datafusion/spark/src/function/math/modulus.rs, which has two gaps in its
zero-divisor handling (both verified against Spark 3.5.8 – 4.2.0 in the issue):

  1. ANSI mode, floating-point divisor. try_rem delegates to Arrow's rem
    kernel in ANSI mode, but Arrow only reports division by zero for integer and
    decimal types. Floating-point divisors follow IEEE 754 and quietly produce
    NaN, while Spark raises REMAINDER_BY_ZERO for a zero divisor of any
    numeric type:

    set datafusion.execution.enable_ansi_mode = true;
    SELECT mod(10.5::float8, 0.0::float8);  -- NaN, Spark raises
  2. -0.0 divisor, both modes. The legacy path nulls out zero divisors via
    eq(right, 0), but Arrow's floating-point comparisons use a total order in
    which -0.0 is distinct from 0.0, so a -0.0 divisor goes unrecognised.
    Spark's isZero is a numeric comparison and treats -0.0 as zero:

    SELECT mod(10.5::float8, -0.0::float8);  -- NaN, Spark returns NULL (legacy)

What changes are included in this PR?

try_rem now detects zero divisors itself, mirroring the shape #23898
established for pmod:

  • A new is_zero helper counts -0.0 as zero for the floating-point types
    (via a negative_zero companion, same as fix(spark): correct pmod overflow, ANSI zero divisor and negative zero handling #23898).
  • In ANSI mode, any row with a zero divisor raises ArrowError::DivideByZero
    — the same error Arrow's rem already produces for integers today, so the
    message stays uniform across types. The check is masked by the validity of
    the dividend because Spark's remainder expressions are null intolerant: a
    NULL dividend short-circuits to NULL before the divisor is validated, so
    mod(NULL, 0) must return NULL rather than raise.
  • Both modes substitute NULL for zero divisors before calling Arrow's rem,
    so the kernel never sees a zero divisor: legacy mode gets NULLs, and ANSI
    mode has already raised on the rows that required it.

Note on overlap with #23898: that PR rewrites pmod to no longer use
try_rem and adds identical is_zero/negative_zero helpers. This PR is
independent of it — mod is fixed either way — but whichever lands second
should dedupe the helpers in a rebase. Until #23898 lands, pmod also picks
up the -0.0 and ANSI floating-point zero-divisor fixes through the shared
helper.

Out of scope: #23897 (reproducing Spark's exact ANSI error text) is a
repository-wide error-message policy question, as noted in that issue.

Are these changes tested?

Yes:

  • New unit tests in modulus.rs: ANSI floating-point zero divisor raises;
    -0.0 divisor returns NULL in legacy mode and raises in ANSI mode; a NULL
    dividend with a zero divisor returns NULL in ANSI mode (integer and float).
  • New sqllogictest cases in spark/math/mod.slt covering the same behavior at
    SQL level.
  • Verified cargo test -p datafusion-spark, the spark/math/mod.slt and
    spark/math/pmod.slt sqllogictests, ./dev/rust_lint.sh, and the extended
    workspace suite (ci profile with
    avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption)
    — all green.

Are there any user-facing changes?

Only the bug fixes, and only for the Spark mod/pmod functions: in ANSI
mode a floating-point zero divisor now raises instead of returning NaN, and a
-0.0 divisor is treated as zero in both modes (NULL in legacy mode, an error
in ANSI mode), matching Spark. No API changes.

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) spark labels Jul 30, 2026
@codecov-commenter

codecov-commenter commented Jul 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.40580% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.19%. Comparing base (b9399dc) to head (07126ed).

Files with missing lines Patch % Lines
datafusion/spark/src/function/math/modulus.rs 88.40% 6 Missing and 10 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #23987      +/-   ##
==========================================
- Coverage   81.19%   81.19%   -0.01%     
==========================================
  Files        1110     1110              
  Lines      388750   388879     +129     
  Branches   388750   388879     +129     
==========================================
+ Hits       315657   315750      +93     
- Misses      54506    54533      +27     
- Partials    18587    18596       +9     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@u70b3
u70b3 force-pushed the fix/spark-mod-zero-divisor branch from 115cecb to 612ea63 Compare July 31, 2026 11:31
@kumarUjjawal

Copy link
Copy Markdown
Contributor

Hi @u70b3 Thank you for working on this. I believe there's already a PR for this here #23898

@u70b3

u70b3 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for checking, @kumarUjjawal. There is some implementation overlap, but the two PRs fix different functions.

#23898 is scoped to pmod: it rewrites spark_pmod so that it no longer uses try_rem, while leaving mod behavior unchanged. This PR fixes mod by correcting try_rem and adds coverage in mod.slt, closing #23894.

The shared overlap is the negative_zero/is_zero helpers. I will rebase and deduplicate those helpers after whichever PR lands first.

@u70b3
u70b3 force-pushed the fix/spark-mod-zero-divisor branch 4 times, most recently from 0270100 to faf495e Compare August 7, 2026 05:40

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@u70b3,

Thanks for working on this. The changes look good overall, especially the handling of both positive and negative floating-point zero divisors and the added regression coverage. I have one non-blocking suggestion to extend the coverage to pmod as well.

let neg = lt(&result, &zero)?;
let plus = zip(&neg, right, &zero)?;
let result = add(&plus, &result)?;
let result = try_rem(&result, right, enable_ansi_mode)?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work covering these cases for mod. Since pmod also reaches the updated try_rem helper twice, could we add similar coverage for that path too? It would be useful to include pmod SQLLogic or unit cases for ANSI float 0.0, legacy -0.0, and ANSI NULL % 0.0. That should help make sure the second remainder path stays protected as well.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call — added coverage for the pmod path in the latest push:

  • Unit tests mirroring the mod ones: ANSI float 0.0 raises, legacy -0.0 returns NULL, ANSI -0.0 raises, and an ANSI NULL dividend with a zero divisor returns NULL without raising (both int and float).
  • SQLLogic cases in spark/math/pmod.slt for the same scenarios.

Also rebased onto the latest main.

In ANSI mode a floating-point zero divisor quietly returned NaN because
Arrow's rem only reports division by zero for integer and decimal types,
and a -0.0 divisor went unrecognised in both modes because Arrow's
floating-point comparisons order totally, so -0.0 is distinct from 0.0.

try_rem now detects zero divisors itself via an is_zero helper that
counts -0.0 as zero, raises for a zero divisor of any numeric type in
ANSI mode (masked by the validity of the dividend, since Spark's
remainder is null intolerant), and substitutes NULL for zero divisors
before calling Arrow's rem so the kernel never sees one.

Closes apache#23894.
@u70b3
u70b3 force-pushed the fix/spark-mod-zero-divisor branch from faf495e to 07126ed Compare August 15, 2026 08:25
@u70b3
u70b3 requested a review from kosiew August 15, 2026 10:41
@kosiew

kosiew commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Thanks @u70b3

@kosiew
kosiew added this pull request to the merge queue Aug 16, 2026
Merged via the queue into apache:main with commit 2da78d6 Aug 16, 2026
37 checks passed
imtherealnaska pushed a commit to imtherealnaska/datafusion that referenced this pull request Aug 16, 2026
…pache#23987)

## Which issue does this PR close?

- Closes apache#23894

## Rationale for this change

`datafusion-spark`'s `mod` routes through the shared `try_rem` helper in
`datafusion/spark/src/function/math/modulus.rs`, which has two gaps in
its
zero-divisor handling (both verified against Spark 3.5.8 – 4.2.0 in the
issue):

1. **ANSI mode, floating-point divisor.** `try_rem` delegates to Arrow's
`rem`
kernel in ANSI mode, but Arrow only reports division by zero for integer
and
decimal types. Floating-point divisors follow IEEE 754 and quietly
produce
`NaN`, while Spark raises `REMAINDER_BY_ZERO` for a zero divisor of any
   numeric type:

   ```sql
   set datafusion.execution.enable_ansi_mode = true;
   SELECT mod(10.5::float8, 0.0::float8);  -- NaN, Spark raises
   ```

2. **`-0.0` divisor, both modes.** The legacy path nulls out zero
divisors via
`eq(right, 0)`, but Arrow's floating-point comparisons use a total order
in
which `-0.0` is distinct from `0.0`, so a `-0.0` divisor goes
unrecognised.
   Spark's `isZero` is a numeric comparison and treats `-0.0` as zero:

   ```sql
SELECT mod(10.5::float8, -0.0::float8); -- NaN, Spark returns NULL
(legacy)
   ```

## What changes are included in this PR?

`try_rem` now detects zero divisors itself, mirroring the shape apache#23898
established for `pmod`:

- A new `is_zero` helper counts `-0.0` as zero for the floating-point
types
  (via a `negative_zero` companion, same as apache#23898).
- In ANSI mode, any row with a zero divisor raises
`ArrowError::DivideByZero`
— the same error Arrow's `rem` already produces for integers today, so
the
message stays uniform across types. The check is masked by the validity
of
the dividend because Spark's remainder expressions are null intolerant:
a
NULL dividend short-circuits to NULL before the divisor is validated, so
  `mod(NULL, 0)` must return NULL rather than raise.
- Both modes substitute NULL for zero divisors before calling Arrow's
`rem`,
so the kernel never sees a zero divisor: legacy mode gets NULLs, and
ANSI
  mode has already raised on the rows that required it.

Note on overlap with apache#23898: that PR rewrites `pmod` to no longer use
`try_rem` and adds identical `is_zero`/`negative_zero` helpers. This PR
is
independent of it — `mod` is fixed either way — but whichever lands
second
should dedupe the helpers in a rebase. Until apache#23898 lands, `pmod` also
picks
up the `-0.0` and ANSI floating-point zero-divisor fixes through the
shared
helper.

Out of scope: apache#23897 (reproducing Spark's exact ANSI error text) is a
repository-wide error-message policy question, as noted in that issue.

## Are these changes tested?

Yes:

- New unit tests in `modulus.rs`: ANSI floating-point zero divisor
raises;
`-0.0` divisor returns NULL in legacy mode and raises in ANSI mode; a
NULL
dividend with a zero divisor returns NULL in ANSI mode (integer and
float).
- New sqllogictest cases in `spark/math/mod.slt` covering the same
behavior at
  SQL level.
- Verified `cargo test -p datafusion-spark`, the `spark/math/mod.slt`
and
`spark/math/pmod.slt` sqllogictests, `./dev/rust_lint.sh`, and the
extended
  workspace suite (ci profile with

`avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption`)
  — all green.

## Are there any user-facing changes?

Only the bug fixes, and only for the Spark `mod`/`pmod` functions: in
ANSI
mode a floating-point zero divisor now raises instead of returning NaN,
and a
`-0.0` divisor is treated as zero in both modes (NULL in legacy mode, an
error
in ANSI mode), matching Spark. No API changes.
andygrove added a commit to andygrove/datafusion that referenced this pull request Sep 3, 2026
…o handling

Audit `pmod` against Apache Spark 4.2.0 and fix three divergences.

Spark evaluates `(r + n) % n` with Java arithmetic, which wraps around for
`int` and `long` and promotes `byte` and `short` to `int`. Arrow's checked
`add` reported an overflow instead, so `pmod(-1, -2147483648)` errored where
Spark returns `2147483647`. Use `add_wrapping`, and widen the narrow integer
types before the addition so Java's operand promotion is reproduced.

Arrow's `rem` only reports division by zero for integer and decimal types, so
in ANSI mode a floating point zero divisor quietly produced `NaN` where Spark
raises. Check the divisor directly, masked by the validity of the dividend so
the null intolerant short circuit is preserved and `pmod(NULL, 0)` stays NULL.

Arrow's comparison kernels order floating point values totally, so `-0.0` was
neither recognised as a zero divisor nor kept out of the negative branch. Treat
`-0.0` as zero when testing the divisor, and select the adjusted value only
where the remainder is negative so a `-0.0` result keeps its sign.

`pmod` no longer shares `try_rem` with `mod`, whose behavior is unchanged.

Rebase note: apache#23987 landed the same ANSI-zero-divisor and `-0.0` fixes for
`mod` while this branch was open, by rewriting the shared `try_rem` and adding
`negative_zero` and `is_zero`. This rebase keeps main's `try_rem` and its
helpers rather than the versions developed here, so `mod` is untouched; the
`is_zero` helper is identical either way. What remains specific to this branch
is the Java overflow and operand promotion handling, the `-0.0` result sign,
and the `pmod` test coverage. The two test suites are kept in full.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

spark sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] mod returns NaN instead of raising for a zero floating point divisor in ANSI mode, and ignores -0.0 divisors

4 participants