[SPARK-59686][SQL] Clamp the rounding scale in round and bround - #58939
Open
SEPURI-SAI-KRISHNA wants to merge 2 commits into
Open
SEPURI-SAI-KRISHNA wants to merge 2 commits into
SEPURI-SAI-KRISHNA wants to merge 2 commits into
Conversation
round and bround passed the user supplied scale straight to BigDecimal.setScale, which throws once the magnitude passes roughly 1e9. Neighbouring scales return the result fine, so a well defined answer became an error: round(1.5, -10000000) is 0.0 while round(1.5, -1000000000) raised java.lang.ArithmeticException. On the ANSI integral path the exception was wrapped by MathUtils.withOverflow and surfaced as a spurious ARITHMETIC_OVERFLOW, even though nothing overflowed and the result is representable. Everywhere else, including the integral types in non-ANSI mode, the raw java.lang.ArithmeticException reached the user. Clamp the scale where it is read. _scale is a single protected lazy val that every type branch uses and that codegen interpolates, so clamping there covers interpreted and generated code for all numeric types. The clamp cannot change a result: no finite value has more than 309 integral digits, and the exact decimal expansion of a finite double needs at most 1074 fractional digits. It also stops -_scale in dataType from overflowing for a scale of Int.MinValue.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
roundandbroundpass the user supplied scale straight toBigDecimal.setScale, which throws once the magnitude passes roughly 1e9. This PR clamps the scale to a bound past which it cannot change the result.The clamp goes on
RoundBase._scale, a singleprotected lazy valthat every type branch reads and that codegen interpolates as a literal, so one change covers interpreted and generated code for all numeric types.Why are the changes needed?
Neighbouring scales already return the answer, so today a well defined result becomes an error:
The answers are not in doubt. A scale far to the left of the decimal point rounds any finite value to zero, and a scale past the digits a value carries leaves it unchanged, which is exactly what the smaller scales return.
There are two symptoms with one cause:
MathUtils.withOverflowand surfaces asARITHMETIC_OVERFLOW. That is spurious: nothing overflowed and the result is representable.FLOAT,DOUBLE,DECIMALwith a negative scale, and the integral types in non-ANSI mode, the rawjava.lang.ArithmeticExceptionreaches the user. It is not aSparkThrowable, so it carries no error condition and no SQLSTATE.RoundBase.dataTypeis affected too: it computes-_scale + 1, and forInt.MinValuethat negation overflows, so aDECIMALinput gets a narrower result precision than intended. Clamping fixes that as well.Does this PR introduce any user-facing change?
Yes. Scales beyond the clamp bound now return the rounded value instead of raising. No input that previously produced a value changes, since the clamp is chosen so that it cannot alter a result.
No migration guide entry, following #58047, which likewise replaced a spurious error with the correct value. #57832 added one because it changed one error condition into another, which is not the case here.
How was this patch tested?
Added
round/bround with an extreme scaletoMathExpressionsSuite, covering both signs atInt.MinValue,Int.MaxValueand 1e9, acrossDOUBLE,FLOAT,LONGandINT, in both ANSI and non-ANSI mode.The clamp is only correct if it preserves results, so the test also pins the two values most sensitive to it:
Double.MinPositiveValue, the subnormal with the longest exact decimal expansion, andDouble.MaxValue. I checked separately that clamping is bit identical to the unclamped scale for both, on either sign.Reverting the clamp makes the new test fail, which is the behaviour being fixed.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)